How does that fare for non-English queries? E.g. are two chars still not enough ... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

Kalabasa 9 months ago | parent | context | favorite | on: PruningRadixTrie – Faster Radix trie for prefix se...

How does that fare for non-English queries? E.g. are two chars still not enough for Chinese languages?

meepmorp 9 months ago [–]

I've seen 1.5 chars/token used as a rule of thumb for estimating token counts in Chinese text.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact