/home/siddhi - artificial-intelligence

Software, Geekery and other miscellany

AI: Tokenisation

There is an interesting parallel between AI and written scripts.

Logographic scripts, like Chinese or Japanese Kanji, there is a separate 'glyph' to represent each word. If you can read the script, you can understand the meaning even if you don't know how to pronounce the word.

Phonetic scripts, like …