SELENE lunar lab

Text Token Indexing

Language AI Language systems · Intermediate

Token Indexing with Vocabularies Token indexing using vocabularies in the context of machine learning for text is the process of mapping words, subwords, or characters in a text to unique numerical identifiers based… excerpt

Run in Colab

The notebook Open on its own page View source Download standalone .ipynb