SELENE lunar lab

Text Tokenization

Language AI Text processing · Beginner

Cut text into the units a model will actually see.

Run in Colab

Part of this learner path:

  • Language models end to end

The notebook Open on its own page View source Download standalone .ipynb