Tokenization is the very first step. It is how we convert human-readable text into small, structured pieces (like syllables or words) that a computer program can understand.

Artificial intelligence models and neural networks cannot read raw letters. They only understand math and numbers. We tokenize text to chop it up into smaller pieces, and then we use a dictionary (called a [[#^vocab|vocabulary]]) to assign a unique number to every single piece.

We use a tool called a [[#^tokenizer|tokenizer]] (specifically, [[#^bpe|Byte Pair Encoding or BPE]]). It acts like a smart dictionary that breaks down rare words into smaller chunks and keeps common words whole.

• Description: A regular text string, exactly as a human would write or read it. This will be our master example sentence.

• Description: A simple list of numbers (a 1-dimensional [[#^tensor|tensor]]). Each number directly corresponds to a specific text piece from our vocabulary dictionary.

1. The tokenizer looks at the raw sentence.
2. It chops the sentence up based on spaces, punctuation, and common word fragments.
3. It looks up each fragment in the vocabulary and swaps it out for its ID number.

graph TD
    A[Raw Text: 'The quick, brown...'] -->|Chop it up| B['The', ' quick', ',', ' brown', ...]
    B -->|Look up in Dictionary| C[Token IDs: 464, 2068, 11, 7586...]