Chapter 3.1 - Tokenization & Input Prompt

Tokenization is the very first step. Tokenization serves as the foundational mechanism to transform human-readable text into small, structured pieces (like syllables or words) that a computer program can understand.

Modern neural architectures and language models inherently cannot process raw letters. They only understand math and numbers. We tokenize text to chop it up into smaller pieces, and then we use a dictionary (called a [[#^vocab|vocabulary]]) to assign a unique number to every single piece.

The standard industry approach relies on a specialized utility known as a [[#^tokenizer|tokenizer]] (specifically, [[#^bpe|Byte Pair Encoding or BPE]]). It acts like a smart dictionary that breaks down rare words into smaller chunks and keeps common words whole.

• Description: A regular text string, exactly as a human would write or read it. This will be our master example sentence.

• Description: A simple list of numbers (a 1-dimensional [[#^tensor|tensor]]). Each number directly corresponds to a specific text piece from our vocabulary dictionary.

1. The tokenizer looks at the raw sentence.
2. It chops the sentence up based on spaces, punctuation, and common word fragments.
3. It looks up each fragment in the vocabulary and swaps it out for its ID number.

graph TD
    A[Raw Text: 'The quick, brown...'] -->|Chop it up| B['The', ' quick', ',', ' brown', ...]
    B -->|Look up in Dictionary| C[Token IDs: 464, 2068, 11, 7586...]