**Tokenization** is the process of converting raw text into these [[🔎 Tokens|token]] units for the model to process. It works like this: 1. The model has a fixed "vocabulary" of tokens (e.g. 100,000 for models like GPT) 2. When you input text, the tokenizer algorithm splits it into chunks according to its vocabulary 3. Each token is converted to a numerical ID that the model can work with 4. The model processes these numerical IDs, not the actual text The tokenization process is designed to balance [[🔎 Efficiency|efficiency]] (using fewer tokens) with preserving meaning (not losing information). ## Why Tokenization Matters Understanding tokenization helps you: 1. **Optimize your [[prompts]]**: Since most LLMs have context window limits (like 4,096 or 8,192 tokens), knowing how text gets tokenized helps you fit more information within these constraints 2. **Manage costs**: API calls to models like GPT-4 are priced per token, so efficient prompting saves money 3. **Understand model limitations**: Some tokenization schemes handle certain languages better than others, which affects performance 4. **Predict behavior**: The way text is tokenized influences how the model "sees" and processes information