Core concepts
Tokenizer
Definition
A tokenizer is the component that splits text into tokens before a model processes it. Different model families use different tokenizers, which is why the same text can have different token counts on GPT versus Claude.
Tokenizers are trained on large text corpora to find the most efficient set of subword fragments. The dominant approach is byte-pair encoding (BPE), which starts from individual characters and repeatedly merges the most frequent adjacent pair until it reaches a target vocabulary size — typically 50,000 to 200,000 tokens.
Because each model family trains its own tokenizer, counts differ between providers. This matters when you are comparing costs: the same prompt may be 1,000 tokens on one model and 1,150 on another, which changes the real price per request even if the headline rate looks identical.
Tokenizer efficiency also varies dramatically by language. English is heavily represented in training data and tokenizes efficiently. Languages with non-Latin scripts, and code with unusual whitespace, can consume several times more tokens for the same information.
Related terms
Token
A token is the unit of text an AI model reads and writes — roughly four characters or three-quarters of a word in English. Models price by the token and cap context by the token, so token count determines both what you pay and what fits.
Context window
The context window is the maximum number of tokens a model can consider at once — your prompt, any attached documents, the conversation history, and the response it generates. Exceed it and the earliest content gets dropped.
Inference cost
Inference cost is what you pay to run a model on a request, almost always billed per million input and output tokens. Output tokens typically cost three to five times more than input tokens.
Prompt
A prompt is the input you give an AI model to produce an output. Effective prompts specify five things: the role the model should adopt, relevant context, the specific task, the output format, and any constraints.
Prompt engineering
Prompt engineering is the practice of designing model inputs that reliably produce the output you want. It combines clear instruction-writing, structured formatting, worked examples, and systematic testing.
System prompt
A system prompt is a standing instruction that shapes a model's behaviour across an entire conversation, separate from individual user messages. It typically defines persona, rules, tone and output format.
Put this into practice
Understanding the term is step one. Our free courses and tools let you actually use it.