Tokenization and context windows: budget the formatted conversation

A tokenizer converts text into the token IDs a model consumes. A context window limits the token sequence the model can process under its supported configuration. Words, characters and tokens are different units, so budget the actual formatted input with the tokenizer belonging to the model.

We will use the public SmolLM2-135M-Instruct tokenizer at revision 12fd25f77366fa6b3b4b768ec3050bf629380bac. The first run downloads tokenizer and configuration files; it does not download model weights or send the prompts to a hosted generation API. The tokenizer's outputs are specific to this revision.

Count the conversation the model actually receives

Chat models use control tokens and a conversation template to distinguish speakers and mark where generation begins. Counting only the user's visible sentence omits system text, role markers and other template content.

Our demonstration sets an application budget of 96 total tokens and reserves 32 for an answer. That is a deliberately small teaching budget, not a claim about the model's maximum supported context. The code verifies that the chosen budget does not exceed the pinned model configuration.

Input tokens plus the maximum requested output must fit the applicable total limit. A model or serving system may impose additional restrictions; check those separately when moving from this local example to an inference service.

Run the pinned tokenizer

Install transformers and its tokenizer dependencies in an isolated environment. The checked environment uses Transformers 5.20.2 and tokenizers 0.23.3. No access token is required for this public repository. Pin versions for reproducing the displayed counts.

from transformers import AutoTokenizer, AutoConfig
name = 'HuggingFaceTB/SmolLM2-135M-Instruct'
revision = '12fd25f77366fa6b3b4b768ec3050bf629380bac'
tokenizer = AutoTokenizer.from_pretrained(name, revision=revision, token=False)
config = AutoConfig.from_pretrained(name, revision=revision, token=False)
texts = ['machine learning', 'mechanical inspection', 'a+b=c', '机器学习']
for text in texts:
    ids = tokenizer.encode(text, add_special_tokens=False)
    print(repr(text), 'characters=' + str(len(text)), 'tokens=' + str(len(ids)))
messages = [{'role': 'system', 'content': 'Answer using the supplied notes.'},
            {'role': 'user', 'content': 'Explain why a held-out test is useful.'}]
visible = sum(len(tokenizer.encode(m['content'], add_special_tokens=False)) for m in messages)
formatted = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)
limit, reserve = 96, 32
available = limit-reserve-len(formatted)
print('visible content tokens:', visible)
print('formatted input tokens:', len(formatted))
print('remaining input budget:', available)
print('application budget within model config:', limit <= config.max_position_embeddings)
assert available >= 0
assert limit <= config.max_position_embeddings

Checked output:

'machine learning' characters=16 tokens=2
'mechanical inspection' characters=21 tokens=3
'a+b=c' characters=5 tokens=5
'机器学习' characters=4 tokens=9
visible content tokens: 16
formatted input tokens: 2
remaining input budget: 62
application budget within model config: True

Explain overhead and language differences

The formatted token count includes the template's role boundaries and generation prompt. Tokenizing a rendered template again with automatic special-token insertion can duplicate control tokens in some workflows. Prefer the tokenizer's documented chat-template path and inspect exactly what is passed into generation.

Token counts also vary with whitespace, punctuation, script and vocabulary. A fixed words-to-tokens multiplier is at best an estimate for a specific kind of text. The Chinese example is included to make that limitation visible, not to assess language quality.

A large supported context is a capacity limit, not a promise that the model will use every fact equally well. Evaluate whether it can find and apply the relevant evidence at the positions and lengths your application uses. Reducing irrelevant context can improve auditability even when the full input technically fits.

Trim evidence without destroying its meaning

When adding retrieved documents, reserve the output budget first, then account for the instructions, question and formatting. Choose evidence at sensible document or passage boundaries. Truncating blindly in the middle of a sentence can remove a qualification or detach an answer from its source.

Do not remove a system constraint merely to fit another document. Keep a clear priority policy and fail visibly when essential input cannot fit. A summarization stage is another model operation with its own possibility of omitted or changed facts, so evaluate it rather than assuming it is lossless compression.

Extend the budget check

Add a second user turn and count the full history again. Then add a retrieval passage whose token count exceeds the remaining budget and reject it before generation. Compare the result with a different tokenizer only after identifying the corresponding model and chat template.

The Generative AI syllabus covers tokenization and LLM workflows; lessons are being published. Continue with a local RAG pipeline to budget actual evidence and LLM evaluation datasets to test behavior beyond token counts.

References

SmolLM2 model repository — Public model, tokenizer files and configuration; this example pins a revision.

Transformers chat templates — Role formatting, generation prompts and special-token handling.