On this page 5 sections
A model card is a strange document. It is written by the people who most want you to use the model, and it is also the only technical description you will get. Reading one well means treating every number as a claim with conditions attached, then finding the conditions.
I’ll use a card for a fictional open-weight model, Alder-14B, as the running example. Its headline claims: a 128,000-token context window, a reasoning benchmark score of 81.2, “strong multilingual performance” and a permissive license.
Start with the setup, not the score
Every benchmark number depends on how it was produced. Before comparing Alder’s 81.2 to anything, find out:
- Shots. Was it zero-shot, or did the prompt include five worked examples? Five-shot scores are often several points higher.
- Sampling. Was the score the best of several attempts? “Pass at 5” and “pass at 1” are different products.
- Prompt format. Did they use a custom template you will not be using?
- Baselines. Are the comparison models current, and were they run with the same settings, or copied from older papers?
Alder’s card, read closely, reported 81.2 with eight examples and majority voting over five samples. Run once, zero-shot, the way most products call a model, our own number was 72.6. Neither figure is a lie. Only one describes your use.
Supported context versus useful context
“Supports 128k context” usually means the model accepts that many tokens without error, and perhaps finds a single planted sentence in a long document. That is a narrow test. We ran a simple multi-hop task: two facts, placed far apart, that had to be combined. Accuracy was 81% at 8,000 tokens, 74% at 32,000 and 54% at 100,000.
Your application decides which number matters. If you only ever need 8,000 tokens, the long-context claim is irrelevant. If you planned to send whole codebases, it is the most important line on the card.
Contamination and freshness
Ask whether the benchmarks could have leaked into training data. Good cards say how they checked. Many say nothing. A test set published three years ago and widely mirrored online is weak evidence, however high the score. Also check the knowledge cutoff, because it affects what the model will confidently get wrong.
The checklist
| Card says | Ask |
|---|---|
| Benchmark score X | Shots, sampling, prompt format, baseline versions? |
| Supports N tokens | Tested on what task, at what lengths? |
| Multilingual | Which languages, measured how, at what quality gap to English? |
| Safety evaluated | Against which categories, by whom, with what results? |
| Permissive license | Commercial use, user thresholds, output restrictions? |
| Runs on consumer hardware | At which quantization, how much memory, how fast? |
Turn claims into your own tests
The useful output of reading a card is not a verdict but a short test plan. For Alder, ours fit in a file:
{
"model": "alder-14b-q4",
"tests": [
{ "name": "reasoning_zero_shot", "cases": 150, "baseline": "current-prod" },
{ "name": "multihop_context", "lengths": [8000, 32000, 100000] },
{ "name": "spanish_support_replies", "cases": 80, "grader": "human" }
],
"license_review": "pending legal"
}It took two days to run. The model was a good fit for short Spanish support replies and a poor one for the long-document feature that had motivated the evaluation. The card had not been wrong about anything. It had just been answering a different question from ours.