On this page 4 sections
When a chat feature feels slow, teams usually blame the model. Sometimes that is right. More often the time is spread across six steps nobody owns, and the model gets the blame because it is the only part with a name.
A latency budget fixes that. You decide in advance how long each step is allowed to take, write it down, and treat overruns as bugs.
What users actually notice
Three thresholds from decades of interface research still hold up well for chat:
- About 100 ms: feels instant. Use it for acknowledging input, like showing the sent message.
- About 1 second: the user’s train of thought survives. Something meaningful should be on screen by then.
- About 10 seconds: attention wanders. Anything longer needs progress information or the user leaves.
For chat, the number that matters most is time to first token: how long until words start appearing. Once text is streaming, users are surprisingly patient, because they are reading. People read roughly four to five words per second, so a model streaming 30 tokens per second is already well ahead of them.
A sample budget
Here is the budget one team used for a support assistant with retrieval. The target was a first token at 1.5 seconds on the 95th percentile, not the median.
| Step | p95 budget | Notes |
|---|---|---|
| Network to API | 80 ms | Regional endpoint |
| Auth and rate limiting | 40 ms | Cached session lookup |
| Input safety check | (120 ms) | Small classifier, runs in parallel, not on the critical path |
| Query rewrite | 250 ms | Small, fast model |
| Retrieval | 200 ms | Vector search plus rerank of top 20 |
| Model prefill to first token | 700 ms | About 5,000 input tokens |
| Rendering | 50 ms | First paint of streamed text |
| Critical path | 1,320 ms | 180 ms slack |
Because the safety check runs alongside the query rewrite and retrieval, its 120 ms never adds to the total. Small parallel steps like this are where the slack in a budget comes from.
{
"budget_ms": { "network": 80, "auth": 40, "rewrite": 250, "retrieval": 200, "ttft": 700, "render": 50 },
"percentile": 95,
"alert_when_over_for_minutes": 15
}What to show while they wait
Even a good budget leaves a second of waiting. That second can feel productive or broken, depending on the interface.
- Show real steps, briefly. “Searching 3 help articles” is honest and reassuring. A generic spinner says nothing.
- Never fake progress. A typing animation that starts before the model has produced anything trains users to distrust the animation.
- Stream the answer, not the reasoning. Long visible thinking may look busy, but users want the first useful sentence.
- Keep the input usable. Let people type the next question or cancel. A frozen box doubles the perceived wait.
Where the time usually hides
When teams first measure their pipeline end to end, the surprises repeat. A query rewrite step using the same large model as the main answer, adding a full second. Retrieval calls made one after another instead of together. A guardrail service in a different region. A cold start on a serverless function that runs once a minute.
None of these show up if you only time the model call. Trace every request from the moment the user presses enter to the first painted character, break the trace into the steps in your budget, and give each step an owner. The model will still be the biggest line, but it will no longer be the only suspect.