When a candidate replies to a screening question, every millisecond of silence costs engagement. Six months ago our p95 response time was 3.8 seconds. Today it is 1.1 seconds. This post covers the three changes that got us there.
1. Streaming everywhere
We moved from request-response to token streaming across the whole stack, including the WhatsApp bridge. The first token now reaches the messaging provider before the model has finished thinking about the last one.
2. Speculative prefetching
While a candidate is typing, we speculatively evaluate the most likely next workflow states. When the message lands, the evaluation for the winning branch is usually already warm — the typing_started webhook gives us a two-second head start on average.
3. A cache that understands conversations
Classic response caches are useless when every conversation is unique. Instead we cache at the semantic layer: criteria evaluations, entity extractions, and template renders are keyed by meaning, not by string.
The fastest inference is the one you never run.
Combined, these changes cut p95 latency by 70% while reducing inference spend by a third. Next up: applying the same speculative machinery to interview scheduling.
