This note tells a debugging story from model serving. A replay system recorded production traffic and ran it again at the same throughput, yet reported a p99 latency only half of what production showed. Several hypotheses failed before profiling showed that CUDA kernels from different requests were interleaved, which stretches each request. A small worked example with fixed intervals and burst traffic makes the point. The fix was to replay traffic as a Poisson process instead of at fixed intervals, which brought the gap under five percent. The closing lessons argue that traffic distribution matters as much as throughput, and that queuing theory belongs in the toolbox of anyone tuning GPU serving.
Problem Summary
Context
- System: Model serving - deploying AI models to production
- Goal: Optimize latency and throughput on GPUs
- Latency: must be capped at a given level (a business requirement)
- Throughput: determines how many GPUs to buy (the higher, the more you save)
The mystery
- Dev system: Records production traffic and replays it at different throughputs to measure p99 latency
- Result: The replay system's p99 latency was only half of production's p99 latency at the same throughput
- Debugging attempts: Reviewed data and code and came up with many hypotheses, but none explained the difference
Finding the Cause
Observation from profiling
- When analyzing CUDA kernels, the kernels of different requests were found to be interleaved with one another
- Insight: A GPU can only run kernels sequentially
- When 2 requests are processed at the same time → each request's latency is stretched
- Conclusion: Traffic pattern matters a great deal; it doesn't depend on throughput alone
Worked example
Assume: 10 qps (queries per second), 100ms per request on average, 4 requests received in 400ms, and the GPU handles each request alone in 50ms
System 1 - Fixed intervals:
- A new request arrives at a steady 100ms interval
- The GPU can process each request in 50ms (the other 50ms is idle)
- Latency: 50ms
System 2 - Burst traffic:
- 4 requests arrive at the same time
- The GPU has to process 4 requests concurrently, with their kernels interleaved
- Latency: 50ms × 4 = 200ms
Solution
Queuing Theory
- Traffic in real systems typically follows a Poisson process with a Poisson distribution
- The replay system initially used fixed intervals → it did not reflect the real traffic pattern
Implementation
- Reimplemented the replay code to follow a Poisson distribution instead of fixed intervals
- Result: The gap between production and replay dropped to under 5%
Lessons
- Foundational math and statistics knowledge matters a lot in practice
- Traffic pattern strongly affects latency; it doesn't depend on throughput alone
- When benchmarking/replaying traffic, simulate the real traffic's distribution correctly
- Queuing theory is a useful tool for understanding and optimizing systems
References
- Queueing Theory
- Poisson Point Process
- Central Limit Theorem (to understand why real traffic follows a Poisson distribution)
See also: DoorDash Feature Store Redis Optimization · Monitoring Tracking