Latency Budget Allocation
A real-time inference pipeline is a chain of dependent operations. Each has a latency distribution. The end-to-end latency is not the sum of medians — it's a convolution of distributions, and tail behavior dominates.
When your SLA is p95 ≤ 12ms and you have 6 pipeline stages, allocating the latency budget is a non-trivial problem. Naive allocation (2ms per stage) ignores the variance structure. A single high-variance stage can blow the entire budget.
Formalization
We model each stage i as having latency distribution Lᵢ with known mean μᵢ and variance σᵢ². The pipeline latency is:
L_total = Σ Lᵢ + Σ queueing_delays
For a target p95 SLA, we derive per-stage budget allocations that minimize the probability of SLA violation, subject to the constraint that allocations sum to the total budget.
The Scheduling Policy
The optimal allocation assigns tighter budgets to high-variance stages (to reduce tail risk) and looser budgets to low-variance stages. In practice, we implement this via an admission control mechanism: stages that are predicted to exceed their budget shed load to a fallback (lower-quality but faster) path.
Results
Applied to Neural Fabric's inference pipeline: SLA violation rate reduced from 0.8% to 0.04% with no change in median latency.