Surprising Economics of Load-Balanced Systems

# Summary In load-balanced systems modeled as M/M/c queuing systems, mean request latency decreases asymptotically toward the service time (one second) as the number of servers increases, even when load scales proportionally with server count. This counterintuitive result means larger systems achieve better latency at the same utilization rates, with percentile latencies improving at similar rates, making this one of the rare distributed systems problems that becomes easier at scale.