Most people think system design is about building the “best” architecture. It’s not—instead, it’s about making the right tradeoffs for a given workload.
1. Performance, Latency & Throughput
Latency vs Throughput
Batching many requests together can reduce network, serialization & processing overhead and increase throughput. Yet the downside is individual requests wait longer while a batch forms. Use batching when throughput matters and individual requests when low latency matters.
Latency vs Tail Latency
A system can have excellent median latency while a tiny number of slow requests create poor p95 and/or p99 latency. Redundant/hedged requests can reduce these long tails, but they consume extra infrastructure capacity.
Compression vs CPU Usage
Compressing data reduces network bandwidth & storage requirements. Yet compression and decompression require CPU and add processing latency. Use compression when bandwidth/storage matters, and uncompressed data when low CPU usage matters.