Most people think system design is about building the “best” architecture. It’s not—instead, it’s about making the right tradeoffs for a given workload and scale.
1. Performance, Latency & Throughput
// WORKLOAD EFFICIENCY Latency vs Throughput
Batching many requests together can reduce network, serialization & processing overhead and increase throughput. However, individual requests must wait longer in the batch buffer before processing begins, degrading single-request latency. Use batching when throughput matters and individual requests when low latency matters.
// OUTLIER MITIGATION Median Latency vs Tail Latency (p99)
A system can have a stellar median (p50) response time while outlier requests suffer from high tail latency (p99) due to GC pauses or lock contention. Hedged requests suppress tails but consume extra capacity.
// RESOURCE UTILIZATION Compression vs CPU Usage
Compressing payloads reduces network bandwidth and storage footprints. Yet, compression and decompression algorithms consume valuable CPU cycles and add processing latency.
Choose compression for network-bound and storage-heavy services, and raw payloads when CPU efficiency is paramount.
// PROCESSING MODELS Batch Processing vs Stream Processing
Batch processing optimizes bulk resource utilization with scheduled delays. Stream processing enables real-time insights as events arrive, requiring continuous resource allocation and higher operational costs.
// TASK EXECUTION Concurrency vs Parallelism
Concurrency structures multiple tasks to make progress interleaved over time (ideal for I/O-bound flows). Parallelism executes multiple tasks simultaneously across multi-core CPUs (ideal for CPU-bound computations).
// ECONOMIC BALANCE Performance vs Cost
Achieving ultra-low latency and massive throughput requires expensive over-provisioned infrastructure, whereas cost-optimized architectures accept slightly reduced performance metrics.
// MEMORY VS IO In-Memory vs Disk-Based Storage
In-memory storage (RAM) provides microsecond access speeds and massive IOPS, but is volatile, expensive, and limited by physical memory capacities. Disk-based storage is cost-effective, persistent, and scales massively, but suffers from high seek and I/O latency penalties.
// HARDWARE ACCELERATION CPU vs GPU/TPU Processing
CPUs excel at complex, sequential branching logic and general application workflows. GPUs and TPUs provide massive parallel throughput ideal for matrix math, machine learning, and cryptography, but introduce high host-to-device data transfer overhead.
// THREADING MODELS Thread-Per-Request vs Event-Loop (Non-Blocking)
Thread-per-request models simplify imperative code flow, but hit hard concurrency limits due to thread stack memory consumption. Event loops handle millions of concurrent connections efficiently via non-blocking I/O, but risk application freezes if blocking operations occur on the event thread.
// RESOURCE SHARING Shared-Memory vs Shared-Nothing Architecture
Shared-memory systems allow fast data visibility across cores or nodes, requiring complex synchronization locks. Shared-nothing architectures isolate resources completely to achieve effortless horizontal scaling, shifting communication entirely to message passing.
// RESOURCE CONTROLS Rate Limiting vs Throttling
Rate limiting strictly rejects excess requests with HTTP 429 errors to protect upstream servers. Throttling queues and slows down traffic processing gracefully to absorb surges without immediately failing user requests.
// ASSET DELIVERY Pre-computation (Caching) vs On-the-Fly Computation
Pre-computing results delivers instant sub-millisecond response times, requiring massive storage space and complex cache invalidation pipelines. On-the-fly computation saves storage and handles dynamic queries seamlessly, but spikes CPU usage and increases request latency.
2. Data Management & Consistency
// PARTITION TOLERANCE Consistency vs Availability (CAP Theorem)
During a network partition, choose between Consistency (CP) (rejecting writes to prevent split-brain) or Availability (AP) (accepting diverged writes and reconciling later).
// REPLICATION LAG Strong Consistency vs Eventual Consistency
Strong consistency guarantees up-to-date reads at the cost of higher latency. Eventual consistency provides high read performance via asynchronous replication with short windows of stale data.
// DATA INTEGRITY ACID vs BASE Guarantees
ACID transactions ensure absolute atomicity within single instances, whereas BASE architecture trades strict isolation for horizontal scalability and availability.
// SCHEMA ENGINE SQL (Relational) vs NoSQL (Non-Relational)
Relational databases enforce strict schemas and relational integrity for complex joins. NoSQL engines offer flexible documents and effortless horizontal sharding at the cost of complex multi-table queries.
// QUERYING VS DUPLICATION Normalization vs Denormalization
Normalization prevents data anomalies and minimizes storage footprint, requiring expensive runtime JOINs. Denormalization duplicates data to speed up reads, shifting complexity to write-time sync logic.
// INDEX OVERHEAD Indexes vs Write Performance
Adding indexes speeds up read searches significantly, but every insert/update requires modifying index trees, directly degrading write throughput and space efficiency.
// CACHE VALIDATION Cache-Aside vs Write-Through Caching
Cache-Aside avoids caching cold data by loading lazily via the app layer (at the cost of cache-miss latency). Write-Through keeps cache completely fresh by writing synchronously to cache and DB, increasing write latency.
// STORAGE MEDIA Block vs Object vs File Storage
Block storage provides raw high-performance disk volumes for databases. Object storage offers massive, cheap flat-namespace scalability for unstructured blobs. File storage provides hierarchical NAS shared access.
// WRITE ACCELERATION Write-Back (Write-Behind) vs Write-Through Caching
Write-Back caching buffers writes in memory for high write throughput and low latency, risking data loss if the cache node crashes before flushing to disk. Write-Through guarantees persistence by updating storage immediately, trading write speed for safety.
// SEARCH ENGINE ARCHITECTURE B-Trees vs LSM-Trees (Log-Structured Merge-Trees)
B-Trees optimize random read performance and point lookups, but suffer from high write amplification and random disk fragmentation. LSM-Trees optimize sequential write throughput via append-only logs, requiring background compaction phases that can steal CPU and I/O resources.
// TRANSACTIONAL COORDINATION Two-Phase Commit (2PC) vs Saga Pattern
Two-Phase Commit (2PC) enforces strict distributed ACID transaction atomicity, but creates a severe availability bottleneck and lock contention across microservices. Sagas execute local transactions sequentially with compensating rollback actions, enabling high availability while exposing the system to temporary data inconsistency windows.
// EVENT SOURCING Event Sourcing vs State-Based Persistence
Event sourcing stores every immutable state change as an event log, providing absolute audit trails and time-travel debugging, but requires complex snapshotting and projection rebuilding to read current state. State-based persistence stores only the current snapshot for lightning-fast reads, discarding historical transitions.
// DATABASE QUERY PATTERNS OLTP (Transactional) vs OLAP (Analytical) Databases
OLTP databases (row-oriented) are optimized for fast low-latency inserts, updates, and small point lookups. OLAP databases (column-oriented) compress and scan billions of rows efficiently for heavy aggregations, but perform poorly on individual row writes and updates.
3. Scaling & Architecture Patterns
// TOPOLOGY CHOICE Monolith vs Microservices
Monoliths offer zero network overhead and simple deployments, but become brittle at scale. Microservices provide independent scaling and fault isolation, introducing distributed system and network complexity.
// HARDWARE CAPACITY Vertical Scaling (Scale-Up) vs Horizontal Scaling (Scale-Out)
Vertical scaling adds machine CPU/RAM with zero partitioning complexity, but hits physical hardware ceilings and single-point-of-failure risks. Horizontal scaling adds commodity servers for infinite elasticity, requiring distributed load balancing and state management.
// SERVER MEMORY MODEL Stateless vs Stateful Architecture
Stateless services allow seamless auto-scaling and failover by storing sessions externally. Stateful designs keep context locally for ultra-low latency, making server migration and replacement difficult.
// DATA PARTITIONING Range-Based vs Hash-Based Sharding
Range sharding groups sequential keys for efficient range queries but causes write hotspots. Hash sharding distributes load uniformly across shards, destroying range query performance.
// COMMUNICATION STYLES Synchronous (REST/gRPC) vs Asynchronous (Kafka/MQ)
Synchronous calls provide immediate feedback and simple tracing, coupling services tightly. Asynchronous event streams decouple producers and buffer traffic spikes, adding debugging and out-of-order handling complexity.
// REAL-TIME PROTOCOLS Long Polling vs WebSockets vs Server-Sent Events (SSE)
Long polling works universally behind proxies with higher HTTP overhead. WebSockets provide full-duplex binary streams for low latency (chat/gaming). SSE provides lightweight server-to-client event streaming.
// API CONTRACTS REST vs GraphQL vs gRPC
REST offers standard resource routing and HTTP caching. GraphQL lets clients fetch exact shapes and avoid over-fetching at the cost of complex query parsing and caching hurdles. gRPC provides high-speed binary serialization over HTTP/2 for internal microservices.
// EVENT DELIVERY Push vs Pull Architecture
Push models deliver updates immediately with low latency, risking client overload during traffic surges. Pull models let consumers fetch at their own stable rate, introducing polling chatter and response latency.
// TRAFFIC DISPATCH Client-Side vs Server-Side Load Balancing
Client-side load balancing embeds discovery logic in clients to remove proxy hops, requiring complex multi-language client libraries. Server-side load balancing uses transparent proxies (Nginx/ALB), adding an extra network hop.
// EDGE CACHING CDN vs Origin Server Delivery
CDNs cache static assets globally near users to drastically reduce edge latency and shield origin servers, requiring cache invalidation strategies for dynamic data.
// DEPLOYMENT TOPOLOGY Serverless (FaaS) vs Containerized (ECS/EKS) Deployment
Serverless architectures eliminate server management and scale to zero automatically with pay-per-execution economics, but suffer from cold-start latency penalties and vendor lock-in. Containerized deployments provide constant warm capacity, predictable execution environments, and multi-cloud portability, requiring manual cluster tuning and continuous patching.
// SERVICE MESH API Gateway vs Service Mesh Architecture
API gateways manage North-South external traffic, routing, auth, and rate-limiting at the perimeter. Service meshes (Istio/Linkerd) manage East-West internal microservice traffic via sidecar proxies, providing mutual TLS and telemetry while adding memory footprint and proxy latency.
// RESILIENCE PATTERNS Circuit Breakers vs Retries with Backoff
Circuit breakers fail fast when downstream services are unhealthy, preventing cascading outages across systems. Retries with exponential backoff and jitter handle transient network glitches successfully, but can accidentally overwhelm recovering services if uncoordinated.
// AVAILABILITY ZONES Active-Active Multi-Region vs Active-Passive Disaster Recovery
Active-active multi-region setups provide zero-downtime failover and low global latency, requiring complex cross-region data replication and conflict resolution. Active-passive DR setups simplify data consistency by keeping secondary sites idle, resulting in downtime and recovery time objectives (RTO/RPO) during failover events.
// DATA ENCRYPTION Encryption-in-Transit vs Encryption-at-Rest
Encryption-in-transit (TLS) secures data packets moving across networks against interception, consuming CPU cycles for handshake crypto operations. Encryption-at-rest protects stored database volumes and S3 buckets against physical theft, requiring secure key management service (KMS) infrastructure and key rotation policies.
// IDENTITY & ACCESS JWT (Stateless Tokens) vs Opaque Session IDs (Stateful)
JWTs enable decentralized, stateless authentication verified via cryptographic signatures, making instant token revocation extremely difficult before expiration. Opaque session IDs look up user state in a central Redis cache on every request, allowing instant revocation and session termination at the cost of database/cache lookup latency.
// OBSERVABILITY PIPELINES Metrics & Traces Sampling vs Full Observability Ingestion
Sampling traces and logs (e.g., keeping 1% of traffic) slashes storage and telemetry infrastructure costs dramatically, but risks missing rare, sporadic production error traces. Full 100% telemetry ingestion ensures complete debugging visibility, running up massive storage and network bandwidth bills.
// IDEMPOTENCY DESIGN Idempotent APIs vs Non-Idempotent Operations
Idempotent APIs ensure duplicate requests (due to retries) produce identical outcomes without side effects, requiring deduplication tables and unique request ID tracking. Non-idempotent operations simplify code logic, but risk double-charging customers or duplicating records when network timeouts trigger retries.