Engineering Notes

System Design Bascis - Architecture, Trade-offs

Properties1
tagssystem-design

8 min read

Core Philosophy

System design is not about memorizing architecture diagrams. It is about making the right trade-offs under real-world constraints.

When learning system design, it is tempting to start with:

“Should I use Kafka, Redis, Kubernetes, microservices, or sharding?”

A better starting point is always:

“What problem am I actually trying to solve?”

The Five Key Dimensions of System Design

graph TD
    SD[System Design Dimensions] --> Scalability
    SD --> Throughput
    SD --> Availability
    SD --> Resilience
    SD --> Performance

Scalability: Can we grow?

A system that works smoothly for 10K users may collapse at 10M. Scalability is your architectural capability to handle increasing load by adding resources.

Key Considerations

  • Horizontal scaling
  • Load balancing
  • Stateless services
  • Caching
  • Data partitioning
  • Autoscaling
  • Capacity planning

The defining question: What becomes the bottleneck as demand grows?

Throughput: How much can we process?

Evaluating system capacity under load requires clear performance metrics:

  • RPS / TPS: Requests or transactions per second.
  • Concurrency: Simultaneous active requests.
  • Resource Utilization: CPU, memory, network I/O bounds.
  • Optimization Levers: Caching, message queues, asynchronous processing, and batching. Always identify the 1.1 Bottleneck before optimizing.

Optimization Levers

Caching, queues, asynchronous processing, batching, and additional workers can improve throughput. Always find the bottleneck first.

Availability: Can users access the system when needed?

For production systems, downtime carries a heavy business cost.

  • The 99.99% SLA Rule: 99.99% availability permits roughly 52 minutes of total downtime per year.

    • 525,600 mins a year times .0001 = 52.56 minutes

  • Core Mechanisms: Redundancy, data replication, automated failover, and eliminating single points of failure (SPOFs).

Critical Definitions

  • RTO (Recovery Time Objective): How quickly must we recover?
  • RPO (Recovery Point Objective): How much data can we afford to lose?

Resilience: What happens when things fail?

Networks fail, dependencies time out, databases go down, and bad deployments happen. Resilient systems assume failure from day one.

Resilience Patterns

  • Timeouts & retries with exponential backoff
  • Circuit breakers
  • Failure isolation & bulkhead patterns
  • Graceful degradation
  • Comprehensive observability

Performance: How fast is the experience?

A system can be scalable and highly available while still feeling sluggish to the end user. Focus on user-perceived latency over simple averages.

Metrics

  • P95 / P99 latency
  • Cache hit ratio
  • Database performance & indexing
  • CDNs and read replicas
  • Efficient data access paths

Production Case Study: The Exam Result Day Server Spike

Consider a massive public exam result portal where millions of students hit the web server simultaneously at 10:00 AM.

  • The Failure Pattern: The backend database connection pool is hardcoded or statically sized for 500 concurrent connections, but 500,000 requests arrive instantly.

  • The Cascading Impact:

    • Because the system is not scalable, it cannot distribute load across nodes.

    • Because workers are maxed out, throughput collapses as requests queue up or time out.

    • Because queues grow, response performance degrades from 200ms to 45 seconds.

    • Eventually, the OS exhausts file descriptors, memory overflows, processes crash, and the system becomes completely unavailable.

The Core Insight

System design dimensions are deeply intertwined. A bottleneck in Scalability directly triggers a drop in Throughput, spikes Latency (Performance), exhausts connection pools leading to Unavailability, and causes cascading timeouts (Resilience failures).

Clarifying the Relationships: Are They Proportional?

1. Scalability vs. Availability & Load

  • Is a non-scalable system always unavailable? Not always, but it has a hard ceiling. If your fixed-size server can handle 1,000 RPS, request 1,001 will either queue or fail. Under sustained high load, lack of scalability guarantees unavailability. Scalability is your ability to dynamically add resources (scale out) to prevent that ceiling from causing an outage.

2. Throughput vs. Performance

  • Does high throughput mean high performance? Not necessarily.
    • Throughput is volume (how many total SQL queries or API responses your cluster pushes per second).
    • Performance is speed/latency (how long a single request takes).
    • Production Catch: You can artificially drive up throughput by batching heavy database writes together, but individual user latency might actually get worse because requests sit in a batch buffer longer. Conversely, a system can have blazing-fast performance (1ms response time) but low throughput if it only has one worker thread.

3. Scalability vs. Resiliency

  • Does lack of scalability mean lack of resiliency? Yes, under high load scenarios. Resiliency assumes that components fail (a database node dies, a network switch drops packets, an upstream payment gateway times out). Scalability handles volume growth. However, if your system isn’t scalable, a traffic spike acts like a systemic failure—overwhelming resources until the application falls over, demonstrating zero resiliency to load.

Production-Level System Definitions & Contrast

DimensionProduction MeaningReal-World Failure Example
ScalabilityThe architectural capability to maintain performance by adding hardware resources (CPU, RAM, nodes) as load increases.An e-commerce checkout service crashes on Black Friday because its database connection pool is hardcoded to max 50 connections.
ThroughputThe rate of successful data or request processing over time (e.g., 50,000 Order Transactions Per Second).Payment gateway queues back up because worker threads spend 80% of their time waiting on an unindexed database table scan.
AvailabilityThe percentage of total time a service is operational and accessible ().A cloud region goes down, but multi-region DNS failover keeps the application online ( SLA).
ResilienceThe ability of a system to gracefully handle and recover from partial component failures without cascading crashes.A third-party shipping API hangs; the circuit breaker trips instantly, falling back to a cached shipping rate instead of locking up all server threads.
PerformanceThe time elapsed to execute a specific transaction (measured via P95/P99 latency metrics).A user dashboard takes 8 seconds to load because it executes 40 synchronous sequential SQL queries instead of parallelizing or caching.

Production Cheat Sheet for Interviews

The Production Mental Model

  1. Requirements First: What is our peak traffic (Throughput)? What is our acceptable delay (Performance)?
  2. Find the Bottleneck: Is it CPU-bound (needs Horizontal Scaling)? Is it Disk I/O-bound (needs Caching/Indexing)?
  3. Isolate Failures: If a downstream service dies, use Circuit Breakers and Retries (Resilience) so it doesn’t drag down the whole cluster (Availability).

The Matrix of Competing Trade-offs

Decision / PatternPrimary BenefitThe Trade-off / Cost
CachingImproves latency & reduces database loadIntroduces cache invalidation & consistency challenges
ReplicationImproves availability & read scalingIncreases storage cost & sync lag complexity
Async ProcessingImproves throughput & decouples systemsAdds operational complexity & eventual consistency
MicroservicesEnables independent scaling & team autonomyIntroduces distributed-system challenges (network partitions, tracing)

Architecture

Architectural Reality

There is no perfect architecture. There is only an architecture that makes the right trade-offs for a given set of requirements.

The Ultimate Mental Model

Understand requirements → Define metrics → Find bottlenecks → Choose patterns → Design for failure → Measure → Iterate

Good system design is not about knowing every technology. It is about knowing why you are choosing one.

Architectural Deep Dive Eg : Kafka vs. Redis

When choosing event backlines or caches, evaluate core access patterns and durability requirements.

DimensionApache KafkaRedis (Streams / Pub-Sub / Data Structures)
Primary NatureDistributed Commit Log / Event Streaming PlatformIn-memory Data Store & Message Broker
Data PersistenceDisk-backed & durable (retains data for days/weeks/indefinitely)In-memory (optional snapshotting/AOF, built for speed)
Consumption ModelConsumer Groups (pull-based, independent parallel reads)Pub/Sub (ephemeral) or Streams (consumer groups in RAM)
Throughput / LatencyMassive batch throughput, millisecond latencyUltra-low latency (microseconds), high RAM throughput
Primary Problem SolvedEvent decoupling, event sourcing, durable pipelinesReal-time caching, session stores, fast message queues, rate-limiting

How to Decide Using “What problem am I trying to solve?”

Instead of asking “Should I use Kafka?”, break down your problem using the dimensions:

Scenario A: Choose Kafka if your problem is…

  1. Durable Event Streaming & Event Sourcing: “I need every single event recorded immutably, and multiple downstream services (analytics, billing, search indexing) need to consume the exact same event stream independently at their own speed.”
  2. Decoupling High-Volume Producers and Consumers: “Our payment gateway generates 50,000 TPS during peak hours, but our fraud detection service can only process 5,000 TPS. I need a durable buffer that won’t drop messages or overwhelm RAM.”
  3. Replayability: “If our analytics service crashes for 6 hours, I need it to restart and replay yesterday’s event stream from offset X without data loss.”

Scenario B: Choose Redis if your problem is…

  1. Low-Latency Caching & Session Management: “I need to reduce database read load for user profile lookups with sub-millisecond response times, where stale data for a few seconds is acceptable or managed via TTLs.”
  2. Real-time Ephemeral Messaging / Pub-Sub: “I need to broadcast real-time chat messages or live driver locations to connected WebSocket clients where message persistence past a few seconds doesn’t matter.”
  3. Distributed Locking & Rate Limiting: “I need an atomic, ultra-fast in-memory counter to enforce API rate limits (e.g., max 100 requests/minute per user) or coordinate distributed locks.”

The Decision Rule of Thumb

  • If you need disk durability, long-term retention, and independent consumer replay groups Kafka.
  • If you need sub-millisecond latency, transient message routing, in-memory caching, or atomic counters Redis.
  • Architectural note: They are not mutually exclusive. Many modern architectures use Kafka as the source of truth / event backbone, and Redis as the downstream cache or real-time delivery layer for specific components.

Recommended Architecture Resources

  1. ByteByteGo (Alex Xu’s System Design Visuals): Highly polished, clean architecture diagrams breaking down real-world systems (like Twitter, Netflix, WhatsApp) with clear callouts on trade-offs.
  2. System Design Primer (GitHub - Donne Martin): The gold-standard open-source repository containing comprehensive text notes, study guides, and architecture flows.
  3. The Architecture Notes (Grokking the System Design Interview): Excellent visual breakdowns of component interactions (Load Balancers, CDNs, Consistent Hashing).

Graph View