Module 21: The Proving Grounds (Capstone Architectural Simulation)

PREREQUISITE STATEMENT: Read this module after completing all prior 20 modules. Having mastered code-level design, database internals, caching topologies, distributed routing, cloud infrastructure, observability operations, and architectural governance, you are ready to apply this knowledge to solve three complex enterprise system designs.

---

1. Introduction: The Capstone Challenge

Architectural competence is not measured by memorizing terminology or multiple-choice exams. It is proven by design execution. The Proving Grounds present three real-world enterprise scenarios. You must select one, model its end-to-end topology in the interactive sandbox editor, and draft a Software Design Specification (SDS) documenting your trade-offs, constraints, and failure recovery playbooks.

Your submission will be evaluated against five architectural metrics:

1. Correctness: Does the design satisfy all business rules and consistency requirements?
2. Resiliency: How does the system handle component failures, network splits, and resource saturation?
3. Scale Capability: Is the infrastructure partitioned horizontally to avoid single-point scaling limits?
4. Cost Efficiency (FinOps): Are resources budgeted appropriately, avoiding over-provisioned cluster configurations?
5. Design Clarity: Are service boundaries, data directions, and protocol handshakes clearly documented?

---

2. Capstone Project Choices

Select and complete one of the following three industrial scenarios:

---

Challenge 17.1: The Greenfield Ticket Multi-Processor

[ Client Traffic ] ---> [ Edge Rate Limiter ] ---> [ Ingestion Queue (Kafka) ] 
                                                          |
                                                    (Worker Pool)
                                                          |
                                                          v
                                            [ Redis Distributed Lock ]
                                                          |
                                                (Write to Postgres DB)

1. The Scenario & Scale

An entertainment platform is launching sales for a global stadium concert.

• Scale: 100,000+ requests per second are expected at the millisecond sales open.
• Inventory: The stadium has a capacity of exactly 80,000 seats.
• Constraint: Zero overallocations or double-bookings are permitted. The system must remain available to clients under peak load, degrade gracefully, and enforce strict correctness.

2. Architectural Blueprint Requirements

• Traffic Ingress: Place an Edge Rate Limiter to drop malicious DDoS and duplicate bots.
• Ingestion Queue Gate: Do not allow client connections to access the database directly. Route requests into an Ingestion Queue (e.g. Apache Kafka or AWS SQS) to buffer traffic, converting a high-concurrency write spike into a flat, predictable consumer load.
• Distributed Inventory Lock: Implement a Redis Distributed Lock (using the Redlock algorithm or optimistic concurrency locking via Lua scripts) to validate seat availability before executing database writes:
  • Lock key format: lock:seat:<seat_number>.
  • Set a short Time-To-Live (TTL) on the lock (e.g., 10 minutes) representing the user checkout safety window. If checkout is not completed, the lock expires automatically, returning the seat to inventory.
• Finalized Purchase Ledger: Use a relational database (PostgreSQL/Postgres with transaction isolation levels set to REPEATABLE READ or SERIALIZABLE) to store finalized checkout records and ledger balances, ensuring ACID compliance.

---

Challenge 17.2: The Brownfield Banking Ledger Decoupling

                      [ Strangler Routing Gateway ]
                               /         \
                 (Legacy Path)             (Decoupled Path)
                     /                             \
        [ Legacy Core Monolith ]            [ Account Microservice ]
                                                   |
                                            (Saga Orchestrator)
                                                   |
                                            [ Transaction Service ]

1. The Scenario & Scale

A commercial bank is modernizing its core banking ledger.

• Scale: The system processes 5,000 transactions per second.
• Constraints: The legacy monolith database must remain online during the migration. The new architecture must decouple the system into autonomous microservices (AccountService, TransactionService, AuditLedgerService) while guaranteeing 100% data consistency and zero lost operations.

2. Architectural Blueprint Requirements

• Strangler Fig Routing: Deploy a Routing Gateway in front of the banking core. Intercept incoming queries and route new account features to the decoupled services while leaving old savings operations on the legacy core.
• Orchestrated Saga Pattern: Implement a Saga Orchestrator to coordinate distributed funds transfers between decoupled services:
  1. Action 1: AccountService debits Account A (Pending state).
  2. Action 2: TransactionService credits Account B (Pending state).
  3. Action 3: AuditLedgerService logs the transfer.
  4. If all actions succeed, transition account balances to Committed.
  5. If Action 2 fails (e.g., Account B is closed), the orchestrator triggers Compensating Transactions (executes a credit back to Account A to restore state).
• Anti-Corruption Layer (ACL): Build an ACL to translate binary EBCDIC mainframe data structures from the legacy core into clean JSON/Protobuf domain models used by the new microservices.

---

Challenge 17.3: The Global Video Streaming Delivery Fabric

[ Upload Client ] ---> [ S3 Ingestion ] ---> [ Event Lambda ] ---> [ Transcoding Workers ]
                                                                          |
                                                                    (Output to S3)
                                                                          |
[ Stream Client ] <--- [ Edge CDN Cache ] <-------------------------------+

1. The Scenario & Scale

A media streaming service processes uploaded video files and streams them globally to millions of concurrent users.

• Scale: Petabytes of data, global distribution.
• Constraints: Videos must start playing in sub-second latency anywhere in the world, automatically adjusting to client bandwidth variations (Adaptive Bitrate Streaming).

2. Architectural Blueprint Requirements

• Ingestion & Notification pipeline: Uploads are written to an S3 Ingestion Bucket. File creation events trigger Serverless Lambdas that insert jobs into a Transcoding Worker queue.
• Multi-Bitrate Transcoding: Dynamic workers transcode raw video into HLS (HTTP Live Streaming) and MPEG-DASH formats at different resolutions (1080p, 720p, 480p), segmenting video into 2-second .ts chunks.
• Multi-Tier Edge CDN Caching: Route client streaming traffic through a Global Content Delivery Network (CDN) like CloudFront. Store video chunks at edge caches globally to bypass light-propagation fiber latency.
• Distributed Metadata persistent store: Store user profiles, play history, and catalog search data in a globally replicated database (such as DynamoDB Global Tables or CockroachDB) to maintain low read latencies.

---

3. Software Design Specification (SDS) Document Outline

To document your Capstone architecture, your Software Design Specification (SDS) must adhere to the following outline:

# Software Design Specification: [Capstone Title]

## 1. System Requirements & Boundary Constraints
*   Document the functional limits (e.g. maximum transactions/second) and constraints.
*   Define the target SLA goals (e.g. RTO, RPO, Latency percentiles).

## 2. Target System Architecture
*   Explain the role of each service, gateway, database, queue, and cache.
*   Document the communication protocols used (REST, gRPC, AMQP).

## 3. Distributed Transactions & Consistency Strategy
*   Detail how data consistency is maintained (e.g. Sagas, Redis locks, quorums).
*   Specify database isolation levels and write replication paths.

## 4. Failure Modes & Self-Healing Playbook
*   Analyze potential failure modes (network splits, crashed nodes, slow databases).
*   Define the circuit breaker thresholds and fallback behaviors.

---

4. Hands-on Architecture Challenge

Scenario Description

Refactor the starting sandbox layout to represent the complete, detailed end-to-end architecture topology for Challenge 17.1: The Greenfield Ticket Multi-Processor.

Your Goal:

1. Model the complete request flow:
   • Client $\rightarrow$ APIGateway (with rate limiter).
   • APIGateway $\rightarrow$ IngestionQueue (Kafka Topic).
   • IngestionQueue $\rightarrow$ WorkerPool (Multiple parallel consumers).
   • WorkerPool $\rightarrow$ RedisLockStore (Validating seat reservations).
   • WorkerPool $\rightarrow$ PostgresLedgerDB (Persisting finalized ticket purchases).
2. Use Mermaid's flowchart (graph TD or similar) syntax.
3. Ensure all database nodes, brokers, gateways, and worker boundaries are represented.

---

5. Practice Challenge Template

Use this template in your sandbox to model the Ticketing architecture:

graph TD
    Client[Client App] -->|1. Submit Purchase| APIGateway[Edge API Gateway]
    
    subgraph Gateway_Ingress [Gateway Boundary]
        APIGateway -->|2. Check Limit| RateLimiter[Token Bucket Limiter]
    end
    
    RateLimiter -->|3. Accept & Queue| KafkaBroker[Kafka Ingestion: ticket-sales-topic]
    
    subgraph Consumer_Processing [Worker Pool Boundary]
        KafkaBroker -.->|4. Pull Job| Worker1[Worker Node 1]
        KafkaBroker -.->|4. Pull Job| Worker2[Worker Node 2]
        
        Worker1 -->|5a. Acquire Lock| RedisLock[Redis Distributed Lock]
        Worker2 -->|5a. Acquire Lock| RedisLock
        
        Worker1 -->|5b. Write Ticket| RelationalDB[(PostgreSQL ACID DB)]
        Worker2 -->|5b. Write Ticket| RelationalDB
    end
    
    style Client fill:#9f9,stroke:#333,stroke-width:2px
    style APIGateway fill:#9ff,stroke:#333,stroke-width:3px
    style KafkaBroker fill:#9ff,stroke:#333,stroke-width:2px
    style Worker1 fill:#9f9,stroke:#333,stroke-width:2px
    style Worker2 fill:#9f9,stroke:#333,stroke-width:2px
    style RedisLock fill:#9ff,stroke:#333,stroke-width:2px
    style RelationalDB fill:#9f9,stroke:#333,stroke-width:3px

---

---

6. The Back-of-the-Envelope Estimation Framework

System design interviews and architecture proposals both require rapid, defensible estimates of scale. The back-of-the-envelope estimation is not guesswork — it is structured arithmetic built on a small set of universal constants. Engineers who can produce credible numbers in under five minutes signal a level of operational fluency that separates senior from principal candidates.

6.1 Universal Constants to Memorise

Constant	Value	Notes
Bytes in 1 KB	1,000 (or 1,024)	Use 1,000 for rough estimates
Bytes in 1 GB	10⁹ bytes	1 billion bytes
Bytes in 1 TB	10¹² bytes
Seconds per day	86,400	≈ 100,000 for quick math
Seconds per year	31,536,000	≈ 32 million
Average HTTP request size	1–10 KB	Varies by payload
Average DB row (user record)	1 KB	With indexes, ~2–5 KB
JPEG thumbnail (100×100 px)	~10 KB
1080p video per second	~5 MB (uncompressed) / ~0.5 MB (H.264)

6.2 QPS (Queries Per Second) Estimation

The formula for QPS derivation from user volume:

QPS = (Daily Active Users × Actions per User per Day) / 86,400 seconds

Worked example — Twitter-scale read system:

• Daily Active Users (DAU): 300 million
• Average timeline reads per user per day: 10
• Read QPS = (300,000,000 × 10) / 86,400 ≈ 34,700 QPS (call it 35,000 QPS)
• Peak QPS (assume 3× average during peak hours): ≈ 105,000 QPS

This immediately tells you the system needs horizontal read scaling — a single PostgreSQL primary can sustain roughly 10,000–15,000 simple reads/s, so you need read replicas or a caching layer (Redis) to absorb the read fan-out.

public class QpsEstimator
{
    public record QpsResult(double AverageQps, double PeakQps);

    /// <param name="dailyActiveUsers">Total DAU.</param>
    /// <param name="actionsPerUserPerDay">Average number of requests per user per day.</param>
    /// <param name="peakMultiplier">Peak-to-average ratio (typically 2x–5x).</param>
    public static QpsResult Estimate(
        long dailyActiveUsers,
        double actionsPerUserPerDay,
        double peakMultiplier = 3.0)
    {
        const double secondsPerDay = 86_400;
        double avgQps  = (dailyActiveUsers * actionsPerUserPerDay) / secondsPerDay;
        double peakQps = avgQps * peakMultiplier;
        return new QpsResult(Math.Round(avgQps, 0), Math.Round(peakQps, 0));
    }
}

// Output for Twitter-scale example:
// AverageQps = 34,722   PeakQps = 104,167

6.3 Storage Sizing Formulas

Storage estimation always follows the same three-step pattern: (1) row size × (2) rows per day × (3) retention period.

Worked example — URL shortener service:

• New URLs shortened per day: 100 million
• Average URL record size: 500 bytes (short key 7 bytes + long URL ~200 bytes + metadata + indexes)
• Daily storage: 100,000,000 × 500 bytes = 50 GB/day
• 5-year retention: 50 GB × 365 × 5 = 91.25 TB total storage

This drives the database technology choice. Relational databases struggle above ~5 TB without significant operational investment. At 91 TB, a key-value store (DynamoDB, Cassandra) with partitioning by short key hash is the natural fit.

public class StorageEstimator
{
    public record StorageResult(double DailyGb, double TotalTb);

    public static StorageResult Estimate(
        long writeOperationsPerDay,
        int averageRowSizeBytes,
        int retentionYears)
    {
        double dailyBytes = (double)writeOperationsPerDay * averageRowSizeBytes;
        double dailyGb    = dailyBytes / 1e9;
        double totalGb    = dailyGb * 365 * retentionYears;
        double totalTb    = totalGb / 1000;
        return new StorageResult(Math.Round(dailyGb, 2), Math.Round(totalTb, 2));
    }
}

// URL shortener: Estimate(100_000_000, 500, 5)
// → DailyGb = 50.0, TotalTb = 91.25

6.4 Bandwidth Calculations

Bandwidth is the product of QPS and average response size.

Formula:

Inbound bandwidth  = Write QPS × average write payload size
Outbound bandwidth = Read QPS × average response size

Worked example — image hosting service:

• Upload QPS: 1,000 images/s, average image size: 2 MB → Inbound: 2 GB/s
• Serve QPS: 50,000 reads/s, average thumbnail response: 15 KB → Outbound: 750 MB/s

The outbound bandwidth of 750 MB/s (≈ 6 Gbps) tells you immediately that a CDN is not optional — it is the only economically viable mechanism to absorb this read bandwidth without paying origin-server egress fees at scale.

These three estimations (QPS, storage, bandwidth) are the minimum you must complete in the first 5 minutes of any system design discussion before proposing architecture. Numbers constrain choices; without them, every architectural decision is an opinion.

---

7. Portfolio Architecture Documentation

Most senior engineers have built significant systems but cannot articulate their decisions in writing. A portfolio that documents why you made specific choices — not just what you built — is one of the strongest differentiators when applying for principal or staff engineer roles.

7.1 What to Document

Your portfolio entry for a system you built must answer five questions:

1. Context: What business problem did the system solve? What were the scale constraints?
2. Architecture Drivers: What non-functional requirements (latency, availability, consistency) constrained the design?
3. Decision Rationale: For each major decision (database choice, communication protocol, caching strategy), explain the trade-off that led to that choice.
4. What Went Wrong: Honest retrospective of incidents, scaling failures, or design decisions you would reverse.
5. What You Would Do Differently: Given what you know now, what would you change?

Question 4 and 5 are the most differentiating. Candidates who only describe successes are unconvincing. Candidates who can articulate failure modes and derive lessons from them demonstrate the kind of operational maturity that hiring managers at senior levels are evaluating.

7.2 The C4 Model Written Narrative

The C4 model (Context, Containers, Components, Code) is primarily a diagramming framework, but its most underused application is as a written narrative structure. Each level of the C4 model corresponds to a section of your portfolio documentation:

Level 1 — System Context (The Why):

Write 2–3 paragraphs describing the external actors (users, third-party systems) that interact with your system and what they need from it. Avoid technical jargon here. This section should be readable by a product manager.

Example: "The Order Processing System handles purchase requests from three million monthly active customers of a B2C retail platform. It interfaces with a third-party payment provider (Stripe) for card authorisation and a logistics partner API (DPD) for shipment booking. The primary non-functional requirement was 99.9% checkout availability, which translates to a maximum of 8.7 hours of downtime per year."

Level 2 — Container Diagram (The What):

Enumerate each deployable unit (service, database, message broker, CDN). For each container, write one paragraph explaining: its single responsibility, the technology choice and why, and its upstream/downstream dependencies.

// A C# record that models a C4 Container entry, suitable for
// generating portfolio documentation programmatically.
public record C4Container(
    string Name,
    string Technology,
    string Responsibility,
    string TechnologyRationale,
    string[] UpstreamDependencies,
    string[] DownstreamDependencies);

// Example: documenting the Order Service container.
var orderService = new C4Container(
    Name: "Order Service",
    Technology: "ASP.NET Core 8, C#",
    Responsibility: "Accepts purchase requests, validates inventory, " +
                    "publishes OrderPlaced events, and returns checkout confirmation.",
    TechnologyRationale: "ASP.NET Core was chosen over Node.js because " +
                          "the team's existing C# expertise reduced onboarding " +
                          "time, and the strong typing system enforced domain " +
                          "model correctness at compile time rather than runtime.",
    UpstreamDependencies: ["API Gateway", "Identity Service"],
    DownstreamDependencies: ["Order Database (PostgreSQL)", "Event Bus (Azure Service Bus)"]
);

Level 3 — Component Narrative (The How):

For the most architecturally significant container, describe its internal structure: the bounded context it implements, the primary classes/modules, and how the dependency rule is enforced (e.g., Domain layer has zero external references).

7.3 Quantifying Your Portfolio

Every portfolio statement should be accompanied by a number. Vague claims ("improved performance") are ignored. Quantified claims ("reduced P99 checkout latency from 820ms to 45ms by replacing synchronous HTTP fan-out with an event-driven outbox pattern") are remembered.

The DORA metrics from Module 15 are your friend here: if your migration increased deployment frequency from monthly to daily and reduced MTTR from 4 hours to 22 minutes, those two numbers tell the story of the modernisation more powerfully than three paragraphs of narrative.

---

8. Communicating Trade-offs Under Time Pressure

The system design interview is not a test of whether you know every technology. It is a test of how you think about constraints. The engineers interviewing you have spent years building systems; they can identify immediately when a candidate is pattern-matching to memorised solutions versus reasoning from first principles.

8.1 The Structured Decision Framework

When asked to choose between two approaches, resist the instinct to pick one immediately. Instead, use this four-step structure:

1. State the constraint that makes this a non-trivial choice. ("The tension here is between write throughput and read consistency.")
2. Enumerate the options you are considering. ("I see three viable approaches: synchronous replication, asynchronous replication with eventual consistency, or a synchronous primary with async replicas for reads.")
3. State the trade-off explicitly. ("Synchronous replication gives us linearizable reads but increases write latency by the round-trip time to the secondary. Async replication drops write latency but means reads from replicas may lag by 10–500ms depending on network conditions.")
4. Make a decision and cite the constraint that drives it. ("Given that the stated SLA is 99.9% checkout availability and writes are on the critical path, I would choose async replication with read-your-writes routing to the primary for post-write reads. We accept eventual consistency on read replicas for analytics queries.")

8.2 The STAR Format for Architecture Decisions

The STAR format (Situation, Task, Action, Result) was designed for behavioural interviews, but it is equally effective for communicating architecture decisions in a structured, time-bounded way. The architecture variant maps as follows:

STAR Component	Architecture Equivalent	Time Budget
Situation	Business context and scale constraints	30 seconds
Task	The specific architectural problem you needed to solve	20 seconds
Action	The decision you made and the trade-offs you evaluated	90 seconds
Result	Quantified outcome (latency, cost, availability, incident reduction)	30 seconds

Total: under 3 minutes. This is the format for delivering an architecture narrative in a panel interview, an ARB presentation, or an engineering all-hands.

8.3 Example: STAR-Formatted Architecture Decision

Situation: "We ran an e-commerce platform processing 8,000 orders per day. During marketing campaign events, the checkout API would time out because the synchronous billing service call added 700ms to every checkout request, and when the billing service degraded, checkout availability fell below 95%."

Task: "I needed to redesign the checkout flow to decouple availability from the billing service without losing any payment events or breaking PCI-DSS audit requirements."

Action: "I evaluated three options: circuit-breaking with retries (rejected: still blocks the user); client-side polling (rejected: poor UX, complex state management); and asynchronous event-driven checkout with the Transactional Outbox pattern. I chose the outbox pattern. The Order Service writes the order and the outbox event in a single local database transaction. A separate relay process reads unpublished events and delivers them to the Billing Service via a durable message bus. The checkout API response drops from 700ms to under 40ms because it only waits for the local database commit."

Result: "Checkout P99 latency dropped from 820ms to 38ms. Checkout availability during our next marketing campaign was 99.97%. The billing reconciliation lag was under 3 seconds at peak, which the finance team accepted as within SLA."

// Structural representation of a STAR decision record —
// useful for documenting decisions in a portfolio database.
public record StarDecisionRecord(
    string ServiceName,
    string Situation,
    string Task,
    string ActionTaken,
    string[] AlternativesConsidered,
    string[] AlternativeRejectionReasons,
    string QuantifiedResult,
    DateTime DecidedAtUtc);

// This record can be serialized to the ADR repository, linking
// the narrative directly to the corresponding ADR number.

8.4 What Interviewers Are Actually Evaluating

Senior and principal-level architecture interviews are not looking for a correct answer (most problems have multiple valid solutions). They are evaluating four signals:

1. Constraint identification: Did you ask about QPS, consistency requirements, and team capability before proposing a solution?
2. Trade-off articulation: Can you explain what you give up by choosing your approach?
3. Failure mode awareness: Did you consider what happens when components fail, not just when they succeed?
4. Communication clarity: Can a non-expert understand your explanation? Can an expert find nothing technically imprecise?

The MPC programme trains all four of these signals explicitly. The Capstone submission is your first opportunity to demonstrate them in a structured, evaluated context.

---

CONGRATULATIONS: Complete this Capstone challenge in your sandbox, finalize your Software Design Specification, and submit your work for evaluation. Upon successful validation, the system will issue your public, verified Macro Patterns Consortium Systems Architect Certificate, signaling your design competency to global recruitment partners.