Module 14: Fault Tolerance & Self-Healing Infrastructure

PREREQUISITE STATEMENT: Read this module after completing Module 13 (Edge Gateways). While Edge API Gateways control incoming traffic rates, internal network splits and database locks will still occur. This module teaches you how to design self-healing backend codebases that survive downstream failures without propagating outages.

---

1. Introduction: The Cascading Failure Mechanism

In a monolithic architecture, if a database query slows down, the thread pool blocks but stays within a single system process. In a distributed microservice architecture, calling another service over the network is an unreliable operation. If Service C (e.g., a payment gateway) experiences a latency spike, the upstream Service B (e.g., an order service) blocks its worker threads while waiting for Service C to respond. In turn, Service A (e.g., the API Gateway) blocks its sockets waiting for Service B, causing a cascading failure that can crash the entire system:

[ Client ] ---> [ API Gateway (Blocked) ] ---> [ Order Service (Thread Exhausted) ] ---> [ Payment Service (Hangs) ]

To build a fault-tolerant system, you must design for failure. Your software must isolate dependencies, fail fast when downstream services are unhealthy, and degrade gracefully to preserve core functionality.

---

2. Resiliency Patterns

To prevent cascading system failures, architects rely on four primary resiliency patterns:

                  [ Distributed Resiliency Patterns ]
                                   |
         +-------------------+-----+-------------------+
         |                   |                         |
  [ Circuit Breaker ]   [ Bulkheads ]         [ Exponential Backoff ]
  - Fail fast early    - Isolate pools       - Back off retries
  - Protect downstream - Prevent contagion   - Add random jitter

A. The Circuit Breaker Pattern

Inspired by electrical circuit breakers, this pattern prevents a service from repeatedly calling a downstream dependency that is highly likely to fail. The circuit breaker operates as a state machine with three primary states:

stateDiagram-v2
    [*] --> Closed : System Normal
    Closed --> Open : Failure Rate > Threshold
    Open --> HalfOpen : Cooldown Period Expired
    HalfOpen --> Closed : Trial Requests Succeed
    HalfOpen --> Open : Trial Request Fails

1. Closed State: Normal operation. Requests flow through to the downstream service. The circuit breaker monitors success/failure rates over a rolling time window (e.g., 100 requests).
2. Open State: When the failure rate exceeds a configured threshold (e.g., 50% of requests fail), the breaker trips. Subsequent requests fail fast immediately, returning a fallback value or error response without calling the downstream service, saving network resources.
3. Half-Open State: After a cooldown period (e.g., 30 seconds), the breaker enters Half-Open. It permits a limited number of trial requests to pass. If they succeed, the breaker resets to Closed. If any fail, it trips back to Open, restarting the cooldown timer.

---

B. Bulkheads

Named after the partition walls in ship hulls that prevent a single hull breach from sinking the entire vessel, the Bulkhead pattern isolates resources into dedicated, bounded pools:

           [ Unisolated Architecture ]
           Shared Thread Pool ---> [ Calling Service A ]
                              ---> [ Calling Service B (Blocked / Saturated) ]
                              
           [ Bulkhead Isolated Architecture ]
           Thread Pool A (10 Threads) ---> [ Service A (Healthy) ]
           Thread Pool B (10 Threads) ---> [ Service B (Saturated / Blocked) ]

• Thread Pool Bulkheads: Assign a dedicated pool of worker threads to each downstream dependency. If Service B hangs, it can only saturate its own thread pool (e.g. 10 threads), leaving Thread Pool A fully available to service requests for Service A.
• Semaphore Bulkheads: Limit the maximum number of concurrent requests allowed to a service. If the limit is reached, incoming requests are rejected immediately, preventing resource saturation.

---

C. Retries with Exponential Backoff and Jitter

When a network call fails due to a transient blip, retrying immediately can overload the recovering server, causing a retry storm.

Architects implement Exponential Backoff to increase the wait time between retries, combined with Jitter (randomness) to prevent synchronized retry waves:

$$\text{Delay}{\text{exponential}} = \text{BaseDelay} \times 2^{\text{Attempt}}$$ $$\text{Delay}{\text{jitter}} = \text{Random}(0, \min(\text{MaxDelay}, \text{Delay}_{\text{exponential}}))$$

gantt
    title Retry Backoff Timeline with Decorrelated Jitter
    dateFormat  X
    axisFormat %s

    section Call Attempt
    Attempt 1 (Fail)   :a1, 0, 1
    Wait (Jitter 120ms):a2, 1, 2
    Attempt 2 (Fail)   :a3, 2, 3
    Wait (Jitter 450ms):a4, 3, 5
    Attempt 3 (Success):a5, 5, 6

---

3. C# Production Resiliency Implementation (Polly Engine)

Below is a complete C# production implementation utilizing the Polly fault-handling library. It constructs a combined resilience pipeline combining Timeout, Bulkhead Isolation, Circuit Breaker, and Exponential Backoff with Decorrelated Jitter.

using System;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using Polly;
using Polly.Bulkhead;
using Polly.CircuitBreaker;
using Polly.Retry;
using Polly.Wrap;

namespace MacroPatternsConsortium.Resiliency
{
    public class ResilientPaymentClient
    {
        private readonly HttpClient _httpClient;
        private readonly AsyncPolicyWrap<HttpResponseMessage> _resiliencePipeline;

        public ResilientPaymentClient(HttpClient httpClient)
        {
            _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));

            // 1. Timeout Policy: Limit max execution time per attempt to 2 seconds
            var timeoutPolicy = Policy
                .TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(2));

            // 2. Retry Policy: Retry 3 times with exponential backoff + decorrelated jitter
            var retryPolicy = Policy<HttpResponseMessage>
                .Handle<HttpRequestException>()
                .OrResult(r => (int)r.StatusCode >= 500)
                .WaitAndRetryAsync(
                    retryCount: 3,
                    sleepDurationProvider: (attempt, outcome, ctx) =>
                    {
                        var random = new Random();
                        var baseDelay = TimeSpan.FromMilliseconds(200 * Math.Pow(2, attempt));
                        var jitter = TimeSpan.FromMilliseconds(random.Next(0, 100));
                        return baseDelay + jitter;
                    },
                    onRetryAsync: (outcome, timespan, attempt, ctx) =>
                    {
                        Console.WriteLine($"[Retry] Attempt {attempt} failed. Waiting {timespan.TotalMilliseconds}ms.");
                        return Task.CompletedTask;
                    });

            // 3. Circuit Breaker Policy: Trip after 50% failures in a 30s window (Open for 15s)
            var circuitBreakerPolicy = Policy<HttpResponseMessage>
                .Handle<HttpRequestException>()
                .OrResult(r => (int)r.StatusCode >= 500)
                .AdvancedCircuitBreakerAsync(
                    failureThreshold: 0.5, // 50%
                    samplingDuration: TimeSpan.FromSeconds(30),
                    minimumThroughput: 10,
                    durationOfBreak: TimeSpan.FromSeconds(15),
                    onBreak: (outcome, timespan) => Console.WriteLine($"[CircuitBreaker] TRIPPED OPEN for {timespan.TotalSeconds}s!"),
                    onReset: () => Console.WriteLine("[CircuitBreaker] RESET to CLOSED."),
                    onHalfOpen: () => Console.WriteLine("[CircuitBreaker] ENTERED HALF-OPEN mode. Testing trial traffic."));

            // 4. Bulkhead Policy: Max 20 concurrent executions, 10 queued requests
            var bulkheadPolicy = Policy
                .BulkheadAsync<HttpResponseMessage>(
                    maxParallelization: 20,
                    maxQueuingActions: 10,
                    onBulkheadRejectedAsync: ctx =>
                    {
                        Console.WriteLine("[Bulkhead] Capacity exceeded! Request rejected fast.");
                        return Task.CompletedTask;
                    });

            // 5. Combine Policies into a unified Resilience Pipeline
            // Execution order: Bulkhead -> CircuitBreaker -> Retry -> Timeout -> Call
            _resiliencePipeline = Policy.WrapAsync(bulkheadPolicy, circuitBreakerPolicy, retryPolicy, timeoutPolicy);
        }

        public async Task<HttpResponseMessage> ProcessPaymentAsync(string payload, CancellationToken ct = default)
        {
            try
            {
                return await _resiliencePipeline.ExecuteAsync(async (cancellationToken) =>
                {
                    var request = new HttpRequestMessage(HttpMethod.Post, "https://payment-internal.vpc/v1/charge")
                    {
                        Content = new StringContent(payload, System.Text.Encoding.UTF8, "application/json")
                    };

                    return await _httpClient.SendAsync(request, cancellationToken);
                }, ct);
            }
            catch (BrokenCircuitException)
            {
                // Fallback Soft Failure
                return new HttpResponseMessage(System.Net.HttpStatusCode.ServiceUnavailable)
                {
                    Content = new StringContent("{\"error\":\"Payment gateway temporarily offline (Circuit Open). Try again shortly.\"}")
                };
            }
            catch (BulkheadRejectedException)
            {
                return new HttpResponseMessage(System.Net.HttpStatusCode.TooManyRequests)
                {
                    Content = new StringContent("{\"error\":\"System concurrency capacity reached. Request rejected fast.\"}")
                };
            }
        }
    }
}

---

4. Service Mesh Resiliency (Envoy Proxy Configuration)

While application-level libraries (like Polly) enforce code resilience, a Service Mesh (e.g., Istio with Envoy Sidecars) enforces fault tolerance transparently at the infrastructure network layer without altering business code:

graph LR
    subgraph Service_Pod_A [App Pod A]
        AppA[Order Service Code] -->|Local Loopback| EnvoyA[Envoy Sidecar Proxy]
    end

    subgraph Service_Pod_B [App Pod B]
        EnvoyB[Envoy Sidecar Proxy] -->|Local Loopback| AppB[Payment Service Code]
    end

    EnvoyA -->|mTLS + Circuit Breaker + Retries| EnvoyB

    style Service_Pod_A fill:#0f172a,stroke:#3b82f6
    style Service_Pod_B fill:#0f172a,stroke:#34d399

Declarative Envoy Outlier Detection & Retry Spec

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: payment-service-resiliency
  namespace: mpc-production
spec:
  host: payment-service.vpc
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 100
      http:
        http1MaxPendingRequests: 10
        maxRequestsPerConnection: 10
    outlierDetection:
      consecutive5xxErrors: 3
      interval: 10s
      baseEjectionTime: 30s
      maxEjectionPercent: 50
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payment-service-retry-route
  namespace: mpc-production
spec:
  hosts:
  - payment-service.vpc
  http:
  - route:
    - destination:
        host: payment-service.vpc
    timeout: 2.5s
    retries:
      attempts: 3
      perTryTimeout: 800ms
      retryOn: 5xx,connect-failure,refused-stream

---

5. Enterprise High-Availability & Disaster Recovery (HA/DR) Runbook

Architects must define quantitative Recovery Point Objectives (RPO) and Recovery Time Objectives (RTO) for corporate disaster recovery.

       [ Disruption Event ]
             |
   |<-- RPO -->|                |<------------ RTO ------------>|
   | Data Loss Window           | Time to Restore Operation     |

HA/DR SLA Matrix

SLA Level	Target RTO	Target RPO	Infrastructure Architecture	Annual Downtime Cap
Tier 0 (Mission Critical)	$< 30 \text{ seconds}$	$0 \text{ (Zero Data Loss)}$	Active-Active Multi-Region with TrueTime / Paxos Quorums	5.26 minutes (99.999%)
Tier 1 (Core Business)	$< 15 \text{ minutes}$	$< 5 \text{ seconds}$	Active-Passive Regional Failover with async CDC replication	52.6 minutes (99.99%)
Tier 2 (Internal Systems)	$< 4 \text{ hours}$	$< 1 \text{ hour}$	Cold Standby, Automated Terraform Infrastructure Re-creation	8.76 hours (99.9%)

---

6. Hands-on Architecture Challenge

Scenario Description

A client application (ClientApp) sends high-frequency checkout calls to OrderService. OrderService synchronously calls an unstable third-party payment endpoint (ExternalPaymentGateway). When the payment gateway experiences latency, OrderService thread pools saturate, causing global system outages.

Your Goal:

1. Wrap the call to ExternalPaymentGateway inside a Circuit Breaker and Bulkhead Isolation pattern.
2. Define the three circuit breaker transitions:
   • Closed: Requests pass normally.
   • Open: If > 50% fail, fail fast and return a fallback payment receipt (HTTP 202 Accepted - Pending Offline Processing).
   • Half-Open: After 15 seconds, permit 1 test request to verify dependency recovery.
3. Draw the circuit breaker routing state machine in the architecture diagram.

---

7. Practice Challenge Template

Use this template in your sandbox to model the resiliency topology:

graph TD
    subgraph Client_Layer [Client Layer]
        Client[Client Browser / Mobile] -->|1. Post Checkout| OrderService[Order Service]
    end

    subgraph Resilience_Boundary [Polly / Envoy Resilience Boundary]
        OrderService -->|2. Execute via Policy| Bulkhead[Bulkhead Isolation (20 Threads)]
        Bulkhead -->|3. Check State| CircuitBreaker{Circuit Breaker}
        
        CircuitBreaker -->|State: CLOSED| PaymentGateway[External Payment Gateway]
        CircuitBreaker -->|State: OPEN (Fail Fast)| FallbackHandler[Fallback Soft Handler]
    end

    FallbackHandler -->|4. Return HTTP 202 Pending| Client
    PaymentGateway -->|4. Return HTTP 200 Success| Client

    style Client_Layer fill:#0f172a,stroke:#3b82f6,stroke-width:2px
    style Resilience_Boundary fill:#1e1b4b,stroke:#6366f1,stroke-width:2px
    style CircuitBreaker fill:#374151,stroke:#f59e0b,stroke-width:2px
    style FallbackHandler fill:#064e3b,stroke:#10b981,stroke-width:2px

---

NEXT MODULE BRIDGE: Building self-healing codebases handles runtime operational failures, but legacy environments present existing brownfield debt. Proceed to Module 15: Environmental Assessment (Greenfield vs. Brownfield) to master the Strangler Fig Pattern and Anti-Corruption Layers.