Module 16: Governance, ADRs, & The Architecture Review Board (ARB)

PREREQUISITE STATEMENT: Read this module after completing Module 15 (Environmental Assessment). Selecting the technical patterns for Greenfield or Brownfield migrations is an engineering task; driving team alignment, managing cloud budgets, and documenting decisions for future engineers is a governance task. This module teaches you how to establish architectural standards within an enterprise organization.

---

1. Introduction: What is Architectural Governance?

In a small startup, system alignment is simple. A few engineers can agree on databases and libraries over a lunch conversation. In a large enterprise with hundreds of developers distributed across multiple autonomous teams, this informal alignment breaks down. Without structured governance, the system decays:

• Technology Sprawl: Team A uses Node.js/Postgres, Team B uses Java/Oracle, Team C uses Python/MongoDB, and Team D uses Rust/DynamoDB. The organization loses the ability to share code, move engineers between teams, or negotiate bulk licensing agreements.
• Compliance Breaches: Individual developers might inadvertently write data to non-compliant storage regions, violating laws like GDPR, HIPAA, or PCI-DSS.
• Technical Debt Accumulation: Short-term product deadlines lead to hacks that compromise core structural patterns (e.g., direct cross-database queries), creating tight coupling that blocks release cycles.
• FinOps / Cloud Cost Sprawl: Without cost boundaries, teams deploy expensive, over-provisioned cluster configurations, causing infrastructure billing to escalate.

Architectural governance is the process of aligning software engineering choices with corporate strategies, security policies, cost constraints, and long-term tech stack standards.

---

2. Architectural Decision Records (ADRs)

A major challenge in software engineering is understanding why a decision was made. Years after a system is built, a new engineer might look at an unusual routing structure or database configuration and assume it was a mistake, only to break the system by refactoring it because they did not understand the original constraints.

To prevent this, architects use Architectural Decision Records (ADRs), a concept popularized by Michael Nygard.

                  [ The Git-Based ADR Workflow ]
                  
   [ Code Change ] + [ ADR Markdown File in /docs/adr/ ]
          |
   [ Git Pull Request ] ---> [ Code Review & Governance Sign-off ]
          |
   [ Merge to Main ]   ---> [ Permanent Architectural Ledger ]

• Locality of Documentation: ADRs are not stored in external wikis or intranets (which quickly become out-of-date and are disconnected from code). Instead, ADRs are stored directly in the source control repository (typically under /docs/adr/) as lightweight Markdown files.
• The PR Lifecycle: Every major architectural change requires a corresponding ADR file. The ADR is committed in the same Git branch as the code changes, allowing the team to review the architectural rationale during the Pull Request code review process.
• The Permanent Ledger: Once merged to main, the directory serves as a chronological, searchable ledger documenting the evolution of the system architecture.

---

Production-Grade ADR Example: ADR-004-Event-Driven-Checkout.md

Below is an enterprise-grade ADR documenting the decision to migrate from synchronous HTTP to asynchronous event streaming for a checkout system:

# ADR-004: Asynchronous Checkout Event Streaming

## Status
Accepted

## Context
Our current e-commerce checkout flow uses synchronous HTTP POST calls between the Order Service, the Inventory Service, and the Billing Service. 
Under peak traffic events (such as marketing campaigns), the Billing Service experiences thread saturation, causing checkout requests to block and time out. 
This temporal coupling reduces checkout availability to the product of all three downstream service availabilities, violating our target P99 latency SLA of < 200ms.

We considered three alternatives:
1.  **Scaling up the Billing Service instances (Vertical/Horizontal):** High compute cost, and does not solve the availability coupling problem if the billing database experiences lock contention.
2.  **Implementing HTTP retry loops with jitter:** Increases client wait times, violating our latency SLA.
3.  **Migrating to Asynchronous Event Streaming (Chosen):** Decouples services temporally.

## Decision
We will replace the synchronous HTTP calls from the Order Service to the Billing and Inventory Services with an asynchronous event-driven model using Apache Kafka. 

*   The Order Service will commit checkout details to its local database and immediately publish an `OrderPlaced` event to a partitioned Kafka topic (`orders.v1`).
*   The Billing and Inventory services will run as independent consumers subscribing to the `orders.v1` topic, processing events asynchronously.
*   We will implement the Transactional Outbox pattern in the Order Service to guarantee at-least-once delivery without dual-write failure states.

## Consequences
*   **Positive:** The checkout API response time will decrease from > 800ms to < 50ms, as it only requires a local write and database transaction commit.
*   **Positive:** Downstream outages in the Billing Service will no longer block checkout queries; events will buffer in Kafka until the billing nodes recover.
*   **Negative:** Introduces eventual consistency. The client UI will receive an immediate checkout success status, but inventory reservation and payment processing will complete asynchronously. The frontend must handle payment failure notifications via WebSockets or email alerts.
*   **Negative:** Increased infrastructure complexity; requires maintaining a Kafka cluster and monitoring consumer group offsets.

---

3. The Architecture Review Board (ARB) & FinOps

A. The ARB Process

The Architecture Review Board (ARB) is a steering committee composed of senior architects, security officers, and engineering managers:

• The Goal: The ARB reviews proposed ADRs for major system alterations (e.g., adding a new database engine, altering authentication protocols, migrating to a new cloud provider).
• The Presentation: The proposing engineer submits the ADR to the board. The board evaluates the design against corporate standards, security policies, and total cost of ownership (TCO) constraints.
• The Resolution: The ARB either accepts the ADR, rejects it with specific feedback, or requests modifications, ensuring all system changes align with global engineering standards.

B. FinOps: Cost as an Architectural Metric

A modern architect must treat cloud infrastructure costs as a primary engineering constraint:

• Compute Costs: Choosing between Serverless (AWS Lambda, Google Cloud Run) and Container Orchestration (Kubernetes). Serverless is cheap for low, bursty workloads, but becomes highly expensive for constant, high-throughput systems where dedicated container nodes are more cost-effective.
• Storage Egress: Designing systems to avoid transferring raw datasets across geographic regions, which generates high network egress bills.
• The FinOps Ledger: Every architecture proposal must include a cost estimation showing the monthly infrastructure bill based on projected user scale.

---

4. Documentation Standard: Enterprise ARB Agenda

Below is a template for documenting an ARB Review Agenda & Architecture Decisions Log:

ARB Review Ledger

ADR ID	Proposal Title	Submitting Team	Sponsoring Architect	ARB Status	Long-term Consequences	FinOps Cost Impact
ADR-004	Asynchronous Checkout Event Streaming	Checkout Platform Team	Jane Doe (Principal Architect)	Accepted	Decouples checkout latency; introduces eventual consistency complexity.	+ $150/month (Kafka Cluster resources)
ADR-005	Adopt DynamoDB for User Session Cache	Identity Security Team	John Smith (Lead Architect)	Accepted	Enables fast session retrieval; requires strict TTL configuration.	- $80/month (Replaces over-provisioned ElastiCache nodes)
ADR-006	Shared Database for Inventory and Orders	Inventory Team	Bob Johnson (Senior Engineer)	Rejected	Violates database-per-service microservice boundaries; introduces tight schema coupling.	N/A

---

5. Hands-on Architecture Challenge

Scenario Description

You are designing the workflow governance for your engineering team. You must model the complete ADR Lifecycle Workflow from draft to final execution or deprecation.

Your Goal:

1. Model the ADR States using flowchart layout shapes:
   • Proposed (Drafting state).
   • Review (ARB review committee).
   • Accepted (Decision approved and active).
   • Rejected (Decision declined).
   • Superseded (Decision replaced by a newer ADR).
2. Define the Workflow Transitions:
   • From Proposed to Review.
   • From Review to Accepted (if approved).
   • From Review to Rejected (if declined).
   • From Rejected back to Proposed (if refactored for changes).
   • From Accepted to Superseded (when a new ADR overrides it).
3. Draw this governance lifecycle using the diagram editor's graph syntax.

---

6. Practice Challenge Template

Use this template in your sandbox to model the ADR governance lifecycle:

graph TD
    Proposed[Proposed / Draft State] -->|Submit for Review| Review[ARB Review Phase]
    
    Review -->|Approved by Committee| Accepted[Accepted / Active Decision]
    Review -->|Declined by Committee| Rejected[Rejected / Needs Revision]
    
    Rejected -->|Refactor & Resubmit| Proposed
    
    Accepted -->|Overridden by New ADR| Superseded[Superseded / Deprecated]
    
    style Proposed fill:#9ff,stroke:#333,stroke-width:2px
    style Review fill:#9ff,stroke:#333,stroke-width:2px
    style Accepted fill:#9f9,stroke:#333,stroke-width:3px
    style Rejected fill:#f99,stroke:#333,stroke-width:2px
    style Superseded fill:#ccc,stroke:#333,stroke-width:2px

---

---

7. Architecture Fitness Functions

An Architecture Fitness Function is an objective, automated test that verifies a specific structural property of a system. The concept was introduced in Building Evolutionary Architectures (Ford, Parsons, Kua) and is the evolution of governance from a manual review process into a CI-enforced constraint.

The critical insight is that fitness functions move governance left: instead of waiting for an ARB meeting to catch a cross-domain dependency violation, the build pipeline fails immediately when an engineer introduces one. This is the same principle behind type checking—you do not wait for a code review to discover a type mismatch; the compiler tells you instantly.

7.1 Categories of Fitness Functions

Category	What it Verifies	Execution Point
Structural	Assembly/namespace dependencies do not cross domain boundaries	CI pipeline, every PR
Cyclic	No circular dependencies exist between packages or services	CI pipeline, every PR
Stability	Abstractions in stable modules are not changed without versioning	CI gate on merge
Latency	P99 response times remain within SLA thresholds	Performance test suite, nightly
Security	No dependency on banned/vulnerable package versions	CI pipeline, every PR

7.2 Implementing Structural Fitness Functions in .NET

The .NET ecosystem does not ship a direct ArchUnit port, but NetArchTest (dotnet add package NetArchTest.Rules) provides a fluent API for expressing dependency constraints as xUnit or MSTest facts. These tests live in a dedicated Architecture.Tests project and run in CI alongside unit and integration tests.

// Architecture.Tests/LayeringTests.cs
using NetArchTest.Rules;
using Xunit;

public class LayeringFitnessTests
{
    private const string DomainNamespace      = "MyApp.Domain";
    private const string ApplicationNamespace = "MyApp.Application";
    private const string InfraNamespace       = "MyApp.Infrastructure";
    private const string ApiNamespace         = "MyApp.Api";

    /// <summary>
    /// Fitness function: Domain layer must not reference Application,
    /// Infrastructure, or Api layers. Domain entities must be pure.
    /// </summary>
    [Fact]
    public void Domain_MustNot_DependOnOuterLayers()
    {
        var result = Types.InCurrentDomain()
            .That().ResideInNamespace(DomainNamespace)
            .ShouldNot().HaveDependencyOnAny(
                ApplicationNamespace,
                InfraNamespace,
                ApiNamespace)
            .GetResult();

        Assert.True(result.IsSuccessful,
            "Domain layer has illegal outward dependency: " +
            string.Join(", ", result.FailingTypeNames ?? []));
    }

    /// <summary>
    /// Fitness function: Infrastructure layer must not be directly
    /// referenced by the Api layer — all access must go via Application.
    /// </summary>
    [Fact]
    public void Api_MustNot_ReferenceInfrastructureDirectly()
    {
        var result = Types.InCurrentDomain()
            .That().ResideInNamespace(ApiNamespace)
            .ShouldNot().HaveDependencyOn(InfraNamespace)
            .GetResult();

        Assert.True(result.IsSuccessful,
            "API layer directly references Infrastructure: " +
            string.Join(", ", result.FailingTypeNames ?? []));
    }

    /// <summary>
    /// Fitness function: All classes that implement IRepository must live
    /// in the Infrastructure namespace (never in Domain or Application).
    /// </summary>
    [Fact]
    public void Repositories_MustLive_InInfrastructure()
    {
        var result = Types.InCurrentDomain()
            .That().ImplementInterface(typeof(IRepository<>))
            .Should().ResideInNamespace(InfraNamespace)
            .GetResult();

        Assert.True(result.IsSuccessful,
            "Repository implementation found outside Infrastructure: " +
            string.Join(", ", result.FailingTypeNames ?? []));
    }
}

7.3 Cycle Detection as a Fitness Function

Cyclic dependencies between packages produce a system where no component can be independently deployed, tested, or reused. Detecting cycles as a CI gate prevents this class of architectural decay from entering the codebase.

[Fact]
public void NoCircularDependencies_BetweenNamespaces()
{
    // NetArchTest does not ship a built-in cycle detector, but you can
    // build one by reflecting on all assemblies and constructing a directed
    // dependency graph, then running DFS to detect back-edges.

    var assemblies = AppDomain.CurrentDomain.GetAssemblies()
        .Where(a => a.FullName!.StartsWith("MyApp."))
        .ToList();

    var graph = new Dictionary<string, HashSet<string>>();

    foreach (var assembly in assemblies)
    {
        var name = assembly.GetName().Name!;
        graph[name] = assembly
            .GetReferencedAssemblies()
            .Where(r => r.Name!.StartsWith("MyApp."))
            .Select(r => r.Name!)
            .ToHashSet();
    }

    var cycles = DetectCycles(graph);
    Assert.Empty(cycles);
}

private static IEnumerable<string> DetectCycles(
    Dictionary<string, HashSet<string>> graph)
{
    var visited = new HashSet<string>();
    var stack   = new HashSet<string>();
    var cycles  = new List<string>();

    foreach (var node in graph.Keys)
        Dfs(node, graph, visited, stack, cycles);

    return cycles;
}

private static void Dfs(
    string node,
    Dictionary<string, HashSet<string>> graph,
    HashSet<string> visited,
    HashSet<string> stack,
    List<string> cycles)
{
    if (stack.Contains(node))  { cycles.Add(node); return; }
    if (visited.Contains(node)) return;

    visited.Add(node);
    stack.Add(node);

    if (graph.TryGetValue(node, out var neighbours))
        foreach (var n in neighbours)
            Dfs(n, graph, visited, stack, cycles);

    stack.Remove(node);
}

---

8. Technology Radar Adoption

Thoughtworks popularized the Technology Radar as a quarterly snapshot of where the organization stands on its tooling choices. The radar has four concentric rings representing adoption maturity, and four quadrants: Techniques, Platforms, Tools, Languages & Frameworks.

8.1 The Four Rings

Ring	Meaning	Governance Action
Adopt	Proven in production; recommended for all new projects	Default choice; ADR not required for standard use
Trial	Validated in a pilot; limited production use permitted	Requires ADR; ARB notified
Assess	Being evaluated; proof-of-concept only	No production use; spike timebox ≤ 2 sprints
Hold	Deprecated or problematic; no new projects	Existing use reviewed quarterly for migration path

8.2 Governance Process for Radar Changes

A technology does not move between rings by committee consensus alone. Each ring transition requires evidence:

• Assess → Trial: A written spike report documenting performance benchmarks, security review findings, and operational complexity assessment.
• Trial → Adopt: Successful use in ≥ 2 independent production services over ≥ 3 months, with an incident retrospective showing the technology did not contribute to outages.
• Adopt → Hold: A mandatory migration path ADR must be drafted before the Hold decision is published. Teams cannot be told to stop using a technology without a concrete replacement timeline.

// Example: representing a Technology Radar entry as a C# record,
// suitable for serialization into a governance database or API.
public record TechnologyEntry(
    string Name,
    TechQuadrant Quadrant,
    TechRing Ring,
    string Rationale,
    DateTime LastReviewedUtc,
    string? MigrationTargetName);   // null unless Ring == Hold

public enum TechQuadrant { Techniques, Platforms, Tools, LanguagesAndFrameworks }

public enum TechRing { Adopt, Trial, Assess, Hold }

// Governance rule: Hold entries must have a migration target defined.
public static class TechnologyRadarValidator
{
    public static IEnumerable<string> Validate(IEnumerable<TechnologyEntry> entries)
    {
        foreach (var entry in entries.Where(e => e.Ring == TechRing.Hold))
        {
            if (string.IsNullOrWhiteSpace(entry.MigrationTargetName))
                yield return $"'{entry.Name}' is on Hold but has no migration target. " +
                             "An ADR must define the replacement technology.";
        }
    }
}

8.3 Integrating the Radar with the ARB

The Architecture Review Board runs a Technology Radar review meeting quarterly. The agenda has three standing items: (1) proposals to promote technologies up the rings based on evidence submitted by teams; (2) proposals to demote technologies based on security CVEs, licensing changes, or operational incidents; and (3) review of "Hold" entries to confirm migration timelines are on track.

Publishing the radar as a static website—versioned in Git—means engineers can always check the current organizational stance on any technology without submitting a ticket or asking a senior architect.

---

9. Cost Governance: FinOps Integration

Cloud billing is a first-class architectural concern. A system that is technically elegant but financially unsustainable will be decommissioned regardless of its design quality. FinOps is the practice of bringing financial accountability to cloud spending through cross-functional collaboration between engineering, finance, and product teams.

9.1 The Architectural Cost Ledger

Every ADR that introduces or changes infrastructure must include a cost estimate. This is the Architectural Cost Ledger: a structured table embedded in the ADR that projects monthly cloud expenditure at target scale.

Resource	Unit Cost	Projected Usage/Month	Estimated Monthly Cost
Azure Service Bus (Standard)	$0.10 / million operations	500M operations	$50
Azure SQL (General Purpose, 8 vCores)	$0.459 / vCore-hour	720 hours × 8 vCores	$2,650
Azure Blob Storage (LRS)	$0.018 / GB	10,000 GB	$180
Azure CDN (outbound)	$0.075 / GB	50,000 GB	$3,750
Total Projected Monthly			$6,630

This projection is reviewed by the ARB alongside the technical design. If the cost exceeds the allocated engineering budget, the design must be revised—perhaps using a lower storage tier, reducing CDN footprint, or batching operations to reduce per-operation billing.

9.2 Egress Fee Modeling in C#

Network egress (data transferred out of a cloud region to the internet or to another region) is one of the most frequently under-estimated cost drivers. The following model calculates projected monthly egress costs given a data transfer profile:

public record EgressTier(decimal GbThreshold, decimal PricePerGb);

public class EgressCostCalculator
{
    // Azure outbound egress pricing tiers (illustrative; verify current pricing).
    private static readonly IReadOnlyList<EgressTier> AzureTiers =
    [
        new EgressTier(100m,       0.00m),   // First 100 GB free
        new EgressTier(10_000m,    0.087m),  // 100 GB – 10 TB
        new EgressTier(50_000m,    0.083m),  // 10 TB  – 50 TB
        new EgressTier(decimal.MaxValue, 0.07m)   // 50 TB+
    ];

    /// <summary>
    /// Calculates the monthly egress cost for a given total transfer volume.
    /// </summary>
    /// <param name="totalGb">Total outbound data in gigabytes per month.</param>
    /// <returns>Estimated USD cost.</returns>
    public decimal Calculate(decimal totalGb)
    {
        decimal remaining = totalGb;
        decimal cost      = 0m;
        decimal consumed  = 0m;

        foreach (var tier in AzureTiers)
        {
            if (remaining <= 0m) break;

            decimal tierCapacity = tier.GbThreshold - consumed;
            decimal chargedInTier = Math.Min(remaining, tierCapacity);

            cost      += chargedInTier * tier.PricePerGb;
            remaining -= chargedInTier;
            consumed  += chargedInTier;
        }

        return Math.Round(cost, 2);
    }
}

// Usage: projecting egress cost for a video streaming service.
public class EgressCostProjection
{
    public void PrintMonthlyProjection()
    {
        var calculator = new EgressCostCalculator();

        // Scenario: 2M daily active users, average 3 GB streamed per session.
        decimal dailyGb   = 2_000_000m * 3m;
        decimal monthlyGb = dailyGb * 30m;

        decimal monthlyCost = calculator.Calculate(monthlyGb);

        Console.WriteLine($"Monthly egress volume : {monthlyGb:N0} GB");
        Console.WriteLine($"Estimated egress cost : ${monthlyCost:N2}");
        // Expected output: ~$12.6M/month — a clear signal to invest in a CDN
        // that caches at the edge, dramatically reducing origin egress.
    }
}

9.3 The Build-vs-Buy Cost Decision Framework

When evaluating whether to build a capability internally or adopt a managed service, the ARB uses a total cost of ownership (TCO) model over a 3-year horizon:

Build cost = (Engineer hours × hourly rate) + Operational overhead (on-call, patches, upgrades)

Buy cost = (Monthly SaaS/managed service fee × 36) + Integration engineering cost

The architectural recommendation is: prefer managed services for undifferentiated infrastructure (queues, caches, identity providers, CDNs). Reserve internal engineering capacity for the capabilities that directly differentiate the product. A message broker is not a competitive advantage; the business logic it carries might be.

Module 16 → Module 17: Once your engineering teams are aligned on architectural governance and ADR processes, you must learn how to design and cost-optimize virtualized cloud networks and compute boundaries. Proceed to Module 17: Cloud Architecture Fundamentals to begin Phase 6.