Module 20: Container Orchestration & Cloud Native Runtimes

PREREQUISITE STATEMENT: Read this module after completing Module 19 (Enterprise Observability). Operating telemetry pipelines at scale requires understanding the containerized runtimes and cluster orchestration engines hosting your microservice workloads.

---

1. Containerization Mechanics & Linux Kernel Primitives

A container is not a lightweight virtual machine. A VM uses a hypervisor (e.g. KVM, ESXi) to emulate physical hardware (virtual CPUs, RAM, NICs) and runs an independent operating system kernel. A container is a set of standard host OS processes isolated by two core Linux kernel primitives: Namespaces and Control Groups (cgroups).

       [ Virtual Machine Architecture ]             [ Container Architecture ]
  +---------------------------------------+    +---------------------------------------+
  |  App A  |  App B  |  App C            |    |  App A  |  App B  |  App C            |
  | GuestOS | GuestOS | GuestOS           |    | Bin/Lib | Bin/Lib | Bin/Lib           |
  +---------------------------------------+    +---------------------------------------+
  |  Hypervisor (KVM / VMware)            |    |  Container Engine (containerd/CRI-O)  |
  +---------------------------------------+    +---------------------------------------+
  |  Host OS Kernel                       |    |  Host OS Kernel (Namespaces/cgroups)  |
  +---------------------------------------+    +---------------------------------------+
  |  Physical Hardware                    |    |  Physical Hardware                    |
  +---------------------------------------+    +---------------------------------------+

The 6 Core Linux Namespaces

Namespace Primitive	Flag	Isolation Boundary	Architectural Function
PID (Process ID)	CLONE_NEWPID	Process Tree	Container process sees itself as PID 1, isolated from host PIDs.
NET (Networking)	CLONE_NEWNET	Network Interfaces	Container receives its own virtual NIC (eth0), IP address, and routing table.
MNT (Mount)	CLONE_NEWNS	File System Mounts	Container receives an isolated root file system (/) via chroot/pivot_root.
IPC (Inter-Process)	CLONE_NEWIPC	System V IPC / POSIX Queues	Prevents container processes from accessing host shared memory blocks.
UTS (Hostname)	CLONE_NEWUTS	Hostname and NIS Domain	Allows container to maintain its own hostname independently.
USER (User IDs)	CLONE_NEWUSER	UID / GID Mappings	Maps root user (UID 0) inside container to a non-privileged UID on host.

---

2. Linux Control Groups (cgroups v2)

While Namespaces isolate what a process can see, Control Groups (cgroups) limit and account for how much physical hardware resources (CPU, Memory, Disk I/O) a container can consume.

CPU Limits vs. Memory Hard Limits

                                [ Resource Limit Enforcement ]
                                              |
                   +--------------------------+--------------------------+
                   |                                                     |
         [ Memory Limit (cgroup hard limit) ]                [ CPU Limit (CFS quota) ]
         - Enforced by Linux OOM Killer                     - Enforced by Completely Fair Scheduler
         - Triggers OOMKilled Process Termination           - Triggers CPU Throttling (Latency Spike)
         - Container crashes immediately                    - Process slows down, does not crash

1. Memory Limits (memory.max): Hard memory ceiling. If a container process attempts to allocate memory beyond its configured limit, the kernel OOM (Out Of Memory) Killer sends a SIGKILL (Signal 9) to PID 1, resulting in an immediate container crash (OOMKilled).
2. CPU Limits (cpu.max): Time-slice quota evaluated over a 100ms period by the Completely Fair Scheduler (CFS). If a container exceeds its quota, its thread execution is paused (throttled) until the next 100ms window begins, manifesting as latency spikes rather than container crashes.

---

3. Kubernetes Control Plane Architecture

Kubernetes orchestrates containerized workloads across a cluster of nodes using a declarative control plane loop:

graph TD
    subgraph Control_Plane [Kubernetes Control Plane Node]
        APIServer[kube-apiserver] <==> ETCD[(etcd State Store)]
        Scheduler[kube-scheduler] -->|Watch & Bind Pods| APIServer
        KCM[kube-controller-manager] -->|Enforce Desired State| APIServer
    end

    subgraph Worker_Node_1 [Worker Node 1]
        Kubelet1[kubelet] -->|Read Pod Spec| APIServer
        ContainerRuntime1[containerd] -->|CRI Calls| Pod1[Pod Workload A]
        KubeProxy1[kube-proxy] -->|iptables/IPVS Rules| ContainerRuntime1
    end

    subgraph Worker_Node_2 [Worker Node 2]
        Kubelet2[kubelet] -->|Read Pod Spec| APIServer
        ContainerRuntime2[containerd] -->|CRI Calls| Pod2[Pod Workload B]
    end

    style Control_Plane fill:#0f172a,stroke:#3b82f6,stroke-width:2px
    style Worker_Node_1 fill:#1e293b,stroke:#34d399,stroke-width:2px
    style Worker_Node_2 fill:#1e293b,stroke:#34d399,stroke-width:2px

Control Plane Components

• kube-apiserver: The central REST API front-end. Validates and persists declarative JSON/YAML manifests to etcd.
• etcd: Consistent, highly available key-value store (using Raft consensus) storing the total cluster state.
• kube-scheduler: Assigns unscheduled Pods to optimal worker nodes based on resource requests, taints, tolerations, and affinity rules.
• kube-controller-manager: Runs background controller loops (DeploymentController, ReplicaSetController) to reconcile actual cluster state with desired state.
• kubelet: Agent running on every worker node. Ensures that containers described in PodSpecs are running and healthy.

---

4. Production Multi-Stage C# Dockerfile

Below is a production multi-stage C# Dockerfile targeting .NET 8. It compiles binaries in a heavy SDK image and outputs a minimal, non-root, hardened distroless runtime container.

# Stage 1: Build & Compile
FROM mcr.microsoft.com/dotnet/sdk:8.0-alpine AS build
WORKDIR /src

# Copy project files first for efficient layer caching
COPY ["MacroPatternsConsortium.OrderService.csproj", "./"]
RUN dotnet restore "MacroPatternsConsortium.OrderService.csproj"

# Copy full source and publish release binary
COPY . .
RUN dotnet publish "MacroPatternsConsortium.OrderService.csproj" \
    -c Release \
    -o /app/publish \
    /p:UseAppHost=false

# Stage 2: Runtime Image (Hardened & Non-Root)
FROM mcr.microsoft.com/dotnet/aspnet:8.0-alpine AS final
WORKDIR /app

# Create non-privileged system user for security
RUN addgroup -g 10001 -S appgroup && \
    adduser -u 10001 -S appuser -G appgroup

COPY --from=build --chown=appuser:appgroup /app/publish .

# Switch to non-root user (Security Best Practice)
USER 10001:10001

ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080

ENTRYPOINT ["dotnet", "MacroPatternsConsortium.OrderService.dll"]

---

5. Declarative Kubernetes Production Manifests

Below are complete, production-grade Kubernetes manifests defining a Deployment, Horizontal Pod Autoscaler (HPA), Service, and Ingress.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-service-deployment
  namespace: mpc-production
  labels:
    app: order-service
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  selector:
    matchLabels:
      app: order-service
  template:
    metadata:
      labels:
        app: order-service
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
      containers:
      - name: order-service
        image: registry.macropatternsconsortium.com/order-service:1.2.0
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
          name: http-port
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
          limits:
            cpu: 1000m
            memory: 512Mi
        startupProbe:
          httpGet:
            path: /health/startup
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 10
        livenessProbe:
          httpGet:
            path: /health/liveness
            port: 8080
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 8080
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 2
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service-hpa
  namespace: mpc-production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service-deployment
  minReplicas: 3
  maxReplicas: 25
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
---
apiVersion: v1
kind: Service
metadata:
  name: order-service-svc
  namespace: mpc-production
spec:
  type: ClusterIP
  selector:
    app: order-service
  ports:
  - port: 80
    targetPort: 8080
    name: http
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: order-service-ingress
  namespace: mpc-production
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/limit-rps: "100"
spec:
  tls:
  - hosts:
    - api.macropatternsconsortium.com
    secretName: mpc-tls-cert
  rules:
  - host: api.macropatternsconsortium.com
    http:
      paths:
      - path: /v1/orders
        pathType: Prefix
        backend:
          service:
            name: order-service-svc
            port:
              number: 80

---

6. Probes & Health Lifecycle Mechanics

               [ Kubernetes Container Health Lifecycle ]
               
  [ Pod Scheduled ] 
        |
        v
  [ Startup Probe ] ------( Fails x Threshold )------> [ Container Restarted ]
        | (Passes)
        v
  +---------------------------------------------------+
  |  [ Liveness Probe ]  ---> Fails -> Restart Pod   |
  |  [ Readiness Probe ] ---> Fails -> Remove Endpoints|
  +---------------------------------------------------+

Probe Responsibilities

1. Startup Probe: Disables Liveness and Readiness probes until the container finishes slow initialization tasks (e.g. database migration, warming cache).
2. Liveness Probe: Determines if the container process is deadlocked or unrecoverable. If it fails, Kubernetes kills and restarts the container.
3. Readiness Probe: Determines if the container is ready to accept user network traffic. If it fails, Kubernetes temporarily removes the Pod's IP address from the Service endpoint list without restarting the container.

---

7. Hands-on Architecture Challenge

Scenario Description

An enterprise microservice deployment suffers from traffic drops during deployment updates. When new Pods spin up, the Ingress Controller immediately sends HTTP requests before the app finishes initializing its C# singleton connections, causing 502 Bad Gateway responses.

Your Goal:

1. Fix the deployment topology by adding a Readiness Probe to the container specification.
2. Configure Zero-Downtime Rolling Update Strategy:
   • Set maxSurge: 25% (spin up extra new pods before killing old pods).
   • Set maxUnavailable: 0 (guarantee 100% existing capacity stays live during updates).
3. Connect the Ingress -> ClusterIP Service -> Pod Replicas flow in the architecture diagram.

---

8. Practice Challenge Template

Use this template in your sandbox to model the container network topology:

graph TD
    subgraph External_Traffic [External Traffic]
        Client[Public Web Client]
    end

    subgraph Cluster_Ingress [Cluster Ingress Gateway]
        Ingress[Ingress Controller (NGINX)]
    end

    subgraph Service_Tier [ClusterIP Service Routing]
        Service[Service: order-service-svc (Port 80)]
    end

    subgraph Pod_Replicas [Pod Replica Set (MaxSurge: 25%, MaxUnavailable: 0)]
        Pod1[Pod 1 (Readiness: OK)]
        Pod2[Pod 2 (Readiness: OK)]
        Pod3[Pod 3 (Readiness: Init...)]
    end

    Client -->|HTTPS /v1/orders| Ingress
    Ingress -->|ClusterIP Routing| Service
    Service -->|Traffic Routed| Pod1
    Service -->|Traffic Routed| Pod2
    Service -.->|Blocked by Readiness Probe| Pod3

    style External_Traffic fill:#0f172a,stroke:#3b82f6,stroke-width:2px
    style Cluster_Ingress fill:#1e1b4b,stroke:#6366f1,stroke-width:2px
    style Service_Tier fill:#0f172a,stroke:#34d399,stroke-width:2px
    style Pod_Replicas fill:#1e293b,stroke:#f59e0b,stroke-width:2px

---

NEXT MODULE BRIDGE: Master container runtimes and Kubernetes control loops before advancing to enterprise architectural evaluation. Proceed to Module 21: Architecture Gap Analysis to learn technical debt valuation frameworks.