Health Checks and Graceful Shutdown

In modern cloud-native architectures, service instances are dynamically created, destroyed, and rolled out continuously. Maintaining high availability across these volatile environments requires a robust synergy between health checks and graceful shutdowns. Without proper implementation, rolling updates or auto-scaling events can easily trigger frustrating 502 Bad Gateway errors, connection resets, and dropped user requests.

Within Kubernetes, health checks are typically handled through three distinct types of probes designed to monitor container lifecycles:

• livenessProbe: Evaluates whether the container is operating normally. If failure thresholds are repeatedly breached, the kubelet will proactively restart the container. Crucially, liveness probes should avoid checking external dependencies like databases, as a transient downstream outage could otherwise cascade into a cluster-wide restart loop.
• readinessProbe: Determines if an instance is fully prepared to accept inbound traffic. When this probe fails, the Pod is automatically removed from the associated Service's Endpoints, cutting off new request routing.
• startupProbe: Tailored for legacy or resource-intensive applications with prolonged initialization phases. While the startup probe is running, both liveness and readiness probes are suppressed, preventing premature container termination before the application has fully initialized.

Probes can be executed via various mechanisms, including HTTP GET requests, TCP sockets, command execution, or gRPC. To prevent resource exhaustion, health check endpoints should always remain lightweight and avoid complex queries or blocking operations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      terminationGracePeriodSeconds: 40
      containers:
      - name: app
        image: demo:1.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          periodSeconds: 2
          failureThreshold: 1
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          periodSeconds: 10
          failureThreshold: 3
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh", "-c", "sleep 5"]

In the configuration above, the /readyz endpoint triggers an immediate removal from the load balancer upon a single failure, while /healthz requires three consecutive failures before initiating a container restart. The sleep 5 command inside the preStop hook introduces a deliberate buffer, giving the routing layer sufficient time to update its endpoint lists and minimizing the chance of routing new traffic to a terminating Pod.

The Core Workflow of Graceful Shutdowns

The primary objective of a graceful shutdown is not an immediate exit, but a controlled exit sequence: first halting incoming traffic, then finishing active requests, and finally releasing system resources. A standard shutdown lifecycle follows these steps:

1. The application receives a SIGTERM signal or a termination notification.
2. The readiness state is explicitly toggled to failure, signaling the load balancer or service mesh to deregister the instance.
3. The application pauses briefly to allow routers, Ingress controllers, and clients to synchronize their routing tables.
4. The server stops accepting new connections while keeping existing persistent connections open.
5. The process waits for ongoing requests to complete or until a predefined shutdown timeout is reached.
6. Underlying resources—such as database connection pools, message queue consumers, and file descriptors—are safely closed.
7. The process exits normally. If the shutdown exceeds the allotted timeout, the orchestration platform forcefully terminates it.

When deleting a Pod in Kubernetes, the kubelet first executes the configured preStop hook, dispatches a SIGTERM signal to the container, and starts the terminationGracePeriodSeconds countdown before resorting to a forced SIGKILL.

Code Example: Graceful Shutdown in a Go Service

package main

import (
    "context"
    "fmt"
    "net/http"
    "os"
    "os/signal"
    "sync/atomic"
    "syscall"
    "time"
)

func main() {
    var ready atomic.Bool
    ready.Store(true)

    mux := http.NewServeMux()
    mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
    })
    mux.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
        if !ready.Load() {
            http.Error(w, "not ready", http.StatusServiceUnavailable)
            return
        }
        w.WriteHeader(http.StatusOK)
    })
    mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        time.Sleep(2 * time.Second)
        fmt.Fprintln(w, "ok")
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            fmt.Println("listen error:", err)
        }
    }()

    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
    <-sig

    ready.Store(false) // Mark readiness as false to allow traffic deregistration
    time.Sleep(5 * time.Second) // Buffer for load balancers to update endpoints

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        fmt.Println("server shutdown error:", err)
    }

    fmt.Println("server gracefully stopped")
}