Zero-Downtime Deploys for a Production Web App

What you'll build / learn

You'll deploy a production web app to Kubernetes with correct readiness/liveness/startup probes, then ship a new version with zero dropped requests using either a rolling update or a blue-green cutover. You'll also learn the expand/contract migration pattern so schema changes never break the running version, plus how to roll back in seconds.

This is platform-agnostic in concept, but the commands target Kubernetes (the most common substrate for this). The same principles apply to ECS, Nomad, or a hand-rolled load balancer + systemd setup.

Prerequisites

• A Kubernetes cluster, v1.27+ (kind, minikube, EKS, GKE, or AKS all work). Check with kubectl version.
• kubectl configured against that cluster and pointed at a non-default namespace you own.
• A container image of your app that exposes an HTTP server. The Deployment below uses three distinct health endpoints — /healthz (startup), /readyz (readiness), and /livez (liveness). Your app must actually implement all three routes, or you can collapse them onto fewer paths intentionally and update the manifest to match. Probing a route that doesn't exist will make pods fail their probes.
• A relational database you control (Postgres in examples). Migrations run via your own tool (Flyway, Liquibase, Rails, Django, Alembic, Prisma — the pattern matters more than the tool).
• Familiarity with Deployments, Services, and kubectl rollout. This is an advanced guide; basic kubectl usage is assumed.

Create a working namespace:

kubectl create namespace shop
kubectl config set-context --current --namespace=shop

1. Get health checks right — this is the foundation

Zero downtime is impossible without the orchestrator knowing exactly when a pod can serve traffic and when it's dead. Kubernetes gives you three distinct probes. Conflating them is the #1 cause of self-inflicted outages.

Key rule: your readiness endpoint should check downstream dependencies it must have (DB connection pool, etc.), while liveness should be cheap and local — never have liveness call the database, or a DB blip will restart every pod simultaneously.

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 4
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
        version: v1
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: web
          image: registry.example.com/shop/web:1.4.0
          ports:
            - containerPort: 8080
          startupProbe:
            httpGet:
              path: /healthz
              port: 8080
            failureThreshold: 30
            periodSeconds: 2   # allows up to 60s to boot
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          lifecycle:
            preStop:
              exec:
                command: ["sh", "-c", "sleep 10"]
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              memory: "512Mi"

The version: v1 label is just an application-version tag on the pods of this single Deployment; the Service in this section and Section 2 selects pods purely on app: web. (Section 3's blue-green flow replaces that version tag with version: blue|green on two separate Deployments — keep them straight, we make this explicit below.)

Two non-obvious details that prevent dropped connections:

• preStop sleep + terminationGracePeriodSeconds. When a pod is deleted, Kubernetes simultaneously sends SIGTERM and removes the pod from Service endpoints — but endpoint removal propagates asynchronously to kube-proxy and ingress controllers. The preStop sleep keeps the old pod serving for ~10s while that propagation completes, so in-flight requests aren't cut. Your app must also handle SIGTERM by draining connections, not exiting instantly.
• failureThreshold on startupProbe decouples slow boots from liveness, so a slow JVM/Node cold start doesn't get killed mid-startup.

Apply and expose it:

kubectl apply -f deployment.yaml
kubectl expose deployment web --port=80 --target-port=8080

2. Rolling updates: the default safe path

A RollingUpdate replaces pods incrementally. Configure the surge/unavailable budget so capacity never dips below your needs:

# patch into spec: of the Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0   # never drop below desired replica count
      maxSurge: 1         # add one extra pod at a time
  minReadySeconds: 15     # pod must stay ready 15s before counting as available

maxUnavailable: 0 with maxSurge: 1 means Kubernetes always brings up a new ready pod before terminating an old one — strict zero capacity loss, at the cost of needing room for one extra pod. minReadySeconds guards against pods that pass readiness then immediately crash.

Apply, then ship a new image and watch the rollout:

kubectl apply -f deployment.yaml
kubectl set image deployment/web web=registry.example.com/shop/web:1.5.0
kubectl rollout status deployment/web --timeout=120s

kubectl rollout status blocks until the new ReplicaSet is fully available or fails — wire this into CI so a bad deploy fails the pipeline.

Rollback is built in, because Kubernetes keeps prior ReplicaSets:

kubectl rollout undo deployment/web            # back one revision
kubectl rollout history deployment/web         # list revisions
kubectl rollout undo deployment/web --to-revision=3

3. Blue-green: when you need an instant, atomic switch

Rolling updates briefly run both versions side by side. If your release cannot tolerate two versions live at once (rare, usually a sign of a non-backward-compatible change you should fix), use blue-green: run the full new version in parallel, verify it, then flip the Service selector in one atomic operation.

Blue-green changes the labeling model from Section 1. Instead of one web Deployment, you run two separately-named Deployments — web-blue and web-green — each tagging its pods with a version: blue or version: green label. The Service selects exactly one color. Create both Deployments explicitly:

# web-blue.yaml — the currently-live version
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-blue
  labels:
    app: web
    version: blue
spec:
  replicas: 4
  selector:
    matchLabels:
      app: web
      version: blue
  template:
    metadata:
      labels:
        app: web
        version: blue
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: web
          image: registry.example.com/shop/web:1.4.0
          ports:
            - containerPort: 8080
          # ...same startup/readiness/liveness probes, preStop, resources as Section 1

# web-green.yaml — the new version, identical except name, version label, and image
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-green
  labels:
    app: web
    version: green
spec:
  replicas: 4
  selector:
    matchLabels:
      app: web
      version: green
  template:
    metadata:
      labels:
        app: web
        version: green
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: web
          image: registry.example.com/shop/web:1.5.0
          ports:
            - containerPort: 8080
          # ...same probes/preStop/resources as Section 1

The Service selector must target a specific color so it only ever resolves to one Deployment's pods:

# service.yaml — selector pins to a color
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
    version: blue   # currently serving blue
  ports:
    - port: 80
      targetPort: 8080

Apply blue and the Service first (this is your steady state), then deploy green, wait for it to be fully ready, smoke-test it, and cut over by patching the live Service selector. port-forward is fine here because green is not being rolled during the test — you're validating one stable pod before the switch, not measuring zero-downtime behavior:

# 0. steady state: blue + Service live
kubectl apply -f web-blue.yaml
kubectl apply -f service.yaml

# 1. deploy green and wait for it to be fully Ready
kubectl apply -f web-green.yaml
kubectl rollout status deployment/web-green --timeout=120s

# 2. smoke-test green directly (port-forward pins to one green pod — fine pre-cutover)
kubectl port-forward deploy/web-green 9090:8080 &
curl -fsS http://localhost:9090/readyz   # validate before cutover

# 3. atomic switch — endpoints repoint instantly
kubectl patch service web -p '{"spec":{"selector":{"app":"web","version":"green"}}}'

Rollback is equally instant — patch the selector back to blue:

kubectl patch service web -p '{"spec":{"selector":{"app":"web","version":"blue"}}}'

Keep blue running until you're confident, then scale it to zero (kubectl scale deployment/web-blue --replicas=0).

Rolling should be your default. Reach for blue-green only when justified.

4. Database migrations: the part that actually causes outages

The orchestrator can do everything right and you'll still cause downtime if a migration breaks the version that's currently serving. During any rolling deploy, old and new code run against the same database simultaneously. The schema must therefore be compatible with both.

Advertisement

AirDroid Personal Best mobile device management suite — only $3.99/mo →

Use the expand/contract (a.k.a. parallel change) pattern. Never combine schema and code changes that aren't backward compatible in a single release.

Example: renaming users.fullname to users.full_name.

Phase 1 — Expand (deploy before/with new code):

-- additive only; old code untouched, new code can use either
ALTER TABLE users ADD COLUMN full_name text;

Now backfill. A single bulk UPDATE users SET full_name = fullname WHERE full_name IS NULL; is only acceptable on small tables — on a large one it is a single long transaction that holds row locks for its entire duration, bloats the table with dead tuples, and floods replication with one giant change set (lagging replicas). On a real production table, backfill in committed batches. PostgreSQL stored procedures (PG11+) can COMMIT inside a loop, which a DO block cannot:

-- batched backfill: commits after every chunk so locks are short and
-- replication stays current. Run CALL outside any explicit transaction.
CREATE PROCEDURE backfill_full_name() AS $$
DECLARE
  rows_updated integer;
BEGIN
  LOOP
    UPDATE users
    SET full_name = fullname
    WHERE id IN (
      SELECT id FROM users
      WHERE full_name IS NULL
      LIMIT 10000
    );
    GET DIAGNOSTICS rows_updated = ROW_COUNT;
    COMMIT;                 -- release locks, let replicas catch up
    EXIT WHEN rows_updated = 0;
  END LOOP;
END;
$$ LANGUAGE plpgsql;

CALL backfill_full_name();   -- must not run inside a transaction block

DROP PROCEDURE backfill_full_name();

If your migration tool wraps everything in a transaction, run the backfill as an external script (e.g. a loop in your deploy job invoking psql) so each batch commits independently.

Phase 2 — Migrate code: ship a version that writes to both columns and reads from full_name, falling back to fullname. Now both old and new pods work.

Phase 3 — Contract (a later, separate release, after old code is fully gone):

ALTER TABLE users DROP COLUMN fullname;

Hard rules for online migrations on Postgres:

• Adding a column with a non-volatile default is fast in modern Postgres (11+). Enforcing NOT NULL is trickier: ALTER TABLE ... ALTER COLUMN ... SET NOT NULL always takes an ACCESS EXCLUSIVE lock, and there is no NOT VALID/VALIDATE CONSTRAINT form for a column NOT NULL (that syntax exists only for CHECK and foreign-key constraints). The lock-light pattern is: add a CHECK constraint as NOT VALID (instant, no scan under a brief lock), VALIDATE it separately (scans with only a SHARE UPDATE EXCLUSIVE lock that doesn't block writes), then on PG12+ run SET NOT NULL, which uses the validated CHECK to skip the full table scan:

-- run the whole sequence with a short lock_timeout so any step that can't
-- grab its lock fails fast instead of queueing behind/ahead of live traffic
SET lock_timeout = '3s';

-- 1. backfill first (batched, as above) so the column has no NULLs

-- 2. add the check as NOT VALID — brief lock, no scan
ALTER TABLE users
  ADD CONSTRAINT users_full_name_not_null CHECK (full_name IS NOT NULL) NOT VALID;

-- 3. validate without blocking writes (SHARE UPDATE EXCLUSIVE)
ALTER TABLE users VALIDATE CONSTRAINT users_full_name_not_null;

-- 4. PG12+: SET NOT NULL reuses the validated CHECK to SKIP THE FULL SCAN,
--    but it STILL takes a brief ACCESS EXCLUSIVE lock on the table.
--    On a hot table, run it with lock_timeout (set above) so it can't stall
--    behind a long-running query and block every writer.
ALTER TABLE users ALTER COLUMN full_name SET NOT NULL;

-- 5. optional: drop the now-redundant CHECK
ALTER TABLE users DROP CONSTRAINT users_full_name_not_null;

• Create indexes concurrently to avoid locking writes:

CREATE INDEX CONCURRENTLY idx_users_full_name ON users (full_name);

CONCURRENTLY cannot run inside a transaction block — tell your migration tool to disable its transaction wrapper for that step.

• Set a lock_timeout (shown above) so a migration that can't grab a lock fails fast instead of blocking your whole app.

Run migrations as a separate, gated step — not inside your app's startup. In Kubernetes, a Job (or Helm pre-install/pre-upgrade hook) is the right primitive:

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-1-5-0
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/shop/web:1.5.0
          command: ["./migrate", "up"]
          envFrom:
            - secretRef:
                name: db-credentials

kubectl apply -f migrate-job.yaml
kubectl wait --for=condition=complete job/migrate-1-5-0 --timeout=300s

Because every phase is backward compatible, rolling back code never requires rolling back the schema — the old version still works against the expanded schema. This is the single most important property for safe deploys.

5. Verify it works

Generate continuous traffic during a deploy and confirm zero errors. Do not use kubectl port-forward for this test. port-forward service/web resolves the Service to a single backing pod and pins the connection to it; it does not load-balance across endpoints. When that pod is rolled out it will be terminated, breaking the forward and producing connection errors even with a perfectly configured zero-downtime deploy — exactly the false negative you're trying to avoid.

Instead, drive traffic through a real load-balanced path: hit the ClusterIP DNS name (web.shop.svc.cluster.local) from a pod inside the cluster, so every request is balanced across all ready endpoints just like production traffic.

# Run an in-cluster curl loop against the Service's ClusterIP (load-balanced)
kubectl run loadtest --rm -it --restart=Never --image=curlimages/curl:8.10.1 -- \
  sh -c 'while true; do \
    code=$(curl -s -o /dev/null -w "%{http_code}" http://web.shop.svc.cluster.local/); \
    echo "$(date +%T) $code"; \
    [ "$code" != "200" ] && echo "DROPPED REQUEST"; \
    sleep 0.2; \
  done'

Trigger a rollout in another terminal (kubectl set image deployment/web web=registry.example.com/shop/web:1.5.0). A correctly configured deploy shows an unbroken stream of 200s. For higher confidence and concurrency, run a small in-cluster load tester instead of a serial curl loop:

kubectl run fortio --rm -it --restart=Never --image=fortio/fortio:latest -- \
  load -qps 200 -t 120s http://web.shop.svc.cluster.local/

Fortio reports the response-code histogram; you want 100.0% 200 with no non-2xx codes across the rollout window.

Then confirm state:

kubectl rollout status deployment/web      # 'successfully rolled out'
kubectl get pods -l app=web                 # all Running, all 1/1 Ready

# Endpoints API (legacy but still populated in current clusters):
kubectl get endpoints web                   # only ready pod IPs listed

# Modern equivalent — EndpointSlices supersede Endpoints:
kubectl get endpointslices -l kubernetes.io/service-name=web

Expected: rollout status reports success, every pod is 1/1, only ready pod IPs appear in the endpoints/slices, and your in-cluster traffic generator logged no non-200 responses.

6. Troubleshooting

Requests dropped during rollout despite maxUnavailable: 0. Almost always missing connection draining. Confirm your app traps SIGTERM and finishes in-flight requests, and that you have a preStop sleep ≥ the time your ingress/kube-proxy needs to deregister the endpoint. Verify the pod actually exits gracefully: kubectl logs <pod> --previous. Also double-check you aren't testing through port-forward, which will show false drops (see Section 5).

New pods never become Ready; rollout hangs. The readiness probe is failing. Inspect events: kubectl describe pod <pod> and look for Readiness probe failed. Common causes: probe path/port mismatch (e.g. /readyz not implemented), readiness checking a dependency that isn't reachable, or timeoutSeconds too low for a busy endpoint. Don't "fix" it by deleting the readiness probe — that just hides the breakage.

Migration Job blocks all traffic. A schema change took a heavy lock (e.g. a bare SET NOT NULL scan, a non-concurrent index build, or a single unbatched bulk backfill). Cancel it, set lock_timeout, and re-run using the additive/expand approach above (batched backfill → CHECK NOT VALID → VALIDATE → SET NOT NULL). Check for blockers with SELECT * FROM pg_stat_activity WHERE wait_event_type = 'Lock';.

kubectl rollout undo doesn't restore old behavior. The schema already moved forward incompatibly — you skipped expand/contract. This is why migrations must stay backward compatible: a code rollback alone can't fix it. Recover by rolling forward with a fixed version, or restoring the DB if data integrity is at risk.

7. Next steps

• Progressive delivery: add Argo Rollouts or Flagger for automated canary deploys with metric-based analysis and automatic abort.
• PodDisruptionBudgets: add a PodDisruptionBudget so node drains/upgrades respect your minimum availability.
• Pre-deploy gates in CI: run the migration Job and kubectl rollout status --timeout as required pipeline steps so a failed deploy fails the build.
• Observability: alert on error-rate and latency per version label so a bad canary is caught in seconds, not from customer reports.