Developer Docs

Hybrid Staging Infrastructure

The Staging Environment Architecture document describes what runs in staging: a 34-container Docker Compose topology built from production targets. This document describes where it runs, and how a 1 GB VPS reaches it.

Staging compute lives on a single Mac running Podman (just like development), and a small Debian VPS handles only DNS, TLS termination, and reverse-proxy routing. Persistent reverse SSH tunnels stitch them together. The pattern lets us keep beta indistinguishable from production for external users while spending under $5/month on infrastructure.

The implementation lives in ngwenya-infra — Ansible playbooks, a build orchestrator, and a Makefile that drives the whole loop. This doc is the engineer-facing tour.

The Three Tiers

Visitor ──► beta-*.mallnline.com ──► VPS (NGINX + Certbot) ──[ reverse SSH tunnel ]──► Mac (Podman)
Tier What it owns Sizing
VPS DNS-terminated public IP, NGINX vhosts, Let's Encrypt certs, 80→443 redirects, styled 502 fallback page Debian 12, 1 GB RAM, 12 GB SSD
Tunnels 8 long-running autossh processes, wrapped in macOS LaunchAgents for crash-restart and login-resume n/a — runtime artifact
Mac The 30+ container Ngwenya stack (subgraphs, gateway, frontend, portals, databases, observability) 500 GB virtual disk on the Podman applehv VM

The VPS doesn't run any application code. The Mac doesn't accept any inbound connections. The reverse tunnel is what makes the Mac reachable from the public internet without VPNs, port forwarding, or a static IP at the operator's residence.

The Request Path

A GET https://beta.mallnline.com/ from a real Visitor:

  1. DNS resolves beta.mallnline.com to the VPS IPv4.
  2. TLS handshake — NGINX presents the Let's Encrypt cert. One cert covers six SANs (one per beta-* subdomain). See SSL strategy.
  3. vhost match — /etc/nginx/sites-enabled/beta.mallnline.com.conf is the Ansible-managed server block. It calls proxy_pass http://127.0.0.1:4173.
  4. 127.0.0.1:4173 on the VPS is not a real backend — it's an sshd reverse-forward listener created by the autossh client logged in from the Mac. The kernel writes the bytes into that channel.
  5. Tunnel hop — the autossh process (com.mallnline.tunnel.frontend LaunchAgent on the Mac) forwards the channel to its local target: [::1]:4173.
  6. gvproxy bridge — Podman's userland helper listens on the IPv6 wildcard for port 4173 and bridges into the Podman VM, where the staging-frontend SvelteKit container is bound.
  7. SvelteKit renders the response, which retraces the same chain back to the Visitor.

If any link breaks (container down, tunnel disconnected, sshd refused forwarding), NGINX falls back to the styled Beta Platform Availability 502 page rather than the default ugly one.

Tunnel Map

graph LR
    subgraph Public
      beta[beta.mallnline.com]
      api[beta-api.mallnline.com]
      dev[beta-dev.mallnline.com]
      sup[beta-support.mallnline.com]
      graf[beta-grafana.mallnline.com]
      glitch[beta-glitchtip.mallnline.com]
    end

    subgraph VPS["VPS · 127.0.0.1 sshd"]
      v4173[":4173"]
      v30000[":30000"]
      v4321[":4321"]
      v4322[":4322"]
      v3100[":3100"]
      v8000[":8000"]
    end

    subgraph Mac["Mac Podman · staging-* containers"]
      front[Frontend SvelteKit]
      gateway[GraphQL Gateway]
      devPortal[Dev Portal Astro]
      support[Support Center Astro]
      grafana[Grafana]
      glitchtip[GlitchTip]
    end

    beta-->v4173-. tunnel .->front
    api-->v30000-. tunnel .->gateway
    dev-->v4321-. tunnel .->devPortal
    sup-->v4322-. tunnel .->support
    graf-->v3100-. tunnel .->grafana
    glitch-->v8000-. tunnel .->glitchtip

Two more tunnels (Prometheus 9090, MinIO console 9001) exist as operator-only debugging channels — no public NGINX vhost, only reachable via ssh -L from an operator's workstation.

Why `[::1]` is Load-Bearing

Each autossh process runs with:

autossh -M 0 -N \
  -R 127.0.0.1:<vps_port>:[::1]:<local_port> \
  ...

The VPS bind stays IPv4 (127.0.0.1:port) because that's what NGINX proxy_pass http://127.0.0.1:port expects, and binding to loopback hides the staging ports from the public internet.

The Mac-side target is pinned to [::1] because Podman's gvproxy helper binds container ports on the IPv6 wildcard only:

gvproxy  IPv6  TCP *:4173 (LISTEN)

macOS dual-stack handling means 127.0.0.1 connections usually succeed against an IPv6-only socket — but ssh -R resolves localhost strictly, and a misconfigured getaddrinfo path can fail intermittently. Pinning to [::1] removes the ambiguity entirely.

This was an outage the first time it happened. Now it's documented.

One Cert, Many SANs

Six subdomains share a single Let's Encrypt certificate stored under /etc/letsencrypt/live/beta.mallnline.com/. The cert's subjectAltName extension lists every covered domain:

DNS:beta-api.mallnline.com, DNS:beta-dev.mallnline.com, DNS:beta-glitchtip.mallnline.com,
DNS:beta-grafana.mallnline.com, DNS:beta-support.mallnline.com, DNS:beta.mallnline.com

Adding a new subdomain triggers a certbot --nginx --expand call inside the vps-certbot Ansible role. The role first reads the existing cert's SAN list:

- name: Read SAN list from existing certificate (if any)
  ansible.builtin.command: >
    openssl x509 -noout -ext subjectAltName
    -in /etc/letsencrypt/live/{{ cert_primary_domain }}/fullchain.pem
  register: cert_san_raw
  failed_when: false

…computes the gap against cert_domains (the desired SAN list, derived from services.yml), and only triggers issuance when the gap is non-empty. The result: re-running make bootstrap after adding a subdomain is a one-step operation, idempotent on no-change.

Auto-renewal is handled by certbot.timer (systemd, fires every 12h, renews anything within 30 days of expiry) with a deploy hook that reloads NGINX after success.

Smart Build Orchestrator

Staging builds 24 microservices across ngwenya-federation, ngwenya-front, ngwenya-dev, and ngwenya-support. A naïve "rebuild everything" pass takes ~15 minutes. The orchestrator at ansible/scripts/build_orchestrator.sh reduces typical iteration to under 30 seconds through three mechanisms:

1. Shared BuildKit cache mounts

RUN --mount=type=cache,id=pnpm,target=/pnpm/store pnpm install -r

A single persistent pnpm store backs all 19 NestJS service builds. First build pays the ~1.5 GB download tax; every subsequent build hits the cache. The store survives podman system prune because it lives in BuildKit's internal state, not in image layers.

2. Selective context copying

Each service's Dockerfile copies only its own apps/<service>/ subtree plus shared libs/ and patches/. A change inside apps/malets/ invalidates only staging-malets — every other NestJS image hits 100% layer cache.

3. Per-repo SHA tracking

logs/.build_state.json records the git HEAD of each repo at last successful build. The orchestrator diffs the working SHA against that and categorizes the changes:

Path pattern Trigger
package.json, pnpm-lock.yaml, libs/, patches/ Rebuild all 19 NestJS services
Cargo.toml, Cargo.lock Rebuild all 4 Rust services
apps/<name>/... Rebuild only <name>

This is the difference between "every commit costs 15 minutes" and "every commit costs the time of the one service you actually touched."

Diagnostics

ansible/diagnose.sh is the canonical first response to "staging is broken." It walks six layers — Mac containers, local port bindings, LaunchAgent state, VPS SSH, VPS port bindings, remote HTTPS — and tags each step green/red. The script is ~200 lines of pure bash, runs in under 10 seconds, and produces actionable output instead of requiring an operator to know the topology.

Typical clean-run output:

[1/6] Checking Local Container Status (Podman)...        ✅ All 32 containers running
[2/6] Verifying Local Port Bindings (Mac)...             ✅ 8 ports listening
[3/6] Checking macOS LaunchAgent (Tunnel) States...      ✅ 8 agents registered, all .err files clean
[4/6] Verifying VPS SSH Connectivity...                  ✅ Connected, Debian 6.1.0-48
[5/6] Checking VPS Port Bindings...                      ✅ All 8 reverse listeners bound on 127.0.0.1
[6/6] Verifying Remote Endpoints & HTTPS Responses...    ✅ 5/6 200|302, 1/6 405 (GraphQL POST-only)

The 405 on the GraphQL endpoint is intentional — the script uses GET; GraphQL only accepts POST. It's a "tunnel + nginx + cert all working" signal, not a failure.

Operator Make Targets

The entire surface is wrapped in Makefile targets that hide the Ansible invocations:

Target Effect
make bootstrap One-time VPS setup (SSH + UFW + NGINX + Certbot SAN expansion)
make build-changed Smart incremental build via the orchestrator
make deploy Containers + tunnels (the daily verb)
make deploy-quick Tunnels + containers without re-building
make teardown Kill switch — visitors see the 502 fallback
make tunnels-up / tunnels-down Just the LaunchAgent layer
make health Local healthcheck + remote HTTPS probes
make backup Mongo/Postgres dumps from running containers

This matches the daily-driver pattern used by Local Development Environment and Staging Environment Architecture — operators don't memorize Ansible plays, they memorize verbs.

When Hybrid Stops Making Sense

The hybrid model is right for staging today. It becomes wrong when any of these become true:

  • Sustained external traffic — the operator's home internet becomes a bottleneck.
  • Compliance / 24/7 SLA — a laptop is not a "data center."
  • Multi-operator deploys — handing off staging means handing off a physical machine.
  • Persistent test data — backup/restore is much harder when state lives on a laptop.

When that moment arrives, the same Ansible repository drives 02-deploy-vps.yml, which deploys the full stack to a fat VPS or cluster. The tunnel layer disappears, but the build orchestrator and certbot logic carry forward.