Developer Docs

Tower Health Observability

The Tower Health Observability system is a background Rust task inside the Tower subgraph that continuously probes all 34 platform services using a dual-layer architecture — checking both process liveness (direct probes) and Gateway schema composition (federated probes). The results are exposed as federated GraphQL and rendered in the Tower admin dashboard as a real-time health grid.

Design Principle: The dual-layer approach was inspired by the existing make smoke-test infrastructure (scripts/test/smoke-test.sh). Where the CLI tests each subgraph through the Gateway and accepts Authentication required as a "reachable" signal, Tower does the same — plus a direct probe to the process port for liveness.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Tower Health Monitor                         │
│                  (tokio::spawn background task)                 │
│                                                                 │
│  For each registered service:                                   │
│                                                                 │
│  ┌──── Direct Probe ────┐    ┌──── Federated Probe ────┐       │
│  │ POST {port}/graphql   │    │ POST gateway:30000/graphql│      │
│  │ Body: smoke query     │    │ Body: smoke query        │      │
│  │ Timeout: 8s           │    │ (routed through Gateway) │      │
│  │ ✓ 200 = UP            │    │ Timeout: 8s              │      │
│  │ ✓ "Auth required" = UP│    │ ✓ 200 = UP               │      │
│  │ ✗ timeout/error = DOWN│    │ ✓ "Auth required" = UP   │      │
│  └───────────────────────┘    │ ✗ timeout/error = DOWN   │      │
│                                └──────────────────────────┘      │
│                                                                 │
│  Result → SharedHealthState (Arc<RwLock<PlatformHealthOverview>>)│
└─────────────────────────────────────────────────────────────────┘
          │
          ▼ Exposed via federated GraphQL
┌─────────────────────┐     ┌───────────────────────────┐
│  platformHealth      │     │  PlatformHealthWidget     │
│  (GraphQL Query)     │────▶│  (Svelte 5 component)     │
│  via Hive Gateway    │     │  Grouped grid + modal     │
└─────────────────────┘     └───────────────────────────┘

Diagnostic Matrix

The dual-layer design produces four diagnostic states that precisely isolate failure modes:

Direct Federated Diagnosis Action
✅ UP ✅ UP Fully operational No action needed
✅ UP ❌ DOWN Process alive — federation issue Check logs/main.log for schema composition errors. The Gateway likely failed to compose this subgraph's schema. Run make restart-gateway to re-compose.
❌ DOWN ✅ UP Stale gateway cache Process crashed but the Gateway is serving cached responses. The cache will expire soon — restart the service.
❌ DOWN ❌ DOWN Service down Check logs/{service}.log for crash output. Restart with make restart-{service} or make services-force.

TIP

The UP/DOWN (direct alive, federated failing) state is the most actionable diagnostic — it catches schema composition failures that make smoke-test alone would miss, since the CLI test only probes through the Gateway.

Smoke Query Mapping

Each subgraph uses a service-specific GraphQL query that mirrors the smoke test script. This ensures parity between CLI-based testing and the live dashboard:

Service Direct Query Notes
malets { maletsTest } Dedicated health-check resolver
products { productsTest } Dedicated health-check resolver
services { servicesTest } Dedicated health-check resolver
ucart { ucartTest } Dedicated health-check resolver
media { mediaHealthCheck } Dedicated health-check resolver
blogs { blogsTest } Dedicated health-check resolver
community { communityTest } Dedicated health-check resolver
organizations { organizationsTest } Dedicated health-check resolver
nodes { nodesTest } Dedicated health-check resolver
murchases order(id:"__tower__") Auth-error = reachable
payments userSubscription(...) Auth-error = reachable
search searchDocs(query:"__tower__") Uses unified search index
alerts alertLog(id:"__tower__") Auth-error = reachable
experiences { myBookings { id } } Auth-error = reachable
auth { mySession { ... } } Auth-error = reachable
uchat { myConversations(...) } Auth-error = reachable. uChat uses Matrix/Conduit underneath
intelligence { forYouFeed(...) } Rust service — same tech stack as Tower
tower { __typename } Self-check via introspection
gateway { __schema { queryType { name } } } Hive Gateway introspection
imaging, video, crypto, scim { __typename } Ancillary services not in the Gateway supergraph — direct probe only

Auth-Error Acceptance

For queries requiring authentication (murchases, payments, alerts, experiences, auth, uchat), the health monitor treats responses containing "Authentication required" or "Forbidden" as UP — the presence of an auth error proves the process is alive and the GraphQL schema is composed.

Service Registry

All 34 monitored components are organized into five categories:

Category Services
Subgraphs (23) malets, products, services, ucart, media, blogs, community, organizations, nodes, murchases, payments, search, alerts, experiences, imaging, video, crypto, scim, auth, uchat, intelligence, gateway, tower
Infrastructure (3) PostgreSQL, Redis, MongoDB
Observability (5) Meilisearch, MinIO, GlitchTip, Prometheus, Grafana
Frontends (1) Frontend (SvelteKit)
Portals (2) Dev Portal, Support Center

Configuration

Variable Default Description
HEALTH_CHECK_INTERVAL 30 Probe interval in seconds
TOWER_TARGET_ENV local local for host-based probing (localhost ports), docker for container DNS

IMPORTANT

TOWER_TARGET_ENV=local is required for development. In Docker/staging, set to docker so services resolve via container names instead of localhost.

Frontend Dashboard

The PlatformHealthWidget.svelte component renders the health data as a premium admin dashboard:

Layout

  • Header: Shows cluster uptime percentage and aggregate KPI cards (Monitored / Healthy / Down)
  • Grouped Grid: Services organized by category with collapsible sections and an {n}/{total} online counter
  • Dual-Status Badges: Each subgraph card displays D (Direct) and F (Federated) micro-badges, color-coded green/red
  • Service Detail Modal: Click any card to see probe details, response times, timestamps, diagnostic message, and a link to the Developer Portal subgraph overview doc

Theming

All colors use the Mallnline design token system ($mall/tokens.css) — var(--mall-text), var(--mall-surface), var(--mall-text-secondary) — ensuring consistent rendering in both light and dark modes.

GraphQL Queries

query GetPlatformHealth {
  platformHealth {
    totalServices
    healthyCount
    unhealthyCount
    degradedCount
    uptimePercentage
    checkedAt
    services {
      name
      category
      status
      directStatus
      federatedStatus
      responseTimeMs
      federatedResponseTimeMs
      lastCheckedAt
      lastSuccessAt
      consecutiveFailures
      target
    }
  }
}

Gateway Metrics (Detail Modal)

When viewing the gateway service, additional metrics are fetched:

query GetGatewayMetrics {
  gatewayMetrics {
    totalRequests
    totalErrors
    cacheHitRatio
    apqHitRatio
    uptimeFormatted
  }
}

These metrics are sourced from the Gateway's GET /admin/health and Prometheus scrape endpoint.