Skip to main content
This page explains how HyphaeDB is operated: how the process is packaged and sized, how it shuts down cleanly, and which settings can change at runtime. The principle that organizes all of it is simple — a planned shutdown is a graceful drain, not a crash; an unplanned crash is the high availability and disaster recovery path (see /operations/high-availability and /operations/durability-and-dr).

​
Topology

A deployment is one hyphae-server container talking to an external PostgreSQL with pgvector, plus an optional TEI embedding sidecar. The server holds the in-memory HNSW mesh; PostgreSQL is the system of record. A high-availability standby is simply a second copy of the same container.

​
Embedding packaging modes

The embedding model is 400 MB–2 GB and changes far less often than the binary, so it does not belong in the app image. Three delivery modes are selected by configuration, with no application change:

​
Resource model

Resident memory is dominated by three terms: the HNSW footprint, the embedding model’s resident set, and base runtime — plus headroom:
Each L0 node costs roughly 7 KiB (a 1024-dim f32 embedding plus its L0 adjacency). The embedding model adds ~0.4–2 GiB resident (or ~0 in the server when using the TEI sidecar). Base runtime is ~256 MiB, with ~50% headroom over the steady-state estimate. The node ceiling of 1,000,000 is a memory ceiling by design: cold nodes evict from RAM above it while their PostgreSQL row stays, so resident memory is bounded independent of corpus size (see /operations/data-lifecycle). For CPU, request 2 and limit 4 cores as a V1 default, sized for 16 concurrent gossip propagations plus the RPC layer and embedding.

​
Health probes

Liveness and readiness have different jobs, and wiring them correctly is load-bearing:
  • Liveness (/healthz) returns 200 {"status":"ok"} and gates on nothing external — it stays 200 throughout rehydration, so Kubernetes never kills a pod that is correctly rebuilding its mesh.
  • Readiness (/readyz, plus grpc.health.v1) aggregates a probe set — storage (a live SELECT 1 against the pool), embedding (the configured provider, probed the same way boot coerces it), audit (Down only if the audit writer has died), and graph (the rehydration state) — and returns ready only when all are Up, so the load balancer never routes to an instance that cannot actually serve. The response body names each probe and its state, which is the first thing to read when a pod won’t go ready.
  • gRPC health mirrors the same verdict on the standard grpc.health.v1 service and is fail-closed: the overall service reports NOT_SERVING from the first instant of boot until readiness is genuinely up, and flips back to NOT_SERVING the moment a drain starts.
Probe evaluation is cached for one second with single-flight de-duplication, so an aggressive prober (or many of them) cannot amplify into a SELECT 1 storm. A startupProbe on /healthz with a generous failure threshold fully decouples liveness from a slow cold rehydration. Prometheus metrics are served from GET /metrics on the REST port when HYPHAEDB_TELEMETRY_MODE=prometheus (the compose default) — see /operations/configuration.

​
Graceful shutdown

On SIGTERM or SIGINT (a deploy, rollout, or node drain) the server runs an ordered six-step drain. The order matters — readiness flips first so the load balancer stops sending work before anything else winds down.
1

Flip readiness to 503

/readyz returns 503 and gRPC health flips to NOT_SERVING; the load balancer removes the instance from rotation while in-flight requests keep completing.
2

Drain the in-flight gossip queue

Reject new propagations but let in-flight walks finish, bounded by the drain timeout (10s default).
3

Signal connected agents (deferred)

The StreamClosing control frame is not yet implemented — today the inbox stream simply ends and the SDKs’ reconnect loop re-establishes it against the next instance.
4

Checkpoint a snapshot

When a snapshot directory is configured (HYPHAEDB_SNAPSHOT_DIR), write a mesh snapshot so the next boot replays only a small delta instead of a full scan. Best-effort: a snapshot failure is logged, never blocks the exit.
5

Release the HA lease (deferred)

A logged no-op until high availability lands — there is no lease to release in a single-instance deployment.
6

Close the pool and exit

Stop the servers gracefully (in-flight requests finish), close the PostgreSQL pool last, exit cleanly.
A planned shutdown loses no acknowledged write: deliveries are recorded synchronously and idempotently, so a drain timeout can only drop the not-yet-delivered tail of a walk, which the sweep re-drives on restart. A second SIGTERM/SIGINT during the drain escalates to an immediate exit so an operator can force a stuck drain.
terminationGracePeriodSeconds (default 30s) is sized above the worst-case drain, snapshot, and lease release. The drain timeout (10s) must be strictly less than the grace period — the server refuses to start otherwise.

​
Config reload stance

Structural config — HNSW dimensions, embedding model and provider, and the storage backend — is immutable for the process lifetime. Changing it requires the re-embed migration plus a restart. SIGHUP reloads only non-structural tunables — today that is the gossip tunables (sigma_min, k_neighbors) and the retention TTLs; log filter, trace sampling, and sweep cadence are in the spec’s reloadable set but currently rejected with an explicit restart-to-apply message rather than silently accepted. The reload is all-or-nothing: a SIGHUP whose diff touches a structural field changes nothing and logs a config error naming the offending path (never its value). See /operations/configuration.

​
Operational artifacts

The repository ships these under deploy/, each validated in CI (kubeconform, helm lint + template invariants, terraform validate, promtool):
  • Dockerfile (distroless, no model weights, non-root) and docker-compose.yml — the self-host quickstart, with a snapshot-cache volume pre-wired.
  • k8s/ raw manifests — Deployment with the TEI embedding sidecar as a native sidecar (restartPolicy: Always init container), the exact probe timings from the spec, a PodDisruptionBudget (gated on replicaCount > 1), and Services.
  • helm/hyphaedb/ — a Helm chart kept contract-identical to the raw manifests by a CI parity check.
  • terraform/aws/ and terraform/gcp/ — managed-Postgres reference stacks (RDS / Cloud SQL with pgvector, automated backups on by default).
  • marketplace/ — the AWS/GCP marketplace listing groundwork.
  • alerts/hyphaedb.rules.yml — six Prometheus alert rules over the standard metrics.
  • runbooks/ — operator runbooks: crash-and-rehydrate, PostgreSQL unreachable, OOM at the node ceiling, slow rehydration, and disaster-recovery restore.

​
Source

This page is a teaching restatement of the deployment-operations spec; that spec is authoritative for any detail here.