Observability & metrics

HyphaeDB is instrumented through the metrics facade: application crates call counter!/gauge!/histogram! and never name an exporter. Exactly one crate — hyphae-telemetry — installs the concrete recorder and subscriber, chosen once at main(). This keeps the product code backend-agnostic: switching from stdout to Prometheus to OTLP is a deployment choice, not a code change.

​

Scraping metrics

The Prometheus exporter is feature-gated so the default build pulls in no HTTP-exporter dependency:

• Build the server with --features prometheus.
• Set the observability mode to prometheus.
• The telemetry layer then mounts a /metrics scrape endpoint (via prometheus_router) that renders the current registry in Prometheus text format. Its bind/port is governed by the PrometheusConfig section — see Configuration.

Without the feature, mode = "prometheus" is a boot error (the server refuses to start rather than pretend to export), and the default mode installs a no-op/stdout recorder so metric calls are cheap and harmless.

​

The naming contract

Metric names are not free-form. hyphae-telemetry owns a canonical registry that every name must pass, and CI enforces it as a bidirectional contract (the metrics_conformance guard):

• LIVE (89 names today) — declared and emitted by live code. The guard fails if a LIVE name is never emitted, or if code emits a name that is not declared LIVE. Declaration and reality cannot drift.
• RESERVED (15 names today) — declared but not yet emitted, each with a // pending: marker naming the emit site still to be wired. The guard fails if a RESERVED name is emitted (promote it to LIVE first) — so a half-wired metric can’t sneak in undocumented.

Every name also passes a shape check (dotted hyphae.<subsystem>.<name> segments, a .total suffix for counters, .seconds/.bytes unit suffixes). This is why the catalog below can be trusted: an unmarked name in it is a name the build guarantees is emitted, and the handful of RESERVED names listed are marked as such inline.

​

Bounded labels: the cardinality rule

A metric label may only be one of a fixed allowlist of bounded, enumerable keys — layer, cell_type, transport, method, code, outcome, action, result, decision, detector, stance, scope, reason, provider, model, and a handful more. Per-entity identifiers — agent_id, tenant_id, node_id, a target — are never labels: they would make the time-series cardinality unbounded. That per-actor detail lives in the audit log and on tracing event attributes instead. The rule is enforced at runtime, not just by convention: in release builds the recorder drops a disallowed label and increments hyphae.telemetry.dropped_labels.total{metric} (naming the offending instrument), so a cardinality mistake is visible rather than a silent metrics-store blowup. A debug build panics on it, so it never reaches a PR.

​

Catalog by subsystem

A representative slice — the operationally important metrics, grouped. All names are verbatim from the registry.

​

Health role gauge

hyphae.health.role is a gauge whose value is the role — 0 standby, 1 primary, 2 fenced — not a labelled series. It pairs with High availability: a load balancer routes to a ready primary, and this gauge tells you which instance believes it holds the lease.

​

Traces

Distributed tracing exports via OTLP under --features otlp (off by default). The tracing subscriber is installed by the same telemetry layer; every audited action is also a tracing event at target = "audit". The metrics→OTLP bridge is not yet wired — metrics and traces export on independent backends today.

​

Source

This page is a teaching restatement of the observability spec and the observability-metrics-completeness spec; those specs are authoritative for any detail here.