Observability

By the end of this page you will know how to scrape ScramDB's metrics, check liveness, and configure logging, with the exact set of metric names that exist today so you do not build a dashboard around a counter that isn't there.

The metrics and health endpoint

ScramDB serves both /metrics and /health on the same HTTP listener, one port, one process.

Setting	Default	How to set it
Port	9090	[general] metrics_port in TOML, or --metrics-port <PORT> on the command line. Set to 0 to disable the endpoint entirely.
Bind address	0.0.0.0 (all interfaces)	Not configurable. If you run ScramDB without a firewall or network policy in front of it, this port is reachable from outside the host.

/metrics

curl -s http://127.0.0.1:9090/metrics | head -20

Expected output: Prometheus text exposition format (Content-Type: text/plain; version=0.0.4; charset=utf-8), starting with lines like # HELP scramdb_queries_total .... If it fails: connection refused means either the server isn't running, metrics_port is set to 0, or a firewall is blocking the port. Remember the listener binds 0.0.0.0, so "connection refused from outside the host" is more likely a firewall rule than the endpoint being loopback-only.

Any path other than /metrics or /health returns 404 Not Found.

/health

curl -s http://127.0.0.1:9090/health

Expected output: ok (plain text, no JSON, no fields, HTTP 200). There is no separate liveness/readiness distinction, this one endpoint returns ok as long as the HTTP listener thread is alive. It does not check database or storage health internally: a 200 ok proves the process and its HTTP listener are up, not that queries are succeeding. For query success/failure, use the scramdb_queries_ok / scramdb_queries_err counters on /metrics instead.

Metric reference

These are the metric names actually emitted on /metrics today.

Core query and connection metrics (always present)

Metric	Type	Meaning
scramdb_queries_total	counter	Total queries received.
scramdb_queries_ok	counter	Queries that completed successfully.
scramdb_queries_err	counter	Queries that errored.
scramdb_queries_active	gauge	Queries currently executing.
scramdb_query_avg_us	gauge	Rolling average query latency in microseconds.
scramdb_connections_total	counter	Total connections accepted since startup.
scramdb_connections_active	gauge	Currently open connections.
scramdb_bytes_sent	counter	Total bytes sent to clients.
scramdb_uptime_seconds	gauge	Seconds since the process started.

Execution-tier metrics (always present)

Metric	Type	Meaning
scramdb_chunks_jit_total	counter	Morsel-chunks executed as compiled JIT machine code.
scramdb_chunks_vm_total	counter	Morsel-chunks executed by the bytecode interpreter.
scramdb_compiled_calls_total	counter	Sub-region JIT invocations.

These are the counters used in the JIT scrape-delta technique described on Query Performance.

WAL archive metrics

Metric	Type	Meaning
scramdb_wal_archive_enabled	gauge	1 if WAL archival is enabled, 0 otherwise. Always emitted.
scramdb_wal_archive_errors_total	counter	Archive attempts that failed. Always emitted.
scramdb_wal_archive_segments_archived_total	counter	Segments successfully shipped to the destination. Always emitted.
scramdb_wal_archive_segments_behind	gauge	Segments not yet archived. Emitted only once a real measurement exists.
scramdb_wal_archive_bytes_behind	gauge	Bytes not yet archived. Emitted only once a real measurement exists.
scramdb_wal_archive_last_archived_lsn	gauge	LSN of the last successfully archived segment. Emitted only once a real measurement exists.

The last three metrics being absent from a scrape means "not measured yet," not zero. This is deliberate: the endpoint does not fabricate a 0 for a value it has not actually computed. Treat an absent metric name the same way, as unmeasured, everywhere on this endpoint, not just for WAL archive.

Resilience metrics (always present, all zero on a healthy process)

Metric	Type	Meaning
scramdb_parallel_join_timeout_total	counter	Parallel join operations that hit their timeout.
scramdb_pool_watchdog_heals_total	counter	Buffer pool watchdog self-heal actions taken.
scramdb_pool_watchdog_heal_budget_exhausted_total	counter	Times the watchdog's heal budget ran out.
scramdb_pool_watchdog_escalations_total	counter	Times the watchdog escalated beyond a self-heal.

Licensing metrics

Metric	Type	Meaning
scramdb_license_valid	gauge	1 if the active license is valid.
scramdb_license_edition	gauge/label	Active license edition.
scramdb_license_node_count	gauge	Nodes currently counted against the license.
scramdb_license_max_nodes	gauge	Node limit the license allows.
scramdb_license_read_only_active	gauge	1 if the license has forced the deployment read-only.
scramdb_license_days_of_term_remaining	gauge	Emitted only for a term-limited license.
scramdb_license_data_bytes	gauge	Emitted only for a license with a data-size cap.
scramdb_license_data_cap	gauge	Emitted only for a license with a data-size cap.

Cluster mode

Running as a cluster node, additional scramdb_cluster_* metrics are appended to the same /metrics response after the single-node metrics above. Every failure mode below has a counter: a stalled compaction, a quarantined group, a dropped partition frame. Alerting on a ScramDB cluster means reading a number, never grepping logs for a string match.

Unlike the WAL archive metrics above, every scramdb_cluster_* metric is always present once [cluster] is configured, whether or not anything interesting has happened yet. A fresh cluster reports every counter at 0 and every gauge at its idle value; there's no "not measured yet" gap to account for in this family.

Peer transport

Metric	Type	Meaning
scramdb_cluster_peer_connects_total	counter	Peer connections established over the cluster transport.
scramdb_cluster_peer_disconnects_total	counter	Peer connections closed.
scramdb_cluster_reconnect_attempts_total	counter	Reconnect attempts after a lost peer connection.
scramdb_cluster_frames_sent_total	counter	Frames sent to peers.
scramdb_cluster_frames_received_total	counter	Frames received from peers.
scramdb_cluster_bytes_sent_total	counter	Bytes sent to peers.
scramdb_cluster_bytes_received_total	counter	Bytes received from peers.
scramdb_cluster_send_queue_bytes	gauge	Outbound send-queue occupancy across all peer connections, right now. Bounded by send_queue_bytes in [cluster] config (default 64MB) per connection; a value pinned near that bound is backpressure, not idle capacity.
scramdb_cluster_partition_drops_total	counter	Frames silently dropped by an injected network partition, outbound and inbound combined. Zero on every normal deployment; nonzero means a chaos/test partition is, or was, active.
scramdb_cluster_handshake_latency_seconds	histogram	Peer handshake latency. 11 buckets from 1ms to 5s (le="0.001" through le="5.000") plus +Inf.

Raft group health and recovery

Metric	Type	Meaning
scramdb_cluster_group_quarantines_total	counter	Raft group storage directories quarantined at boot for deeper corruption: a bad header, a mid-prefix CRC failure, an undecodable hard state. Does not count a benign torn-tail truncation; that's the next row.
scramdb_cluster_quarantined_groups	gauge	Groups currently quarantined, awaiting a membership-replace rejoin. This is the line to alert on.
scramdb_cluster_group_rejoins_total	counter	Quarantined groups that completed their membership-replace rejoin.
scramdb_cluster_torn_tail_truncations_total	counter	Benign torn-tail truncations at raft log open: the normal crash-recovery path, made visible instead of silent.

scramdb_cluster_quarantined_groups is restart-honest. It is never incremented or decremented as events happen; every host-reconcile round recomputes it from scratch, from the actual set of groups gated right now, and overwrites the gauge with that count. A crash that loses an in-flight update, or a process restart mid-recovery, can't leave this gauge stuck on a stale nonzero reading or silently reset to a wrong zero: the very next reconcile round always reflects what's really quarantined at that moment, never a running total of past events.

Group commit and durability

Metric	Type	Meaning
scramdb_cluster_group_commit_ops_total	counter	Raft ops served, summed across every group-commit wake-up. Divide by group_commit_wakeups_total for the achieved batch size. A lone proposal (nothing to batch with) contributes 1 op and 0 wake-ups, so an idle system reports no batching instead of a fake 1.0.
scramdb_cluster_group_commit_wakeups_total	counter	Wake-ups that served more than one op, i.e. wake-ups that actually batched.
scramdb_cluster_group_commit_max_batch	gauge	The largest batch any wake-up has served so far, a high-water mark. Pinned at your configured cap says "raise the cap"; comfortably under it says the batching window is already draining faster than proposals arrive.
scramdb_cluster_durability_flushes_total	counter	Durability flushes, one per fsync. Divide into group_commit_ops_total for fsyncs-per-commit, the number batching exists to lower.
scramdb_cluster_log_compactions_total	counter	Raft log compactions performed.
scramdb_cluster_log_entries_compacted_total	counter	Raft log entries reclaimed by compaction, summed.

Zero log_compactions_total is normal on a cluster whose log never reaches the compaction trigger. A flat zero on a cluster that's otherwise busy (group_commit_ops_total climbing) is not normal: it means compaction is blocked, almost always a lagging replica holding the compaction boundary down, and the raft log is growing without bound. The blocked case also logs loudly on its own, but the metric is what you'd wire an alert to; see Alerting on cluster metrics below.

Adaptive routing

Metric	Type	Meaning
scramdb_cluster_routes_total	counter/label	Statements routed, labelled route.

route takes exactly one of five values: oltp, point, local_replica, forward_replica, scatter. That's the whole label space, fixed by construction, the keys are the five route kinds the planner can choose, never a node id or a group id, so this metric's cardinality can never grow with the size of the cluster. A route kind the counter doesn't recognize is dropped rather than bucketed into a sixth catch-all series, so a routing path that forgot to name itself shows up as a silent gap here, not a mislabeled entry.

Distributed shuffle (AQE)

Metric	Type	Meaning
scramdb_cluster_shuffle_consumer_fragments_total	counter	Consumer fragments dispatched for a materialized shuffle join, one per coalesce group. Equals the participant count with coalescing off; lower with it on. Zero on a streaming query.
scramdb_cluster_shuffle_aqe_coalesce_groups_total	counter	Coalesce groups actually acted on: each merged more than one small adjacent partition into a single consumer fragment.
scramdb_cluster_shuffle_aqe_skew_split_flagged_total	counter	Partitions flagged skewed (over 5x the median and over 256MB) but not acted on: skew-split off, or a single-node participant set with nothing to split across. Logged and metered even when nothing follows.
scramdb_cluster_shuffle_aqe_skew_split_acted_total	counter	Sub-consumers dispatched for a skew-split partition that was acted on: the larger side split into disjoint row ranges, the smaller side replicated. Summed across every acted partition.
scramdb_cluster_shuffle_aqe_broadcast_demote_flagged_total	counter	Broadcast sides flagged over budget, a broadcast-demote candidate. Logged and metered, not acted on yet.
scramdb_cluster_shuffle_straggler_backups_total	counter	Speculative backup consumer fragments dispatched because a primary consumer didn't return within the straggler-backup delay, on a different live node reading the same frozen spools.

Learner-read freshness

Metric	Type	Meaning
scramdb_cluster_learner_freshness_seconds	histogram	now - safe_ts_wall, observed once per bounded-stale or wait-index learner_read read (both modes). The OLTP path (learner_read off) never touches this histogram. Same buckets as the handshake histogram, 1ms to 5s plus +Inf, a natural fit for replication lag.

This is the same lag SHOW FRESHNESS reports per data group in Clustering, captured here as a scrapeable distribution instead of a point-in-time value.

Alerting on cluster metrics

Three conditions worth paging on. Each maps to a documented design decision rather than a guessed threshold:

groups:
  - name: scramdb-cluster
    rules:
      - alert: ScramDBGroupsQuarantined
        expr: scramdb_cluster_quarantined_groups > 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $value }} raft group(s) quarantined"
          description: >
            A group's storage was quarantined at boot for corruption and has
            not finished its membership-replace rejoin. This gauge is
            recomputed from the real gated set every reconcile round, not
            decremented, so restarting the node will not clear a stale
            reading; only a completed rejoin does.

      - alert: ScramDBLogCompactionStalled
        expr: |
          rate(scramdb_cluster_log_compactions_total[15m]) == 0
          and rate(scramdb_cluster_group_commit_ops_total[15m]) > 0
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "Raft log compaction has not run on a busy node in 30 minutes"
          description: >
            The node is committing but not compacting. This almost always
            means a lagging replica is holding the compaction boundary down;
            the raft log grows unbounded until that replica catches up or is
            replaced.

      - alert: ScramDBPartitionDropsDetected
        expr: increase(scramdb_cluster_partition_drops_total[5m]) > 0
        labels:
          severity: critical
        annotations:
          summary: "Cluster transport frames are being silently dropped"
          description: >
            This counter only moves when a network partition is injected
            (chaos testing) or something outside the engine is dropping
            cluster transport frames. It is zero in every normal deployment,
            so any increase is worth paging on.

What is not on this endpoint

• GPU metrics are not exposed today. There is no scramdb_gpu_* family on /metrics, even though [gpu] metrics_enabled defaults to true. See GPU Acceleration for the log-based way to confirm GPU dispatch instead.
• No buffer-pool hit-ratio, cache-hit-ratio, or per-worker-utilization metric exists. Memory and parallelism verification on this site relies on \timing and OS-level observation instead, see Memory and Parallelism.

Log levels and format

SCRAMDB_LOG=debug SCRAMDB_LOG_FORMAT=json scramdb -c scramdb-config.toml --pg-address 127.0.0.1:5432 --pg-no-auth

Expected: stderr lines shaped like {"ts":"...","level":"INFO","target":"...","msg":"Metrics endpoint listening on http://0.0.0.0:9090/metrics"}.

SCRAMDB_LOG

Controls the level filter. Accepted values (case-insensitive): off, error, warn, info, debug, trace. Default when unset: info. An unrecognized value does not fail loud, it silently falls back to info, so a typo in this env var will not show up as an error, it just quietly serves you the default level.

SCRAMDB_LOG_FORMAT

Set to json (case-insensitive) for structured JSON output. Anything else, including leaving it unset, produces the default plain-text format.

Plain-text format: [{level}] [{timestamp}] {message}, for example:

[INFO] [2026-08-02T10:15:30.123Z] Metrics endpoint listening on http://0.0.0.0:9090/metrics

JSON format: {"ts":"<timestamp>","level":"<LEVEL>","target":"<module path>","msg":"<message>"}. Only backslash (\) and double-quote (") characters in the message are escaped. No other JSON escaping is performed, so a log message containing a raw newline or control character is not escaped and can produce an invalid JSON line in that rare case. Keep this in mind if you feed the JSON log output into a strict JSON-line parser downstream.

Logging is asynchronous: a dedicated low-priority background thread drains log messages and writes to stderr, so logging never blocks a query worker. Output always goes to stderr, never directly to a file, redirect it yourself (scramdb ... 2> scramdb.log) or capture it with your process supervisor.

Related diagnostic env vars

Two additional env vars are useful for deeper diagnosis alongside SCRAMDB_LOG, both covered on other pages in this section:

• SCRAMDB_GPU_FORCE=1, forces GPU pipeline-shape eligibility for benchmarking. See GPU Acceleration.
• SCRAMDB_JIT_PERFMAP=1, emits a perf-compatible jitdump file so perf inject --jit can resolve JIT-compiled kernel symbols in a flamegraph. This is a profiling aid for deep JIT investigation, most readers tuning JIT cache size will not need it.

Next

See Query Performance for how the JIT chunk counters are used in practice, GPU Acceleration for why GPU verification uses logs instead of this endpoint, or Clustering for how the cluster counters above map onto a Docker Compose or Kubernetes deployment.