Skip to main content

Observability

By the end of this page you will know how to scrape ScramDB's metrics, check liveness, and configure logging, with the exact set of metric names that exist today so you do not build a dashboard around a counter that isn't there.

The metrics and health endpoint​

ScramDB serves both /metrics and /health on the same HTTP listener, one port, one process.

SettingDefaultHow to set it
Port9090[general] metrics_port in TOML, or --metrics-port <PORT> on the command line. Set to 0 to disable the endpoint entirely.
Bind address127.0.0.1[general] metrics_listen_address (an IP address, or "*" for every interface), or --metrics-listen-address. The shipped image and the Helm chart set "*" so a scraper can reach the pod.
Tokennone[general] metrics_bearer_token_file, or --metrics-bearer-token-file: every request then needs Authorization: Bearer <token> and is answered 401 without it. The file must not be readable by every user.

/metrics​

bash
curl -s http://127.0.0.1:9090/metrics | head -20

Expected output: Prometheus text exposition format (Content-Type: text/plain; version=0.0.4; charset=utf-8), starting with lines like # HELP scramdb_queries_total .... If it fails: connection refused means either the server isn't running, metrics_port is set to 0, or a firewall is blocking the port. Remember the listener binds 0.0.0.0, so "connection refused from outside the host" is more likely a firewall rule than the endpoint being loopback-only.

Any path other than /metrics or /health returns 404 Not Found.

/health​

bash
curl -s http://127.0.0.1:9090/health

Expected output: ok (plain text, no JSON, no fields, HTTP 200). On a cluster node whose commit protocol has stopped after a failure (scramdb_cluster_dilith_host_stopped at 1), the answer is HTTP 503 Service Unavailable with the body the commit protocol stopped on this node: that node can no longer commit transactions and must be restarted, so a load balancer or orchestrator probing /health takes it out of service. The answer is 503 too while the server's log is failed (scramdb_wal_failed above 0): the body is the log failed and takes no writes until it heals or the server is restarted, a colon and the cause the log failed with. Such a server still answers reads and takes writes again by itself once its storage works, so /health returns to 200 ok without a restart (see Storage Engine and Error codes). There is no separate liveness/readiness distinction. Beyond those two cases it does not check database or storage health internally: a 200 ok proves the process and its HTTP listener are up, not that queries are succeeding. For query success/failure, use the scramdb_queries_ok / scramdb_queries_err counters on /metrics instead.

The endpoint reports on itself on the same /metrics body, always emitted:

MetricTypeMeaning
scramdb_metrics_endpoint_requests_totalcounterRequests the metrics endpoint answered.
scramdb_metrics_endpoint_request_timeouts_totalcounterClients disconnected because their request did not arrive in time.
scramdb_metrics_endpoint_response_timeouts_totalcounterClients disconnected because they did not take their response in time.
scramdb_metrics_endpoint_client_errors_totalcounterClient connections that failed before their answer was written.
scramdb_metrics_endpoint_accept_failures_totalcounterConnections the metrics endpoint could not accept.
scramdb_metrics_endpoint_render_secondshistogramTime to render one /metrics answer.

Metric reference​

Every metric named on this page is really emitted on /metrics, so you can build a dashboard from these names without checking first. The reverse does not hold: the endpoint also carries deep diagnostic counters that are not part of the supported surface and can change between releases. The families below are the ones worth alerting on and graphing.

Core query and connection metrics (always present)​

MetricTypeMeaning
scramdb_queries_totalcounterTotal queries received.
scramdb_queries_okcounterQueries that completed successfully.
scramdb_queries_errcounterQueries that errored.
scramdb_queries_activegaugeQueries currently executing.
scramdb_query_avg_usgaugeRolling average query latency in microseconds.
scramdb_connections_totalcounterTotal connections accepted since startup.
scramdb_connections_activegaugeCurrently open connections.
scramdb_bytes_sentcounterTotal bytes sent to clients.
scramdb_bytes_receivedcounterTotal bytes received from clients.
scramdb_uptime_secondsgaugeSeconds since the process started.

Execution-tier metrics (always present)​

MetricTypeMeaning
scramdb_chunks_jit_totalcounterMorsel-chunks executed as compiled JIT machine code.
scramdb_chunks_vm_totalcounterMorsel-chunks executed by the bytecode interpreter.
scramdb_compiled_calls_totalcounterSub-region JIT invocations.
scramdb_interpreter_aggregate_top_cuts_totalcounterInterpreter aggregate partitions cut to the top groups of the query's ORDER BY with LIMIT.
scramdb_compiled_aggregate_top_cuts_totalcounterCompiled aggregate partitions cut to the top groups of the query's ORDER BY with LIMIT.

These are the counters used in the JIT scrape-delta technique described on Query Performance.

Client connections (always present)​

Every client connection holds one of [general] max_connections places while its session lives, the last superuser_reserved_connections of them kept for superusers. Every connection is served on the network threads ([network] threads, by default the CPU budget's IO share: 2 threads on a 4-core machine, at most 8 by default), each serving many connections at once. An idle session waits for its client holding no thread and no buffer (it is parked); its next message, a deadline, a notification or a close wakes it.

MetricTypeMeaning
scramdb_max_connectionsgaugeConnection places the client listener enforces (max_connections).
scramdb_connection_places_takengaugeConnection places held by sessions.
scramdb_connection_startinggaugeConnections still negotiating SSL, sending their startup packet or authenticating.
scramdb_connection_parkedgaugeIdle sessions waiting for their client with no buffer held.
scramdb_connection_wakes_totalcounterIdle sessions woken by a close, a notification or the end of a COPY.
scramdb_connection_idle_wakes_totalcounterTimes an idle session was woken by its own deadline. An idle server with no deadlines set holds it still.
scramdb_connection_refused_too_many_totalcounterConnections refused because every connection place was taken.
scramdb_connection_refused_reserved_totalcounterConnections refused because only the places reserved for superusers were left.
scramdb_connection_refused_role_limit_totalcounterConnections refused by their role's connection limit.
scramdb_connection_refused_database_limit_totalcounterConnections refused by their database's connection limit (CREATE DATABASE ... CONNECTION LIMIT).
scramdb_connection_refused_no_login_totalcounterConnections refused because their user names no role, or a role that may not log in (NOLOGIN); both FATAL 28000, as PostgreSQL.
scramdb_connection_refused_database_connect_totalcounterConnections refused because the role holds no CONNECT privilege on the database it asked for (FATAL 42501, as PostgreSQL).
scramdb_connection_refused_server_setting_totalcounterConnections refused because they asked to change a setting fixed when the server started (max_connections or superuser_reserved_connections in their startup options, FATAL 55P02).
scramdb_connection_refused_startup_permission_totalcounterConnections refused because a role without the superuser attribute asked, in its startup options, to change a superuser-only setting such as track_counts (FATAL 42501).
scramdb_connection_refused_startup_setting_totalcounterConnections refused because a setting their startup packet sent (TimeZone, DateStyle or any other) had a value the setting refuses (FATAL, the setting's own code, as PostgreSQL).
scramdb_connection_stored_setting_failures_totalcounterSettings stored for a role, a database or the server that a login could not apply and skipped.
scramdb_connection_reload_setting_failures_totalcounterSettings a configuration reload changed that an open session could not apply and kept as they were.
scramdb_connection_hba_rejects_totalcounterConnections rejected by the host-based authentication rules when accepted.
scramdb_connection_idle_session_timeouts_totalcounterSessions closed because they stayed idle outside a transaction past idle_session_timeout (FATAL 57P05).
scramdb_connection_startup_timeouts_totalcounterConnections closed because authentication did not finish within authentication_timeout.
scramdb_connection_tls_failures_totalcounterClient TLS handshakes that failed.
scramdb_connection_messages_refused_totalcounterClient messages refused because the server could not reserve memory to receive them.
scramdb_connection_ended_client_gone_totalcounterSessions whose client closed its connection without a Terminate message.
scramdb_connection_ended_socket_error_totalcounterSessions whose client connection failed.
scramdb_connection_ended_terminated_totalcounterSessions closed by pg_terminate_backend or a server shutdown.
scramdb_connection_ended_in_transaction_totalcounterSessions that ended with a transaction open, which was rolled back (its row, table and advisory locks released with it).
scramdb_connection_startup_secondshistogramTime from accepting a connection to its session being ready for queries.

The network threads themselves:

MetricTypeMeaning
scramdb_network_threadsgaugeThreads serving client connections.
scramdb_network_tasksgaugeConnection and accept tasks alive on the network threads.
scramdb_network_io_in_flightgaugeSocket operations handed to the kernel that have not ended.
scramdb_network_handoffs_totalcounterTimes a network thread's work moved to another thread because one task ran too long.
scramdb_network_long_polls_totalcounterTasks that held a network thread past the hand-off bound.
scramdb_network_io_errors_totalcounterSocket operation errors on the network threads, by error kind (kind label).
scramdb_network_backendgaugeThe IO mechanism the network threads run on, 1 for the one in use (backend label: io_uring or epoll).
scramdb_network_off_shard_totalcounterClient messages whose work ran on a query worker because it scales with the message (a large statement's parse, a describe, a login).
scramdb_network_inline_reads_totalcounterPoint reads answered on the network thread that received them, because every step of them was in memory.
scramdb_network_moved_reads_totalcounterPoint reads that took a query worker because a step of them could wait (a page to load, a lock, a first use).
scramdb_network_blocking_waits_totalcounterTimes a network thread waited on something that blocks. It stays at 0 in normal operation; if it rises, contact support with the server log.
scramdb_metrics_scrapes_refused_totalcounterMetrics requests answered 503 because the endpoint was already serving its most concurrent scrapes.

scramdb_network_handoffs_total and scramdb_network_long_polls_total stay at 0 in normal operation: statements run on the query workers, and a network thread only reads, writes and answers point operations by key. If either rises, contact support with the server log.

refused_too_many_total climbing means clients open more connections than max_connections allows: put a pooler in front, or raise the limit. idle_session_timeouts_total moves only when a session sets idle_session_timeout (off by default).

Statements (always present)​

A session hands each statement to a query worker and waits for its result; a statement timeout, a cancel, the transaction timeout and a lost client all end that wait at once.

MetricTypeMeaning
scramdb_statement_wait_secondshistogramTime a session waited for its statement's result from a query worker.
scramdb_statement_timeouts_totalcounterStatements cancelled because they ran past statement_timeout.
scramdb_statement_cancels_totalcounterStatements a client, a forced close or the memory watchdog cancelled while they ran.
scramdb_statement_waits_cancelled_totalcounterWaits inside statements that ended because their statement was cancelled, timed out or given up.
scramdb_statement_worker_kept_totalcounterStatements taken by the query worker that ran their session's previous statement. A session's statements stay on one worker while it is free, which keeps its data in that core's cache.
scramdb_statement_worker_moved_totalcounterStatements taken by a query worker other than the one that ran their session's previous statement: a session's first statement, or one whose worker was busy.
scramdb_prepared_shape_memos_totalcounterPrepared statements that took the decisions kept for their statement shape at Parse (who may run it, where it routes), so an Execute does not decide them again.
scramdb_shape_gate_judgments_totalcounterStatement privilege checks made in full because no current check of the same statement shape was kept. It moves when a shape is first seen and after a grant, a role change or a schema change.
scramdb_shape_point_classifications_totalcounterStatement shapes classified for the point-operation lane.
scramdb_shape_write_routes_totalcounterWrite routes decided and kept for their statement shape.
scramdb_merge_cancels_totalcounterStatements cancelled while merging their grouped partial results: it moves when the merge of a GROUP BY saw the cancel and stopped.
scramdb_merge_serialization_failures_totalcounterMERGE statements that ended with 40001: a REPEATABLE READ or SERIALIZABLE statement whose row another transaction changed after the snapshot (PostgreSQL answers the same), a distributed vote refused, or a changed row the engine cannot judge again. A READ COMMITTED statement over a row that changed is judged again and is not counted.
scramdb_statement_transaction_timeouts_totalcounterSessions closed because their transaction ran past transaction_timeout during a statement.
scramdb_statement_clients_gone_totalcounterSessions closed because their client went away during a statement.
scramdb_statement_late_results_dropped_totalcounterResults of statements a session had already stopped waiting for, dropped when they arrived.
scramdb_statement_implicit_blocks_totalcounterMulti-statement queries and pipelines run as one implicit transaction, as PostgreSQL runs them.
scramdb_statement_implicit_blocks_rolled_back_totalcounterImplicit transactions rolled back because one of their statements failed.

Row-by-row writes (always present)​

A write whose computed value reads the rows the statement has already written (a VOLATILE function in an INSERT, a COPY column default that counts the table) is written one row at a time instead of in batches. Every other statement keeps the batch path and counts nothing here.

MetricTypeMeaning
scramdb_dml_row_by_row_statements_totalcounterUPDATE, DELETE and MERGE statements run one row at a time because a routine they call can see their own writes.
scramdb_insert_serial_statements_totalcounterINSERT statements written one row at a time because a value they compute reads the rows they already wrote.
scramdb_insert_serial_rows_totalcounterRows written by INSERT statements that wrote one row at a time.
scramdb_copy_serial_statements_totalcounterCOPY statements loaded one row at a time because a column default reads the rows already loaded.
scramdb_copy_serial_rows_totalcounterRows loaded by COPY statements that loaded one row at a time.

An UPDATE or DELETE that finds its rows through an index and can show the order it meets them in (a row trigger, a RETURNING list, a row that can be refused) takes them in the order the table lists them (see the order of triggers and keys). A statement whose rows the index gave in that order, or that nothing can show an order to, counts nothing here.

MetricTypeMeaning
scramdb_dml_heap_ordered_statements_totalcounterUPDATE and DELETE statements whose rows were sorted into the table's order.
scramdb_dml_key_ordered_statements_totalcounterUPDATE and DELETE statements whose rows were sorted into the key order of a table other nodes hold too.
scramdb_cluster_local_key_ordered_reads_totalcounterReads of a table this node holds whole that were run in the key order the other nodes answer in.

On a cluster, a node that holds every bucket of a table together with other nodes meets the rows of such a statement in the key order, which is what a node that holds only part of the table answers in, so every node of the cluster gives one answer to one statement. The two counters above count the statements and the reads that took that order.

A write driven by a join reads its target rows again by key after the join when the join read only the columns it needed. These metrics count that completion.

MetricTypeMeaning
scramdb_join_write_completions_totalcounterJoin write batches completed with their target rows after a column-limited read.
scramdb_join_write_completed_rows_totalcounterPair rows of join writes completed with a target row.
scramdb_join_write_completed_own_rows_totalcounterPair rows of join writes whose target row is the transaction's own write.
scramdb_join_write_fetch_rounds_totalcounterRounds a cluster join write made to fetch its target rows from the nodes that hold them.
scramdb_join_write_completion_misses_totalcounterJoin write batches refused with a retryable error because a target row could not be read.
scramdb_join_write_completion_secondshistogramTime one join write batch took to complete its target rows.
scramdb_join_write_pairs_read_again_totalcounterREAD COMMITTED join writes that read their pairs again because a target row could not be read.

A rate that climbs with a bulk load means a column default or a select item calls a VOLATILE function that reads its own table: make the function STABLE if it need not see the rows written so far, and the load returns to the batch path.

Rows COPY left out (always present)​

MetricTypeMeaning
scramdb_copy_ignored_rows_totalcounterRows COPY ... FROM left out because a field did not convert under ON_ERROR ignore.

A rate that climbs means loads carry malformed rows the statements chose to skip: read the NOTICE lines of LOG_VERBOSITY verbose to find them, or drop ON_ERROR ignore and let the load fail at the first one.

Transition tables (always present)​

The rows an AFTER trigger, statement-level or row-level, reads through REFERENCING OLD TABLE or NEW TABLE stay in memory while the statement's memory budget grants them and go to disk past it.

MetricTypeMeaning
scramdb_transition_spilled_relations_totalcounterTransition tables of triggers that went past the memory budget to disk.
scramdb_transition_spilled_bytes_totalcounterBytes of transition table rows written to disk.

A rate that climbs means statements change more rows than their memory budget holds: the trigger still reads every row, the statement is slower by the file's write and read. Raise storage.execution_memory_bytes or change fewer rows per statement.

Prepared statements (always present)​

A prepared statement is parsed once, the first time a Bind needs it; every later execution hands the engine that parse with the Bind's values in place, so a statement executed many times is never parsed again.

MetricTypeMeaning
scramdb_prepared_template_parses_totalcounterPrepared statements parsed, once each, the first time a Bind needed the parse.
scramdb_prepared_handoffs_totalcounterPrepared statement executions handed to the engine already parsed.
scramdb_prepared_handoffs_unused_totalcounterPrepared statement executions whose parse the engine did not use.
scramdb_prepared_reparses_totalcounterTimes the engine parsed an already parsed prepared statement execution again.
scramdb_prepared_declared_resolutions_totalcounterPrepared statements' declared user types looked up in the catalog, at Parse and again after a type changed.
scramdb_prepared_descriptions_totalcounterPrepared statements' result descriptions derived by planning, once per statement and catalog state; every other Describe reuses it.
scramdb_prepared_descriptions_on_shard_totalcounterDescribes of prepared statements answered from their kept description without a query worker.

On a workload of prepared statements, handoffs_total grows with the executions while template_parses_total grows only with the distinct statements; handoffs_unused_total and reparses_total staying near zero confirm the parse is reused.

SQL parses (always present)​

The server reads a stored definition (a view's body, a column default, a CHECK, a generated column, a trigger's WHEN and its function's body, a row policy, an index expression, a function body) once, the first time a statement needs it after it is created or after a restart, and keeps the parsed form for every later statement, row and call in every session.

MetricTypeMeaning
scramdb_sql_parses_total{source}counterSQL texts the parser read, labeled source: statement (the statements clients send and the text the server runs for them) and definition (stored definitions, read once each).

On a steady workload {source="definition"} stays flat: it grows only when a definition is created, replaced or renamed, after a restart, or when the catalog's memory budget had to drop a parsed definition no statement had read for a while.

Plan cache (always present)​

Plans, their compiled code and plan markers are cached per statement and bounded in bytes by [optimizer] plan_cache_memory.

MetricTypeMeaning
scramdb_plan_cache_hits_totalcounterStatements that reused a cached plan.
scramdb_plan_cache_misses_totalcounterStatements that found no current cached plan and were planned.
scramdb_plan_cache_evictions_totalcounterCached plans, plan markers, plan choices, point-op templates, lowered expressions and statement shapes dropped to stay within the plan cache memory budget.
scramdb_plan_cache_bytesgaugeMemory held by cached plans, their compiled modules, plan markers, plan choices, point-op templates, lowered expressions and statement shapes.
scramdb_plan_cache_budget_bytesgaugeMemory the plan cache may hold.
scramdb_plan_cache_entriesgaugePlans, plan markers, plan choices, point-op templates, lowered expressions and statement shapes the plan cache holds.
scramdb_plans_total{kind}counterStatements planned, labeled kind: generic (a statement with parameters planned once with a slot for each, Plan Cache), custom (a statement with parameters planned with its values written in) and plain (a statement without parameters). On a steady workload of prepared statements generic and custom stay flat.
scramdb_write_lowerings_totalcounterTimes an UPDATE or DELETE had its condition, row security fence and SET list lowered from its plan. A plan lowers them once, for the first execution that meets its settings, so on a steady workload this stays flat.
scramdb_dml_expression_lowerings_totalcounterTimes an expression of a write (a VALUES cell, a CHECK, a RETURNING item) was lowered.
scramdb_dml_expression_lowering_hits_totalcounterTimes an expression of a write was taken from the lowered expressions the plan cache keeps. On a steady workload of one statement shape this rises while the lowerings stay flat.
scramdb_point_inserts_totalcounterOne-row INSERTs inside a transaction answered by the point lane, without planning. A VALUES row may omit nullable columns that have no default and may use CURRENT_TIMESTAMP, now(), LOCALTIMESTAMP or CURRENT_DATE.

Notifications (LISTEN and NOTIFY)​

LISTEN and NOTIFY report process-wide, always emitted:

MetricTypeMeaning
scramdb_notify_notifications_committed_totalcounterNotifications sent by committed transactions.
scramdb_notify_notifications_delivered_totalcounterNotifications written to listening clients.
scramdb_notify_batches_replicated_totalcounterCommitted transactions whose notifications were sent to the other nodes.
scramdb_notify_replication_failures_totalcounterCommitted transactions whose notifications could not be sent to the other nodes.
scramdb_notify_queue_full_totalcounterNotifications refused because the notification queue was full.
scramdb_notify_listening_sessionsgaugeSessions listening to at least one channel.
scramdb_notify_held_bytesgaugeBytes of notifications queued or waiting to be sent to a listener.
scramdb_notify_batches_unheard_totalcounterCommitted transactions whose notifications no session was listening to.
scramdb_notify_batches_unrouted_totalcounterNotification deliveries lost because the session had ended or the database was not known.
scramdb_notify_notifications_unlistened_totalcounterNotifications skipped because the session stopped listening to their channel first.
scramdb_notify_notifications_abandoned_totalcounterNotifications queued for a session that ended before it received them.
scramdb_notify_listeners_rewoken_totalcounterIdle sessions woken by a periodic check.

A notification's bytes are held from the execution memory budget until the last listener has received it, so held_bytes that keeps growing means a session listens and never lets its client read (a client blocked inside a long transaction receives nothing until it ends); queue_full_total counts the NOTIFY statements refused with 54000 when that budget had no room. On a cluster, replication_failures_total counts committed transactions whose notifications were not sent to the other nodes: the committing client gets a warning (08006) with its commit.

Every notification that does not reach a listening client is counted where it is lost. batches_unheard_total and notifications_unlistened_total count what PostgreSQL does not send either (nobody listened; the session stopped listening before the send). batches_unrouted_total and notifications_abandoned_total count notifications that had no listener left to receive them (a listener that ended between the commit and the delivery, a replicated batch for a database this node does not know) and each writes one warning per half minute. listeners_rewoken_total counts sessions woken by a periodic check; it stays at 0 in normal operation, and if it rises, contact support with the server log.

WAL archive metrics​

MetricTypeMeaning
scramdb_wal_archive_enabledgauge1 if WAL archival is enabled, 0 otherwise. Always emitted.
scramdb_wal_archive_errors_totalcounterArchive attempts that failed. Always emitted.
scramdb_wal_archive_segments_archived_totalcounterSegments successfully shipped to the destination. Always emitted.
scramdb_wal_archive_segments_behindgaugeSegments not yet archived. Emitted only once a real measurement exists.
scramdb_wal_archive_bytes_behindgaugeBytes not yet archived. Emitted only once a real measurement exists.
scramdb_wal_archive_last_archived_lsngaugeLSN of the last successfully archived segment. Emitted only once a real measurement exists.
scramdb_wal_archive_segments_pruned_totalcounterWAL segments removed past the archive retention window ([storage.wal.archive] retention). Always emitted.
scramdb_wal_archive_prune_failures_totalcounterArchive retention passes that failed and will retry. Always emitted; zero on a healthy archive.
scramdb_wal_archive_raw_bytes_totalcounterBytes of closed WAL segments handed to the archive, before packing. Always emitted.
scramdb_wal_archive_stored_bytes_totalcounterBytes the archive stored for those segments, after packing; the ratio of the two counters is what [storage.wal.archive] compress saves. Always emitted.
scramdb_wal_archive_pruned_bytes_totalcounterBytes of archived segments removed by the retention pass, by age or by the size bound. Always emitted.
scramdb_wal_archive_over_bound_totalcounterTimes the archive stayed past its size bound because a base backup or a branch still needs its segments, counted once for each stretch it stays past. Always emitted; zero when nothing holds the archive past its bound.
scramdb_wal_archive_over_bound_bytesgaugeBytes the archive holds past its size bound that the retention pass may not remove. Emitted once a retention pass has compared the archive with its bound.

The periodic checkpoint scheduler ([storage.wal.checkpoint], on by default) reports on the same endpoint, always emitted:

MetricTypeMeaning
scramdb_wal_checkpoint_interval_totalcounterCheckpoints the scheduler ran because its interval elapsed.
scramdb_wal_checkpoint_wal_bytes_totalcounterCheckpoints the scheduler ran because the WAL grew past its threshold.
scramdb_wal_checkpoint_behind_totalcounterCheckpoints run at full disk priority because the WAL grew past twice its threshold.
scramdb_wal_checkpoint_failures_totalcounterCheckpoints the scheduler started that failed.
scramdb_wal_checkpoint_threshold_bytesgaugeWAL bytes written since the last checkpoint that start the next one.
scramdb_wal_checkpoint_secondshistogramTime one scheduled checkpoint took.
scramdb_wal_last_checkpoint_unix_msgaugeUnix time in milliseconds of the last checkpoint that completed, 0 before the first.

The cold tier (Storage Tiering) reports on the same endpoint, always emitted; every failure counter stays at 0 on a healthy tier:

MetricTypeMeaning
scramdb_cold_segments_demoted_totalcounterSealed segments copied to cold storage.
scramdb_cold_bytes_demoted_totalcounterBytes of sealed segments copied to cold storage.
scramdb_cold_segments_evicted_totalcounterSegments whose local file was removed because cold storage holds them.
scramdb_cold_bytes_evicted_totalcounterBytes of local disk freed by segments cold storage holds.
scramdb_cold_segments_read_back_totalcounterSegments read back from cold storage to local disk.
scramdb_cold_bytes_read_back_totalcounterBytes read back from cold storage to local disk.
scramdb_cold_demotion_failures_totalcounterSegment copies to cold storage that failed and will be tried again.
scramdb_cold_read_back_failures_totalcounterSegment reads from cold storage that failed after every attempt.
scramdb_cold_read_back_retries_totalcounterSegment reads from cold storage attempted again after a failure.
scramdb_cold_read_back_mismatches_totalcounterCold storage objects that did not hold the bytes their segment's marker names.
scramdb_cold_transfer_timeouts_totalcounterTransfers to or from cold storage that ran past their deadline.

Operators past their memory (work_mem, see Memory) report their spills, always emitted:

MetricTypeMeaning
scramdb_sort_spill_runs_totalcounterSorted runs written to disk by sorts past their memory.
scramdb_sort_spill_bytes_written_totalcounterBytes of sorted runs written to disk.
scramdb_join_build_spill_partitions_totalcounterHash join build partitions written to disk to stay within their memory.

The compiled-function cache of [udf] (aot_cache_dir, aot_cache_max_bytes) reports on the same endpoint:

MetricTypeMeaning
scramdb_udf_compile_cache_bytesgaugeBytes of compiled function modules held in the on-disk compile cache.
scramdb_udf_compile_cache_limit_bytesgaugeThe most bytes the compile cache holds (aot_cache_max_bytes).
scramdb_udf_compile_cache_hits_totalcounterFunction modules loaded from the compile cache.
scramdb_udf_compile_cache_misses_totalcounterFunction modules compiled because the cache did not hold them.
scramdb_udf_compile_cache_evictions_totalcounterModules removed to keep the cache within its bound.
scramdb_udf_compile_cache_write_failures_totalcounterCompiled modules that could not be saved to the cache.
scramdb_udf_catalog_saves_totalcounterSaves of the installed modules and functions, one per INSTALL MODULE, DROP MODULE, CREATE FUNCTION and DROP FUNCTION of a module-backed language.
scramdb_udf_catalog_save_failures_totalcounterModule and function changes undone because they could not be saved (the statement failed with 58030).
scramdb_udf_catalog_modules_loaded_totalcounterInstalled modules loaded at startup.
scramdb_udf_catalog_functions_loaded_totalcounterFunctions loaded at startup.
scramdb_udf_catalog_load_failures_totalcounterSaved modules and functions that could not be loaded at startup; the server log names each.
scramdb_udf_catalog_module_files_removed_totalcounterSaved module files removed because no module uses them any more.
scramdb_udf_catalog_changes_refused_totalcounterModule and function changes refused because the saved modules and functions could not be read at startup; they stay as saved until the file is repaired or moved aside and the server restarted.
scramdb_udf_guest_abandoned_totalcounterPython and Ruby function calls left to finish alone after their statement was cancelled or timed out.
scramdb_udf_guest_abandoned_runninggaugeAbandoned Python and Ruby function calls still running; a call that would exceed the bound waits for one to end.
scramdb_udf_guest_slot_waits_totalcounterFunction calls that waited for an abandoned call to end before starting.
scramdb_udf_guest_faults_total{language}counterFunction calls whose own code failed (an exception, a trap or a breached limit), by runtime: javascript (TypeScript counts here too), python or ruby. A rate means a function to find from the statement's error and the server log. A call that was cancelled or refused for load or for memory is not counted.
scramdb_udf_epoch_ticks_totalcounterTicks of the function engine's deadline clock, which ticks only while a function runs.
scramdb_udf_epoch_ticker_wakes_totalcounterTimes the function engine's deadline clock woke for a call after being idle.

The WAL ring reports on the same endpoint, always emitted. The bound is what the memory budget reserves for it: the ring, 64 times [storage.wal] flush_threshold_bytes (sized for the machine unless pinned), and its completion slots (a sixteenth of the ring). append_waits_total rising means the WAL device is not keeping up with the write rate. A stall is the log making no progress for far longer than a flush usually takes (at least a second at the default epoch): it is reported, never treated as a failure, and every commit waiting for it completes when the device answers. An interrupted log write leaves padding in its place, which every reader skips:

MetricTypeMeaning
scramdb_wal_buffer_bytesgaugeBytes of log records waiting in the WAL ring at its last flush.
scramdb_wal_buffer_bound_bytesgaugeBytes the WAL ring and its completion slots hold, as the memory budget reserves them.
scramdb_wal_append_waits_totalcounterWAL appends that waited for a flush to make room in the ring.
scramdb_wal_flush_threshold_bytesgaugeBytes of waiting log records that start a WAL flush early, as configured or sized.
scramdb_wal_flush_stalls_totalcounterTimes the WAL made no progress for far longer than a flush usually takes.
scramdb_wal_flush_stalledgaugeWAL writers whose log has made no progress for far longer than usual right now.
scramdb_wal_padding_records_totalcounterLog records whose append was interrupted and whose place was filled with padding.
scramdb_wal_schema_records_totalcounterRecords of a table's columns written to the log. A row record names its table's columns by one of them, so the count follows new and altered tables, not rows.
scramdb_wal_flush_syncs_totalcounterData syncs of the WAL that flush rounds made, one per round that wrote log records.
scramdb_wal_flusher_wakes_totalcounterTimes the idle WAL flush thread was woken by a log append or a commit. A commit that syncs its own records wakes no one, so on a transactional workload this stays far below the commit count.
scramdb_wal_rotation_syncs_totalcounterData and directory syncs of the WAL made when a file filled and the next one began.
scramdb_storage_syncs_totalcounterData syncs the storage layer made to disk: the log, segment files and replaced files. A commit costs one, shared with the commits that land beside it.
scramdb_wal_flush_sync_secondshistogramTime one flush round's WAL data sync took.
scramdb_wal_disk_full_episodes_totalcounterTimes the WAL stopped writing because the disk had no room.
scramdb_wal_disk_full_resumes_totalcounterTimes the WAL wrote again after it had stopped for lack of room.
scramdb_wal_disk_fullgaugeWAL writers refusing writes right now because the disk has no room.
scramdb_wal_failedgaugeWAL writers refusing writes now because their log failed a write or a sync. 0 on a healthy server. While it is above 0, writes are refused (58030, see Error codes) and /health answers 503; alert on it.
scramdb_wal_healinggaugeWAL writers trying right now to write the records a failed sync left into a new file.
scramdb_wal_heal_attempts_totalcounterTimes the WAL tried to write the records a failed sync left into a new file.
scramdb_wal_heals_totalcounterTimes the WAL healed after a failed sync by writing its records into a new file.

Every sync of the WAL is counted: the sum of flush_syncs_total and rotation_syncs_total over the transactions committed in the same window is the syncs per committed transaction, and flush_sync_seconds is how long the device takes to make one flush round durable.

A failed sync does not end the log. The records the failed sync had not made durable are written again into a new file at once, with one warning in the server log, and the commit that waited for them answers as usual (scramdb_wal_heal_attempts_total and scramdb_wal_heals_total each count one). When that cannot be done either, scramdb_wal_failed reads 1, the server logs one error line and the log tries again on a pause that starts at 100 milliseconds and doubles up to 10 seconds, with scramdb_wal_healing at 1 during each try. When a try succeeds the log takes writes again and scramdb_wal_failed returns to 0. A rising scramdb_wal_heals_total is a storage device that failed a sync and recovered: look at the device even when no statement was refused. A log that cannot be repaired reads scramdb_wal_failed at 1 until the server is restarted and logs one error line with the cause; the commits that waited answer 08007 and every later write 58030, so no commit hangs behind it. A log that is only slow is scramdb_wal_flush_stalled above 0: nothing has failed and commits wait for the device.

Catalog file writes (each table's catalog file and deleted-row record, the table list, sequences, databases and the commit timestamp mark) report on the same endpoint, always emitted. Concurrent changes to one file share one write, so coalesced_total rising while writes_total stays flat under heavy DDL or checkpointing is expected:

MetricTypeMeaning
scramdb_catalog_writes_totalcounterCatalog files written.
scramdb_catalog_write_failures_totalcounterCatalog file writes that failed. Zero on a healthy disk; each failure is returned to the statements that asked for the write.
scramdb_catalog_coalesced_totalcounterCatalog changes made durable by a write another caller did.
scramdb_catalog_write_secondshistogramTime one catalog file write took (write, sync, rename, directory sync).
scramdb_catalog_column_records_dropped_totalcounterTable column records a checkpoint dropped from the table entries.
scramdb_catalog_column_records_keptgaugeTable column records the table entries held after the last checkpoint.

The segments_behind, bytes_behind and last_archived_lsn metrics being absent from a scrape means "not measured yet," not zero. This is deliberate: the endpoint does not fabricate a 0 for a value it has not actually computed. Treat an absent metric name the same way, as unmeasured, everywhere on this endpoint, not just for WAL archive.

Disk bytes​

Every byte the store keeps on disk is booked by class, by the same book the scram_disk view reads, so a scrape, a query and the files agree. The classes are table_segments (table segment files), wal, archive (archived log segments), base_backups, branch_staging, vector_indexes, spill, caches, raft_log (the replicated log of a cluster node), logs (the file the server's output goes to: the background server's log or a redirect; absent when the output is a terminal or a pipe) and metadata. A census walks the data directory once a minute ([storage.disk] census_interval) and corrects a class whose books differ from its files, counting each correction; a figure that has not been measured yet is absent from the scrape, never 0.

MetricTypeMeaning
scramdb_disk_bytes{class}gaugeBytes on disk by class.
scramdb_disk_limit_bytes{class}gaugeBytes a class may hold before it is reported over its bound. Absent for a class with no bound derived yet.
scramdb_disk_over_bound{class}gauge1 while a class holds more than its bound, otherwise 0.
scramdb_disk_over_bound_episodes_total{class}counterTimes a class went past its bound.
scramdb_disk_corrections_total{class}counterTimes the census found a class's books disagreeing with its files and corrected them. If it rises, contact support with the server log.
scramdb_disk_correction_bytes_total{class}counterBytes the census moved when it corrected a class's books.
scramdb_disk_database_bytes{database_oid}gaugeBytes of table files each database holds. The archive is shared by every database and is counted in its own class, not here.
scramdb_disk_volume_available_bytesgaugeBytes free on the volume holding the data directory at the last census.
scramdb_disk_volume_pressure_episodes_totalcounterTimes the volume could not hold every class at its bound.
scramdb_disk_census_age_secondsgaugeSeconds since the census last finished.
scramdb_disk_census_last_microsgaugeMicroseconds the last census took.
scramdb_disk_census_runs_totalcounterCensuses completed.
scramdb_disk_census_errors_totalcounterCensuses that could not read part of the data directory.
scramdb_disk_census_table_corrections_totalcounterTables whose books the census corrected to the files they hold.
scramdb_wal_pins_heldgaugePins held now that keep log segments from being reclaimed: a branch at a past time staging its replay. Always emitted.
scramdb_wal_pin_oldest_age_secondsgaugeSeconds the oldest pin has been held. Absent while no pin is held.
scramdb_wal_pins_lapsed_totalcounterPins released because their holder stopped renewing them within [storage.disk] pin_lease. Zero on a healthy server; a rise means a branch staging stopped without letting go, and the pin it held was released so it could not keep the log from being reclaimed.

Only the archive is held to its bound by the server, which removes the oldest segments no base backup or branch needs once [storage.wal.archive] max_size (a quarter of the volume unless set) is passed; an archive in an object store is pruned by age alone. The other classes report over_bound and the episode counters; none of them is trimmed for being over. A branch whose pin lapses fails with the reason instead of finishing on segments that may be gone.

A table's bytes and the archive's move in the same step as their files; the log, branch staging, vector indexes and the other classes are read from the files by the census, so they trail a write by up to one census interval. A dropped table or database frees its files at once, and its bytes leave the books in the same step, unless a clone or a snapshot still reads them; those bytes stay on table_segments until that reader is gone.

scram_disk answers the same figures in SQL, one row per class, one per database that holds table files, and one for the free space of the volume:

ColumnMeaning
scopeclass, database or volume.
nameThe class, the database's name, or available for the volume.
bytesBytes held, or free for the volume. NULL until measured.
limit_bytesThe class's bound. NULL for a database, for the volume and for a class with no bound derived yet.
over_boundWhether a class holds more than its bound. NULL for a database and the volume.
episodesTimes a class went past its bound. NULL for a database and the volume.
sql
SELECT name, bytes, limit_bytes, over_bound FROM scram_disk WHERE scope = 'class' ORDER BY bytes DESC NULLS LAST;
SELECT name, bytes FROM scram_disk WHERE scope = 'database' ORDER BY bytes DESC;

A superuser or a member of pg_read_all_stats (which pg_monitor includes) sees every row. Any other role sees one database row, the size of the database it is connected to, and no class or volume row, as PostgreSQL gives a database size only to a role that may connect to it and keeps the rest of the node's statistics for pg_read_all_stats.

The archive and base backup sizes are on the node's own disk. A cluster node's view and scrape describe that node's disk; a learner's store keeps its own books and is not on them yet.

Object store and function IO threads​

The object store client (the s3://, gs:// and az:// destinations of the WAL archive, base backups, restores, branch staging and the cold tier) makes its requests on the thread that needs them, so it starts no thread of its own. Reads of remote files over http:// and https:// for an external table run on a one or two thread pool that starts with the first such read, so a query worker never waits on the network. The file, clock and network calls user functions make run on a small pool of worker threads of their own, which starts with the first function that needs it. Always emitted: zero until a pool starts.

MetricTypeMeaning
scramdb_object_store_io_threadsgaugeThreads serving remote file reads for query workers.
scramdb_udf_io_workersgaugeWorker threads serving functions' file, clock and network calls: the network runtime's shards, which also serve client connections.
scramdb_udf_io_busy_microseconds_totalcounterTime the network runtime's shards spent running work, functions' IO and client connections together.

Every object store request is counted by backend (local, memory, http, s3, gcs, azure) and operation, with a series emitted once a request of that kind was made:

MetricTypeMeaning
scramdb_object_store_requests_total{backend,op,outcome}counterObject store requests by backend, operation (put, put_if_absent, get, get_range, head, list, list_dirs, delete, begin, part, complete, abort) and outcome (ok, not_found, already_exists, precondition_failed, denied, throttled, transport, server, invalid, io).
scramdb_object_store_request_seconds{backend,op}histogramTime of object store requests.
scramdb_object_store_bytes_in_total{backend}counterBytes read from object stores.
scramdb_object_store_bytes_out_total{backend}counterBytes written to object stores.
scramdb_object_store_http_requests_total{backend}counterHTTP requests sent to object stores, retries included.
scramdb_object_store_retries_total{backend}counterRequests sent again after a failure that answered nothing.
scramdb_object_store_timeouts_total{backend}counterRequests that waited past their bound.

A busy counter's rate divided by its worker gauge is that pool's utilization.

Every upload and download through the object store client is counted, and so is every HTTP read of a remote file. A request is bounded: 5 s to connect and 30 s of silence. A read that got no answer (a connection that did not open or broke, a name that did not resolve, a timeout, or a 408, 429, 500, 502, 503 or 504) is sent again, up to 10 times within 180 s, backing off from 100 ms to 15 s; an object store request keeps the same bounds. Always emitted: zero until the first request.

MetricTypeMeaning
scramdb_object_store_uploads_total / scramdb_object_store_upload_failures_totalcounterUploads to the object store, and the ones that failed.
scramdb_object_store_uploads_aborted_total / scramdb_object_store_abort_failures_totalcounterUnfinished multipart uploads aborted after a failure, and the aborts that failed (the object store keeps those parts until its own cleanup).
scramdb_object_store_uploaded_bytes_totalcounterBytes uploaded.
scramdb_object_store_downloads_total / scramdb_object_store_download_failures_totalcounterDownloads from the object store, and the ones that failed.
scramdb_object_store_downloaded_bytes_totalcounterBytes downloaded.
scramdb_object_store_upload_seconds / scramdb_object_store_download_secondshistogramTime an upload or a download took.
scramdb_remote_file_requests_total / scramdb_remote_file_request_failures_totalcounterHTTP requests for remote files, and the ones that failed after their retries.
scramdb_remote_file_request_retries_totalcounterHTTP requests for remote files sent again.
scramdb_remote_file_request_timeouts_totalcounterHTTP requests for remote files that got no answer within their bounds.
scramdb_remote_file_bodies_opengaugeRemote file answers being read now, each holding one connection.

Server stalls (always present, all zero on a healthy process)​

A sampler reads the statement counters once a second. The server is stalled when statements are in flight and none has finished for longer than a bound that follows the finishing pace the server itself has shown (between 15 and 60 seconds), and either at least two statements are in flight or one waits for a memory grant or a lock. A single long statement that waits on nothing is not a stall. The log carries one warning when a stall begins, naming how many statements wait and the longest-running session, and one line when it ends.

MetricTypeMeaning
scramdb_server_stalls_totalcounterTimes statements were in flight and none finished for longer than the stall bound.
scramdb_server_stalled_secondsgaugeSeconds the server has had statements in flight with none finished, 0 when it is not stalled.
scramdb_server_stall_bound_millisecondsgaugeTime with no statement finished after which the server reports a stall.
scramdb_timer_registrations_queuedgaugeTimer registrations waiting for the timer thread to take them. Near 0 on a healthy server; if it keeps growing, contact support with the server log.

Error shapes (always present)​

MetricTypeMeaning
scramdb_error_shapes_totalcounterStatements answered with a recognized error shape, by shape.

The shape label is one of disk_full, spill_unavailable, prepared_corrupt, prepared_busy, prepared_rollback_only, prepared_unsettled, prepared_commit_pending, prepared_storage, cluster_in_doubt, wire_version_refused, generated_column_write, set_list_misuse, update_subselect_unsupported, branch_statement_unsupported, cursor_misuse, routine_variable_misuse, routine_transaction_termination, routine_type_reference, from_clause_entry, duplicate_table_name, untyped_array, merge_row_twice, clone_not_ready and internal. It counts only errors one classifier recognizes, never every error: a statement that fails any other way is not counted here.

Resilience metrics (always present, all zero on a healthy process)​

MetricTypeMeaning
scramdb_parallel_join_timeout_totalcounterParallel join operations that hit their timeout.
scramdb_pool_watchdog_heals_totalcounterTimes the pool watchdog woke a worker thread that seemed stalled.
scramdb_pool_watchdog_samples_totalcounterSampling passes the pool watchdog made over its workers. While work is outstanding it makes one about every 125 ms; an idle pool holds it still.
scramdb_pool_watchdog_heal_budget_exhausted_totalcounterTimes the watchdog's heal budget ran out.
scramdb_pool_watchdog_escalations_totalcounterTimes the watchdog escalated beyond a self-heal.
scramdb_pool_idle_wakes_totalcounterTimes an idle worker thread woke from its wait to settle owed cleanup and found no task. A pool with nothing to do holds it still.

Crashes and fatal signals​

A statement that fails on an unexpected condition ends alone, never the server. It answers XX000 with a sentence pointing to the server log (internal error: ...; the server log has details), the memory it had reserved is released, and the connection and the worker go on. The log carries one error line with the statement's shape: the session, a fingerprint, and the start of the statement with every literal replaced by ?, so no value a client sent reaches the log. A statement nested too deep for the stack, whether in the parser, in an expression, in a chain of views, in routine calls or in an engine rewrite, answers 54001 (stack depth limit exceeded) instead; every thread that runs query work has a 64 MiB stack and a deeper descent continues on a new segment, counted as stack_segments on scramdb_memory_charged_bytes, so the depth a statement may reach is the session's max_stack_depth, not the size of a thread's stack.

MetricTypeMeaning
scramdb_panics_caught_total{site}counterPanics that ended one statement or job instead of the process, by where they were caught: statement, connection, worker, copy_loader, compiled_call, fan_out (a fragment of a distributed statement, including a request served for another node) or background.
scramdb_stack_depth_refusals_total{site}counterStatements ended with a stack depth limit error instead of crashing, by the guard that refused them: parse, expression (an expression, or a chain of views, nested too deep), rewrite, routine or call.
scramdb_fatal_signals_previous_rungaugeFatal signals the previous run of this server ended on, found at this start.
scramdb_core_dump_limit_bytesgaugeLargest core dump the process may write, in bytes; 0 when core dumps are off.

scramdb_panics_caught_total stays at zero in normal operation; if it rises, contact support with the server log. A growing scramdb_stack_depth_refusals_total means clients send nesting the server has to refuse; the site label says which kind.

A fatal signal (SIGSEGV, SIGBUS, SIGILL, SIGFPE, SIGABRT) still ends the process, but leaves a record first. The signal, the faulting address, the thread, the statement shape and a stack go to the log and to fatal-signal.log in [storage] basedir. The next start finds the file, logs one error naming the signal, keeps it as fatal-signal.previous.log and reports it in scramdb_fatal_signals_previous_run, so a restart never hides that the previous run was killed. The release binary is stripped, so the stack lists module and offset; resolve them with addr2line against the unstripped binary of the same build. core_dump_limit_mib in [general] (default 1024, 0 turns core dumps off) bounds the core the kernel may write, so a crash of a large process cannot fill the data volume and starve the restart. See Configuration file.

Query execution and JIT​

MetricTypeMeaning
scramdb_jit_compile_submits_totalcounterCompilation requests handed to the JIT worker. On a warm, repeating workload this should stop growing; continued growth means the compiled-artifact cache is not serving and is worth raising compiled_cache_memory for.
scramdb_chunks_jit_t1_totalcounterChunks served by tier 1 compiled code, which is the fast-to-produce tier or a cached artifact loaded from disk.
scramdb_chunks_jit_t2_totalcounterChunks served by tier 2 compiled code, the fully optimized tier.
scramdb_jit_unavailablegauge1 when the JIT is enabled but this server's LLVM cannot compile for its CPU: statements then run on the bytecode interpreter alone. 0 otherwise.
scramdb_jit_t1_compile_secondshistogramTime to compile one module for a quick start (tier 1).
scramdb_jit_t2_compile_secondshistogramTime to compile one module fully optimized (tier 2).
scramdb_jit_jetz_load_secondshistogramTime to load one compiled module from the on-disk compile cache.
scramdb_vm_chunk_rows_totalcounterRows in the batches run by the interpreter.
scramdb_vm_chunk_nanoseconds_totalcounterTime spent running batches in the interpreter, in nanoseconds.
scramdb_jit_chunk_rows_totalcounterRows in the batches run as compiled machine code.
scramdb_jit_chunk_nanoseconds_totalcounterTime spent running batches as compiled machine code, in nanoseconds. A statement is compiled the first time it runs when its rows times the measured difference per row exceed the measured compile time, and otherwise the second time.
scramdb_jit_jetbc_load_secondshistogramTime to load and compile one cached intermediate module from the on-disk compile cache.
scramdb_jit_verifier_rejections_total{tier}counterCompiled kernels the code verifier rejected, labelled tier (fused, whole_function, expression_region, module). The statement still runs on the interpreter. It stays at zero in normal operation; if it rises, contact support with the server log.
scramdb_udf_interpreter_warmups_totalcounterInterpreted function runtimes started, by language (python, ruby): each starts once per process, the first time a function of that language runs.
scramdb_jit_compile_memory_charges_totalcounterCompiles whose working memory was charged to the compiled-code cache budget.
scramdb_jit_compile_memory_in_flight_bytesgaugeWorking memory held by the compiles running now. It counts against compiled_cache_memory together with the cached code, and cached code is evicted to make room for it.
scramdb_jit_cache_bytesgaugeBytes the compiled-code caches hold now: their bytecode, the compiled code and the running compiles. The compiled code and the compiles are measured from the native heap where the system reports it, so the figure is what the budget is enforced on. Read it against scramdb_jit_cache_budget_bytes.
scramdb_jit_cache_budget_bytesgaugeThe budget of the compiled-code caches (compiled_cache_memory).
scramdb_jit_cache_hits_totalcounterLookups the compiled-code caches answered from a cached entry.
scramdb_jit_cache_misses_totalcounterLookups that found no cached entry and installed a new one.
scramdb_jit_cache_evictions_totalcounterCached entries the compiled-code caches evicted to stay within their budget.
scramdb_jit_disk_cache_pruned_files_totalcounterCompiled-code files removed from disk, oldest first, to stay within compiled_cache_disk.
scramdb_jit_compiles_in_flightgaugeCompiles running now.
scramdb_jit_compile_memory_over_budget_totalcounterCompiles that started while the running compiles alone needed more than compiled_cache_memory. A compile is never refused or delayed for memory, so a growing count means the budget is too small for the compile concurrency: raise compiled_cache_memory or lower jit_compile_threads.
scramdb_jit_compile_threadsgaugeThreads of the query compile pool ([execution] jit_compile_threads, one to eight). They run on the query cores below the query threads' priority, never on the cores a cluster node reserves for its own traffic.
scramdb_jit_compile_queuedgaugeQuery compiles waiting for a compile thread.
scramdb_jit_compile_queued_bytesgaugeBytes held by the query compiles waiting for a compile thread. Bounded by compiled_cache_memory.
scramdb_jit_compile_refused_totalcounterQuery compiles not queued because the compile queue was full. The statement runs on the interpreter, never waits, and is compiled on a later run once the queue has room; a steadily growing count means compiles cannot keep up with new statement shapes: raise jit_compile_threads or compiled_cache_memory.
scramdb_vm_agg_flushes_totalcounterInterpreter-side grouped aggregate tables flushed as partial results on reaching their memory budget.
scramdb_preagg_flushes_totalcounterPer-worker aggregate tables flushed as partial results for exceeding their share of the execution budget.
scramdb_preagg_merges_on_caller_totalcounterSmall pre-aggregate merges run on the calling thread.
scramdb_preagg_merge_empty_lanes_totalcounterEmpty pre-aggregate lanes skipped at merge.
scramdb_preagg_merged_sources_totalcounterWorker partials folded together by pre-aggregate merges. Compare its rate with the pre-aggregate merges: a higher figure means merges fold several partials, which is a parallel merge at work.
scramdb_agg_resident_bytesgaugeLive aggregate hash-table bytes: buckets, key slots, tags and string arenas.
scramdb_join_ordering_cartesian_fallbacks_totalcounterTimes the planner could not connect a join graph and fell back to emitting a cross product. A query whose SQL does not ask for a cross join leaves this at zero.
scramdb_split_distinct_rewrites_totalcounterDISTINCT aggregates the planner split into two aggregation levels.
scramdb_probe_side_materializations_totalcounterOuter-join probe sides that could not stream and were written to a temporary relation first.
scramdb_join_input_materializations_totalcounterJoin inputs of any position that could not stream and were written to a temporary relation first. Falling over time on a fixed workload is the plan getting better.
scramdb_stream_topk_rows_totalcounterRows accepted by the streaming top-K sink serving ORDER BY ... LIMIT.
scramdb_topk_threshold_publishes_totalcounterTop-K thresholds published into the scan so it can skip pages that cannot beat the current best rows.
scramdb_late_fetch_rows_totalcounterRows served by late materialization: a narrow scan that fetches the remaining columns only for rows that survived.
scramdb_stream_queue_bytesgaugeBytes of worker results queued ahead of the client, bounded by backpressure.
scramdb_stream_gate_forced_admits_totalcounterResults pushed past their statement's budget because the queue was empty and there was nothing left to drain.

Startup​

MetricTypeMeaning
scramdb_startup_subsystems_pendinggaugeSubsystems a client can reach (vector indexes, the memory watchdog, the function engine) that are not ready yet. The PostgreSQL port accepts no client until this is zero.
scramdb_startup_listener_wait_microsgaugeMicroseconds the client listener waited for startup to finish before accepting clients. Zero on an ordinary boot, where every subsystem is ready before the port is opened.
scramdb_recovery_redo_divergence_quarantines_totalcounterTables quarantined during boot's log replay because their data could not be verified. Quarantining one table never stops any other table from booting; a nonzero count here means one table needs a look before its data is trusted again.
scramdb_recovery_redo_divergence_quarantines_persisted_totalcounterThe same divergence, found by a bounded replay (a restore or a branch at a past time) rather than an ordinary boot: the table is quarantined and the quarantine is written to disk, so every later reopen of the restored or branched directory still finds it quarantined, not just the process that ran the replay. A nonzero count here means a restore or a branch printed at least one quarantined table in its own output.
scramdb_boot_tables_quarantinedgaugeTables the last boot left quarantined. It is the boot's count and does not fall when a table is dropped or recreated; scram_quarantined_tables is the live list. Alert on any.
scramdb_boot_empty_segment_files_recreated_totalcounterEmpty active data files recreated at boot because a power cut took back their directory entry. Expected after a power loss, nothing was lost.
scramdb_boot_empty_segment_file_failures_totalcounterEmpty active data files boot could not recreate; the table's first write then answers with the real error. Alert on any.
scramdb_boot_lost_segment_files_rebuilt_totalcounterData files missing at boot that the log's page images rebuilt. Nonzero after a power loss or a lost file: the table was served, no row was lost.
scramdb_boot_quarantines_lifted_by_log_totalcounterTables held aside at boot for a short or missing data file and served again once the log named the segment.
scramdb_boot_orphan_segment_filesgaugeData files the last boot found on disk that no table lists and no clone shares. The boot only counts them and removes nothing; a nonzero value after a clean run is worth a look.
scramdb_boot_orphan_segment_bytesgaugeBytes of those data files.
scramdb_boot_orphan_table_dirsgaugeTable directories the last boot found that no table uses and no clone lists from.
scramdb_boot_orphan_scan_shortgauge1 when the last boot could not read a directory while counting unlisted files, so the three orphan figures above may be short: read them as a floor, never as zero.
scramdb_boot_leftover_files_removed_totalcounterTemporary files a crash left in the directories of the tables the catalog lists, which the boot removed. Expected after a kill or a power cut.
scramdb_table_repairs_started_totalcounterTable repairs from a peer that began on this member (a cluster member that lost a table's files refills the table from a healthy member). A restart in the middle of a repair counts a second start. A learner's copy of the table is repaired the same way.
scramdb_table_repairs_completed_totalcounterTable repairs from a peer that finished and left the table whole.
scramdb_table_repairs_failed_totalcounterTable repairs from a peer that ended without finishing and were left for the next try. Alert on a rising rate: the member serves none of the group's reads until the copy lands.
scramdb_table_repairs_in_flightgaugeTables on this member that wait for their rows from a peer. Above zero for long means no peer answers the copy.
scramdb_table_repairs_blockedgaugeShard groups on this member whose quarantined tables cannot be repaired from a peer yet: no other member holds the group, a sibling group of the same table has none, or the group's own repair record cannot be read. The table stays quarantined and a damaged record stops its own group only. Alert on any.
scramdb_quarantined_tables_rebuilt_totalcounterQuarantined tables emptied so a peer could refill them.

No statement fails because a subsystem is still being installed: a client whose connection reaches the listener before startup has finished waits and is served once it has.

A table quarantined at boot - for this reason or for a segment whose file plainly diverges from its catalog entry and cannot be rebuilt from the log - refuses every access (scan, insert, delete, VACUUM, ANALYZE) with an error naming the table and the segment, until it is restored from a backup or dropped and recreated (a cluster member whose peers hold the table refills it from them by itself, see scramdb_table_repairs_in_flight). Boot says so once, on one line, naming each such table with its database; querying the scram_quarantined_tables system view (columns database, table_name, segment_id, reason) lists every table currently quarantined in every database, and why, to a superuser or a member of pg_read_all_stats; any other role sees the tables of its own database only.

Scans and storage​

MetricTypeMeaning
scramdb_fsync_skipped_totalcounterFlushes to disk skipped because fsync is off. Always emitted; zero on a server that flushes.
scramdb_async_commits_totalcounterCommits answered before their log flush because their session set synchronous_commit to off. Always emitted; zero when no session does.
scramdb_index_delete_lock_acquisitions_totalcounterBatches of entries removed from ordered and hashed indexes. Index reads never wait on these removals.
scramdb_index_delete_entries_totalcounterEntries asked to be removed from ordered and hashed indexes.
scramdb_index_read_gate_raised_totalcounterIndex key reads that held the index's writers back to finish: a read of a key that writers keep changing holds back changes to that index until it has read the key once. Rare outside heavy writes on one key.
scramdb_index_read_gate_writer_waits_totalcounterIndex changes that waited for such a key read to finish. Each wait lasts one read of the key.
scramdb_index_rebuilt_at_open_totalcounterPersisted indexes rebuilt from their table's rows at open, never from stored key bytes. An ordinary index (ART, B-tree, hash, GIN) keeps no on-disk byte format: every server startup and every CoW branch fork rebuilds it from the current rows, so a key-encoding change can never leave a stale entry behind.
scramdb_index_rebuild_failed_at_open_totalcounterIndexes whose rebuild at open failed; each is served scan-only until REINDEX repairs it.
scramdb_index_reindexed_totalcounterIndex rebuilds finished by REINDEX on this node: one per ordinary index and one per part of a vector index. REINDEX SCHEMA and REINDEX DATABASE add one per index of every table they name, REINDEX SYSTEM none. On a cluster REINDEX of a distributed table's index, table, schema or database runs on every node, so each node's own count moves.
scramdb_table_clustered_totalcounterTable rewrites finished by CLUSTER on this node: one per table, one per partition for a partitioned table, one per marked table for CLUSTER with no table. ALTER TABLE ... CLUSTER ON and SET WITHOUT CLUSTER move a mark and count nothing, a refused statement counts nothing, and a rewrite a transaction block later rolls back was finished and is counted. On a cluster, CLUSTER of a distributed table runs on every node, so each node's own count moves.
scramdb_zone_pages_considered_totalcounterPages evaluated against zone-map predicates.
scramdb_zone_pages_pruned_totalcounterPages skipped by zone-map pruning and never read. Against considered, this is your page-pruning hit rate.
scramdb_selective_pages_read_totalcounterPages served by reading only the projected columns rather than the whole page.
scramdb_selective_bytes_saved_totalcounterBytes that column-selective reading did not have to fetch.
scramdb_selective_ranges_read_totalcounterRead requests column-selective reading issued. Divided by selective_pages_read_total it is the number of disk reads per page.
scramdb_read_pool_active_unitsgaugeUnits currently holding a read-buffer-pool slot, either waiting for concurrency or actively holding pages.
scramdb_read_pool_unit_waits_totalcounterTimes a unit waited noticeably long for a read-pool concurrency slot. This is a throughput signal, not a safety one: page acquisition itself is never queued.
scramdb_page_pool_magazine_sheds_totalcounterPage returns under memory pressure that emptied a thread's idle read buffers into the shared pool. It rises while the server is reclaiming idle per-thread buffers in a pressure episode; it stays flat when the threads holding idle buffers made no page traffic of their own.
scramdb_row_fetches_totalcounterFetches of rows by their row id, for index probes, row re-reads and re-ranking.
scramdb_row_fetch_rows_totalcounterRows returned by fetches of rows by their row id.
scramdb_row_fetch_pages_totalcounterPages read by fetches of rows by their row id.
scramdb_row_fetch_page_bytes_totalcounterBytes of the pages read by fetches of rows by their row id.
scramdb_row_fetch_decoded_bytes_totalcounterBytes decoded from those pages by fetches of rows by their row id. Against page_bytes_total, how much of each page a fetch had to decode.
scramdb_row_place_arrays_loaded_totalcounterMerged segments whose row places were read from disk. A row keeps its id when a compaction moves it: each merged segment keeps the ids of its rows beside it, and the first read by id of one this server did not write since it started reads them once (memory family row_places).
scramdb_row_id_reads_relocated_totalcounterReads by row id that ran again because a compaction swapped the table's segments while they read. Each runs again at once; one whose table keeps changing under it for longer than [resilience] deadlock_timeout (configuration file) ends with 40001, which the client retries.
scramdb_values_out_of_line_written_totalcounterValues too long for their page written out of line, in a chain of pages of their own.
scramdb_value_chain_pages_written_totalcounterPages written to hold values stored out of line.
scramdb_values_out_of_line_bytes_written_totalcounterBytes of the values written out of line.
scramdb_value_chain_bytes_stored_totalcounterBytes stored for values written out of line, after compression. Against bytes_written_total, the compression those values got.
scramdb_values_out_of_line_read_totalcounterValues stored out of line read back.
scramdb_value_chain_pages_read_totalcounterPages read to read values stored out of line.
scramdb_values_out_of_line_bytes_read_totalcounterBytes of the values stored out of line read back.

Rows fetched by address are counted by each query worker on its own, so the counters cost nothing measurable however many workers fetch. A text, binary, vector or array value longer than fits its page is stored out of line, in a chain of pages of its row's segment, and read back only by the rows that need it; the out-of-line counters are zero on a workload with no such values.

Segment files and checkpoints (always present)​

How data files are referenced, flushed and removed, and how a CHECKPOINT walks the tables. All zero or unchanged on a process that has not written, cloned or checkpointed.

MetricTypeMeaning
scramdb_segment_files_referencedgaugeData files with at least one reference, from a table, a clone or a running scan. A count of files; the bytes are scramdb_segment_bytes_referenced.
scramdb_segment_files_sharedgaugeData files referenced more than once, by a clone or a running scan.
scramdb_segment_bytes_referencedgaugeBytes, in whole pages, of the data files that have been filled and at least one reference holds, each file counted once. The file a table is still writing counts as a file and adds no bytes until it is full. Chart it beside the data directory size.
scramdb_segment_bytes_sharedgaugeBytes of those files that more than one table lists: the disk a clone shares with its parent instead of copying.
scramdb_segment_files_freed_totalcounterData files removed because their last reference was released.
scramdb_segment_bytes_freed_totalcounterBytes of the data files removed because their last reference was released. Read it beside the data directory size.
scramdb_segment_bytes_kept_totalcounterBytes of data files a drop or truncate left in place because a clone or a running scan still uses them. Expected with clones.
scramdb_segment_deletions_deferred_totalcounterReleases that kept the file because another reference remains. A clone or a scan holding a parent's file shows here.
scramdb_segment_double_releases_totalcounterUnexpected releases of data file references. The file is kept. If it rises, contact support with the server log.
scramdb_segment_untracked_releases_totalcounterUnexpected releases of data file references. The file is kept. If it rises, contact support with the server log.
scramdb_segment_unreferenced_listings_totalcounterReads and clones that were stopped because the server could not account for a file of the table. Restart the server; if it happens again, contact support with the server log.
scramdb_segment_releases_deferred_totalcounterSegment releases that waited for the statements holding their table's rows to finish.
scramdb_segment_swaps_superseded_totalcounterCompactions whose result a drop, truncate or rename took over after a failed save. The one that took over releases the replaced files at once (scramdb_segment_superseded_released_total); if its own write failed they stay on disk and the next boot counts them.
scramdb_segment_superseded_released_totalcounterData file references released for a compaction that a drop, truncate or rename took over. Against scramdb_segment_swaps_superseded_total it should keep pace; a gap is a takeover whose own write failed.
scramdb_segment_file_syncs_totalcounterData files flushed to disk by a checkpoint. Zero with fsync off.
scramdb_segment_dir_syncs_totalcounterDirectories flushed to disk so that newly created data files survive a power cut and removed ones stay removed. Zero with fsync off.
scramdb_segment_creations_covered_totalcounterData file creations made durable by a directory flush. Against the directory flushes it shows the batching: many creations under one flush is the design.
scramdb_segment_removals_covered_totalcounterData file removals made durable by a directory flush, so a power cut never brings a removed file back. Against the directory flushes it shows the batching, as the creations do.
scramdb_segment_flush_last_filesgaugeData files flushed by the last checkpoint that flushed any.
scramdb_segment_flush_last_dirsgaugeDirectories flushed by the last flush group that flushed any.
scramdb_checkpoint_table_passes_repeated_totalcounterCheckpoint passes over the tables that ran again because DDL ran during them. Expected under DDL load.
scramdb_checkpoint_table_passes_closed_totalcounterCheckpoint passes that held DDL off for their own length because DDL kept overlapping them. A steady rate means DDL keeps meeting checkpoints.
scramdb_checkpoint_table_pass_closed_secondshistogramTime each stretch of a checkpoint's closed pass held DDL off. The pass walks one table at a time and lets DDL run between stretches of about 100 ms, so the tail should stay near that plus one table's flush; a long tail is one slow table.
scramdb_checkpoint_dropped_index_flushes_skipped_totalcounterVector index flushes skipped because the index was being dropped. Expected under DROP load.
scramdb_quarantined_tables_cloned_totalcounterQuarantined tables a database clone, or a branch at a past time, carried into the new database as quarantined. Alert on any: the table is unreadable in both databases until dropped or recreated.
scramdb_table_drop_catalog_failures_totalcounterTable drops whose catalog entry could not be removed, left for the next start to finish. Alert on any: the table is still listed on disk.

Branches and clones (always present)​

A CREATE DATABASE ... CLONE on a single node or on any node of a cluster; see Branching. All zero on a process that never clones.

MetricTypeMeaning
scramdb_branch_clone_gate_waits_totalcounterDDL statements that waited for a database clone or a checkpoint to finish walking the tables. A rise that follows branch creation or a checkpoint is DDL held back by it, for the length of that walk.
scramdb_branch_clone_rollbacks_totalcounterDatabase clones undone because a step failed. The statement answered an error that names the step, and nothing of the clone is left.
scramdb_branch_clone_cleanup_failures_totalcounterDatabase clones whose undo could not finish and left a half-made database to drop. Alert on any: the database refuses connections until DROP DATABASE or the next start removes it.
scramdb_branch_clone_admission_parked_totalcounterDatabase clones that waited for memory before they started. A rise is clones arriving faster than the memory budget serves them.
scramdb_branch_lifecycle_total{op,outcome}counterBranch lifecycle operations by kind (create, rename, expire, drop, reap, swap, resume: a drop a restart finished or could not finish) and outcome (ok, error).
scramdb_branch_expiry_backloggaugeBranches whose expiry has passed and that the reaper has not dropped yet. Alert when it stays above zero.
scramdb_branch_countgaugeDatabases on this node cut from another database (branches and clones).
scramdb_merge_record_retentiongaugeMerge records each database keeps, newest first ([agent] merge_record_retention).
scramdb_merge_records_pruned_totalcounterBranch merge and refresh records removed to stay within the merge record retention.
scramdb_retired_agent_historygaugeRetired agent records the server keeps, newest first ([agent] retired_agent_history).
scramdb_retired_agents_pruned_totalcounterRetired agent records removed to stay within the retired agent history.
scramdb_module_counters_pruned_totalcounterRun counters of modules that were dropped and removed to bound the counters.
scramdb_branch_bookkeeping_rows_pruned_totalcounterBookkeeping rows of dropped branches removed.
scramdb_branch_clone_admission_refused_totalcounterDatabase clones refused for lack of memory. It stays at zero: a clone that finds memory short waits for it, until its statement_timeout or a cancel ends the wait (SQLSTATE 57014, nothing of the clone left).
scramdb_branch_clone_rotation_fallbacks_totalcounterClones that had to start a new segment of a parent table because the rows it buffered did not fit the segment it was writing. Rare: most clones take the segment as it stands.
scramdb_branch_active_segments_shared_totalcounterSegments a clone took from its parent as they stood, without the parent starting a new one.
scramdb_branch_tables_inherited_totalcounterTables of cloned databases made live with no file of their own. Each gets its files at its first write.
scramdb_branch_files_born_totalcounterSegment files of cloned tables made by their first write.
scramdb_branch_index_viewsgaugeBranch indexes that read through their parent's index now. Falls as branches fold or drop.
scramdb_branch_index_views_registered_totalcounterBranch indexes registered as a view over their parent's index, on a single node and on each node of a cluster. A clone adds one per index of each table.
scramdb_branch_index_fallback_rebuilds_totalcounterBranch indexes rebuilt at clone time because the parent could not serve a view. 0 on a healthy store; each one cost time that grows with the table.
scramdb_branch_index_view_probes_totalcounterIndex reads answered through a branch's view of its parent's index.
scramdb_branch_index_folds_totalcounterBranch indexes that finished copying their parent's entries into their own.
scramdb_branch_index_fold_entries_totalcounterEntries copied into branch indexes by their folds.
scramdb_branch_index_fold_cancelled_totalcounterFolds stopped early by a drop, a shutdown or the maintenance memory limit. They resume; a rate around a shutdown or a memory squeeze is expected.
scramdb_branch_index_fold_failed_totalcounterFolds that stopped on an error and will run again. A steady rise with scramdb_branch_hold_retained_bytes rising too is a fold that cannot finish: read the server log.
scramdb_branch_index_fold_secondshistogramTime one branch index fold took to finish.
scramdb_branch_fork_holdsgaugeBranches whose index views read their parent's retired entries.
scramdb_branch_hold_retained_bytesgaugeBytes of parent segments that branches still list and keep on disk. They are given back when the branch folds or drops.
scramdb_branch_relocation_watchesgaugeParent tables a clone in progress keeps the retired entries of. 0 when no clone runs.
scramdb_branch_hold_pinsgaugeParent indexes pinned for clones in progress on this node. 0 when no clone runs; a value that stays above 0 for minutes is a clone that never ended.
scramdb_branch_vector_viewsgaugeBranch vector indexes that read through their parent's index now.
scramdb_branch_vector_views_registered_totalcounterBranch vector indexes registered as a view over their parent's index.
scramdb_branch_vector_fallback_rebuilds_totalcounterBranch vector indexes rebuilt at clone time because the parent could not serve a view. 0 on a healthy store.
scramdb_branch_vector_unservable_totalcounterBranch vector index reads answered from the table instead of the index. The answer is exact and slower.
scramdb_branch_vector_folds_totalcounterBranch vector indexes that finished rebuilding as their own.
scramdb_branch_vector_fold_failed_totalcounterBranch vector index folds that stopped on an error and will run again.
scramdb_branch_vector_retained_bytesgaugeBytes of replaced vector index memory kept charged for branches still reading it. 0 once the last branch folds or drops.
scramdb_branch_vector_searches_totalcounterSearches answered through a branch's vector index.
scramdb_database_objects_forked_totalcounterCatalog objects a branch or clone gave to its new database: functions, procedures, types, schemas, event triggers, default privileges, their privileges, owners and comments.
scramdb_database_objects_dropped_totalcounterCatalog objects removed with a dropped database, the same kinds.
scramdb_database_layouts_migrated_totalcounterSaved catalog files read in the layout from before databases owned them, once each at the start that read them: every object in them became the default database's.
scramdb_schema_renames_resumed_totalcounterSchema renames a start finished because the run before it stopped part way through one: the schema's members were moved to the new name from where they stopped.
scramdb_schema_rename_resume_failures_totalcounterSchema renames a start could not finish. The rename stays recorded and the next start tries again; a rise needs an operator.
scramdb_type_rekeys_resumed_totalcounterType renames and moves a start finished because the run before it stopped part way through one: the stored texts and columns that name the type were brought to the new name from where they stopped.
scramdb_type_rekey_resume_failures_totalcounterType renames and moves a start could not finish. The change stays recorded and the next start tries again; a rise needs an operator.
scramdb_enum_renumbers_totalcounterEnum types an ALTER TYPE ... ADD VALUE laid out again with every stored value, because the slot between two labels had no ordinal left.
scramdb_enum_renumbers_resumed_totalcounterEnum renumberings a start settled because the run before it stopped part way through one: the stored values were brought to the type's labels from where they stopped.
scramdb_enum_renumber_resume_failures_totalcounterEnum renumberings a start could not settle. The change stays recorded and the next start tries again; a rise needs an operator.
scramdb_enum_renumbers_refused_totalcounterEnum value additions a cluster refused because a distributed table is placed by the enum, or a table appeared while the statement was decided.
scramdb_sequence_follows_resumed_totalcounterSequence renames and moves a start finished because the run before it stopped part way through one: the column defaults, views, domains and other stored texts that draw from the sequence were brought to its new name from where they stopped.
scramdb_sequence_follow_resume_failures_totalcounterSequence renames and moves a start could not finish. The change stays recorded and the next start tries again; a rise needs an operator.
scramdb_collation_follows_resumed_totalcounterCollation renames and moves a start found unfinished because the run before it stopped part way through one: the registry had the new name and the stored texts might still name the old one.
scramdb_collation_follows_finished_totalcounterCollation renames and moves a start brought to their new name in every stored text (the COLLATE clauses of checks, defaults, generated columns, views and index expressions). A resumed count above this one needs an operator: the change stays recorded and the next start tries again.
scramdb_array_names_repaired_totalcounterArray type names a start put right because the run before it stopped part way through a create or a rename that moved an array aside: the names of the arrays and the types they belong to were brought together from where they stopped.
scramdb_array_name_repair_failures_totalcounterDatabases whose array type names a start could not put right. The stored names stay as they were and the next start tries again; a rise needs an operator.
scramdb_partition_bounds_refreshed_totalcounterPartitioned tables whose stored bounds a start read again from their own text, because the bytes a bound of an array key keeps were saved before arrays carried their lower bounds and dimensions. It rises once per such table after an upgrade and stays flat after.
scramdb_partition_bound_refresh_failures_totalcounterPartitioned tables whose stored bounds a start could not read again. The bounds stay as they were and the next start tries again; a rise needs an operator.

A function, type, schema, event trigger or default privilege belongs to the database that made it, so a branch starts with a copy of its parent's and the two change apart from then on. database_objects_forked_total rises by the objects each branch or clone received, and database_objects_dropped_total by those a DROP DATABASE took with it. database_layouts_migrated_total moves only at the first start after an upgrade from a release that kept them for the whole server, and stays flat after.

Internal tables (always present)​

The engine keeps the self-driving loop's decision record and the merge records in tables of its own, and holds each to a bound: the newest rows stay, a row past its age bound goes, the oldest first and a page of rows at a time, and the space of the removed rows is asked back from the background merge. One series per table, labelled table: autonomy.ticks, autonomy.actions, autonomy.setpoints, autonomy.explanations and autonomy.shadow for the decision record (bounded by [autonomy] audit_max_rows and audit_retention) and scram_lo.scram_branch_merges for the merge records of every database (bounded by [agent] merge_record_retention).

MetricTypeMeaning
scramdb_internal_table_rows_trimmed_totalcounterRows removed from one of the engine's own tables to keep it within its bound.
scramdb_internal_table_reclaims_requested_totalcounterTimes the space of one of the engine's own tables was asked to be reclaimed after a trim.
scramdb_internal_table_row_boundgaugeRows one of the engine's own tables keeps, newest first.
scramdb_internal_table_age_bound_secondsgaugeSeconds a row of one of the engine's own tables is kept, for a table with an age bound.
scramdb_merge_records_kept_past_boundgaugeMerge records past the bound that the last trim pass read and kept, by what holds them (reason: branch_base, newest_through_path, refresh_above_base, in_flight).
scramdb_autonomy_trim_kept_age_secondsgaugeSeconds the oldest unfinished decision-record trim has been kept, 0 when none.

A merge record can stay past its bound while something still reads it: a direct merge its branch's merge base has not passed (branch_base), the newest merge of a branch into a database above its parent (newest_through_path), a refresh above its branch's own base (refresh_above_base) or a merge in flight (in_flight). A table over its bound is explained by these reasons. A pass of the decision-record trim whose statement outlasts a tick is kept and goes on at the next one from where it stood, never from its first statement; after ten ticks the server logs one warning naming the table, and scramdb_autonomy_trim_kept_age_seconds reads how long the oldest kept pass has been kept. If it keeps growing, contact support with the server log.

The merge records have no age bound, so scramdb_internal_table_age_bound_seconds has no series for them. A reclaim request counts when the table was asked, not when its bytes came back: what a merge freed is counted per table in Compaction by table.

Security decisions (always present)​

What the access checks decided in this process; see Roles and privileges. The refusal and rule-change counters are zero on a process nobody was refused on and nobody changed a grant on; the access-check counters count every statement checked.

MetricTypeMeaning
scramdb_privilege_refusals_totalcounterStatements refused with a permission error (SQLSTATE 42501) and answered to a client, once each. A refusal a PL/pgSQL handler catches, and a statement a node relays to another, do not count again. A sustained rate is a client or pool running as the wrong role, or a grant that was never made; the server log names the role and the object.
scramdb_row_security_scans_totalcounterTable scans planned with a row-security policy applied. A plan served from the plan cache is not planned again, so this is a lower bound of the filtered executions. Zero on a node whose tables use policies and whose clients run as their own roles means the policies are bypassed (the role owns the table, holds BYPASSRLS or is a superuser).
scramdb_acl_changes_totalcounterChanges to the access rules: roles, grants and default privileges. A statement that changed nothing (refused before its first write) does not count; one that was undone counts once, with its rewind.
scramdb_acl_rewinds_totalcounterAccess-rule changes taken back after a later step failed or a transaction rolled back. A GRANT, REVOKE or role statement is applied whole or not at all: when the save of the rules is refused, the statement is undone and the client gets the store's error. A rate with no rolled-back transactions to explain it means the catalog store is refusing writes (a full disk or a stopped store); the server log names the cause.
scramdb_access_checks_totalcounterStatement access lists judged against the role's privileges: one per statement planned (its reads, as the planner bound them), per write, COPY, TRUNCATE or locking clause judged at its start, and per point query judged on its first run. A plan served from the plan cache under unchanged rules is counted in the next metric instead.
scramdb_access_check_reuses_totalcounterAccess checks a cached plan or a cached point query answered with its earlier answer, because no GRANT, REVOKE, membership or owner change happened since: one compare, no privilege lookup. A GRANT or REVOKE makes the next run of every cached plan judged again.
scramdb_security_seam_disagreements_totalcounterDebug builds only (always zero in a release build).
scramdb_agent_quota_refusals_totalcounterStatements of an AI agent refused because its branch reached its size quota (SQLSTATE 53100).

IN subqueries (always present)​

What became of the result of each x IN (SELECT ...) whose subquery does not read the enclosing query. The subquery runs once per statement: its result is a typed list the statement holds, or, when the list would not fit the statement's memory budget or its column cannot be held as one, a relation of the statement that the join reads; a relation the budget refuses goes to temporary files. All zero on a process that ran no such subquery.

MetricTypeMeaning
scramdb_in_subquery_lists_totalcounterIN subquery results held as a typed list of the statement.
scramdb_in_subquery_relations_totalcounterIN subquery results held as a relation of the statement for the semi join. A rising share against the lists means results are large or of a type no list holds; check work_mem.
scramdb_in_subquery_relations_spilled_totalcounterIN subquery relations written to temporary files because memory ran short. Any rate is memory pressure on the node.
scramdb_in_subquery_lists_received_totalcounterTyped lists received with a fragment from another node (a cluster statement that reads a list on a node other than the one coordinating it).
scramdb_in_subquery_held_bytesgaugeBytes running statements hold for their IN subquery lists and relations, charged to their memory budget where they have one. Returns to its start when the statements end; a value that stays up with no statement running is a leak.

Snapshot index reads​

An index lists only the latest version of each row. A REPEATABLE READ or SERIALIZABLE read whose snapshot is older than a change finds the version it sees through the index's record of superseded entries, instead of scanning the table.

MetricTypeMeaning
scramdb_snapshot_index_probes_versioned_totalcounterSnapshot index probes answered from the index and its record of superseded entries.
scramdb_snapshot_index_probes_scanned_totalcounterSnapshot index probes answered by scanning the table.
scramdb_retired_versions_recorded_totalcounterSuperseded index entries kept for snapshots that can still see them.
scramdb_retired_versions_released_totalcounterSuperseded index entries released by compaction or with their index.
scramdb_retired_versions_entriesgaugeSuperseded index entries currently kept for snapshots.
scramdb_staged_read_settled_gate_entries_totalcounterStaged-entry reads that outran the bounded retry under a storm of stage changes and took the reader-priority gate. A rate means readers wait behind a heavy stream of stage changes.

probes_scanned_total climbing against probes_versioned_total means snapshot reads are scanning whole tables instead of probing their index.

Read snapshot pins​

The snapshot of a statement, of a REPEATABLE READ or SERIALIZABLE transaction, of a streaming result and of a distributed fragment's local scan is pinned while it is in use, so compaction keeps every row version it can still see.

MetricTypeMeaning
scramdb_read_pins_taken_totalcounterRead snapshot pins registered.
scramdb_read_pins_released_totalcounterRead snapshot pins released.
scramdb_read_pin_redraws_totalcounterSnapshot draws that moved their pin off the floor registered before the draw.
scramdb_read_pins_heldgaugeRead snapshot pins held now.
scramdb_retention_holds_taken_totalcounterRetention holds taken for the shard groups hosted on this node.
scramdb_retention_holds_released_totalcounterRetention holds released.
scramdb_retention_holds_heldgaugeRetention holds held now for the shard groups hosted on this node.
scramdb_retention_boot_holds_armed_totalcounterStores that kept every row version at start until their shard groups were pinned.
scramdb_retention_boot_holds_released_totalcounterStart-up retention holds released.
scramdb_retention_boot_holds_heldgaugeStores keeping every row version until their shard groups are pinned.

read_pins_held that keeps growing on a steady workload means snapshots are not being released, which holds old row versions on disk; look for long-open transactions or unread result streams.

On a cluster node, each shard group hosted here also holds the row versions between the oldest read position the group can still serve and its newest commit, since a read at such a position can reach this node long after it was chosen; a quiet group holds only its last window, never what busier groups write after it. retention_holds_held follows the groups hosted here. At start a store keeps every row version until each shard group it holds rows of is hosted again or placed on another node, and it logs one line saying so; retention_boot_holds_held above zero long after start means a group was never hosted again (its restarts exhausted, or it is quarantined), and compaction on that store frees nothing until it is.

Memory admission​

Heavy statements take a memory grant before they run and queue in arrival order when the pool is full. A statement is never refused for lack of memory while the pool can make progress; one that waits longer than admission_max_park_ms ends with a "server is busy" error, and one that holds its grant longer than admission_max_hold_ms while others wait is cancelled. A waiting statement shows IPC / admission in pg_stat_activity with the size of its share in admission_ticket_bytes, and the log names the statement holding the memory when a statement starts to wait. Catalog and system-view reads never take a grant. A statement whose data grows past what its grant covered is charged for the growth while it runs: it never waits for it and never takes it ahead of a statement that is queued, and when the gate has no room it keeps running at the share it has, spilling what does not fit, and a later run of the same shape is charged more. A shape the server has not run before starts with a small share (the pool split over the query cores) and grows while it runs as long as the pool has room; it never waits for more, it spills what does not fit, so a burst of new statements runs together. A wait is counted from the later of its start and the last time memory went back to the pool, so a pool that is still finishing work is not mistaken for a stalled one. A hash join whose build side does not fit in memory runs in rounds, each within the memory the server has free, and answers every row.

MetricTypeMeaning
scramdb_admission_admitted_totalcounterHeavy statements that took a grant.
scramdb_admission_queued_totalcounterHeavy statements that had to wait for one, counted once each when they first park.
scramdb_admission_parked_waitersgaugeStatements parked waiting for a grant right now. Emitted only once a real measurement exists.
scramdb_admission_outstanding_grant_bytesgaugeBytes currently granted and not yet handed back.
scramdb_admission_waited_past_deadline_totalcounterStatements that waited longer than the warning threshold. Nothing is refused on that threshold; it exists so a long wait is visible.
scramdb_admission_cancelled_while_queued_totalcounterStatements whose own cancellation fired while they were still waiting.
scramdb_maintenance_memory_waits_totalcounterCREATE INDEX builds (vector indexes and their REINDEX included), CLUSTER rewrites and ANALYZE runs whose working memory (storage.memory.maintenance_bytes) did not fit the pool when asked, and that waited for it instead of failing. The automatic analysis a COPY or bulk INSERT asks for counts here too, and runs once the pool has room.
scramdb_analyze_on_load_waits_totalcounterAutomatic analyses of a COPY or bulk INSERT ([statistics] analyze_on_load) that waited for the load's transaction to commit, so they read the loaded rows. Growth is normal: a load inside a transaction block counts once while it waits for its COMMIT.
scramdb_auto_analyze_wakes_totalcounterTimes the automatic analysis thread woke to look for work. An idle server holds it nearly still: the thread wakes once per [statistics] auto_analyze_check_interval, at once for an analyze-on-load request, and every 200 ms only while a request is waiting for its load's transaction or for memory.
scramdb_analyze_samples_totalcounterTable samples drawn by ANALYZE and by automatic analysis, one per analysed table.
scramdb_analyze_sample_pages_totalcounterPages those samples read.
scramdb_analyze_sample_reserved_bytes_totalcounterWorking memory those samples reserved before reading their first page. Divided by scramdb_analyze_samples_total it is what one analysis asks for: it follows the table's own size and row width, so a small table asks for kilobytes and never for the whole maintenance_bytes.
scramdb_analyze_samples_short_totalcounterSamples that read fewer pages than the [statistics] settings ask for, because the reserved memory held fewer or a page could not be read. The statistics such a sample gives are the page-header figures plus what was read.
scramdb_analyze_statistics_reused_totalcounterTables that took their saved statistics instead of being analysed: the tables of a restarted server, and the tables of a database made by CREATE DATABASE ... CLONE, which plan from their source's statistics until their own rows drift.
scramdb_admission_bypassed_totalcounterStatements served without consulting the ledger, either because they were light enough to stream or because the gate is switched off.
scramdb_admission_inherited_nested_totalcounterNested statements served inside their parent's grant instead of taking a second one. Growth is normal on a workload full of CTEs and routines.
scramdb_admission_released_at_seam_totalcounterGrants handed back cleanly at statement completion. On ordinary workloads this climbs in step with admitted_total.
scramdb_admission_measured_grants_totalcounterStatements admitted with a grant sized from their measured memory.
scramdb_admission_measured_growths_totalcounterMeasured grants that had to grow while their statement ran.
scramdb_admission_ticket_cap_bytesgaugeMost memory the admission gate grants one statement (admission_ticket_percent of the execution budget, or its share at heavy_concurrency, whichever is smaller).
scramdb_admission_park_bound_millisecondsgaugeLongest a statement waits for a memory grant before it is ended; adapts to the waits observed, 0 when unbounded.
scramdb_admission_hold_bound_millisecondsgaugeLongest a statement holds a memory grant while others wait before it is ended; adapts to the holds observed, 0 when unbounded.
scramdb_admission_parked_too_long_totalcounterStatements ended with a "server is busy" error after waiting longer than the park bound.
scramdb_admission_hold_cancelled_totalcounterStatements cancelled for holding a memory grant past the hold bound while others waited.
scramdb_admission_unenforceable_holds_totalcounterHolds that ran past the hold bound for a statement with no client session.
scramdb_admission_small_admitted_totalcounterStatements admitted at once because their measured memory is a small share of the gate.
scramdb_admission_topups_totalcounterRunning statements whose memory grant was raised because their state outgrew it. A steady rate at one shape is a shape whose data keeps growing; it is charged more on its next run.
scramdb_admission_topup_waits_totalcounterWorkers that waited for another worker of the same statement to finish raising its memory grant. The wait is short (one charge); a steady rate means one statement's workers outgrow their share together.
scramdb_admission_topups_refused_totalcounterRunning statements whose memory grant could not be raised, because the gate had no room to give or the wait for another worker's raise ran out. They keep the share they have; a rate beside a full gate means storage.execution_memory_bytes (or the container limit) is small for the shapes.
scramdb_admission_heavy_concurrencygaugeHeavy statements the gate currently admits at once. Live-settable at runtime, independent of the storage.memory.heavy_concurrency config key's value at startup.

Allocator accounting​

Every byte ScramDB's own allocator hands out is counted exactly, the instant it is allocated; see Memory for how the figures below feed the budgets on that page.

MetricTypeMeaning
scramdb_heap_live_bytesgaugeThe exact process heap total: every byte the allocator has handed out and not yet freed.
scramdb_heap_reserve_bytesgaugeBytes the allocator has committed but not yet handed to a live allocation (its own retention).
scramdb_heap_overflow_cellsgaugeThreads currently sharing the heap accounting overflow cell instead of their own.
scramdb_heap_context_overflows_totalcounterCharge contexts created after the context table was full, sharing a class's permanent slot.
scramdb_heap_class_bytesgaugeLive bytes of each heap class (statement, index_build, vector_index, background, cache, cluster, autonomy, unattributed), labeled class. unattributed is the live heap beyond the other classes and the escaped bytes, never below 0, so the classes and scramdb_heap_escaped_bytes add up to scramdb_heap_live_bytes.
scramdb_heap_escaped_bytesgaugeBytes still resident under a class whose owning statement, build or job already ended, labeled class.
scramdb_heap_unpaired_small_bytes{class,direction}gaugeSmall-block bytes an ended context left for another to free (held), or freed beyond its own charge (freed), by class. They balance across contexts over time.
scramdb_heap_unscoped_bytesgaugeBytes allocated outside every scope and not yet freed outside one: the raw counter behind the unattributed class, in which a block allocated outside every scope and freed inside one stays.
scramdb_heap_class_overrun_bytesgaugeBytes by which the named classes and escaped bytes read past the live heap. 0 while they add up; nothing is refused because of it.
scramdb_heap_uncharged_bytesgaugeBytes live statements hold in their allocator context past what they reserved through the execution-memory budget; the budgets already decide on the larger figure, so this shows how much work runs ahead of its reservations.
scramdb_heap_vector_bytesgaugeBytes the vectors of each class hold right now (their values, long strings and array elements), labeled class. A statement's share counts against its memory: its operators spill once it nears its budget, and it stops with out of memory (53200) past the execution memory budget.
scramdb_heap_vector_escaped_bytesgaugeVector bytes still held after the statement, build or job that made them ended, labeled class: a result handed to its client, or a value a cache kept.
scramdb_heap_vector_over_limit_totalcounterMemory accounts whose vectors went past their limit, labeled class; a statement one is a statement answered out of memory (53200) instead of growing the server.
scramdb_memory_uncharged_bytes{family}gaugeBytes a memory family holds beyond what it reserved, labeled family: execution_pool (statements, background work and index builds past the execution pool's reservations and the maintenance work's charge) and cache (the cache class past the compiled-code and plan caches' own accounting and the fused buffer pool's retained blocks). It reads zero when the family reserved at least what the allocator measured for it. The execution_pool series is absent until the server has an execution pool.
scramdb_maintenance_memory_bytesgaugeWorking memory compaction and other maintenance work hold now. It is reserved when a pass takes a table and given back when the pass ends.
scramdb_native_heap_bytesgaugeBytes the C allocator holds for native code, labeled state: in_use (handed out and not freed) and free (held and reusable). The query compiler and the code it compiled live there. The free bytes are returned to the system with the allocator's retained pages. Absent until the memory watchdog has sampled it, and on a system whose C allocator does not report it.
scramdb_memory_file_resident_bytesgaugeResident file-backed memory: the binary's code and mapped files. The kernel can drop it, so it is no allocation and is not part of the unattributed figures.
scramdb_memory_unattributed_bytesgaugeAnonymous memory (resident, shared and swapped out, from the process's status file; file-backed pages are scramdb_memory_file_resident_bytes) the process holds beyond everything the engine's own counters name: the execution pool's reservations, maintenance work, the page and buffer pools, the aggregate tables, the streamed-result queue, the compiled-code and plan caches, the native heap (scramdb_native_heap_bytes) and the families of scramdb_memory_charged_bytes. It is the figure the memory watchdog's breach line gives as held by no live counter. What remains is thread stacks, mappings outside the native heap and the allocator's own overhead (its retained pages are a named holder and are returned every ten seconds past 256 MiB); read it beside scramdb_vector_memory_in_use_bytes before suspecting a leak. Absent until the memory watchdog has taken its first sample; it is never rendered as zero for a figure not yet known.
scramdb_memory_anon_unattributed_bytesgaugeThe same figure as scramdb_memory_unattributed_bytes, sampled with the memory watchdog.
scramdb_memory_ceiling_effective_bytesgaugeThe memory ceiling the watchdog enforces now: the configured ceiling, or less while the machine has less memory free than the server was sized for. Absent until the first sample.
scramdb_memory_pressure_some_avg10gaugeThe percent of the last ten seconds in which some task on the machine stalled on memory, from the memory pressure stall file. Absent where that file cannot be read.
scramdb_memory_shed_total{step}counterTimes the watchdog gave memory back before cancelling a statement, labeled step: allocator_purge, subplan_cache, plan_cache, compiled_code and buffer_pool, in the order they run. Absent until the watchdog is armed.
scramdb_memory_charged_bytes{family}gaugeBytes an owner of process memory has charged, labeled family: session_buffers (the read and write buffers each connection thread keeps between statements), catalog (the table definitions the schemas hold and the parsed stored definitions: view bodies, defaults, checks, policies, trigger conditions and function bodies), udf_daemons (the memory reserved for the function daemons' shards, a reservation and not what the guests touched), fused_pool (the retained blocks of the compiled kernels' buffer pool) object_store_pool (the remote-read client's idle connection buffers and one connection's buffers for each open response body) subplan_cache (the finished results the subplan cache keeps for reuse, counted by their size in memory), stack_segments (the stack segments a deeply nested statement grew while it was read, planned and run, a reservation and not what the statement touched; the segments the SQL reader grows for itself are not counted), regex_exec (the working memory of a regular-expression search that has a back-reference or a look-around, the scratch it leaves for the next search, and the character keys a pattern holds while it is compiled), function_scratch (the buffer a string or binary function writes each answer into, at the largest answer of each running batch) and row_places (where each row of a merged segment sits, four bytes per row position of the segments a compaction merged, for the merged segments read by row id since the server started), index_buffers (the nodes and entries every index holds in memory before they are written into a run file), row_locks (the row locks each open transaction holds: per row an UPDATE, DELETE, unique-key INSERT, COPY or locking SELECT locked, its lock entry and the key kept to release it, an estimate per lock), tag_map (the allocator's map from a block to the context it was charged to, 2 MiB each time it grows, never given back), wal_buffers (the write-ahead log writer's ring and its completion slots, sized by the memory budget) and vector_arenas (the vector indexes' arenas and row-id ledgers, outside the allocator, without their write buffers, which the execution pool counts). The catalog, object_store_pool, subplan_cache and vector_arenas figures are estimates from the stored sizes and the client's own settings, not allocator readings.
scramdb_memory_budget_bytes{family}gaugeThe byte budget an owner of process memory is sized to, labeled family, derived from the machine: the connection places for session_buffers, a split of the runtime memory margin by what each uses for catalog, fused_pool, subplan_cache and regex_exec, the function daemons' own memory envelope for udf_daemons and the client's connection pool plus one open body per core for object_store_pool. A family appears only once its budget is known, so stack_segments, function_scratch, row_places and row_locks, which have none, are absent here. The budget of index_buffers is the index memory (index_memory_bytes, a share of the maintenance memory unless set): past it the largest buffers are written out until the family is at three quarters of it. Nothing is refused at a budget except the fused pool, which frees a block instead of keeping it, the subplan cache, which drops its entries and does not keep a result larger than the whole budget, a regular-expression search, which drops the end positions it kept and works them out again, and the catalog's parsed definitions, which drop the ones no statement read since the last pass and parse one again when it is next read; an owner above its budget shows on this pair of series.
scramdb_memory_refused_bytes_total{family}counterBytes a family's budget refused, labeled family, since the process started: the fused pool's blocks it freed instead of keeping, the subplan cache's results it did not keep, the end positions a regular-expression search dropped to work them out again and the parsed definitions the catalog did not keep. A family that is refused is given room at the next margin split.
scramdb_memory_margin_bytesgaugeThe runtime memory margin the table definitions, the fused pool, the subplan cache and the regular-expression working memory draw their budgets from. Absent until the margin is resolved.
scramdb_memory_margin_pool_bytesgaugeThe part of the runtime memory margin divided among those four at the last split: the margin less the resident memory no counter names.
scramdb_memory_margin_split{reason}gaugeOne series per reason, one for the reason the last margin split came out as it did and zero for the others: even (nothing is used yet, the pool is divided evenly), follows_use (each budget is its family's use plus an equal share of what is unclaimed) and over_pool (the families want more than the pool, so each keeps what it holds and the rest is divided by what was refused).
scramdb_execution_pool_allocated_bytesgaugeBytes the execution memory pool currently holds against its budget (the larger of what was reserved and what the allocator measured).
scramdb_execution_pool_budget_bytesgaugeThe execution memory pool's total budget (execution_memory_bytes).
scramdb_write_set_returned_bytes_totalcounterWrite-set bytes given back to the execution memory pool before their transaction ended: an UPDATE that made a row smaller, and the part of a WITH statement's reservation its arms did not keep. Zero on a node whose transactions only grow what they write.
scramdb_execution_pool_essential_breach_episodes_totalcounterTimes the execution memory pool went past its budget and its small overshoot allowance. Reported once per crossing, never once per allocation; if the count keeps climbing, raise execution_memory_bytes or run fewer heavy queries at once.

Joins over the memory budget and buffer reuse (always present)​

A hash join whose build side does not fit the execution-memory budget spills and runs in rounds, and several such joins in one probe stage share one round schedule. When the joins' keys are columns of the probe table itself, the probe side is partitioned to disk once and each pass reads only the rows its rounds can match; the query workers reuse their interpreters and join buffers across morsels. A join whose output for one batch of its probe side is larger than one output batch (a cross join, a key the build side holds many times) emits it in bounded batches, each charged to the statement's memory while it is held, and a plain query over such a join streams them to the client as they are made.

MetricTypeMeaning
scramdb_join_round_plans_totalcounterHash join builds that did not fit the execution-memory budget and ran in rounds.
scramdb_join_round_passes_totalcounterPasses over a probe side made by hash joins running in rounds.
scramdb_join_round_multi_plan_stages_totalcounterProbe stages that ran several oversized hash joins in rounds together.
scramdb_join_round_chunked_plans_totalcounterHash join builds that split one partition into several rounds to stay within the execution-memory budget.
scramdb_join_round_partitioned_probes_totalcounterProbe stages of hash joins in rounds whose probe side was partitioned to disk once instead of read on every pass.
scramdb_join_output_steps_totalcounterJoin output batches emitted by a probe whose output was larger than one batch.
scramdb_unnest_output_steps_totalcounterUnnest output batches emitted for arrays whose elements were more than one batch.
scramdb_join_output_step_bytesgaugeBytes of join and unnest output batches the query workers hold right now.
scramdb_join_output_step_peak_bytesgaugeMost bytes of join and unnest output batches the query workers held at once.
scramdb_execution_interpreters_built_totalcounterInterpreters the query workers built to run morsels on the interpreter.
scramdb_execution_join_selection_grows_totalcounterTimes a query worker grew its reused join selection buffer.
scramdb_execution_agg_rows_folded_in_runs_totalcounterGrouped aggregate rows folded into the previous row's group without a hash lookup, because their key equals the previous row's.
scramdb_execution_agg_rows_folded_in_runs_compiled_totalcounterThe same, for the rows the compiled tier folded.
scramdb_execution_agg_run_checks_switched_off_totalcounterGrouped aggregate batches whose consecutive-key check switched off for lack of runs, so input not ordered on its keys pays only a short warm-up of comparisons per batch.
scramdb_execution_call_argument_views_totalcounterBatch arguments handed to scalar functions as zero-copy views instead of copies.

The build counters stay bounded by the workers and the batch sizes, never by the rows a statement reads: The folded-row counters grow on input grouped or sorted by its grouping key, where most rows skip the hash table.

Set-returning routines (always present)​

A RETURNS SETOF or RETURNS TABLE PL/pgSQL function hands its rows to the query in bounded batches, the routine pausing while the query works on the last one, so these counters stay bounded by the batch size, never by the rows a function answers.

MetricTypeMeaning
scramdb_setof_routine_calls_totalcounterSet-returning routine calls started by a query.
scramdb_setof_routine_batches_totalcounterRow batches set-returning routines handed to their queries.
scramdb_setof_routine_rows_totalcounterRows set-returning routines handed to their queries.
scramdb_setof_routine_bytes_totalcounterEstimated bytes of the rows set-returning routines handed to their queries.
scramdb_setof_routine_abandoned_totalcounterSet-returning routine calls stopped before their last row was read, by a LIMIT, a cancel or an error.
scramdb_setof_routine_open_callsgaugeSet-returning routine calls started and neither finished nor stopped right now.
scramdb_user_aggregate_partial_states_totalcounterPartial transition states merged by user-defined aggregates: the path an aggregate with a COMBINEFUNC takes, one state per group and reader instead of every row.
scramdb_user_aggregate_rows_gathered_totalcounterRows user-defined aggregates collected and fed to their transition function in sorted order: the path of an aggregate with no COMBINEFUNC, or with ORDER BY, DISTINCT or WITHIN GROUP. A group larger than the aggregate memory budget is fed from a stream on disk.
scramdb_user_aggregate_bindingsgaugeUser aggregate bindings held, live and waiting to be freed after a replace or drop.

A LIMIT over a function that answers millions of rows counts one abandoned call and a few small batches.

Spilling to disk​

A query that outgrows its memory budget writes to disk rather than failing. These count that happening.

MetricTypeMeaning
scramdb_agg_spill_partition_flushes_totalcounterAggregate partitions written to disk instead of breaching the memory budget.
scramdb_agg_spill_bytes_written_totalcounterBytes written to aggregate spill files.
scramdb_agg_spill_rows_written_totalcounterRows written to aggregate spill files.
scramdb_agg_spill_fragment_partition_flushes_totalcounterFragment partitions written to disk for the same reason.
scramdb_agg_spill_fragment_bytes_written_totalcounterBytes written to fragment spill files.
scramdb_agg_spill_fragment_rows_written_totalcounterEntries written to fragment spill files.
scramdb_agg_spill_grace_partitions_totalcounterAggregate partitions merged in several passes because they did not fit their memory.
scramdb_agg_spill_grace_passes_totalcounterMerge passes run over aggregate partitions split by hash range. Divided by the partitions above, the passes a partition took.
scramdb_spill_tape_live_bytesgaugeBytes currently resident in every live spill file. Alert on it if the volume is small.
scramdb_fragment_spill_mgr_init_failures_totalcounterTimes the fragment spill directory could not be initialized and was retried. Zero on a healthy process; non-zero points at temp_dir permissions or space.
scramdb_final_agg_spill_mgr_init_failures_totalcounterThe same, for the final aggregate spill directory.

Bulk load​

MetricTypeMeaning
scramdb_copy_one_pass_chunks_totalcounterChunks processed by the single-read COPY parser. Zero means that path never engaged, either because it is switched off or because no eligible COPY has run.
scramdb_copy_one_pass_reparses_totalcounterChunks that had to be parsed a second time because the parser's guess about where quoting stood at the chunk boundary was wrong. Against chunks_total this is the miss rate; a file with heavy embedded quoting raises it.
scramdb_copy_one_pass_reparse_bytes_totalcounterBytes re-read by those second passes, tracked separately so load throughput is never overstated.
scramdb_copy_line_scan_bytes_totalcounterBytes of COPY input read again to find the line number of a failing row. A COPY that succeeds adds nothing; a failing one usually adds only its first record.
scramdb_copy_ordered_chunks_totalcounterCOPY chunks handed over in file order because the table's triggers or column defaults need the rows in input order.
scramdb_copy_ordered_wait_nanos_totalcounterNanoseconds COPY parse workers waited for earlier chunks to be handed over in file order.
scramdb_copy_staged_bytesgaugeBytes of staged COPY rows and key runs held on this node.
scramdb_copy_stages_opengaugeCOPY stages holding rows that are not committed yet: a node-local COPY's, and on a cluster node also every replica's stage of a cluster COPY and the coordinating node's stage of the COPY's key checks.
scramdb_copy_staged_rows_totalcounterRows COPY staged.
scramdb_copy_stages_attached_totalcounterCOPY stages committed into their tables.
scramdb_copy_stages_dropped_totalcounterCOPY stages discarded without a commit.
scramdb_copy_staging_refusals_totalcounterCOPY batches refused because the staging area was full. Each one failed its COPY with the disk-full error, leaving nothing behind.
scramdb_copy_stages_recovered_totalcounterCOPY stages a restart found and freed.
scramdb_copy_stages_restored_totalcounterCommitted COPY stages a restore or a branch rebuilt from the WAL.
scramdb_copy_attach_secondshistogramTime to attach one COPY stage to its table at commit.

A COPY writes its rows into staged segments outside the table, invisible to every reader, and attaches them to the table in one step when its transaction commits; a rollback or a crash before the commit frees them whole. copy_staged_bytes is the disk those uncommitted rows hold now. A COPY into a cluster table stages its rows on every replica of every shard group it writes and commits them together (see COPY into a distributed table); copy_stages_open above zero while no COPY is running means a stage was left behind, which a group leader collects after [cluster.dilith] copy_stage_idle_timeout (scramdb_cluster_dilith_copy_stages_collected_total).

A COPY into a table with a trigger, or with a column default that calls a VOLATILE function, still parses a large file across parallel lanes but hands the lanes' chunks to the statement in file order, one at a time, so the trigger or default sees its rows in input order. copy_ordered_chunks_total counts those chunks and copy_ordered_wait_nanos_total the time the parse workers spent waiting for their turn. A wait that is large against the load's wall time means the statement's own per-row work (a row trigger) is what bounds the load, and more parse lanes will not make it faster; a table with only statement triggers or a cheap default spends little waiting. Neither counter moves for a table without triggers.

Import (COPY ... FROM STDIN)​

MetricTypeMeaning
scramdb_copy_in_started_totalcounterCOPY FROM STDIN statements that started receiving rows from a client.
scramdb_copy_in_completed_totalcounterCOPY FROM STDIN statements that loaded their rows.
scramdb_copy_in_failed_totalcounterCOPY FROM STDIN statements that failed, including those the client aborted.
scramdb_copy_in_client_aborts_totalcounterCOPY FROM STDIN statements the client ended with CopyFail.
scramdb_copy_in_bytes_totalcounterBytes of COPY FROM STDIN input handed to loads.
scramdb_copy_in_slices_totalcounterSlices of COPY FROM STDIN input handed to loads.
scramdb_copy_in_client_waits_totalcounterTimes a COPY FROM STDIN's input waited for its load to take the slices before it.

An import queues at most [execution] copy_pipeline_depth slices of input for its load and makes the client wait while the queue is full, so copy_in_client_waits_total climbing against copy_in_started_total means the load is slower than the client sends; the server is not buffering the difference.

Export (COPY ... TO STDOUT)​

MetricTypeMeaning
scramdb_copy_out_streams_totalcounterCOPY TO STDOUT statements that started streaming rows to a client.
scramdb_copy_out_rows_totalcounterRows COPY TO STDOUT statements streamed to clients.
scramdb_copy_out_bytes_totalcounterBytes of COPY TO STDOUT output streamed to clients.
scramdb_copy_out_failed_totalcounterCOPY TO STDOUT statements that ended with an error after they started streaming.
scramdb_copy_out_client_waits_totalcounterTimes a COPY TO STDOUT waited for its client to take rows before rendering more.
scramdb_copy_out_secondshistogramTime from a COPY TO STDOUT's CopyOutResponse to its CopyDone.

An export holds [execution] copy_pipeline_depth buffers and waits for the client when they are full, so copy_out_client_waits_total climbing against copy_out_streams_total means clients read slower than the server renders; the export is not buffering the difference.

Programs (COPY ... PROGRAM)​

MetricTypeMeaning
scramdb_copy_program_childrengaugePrograms started by COPY PROGRAM that have not been reaped yet.
scramdb_copy_program_descriptorsgaugePipe descriptors the server holds to the programs COPY PROGRAM runs.
scramdb_copy_program_started_totalcounterPrograms started by COPY PROGRAM.
scramdb_copy_program_failed_totalcounterCOPY PROGRAM statements that failed in their program or its pipe.
scramdb_copy_program_cancelled_totalcounterCOPY PROGRAM statements cancelled, their program killed.

Both gauges stay at zero when no COPY PROGRAM runs. A value that does not return to zero after the statements ended means a program is still alive: read the process list on the server for the sh -c children of the engine.

System routines and server statements​

MetricTypeMeaning
scramdb_system_calls_totalcounterSystem routine calls evaluated, one per row a call ran for.
scramdb_system_file_bytes_read_totalcounterBytes the server file functions read from files on the server.
scramdb_config_reloads_totalcounterConfiguration reloads run by pg_reload_conf() or a SIGHUP.
scramdb_wal_switches_totalcounterWrite-ahead log file switches run by pg_switch_wal().
scramdb_alter_system_totalcounterALTER SYSTEM statements that changed the settings written to the auto file.
scramdb_alter_system_write_failures_totalcounterALTER SYSTEM statements the store could not write to the auto file, and statements replayed from another node that a node with an unreadable auto file skipped.
scramdb_tablespace_moves_totalcounterTables, materialized views and indexes whose storage or tablespace placement changed by SET TABLESPACE.
scramdb_tablespace_moved_bytes_totalcounterBytes of table storage copied by SET TABLESPACE.

scramdb_tablespace_moves_total counts a relation once, however many files it holds, and an index adds one move and no bytes (an index owns no file, so its placement is a record). A relation already in the target and a refused statement add nothing, and neither does the undo of a move or the placement a REINDEX (TABLESPACE) makes. A move copies the table's files into the target before it removes the old ones, so watch the free space of the target's volume while a large SET TABLESPACE runs.

A call refused for a missing EXECUTE or setting privilege is not counted here: it is a 42501 and counts once in scramdb_privilege_refusals_total. A rising scramdb_alter_system_write_failures_total means the store refuses the write (a full disk or a stopped store): the statement failed and no setting changed, so read the server log for the cause.

Large objects​

MetricTypeMeaning
scramdb_large_objects_created_totalcounterLarge objects created.
scramdb_large_objects_unlinked_totalcounterLarge objects removed.
scramdb_large_object_bytes_read_totalcounterBytes read from large objects.
scramdb_large_object_bytes_written_totalcounterBytes written to large objects.
scramdb_large_object_pages_written_totalcounterLarge object pages stored.
scramdb_large_object_descriptors_opened_totalcounterLarge object descriptors opened.
scramdb_large_object_descriptors_opengaugeLarge object descriptors open in running transactions.
scramdb_large_object_import_bytes_totalcounterBytes lo_import read from server files into large objects.
scramdb_large_object_export_bytes_totalcounterBytes lo_export wrote from large objects to server files.
scramdb_large_object_faults_totalcounterLarge object calls that failed with an error.
scramdb_connection_function_calls_totalcounterFunction call messages received.
scramdb_connection_function_call_failures_totalcounterFunction call messages answered with an error.

The counters count work done: a write that was rolled back is still counted. The descriptor gauge returns to zero as transactions end and connections close, so a value that stays above zero with no transaction open points to a session that holds a transaction open; read pg_stat_activity. A function call message is counted in the two connection counters and, for a large-object function, in the families above too.

Foreign keys​

MetricTypeMeaning
scramdb_fk_referencing_keys_checked_totalcounterReferencing keys checked against the table they reference.
scramdb_fk_referenced_keys_checked_totalcounterReferenced keys that left or changed and were checked for rows still naming them.
scramdb_fk_violations_totalcounterStatements and commits refused because a foreign key was violated.
scramdb_fk_actions_totalcounterReferential actions (CASCADE, SET NULL, SET DEFAULT) run on referencing tables.
scramdb_fk_deferred_checks_totalcounterForeign key checks deferred to the transaction's commit point.
scramdb_fk_statement_end_checks_totalcounterForeign key checks held for the end of their statement, as when a WITH statement inserts a parent row in one arm and its child in another.
scramdb_fk_key_shares_totalcounterReferenced keys a referencing check held shared while it ran, so a concurrent delete of the key waits for it.
scramdb_fk_key_share_waits_totalcounterReferencing checks that waited for a transaction removing the key they reference.
scramdb_fk_leaving_key_locks_totalcounterReferenced keys held exclusively because they left or changed.
scramdb_fk_removed_after_snapshot_totalcounterReferencing checks refused with 40001 because a key their snapshot held was removed by a later commit.
scramdb_fk_probe_secondshistogramTime one foreign key probe statement took.
scramdb_unique_deferred_keys_totalcounterKeys of a deferrable unique constraint held for the end of their statement or the commit point.
scramdb_unique_deferred_keys_checked_totalcounterHeld keys of a deferrable unique constraint checked for a second row.
scramdb_unique_deferred_violations_totalcounterStatements and commits refused because a held key of a deferrable unique constraint was taken twice.

Triggers​

MetricTypeMeaning
scramdb_trigger_statement_end_events_totalcounterAFTER ROW and AFTER STATEMENT trigger events held for the end of their statement, as when a WITH statement writes a table in one arm and the trigger counts rows another arm wrote.
scramdb_trigger_deferred_events_totalcounterAFTER ROW events of deferrable constraint triggers held for the end of their transaction, to run at COMMIT or at SET CONSTRAINTS ... IMMEDIATE. A rate is the application's own use of deferred constraint triggers; a count that climbs while no transaction commits is memory held by long open transactions.
scramdb_trigger_key_units_totalcounterRuns of foreign-key checks and actions judged among the AFTER ROW triggers of a statement. It stays at 0 for a statement whose triggers all sort behind its keys, and rises once a trigger named ahead of a key makes the statement judge the key row by row, as PostgreSQL does.

Prepared transactions (always present)​

What a restart did with the transactions left in PREPARE TRANSACTION (see Prepared transactions): the ones it found, finished, put back and cannot use.

MetricTypeMeaning
scramdb_prepared_xacts_recovered_totalcounterPrepared transactions found on disk at startup and registered again.
scramdb_prepared_xacts_resolved_committed_totalcounterPrepared transactions whose COMMIT PREPARED was found decided and finished.
scramdb_prepared_xacts_put_back_totalcounterPrepared transactions whose interrupted COMMIT PREPARED was found undecided and put back to prepared.
scramdb_prepared_xacts_leftovers_removed_totalcounterFiles of prepared transactions removed because no transaction owned them.
scramdb_prepared_xacts_corrupt_found_totalcounterPrepared transaction records found unreadable or without their changes.
scramdb_prepared_xacts_corruptgaugePrepared transactions whose record cannot be used and can only be rolled back.
scramdb_prepared_xacts_orphan_gidsgaugePrepared transaction identifiers listed without any record of their changes.
scramdb_prepared_xacts_oldest_age_secondsgaugeSeconds since the oldest prepared transaction of this node or its cluster was prepared, 0 when there is none; the cluster's part is refreshed periodically, not at each scrape. The node's own records are read at each scrape. A prepared transaction of a cluster table is kept by the cluster's registry, which each node reads about once a minute, so a cluster transaction prepared or ended just now shows up to about a minute late; pg_prepared_xacts lists them at once.

recovered_total moves only at startup, and leftovers_removed_total only after a stop during PREPARE TRANSACTION or COMMIT PREPARED. corrupt and orphan_gids stay above zero until each listed identifier is ended with ROLLBACK PREPARED; alert on them, and read the server log for the identifiers.

Transactions and locks (always present)​

MetricTypeMeaning
scramdb_transactions_committed_totalcounterTransactions that committed.
scramdb_transactions_aborted_totalcounterTransactions that rolled back or failed to commit.
scramdb_lock_waits_totalcounterRow and table lock requests that found the lock held and waited.

Transaction blocks (always present)​

What an open transaction block hides from other sessions while it is open, and what a restart decided about a block it cut short (see DDL inside a transaction block).

MetricTypeMeaning
scramdb_block_visible_refused_totalcounterStatements refused because they named an object another transaction's open block created.
scramdb_block_claims_livegaugeObjects other than tables (views, functions, sequences, indexes, types, schemas) made in open transaction blocks that other sessions cannot see yet.
scramdb_block_marks_kept_totalcounterTables made in a transaction block that a restart kept because the block committed.
scramdb_block_marks_undone_totalcounterTables made or dropped in a transaction block that a restart took back.
scramdb_block_marks_finished_totalcounterTables dropped in a transaction block that a restart finished dropping.
scramdb_block_marks_failed_totalcounterTables a restart could not resolve after a transaction block was cut short.
scramdb_block_commit_records_removed_totalcounterCommit records of transaction blocks with cluster DDL that were removed once the block had ended. A block with no cluster DDL writes none and removes none.
scramdb_replica_apply_live_duplicates_totalcounterPrimary keys a replica apply found with more than one live version. It stays at zero in normal operation; if it rises, contact support with the server log.
scramdb_block_boot_ends_totalcounterEnds the cluster decided for transaction blocks a restart cut short.
scramdb_block_boot_end_retries_totalcounterAttempts to end a cut-short transaction block that the cluster did not decide.
scramdb_block_round_give_backs_totalcounterLock rounds of a rolled-back savepoint whose table locks the other nodes gave back.
scramdb_block_round_give_back_failures_totalcounterHand-backs of a rolled-back savepoint's locks that did not reach every node.
scramdb_block_commits_named_totalcounterTransaction block commits named through the cluster before their submission.
scramdb_block_commits_refused_totalcounterTransaction block commits not submitted because another node ended the block first.
scramdb_block_departed_ends_totalcounterTransaction blocks of nodes that left the cluster ended as their commit ended.

refused_total rises under real concurrency, when a client names a table another session is still creating; a steady rate on a name that never existed is a client bug. claims_live is zero whenever no block is creating an object. The marks and boot counters move only at startup, after a node stopped inside a transaction block: expect a rise next to a restart, alert on any failed_total, and read the server log for the table. A sustained boot_end_retries_total on a cluster means the cluster has no leader or quorum to decide the block's end yet.

Deep diagnostics​

These are emitted but are not part of the supported surface, and their names and meanings can change between releases. They are here so a number you see on the endpoint is never unexplained; do not build alerts on them.

MetricTypeMeaning
scramdb_header_sidecar_loads_totalcounterSegments whose page headers were loaded entirely from their persisted sidecar file: one sequential read instead of one per page.
scramdb_header_sidecar_partial_loads_totalcounterSidecars that covered only part of what was asked for, the ordinary case for a table still growing.
scramdb_header_sidecar_fallbacks_totalcounterSidecar reads that fell back to walking page headers because the file was absent, stale or corrupt.
scramdb_header_sidecar_writes_totalcounterHeader sidecars written.
scramdb_header_sidecar_write_errors_totalcounterHeader sidecar writes that failed. The segment is walked next time instead.
scramdb_header_prefix_pages_walked_totalcounterPages read by the per-page header walk. A warm store should stop moving it.
scramdb_page_zero_bytes_full_totalcounterBytes zeroed over a whole page buffer on release.
scramdb_page_zero_bytes_partial_totalcounterBytes zeroed over only the recorded written region. On a warm selective scan this should dominate the full counter.
scramdb_scan_prune_nanos_totalcounterCore-nanoseconds spent choosing which pages to read. Summed across workers, so it exceeds wall-clock time on a parallel scan.
scramdb_scan_acquire_nanos_totalcounterCore-nanoseconds spent acquiring pages, pinning them when resident and reading them when not.
scramdb_scan_decode_nanos_totalcounterCore-nanoseconds spent turning page bytes into chunks.
scramdb_agg_fold_rows_totalcounterPartial aggregate rows consumed by the final fold.
scramdb_agg_fold_chunks_parallel_totalcounterFold chunks large enough to spread across cores.
scramdb_agg_fold_chunks_in_place_totalcounterFold chunks small enough to fold on one core.
scramdb_agg_fold_distinct_values_offered_totalcounterDISTINCT values offered to a fold's deduplication set.

Licensing metrics​

MetricTypeMeaning
scramdb_license_validgauge1 if the active license is valid.
scramdb_license_editiongauge/labelActive license edition.
scramdb_license_node_countgaugeNodes currently counted against the license.
scramdb_license_max_nodesgaugeNode limit the license allows.
scramdb_license_read_only_activegauge1 if the license has forced the deployment read-only.
scramdb_license_days_of_term_remaininggaugeEmitted only for a term-limited license.
scramdb_license_data_bytesgaugeEmitted only for a license with a data-size cap.
scramdb_license_data_capgaugeEmitted only for a license with a data-size cap.

Cluster mode​

Running as a cluster node, additional scramdb_cluster_* metrics are appended to the same /metrics response after the single-node metrics above. Every failure mode below has a counter: a stalled compaction, a quarantined group, a dropped partition frame. Alerting on a ScramDB cluster means reading a number, never grepping logs for a string match.

Unlike the WAL archive metrics above, every scramdb_cluster_* metric is always present once [cluster] is configured, whether or not anything interesting has happened yet. A fresh cluster reports every counter at 0 and every gauge at its idle value; there's no "not measured yet" gap to account for in this family.

Peer transport​

MetricTypeMeaning
scramdb_cluster_peer_connects_totalcounterPeer connections established over the cluster transport.
scramdb_cluster_peer_disconnects_totalcounterPeer connections closed.
scramdb_cluster_reconnect_attempts_totalcounterReconnect attempts after a lost peer connection.
scramdb_cluster_frames_sent_totalcounterFrames sent to peers.
scramdb_cluster_frames_received_totalcounterFrames received from peers.
scramdb_cluster_bytes_sent_totalcounterBytes sent to peers.
scramdb_cluster_bytes_received_totalcounterBytes received from peers.
scramdb_cluster_send_queue_bytesgaugeOutbound send-queue occupancy across all peer connections, right now. Bounded by send_queue_bytes in [cluster] config (default 64MB) per connection; a value pinned near that bound is backpressure, not idle capacity.
scramdb_cluster_partition_drops_totalcounterFrames dropped by a test partition, outbound and inbound combined. Zero on every normal deployment.
scramdb_cluster_fragment_boot_refusals_totalcounterFragment requests from other nodes answered with a refusal because this node was still starting.
scramdb_cluster_forward_boot_refusals_totalcounterRequests other nodes forwarded to this node for the metadata group (a DDL, a lock, a node number claim, a member removal) answered as not made because this node was still starting: the sender asks the next node at once instead of waiting out its forward timeout.
scramdb_cluster_handshake_latency_secondshistogramPeer handshake latency. 11 buckets from 1ms to 5s (le="0.001" through le="5.000") plus +Inf.
scramdb_cluster_connections_closed_totalcounterPeer connections closed, by cause: ping_deadline (no byte from the peer within its dead-peer deadline, the stalled-process case), tcp_timeout (the kernel's user timeout or keepalive gave up), peer_closed, local (shutdown or a duplicate connection), io_error, protocol_error.
scramdb_cluster_frames_dropped_on_close_totalcounterFrames still queued on a peer connection when it closed, dropped and reported to the subsystems that sent them.
scramdb_cluster_peer_events_totalcounterPeer connection events published to the cluster's subsystems, by kind: connected and lost.
scramdb_cluster_peer_event_lag_totalcounterTimes a subsystem missed connection events and refreshed its view of every peer. Zero in normal operation.
scramdb_cluster_liveness_pings_sent_totalcounterLiveness pings sent on peer connections that had sent nothing for ping_interval.
scramdb_cluster_liveness_ping_rtt_secondshistogramRound trip of a liveness ping, from its send to its answer. 18 buckets from 10µs to 10s plus +Inf.
scramdb_cluster_dead_peer_deadline_connectionsgaugeOpen peer connections by how their dead-peer deadline is set, source: floor (the adaptive deadline at dead_peer_timeout_min), measured (lifted by a long round trip), ceiling (held at dead_peer_timeout_max), pinned (a fixed dead_peer_timeout).
scramdb_cluster_dead_peer_deadline_secondshistogramEvery dead-peer deadline put in force, at a connection's start and whenever its measured round trip moves it.
scramdb_cluster_socket_option_failures_totalcounterSocket options (TCP_NODELAY, keepalive, user timeout) the kernel refused on a peer connection. Zero on Linux.
scramdb_cluster_peers_rememberedgaugePeers whose connection closed within the grace period and whose handshake details are kept.
scramdb_cluster_peers_remembered_expired_totalcounterRemembered peers dropped after staying disconnected past the grace period.
scramdb_cluster_peers_left_totalcounterPeers forgotten because they left the cluster membership.
scramdb_cluster_dial_backsgaugePeers this node is dialing back because this node must open their connection.
scramdb_cluster_dial_backs_started_totalcounterTimes this node started dialing a peer back.
scramdb_cluster_dial_backs_ended_totalcounterTimes this node stopped dialing a peer back.
scramdb_cluster_peer_identity_refusals_totalcounterPeer connections refused because the certificate does not carry the node name the peer announced. Nonzero means a node's certificate is missing its name; see TLS between cluster nodes.
scramdb_cluster_duplicate_connections_refused_totalcounterPeer connections refused because this node already accepted one from the same peer on the same lane. A peer that opens a lane twice at once (two seed addresses for one node, or a reconnect racing the connection it replaces) keeps the first and drops the second, so a few are normal; a steady climb means a peer keeps reopening a lane it already holds.
scramdb_cluster_transport_threadsgaugeThreads the cluster transport runs for its listeners, dials, handshakes and peer connections.
scramdb_cluster_transport_thread_start_failures_totalcounterThreads the cluster transport could not start.

Traffic lanes​

Every pair of nodes talks over three connections, one per traffic lane, each on its own port: control (membership, consensus, the commit protocol, forwarded DDL and lock calls, shuffle fetches), interactive (replies to forwarded point reads) and bulk (query data exchange, fragment dispatch with its broadcast join side, snapshot transfer, the rows a cluster COPY stages). A message larger than [cluster.transport] chunk_size travels as chunks, and a lane's connection takes its streams in turn a chunk at a time. Every stream has its own window of its lane, so a subsystem that stops reading holds only its window and stops only its own senders.

MetricTypeMeaning
scramdb_cluster_lane_bytes_sent_totalcounterBytes written to peers, by lane.
scramdb_cluster_lane_frames_sent_totalcounterFrames written to peers, by lane.
scramdb_cluster_lane_bytes_received_totalcounterBytes read from peers, by lane.
scramdb_cluster_lane_frames_received_totalcounterFrames read from peers, by lane.
scramdb_cluster_lane_chunks_sent_totalcounterChunks of large messages written to peers, by lane.
scramdb_cluster_lane_chunks_received_totalcounterChunks of large messages read from peers, by lane.
scramdb_cluster_lane_messages_reassembled_totalcounterLarge messages reassembled from chunks, by lane.
scramdb_cluster_lane_credit_frames_sent_totalcounterFrames that returned stream credit to a peer, by lane.
scramdb_cluster_lane_connectionsgaugeOpen peer connections, by lane.
scramdb_cluster_lane_older_protocol_connectionsgaugeOpen peer connections that speak an older cluster protocol version, by traffic lane.
scramdb_cluster_lane_protocol_fallbacks_totalcounterDials made again with the older cluster protocol version a peer asked for, by traffic lane.
scramdb_cluster_lane_stream_window_bytesgaugeReceive window granted to each stream on the newest connection, by lane: the lane's size shared by the peers then connected.
scramdb_cluster_lane_queued_bytesgaugeBytes held by each traffic queue, by lane and direction: send (queued to send) or receive (received and not yet consumed).
scramdb_cluster_lane_capacity_bytesgaugeSize of each traffic queue, by lane and direction.
scramdb_cluster_lane_backpressure_totalcounterSends refused because a traffic queue or a stream window was full, by lane and stream (membership, metadata_consensus, shard_consensus, commit_protocol, ddl_forward, lock_forward, shuffle_fetch, point_read, exchange, fragment_dispatch, snapshot_transfer, copy_staging, other).
scramdb_cluster_lane_drain_waits_totalcounterSends that waited for a full traffic queue to drain instead of failing.
scramdb_cluster_lane_drain_wait_secondshistogramTime a send waited for a full traffic queue to drain.
scramdb_cluster_lane_connection_threadsgaugeReader and writer threads of peer connections, by lane: two per open connection.

backpressure_total on the bulk lane is flow control doing its job under a large query. On the control lane it should stay at zero: a nonzero shard_consensus or metadata_consensus count means consensus traffic found its queue full, and each such event is also logged once.

Raft group health and recovery​

MetricTypeMeaning
scramdb_cluster_group_quarantines_totalcounterRaft group storage directories quarantined: at boot for deeper corruption (a bad header, a mid-prefix CRC failure, an undecodable hard state), or at runtime when a group stopped for a reason a restart cannot fix. Does not count a benign torn-tail truncation; that's a row below.
scramdb_cluster_quarantined_groupsgaugeGroups currently quarantined, awaiting a membership-replace rejoin. This is the line to alert on.
scramdb_cluster_group_rejoins_totalcounterQuarantined groups that completed their membership-replace rejoin.
scramdb_cluster_term_changes_totalcounterMetadata group (group0) raft term changes this node has observed: an election, a term bump from a higher-term message, or a step-down. Node-local, not a cluster-wide total: each node counts its own view of the same election.
scramdb_cluster_group_actor_restarts_totalcounterShard groups this node restarted in place after a transient stop, within a bounded restart budget; one that keeps stopping is quarantined instead.
scramdb_cluster_group_disk_full_resumes_totalcounterShard groups this node resumed in place after a full disk, once a free-space recheck found room again. Never counted as a quarantine or a rejoin: the group's storage was never touched.
scramdb_cluster_torn_tail_truncations_totalcounterBenign torn-tail truncations at raft log open: the normal crash-recovery path, made visible instead of silent.

scramdb_cluster_quarantined_groups is restart-honest. It is never incremented or decremented as events happen; every host-reconcile round recomputes it from scratch, from the actual set of groups gated right now, and overwrites the gauge with that count. A crash that loses an in-flight update, or a process restart mid-recovery, can't leave this gauge stuck on a stale nonzero reading or silently reset to a wrong zero: the very next reconcile round always reflects what's really quarantined at that moment, never a running total of past events.

Consensus threads​

Every replication group and the metadata group of a node run on one pool of consensus threads ([cluster.consensus] threads), never a thread per group. A group runs one step at a time and hands its thread back, so a slow group holds one thread and never another group.

MetricTypeMeaning
scramdb_cluster_consensus_pool_threadsgaugeThreads of the pool that drives the consensus groups.
scramdb_cluster_consensus_pool_queuedgaugeConsensus groups and cluster tasks waiting for a consensus thread. Near zero on a healthy node; a value that stays high means the threads cannot keep up.
scramdb_cluster_consensus_pool_panics_totalcounterConsensus groups and cluster tasks that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log.
scramdb_cluster_consensus_pool_idle_wakes_totalcounterTimes an idle consensus thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer.
scramdb_cluster_consensus_mailbox_refused_totalcounterMessages a consensus group refused because its queue was full or closed.

Exchange threads​

The distributed query work other nodes send a node (their fragments, the rows arriving, and the results sent back) runs on its own pool of exchange threads ([cluster.exchange] threads), never on the consensus threads and never on the threads that run queries.

MetricTypeMeaning
scramdb_cluster_exchange_pool_threadsgaugeThreads of the pool that runs distributed query work sent by other nodes.
scramdb_cluster_exchange_pool_queuedgaugeDistributed query tasks waiting for an exchange thread. Near zero on a healthy node; a value that stays high means the threads cannot keep up.
scramdb_cluster_exchange_pool_panics_totalcounterDistributed query tasks that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log.
scramdb_cluster_exchange_pool_idle_wakes_totalcounterTimes an idle exchange thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer.

Cluster services​

Requests other nodes send this node's cluster services (lock and DDL forwarding, snapshot transfer, query fragments and shuffle fetches) are served on engine threads, each service with its own bound on the requests it serves at once.

MetricTypeMeaning
scramdb_cluster_stream_requests_totalcounterRequests from other nodes this node's cluster services started serving.
scramdb_cluster_stream_requests_in_flightgaugeRequests from other nodes this node's cluster services are serving now.
scramdb_cluster_stream_requests_waited_totalcounterRequests that waited for a free place because their service was at its bound.
scramdb_cluster_stream_frames_refused_totalcounterFrames dropped because their service had stopped or its queue was full.

Cluster-wide LOCK TABLE grants and transaction advisory locks are released at the end of the transaction or the session that holds them, a session that ends with its transaction open included:

MetricTypeMeaning
scramdb_cluster_lock_grants_released_totalcounterCluster-wide lock grants given back at a transaction or session end or a savepoint rollback.
scramdb_cluster_lock_grant_release_failures_totalcounterCluster-wide lock grants whose release at the end of a transaction or session failed.
scramdb_cluster_lock_detached_releases_totalcounterSessions that ended holding cluster-wide lock grants, released after they closed.
scramdb_cluster_lock_wait_withdrawals_totalcounterCluster-wide lock waits that ended without a grant and withdrew their registered wait.
scramdb_cluster_lock_wait_withdrawal_failures_totalcounterCluster-wide lock waits whose registered wait could not be withdrawn.
scramdb_cluster_lock_wait_registration_failures_totalcounterCluster-wide lock waits left unregistered, which end at their lock timeout when they cannot be granted.

Group commit and durability​

MetricTypeMeaning
scramdb_cluster_group_commit_ops_totalcounterRaft ops served, summed across every group-commit wake-up. Divide by group_commit_wakeups_total for the achieved batch size. A lone proposal (nothing to batch with) contributes 1 op and 0 wake-ups, so an idle system reports no batching instead of a fake 1.0.
scramdb_cluster_group_commit_wakeups_totalcounterWake-ups that served more than one op, i.e. wake-ups that actually batched.
scramdb_cluster_group_commit_max_batchgaugeThe largest batch any wake-up has served so far, a high-water mark. Pinned at your configured cap says "raise the cap"; comfortably under it says the batching window is already draining faster than proposals arrive.
scramdb_cluster_durability_flushes_totalcounterDurability flushes, one per fsync. Divide into group_commit_ops_total for fsyncs-per-commit, the number batching exists to lower.
scramdb_cluster_log_compactions_totalcounterRaft log compactions performed.
scramdb_cluster_log_entries_compacted_totalcounterRaft log entries reclaimed by compaction, summed.

Zero log_compactions_total is normal on a cluster whose log never reaches the compaction trigger. A flat zero on a cluster that's otherwise busy (group_commit_ops_total climbing) is not normal: compaction is waiting, either for a state snapshot that keeps failing (state_snapshot_failures_total or state_snapshots_abandoned_total climbing: a full disk, or a [cluster.log_compaction] memory_bytes budget too small for the group's state) or for a replica that is behind but still inside the lag limit. A replica lagging further than replica_lag_limit_entries no longer holds the log: when it returns it is repaired from a state snapshot, so a dead replica bounds the log instead of growing it forever. The waiting case also logs on its own, but the metric is what you'd wire an alert to; see Alerting on cluster metrics below.

Shared raft log​

Every shard group a node hosts writes its replication log into the node's one shared raft log: one writer makes a whole batch of appends from every group durable with one data sync, so a transaction that touches several groups costs the node one sync, not one per group. Its segment files are reused a whole segment at a time once nothing in them is needed; [cluster.consensus] log_segment_size sets their size (by default a share of the data volume). The metadata group keeps its own log.

MetricTypeMeaning
scramdb_cluster_shared_log_drains_totalcounterBatches the writer of the node's shared raft log wrote, each ending in one sync.
scramdb_cluster_shared_log_syncs_totalcounterData syncs of the node's shared raft log.
scramdb_cluster_shared_log_group_syncs_totalcounterGroups whose writes a sync of the node's shared raft log covered, summed over its syncs. Divided by syncs_total, the mean groups one sync made durable.
scramdb_cluster_shared_log_records_totalcounterRecords written to the node's shared raft log.
scramdb_cluster_shared_log_bytes_written_totalcounterBytes written to the node's shared raft log.
scramdb_cluster_shared_log_copied_bytes_totalcounterEntry payload bytes copied into write buffers of the node's shared raft log; larger payloads are written in place.
scramdb_cluster_shared_log_truncations_totalcounterTruncate records written for a replica's conflicting log suffix.
scramdb_cluster_shared_log_segments_created_totalcounterSegment files the node's shared raft log started.
scramdb_cluster_shared_log_segments_removed_totalcounterSegment files of the node's shared raft log removed once nothing in them was needed.
scramdb_cluster_shared_log_dir_sync_failures_totalcounterFailed directory syncs after the node's shared raft log removed segments; each is retried at the next write.
scramdb_cluster_shared_log_compactions_requested_totalcounterGroups asked to compact their log because they held the oldest segment of the node's shared raft log.
scramdb_cluster_shared_log_relocated_records_totalcounterRecords rewritten at the head of the node's shared raft log so an old segment could be removed.
scramdb_cluster_shared_log_relocated_bytes_totalcounterBytes rewritten at the head of the node's shared raft log so an old segment could be removed.
scramdb_cluster_shared_log_sync_failures_totalcounterFailed writes or syncs of the node's shared raft log; the first one stops every group on the node.
scramdb_cluster_shared_log_torn_bytes_totalcounterBytes of an unfinished write cut from the end of the node's shared raft log at start.
scramdb_cluster_shared_log_migrated_groups_totalcounterGroups whose own raft log was moved into the node's shared raft log.
scramdb_cluster_shared_log_segmentsgaugeSegment files of the node's shared raft log.
scramdb_cluster_shared_log_bytesgaugeBytes of the node's shared raft log on disk.
scramdb_cluster_shared_log_live_bytesgaugeBytes of the node's shared raft log still needed by a group.
scramdb_cluster_shared_log_groupsgaugeGroups with records in the node's shared raft log or a store open on it.
scramdb_cluster_shared_log_stoppedgauge1 once a failed write or sync stopped the node's shared raft log, else 0.
scramdb_cluster_shared_log_segment_bytesgaugeThe segment size of the node's shared raft log in force.
scramdb_cluster_shared_log_scan_millisecondsgaugeHow long the node's shared raft log took to read its segments at start.
scramdb_cluster_shared_log_sync_secondshistogramTime one data sync of the node's shared raft log took.
scramdb_cluster_shared_log_drain_secondshistogramTime from the start of a batch's write to its sync returning.

shared_log_stopped at 1 is the one to page on: a write or sync of the log failed, every shard group on the node stopped rather than trust a disk whose state is no longer known, and the node must be restarted (nothing is retried). shared_log_bytes far above shared_log_live_bytes while compactions_requested_total climbs means an idle group holds the oldest segment; the log asks it to compact and then moves its few live records forward, which relocated_bytes_total counts.

State snapshots and log compaction​

A data group compacts its replication log behind a snapshot of its commit state: the open transactions' locks and staged writes, the recent versions of every row it wrote, and its read and commit watermarks, taken at one applied log position. A restart restores the newest snapshot and replays only the log above it; a replica that fell behind the compaction point installs the leader's snapshot and rebuilds the rows it missed from it instead of copying the whole group. The cluster metadata log compacts the same way. When a group compacts, how far a replica may lag and how much memory snapshots may hold are set under [cluster.log_compaction]; each count adapts to the group's own snapshot and entry sizes unless you pin it.

MetricTypeMeaning
scramdb_cluster_state_snapshots_taken_totalcounterState snapshots taken for a log compaction.
scramdb_cluster_state_snapshot_failures_totalcounterState snapshots that could not be taken (an I/O error, a full disk).
scramdb_cluster_state_snapshots_abandoned_totalcounterState snapshots given up to stay within their memory budget.
scramdb_cluster_state_snapshot_last_bytesgaugeSize of the most recent state snapshot taken on this node.
scramdb_cluster_state_snapshot_bytes_totalcounterBytes of every state snapshot taken.
scramdb_cluster_state_snapshot_duration_secondshistogramTime from the start of a state snapshot to its completion. A snapshot is written in small steps between applied batches, so the group keeps applying while it runs.
scramdb_cluster_state_snapshot_step_secondshistogramTime one of those steps held a group's apply.
scramdb_cluster_state_snapshot_memory_bytesgaugeMemory held right now by state snapshots being taken, sent or received.
scramdb_cluster_state_snapshot_memory_limit_bytesgaugeMemory they may hold at once (memory_bytes, or the derived default).
scramdb_cluster_state_snapshot_restores_totalcounterGroups restored from a state snapshot at startup.
scramdb_cluster_state_snapshot_missing_totalcounterGroups found at startup with a compacted log and no state snapshot to restore from. Zero in a healthy deployment; anything else means that replica needs a full copy.
scramdb_cluster_state_snapshot_restore_secondshistogramTime to restore one group's state at startup.
scramdb_cluster_log_entries_replayed_totalcounterLog entries replayed at startup on top of the restored state.
scramdb_cluster_log_torn_tail_replays_totalcounterGroups found at startup with their log cut below what they had applied (a crash, or a disk that lost part of the log). The group takes the cut entries from its peers again and replays them without applying their rows a second time.
scramdb_cluster_log_compactions_awaiting_snapshot_totalcounterLog compactions held back until a state snapshot covers the open transactions.
scramdb_cluster_log_compactions_past_lagging_replica_totalcounterLog compactions that passed a replica lagging beyond the lag limit. That replica is repaired from the snapshot when it returns.
scramdb_cluster_state_snapshot_installs_totalcounterState snapshots from a leader installed on this node.
scramdb_cluster_state_snapshot_installs_incomplete_totalcounterSnapshot installs whose skipped rows come from a full copy (every install: a snapshot carries no row).
scramdb_cluster_state_snapshot_installs_resumed_totalcounterFull copies resumed at startup after a restart interrupted them.
scramdb_cluster_state_snapshot_installs_superseded_totalcounterPending full copies replaced by a newer install before they completed.
scramdb_cluster_state_snapshot_install_intent_failures_totalcounterSnapshot installs left unapplied because their recovery record could not be written. The install is tried again; a rate points at the data volume.
scramdb_cluster_state_snapshot_fetches_totalcounterState snapshots fetched from a peer, in bounded chunks.
scramdb_cluster_state_snapshot_fetch_failures_totalcounterState snapshot fetches from a peer that failed; the fetch retries with a growing pause.
scramdb_cluster_state_snapshot_fetched_bytes_totalcounterBytes of state snapshots fetched from peers.
scramdb_cluster_state_snapshot_chunks_served_totalcounterState snapshot chunks served to peers.
scramdb_cluster_replica_copies_started_totalcounterFull copies of a shard group's rows this node started after a snapshot install.
scramdb_cluster_replica_copies_completed_totalcounterFull copies of a shard group's rows this node completed.
scramdb_cluster_replicas_awaiting_rowsgaugeShard group replicas on this node waiting for a full copy; they serve no read. A value that stays above zero means a copy cannot finish: check replica_copy_attempt_failures_total here and replica_copy_build_failures_total on the group's voters.
scramdb_cluster_replica_copy_attempt_failures_totalcounterAttempts to copy a shard group's rows from a peer that failed and were retried.
scramdb_cluster_replica_copy_rows_totalcounterRows this node received in full copies of shard groups.
scramdb_cluster_replica_copy_bytes_totalcounterBytes this node received in full copies of shard groups.
scramdb_cluster_replica_copy_secondshistogramTime from a snapshot install to its full copy being complete.
scramdb_cluster_replica_copies_built_totalcounterFull copies of a shard group's rows this node built for a peer.
scramdb_cluster_replica_copy_build_failures_totalcounterFull copies of a shard group's rows this node could not build for a peer.
scramdb_cluster_replica_copy_chunks_served_totalcounterChunks of full copies this node served to peers.
scramdb_cluster_metadata_log_compactions_totalcounterCluster metadata log compactions.
scramdb_cluster_metadata_state_restores_totalcounterCluster metadata states restored or installed from a state snapshot.
scramdb_cluster_log_compaction_trigger_entriesgaugeRetained log entries at which a data group last decided to compact.
scramdb_cluster_replica_lag_limit_entriesgaugeLog entries a replica may lag before it stops holding the leader's log.
scramdb_cluster_log_compaction_trigger_sourcegaugeWhy the trigger has its value: 0 configured, 1 the floor (nothing measured yet, or measured below it), 2 measured from the group's snapshot and entry sizes, 3 the ceiling.

Cluster archive​

With [storage.wal.archive] enabled, each shard group's leader ships the group's committed log, and the metadata group's leader the metadata log, to the archive destination, for cluster restore. Each segment lands under a key only one leader can create, so a leader that lost its leadership can never overwrite what the new one shipped, and once it finds the group archived in a newer term it stops archiving that group.

MetricTypeMeaning
scramdb_cluster_archive_segments_totalcounterLog segments this node shipped to the cluster archive.
scramdb_cluster_archive_entries_totalcounterLog entries this node shipped to the cluster archive.
scramdb_cluster_archive_bytes_totalcounterBytes of log segments this node shipped to the cluster archive.
scramdb_cluster_archive_fenced_totalcounterLog segments refused because another leader had already archived that log position.
scramdb_cluster_archive_superseded_totalcounterGroups this node stopped archiving because a leader of a newer term archives them.
scramdb_cluster_archive_bases_totalcounterGroup bases (a state image and a full copy of the rows) this node shipped.
scramdb_cluster_archive_images_totalcounterGroup state images this node shipped alone, between bases.
scramdb_cluster_archive_gaps_totalcounterTimes a group's log was compacted past the archive and no base could be shipped.
scramdb_cluster_archive_errors_totalcounterFailed attempts to write to the cluster archive.
scramdb_cluster_archive_pruned_totalcounterArchive objects the retention pass removed.
scramdb_cluster_archive_time_samples_totalcounterTime index samples this node shipped to the cluster archive.
scramdb_cluster_archive_fence_asks_totalcounterFences this node asked of idle groups behind its newest acknowledged commit.
scramdb_cluster_archive_cycles_totalcounterPasses this node's archiver made over its groups and its time index.
scramdb_cluster_archive_lag_entriesgaugeCommitted log entries of the groups this node leads that are not archived yet.
scramdb_cluster_archive_groupsgaugeGroups whose log this node ships now.
scramdb_cluster_archive_ship_secondshistogramTime to ship one log segment to the cluster archive.

A few fenced_total around a leader change are expected: the old leader's last attempt is refused because the new leader already archived that position. gaps_total above zero means a group's history has a hole a restore cannot cross (the group compacted its log past what was archived and no base could be shipped): a restore to a point inside that hole is refused, and points after the group's next base restore again. lag_entries growing without bound means the destination cannot keep up. cycles_total counts the archiver's passes, and fence_asks_total the requests it made to a group that stood behind its newest acknowledged commit so that the latest restorable point reaches that commit; the second stays flat on a cluster that is idle (Cluster Restore).

Distributed transaction commit​

What the commit protocol of distributed transactions ([cluster.dilith] in the config file) did on this node.

MetricTypeMeaning
scramdb_cluster_dilith_commits_totalcounterTransactions committed.
scramdb_cluster_dilith_aborts_totalcounterTransaction attempts aborted, by reason: conflict (another transaction held a pending write or a read fence the vote needed), stale_read (a version above a read position, or a read below the retention floor), outdated_vote (older than the group's memory of decisions), wait_die (the younger of two transactions on one row), recovery (a recovery found a group that never voted).
scramdb_cluster_dilith_commit_latency_secondshistogramTime from a commit request to its answer.
scramdb_cluster_dilith_vote_round_secondshistogramTime from sending a transaction's votes to its last vote answer.
scramdb_cluster_dilith_votes_appended_totalcounterVote entries appended by group leaders.
scramdb_cluster_dilith_precheck_refusals_totalcounterVotes a group leader refused before appending anything.
scramdb_cluster_dilith_parks_totalcounterRequests parked at a group behind an undecided transaction.
scramdb_cluster_dilith_write_free_certified_totalcounterTransactions that wrote nothing certified without a log entry.
scramdb_cluster_dilith_fences_appended_totalcounterFence entries appended for analytical cuts.
scramdb_cluster_dilith_fences_avoided_totalcounterAnalytical cuts a group already covered without a fence entry.
scramdb_cluster_dilith_recoveries_started_totalcounterRecoveries started for undecided transactions.
scramdb_cluster_dilith_recoveries_decided_totalcounterRecoveries that reached a verdict.
scramdb_cluster_dilith_recoveries_behind_waiter_totalcounterRecoveries started by a request that waited too long behind an undecided transaction.
scramdb_cluster_dilith_recoveries_abandoned_vote_totalcounterRecoveries started for votes pending too long with nobody waiting.
scramdb_cluster_dilith_recoveries_handed_over_totalcounterRecoveries handed to the group leader that holds the pending vote.
scramdb_cluster_dilith_recoveries_from_record_totalcounterRecoveries decided from the leader's own record of the transaction.
scramdb_cluster_dilith_status_probes_totalcounterStatus queries for transactions whose outcome a client did not receive.
scramdb_cluster_dilith_sweeps_totalcounterSweep entries appended to decide transactions that never voted at a group.
scramdb_cluster_dilith_sweeps_refused_totalcounterSweeps refused because they were older than the group's memory of decisions.
scramdb_cluster_dilith_sweeps_resent_totalcounterSweeps sent again with a fresher decision count.
scramdb_cluster_dilith_votes_resent_totalcounterVote requests sent again to groups that had not answered, one per group.
scramdb_cluster_dilith_outcomes_resent_totalcounterTransaction outcomes sent again to groups that had not acknowledged them, one per group.
scramdb_cluster_dilith_aborts_without_record_totalcounterAbort outcomes acknowledged without recording anything at a group holding no record of the transaction.
scramdb_cluster_dilith_decisions_forgotten_totalcounterDecisions forgotten once outside the decision window.
scramdb_cluster_dilith_staged_vote_rounds_totalcounterVote rounds that asked the groups likely to refuse before the others.
scramdb_cluster_dilith_copy_stage_entries_totalcounterCOPY stage entries applied on this node's replicas.
scramdb_cluster_dilith_copy_stages_dropped_totalcounterCOPY stages dropped on this node's replicas without a commit.
scramdb_cluster_dilith_copy_stage_vote_refusals_totalcounterVotes refused because the COPY rows they name were not all staged or conflicted.
scramdb_cluster_dilith_copy_stages_collected_totalcounterCOPY stages a group leader on this node dropped after their owner stopped renewing them.
scramdb_cluster_dilith_copy_stages_refused_full_totalcounterCOPY stage entries a group leader on this node refused because its staging area had no room.
scramdb_cluster_dilith_copy_stages_refused_stale_totalcounterCOPY stage entries a group leader on this node sent back to be checked again because the table moved on since their rows were checked.
scramdb_cluster_dilith_owed_commits_collected_totalcounterCommits whose discharge was lost that a group leader on this node collected.
scramdb_cluster_dilith_block_verdicts_released_totalcounterCommits of this node's transaction blocks whose verdict was held until the block's end learned the outcome and released it.
scramdb_cluster_dilith_block_verdicts_heldgaugeCommits on this node's replicas held for their transaction block's end, which no age collects.
scramdb_cluster_dilith_table_entriesgaugeEntries each commit protocol table on this node holds, by table.
scramdb_cluster_dilith_owed_commitsgaugeCommits on this node's replicas not yet discharged.
scramdb_cluster_dilith_pending_votesgaugeVotes accepted on this node's replicas and not yet decided.
scramdb_cluster_dilith_recent_verdictsgaugeRecent transaction outcomes this node remembers to answer recoveries.
scramdb_cluster_dilith_discharges_queuedgaugeDischarge notices waiting for their batch to be sent.
scramdb_cluster_dilith_refusing_groupsgaugeGroups that refused this node's votes recently.
scramdb_cluster_dilith_state_bytesgaugeMemory the commit protocol's tables hold on this node, by part: pending_votes (votes accepted and not decided, with their writes and read fences), owed_commits (commits not yet discharged), kept (the decisions and versions the decision and retention windows keep, and COPY stage records), outcomes (outcomes and discharges a group leader holds for its next entry or has in flight), reserved (room reserved for vote appends in flight), waiting (requests parked at a group, and votes waiting for room), scratch (the node's reusable decode and step buffers).
scramdb_cluster_dilith_state_share_bytesgaugeMemory the commit protocol's tables may hold on this node ([cluster.dilith] protocol_state_memory).
scramdb_cluster_dilith_state_peak_bytesgaugeMost memory the commit protocol's tables and reserved room have held on this node.
scramdb_cluster_dilith_votes_waiting_for_memorygaugeVotes waiting on this node for the commit protocol's memory share to have room.
scramdb_cluster_dilith_memory_parks_totalcounterVotes a group leader on this node held until the commit protocol's memory share had room.
scramdb_cluster_dilith_memory_past_share_totalcounterVotes a group leader on this node let past a full memory share so the oldest waiting transaction could go on.

aborts_total{reason="conflict"} and {reason="wait_die"} count contention between transactions on the same rows. A recoveries_started_total that keeps growing on a healthy network means transactions are being left undecided, and is worth a look at the nodes' logs. copy_stages_collected_total rising means COPY statements stop or lose their node while staging; copy_stage_vote_refusals_total rising means a COPY's vote meets concurrent writers of the rows it stages, and the COPY fails with 40001. copy_stages_refused_stale_total counts batches a COPY checked again because its groups moved on while the batch travelled; the COPY goes on. copy_stages_refused_full_total rising means the staging area is too small for the COPYs running (53100). owed_commits_collected_total rising means nodes stop right after answering commits; every such commit is finished by its groups. A transaction block's commit that changed the schema is held on its groups until the block's own end has learned whether it committed, however late that is: block_verdicts_held counts them and block_verdicts_released_total their releases. A held count that keeps growing while releases stay flat means blocks whose COMMIT was answered in doubt cannot learn their outcome yet, usually a node cut off from the cluster; they are released when it can, or when that node leaves the cluster. table_entries shows every table of the commit protocol and what it holds: each one stays bounded by the work in flight.

The commit protocol's tables live in a memory share of their own (state_share_bytes), counted in the process memory budget. A group leader appends a new vote only while the share has room; a vote that finds none waits, oldest transaction first, and is appended as soon as a decision, a discharge or the protocol's windows free room. It is never refused for memory. votes_waiting_for_memory above zero for long, or memory_parks_total climbing, means the share is too small for the transactions in flight: raise [cluster.dilith] protocol_state_memory. memory_past_share_total counts the oldest waiting transaction let through a full share when nothing on the node could free room without it, which is what keeps the cluster moving; its rise says the same.

The commit protocol on this node​

The commit protocol runs as one task on the node's consensus threads, one run at a time, and serves every shard group this node holds a replica of.

MetricTypeMeaning
scramdb_cluster_dilith_run_secondshistogramTime one run of the commit protocol's host took.
scramdb_cluster_dilith_runs_totalcounterRuns of the commit protocol's host.
scramdb_cluster_dilith_run_inputs_totalcounterInputs the commit protocol's host stepped, over all runs.
scramdb_cluster_dilith_queuedgaugeInputs waiting for the commit protocol's host, by kind: control, decide, frame, client.
scramdb_cluster_dilith_step_errors_totalcounterInputs the commit protocol could not step, by input: apply, message, timer, leadership, append_decided, restored, client, hosting.
scramdb_cluster_dilith_reply_slotsgaugeClient calls waiting for an answer from the commit protocol.
scramdb_cluster_dilith_memory_bytesgaugeBytes the commit protocol holds on this node, as last measured.
scramdb_cluster_dilith_appends_in_flightgaugeLog entries the commit protocol proposed whose decision has not come back.
scramdb_cluster_dilith_timersgaugeTimers the commit protocol keeps armed.
scramdb_cluster_dilith_frames_refused_totalcounterFrames the transport refused to send; the protocol sends again.
scramdb_cluster_dilith_appends_lost_totalcounterProposed log entries lost because the group's leader could not take them.
scramdb_cluster_commits_in_doubt_totalcounterCommits whose outcome was not known when the client stopped waiting.
scramdb_cluster_dilith_commit_progress_marks_totalcounterProgress marks of commit rounds that reached a waiting client: a vote recorded, or a vote applied on this node.
scramdb_cluster_dilith_commit_wait_renewals_totalcounterTimes a waiting commit kept waiting past its patience because its round had progressed.
scramdb_cluster_dilith_commits_in_flightgaugeCommits this node's commit protocol drives whose outcome is not decided yet, as last measured.
scramdb_cluster_dilith_commits_abandonedgaugeCommits still in flight here whose client stopped waiting for the outcome, as last measured.
scramdb_cluster_dilith_calls_abandoned_totalcounterCalls to the commit protocol whose caller stopped waiting, forgotten with what waited to answer them.
scramdb_cluster_dilith_outcome_waits_totalcounterWaits for the commits already answered to reach every group.
scramdb_cluster_dilith_outcome_wait_timeouts_totalcounterWaits for commit outcomes that gave up at their deadline with a group yet to acknowledge.
scramdb_cluster_dilith_outcomes_owedgaugeCommits answered to their client whose outcome some group has yet to acknowledge, as last measured.
scramdb_cluster_dilith_stage_rounds_resent_totalcounterRounds of staged rows sent again because their answers did not come in time.
scramdb_cluster_dilith_copy_stage_entries_wide_totalcounterCOPY stage entries a group leader on this node appended without their keys because its memory share had no room for them.
scramdb_cluster_dilith_stage_requests_lost_totalcounterRequests of staged rows asked again at once because their shard group changed leader.
scramdb_cluster_dilith_host_stoppedgauge1 when the commit protocol on this node stopped after a failure.
scramdb_cluster_dilith_hosted_groupsgaugeShard groups the commit protocol hosts on this node.
scramdb_cluster_dilith_groups_hosted_totalcounterShard groups the commit protocol started hosting on this node.
scramdb_cluster_dilith_groups_released_totalcounterShard groups the commit protocol stopped hosting on this node.
scramdb_cluster_dilith_decide_requests_dropped_totalcounterShard group requests dropped because the group is not hosted here or started again.
scramdb_cluster_dilith_node_numbergaugeThis node's number in transaction ids, 0 until the cluster gave it one.
scramdb_cluster_dilith_node_number_claims_totalcounterRequests this node made to the cluster for its node number.
scramdb_cluster_dilith_node_number_waits_totalcounterCommit attempts that waited for this node's node number instead of starting at once.

dilith_host_stopped at 1 means the node can no longer commit transactions and must be restarted; every waiting client was answered with an error. A commits_in_doubt_total that grows means clients stopped waiting before their commit's outcome was known: the commit may still have happened. A commit waits as long as its round keeps progressing ([cluster.transactions]): commit_progress_marks_total counts the steps that kept clients waiting and commit_wait_renewals_total the waits that went past their patience because of them, so renewals that grow while commits_in_doubt_total stays flat are large or slow commits that still finish, and in-doubt commits that grow without marks are rounds that stalled. stage_rounds_resent_total grows when a node sending a COPY's rows, or a transaction's rows too large for one commit message, lost a round to a leader change it could not see or to a node that stopped; one that grows while no leader changed means rounds are taking far longer than the rounds before them.

Committed entries through the commit protocol​

Each shard group hands its committed log entries to the commit protocol, which decides the rows they install, and hands the protocol's own entries to the group's log.

MetricTypeMeaning
scramdb_cluster_dilith_decide_requests_totalcounterBatches of committed entries the data groups handed to the commit protocol.
scramdb_cluster_dilith_decide_entries_totalcounterCommitted entries the data groups handed to the commit protocol.
scramdb_cluster_dilith_decided_rows_totalcounterRows the commit protocol decided for the data groups to install.
scramdb_cluster_dilith_decide_outstandinggaugeRequests of the data groups the commit protocol has not answered yet.
scramdb_cluster_dilith_decide_secondshistogramTime from handing committed entries to the commit protocol to receiving their rows.
scramdb_cluster_dilith_decide_failures_totalcounterData groups stopped because the commit protocol could not decide an entry of theirs.
scramdb_cluster_dilith_leadership_reports_totalcounterLeader changes of the data groups reported to the commit protocol.
scramdb_cluster_dilith_proposal_batches_totalcounterBatches of commit protocol entries handed to a data group's log.
scramdb_cluster_dilith_proposal_batches_refused_totalcounterBatches of commit protocol entries a data group's log could not take.

Applying committed writes​

A committed entry is applied in two steps on the node's apply threads ([cluster.apply] threads), never on the threads that run consensus, replication and client connections. First it is decided: the commit protocol steps it in memory, in log order (the session that committed it already has its answer, given once every group the transaction touched held a durable yes vote). Then its rows are written, and one data log sync per batch makes a whole batch of them durable before the group's applied position moves. Reads wait for that applied position, so a row is never read before it is applied and synced. On a cluster node the consensus threads keep a share of the cores of their own ([cluster.apply] reserve_io_cores), so heavy queries cannot delay them.

MetricTypeMeaning
scramdb_cluster_apply_decide_to_durable_secondshistogramTime from deciding a batch of committed entries to its effects being durable: how far the applied position trails the answers sessions already have.
scramdb_cluster_apply_effect_batches_totalcounterBatches of committed entries whose effects were made durable.
scramdb_cluster_apply_effect_entries_totalcounterCommitted entries whose effects were made durable. Divide by apply_effect_batches_total for the mean batch size.
scramdb_cluster_apply_data_syncs_totalcounterData log syncs the apply path requested for those effects. Divide by apply_effect_batches_total: one per batch (zero for a batch with nothing to write) is the design; more means a write path is syncing on its own.
scramdb_cluster_apply_last_batch_entriesgaugeEntries in the most recent effect batch.
scramdb_cluster_apply_last_batch_data_syncsgaugeData log syncs the most recent effect batch requested.
scramdb_cluster_apply_decided_batches_waitinggaugeDecided batches waiting for their effects. Climbing steadily means the disk or the apply threads cannot keep up; each group's backlog stays bounded by its apply mailbox either way.
scramdb_cluster_apply_stops_totalcounterGroups whose apply stopped on an error. Except a full disk (the row below), such a replica is quarantined and rebuilt from the leader's own snapshot; see the logs for the reason.
scramdb_cluster_apply_stops_disk_full_totalcounterThe subset of scramdb_cluster_apply_stops_total caused by a write meeting a full disk on this node: the node's own log names the index it stopped at. Its storage is never quarantined; free space on that node and it resumes on its own from that same index (scramdb_cluster_group_disk_full_resumes_total), no restart needed.
scramdb_cluster_apply_pool_threadsgaugeApply threads running.
scramdb_cluster_apply_pool_queuedgaugeGroups waiting for an apply thread.
scramdb_cluster_apply_pool_idle_wakes_totalcounterTimes an idle apply thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer.
scramdb_cluster_apply_effect_slots_reused_totalcounterInstalled rows whose effect was built in a slot an apply thread keeps, its key buffer reused.
scramdb_cluster_apply_effect_slots_fresh_totalcounterInstalled rows whose effect needed a new slot, so its key buffer allocated. Past warm-up it grows only when a batch is larger than any before it on that thread.
scramdb_cluster_apply_effect_images_decoded_totalcounterInstalled rows whose image decode had to allocate memory of its own.
scramdb_replica_apply_delete_batches_totalcounterBatches of deleted rows the replica apply removed from storage in one call.
scramdb_cluster_io_runtime_busy_microseconds_totalcounterTime the consensus and connection threads spent running work. Its rate divided by io_runtime_workers is their utilization.
scramdb_cluster_io_runtime_workersgaugeConsensus and connection threads.

Replication flow control​

A leader streams entries to each follower inside a bounded in-flight window (max_inflight_bytes and max_inflight_msgs in [cluster.consensus]), which by default adapts under that ceiling to the bandwidth-delay product measured toward each follower. Each follower's stream is in one of three states: probe (finding where the follower's log ends, one message outstanding), replicate (streaming inside the window) or snapshot (a snapshot is outstanding). The counters and gauges below sum every group this node leads.

MetricTypeMeaning
scramdb_cluster_replication_transitions_totalcounterFollower stream state changes, labelled state (probe, replicate, snapshot) and cause: into probe by leader_start, reject (the follower refused an append or a snapshot), connection_loss, stall (two heartbeat checks with entries in flight and no answer) or incarnation (the follower answered as a newer start of the replica than the leader had heard, so what the leader believed it held is void and its stream restarts); into replicate by acknowledged or snapshot_installed; into snapshot by behind_compaction.
scramdb_cluster_replication_sends_paused_totalcounterSends held back because a follower's in-flight window was full.
scramdb_cluster_replication_append_messages_totalcounterAppendEntries messages sent to followers, heartbeats included.
scramdb_cluster_replication_append_entries_totalcounterLog entries sent to followers.
scramdb_cluster_replication_append_bytes_totalcounterLog entry bytes sent to followers. Divided by the bytes proposed and by the follower count, this is the copies of each entry the network carried: about one in steady operation.
scramdb_cluster_replication_rejects_totalcounterAppend and snapshot rejects received from followers.
scramdb_cluster_replication_snapshot_resends_totalcounterSnapshots sent again to a follower that had not answered within the resend backoff.
scramdb_cluster_replication_incarnation_changes_totalcounterFollower responses that came from a different start of the replica than the one the leader held: one per follower restart, each restarting that follower's stream. A short burst follows a restart that raced a response still in flight.
scramdb_cluster_replication_unconfirmed_responses_totalcounterFollower responses held back until a round trip the leader began confirms the replica's current start. A few follow every leader election and every follower restart; a sustained rate means a follower never confirms (its responses do not echo what the leader sent).
scramdb_cluster_replication_incarnations_confirmed_totalcounterReplica starts confirmed by a round trip from the leader: one per follower per leadership, and one more each time a follower restarts.
scramdb_cluster_replication_inflight_messagesgaugeUnacknowledged AppendEntries messages to followers, right now.
scramdb_cluster_replication_inflight_bytesgaugeUnacknowledged log entry bytes sent to followers, right now.
scramdb_cluster_replication_window_bytesgaugeThe in-flight window toward a peer, summed over the groups this node leads, labelled peer and reason: ceiling (the configured max_inflight_bytes, or not yet measured), measured (twice the measured bandwidth-delay product), floor (two of the stream's own messages) or pinned (adaptive_inflight_window = false).
scramdb_cluster_replication_round_trip_millisecondsgaugeThe smoothed replication round trip to a peer, from sending an AppendEntries to its acknowledgement (the follower's fsync included), as last measured.
scramdb_cluster_replication_entries_committed_before_leader_sync_totalcounterLog entries committed on followers' synced copies before the leader's own copy reached disk (leader_sends_before_sync in [cluster.consensus]).
scramdb_cluster_replication_leader_unsynced_entriesgaugeLog entries the groups this node leads have sent but not yet synced to their own disk.
scramdb_cluster_replication_lag_entriesgaugeLog entries a peer has not acknowledged, summed over the groups this node leads, labelled peer. A peer whose lag keeps growing is falling behind.
scramdb_cluster_replication_leader_sync_secondshistogramTime from a leader writing new log entries to their sync to disk.
scramdb_cluster_election_timeout_min_millisecondsgaugeLower bound of the election timeout in force (0 until a replication group has started on this node).
scramdb_cluster_election_timeout_max_millisecondsgaugeUpper bound of the election timeout in force.

sends_paused_total climbing together with window_bytes{reason="ceiling"} toward a peer says the ceiling, not the link, is the limit: raise max_inflight_bytes for a long, fast link. Steady growth of transitions_total{cause="stall"} or {cause="reject"} on a healthy network says a follower keeps losing its place and is worth a look at its disk and its connection. A restarted follower shows one transitions_total{cause="incarnation"} and one more incarnations_confirmed_total on the leader; growth of either without restarts says a node keeps restarting or two processes answer as one node.

Consensus core buffers​

Consensus reuses its working buffers across steps, so a steady stream of heartbeats, appends and commits allocates nothing for its step output; only a burst above the recent load, or the first load after a quiet spell, needs a fresh allocation, and that spare capacity is handed back again once the burst passes. A received frame is decoded without copying the commands and snapshot data it carries. These figures are for the whole process, summed across it.

MetricTypeMeaning
scramdb_cluster_raft_step_vectors_reused_totalcounterRaft step output vectors served from a kept buffer instead of a new allocation. Counted as each vector is taken.
scramdb_cluster_raft_step_vectors_fresh_totalcounterRaft step output vectors started without a kept buffer, so their first element allocates. Counted as each vector is taken. It stays flat on a steady load and grows when the load keeps more step output in flight than it recently did (a burst, or the first load after a quiet spell).
scramdb_cluster_raft_step_buffer_bytesgaugeBytes the consensus threads keep for Raft step output.
scramdb_cluster_raft_frame_blob_bytes_shared_totalcounterCommand and snapshot bytes decoded from received Raft frames without a copy.
scramdb_cluster_raft_frame_blob_bytes_copied_totalcounterCommand and snapshot bytes copied out of received Raft frames. It stays at zero on a node: every receive path decodes from the frame it was given.
scramdb_cluster_raft_frames_encoded_totalcounterCluster messages encoded into a consensus thread's kept frame buffer.
scramdb_cluster_raft_frame_buffer_growths_totalcounterTimes a consensus thread's kept frame buffer grew to hold a message.
scramdb_cluster_raft_frame_entry_vectors_decoded_totalcounterEntry vectors the Raft frame decoder allocated for received appends, one per append that carries entries.

Heartbeats and quiet groups​

A node sends one heartbeat frame per peer node each heartbeat interval, carrying a line for every group it leads with that peer, instead of one heartbeat per group ([cluster.consensus] coalesced_heartbeats). A frame names its group set by a hash and lists the groups in full only after the set changed. When a group's commit index advances, a read needs confirming or a small append waits, the frame leaves early: at once while the connection to that peer has nothing queued, otherwise after heartbeat_flush_delay. A small append rides inside the frame instead of taking a frame of its own (carry_small_appends), and a follower's reply to an append can ride back inside the answer frame it sends anyway. An idle group (nothing in flight, every member caught up) goes quiet: it stops its heartbeats and its members' election timers, and its peer is refreshed once per election_timeout_min (quiescence). See [cluster.consensus].

MetricTypeMeaning
scramdb_cluster_heartbeat_frames_sent_totalcounterHeartbeat frames this node sent, each covering every group it leads with that peer.
scramdb_cluster_heartbeat_group_lines_sent_totalcounterGroup lines the heartbeat frames of this node carried.
scramdb_cluster_heartbeat_bytes_sent_totalcounterBytes of heartbeat and answer frames this node sent.
scramdb_cluster_heartbeat_full_lists_sent_totalcounterHeartbeat frames that listed every group this node leads with the peer, after that set changed.
scramdb_cluster_heartbeat_quiet_frames_sent_totalcounterHeartbeat frames with no group line, sent to keep a peer's quiet groups quiet.
scramdb_cluster_heartbeat_flushes_totalcounterHeartbeat frames sent early because a commit index advanced, a read needed confirming or a small append waited.
scramdb_cluster_heartbeat_immediate_flushes_totalcounterEarly heartbeat frames sent at once because the connection to the peer had nothing queued.
scramdb_cluster_heartbeat_flush_delay_secondshistogramTime from a commit index advancing, or a read or small append waiting, to the heartbeat frame carrying it.
scramdb_cluster_heartbeat_answers_sent_totalcounterHeartbeat answer frames this node sent to the nodes leading its groups.
scramdb_cluster_heartbeat_frames_received_totalcounterHeartbeat frames this node received.
scramdb_cluster_heartbeat_answers_received_totalcounterHeartbeat answer frames this node received.
scramdb_cluster_heartbeat_round_trip_secondshistogramTime from sending a heartbeat frame to a peer to the peer's answer.
scramdb_cluster_heartbeat_carried_appends_totalcounterSmall appends sent inside heartbeat frames instead of frames of their own.
scramdb_cluster_heartbeat_carried_bytes_totalcounterBytes of the appends sent inside heartbeat frames.
scramdb_cluster_heartbeat_append_replies_totalcounterReplies this node's groups sent to appends of the nodes leading them.
scramdb_cluster_heartbeat_append_reply_frames_totalcounterFrames this node sent for its groups' append replies: a reply sent on its own, or an answer frame carrying replies and no heartbeat answer.
scramdb_cluster_heartbeat_carried_replies_totalcounterAppend replies sent inside heartbeat answer frames instead of frames of their own.
scramdb_cluster_heartbeat_carried_replies_received_totalcounterAppend replies this node received inside heartbeat answer frames.
scramdb_cluster_heartbeat_send_refused_totalcounterHeartbeat and answer frames the transport refused because its queue was full or the peer was not connected.
scramdb_cluster_heartbeat_delivery_refused_totalcounterHeartbeat lines a group could not take because its queue was full or it was gone.
scramdb_cluster_heartbeat_malformed_frames_totalcounterHeartbeat frames dropped because they did not decode.
scramdb_cluster_heartbeat_stale_views_totalcounterHeartbeat frames whose group-set hash did not match the set this node held, which asks for the full list.
scramdb_cluster_heartbeat_quiesce_entered_totalcounterGroups this node leads that went quiet: every member caught up and idle, no heartbeats and no timers.
scramdb_cluster_heartbeat_quiesce_left_totalcounterQuiet groups this node leads that woke up.
scramdb_cluster_heartbeat_woken_by_set_change_totalcounterQuiet groups on this node woken because the node leading them stopped listing them.
scramdb_cluster_heartbeat_woken_by_missed_refresh_totalcounterQuiet groups on this node woken because the node leading them missed two refreshes.
scramdb_cluster_heartbeat_led_groupsgaugeGroups this node leads whose heartbeats its frames carry.
scramdb_cluster_heartbeat_quiescent_groupsgaugeGroups this node leads that are quiet.
scramdb_cluster_heartbeat_peersgaugePeer nodes this node sends heartbeat frames to or receives them from.
scramdb_cluster_heartbeat_interval_millisecondsgaugeThe heartbeat interval in force (heartbeat_interval).
scramdb_cluster_heartbeat_flush_delay_microsecondsgaugeHow long an early heartbeat frame waits to gather more groups, in force (heartbeat_flush_delay).
scramdb_cluster_heartbeat_quiet_refresh_millisecondsgaugeHow often a peer whose groups here are all quiet is refreshed, in force.

group_lines_sent_total divided by frames_sent_total is how many groups one frame carries: the saving over one heartbeat per group. On an idle cluster quiescent_groups approaches led_groups and the heartbeat traffic falls to one small refresh per peer. woken_by_missed_refresh_total rising on a healthy network means a leader's node stalls; send_refused_total, delivery_refused_total and malformed_frames_total stay at zero on a healthy node.

Read-index forwarding​

A replica that is not its group's leader, a follower or a learner, forwards its read-index requests to the leader, which answers them as it answers its own. A forwarded request with no answer within two election timeouts plus the round trip the replica measured to the leader, or whose connection to the leader closed, is refused. An answer that arrives after its request was refused still measures the round trip, so on a slow link the next request waits long enough; forward_late_total rising means a replica's link to its leader is slower than two election timeouts.

MetricTypeMeaning
scramdb_cluster_read_index_forwarded_totalcounterRead-index requests this node forwarded to a group's leader.
scramdb_cluster_read_index_forward_unanswered_totalcounterForwarded read-index requests that ended without the leader's answer.
scramdb_cluster_read_index_forward_timed_out_totalcounterForwarded read-index requests given up after their wait with no answer from the leader. A request the leader's node cannot take is refused at once instead and counted only as unanswered, so a rise here means messages were lost.
scramdb_cluster_read_index_forward_late_totalcounterLeader answers to forwarded read-index requests that arrived after the request was given up.
scramdb_cluster_read_index_served_for_peers_totalcounterRead-index requests this node answered as a group's leader for another replica.
scramdb_cluster_read_index_forward_secondshistogramTime from forwarding a read-index request to the leader's answer.

Requests to the metadata and shard groups​

Every request a node sends its metadata group or one of its shard groups has a deadline adapted to the group's election timing. A request the group did not take in time changed nothing and is retried by its caller; a proposal whose answer did not come may still commit, so it is never sent again; a read confirmation is asked again until the deadline. A DDL statement forwarded to the metadata group's leader is bounded the same way. Process-wide, always present, all zero on a healthy cluster:

MetricTypeMeaning
scramdb_group0_reply_timeouts_totalcounterMetadata group proposals whose answer did not come in time; each may still commit.
scramdb_group0_mailbox_timeouts_totalcounterMetadata group requests the group did not take in time; none changed anything.
scramdb_group0_read_retries_totalcounterMetadata group read confirmations asked again after an answer did not come.
scramdb_group0_apply_wait_timeouts_totalcounterWaits for this node to apply a committed metadata group entry that ran out of time.
scramdb_shard_group_reply_timeouts_totalcounterShard group proposals whose answer did not come in time; each may still commit.
scramdb_shard_group_mailbox_timeouts_totalcounterShard group requests the group did not take in time; none changed anything.
scramdb_shard_group_read_retries_totalcounterShard group read points asked again after an answer did not come.
scramdb_shard_group_apply_wait_timeouts_totalcounterWaits for this node to apply a committed shard group entry that ran out of time.
scramdb_ddl_forward_answer_timeouts_totalcounterForwarded metadata group proposals whose answer did not come in time.
scramdb_ddl_forward_in_doubt_totalcounterForwarded metadata group proposals whose leader could not learn their outcome.

reply_timeouts_total or apply_wait_timeouts_total rising means a group is slow to commit or a node is slow to apply; mailbox_timeouts_total rising means the group itself is falling behind on incoming requests.

A metadata group proposal this node's answer did not reach in time is sent again under its first attempt's id, so a leader that already applied it once applies it once, never twice, and a resend past its own deadline is refused rather than applied late:

MetricTypeMeaning
scramdb_group0_proposal_resends_totalcounterMetadata proposals sent again under the id of their first attempt.
scramdb_group0_proposal_duplicates_totalcounterMetadata log entries applied as a repeat of a proposal already applied.
scramdb_group0_proposal_expired_totalcounterMetadata proposals refused because they arrived after their deadline.
scramdb_group0_proposal_window_idsgaugeMetadata proposal ids this node keeps to answer a repeat.

How far a node has come through the metadata group's log. A schema change reaches a node's catalog after its log entry applies there, so the two schema figures differ while the catalog catches up. Across the nodes of a settled cluster each figure is the same; a node whose figure stays below the others is behind on the metadata log or on its catalog:

MetricTypeMeaning
scramdb_group0_applied_indexgaugeHighest metadata log entry this node has applied.
scramdb_group0_schema_changes_appliedgaugeSchema changes this node's metadata log has applied.
scramdb_catalog_schema_changes_appliedgaugeSchema changes this node's catalog has applied.

Cluster sequences​

A sequence a cluster made is one sequence: every node takes runs of its values from the metadata group and hands them out locally. Process-wide, always present:

MetricTypeMeaning
scramdb_cluster_sequence_grants_totalcounterRuns of sequence values this node took from the cluster.
scramdb_cluster_sequence_repositions_totalcounterSequence positions this node set cluster-wide.
scramdb_cluster_sequence_failures_totalcounterRequests about a cluster sequence that ended in an error.
scramdb_cluster_sequence_discarded_values_totalcounterSequence values this node dropped unused because the sequence was moved.

grants_total rising much faster than the draw rate divided by a few per second means a sequence is drawn from in tiny bursts; discarded_values_total counts the gaps a setval or RESTART leaves in what a node had already taken.

Sequence values given back​

A bulk INSERT that fails part way gives back the sequence values it drew for the rows after the one it failed at, so the next draw continues where PostgreSQL's would. Process-wide, always present, on a single node and a cluster alike:

MetricTypeMeaning
scramdb_sequence_rewound_values_totalcounterSequence values a failed statement gave back to the sequence.
scramdb_sequence_refused_values_totalcounterSequence values a failed statement could not give back because another draw followed.
scramdb_sequence_unrewound_statements_totalcounterFailed statements whose sequence draws could not be tied to their rows, so none were given back.

A value that stays drawn is a gap, never a duplicate. refused_values_total rising means concurrent sessions draw from the same sequences that failing statements draw from; unrewound_statements_total rising means failing statements draw one sequence for more than one column or in an order the rows cannot be matched to.

INSERT replays​

An INSERT whose source raised a data exception (division by zero, an integer out of range, an invalid cast) is run again row by row, so the error reported is the one of the first row that fails, as PostgreSQL reports it. Process-wide, always present, on a single node and a cluster alike:

MetricTypeMeaning
scramdb_insert_replay_statements_totalcounterINSERT statements run again row by row after their source raised an error.
scramdb_insert_replay_earlier_error_totalcounterReplays that raised the error of a row before the one the source failed at.
scramdb_insert_replay_source_error_totalcounterReplays that kept the source's own error, because no earlier row failed or the replay could not run.

A statement that succeeds never replays. statements_total rising in steady state means clients send inserts that fail on a data exception; source_error_total close to statements_total means the replay seldom finds an earlier row to blame, and a rise in it with a source larger than the memory budget means the replay could not hold the source.

Read positions​

Every read of a cluster table happens at a read position of the commit protocol. A fresh position first has each group's leader confirm the group's latest commit, brings a replica up to it, and waits out any commit still in flight on what is read, so a read never misses a commit already acknowledged to any client, on any node; a reader then serves each group it scans at that position before it reads. For a group this node holds no replica of, the position is asked of the group's leader first, then of its followers, and of its learners last: the leader has nothing to catch up, while a learner may trail its group by any amount, so a learner that has fallen behind does not hold up a statement while a voter of the group can answer. A replica that refuses the position (it does not run the group yet, it is starting, it could not catch up in time; counted on that node by scramdb_cluster_read_position_refusals_total), or fails while still connected, is passed over for the group's next replica. When no replica of such a group is connected to this node (a partition that just healed while its connections form again, a node restarting), the read waits for one to connect (scramdb_cluster_read_position_connect_waits_total). When every replica of the group has refused, as happens for a table created a moment before whose replicas are still starting, the read asks them all again after a pause of a few round trips that grows with each try, and serves the group itself once its own replica of it runs (scramdb_cluster_read_position_reask_waits_total). A replica that has not answered when the read's re-send interval lapses (the interval this node's measured round trip sets) is not waited for alone: the group's next replica is asked beside it, the first answer is taken and the other request is dropped (scramdb_cluster_read_position_hedges_total counts those requests, scramdb_cluster_read_position_asks_in_flight the ones that have waited long enough to ask another replica and have not ended). A replica answers a peer's request within half a read wait, so its reason reaches the asking node while that node still waits. Either way the read waits for at most its read wait, and then fails with 40001 naming the group and, when a replica gave one, what it was waiting for. Nothing here writes to a log unless a group is behind the position, where one fence entry brings it up (scramdb_cluster_dilith_fences_appended_total).

MetricTypeMeaning
scramdb_cluster_read_positions_totalcounterFresh read positions chosen by this node.
scramdb_cluster_read_position_secondshistogramTime to choose a fresh read position, from the first barrier to the last answer.
scramdb_cluster_read_position_remote_groups_totalcounterShard groups whose fresh read position another node took for this node (groups this node holds no replica of).
scramdb_cluster_read_position_connect_waits_totalcounterFresh read positions that waited for a replica of a shard group to connect.
scramdb_cluster_read_position_reask_waits_totalcounterFresh read positions that asked a shard group again after every replica refused it.
scramdb_cluster_read_position_refusals_totalcounterFresh read positions refused here because this node's replica could not serve them.
scramdb_cluster_read_position_hedges_totalcounterShard group positions asked of a next replica before an earlier one had answered.
scramdb_cluster_read_position_asks_in_flightgaugePosition requests of reads that waited long enough to ask another replica and have not ended.
scramdb_cluster_read_position_raised_totalcounterTransactions whose read position rose to reach a table they had not read before.
scramdb_cluster_read_barrier_retries_totalcounterRead barriers asked again because a shard group had no leader to confirm them.
scramdb_cluster_read_serves_totalcounterShard groups served at a read position on this node.
scramdb_cluster_read_serve_secondshistogramTime a shard group took to reach a read position on this node.
scramdb_cluster_read_failures_totalcounterReads refused because a shard group could not be confirmed or served in time.
scramdb_cluster_read_registered_ranges_totalcounterTable reads registered with a transaction's commit for validation.
scramdb_cluster_read_skip_serves_totalcounterShard groups served for SKIP LOCKED or NOWAIT reads without waiting for a commit.
scramdb_cluster_read_skip_waits_totalcounterSKIP LOCKED or NOWAIT reads of a shard group that waited for a commit in flight.
scramdb_cluster_read_skip_fence_waits_totalcounterWaits of SKIP LOCKED or NOWAIT reads for a shard group to be fenced at their cut.
scramdb_cluster_read_skipped_keys_totalcounterKeys a SKIP LOCKED or NOWAIT read found written by a commit in flight.
scramdb_cluster_read_skipped_rows_totalcounterRows SKIP LOCKED left out because a commit in flight writes them.
scramdb_cluster_read_nowait_refusals_totalcounterNOWAIT statements refused because a commit in flight writes a row they select.
scramdb_cluster_read_join_spills_totalcounterReads over more shard groups than fit inline, whose answers spilled to the heap: reads that touch more than eight groups at once.
scramdb_cluster_read_horizon_serves_totalcounterShard group watermarks taken here for a transaction snapshot, waiting for no commit.
scramdb_cluster_read_unreached_groups_totalcounterShard groups a transaction snapshot could reach on no replica.
scramdb_cluster_read_fence_waits_totalcounterWaits for a shard group to be fenced at a read position, waiting for no commit.
scramdb_cluster_read_fence_remote_groups_totalcounterShard groups hosted on other nodes that a transaction snapshot asked to fence at its cut.
scramdb_cluster_read_fence_request_retries_totalcounterShard groups whose fence request to another node was asked again after it went unanswered or was refused.
scramdb_cluster_read_range_serves_totalcounterKey ranges served at a read position that waited only for commits writing them.

A read that finds no leader for a group keeps asking until its wait runs out (one election timeout plus the time a group may hold requests without a leader plus the time an undecided transaction may block it, about ten seconds with the defaults) and is then refused with SQLSTATE 40001; a replica that cannot catch up is refused with 57P03. Growth of read_failures_total together with read_barrier_retries_total means a group is without a leader. read_position_raised_total counts transactions that read a table only after an earlier statement had fixed their position: such a transaction's earlier reads are validated at their own position when it commits. Under READ COMMITTED it also counts statements whose position rose because a table they reached later, such as a foreign key's parent, was fresher, so a row committed before the statement began is never missed.

SKIP LOCKED and NOWAIT read the tables they lock without waiting for any commit in flight: the keys such commits write come back with the rows (read_skipped_keys_total), and the statement leaves those rows out (read_skipped_rows_total) or refuses with 55P03 (read_nowait_refusals_total). A shard group where a commit in flight writes a table the statement reads without locking it, such as a join's other side, is read the ordinary way, waiting for that commit, and counts in read_skip_waits_total.

A REPEATABLE READ or SERIALIZABLE transaction takes its snapshot from every shard group's confirmed watermark without waiting for any commit in flight (read_horizon_serves_total), then fences every group at it: the groups this node holds directly (read_fence_waits_total counts the waits for a fence on its way), the others by the nodes that hold them, which the transaction waits to acknowledge before it reads (read_fence_remote_groups_total counts the groups asked). A request that goes unanswered is sent again after the retry interval (read_fence_request_retries_total), so a table that another node commits to after the snapshot is never shown by it. A read of a few keys at that snapshot waits only for a commit in flight on those keys (read_range_serves_total). A group the snapshot can reach on no replica, such as one cut off by a partition, is counted in read_unreached_groups_total, and the transaction's reads are checked at COMMIT, which fails with 40001 if one of them missed a commit. A group whose replicas the snapshot reaches but which all refuse it, such as the groups of a table created a moment before whose replicas are still starting, is not left out: it is asked again at the horizon retry interval (horizon_retry, 90 ms by default) until a replica answers, for at most the read wait.

Adaptive routing​

MetricTypeMeaning
scramdb_cluster_routes_totalcounter/labelRouting decisions, labelled route: one per statement, and one per scatter for a statement that scatters more than once (each outer row of a k-NN LATERAL join).

route takes exactly one of five values: oltp, point, local_replica, forward_replica, scatter. That's the whole label space, fixed by construction, the keys are the five route kinds the planner can choose, never a node id or a group id, so this metric's cardinality can never grow with the size of the cluster.

Where fragments run​

A distributed read runs each bucket's fragment on one replica that holds the bucket: a learner before a follower before the shard group's leader, in this node's region first, spread evenly (see Where fragments run). [cluster] fragment_any_replica = false keeps fragments on the buckets' owners.

MetricTypeMeaning
scramdb_cluster_exchange_placed_learner_buckets_totalcounterBuckets of distributed reads this node placed on a learner replica.
scramdb_cluster_exchange_placed_follower_buckets_totalcounterBuckets of distributed reads this node placed on a follower replica.
scramdb_cluster_exchange_placed_leader_buckets_totalcounterBuckets of distributed reads this node placed on a shard group's leader.
scramdb_cluster_exchange_placement_skips_totalcounterReplicas placement passed over because they could not serve the read position.
scramdb_cluster_exchange_fragment_redispatches_totalcounterFragments run on another replica after one could not serve them.
scramdb_cluster_exchange_fragments_served_learner_totalcounterFragments this node served from its learner store.
scramdb_cluster_exchange_round_replacements_totalcounterShuffle and two-phase read rounds run again on other replicas after a node could not serve its buckets.
scramdb_cluster_exchange_fragments_refused_totalcounterFragments this node refused because its replica could not serve the read position.
scramdb_cluster_fragment_wire_version_refusals_totalcounterFragments this node refused because their sender speaks a newer wire protocol version than this node reads.
scramdb_cluster_wire_versiongaugeThe wire protocol version this node speaks.
scramdb_cluster_wire_version_minimumgaugeThe smallest wire protocol version any cluster member has recorded (0 while a member has recorded none).
scramdb_cluster_route_hash_v1_totalcounterShuffles and runtime filters this node routed by the older hash because a member it used speaks an older protocol version. It stops rising once every node runs the newer release (Upgrading).
scramdb_cluster_route_hash_v2_totalcounterShuffles and runtime filters this node routed by the newer hash because every member it used speaks the newer protocol version.
scramdb_cluster_exchange_shipped_tables_totalcounterTables this node sent with a query's fragments because no replica held them whole.
scramdb_cluster_exchange_statistics_views_published_totalcounterTable statistics this node sent to the other nodes after analysing a table.
scramdb_cluster_exchange_statistics_views_installed_totalcounterTable statistics another node sent that this node now plans with.
scramdb_cluster_exchange_statistics_views_stale_totalcounterTable statistics another node sent that this node already held a newer view of.
scramdb_cluster_exchange_statistics_views_oversize_totalcounterTable statistics too large to send to the other nodes.

A distributed table has one statistics view: an ANALYZE on any node measures the buckets that node holds, scales the result to the whole table and sends it to the other members, and the newest analysis wins on every node. statistics_views_published_total counts the analyses this node sent and statistics_views_installed_total the ones it took from another node. statistics_views_stale_total is expected after two nodes analyse the same table close together. Any statistics_views_oversize_total means a table's statistics did not fit in one message, so the other nodes keep their own view until they analyse the table themselves. The row count in a shared view is an estimate.

placed_leader_buckets_total growing against the learner and follower counts means the learners and followers cannot serve the reads (behind their group, or waiting for a full copy of its rows), so analytical scans land on the leaders; fragments_refused_total that stays high on one node says that node's replicas lag.

Every node records the wire protocol version it speaks in the metadata group when it starts, and every fragment a node sends carries that version to nodes that have recorded one. A node that receives a fragment from a newer version than it reads refuses it with a retryable message, and the sender runs that statement's buckets on the other replicas that hold them, the sender's own node for buckets it holds. wire_version_refusals_total therefore moves only while nodes of different versions run together, as in a rolling upgrade: a rate on the older nodes is expected until the upgrade finishes, and it is not an error. wire_version_minimum is the smallest version any member has recorded, and a cluster uses a feature that older nodes cannot read only once that minimum allows it; a minimum that stays below wire_version on the upgraded nodes long after an upgrade means a member is still running the older build or never recorded its version. A read whose every replica of a bucket refuses, with no replica on the sender, fails with a retryable error, so finish an upgrade rather than leave the cluster mixed.

Distributed shuffle joins​

How a shuffle join across nodes adapted while it ran ([cluster.distributed_join] in the config file).

MetricTypeMeaning
scramdb_cluster_shuffle_consumer_fragments_totalcounterConsumer fragments dispatched for a materialized shuffle join, one per coalesce group. Equals the participant count with coalescing off; lower with it on. Zero on a streaming query.
scramdb_cluster_shuffle_aqe_coalesce_groups_totalcounterCoalesce groups actually acted on: each merged more than one small adjacent partition into a single consumer fragment.
scramdb_cluster_shuffle_aqe_skew_split_flagged_totalcounterPartitions flagged skewed (over 5x the median and over 256MB) but not acted on: skew-split off, or a single-node participant set with nothing to split across. Logged and metered even when nothing follows.
scramdb_cluster_shuffle_aqe_skew_split_acted_totalcounterSub-consumers dispatched for a skew-split partition that was acted on: the larger side split into disjoint row ranges, the smaller side replicated. Summed across every acted partition.
scramdb_cluster_shuffle_aqe_broadcast_demote_flagged_totalcounterBroadcast sides flagged over budget, a broadcast-demote candidate. Logged and metered, not acted on yet.
scramdb_cluster_shuffle_straggler_backups_totalcounterSpeculative backup consumer fragments dispatched because a primary consumer didn't return within the straggler-backup delay, on a different live node reading the same frozen spools.

Node operations and cluster TRUNCATE​

The whole-cluster vector statistics views, a cluster TRUNCATE and the rows of a transaction block's CREATE TABLE AS or REFRESH reach every other node through the same exchange the distributed queries use.

MetricTypeMeaning
scramdb_cluster_exchange_node_ops_sent_total / scramdb_cluster_exchange_node_ops_failed_totalcounterNode operations this node asked a peer to run, and those a peer could not run or never answered.
scramdb_cluster_exchange_node_ops_served_total / scramdb_cluster_exchange_node_ops_serve_failed_totalcounterNode operations this node ran for a peer, and those it could not.
scramdb_cluster_exchange_view_gathers_total / scramdb_cluster_exchange_view_nodes_unreachable_totalcounterWhole-cluster statistics view reads this node gathered, and nodes such a read reported as unreachable.
scramdb_cluster_exchange_view_gather_secondshistogramTime to gather one whole-cluster statistics view read.
scramdb_cluster_exchange_truncates_total / scramdb_cluster_exchange_truncate_failures_totalcounterCluster TRUNCATE statements this node coordinated, and those that did not complete on every node.
scramdb_cluster_exchange_truncate_secondshistogramTime for one cluster TRUNCATE to reach every node holding the table.
scramdb_cluster_exchange_block_fills_published_totalcounterSchema changes of a transaction block whose rows this node kept for the other nodes.
scramdb_cluster_exchange_block_fills_served_totalcounterSchema changes of a transaction block whose rows this node sent to another node.
scramdb_cluster_exchange_block_fills_pulled_totalcounterSchema changes of a transaction block whose rows this node took from the node that runs it.
scramdb_cluster_exchange_block_fill_fallbacks_totalcounterSchema changes of a transaction block that read their rows again here because the node that runs the block had none to give.
scramdb_cluster_exchange_producer_failures_propagated_total / scramdb_cluster_exchange_producer_failure_undelivered_totalcounterFailed shuffle producers that handed their error to every consumer at once, and consumers they could not reach (those fall back to their own receive deadline).

A CREATE TABLE AS, SELECT ... INTO, CREATE MATERIALIZED VIEW or REFRESH MATERIALIZED VIEW inside a transaction block on a cluster is evaluated once, by the node that runs the block, and every other node takes its rows. On that node block_fills_published_total counts the statements and block_fills_served_total the times it sent their rows to another node; on every other node block_fills_pulled_total counts the statements whose rows it took, so one block statement on a three-node cluster adds one published and at least two served on the node that ran the block, and one pulled on each other node. block_fill_fallbacks_total stays at zero in normal operation; if it rises, contact support with the server log.

Cluster branches and table commands​

A CREATE DATABASE ... CLONE or a scram.branch_at on a cluster, and the table commands that carry a branch, a CLONE's fork hold and a cluster TRUNCATE through each shard group's log.

MetricTypeMeaning
scramdb_cluster_branches_created_totalcounterBranch databases this node created on the cluster.
scramdb_cluster_branch_failures_totalcounterBranch databases this node could not create on the cluster.
scramdb_cluster_branch_tables_forked_totalcounterTables forked into a branch database on this node's stores.
scramdb_cluster_branch_pending_rows_totalcounterRows a shard group gave a branch after this node's store forked it, before the group's fork entry.
scramdb_cluster_branch_shared_stages_totalcounterCOPY segments shared into a branch as they attached after this node's store forked it.
scramdb_cluster_branch_holds_taken_totalcounterFork holds taken on this node's stores, each keeping row versions until its branch is forked.
scramdb_cluster_branch_holds_released_totalcounterFork holds released on this node's stores.
scramdb_cluster_branch_clones_abandoned_totalcounterCLONE statements this node abandoned, their fork holds released on every node.
scramdb_cluster_branch_holds_kept_totalcounterFork hold checks that kept a hold because this node's store had not finished forking its branch.
scramdb_cluster_branch_fork_entry_failures_totalcounterBranch forks a shard group's entry could not make on this node, each given up as the branch's failure.
scramdb_cluster_branch_archive_builds_totalcounterBranches at a past time this node built from the cluster archive.
scramdb_cluster_branch_archive_rows_totalcounterRows this node read from the cluster archive into a branch at a past time.
scramdb_cluster_dropped_database_rows_skipped_totalcounterRows of a dropped database a shard group skipped on this node.
scramdb_cluster_branch_fork_recordsgaugeBranch fork entries the shard groups on this node still record one by one. A record is folded away once the store holding its group settled the branch (made it ready, or dropped it), so this stays at the branches in progress.
scramdb_cluster_branch_fork_records_pruned_totalcounterBranch fork records folded into a group's floor once their branch was settled.
scramdb_cluster_dropped_databases_recordedgaugeDropped databases this node's stores record one by one; each is released once its drop is decided.
scramdb_cluster_table_commands_proposed_totalcounterTable commands this node placed in a shard group's log.
scramdb_cluster_table_command_failures_totalcounterTable commands this node could not place in a shard group's log before the deadline.
scramdb_cluster_table_command_redirects_totalcounterTable command proposals a node turned away because its replica did not lead the group.

A fork hold keeps every row version of the parent's tables on a store, from before the branch's cut is chosen until the store has forked them all. At the end of the hold's lease (two minutes) the node asks the metadata group what became of the CLONE: an abandoned clone releases the hold on every node, and a branch that was made releases it once the store has finished forking it. A hold whose store has not finished forking its branch stays until the fork finishes. Each check that finds such a hold counts on holds_kept_total and logs once, scramdb_compaction_versions_held_total counts the row versions merges kept only because a hold stood, and scramdb_cluster_ddl_apply_stuck shows the change the fork waits for. The hold ends when the cause is repaired or the branch is dropped.

Table definition changes​

An ALTER TABLE on a cluster: the commits it holds back on each node, the fence it places in each of a distributed table's shard groups, and the rows each node installs across a change.

MetricTypeMeaning
scramdb_cluster_schema_changes_pendinggaugeTables whose definition change is under way on this node's stores.
scramdb_cluster_schema_commits_refused_totalcounterCommits refused because a table they wrote changed its definition while they ran.
scramdb_cluster_schema_commit_drains_totalcounterWaits for the commits under way before a table's definition change.
scramdb_cluster_schema_commit_drain_timeouts_totalcounterWaits for the commits under way that ended at their deadline.
scramdb_cluster_schema_fences_applied_totalcounterTable definition change fences this node's stores applied from shard group logs.
scramdb_cluster_schema_fence_waits_totalcounterFences that waited for this node's catalog to take the change first.
scramdb_cluster_renamed_table_rows_totalcounterRows of a renamed distributed table installed under its new name.
scramdb_cluster_widened_rows_totalcounterRows written before a column was added, widened to the table's shape as installed.
scramdb_cluster_schema_marks_restored_totalcounterTable change marks this node's stores restored from disk when they opened.
scramdb_cluster_schema_mark_write_failures_totalcounterWrites of the table change marks to disk that failed.
scramdb_cluster_ddl_apply_stuckgaugeDecided schema changes this node failed to apply and is trying again. 0 on a healthy node; alert on a value above 0 that lasts.
scramdb_cluster_ddl_apply_retries_totalcounterTries again of decided schema changes that failed to apply on this node.
scramdb_cluster_ddl_stuck_refusals_totalcounterStatements refused because a decided schema change of a table they name is not applied here yet. Each answered 40001.
scramdb_cluster_replica_key_waits_totalcounterShard group applies that waited for the key index of their table to be made on this node.
scramdb_cluster_ddl_apply_lock_waits_totalcounterRelation locks a decided schema change's apply had to wait for on this node.
scramdb_cluster_ddl_columnar_store_waitinggaugeDecided schema changes waiting for the columnar store's log, which hold this node's later ones back.
scramdb_cluster_apply_lanes_at_fencegaugeShard group apply lanes waiting at a schema change for this node's catalog, holding no thread.
scramdb_cluster_name_barriers_totalcounterStatements that asked the metadata group how far it had decided before a missing name was answered.
scramdb_cluster_name_barrier_catch_ups_totalcounterStatements that ran again after this node applied what the metadata group had decided.
scramdb_cluster_name_barriers_unreached_totalcounterMissing-name checks that could not reach the metadata group within the statement's bound.
scramdb_cluster_name_barriers_unapplied_totalcounterMissing-name checks that ended with this node still behind the metadata group.

A DROP TABLE of a distributed table holds back and drains the table's commits and fences its shard groups the same way before the table goes, so it moves schema_commit_drains_total and schema_fences_applied_total too.

A schema change the cluster decided is applied on every node. A node that cannot apply one (a failing disk, no room, a wait that ran out) does not record it as applied: it logs one error line, counts the change on scramdb_cluster_ddl_apply_stuck and tries again on a pause that starts at 100 milliseconds and doubles up to the metadata group's reply deadline, each try counted on scramdb_cluster_ddl_apply_retries_total. Until it applies, a statement that names a table the change touches waits for it, holding no lock, and when its wait ends first (its lock_timeout or statement_timeout, or an hour when it set neither) it is answered 40001 (table "name" is not up to date on this node yet) and counted on scramdb_cluster_ddl_stuck_refusals_total; other tables are served as usual, and the gauge falls back to 0 once the change applies. A change the statement itself refuses, the same on every node, is not retried and never counts here. A change that every node fails to apply for a reason of its own is held on all of them: every node reads the gauge above 0, refuses the statements that name the table once their wait is over and serves the rest, and the schema changes decided after it wait behind it. No statement gives such a change up; remove the cause the server log names (free the disk, for instance) and it applies on its own.

A node that finds a relation missing asks the metadata group how far it has decided before it answers 42P01 (see Changing a table on a cluster). name_barriers_total counts those questions, and each ends in one of four ways: the node was current and the miss stands, name_barrier_catch_ups_total (the node applied what the group had decided and the statement ran again), name_barriers_unapplied_total (the node was still behind at the end of the statement's wait and the statement was answered 40001) or name_barriers_unreached_total (the group's leader did not answer within the statement's wait, and the miss is answered as before). A catch_ups_total or unapplied_total that keeps growing means this node applies the metadata group's decisions late.

scramdb_cluster_apply_lanes_at_fence reads how many shard group apply lanes of this node wait at a schema change for the node's catalog to take it first. A lane waiting there holds no apply thread, so a node with few apply threads keeps serving its other groups, and the lane goes on by itself once the catalog has taken the change. It is 0 on a node with no change in progress; a value above 0 that lasts means the node's catalog has not yet taken a change (a rewrite of a large table is slow to apply, and a change the node cannot apply shows on scramdb_cluster_ddl_apply_stuck). scramdb_cluster_ddl_apply_lock_waits_total does not count the wait behind the hold of the change's own round, so a value above 0 names a lock that nothing drained before the change.

The marks that say a table was dropped or changed are written to disk before a node answers the round that proposes the change, so a node that restarts between its answer and the decision still refuses a stale write to the table. schema_marks_restored_total counts the marks read back at start. Any schema_mark_write_failures_total means a mark could not be written to that node's disk. The statement fails and can be retried. Check the disk of that node.

UPDATE and DELETE on a partially held table​

An UPDATE or DELETE on a node that holds only some buckets of a distributed table reads its rows from every owner (see UPDATE and DELETE across nodes). A node that holds every bucket of the table runs the single-node path and moves none of these.

MetricTypeMeaning
scramdb_cluster_dml_statements_distributed_totalcounterUPDATE and DELETE statements that read their rows from every owner.
scramdb_cluster_dml_candidates_remote_total / scramdb_cluster_dml_candidates_local_totalcounterRows such statements received from other nodes, and read from this node's own buckets.
scramdb_cluster_dml_candidate_batches_totalcounterRow batches such statements processed.
scramdb_cluster_dml_candidate_fetch_secondshistogramTime a statement waited for its next batch of rows.
scramdb_cluster_dml_candidate_spilled_bytes_totalcounterRow bytes set aside on disk because the statement's memory budget was spoken for.
scramdb_cluster_dml_owners_pruned_totalcounterOwners a statement skipped because its WHERE names one bucket.
scramdb_cluster_dml_locking_selects_totalcounterLocking SELECT statements (FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE) that read a distributed table's rows without locking them.
scramdb_cluster_dml_locking_read_rows_totalcounterRows such statements took and registered one by one for their commit to check (a SERIALIZABLE statement without SKIP LOCKED has its whole scan checked instead and adds nothing; rows SKIP LOCKED left out are not counted).
scramdb_cluster_dml_own_writes_reads_totalcounterQueries that read a partially held table together with their transaction's own uncommitted changes to it.
scramdb_cluster_dml_candidate_readers_replaced_totalcounterReaders of an UPDATE, DELETE or uniqueness check that could not serve their buckets and were read again on another replica.

A growing candidate_spilled_bytes_total means large UPDATE or DELETE statements are running close to the execution memory budget.

The row-lock family these statements used to report (scramdb_cluster_dml_row_lock_batches_total, scramdb_cluster_dml_row_locks_total, scramdb_cluster_dml_row_lock_waits_total, scramdb_cluster_dml_row_lock_wait_seconds, scramdb_cluster_dml_row_lock_conflicts_total, scramdb_cluster_dml_row_locks_released_total) and scramdb_cluster_dml_statement_restarts_total are gone: rows of a distributed table are no longer locked, so no statement waits, restarts or releases a lock (see Row locks on distributed tables). Writers that reach the same rows now show up as 40001 failures at commit: watch scramdb_cluster_dilith_aborts_total and your clients' retry counts instead, and remove any alert or dashboard panel that reads the removed names.

Distributed statements​

MetricTypeMeaning
scramdb_cluster_read_stability_retries_totalcounterReads re-run because a replica applied writes while they scanned.
scramdb_cluster_read_stability_failures_totalcounterReads that failed because replicas kept applying writes through every re-run. The statement fails with 40001 and can be retried.
scramdb_cluster_serialization_retries_totalcounterAutocommit statements sent again, and COPY votes voted again, after a serialization failure.
scramdb_cluster_placement_waits_totalcounterStatements that waited for this node to learn where a table they name is placed, right after it restarted or right after the table was created.
scramdb_cluster_catalog_waits_totalcounterReader slices that waited for this node to apply the definition of a table it places.
scramdb_cluster_rows_moved_totalcounterRows an UPDATE moved to a new primary key or shard group.
scramdb_cluster_unique_probes_totalcounterUniqueness checks of new keys sent to the nodes that hold their buckets.
scramdb_cluster_unique_violations_totalcounterNew keys refused because a row on another node already holds them.
scramdb_cluster_inserts_positioned_by_key_totalcounterInserted batches whose keys were checked in their own shard groups only, not in every group of the table.
scramdb_cluster_upsert_remote_conflicts_totalcounterINSERT ... ON CONFLICT rows whose conflicting row is stored on another node.
scramdb_cluster_on_conflict_keys_overtaken_totalcounterINSERT ... ON CONFLICT statements refused because another transaction committed one of their conflict keys after the statement judged it free.
scramdb_cluster_own_writes_spilled_bytes_totalcounterBytes of a transaction's view of a table a query set aside on disk for want of memory.
scramdb_cluster_general_statements_totalcounterStatements answered on their coordinator over rows streamed from every shard group.
scramdb_cluster_general_fast_path_declines_totalcounterStatements a faster distributed path handed to the general path.
scramdb_cluster_general_rows_streamed_totalcounterRows streamed to a coordinator for the statements it answered over them.
scramdb_cluster_general_spilled_bytes_totalcounterBytes of streamed rows a coordinator set aside on disk for want of memory.
scramdb_cluster_general_readers_refused_totalcounterReaders of a streamed relation whose replica refused the read position. Their buckets were read on another replica.
scramdb_cluster_general_readers_unreached_totalcounterReaders of a streamed relation that were not reached or stopped sending. Their buckets were read on another replica.
scramdb_cluster_general_readers_failed_totalcounterReaders of a streamed relation whose read failed, failing the statement.
scramdb_cluster_general_reads_exhausted_totalcounterStreamed reads that failed because no replica was left to read a bucket. The statement fails with 40001 and can be retried.

Prepared transactions​

PREPARE TRANSACTION, COMMIT PREPARED and ROLLBACK PREPARED on a cluster's distributed tables (see Prepared transactions), counted on the node that ran the statement; the holds are counted on every node that keeps a replica of the groups they live in.

MetricTypeMeaning
scramdb_cluster_prepare_succeeded_totalcounterPREPARE TRANSACTION statements that prepared their transaction.
scramdb_cluster_prepare_refused_totalcounterPREPARE TRANSACTION statements refused, their transaction rolled back.
scramdb_cluster_prepare_in_doubt_totalcounterPREPARE TRANSACTION statements whose outcome was not known when they answered (08007).
scramdb_cluster_commit_prepared_succeeded_totalcounterCOMMIT PREPARED statements that committed a prepared transaction.
scramdb_cluster_commit_prepared_refused_totalcounterCOMMIT PREPARED statements that were refused, such as an unknown transaction name.
scramdb_cluster_commit_prepared_in_doubt_totalcounterCOMMIT PREPARED statements whose outcome was not known when they answered (08007).
scramdb_cluster_rollback_prepared_succeeded_totalcounterROLLBACK PREPARED statements that rolled a prepared transaction back.
scramdb_cluster_rollback_prepared_refused_totalcounterROLLBACK PREPARED statements that were refused.
scramdb_cluster_rollback_prepared_in_doubt_totalcounterROLLBACK PREPARED statements whose outcome was not known when they answered (08007).
scramdb_cluster_dilith_prepared_holds_created_totalcounterPrepared transactions whose hold this node's replicas made.
scramdb_cluster_dilith_prepared_holds_consumed_totalcounterPrepared transactions committed on this node's replicas, their holds ended.
scramdb_cluster_dilith_prepared_holds_released_totalcounterPrepared transactions rolled back on this node's replicas, their holds ended.
scramdb_cluster_dilith_prepared_hold_vote_refusals_totalcounterVotes this node's replicas refused because they met a prepared transaction's hold.
scramdb_cluster_dilith_prepared_hold_precheck_refusals_totalcounterVotes a group leader on this node refused before appending because they met a prepared transaction's hold.

A prepared transaction's hold keeps its writes from being overtaken while it waits for COMMIT PREPARED or ROLLBACK PREPARED, from whichever node the verb arrives. pg_prepared_xacts lists the prepared transactions of the current database from the cluster's registry: one whose prepared time keeps growing is a transaction nobody resolved.

Table migrations and re-keys​

A table definition change that reshapes rows (a re-key, a column type change, the upgrade of an older table) is staged, decided once, fenced on every node and finished by the group leader if the node that began it died. These metrics show the progress.

MetricTypeMeaning
scramdb_cluster_migration_tables_pendinggaugeTables with a migration on the cluster that is not finished for this node. SHOW CLUSTER lists the same count per node as migrations_pending.
scramdb_cluster_migration_rows_moved_totalcounterRows this node moved or staged for a table migration.
scramdb_cluster_rekey_row_keys_retired_totalcounterHidden row keys this node retired once every group it hosts of the table applied its re-key entry.
scramdb_cluster_rekey_stage_mismatches_totalcounterRe-key entries a replica stopped at because its stage was not the one the entry names. Zero on a healthy cluster.
scramdb_cluster_copy_stage_mismatches_totalcounterCOPY stage attaches a replica stopped at because its stage was not the one the vote names. Zero on a healthy cluster.
scramdb_cluster_stage_entries_carried_totalcounterEntries of an open stage (a COPY's or a re-key's) a state snapshot carried to a replica that installed it.

A scramdb_cluster_migration_tables_pending that stays above 0 for an hour means a migration is waiting on a node that is down or fenced. The mismatch counters stay at zero in normal operation; if one rises, contact support with the server log.

The holds this node's replicas keep are the prepared_holds entry of scramdb_cluster_dilith_table_entries. A growing scramdb_cluster_dilith_prepared_hold_vote_refusals_total means writers keep meeting what prepared transactions touched; commit or roll the prepared transactions back sooner.

Columnar mirror​

A node that keeps a columnar copy of the rows it replicates (a hybrid voter, columnar_replica = true, or a learner) copies committed entries into it on its own mirror threads.

MetricTypeMeaning
scramdb_cluster_columnar_mirror_threadsgaugeThreads that copy committed entries into this node's columnar store.
scramdb_cluster_columnar_mirror_queuedgaugeColumnar mirror lanes waiting for a mirror thread.
scramdb_cluster_columnar_mirror_panics_totalcounterColumnar mirror lanes that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log.
scramdb_cluster_columnar_mirror_idle_wakes_totalcounterTimes an idle columnar mirror thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer.

Learner-read freshness​

MetricTypeMeaning
scramdb_cluster_learner_freshness_secondshistogramHow long ago each queried group's read floor last advanced, observed once per stale or strong learner_read read. The floor is the commit protocol's closed position at the replica: every commit at or below it is installed there and none can land there later. The OLTP path (learner_read off) never touches this histogram. Same buckets as the handshake histogram, 1ms to 5s plus +Inf, a natural fit for replication lag.

This is the same lag SHOW FRESHNESS reports per data group in Clustering, captured here as a scrapeable distribution instead of a point-in-time value.

Faults, retries and reachability​

How a node sees its peers and its consensus groups, and every retry, timeout and re-dispatch of the cluster's network mechanisms. Always emitted in cluster mode. On a healthy cluster the failure and retry counters stay at zero, and the gauges show the peers and members that are up. A fault that repeats is logged once, as a warning when it begins, and counted every time; a lost peer that is heard again is logged once at info.

MetricTypeMeaning
scramdb_cluster_peers_reachablegaugePeers with an open cluster connection.
scramdb_cluster_peer_reachablegaugeWhether a peer connected or lost now has an open cluster connection (1 or 0), labelled peer.
scramdb_cluster_peer_round_trip_microsecondsgaugeRound trip of the newest answered liveness ping to a peer, labelled peer. Absent until a ping is answered.
scramdb_cluster_peer_silence_millisecondsgaugeTime since a peer's cluster connection last received a byte, labelled peer.
scramdb_cluster_peers_lostgaugePeers that were connected and have not been heard from since their connection was lost.
scramdb_cluster_peer_losses_totalcounterTimes a connected peer became unreachable. A partition is one loss per peer, however often its connections close.
scramdb_cluster_dial_failures_totalcounterConnection attempts to peers that failed, labelled cause.
scramdb_cluster_inbound_handshake_failures_totalcounterConnections from peers that failed or were refused during the handshake, labelled cause.
scramdb_cluster_accept_errors_totalcounterTimes a cluster listener failed to accept a connection.
scramdb_cluster_groups_without_leadergaugeConsensus groups on this node that know no leader.
scramdb_cluster_group_leader_losses_totalcounterTimes a consensus group on this node lost its leader.
scramdb_cluster_stalled_groupsgaugeConsensus groups on this node that made no progress for as long as a request waits for its answer (fifteen election timeouts, at least 5 s).
scramdb_cluster_group_stalls_totalcounterTimes a consensus group on this node stalled.
scramdb_cluster_retries_totalcounterTimes a cluster mechanism tried again after a failure, labelled mechanism.
scramdb_cluster_retries_exhausted_totalcounterTimes a cluster mechanism gave up after its retries or its deadline ran out, labelled mechanism.
scramdb_cluster_timeouts_totalcounterWaits of a cluster mechanism that reached their deadline, labelled mechanism.
scramdb_cluster_redispatches_totalcounterWork placed again on another node after its first node failed, labelled mechanism.
scramdb_cluster_retry_rate_per_secondgaugeRetries a second across every mechanism over the last ten seconds. A sustained rise is a retry storm.
scramdb_group0_reply_timeouts_total / scramdb_shard_group_reply_timeouts_totalcounterMetadata group and shard group proposals whose answer did not come in time. Each may still commit.
scramdb_group0_mailbox_timeouts_total / scramdb_shard_group_mailbox_timeouts_totalcounterRequests the group did not take in time. None changed anything.
scramdb_group0_read_retries_total / scramdb_shard_group_read_retries_totalcounterRead confirmations asked again after an answer did not come.
scramdb_group0_apply_wait_timeouts_total / scramdb_shard_group_apply_wait_timeouts_totalcounterWaits for this node to apply a committed entry that ran out of time.
scramdb_cluster_swim_members_alive / _suspect / _deadgaugeCluster members this node's failure detector sees alive, suspects, and has declared failed.
scramdb_cluster_swim_local_healthgaugeThis node's failure detector health score. Above zero, the node itself is missing probe answers.
scramdb_cluster_swim_member_up_total / _suspect_total / _down_totalcounterTimes a member came up, became suspect, or was declared failed in this node's failure detector.
scramdb_cluster_swim_malformed_messages_totalcounterFailure detector messages from peers that could not be read and were dropped.
scramdb_cluster_swim_send_failures_totalcounterFailure detector messages this node could not hand to a peer's connection.
scramdb_cluster_swim_joins_dropped_totalcounterMember joins dropped because the failure detector's join queue was full.
scramdb_cluster_swim_events_coalesced_totalcounterMembership events replaced by a newer event of the same member before the listener took them.
scramdb_cluster_swim_events_evicted_totalcounterMembership events dropped because the listener fell behind and the event queue was full.
scramdb_cluster_swim_events_pending / _capacitygaugeMembers with a membership event not yet taken, and the most members the queue holds (sized from the cluster).
scramdb_cluster_exchange_partial_chunksgaugeExchange streams holding part of a chunk in reassembly on this node.
scramdb_cluster_exchange_malformed_frames_total / scramdb_cluster_exchange_orphan_frames_totalcounterExchange frames dropped because they could not be read, or because no query on this node was waiting for their stream.
scramdb_cluster_exchange_out_of_order_total / _undecodable_chunks_total / _reassembly_overflows_totalcounterExchange streams failed because a frame arrived out of order, a chunk could not be decoded, or a chunk grew past the reassembly limit.
scramdb_cluster_exchange_producer_failures_received_totalcounterExchange streams whose producer reported a failure to this node.
scramdb_cluster_stream_frames_malformed_totalcounterFrames of a cluster service from peers that could not be read and were dropped.
scramdb_cluster_point_forwards_totalcounterPoint reads this node forwarded to their group's leader.
scramdb_cluster_point_forward_send_failures_total / _remote_errors_total / _reply_send_failures_totalcounterForwarded point reads that could not be sent, that the leader answered with an error, and answers to other nodes' forwarded reads that could not be sent back. A forwarded read that could not be sent or got no answer is answered on the general path instead; an error the leader reported is the statement's answer.

The cause of a failed connection is one of connect_timeout, refused, reset, unreachable, unresolved, handshake_timeout, rejected, version (the two nodes share no cluster protocol version: upgrade the older one), identity, tls, protocol, closed and other.

The mechanism is one of the following, and each family lists only the mechanisms that can record it, so no sample reads as a measured zero:

  • dial and lane_dial: connecting to a peer;
  • point_forward, fragment and fetch: a distributed statement's requests;
  • install: fetching a state snapshot;
  • drain and autojoin: membership changes;
  • metadata_group and shard_group: requests to a consensus group;
  • commit_vote and commit_outcome: the commit protocol's resends.

A fragment timeout is a reader that missed its deadline. Its buckets are read on another replica, which is a fragment re-dispatch.

Alerting on cluster metrics​

Three conditions worth paging on. Each maps to a documented design decision rather than a guessed threshold:

yaml
groups:
- name: scramdb-cluster
rules:
- alert: ScramDBGroupsQuarantined
expr: scramdb_cluster_quarantined_groups > 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $value }} raft group(s) quarantined"
description: >
A group's storage was quarantined at boot for corruption and has
not finished its membership-replace rejoin. This gauge is
recomputed from the real gated set every reconcile round, not
decremented, so restarting the node will not clear a stale
reading; only a completed rejoin does.

- alert: ScramDBLogCompactionStalled
expr: |
rate(scramdb_cluster_log_compactions_total[15m]) == 0
and rate(scramdb_cluster_group_commit_ops_total[15m]) > 0
for: 30m
labels:
severity: warning
annotations:
summary: "Raft log compaction has not run on a busy node in 30 minutes"
description: >
The node is committing but not compacting. Either its state
snapshots keep failing (check
scramdb_cluster_state_snapshot_failures_total and
scramdb_cluster_state_snapshots_abandoned_total: a full disk or a
memory_bytes budget too small), or a replica is behind but still
inside the lag limit. A replica lagging past the limit stops
holding the log and is repaired from a snapshot when it returns.

- alert: ScramDBPartitionDropsDetected
expr: increase(scramdb_cluster_partition_drops_total[5m]) > 0
labels:
severity: critical
annotations:
summary: "Cluster transport frames are being silently dropped"
description: >
This counter only moves when a network partition is injected
(chaos testing) or something outside the engine is dropping
cluster transport frames. It is zero in every normal deployment,
so any increase is worth paging on.

Index memory (always present)​

The memory each index holds, read when /metrics is scraped. A primary key, a unique key and every btree, hash and gin index are kept in memory, so this is all the memory they use. Vector indexes report their own memory under Vector index metrics.

MetricTypeMeaning
scramdb_index_memory_bytesgaugeMemory one index holds, labelled database, schema, table, index and method. Only the 1000 largest indexes get a series.
scramdb_index_memory_other_bytesgaugeMemory held by the indexes past those 1000, which have no series of their own.
scramdb_index_memory_other_indexesgaugeHow many indexes are past those 1000.
scramdb_index_memory_total_bytesgaugeMemory held by every index on the node.

The same figure is in SQL. scram_indexes lists each index of the current database with schemaname, relname, indexrelname, method, entries, memory_bytes, and it shows only the tables the reading role may see. pg_relation_size('an_index') gives that index's memory, and pg_indexes_size('a_table') adds up a table's indexes:

sql
SELECT indexrelname, method, entries, pg_size_pretty(memory_bytes)
FROM scram_indexes ORDER BY memory_bytes DESC LIMIT 10;

Index run files hold the sorted, on-disk part of an index too large to keep in memory. Their counters are always present and stay at zero until an index has a run file:

MetricTypeMeaning
scramdb_index_run_filter_probes_totalcounterIndex run filter probes made.
scramdb_index_run_filter_negatives_totalcounterIndex run filter probes that ruled a key out.
scramdb_index_run_filter_false_positives_totalcounterIndex run filter probes that passed a key the run does not hold.
scramdb_index_run_data_pages_read_totalcounterIndex run data pages read by lookups and scans.
scramdb_index_run_fence_pages_read_totalcounterIndex run fence pages read by lookups and seeks.

GIN index metrics​

Process-wide totals over every GIN index (USING gin) on the node, always present, on the same /metrics body. The memory each GIN index holds is in Index memory; EXPLAIN shows which index a query probes.

MetricTypeMeaning
scramdb_gin_probes_totalcounterGIN index probes run.
scramdb_gin_candidates_totalcounterCandidate rows the posting lists offered.
scramdb_gin_recheck_rejected_totalcounterCandidates the recheck of the query's filter rejected.
scramdb_gin_rows_returned_totalcounterRows GIN probes returned after their recheck.
scramdb_gin_snapshot_scans_totalcounterGIN-planned reads inside a REPEATABLE READ or SERIALIZABLE transaction, answered by scanning the table as of the snapshot.
scramdb_gin_entries_inserted_total / scramdb_gin_entries_deleted_totalcounterIndex entries added / removed by writes.
scramdb_gin_probe_secondshistogramTime one GIN probe took, posting lists to returned rows.

A high recheck_rejected_total against candidates_total means the index narrows poorly for your filters; a rising snapshot_scans_total means snapshot transactions are reading whole tables where a READ COMMITTED read would probe the index.

Self-driving (always present)​

Ejder, the self-driving loop (Self-driving with Ejder). A node that does not run it reports scramdb_autonomy_running at 0 and zeros, never an absent series.

MetricTypeMeaning
scramdb_autonomy_runninggaugeWhether the self-driving loop runs on this node, 1 or 0.
scramdb_autonomy_parkedgaugeWhether the loop stopped deciding after repeated failures, 1 or 0.
scramdb_autonomy_ticks_totalcounterDecisions the loop took.
scramdb_autonomy_tick_secondshistogramTime one decision took, in seconds.
scramdb_autonomy_tick_interval_msgaugeThe interval between decisions in force now, in milliseconds.
scramdb_autonomy_self_cost_permillegaugeShare of one core the last decision cost, in thousandths.
scramdb_autonomy_throttled_totalcounterDecisions after which the loop slowed its own interval.
scramdb_autonomy_setpoints_applied_totalcounterSetting changes the loop applied.
scramdb_autonomy_setpoints_reverted_totalcounterSettings put back because the change made things worse.
scramdb_autonomy_open_windowsgaugeChanges whose effect is being checked now.
scramdb_autonomy_windows_evicted_totalcounterChecks dropped unfinished because too many were open at once.
scramdb_autonomy_state_refused_totalcounterDecisions whose state was over its size bound and was not kept.
scramdb_autonomy_audit_failures_totalcounterDecision records that could not be written.
scramdb_autonomy_finishes_dropped_totalcounterEnds of background changes dropped before they were recorded.
scramdb_autonomy_records_dropped_totalcounterDecisions not written to the recording because it was full or failed.
scramdb_autonomy_tick_failures_totalcounterDecisions that failed, by reason: unavailable, compile, timeout, fuel, memory, output_too_large, trap or frame.
scramdb_autonomy_actions_totalcounterActions the loop proposed, by outcome: dispatched, advised, refused, accepted, reverted, expired, completed or failed.

Index builds and compaction​

An index build and a compaction merge never work on one table at the same time, as in PostgreSQL, where CREATE INDEX and VACUUM take conflicting locks. Background compaction yields: a merge running when a build starts stops, and the table is merged on a later pass. An explicit VACUUM waits for the builds on its table to finish, and a build waits for a running VACUUM, for at most lock_timeout when it is set (then it fails with SQLSTATE 55P03). Process-wide, always present:

MetricTypeMeaning
scramdb_compaction_merges_totalcounterBackground merges that completed, whatever started them.
scramdb_compaction_rows_reclaimed_totalcounterRows background merges dropped as dead.
scramdb_compaction_versions_held_totalcounterRow versions merges kept only because a retention hold stood, counted once per merge batch.
scramdb_compaction_merges_yielded_to_index_builds_totalcounterBackground merges stopped or skipped because an index build held their table.
scramdb_index_builds_waited_for_merges_totalcounterIndex builds that waited for a merge running on their table to end.
scramdb_index_build_merge_wait_timeouts_totalcounterIndex builds that stopped waiting for a merge on their table at their lock timeout.
scramdb_vacuums_waited_for_index_builds_totalcounterVACUUM runs that waited for the index builds holding their table to end.

Two table storage parameters also steer the background merge. A table that took in rows past its autovacuum_vacuum_insert_threshold and autovacuum_vacuum_insert_scale_factor has its small segments merged without waiting for the usual number, and a table with an autovacuum_vacuum_cost_delay pauses after a merge for that delay for each autovacuum_vacuum_cost_limit pages the merge read. Process-wide, always present:

MetricTypeMeaning
scramdb_compaction_insert_merges_totalcounterBackground merges started by a table's insert threshold instead of the usual number of small segments.
scramdb_compaction_pace_pause_secondshistogramTime background merges paused to keep to their table's cost delay.

A separate fence orders a compaction's row relocation against a concurrent write that resolves a row's physical position by table (DELETE, UPDATE, ON CONFLICT DO UPDATE/DO SELECT, a replica's apply): the write holds the fence until its commit is durable, and a compaction raises it before it swaps its merged segments in, so a swap never relocates a row out from under a write already under way. A statement that reads a table's row identities (a ctid column, a TABLESAMPLE draw) holds nothing: a compaction moves a row and keeps its identity, so a row has one ctid and one place in a sample whatever compaction runs meanwhile, and VACUUM FULL does not wait for such a reader. Process-wide, always present:

MetricTypeMeaning
scramdb_writes_waited_for_compaction_totalcounterStatements that waited for a compaction to finish moving their table's rows.
scramdb_write_compaction_wait_timeouts_totalcounterStatements that stopped waiting for a compaction at their lock timeout.
scramdb_compaction_guarded_swaps_totalcounterCompaction swaps made while the statements using their table's rows were held back.
scramdb_compaction_swaps_abandoned_totalcounterBackground compaction batches dropped because the statements holding their table's rows did not finish in time.
scramdb_vacuum_lock_timeouts_totalcounterVACUUM FULL runs stopped at their lock timeout while statements held the table's rows.
scramdb_compaction_late_deletes_carried_totalcounterRows deleted while a compaction ran that it carried to their new places.
scramdb_compaction_writer_drain_secondshistogramTime a compaction waited for the statements holding its table's rows to finish.
scramdb_compaction_writer_pause_secondshistogramTime a compaction held back the statements using its table's rows while it swapped.
scramdb_write_compaction_wait_secondshistogramTime a statement waited for a compaction before it could read or write its rows.
scramdb_compaction_writer_tablesgaugeTables with writer counters allocated.

swaps_abandoned_total or vacuum_lock_timeouts_total rising means the statements holding a table's rows (its writers) never drain within the wait bound: background compaction backs off quietly, VACUUM FULL fails loudly with 55P03.

Compaction by table​

What the merges of each table gave back, counted where a merge completes whether the background pass or a VACUUM ran it, and read when /metrics is scraped. A table has a series once a merge of it has finished since the server started, and loses it when the table is dropped. Each family is labelled database, schema and table, and only the 1000 tables that reclaimed the most bytes get a series.

MetricTypeMeaning
scramdb_table_compaction_rewrites_totalcounterMerges of a table that completed since the server started, by the background pass or a VACUUM.
scramdb_table_compaction_rows_reclaimed_totalcounterRows the merges of a table dropped as dead since the server started.
scramdb_table_compaction_bytes_reclaimed_totalcounterBytes of segments the merges of a table freed past the bytes they wrote since the server started.
scramdb_table_compaction_other_tablesgaugeTables merged since the server started that have no compaction series of their own, past the series bound.

scramdb_table_compaction_other_tables is how many tables are past the 1000: a table is left out of the list, never out of the count. It reads 0 while every merged table has its series.

Query worker pool​

The engine's core-pinned query worker pool ([execution] workers), process-wide, always present:

MetricTypeMeaning
scramdb_query_worker_countgaugePinned workers the engine's query worker pool runs.
scramdb_query_worker_queue_depthgaugeOutstanding tasks queued across the query worker pool's pinned workers, summed.
scramdb_query_worker_jobs_total{worker}counterJobs each query worker finished, one series per worker. A server under load shows every worker moving; one series far above the rest is work that is not spreading.
scramdb_query_dispatches_idle_totalcounterQuery dispatches placed on an idle query worker.
scramdb_query_dispatches_busy_totalcounterQuery dispatches placed on a busy query worker because every probed worker was busy: a rising share means the pool is saturated.
scramdb_query_worker_cpu_quota_cores_permillegaugeThis engine's claimed cores per thousand of the machine's full, unconstrained cpuset: 1000 unconstrained, lower under a CPU limit (a container --cpus, a Kubernetes CPU request).

worker_queue_depth climbing while worker_count holds steady means query work is arriving faster than the pool drains it; cpu_quota_cores_permille well below 1000 on a box that looks otherwise idle means the CFS quota, not the workload, is the ceiling on worker_count.

A table's parallel_workers storage parameter limits the workers a scan of it runs on (0 is a serial scan). Process-wide, always present, counting only scans that ran under such a parameter:

MetricTypeMeaning
scramdb_scan_capped_dispatches_totalcounterScans that ran under their table's parallel_workers.
scramdb_scan_capped_workers_totalcounterWorkers given to scans that ran under their table's parallel_workers.

The second over the first is the width a capped scan ran at.

Stand-in query workers​

A fixed-pool query worker whose core is parked on a lock wait hands its core to a stand-in worker for the wait's duration, so a lock contended by one statement does not idle every query on that core. Process-wide, always present:

MetricTypeMeaning
scramdb_query_standinsgaugeStand-in query worker threads alive right now, running or idle-cached.
scramdb_query_standin_starts_totalcounterStand-in query worker threads started.
scramdb_query_relief_standinsgaugeRelief query workers alive right now: started when a queued statement waited a full watchdog sample while every worker was busy, at most one per core.
scramdb_query_relief_starts_totalcounterRelief query workers started. A steady rise means long statements often hold every worker.
scramdb_pool_standinsgaugeStand-in threads of the core-pinned pool alive right now.
scramdb_pool_standin_starts_totalcounterStand-in threads of the core-pinned pool started.
scramdb_pool_job_parks_totalcounterJobs of the core-pinned pool that parked and handed their worker's queue on.
scramdb_query_worker_idle_wakes_totalcounterTimes an idle query worker thread woke from its wait, with or without work. An idle server holds it still apart from one wake per worker to hand freed memory back to the system; it moves with statements arriving, never on a timer.

Vector index metrics​

Every anode/manode vector index family lives on the scramdb_vector_ prefix, appended to the same /metrics body as everything above. One sample feeds both the Prometheus families below and the system views in the next section, so a number never disagrees between them. A family the sample has not measured (a recall job that never ran, a per-index rate with no traffic yet) is omitted from the scrape entirely rather than rendered as a fabricated 0, the same rule the WAL archive metrics above follow. Cardinality is bounded by the number of indexes (and cluster parts) open on the node, reported by scramdb_vector_series so you can see the cost.

Node totals (no index label, always present):

MetricTypeMeaning
scramdb_vector_memory_budget_total_bytesgaugeThe memory ceiling configured for all vector indexes on this node together.
scramdb_vector_memory_reserved_bytesgaugeMemory currently reserved by vector index arenas against the total ceiling.
scramdb_vector_memory_in_use_bytesgaugeMemory vector indexes use right now: their arenas, row maps and write buffers together. The write buffers are charged to execution memory, not to the vector ceiling. A build's working memory (the model training sample, the convert buffers) is not counted here: it is charged to storage.memory.maintenance_bytes, like every index build.
scramdb_vector_memory_uncharged_bytesgaugeMemory vector indexes use beyond what the node's vector budget reserved. It reads zero while every byte an index uses is reserved.
scramdb_vector_memory_engine_bytesgaugeMemory the engine holds for vector index ledgers, bitmaps and buffers.
scramdb_vector_memory_budget_refusals_totalcounterIndex builds, rebuilds and arena regrows refused because the total ceiling would be exceeded.
scramdb_vector_indexesgaugeVector indexes open on this node.
scramdb_vector_regrows_total / scramdb_vector_regrows_runningcounter / gaugeArena regrows completed / running on this node.
scramdb_vector_scans_total{tier}counterIndexed vector searches served, labelled exact, traversal, or postfilter.
scramdb_vector_scan_secondshistogramTime an indexed vector search took end to end.
scramdb_vector_rerank_rows_total / scramdb_vector_rerank_secondscounter / histogramCandidate rows re-ranked exactly after an index search, and how long it took.
scramdb_vector_widening_rounds_totalcounterExtra index rounds taken when a filtered search had fewer than the requested rows.
scramdb_vector_underfilled_totalcounterFiltered searches that returned fewer rows than requested after the scan limit.
scramdb_vector_knn_join_batches_total / scramdb_vector_knn_join_queries_totalcounterBatches / individual query vectors sent to an index by a nearest-neighbour join.
scramdb_vector_threshold_scans_totalcounterDistance threshold searches served by an index.
scramdb_vector_exact_scans_total / scramdb_vector_exact_scan_secondscounter / histogramNearest-neighbour searches the vector operator answered with an exact scan, and how long they took. A table scan with no vector operator is not counted.
scramdb_vector_distributed_scans_total / scramdb_vector_distributed_merge_rows_totalcounterNearest-neighbour statements fanned out to cluster readers, and rows merged from them.
scramdb_vector_distributed_redispatches_totalcounterReaders of a distributed nearest-neighbour search whose buckets another replica served, after the reader missed its deadline or could not be reached.
scramdb_vector_distributed_failures_totalcounterDistributed nearest-neighbour searches that failed because a reader failed or no replica of a bucket was left.
scramdb_vector_builds_running / scramdb_vector_build_secondsgauge / histogramVector index builds and rebuilds running, and how long one takes.
scramdb_vector_reconcile_rows_totalcounterRows replayed into vector indexes at startup to catch up with the table.
scramdb_vector_build_owed_deletes_totalcounterDeleted rows an index build kept for read snapshots older than the delete, so those snapshots still find them.
scramdb_vector_delete_tail_reclaims_totalcounterReclaims that rebuilt a vector index's delete tail (the deleted rows kept for older snapshots) once some of those rows were no longer needed.
scramdb_vector_delete_tail_batches_dropped_total / scramdb_vector_delete_tail_batches_relinked_totalcounterDelete tail batches a reclaim dropped whole because no read snapshot needs them, and batches it kept whole without copying them.
scramdb_vector_delete_tail_batches_merged_total / scramdb_vector_delete_tail_boundary_batches_folded_totalcounterSmall delete tail batches a reclaim copied into a larger one, and batches it read row by row because a read snapshot's boundary cut through them. Every batch a reclaim judges lands in exactly one of these four counters.
scramdb_vector_fit_sample_bytes_totalcounterBytes of training samples vector index model fits charged to execution memory: a build's fits through storage.memory.maintenance_bytes, every other fit (a streaming INSERT or COPY that trains the model, a background retrain) to the execution pool directly.
scramdb_vector_setting{key} / scramdb_vector_setting_source{key,source}gaugeThe value in force for a [vector] setting, and which layer set it (config, override, setpoint).
scramdb_vector_seriesgaugePrometheus series the vector index families currently emit, computed from the same scrape it appears in.

Per index (labels index, table, column, method, group):

MetricTypeMeaning
scramdb_vector_index_state{state}gaugeWhether the index is in this state (building, ready, regrowing, rebuilding, tail, invalid); one state reads 1.
scramdb_vector_index_live_vectors / scramdb_vector_index_tombstonesgaugeVectors the index holds and can return, and deleted vectors it still carries until compaction.
scramdb_vector_index_base_rowsgaugeRows a branch's index reads through the index it was cut from, until it folds; live_vectors counts only the rows the branch's own index holds.
scramdb_vector_index_unindexed_rowsgaugeRows served exactly because the index has no memory left to hold them (the tail state).
scramdb_vector_index_ledger_live_rows / scramdb_vector_index_delete_tail_rowsgaugeRows the index's row map holds that are not deleted (what a rebuild sizes the index for), and deleted rows the index keeps for read snapshots older than the delete until a reclaim finds no snapshot needs them.
scramdb_vector_index_arena_slots / scramdb_vector_index_arena_used_permillegaugeSlots the index's arena is promised, and the share in use, in thousandths.
scramdb_vector_index_regrows_total / scramdb_vector_index_regrow_runningcounter / gaugeTimes the arena was reopened with a larger promise, and whether it is happening now.
scramdb_vector_index_memory_budget_bytes / scramdb_vector_index_memory_in_use_bytesgaugeThe memory ceiling the index runs under, and resident memory it uses right now.
scramdb_vector_index_memory_plane_bytes{plane}gaugeResident memory by plane: codes, graph, exact_vectors, directory, slot_state, coarse_model, journal_buffers.
scramdb_vector_index_ledger_bytes / scramdb_vector_index_disk_bytes / scramdb_vector_index_journal_bytesgaugeMemory the row-id ledger uses, bytes the index occupies on disk, and bytes of write history kept before compaction.
scramdb_vector_index_ops_total{op} / scramdb_vector_index_ops_failed_total{op}counterOperations completed / refused or failed, op one of search, insert, update, delete, flush.
scramdb_vector_index_search_qpsgaugeSearches per second over the recent window.
scramdb_vector_index_search_latency_seconds{quantile} / scramdb_vector_index_write_latency_seconds{quantile}gaugeSearch / write latency over the recent window, quantile one of 0.5, 0.95, 0.99, 0.999, max.
scramdb_vector_index_stalenessgaugeWrites accepted but not yet searchable. Reads 0: a write is searchable by the very next query, so there is nothing for this gauge to ever report as pending.
scramdb_vector_index_compactions_total / scramdb_vector_index_compaction_runningcounter / gaugeCompactions completed, and whether one is running now.
scramdb_vector_index_fits_totalcounterCoarse model fits the index completed.
scramdb_vector_index_maintenance_running / scramdb_vector_index_graph_armedgaugeWhether the index's own maintenance is running, and whether its navigation graph is built and in use.
scramdb_vector_index_edge_retries_totalcounterNavigation graph edge publications retried because another writer changed the same edge list first. A retry never loses the edge; a high rate means many threads link into the same nodes at once.
scramdb_vector_index_unplaced_slotsgaugeVectors the index's last model fit could not place in the partition it chose for them, for want of room there. While that is a material share of the index, the navigation graph stays off.
scramdb_vector_index_io_reads_total / scramdb_vector_index_io_bytes_read_totalcounterReads and bytes read for vectors spilled to disk.
scramdb_vector_index_cache_hit_ratiogaugeShare of vector reads served from memory.
scramdb_vector_index_simd_level{level}gaugeThe instruction set the index runs with; the matching level reads 1.
scramdb_vector_index_knob{knob}gaugeThe search knob value in force for the index.
scramdb_vector_index_estimator_agreementgaugeShare of candidates whose estimated order matched the exact order in recent searches. Emitted only once a real measurement exists.
scramdb_vector_index_recall{k}gaugeRecall measured by the last recall job for this index. Absent until a recall job has actually run, never a guessed value.
scramdb_vector_index_last_build_wall_microseconds / scramdb_vector_index_last_build_cpu_microsecondsgaugeWall time and CPU time (summed over every thread that worked on it) of the index's last build in this process. Absent until the index is built here: a restart reads absent until the next build, never a stale or guessed figure.
scramdb_vector_index_last_build_peak_memory_bytes / scramdb_vector_index_last_build_rowsgaugeResident memory of the index at the end of its last build (its peak: a build only adds), and rows it indexed.
scramdb_vector_index_search_secondshistogramTime one search of the index took, end to end.
scramdb_vector_index_last_flush_unix_seconds / scramdb_vector_index_rows_since_flushgaugeWhen the index last made its writes durable, and rows written since then.
scramdb_vector_index_last_reindex_unix_secondsgaugeWhen the index was last rebuilt. Emitted only once a REINDEX has actually run.

Cluster (labels index, table, group; present only once a distributed vector index part exists):

MetricTypeMeaning
scramdb_vector_group_owner_is_selfgaugeWhether this node owns the bucket group's index part.
scramdb_vector_group_part_state{state}gaugeThe state of the index part for the bucket group (building, ready, regrowing, rebuilding, tail, invalid, absent, waiting_memory: a hosted bucket group this pass's own memory admission left out of the count it admitted to build).
scramdb_vector_group_rebuilds_total / scramdb_vector_group_rebuild_runningcounter / gaugeTimes the index part was rebuilt after the bucket group moved, and whether one is running now.
scramdb_vector_group_learner_readygaugeWhether a learner may serve vector searches for the bucket group.
scramdb_vector_group_memory_in_use_bytesgaugeResident memory the index part for the bucket group uses.
scramdb_vector_cluster_parts / scramdb_vector_cluster_parts_missinggaugeIndex parts this node holds for distributed tables, and bucket groups it owns whose part is not built.
scramdb_vector_cluster_min_recall{index,k}gaugeThe lowest recall any of this node's parts of the index measured at its last recall run. Absent until measured; the whole cluster's lowest is the lowest over every node.

Recall job (present only on a cluster node, where the job runs):

MetricTypeMeaning
scramdb_vector_recall_parts_measured_total / scramdb_vector_recall_parts_skipped_totalcounterIndex parts the recall job measured on this node, and parts a pass was due to measure and could not (each such part says why in scram_vector_index_parts.recall_note).
scramdb_vector_recall_pass_cpu_secondshistogramCPU time of one recall pass on this node.
scramdb_vector_recall_next_wait_seconds{reason}gaugeSeconds until the job next measures a part, and why: churn (the wait follows how much of a part changed), cost (the one-percent CPU duty cycle binds), ceiling (a lightly changed part waits the hour ceiling), pinned (vector.recall_interval_secs), first (a part never measured), unchanged (nothing changed since the last measurement, so nothing is due: +Inf), or off.
scramdb_vector_recall_probes{reason}gaugeProbe queries the last measurement sampled, and why: precision (enough for a one-point standard error at the part's last recall), floor (8), ceiling (64), memory (the vector memory budget held fewer), or pinned (vector.recall_probes).

Vector index system views​

The same sample backs five read-only system views, so a dashboard that prefers SQL over scraping Prometheus sees the identical numbers. Every row leads with node, the node that measured it.

On a cluster, the four statistics views answer for the whole cluster from whichever node you query: that node's rows and every other member's, gathered at query time. A member that does not answer within vector.stats_node_timeout_ms (default 5000) contributes one row with its node, the word unreachable in state (or metric), the reason in error, and NULL everywhere else, never a silently shorter result. scram_vector_settings is the queried node's own settings. On a single node every view is that node's rows.

ViewColumns
scram_vector_settingsnode, key, value, source, unit, live, description - every [vector] key, the value in force, and which layer set it
scram_vector_indexesnode, index, schema, table, column, method, metric, opclass, state, arena_slots, arena_used_permille, regrows, live, base_rows, tombstones, unindexed_rows, memory_budget, memory_in_use, disk_bytes, journal_bytes, search_qps, p50_us, p99_us, staleness, compactions, fits, graph_armed, simd_level, estimator_agreement, recall_at_10, last_flush, last_reindex, options, build_wall_us, build_cpu_us, build_peak_memory, build_rows, search_count, search_p99_us, error - one row per index per node; a distributed index's parts on a node fold into its row (build cost summed, NULL unless every part was built by that process; recall the lowest part's)
scram_vector_index_partsnode, index, schema, table, group, owner, is_self, state, live, memory_in_use, rebuilds, learner_ready, rebuild_running, recall_at_10, recall_probes, recall_measured_at, recall_note, error - one row per bucket group each node hosts; is_self marks the rows of the node you queried; recall_note says why a part has no recall figure (too few vectors, over vector.recall_max_rows, not ready, or no memory); empty on a single node
scram_vector_memorynode, total, reserved, in_use, engine_bytes, refusals, error - one summary row per node
scram_vector_activitynode, metric, value, unit, description, error - the node-wide vector counters, one row per counter per node
sql
SELECT node, index, state, live, memory_in_use, search_qps FROM scram_vector_indexes;
SELECT node, "group", state, live, recall_at_10, recall_note
FROM scram_vector_index_parts WHERE index = 'docs_embedding_idx' OR error IS NOT NULL;
SELECT key, value, source, live FROM scram_vector_settings WHERE key = 'vector.over_fetch';

EXPLAIN on an indexed vector query names the tier chosen and the candidate count; EXPLAIN ANALYZE adds candidates fetched, widening rounds taken, re-rank time, and whether the answer came back underfilled. See Vector Search for the query shapes these views and counters describe, and Tuning: Vector Indexes for how to act on them.

GPU metrics (present once a GPU is in use)​

On a server that brought up a GPU at startup, with [gpu] metrics_enabled = true (the default). A server without one shows no scramdb_gpu_* family at all.

MetricTypeMeaning
scramdb_gpu_kernel_launches_totalcounterGPU kernels launched.
scramdb_gpu_kernel_errors_totalcounterGPU kernel launches the device refused.
scramdb_gpu_fallback_totalcounterTimes work admitted to the GPU ran on the CPU instead (a morsel holding a NULL, a condition the GPU row filter cannot read exactly, an unhealthy device).
scramdb_gpu_oom_totalcounterGPU memory requests refused for lack of device memory.
scramdb_gpu_transfer_htod_bytes_totalcounterBytes copied from host memory to GPU memory.
scramdb_gpu_transfer_dtoh_bytes_totalcounterBytes copied from GPU memory to host memory.
scramdb_gpu_vram_used_bytesgaugeGPU memory held by queries now.
scramdb_gpu_vram_budget_bytesgaugeGPU memory queries may hold.

A query ran on the GPU when scramdb_gpu_kernel_launches_total moved across it and scramdb_gpu_fallback_total did not; see GPU Acceleration.

What is not on this endpoint​

  • No buffer-pool hit-ratio, cache-hit-ratio, or per-worker-utilization metric exists. Memory and parallelism verification on this site relies on \timing and OS-level observation instead, see Memory and Parallelism.

Log levels and format​

bash
SCRAMDB_LOG=debug SCRAMDB_LOG_FORMAT=json scramdb -c scramdb-config.toml --pg-address 127.0.0.1:5432 --pg-no-auth

Expected: stderr lines shaped like {"ts":"...","level":"INFO","target":"...","msg":"Metrics endpoint listening on http://0.0.0.0:9090/metrics"}. Every record is exactly one line: a control character in a message is escaped inside the JSON string.

SCRAMDB_LOG​

Controls the level filter. Accepted values (case-insensitive): off, error, warn, info, debug, trace. Default when unset: warn, so a release binary starts quiet. An unrecognized value does not fail loud, it silently falls back to warn, so a typo in this env var will not show up as an error, it just quietly serves you the default level.

SCRAMDB_LOG_FORMAT​

Set to json (case-insensitive) for structured JSON output. Anything else, including leaving it unset, produces the default plain-text format.

Plain-text format: [{level}] [{timestamp}] {message}, for example:

text
[INFO] [2026-08-02T10:15:30.123Z] Metrics endpoint listening on http://0.0.0.0:9090/metrics

The timestamp of both formats is UTC, to the millisecond, marked Z, whatever zone the host is set to.

JSON format: {"ts":"<timestamp>","level":"<LEVEL>","target":"<module path>","msg":"<message>"}.

A WARN or ERROR line logged while a statement runs ends with the statement's context, (pid N, database D, user U, statement shape <16 hex digits> "<masked start>"), so a line can be tied to its session and statement shape. The masked start never holds a client value. Lines at INFO and below, and lines outside a statement, carry no context.

Logging is asynchronous: a dedicated low-priority background thread drains log messages and writes to stderr, so logging never blocks a query worker. Output always goes to stderr, never directly to a file, redirect it yourself (scramdb ... 2> scramdb.log) or capture it with your process supervisor.

One additional env var is useful for deeper diagnosis alongside SCRAMDB_LOG, covered on another page in this section:

  • SCRAMDB_GPU_FORCE=1, forces GPU pipeline-shape eligibility for benchmarking. See GPU Acceleration.

Next​

See Query Performance for how the JIT chunk counters are used in practice, GPU Acceleration for how to tell from the scramdb_gpu_* counters whether a query ran on the GPU, or Clustering for how the cluster counters above map onto a Docker Compose or Kubernetes deployment.

On this page