Observability
By the end of this page you will know how to scrape ScramDB's metrics, check liveness, and configure logging, with the exact set of metric names that exist today so you do not build a dashboard around a counter that isn't there.
The metrics and health endpoint
ScramDB serves both /metrics and /health on the same HTTP listener, one port, one process.
| Setting | Default | How to set it |
|---|---|---|
| Port | 9090 | [general] metrics_port in TOML, or --metrics-port <PORT> on the command line. Set to 0 to disable the endpoint entirely. |
| Bind address | 127.0.0.1 | [general] metrics_listen_address (an IP address, or "*" for every interface), or --metrics-listen-address. The shipped image and the Helm chart set "*" so a scraper can reach the pod. |
| Token | none | [general] metrics_bearer_token_file, or --metrics-bearer-token-file: every request then needs Authorization: Bearer <token> and is answered 401 without it. The file must not be readable by every user. |
/metrics
curl -s http://127.0.0.1:9090/metrics | head -20
Expected output: Prometheus text exposition format (Content-Type: text/plain; version=0.0.4; charset=utf-8), starting with lines like # HELP scramdb_queries_total .... If it fails: connection refused means either the server isn't running, metrics_port is set to 0, or a firewall is blocking the port. Remember the listener binds 0.0.0.0, so "connection refused from outside the host" is more likely a firewall rule than the endpoint being loopback-only.
Any path other than /metrics or /health returns 404 Not Found.
/health
curl -s http://127.0.0.1:9090/health
Expected output: ok (plain text, no JSON, no fields, HTTP 200). On a cluster node whose commit protocol has stopped after a failure (scramdb_cluster_dilith_host_stopped at 1), the answer is HTTP 503 Service Unavailable with the body the commit protocol stopped on this node: that node can no longer commit transactions and must be restarted, so a load balancer or orchestrator probing /health takes it out of service. The answer is 503 too while the server's log is failed (scramdb_wal_failed above 0): the body is the log failed and takes no writes until it heals or the server is restarted, a colon and the cause the log failed with. Such a server still answers reads and takes writes again by itself once its storage works, so /health returns to 200 ok without a restart (see Storage Engine and Error codes). There is no separate liveness/readiness distinction. Beyond those two cases it does not check database or storage health internally: a 200 ok proves the process and its HTTP listener are up, not that queries are succeeding. For query success/failure, use the scramdb_queries_ok / scramdb_queries_err counters on /metrics instead.
The endpoint reports on itself on the same /metrics body, always emitted:
| Metric | Type | Meaning |
|---|---|---|
scramdb_metrics_endpoint_requests_total | counter | Requests the metrics endpoint answered. |
scramdb_metrics_endpoint_request_timeouts_total | counter | Clients disconnected because their request did not arrive in time. |
scramdb_metrics_endpoint_response_timeouts_total | counter | Clients disconnected because they did not take their response in time. |
scramdb_metrics_endpoint_client_errors_total | counter | Client connections that failed before their answer was written. |
scramdb_metrics_endpoint_accept_failures_total | counter | Connections the metrics endpoint could not accept. |
scramdb_metrics_endpoint_render_seconds | histogram | Time to render one /metrics answer. |
Metric reference
Every metric named on this page is really emitted on /metrics, so you can build a dashboard from these names without checking first. The reverse does not hold: the endpoint also carries deep diagnostic counters that are not part of the supported surface and can change between releases. The families below are the ones worth alerting on and graphing.
Core query and connection metrics (always present)
| Metric | Type | Meaning |
|---|---|---|
scramdb_queries_total | counter | Total queries received. |
scramdb_queries_ok | counter | Queries that completed successfully. |
scramdb_queries_err | counter | Queries that errored. |
scramdb_queries_active | gauge | Queries currently executing. |
scramdb_query_avg_us | gauge | Rolling average query latency in microseconds. |
scramdb_connections_total | counter | Total connections accepted since startup. |
scramdb_connections_active | gauge | Currently open connections. |
scramdb_bytes_sent | counter | Total bytes sent to clients. |
scramdb_bytes_received | counter | Total bytes received from clients. |
scramdb_uptime_seconds | gauge | Seconds since the process started. |
Execution-tier metrics (always present)
| Metric | Type | Meaning |
|---|---|---|
scramdb_chunks_jit_total | counter | Morsel-chunks executed as compiled JIT machine code. |
scramdb_chunks_vm_total | counter | Morsel-chunks executed by the bytecode interpreter. |
scramdb_compiled_calls_total | counter | Sub-region JIT invocations. |
scramdb_interpreter_aggregate_top_cuts_total | counter | Interpreter aggregate partitions cut to the top groups of the query's ORDER BY with LIMIT. |
scramdb_compiled_aggregate_top_cuts_total | counter | Compiled aggregate partitions cut to the top groups of the query's ORDER BY with LIMIT. |
These are the counters used in the JIT scrape-delta technique described on Query Performance.
Client connections (always present)
Every client connection holds one of [general] max_connections places while its session lives, the last superuser_reserved_connections of them kept for superusers. Every connection is served on the network threads ([network] threads, by default the CPU budget's IO share: 2 threads on a 4-core machine, at most 8 by default), each serving many connections at once. An idle session waits for its client holding no thread and no buffer (it is parked); its next message, a deadline, a notification or a close wakes it.
| Metric | Type | Meaning |
|---|---|---|
scramdb_max_connections | gauge | Connection places the client listener enforces (max_connections). |
scramdb_connection_places_taken | gauge | Connection places held by sessions. |
scramdb_connection_starting | gauge | Connections still negotiating SSL, sending their startup packet or authenticating. |
scramdb_connection_parked | gauge | Idle sessions waiting for their client with no buffer held. |
scramdb_connection_wakes_total | counter | Idle sessions woken by a close, a notification or the end of a COPY. |
scramdb_connection_idle_wakes_total | counter | Times an idle session was woken by its own deadline. An idle server with no deadlines set holds it still. |
scramdb_connection_refused_too_many_total | counter | Connections refused because every connection place was taken. |
scramdb_connection_refused_reserved_total | counter | Connections refused because only the places reserved for superusers were left. |
scramdb_connection_refused_role_limit_total | counter | Connections refused by their role's connection limit. |
scramdb_connection_refused_database_limit_total | counter | Connections refused by their database's connection limit (CREATE DATABASE ... CONNECTION LIMIT). |
scramdb_connection_refused_no_login_total | counter | Connections refused because their user names no role, or a role that may not log in (NOLOGIN); both FATAL 28000, as PostgreSQL. |
scramdb_connection_refused_database_connect_total | counter | Connections refused because the role holds no CONNECT privilege on the database it asked for (FATAL 42501, as PostgreSQL). |
scramdb_connection_refused_server_setting_total | counter | Connections refused because they asked to change a setting fixed when the server started (max_connections or superuser_reserved_connections in their startup options, FATAL 55P02). |
scramdb_connection_refused_startup_permission_total | counter | Connections refused because a role without the superuser attribute asked, in its startup options, to change a superuser-only setting such as track_counts (FATAL 42501). |
scramdb_connection_refused_startup_setting_total | counter | Connections refused because a setting their startup packet sent (TimeZone, DateStyle or any other) had a value the setting refuses (FATAL, the setting's own code, as PostgreSQL). |
scramdb_connection_stored_setting_failures_total | counter | Settings stored for a role, a database or the server that a login could not apply and skipped. |
scramdb_connection_reload_setting_failures_total | counter | Settings a configuration reload changed that an open session could not apply and kept as they were. |
scramdb_connection_hba_rejects_total | counter | Connections rejected by the host-based authentication rules when accepted. |
scramdb_connection_idle_session_timeouts_total | counter | Sessions closed because they stayed idle outside a transaction past idle_session_timeout (FATAL 57P05). |
scramdb_connection_startup_timeouts_total | counter | Connections closed because authentication did not finish within authentication_timeout. |
scramdb_connection_tls_failures_total | counter | Client TLS handshakes that failed. |
scramdb_connection_messages_refused_total | counter | Client messages refused because the server could not reserve memory to receive them. |
scramdb_connection_ended_client_gone_total | counter | Sessions whose client closed its connection without a Terminate message. |
scramdb_connection_ended_socket_error_total | counter | Sessions whose client connection failed. |
scramdb_connection_ended_terminated_total | counter | Sessions closed by pg_terminate_backend or a server shutdown. |
scramdb_connection_ended_in_transaction_total | counter | Sessions that ended with a transaction open, which was rolled back (its row, table and advisory locks released with it). |
scramdb_connection_startup_seconds | histogram | Time from accepting a connection to its session being ready for queries. |
The network threads themselves:
| Metric | Type | Meaning |
|---|---|---|
scramdb_network_threads | gauge | Threads serving client connections. |
scramdb_network_tasks | gauge | Connection and accept tasks alive on the network threads. |
scramdb_network_io_in_flight | gauge | Socket operations handed to the kernel that have not ended. |
scramdb_network_handoffs_total | counter | Times a network thread's work moved to another thread because one task ran too long. |
scramdb_network_long_polls_total | counter | Tasks that held a network thread past the hand-off bound. |
scramdb_network_io_errors_total | counter | Socket operation errors on the network threads, by error kind (kind label). |
scramdb_network_backend | gauge | The IO mechanism the network threads run on, 1 for the one in use (backend label: io_uring or epoll). |
scramdb_network_off_shard_total | counter | Client messages whose work ran on a query worker because it scales with the message (a large statement's parse, a describe, a login). |
scramdb_network_inline_reads_total | counter | Point reads answered on the network thread that received them, because every step of them was in memory. |
scramdb_network_moved_reads_total | counter | Point reads that took a query worker because a step of them could wait (a page to load, a lock, a first use). |
scramdb_network_blocking_waits_total | counter | Times a network thread waited on something that blocks. It stays at 0 in normal operation; if it rises, contact support with the server log. |
scramdb_metrics_scrapes_refused_total | counter | Metrics requests answered 503 because the endpoint was already serving its most concurrent scrapes. |
scramdb_network_handoffs_total and scramdb_network_long_polls_total stay at 0 in normal operation: statements run on the query workers, and a network thread only reads, writes and answers point operations by key. If either rises, contact support with the server log.
refused_too_many_total climbing means clients open more connections than max_connections allows: put a pooler in front, or raise the limit. idle_session_timeouts_total moves only when a session sets idle_session_timeout (off by default).
Statements (always present)
A session hands each statement to a query worker and waits for its result; a statement timeout, a cancel, the transaction timeout and a lost client all end that wait at once.
| Metric | Type | Meaning |
|---|---|---|
scramdb_statement_wait_seconds | histogram | Time a session waited for its statement's result from a query worker. |
scramdb_statement_timeouts_total | counter | Statements cancelled because they ran past statement_timeout. |
scramdb_statement_cancels_total | counter | Statements a client, a forced close or the memory watchdog cancelled while they ran. |
scramdb_statement_waits_cancelled_total | counter | Waits inside statements that ended because their statement was cancelled, timed out or given up. |
scramdb_statement_worker_kept_total | counter | Statements taken by the query worker that ran their session's previous statement. A session's statements stay on one worker while it is free, which keeps its data in that core's cache. |
scramdb_statement_worker_moved_total | counter | Statements taken by a query worker other than the one that ran their session's previous statement: a session's first statement, or one whose worker was busy. |
scramdb_prepared_shape_memos_total | counter | Prepared statements that took the decisions kept for their statement shape at Parse (who may run it, where it routes), so an Execute does not decide them again. |
scramdb_shape_gate_judgments_total | counter | Statement privilege checks made in full because no current check of the same statement shape was kept. It moves when a shape is first seen and after a grant, a role change or a schema change. |
scramdb_shape_point_classifications_total | counter | Statement shapes classified for the point-operation lane. |
scramdb_shape_write_routes_total | counter | Write routes decided and kept for their statement shape. |
scramdb_merge_cancels_total | counter | Statements cancelled while merging their grouped partial results: it moves when the merge of a GROUP BY saw the cancel and stopped. |
scramdb_merge_serialization_failures_total | counter | MERGE statements that ended with 40001: a REPEATABLE READ or SERIALIZABLE statement whose row another transaction changed after the snapshot (PostgreSQL answers the same), a distributed vote refused, or a changed row the engine cannot judge again. A READ COMMITTED statement over a row that changed is judged again and is not counted. |
scramdb_statement_transaction_timeouts_total | counter | Sessions closed because their transaction ran past transaction_timeout during a statement. |
scramdb_statement_clients_gone_total | counter | Sessions closed because their client went away during a statement. |
scramdb_statement_late_results_dropped_total | counter | Results of statements a session had already stopped waiting for, dropped when they arrived. |
scramdb_statement_implicit_blocks_total | counter | Multi-statement queries and pipelines run as one implicit transaction, as PostgreSQL runs them. |
scramdb_statement_implicit_blocks_rolled_back_total | counter | Implicit transactions rolled back because one of their statements failed. |
Row-by-row writes (always present)
A write whose computed value reads the rows the statement has already written (a VOLATILE function in an INSERT, a COPY column default that counts the table) is written one row at a time instead of in batches. Every other statement keeps the batch path and counts nothing here.
| Metric | Type | Meaning |
|---|---|---|
scramdb_dml_row_by_row_statements_total | counter | UPDATE, DELETE and MERGE statements run one row at a time because a routine they call can see their own writes. |
scramdb_insert_serial_statements_total | counter | INSERT statements written one row at a time because a value they compute reads the rows they already wrote. |
scramdb_insert_serial_rows_total | counter | Rows written by INSERT statements that wrote one row at a time. |
scramdb_copy_serial_statements_total | counter | COPY statements loaded one row at a time because a column default reads the rows already loaded. |
scramdb_copy_serial_rows_total | counter | Rows loaded by COPY statements that loaded one row at a time. |
An UPDATE or DELETE that finds its rows through an index and can show the order it meets them in (a row trigger, a RETURNING list, a row that can be refused) takes them in the order the table lists them (see the order of triggers and keys). A statement whose rows the index gave in that order, or that nothing can show an order to, counts nothing here.
| Metric | Type | Meaning |
|---|---|---|
scramdb_dml_heap_ordered_statements_total | counter | UPDATE and DELETE statements whose rows were sorted into the table's order. |
scramdb_dml_key_ordered_statements_total | counter | UPDATE and DELETE statements whose rows were sorted into the key order of a table other nodes hold too. |
scramdb_cluster_local_key_ordered_reads_total | counter | Reads of a table this node holds whole that were run in the key order the other nodes answer in. |
On a cluster, a node that holds every bucket of a table together with other nodes meets the rows of such a statement in the key order, which is what a node that holds only part of the table answers in, so every node of the cluster gives one answer to one statement. The two counters above count the statements and the reads that took that order.
A write driven by a join reads its target rows again by key after the join when the join read only the columns it needed. These metrics count that completion.
| Metric | Type | Meaning |
|---|---|---|
scramdb_join_write_completions_total | counter | Join write batches completed with their target rows after a column-limited read. |
scramdb_join_write_completed_rows_total | counter | Pair rows of join writes completed with a target row. |
scramdb_join_write_completed_own_rows_total | counter | Pair rows of join writes whose target row is the transaction's own write. |
scramdb_join_write_fetch_rounds_total | counter | Rounds a cluster join write made to fetch its target rows from the nodes that hold them. |
scramdb_join_write_completion_misses_total | counter | Join write batches refused with a retryable error because a target row could not be read. |
scramdb_join_write_completion_seconds | histogram | Time one join write batch took to complete its target rows. |
scramdb_join_write_pairs_read_again_total | counter | READ COMMITTED join writes that read their pairs again because a target row could not be read. |
A rate that climbs with a bulk load means a column default or a select item calls a VOLATILE function that reads its own table: make the function STABLE if it need not see the rows written so far, and the load returns to the batch path.
Rows COPY left out (always present)
| Metric | Type | Meaning |
|---|---|---|
scramdb_copy_ignored_rows_total | counter | Rows COPY ... FROM left out because a field did not convert under ON_ERROR ignore. |
A rate that climbs means loads carry malformed rows the statements chose to skip: read the NOTICE lines of LOG_VERBOSITY verbose to find them, or drop ON_ERROR ignore and let the load fail at the first one.
Transition tables (always present)
The rows an AFTER trigger, statement-level or row-level, reads through REFERENCING OLD TABLE or NEW TABLE stay in memory while the statement's memory budget grants them and go to disk past it.
| Metric | Type | Meaning |
|---|---|---|
scramdb_transition_spilled_relations_total | counter | Transition tables of triggers that went past the memory budget to disk. |
scramdb_transition_spilled_bytes_total | counter | Bytes of transition table rows written to disk. |
A rate that climbs means statements change more rows than their memory budget holds: the trigger still reads every row, the statement is slower by the file's write and read. Raise storage.execution_memory_bytes or change fewer rows per statement.
Prepared statements (always present)
A prepared statement is parsed once, the first time a Bind needs it; every later execution hands the engine that parse with the Bind's values in place, so a statement executed many times is never parsed again.
| Metric | Type | Meaning |
|---|---|---|
scramdb_prepared_template_parses_total | counter | Prepared statements parsed, once each, the first time a Bind needed the parse. |
scramdb_prepared_handoffs_total | counter | Prepared statement executions handed to the engine already parsed. |
scramdb_prepared_handoffs_unused_total | counter | Prepared statement executions whose parse the engine did not use. |
scramdb_prepared_reparses_total | counter | Times the engine parsed an already parsed prepared statement execution again. |
scramdb_prepared_declared_resolutions_total | counter | Prepared statements' declared user types looked up in the catalog, at Parse and again after a type changed. |
scramdb_prepared_descriptions_total | counter | Prepared statements' result descriptions derived by planning, once per statement and catalog state; every other Describe reuses it. |
scramdb_prepared_descriptions_on_shard_total | counter | Describes of prepared statements answered from their kept description without a query worker. |
On a workload of prepared statements, handoffs_total grows with the executions while template_parses_total grows only with the distinct statements; handoffs_unused_total and reparses_total staying near zero confirm the parse is reused.
SQL parses (always present)
The server reads a stored definition (a view's body, a column default, a CHECK, a generated column, a trigger's WHEN and its function's body, a row policy, an index expression, a function body) once, the first time a statement needs it after it is created or after a restart, and keeps the parsed form for every later statement, row and call in every session.
| Metric | Type | Meaning |
|---|---|---|
scramdb_sql_parses_total{source} | counter | SQL texts the parser read, labeled source: statement (the statements clients send and the text the server runs for them) and definition (stored definitions, read once each). |
On a steady workload {source="definition"} stays flat: it grows only when a definition is created, replaced or renamed, after a restart, or when the catalog's memory budget had to drop a parsed definition no statement had read for a while.
Plan cache (always present)
Plans, their compiled code and plan markers are cached per statement and bounded in bytes by [optimizer] plan_cache_memory.
| Metric | Type | Meaning |
|---|---|---|
scramdb_plan_cache_hits_total | counter | Statements that reused a cached plan. |
scramdb_plan_cache_misses_total | counter | Statements that found no current cached plan and were planned. |
scramdb_plan_cache_evictions_total | counter | Cached plans, plan markers, plan choices, point-op templates, lowered expressions and statement shapes dropped to stay within the plan cache memory budget. |
scramdb_plan_cache_bytes | gauge | Memory held by cached plans, their compiled modules, plan markers, plan choices, point-op templates, lowered expressions and statement shapes. |
scramdb_plan_cache_budget_bytes | gauge | Memory the plan cache may hold. |
scramdb_plan_cache_entries | gauge | Plans, plan markers, plan choices, point-op templates, lowered expressions and statement shapes the plan cache holds. |
scramdb_plans_total{kind} | counter | Statements planned, labeled kind: generic (a statement with parameters planned once with a slot for each, Plan Cache), custom (a statement with parameters planned with its values written in) and plain (a statement without parameters). On a steady workload of prepared statements generic and custom stay flat. |
scramdb_write_lowerings_total | counter | Times an UPDATE or DELETE had its condition, row security fence and SET list lowered from its plan. A plan lowers them once, for the first execution that meets its settings, so on a steady workload this stays flat. |
scramdb_dml_expression_lowerings_total | counter | Times an expression of a write (a VALUES cell, a CHECK, a RETURNING item) was lowered. |
scramdb_dml_expression_lowering_hits_total | counter | Times an expression of a write was taken from the lowered expressions the plan cache keeps. On a steady workload of one statement shape this rises while the lowerings stay flat. |
scramdb_point_inserts_total | counter | One-row INSERTs inside a transaction answered by the point lane, without planning. A VALUES row may omit nullable columns that have no default and may use CURRENT_TIMESTAMP, now(), LOCALTIMESTAMP or CURRENT_DATE. |
Notifications (LISTEN and NOTIFY)
LISTEN and NOTIFY report process-wide, always emitted:
| Metric | Type | Meaning |
|---|---|---|
scramdb_notify_notifications_committed_total | counter | Notifications sent by committed transactions. |
scramdb_notify_notifications_delivered_total | counter | Notifications written to listening clients. |
scramdb_notify_batches_replicated_total | counter | Committed transactions whose notifications were sent to the other nodes. |
scramdb_notify_replication_failures_total | counter | Committed transactions whose notifications could not be sent to the other nodes. |
scramdb_notify_queue_full_total | counter | Notifications refused because the notification queue was full. |
scramdb_notify_listening_sessions | gauge | Sessions listening to at least one channel. |
scramdb_notify_held_bytes | gauge | Bytes of notifications queued or waiting to be sent to a listener. |
scramdb_notify_batches_unheard_total | counter | Committed transactions whose notifications no session was listening to. |
scramdb_notify_batches_unrouted_total | counter | Notification deliveries lost because the session had ended or the database was not known. |
scramdb_notify_notifications_unlistened_total | counter | Notifications skipped because the session stopped listening to their channel first. |
scramdb_notify_notifications_abandoned_total | counter | Notifications queued for a session that ended before it received them. |
scramdb_notify_listeners_rewoken_total | counter | Idle sessions woken by a periodic check. |
A notification's bytes are held from the execution memory budget until the last listener has received it, so held_bytes that keeps growing means a session listens and never lets its client read (a client blocked inside a long transaction receives nothing until it ends); queue_full_total counts the NOTIFY statements refused with 54000 when that budget had no room. On a cluster, replication_failures_total counts committed transactions whose notifications were not sent to the other nodes: the committing client gets a warning (08006) with its commit.
Every notification that does not reach a listening client is counted where it is lost. batches_unheard_total and notifications_unlistened_total count what PostgreSQL does not send either (nobody listened; the session stopped listening before the send). batches_unrouted_total and notifications_abandoned_total count notifications that had no listener left to receive them (a listener that ended between the commit and the delivery, a replicated batch for a database this node does not know) and each writes one warning per half minute. listeners_rewoken_total counts sessions woken by a periodic check; it stays at 0 in normal operation, and if it rises, contact support with the server log.
WAL archive metrics
| Metric | Type | Meaning |
|---|---|---|
scramdb_wal_archive_enabled | gauge | 1 if WAL archival is enabled, 0 otherwise. Always emitted. |
scramdb_wal_archive_errors_total | counter | Archive attempts that failed. Always emitted. |
scramdb_wal_archive_segments_archived_total | counter | Segments successfully shipped to the destination. Always emitted. |
scramdb_wal_archive_segments_behind | gauge | Segments not yet archived. Emitted only once a real measurement exists. |
scramdb_wal_archive_bytes_behind | gauge | Bytes not yet archived. Emitted only once a real measurement exists. |
scramdb_wal_archive_last_archived_lsn | gauge | LSN of the last successfully archived segment. Emitted only once a real measurement exists. |
scramdb_wal_archive_segments_pruned_total | counter | WAL segments removed past the archive retention window ([storage.wal.archive] retention). Always emitted. |
scramdb_wal_archive_prune_failures_total | counter | Archive retention passes that failed and will retry. Always emitted; zero on a healthy archive. |
scramdb_wal_archive_raw_bytes_total | counter | Bytes of closed WAL segments handed to the archive, before packing. Always emitted. |
scramdb_wal_archive_stored_bytes_total | counter | Bytes the archive stored for those segments, after packing; the ratio of the two counters is what [storage.wal.archive] compress saves. Always emitted. |
scramdb_wal_archive_pruned_bytes_total | counter | Bytes of archived segments removed by the retention pass, by age or by the size bound. Always emitted. |
scramdb_wal_archive_over_bound_total | counter | Times the archive stayed past its size bound because a base backup or a branch still needs its segments, counted once for each stretch it stays past. Always emitted; zero when nothing holds the archive past its bound. |
scramdb_wal_archive_over_bound_bytes | gauge | Bytes the archive holds past its size bound that the retention pass may not remove. Emitted once a retention pass has compared the archive with its bound. |
The periodic checkpoint scheduler ([storage.wal.checkpoint], on by default) reports on the same endpoint, always emitted:
| Metric | Type | Meaning |
|---|---|---|
scramdb_wal_checkpoint_interval_total | counter | Checkpoints the scheduler ran because its interval elapsed. |
scramdb_wal_checkpoint_wal_bytes_total | counter | Checkpoints the scheduler ran because the WAL grew past its threshold. |
scramdb_wal_checkpoint_behind_total | counter | Checkpoints run at full disk priority because the WAL grew past twice its threshold. |
scramdb_wal_checkpoint_failures_total | counter | Checkpoints the scheduler started that failed. |
scramdb_wal_checkpoint_threshold_bytes | gauge | WAL bytes written since the last checkpoint that start the next one. |
scramdb_wal_checkpoint_seconds | histogram | Time one scheduled checkpoint took. |
scramdb_wal_last_checkpoint_unix_ms | gauge | Unix time in milliseconds of the last checkpoint that completed, 0 before the first. |
The cold tier (Storage Tiering) reports on the same endpoint, always emitted; every failure counter stays at 0 on a healthy tier:
| Metric | Type | Meaning |
|---|---|---|
scramdb_cold_segments_demoted_total | counter | Sealed segments copied to cold storage. |
scramdb_cold_bytes_demoted_total | counter | Bytes of sealed segments copied to cold storage. |
scramdb_cold_segments_evicted_total | counter | Segments whose local file was removed because cold storage holds them. |
scramdb_cold_bytes_evicted_total | counter | Bytes of local disk freed by segments cold storage holds. |
scramdb_cold_segments_read_back_total | counter | Segments read back from cold storage to local disk. |
scramdb_cold_bytes_read_back_total | counter | Bytes read back from cold storage to local disk. |
scramdb_cold_demotion_failures_total | counter | Segment copies to cold storage that failed and will be tried again. |
scramdb_cold_read_back_failures_total | counter | Segment reads from cold storage that failed after every attempt. |
scramdb_cold_read_back_retries_total | counter | Segment reads from cold storage attempted again after a failure. |
scramdb_cold_read_back_mismatches_total | counter | Cold storage objects that did not hold the bytes their segment's marker names. |
scramdb_cold_transfer_timeouts_total | counter | Transfers to or from cold storage that ran past their deadline. |
Operators past their memory (work_mem, see Memory) report their spills, always emitted:
| Metric | Type | Meaning |
|---|---|---|
scramdb_sort_spill_runs_total | counter | Sorted runs written to disk by sorts past their memory. |
scramdb_sort_spill_bytes_written_total | counter | Bytes of sorted runs written to disk. |
scramdb_join_build_spill_partitions_total | counter | Hash join build partitions written to disk to stay within their memory. |
The compiled-function cache of [udf] (aot_cache_dir, aot_cache_max_bytes) reports on the same endpoint:
| Metric | Type | Meaning |
|---|---|---|
scramdb_udf_compile_cache_bytes | gauge | Bytes of compiled function modules held in the on-disk compile cache. |
scramdb_udf_compile_cache_limit_bytes | gauge | The most bytes the compile cache holds (aot_cache_max_bytes). |
scramdb_udf_compile_cache_hits_total | counter | Function modules loaded from the compile cache. |
scramdb_udf_compile_cache_misses_total | counter | Function modules compiled because the cache did not hold them. |
scramdb_udf_compile_cache_evictions_total | counter | Modules removed to keep the cache within its bound. |
scramdb_udf_compile_cache_write_failures_total | counter | Compiled modules that could not be saved to the cache. |
scramdb_udf_catalog_saves_total | counter | Saves of the installed modules and functions, one per INSTALL MODULE, DROP MODULE, CREATE FUNCTION and DROP FUNCTION of a module-backed language. |
scramdb_udf_catalog_save_failures_total | counter | Module and function changes undone because they could not be saved (the statement failed with 58030). |
scramdb_udf_catalog_modules_loaded_total | counter | Installed modules loaded at startup. |
scramdb_udf_catalog_functions_loaded_total | counter | Functions loaded at startup. |
scramdb_udf_catalog_load_failures_total | counter | Saved modules and functions that could not be loaded at startup; the server log names each. |
scramdb_udf_catalog_module_files_removed_total | counter | Saved module files removed because no module uses them any more. |
scramdb_udf_catalog_changes_refused_total | counter | Module and function changes refused because the saved modules and functions could not be read at startup; they stay as saved until the file is repaired or moved aside and the server restarted. |
scramdb_udf_guest_abandoned_total | counter | Python and Ruby function calls left to finish alone after their statement was cancelled or timed out. |
scramdb_udf_guest_abandoned_running | gauge | Abandoned Python and Ruby function calls still running; a call that would exceed the bound waits for one to end. |
scramdb_udf_guest_slot_waits_total | counter | Function calls that waited for an abandoned call to end before starting. |
scramdb_udf_guest_faults_total{language} | counter | Function calls whose own code failed (an exception, a trap or a breached limit), by runtime: javascript (TypeScript counts here too), python or ruby. A rate means a function to find from the statement's error and the server log. A call that was cancelled or refused for load or for memory is not counted. |
scramdb_udf_epoch_ticks_total | counter | Ticks of the function engine's deadline clock, which ticks only while a function runs. |
scramdb_udf_epoch_ticker_wakes_total | counter | Times the function engine's deadline clock woke for a call after being idle. |
The WAL ring reports on the same endpoint, always emitted. The bound is what the memory budget reserves for it: the ring, 64 times [storage.wal] flush_threshold_bytes (sized for the machine unless pinned), and its completion slots (a sixteenth of the ring). append_waits_total rising means the WAL device is not keeping up with the write rate. A stall is the log making no progress for far longer than a flush usually takes (at least a second at the default epoch): it is reported, never treated as a failure, and every commit waiting for it completes when the device answers. An interrupted log write leaves padding in its place, which every reader skips:
| Metric | Type | Meaning |
|---|---|---|
scramdb_wal_buffer_bytes | gauge | Bytes of log records waiting in the WAL ring at its last flush. |
scramdb_wal_buffer_bound_bytes | gauge | Bytes the WAL ring and its completion slots hold, as the memory budget reserves them. |
scramdb_wal_append_waits_total | counter | WAL appends that waited for a flush to make room in the ring. |
scramdb_wal_flush_threshold_bytes | gauge | Bytes of waiting log records that start a WAL flush early, as configured or sized. |
scramdb_wal_flush_stalls_total | counter | Times the WAL made no progress for far longer than a flush usually takes. |
scramdb_wal_flush_stalled | gauge | WAL writers whose log has made no progress for far longer than usual right now. |
scramdb_wal_padding_records_total | counter | Log records whose append was interrupted and whose place was filled with padding. |
scramdb_wal_schema_records_total | counter | Records of a table's columns written to the log. A row record names its table's columns by one of them, so the count follows new and altered tables, not rows. |
scramdb_wal_flush_syncs_total | counter | Data syncs of the WAL that flush rounds made, one per round that wrote log records. |
scramdb_wal_flusher_wakes_total | counter | Times the idle WAL flush thread was woken by a log append or a commit. A commit that syncs its own records wakes no one, so on a transactional workload this stays far below the commit count. |
scramdb_wal_rotation_syncs_total | counter | Data and directory syncs of the WAL made when a file filled and the next one began. |
scramdb_storage_syncs_total | counter | Data syncs the storage layer made to disk: the log, segment files and replaced files. A commit costs one, shared with the commits that land beside it. |
scramdb_wal_flush_sync_seconds | histogram | Time one flush round's WAL data sync took. |
scramdb_wal_disk_full_episodes_total | counter | Times the WAL stopped writing because the disk had no room. |
scramdb_wal_disk_full_resumes_total | counter | Times the WAL wrote again after it had stopped for lack of room. |
scramdb_wal_disk_full | gauge | WAL writers refusing writes right now because the disk has no room. |
scramdb_wal_failed | gauge | WAL writers refusing writes now because their log failed a write or a sync. 0 on a healthy server. While it is above 0, writes are refused (58030, see Error codes) and /health answers 503; alert on it. |
scramdb_wal_healing | gauge | WAL writers trying right now to write the records a failed sync left into a new file. |
scramdb_wal_heal_attempts_total | counter | Times the WAL tried to write the records a failed sync left into a new file. |
scramdb_wal_heals_total | counter | Times the WAL healed after a failed sync by writing its records into a new file. |
Every sync of the WAL is counted: the sum of flush_syncs_total and rotation_syncs_total over the transactions committed in the same window is the syncs per committed transaction, and flush_sync_seconds is how long the device takes to make one flush round durable.
A failed sync does not end the log. The records the failed sync had not made durable are written again into a new file at once, with one warning in the server log, and the commit that waited for them answers as usual (scramdb_wal_heal_attempts_total and scramdb_wal_heals_total each count one). When that cannot be done either, scramdb_wal_failed reads 1, the server logs one error line and the log tries again on a pause that starts at 100 milliseconds and doubles up to 10 seconds, with scramdb_wal_healing at 1 during each try. When a try succeeds the log takes writes again and scramdb_wal_failed returns to 0. A rising scramdb_wal_heals_total is a storage device that failed a sync and recovered: look at the device even when no statement was refused. A log that cannot be repaired reads scramdb_wal_failed at 1 until the server is restarted and logs one error line with the cause; the commits that waited answer 08007 and every later write 58030, so no commit hangs behind it. A log that is only slow is scramdb_wal_flush_stalled above 0: nothing has failed and commits wait for the device.
Catalog file writes (each table's catalog file and deleted-row record, the table list, sequences, databases and the commit timestamp mark) report on the same endpoint, always emitted. Concurrent changes to one file share one write, so coalesced_total rising while writes_total stays flat under heavy DDL or checkpointing is expected:
| Metric | Type | Meaning |
|---|---|---|
scramdb_catalog_writes_total | counter | Catalog files written. |
scramdb_catalog_write_failures_total | counter | Catalog file writes that failed. Zero on a healthy disk; each failure is returned to the statements that asked for the write. |
scramdb_catalog_coalesced_total | counter | Catalog changes made durable by a write another caller did. |
scramdb_catalog_write_seconds | histogram | Time one catalog file write took (write, sync, rename, directory sync). |
scramdb_catalog_column_records_dropped_total | counter | Table column records a checkpoint dropped from the table entries. |
scramdb_catalog_column_records_kept | gauge | Table column records the table entries held after the last checkpoint. |
The segments_behind, bytes_behind and last_archived_lsn metrics being absent from a scrape means "not measured yet," not zero. This is deliberate: the endpoint does not fabricate a 0 for a value it has not actually computed. Treat an absent metric name the same way, as unmeasured, everywhere on this endpoint, not just for WAL archive.
Disk bytes
Every byte the store keeps on disk is booked by class, by the same book the scram_disk view reads, so a scrape, a query and the files agree. The classes are table_segments (table segment files), wal, archive (archived log segments), base_backups, branch_staging, vector_indexes, spill, caches, raft_log (the replicated log of a cluster node), logs (the file the server's output goes to: the background server's log or a redirect; absent when the output is a terminal or a pipe) and metadata. A census walks the data directory once a minute ([storage.disk] census_interval) and corrects a class whose books differ from its files, counting each correction; a figure that has not been measured yet is absent from the scrape, never 0.
| Metric | Type | Meaning |
|---|---|---|
scramdb_disk_bytes{class} | gauge | Bytes on disk by class. |
scramdb_disk_limit_bytes{class} | gauge | Bytes a class may hold before it is reported over its bound. Absent for a class with no bound derived yet. |
scramdb_disk_over_bound{class} | gauge | 1 while a class holds more than its bound, otherwise 0. |
scramdb_disk_over_bound_episodes_total{class} | counter | Times a class went past its bound. |
scramdb_disk_corrections_total{class} | counter | Times the census found a class's books disagreeing with its files and corrected them. If it rises, contact support with the server log. |
scramdb_disk_correction_bytes_total{class} | counter | Bytes the census moved when it corrected a class's books. |
scramdb_disk_database_bytes{database_oid} | gauge | Bytes of table files each database holds. The archive is shared by every database and is counted in its own class, not here. |
scramdb_disk_volume_available_bytes | gauge | Bytes free on the volume holding the data directory at the last census. |
scramdb_disk_volume_pressure_episodes_total | counter | Times the volume could not hold every class at its bound. |
scramdb_disk_census_age_seconds | gauge | Seconds since the census last finished. |
scramdb_disk_census_last_micros | gauge | Microseconds the last census took. |
scramdb_disk_census_runs_total | counter | Censuses completed. |
scramdb_disk_census_errors_total | counter | Censuses that could not read part of the data directory. |
scramdb_disk_census_table_corrections_total | counter | Tables whose books the census corrected to the files they hold. |
scramdb_wal_pins_held | gauge | Pins held now that keep log segments from being reclaimed: a branch at a past time staging its replay. Always emitted. |
scramdb_wal_pin_oldest_age_seconds | gauge | Seconds the oldest pin has been held. Absent while no pin is held. |
scramdb_wal_pins_lapsed_total | counter | Pins released because their holder stopped renewing them within [storage.disk] pin_lease. Zero on a healthy server; a rise means a branch staging stopped without letting go, and the pin it held was released so it could not keep the log from being reclaimed. |
Only the archive is held to its bound by the server, which removes the oldest segments no base backup or branch needs once [storage.wal.archive] max_size (a quarter of the volume unless set) is passed; an archive in an object store is pruned by age alone. The other classes report over_bound and the episode counters; none of them is trimmed for being over. A branch whose pin lapses fails with the reason instead of finishing on segments that may be gone.
A table's bytes and the archive's move in the same step as their files; the log, branch staging, vector indexes and the other classes are read from the files by the census, so they trail a write by up to one census interval. A dropped table or database frees its files at once, and its bytes leave the books in the same step, unless a clone or a snapshot still reads them; those bytes stay on table_segments until that reader is gone.
scram_disk answers the same figures in SQL, one row per class, one per database that holds table files, and one for the free space of the volume:
| Column | Meaning |
|---|---|
scope | class, database or volume. |
name | The class, the database's name, or available for the volume. |
bytes | Bytes held, or free for the volume. NULL until measured. |
limit_bytes | The class's bound. NULL for a database, for the volume and for a class with no bound derived yet. |
over_bound | Whether a class holds more than its bound. NULL for a database and the volume. |
episodes | Times a class went past its bound. NULL for a database and the volume. |
SELECT name, bytes, limit_bytes, over_bound FROM scram_disk WHERE scope = 'class' ORDER BY bytes DESC NULLS LAST;
SELECT name, bytes FROM scram_disk WHERE scope = 'database' ORDER BY bytes DESC;
A superuser or a member of pg_read_all_stats (which pg_monitor includes) sees every row. Any other role sees one database row, the size of the database it is connected to, and no class or volume row, as PostgreSQL gives a database size only to a role that may connect to it and keeps the rest of the node's statistics for pg_read_all_stats.
The archive and base backup sizes are on the node's own disk. A cluster node's view and scrape describe that node's disk; a learner's store keeps its own books and is not on them yet.
Object store and function IO threads
The object store client (the s3://, gs:// and az:// destinations of the WAL archive, base backups, restores, branch staging and the cold tier) makes its requests on the thread that needs them, so it starts no thread of its own. Reads of remote files over http:// and https:// for an external table run on a one or two thread pool that starts with the first such read, so a query worker never waits on the network. The file, clock and network calls user functions make run on a small pool of worker threads of their own, which starts with the first function that needs it. Always emitted: zero until a pool starts.
| Metric | Type | Meaning |
|---|---|---|
scramdb_object_store_io_threads | gauge | Threads serving remote file reads for query workers. |
scramdb_udf_io_workers | gauge | Worker threads serving functions' file, clock and network calls: the network runtime's shards, which also serve client connections. |
scramdb_udf_io_busy_microseconds_total | counter | Time the network runtime's shards spent running work, functions' IO and client connections together. |
Every object store request is counted by backend (local, memory, http, s3, gcs, azure) and operation, with a series emitted once a request of that kind was made:
| Metric | Type | Meaning |
|---|---|---|
scramdb_object_store_requests_total{backend,op,outcome} | counter | Object store requests by backend, operation (put, put_if_absent, get, get_range, head, list, list_dirs, delete, begin, part, complete, abort) and outcome (ok, not_found, already_exists, precondition_failed, denied, throttled, transport, server, invalid, io). |
scramdb_object_store_request_seconds{backend,op} | histogram | Time of object store requests. |
scramdb_object_store_bytes_in_total{backend} | counter | Bytes read from object stores. |
scramdb_object_store_bytes_out_total{backend} | counter | Bytes written to object stores. |
scramdb_object_store_http_requests_total{backend} | counter | HTTP requests sent to object stores, retries included. |
scramdb_object_store_retries_total{backend} | counter | Requests sent again after a failure that answered nothing. |
scramdb_object_store_timeouts_total{backend} | counter | Requests that waited past their bound. |
A busy counter's rate divided by its worker gauge is that pool's utilization.
Every upload and download through the object store client is counted, and so is every HTTP read of a remote file. A request is bounded: 5 s to connect and 30 s of silence. A read that got no answer (a connection that did not open or broke, a name that did not resolve, a timeout, or a 408, 429, 500, 502, 503 or 504) is sent again, up to 10 times within 180 s, backing off from 100 ms to 15 s; an object store request keeps the same bounds. Always emitted: zero until the first request.
| Metric | Type | Meaning |
|---|---|---|
scramdb_object_store_uploads_total / scramdb_object_store_upload_failures_total | counter | Uploads to the object store, and the ones that failed. |
scramdb_object_store_uploads_aborted_total / scramdb_object_store_abort_failures_total | counter | Unfinished multipart uploads aborted after a failure, and the aborts that failed (the object store keeps those parts until its own cleanup). |
scramdb_object_store_uploaded_bytes_total | counter | Bytes uploaded. |
scramdb_object_store_downloads_total / scramdb_object_store_download_failures_total | counter | Downloads from the object store, and the ones that failed. |
scramdb_object_store_downloaded_bytes_total | counter | Bytes downloaded. |
scramdb_object_store_upload_seconds / scramdb_object_store_download_seconds | histogram | Time an upload or a download took. |
scramdb_remote_file_requests_total / scramdb_remote_file_request_failures_total | counter | HTTP requests for remote files, and the ones that failed after their retries. |
scramdb_remote_file_request_retries_total | counter | HTTP requests for remote files sent again. |
scramdb_remote_file_request_timeouts_total | counter | HTTP requests for remote files that got no answer within their bounds. |
scramdb_remote_file_bodies_open | gauge | Remote file answers being read now, each holding one connection. |
Server stalls (always present, all zero on a healthy process)
A sampler reads the statement counters once a second. The server is stalled when statements are in flight and none has finished for longer than a bound that follows the finishing pace the server itself has shown (between 15 and 60 seconds), and either at least two statements are in flight or one waits for a memory grant or a lock. A single long statement that waits on nothing is not a stall. The log carries one warning when a stall begins, naming how many statements wait and the longest-running session, and one line when it ends.
| Metric | Type | Meaning |
|---|---|---|
scramdb_server_stalls_total | counter | Times statements were in flight and none finished for longer than the stall bound. |
scramdb_server_stalled_seconds | gauge | Seconds the server has had statements in flight with none finished, 0 when it is not stalled. |
scramdb_server_stall_bound_milliseconds | gauge | Time with no statement finished after which the server reports a stall. |
scramdb_timer_registrations_queued | gauge | Timer registrations waiting for the timer thread to take them. Near 0 on a healthy server; if it keeps growing, contact support with the server log. |
Error shapes (always present)
| Metric | Type | Meaning |
|---|---|---|
scramdb_error_shapes_total | counter | Statements answered with a recognized error shape, by shape. |
The shape label is one of disk_full, spill_unavailable, prepared_corrupt, prepared_busy, prepared_rollback_only, prepared_unsettled, prepared_commit_pending, prepared_storage, cluster_in_doubt, wire_version_refused, generated_column_write, set_list_misuse, update_subselect_unsupported, branch_statement_unsupported, cursor_misuse, routine_variable_misuse, routine_transaction_termination, routine_type_reference, from_clause_entry, duplicate_table_name, untyped_array, merge_row_twice, clone_not_ready and internal. It counts only errors one classifier recognizes, never every error: a statement that fails any other way is not counted here.
Resilience metrics (always present, all zero on a healthy process)
| Metric | Type | Meaning |
|---|---|---|
scramdb_parallel_join_timeout_total | counter | Parallel join operations that hit their timeout. |
scramdb_pool_watchdog_heals_total | counter | Times the pool watchdog woke a worker thread that seemed stalled. |
scramdb_pool_watchdog_samples_total | counter | Sampling passes the pool watchdog made over its workers. While work is outstanding it makes one about every 125 ms; an idle pool holds it still. |
scramdb_pool_watchdog_heal_budget_exhausted_total | counter | Times the watchdog's heal budget ran out. |
scramdb_pool_watchdog_escalations_total | counter | Times the watchdog escalated beyond a self-heal. |
scramdb_pool_idle_wakes_total | counter | Times an idle worker thread woke from its wait to settle owed cleanup and found no task. A pool with nothing to do holds it still. |
Crashes and fatal signals
A statement that fails on an unexpected condition ends alone, never the server. It answers XX000 with a sentence pointing to the server log (internal error: ...; the server log has details), the memory it had reserved is released, and the connection and the worker go on. The log carries one error line with the statement's shape: the session, a fingerprint, and the start of the statement with every literal replaced by ?, so no value a client sent reaches the log. A statement nested too deep for the stack, whether in the parser, in an expression, in a chain of views, in routine calls or in an engine rewrite, answers 54001 (stack depth limit exceeded) instead; every thread that runs query work has a 64 MiB stack and a deeper descent continues on a new segment, counted as stack_segments on scramdb_memory_charged_bytes, so the depth a statement may reach is the session's max_stack_depth, not the size of a thread's stack.
| Metric | Type | Meaning |
|---|---|---|
scramdb_panics_caught_total{site} | counter | Panics that ended one statement or job instead of the process, by where they were caught: statement, connection, worker, copy_loader, compiled_call, fan_out (a fragment of a distributed statement, including a request served for another node) or background. |
scramdb_stack_depth_refusals_total{site} | counter | Statements ended with a stack depth limit error instead of crashing, by the guard that refused them: parse, expression (an expression, or a chain of views, nested too deep), rewrite, routine or call. |
scramdb_fatal_signals_previous_run | gauge | Fatal signals the previous run of this server ended on, found at this start. |
scramdb_core_dump_limit_bytes | gauge | Largest core dump the process may write, in bytes; 0 when core dumps are off. |
scramdb_panics_caught_total stays at zero in normal operation; if it rises, contact support with the server log. A growing scramdb_stack_depth_refusals_total means clients send nesting the server has to refuse; the site label says which kind.
A fatal signal (SIGSEGV, SIGBUS, SIGILL, SIGFPE, SIGABRT) still ends the process, but leaves a record first. The signal, the faulting address, the thread, the statement shape and a stack go to the log and to fatal-signal.log in [storage] basedir. The next start finds the file, logs one error naming the signal, keeps it as fatal-signal.previous.log and reports it in scramdb_fatal_signals_previous_run, so a restart never hides that the previous run was killed. The release binary is stripped, so the stack lists module and offset; resolve them with addr2line against the unstripped binary of the same build. core_dump_limit_mib in [general] (default 1024, 0 turns core dumps off) bounds the core the kernel may write, so a crash of a large process cannot fill the data volume and starve the restart. See Configuration file.
Query execution and JIT
| Metric | Type | Meaning |
|---|---|---|
scramdb_jit_compile_submits_total | counter | Compilation requests handed to the JIT worker. On a warm, repeating workload this should stop growing; continued growth means the compiled-artifact cache is not serving and is worth raising compiled_cache_memory for. |
scramdb_chunks_jit_t1_total | counter | Chunks served by tier 1 compiled code, which is the fast-to-produce tier or a cached artifact loaded from disk. |
scramdb_chunks_jit_t2_total | counter | Chunks served by tier 2 compiled code, the fully optimized tier. |
scramdb_jit_unavailable | gauge | 1 when the JIT is enabled but this server's LLVM cannot compile for its CPU: statements then run on the bytecode interpreter alone. 0 otherwise. |
scramdb_jit_t1_compile_seconds | histogram | Time to compile one module for a quick start (tier 1). |
scramdb_jit_t2_compile_seconds | histogram | Time to compile one module fully optimized (tier 2). |
scramdb_jit_jetz_load_seconds | histogram | Time to load one compiled module from the on-disk compile cache. |
scramdb_vm_chunk_rows_total | counter | Rows in the batches run by the interpreter. |
scramdb_vm_chunk_nanoseconds_total | counter | Time spent running batches in the interpreter, in nanoseconds. |
scramdb_jit_chunk_rows_total | counter | Rows in the batches run as compiled machine code. |
scramdb_jit_chunk_nanoseconds_total | counter | Time spent running batches as compiled machine code, in nanoseconds. A statement is compiled the first time it runs when its rows times the measured difference per row exceed the measured compile time, and otherwise the second time. |
scramdb_jit_jetbc_load_seconds | histogram | Time to load and compile one cached intermediate module from the on-disk compile cache. |
scramdb_jit_verifier_rejections_total{tier} | counter | Compiled kernels the code verifier rejected, labelled tier (fused, whole_function, expression_region, module). The statement still runs on the interpreter. It stays at zero in normal operation; if it rises, contact support with the server log. |
scramdb_udf_interpreter_warmups_total | counter | Interpreted function runtimes started, by language (python, ruby): each starts once per process, the first time a function of that language runs. |
scramdb_jit_compile_memory_charges_total | counter | Compiles whose working memory was charged to the compiled-code cache budget. |
scramdb_jit_compile_memory_in_flight_bytes | gauge | Working memory held by the compiles running now. It counts against compiled_cache_memory together with the cached code, and cached code is evicted to make room for it. |
scramdb_jit_cache_bytes | gauge | Bytes the compiled-code caches hold now: their bytecode, the compiled code and the running compiles. The compiled code and the compiles are measured from the native heap where the system reports it, so the figure is what the budget is enforced on. Read it against scramdb_jit_cache_budget_bytes. |
scramdb_jit_cache_budget_bytes | gauge | The budget of the compiled-code caches (compiled_cache_memory). |
scramdb_jit_cache_hits_total | counter | Lookups the compiled-code caches answered from a cached entry. |
scramdb_jit_cache_misses_total | counter | Lookups that found no cached entry and installed a new one. |
scramdb_jit_cache_evictions_total | counter | Cached entries the compiled-code caches evicted to stay within their budget. |
scramdb_jit_disk_cache_pruned_files_total | counter | Compiled-code files removed from disk, oldest first, to stay within compiled_cache_disk. |
scramdb_jit_compiles_in_flight | gauge | Compiles running now. |
scramdb_jit_compile_memory_over_budget_total | counter | Compiles that started while the running compiles alone needed more than compiled_cache_memory. A compile is never refused or delayed for memory, so a growing count means the budget is too small for the compile concurrency: raise compiled_cache_memory or lower jit_compile_threads. |
scramdb_jit_compile_threads | gauge | Threads of the query compile pool ([execution] jit_compile_threads, one to eight). They run on the query cores below the query threads' priority, never on the cores a cluster node reserves for its own traffic. |
scramdb_jit_compile_queued | gauge | Query compiles waiting for a compile thread. |
scramdb_jit_compile_queued_bytes | gauge | Bytes held by the query compiles waiting for a compile thread. Bounded by compiled_cache_memory. |
scramdb_jit_compile_refused_total | counter | Query compiles not queued because the compile queue was full. The statement runs on the interpreter, never waits, and is compiled on a later run once the queue has room; a steadily growing count means compiles cannot keep up with new statement shapes: raise jit_compile_threads or compiled_cache_memory. |
scramdb_vm_agg_flushes_total | counter | Interpreter-side grouped aggregate tables flushed as partial results on reaching their memory budget. |
scramdb_preagg_flushes_total | counter | Per-worker aggregate tables flushed as partial results for exceeding their share of the execution budget. |
scramdb_preagg_merges_on_caller_total | counter | Small pre-aggregate merges run on the calling thread. |
scramdb_preagg_merge_empty_lanes_total | counter | Empty pre-aggregate lanes skipped at merge. |
scramdb_preagg_merged_sources_total | counter | Worker partials folded together by pre-aggregate merges. Compare its rate with the pre-aggregate merges: a higher figure means merges fold several partials, which is a parallel merge at work. |
scramdb_agg_resident_bytes | gauge | Live aggregate hash-table bytes: buckets, key slots, tags and string arenas. |
scramdb_join_ordering_cartesian_fallbacks_total | counter | Times the planner could not connect a join graph and fell back to emitting a cross product. A query whose SQL does not ask for a cross join leaves this at zero. |
scramdb_split_distinct_rewrites_total | counter | DISTINCT aggregates the planner split into two aggregation levels. |
scramdb_probe_side_materializations_total | counter | Outer-join probe sides that could not stream and were written to a temporary relation first. |
scramdb_join_input_materializations_total | counter | Join inputs of any position that could not stream and were written to a temporary relation first. Falling over time on a fixed workload is the plan getting better. |
scramdb_stream_topk_rows_total | counter | Rows accepted by the streaming top-K sink serving ORDER BY ... LIMIT. |
scramdb_topk_threshold_publishes_total | counter | Top-K thresholds published into the scan so it can skip pages that cannot beat the current best rows. |
scramdb_late_fetch_rows_total | counter | Rows served by late materialization: a narrow scan that fetches the remaining columns only for rows that survived. |
scramdb_stream_queue_bytes | gauge | Bytes of worker results queued ahead of the client, bounded by backpressure. |
scramdb_stream_gate_forced_admits_total | counter | Results pushed past their statement's budget because the queue was empty and there was nothing left to drain. |
Startup
| Metric | Type | Meaning |
|---|---|---|
scramdb_startup_subsystems_pending | gauge | Subsystems a client can reach (vector indexes, the memory watchdog, the function engine) that are not ready yet. The PostgreSQL port accepts no client until this is zero. |
scramdb_startup_listener_wait_micros | gauge | Microseconds the client listener waited for startup to finish before accepting clients. Zero on an ordinary boot, where every subsystem is ready before the port is opened. |
scramdb_recovery_redo_divergence_quarantines_total | counter | Tables quarantined during boot's log replay because their data could not be verified. Quarantining one table never stops any other table from booting; a nonzero count here means one table needs a look before its data is trusted again. |
scramdb_recovery_redo_divergence_quarantines_persisted_total | counter | The same divergence, found by a bounded replay (a restore or a branch at a past time) rather than an ordinary boot: the table is quarantined and the quarantine is written to disk, so every later reopen of the restored or branched directory still finds it quarantined, not just the process that ran the replay. A nonzero count here means a restore or a branch printed at least one quarantined table in its own output. |
scramdb_boot_tables_quarantined | gauge | Tables the last boot left quarantined. It is the boot's count and does not fall when a table is dropped or recreated; scram_quarantined_tables is the live list. Alert on any. |
scramdb_boot_empty_segment_files_recreated_total | counter | Empty active data files recreated at boot because a power cut took back their directory entry. Expected after a power loss, nothing was lost. |
scramdb_boot_empty_segment_file_failures_total | counter | Empty active data files boot could not recreate; the table's first write then answers with the real error. Alert on any. |
scramdb_boot_lost_segment_files_rebuilt_total | counter | Data files missing at boot that the log's page images rebuilt. Nonzero after a power loss or a lost file: the table was served, no row was lost. |
scramdb_boot_quarantines_lifted_by_log_total | counter | Tables held aside at boot for a short or missing data file and served again once the log named the segment. |
scramdb_boot_orphan_segment_files | gauge | Data files the last boot found on disk that no table lists and no clone shares. The boot only counts them and removes nothing; a nonzero value after a clean run is worth a look. |
scramdb_boot_orphan_segment_bytes | gauge | Bytes of those data files. |
scramdb_boot_orphan_table_dirs | gauge | Table directories the last boot found that no table uses and no clone lists from. |
scramdb_boot_orphan_scan_short | gauge | 1 when the last boot could not read a directory while counting unlisted files, so the three orphan figures above may be short: read them as a floor, never as zero. |
scramdb_boot_leftover_files_removed_total | counter | Temporary files a crash left in the directories of the tables the catalog lists, which the boot removed. Expected after a kill or a power cut. |
scramdb_table_repairs_started_total | counter | Table repairs from a peer that began on this member (a cluster member that lost a table's files refills the table from a healthy member). A restart in the middle of a repair counts a second start. A learner's copy of the table is repaired the same way. |
scramdb_table_repairs_completed_total | counter | Table repairs from a peer that finished and left the table whole. |
scramdb_table_repairs_failed_total | counter | Table repairs from a peer that ended without finishing and were left for the next try. Alert on a rising rate: the member serves none of the group's reads until the copy lands. |
scramdb_table_repairs_in_flight | gauge | Tables on this member that wait for their rows from a peer. Above zero for long means no peer answers the copy. |
scramdb_table_repairs_blocked | gauge | Shard groups on this member whose quarantined tables cannot be repaired from a peer yet: no other member holds the group, a sibling group of the same table has none, or the group's own repair record cannot be read. The table stays quarantined and a damaged record stops its own group only. Alert on any. |
scramdb_quarantined_tables_rebuilt_total | counter | Quarantined tables emptied so a peer could refill them. |
No statement fails because a subsystem is still being installed: a client whose connection reaches the listener before startup has finished waits and is served once it has.
A table quarantined at boot - for this reason or for a segment whose file plainly diverges from its catalog entry and cannot be rebuilt from the log - refuses every access (scan, insert, delete, VACUUM, ANALYZE) with an error naming the table and the segment, until it is restored from a backup or dropped and recreated (a cluster member whose peers hold the table refills it from them by itself, see scramdb_table_repairs_in_flight). Boot says so once, on one line, naming each such table with its database; querying the scram_quarantined_tables system view (columns database, table_name, segment_id, reason) lists every table currently quarantined in every database, and why, to a superuser or a member of pg_read_all_stats; any other role sees the tables of its own database only.
Scans and storage
| Metric | Type | Meaning |
|---|---|---|
scramdb_fsync_skipped_total | counter | Flushes to disk skipped because fsync is off. Always emitted; zero on a server that flushes. |
scramdb_async_commits_total | counter | Commits answered before their log flush because their session set synchronous_commit to off. Always emitted; zero when no session does. |
scramdb_index_delete_lock_acquisitions_total | counter | Batches of entries removed from ordered and hashed indexes. Index reads never wait on these removals. |
scramdb_index_delete_entries_total | counter | Entries asked to be removed from ordered and hashed indexes. |
scramdb_index_read_gate_raised_total | counter | Index key reads that held the index's writers back to finish: a read of a key that writers keep changing holds back changes to that index until it has read the key once. Rare outside heavy writes on one key. |
scramdb_index_read_gate_writer_waits_total | counter | Index changes that waited for such a key read to finish. Each wait lasts one read of the key. |
scramdb_index_rebuilt_at_open_total | counter | Persisted indexes rebuilt from their table's rows at open, never from stored key bytes. An ordinary index (ART, B-tree, hash, GIN) keeps no on-disk byte format: every server startup and every CoW branch fork rebuilds it from the current rows, so a key-encoding change can never leave a stale entry behind. |
scramdb_index_rebuild_failed_at_open_total | counter | Indexes whose rebuild at open failed; each is served scan-only until REINDEX repairs it. |
scramdb_index_reindexed_total | counter | Index rebuilds finished by REINDEX on this node: one per ordinary index and one per part of a vector index. REINDEX SCHEMA and REINDEX DATABASE add one per index of every table they name, REINDEX SYSTEM none. On a cluster REINDEX of a distributed table's index, table, schema or database runs on every node, so each node's own count moves. |
scramdb_table_clustered_total | counter | Table rewrites finished by CLUSTER on this node: one per table, one per partition for a partitioned table, one per marked table for CLUSTER with no table. ALTER TABLE ... CLUSTER ON and SET WITHOUT CLUSTER move a mark and count nothing, a refused statement counts nothing, and a rewrite a transaction block later rolls back was finished and is counted. On a cluster, CLUSTER of a distributed table runs on every node, so each node's own count moves. |
scramdb_zone_pages_considered_total | counter | Pages evaluated against zone-map predicates. |
scramdb_zone_pages_pruned_total | counter | Pages skipped by zone-map pruning and never read. Against considered, this is your page-pruning hit rate. |
scramdb_selective_pages_read_total | counter | Pages served by reading only the projected columns rather than the whole page. |
scramdb_selective_bytes_saved_total | counter | Bytes that column-selective reading did not have to fetch. |
scramdb_selective_ranges_read_total | counter | Read requests column-selective reading issued. Divided by selective_pages_read_total it is the number of disk reads per page. |
scramdb_read_pool_active_units | gauge | Units currently holding a read-buffer-pool slot, either waiting for concurrency or actively holding pages. |
scramdb_read_pool_unit_waits_total | counter | Times a unit waited noticeably long for a read-pool concurrency slot. This is a throughput signal, not a safety one: page acquisition itself is never queued. |
scramdb_page_pool_magazine_sheds_total | counter | Page returns under memory pressure that emptied a thread's idle read buffers into the shared pool. It rises while the server is reclaiming idle per-thread buffers in a pressure episode; it stays flat when the threads holding idle buffers made no page traffic of their own. |
scramdb_row_fetches_total | counter | Fetches of rows by their row id, for index probes, row re-reads and re-ranking. |
scramdb_row_fetch_rows_total | counter | Rows returned by fetches of rows by their row id. |
scramdb_row_fetch_pages_total | counter | Pages read by fetches of rows by their row id. |
scramdb_row_fetch_page_bytes_total | counter | Bytes of the pages read by fetches of rows by their row id. |
scramdb_row_fetch_decoded_bytes_total | counter | Bytes decoded from those pages by fetches of rows by their row id. Against page_bytes_total, how much of each page a fetch had to decode. |
scramdb_row_place_arrays_loaded_total | counter | Merged segments whose row places were read from disk. A row keeps its id when a compaction moves it: each merged segment keeps the ids of its rows beside it, and the first read by id of one this server did not write since it started reads them once (memory family row_places). |
scramdb_row_id_reads_relocated_total | counter | Reads by row id that ran again because a compaction swapped the table's segments while they read. Each runs again at once; one whose table keeps changing under it for longer than [resilience] deadlock_timeout (configuration file) ends with 40001, which the client retries. |
scramdb_values_out_of_line_written_total | counter | Values too long for their page written out of line, in a chain of pages of their own. |
scramdb_value_chain_pages_written_total | counter | Pages written to hold values stored out of line. |
scramdb_values_out_of_line_bytes_written_total | counter | Bytes of the values written out of line. |
scramdb_value_chain_bytes_stored_total | counter | Bytes stored for values written out of line, after compression. Against bytes_written_total, the compression those values got. |
scramdb_values_out_of_line_read_total | counter | Values stored out of line read back. |
scramdb_value_chain_pages_read_total | counter | Pages read to read values stored out of line. |
scramdb_values_out_of_line_bytes_read_total | counter | Bytes of the values stored out of line read back. |
Rows fetched by address are counted by each query worker on its own, so the counters cost nothing measurable however many workers fetch. A text, binary, vector or array value longer than fits its page is stored out of line, in a chain of pages of its row's segment, and read back only by the rows that need it; the out-of-line counters are zero on a workload with no such values.
Segment files and checkpoints (always present)
How data files are referenced, flushed and removed, and how a CHECKPOINT walks the tables. All zero or unchanged on a process that has not written, cloned or checkpointed.
| Metric | Type | Meaning |
|---|---|---|
scramdb_segment_files_referenced | gauge | Data files with at least one reference, from a table, a clone or a running scan. A count of files; the bytes are scramdb_segment_bytes_referenced. |
scramdb_segment_files_shared | gauge | Data files referenced more than once, by a clone or a running scan. |
scramdb_segment_bytes_referenced | gauge | Bytes, in whole pages, of the data files that have been filled and at least one reference holds, each file counted once. The file a table is still writing counts as a file and adds no bytes until it is full. Chart it beside the data directory size. |
scramdb_segment_bytes_shared | gauge | Bytes of those files that more than one table lists: the disk a clone shares with its parent instead of copying. |
scramdb_segment_files_freed_total | counter | Data files removed because their last reference was released. |
scramdb_segment_bytes_freed_total | counter | Bytes of the data files removed because their last reference was released. Read it beside the data directory size. |
scramdb_segment_bytes_kept_total | counter | Bytes of data files a drop or truncate left in place because a clone or a running scan still uses them. Expected with clones. |
scramdb_segment_deletions_deferred_total | counter | Releases that kept the file because another reference remains. A clone or a scan holding a parent's file shows here. |
scramdb_segment_double_releases_total | counter | Unexpected releases of data file references. The file is kept. If it rises, contact support with the server log. |
scramdb_segment_untracked_releases_total | counter | Unexpected releases of data file references. The file is kept. If it rises, contact support with the server log. |
scramdb_segment_unreferenced_listings_total | counter | Reads and clones that were stopped because the server could not account for a file of the table. Restart the server; if it happens again, contact support with the server log. |
scramdb_segment_releases_deferred_total | counter | Segment releases that waited for the statements holding their table's rows to finish. |
scramdb_segment_swaps_superseded_total | counter | Compactions whose result a drop, truncate or rename took over after a failed save. The one that took over releases the replaced files at once (scramdb_segment_superseded_released_total); if its own write failed they stay on disk and the next boot counts them. |
scramdb_segment_superseded_released_total | counter | Data file references released for a compaction that a drop, truncate or rename took over. Against scramdb_segment_swaps_superseded_total it should keep pace; a gap is a takeover whose own write failed. |
scramdb_segment_file_syncs_total | counter | Data files flushed to disk by a checkpoint. Zero with fsync off. |
scramdb_segment_dir_syncs_total | counter | Directories flushed to disk so that newly created data files survive a power cut and removed ones stay removed. Zero with fsync off. |
scramdb_segment_creations_covered_total | counter | Data file creations made durable by a directory flush. Against the directory flushes it shows the batching: many creations under one flush is the design. |
scramdb_segment_removals_covered_total | counter | Data file removals made durable by a directory flush, so a power cut never brings a removed file back. Against the directory flushes it shows the batching, as the creations do. |
scramdb_segment_flush_last_files | gauge | Data files flushed by the last checkpoint that flushed any. |
scramdb_segment_flush_last_dirs | gauge | Directories flushed by the last flush group that flushed any. |
scramdb_checkpoint_table_passes_repeated_total | counter | Checkpoint passes over the tables that ran again because DDL ran during them. Expected under DDL load. |
scramdb_checkpoint_table_passes_closed_total | counter | Checkpoint passes that held DDL off for their own length because DDL kept overlapping them. A steady rate means DDL keeps meeting checkpoints. |
scramdb_checkpoint_table_pass_closed_seconds | histogram | Time each stretch of a checkpoint's closed pass held DDL off. The pass walks one table at a time and lets DDL run between stretches of about 100 ms, so the tail should stay near that plus one table's flush; a long tail is one slow table. |
scramdb_checkpoint_dropped_index_flushes_skipped_total | counter | Vector index flushes skipped because the index was being dropped. Expected under DROP load. |
scramdb_quarantined_tables_cloned_total | counter | Quarantined tables a database clone, or a branch at a past time, carried into the new database as quarantined. Alert on any: the table is unreadable in both databases until dropped or recreated. |
scramdb_table_drop_catalog_failures_total | counter | Table drops whose catalog entry could not be removed, left for the next start to finish. Alert on any: the table is still listed on disk. |
Branches and clones (always present)
A CREATE DATABASE ... CLONE on a single node or on any node of a cluster; see Branching. All zero on a process that never clones.
| Metric | Type | Meaning |
|---|---|---|
scramdb_branch_clone_gate_waits_total | counter | DDL statements that waited for a database clone or a checkpoint to finish walking the tables. A rise that follows branch creation or a checkpoint is DDL held back by it, for the length of that walk. |
scramdb_branch_clone_rollbacks_total | counter | Database clones undone because a step failed. The statement answered an error that names the step, and nothing of the clone is left. |
scramdb_branch_clone_cleanup_failures_total | counter | Database clones whose undo could not finish and left a half-made database to drop. Alert on any: the database refuses connections until DROP DATABASE or the next start removes it. |
scramdb_branch_clone_admission_parked_total | counter | Database clones that waited for memory before they started. A rise is clones arriving faster than the memory budget serves them. |
scramdb_branch_lifecycle_total{op,outcome} | counter | Branch lifecycle operations by kind (create, rename, expire, drop, reap, swap, resume: a drop a restart finished or could not finish) and outcome (ok, error). |
scramdb_branch_expiry_backlog | gauge | Branches whose expiry has passed and that the reaper has not dropped yet. Alert when it stays above zero. |
scramdb_branch_count | gauge | Databases on this node cut from another database (branches and clones). |
scramdb_merge_record_retention | gauge | Merge records each database keeps, newest first ([agent] merge_record_retention). |
scramdb_merge_records_pruned_total | counter | Branch merge and refresh records removed to stay within the merge record retention. |
scramdb_retired_agent_history | gauge | Retired agent records the server keeps, newest first ([agent] retired_agent_history). |
scramdb_retired_agents_pruned_total | counter | Retired agent records removed to stay within the retired agent history. |
scramdb_module_counters_pruned_total | counter | Run counters of modules that were dropped and removed to bound the counters. |
scramdb_branch_bookkeeping_rows_pruned_total | counter | Bookkeeping rows of dropped branches removed. |
scramdb_branch_clone_admission_refused_total | counter | Database clones refused for lack of memory. It stays at zero: a clone that finds memory short waits for it, until its statement_timeout or a cancel ends the wait (SQLSTATE 57014, nothing of the clone left). |
scramdb_branch_clone_rotation_fallbacks_total | counter | Clones that had to start a new segment of a parent table because the rows it buffered did not fit the segment it was writing. Rare: most clones take the segment as it stands. |
scramdb_branch_active_segments_shared_total | counter | Segments a clone took from its parent as they stood, without the parent starting a new one. |
scramdb_branch_tables_inherited_total | counter | Tables of cloned databases made live with no file of their own. Each gets its files at its first write. |
scramdb_branch_files_born_total | counter | Segment files of cloned tables made by their first write. |
scramdb_branch_index_views | gauge | Branch indexes that read through their parent's index now. Falls as branches fold or drop. |
scramdb_branch_index_views_registered_total | counter | Branch indexes registered as a view over their parent's index, on a single node and on each node of a cluster. A clone adds one per index of each table. |
scramdb_branch_index_fallback_rebuilds_total | counter | Branch indexes rebuilt at clone time because the parent could not serve a view. 0 on a healthy store; each one cost time that grows with the table. |
scramdb_branch_index_view_probes_total | counter | Index reads answered through a branch's view of its parent's index. |
scramdb_branch_index_folds_total | counter | Branch indexes that finished copying their parent's entries into their own. |
scramdb_branch_index_fold_entries_total | counter | Entries copied into branch indexes by their folds. |
scramdb_branch_index_fold_cancelled_total | counter | Folds stopped early by a drop, a shutdown or the maintenance memory limit. They resume; a rate around a shutdown or a memory squeeze is expected. |
scramdb_branch_index_fold_failed_total | counter | Folds that stopped on an error and will run again. A steady rise with scramdb_branch_hold_retained_bytes rising too is a fold that cannot finish: read the server log. |
scramdb_branch_index_fold_seconds | histogram | Time one branch index fold took to finish. |
scramdb_branch_fork_holds | gauge | Branches whose index views read their parent's retired entries. |
scramdb_branch_hold_retained_bytes | gauge | Bytes of parent segments that branches still list and keep on disk. They are given back when the branch folds or drops. |
scramdb_branch_relocation_watches | gauge | Parent tables a clone in progress keeps the retired entries of. 0 when no clone runs. |
scramdb_branch_hold_pins | gauge | Parent indexes pinned for clones in progress on this node. 0 when no clone runs; a value that stays above 0 for minutes is a clone that never ended. |
scramdb_branch_vector_views | gauge | Branch vector indexes that read through their parent's index now. |
scramdb_branch_vector_views_registered_total | counter | Branch vector indexes registered as a view over their parent's index. |
scramdb_branch_vector_fallback_rebuilds_total | counter | Branch vector indexes rebuilt at clone time because the parent could not serve a view. 0 on a healthy store. |
scramdb_branch_vector_unservable_total | counter | Branch vector index reads answered from the table instead of the index. The answer is exact and slower. |
scramdb_branch_vector_folds_total | counter | Branch vector indexes that finished rebuilding as their own. |
scramdb_branch_vector_fold_failed_total | counter | Branch vector index folds that stopped on an error and will run again. |
scramdb_branch_vector_retained_bytes | gauge | Bytes of replaced vector index memory kept charged for branches still reading it. 0 once the last branch folds or drops. |
scramdb_branch_vector_searches_total | counter | Searches answered through a branch's vector index. |
scramdb_database_objects_forked_total | counter | Catalog objects a branch or clone gave to its new database: functions, procedures, types, schemas, event triggers, default privileges, their privileges, owners and comments. |
scramdb_database_objects_dropped_total | counter | Catalog objects removed with a dropped database, the same kinds. |
scramdb_database_layouts_migrated_total | counter | Saved catalog files read in the layout from before databases owned them, once each at the start that read them: every object in them became the default database's. |
scramdb_schema_renames_resumed_total | counter | Schema renames a start finished because the run before it stopped part way through one: the schema's members were moved to the new name from where they stopped. |
scramdb_schema_rename_resume_failures_total | counter | Schema renames a start could not finish. The rename stays recorded and the next start tries again; a rise needs an operator. |
scramdb_type_rekeys_resumed_total | counter | Type renames and moves a start finished because the run before it stopped part way through one: the stored texts and columns that name the type were brought to the new name from where they stopped. |
scramdb_type_rekey_resume_failures_total | counter | Type renames and moves a start could not finish. The change stays recorded and the next start tries again; a rise needs an operator. |
scramdb_enum_renumbers_total | counter | Enum types an ALTER TYPE ... ADD VALUE laid out again with every stored value, because the slot between two labels had no ordinal left. |
scramdb_enum_renumbers_resumed_total | counter | Enum renumberings a start settled because the run before it stopped part way through one: the stored values were brought to the type's labels from where they stopped. |
scramdb_enum_renumber_resume_failures_total | counter | Enum renumberings a start could not settle. The change stays recorded and the next start tries again; a rise needs an operator. |
scramdb_enum_renumbers_refused_total | counter | Enum value additions a cluster refused because a distributed table is placed by the enum, or a table appeared while the statement was decided. |
scramdb_sequence_follows_resumed_total | counter | Sequence renames and moves a start finished because the run before it stopped part way through one: the column defaults, views, domains and other stored texts that draw from the sequence were brought to its new name from where they stopped. |
scramdb_sequence_follow_resume_failures_total | counter | Sequence renames and moves a start could not finish. The change stays recorded and the next start tries again; a rise needs an operator. |
scramdb_collation_follows_resumed_total | counter | Collation renames and moves a start found unfinished because the run before it stopped part way through one: the registry had the new name and the stored texts might still name the old one. |
scramdb_collation_follows_finished_total | counter | Collation renames and moves a start brought to their new name in every stored text (the COLLATE clauses of checks, defaults, generated columns, views and index expressions). A resumed count above this one needs an operator: the change stays recorded and the next start tries again. |
scramdb_array_names_repaired_total | counter | Array type names a start put right because the run before it stopped part way through a create or a rename that moved an array aside: the names of the arrays and the types they belong to were brought together from where they stopped. |
scramdb_array_name_repair_failures_total | counter | Databases whose array type names a start could not put right. The stored names stay as they were and the next start tries again; a rise needs an operator. |
scramdb_partition_bounds_refreshed_total | counter | Partitioned tables whose stored bounds a start read again from their own text, because the bytes a bound of an array key keeps were saved before arrays carried their lower bounds and dimensions. It rises once per such table after an upgrade and stays flat after. |
scramdb_partition_bound_refresh_failures_total | counter | Partitioned tables whose stored bounds a start could not read again. The bounds stay as they were and the next start tries again; a rise needs an operator. |
A function, type, schema, event trigger or default privilege belongs to the database that made it, so a branch starts with a copy of its parent's and the two change apart from then on. database_objects_forked_total rises by the objects each branch or clone received, and database_objects_dropped_total by those a DROP DATABASE took with it. database_layouts_migrated_total moves only at the first start after an upgrade from a release that kept them for the whole server, and stays flat after.
Internal tables (always present)
The engine keeps the self-driving loop's decision record and the merge records in tables of its own, and holds each to a bound: the newest rows stay, a row past its age bound goes, the oldest first and a page of rows at a time, and the space of the removed rows is asked back from the background merge. One series per table, labelled table: autonomy.ticks, autonomy.actions, autonomy.setpoints, autonomy.explanations and autonomy.shadow for the decision record (bounded by [autonomy] audit_max_rows and audit_retention) and scram_lo.scram_branch_merges for the merge records of every database (bounded by [agent] merge_record_retention).
| Metric | Type | Meaning |
|---|---|---|
scramdb_internal_table_rows_trimmed_total | counter | Rows removed from one of the engine's own tables to keep it within its bound. |
scramdb_internal_table_reclaims_requested_total | counter | Times the space of one of the engine's own tables was asked to be reclaimed after a trim. |
scramdb_internal_table_row_bound | gauge | Rows one of the engine's own tables keeps, newest first. |
scramdb_internal_table_age_bound_seconds | gauge | Seconds a row of one of the engine's own tables is kept, for a table with an age bound. |
scramdb_merge_records_kept_past_bound | gauge | Merge records past the bound that the last trim pass read and kept, by what holds them (reason: branch_base, newest_through_path, refresh_above_base, in_flight). |
scramdb_autonomy_trim_kept_age_seconds | gauge | Seconds the oldest unfinished decision-record trim has been kept, 0 when none. |
A merge record can stay past its bound while something still reads it: a direct merge its branch's merge base has not passed (branch_base), the newest merge of a branch into a database above its parent (newest_through_path), a refresh above its branch's own base (refresh_above_base) or a merge in flight (in_flight). A table over its bound is explained by these reasons. A pass of the decision-record trim whose statement outlasts a tick is kept and goes on at the next one from where it stood, never from its first statement; after ten ticks the server logs one warning naming the table, and scramdb_autonomy_trim_kept_age_seconds reads how long the oldest kept pass has been kept. If it keeps growing, contact support with the server log.
The merge records have no age bound, so scramdb_internal_table_age_bound_seconds has no series for them. A reclaim request counts when the table was asked, not when its bytes came back: what a merge freed is counted per table in Compaction by table.
Security decisions (always present)
What the access checks decided in this process; see Roles and privileges. The refusal and rule-change counters are zero on a process nobody was refused on and nobody changed a grant on; the access-check counters count every statement checked.
| Metric | Type | Meaning |
|---|---|---|
scramdb_privilege_refusals_total | counter | Statements refused with a permission error (SQLSTATE 42501) and answered to a client, once each. A refusal a PL/pgSQL handler catches, and a statement a node relays to another, do not count again. A sustained rate is a client or pool running as the wrong role, or a grant that was never made; the server log names the role and the object. |
scramdb_row_security_scans_total | counter | Table scans planned with a row-security policy applied. A plan served from the plan cache is not planned again, so this is a lower bound of the filtered executions. Zero on a node whose tables use policies and whose clients run as their own roles means the policies are bypassed (the role owns the table, holds BYPASSRLS or is a superuser). |
scramdb_acl_changes_total | counter | Changes to the access rules: roles, grants and default privileges. A statement that changed nothing (refused before its first write) does not count; one that was undone counts once, with its rewind. |
scramdb_acl_rewinds_total | counter | Access-rule changes taken back after a later step failed or a transaction rolled back. A GRANT, REVOKE or role statement is applied whole or not at all: when the save of the rules is refused, the statement is undone and the client gets the store's error. A rate with no rolled-back transactions to explain it means the catalog store is refusing writes (a full disk or a stopped store); the server log names the cause. |
scramdb_access_checks_total | counter | Statement access lists judged against the role's privileges: one per statement planned (its reads, as the planner bound them), per write, COPY, TRUNCATE or locking clause judged at its start, and per point query judged on its first run. A plan served from the plan cache under unchanged rules is counted in the next metric instead. |
scramdb_access_check_reuses_total | counter | Access checks a cached plan or a cached point query answered with its earlier answer, because no GRANT, REVOKE, membership or owner change happened since: one compare, no privilege lookup. A GRANT or REVOKE makes the next run of every cached plan judged again. |
scramdb_security_seam_disagreements_total | counter | Debug builds only (always zero in a release build). |
scramdb_agent_quota_refusals_total | counter | Statements of an AI agent refused because its branch reached its size quota (SQLSTATE 53100). |
IN subqueries (always present)
What became of the result of each x IN (SELECT ...) whose subquery does not read the enclosing query. The subquery runs once per statement: its result is a typed list the statement holds, or, when the list would not fit the statement's memory budget or its column cannot be held as one, a relation of the statement that the join reads; a relation the budget refuses goes to temporary files. All zero on a process that ran no such subquery.
| Metric | Type | Meaning |
|---|---|---|
scramdb_in_subquery_lists_total | counter | IN subquery results held as a typed list of the statement. |
scramdb_in_subquery_relations_total | counter | IN subquery results held as a relation of the statement for the semi join. A rising share against the lists means results are large or of a type no list holds; check work_mem. |
scramdb_in_subquery_relations_spilled_total | counter | IN subquery relations written to temporary files because memory ran short. Any rate is memory pressure on the node. |
scramdb_in_subquery_lists_received_total | counter | Typed lists received with a fragment from another node (a cluster statement that reads a list on a node other than the one coordinating it). |
scramdb_in_subquery_held_bytes | gauge | Bytes running statements hold for their IN subquery lists and relations, charged to their memory budget where they have one. Returns to its start when the statements end; a value that stays up with no statement running is a leak. |
Snapshot index reads
An index lists only the latest version of each row. A REPEATABLE READ or SERIALIZABLE read whose snapshot is older than a change finds the version it sees through the index's record of superseded entries, instead of scanning the table.
| Metric | Type | Meaning |
|---|---|---|
scramdb_snapshot_index_probes_versioned_total | counter | Snapshot index probes answered from the index and its record of superseded entries. |
scramdb_snapshot_index_probes_scanned_total | counter | Snapshot index probes answered by scanning the table. |
scramdb_retired_versions_recorded_total | counter | Superseded index entries kept for snapshots that can still see them. |
scramdb_retired_versions_released_total | counter | Superseded index entries released by compaction or with their index. |
scramdb_retired_versions_entries | gauge | Superseded index entries currently kept for snapshots. |
scramdb_staged_read_settled_gate_entries_total | counter | Staged-entry reads that outran the bounded retry under a storm of stage changes and took the reader-priority gate. A rate means readers wait behind a heavy stream of stage changes. |
probes_scanned_total climbing against probes_versioned_total means snapshot reads are scanning whole tables instead of probing their index.
Read snapshot pins
The snapshot of a statement, of a REPEATABLE READ or SERIALIZABLE transaction, of a streaming result and of a distributed fragment's local scan is pinned while it is in use, so compaction keeps every row version it can still see.
| Metric | Type | Meaning |
|---|---|---|
scramdb_read_pins_taken_total | counter | Read snapshot pins registered. |
scramdb_read_pins_released_total | counter | Read snapshot pins released. |
scramdb_read_pin_redraws_total | counter | Snapshot draws that moved their pin off the floor registered before the draw. |
scramdb_read_pins_held | gauge | Read snapshot pins held now. |
scramdb_retention_holds_taken_total | counter | Retention holds taken for the shard groups hosted on this node. |
scramdb_retention_holds_released_total | counter | Retention holds released. |
scramdb_retention_holds_held | gauge | Retention holds held now for the shard groups hosted on this node. |
scramdb_retention_boot_holds_armed_total | counter | Stores that kept every row version at start until their shard groups were pinned. |
scramdb_retention_boot_holds_released_total | counter | Start-up retention holds released. |
scramdb_retention_boot_holds_held | gauge | Stores keeping every row version until their shard groups are pinned. |
read_pins_held that keeps growing on a steady workload means snapshots are not being released, which holds old row versions on disk; look for long-open transactions or unread result streams.
On a cluster node, each shard group hosted here also holds the row versions between the oldest read position the group can still serve and its newest commit, since a read at such a position can reach this node long after it was chosen; a quiet group holds only its last window, never what busier groups write after it. retention_holds_held follows the groups hosted here. At start a store keeps every row version until each shard group it holds rows of is hosted again or placed on another node, and it logs one line saying so; retention_boot_holds_held above zero long after start means a group was never hosted again (its restarts exhausted, or it is quarantined), and compaction on that store frees nothing until it is.
Memory admission
Heavy statements take a memory grant before they run and queue in arrival order when the pool is full. A statement is never refused for lack of memory while the pool can make progress; one that waits longer than admission_max_park_ms ends with a "server is busy" error, and one that holds its grant longer than admission_max_hold_ms while others wait is cancelled. A waiting statement shows IPC / admission in pg_stat_activity with the size of its share in admission_ticket_bytes, and the log names the statement holding the memory when a statement starts to wait. Catalog and system-view reads never take a grant. A statement whose data grows past what its grant covered is charged for the growth while it runs: it never waits for it and never takes it ahead of a statement that is queued, and when the gate has no room it keeps running at the share it has, spilling what does not fit, and a later run of the same shape is charged more. A shape the server has not run before starts with a small share (the pool split over the query cores) and grows while it runs as long as the pool has room; it never waits for more, it spills what does not fit, so a burst of new statements runs together. A wait is counted from the later of its start and the last time memory went back to the pool, so a pool that is still finishing work is not mistaken for a stalled one. A hash join whose build side does not fit in memory runs in rounds, each within the memory the server has free, and answers every row.
| Metric | Type | Meaning |
|---|---|---|
scramdb_admission_admitted_total | counter | Heavy statements that took a grant. |
scramdb_admission_queued_total | counter | Heavy statements that had to wait for one, counted once each when they first park. |
scramdb_admission_parked_waiters | gauge | Statements parked waiting for a grant right now. Emitted only once a real measurement exists. |
scramdb_admission_outstanding_grant_bytes | gauge | Bytes currently granted and not yet handed back. |
scramdb_admission_waited_past_deadline_total | counter | Statements that waited longer than the warning threshold. Nothing is refused on that threshold; it exists so a long wait is visible. |
scramdb_admission_cancelled_while_queued_total | counter | Statements whose own cancellation fired while they were still waiting. |
scramdb_maintenance_memory_waits_total | counter | CREATE INDEX builds (vector indexes and their REINDEX included), CLUSTER rewrites and ANALYZE runs whose working memory (storage.memory.maintenance_bytes) did not fit the pool when asked, and that waited for it instead of failing. The automatic analysis a COPY or bulk INSERT asks for counts here too, and runs once the pool has room. |
scramdb_analyze_on_load_waits_total | counter | Automatic analyses of a COPY or bulk INSERT ([statistics] analyze_on_load) that waited for the load's transaction to commit, so they read the loaded rows. Growth is normal: a load inside a transaction block counts once while it waits for its COMMIT. |
scramdb_auto_analyze_wakes_total | counter | Times the automatic analysis thread woke to look for work. An idle server holds it nearly still: the thread wakes once per [statistics] auto_analyze_check_interval, at once for an analyze-on-load request, and every 200 ms only while a request is waiting for its load's transaction or for memory. |
scramdb_analyze_samples_total | counter | Table samples drawn by ANALYZE and by automatic analysis, one per analysed table. |
scramdb_analyze_sample_pages_total | counter | Pages those samples read. |
scramdb_analyze_sample_reserved_bytes_total | counter | Working memory those samples reserved before reading their first page. Divided by scramdb_analyze_samples_total it is what one analysis asks for: it follows the table's own size and row width, so a small table asks for kilobytes and never for the whole maintenance_bytes. |
scramdb_analyze_samples_short_total | counter | Samples that read fewer pages than the [statistics] settings ask for, because the reserved memory held fewer or a page could not be read. The statistics such a sample gives are the page-header figures plus what was read. |
scramdb_analyze_statistics_reused_total | counter | Tables that took their saved statistics instead of being analysed: the tables of a restarted server, and the tables of a database made by CREATE DATABASE ... CLONE, which plan from their source's statistics until their own rows drift. |
scramdb_admission_bypassed_total | counter | Statements served without consulting the ledger, either because they were light enough to stream or because the gate is switched off. |
scramdb_admission_inherited_nested_total | counter | Nested statements served inside their parent's grant instead of taking a second one. Growth is normal on a workload full of CTEs and routines. |
scramdb_admission_released_at_seam_total | counter | Grants handed back cleanly at statement completion. On ordinary workloads this climbs in step with admitted_total. |
scramdb_admission_measured_grants_total | counter | Statements admitted with a grant sized from their measured memory. |
scramdb_admission_measured_growths_total | counter | Measured grants that had to grow while their statement ran. |
scramdb_admission_ticket_cap_bytes | gauge | Most memory the admission gate grants one statement (admission_ticket_percent of the execution budget, or its share at heavy_concurrency, whichever is smaller). |
scramdb_admission_park_bound_milliseconds | gauge | Longest a statement waits for a memory grant before it is ended; adapts to the waits observed, 0 when unbounded. |
scramdb_admission_hold_bound_milliseconds | gauge | Longest a statement holds a memory grant while others wait before it is ended; adapts to the holds observed, 0 when unbounded. |
scramdb_admission_parked_too_long_total | counter | Statements ended with a "server is busy" error after waiting longer than the park bound. |
scramdb_admission_hold_cancelled_total | counter | Statements cancelled for holding a memory grant past the hold bound while others waited. |
scramdb_admission_unenforceable_holds_total | counter | Holds that ran past the hold bound for a statement with no client session. |
scramdb_admission_small_admitted_total | counter | Statements admitted at once because their measured memory is a small share of the gate. |
scramdb_admission_topups_total | counter | Running statements whose memory grant was raised because their state outgrew it. A steady rate at one shape is a shape whose data keeps growing; it is charged more on its next run. |
scramdb_admission_topup_waits_total | counter | Workers that waited for another worker of the same statement to finish raising its memory grant. The wait is short (one charge); a steady rate means one statement's workers outgrow their share together. |
scramdb_admission_topups_refused_total | counter | Running statements whose memory grant could not be raised, because the gate had no room to give or the wait for another worker's raise ran out. They keep the share they have; a rate beside a full gate means storage.execution_memory_bytes (or the container limit) is small for the shapes. |
scramdb_admission_heavy_concurrency | gauge | Heavy statements the gate currently admits at once. Live-settable at runtime, independent of the storage.memory.heavy_concurrency config key's value at startup. |
Allocator accounting
Every byte ScramDB's own allocator hands out is counted exactly, the instant it is allocated; see Memory for how the figures below feed the budgets on that page.
| Metric | Type | Meaning |
|---|---|---|
scramdb_heap_live_bytes | gauge | The exact process heap total: every byte the allocator has handed out and not yet freed. |
scramdb_heap_reserve_bytes | gauge | Bytes the allocator has committed but not yet handed to a live allocation (its own retention). |
scramdb_heap_overflow_cells | gauge | Threads currently sharing the heap accounting overflow cell instead of their own. |
scramdb_heap_context_overflows_total | counter | Charge contexts created after the context table was full, sharing a class's permanent slot. |
scramdb_heap_class_bytes | gauge | Live bytes of each heap class (statement, index_build, vector_index, background, cache, cluster, autonomy, unattributed), labeled class. unattributed is the live heap beyond the other classes and the escaped bytes, never below 0, so the classes and scramdb_heap_escaped_bytes add up to scramdb_heap_live_bytes. |
scramdb_heap_escaped_bytes | gauge | Bytes still resident under a class whose owning statement, build or job already ended, labeled class. |
scramdb_heap_unpaired_small_bytes{class,direction} | gauge | Small-block bytes an ended context left for another to free (held), or freed beyond its own charge (freed), by class. They balance across contexts over time. |
scramdb_heap_unscoped_bytes | gauge | Bytes allocated outside every scope and not yet freed outside one: the raw counter behind the unattributed class, in which a block allocated outside every scope and freed inside one stays. |
scramdb_heap_class_overrun_bytes | gauge | Bytes by which the named classes and escaped bytes read past the live heap. 0 while they add up; nothing is refused because of it. |
scramdb_heap_uncharged_bytes | gauge | Bytes live statements hold in their allocator context past what they reserved through the execution-memory budget; the budgets already decide on the larger figure, so this shows how much work runs ahead of its reservations. |
scramdb_heap_vector_bytes | gauge | Bytes the vectors of each class hold right now (their values, long strings and array elements), labeled class. A statement's share counts against its memory: its operators spill once it nears its budget, and it stops with out of memory (53200) past the execution memory budget. |
scramdb_heap_vector_escaped_bytes | gauge | Vector bytes still held after the statement, build or job that made them ended, labeled class: a result handed to its client, or a value a cache kept. |
scramdb_heap_vector_over_limit_total | counter | Memory accounts whose vectors went past their limit, labeled class; a statement one is a statement answered out of memory (53200) instead of growing the server. |
scramdb_memory_uncharged_bytes{family} | gauge | Bytes a memory family holds beyond what it reserved, labeled family: execution_pool (statements, background work and index builds past the execution pool's reservations and the maintenance work's charge) and cache (the cache class past the compiled-code and plan caches' own accounting and the fused buffer pool's retained blocks). It reads zero when the family reserved at least what the allocator measured for it. The execution_pool series is absent until the server has an execution pool. |
scramdb_maintenance_memory_bytes | gauge | Working memory compaction and other maintenance work hold now. It is reserved when a pass takes a table and given back when the pass ends. |
scramdb_native_heap_bytes | gauge | Bytes the C allocator holds for native code, labeled state: in_use (handed out and not freed) and free (held and reusable). The query compiler and the code it compiled live there. The free bytes are returned to the system with the allocator's retained pages. Absent until the memory watchdog has sampled it, and on a system whose C allocator does not report it. |
scramdb_memory_file_resident_bytes | gauge | Resident file-backed memory: the binary's code and mapped files. The kernel can drop it, so it is no allocation and is not part of the unattributed figures. |
scramdb_memory_unattributed_bytes | gauge | Anonymous memory (resident, shared and swapped out, from the process's status file; file-backed pages are scramdb_memory_file_resident_bytes) the process holds beyond everything the engine's own counters name: the execution pool's reservations, maintenance work, the page and buffer pools, the aggregate tables, the streamed-result queue, the compiled-code and plan caches, the native heap (scramdb_native_heap_bytes) and the families of scramdb_memory_charged_bytes. It is the figure the memory watchdog's breach line gives as held by no live counter. What remains is thread stacks, mappings outside the native heap and the allocator's own overhead (its retained pages are a named holder and are returned every ten seconds past 256 MiB); read it beside scramdb_vector_memory_in_use_bytes before suspecting a leak. Absent until the memory watchdog has taken its first sample; it is never rendered as zero for a figure not yet known. |
scramdb_memory_anon_unattributed_bytes | gauge | The same figure as scramdb_memory_unattributed_bytes, sampled with the memory watchdog. |
scramdb_memory_ceiling_effective_bytes | gauge | The memory ceiling the watchdog enforces now: the configured ceiling, or less while the machine has less memory free than the server was sized for. Absent until the first sample. |
scramdb_memory_pressure_some_avg10 | gauge | The percent of the last ten seconds in which some task on the machine stalled on memory, from the memory pressure stall file. Absent where that file cannot be read. |
scramdb_memory_shed_total{step} | counter | Times the watchdog gave memory back before cancelling a statement, labeled step: allocator_purge, subplan_cache, plan_cache, compiled_code and buffer_pool, in the order they run. Absent until the watchdog is armed. |
scramdb_memory_charged_bytes{family} | gauge | Bytes an owner of process memory has charged, labeled family: session_buffers (the read and write buffers each connection thread keeps between statements), catalog (the table definitions the schemas hold and the parsed stored definitions: view bodies, defaults, checks, policies, trigger conditions and function bodies), udf_daemons (the memory reserved for the function daemons' shards, a reservation and not what the guests touched), fused_pool (the retained blocks of the compiled kernels' buffer pool) object_store_pool (the remote-read client's idle connection buffers and one connection's buffers for each open response body) subplan_cache (the finished results the subplan cache keeps for reuse, counted by their size in memory), stack_segments (the stack segments a deeply nested statement grew while it was read, planned and run, a reservation and not what the statement touched; the segments the SQL reader grows for itself are not counted), regex_exec (the working memory of a regular-expression search that has a back-reference or a look-around, the scratch it leaves for the next search, and the character keys a pattern holds while it is compiled), function_scratch (the buffer a string or binary function writes each answer into, at the largest answer of each running batch) and row_places (where each row of a merged segment sits, four bytes per row position of the segments a compaction merged, for the merged segments read by row id since the server started), index_buffers (the nodes and entries every index holds in memory before they are written into a run file), row_locks (the row locks each open transaction holds: per row an UPDATE, DELETE, unique-key INSERT, COPY or locking SELECT locked, its lock entry and the key kept to release it, an estimate per lock), tag_map (the allocator's map from a block to the context it was charged to, 2 MiB each time it grows, never given back), wal_buffers (the write-ahead log writer's ring and its completion slots, sized by the memory budget) and vector_arenas (the vector indexes' arenas and row-id ledgers, outside the allocator, without their write buffers, which the execution pool counts). The catalog, object_store_pool, subplan_cache and vector_arenas figures are estimates from the stored sizes and the client's own settings, not allocator readings. |
scramdb_memory_budget_bytes{family} | gauge | The byte budget an owner of process memory is sized to, labeled family, derived from the machine: the connection places for session_buffers, a split of the runtime memory margin by what each uses for catalog, fused_pool, subplan_cache and regex_exec, the function daemons' own memory envelope for udf_daemons and the client's connection pool plus one open body per core for object_store_pool. A family appears only once its budget is known, so stack_segments, function_scratch, row_places and row_locks, which have none, are absent here. The budget of index_buffers is the index memory (index_memory_bytes, a share of the maintenance memory unless set): past it the largest buffers are written out until the family is at three quarters of it. Nothing is refused at a budget except the fused pool, which frees a block instead of keeping it, the subplan cache, which drops its entries and does not keep a result larger than the whole budget, a regular-expression search, which drops the end positions it kept and works them out again, and the catalog's parsed definitions, which drop the ones no statement read since the last pass and parse one again when it is next read; an owner above its budget shows on this pair of series. |
scramdb_memory_refused_bytes_total{family} | counter | Bytes a family's budget refused, labeled family, since the process started: the fused pool's blocks it freed instead of keeping, the subplan cache's results it did not keep, the end positions a regular-expression search dropped to work them out again and the parsed definitions the catalog did not keep. A family that is refused is given room at the next margin split. |
scramdb_memory_margin_bytes | gauge | The runtime memory margin the table definitions, the fused pool, the subplan cache and the regular-expression working memory draw their budgets from. Absent until the margin is resolved. |
scramdb_memory_margin_pool_bytes | gauge | The part of the runtime memory margin divided among those four at the last split: the margin less the resident memory no counter names. |
scramdb_memory_margin_split{reason} | gauge | One series per reason, one for the reason the last margin split came out as it did and zero for the others: even (nothing is used yet, the pool is divided evenly), follows_use (each budget is its family's use plus an equal share of what is unclaimed) and over_pool (the families want more than the pool, so each keeps what it holds and the rest is divided by what was refused). |
scramdb_execution_pool_allocated_bytes | gauge | Bytes the execution memory pool currently holds against its budget (the larger of what was reserved and what the allocator measured). |
scramdb_execution_pool_budget_bytes | gauge | The execution memory pool's total budget (execution_memory_bytes). |
scramdb_write_set_returned_bytes_total | counter | Write-set bytes given back to the execution memory pool before their transaction ended: an UPDATE that made a row smaller, and the part of a WITH statement's reservation its arms did not keep. Zero on a node whose transactions only grow what they write. |
scramdb_execution_pool_essential_breach_episodes_total | counter | Times the execution memory pool went past its budget and its small overshoot allowance. Reported once per crossing, never once per allocation; if the count keeps climbing, raise execution_memory_bytes or run fewer heavy queries at once. |
Joins over the memory budget and buffer reuse (always present)
A hash join whose build side does not fit the execution-memory budget spills and runs in rounds, and several such joins in one probe stage share one round schedule. When the joins' keys are columns of the probe table itself, the probe side is partitioned to disk once and each pass reads only the rows its rounds can match; the query workers reuse their interpreters and join buffers across morsels. A join whose output for one batch of its probe side is larger than one output batch (a cross join, a key the build side holds many times) emits it in bounded batches, each charged to the statement's memory while it is held, and a plain query over such a join streams them to the client as they are made.
| Metric | Type | Meaning |
|---|---|---|
scramdb_join_round_plans_total | counter | Hash join builds that did not fit the execution-memory budget and ran in rounds. |
scramdb_join_round_passes_total | counter | Passes over a probe side made by hash joins running in rounds. |
scramdb_join_round_multi_plan_stages_total | counter | Probe stages that ran several oversized hash joins in rounds together. |
scramdb_join_round_chunked_plans_total | counter | Hash join builds that split one partition into several rounds to stay within the execution-memory budget. |
scramdb_join_round_partitioned_probes_total | counter | Probe stages of hash joins in rounds whose probe side was partitioned to disk once instead of read on every pass. |
scramdb_join_output_steps_total | counter | Join output batches emitted by a probe whose output was larger than one batch. |
scramdb_unnest_output_steps_total | counter | Unnest output batches emitted for arrays whose elements were more than one batch. |
scramdb_join_output_step_bytes | gauge | Bytes of join and unnest output batches the query workers hold right now. |
scramdb_join_output_step_peak_bytes | gauge | Most bytes of join and unnest output batches the query workers held at once. |
scramdb_execution_interpreters_built_total | counter | Interpreters the query workers built to run morsels on the interpreter. |
scramdb_execution_join_selection_grows_total | counter | Times a query worker grew its reused join selection buffer. |
scramdb_execution_agg_rows_folded_in_runs_total | counter | Grouped aggregate rows folded into the previous row's group without a hash lookup, because their key equals the previous row's. |
scramdb_execution_agg_rows_folded_in_runs_compiled_total | counter | The same, for the rows the compiled tier folded. |
scramdb_execution_agg_run_checks_switched_off_total | counter | Grouped aggregate batches whose consecutive-key check switched off for lack of runs, so input not ordered on its keys pays only a short warm-up of comparisons per batch. |
scramdb_execution_call_argument_views_total | counter | Batch arguments handed to scalar functions as zero-copy views instead of copies. |
The build counters stay bounded by the workers and the batch sizes, never by the rows a statement reads: The folded-row counters grow on input grouped or sorted by its grouping key, where most rows skip the hash table.
Set-returning routines (always present)
A RETURNS SETOF or RETURNS TABLE PL/pgSQL function hands its rows to the query in bounded batches, the routine pausing while the query works on the last one, so these counters stay bounded by the batch size, never by the rows a function answers.
| Metric | Type | Meaning |
|---|---|---|
scramdb_setof_routine_calls_total | counter | Set-returning routine calls started by a query. |
scramdb_setof_routine_batches_total | counter | Row batches set-returning routines handed to their queries. |
scramdb_setof_routine_rows_total | counter | Rows set-returning routines handed to their queries. |
scramdb_setof_routine_bytes_total | counter | Estimated bytes of the rows set-returning routines handed to their queries. |
scramdb_setof_routine_abandoned_total | counter | Set-returning routine calls stopped before their last row was read, by a LIMIT, a cancel or an error. |
scramdb_setof_routine_open_calls | gauge | Set-returning routine calls started and neither finished nor stopped right now. |
scramdb_user_aggregate_partial_states_total | counter | Partial transition states merged by user-defined aggregates: the path an aggregate with a COMBINEFUNC takes, one state per group and reader instead of every row. |
scramdb_user_aggregate_rows_gathered_total | counter | Rows user-defined aggregates collected and fed to their transition function in sorted order: the path of an aggregate with no COMBINEFUNC, or with ORDER BY, DISTINCT or WITHIN GROUP. A group larger than the aggregate memory budget is fed from a stream on disk. |
scramdb_user_aggregate_bindings | gauge | User aggregate bindings held, live and waiting to be freed after a replace or drop. |
A LIMIT over a function that answers millions of rows counts one abandoned call and a few small batches.
Spilling to disk
A query that outgrows its memory budget writes to disk rather than failing. These count that happening.
| Metric | Type | Meaning |
|---|---|---|
scramdb_agg_spill_partition_flushes_total | counter | Aggregate partitions written to disk instead of breaching the memory budget. |
scramdb_agg_spill_bytes_written_total | counter | Bytes written to aggregate spill files. |
scramdb_agg_spill_rows_written_total | counter | Rows written to aggregate spill files. |
scramdb_agg_spill_fragment_partition_flushes_total | counter | Fragment partitions written to disk for the same reason. |
scramdb_agg_spill_fragment_bytes_written_total | counter | Bytes written to fragment spill files. |
scramdb_agg_spill_fragment_rows_written_total | counter | Entries written to fragment spill files. |
scramdb_agg_spill_grace_partitions_total | counter | Aggregate partitions merged in several passes because they did not fit their memory. |
scramdb_agg_spill_grace_passes_total | counter | Merge passes run over aggregate partitions split by hash range. Divided by the partitions above, the passes a partition took. |
scramdb_spill_tape_live_bytes | gauge | Bytes currently resident in every live spill file. Alert on it if the volume is small. |
scramdb_fragment_spill_mgr_init_failures_total | counter | Times the fragment spill directory could not be initialized and was retried. Zero on a healthy process; non-zero points at temp_dir permissions or space. |
scramdb_final_agg_spill_mgr_init_failures_total | counter | The same, for the final aggregate spill directory. |
Bulk load
| Metric | Type | Meaning |
|---|---|---|
scramdb_copy_one_pass_chunks_total | counter | Chunks processed by the single-read COPY parser. Zero means that path never engaged, either because it is switched off or because no eligible COPY has run. |
scramdb_copy_one_pass_reparses_total | counter | Chunks that had to be parsed a second time because the parser's guess about where quoting stood at the chunk boundary was wrong. Against chunks_total this is the miss rate; a file with heavy embedded quoting raises it. |
scramdb_copy_one_pass_reparse_bytes_total | counter | Bytes re-read by those second passes, tracked separately so load throughput is never overstated. |
scramdb_copy_line_scan_bytes_total | counter | Bytes of COPY input read again to find the line number of a failing row. A COPY that succeeds adds nothing; a failing one usually adds only its first record. |
scramdb_copy_ordered_chunks_total | counter | COPY chunks handed over in file order because the table's triggers or column defaults need the rows in input order. |
scramdb_copy_ordered_wait_nanos_total | counter | Nanoseconds COPY parse workers waited for earlier chunks to be handed over in file order. |
scramdb_copy_staged_bytes | gauge | Bytes of staged COPY rows and key runs held on this node. |
scramdb_copy_stages_open | gauge | COPY stages holding rows that are not committed yet: a node-local COPY's, and on a cluster node also every replica's stage of a cluster COPY and the coordinating node's stage of the COPY's key checks. |
scramdb_copy_staged_rows_total | counter | Rows COPY staged. |
scramdb_copy_stages_attached_total | counter | COPY stages committed into their tables. |
scramdb_copy_stages_dropped_total | counter | COPY stages discarded without a commit. |
scramdb_copy_staging_refusals_total | counter | COPY batches refused because the staging area was full. Each one failed its COPY with the disk-full error, leaving nothing behind. |
scramdb_copy_stages_recovered_total | counter | COPY stages a restart found and freed. |
scramdb_copy_stages_restored_total | counter | Committed COPY stages a restore or a branch rebuilt from the WAL. |
scramdb_copy_attach_seconds | histogram | Time to attach one COPY stage to its table at commit. |
A COPY writes its rows into staged segments outside the table, invisible to every reader, and attaches them to the table in one step when its transaction commits; a rollback or a crash before the commit frees them whole. copy_staged_bytes is the disk those uncommitted rows hold now. A COPY into a cluster table stages its rows on every replica of every shard group it writes and commits them together (see COPY into a distributed table); copy_stages_open above zero while no COPY is running means a stage was left behind, which a group leader collects after [cluster.dilith] copy_stage_idle_timeout (scramdb_cluster_dilith_copy_stages_collected_total).
A COPY into a table with a trigger, or with a column default that calls a VOLATILE function, still parses a large file across parallel lanes but hands the lanes' chunks to the statement in file order, one at a time, so the trigger or default sees its rows in input order. copy_ordered_chunks_total counts those chunks and copy_ordered_wait_nanos_total the time the parse workers spent waiting for their turn. A wait that is large against the load's wall time means the statement's own per-row work (a row trigger) is what bounds the load, and more parse lanes will not make it faster; a table with only statement triggers or a cheap default spends little waiting. Neither counter moves for a table without triggers.
Import (COPY ... FROM STDIN)
| Metric | Type | Meaning |
|---|---|---|
scramdb_copy_in_started_total | counter | COPY FROM STDIN statements that started receiving rows from a client. |
scramdb_copy_in_completed_total | counter | COPY FROM STDIN statements that loaded their rows. |
scramdb_copy_in_failed_total | counter | COPY FROM STDIN statements that failed, including those the client aborted. |
scramdb_copy_in_client_aborts_total | counter | COPY FROM STDIN statements the client ended with CopyFail. |
scramdb_copy_in_bytes_total | counter | Bytes of COPY FROM STDIN input handed to loads. |
scramdb_copy_in_slices_total | counter | Slices of COPY FROM STDIN input handed to loads. |
scramdb_copy_in_client_waits_total | counter | Times a COPY FROM STDIN's input waited for its load to take the slices before it. |
An import queues at most [execution] copy_pipeline_depth slices of input for its load and makes the client wait while the queue is full, so copy_in_client_waits_total climbing against copy_in_started_total means the load is slower than the client sends; the server is not buffering the difference.
Export (COPY ... TO STDOUT)
| Metric | Type | Meaning |
|---|---|---|
scramdb_copy_out_streams_total | counter | COPY TO STDOUT statements that started streaming rows to a client. |
scramdb_copy_out_rows_total | counter | Rows COPY TO STDOUT statements streamed to clients. |
scramdb_copy_out_bytes_total | counter | Bytes of COPY TO STDOUT output streamed to clients. |
scramdb_copy_out_failed_total | counter | COPY TO STDOUT statements that ended with an error after they started streaming. |
scramdb_copy_out_client_waits_total | counter | Times a COPY TO STDOUT waited for its client to take rows before rendering more. |
scramdb_copy_out_seconds | histogram | Time from a COPY TO STDOUT's CopyOutResponse to its CopyDone. |
An export holds [execution] copy_pipeline_depth buffers and waits for the client when they are full, so copy_out_client_waits_total climbing against copy_out_streams_total means clients read slower than the server renders; the export is not buffering the difference.
Programs (COPY ... PROGRAM)
| Metric | Type | Meaning |
|---|---|---|
scramdb_copy_program_children | gauge | Programs started by COPY PROGRAM that have not been reaped yet. |
scramdb_copy_program_descriptors | gauge | Pipe descriptors the server holds to the programs COPY PROGRAM runs. |
scramdb_copy_program_started_total | counter | Programs started by COPY PROGRAM. |
scramdb_copy_program_failed_total | counter | COPY PROGRAM statements that failed in their program or its pipe. |
scramdb_copy_program_cancelled_total | counter | COPY PROGRAM statements cancelled, their program killed. |
Both gauges stay at zero when no COPY PROGRAM runs. A value that does not return to zero after the statements ended means a program is still alive: read the process list on the server for the sh -c children of the engine.
System routines and server statements
| Metric | Type | Meaning |
|---|---|---|
scramdb_system_calls_total | counter | System routine calls evaluated, one per row a call ran for. |
scramdb_system_file_bytes_read_total | counter | Bytes the server file functions read from files on the server. |
scramdb_config_reloads_total | counter | Configuration reloads run by pg_reload_conf() or a SIGHUP. |
scramdb_wal_switches_total | counter | Write-ahead log file switches run by pg_switch_wal(). |
scramdb_alter_system_total | counter | ALTER SYSTEM statements that changed the settings written to the auto file. |
scramdb_alter_system_write_failures_total | counter | ALTER SYSTEM statements the store could not write to the auto file, and statements replayed from another node that a node with an unreadable auto file skipped. |
scramdb_tablespace_moves_total | counter | Tables, materialized views and indexes whose storage or tablespace placement changed by SET TABLESPACE. |
scramdb_tablespace_moved_bytes_total | counter | Bytes of table storage copied by SET TABLESPACE. |
scramdb_tablespace_moves_total counts a relation once, however many files it holds, and an index adds one move and no bytes (an index owns no file, so its placement is a record). A relation already in the target and a refused statement add nothing, and neither does the undo of a move or the placement a REINDEX (TABLESPACE) makes. A move copies the table's files into the target before it removes the old ones, so watch the free space of the target's volume while a large SET TABLESPACE runs.
A call refused for a missing EXECUTE or setting privilege is not counted here: it is a 42501 and counts once in scramdb_privilege_refusals_total. A rising scramdb_alter_system_write_failures_total means the store refuses the write (a full disk or a stopped store): the statement failed and no setting changed, so read the server log for the cause.
Large objects
| Metric | Type | Meaning |
|---|---|---|
scramdb_large_objects_created_total | counter | Large objects created. |
scramdb_large_objects_unlinked_total | counter | Large objects removed. |
scramdb_large_object_bytes_read_total | counter | Bytes read from large objects. |
scramdb_large_object_bytes_written_total | counter | Bytes written to large objects. |
scramdb_large_object_pages_written_total | counter | Large object pages stored. |
scramdb_large_object_descriptors_opened_total | counter | Large object descriptors opened. |
scramdb_large_object_descriptors_open | gauge | Large object descriptors open in running transactions. |
scramdb_large_object_import_bytes_total | counter | Bytes lo_import read from server files into large objects. |
scramdb_large_object_export_bytes_total | counter | Bytes lo_export wrote from large objects to server files. |
scramdb_large_object_faults_total | counter | Large object calls that failed with an error. |
scramdb_connection_function_calls_total | counter | Function call messages received. |
scramdb_connection_function_call_failures_total | counter | Function call messages answered with an error. |
The counters count work done: a write that was rolled back is still counted. The descriptor gauge returns to zero as transactions end and connections close, so a value that stays above zero with no transaction open points to a session that holds a transaction open; read pg_stat_activity. A function call message is counted in the two connection counters and, for a large-object function, in the families above too.
Foreign keys
| Metric | Type | Meaning |
|---|---|---|
scramdb_fk_referencing_keys_checked_total | counter | Referencing keys checked against the table they reference. |
scramdb_fk_referenced_keys_checked_total | counter | Referenced keys that left or changed and were checked for rows still naming them. |
scramdb_fk_violations_total | counter | Statements and commits refused because a foreign key was violated. |
scramdb_fk_actions_total | counter | Referential actions (CASCADE, SET NULL, SET DEFAULT) run on referencing tables. |
scramdb_fk_deferred_checks_total | counter | Foreign key checks deferred to the transaction's commit point. |
scramdb_fk_statement_end_checks_total | counter | Foreign key checks held for the end of their statement, as when a WITH statement inserts a parent row in one arm and its child in another. |
scramdb_fk_key_shares_total | counter | Referenced keys a referencing check held shared while it ran, so a concurrent delete of the key waits for it. |
scramdb_fk_key_share_waits_total | counter | Referencing checks that waited for a transaction removing the key they reference. |
scramdb_fk_leaving_key_locks_total | counter | Referenced keys held exclusively because they left or changed. |
scramdb_fk_removed_after_snapshot_total | counter | Referencing checks refused with 40001 because a key their snapshot held was removed by a later commit. |
scramdb_fk_probe_seconds | histogram | Time one foreign key probe statement took. |
scramdb_unique_deferred_keys_total | counter | Keys of a deferrable unique constraint held for the end of their statement or the commit point. |
scramdb_unique_deferred_keys_checked_total | counter | Held keys of a deferrable unique constraint checked for a second row. |
scramdb_unique_deferred_violations_total | counter | Statements and commits refused because a held key of a deferrable unique constraint was taken twice. |
Triggers
| Metric | Type | Meaning |
|---|---|---|
scramdb_trigger_statement_end_events_total | counter | AFTER ROW and AFTER STATEMENT trigger events held for the end of their statement, as when a WITH statement writes a table in one arm and the trigger counts rows another arm wrote. |
scramdb_trigger_deferred_events_total | counter | AFTER ROW events of deferrable constraint triggers held for the end of their transaction, to run at COMMIT or at SET CONSTRAINTS ... IMMEDIATE. A rate is the application's own use of deferred constraint triggers; a count that climbs while no transaction commits is memory held by long open transactions. |
scramdb_trigger_key_units_total | counter | Runs of foreign-key checks and actions judged among the AFTER ROW triggers of a statement. It stays at 0 for a statement whose triggers all sort behind its keys, and rises once a trigger named ahead of a key makes the statement judge the key row by row, as PostgreSQL does. |
Prepared transactions (always present)
What a restart did with the transactions left in PREPARE TRANSACTION (see Prepared transactions): the ones it found, finished, put back and cannot use.
| Metric | Type | Meaning |
|---|---|---|
scramdb_prepared_xacts_recovered_total | counter | Prepared transactions found on disk at startup and registered again. |
scramdb_prepared_xacts_resolved_committed_total | counter | Prepared transactions whose COMMIT PREPARED was found decided and finished. |
scramdb_prepared_xacts_put_back_total | counter | Prepared transactions whose interrupted COMMIT PREPARED was found undecided and put back to prepared. |
scramdb_prepared_xacts_leftovers_removed_total | counter | Files of prepared transactions removed because no transaction owned them. |
scramdb_prepared_xacts_corrupt_found_total | counter | Prepared transaction records found unreadable or without their changes. |
scramdb_prepared_xacts_corrupt | gauge | Prepared transactions whose record cannot be used and can only be rolled back. |
scramdb_prepared_xacts_orphan_gids | gauge | Prepared transaction identifiers listed without any record of their changes. |
scramdb_prepared_xacts_oldest_age_seconds | gauge | Seconds since the oldest prepared transaction of this node or its cluster was prepared, 0 when there is none; the cluster's part is refreshed periodically, not at each scrape. The node's own records are read at each scrape. A prepared transaction of a cluster table is kept by the cluster's registry, which each node reads about once a minute, so a cluster transaction prepared or ended just now shows up to about a minute late; pg_prepared_xacts lists them at once. |
recovered_total moves only at startup, and leftovers_removed_total only after a stop during PREPARE TRANSACTION or COMMIT PREPARED. corrupt and orphan_gids stay above zero until each listed identifier is ended with ROLLBACK PREPARED; alert on them, and read the server log for the identifiers.
Transactions and locks (always present)
| Metric | Type | Meaning |
|---|---|---|
scramdb_transactions_committed_total | counter | Transactions that committed. |
scramdb_transactions_aborted_total | counter | Transactions that rolled back or failed to commit. |
scramdb_lock_waits_total | counter | Row and table lock requests that found the lock held and waited. |
Transaction blocks (always present)
What an open transaction block hides from other sessions while it is open, and what a restart decided about a block it cut short (see DDL inside a transaction block).
| Metric | Type | Meaning |
|---|---|---|
scramdb_block_visible_refused_total | counter | Statements refused because they named an object another transaction's open block created. |
scramdb_block_claims_live | gauge | Objects other than tables (views, functions, sequences, indexes, types, schemas) made in open transaction blocks that other sessions cannot see yet. |
scramdb_block_marks_kept_total | counter | Tables made in a transaction block that a restart kept because the block committed. |
scramdb_block_marks_undone_total | counter | Tables made or dropped in a transaction block that a restart took back. |
scramdb_block_marks_finished_total | counter | Tables dropped in a transaction block that a restart finished dropping. |
scramdb_block_marks_failed_total | counter | Tables a restart could not resolve after a transaction block was cut short. |
scramdb_block_commit_records_removed_total | counter | Commit records of transaction blocks with cluster DDL that were removed once the block had ended. A block with no cluster DDL writes none and removes none. |
scramdb_replica_apply_live_duplicates_total | counter | Primary keys a replica apply found with more than one live version. It stays at zero in normal operation; if it rises, contact support with the server log. |
scramdb_block_boot_ends_total | counter | Ends the cluster decided for transaction blocks a restart cut short. |
scramdb_block_boot_end_retries_total | counter | Attempts to end a cut-short transaction block that the cluster did not decide. |
scramdb_block_round_give_backs_total | counter | Lock rounds of a rolled-back savepoint whose table locks the other nodes gave back. |
scramdb_block_round_give_back_failures_total | counter | Hand-backs of a rolled-back savepoint's locks that did not reach every node. |
scramdb_block_commits_named_total | counter | Transaction block commits named through the cluster before their submission. |
scramdb_block_commits_refused_total | counter | Transaction block commits not submitted because another node ended the block first. |
scramdb_block_departed_ends_total | counter | Transaction blocks of nodes that left the cluster ended as their commit ended. |
refused_total rises under real concurrency, when a client names a table another session is still creating; a steady rate on a name that never existed is a client bug. claims_live is zero whenever no block is creating an object. The marks and boot counters move only at startup, after a node stopped inside a transaction block: expect a rise next to a restart, alert on any failed_total, and read the server log for the table. A sustained boot_end_retries_total on a cluster means the cluster has no leader or quorum to decide the block's end yet.
Deep diagnostics
These are emitted but are not part of the supported surface, and their names and meanings can change between releases. They are here so a number you see on the endpoint is never unexplained; do not build alerts on them.
| Metric | Type | Meaning |
|---|---|---|
scramdb_header_sidecar_loads_total | counter | Segments whose page headers were loaded entirely from their persisted sidecar file: one sequential read instead of one per page. |
scramdb_header_sidecar_partial_loads_total | counter | Sidecars that covered only part of what was asked for, the ordinary case for a table still growing. |
scramdb_header_sidecar_fallbacks_total | counter | Sidecar reads that fell back to walking page headers because the file was absent, stale or corrupt. |
scramdb_header_sidecar_writes_total | counter | Header sidecars written. |
scramdb_header_sidecar_write_errors_total | counter | Header sidecar writes that failed. The segment is walked next time instead. |
scramdb_header_prefix_pages_walked_total | counter | Pages read by the per-page header walk. A warm store should stop moving it. |
scramdb_page_zero_bytes_full_total | counter | Bytes zeroed over a whole page buffer on release. |
scramdb_page_zero_bytes_partial_total | counter | Bytes zeroed over only the recorded written region. On a warm selective scan this should dominate the full counter. |
scramdb_scan_prune_nanos_total | counter | Core-nanoseconds spent choosing which pages to read. Summed across workers, so it exceeds wall-clock time on a parallel scan. |
scramdb_scan_acquire_nanos_total | counter | Core-nanoseconds spent acquiring pages, pinning them when resident and reading them when not. |
scramdb_scan_decode_nanos_total | counter | Core-nanoseconds spent turning page bytes into chunks. |
scramdb_agg_fold_rows_total | counter | Partial aggregate rows consumed by the final fold. |
scramdb_agg_fold_chunks_parallel_total | counter | Fold chunks large enough to spread across cores. |
scramdb_agg_fold_chunks_in_place_total | counter | Fold chunks small enough to fold on one core. |
scramdb_agg_fold_distinct_values_offered_total | counter | DISTINCT values offered to a fold's deduplication set. |
Licensing metrics
| Metric | Type | Meaning |
|---|---|---|
scramdb_license_valid | gauge | 1 if the active license is valid. |
scramdb_license_edition | gauge/label | Active license edition. |
scramdb_license_node_count | gauge | Nodes currently counted against the license. |
scramdb_license_max_nodes | gauge | Node limit the license allows. |
scramdb_license_read_only_active | gauge | 1 if the license has forced the deployment read-only. |
scramdb_license_days_of_term_remaining | gauge | Emitted only for a term-limited license. |
scramdb_license_data_bytes | gauge | Emitted only for a license with a data-size cap. |
scramdb_license_data_cap | gauge | Emitted only for a license with a data-size cap. |
Cluster mode
Running as a cluster node, additional scramdb_cluster_* metrics are appended to the same /metrics response after the single-node metrics above. Every failure mode below has a counter: a stalled compaction, a quarantined group, a dropped partition frame. Alerting on a ScramDB cluster means reading a number, never grepping logs for a string match.
Unlike the WAL archive metrics above, every scramdb_cluster_* metric is always present once [cluster] is configured, whether or not anything interesting has happened yet. A fresh cluster reports every counter at 0 and every gauge at its idle value; there's no "not measured yet" gap to account for in this family.
Peer transport
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_peer_connects_total | counter | Peer connections established over the cluster transport. |
scramdb_cluster_peer_disconnects_total | counter | Peer connections closed. |
scramdb_cluster_reconnect_attempts_total | counter | Reconnect attempts after a lost peer connection. |
scramdb_cluster_frames_sent_total | counter | Frames sent to peers. |
scramdb_cluster_frames_received_total | counter | Frames received from peers. |
scramdb_cluster_bytes_sent_total | counter | Bytes sent to peers. |
scramdb_cluster_bytes_received_total | counter | Bytes received from peers. |
scramdb_cluster_send_queue_bytes | gauge | Outbound send-queue occupancy across all peer connections, right now. Bounded by send_queue_bytes in [cluster] config (default 64MB) per connection; a value pinned near that bound is backpressure, not idle capacity. |
scramdb_cluster_partition_drops_total | counter | Frames dropped by a test partition, outbound and inbound combined. Zero on every normal deployment. |
scramdb_cluster_fragment_boot_refusals_total | counter | Fragment requests from other nodes answered with a refusal because this node was still starting. |
scramdb_cluster_forward_boot_refusals_total | counter | Requests other nodes forwarded to this node for the metadata group (a DDL, a lock, a node number claim, a member removal) answered as not made because this node was still starting: the sender asks the next node at once instead of waiting out its forward timeout. |
scramdb_cluster_handshake_latency_seconds | histogram | Peer handshake latency. 11 buckets from 1ms to 5s (le="0.001" through le="5.000") plus +Inf. |
scramdb_cluster_connections_closed_total | counter | Peer connections closed, by cause: ping_deadline (no byte from the peer within its dead-peer deadline, the stalled-process case), tcp_timeout (the kernel's user timeout or keepalive gave up), peer_closed, local (shutdown or a duplicate connection), io_error, protocol_error. |
scramdb_cluster_frames_dropped_on_close_total | counter | Frames still queued on a peer connection when it closed, dropped and reported to the subsystems that sent them. |
scramdb_cluster_peer_events_total | counter | Peer connection events published to the cluster's subsystems, by kind: connected and lost. |
scramdb_cluster_peer_event_lag_total | counter | Times a subsystem missed connection events and refreshed its view of every peer. Zero in normal operation. |
scramdb_cluster_liveness_pings_sent_total | counter | Liveness pings sent on peer connections that had sent nothing for ping_interval. |
scramdb_cluster_liveness_ping_rtt_seconds | histogram | Round trip of a liveness ping, from its send to its answer. 18 buckets from 10µs to 10s plus +Inf. |
scramdb_cluster_dead_peer_deadline_connections | gauge | Open peer connections by how their dead-peer deadline is set, source: floor (the adaptive deadline at dead_peer_timeout_min), measured (lifted by a long round trip), ceiling (held at dead_peer_timeout_max), pinned (a fixed dead_peer_timeout). |
scramdb_cluster_dead_peer_deadline_seconds | histogram | Every dead-peer deadline put in force, at a connection's start and whenever its measured round trip moves it. |
scramdb_cluster_socket_option_failures_total | counter | Socket options (TCP_NODELAY, keepalive, user timeout) the kernel refused on a peer connection. Zero on Linux. |
scramdb_cluster_peers_remembered | gauge | Peers whose connection closed within the grace period and whose handshake details are kept. |
scramdb_cluster_peers_remembered_expired_total | counter | Remembered peers dropped after staying disconnected past the grace period. |
scramdb_cluster_peers_left_total | counter | Peers forgotten because they left the cluster membership. |
scramdb_cluster_dial_backs | gauge | Peers this node is dialing back because this node must open their connection. |
scramdb_cluster_dial_backs_started_total | counter | Times this node started dialing a peer back. |
scramdb_cluster_dial_backs_ended_total | counter | Times this node stopped dialing a peer back. |
scramdb_cluster_peer_identity_refusals_total | counter | Peer connections refused because the certificate does not carry the node name the peer announced. Nonzero means a node's certificate is missing its name; see TLS between cluster nodes. |
scramdb_cluster_duplicate_connections_refused_total | counter | Peer connections refused because this node already accepted one from the same peer on the same lane. A peer that opens a lane twice at once (two seed addresses for one node, or a reconnect racing the connection it replaces) keeps the first and drops the second, so a few are normal; a steady climb means a peer keeps reopening a lane it already holds. |
scramdb_cluster_transport_threads | gauge | Threads the cluster transport runs for its listeners, dials, handshakes and peer connections. |
scramdb_cluster_transport_thread_start_failures_total | counter | Threads the cluster transport could not start. |
Traffic lanes
Every pair of nodes talks over three connections, one per traffic lane, each on its own port: control (membership, consensus, the commit protocol, forwarded DDL and lock calls, shuffle fetches), interactive (replies to forwarded point reads) and bulk (query data exchange, fragment dispatch with its broadcast join side, snapshot transfer, the rows a cluster COPY stages). A message larger than [cluster.transport] chunk_size travels as chunks, and a lane's connection takes its streams in turn a chunk at a time. Every stream has its own window of its lane, so a subsystem that stops reading holds only its window and stops only its own senders.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_lane_bytes_sent_total | counter | Bytes written to peers, by lane. |
scramdb_cluster_lane_frames_sent_total | counter | Frames written to peers, by lane. |
scramdb_cluster_lane_bytes_received_total | counter | Bytes read from peers, by lane. |
scramdb_cluster_lane_frames_received_total | counter | Frames read from peers, by lane. |
scramdb_cluster_lane_chunks_sent_total | counter | Chunks of large messages written to peers, by lane. |
scramdb_cluster_lane_chunks_received_total | counter | Chunks of large messages read from peers, by lane. |
scramdb_cluster_lane_messages_reassembled_total | counter | Large messages reassembled from chunks, by lane. |
scramdb_cluster_lane_credit_frames_sent_total | counter | Frames that returned stream credit to a peer, by lane. |
scramdb_cluster_lane_connections | gauge | Open peer connections, by lane. |
scramdb_cluster_lane_older_protocol_connections | gauge | Open peer connections that speak an older cluster protocol version, by traffic lane. |
scramdb_cluster_lane_protocol_fallbacks_total | counter | Dials made again with the older cluster protocol version a peer asked for, by traffic lane. |
scramdb_cluster_lane_stream_window_bytes | gauge | Receive window granted to each stream on the newest connection, by lane: the lane's size shared by the peers then connected. |
scramdb_cluster_lane_queued_bytes | gauge | Bytes held by each traffic queue, by lane and direction: send (queued to send) or receive (received and not yet consumed). |
scramdb_cluster_lane_capacity_bytes | gauge | Size of each traffic queue, by lane and direction. |
scramdb_cluster_lane_backpressure_total | counter | Sends refused because a traffic queue or a stream window was full, by lane and stream (membership, metadata_consensus, shard_consensus, commit_protocol, ddl_forward, lock_forward, shuffle_fetch, point_read, exchange, fragment_dispatch, snapshot_transfer, copy_staging, other). |
scramdb_cluster_lane_drain_waits_total | counter | Sends that waited for a full traffic queue to drain instead of failing. |
scramdb_cluster_lane_drain_wait_seconds | histogram | Time a send waited for a full traffic queue to drain. |
scramdb_cluster_lane_connection_threads | gauge | Reader and writer threads of peer connections, by lane: two per open connection. |
backpressure_total on the bulk lane is flow control doing its job under a large query. On the control lane it should stay at zero: a nonzero shard_consensus or metadata_consensus count means consensus traffic found its queue full, and each such event is also logged once.
Raft group health and recovery
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_group_quarantines_total | counter | Raft group storage directories quarantined: at boot for deeper corruption (a bad header, a mid-prefix CRC failure, an undecodable hard state), or at runtime when a group stopped for a reason a restart cannot fix. Does not count a benign torn-tail truncation; that's a row below. |
scramdb_cluster_quarantined_groups | gauge | Groups currently quarantined, awaiting a membership-replace rejoin. This is the line to alert on. |
scramdb_cluster_group_rejoins_total | counter | Quarantined groups that completed their membership-replace rejoin. |
scramdb_cluster_term_changes_total | counter | Metadata group (group0) raft term changes this node has observed: an election, a term bump from a higher-term message, or a step-down. Node-local, not a cluster-wide total: each node counts its own view of the same election. |
scramdb_cluster_group_actor_restarts_total | counter | Shard groups this node restarted in place after a transient stop, within a bounded restart budget; one that keeps stopping is quarantined instead. |
scramdb_cluster_group_disk_full_resumes_total | counter | Shard groups this node resumed in place after a full disk, once a free-space recheck found room again. Never counted as a quarantine or a rejoin: the group's storage was never touched. |
scramdb_cluster_torn_tail_truncations_total | counter | Benign torn-tail truncations at raft log open: the normal crash-recovery path, made visible instead of silent. |
scramdb_cluster_quarantined_groups is restart-honest. It is never incremented or decremented as events happen; every host-reconcile round recomputes it from scratch, from the actual set of groups gated right now, and overwrites the gauge with that count. A crash that loses an in-flight update, or a process restart mid-recovery, can't leave this gauge stuck on a stale nonzero reading or silently reset to a wrong zero: the very next reconcile round always reflects what's really quarantined at that moment, never a running total of past events.
Consensus threads
Every replication group and the metadata group of a node run on one pool of consensus threads ([cluster.consensus] threads), never a thread per group. A group runs one step at a time and hands its thread back, so a slow group holds one thread and never another group.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_consensus_pool_threads | gauge | Threads of the pool that drives the consensus groups. |
scramdb_cluster_consensus_pool_queued | gauge | Consensus groups and cluster tasks waiting for a consensus thread. Near zero on a healthy node; a value that stays high means the threads cannot keep up. |
scramdb_cluster_consensus_pool_panics_total | counter | Consensus groups and cluster tasks that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log. |
scramdb_cluster_consensus_pool_idle_wakes_total | counter | Times an idle consensus thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer. |
scramdb_cluster_consensus_mailbox_refused_total | counter | Messages a consensus group refused because its queue was full or closed. |
Exchange threads
The distributed query work other nodes send a node (their fragments, the rows arriving, and the results sent back) runs on its own pool of exchange threads ([cluster.exchange] threads), never on the consensus threads and never on the threads that run queries.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_exchange_pool_threads | gauge | Threads of the pool that runs distributed query work sent by other nodes. |
scramdb_cluster_exchange_pool_queued | gauge | Distributed query tasks waiting for an exchange thread. Near zero on a healthy node; a value that stays high means the threads cannot keep up. |
scramdb_cluster_exchange_pool_panics_total | counter | Distributed query tasks that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log. |
scramdb_cluster_exchange_pool_idle_wakes_total | counter | Times an idle exchange thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer. |
Cluster services
Requests other nodes send this node's cluster services (lock and DDL forwarding, snapshot transfer, query fragments and shuffle fetches) are served on engine threads, each service with its own bound on the requests it serves at once.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_stream_requests_total | counter | Requests from other nodes this node's cluster services started serving. |
scramdb_cluster_stream_requests_in_flight | gauge | Requests from other nodes this node's cluster services are serving now. |
scramdb_cluster_stream_requests_waited_total | counter | Requests that waited for a free place because their service was at its bound. |
scramdb_cluster_stream_frames_refused_total | counter | Frames dropped because their service had stopped or its queue was full. |
Cluster-wide LOCK TABLE grants and transaction advisory locks are released at the end of the transaction or the session that holds them, a session that ends with its transaction open included:
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_lock_grants_released_total | counter | Cluster-wide lock grants given back at a transaction or session end or a savepoint rollback. |
scramdb_cluster_lock_grant_release_failures_total | counter | Cluster-wide lock grants whose release at the end of a transaction or session failed. |
scramdb_cluster_lock_detached_releases_total | counter | Sessions that ended holding cluster-wide lock grants, released after they closed. |
scramdb_cluster_lock_wait_withdrawals_total | counter | Cluster-wide lock waits that ended without a grant and withdrew their registered wait. |
scramdb_cluster_lock_wait_withdrawal_failures_total | counter | Cluster-wide lock waits whose registered wait could not be withdrawn. |
scramdb_cluster_lock_wait_registration_failures_total | counter | Cluster-wide lock waits left unregistered, which end at their lock timeout when they cannot be granted. |
Group commit and durability
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_group_commit_ops_total | counter | Raft ops served, summed across every group-commit wake-up. Divide by group_commit_wakeups_total for the achieved batch size. A lone proposal (nothing to batch with) contributes 1 op and 0 wake-ups, so an idle system reports no batching instead of a fake 1.0. |
scramdb_cluster_group_commit_wakeups_total | counter | Wake-ups that served more than one op, i.e. wake-ups that actually batched. |
scramdb_cluster_group_commit_max_batch | gauge | The largest batch any wake-up has served so far, a high-water mark. Pinned at your configured cap says "raise the cap"; comfortably under it says the batching window is already draining faster than proposals arrive. |
scramdb_cluster_durability_flushes_total | counter | Durability flushes, one per fsync. Divide into group_commit_ops_total for fsyncs-per-commit, the number batching exists to lower. |
scramdb_cluster_log_compactions_total | counter | Raft log compactions performed. |
scramdb_cluster_log_entries_compacted_total | counter | Raft log entries reclaimed by compaction, summed. |
Zero log_compactions_total is normal on a cluster whose log never reaches the compaction trigger. A flat zero on a cluster that's otherwise busy (group_commit_ops_total climbing) is not normal: compaction is waiting, either for a state snapshot that keeps failing (state_snapshot_failures_total or state_snapshots_abandoned_total climbing: a full disk, or a [cluster.log_compaction] memory_bytes budget too small for the group's state) or for a replica that is behind but still inside the lag limit. A replica lagging further than replica_lag_limit_entries no longer holds the log: when it returns it is repaired from a state snapshot, so a dead replica bounds the log instead of growing it forever. The waiting case also logs on its own, but the metric is what you'd wire an alert to; see Alerting on cluster metrics below.
Shared raft log
Every shard group a node hosts writes its replication log into the node's one shared raft log: one writer makes a whole batch of appends from every group durable with one data sync, so a transaction that touches several groups costs the node one sync, not one per group. Its segment files are reused a whole segment at a time once nothing in them is needed; [cluster.consensus] log_segment_size sets their size (by default a share of the data volume). The metadata group keeps its own log.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_shared_log_drains_total | counter | Batches the writer of the node's shared raft log wrote, each ending in one sync. |
scramdb_cluster_shared_log_syncs_total | counter | Data syncs of the node's shared raft log. |
scramdb_cluster_shared_log_group_syncs_total | counter | Groups whose writes a sync of the node's shared raft log covered, summed over its syncs. Divided by syncs_total, the mean groups one sync made durable. |
scramdb_cluster_shared_log_records_total | counter | Records written to the node's shared raft log. |
scramdb_cluster_shared_log_bytes_written_total | counter | Bytes written to the node's shared raft log. |
scramdb_cluster_shared_log_copied_bytes_total | counter | Entry payload bytes copied into write buffers of the node's shared raft log; larger payloads are written in place. |
scramdb_cluster_shared_log_truncations_total | counter | Truncate records written for a replica's conflicting log suffix. |
scramdb_cluster_shared_log_segments_created_total | counter | Segment files the node's shared raft log started. |
scramdb_cluster_shared_log_segments_removed_total | counter | Segment files of the node's shared raft log removed once nothing in them was needed. |
scramdb_cluster_shared_log_dir_sync_failures_total | counter | Failed directory syncs after the node's shared raft log removed segments; each is retried at the next write. |
scramdb_cluster_shared_log_compactions_requested_total | counter | Groups asked to compact their log because they held the oldest segment of the node's shared raft log. |
scramdb_cluster_shared_log_relocated_records_total | counter | Records rewritten at the head of the node's shared raft log so an old segment could be removed. |
scramdb_cluster_shared_log_relocated_bytes_total | counter | Bytes rewritten at the head of the node's shared raft log so an old segment could be removed. |
scramdb_cluster_shared_log_sync_failures_total | counter | Failed writes or syncs of the node's shared raft log; the first one stops every group on the node. |
scramdb_cluster_shared_log_torn_bytes_total | counter | Bytes of an unfinished write cut from the end of the node's shared raft log at start. |
scramdb_cluster_shared_log_migrated_groups_total | counter | Groups whose own raft log was moved into the node's shared raft log. |
scramdb_cluster_shared_log_segments | gauge | Segment files of the node's shared raft log. |
scramdb_cluster_shared_log_bytes | gauge | Bytes of the node's shared raft log on disk. |
scramdb_cluster_shared_log_live_bytes | gauge | Bytes of the node's shared raft log still needed by a group. |
scramdb_cluster_shared_log_groups | gauge | Groups with records in the node's shared raft log or a store open on it. |
scramdb_cluster_shared_log_stopped | gauge | 1 once a failed write or sync stopped the node's shared raft log, else 0. |
scramdb_cluster_shared_log_segment_bytes | gauge | The segment size of the node's shared raft log in force. |
scramdb_cluster_shared_log_scan_milliseconds | gauge | How long the node's shared raft log took to read its segments at start. |
scramdb_cluster_shared_log_sync_seconds | histogram | Time one data sync of the node's shared raft log took. |
scramdb_cluster_shared_log_drain_seconds | histogram | Time from the start of a batch's write to its sync returning. |
shared_log_stopped at 1 is the one to page on: a write or sync of the log failed, every shard group on the node stopped rather than trust a disk whose state is no longer known, and the node must be restarted (nothing is retried). shared_log_bytes far above shared_log_live_bytes while compactions_requested_total climbs means an idle group holds the oldest segment; the log asks it to compact and then moves its few live records forward, which relocated_bytes_total counts.
State snapshots and log compaction
A data group compacts its replication log behind a snapshot of its commit state: the open transactions' locks and staged writes, the recent versions of every row it wrote, and its read and commit watermarks, taken at one applied log position. A restart restores the newest snapshot and replays only the log above it; a replica that fell behind the compaction point installs the leader's snapshot and rebuilds the rows it missed from it instead of copying the whole group. The cluster metadata log compacts the same way. When a group compacts, how far a replica may lag and how much memory snapshots may hold are set under [cluster.log_compaction]; each count adapts to the group's own snapshot and entry sizes unless you pin it.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_state_snapshots_taken_total | counter | State snapshots taken for a log compaction. |
scramdb_cluster_state_snapshot_failures_total | counter | State snapshots that could not be taken (an I/O error, a full disk). |
scramdb_cluster_state_snapshots_abandoned_total | counter | State snapshots given up to stay within their memory budget. |
scramdb_cluster_state_snapshot_last_bytes | gauge | Size of the most recent state snapshot taken on this node. |
scramdb_cluster_state_snapshot_bytes_total | counter | Bytes of every state snapshot taken. |
scramdb_cluster_state_snapshot_duration_seconds | histogram | Time from the start of a state snapshot to its completion. A snapshot is written in small steps between applied batches, so the group keeps applying while it runs. |
scramdb_cluster_state_snapshot_step_seconds | histogram | Time one of those steps held a group's apply. |
scramdb_cluster_state_snapshot_memory_bytes | gauge | Memory held right now by state snapshots being taken, sent or received. |
scramdb_cluster_state_snapshot_memory_limit_bytes | gauge | Memory they may hold at once (memory_bytes, or the derived default). |
scramdb_cluster_state_snapshot_restores_total | counter | Groups restored from a state snapshot at startup. |
scramdb_cluster_state_snapshot_missing_total | counter | Groups found at startup with a compacted log and no state snapshot to restore from. Zero in a healthy deployment; anything else means that replica needs a full copy. |
scramdb_cluster_state_snapshot_restore_seconds | histogram | Time to restore one group's state at startup. |
scramdb_cluster_log_entries_replayed_total | counter | Log entries replayed at startup on top of the restored state. |
scramdb_cluster_log_torn_tail_replays_total | counter | Groups found at startup with their log cut below what they had applied (a crash, or a disk that lost part of the log). The group takes the cut entries from its peers again and replays them without applying their rows a second time. |
scramdb_cluster_log_compactions_awaiting_snapshot_total | counter | Log compactions held back until a state snapshot covers the open transactions. |
scramdb_cluster_log_compactions_past_lagging_replica_total | counter | Log compactions that passed a replica lagging beyond the lag limit. That replica is repaired from the snapshot when it returns. |
scramdb_cluster_state_snapshot_installs_total | counter | State snapshots from a leader installed on this node. |
scramdb_cluster_state_snapshot_installs_incomplete_total | counter | Snapshot installs whose skipped rows come from a full copy (every install: a snapshot carries no row). |
scramdb_cluster_state_snapshot_installs_resumed_total | counter | Full copies resumed at startup after a restart interrupted them. |
scramdb_cluster_state_snapshot_installs_superseded_total | counter | Pending full copies replaced by a newer install before they completed. |
scramdb_cluster_state_snapshot_install_intent_failures_total | counter | Snapshot installs left unapplied because their recovery record could not be written. The install is tried again; a rate points at the data volume. |
scramdb_cluster_state_snapshot_fetches_total | counter | State snapshots fetched from a peer, in bounded chunks. |
scramdb_cluster_state_snapshot_fetch_failures_total | counter | State snapshot fetches from a peer that failed; the fetch retries with a growing pause. |
scramdb_cluster_state_snapshot_fetched_bytes_total | counter | Bytes of state snapshots fetched from peers. |
scramdb_cluster_state_snapshot_chunks_served_total | counter | State snapshot chunks served to peers. |
scramdb_cluster_replica_copies_started_total | counter | Full copies of a shard group's rows this node started after a snapshot install. |
scramdb_cluster_replica_copies_completed_total | counter | Full copies of a shard group's rows this node completed. |
scramdb_cluster_replicas_awaiting_rows | gauge | Shard group replicas on this node waiting for a full copy; they serve no read. A value that stays above zero means a copy cannot finish: check replica_copy_attempt_failures_total here and replica_copy_build_failures_total on the group's voters. |
scramdb_cluster_replica_copy_attempt_failures_total | counter | Attempts to copy a shard group's rows from a peer that failed and were retried. |
scramdb_cluster_replica_copy_rows_total | counter | Rows this node received in full copies of shard groups. |
scramdb_cluster_replica_copy_bytes_total | counter | Bytes this node received in full copies of shard groups. |
scramdb_cluster_replica_copy_seconds | histogram | Time from a snapshot install to its full copy being complete. |
scramdb_cluster_replica_copies_built_total | counter | Full copies of a shard group's rows this node built for a peer. |
scramdb_cluster_replica_copy_build_failures_total | counter | Full copies of a shard group's rows this node could not build for a peer. |
scramdb_cluster_replica_copy_chunks_served_total | counter | Chunks of full copies this node served to peers. |
scramdb_cluster_metadata_log_compactions_total | counter | Cluster metadata log compactions. |
scramdb_cluster_metadata_state_restores_total | counter | Cluster metadata states restored or installed from a state snapshot. |
scramdb_cluster_log_compaction_trigger_entries | gauge | Retained log entries at which a data group last decided to compact. |
scramdb_cluster_replica_lag_limit_entries | gauge | Log entries a replica may lag before it stops holding the leader's log. |
scramdb_cluster_log_compaction_trigger_source | gauge | Why the trigger has its value: 0 configured, 1 the floor (nothing measured yet, or measured below it), 2 measured from the group's snapshot and entry sizes, 3 the ceiling. |
Cluster archive
With [storage.wal.archive] enabled, each shard group's leader ships the group's committed log, and the metadata group's leader the metadata log, to the archive destination, for cluster restore. Each segment lands under a key only one leader can create, so a leader that lost its leadership can never overwrite what the new one shipped, and once it finds the group archived in a newer term it stops archiving that group.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_archive_segments_total | counter | Log segments this node shipped to the cluster archive. |
scramdb_cluster_archive_entries_total | counter | Log entries this node shipped to the cluster archive. |
scramdb_cluster_archive_bytes_total | counter | Bytes of log segments this node shipped to the cluster archive. |
scramdb_cluster_archive_fenced_total | counter | Log segments refused because another leader had already archived that log position. |
scramdb_cluster_archive_superseded_total | counter | Groups this node stopped archiving because a leader of a newer term archives them. |
scramdb_cluster_archive_bases_total | counter | Group bases (a state image and a full copy of the rows) this node shipped. |
scramdb_cluster_archive_images_total | counter | Group state images this node shipped alone, between bases. |
scramdb_cluster_archive_gaps_total | counter | Times a group's log was compacted past the archive and no base could be shipped. |
scramdb_cluster_archive_errors_total | counter | Failed attempts to write to the cluster archive. |
scramdb_cluster_archive_pruned_total | counter | Archive objects the retention pass removed. |
scramdb_cluster_archive_time_samples_total | counter | Time index samples this node shipped to the cluster archive. |
scramdb_cluster_archive_fence_asks_total | counter | Fences this node asked of idle groups behind its newest acknowledged commit. |
scramdb_cluster_archive_cycles_total | counter | Passes this node's archiver made over its groups and its time index. |
scramdb_cluster_archive_lag_entries | gauge | Committed log entries of the groups this node leads that are not archived yet. |
scramdb_cluster_archive_groups | gauge | Groups whose log this node ships now. |
scramdb_cluster_archive_ship_seconds | histogram | Time to ship one log segment to the cluster archive. |
A few fenced_total around a leader change are expected: the old leader's last attempt is refused because the new leader already archived that position. gaps_total above zero means a group's history has a hole a restore cannot cross (the group compacted its log past what was archived and no base could be shipped): a restore to a point inside that hole is refused, and points after the group's next base restore again. lag_entries growing without bound means the destination cannot keep up. cycles_total counts the archiver's passes, and fence_asks_total the requests it made to a group that stood behind its newest acknowledged commit so that the latest restorable point reaches that commit; the second stays flat on a cluster that is idle (Cluster Restore).
Distributed transaction commit
What the commit protocol of distributed transactions ([cluster.dilith] in the config file) did on this node.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_dilith_commits_total | counter | Transactions committed. |
scramdb_cluster_dilith_aborts_total | counter | Transaction attempts aborted, by reason: conflict (another transaction held a pending write or a read fence the vote needed), stale_read (a version above a read position, or a read below the retention floor), outdated_vote (older than the group's memory of decisions), wait_die (the younger of two transactions on one row), recovery (a recovery found a group that never voted). |
scramdb_cluster_dilith_commit_latency_seconds | histogram | Time from a commit request to its answer. |
scramdb_cluster_dilith_vote_round_seconds | histogram | Time from sending a transaction's votes to its last vote answer. |
scramdb_cluster_dilith_votes_appended_total | counter | Vote entries appended by group leaders. |
scramdb_cluster_dilith_precheck_refusals_total | counter | Votes a group leader refused before appending anything. |
scramdb_cluster_dilith_parks_total | counter | Requests parked at a group behind an undecided transaction. |
scramdb_cluster_dilith_write_free_certified_total | counter | Transactions that wrote nothing certified without a log entry. |
scramdb_cluster_dilith_fences_appended_total | counter | Fence entries appended for analytical cuts. |
scramdb_cluster_dilith_fences_avoided_total | counter | Analytical cuts a group already covered without a fence entry. |
scramdb_cluster_dilith_recoveries_started_total | counter | Recoveries started for undecided transactions. |
scramdb_cluster_dilith_recoveries_decided_total | counter | Recoveries that reached a verdict. |
scramdb_cluster_dilith_recoveries_behind_waiter_total | counter | Recoveries started by a request that waited too long behind an undecided transaction. |
scramdb_cluster_dilith_recoveries_abandoned_vote_total | counter | Recoveries started for votes pending too long with nobody waiting. |
scramdb_cluster_dilith_recoveries_handed_over_total | counter | Recoveries handed to the group leader that holds the pending vote. |
scramdb_cluster_dilith_recoveries_from_record_total | counter | Recoveries decided from the leader's own record of the transaction. |
scramdb_cluster_dilith_status_probes_total | counter | Status queries for transactions whose outcome a client did not receive. |
scramdb_cluster_dilith_sweeps_total | counter | Sweep entries appended to decide transactions that never voted at a group. |
scramdb_cluster_dilith_sweeps_refused_total | counter | Sweeps refused because they were older than the group's memory of decisions. |
scramdb_cluster_dilith_sweeps_resent_total | counter | Sweeps sent again with a fresher decision count. |
scramdb_cluster_dilith_votes_resent_total | counter | Vote requests sent again to groups that had not answered, one per group. |
scramdb_cluster_dilith_outcomes_resent_total | counter | Transaction outcomes sent again to groups that had not acknowledged them, one per group. |
scramdb_cluster_dilith_aborts_without_record_total | counter | Abort outcomes acknowledged without recording anything at a group holding no record of the transaction. |
scramdb_cluster_dilith_decisions_forgotten_total | counter | Decisions forgotten once outside the decision window. |
scramdb_cluster_dilith_staged_vote_rounds_total | counter | Vote rounds that asked the groups likely to refuse before the others. |
scramdb_cluster_dilith_copy_stage_entries_total | counter | COPY stage entries applied on this node's replicas. |
scramdb_cluster_dilith_copy_stages_dropped_total | counter | COPY stages dropped on this node's replicas without a commit. |
scramdb_cluster_dilith_copy_stage_vote_refusals_total | counter | Votes refused because the COPY rows they name were not all staged or conflicted. |
scramdb_cluster_dilith_copy_stages_collected_total | counter | COPY stages a group leader on this node dropped after their owner stopped renewing them. |
scramdb_cluster_dilith_copy_stages_refused_full_total | counter | COPY stage entries a group leader on this node refused because its staging area had no room. |
scramdb_cluster_dilith_copy_stages_refused_stale_total | counter | COPY stage entries a group leader on this node sent back to be checked again because the table moved on since their rows were checked. |
scramdb_cluster_dilith_owed_commits_collected_total | counter | Commits whose discharge was lost that a group leader on this node collected. |
scramdb_cluster_dilith_block_verdicts_released_total | counter | Commits of this node's transaction blocks whose verdict was held until the block's end learned the outcome and released it. |
scramdb_cluster_dilith_block_verdicts_held | gauge | Commits on this node's replicas held for their transaction block's end, which no age collects. |
scramdb_cluster_dilith_table_entries | gauge | Entries each commit protocol table on this node holds, by table. |
scramdb_cluster_dilith_owed_commits | gauge | Commits on this node's replicas not yet discharged. |
scramdb_cluster_dilith_pending_votes | gauge | Votes accepted on this node's replicas and not yet decided. |
scramdb_cluster_dilith_recent_verdicts | gauge | Recent transaction outcomes this node remembers to answer recoveries. |
scramdb_cluster_dilith_discharges_queued | gauge | Discharge notices waiting for their batch to be sent. |
scramdb_cluster_dilith_refusing_groups | gauge | Groups that refused this node's votes recently. |
scramdb_cluster_dilith_state_bytes | gauge | Memory the commit protocol's tables hold on this node, by part: pending_votes (votes accepted and not decided, with their writes and read fences), owed_commits (commits not yet discharged), kept (the decisions and versions the decision and retention windows keep, and COPY stage records), outcomes (outcomes and discharges a group leader holds for its next entry or has in flight), reserved (room reserved for vote appends in flight), waiting (requests parked at a group, and votes waiting for room), scratch (the node's reusable decode and step buffers). |
scramdb_cluster_dilith_state_share_bytes | gauge | Memory the commit protocol's tables may hold on this node ([cluster.dilith] protocol_state_memory). |
scramdb_cluster_dilith_state_peak_bytes | gauge | Most memory the commit protocol's tables and reserved room have held on this node. |
scramdb_cluster_dilith_votes_waiting_for_memory | gauge | Votes waiting on this node for the commit protocol's memory share to have room. |
scramdb_cluster_dilith_memory_parks_total | counter | Votes a group leader on this node held until the commit protocol's memory share had room. |
scramdb_cluster_dilith_memory_past_share_total | counter | Votes a group leader on this node let past a full memory share so the oldest waiting transaction could go on. |
aborts_total{reason="conflict"} and {reason="wait_die"} count contention between transactions on the same rows. A recoveries_started_total that keeps growing on a healthy network means transactions are being left undecided, and is worth a look at the nodes' logs. copy_stages_collected_total rising means COPY statements stop or lose their node while staging; copy_stage_vote_refusals_total rising means a COPY's vote meets concurrent writers of the rows it stages, and the COPY fails with 40001. copy_stages_refused_stale_total counts batches a COPY checked again because its groups moved on while the batch travelled; the COPY goes on. copy_stages_refused_full_total rising means the staging area is too small for the COPYs running (53100). owed_commits_collected_total rising means nodes stop right after answering commits; every such commit is finished by its groups. A transaction block's commit that changed the schema is held on its groups until the block's own end has learned whether it committed, however late that is: block_verdicts_held counts them and block_verdicts_released_total their releases. A held count that keeps growing while releases stay flat means blocks whose COMMIT was answered in doubt cannot learn their outcome yet, usually a node cut off from the cluster; they are released when it can, or when that node leaves the cluster. table_entries shows every table of the commit protocol and what it holds: each one stays bounded by the work in flight.
The commit protocol's tables live in a memory share of their own (state_share_bytes), counted in the process memory budget. A group leader appends a new vote only while the share has room; a vote that finds none waits, oldest transaction first, and is appended as soon as a decision, a discharge or the protocol's windows free room. It is never refused for memory. votes_waiting_for_memory above zero for long, or memory_parks_total climbing, means the share is too small for the transactions in flight: raise [cluster.dilith] protocol_state_memory. memory_past_share_total counts the oldest waiting transaction let through a full share when nothing on the node could free room without it, which is what keeps the cluster moving; its rise says the same.
The commit protocol on this node
The commit protocol runs as one task on the node's consensus threads, one run at a time, and serves every shard group this node holds a replica of.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_dilith_run_seconds | histogram | Time one run of the commit protocol's host took. |
scramdb_cluster_dilith_runs_total | counter | Runs of the commit protocol's host. |
scramdb_cluster_dilith_run_inputs_total | counter | Inputs the commit protocol's host stepped, over all runs. |
scramdb_cluster_dilith_queued | gauge | Inputs waiting for the commit protocol's host, by kind: control, decide, frame, client. |
scramdb_cluster_dilith_step_errors_total | counter | Inputs the commit protocol could not step, by input: apply, message, timer, leadership, append_decided, restored, client, hosting. |
scramdb_cluster_dilith_reply_slots | gauge | Client calls waiting for an answer from the commit protocol. |
scramdb_cluster_dilith_memory_bytes | gauge | Bytes the commit protocol holds on this node, as last measured. |
scramdb_cluster_dilith_appends_in_flight | gauge | Log entries the commit protocol proposed whose decision has not come back. |
scramdb_cluster_dilith_timers | gauge | Timers the commit protocol keeps armed. |
scramdb_cluster_dilith_frames_refused_total | counter | Frames the transport refused to send; the protocol sends again. |
scramdb_cluster_dilith_appends_lost_total | counter | Proposed log entries lost because the group's leader could not take them. |
scramdb_cluster_commits_in_doubt_total | counter | Commits whose outcome was not known when the client stopped waiting. |
scramdb_cluster_dilith_commit_progress_marks_total | counter | Progress marks of commit rounds that reached a waiting client: a vote recorded, or a vote applied on this node. |
scramdb_cluster_dilith_commit_wait_renewals_total | counter | Times a waiting commit kept waiting past its patience because its round had progressed. |
scramdb_cluster_dilith_commits_in_flight | gauge | Commits this node's commit protocol drives whose outcome is not decided yet, as last measured. |
scramdb_cluster_dilith_commits_abandoned | gauge | Commits still in flight here whose client stopped waiting for the outcome, as last measured. |
scramdb_cluster_dilith_calls_abandoned_total | counter | Calls to the commit protocol whose caller stopped waiting, forgotten with what waited to answer them. |
scramdb_cluster_dilith_outcome_waits_total | counter | Waits for the commits already answered to reach every group. |
scramdb_cluster_dilith_outcome_wait_timeouts_total | counter | Waits for commit outcomes that gave up at their deadline with a group yet to acknowledge. |
scramdb_cluster_dilith_outcomes_owed | gauge | Commits answered to their client whose outcome some group has yet to acknowledge, as last measured. |
scramdb_cluster_dilith_stage_rounds_resent_total | counter | Rounds of staged rows sent again because their answers did not come in time. |
scramdb_cluster_dilith_copy_stage_entries_wide_total | counter | COPY stage entries a group leader on this node appended without their keys because its memory share had no room for them. |
scramdb_cluster_dilith_stage_requests_lost_total | counter | Requests of staged rows asked again at once because their shard group changed leader. |
scramdb_cluster_dilith_host_stopped | gauge | 1 when the commit protocol on this node stopped after a failure. |
scramdb_cluster_dilith_hosted_groups | gauge | Shard groups the commit protocol hosts on this node. |
scramdb_cluster_dilith_groups_hosted_total | counter | Shard groups the commit protocol started hosting on this node. |
scramdb_cluster_dilith_groups_released_total | counter | Shard groups the commit protocol stopped hosting on this node. |
scramdb_cluster_dilith_decide_requests_dropped_total | counter | Shard group requests dropped because the group is not hosted here or started again. |
scramdb_cluster_dilith_node_number | gauge | This node's number in transaction ids, 0 until the cluster gave it one. |
scramdb_cluster_dilith_node_number_claims_total | counter | Requests this node made to the cluster for its node number. |
scramdb_cluster_dilith_node_number_waits_total | counter | Commit attempts that waited for this node's node number instead of starting at once. |
dilith_host_stopped at 1 means the node can no longer commit transactions and must be restarted; every waiting client was answered with an error. A commits_in_doubt_total that grows means clients stopped waiting before their commit's outcome was known: the commit may still have happened. A commit waits as long as its round keeps progressing ([cluster.transactions]): commit_progress_marks_total counts the steps that kept clients waiting and commit_wait_renewals_total the waits that went past their patience because of them, so renewals that grow while commits_in_doubt_total stays flat are large or slow commits that still finish, and in-doubt commits that grow without marks are rounds that stalled. stage_rounds_resent_total grows when a node sending a COPY's rows, or a transaction's rows too large for one commit message, lost a round to a leader change it could not see or to a node that stopped; one that grows while no leader changed means rounds are taking far longer than the rounds before them.
Committed entries through the commit protocol
Each shard group hands its committed log entries to the commit protocol, which decides the rows they install, and hands the protocol's own entries to the group's log.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_dilith_decide_requests_total | counter | Batches of committed entries the data groups handed to the commit protocol. |
scramdb_cluster_dilith_decide_entries_total | counter | Committed entries the data groups handed to the commit protocol. |
scramdb_cluster_dilith_decided_rows_total | counter | Rows the commit protocol decided for the data groups to install. |
scramdb_cluster_dilith_decide_outstanding | gauge | Requests of the data groups the commit protocol has not answered yet. |
scramdb_cluster_dilith_decide_seconds | histogram | Time from handing committed entries to the commit protocol to receiving their rows. |
scramdb_cluster_dilith_decide_failures_total | counter | Data groups stopped because the commit protocol could not decide an entry of theirs. |
scramdb_cluster_dilith_leadership_reports_total | counter | Leader changes of the data groups reported to the commit protocol. |
scramdb_cluster_dilith_proposal_batches_total | counter | Batches of commit protocol entries handed to a data group's log. |
scramdb_cluster_dilith_proposal_batches_refused_total | counter | Batches of commit protocol entries a data group's log could not take. |
Applying committed writes
A committed entry is applied in two steps on the node's apply threads ([cluster.apply] threads), never on the threads that run consensus, replication and client connections. First it is decided: the commit protocol steps it in memory, in log order (the session that committed it already has its answer, given once every group the transaction touched held a durable yes vote). Then its rows are written, and one data log sync per batch makes a whole batch of them durable before the group's applied position moves. Reads wait for that applied position, so a row is never read before it is applied and synced. On a cluster node the consensus threads keep a share of the cores of their own ([cluster.apply] reserve_io_cores), so heavy queries cannot delay them.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_apply_decide_to_durable_seconds | histogram | Time from deciding a batch of committed entries to its effects being durable: how far the applied position trails the answers sessions already have. |
scramdb_cluster_apply_effect_batches_total | counter | Batches of committed entries whose effects were made durable. |
scramdb_cluster_apply_effect_entries_total | counter | Committed entries whose effects were made durable. Divide by apply_effect_batches_total for the mean batch size. |
scramdb_cluster_apply_data_syncs_total | counter | Data log syncs the apply path requested for those effects. Divide by apply_effect_batches_total: one per batch (zero for a batch with nothing to write) is the design; more means a write path is syncing on its own. |
scramdb_cluster_apply_last_batch_entries | gauge | Entries in the most recent effect batch. |
scramdb_cluster_apply_last_batch_data_syncs | gauge | Data log syncs the most recent effect batch requested. |
scramdb_cluster_apply_decided_batches_waiting | gauge | Decided batches waiting for their effects. Climbing steadily means the disk or the apply threads cannot keep up; each group's backlog stays bounded by its apply mailbox either way. |
scramdb_cluster_apply_stops_total | counter | Groups whose apply stopped on an error. Except a full disk (the row below), such a replica is quarantined and rebuilt from the leader's own snapshot; see the logs for the reason. |
scramdb_cluster_apply_stops_disk_full_total | counter | The subset of scramdb_cluster_apply_stops_total caused by a write meeting a full disk on this node: the node's own log names the index it stopped at. Its storage is never quarantined; free space on that node and it resumes on its own from that same index (scramdb_cluster_group_disk_full_resumes_total), no restart needed. |
scramdb_cluster_apply_pool_threads | gauge | Apply threads running. |
scramdb_cluster_apply_pool_queued | gauge | Groups waiting for an apply thread. |
scramdb_cluster_apply_pool_idle_wakes_total | counter | Times an idle apply thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer. |
scramdb_cluster_apply_effect_slots_reused_total | counter | Installed rows whose effect was built in a slot an apply thread keeps, its key buffer reused. |
scramdb_cluster_apply_effect_slots_fresh_total | counter | Installed rows whose effect needed a new slot, so its key buffer allocated. Past warm-up it grows only when a batch is larger than any before it on that thread. |
scramdb_cluster_apply_effect_images_decoded_total | counter | Installed rows whose image decode had to allocate memory of its own. |
scramdb_replica_apply_delete_batches_total | counter | Batches of deleted rows the replica apply removed from storage in one call. |
scramdb_cluster_io_runtime_busy_microseconds_total | counter | Time the consensus and connection threads spent running work. Its rate divided by io_runtime_workers is their utilization. |
scramdb_cluster_io_runtime_workers | gauge | Consensus and connection threads. |
Replication flow control
A leader streams entries to each follower inside a bounded in-flight window (max_inflight_bytes and max_inflight_msgs in [cluster.consensus]), which by default adapts under that ceiling to the bandwidth-delay product measured toward each follower. Each follower's stream is in one of three states: probe (finding where the follower's log ends, one message outstanding), replicate (streaming inside the window) or snapshot (a snapshot is outstanding). The counters and gauges below sum every group this node leads.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_replication_transitions_total | counter | Follower stream state changes, labelled state (probe, replicate, snapshot) and cause: into probe by leader_start, reject (the follower refused an append or a snapshot), connection_loss, stall (two heartbeat checks with entries in flight and no answer) or incarnation (the follower answered as a newer start of the replica than the leader had heard, so what the leader believed it held is void and its stream restarts); into replicate by acknowledged or snapshot_installed; into snapshot by behind_compaction. |
scramdb_cluster_replication_sends_paused_total | counter | Sends held back because a follower's in-flight window was full. |
scramdb_cluster_replication_append_messages_total | counter | AppendEntries messages sent to followers, heartbeats included. |
scramdb_cluster_replication_append_entries_total | counter | Log entries sent to followers. |
scramdb_cluster_replication_append_bytes_total | counter | Log entry bytes sent to followers. Divided by the bytes proposed and by the follower count, this is the copies of each entry the network carried: about one in steady operation. |
scramdb_cluster_replication_rejects_total | counter | Append and snapshot rejects received from followers. |
scramdb_cluster_replication_snapshot_resends_total | counter | Snapshots sent again to a follower that had not answered within the resend backoff. |
scramdb_cluster_replication_incarnation_changes_total | counter | Follower responses that came from a different start of the replica than the one the leader held: one per follower restart, each restarting that follower's stream. A short burst follows a restart that raced a response still in flight. |
scramdb_cluster_replication_unconfirmed_responses_total | counter | Follower responses held back until a round trip the leader began confirms the replica's current start. A few follow every leader election and every follower restart; a sustained rate means a follower never confirms (its responses do not echo what the leader sent). |
scramdb_cluster_replication_incarnations_confirmed_total | counter | Replica starts confirmed by a round trip from the leader: one per follower per leadership, and one more each time a follower restarts. |
scramdb_cluster_replication_inflight_messages | gauge | Unacknowledged AppendEntries messages to followers, right now. |
scramdb_cluster_replication_inflight_bytes | gauge | Unacknowledged log entry bytes sent to followers, right now. |
scramdb_cluster_replication_window_bytes | gauge | The in-flight window toward a peer, summed over the groups this node leads, labelled peer and reason: ceiling (the configured max_inflight_bytes, or not yet measured), measured (twice the measured bandwidth-delay product), floor (two of the stream's own messages) or pinned (adaptive_inflight_window = false). |
scramdb_cluster_replication_round_trip_milliseconds | gauge | The smoothed replication round trip to a peer, from sending an AppendEntries to its acknowledgement (the follower's fsync included), as last measured. |
scramdb_cluster_replication_entries_committed_before_leader_sync_total | counter | Log entries committed on followers' synced copies before the leader's own copy reached disk (leader_sends_before_sync in [cluster.consensus]). |
scramdb_cluster_replication_leader_unsynced_entries | gauge | Log entries the groups this node leads have sent but not yet synced to their own disk. |
scramdb_cluster_replication_lag_entries | gauge | Log entries a peer has not acknowledged, summed over the groups this node leads, labelled peer. A peer whose lag keeps growing is falling behind. |
scramdb_cluster_replication_leader_sync_seconds | histogram | Time from a leader writing new log entries to their sync to disk. |
scramdb_cluster_election_timeout_min_milliseconds | gauge | Lower bound of the election timeout in force (0 until a replication group has started on this node). |
scramdb_cluster_election_timeout_max_milliseconds | gauge | Upper bound of the election timeout in force. |
sends_paused_total climbing together with window_bytes{reason="ceiling"} toward a peer says the ceiling, not the link, is the limit: raise max_inflight_bytes for a long, fast link. Steady growth of transitions_total{cause="stall"} or {cause="reject"} on a healthy network says a follower keeps losing its place and is worth a look at its disk and its connection. A restarted follower shows one transitions_total{cause="incarnation"} and one more incarnations_confirmed_total on the leader; growth of either without restarts says a node keeps restarting or two processes answer as one node.
Consensus core buffers
Consensus reuses its working buffers across steps, so a steady stream of heartbeats, appends and commits allocates nothing for its step output; only a burst above the recent load, or the first load after a quiet spell, needs a fresh allocation, and that spare capacity is handed back again once the burst passes. A received frame is decoded without copying the commands and snapshot data it carries. These figures are for the whole process, summed across it.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_raft_step_vectors_reused_total | counter | Raft step output vectors served from a kept buffer instead of a new allocation. Counted as each vector is taken. |
scramdb_cluster_raft_step_vectors_fresh_total | counter | Raft step output vectors started without a kept buffer, so their first element allocates. Counted as each vector is taken. It stays flat on a steady load and grows when the load keeps more step output in flight than it recently did (a burst, or the first load after a quiet spell). |
scramdb_cluster_raft_step_buffer_bytes | gauge | Bytes the consensus threads keep for Raft step output. |
scramdb_cluster_raft_frame_blob_bytes_shared_total | counter | Command and snapshot bytes decoded from received Raft frames without a copy. |
scramdb_cluster_raft_frame_blob_bytes_copied_total | counter | Command and snapshot bytes copied out of received Raft frames. It stays at zero on a node: every receive path decodes from the frame it was given. |
scramdb_cluster_raft_frames_encoded_total | counter | Cluster messages encoded into a consensus thread's kept frame buffer. |
scramdb_cluster_raft_frame_buffer_growths_total | counter | Times a consensus thread's kept frame buffer grew to hold a message. |
scramdb_cluster_raft_frame_entry_vectors_decoded_total | counter | Entry vectors the Raft frame decoder allocated for received appends, one per append that carries entries. |
Heartbeats and quiet groups
A node sends one heartbeat frame per peer node each heartbeat interval, carrying a line for every group it leads with that peer, instead of one heartbeat per group ([cluster.consensus] coalesced_heartbeats). A frame names its group set by a hash and lists the groups in full only after the set changed. When a group's commit index advances, a read needs confirming or a small append waits, the frame leaves early: at once while the connection to that peer has nothing queued, otherwise after heartbeat_flush_delay. A small append rides inside the frame instead of taking a frame of its own (carry_small_appends), and a follower's reply to an append can ride back inside the answer frame it sends anyway. An idle group (nothing in flight, every member caught up) goes quiet: it stops its heartbeats and its members' election timers, and its peer is refreshed once per election_timeout_min (quiescence). See [cluster.consensus].
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_heartbeat_frames_sent_total | counter | Heartbeat frames this node sent, each covering every group it leads with that peer. |
scramdb_cluster_heartbeat_group_lines_sent_total | counter | Group lines the heartbeat frames of this node carried. |
scramdb_cluster_heartbeat_bytes_sent_total | counter | Bytes of heartbeat and answer frames this node sent. |
scramdb_cluster_heartbeat_full_lists_sent_total | counter | Heartbeat frames that listed every group this node leads with the peer, after that set changed. |
scramdb_cluster_heartbeat_quiet_frames_sent_total | counter | Heartbeat frames with no group line, sent to keep a peer's quiet groups quiet. |
scramdb_cluster_heartbeat_flushes_total | counter | Heartbeat frames sent early because a commit index advanced, a read needed confirming or a small append waited. |
scramdb_cluster_heartbeat_immediate_flushes_total | counter | Early heartbeat frames sent at once because the connection to the peer had nothing queued. |
scramdb_cluster_heartbeat_flush_delay_seconds | histogram | Time from a commit index advancing, or a read or small append waiting, to the heartbeat frame carrying it. |
scramdb_cluster_heartbeat_answers_sent_total | counter | Heartbeat answer frames this node sent to the nodes leading its groups. |
scramdb_cluster_heartbeat_frames_received_total | counter | Heartbeat frames this node received. |
scramdb_cluster_heartbeat_answers_received_total | counter | Heartbeat answer frames this node received. |
scramdb_cluster_heartbeat_round_trip_seconds | histogram | Time from sending a heartbeat frame to a peer to the peer's answer. |
scramdb_cluster_heartbeat_carried_appends_total | counter | Small appends sent inside heartbeat frames instead of frames of their own. |
scramdb_cluster_heartbeat_carried_bytes_total | counter | Bytes of the appends sent inside heartbeat frames. |
scramdb_cluster_heartbeat_append_replies_total | counter | Replies this node's groups sent to appends of the nodes leading them. |
scramdb_cluster_heartbeat_append_reply_frames_total | counter | Frames this node sent for its groups' append replies: a reply sent on its own, or an answer frame carrying replies and no heartbeat answer. |
scramdb_cluster_heartbeat_carried_replies_total | counter | Append replies sent inside heartbeat answer frames instead of frames of their own. |
scramdb_cluster_heartbeat_carried_replies_received_total | counter | Append replies this node received inside heartbeat answer frames. |
scramdb_cluster_heartbeat_send_refused_total | counter | Heartbeat and answer frames the transport refused because its queue was full or the peer was not connected. |
scramdb_cluster_heartbeat_delivery_refused_total | counter | Heartbeat lines a group could not take because its queue was full or it was gone. |
scramdb_cluster_heartbeat_malformed_frames_total | counter | Heartbeat frames dropped because they did not decode. |
scramdb_cluster_heartbeat_stale_views_total | counter | Heartbeat frames whose group-set hash did not match the set this node held, which asks for the full list. |
scramdb_cluster_heartbeat_quiesce_entered_total | counter | Groups this node leads that went quiet: every member caught up and idle, no heartbeats and no timers. |
scramdb_cluster_heartbeat_quiesce_left_total | counter | Quiet groups this node leads that woke up. |
scramdb_cluster_heartbeat_woken_by_set_change_total | counter | Quiet groups on this node woken because the node leading them stopped listing them. |
scramdb_cluster_heartbeat_woken_by_missed_refresh_total | counter | Quiet groups on this node woken because the node leading them missed two refreshes. |
scramdb_cluster_heartbeat_led_groups | gauge | Groups this node leads whose heartbeats its frames carry. |
scramdb_cluster_heartbeat_quiescent_groups | gauge | Groups this node leads that are quiet. |
scramdb_cluster_heartbeat_peers | gauge | Peer nodes this node sends heartbeat frames to or receives them from. |
scramdb_cluster_heartbeat_interval_milliseconds | gauge | The heartbeat interval in force (heartbeat_interval). |
scramdb_cluster_heartbeat_flush_delay_microseconds | gauge | How long an early heartbeat frame waits to gather more groups, in force (heartbeat_flush_delay). |
scramdb_cluster_heartbeat_quiet_refresh_milliseconds | gauge | How often a peer whose groups here are all quiet is refreshed, in force. |
group_lines_sent_total divided by frames_sent_total is how many groups one frame carries: the saving over one heartbeat per group. On an idle cluster quiescent_groups approaches led_groups and the heartbeat traffic falls to one small refresh per peer. woken_by_missed_refresh_total rising on a healthy network means a leader's node stalls; send_refused_total, delivery_refused_total and malformed_frames_total stay at zero on a healthy node.
Read-index forwarding
A replica that is not its group's leader, a follower or a learner, forwards its read-index requests to the leader, which answers them as it answers its own. A forwarded request with no answer within two election timeouts plus the round trip the replica measured to the leader, or whose connection to the leader closed, is refused. An answer that arrives after its request was refused still measures the round trip, so on a slow link the next request waits long enough; forward_late_total rising means a replica's link to its leader is slower than two election timeouts.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_read_index_forwarded_total | counter | Read-index requests this node forwarded to a group's leader. |
scramdb_cluster_read_index_forward_unanswered_total | counter | Forwarded read-index requests that ended without the leader's answer. |
scramdb_cluster_read_index_forward_timed_out_total | counter | Forwarded read-index requests given up after their wait with no answer from the leader. A request the leader's node cannot take is refused at once instead and counted only as unanswered, so a rise here means messages were lost. |
scramdb_cluster_read_index_forward_late_total | counter | Leader answers to forwarded read-index requests that arrived after the request was given up. |
scramdb_cluster_read_index_served_for_peers_total | counter | Read-index requests this node answered as a group's leader for another replica. |
scramdb_cluster_read_index_forward_seconds | histogram | Time from forwarding a read-index request to the leader's answer. |
Requests to the metadata and shard groups
Every request a node sends its metadata group or one of its shard groups has a deadline adapted to the group's election timing. A request the group did not take in time changed nothing and is retried by its caller; a proposal whose answer did not come may still commit, so it is never sent again; a read confirmation is asked again until the deadline. A DDL statement forwarded to the metadata group's leader is bounded the same way. Process-wide, always present, all zero on a healthy cluster:
| Metric | Type | Meaning |
|---|---|---|
scramdb_group0_reply_timeouts_total | counter | Metadata group proposals whose answer did not come in time; each may still commit. |
scramdb_group0_mailbox_timeouts_total | counter | Metadata group requests the group did not take in time; none changed anything. |
scramdb_group0_read_retries_total | counter | Metadata group read confirmations asked again after an answer did not come. |
scramdb_group0_apply_wait_timeouts_total | counter | Waits for this node to apply a committed metadata group entry that ran out of time. |
scramdb_shard_group_reply_timeouts_total | counter | Shard group proposals whose answer did not come in time; each may still commit. |
scramdb_shard_group_mailbox_timeouts_total | counter | Shard group requests the group did not take in time; none changed anything. |
scramdb_shard_group_read_retries_total | counter | Shard group read points asked again after an answer did not come. |
scramdb_shard_group_apply_wait_timeouts_total | counter | Waits for this node to apply a committed shard group entry that ran out of time. |
scramdb_ddl_forward_answer_timeouts_total | counter | Forwarded metadata group proposals whose answer did not come in time. |
scramdb_ddl_forward_in_doubt_total | counter | Forwarded metadata group proposals whose leader could not learn their outcome. |
reply_timeouts_total or apply_wait_timeouts_total rising means a group is slow to commit or a node is slow to apply; mailbox_timeouts_total rising means the group itself is falling behind on incoming requests.
A metadata group proposal this node's answer did not reach in time is sent again under its first attempt's id, so a leader that already applied it once applies it once, never twice, and a resend past its own deadline is refused rather than applied late:
| Metric | Type | Meaning |
|---|---|---|
scramdb_group0_proposal_resends_total | counter | Metadata proposals sent again under the id of their first attempt. |
scramdb_group0_proposal_duplicates_total | counter | Metadata log entries applied as a repeat of a proposal already applied. |
scramdb_group0_proposal_expired_total | counter | Metadata proposals refused because they arrived after their deadline. |
scramdb_group0_proposal_window_ids | gauge | Metadata proposal ids this node keeps to answer a repeat. |
How far a node has come through the metadata group's log. A schema change reaches a node's catalog after its log entry applies there, so the two schema figures differ while the catalog catches up. Across the nodes of a settled cluster each figure is the same; a node whose figure stays below the others is behind on the metadata log or on its catalog:
| Metric | Type | Meaning |
|---|---|---|
scramdb_group0_applied_index | gauge | Highest metadata log entry this node has applied. |
scramdb_group0_schema_changes_applied | gauge | Schema changes this node's metadata log has applied. |
scramdb_catalog_schema_changes_applied | gauge | Schema changes this node's catalog has applied. |
Cluster sequences
A sequence a cluster made is one sequence: every node takes runs of its values from the metadata group and hands them out locally. Process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_sequence_grants_total | counter | Runs of sequence values this node took from the cluster. |
scramdb_cluster_sequence_repositions_total | counter | Sequence positions this node set cluster-wide. |
scramdb_cluster_sequence_failures_total | counter | Requests about a cluster sequence that ended in an error. |
scramdb_cluster_sequence_discarded_values_total | counter | Sequence values this node dropped unused because the sequence was moved. |
grants_total rising much faster than the draw rate divided by a few per second means a sequence is drawn from in tiny bursts; discarded_values_total counts the gaps a setval or RESTART leaves in what a node had already taken.
Sequence values given back
A bulk INSERT that fails part way gives back the sequence values it drew for the rows after the one it failed at, so the next draw continues where PostgreSQL's would. Process-wide, always present, on a single node and a cluster alike:
| Metric | Type | Meaning |
|---|---|---|
scramdb_sequence_rewound_values_total | counter | Sequence values a failed statement gave back to the sequence. |
scramdb_sequence_refused_values_total | counter | Sequence values a failed statement could not give back because another draw followed. |
scramdb_sequence_unrewound_statements_total | counter | Failed statements whose sequence draws could not be tied to their rows, so none were given back. |
A value that stays drawn is a gap, never a duplicate. refused_values_total rising means concurrent sessions draw from the same sequences that failing statements draw from; unrewound_statements_total rising means failing statements draw one sequence for more than one column or in an order the rows cannot be matched to.
INSERT replays
An INSERT whose source raised a data exception (division by zero, an integer out of range, an invalid cast) is run again row by row, so the error reported is the one of the first row that fails, as PostgreSQL reports it. Process-wide, always present, on a single node and a cluster alike:
| Metric | Type | Meaning |
|---|---|---|
scramdb_insert_replay_statements_total | counter | INSERT statements run again row by row after their source raised an error. |
scramdb_insert_replay_earlier_error_total | counter | Replays that raised the error of a row before the one the source failed at. |
scramdb_insert_replay_source_error_total | counter | Replays that kept the source's own error, because no earlier row failed or the replay could not run. |
A statement that succeeds never replays. statements_total rising in steady state means clients send inserts that fail on a data exception; source_error_total close to statements_total means the replay seldom finds an earlier row to blame, and a rise in it with a source larger than the memory budget means the replay could not hold the source.
Read positions
Every read of a cluster table happens at a read position of the commit protocol. A fresh position first has each group's leader confirm the group's latest commit, brings a replica up to it, and waits out any commit still in flight on what is read, so a read never misses a commit already acknowledged to any client, on any node; a reader then serves each group it scans at that position before it reads. For a group this node holds no replica of, the position is asked of the group's leader first, then of its followers, and of its learners last: the leader has nothing to catch up, while a learner may trail its group by any amount, so a learner that has fallen behind does not hold up a statement while a voter of the group can answer. A replica that refuses the position (it does not run the group yet, it is starting, it could not catch up in time; counted on that node by scramdb_cluster_read_position_refusals_total), or fails while still connected, is passed over for the group's next replica. When no replica of such a group is connected to this node (a partition that just healed while its connections form again, a node restarting), the read waits for one to connect (scramdb_cluster_read_position_connect_waits_total). When every replica of the group has refused, as happens for a table created a moment before whose replicas are still starting, the read asks them all again after a pause of a few round trips that grows with each try, and serves the group itself once its own replica of it runs (scramdb_cluster_read_position_reask_waits_total). A replica that has not answered when the read's re-send interval lapses (the interval this node's measured round trip sets) is not waited for alone: the group's next replica is asked beside it, the first answer is taken and the other request is dropped (scramdb_cluster_read_position_hedges_total counts those requests, scramdb_cluster_read_position_asks_in_flight the ones that have waited long enough to ask another replica and have not ended). A replica answers a peer's request within half a read wait, so its reason reaches the asking node while that node still waits. Either way the read waits for at most its read wait, and then fails with 40001 naming the group and, when a replica gave one, what it was waiting for. Nothing here writes to a log unless a group is behind the position, where one fence entry brings it up (scramdb_cluster_dilith_fences_appended_total).
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_read_positions_total | counter | Fresh read positions chosen by this node. |
scramdb_cluster_read_position_seconds | histogram | Time to choose a fresh read position, from the first barrier to the last answer. |
scramdb_cluster_read_position_remote_groups_total | counter | Shard groups whose fresh read position another node took for this node (groups this node holds no replica of). |
scramdb_cluster_read_position_connect_waits_total | counter | Fresh read positions that waited for a replica of a shard group to connect. |
scramdb_cluster_read_position_reask_waits_total | counter | Fresh read positions that asked a shard group again after every replica refused it. |
scramdb_cluster_read_position_refusals_total | counter | Fresh read positions refused here because this node's replica could not serve them. |
scramdb_cluster_read_position_hedges_total | counter | Shard group positions asked of a next replica before an earlier one had answered. |
scramdb_cluster_read_position_asks_in_flight | gauge | Position requests of reads that waited long enough to ask another replica and have not ended. |
scramdb_cluster_read_position_raised_total | counter | Transactions whose read position rose to reach a table they had not read before. |
scramdb_cluster_read_barrier_retries_total | counter | Read barriers asked again because a shard group had no leader to confirm them. |
scramdb_cluster_read_serves_total | counter | Shard groups served at a read position on this node. |
scramdb_cluster_read_serve_seconds | histogram | Time a shard group took to reach a read position on this node. |
scramdb_cluster_read_failures_total | counter | Reads refused because a shard group could not be confirmed or served in time. |
scramdb_cluster_read_registered_ranges_total | counter | Table reads registered with a transaction's commit for validation. |
scramdb_cluster_read_skip_serves_total | counter | Shard groups served for SKIP LOCKED or NOWAIT reads without waiting for a commit. |
scramdb_cluster_read_skip_waits_total | counter | SKIP LOCKED or NOWAIT reads of a shard group that waited for a commit in flight. |
scramdb_cluster_read_skip_fence_waits_total | counter | Waits of SKIP LOCKED or NOWAIT reads for a shard group to be fenced at their cut. |
scramdb_cluster_read_skipped_keys_total | counter | Keys a SKIP LOCKED or NOWAIT read found written by a commit in flight. |
scramdb_cluster_read_skipped_rows_total | counter | Rows SKIP LOCKED left out because a commit in flight writes them. |
scramdb_cluster_read_nowait_refusals_total | counter | NOWAIT statements refused because a commit in flight writes a row they select. |
scramdb_cluster_read_join_spills_total | counter | Reads over more shard groups than fit inline, whose answers spilled to the heap: reads that touch more than eight groups at once. |
scramdb_cluster_read_horizon_serves_total | counter | Shard group watermarks taken here for a transaction snapshot, waiting for no commit. |
scramdb_cluster_read_unreached_groups_total | counter | Shard groups a transaction snapshot could reach on no replica. |
scramdb_cluster_read_fence_waits_total | counter | Waits for a shard group to be fenced at a read position, waiting for no commit. |
scramdb_cluster_read_fence_remote_groups_total | counter | Shard groups hosted on other nodes that a transaction snapshot asked to fence at its cut. |
scramdb_cluster_read_fence_request_retries_total | counter | Shard groups whose fence request to another node was asked again after it went unanswered or was refused. |
scramdb_cluster_read_range_serves_total | counter | Key ranges served at a read position that waited only for commits writing them. |
A read that finds no leader for a group keeps asking until its wait runs out (one election timeout plus the time a group may hold requests without a leader plus the time an undecided transaction may block it, about ten seconds with the defaults) and is then refused with SQLSTATE 40001; a replica that cannot catch up is refused with 57P03. Growth of read_failures_total together with read_barrier_retries_total means a group is without a leader. read_position_raised_total counts transactions that read a table only after an earlier statement had fixed their position: such a transaction's earlier reads are validated at their own position when it commits. Under READ COMMITTED it also counts statements whose position rose because a table they reached later, such as a foreign key's parent, was fresher, so a row committed before the statement began is never missed.
SKIP LOCKED and NOWAIT read the tables they lock without waiting for any commit in flight: the keys such commits write come back with the rows (read_skipped_keys_total), and the statement leaves those rows out (read_skipped_rows_total) or refuses with 55P03 (read_nowait_refusals_total). A shard group where a commit in flight writes a table the statement reads without locking it, such as a join's other side, is read the ordinary way, waiting for that commit, and counts in read_skip_waits_total.
A REPEATABLE READ or SERIALIZABLE transaction takes its snapshot from every shard group's confirmed watermark without waiting for any commit in flight (read_horizon_serves_total), then fences every group at it: the groups this node holds directly (read_fence_waits_total counts the waits for a fence on its way), the others by the nodes that hold them, which the transaction waits to acknowledge before it reads (read_fence_remote_groups_total counts the groups asked). A request that goes unanswered is sent again after the retry interval (read_fence_request_retries_total), so a table that another node commits to after the snapshot is never shown by it. A read of a few keys at that snapshot waits only for a commit in flight on those keys (read_range_serves_total). A group the snapshot can reach on no replica, such as one cut off by a partition, is counted in read_unreached_groups_total, and the transaction's reads are checked at COMMIT, which fails with 40001 if one of them missed a commit. A group whose replicas the snapshot reaches but which all refuse it, such as the groups of a table created a moment before whose replicas are still starting, is not left out: it is asked again at the horizon retry interval (horizon_retry, 90 ms by default) until a replica answers, for at most the read wait.
Adaptive routing
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_routes_total | counter/label | Routing decisions, labelled route: one per statement, and one per scatter for a statement that scatters more than once (each outer row of a k-NN LATERAL join). |
route takes exactly one of five values: oltp, point, local_replica, forward_replica, scatter. That's the whole label space, fixed by construction, the keys are the five route kinds the planner can choose, never a node id or a group id, so this metric's cardinality can never grow with the size of the cluster.
Where fragments run
A distributed read runs each bucket's fragment on one replica that holds the bucket: a learner before a follower before the shard group's leader, in this node's region first, spread evenly (see Where fragments run). [cluster] fragment_any_replica = false keeps fragments on the buckets' owners.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_exchange_placed_learner_buckets_total | counter | Buckets of distributed reads this node placed on a learner replica. |
scramdb_cluster_exchange_placed_follower_buckets_total | counter | Buckets of distributed reads this node placed on a follower replica. |
scramdb_cluster_exchange_placed_leader_buckets_total | counter | Buckets of distributed reads this node placed on a shard group's leader. |
scramdb_cluster_exchange_placement_skips_total | counter | Replicas placement passed over because they could not serve the read position. |
scramdb_cluster_exchange_fragment_redispatches_total | counter | Fragments run on another replica after one could not serve them. |
scramdb_cluster_exchange_fragments_served_learner_total | counter | Fragments this node served from its learner store. |
scramdb_cluster_exchange_round_replacements_total | counter | Shuffle and two-phase read rounds run again on other replicas after a node could not serve its buckets. |
scramdb_cluster_exchange_fragments_refused_total | counter | Fragments this node refused because its replica could not serve the read position. |
scramdb_cluster_fragment_wire_version_refusals_total | counter | Fragments this node refused because their sender speaks a newer wire protocol version than this node reads. |
scramdb_cluster_wire_version | gauge | The wire protocol version this node speaks. |
scramdb_cluster_wire_version_minimum | gauge | The smallest wire protocol version any cluster member has recorded (0 while a member has recorded none). |
scramdb_cluster_route_hash_v1_total | counter | Shuffles and runtime filters this node routed by the older hash because a member it used speaks an older protocol version. It stops rising once every node runs the newer release (Upgrading). |
scramdb_cluster_route_hash_v2_total | counter | Shuffles and runtime filters this node routed by the newer hash because every member it used speaks the newer protocol version. |
scramdb_cluster_exchange_shipped_tables_total | counter | Tables this node sent with a query's fragments because no replica held them whole. |
scramdb_cluster_exchange_statistics_views_published_total | counter | Table statistics this node sent to the other nodes after analysing a table. |
scramdb_cluster_exchange_statistics_views_installed_total | counter | Table statistics another node sent that this node now plans with. |
scramdb_cluster_exchange_statistics_views_stale_total | counter | Table statistics another node sent that this node already held a newer view of. |
scramdb_cluster_exchange_statistics_views_oversize_total | counter | Table statistics too large to send to the other nodes. |
A distributed table has one statistics view: an ANALYZE on any node measures the buckets that node holds, scales the result to the whole table and sends it to the other members, and the newest analysis wins on every node. statistics_views_published_total counts the analyses this node sent and statistics_views_installed_total the ones it took from another node. statistics_views_stale_total is expected after two nodes analyse the same table close together. Any statistics_views_oversize_total means a table's statistics did not fit in one message, so the other nodes keep their own view until they analyse the table themselves. The row count in a shared view is an estimate.
placed_leader_buckets_total growing against the learner and follower counts means the learners and followers cannot serve the reads (behind their group, or waiting for a full copy of its rows), so analytical scans land on the leaders; fragments_refused_total that stays high on one node says that node's replicas lag.
Every node records the wire protocol version it speaks in the metadata group when it starts, and every fragment a node sends carries that version to nodes that have recorded one. A node that receives a fragment from a newer version than it reads refuses it with a retryable message, and the sender runs that statement's buckets on the other replicas that hold them, the sender's own node for buckets it holds. wire_version_refusals_total therefore moves only while nodes of different versions run together, as in a rolling upgrade: a rate on the older nodes is expected until the upgrade finishes, and it is not an error. wire_version_minimum is the smallest version any member has recorded, and a cluster uses a feature that older nodes cannot read only once that minimum allows it; a minimum that stays below wire_version on the upgraded nodes long after an upgrade means a member is still running the older build or never recorded its version. A read whose every replica of a bucket refuses, with no replica on the sender, fails with a retryable error, so finish an upgrade rather than leave the cluster mixed.
Distributed shuffle joins
How a shuffle join across nodes adapted while it ran ([cluster.distributed_join] in the config file).
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_shuffle_consumer_fragments_total | counter | Consumer fragments dispatched for a materialized shuffle join, one per coalesce group. Equals the participant count with coalescing off; lower with it on. Zero on a streaming query. |
scramdb_cluster_shuffle_aqe_coalesce_groups_total | counter | Coalesce groups actually acted on: each merged more than one small adjacent partition into a single consumer fragment. |
scramdb_cluster_shuffle_aqe_skew_split_flagged_total | counter | Partitions flagged skewed (over 5x the median and over 256MB) but not acted on: skew-split off, or a single-node participant set with nothing to split across. Logged and metered even when nothing follows. |
scramdb_cluster_shuffle_aqe_skew_split_acted_total | counter | Sub-consumers dispatched for a skew-split partition that was acted on: the larger side split into disjoint row ranges, the smaller side replicated. Summed across every acted partition. |
scramdb_cluster_shuffle_aqe_broadcast_demote_flagged_total | counter | Broadcast sides flagged over budget, a broadcast-demote candidate. Logged and metered, not acted on yet. |
scramdb_cluster_shuffle_straggler_backups_total | counter | Speculative backup consumer fragments dispatched because a primary consumer didn't return within the straggler-backup delay, on a different live node reading the same frozen spools. |
Node operations and cluster TRUNCATE
The whole-cluster vector statistics views, a cluster TRUNCATE and the rows of a transaction block's
CREATE TABLE AS or REFRESH reach every other node through the same exchange the distributed queries use.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_exchange_node_ops_sent_total / scramdb_cluster_exchange_node_ops_failed_total | counter | Node operations this node asked a peer to run, and those a peer could not run or never answered. |
scramdb_cluster_exchange_node_ops_served_total / scramdb_cluster_exchange_node_ops_serve_failed_total | counter | Node operations this node ran for a peer, and those it could not. |
scramdb_cluster_exchange_view_gathers_total / scramdb_cluster_exchange_view_nodes_unreachable_total | counter | Whole-cluster statistics view reads this node gathered, and nodes such a read reported as unreachable. |
scramdb_cluster_exchange_view_gather_seconds | histogram | Time to gather one whole-cluster statistics view read. |
scramdb_cluster_exchange_truncates_total / scramdb_cluster_exchange_truncate_failures_total | counter | Cluster TRUNCATE statements this node coordinated, and those that did not complete on every node. |
scramdb_cluster_exchange_truncate_seconds | histogram | Time for one cluster TRUNCATE to reach every node holding the table. |
scramdb_cluster_exchange_block_fills_published_total | counter | Schema changes of a transaction block whose rows this node kept for the other nodes. |
scramdb_cluster_exchange_block_fills_served_total | counter | Schema changes of a transaction block whose rows this node sent to another node. |
scramdb_cluster_exchange_block_fills_pulled_total | counter | Schema changes of a transaction block whose rows this node took from the node that runs it. |
scramdb_cluster_exchange_block_fill_fallbacks_total | counter | Schema changes of a transaction block that read their rows again here because the node that runs the block had none to give. |
scramdb_cluster_exchange_producer_failures_propagated_total / scramdb_cluster_exchange_producer_failure_undelivered_total | counter | Failed shuffle producers that handed their error to every consumer at once, and consumers they could not reach (those fall back to their own receive deadline). |
A CREATE TABLE AS, SELECT ... INTO, CREATE MATERIALIZED VIEW or REFRESH MATERIALIZED VIEW inside a transaction block on a cluster is evaluated once, by the node that runs the block, and every other node takes its rows. On that node block_fills_published_total counts the statements and block_fills_served_total the times it sent their rows to another node; on every other node block_fills_pulled_total counts the statements whose rows it took, so one block statement on a three-node cluster adds one published and at least two served on the node that ran the block, and one pulled on each other node. block_fill_fallbacks_total stays at zero in normal operation; if it rises, contact support with the server log.
Cluster branches and table commands
A CREATE DATABASE ... CLONE or a scram.branch_at on a cluster, and the table commands that
carry a branch, a CLONE's fork hold and a cluster TRUNCATE through each shard group's log.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_branches_created_total | counter | Branch databases this node created on the cluster. |
scramdb_cluster_branch_failures_total | counter | Branch databases this node could not create on the cluster. |
scramdb_cluster_branch_tables_forked_total | counter | Tables forked into a branch database on this node's stores. |
scramdb_cluster_branch_pending_rows_total | counter | Rows a shard group gave a branch after this node's store forked it, before the group's fork entry. |
scramdb_cluster_branch_shared_stages_total | counter | COPY segments shared into a branch as they attached after this node's store forked it. |
scramdb_cluster_branch_holds_taken_total | counter | Fork holds taken on this node's stores, each keeping row versions until its branch is forked. |
scramdb_cluster_branch_holds_released_total | counter | Fork holds released on this node's stores. |
scramdb_cluster_branch_clones_abandoned_total | counter | CLONE statements this node abandoned, their fork holds released on every node. |
scramdb_cluster_branch_holds_kept_total | counter | Fork hold checks that kept a hold because this node's store had not finished forking its branch. |
scramdb_cluster_branch_fork_entry_failures_total | counter | Branch forks a shard group's entry could not make on this node, each given up as the branch's failure. |
scramdb_cluster_branch_archive_builds_total | counter | Branches at a past time this node built from the cluster archive. |
scramdb_cluster_branch_archive_rows_total | counter | Rows this node read from the cluster archive into a branch at a past time. |
scramdb_cluster_dropped_database_rows_skipped_total | counter | Rows of a dropped database a shard group skipped on this node. |
scramdb_cluster_branch_fork_records | gauge | Branch fork entries the shard groups on this node still record one by one. A record is folded away once the store holding its group settled the branch (made it ready, or dropped it), so this stays at the branches in progress. |
scramdb_cluster_branch_fork_records_pruned_total | counter | Branch fork records folded into a group's floor once their branch was settled. |
scramdb_cluster_dropped_databases_recorded | gauge | Dropped databases this node's stores record one by one; each is released once its drop is decided. |
scramdb_cluster_table_commands_proposed_total | counter | Table commands this node placed in a shard group's log. |
scramdb_cluster_table_command_failures_total | counter | Table commands this node could not place in a shard group's log before the deadline. |
scramdb_cluster_table_command_redirects_total | counter | Table command proposals a node turned away because its replica did not lead the group. |
A fork hold keeps every row version of the parent's tables on a store, from before the branch's cut is chosen until the store has forked them all. At the end of the hold's lease (two minutes) the node asks the metadata group what became of the CLONE: an abandoned clone releases the hold on every node, and a branch that was made releases it once the store has finished forking it. A hold whose store has not finished forking its branch stays until the fork finishes. Each check that finds such a hold counts on holds_kept_total and logs once, scramdb_compaction_versions_held_total counts the row versions merges kept only because a hold stood, and scramdb_cluster_ddl_apply_stuck shows the change the fork waits for. The hold ends when the cause is repaired or the branch is dropped.
Table definition changes
An ALTER TABLE on a cluster: the commits it holds back on each node, the fence it places in each of a distributed table's shard groups, and the rows each node installs across a change.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_schema_changes_pending | gauge | Tables whose definition change is under way on this node's stores. |
scramdb_cluster_schema_commits_refused_total | counter | Commits refused because a table they wrote changed its definition while they ran. |
scramdb_cluster_schema_commit_drains_total | counter | Waits for the commits under way before a table's definition change. |
scramdb_cluster_schema_commit_drain_timeouts_total | counter | Waits for the commits under way that ended at their deadline. |
scramdb_cluster_schema_fences_applied_total | counter | Table definition change fences this node's stores applied from shard group logs. |
scramdb_cluster_schema_fence_waits_total | counter | Fences that waited for this node's catalog to take the change first. |
scramdb_cluster_renamed_table_rows_total | counter | Rows of a renamed distributed table installed under its new name. |
scramdb_cluster_widened_rows_total | counter | Rows written before a column was added, widened to the table's shape as installed. |
scramdb_cluster_schema_marks_restored_total | counter | Table change marks this node's stores restored from disk when they opened. |
scramdb_cluster_schema_mark_write_failures_total | counter | Writes of the table change marks to disk that failed. |
scramdb_cluster_ddl_apply_stuck | gauge | Decided schema changes this node failed to apply and is trying again. 0 on a healthy node; alert on a value above 0 that lasts. |
scramdb_cluster_ddl_apply_retries_total | counter | Tries again of decided schema changes that failed to apply on this node. |
scramdb_cluster_ddl_stuck_refusals_total | counter | Statements refused because a decided schema change of a table they name is not applied here yet. Each answered 40001. |
scramdb_cluster_replica_key_waits_total | counter | Shard group applies that waited for the key index of their table to be made on this node. |
scramdb_cluster_ddl_apply_lock_waits_total | counter | Relation locks a decided schema change's apply had to wait for on this node. |
scramdb_cluster_ddl_columnar_store_waiting | gauge | Decided schema changes waiting for the columnar store's log, which hold this node's later ones back. |
scramdb_cluster_apply_lanes_at_fence | gauge | Shard group apply lanes waiting at a schema change for this node's catalog, holding no thread. |
scramdb_cluster_name_barriers_total | counter | Statements that asked the metadata group how far it had decided before a missing name was answered. |
scramdb_cluster_name_barrier_catch_ups_total | counter | Statements that ran again after this node applied what the metadata group had decided. |
scramdb_cluster_name_barriers_unreached_total | counter | Missing-name checks that could not reach the metadata group within the statement's bound. |
scramdb_cluster_name_barriers_unapplied_total | counter | Missing-name checks that ended with this node still behind the metadata group. |
A DROP TABLE of a distributed table holds back and drains the table's commits and fences its shard groups the same way before the table goes, so it moves schema_commit_drains_total and schema_fences_applied_total too.
A schema change the cluster decided is applied on every node. A node that cannot apply one (a failing disk, no room, a wait that ran out) does not record it as applied: it logs one error line, counts the change on scramdb_cluster_ddl_apply_stuck and tries again on a pause that starts at 100 milliseconds and doubles up to the metadata group's reply deadline, each try counted on scramdb_cluster_ddl_apply_retries_total. Until it applies, a statement that names a table the change touches waits for it, holding no lock, and when its wait ends first (its lock_timeout or statement_timeout, or an hour when it set neither) it is answered 40001 (table "name" is not up to date on this node yet) and counted on scramdb_cluster_ddl_stuck_refusals_total; other tables are served as usual, and the gauge falls back to 0 once the change applies. A change the statement itself refuses, the same on every node, is not retried and never counts here. A change that every node fails to apply for a reason of its own is held on all of them: every node reads the gauge above 0, refuses the statements that name the table once their wait is over and serves the rest, and the schema changes decided after it wait behind it. No statement gives such a change up; remove the cause the server log names (free the disk, for instance) and it applies on its own.
A node that finds a relation missing asks the metadata group how far it has decided before it answers 42P01 (see Changing a table on a cluster). name_barriers_total counts those questions, and each ends in one of four ways: the node was current and the miss stands, name_barrier_catch_ups_total (the node applied what the group had decided and the statement ran again), name_barriers_unapplied_total (the node was still behind at the end of the statement's wait and the statement was answered 40001) or name_barriers_unreached_total (the group's leader did not answer within the statement's wait, and the miss is answered as before). A catch_ups_total or unapplied_total that keeps growing means this node applies the metadata group's decisions late.
scramdb_cluster_apply_lanes_at_fence reads how many shard group apply lanes of this node wait at a schema change for the node's catalog to take it first. A lane waiting there holds no apply thread, so a node with few apply threads keeps serving its other groups, and the lane goes on by itself once the catalog has taken the change. It is 0 on a node with no change in progress; a value above 0 that lasts means the node's catalog has not yet taken a change (a rewrite of a large table is slow to apply, and a change the node cannot apply shows on scramdb_cluster_ddl_apply_stuck). scramdb_cluster_ddl_apply_lock_waits_total does not count the wait behind the hold of the change's own round, so a value above 0 names a lock that nothing drained before the change.
The marks that say a table was dropped or changed are written to disk before a node answers the round that proposes the change, so a node that restarts between its answer and the decision still refuses a stale write to the table. schema_marks_restored_total counts the marks read back at start. Any schema_mark_write_failures_total means a mark could not be written to that node's disk. The statement fails and can be retried. Check the disk of that node.
UPDATE and DELETE on a partially held table
An UPDATE or DELETE on a node that holds only some buckets of a distributed table reads its
rows from every owner (see
UPDATE and DELETE across nodes).
A node that holds every bucket of the table runs the single-node path and moves none of these.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_dml_statements_distributed_total | counter | UPDATE and DELETE statements that read their rows from every owner. |
scramdb_cluster_dml_candidates_remote_total / scramdb_cluster_dml_candidates_local_total | counter | Rows such statements received from other nodes, and read from this node's own buckets. |
scramdb_cluster_dml_candidate_batches_total | counter | Row batches such statements processed. |
scramdb_cluster_dml_candidate_fetch_seconds | histogram | Time a statement waited for its next batch of rows. |
scramdb_cluster_dml_candidate_spilled_bytes_total | counter | Row bytes set aside on disk because the statement's memory budget was spoken for. |
scramdb_cluster_dml_owners_pruned_total | counter | Owners a statement skipped because its WHERE names one bucket. |
scramdb_cluster_dml_locking_selects_total | counter | Locking SELECT statements (FOR UPDATE, FOR NO KEY UPDATE, FOR SHARE, FOR KEY SHARE) that read a distributed table's rows without locking them. |
scramdb_cluster_dml_locking_read_rows_total | counter | Rows such statements took and registered one by one for their commit to check (a SERIALIZABLE statement without SKIP LOCKED has its whole scan checked instead and adds nothing; rows SKIP LOCKED left out are not counted). |
scramdb_cluster_dml_own_writes_reads_total | counter | Queries that read a partially held table together with their transaction's own uncommitted changes to it. |
scramdb_cluster_dml_candidate_readers_replaced_total | counter | Readers of an UPDATE, DELETE or uniqueness check that could not serve their buckets and were read again on another replica. |
A growing candidate_spilled_bytes_total means large UPDATE or DELETE statements are running
close to the execution memory budget.
The row-lock family these statements used to report (scramdb_cluster_dml_row_lock_batches_total,
scramdb_cluster_dml_row_locks_total, scramdb_cluster_dml_row_lock_waits_total,
scramdb_cluster_dml_row_lock_wait_seconds, scramdb_cluster_dml_row_lock_conflicts_total,
scramdb_cluster_dml_row_locks_released_total) and scramdb_cluster_dml_statement_restarts_total
are gone: rows of a distributed table are no longer locked, so no statement waits, restarts or
releases a lock (see
Row locks on distributed tables).
Writers that reach the same rows now show up as 40001 failures at commit: watch
scramdb_cluster_dilith_aborts_total and your clients' retry counts instead, and remove any alert
or dashboard panel that reads the removed names.
Distributed statements
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_read_stability_retries_total | counter | Reads re-run because a replica applied writes while they scanned. |
scramdb_cluster_read_stability_failures_total | counter | Reads that failed because replicas kept applying writes through every re-run. The statement fails with 40001 and can be retried. |
scramdb_cluster_serialization_retries_total | counter | Autocommit statements sent again, and COPY votes voted again, after a serialization failure. |
scramdb_cluster_placement_waits_total | counter | Statements that waited for this node to learn where a table they name is placed, right after it restarted or right after the table was created. |
scramdb_cluster_catalog_waits_total | counter | Reader slices that waited for this node to apply the definition of a table it places. |
scramdb_cluster_rows_moved_total | counter | Rows an UPDATE moved to a new primary key or shard group. |
scramdb_cluster_unique_probes_total | counter | Uniqueness checks of new keys sent to the nodes that hold their buckets. |
scramdb_cluster_unique_violations_total | counter | New keys refused because a row on another node already holds them. |
scramdb_cluster_inserts_positioned_by_key_total | counter | Inserted batches whose keys were checked in their own shard groups only, not in every group of the table. |
scramdb_cluster_upsert_remote_conflicts_total | counter | INSERT ... ON CONFLICT rows whose conflicting row is stored on another node. |
scramdb_cluster_on_conflict_keys_overtaken_total | counter | INSERT ... ON CONFLICT statements refused because another transaction committed one of their conflict keys after the statement judged it free. |
scramdb_cluster_own_writes_spilled_bytes_total | counter | Bytes of a transaction's view of a table a query set aside on disk for want of memory. |
scramdb_cluster_general_statements_total | counter | Statements answered on their coordinator over rows streamed from every shard group. |
scramdb_cluster_general_fast_path_declines_total | counter | Statements a faster distributed path handed to the general path. |
scramdb_cluster_general_rows_streamed_total | counter | Rows streamed to a coordinator for the statements it answered over them. |
scramdb_cluster_general_spilled_bytes_total | counter | Bytes of streamed rows a coordinator set aside on disk for want of memory. |
scramdb_cluster_general_readers_refused_total | counter | Readers of a streamed relation whose replica refused the read position. Their buckets were read on another replica. |
scramdb_cluster_general_readers_unreached_total | counter | Readers of a streamed relation that were not reached or stopped sending. Their buckets were read on another replica. |
scramdb_cluster_general_readers_failed_total | counter | Readers of a streamed relation whose read failed, failing the statement. |
scramdb_cluster_general_reads_exhausted_total | counter | Streamed reads that failed because no replica was left to read a bucket. The statement fails with 40001 and can be retried. |
Prepared transactions
PREPARE TRANSACTION, COMMIT PREPARED and ROLLBACK PREPARED on a cluster's distributed tables (see Prepared transactions), counted on the node that ran the statement; the holds are counted on every node that keeps a replica of the groups they live in.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_prepare_succeeded_total | counter | PREPARE TRANSACTION statements that prepared their transaction. |
scramdb_cluster_prepare_refused_total | counter | PREPARE TRANSACTION statements refused, their transaction rolled back. |
scramdb_cluster_prepare_in_doubt_total | counter | PREPARE TRANSACTION statements whose outcome was not known when they answered (08007). |
scramdb_cluster_commit_prepared_succeeded_total | counter | COMMIT PREPARED statements that committed a prepared transaction. |
scramdb_cluster_commit_prepared_refused_total | counter | COMMIT PREPARED statements that were refused, such as an unknown transaction name. |
scramdb_cluster_commit_prepared_in_doubt_total | counter | COMMIT PREPARED statements whose outcome was not known when they answered (08007). |
scramdb_cluster_rollback_prepared_succeeded_total | counter | ROLLBACK PREPARED statements that rolled a prepared transaction back. |
scramdb_cluster_rollback_prepared_refused_total | counter | ROLLBACK PREPARED statements that were refused. |
scramdb_cluster_rollback_prepared_in_doubt_total | counter | ROLLBACK PREPARED statements whose outcome was not known when they answered (08007). |
scramdb_cluster_dilith_prepared_holds_created_total | counter | Prepared transactions whose hold this node's replicas made. |
scramdb_cluster_dilith_prepared_holds_consumed_total | counter | Prepared transactions committed on this node's replicas, their holds ended. |
scramdb_cluster_dilith_prepared_holds_released_total | counter | Prepared transactions rolled back on this node's replicas, their holds ended. |
scramdb_cluster_dilith_prepared_hold_vote_refusals_total | counter | Votes this node's replicas refused because they met a prepared transaction's hold. |
scramdb_cluster_dilith_prepared_hold_precheck_refusals_total | counter | Votes a group leader on this node refused before appending because they met a prepared transaction's hold. |
A prepared transaction's hold keeps its writes from being overtaken while it waits for COMMIT PREPARED or ROLLBACK PREPARED, from whichever node the verb arrives. pg_prepared_xacts lists the prepared transactions of the current database from the cluster's registry: one whose prepared time keeps growing is a transaction nobody resolved.
Table migrations and re-keys
A table definition change that reshapes rows (a re-key, a column type change, the upgrade of an older table) is staged, decided once, fenced on every node and finished by the group leader if the node that began it died. These metrics show the progress.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_migration_tables_pending | gauge | Tables with a migration on the cluster that is not finished for this node. SHOW CLUSTER lists the same count per node as migrations_pending. |
scramdb_cluster_migration_rows_moved_total | counter | Rows this node moved or staged for a table migration. |
scramdb_cluster_rekey_row_keys_retired_total | counter | Hidden row keys this node retired once every group it hosts of the table applied its re-key entry. |
scramdb_cluster_rekey_stage_mismatches_total | counter | Re-key entries a replica stopped at because its stage was not the one the entry names. Zero on a healthy cluster. |
scramdb_cluster_copy_stage_mismatches_total | counter | COPY stage attaches a replica stopped at because its stage was not the one the vote names. Zero on a healthy cluster. |
scramdb_cluster_stage_entries_carried_total | counter | Entries of an open stage (a COPY's or a re-key's) a state snapshot carried to a replica that installed it. |
A scramdb_cluster_migration_tables_pending that stays above 0 for an hour means a migration is waiting on a node that is down or fenced. The mismatch counters stay at zero in normal operation; if one rises, contact support with the server log.
The holds this node's replicas keep are the prepared_holds entry of scramdb_cluster_dilith_table_entries. A growing scramdb_cluster_dilith_prepared_hold_vote_refusals_total means writers keep meeting what prepared transactions touched; commit or roll the prepared transactions back sooner.
Columnar mirror
A node that keeps a columnar copy of the rows it replicates (a hybrid voter, columnar_replica = true, or a learner) copies committed entries into it on its own mirror threads.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_columnar_mirror_threads | gauge | Threads that copy committed entries into this node's columnar store. |
scramdb_cluster_columnar_mirror_queued | gauge | Columnar mirror lanes waiting for a mirror thread. |
scramdb_cluster_columnar_mirror_panics_total | counter | Columnar mirror lanes that stopped on an unexpected condition. Zero in normal operation; if it rises, contact support with the server log. |
scramdb_cluster_columnar_mirror_idle_wakes_total | counter | Times an idle columnar mirror thread woke up. An idle pool holds it still; it moves with work arriving, never on a timer. |
Learner-read freshness
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_learner_freshness_seconds | histogram | How long ago each queried group's read floor last advanced, observed once per stale or strong learner_read read. The floor is the commit protocol's closed position at the replica: every commit at or below it is installed there and none can land there later. The OLTP path (learner_read off) never touches this histogram. Same buckets as the handshake histogram, 1ms to 5s plus +Inf, a natural fit for replication lag. |
This is the same lag SHOW FRESHNESS reports per data group in Clustering, captured here as a scrapeable distribution instead of a point-in-time value.
Faults, retries and reachability
How a node sees its peers and its consensus groups, and every retry, timeout and re-dispatch of the cluster's network mechanisms. Always emitted in cluster mode. On a healthy cluster the failure and retry counters stay at zero, and the gauges show the peers and members that are up. A fault that repeats is logged once, as a warning when it begins, and counted every time; a lost peer that is heard again is logged once at info.
| Metric | Type | Meaning |
|---|---|---|
scramdb_cluster_peers_reachable | gauge | Peers with an open cluster connection. |
scramdb_cluster_peer_reachable | gauge | Whether a peer connected or lost now has an open cluster connection (1 or 0), labelled peer. |
scramdb_cluster_peer_round_trip_microseconds | gauge | Round trip of the newest answered liveness ping to a peer, labelled peer. Absent until a ping is answered. |
scramdb_cluster_peer_silence_milliseconds | gauge | Time since a peer's cluster connection last received a byte, labelled peer. |
scramdb_cluster_peers_lost | gauge | Peers that were connected and have not been heard from since their connection was lost. |
scramdb_cluster_peer_losses_total | counter | Times a connected peer became unreachable. A partition is one loss per peer, however often its connections close. |
scramdb_cluster_dial_failures_total | counter | Connection attempts to peers that failed, labelled cause. |
scramdb_cluster_inbound_handshake_failures_total | counter | Connections from peers that failed or were refused during the handshake, labelled cause. |
scramdb_cluster_accept_errors_total | counter | Times a cluster listener failed to accept a connection. |
scramdb_cluster_groups_without_leader | gauge | Consensus groups on this node that know no leader. |
scramdb_cluster_group_leader_losses_total | counter | Times a consensus group on this node lost its leader. |
scramdb_cluster_stalled_groups | gauge | Consensus groups on this node that made no progress for as long as a request waits for its answer (fifteen election timeouts, at least 5 s). |
scramdb_cluster_group_stalls_total | counter | Times a consensus group on this node stalled. |
scramdb_cluster_retries_total | counter | Times a cluster mechanism tried again after a failure, labelled mechanism. |
scramdb_cluster_retries_exhausted_total | counter | Times a cluster mechanism gave up after its retries or its deadline ran out, labelled mechanism. |
scramdb_cluster_timeouts_total | counter | Waits of a cluster mechanism that reached their deadline, labelled mechanism. |
scramdb_cluster_redispatches_total | counter | Work placed again on another node after its first node failed, labelled mechanism. |
scramdb_cluster_retry_rate_per_second | gauge | Retries a second across every mechanism over the last ten seconds. A sustained rise is a retry storm. |
scramdb_group0_reply_timeouts_total / scramdb_shard_group_reply_timeouts_total | counter | Metadata group and shard group proposals whose answer did not come in time. Each may still commit. |
scramdb_group0_mailbox_timeouts_total / scramdb_shard_group_mailbox_timeouts_total | counter | Requests the group did not take in time. None changed anything. |
scramdb_group0_read_retries_total / scramdb_shard_group_read_retries_total | counter | Read confirmations asked again after an answer did not come. |
scramdb_group0_apply_wait_timeouts_total / scramdb_shard_group_apply_wait_timeouts_total | counter | Waits for this node to apply a committed entry that ran out of time. |
scramdb_cluster_swim_members_alive / _suspect / _dead | gauge | Cluster members this node's failure detector sees alive, suspects, and has declared failed. |
scramdb_cluster_swim_local_health | gauge | This node's failure detector health score. Above zero, the node itself is missing probe answers. |
scramdb_cluster_swim_member_up_total / _suspect_total / _down_total | counter | Times a member came up, became suspect, or was declared failed in this node's failure detector. |
scramdb_cluster_swim_malformed_messages_total | counter | Failure detector messages from peers that could not be read and were dropped. |
scramdb_cluster_swim_send_failures_total | counter | Failure detector messages this node could not hand to a peer's connection. |
scramdb_cluster_swim_joins_dropped_total | counter | Member joins dropped because the failure detector's join queue was full. |
scramdb_cluster_swim_events_coalesced_total | counter | Membership events replaced by a newer event of the same member before the listener took them. |
scramdb_cluster_swim_events_evicted_total | counter | Membership events dropped because the listener fell behind and the event queue was full. |
scramdb_cluster_swim_events_pending / _capacity | gauge | Members with a membership event not yet taken, and the most members the queue holds (sized from the cluster). |
scramdb_cluster_exchange_partial_chunks | gauge | Exchange streams holding part of a chunk in reassembly on this node. |
scramdb_cluster_exchange_malformed_frames_total / scramdb_cluster_exchange_orphan_frames_total | counter | Exchange frames dropped because they could not be read, or because no query on this node was waiting for their stream. |
scramdb_cluster_exchange_out_of_order_total / _undecodable_chunks_total / _reassembly_overflows_total | counter | Exchange streams failed because a frame arrived out of order, a chunk could not be decoded, or a chunk grew past the reassembly limit. |
scramdb_cluster_exchange_producer_failures_received_total | counter | Exchange streams whose producer reported a failure to this node. |
scramdb_cluster_stream_frames_malformed_total | counter | Frames of a cluster service from peers that could not be read and were dropped. |
scramdb_cluster_point_forwards_total | counter | Point reads this node forwarded to their group's leader. |
scramdb_cluster_point_forward_send_failures_total / _remote_errors_total / _reply_send_failures_total | counter | Forwarded point reads that could not be sent, that the leader answered with an error, and answers to other nodes' forwarded reads that could not be sent back. A forwarded read that could not be sent or got no answer is answered on the general path instead; an error the leader reported is the statement's answer. |
The cause of a failed connection is one of connect_timeout, refused, reset, unreachable, unresolved, handshake_timeout, rejected, version (the two nodes share no cluster protocol version: upgrade the older one), identity, tls, protocol, closed and other.
The mechanism is one of the following, and each family lists only the mechanisms that can record it, so no sample reads as a measured zero:
dialandlane_dial: connecting to a peer;point_forward,fragmentandfetch: a distributed statement's requests;install: fetching a state snapshot;drainandautojoin: membership changes;metadata_groupandshard_group: requests to a consensus group;commit_voteandcommit_outcome: the commit protocol's resends.
A fragment timeout is a reader that missed its deadline. Its buckets are read on another replica, which is a fragment re-dispatch.
Alerting on cluster metrics
Three conditions worth paging on. Each maps to a documented design decision rather than a guessed threshold:
groups:
- name: scramdb-cluster
rules:
- alert: ScramDBGroupsQuarantined
expr: scramdb_cluster_quarantined_groups > 0
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $value }} raft group(s) quarantined"
description: >
A group's storage was quarantined at boot for corruption and has
not finished its membership-replace rejoin. This gauge is
recomputed from the real gated set every reconcile round, not
decremented, so restarting the node will not clear a stale
reading; only a completed rejoin does.
- alert: ScramDBLogCompactionStalled
expr: |
rate(scramdb_cluster_log_compactions_total[15m]) == 0
and rate(scramdb_cluster_group_commit_ops_total[15m]) > 0
for: 30m
labels:
severity: warning
annotations:
summary: "Raft log compaction has not run on a busy node in 30 minutes"
description: >
The node is committing but not compacting. Either its state
snapshots keep failing (check
scramdb_cluster_state_snapshot_failures_total and
scramdb_cluster_state_snapshots_abandoned_total: a full disk or a
memory_bytes budget too small), or a replica is behind but still
inside the lag limit. A replica lagging past the limit stops
holding the log and is repaired from a snapshot when it returns.
- alert: ScramDBPartitionDropsDetected
expr: increase(scramdb_cluster_partition_drops_total[5m]) > 0
labels:
severity: critical
annotations:
summary: "Cluster transport frames are being silently dropped"
description: >
This counter only moves when a network partition is injected
(chaos testing) or something outside the engine is dropping
cluster transport frames. It is zero in every normal deployment,
so any increase is worth paging on.
Index memory (always present)
The memory each index holds, read when /metrics is scraped. A primary key, a unique key and
every btree, hash and gin index are kept in memory, so this is all the memory they
use. Vector indexes report their own memory under Vector index metrics.
| Metric | Type | Meaning |
|---|---|---|
scramdb_index_memory_bytes | gauge | Memory one index holds, labelled database, schema, table, index and method. Only the 1000 largest indexes get a series. |
scramdb_index_memory_other_bytes | gauge | Memory held by the indexes past those 1000, which have no series of their own. |
scramdb_index_memory_other_indexes | gauge | How many indexes are past those 1000. |
scramdb_index_memory_total_bytes | gauge | Memory held by every index on the node. |
The same figure is in SQL. scram_indexes lists each index of the current database with
schemaname, relname, indexrelname, method, entries, memory_bytes, and it shows only the
tables the reading role may see. pg_relation_size('an_index') gives that index's memory, and
pg_indexes_size('a_table') adds up a table's indexes:
SELECT indexrelname, method, entries, pg_size_pretty(memory_bytes)
FROM scram_indexes ORDER BY memory_bytes DESC LIMIT 10;
Index run files hold the sorted, on-disk part of an index too large to keep in memory. Their counters are always present and stay at zero until an index has a run file:
| Metric | Type | Meaning |
|---|---|---|
scramdb_index_run_filter_probes_total | counter | Index run filter probes made. |
scramdb_index_run_filter_negatives_total | counter | Index run filter probes that ruled a key out. |
scramdb_index_run_filter_false_positives_total | counter | Index run filter probes that passed a key the run does not hold. |
scramdb_index_run_data_pages_read_total | counter | Index run data pages read by lookups and scans. |
scramdb_index_run_fence_pages_read_total | counter | Index run fence pages read by lookups and seeks. |
GIN index metrics
Process-wide totals over every GIN index (USING gin) on the node, always present, on the
same /metrics body. The memory each GIN index holds is in
Index memory; EXPLAIN shows which index a query probes.
| Metric | Type | Meaning |
|---|---|---|
scramdb_gin_probes_total | counter | GIN index probes run. |
scramdb_gin_candidates_total | counter | Candidate rows the posting lists offered. |
scramdb_gin_recheck_rejected_total | counter | Candidates the recheck of the query's filter rejected. |
scramdb_gin_rows_returned_total | counter | Rows GIN probes returned after their recheck. |
scramdb_gin_snapshot_scans_total | counter | GIN-planned reads inside a REPEATABLE READ or SERIALIZABLE transaction, answered by scanning the table as of the snapshot. |
scramdb_gin_entries_inserted_total / scramdb_gin_entries_deleted_total | counter | Index entries added / removed by writes. |
scramdb_gin_probe_seconds | histogram | Time one GIN probe took, posting lists to returned rows. |
A high recheck_rejected_total against candidates_total means the index narrows poorly
for your filters; a rising snapshot_scans_total means snapshot transactions are reading
whole tables where a READ COMMITTED read would probe the index.
Self-driving (always present)
Ejder, the self-driving loop (Self-driving with Ejder). A node that does not run it reports scramdb_autonomy_running at 0 and zeros, never an absent series.
| Metric | Type | Meaning |
|---|---|---|
scramdb_autonomy_running | gauge | Whether the self-driving loop runs on this node, 1 or 0. |
scramdb_autonomy_parked | gauge | Whether the loop stopped deciding after repeated failures, 1 or 0. |
scramdb_autonomy_ticks_total | counter | Decisions the loop took. |
scramdb_autonomy_tick_seconds | histogram | Time one decision took, in seconds. |
scramdb_autonomy_tick_interval_ms | gauge | The interval between decisions in force now, in milliseconds. |
scramdb_autonomy_self_cost_permille | gauge | Share of one core the last decision cost, in thousandths. |
scramdb_autonomy_throttled_total | counter | Decisions after which the loop slowed its own interval. |
scramdb_autonomy_setpoints_applied_total | counter | Setting changes the loop applied. |
scramdb_autonomy_setpoints_reverted_total | counter | Settings put back because the change made things worse. |
scramdb_autonomy_open_windows | gauge | Changes whose effect is being checked now. |
scramdb_autonomy_windows_evicted_total | counter | Checks dropped unfinished because too many were open at once. |
scramdb_autonomy_state_refused_total | counter | Decisions whose state was over its size bound and was not kept. |
scramdb_autonomy_audit_failures_total | counter | Decision records that could not be written. |
scramdb_autonomy_finishes_dropped_total | counter | Ends of background changes dropped before they were recorded. |
scramdb_autonomy_records_dropped_total | counter | Decisions not written to the recording because it was full or failed. |
scramdb_autonomy_tick_failures_total | counter | Decisions that failed, by reason: unavailable, compile, timeout, fuel, memory, output_too_large, trap or frame. |
scramdb_autonomy_actions_total | counter | Actions the loop proposed, by outcome: dispatched, advised, refused, accepted, reverted, expired, completed or failed. |
Index builds and compaction
An index build and a compaction merge never work on one table at the same time, as in
PostgreSQL, where CREATE INDEX and VACUUM take conflicting locks. Background compaction
yields: a merge running when a build starts stops, and the table is merged on a later pass. An
explicit VACUUM waits for the builds on its table to finish, and a build waits for a running
VACUUM, for at most lock_timeout when it is set (then it fails with SQLSTATE 55P03).
Process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_compaction_merges_total | counter | Background merges that completed, whatever started them. |
scramdb_compaction_rows_reclaimed_total | counter | Rows background merges dropped as dead. |
scramdb_compaction_versions_held_total | counter | Row versions merges kept only because a retention hold stood, counted once per merge batch. |
scramdb_compaction_merges_yielded_to_index_builds_total | counter | Background merges stopped or skipped because an index build held their table. |
scramdb_index_builds_waited_for_merges_total | counter | Index builds that waited for a merge running on their table to end. |
scramdb_index_build_merge_wait_timeouts_total | counter | Index builds that stopped waiting for a merge on their table at their lock timeout. |
scramdb_vacuums_waited_for_index_builds_total | counter | VACUUM runs that waited for the index builds holding their table to end. |
Two table storage parameters also steer the background merge. A table that took in rows past its
autovacuum_vacuum_insert_threshold and autovacuum_vacuum_insert_scale_factor has its small
segments merged without waiting for the usual number, and a table with an
autovacuum_vacuum_cost_delay pauses after a merge for that delay for each
autovacuum_vacuum_cost_limit pages the merge read. Process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_compaction_insert_merges_total | counter | Background merges started by a table's insert threshold instead of the usual number of small segments. |
scramdb_compaction_pace_pause_seconds | histogram | Time background merges paused to keep to their table's cost delay. |
A separate fence orders a compaction's row relocation against a concurrent write that resolves a
row's physical position by table (DELETE, UPDATE, ON CONFLICT DO UPDATE/DO SELECT, a
replica's apply): the write holds the fence until its commit is durable, and a compaction raises
it before it swaps its merged segments in, so a swap never relocates a row out from under a write
already under way. A statement that reads a table's row identities (a ctid column, a
TABLESAMPLE draw) holds nothing: a compaction moves a row and keeps its identity, so a row has
one ctid and one place in a sample whatever compaction runs meanwhile, and VACUUM FULL does
not wait for such a reader. Process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_writes_waited_for_compaction_total | counter | Statements that waited for a compaction to finish moving their table's rows. |
scramdb_write_compaction_wait_timeouts_total | counter | Statements that stopped waiting for a compaction at their lock timeout. |
scramdb_compaction_guarded_swaps_total | counter | Compaction swaps made while the statements using their table's rows were held back. |
scramdb_compaction_swaps_abandoned_total | counter | Background compaction batches dropped because the statements holding their table's rows did not finish in time. |
scramdb_vacuum_lock_timeouts_total | counter | VACUUM FULL runs stopped at their lock timeout while statements held the table's rows. |
scramdb_compaction_late_deletes_carried_total | counter | Rows deleted while a compaction ran that it carried to their new places. |
scramdb_compaction_writer_drain_seconds | histogram | Time a compaction waited for the statements holding its table's rows to finish. |
scramdb_compaction_writer_pause_seconds | histogram | Time a compaction held back the statements using its table's rows while it swapped. |
scramdb_write_compaction_wait_seconds | histogram | Time a statement waited for a compaction before it could read or write its rows. |
scramdb_compaction_writer_tables | gauge | Tables with writer counters allocated. |
swaps_abandoned_total or vacuum_lock_timeouts_total rising means the statements holding a
table's rows (its writers) never drain within the wait bound: background compaction backs off
quietly, VACUUM FULL fails loudly with 55P03.
Compaction by table
What the merges of each table gave back, counted where a merge completes whether the background pass or a
VACUUM ran it, and read when /metrics is scraped. A table has a series once a merge of it has finished
since the server started, and loses it when the table is dropped. Each family is labelled database,
schema and table, and only the 1000 tables that reclaimed the most bytes get a series.
| Metric | Type | Meaning |
|---|---|---|
scramdb_table_compaction_rewrites_total | counter | Merges of a table that completed since the server started, by the background pass or a VACUUM. |
scramdb_table_compaction_rows_reclaimed_total | counter | Rows the merges of a table dropped as dead since the server started. |
scramdb_table_compaction_bytes_reclaimed_total | counter | Bytes of segments the merges of a table freed past the bytes they wrote since the server started. |
scramdb_table_compaction_other_tables | gauge | Tables merged since the server started that have no compaction series of their own, past the series bound. |
scramdb_table_compaction_other_tables is how many tables are past the 1000: a table is left out of the
list, never out of the count. It reads 0 while every merged table has its series.
Query worker pool
The engine's core-pinned query worker pool ([execution] workers), process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_query_worker_count | gauge | Pinned workers the engine's query worker pool runs. |
scramdb_query_worker_queue_depth | gauge | Outstanding tasks queued across the query worker pool's pinned workers, summed. |
scramdb_query_worker_jobs_total{worker} | counter | Jobs each query worker finished, one series per worker. A server under load shows every worker moving; one series far above the rest is work that is not spreading. |
scramdb_query_dispatches_idle_total | counter | Query dispatches placed on an idle query worker. |
scramdb_query_dispatches_busy_total | counter | Query dispatches placed on a busy query worker because every probed worker was busy: a rising share means the pool is saturated. |
scramdb_query_worker_cpu_quota_cores_permille | gauge | This engine's claimed cores per thousand of the machine's full, unconstrained cpuset: 1000 unconstrained, lower under a CPU limit (a container --cpus, a Kubernetes CPU request). |
worker_queue_depth climbing while worker_count holds steady means query work is arriving
faster than the pool drains it; cpu_quota_cores_permille well below 1000 on a box that looks
otherwise idle means the CFS quota, not the workload, is the ceiling on worker_count.
A table's parallel_workers storage parameter limits the workers a scan of it runs on (0 is a
serial scan). Process-wide, always present, counting only scans that ran under such a parameter:
| Metric | Type | Meaning |
|---|---|---|
scramdb_scan_capped_dispatches_total | counter | Scans that ran under their table's parallel_workers. |
scramdb_scan_capped_workers_total | counter | Workers given to scans that ran under their table's parallel_workers. |
The second over the first is the width a capped scan ran at.
Stand-in query workers
A fixed-pool query worker whose core is parked on a lock wait hands its core to a stand-in worker for the wait's duration, so a lock contended by one statement does not idle every query on that core. Process-wide, always present:
| Metric | Type | Meaning |
|---|---|---|
scramdb_query_standins | gauge | Stand-in query worker threads alive right now, running or idle-cached. |
scramdb_query_standin_starts_total | counter | Stand-in query worker threads started. |
scramdb_query_relief_standins | gauge | Relief query workers alive right now: started when a queued statement waited a full watchdog sample while every worker was busy, at most one per core. |
scramdb_query_relief_starts_total | counter | Relief query workers started. A steady rise means long statements often hold every worker. |
scramdb_pool_standins | gauge | Stand-in threads of the core-pinned pool alive right now. |
scramdb_pool_standin_starts_total | counter | Stand-in threads of the core-pinned pool started. |
scramdb_pool_job_parks_total | counter | Jobs of the core-pinned pool that parked and handed their worker's queue on. |
scramdb_query_worker_idle_wakes_total | counter | Times an idle query worker thread woke from its wait, with or without work. An idle server holds it still apart from one wake per worker to hand freed memory back to the system; it moves with statements arriving, never on a timer. |
Vector index metrics
Every anode/manode vector index family lives on the scramdb_vector_ prefix,
appended to the same /metrics body as everything above. One sample feeds both the
Prometheus families below and the system views in the next section, so a number never
disagrees between them. A
family the sample has not measured (a recall job that never ran, a per-index rate with
no traffic yet) is omitted from the scrape entirely rather than rendered as a
fabricated 0, the same rule the WAL archive metrics above follow. Cardinality is
bounded by the number of indexes (and cluster parts) open on the node, reported by
scramdb_vector_series so you can see the cost.
Node totals (no index label, always present):
| Metric | Type | Meaning |
|---|---|---|
scramdb_vector_memory_budget_total_bytes | gauge | The memory ceiling configured for all vector indexes on this node together. |
scramdb_vector_memory_reserved_bytes | gauge | Memory currently reserved by vector index arenas against the total ceiling. |
scramdb_vector_memory_in_use_bytes | gauge | Memory vector indexes use right now: their arenas, row maps and write buffers together. The write buffers are charged to execution memory, not to the vector ceiling. A build's working memory (the model training sample, the convert buffers) is not counted here: it is charged to storage.memory.maintenance_bytes, like every index build. |
scramdb_vector_memory_uncharged_bytes | gauge | Memory vector indexes use beyond what the node's vector budget reserved. It reads zero while every byte an index uses is reserved. |
scramdb_vector_memory_engine_bytes | gauge | Memory the engine holds for vector index ledgers, bitmaps and buffers. |
scramdb_vector_memory_budget_refusals_total | counter | Index builds, rebuilds and arena regrows refused because the total ceiling would be exceeded. |
scramdb_vector_indexes | gauge | Vector indexes open on this node. |
scramdb_vector_regrows_total / scramdb_vector_regrows_running | counter / gauge | Arena regrows completed / running on this node. |
scramdb_vector_scans_total{tier} | counter | Indexed vector searches served, labelled exact, traversal, or postfilter. |
scramdb_vector_scan_seconds | histogram | Time an indexed vector search took end to end. |
scramdb_vector_rerank_rows_total / scramdb_vector_rerank_seconds | counter / histogram | Candidate rows re-ranked exactly after an index search, and how long it took. |
scramdb_vector_widening_rounds_total | counter | Extra index rounds taken when a filtered search had fewer than the requested rows. |
scramdb_vector_underfilled_total | counter | Filtered searches that returned fewer rows than requested after the scan limit. |
scramdb_vector_knn_join_batches_total / scramdb_vector_knn_join_queries_total | counter | Batches / individual query vectors sent to an index by a nearest-neighbour join. |
scramdb_vector_threshold_scans_total | counter | Distance threshold searches served by an index. |
scramdb_vector_exact_scans_total / scramdb_vector_exact_scan_seconds | counter / histogram | Nearest-neighbour searches the vector operator answered with an exact scan, and how long they took. A table scan with no vector operator is not counted. |
scramdb_vector_distributed_scans_total / scramdb_vector_distributed_merge_rows_total | counter | Nearest-neighbour statements fanned out to cluster readers, and rows merged from them. |
scramdb_vector_distributed_redispatches_total | counter | Readers of a distributed nearest-neighbour search whose buckets another replica served, after the reader missed its deadline or could not be reached. |
scramdb_vector_distributed_failures_total | counter | Distributed nearest-neighbour searches that failed because a reader failed or no replica of a bucket was left. |
scramdb_vector_builds_running / scramdb_vector_build_seconds | gauge / histogram | Vector index builds and rebuilds running, and how long one takes. |
scramdb_vector_reconcile_rows_total | counter | Rows replayed into vector indexes at startup to catch up with the table. |
scramdb_vector_build_owed_deletes_total | counter | Deleted rows an index build kept for read snapshots older than the delete, so those snapshots still find them. |
scramdb_vector_delete_tail_reclaims_total | counter | Reclaims that rebuilt a vector index's delete tail (the deleted rows kept for older snapshots) once some of those rows were no longer needed. |
scramdb_vector_delete_tail_batches_dropped_total / scramdb_vector_delete_tail_batches_relinked_total | counter | Delete tail batches a reclaim dropped whole because no read snapshot needs them, and batches it kept whole without copying them. |
scramdb_vector_delete_tail_batches_merged_total / scramdb_vector_delete_tail_boundary_batches_folded_total | counter | Small delete tail batches a reclaim copied into a larger one, and batches it read row by row because a read snapshot's boundary cut through them. Every batch a reclaim judges lands in exactly one of these four counters. |
scramdb_vector_fit_sample_bytes_total | counter | Bytes of training samples vector index model fits charged to execution memory: a build's fits through storage.memory.maintenance_bytes, every other fit (a streaming INSERT or COPY that trains the model, a background retrain) to the execution pool directly. |
scramdb_vector_setting{key} / scramdb_vector_setting_source{key,source} | gauge | The value in force for a [vector] setting, and which layer set it (config, override, setpoint). |
scramdb_vector_series | gauge | Prometheus series the vector index families currently emit, computed from the same scrape it appears in. |
Per index (labels index, table, column, method, group):
| Metric | Type | Meaning |
|---|---|---|
scramdb_vector_index_state{state} | gauge | Whether the index is in this state (building, ready, regrowing, rebuilding, tail, invalid); one state reads 1. |
scramdb_vector_index_live_vectors / scramdb_vector_index_tombstones | gauge | Vectors the index holds and can return, and deleted vectors it still carries until compaction. |
scramdb_vector_index_base_rows | gauge | Rows a branch's index reads through the index it was cut from, until it folds; live_vectors counts only the rows the branch's own index holds. |
scramdb_vector_index_unindexed_rows | gauge | Rows served exactly because the index has no memory left to hold them (the tail state). |
scramdb_vector_index_ledger_live_rows / scramdb_vector_index_delete_tail_rows | gauge | Rows the index's row map holds that are not deleted (what a rebuild sizes the index for), and deleted rows the index keeps for read snapshots older than the delete until a reclaim finds no snapshot needs them. |
scramdb_vector_index_arena_slots / scramdb_vector_index_arena_used_permille | gauge | Slots the index's arena is promised, and the share in use, in thousandths. |
scramdb_vector_index_regrows_total / scramdb_vector_index_regrow_running | counter / gauge | Times the arena was reopened with a larger promise, and whether it is happening now. |
scramdb_vector_index_memory_budget_bytes / scramdb_vector_index_memory_in_use_bytes | gauge | The memory ceiling the index runs under, and resident memory it uses right now. |
scramdb_vector_index_memory_plane_bytes{plane} | gauge | Resident memory by plane: codes, graph, exact_vectors, directory, slot_state, coarse_model, journal_buffers. |
scramdb_vector_index_ledger_bytes / scramdb_vector_index_disk_bytes / scramdb_vector_index_journal_bytes | gauge | Memory the row-id ledger uses, bytes the index occupies on disk, and bytes of write history kept before compaction. |
scramdb_vector_index_ops_total{op} / scramdb_vector_index_ops_failed_total{op} | counter | Operations completed / refused or failed, op one of search, insert, update, delete, flush. |
scramdb_vector_index_search_qps | gauge | Searches per second over the recent window. |
scramdb_vector_index_search_latency_seconds{quantile} / scramdb_vector_index_write_latency_seconds{quantile} | gauge | Search / write latency over the recent window, quantile one of 0.5, 0.95, 0.99, 0.999, max. |
scramdb_vector_index_staleness | gauge | Writes accepted but not yet searchable. Reads 0: a write is searchable by the very next query, so there is nothing for this gauge to ever report as pending. |
scramdb_vector_index_compactions_total / scramdb_vector_index_compaction_running | counter / gauge | Compactions completed, and whether one is running now. |
scramdb_vector_index_fits_total | counter | Coarse model fits the index completed. |
scramdb_vector_index_maintenance_running / scramdb_vector_index_graph_armed | gauge | Whether the index's own maintenance is running, and whether its navigation graph is built and in use. |
scramdb_vector_index_edge_retries_total | counter | Navigation graph edge publications retried because another writer changed the same edge list first. A retry never loses the edge; a high rate means many threads link into the same nodes at once. |
scramdb_vector_index_unplaced_slots | gauge | Vectors the index's last model fit could not place in the partition it chose for them, for want of room there. While that is a material share of the index, the navigation graph stays off. |
scramdb_vector_index_io_reads_total / scramdb_vector_index_io_bytes_read_total | counter | Reads and bytes read for vectors spilled to disk. |
scramdb_vector_index_cache_hit_ratio | gauge | Share of vector reads served from memory. |
scramdb_vector_index_simd_level{level} | gauge | The instruction set the index runs with; the matching level reads 1. |
scramdb_vector_index_knob{knob} | gauge | The search knob value in force for the index. |
scramdb_vector_index_estimator_agreement | gauge | Share of candidates whose estimated order matched the exact order in recent searches. Emitted only once a real measurement exists. |
scramdb_vector_index_recall{k} | gauge | Recall measured by the last recall job for this index. Absent until a recall job has actually run, never a guessed value. |
scramdb_vector_index_last_build_wall_microseconds / scramdb_vector_index_last_build_cpu_microseconds | gauge | Wall time and CPU time (summed over every thread that worked on it) of the index's last build in this process. Absent until the index is built here: a restart reads absent until the next build, never a stale or guessed figure. |
scramdb_vector_index_last_build_peak_memory_bytes / scramdb_vector_index_last_build_rows | gauge | Resident memory of the index at the end of its last build (its peak: a build only adds), and rows it indexed. |
scramdb_vector_index_search_seconds | histogram | Time one search of the index took, end to end. |
scramdb_vector_index_last_flush_unix_seconds / scramdb_vector_index_rows_since_flush | gauge | When the index last made its writes durable, and rows written since then. |
scramdb_vector_index_last_reindex_unix_seconds | gauge | When the index was last rebuilt. Emitted only once a REINDEX has actually run. |
Cluster (labels index, table, group; present only once a distributed vector index
part exists):
| Metric | Type | Meaning |
|---|---|---|
scramdb_vector_group_owner_is_self | gauge | Whether this node owns the bucket group's index part. |
scramdb_vector_group_part_state{state} | gauge | The state of the index part for the bucket group (building, ready, regrowing, rebuilding, tail, invalid, absent, waiting_memory: a hosted bucket group this pass's own memory admission left out of the count it admitted to build). |
scramdb_vector_group_rebuilds_total / scramdb_vector_group_rebuild_running | counter / gauge | Times the index part was rebuilt after the bucket group moved, and whether one is running now. |
scramdb_vector_group_learner_ready | gauge | Whether a learner may serve vector searches for the bucket group. |
scramdb_vector_group_memory_in_use_bytes | gauge | Resident memory the index part for the bucket group uses. |
scramdb_vector_cluster_parts / scramdb_vector_cluster_parts_missing | gauge | Index parts this node holds for distributed tables, and bucket groups it owns whose part is not built. |
scramdb_vector_cluster_min_recall{index,k} | gauge | The lowest recall any of this node's parts of the index measured at its last recall run. Absent until measured; the whole cluster's lowest is the lowest over every node. |
Recall job (present only on a cluster node, where the job runs):
| Metric | Type | Meaning |
|---|---|---|
scramdb_vector_recall_parts_measured_total / scramdb_vector_recall_parts_skipped_total | counter | Index parts the recall job measured on this node, and parts a pass was due to measure and could not (each such part says why in scram_vector_index_parts.recall_note). |
scramdb_vector_recall_pass_cpu_seconds | histogram | CPU time of one recall pass on this node. |
scramdb_vector_recall_next_wait_seconds{reason} | gauge | Seconds until the job next measures a part, and why: churn (the wait follows how much of a part changed), cost (the one-percent CPU duty cycle binds), ceiling (a lightly changed part waits the hour ceiling), pinned (vector.recall_interval_secs), first (a part never measured), unchanged (nothing changed since the last measurement, so nothing is due: +Inf), or off. |
scramdb_vector_recall_probes{reason} | gauge | Probe queries the last measurement sampled, and why: precision (enough for a one-point standard error at the part's last recall), floor (8), ceiling (64), memory (the vector memory budget held fewer), or pinned (vector.recall_probes). |
Vector index system views
The same sample backs five read-only system views, so a dashboard that prefers SQL
over scraping Prometheus sees the identical numbers. Every row leads with node, the node
that measured it.
On a cluster, the four statistics views answer for the whole cluster from whichever node you
query: that node's rows and every other member's, gathered at query time. A member that does
not answer within vector.stats_node_timeout_ms (default 5000) contributes one row with its
node, the word unreachable in state (or metric), the reason in error, and NULL
everywhere else, never a silently shorter result. scram_vector_settings is the queried
node's own settings. On a single node every view is that node's rows.
| View | Columns |
|---|---|
scram_vector_settings | node, key, value, source, unit, live, description - every [vector] key, the value in force, and which layer set it |
scram_vector_indexes | node, index, schema, table, column, method, metric, opclass, state, arena_slots, arena_used_permille, regrows, live, base_rows, tombstones, unindexed_rows, memory_budget, memory_in_use, disk_bytes, journal_bytes, search_qps, p50_us, p99_us, staleness, compactions, fits, graph_armed, simd_level, estimator_agreement, recall_at_10, last_flush, last_reindex, options, build_wall_us, build_cpu_us, build_peak_memory, build_rows, search_count, search_p99_us, error - one row per index per node; a distributed index's parts on a node fold into its row (build cost summed, NULL unless every part was built by that process; recall the lowest part's) |
scram_vector_index_parts | node, index, schema, table, group, owner, is_self, state, live, memory_in_use, rebuilds, learner_ready, rebuild_running, recall_at_10, recall_probes, recall_measured_at, recall_note, error - one row per bucket group each node hosts; is_self marks the rows of the node you queried; recall_note says why a part has no recall figure (too few vectors, over vector.recall_max_rows, not ready, or no memory); empty on a single node |
scram_vector_memory | node, total, reserved, in_use, engine_bytes, refusals, error - one summary row per node |
scram_vector_activity | node, metric, value, unit, description, error - the node-wide vector counters, one row per counter per node |
SELECT node, index, state, live, memory_in_use, search_qps FROM scram_vector_indexes;
SELECT node, "group", state, live, recall_at_10, recall_note
FROM scram_vector_index_parts WHERE index = 'docs_embedding_idx' OR error IS NOT NULL;
SELECT key, value, source, live FROM scram_vector_settings WHERE key = 'vector.over_fetch';
EXPLAIN on an indexed vector query names the tier chosen and the candidate count;
EXPLAIN ANALYZE adds candidates fetched, widening rounds taken, re-rank time, and
whether the answer came back underfilled. See Vector Search
for the query shapes these views and counters describe, and Tuning: Vector
Indexes for how to act on them.
GPU metrics (present once a GPU is in use)
On a server that brought up a GPU at startup, with [gpu] metrics_enabled = true (the default). A server without one shows no scramdb_gpu_* family at all.
| Metric | Type | Meaning |
|---|---|---|
scramdb_gpu_kernel_launches_total | counter | GPU kernels launched. |
scramdb_gpu_kernel_errors_total | counter | GPU kernel launches the device refused. |
scramdb_gpu_fallback_total | counter | Times work admitted to the GPU ran on the CPU instead (a morsel holding a NULL, a condition the GPU row filter cannot read exactly, an unhealthy device). |
scramdb_gpu_oom_total | counter | GPU memory requests refused for lack of device memory. |
scramdb_gpu_transfer_htod_bytes_total | counter | Bytes copied from host memory to GPU memory. |
scramdb_gpu_transfer_dtoh_bytes_total | counter | Bytes copied from GPU memory to host memory. |
scramdb_gpu_vram_used_bytes | gauge | GPU memory held by queries now. |
scramdb_gpu_vram_budget_bytes | gauge | GPU memory queries may hold. |
A query ran on the GPU when scramdb_gpu_kernel_launches_total moved across it and scramdb_gpu_fallback_total did not; see GPU Acceleration.
What is not on this endpoint
- No buffer-pool hit-ratio, cache-hit-ratio, or per-worker-utilization metric exists. Memory and parallelism verification on this site relies on
\timingand OS-level observation instead, see Memory and Parallelism.
Log levels and format
SCRAMDB_LOG=debug SCRAMDB_LOG_FORMAT=json scramdb -c scramdb-config.toml --pg-address 127.0.0.1:5432 --pg-no-auth
Expected: stderr lines shaped like {"ts":"...","level":"INFO","target":"...","msg":"Metrics endpoint listening on http://0.0.0.0:9090/metrics"}. Every record is exactly one line: a control character in a message is escaped inside the JSON string.
SCRAMDB_LOG
Controls the level filter. Accepted values (case-insensitive): off, error, warn, info, debug, trace. Default when unset: warn, so a release binary starts quiet. An unrecognized value does not fail loud, it silently falls back to warn, so a typo in this env var will not show up as an error, it just quietly serves you the default level.
SCRAMDB_LOG_FORMAT
Set to json (case-insensitive) for structured JSON output. Anything else, including leaving it unset, produces the default plain-text format.
Plain-text format: [{level}] [{timestamp}] {message}, for example:
[INFO] [2026-08-02T10:15:30.123Z] Metrics endpoint listening on http://0.0.0.0:9090/metrics
The timestamp of both formats is UTC, to the millisecond, marked Z, whatever zone the host is set to.
JSON format: {"ts":"<timestamp>","level":"<LEVEL>","target":"<module path>","msg":"<message>"}.
A WARN or ERROR line logged while a statement runs ends with the statement's context, (pid N, database D, user U, statement shape <16 hex digits> "<masked start>"), so a line can be tied to its session and statement shape. The masked start never holds a client value. Lines at INFO and below, and lines outside a statement, carry no context.
Logging is asynchronous: a dedicated low-priority background thread drains log messages and writes to stderr, so logging never blocks a query worker. Output always goes to stderr, never directly to a file, redirect it yourself (scramdb ... 2> scramdb.log) or capture it with your process supervisor.
Related diagnostic env vars
One additional env var is useful for deeper diagnosis alongside SCRAMDB_LOG, covered on another page in this section:
SCRAMDB_GPU_FORCE=1, forces GPU pipeline-shape eligibility for benchmarking. See GPU Acceleration.
Next
See Query Performance for how the JIT chunk counters are used in practice, GPU Acceleration for how to tell from the scramdb_gpu_* counters whether a query ran on the GPU, or Clustering for how the cluster counters above map onto a Docker Compose or Kubernetes deployment.