Failover: what happens when a node dies

By the end of this page you will know the two timers that govern how fast a ScramDB cluster reacts to a dead node, exactly what a connected client experiences during that window, and how a node rejoins once it comes back.

Two independent timers

Two separate things have to notice a node is gone, and they run on different clocks.

1. Cluster membership detection. Governed by [cluster.swim]'s ping_timeout (500 milliseconds by default), indirect_timeout (500 milliseconds), a dynamic suspicion window bounded by suspicion_min and suspicion_max (1 to 5 seconds), and a 1 second protocol tick. Under defaults, a dead peer typically moves from alive to suspect to dead within a few seconds; this is a dynamic, corroborated timeout, not a fixed one, so treat "a few seconds" as the honest answer rather than an exact figure. You will see this in a node's logs as swim: member up, swim: member suspect, and swim: member down lines naming the peer.

A leader that is alive but cannot write is treated as failed too. Both timers above notice a node that has gone silent. A subtler failure is a leader whose replication is healthy (it still sends heartbeats, so nobody elects around it) but whose local storage has died, so it can never actually apply anything again. ScramDB stops that replica deliberately when its apply path fails: heartbeats cease, and the ordinary election below happens within a round. Without that, the group would keep a leader that could accept nothing, forever.

2. Leadership re-election for the affected shard group. Every shard group, including the cluster's own metadata group, elects a new leader once its current leader goes quiet, using the randomized window between election_timeout_min (1 second) and election_timeout_max (2 seconds) described in Multi-zone. This is independent of, and generally faster than, cluster-wide membership detection: a group's own replicas notice a silent leader directly from the absence of its heartbeats, they do not wait for the membership layer to catch up.

What a connected client sees

There is no separate coordinator or query router process to fail over. Any node accepts a normal PostgreSQL wire connection and internally forwards or coordinates work against whichever node currently leads the relevant data.

• A write whose data is led by a different node is forwarded internally, not rejected back to you to reconnect elsewhere.
• In the common case, a leader failover mid-statement is retried internally and is transparent to the client. This is documented, tested behavior for the ordinary case, not a guarantee that every possible timing is invisible. A statement forwarded to a leader that dies mid-flight does not wait out its full forwarding window: the node re-resolves the leader on every attempt and races the in-flight call against its own view of leadership, so it switches to the new leader as soon as one is elected.
• If internal retries are exhausted, or a genuine write conflict is detected, you get a normal PostgreSQL error with SQLSTATE 40001 (serialization_failure), the same error class PostgreSQL's own serializable isolation already trains a client to expect. The correct, and only necessary, client-side handling is: catch 40001 and retry the transaction. No ScramDB-specific error code or client library is involved; any PostgreSQL driver's normal retry-on-40001 logic works unmodified.
• A crashed leader loses no committed write. Once a new leader is elected, the surviving replicas keep accepting reads and writes with byte-equal data across every replica.
• A crashed voter is never automatically removed from cluster membership. This is a deliberate, permanent invariant, not a bug: automatically dropping a member could put the remaining majority at risk. The dead node stays a phantom member (visible as suspect or dead in membership) until it comes back or an operator explicitly addresses it, covered in Scaling.
• If your own connection was to the node that died (not just to a shard's leader elsewhere), that TCP connection drops. That is unavoidable with a normal PostgreSQL wire connection to a single socket; ScramDB does not ship a connection-level proxy or load balancer, and it does not ship a client-side automatic-failover driver either, it is plain PostgreSQL wire protocol. Routing a reconnect to a live node, whether through a TCP load balancer in front of the node set, a multi-host client connection string, or your own pooler, is your own deployment's responsibility, exactly as it is for any PostgreSQL-wire-protocol cluster.

Recovery: how a crashed node rejoins

Once you restart the crashed node's process (same config, same data directory), it rejoins on its own: no special "rejoin procedure" beyond starting it again. Expect to see the sequence suspect then dead then, on rejoin, up in the survivors' membership logs for that node's id. After rejoining, it catches up through normal log replication for whatever it missed while it was down.

A returning node is not blocked by any license logic; a license only gates whether cluster mode is allowed to start at all, and its node-count ceiling is not something the engine enforces automatically at join time today. Once a node is running cluster mode, rejoin is governed purely by the underlying consensus and replication mechanics above.

That's the ordinary case: restart, and catch up. A node whose own storage comes back corrupted, rather than just behind, goes through a different, still-automatic path.

Storage corruption: quarantine and automatic rejoin

A crash that leaves a node's own on-disk storage for one replication group corrupted (a bad header, a torn write mid-CRC, an undecodable piece of state) is a different failure shape than the plain crash-and-restart above, and ScramDB handles it differently: automatically, without crash-looping, and without losing anything.

What counts as corruption, and what doesn't. An ordinary torn write, the normal result of a process dying mid-append, is not corruption at this level: storage already truncates back to the last verified entry and boots normally from there, the same benign path a plain crash always took. A disk that can't be read at all is not quarantined either; that's an environment failure, not a damaged artifact, and silently rebuilding state on top of unreadable storage would be its own data-loss bug, so it stays a loud, unresolved error instead until it's fixed. Quarantine is specifically for the case in between: the file is readable, and it's wrong.

What happens when it is. The affected group's storage directory (and only that group's, never the whole node's) is renamed to <original-directory>.quarantine-<timestamp> and a fresh, empty one opens in its place. Nothing is deleted; the quarantined copy sits on disk for as long as you want to inspect or salvage it by hand. Every other group this node hosts, including every other table's shard groups, keeps being served normally; one corrupt group never crash-loops a healthy node.

Rejoining is automatic. The fresh replica rejoins the group through membership replace: the group removes its old identity, re-adds it as a learner, lets it catch up from its healthy peers rather than trust anything on the damaged disk, and promotes it back to a voter once it has. Until that promotion lands, the replica is vote-fenced: it cannot grant a vote, grant a pre-vote, or start a campaign of its own, so a replica that just lost its history cannot go on to decide, or corrupt, an election while it's still catching up.

It's on /metrics. scramdb_cluster_group_quarantines_total counts every quarantine that's ever happened, scramdb_cluster_quarantined_groups is the live gauge, back to zero once every quarantined group has rejoined, and scramdb_cluster_group_rejoins_total counts completed rejoins. A cluster whose disks never corrupt anything reads zero across all three, permanently; that's the expected, healthy number, not a sign nothing is being measured.

Log compaction keeps restarts bounded

Every shard group's consensus log grows by one entry per committed write. A log that only ever grows makes every future restart replay more of it than the last, until eventually a restart can't finish inside any reasonable deadline; left unchecked, that's a recovery problem wearing a disk-space costume, not just a storage one. ScramDB's groups compact their own log automatically to bound exactly that.

How it's triggered. The group's current leader checks after every batch of proposals lands (cheap when the log is small: one subtraction and an early return), and attempts compaction once the log holds more than 10,000 entries since the last one.

How it's bounded. Compaction never discards more than every voter and learner in the group has already replicated, the minimum progress across the whole group, with a small margin (1,000 entries) held back besides, so a peer that's only ordinarily behind keeps catching up through normal log replication instead of suddenly needing a full resync. In practice this means compaction is bounded by whichever replica is making the slowest progress: if one peer is behind, the log simply keeps what that peer still needs, and a warning names the specific peer and its last known progress, so there's something concrete to go check rather than a number that just climbs with no explanation. An uncommitted entry is never discarded either way; that boundary is a hard refusal, not a heuristic.

What this means for you. A restart replays a bounded amount of log, not the group's entire history since it was created, on any group that has ever crossed the compaction threshold, precisely because the log itself never grows past the slowest live replica's own reach. scramdb_cluster_log_compactions_total on /metrics counts how many compactions have run; a healthy cluster's number climbs steadily under write load, and that climbing is what it's supposed to do.

What the cluster guarantees under failure

Put together, this is what ScramDB actually guarantees when something goes wrong, stated plainly rather than left as separate mechanisms to piece together yourself:

• A voter lost within quorum keeps serving. No client action needed; the surviving majority carries on.
• A replica lost degrades routing, not correctness. Reads fall back to the group's owners; you get a slower or less local answer, never a wrong one.
• Overload sheds analytics first, transactions last. If something has to give under load, it isn't your write path.
• Storage corruption quarantines one group, never a node, and the group rejoins automatically with nothing deleted, exactly as described above.
• Every refusal is loud and names its reason. There is no silent fallback, and no fabricated result standing in for one that couldn't actually be computed.

Next

• Scaling for what your options are once a node is confirmed gone for good: a graceful shutdown now hands its membership back automatically, and why a crash still leaves a permanent phantom member behind until you act.
• Multi-zone and Multi-region for how far a group's replicas can be from each other before these timers need retuning.