Cluster Restore
By the end of this page you will have restored every node of a ScramDB cluster to one consistent point in the past, the latest point the archive holds or a moment you choose, and started the restored nodes as a new cluster.
A cluster restore brings every shard group, and the cluster's own metadata (its tables, placement and node numbers), back to ONE commit position shared by every group. A transaction that wrote several groups is restored on all of them or on none, so the restored cluster never holds half of a transaction, whichever point you pick. It is the cluster's counterpart of point-in-time recovery on a single node, and it reads the same archive destination.
What the archive holds
With [storage.wal.archive] enabled on every node, each shard group's leader ships the group's committed log to the archive destination, and the leader of the cluster's metadata group ships the metadata log. Only committed log entries are shipped, and each segment is stored under a key only one leader can create, so a node that lost its leadership can never overwrite or interleave what the new leader shipped. When a group's log was compacted past what the archive holds (archiving switched on for a running cluster, for example), the group's leader ships a base: the group's state and a full copy of its rows, and the log continues from there. Each node's own storage log goes to its own place in the same destination, never mixed with another node's.
Point every node at the SAME destination, typically an object store, so the archive holds every group whichever node led it:
[storage.wal.archive]
enabled = true
destination = "s3://my-bucket/scramdb-archive"
# retention = "168h" # keep what a restore to any point of the last seven days needs
A destination left unset archives into each node's own data directory, which holds only the groups that node led: a cluster restore needs one shared destination. The archive's own metrics (scramdb_cluster_archive_* on Observability) show what each node ships, and scramdb_cluster_archive_gaps_total above zero says a group's history has a hole a restore cannot cross.
Choosing a target
| Flag | Restores to |
|---|---|
| none | The latest point every group of the archive can be restored to. |
--target-time <TIMESTAMP> | The state as of that time: every transaction acknowledged by then, and none after (2026-09-24T10:00:00Z, or 2026-09-24 10:00:00 read as UTC). |
--target-position <POSITION> | Exactly that commit position. |
--target-name <NAME> | A named restore point, made on the cluster with scram.create_restore_point (below). |
Give at most one. A time is turned into a point through a time index the cluster records as it commits: every node notes the commits it acknowledges, and every shard group's leader notes how far it has applied its log. The clock only helps you name the moment; it never decides what the restored point contains, which comes from the groups' own logs: a shard group that commits less often than the others gives the restore nothing it committed after the time, and a transaction that wrote several groups is restored on all of them or on none. Transactions and metadata changes (a CREATE TABLE, an ALTER TABLE) are placed against the target to within one millisecond.
Naming a restore point
Before a risky change, name the moment on any node, as a superuser:
SELECT scram.create_restore_point('before-migration');
Expect: one row with the commit position of the point. Every shard group is fenced at that position as the point is made, so a restore to it (--target-name before-migration) holds exactly what committed before it. A name is at most 63 bytes. Making a point under a name already taken keeps the first point, and answers its position; a restore to the name reaches that first point. PostgreSQL's own SELECT pg_create_restore_point('before-migration') makes the same point and answers its position in pg_lsn form.
Step by step
-
Plan the restore, once. From any machine that can read the archive:
bashscramdb restore-cluster plan --archive s3://my-bucket/scramdb-archive \--target-time "2026-09-24T10:00:00Z"Expect:
restore <ID> of <STREAM>: every group at commit position <P>, metadata through log index <M>, followed by the exact command to run on every node. For--target-timethe plan says what it holds on every group:restore <ID> of <STREAM>: the target time's cut at commit position <P>, <N> transaction(s) in flight at the time admitted, metadata through log index <M>, then one line per shard group,group <G>: log index <I> (base <B>), the log position the group had applied by the time (ornothing applied by the time). With no target, a second line names what bounds the point:the newest point every group can be restored to is commit position <P>, where group <G> closes; the other groups close at or after it. The plan is recorded in the archive, so every node restores to the same point even while the archive keeps growing.--streamnames the archived cluster when the archive holds more than one (the command lists them). -
Restore every node, before it starts. On each node, with the node's own configuration file and an empty data directory:
bashscramdb restore-cluster node --config /etc/scramdb/config.toml \--archive s3://my-bucket/scramdb-archive --stream <STREAM> --restore <ID>Expect:
node <NAME> restored to commit position <P>: <N> group(s), <R> row(s). The node gets the restored tables first, then the rows of every shard group it holds a replica of (a learner's into its columnar store), each group's log, and the cluster metadata. A data directory that is not empty is refused: restore into a fresh one.If one table's own data no longer matched what it committed (a torn or truncated segment, for example), the node prints one extra line per affected table rather than failing the whole restore over it:
texttable orders (row store) is quarantined: catalogued with 14 page(s) but the file holds only 0 - torn or truncated - restore it from a backup, or DROP and recreate it, to clear the quarantineA quarantined table refuses every read and write on the node it was found on; every other table, and every other node, restores normally. Restore just that table from a backup that predates the divergence, or
DROP TABLEand recreate it once the restored cluster is up. -
Stop every node of the old cluster (or keep it on a network the restored nodes cannot reach), then start the restored nodes as you normally start them. They form a new cluster at the restored point: every group's log starts with a marker at that position, so every new commit lands after every restored row, and the restored cluster archives into a new stream that records the one it came from and the point it was restored to.
The latest point and idle shard groups
The latest point trails the newest acknowledged commit by a few archive passes (each one [storage.wal.archive] poll_interval, 5s by default), even when a shard group has had no commit for a while: when a group is behind the newest commit a node acknowledged and has not moved since the previous pass, the archiver asks it for a fence, which brings the group's position up to that commit, so a restore right after a commit is not sent back to before it and you need not run a statement that touches every group first. A cluster with nothing new to learn asks nothing: scramdb_cluster_archive_cycles_total counts the passes of a node's archiver and scramdb_cluster_archive_fence_asks_total the requests it made, and the second stays still on an idle cluster (Observability).
Prepared transactions
A transaction prepared with PREPARE TRANSACTION at or before the target is in the restored cluster as it was prepared: pg_prepared_xacts lists it on every node, its rows stay staged, and COMMIT PREPARED or ROLLBACK PREPARED from any node ends it, the commit applying once. A transaction that wrote several shard groups is kept in each of them. One prepared after the target is not there, and one that ended before the target leaves nothing to end.
Tables renamed or rewritten since the base
A restore to a time after a table was renamed, after two tables swapped their names, or after a schema that holds several tables was renamed, puts each table's rows in the table that has them: the restore follows the renames through the archive, so a row logged under an older name is not lost and is not put in the table that took the name since. A CLONE made before the target is restored the same way, its branch holding the renamed table's rows. A restore past a table rewrite (a column's type change, a column added with a volatile default) holds the values the rewrite drew for the rows written before it, including the rows of the last commit before the change. If the node no longer holds the values a rewrite staged, the restore fails with the values a rewrite of table "t" staged are not on this node, so its restored rows would hold the column's default; restore again from the archive and never restores the column's default without saying so.
Verifying you landed where you meant
Query for state that differs on either side of your target, on more than one node: a row committed just before it that must be present, a change just after it that must be absent, and a total that must match what you knew at that moment. Because every group is restored to one position, a transaction you know committed before the target is present on every node whole, and one that committed after it is absent everywhere.
Next
- Point-in-time recovery: the single-node form of the same idea.
- Observability: the cluster archive's metrics.