Skip to main content

Upgrading

From ScramDB 1.0.0 on, every later 1.x release opens a 1.0.0 data directory and its backups with no conversion step and no loss, and reads its write-ahead log and its archive; what a release replays of an older release's log is stated under "Replay across versions". Data written by a newer release than the one reading it is refused with one sentence; it is never read as something else.

What is promised​

  • Forward reads. Segment files, pages, the log, the catalog, vector indexes and cluster records written by 1.0.0 are read by every 1.x release.
  • Replay across versions. A later 1.x release reads the log and the archive an older release wrote, and replays their page images and deletions. It does not replay a committed write of rows that the older release logged and no page holds yet: it stops and says so, naming the table and the log position. For each use:
    • Crash recovery. A log whose row writes are all on pages already replays, and the start goes on. A log that holds a committed write of rows no page holds yet refuses the start, with a message that says to start the older version once, shut it down cleanly, then start the new one. Do that before the upgrade.
    • Point-in-time restore and a base backup's restore. A base backup an older release took is accepted. The restore replays the archived log from the backup with the same rule, up to the time you chose, and stops with the same message when the log up to that time holds such a write. Take a new base backup after the upgrade for restores to times after it. Do a restore to a time before the upgrade with the older release.
    • Branching at a past time. The branch replays the log from the base backup to its time with the same rule, and the statement fails with the same message when that log holds such a write. Take a new base backup after the upgrade for branches at later times. Make a branch at a time before the upgrade with the older release.
  • Data from before 1.0.0 is refused. A data directory with no format version file was written by a pre-release build; the server stops at start and says so. Load its data into a new directory with a dump and restore.
  • Newer data is refused. An older server that meets a directory, log file, page, backup or cluster file of a newer format stops with a message naming the file and saying a newer ScramDB wrote it. Nothing is changed.
  • The data format version. The file FORMAT_VERSION at the root of the data directory holds the on-disk format version (1 for 1.0.0; 2 in the release that keeps the row places of merged segments on disk; 3 in the release whose log records which insert wrote each page image; 4 in the release that writes the log's row and commit records in a fixed layout of their own and logs a table's columns once, in a record of their own, where every row record carried them; 5 in the release whose commit records name the clock that drew their stamp and whose table entries record whether a cluster placed the table; 6 in the release whose vector index ledger marks each record placed once the load that reserved its id has decided its row). A newer server opens an older directory. A release that changes a format raises the number and states the one-way step in its release notes.

Compiled-code caches and other derived files are rebuilt when they do not match; they are not part of the promise. The cluster protocols are negotiated between running nodes, and a node outside the supported range is refused when it connects.

What the first start on this release does​

Data format 6. A vector index keeps a ledger of its rows on disk, and each record of it now carries a placed byte, 0 until the load that reserved the row's id has decided the row's claim and 1 after. A rebuild of the index's row map beside a running load reads only placed records, and recovery voids an id no load placed. A ledger of the older layout cannot be read under that meaning, so the first start on a directory stamped 5 removes every file under the data directory's vector_index/ and then stamps FORMAT_VERSION 6. Each vector index is then built again from its table at that start, as CREATE INDEX builds it: the time that takes is the cost of the upgrade, and no table data is touched. The step is safe to run again, so a start killed before the stamp runs it once more, and it logs one line when it removes something. An older release refuses a directory stamped 6, so roll back from the base backup taken before the upgrade (see Roll back).

Data format 5. Each commit record of the write-ahead log now names the clock that drew its stamp, the server's own or a position of a shard group's log that a cluster member applied, and each table's catalog entry records whether a cluster placed the table. After a crash, recovery judges a commit of the server's own clock exactly, by the commit records of the log and by a settled position that a checkpoint stores in catalog/_settled_position.dat; a commit a cluster applied is judged as it was.

The first start on a directory stamped 4 rewrites every table entry and fork record of the catalog once, then stamps FORMAT_VERSION 5. Each file is replaced as a whole, so a start that is killed before the stamp runs the step again, and every older table is marked as placed by a cluster, so nothing an older store held is judged more strictly than before. A table a local statement makes later is not marked.

On a server that is not a member of a cluster, every table is then judged as the server's own. That happens at the first start, once every commit in the log is decided: the server stores the settled position first, then clears the mark of each older table. If the log still holds a commit a cluster applied, or a table made by a transaction block is waiting to be dropped, it happens at a later start. A cluster member leaves the marks as the step set them. A directory with no settled position (every directory an older release wrote) is judged by the stamped maximum, as it was, until its first checkpoint stores one.

An older release refuses a directory stamped 5, so roll back from the base backup taken before the upgrade (see Roll back).

Log format 8. A log file's header carries a flag saying it takes over the file before it (see Storage Engine). Every reader of the log, a start's replay, a restore, a branch at a past time and the search of a commit time, reads the file left behind only up to where the new file starts. Nothing is rewritten at the upgrade: a log of format 3 to 7 reads as before, its commit records counting for both clocks. A release that does not know format 8 refuses the log by its version check and never reads what the flag hides. An archive made before this format restores as before.

The files of routines and functions. _routines.dat holds the functions and procedures of every database, and its new layout records the language of a routine as one of the package languages (js, ts, python, ruby) as well as sql and plpgsql, and the entry function of a python or ruby procedure, so a procedure renamed or moved keeps running. _udf_catalog.dat holds the functions written in the package languages, and its new layout adds each function's SECURITY DEFINER and SET clause to the entry function of a python or ruby function. Nothing is rewritten at the upgrade: this release reads every older layout of both files with each routine, its identity and its definition kept (a function saved before the clauses reads as one with neither), and writes the new layouts the next time a function or a procedure is created, changed or dropped. A release older than this one cannot read the new layouts.

A catalog file the start cannot read. The catalog keeps a file for each kind of object: the routines, comments, owners, casts, event triggers, types, sequences, schemas, views, the functions of the package languages, the rules in a table's entry, and the roles, privileges, default privileges, routine access lists, server settings, agents and the settings ALTER SYSTEM wrote. A start that finds one and cannot read it (a layout from a newer ScramDB, a file cut short or damaged, or a read that failed) does not start: it answers one sentence that names the file and, for a layout written by a newer ScramDB, says that a newer ScramDB wrote it, and nothing is written to the file. Start the build that wrote the directory, or restore the directory from its backup. A file that is absent is an empty catalog, and the server starts. Do not start an older build over a directory this release has written.

A first start that is killed while it stamps the format version of an empty directory does not leave a directory the next start refuses: the stamp's leftovers are swept, and the next start stamps the directory. A directory that holds data and no stamp is still refused as written before data format versions existed.

Upgrade a single node​

  1. Take a base backup and keep it.
  2. Finish prepared transactions: SELECT gid FROM pg_prepared_xacts, then COMMIT PREPARED or ROLLBACK PREPARED.
  3. Stop the node cleanly.
  4. Start the new version on the same data directory.
  5. Check SELECT version() and query your tables.

You can skip releases: each 1.x release reads everything the 1.x releases before it wrote.

Stop the old node cleanly. A node that stopped cleanly leaves a mark that lets the new release start without replaying the old log. After a crash, the old log can hold writes whose rows are not on disk pages yet. The new release replays the old log's page images and deletions, but it cannot replay a write of rows made by an older version, and refuses the start with a message that says so, names the table and the log position, and tells you to start that version once, shut it down cleanly, then start the new one. Do that, then start the new release.

A table whose catalog file the new release cannot read also stops the start, with a message that names the table and the file; the server never starts with the table missing.

Upgrade a cluster node by node​

Upgrade one node at a time, followers first and the leader of each group last.

  1. Confirm every group has all its replicas healthy and caught up.
  2. Stop the node and start it on the new version with the same storage directory. Wait until it rejoins and every group it hosts is caught up.
  3. Repeat for the next node.

While versions are mixed, the cluster speaks the highest protocol every peer supports. Keep the mixed period short, and do not change schemas during it.

A shuffled join sends each row to a node by a hash of its key, and a distributed runtime filter is built from the same hash. Cluster protocol version 7 uses a newer hash. The node that runs a statement names the hash in what it sends to every node of the statement, and names the newer one only when every node the statement uses speaks version 7. While one of them is older, every node of that statement routes by the older hash, so equal keys still reach one node and the answer is the same. The older hash is used until the last node is upgraded, and a node of the older version is sent no runtime filter of the newer hash and runs that join unfiltered. Nothing needs doing: scramdb_cluster_route_hash_v1_total counts the shuffles and runtime filters a node routed by the older hash and scramdb_cluster_route_hash_v2_total those routed by the newer one, and the first stops rising once the last node runs version 7 (Observability).

A change that only the newer version can read is held back until every member runs it: while an older member is in the cluster, the statement answers that every node needs a newer ScramDB, and runs once they are all upgraded. A member that nevertheless meets a committed change it cannot read stops serving and says to upgrade it, rather than skip the change and fall out of step with the others.

Roll back​

  • Before the first start on the new version, start the old version again. Nothing changed.
  • After the new version served writes, restore the base backup from before the upgrade, and replay the archived log to the point you choose with point-in-time recovery.
  • If the release raised the format version in FORMAT_VERSION, the old server refuses the directory by design. Never lower the number by hand.
  • An older release that meets a catalog file in a layout it does not know refuses the start naming the file (see A catalog file the start cannot read); the base backup is the way back.