Skip to main content

Storage Engine

Tundra is ScramDB's storage engine. It keeps your data in a single columnar copy that is fast to write to and fast to scan, so transactions and analytics run on the same live data without stepping on each other.

One copy for writes and scans​

Traditional setups keep two shapes of your data: a row-oriented copy tuned for transactions and inserts, and a separate column-oriented copy tuned for analytics, kept in sync by a pipeline. Tundra keeps one copy and serves both.

  • Writes are fast. New and changed rows land in a write-optimized area and become visible immediately, so transactions stay quick.
  • Scans are fast. In the background, that data is folded into a compressed columnar layout, the format analytical queries love, so large scans and aggregations fly.
  • Readers and writers never block each other. ScramDB uses multi-version concurrency control (MVCC): a reader sees a stable snapshot of the data while writers keep working. It supports the standard isolation levels you expect, up to serializable.

The practical upshot: you insert an order and analyze it in the same breath, with no copy to keep in sync and no lag between the two.

Skipping the data you do not need​

Tundra keeps lightweight summaries (such as the minimum and maximum values, and other per-block filters) for each block of data. When a query filters on a column, ScramDB uses these summaries to skip whole blocks that cannot match, so a query that touches a small slice of a large table only reads the part it needs.

  • The summaries are cached on disk. Each segment file in your data directory also gets a small .hdr sidecar file beside it, holding a compact copy of the block summaries so ScramDB does not need to re-read every block to rebuild them. A sidecar is never read at startup, is rebuilt automatically if it is missing or out of date, and is removed along with its segment, so you can expect to see a modest number of extra files, and a modest amount of extra disk space, alongside your data.
  • Some aggregates skip rows entirely. ScramDB also keeps an exact minimum, maximum, and sum for each column alongside these summaries, so an aggregate like a minimum, maximum, or total over a whole column can often be answered directly from stored statistics instead of scanning the table. This still holds after VACUUM rewrites and merges the underlying data.

Durable and self-checking​

Your committed data is safe.

  • Crash recovery. Every change is first recorded in a write-ahead log before it is applied. If the server crashes or loses power, ScramDB replays the log on restart and comes back consistent, with no committed data lost. Recovery also removes rows that reached disk but were never acknowledged to any client, so a crash can never resurrect a write nobody was told had succeeded.
  • A commit becomes visible only once it is durable. Other sessions see a transaction's changes only after its commit record is safely in the write-ahead log, on every node and for replicated writes too. With synchronous_commit off for the session, a commit is visible as soon as its record is written, before it is durable, and a crash may lose it whole, never half of it (Synchronous Commit).
  • A failed sync is repaired by writing again into a new file, never by retrying the sync. If a log sync fails, the operating system is saying the data may not be on the disk, and syncing the same file again proves nothing, because the pages it would retry may already have been dropped. ScramDB keeps the bytes the failed sync had not made durable, writes them into a new log file and syncs the file and its directory. When that works, the commits that waited answer as usual and the log goes on in the new file, with one warning in the server log. When it does not, the log is failed: the commits that waited answer 08007 (their outcome is not known yet, and no session reads them), every later write answers 58030 with The server cannot write its log, and reads keep answering what was durable when the log failed. The server tries again on a pause that starts at 100 milliseconds and doubles up to 10 seconds, and once its storage answers it writes the kept bytes, writes a record that takes back each commit it answered in doubt, and takes writes again with no restart. A checkpoint is refused while the log is failed, so a failed server cannot mark unsafe data as safely stored. Otherwise writes stop until the server is restarted on working storage, which replays the durable prefix of the log; the commits that waited answer 08007 at once, so none hangs. Whether the record of a commit answered in doubt reached the disk is only known later, so treat its outcome like that of a lost connection: check before you retry it. scramdb_wal_failed reads 1 while the log is failed and /health answers 503 with the cause (Observability, Error codes).
  • A full disk refuses a write whole, and nothing refused happens later. A write the log has no room for answers 53100, and the log goes on taking writes once there is room, with no restart. A statement that refused for room gives up every record it appended, so a DROP TABLE, a TRUNCATE, an ALTER TABLE ... RENAME TO, a restore point and a background compaction swap that answered 53100 are never found done after a restart, and a CREATE TABLE leaves a whole table or none. A statement that cannot tell, because one record of it is already in the log, answers 08007: check before you run it again (Error codes). A record larger than the log's buffer that the disk refuses is in doubt the same way and never stops the log.
  • A data file is published only once it is durable. A new data file is written and flushed, then its directory is flushed, and only then does the catalog, the log or a checkpoint name it. The log is cut only after that, so a power cut that takes a new file back leaves the log that rebuilds it. The directory flushes are batched: one per directory for a whole commit group or checkpoint, not one per file, and scramdb_segment_dir_syncs_total against scramdb_segment_creations_covered_total shows it. A removal is made durable the same way (scramdb_segment_removals_covered_total), so a power cut never brings a removed file back. A temporary file a crash left in a table's directory is removed at the next start and counted in scramdb_boot_leftover_files_removed_total. With fsync off none of these flushes run (fsync).
  • A lost data file is rebuilt from the log. If a restart finds an active data file missing or short and the log still holds its pages, ScramDB rebuilds the file and serves the table; an empty one is recreated. Only a table whose rows neither the file nor the log can supply is set aside, and it is reported once at startup with its database (troubleshooting).
  • A cluster member that lost a table's files refills it from a peer. The member sets the table aside, asks a healthy member of each shard group that holds the table for its rows at the position the group agrees on, and catches up from the log; its reads of the group wait until it is whole, and no healthy member's rows are touched. A table that several shard groups hold is refilled in each of them, and a group waits while a sibling that holds the same table cannot yet be refilled; a learner's copy is refilled the same way. If no other member holds the group, or the group's own repair record cannot be read, the table stays set aside and scramdb_table_repairs_blocked says so; a record that cannot be read stops its own group and no other.
  • A data file is removed only by its last user. Branches share their parent's frozen data files. A file is deleted only when the last table or branch that lists it lets go. A file left on disk that no table lists is only counted at startup (scramdb_boot_orphan_segment_files), never deleted on a guess. scramdb_segment_bytes_referenced and scramdb_segment_bytes_shared show the bytes the data files hold and the bytes clones share.
  • CHECKPOINT completes. It covers the tables as they were when it began, lets concurrent DDL, bulk loads and clones run beside it or finish first, and never fails on a rename, drop, create, load or clone; a real failure of a disk write still fails it. Under heavy DDL it walks the tables one at a time and lets new DDL run between short stretches (scramdb_checkpoint_table_passes_closed_total, scramdb_checkpoint_table_pass_closed_seconds).
  • Integrity checks. Every stored block carries a checksum. If a disk ever returns corrupted bytes, ScramDB detects it instead of quietly handing back a wrong answer.
  • Room beyond memory. Large queries that would not fit in memory spill to disk automatically and keep running, rather than failing.

Cold tiering to object storage​

Not all data is used equally. The cold tier copies a table's sealed segments (the data files that no longer take writes) to object storage, frees their local disk space at the next start, and reads each one back the first time a query needs it. A copy is the segment file itself, in ScramDB's own format; it is not a Parquet file. The destination can be Amazon S3, Google Cloud Storage, Azure Blob Storage or a local directory.

The cold tier is off by default. Turn it on with cold_enable and a destination in the [storage.cold_config] table. Storage Tiering has the settings, how a copy is read back, and what the tier does not do yet.