Skip to main content

Storage Engine

Tundra is ScramDB's storage engine. It keeps your data in a single columnar copy that is fast to write to and fast to scan, so transactions and analytics run on the same live data without stepping on each other.

One copy for writes and scans​

Traditional setups keep two shapes of your data: a row-oriented copy tuned for transactions and inserts, and a separate column-oriented copy tuned for analytics, kept in sync by a pipeline. Tundra keeps one copy and serves both.

  • Writes are fast. New and changed rows land in a write-optimized area and become visible immediately, so transactions stay quick.
  • Scans are fast. In the background, that data is folded into a compressed columnar layout, the format analytical queries love, so large scans and aggregations fly.
  • Readers and writers never block each other. ScramDB uses multi-version concurrency control (MVCC): a reader sees a stable snapshot of the data while writers keep working. It supports the standard isolation levels you expect, up to serializable.

The practical upshot: you insert an order and analyze it in the same breath, with no copy to keep in sync and no lag between the two.

Skipping the data you do not need​

Tundra keeps lightweight summaries (such as the minimum and maximum values, and other per-block filters) for each block of data. When a query filters on a column, ScramDB uses these summaries to skip whole blocks that cannot match, so a query that touches a small slice of a large table only reads the part it needs.

  • The summaries are cached on disk. Each segment file in your data directory also gets a small .hdr sidecar file beside it, holding a compact copy of the block summaries so ScramDB does not need to re-read every block to rebuild them. A sidecar is never read at startup, is rebuilt automatically if it is missing or out of date, and is removed along with its segment, so you can expect to see a modest number of extra files, and a modest amount of extra disk space, alongside your data.
  • Some aggregates skip rows entirely. ScramDB also keeps an exact minimum, maximum, and sum for each column alongside these summaries, so an aggregate like a minimum, maximum, or total over a whole column can often be answered directly from stored statistics instead of scanning the table. This still holds after VACUUM rewrites and merges the underlying data.

Durable and self-checking​

Your committed data is safe.

  • Crash recovery. Every change is first recorded in a write-ahead log before it is applied. If the server crashes or loses power, ScramDB replays the log on restart and comes back consistent, with no committed data lost. Recovery also removes rows that reached disk but were never acknowledged to any client, so a crash can never resurrect a write nobody was told had succeeded.
  • A commit becomes visible only once it is durable. Other sessions see a transaction's changes only after its commit record is safely in the write-ahead log, on every node and for replicated writes too. With synchronous_commit off, a commit is visible as soon as its record is written, before it is durable.
  • A failing disk stops the writer, it does not get retried. If a log write or its fsync fails (a full disk included), that is the operating system saying the data may not be on the disk. Retrying proves nothing, because the pages it would retry may already have been dropped. ScramDB treats the first such failure as permanent: the statement that met it fails, naming the original error, and nothing it wrote ever becomes visible. From then on every write fails loudly the same way, while reads keep answering from the last state that was both durable and visible when the log failed. Recovery is a restart, which replays the durable prefix of the log. A checkpoint is refused in this state too, so a wedged server cannot mark unsafe data as safely stored on its way down. Whether the failed statement's own log record reached the disk is only known after the restart, so treat its outcome like that of a lost connection: check before you retry it.
  • Integrity checks. Every stored block carries a checksum. If a disk ever returns corrupted bytes, ScramDB detects it instead of quietly handing back a wrong answer.
  • Room beyond memory. Large queries that would not fit in memory spill to disk automatically and keep running, rather than failing.

Automatic cold tiering to the cloud​

Not all data is used equally. Colder, infrequently accessed data is tiered out automatically to compressed Parquet files in cloud object storage, with support for Amazon S3, Google Cloud Storage, and Azure. Your hot working set stays local and fast, while older data is kept economically in the cloud in an open, portable format. You do not manage this by hand.