Skip to main content

Storage Engine

Tundra is ScramDB's storage engine. It keeps your data in a single columnar copy that is fast to write to and fast to scan, so transactions and analytics run on the same live data without stepping on each other.

One copy for writes and scans​

Traditional setups keep two shapes of your data: a row-oriented copy tuned for transactions and inserts, and a separate column-oriented copy tuned for analytics, kept in sync by a pipeline. Tundra keeps one copy and serves both.

  • Writes are fast. New and changed rows land in a write-optimized area and become visible immediately, so transactions stay quick.
  • Scans are fast. In the background, that data is folded into a compressed columnar layout, the format analytical queries love, so large scans and aggregations fly.
  • Readers and writers never block each other. ScramDB uses multi-version concurrency control (MVCC): a reader sees a stable snapshot of the data while writers keep working. It supports the standard isolation levels you expect, up to serializable.

The practical upshot: you insert an order and analyze it in the same breath, with no copy to keep in sync and no lag between the two.

Skipping the data you do not need​

Tundra keeps lightweight summaries (such as the minimum and maximum values, and other per-block filters) for each block of data. When a query filters on a column, ScramDB uses these summaries to skip whole blocks that cannot match, so a query that touches a small slice of a large table only reads the part it needs.

Durable and self-checking​

Your committed data is safe.

  • Crash recovery. Every change is first recorded in a write-ahead log before it is applied. If the server crashes or loses power, ScramDB replays the log on restart and comes back consistent, with no committed data lost. Recovery also removes rows that reached disk but were never acknowledged to any client, so a crash can never resurrect a write nobody was told had succeeded.
  • A failing disk stops the writer, it does not get retried. If a log write or its fsync fails, that is the operating system saying the data may not be on the disk. Retrying proves nothing, because the pages it would retry may already have been dropped. ScramDB treats the first such failure as permanent: the writer stops accepting new durable work and every later attempt fails loudly, naming the original error. Recovery is a restart, which replays the durable prefix of the log. A checkpoint is refused in this state too, so a wedged server cannot mark unsafe data as safely stored on its way down.
  • Integrity checks. Every stored block carries a checksum. If a disk ever returns corrupted bytes, ScramDB detects it instead of quietly handing back a wrong answer.
  • Room beyond memory. Large queries that would not fit in memory spill to disk automatically and keep running, rather than failing.

Automatic cold tiering to the cloud​

Not all data is used equally. Colder, infrequently accessed data is tiered out automatically to compressed Parquet files in cloud object storage, with support for Amazon S3, Google Cloud Storage, and Azure. Your hot working set stays local and fast, while older data is kept economically in the cloud in an open, portable format. You do not manage this by hand.