Engineering/Release/Product/Company/15 min read

Why I Built ScramDB: One Engine for Transactions, Analytics and AI

Theo M. Bulut·July 23, 2026

It all started 8 years ago, when analytical data systems were still largely built around traditional execution models. At that time, many famous open source projects (later made into proprietary systems) didn't exist. I knew that researching and developing a better system will be harder. Literally, the definition of a moonshot project, to be exact. I knew that it will take me years to build - and it took. I had multiple burnouts while facing hard to solve problems. Today, I finally release the first version of ScramDB.

Many systems have no transactions, no OLAP grade analytical performance. So I decided to implement my own system that will break these barriers. A wallbreaker database that:

AI Agents can program it (yes, literally develop programs that can run in the database)
Transactional with high TPS and SERIALIZABLE (even on distributed scale) with proven tests
Adapts to the resources that is given regardless of the heaviness of the queries
Does analytical complex queries literally under 20ms
Can be invincible to any failure (power outages, crashes). ScramDB is adaptive by nature.
Works in a global span without operational overhead
Have majority of Postgres compatibility

So you don't have to use Redis, Postgres, CDC (Debezium or alike), another GraphQL service for exposing your data, Kafka, RabbitMQ, Graph database for AI agents, Vector database for RAG systems, FTS engines, metrics collector, geospatial systems and many more... One system is enough. Deploy different versions and use it for your own needs.

For all these years, as engineers, we made things that served a purpose for a short time. Redis served for specific use case, Postgres served models. We extracted data put to Kafka to have event exchanged between your operational stack and analytical stack, someone needed to ingest that into a secondary system which has no transactional guarantees so your dashboards was not correct either.

Simplification is better than writing full orchestration of services, taking care of your analytical dashboards (or deploying dashes for yourself) while your database is hammered with transactions and expect to have real time data which ain't arriving is still a problem. ScramDB, by all means, wants to make your AI agents, your developers, or anyone developing IT systems, getting closer to the system that they develop and converge it together. Tailored for your needs and working in a greater scaling. Same stack can be used for various purposes, repurposed if you want to. Because ScramDB has the primitives to do so.

A SRE (Site Reliability Engineer) can program ScramDB to:

Autoscale based on load.
Use a self driving Ejder™ system to keep costs under control.
Carry one pager rotation instead of six. No broker, no connector, no warehouse, no vector store, no search cluster, each with its own failure mode and its own 3am. One system, one set of dashboards, one thing to reason about when something goes wrong.
Get the boring things for free. Prometheus metrics and a health endpoint out of the box, structured JSON logs when you want to ship them somewhere, and a graceful shutdown that drains instead of dropping your connections on the floor.
Sleep through a power cut. Committed is committed, recovery replays on its own, and continuous WAL archiving is on from the first boot, so point in time recovery is there before you ever think to configure it.
Take a backup and prove it. Back up to a local path or straight to S3, GCS or Azure, then restore to an exact moment or an exact log position. Not a snapshot you hope is good, a restore you can rehearse.
Grow the cluster without a migration. Add nodes, rebalance, drain one before you take it out. It stays one logical database the whole way, so scaling out never becomes a project with a name and a spreadsheet.

For SWE (Software Engineer) it is a dream to have:

You are writing your program effectively from any programming language that you can call from your services. Or even better, dynamically inject and run programs to your database. All confined. No more writing PDF generation from tabular data. Because your database can generate PDF with a single query.
Write programs that writes itself (it is the AI age). Control your AI from your local database. Navigate code and let AI do the job for you.
Write behavioral contracts for your AI agents, enforced at data access time. Amazing work. Possibilities are endless.
Keep the driver, the ORM and the client you already use. It speaks the PostgreSQL wire protocol, so psql, JDBC and psycopg connect to it today. No new query language to learn, no proprietary SDK to wrap, no rewrite to justify to your team.
Get a real database underneath, not a fast approximation of one. Transactions with MVCC, isolation up to serializable, two phase commit, triggers, stored procedures and a PL/pgSQL runtime. The parts that make Postgres trustworthy are the parts I kept.
Branch the database like you branch the code. Fork it as of right now or as of last Tuesday, run your feature against it, throw it away. Every developer and every CI run gets a full sized copy of real data, without paying for a full sized copy.
Extend the database without the footgun. Postgres lets you load an untrusted C extension straight into the server and take the whole thing down with one bug. Your function here runs sealed, declares what it may touch, and gets nothing else.

For a CISO (Chief Information Security Officer) giving what they need:

AI agents where production data is anonymized and can be seen anonymized by the AI agents.
AI agents can't break the mainline data. It is confined.
Code that they wrote is also confined. Abusive operations are stopped.
Deterministic AI operations. Know what your agent functions do and how long they take, and what they are accessing. All inside your data store, ScramDB.
One copy to defend instead of two. Every warehouse, every replica and every extract is another place your customer data lives, another access model to get right and another thing to remember when a deletion request arrives. Remove the copies and most of that surface disappears with them.
Access control that is actually enforced, not a convention. Roles and grants, row level security, so a tenant cannot read another tenant's rows, TLS on the wire and host based rules for who may connect from where.
Data that never leaves your infrastructure. Run it on your hardware, in your VPC, in the region your regulator cares about. No third party plane sees your rows on the way to being analyzed.
An answer for "what did that function do?". Sealed by default, no network, no filesystem, no ambient anything, and it declares its capabilities up front. What a function may reach is a property you can read, not a promise you have to trust.

A Data Engineer can:

Delete the pipeline. No CDC connector, no nightly ETL, no Kafka topic whose only job is moving your own rows from your own database into your own warehouse. The analytical query reads the row the transaction just committed, because it is the same row.
Stop apologizing for the staleness window. There isn't one. "How fresh is this dashboard?" stops being a question you have to answer.
Develop OCR functions so literally any data can make into tabular and automatically imported by the database.
Branch production instantly and test the migration or the backfill against real data at real volume, then drop the branch. No sampling, no synthetic fixture, no staging environment that drifted from prod six months ago. And if something did land wrong, point-in-time recovery rewinds you to the moment before it.
Do the transformation in SQL, in the database, in the language you already write. Ship a function in JavaScript, Python, Ruby, Rust, Go, C or C++ and call it from a query. That is a whole class of Spark and Flink jobs that no longer need to exist, along with the cluster they ran on.
Reach for the analytics that normally mean another system. Full-text and BM25 relevance, geospatial and H3, embeddings and similarity, clustering, regression, anomaly scoring, and the streaming sketches you actually want at scale: cardinality, quantiles, top-k. All of it callable from SQL, on the live data.
Anonymize at the source. Mask the PII in the query that feeds the model, the notebook or the agent, so the sensitive copy never gets made and never needs deleting later.
Keep one copy to model, secure, partition and pay for. One schema, one access-control story, one bill, and range, list or hash partitioning when a table gets big.
Point the tools you already have at it. It speaks the PostgreSQL wire protocol, so your BI tool, your orchestrator and your driver connect to it the way they connect to Postgres today.

I know that rudimentary story

It starts with one database for transactions, the system your app writes to. Then the questions get harder, dashboards, reporting, ML features, and that database strains under analytical scans. So you stand up a second system: a columnar warehouse. Now your freshest data lives in the wrong place, so you build a pipeline, change-data-capture or nightly ETL, to shuttle it across. Three systems to run, two copies of the truth to reconcile, and a staleness window you spend the rest of the project apologizing for.

I built ScramDB because that entire architecture is a workaround for a limitation that no longer needs to exist. ScramDB is a programmable, distributed, hyperscale UTAP database. This post is about what UTAP means, why it is the right shape for modern and AI-driven workloads, and how every layer of the engine is built to deliver it.

That's why I call it UTAP (Unified Transactional Analytical Processing). No more 10K USD monthly bill, no more multi thousand USD bill for small data even! It is not HTAP anymore. It is literally the next stage of system architecture!

The two-database tax

The split between transactional (OLTP) and analytical (OLAP) systems was never a law of nature. It was a concession to hardware: row stores were good at point writes, column stores were good at big scans, and nobody had built one engine that did both well. So the industry standardized on running both, and paying the tax in between:

A pipeline you did not want to build. CDC connectors, ETL jobs, schema drift, backfills, and the on-call rotation that comes with them.
Data that is always a little bit wrong. Your analytics are minutes or hours behind your writes. For fraud checks, personalization, and agent decisions, "a little bit behind" is the same as wrong.
Two copies to secure, size, and pay for. Twice the storage, twice the access control, twice the surface area for something to break.

For a decade the accepted answer was "that is just how it works." I did not accept it.

UTAP: one engine, one copy

UTAP stands for Unified Transactional Analytical Processing. The idea is simple and the engineering is not: one engine serves transactions and analytics on one live copy of your data. A row commits, and the next analytical query sees it immediately, because there is nothing to copy and nothing to wait for.

That is the whole point of ScramDB. No CDC pipeline. No read replica. No nightly export to a warehouse. Your transactional writes and your heaviest analytical scans hit the same rows, the instant they land. Scale out, and that becomes one logical database spread across nodes, so the model grows without turning back into two systems.

Everything below is in service of making that single engine genuinely good at both jobs, instead of mediocre at each.

Compiled to native code

Most databases interpret your query, walking a tree of operators row by row. ScramDB compiles it. SQL is lowered to a compact bytecode and then, in the background, to native machine code through an LLVM JIT with two optimization tiers, so a warm query runs as tight, purpose-built machine code instead of an interpreter loop. Compiled code is cached to disk and reused across restarts, so you pay the compilation cost once.

Underneath, execution is morsel-driven: each query is split into small work units spread across workers pinned to CPU cores. Adding cores adds throughput close to linearly, with no scheduler overhead fighting you.

Every core, and the GPU

For the heaviest batch work, ScramDB can offload to the GPU. The engine ships GPU kernels for NVIDIA, AMD, and Apple Metal, and decides per query whether the GPU is worth it based on how much data is involved. If there is no eligible GPU, or an operation is not a fit, it falls back to the CPU path automatically. GPU acceleration is built into the engine, not a separate service you have to operate.

The whole PostgreSQL contract

A lot of "Postgres-compatible" systems keep the wire protocol and quietly drop the parts that make Postgres trustworthy. ScramDB keeps the contract:

Real transactions with MVCC and isolation levels from Read Committed up to Serializable.
Two-phase commit for coordinated writes.
Access control that means it: role-based access control and row-level security.
Triggers, stored procedures, and functions, including a PL/pgSQL runtime.

Because it speaks the PostgreSQL wire protocol, you connect with psql, JDBC, psycopg, or any driver you already use. No new query language, no proprietary SDK.

Storage built for both jobs

The storage layer, Tundra, is a custom columnar engine designed to serve fast writes and fast scans from the same data. It has crash recovery through a write-ahead log, block-level integrity checks, and automatic tiering that keeps hot data close and pushes cold data out to compressed Parquet in cloud object storage. You get warehouse-style scan performance without shipping your data to a warehouse.

Program your database

Databases should be extensible without the footgun. Postgres lets you load an untrusted C extension straight into the server, and one bug takes down the database. ScramDB takes a different path: sealed, deterministic user-defined functions that run in a capability-governed sandbox with no ambient access to the network or filesystem.

You write a function in the language you already use, and it is callable straight from SQL. The runtime is polyglot: JavaScript, TypeScript, Rust, Go, C, C++, Python and Ruby, all compiled to sealed WebAssembly and run in the same governed sandbox. And ScramDB ships with a library of analytics packages built in, so much of what you would normally run a separate service for is already a SQL function:

Vector similarity (cosine, dot product) for embeddings
Full-text search with tokenization and BM25 relevance ranking
Geospatial distance and spatial indexing
Machine-learning primitives: clustering, regression, anomaly scoring
Streaming sketches: cardinality, quantiles, and heavy hitters

Branch your data like code

Modern development, and increasingly AI agents, want to spin up a database, try something, and throw it away. ScramDB supports point-in-time recovery and branching: fork a database as of a moment in time, work against it in isolation, and drop it when you are done. It is the git-style workflow, applied to your data. And yeah, I am not selling this as whole company's promise product. It is literally freely baked into the product. Check out the docs.

Distributed when you need it, not before

You should not have to run a cluster to get started, and you should not have to migrate to a new system when you outgrow one node. ScramDB runs happily on a single machine and scales out to a distributed cluster with consensus-based replication and distributed transactions that stay fully serializable (SSI) across nodes, presenting itself as one consistent logical database the whole way up. No stale replicas, no bounded-staleness caveats: the cluster sees one copy of the truth.

Measured, not hand-waved

I hold myself to one rule above the rest: a performance claim ships with a number, or it does not ship. Every optimization is validated against the standard suites. Today, all TPC-DS, TPC-H and TPC-C pass with flying colors, even better than many commercial databases out there with using less resources (even with a single node), correctness-checked, not a hand-picked subset. When I make a claim, you can reproduce it.

Try it

ScramDB is a programmable, distributed, hyperscale UTAP database: one engine for transactions, analytics, and AI, on one copy of your data. If you are tired of building architectures that are not future proof, I built this for you. Bootstrapping your start-up, or scaling your company must not be hard. I came as a titan to solve this for you!

Get started with AI! Yes, stop reading anything. Just go to our webpage click to Start Developing Agentic AI App. and paste it to your favorite AI coding agent to get started. OR go the old way: Get started in a few minutes, or read the docs.