ScramDB vs Apache Spark
Spark SQL is a processing engine over other people’s storage: a driver, executors and a cluster manager, with no persistent storage of its own and no cross-table ACID guarantees in core Spark SQL. It is built for large batch pipelines, and it is the thing your data leaves the database to reach. ScramDB answered the same analytical workload 10 times faster, from one process, on the data it already holds.
Side by side
| ScramDB | Apache Spark | |
|---|---|---|
| What it is | A database | A distributed processing engine over external storage |
| Owns the data | Yes: its own storage, with real transactions behind it | No: files and tables in other systems |
| Transactions | Full ACID up to serializable isolation | Not part of core Spark SQL; it depends on the table format underneath |
| What you deploy | One binary, one process | A driver, executors and a cluster manager such as YARN or Kubernetes |
| Latency | Interactive: a query is a query | Job oriented, with scheduling and startup in the path |
| Constraints and triggers | Foreign keys, CHECK constraints, triggers, procedures | Not part of the engine |
| Wire protocol | PostgreSQL wire protocol, every standard driver and BI tool | JDBC and ODBC through the thrift server, or its own APIs |
| AI and agents | A semantic layer inside the engine that agents connect to directly, bound by the caller’s own permissions | MLlib and the wider ecosystem, outside the SQL surface |
The pipeline exists because the database was slow
Most Spark jobs exist to do something the database could not: scan a lot of rows and aggregate them. When the database answers that in seconds on the live data, a large part of the pipeline stops earning its keep, along with the cluster it runs on and the copy it reads.
When Spark is the right answer
Genuine data engineering at petabyte scale, machine learning pipelines, and transformation work over many heterogeneous sources. Spark is the standard for that and it is Apache 2.0.
Other head to heads
Same wire protocol, same drivers, same SQL. A columnar engine and native-code execution underneath.
ClickHouse cannot hold your transactions, so it lives next to a real database with a pipeline between them. ScramDB is both.
No columnar storage, no vectorized execution, no parallel query in the community server. Every analytical scan is a row walk.
Run it yourself in one line
One static binary, no dependencies. It picks its own config, listens on the postgres port 5432 and runs in the background.
$curl -fsSL https://scramdb.com/install | bash