Distributed Cluster

A ScramDB cluster runs the same UTAP engine across several machines and presents them as one database. It is not a set of loosely synchronized copies. The cluster holds one consistent, serializable copy of your data, and every node can serve your application.

You move to a cluster for two reasons: to stay available when a machine fails, and to scale beyond what one machine can hold. This page walks through both, explains what the consistency guarantees mean in practice, and shows how to deploy a cluster with Docker Compose or Kubernetes.

Licensing

Multi-node clustering is an Enterprise capability. When a node's config has a [cluster] section, it checks its license before it binds any port:

Community (the default, unlicensed edition): the node refuses to start. It logs the reason and exits.
Enterprise or Trial: clustering is unlocked.

Supply the license with the SCRAMDB_LICENSE_KEY environment variable, or license_key under [general] in the config file. The environment variable wins if both are set. Every node in the cluster needs a valid key; there's no "leader has the license" shortcut.

Your license also sets a ceiling on the number of nodes you can run. Treat that ceiling as the number your license entitles you to; contact your account team before scaling past it.

Single-node ScramDB has no license gate at all. Everything on this page is what you get once you add [cluster] and a valid Enterprise/Trial key.

See Cluster licensing for exactly how a node resolves its edition, what each degraded state logs, and how to verify a license is active on a running node.

What stays identical to a single node

Clustering is opt-in and additive. Turning it on does not change how you use the database:

Same SQL and same drivers. The PostgreSQL wire protocol, the SQL you write, and your existing clients all work unchanged. Connect to any node with psql, JDBC, psycopg, or any PostgreSQL driver.
Same engine. Each node runs the identical columnar storage, JIT-compiled execution, and MVCC transactions as a single node. A cluster is that engine replicated and coordinated, not a different product.
Same isolation levels. Read Committed, Repeatable Read, and Serializable behave the same way they do on one node. The difference is that the guarantee now holds across every node, not just within one.

Your application does not need to know how many nodes there are or where a given row lives. You send SQL to any node and get one consistent answer.

Consistency across the cluster

This is the property that makes a ScramDB cluster different from a read-replica setup: the whole cluster is one serializable database, not a primary with stale followers.

Serializable across every node

ScramDB provides full serializable consistency (Serializable Snapshot Isolation, SSI) across the cluster. A transaction that reads and writes data living on several different nodes is still serializable: the result is as if transactions ran one at a time, in some order, over the entire cluster. When two transactions conflict, one is aborted with a serialization failure (PostgreSQL SQLSTATE 40001), exactly as on a single node, and your application retries it. There is no weaker "eventually consistent" mode hiding behind the cluster boundary.

Distributed transactions commit atomically

A transaction that spans multiple nodes commits all-or-nothing using distributed two-phase commit (2PC), coordinated over Raft consensus. Before ScramDB acknowledges a commit, every write is durably replicated to a majority of the nodes that hold that data. A transaction confined to a single node takes a fast path and skips the distributed coordination entirely, so you pay the cross-node cost only when a transaction actually crosses nodes.

One consistent copy, not stale replicas

Each piece of data is replicated across several nodes (the default replication factor is 3) and kept in agreement by Raft consensus. Replication here is for durability and availability, not for serving stale reads: a normal read sees committed data, and the copies do not drift apart. Losing a node does not lose data, because a majority of the replicas still hold the committed history.

For analytics that can tolerate a small, known amount of lag, ScramDB also offers an explicit read path against a columnar replica, enabled per session with SET learner_read. This path is bounded and honest: if the replica has fallen behind its freshness bound, the read is refused rather than served stale, and SHOW FRESHNESS reports the current lag per data group. It is an opt-in for offloading heavy analytical scans, never the default, and never a silently stale answer.

Row-level security holds across nodes

Row-level security (RLS) policies are enforced identically whether a query runs on one node or fans out across many. When a distributed query gathers rows from several nodes, each node applies the same policies for the role that issued the query, so a user sees exactly the rows their policies allow and no more. Distributing a query never widens what a user can see.

Scenario: a highly available cluster

The most common cluster is three nodes for high availability.

Data is replicated across the three nodes (replication factor 3), and each shard commits through Raft, which requires agreement from a majority.
Because a majority of three is two, the cluster keeps serving reads and writes, with full serializable consistency and no data loss, while one node is down. You can lose any single node and stay online.
A crashed node is never removed from the cluster automatically, because automatically dropping a node could put the remaining majority at risk. For planned maintenance or scale-down, drain a node explicitly with ALTER CLUSTER DRAIN (see Draining a node) before you take it offline.

A three-node cluster is the smallest configuration that tolerates a full node failure. How many failures a cluster tolerates is governed by the replication factor: with the default of 3, each shard survives the loss of one of its replicas. Raising the replication factor keeps more copies and tolerates more simultaneous failures, at the cost of more write traffic per commit.

Scenario: scaling out for capacity

When your data or your query load outgrows one machine, add nodes.

Data spreads automatically. Tables are hash-partitioned into buckets, and those buckets are distributed across the nodes. When you add or remove a node, ScramDB rebalances buckets to even out storage and load. Scaling is elastic: the cluster grows and shrinks while it keeps serving.
Queries run in parallel across nodes. An analytical query executes distributed: nodes scan their local data in parallel, and ScramDB streams and combines the partial results (distributed scans, joins, and aggregations) into one answer. Adding nodes adds parallel capacity for large scans and joins.
Writes scale with the data. Because different tables and buckets live on different nodes, write throughput spreads across the cluster instead of funneling through a single machine.

The same query returns the same result whether it runs on one node or many. Distribution changes where work happens, never the answer.

Deploying a cluster

Every node runs the same image: one config template, rendered per node, the same binary everywhere. Cluster mode is selected at startup by three settings that identify the node and its peers, plus a license key.

Docker Compose (three nodes)

The image starts in cluster mode when you set NODE_NAME, ADVERTISE_ADDR, and SEEDS. All three must be set together: setting only one or two of them is a hard failure, not a fallback to single-node mode. 7190 is the cluster transport port nodes use to talk to each other; it's separate from the 5432 pgwire port your client connects to.

services:
  node-a:
    image: scramdb/scramdb:latest
    hostname: node-a
    environment:
      NODE_NAME: node-a
      ADVERTISE_ADDR: "node-a:7190"
      SEEDS: '["node-b:7190", "node-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5432:5432"   # PostgreSQL clients
      - "9090:9090"   # Prometheus metrics
    volumes:
      - node-a-data:/var/lib/scramdb

  node-b:
    image: scramdb/scramdb:latest
    hostname: node-b
    environment:
      NODE_NAME: node-b
      ADVERTISE_ADDR: "node-b:7190"
      SEEDS: '["node-a:7190", "node-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5433:5432"
      - "9091:9090"
    volumes:
      - node-b-data:/var/lib/scramdb

  node-c:
    image: scramdb/scramdb:latest
    hostname: node-c
    environment:
      NODE_NAME: node-c
      ADVERTISE_ADDR: "node-c:7190"
      SEEDS: '["node-a:7190", "node-b:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5434:5432"
      - "9092:9090"
    volumes:
      - node-c-data:/var/lib/scramdb

networks:
  scramdb-cluster:
    driver: bridge

volumes:
  node-a-data:
  node-b-data:
  node-c-data:

Set the license key once in your shell (or an .env file next to the compose file) before bringing the cluster up:

export SCRAMDB_LICENSE_KEY="<your-enterprise-or-trial-key>"
docker compose up -d
docker compose logs -f

Without a valid key, each node logs the license error and exits; the compose file above will not form a cluster on its own.

The nodes are symmetric peers. Leadership is elected at runtime, so there is no required start order, and you can connect to any node:

psql "host=127.0.0.1 port=5432 user=scramdb dbname=scramdb" -c "select 1;"

The three env vars only cover node identity and seeds. To tune anything else cluster-specific, for example [cluster.swim], [cluster.raft], or [storage.wal.archive], mount a complete config file and pass -c explicitly instead of (or in addition to) the env vars; the entrypoint's three-variable path renders a template with just those fields.

Kubernetes

For Kubernetes, run the cluster as a StatefulSet behind a headless Service. Each node gets a stable identity and its own persistent volume, and nodes discover their peers by DNS name through the headless Service, so you do not hardcode addresses. Set SCRAMDB_LICENSE_KEY on every pod, from a Secret.

Draining is a manual step

Scaling a StatefulSet down does not automatically drain the node being removed. Run ALTER CLUSTER DRAIN yourself before you reduce the replica count; see Draining a node below.

Seedless formation

The StatefulSet's ConfigMap carries no seed list. A hardcoded list of peer addresses is only correct at the exact replica count it was written for; scale up or down and it goes stale. Instead, every pod sets bootstrap_expect, the number of other voter-role nodes it waits to see connected before founding the cluster, and finds those peers itself through the headless Service's DNS plus SWIM:

[cluster]
node_name = "__NODE_NAME__"
cluster_listen = "0.0.0.0:7190"
advertise_addr = "__NODE_NAME__.scramdb-headless:7190"
bootstrap_expect = 2
discovery = ["dns", "swim"]
dns_name = "scramdb-headless"
dns_refresh = "5s"
replication_factor = 3
group0_bootstrap_timeout = "60s"

__NODE_NAME__ is the one templated token: the container's own startup command substitutes it with the pod's metadata.name, read through the downward API, before handing the rendered file to scramdb -c. Every pod runs this identical template and derives the identical genesis set, so there's nothing per-node to hand-edit and nothing that drifts out of sync with the replica count, except one number: bootstrap_expect has to equal replicas - 1, and the two live one field apart in the same overlay. Three replicas is bootstrap_expect = 2; five is bootstrap_expect = 4. Change both together. Get it wrong and the cluster doesn't misform silently, it waits out group0_bootstrap_timeout and then refuses, loudly, naming what it was waiting for.

The license Secret

Cluster mode needs the same Enterprise or Trial key on every pod that Docker Compose needs in your shell, just delivered through a Secret instead of an environment variable. Create it once, before the first kubectl apply:

kubectl create secret generic scramdb-license \
  --from-literal=license-key="<your Enterprise token>"

and reference it from the container spec instead of a literal value:

env:
  - name: SCRAMDB_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: scramdb-license
        key: license-key

A missing Secret holds the pod at CreateContainerConfigError, with a kubectl describe pod event naming exactly which Secret it wanted, before the container so much as starts. That's the same loud, name-the-variable refusal a Community-licensed process logs on its own stderr, just caught one layer up by the kubelet instead.

Size each pod's CPU request and limit like any single node: a Guaranteed QoS pod with the kubelet's static CPU manager policy pins exclusive cores per replica, and the default policy quota-bounds them instead. See Parallelism for how ScramDB resolves either into a worker count, and for SCRAMDB_MAX_CORES when you bin-pack more than one node onto the same host.

Geo placement across regions and zones

Every node, in Docker Compose or Kubernetes, can carry an optional region and an optional zone label. A table created through a labelled node homes its voters in that region and commits at region-local quorum latency; ALTER TABLE ... SET (home_region = '<region>') re-homes an existing table live. Nothing about the SQL you write changes either way.

Leaving a label unset is always safe, and so is setting it to an empty string: an empty or whitespace-only value is trimmed and dropped, so the node comes up unlabelled, never a node whose region is literally named "". A compose file or manifest that never mentions REGION/ZONE at all behaves exactly as it always has.

Docker Compose

docker/cluster-config.template.toml renders REGION/ZONE straight into the node's [cluster] section, next to the fields you've already seen:

[cluster]
node_name = "${NODE_NAME}"
advertise_addr = "${ADVERTISE_ADDR}"
seeds = ${SEEDS}
region = "${REGION}"
zone = "${ZONE}"

Set them per service, the same way you set NODE_NAME. A genuinely multi-region compose file just gives different services different values. replication_factor is fixed at 3 in the demo template, so six nodes, three per region, is the shape that keeps every home-region table's replicas fully inside its own region:

services:
  eu-a:
    image: scramdb/scramdb:latest
    hostname: eu-a
    environment:
      NODE_NAME: eu-a
      ADVERTISE_ADDR: "eu-a:7190"
      SEEDS: '["eu-b:7190", "eu-c:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5432:5432"   # PostgreSQL clients
      - "9090:9090"   # Prometheus metrics
    volumes:
      - eu-a-data:/var/lib/scramdb

  eu-b:
    image: scramdb/scramdb:latest
    hostname: eu-b
    environment:
      NODE_NAME: eu-b
      ADVERTISE_ADDR: "eu-b:7190"
      SEEDS: '["eu-a:7190", "eu-c:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5433:5432"
      - "9091:9090"
    volumes:
      - eu-b-data:/var/lib/scramdb

  eu-c:
    image: scramdb/scramdb:latest
    hostname: eu-c
    environment:
      NODE_NAME: eu-c
      ADVERTISE_ADDR: "eu-c:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5434:5432"
      - "9092:9090"
    volumes:
      - eu-c-data:/var/lib/scramdb

  us-a:
    image: scramdb/scramdb:latest
    hostname: us-a
    environment:
      NODE_NAME: us-a
      ADVERTISE_ADDR: "us-a:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5435:5432"
      - "9093:9090"
    volumes:
      - us-a-data:/var/lib/scramdb

  us-b:
    image: scramdb/scramdb:latest
    hostname: us-b
    environment:
      NODE_NAME: us-b
      ADVERTISE_ADDR: "us-b:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-a:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5436:5432"
      - "9094:9090"
    volumes:
      - us-b-data:/var/lib/scramdb

  us-c:
    image: scramdb/scramdb:latest
    hostname: us-c
    environment:
      NODE_NAME: us-c
      ADVERTISE_ADDR: "us-c:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-a:7190", "us-b:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5437:5432"
      - "9095:9090"
    volumes:
      - us-c-data:/var/lib/scramdb

networks:
  scramdb-cluster:
    driver: bridge

volumes:
  eu-a-data:
  eu-b-data:
  eu-c-data:
  us-a-data:
  us-b-data:
  us-c-data:

Bring it up the same way as the three-node cluster: one exported license key, one docker compose up -d. Each node's own boot log confirms what it rendered:

entrypoint: cluster mode - node_name=eu-a advertise_addr=eu-a:7190 seeds=["eu-b:7190",...] region=eu-central zone=eu-central-1a

On a single bridge network, REGION/ZONE change placement, not physical latency: every node is one Docker hop from every other regardless of label. The labels are real, and a table homed through eu-a genuinely keeps its voters on eu-a/eu-b/eu-c; what a single-host demo can't show you is the cross-ocean round trip a real two-region deployment would.

Kubernetes

k8s/overlays/regions-2x2 is the same idea, shaped for a StatefulSet: two pools sharing one headless Service, one flat four-node cluster. The base StatefulSet becomes the eu pool (patched to replicas: 2), and a second StatefulSet, scramdb-us, is the us pool (replicas: 2). Both share the same headless Service and ConfigMap, so pods from both pools resolve each other by DNS as if they were one StatefulSet. Each pool patches just two environment entries on top of the shared container spec:

env:
  - name: REGION
    value: "eu"
  - name: ZONE
    value: "eu-1a"

kubectl apply -k k8s/overlays/regions-2x2

The ConfigMap this overlay ships sets replication_factor = 2, not the base's 3, and bootstrap_expect = 3 (four pods total, each waiting for the other three). That replication factor is a deliberate match to two nodes per region: with RF=2, each region can fully supply a home-region table's replicas from its own pool, so a re-home in either direction is a clean move, never a degraded one, the same property the six-node Compose example above gets from three nodes per region instead.

On one kind cluster both pools land on the same nodes, so this still only demonstrates placement, the same caveat as Compose above. On a real multi-region cluster, pin each pool to its region's actual nodes with a nodeSelector or topologySpreadConstraints; the StatefulSet/ConfigMap shape above doesn't change.

Configuration essentials

Deployments that use a configuration file instead of environment variables turn on cluster mode with a [cluster] section:

[general]
license_key = "<your-enterprise-or-trial-key>"

[cluster]
node_name = "node-a"
advertise_addr = "node-a:7190"
seeds = ["node-b:7190", "node-c:7190"]
replication_factor = 3

The presence of the [cluster] section is what activates multi-node mode, and it's what triggers the license check above. Remove it and the same binary runs as a single node with no license requirement. replication_factor sets how many copies of each piece of data the cluster keeps (default 3); higher means more machine failures tolerated at the cost of more replicated write traffic. [cluster] has several sub-tables ([cluster.swim], [cluster.raft], [cluster.rebalance], [cluster.txn]) for tuning gossip, consensus, rebalancing, and distributed-transaction behavior beyond the defaults; they aren't covered in depth on this page. Two plain top-level fields, region and zone, label a node for geo placement; see Geo placement across regions and zones above. bootstrap_expect replaces a hardcoded seeds list for DNS-discovered formation; see Seedless formation under Kubernetes above.

Operating a cluster

Connect to any node. All nodes are equal peers and serve the same consistent database. Put a load balancer in front of the client port if you want a single endpoint.
Monitor freshness and health. SHOW FRESHNESS reports replication lag per data group, and every node exposes Prometheus metrics on port 9090 plus a plain /health endpoint. See Observability for the full scramdb_cluster_* reference and which counters are worth alerting on.

Draining a node

Use ALTER CLUSTER DRAIN '<node-name>' to move a node's data and responsibilities to the rest of the cluster before you shut it down, whether for maintenance or to scale in:

ALTER CLUSTER DRAIN 'node-b';

Issue the command against the node that currently leads the shard groups you want moved. Draining fully converges the groups that node leads; if leadership for other groups is spread across different nodes, issue DRAIN against each leader in turn to move everything. This is a manual operation today: nothing drains a node automatically, whether from a crash, a Kubernetes scale-down, or any other trigger.

Summary

A ScramDB cluster is one serializable, highly available database built from the same engine as a single node. Transactions are serializable across every node, distributed transactions commit atomically through Raft-backed two-phase commit, replication keeps one consistent copy rather than stale followers, and row-level security holds no matter how a query is distributed. You get availability and capacity without giving up the guarantees, or the SQL, you rely on, on the Enterprise license tier.

Licensing​

What stays identical to a single node​

Consistency across the cluster​

Serializable across every node​

Distributed transactions commit atomically​

One consistent copy, not stale replicas​

Row-level security holds across nodes​

Scenario: a highly available cluster​

Scenario: scaling out for capacity​

Deploying a cluster​

Docker Compose (three nodes)​

Kubernetes​

Seedless formation​

The license Secret​

Geo placement across regions and zones​

Docker Compose​

Kubernetes​

Configuration essentials​

Operating a cluster​

Draining a node​

Summary​

Licensing

What stays identical to a single node

Consistency across the cluster

Serializable across every node

Distributed transactions commit atomically

One consistent copy, not stale replicas

Row-level security holds across nodes

Scenario: a highly available cluster

Scenario: scaling out for capacity

Deploying a cluster

Docker Compose (three nodes)

Kubernetes

Seedless formation

The license Secret

Geo placement across regions and zones

Docker Compose

Kubernetes

Configuration essentials

Operating a cluster

Draining a node

Summary