Skip to main content

Distributed Cluster

A ScramDB cluster runs the same UTAP engine across several machines and presents them as one database. It is not a set of loosely synchronized copies. The cluster holds one consistent, serializable copy of your data, and every node can serve your application.

You move to a cluster for two reasons: to stay available when a machine fails, and to scale beyond what one machine can hold. This page walks through both, explains what the consistency guarantees mean in practice, and shows how to deploy a cluster with Docker Compose or Kubernetes.

Licensing​

Multi-node clustering is an Enterprise capability. When a node's config has a [cluster] section, it checks its license before it binds any port:

  • Community (the default, unlicensed edition): the node refuses to start. It logs the reason and exits.
  • Enterprise or Trial: clustering is unlocked.

Supply the license with the SCRAMDB_LICENSE_KEY environment variable, or license_key under [general] in the config file. The environment variable wins if both are set. Every node in the cluster needs a valid key; there's no "leader has the license" shortcut.

Your license also sets a ceiling on the number of nodes you can run. Treat that ceiling as the number your license entitles you to; contact your account team before scaling past it.

Single-node ScramDB has no license gate at all. Everything on this page is what you get once you add [cluster] and a valid Enterprise/Trial key.

See Cluster licensing for exactly how a node resolves its edition, what each degraded state logs, and how to verify a license is active on a running node.

What stays identical to a single node​

Clustering is opt-in and additive. Turning it on does not change how you use the database:

  • Same SQL and same drivers. The PostgreSQL wire protocol, the SQL you write, and your existing clients all work unchanged. Connect to any node with psql, JDBC, psycopg, or any PostgreSQL driver.
  • Same engine. Each node runs the identical columnar storage, JIT-compiled execution, and MVCC transactions as a single node. A cluster is that engine replicated and coordinated, not a different product.
  • Same isolation levels. Read Committed, Repeatable Read, and Serializable behave the same way they do on one node. The difference is that the guarantee now holds across every node, not just within one.

Your application does not need to know how many nodes there are or where a given row lives. You send SQL to any node and get one consistent answer.

Consistency across the cluster​

This is the property that makes a ScramDB cluster different from a read-replica setup: the whole cluster is one serializable database, not a primary with stale followers.

Serializable across every node​

ScramDB provides full serializable consistency (Serializable Snapshot Isolation, SSI) across the cluster. A transaction that reads and writes data living on several different nodes is still serializable: the result is as if transactions ran one at a time, in some order, over the entire cluster. When two transactions conflict, one is aborted with a serialization failure (PostgreSQL SQLSTATE 40001), exactly as on a single node, and your application retries it. There is no weaker "eventually consistent" mode hiding behind the cluster boundary.

Distributed transactions commit atomically​

A transaction that spans multiple nodes commits all-or-nothing in one round of the cluster's commit protocol: every shard group it touched logs a vote carrying its writes over Raft consensus, and the transaction is committed once every one of them holds a durable yes vote. Before ScramDB acknowledges a commit, every write is durably replicated to a majority of the nodes that hold that data. A transaction confined to a single shard group commits with one entry in that group's log and skips the distributed coordination entirely, so you pay the cross-node cost only when a transaction actually crosses nodes.

One consistent copy, not stale replicas​

Each piece of data is replicated across several nodes (the default replication factor is 3) and kept in agreement by Raft consensus. Replication here is for durability and availability, not for serving stale reads: a normal read sees committed data, and the copies do not drift apart. Losing a node does not lose data, because a majority of the replicas still hold the committed history.

For analytics that can tolerate a small, known amount of lag, ScramDB also offers an explicit read path against a columnar replica, enabled per session with SET learner_read. This path is bounded and honest: if the replica has fallen behind its freshness bound, the read is refused rather than served stale, and SHOW FRESHNESS reports the current lag per data group. It is an opt-in for offloading heavy analytical scans, never the default, and never a silently stale answer.

Row-level security holds across nodes​

Row-level security (RLS) policies are enforced identically whether a query runs on one node or fans out across many. When a distributed query gathers rows from several nodes, each node applies the same policies for the role that issued the query, so a user sees exactly the rows their policies allow and no more. Distributing a query never widens what a user can see.

Scenario: a highly available cluster​

The most common cluster is three nodes for high availability.

  • Data is replicated across the three nodes (replication factor 3), and each shard commits through Raft, which requires agreement from a majority.
  • Because a majority of three is two, the cluster keeps serving reads and writes, with full serializable consistency and no data loss, while one node is down. You can lose any single node and stay online.
  • A crashed node is never removed from the cluster automatically, because automatically dropping a node could put the remaining majority at risk. A node that stops gracefully is a different case: it drains its own membership on the way out, so a planned stop or a Kubernetes scale-down hands the seat back on its own. ALTER CLUSTER DRAIN is the on-demand form, for a node that is already gone or one whose buckets you want moved before you touch it (see Draining a node).

A three-node cluster is the smallest configuration that tolerates a full node failure. How many failures a cluster tolerates is governed by the replication factor: with the default of 3, each shard survives the loss of one of its replicas. Raising the replication factor keeps more copies and tolerates more simultaneous failures, at the cost of more write traffic per commit.

Scenario: scaling out for capacity​

When your data or your query load outgrows one machine, add nodes.

  • Data spreads automatically. Tables are hash-partitioned into buckets, and those buckets are distributed across the nodes. When you add or remove a node, ScramDB rebalances buckets to even out storage and load. Scaling is elastic: the cluster grows and shrinks while it keeps serving.
  • Queries run in parallel across nodes. An analytical query executes distributed: nodes scan their local data in parallel, and ScramDB streams and combines the partial results (distributed scans, joins, and aggregations) into one answer. Adding nodes adds parallel capacity for large scans and joins.
  • Writes scale with the data. Because different tables and buckets live on different nodes, write throughput spreads across the cluster instead of funneling through a single machine.

The same query returns the same result whether it runs on one node or many. Distribution changes where work happens, never the answer.

Deploying a cluster​

Every node runs the same image: one config template, rendered per node, the same binary everywhere. Cluster mode is selected at startup by three settings that identify the node and its peers, plus a license key.

Docker Compose (three nodes)​

The image starts in cluster mode when you set NODE_NAME, ADVERTISE_ADDR, and SEEDS. All three must be set together: setting only one or two of them is a hard failure, not a fallback to single-node mode. Nodes talk to each other on three ports: 7190 (control traffic, the port ADVERTISE_ADDR and SEEDS name), 7191 (interactive traffic) and 7192 (bulk traffic), all separate from the 5432 pgwire port your client connects to. They stay on the compose network, where the nodes reach each other, and are not published to the host. Ports lists every port a node opens and the firewall rules between machines.

services:
  node-a:
    image: scramdb/scramdb:latest
    hostname: node-a
    environment:
      NODE_NAME: node-a
      ADVERTISE_ADDR: "node-a:7190"
      SEEDS: '["node-b:7190", "node-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5432:5432"   # PostgreSQL clients
      - "9090:9090"   # Prometheus metrics
      - "9191:9191"   # Semantic AI MCP
    stop_grace_period: 120s
    volumes:
      - node-a-data:/var/lib/scramdb

  node-b:
    image: scramdb/scramdb:latest
    hostname: node-b
    environment:
      NODE_NAME: node-b
      ADVERTISE_ADDR: "node-b:7190"
      SEEDS: '["node-a:7190", "node-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5433:5432"
      - "9091:9090"
      - "9291:9191"
    stop_grace_period: 120s
    volumes:
      - node-b-data:/var/lib/scramdb

  node-c:
    image: scramdb/scramdb:latest
    hostname: node-c
    environment:
      NODE_NAME: node-c
      ADVERTISE_ADDR: "node-c:7190"
      SEEDS: '["node-a:7190", "node-b:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
    networks: [scramdb-cluster]
    ports:
      - "5434:5432"
      - "9092:9090"
      - "9391:9191"
    stop_grace_period: 120s
    volumes:
      - node-c-data:/var/lib/scramdb

networks:
  scramdb-cluster:
    driver: bridge

volumes:
  node-a-data:
  node-b-data:
  node-c-data:

stop_grace_period is not decoration. A node hands its bucket ownership and its voter seat back before it exits, and that drain, the query drain and the JIT quiesce each get their own timeout, so a graceful stop can want around forty seconds. Compose's ten second default would kill it mid-drain and leave a configured voter the cluster keeps counting, which costs quorum the next time you scale in.

Port 9191 is the Semantic AI MCP server, which the engine starts in-process on every node. It has no HTTP authentication unless you set MCP_AUTH=basic, so publish it only on a network you control, or set that variable. See Semantic AI.

Set the license key once in your shell (or an .env file next to the compose file) before bringing the cluster up:

export SCRAMDB_LICENSE_KEY="<your-enterprise-or-trial-key>"
docker compose up -d

Without a valid key, each node logs the license error and exits; the compose file above will not form a cluster on its own.

The nodes are symmetric peers. Leadership is elected at runtime, so there is no required start order. A node accepts clients once the cluster has formed, and then you can connect to any node:

psql "host=127.0.0.1 port=5432 user=scramdb dbname=scramdb" \
  -c "SELECT node_id, role, state FROM scram.nodes ORDER BY node_id;"

Expect one voter row per node, each live. The formation lines (cluster transport listening, swim: member up) log at info; set SCRAMDB_LOG: info on the services to see them in docker compose logs.

The three env vars only cover node identity and seeds. To tune anything else cluster-specific, for example [cluster.swim], [cluster.consensus], or [storage.wal.archive], mount a complete config file and pass -c with its path as the service's command: instead of the env vars; the entrypoint's three-variable path renders a template with just those fields, and it refuses to start a container that sets the env vars and names a config file too.

Kubernetes​

Use the Helm chart. It is the supported path, and Kubernetes deployment is the full guide.

kubectl create secret generic scramdb-license \
  --from-literal=license-key="<your Enterprise or Trial token>"

helm repo add scramdb https://charts.scramdb.com
helm install scramdb scramdb/scramdb \
  --set architecture=cluster \
  --set replicaCount=3 \
  --set license.existingSecret=scramdb-license \
  --set resourcesPreset=medium

The rest of this section explains what that produces, because the shape matters whether you use the chart or write the manifests yourself: a StatefulSet behind a headless Service, each node with a stable identity and its own persistent volume, discovering its peers by DNS name so no address is hardcoded, and SCRAMDB_LICENSE_KEY on every pod from a Secret.

Scale-down drains itself

A terminating pod hands its bucket ownership and its voter seat back before it exits, so reducing the replica count needs no command of its own. What it does need is a terminationGracePeriodSeconds long enough to finish: see Scaling for the arithmetic, and Draining a node for the on-demand form.

Seedless formation​

The StatefulSet's ConfigMap carries no seed list. A hardcoded list of peer addresses is only correct at the exact replica count it was written for; scale up or down and it goes stale. Instead, every pod sets bootstrap_expect, the number of other voter-role nodes it waits to see connected before founding the cluster, and finds those peers itself through the headless Service's DNS plus SWIM:

[cluster]
node_name = "__NODE_NAME__"
cluster_listen = "0.0.0.0:7190"
cluster_interactive_listen = "0.0.0.0:7191"
cluster_bulk_listen = "0.0.0.0:7192"
advertise_addr = "__NODE_NAME__.scramdb-headless:7190"
bootstrap_expect = 2
discovery = ["dns", "swim"]
dns_name = "scramdb-headless"
dns_refresh = "5s"
replication_factor = 3
group0_bootstrap_timeout = "60s"

__NODE_NAME__ is the one templated token: the pod substitutes it with its own metadata.name, read through the downward API, before it hands the rendered file to scramdb -c (the Helm chart does the same in an init container). All three transport ports are set rather than left to follow cluster_listen, so a taken port stops the pod instead of moving a lane to a port the Service and a network policy would not carry. Every pod runs this identical template and derives the identical genesis set, so there is nothing per-node to hand-edit.

One number does have to match the topology: bootstrap_expect is replicas - 1. Three replicas is bootstrap_expect = 2; five is bootstrap_expect = 4. The Helm chart derives it from the replica count, summed across every region pool, so it cannot drift; if you hand-write the manifests, change both together. Get it wrong and the cluster does not misform silently, it waits out group0_bootstrap_timeout and then refuses, loudly, naming what it was waiting for.

Scaling up afterwards needs no config edit at all. A node that finds an already-formed cluster boots as an elastic joiner instead of founding a rival one, and the running cluster adds it; bootstrap_expect only ever decides who founds a cluster from nothing.

The license Secret​

Cluster mode needs the same Enterprise or Trial key on every pod that Docker Compose needs in your shell, just delivered through a Secret instead of an environment variable. Create it once, before the first kubectl apply:

kubectl create secret generic scramdb-license \
  --from-literal=license-key="<your Enterprise token>"

and reference it from the container spec instead of a literal value:

env:
  - name: SCRAMDB_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: scramdb-license
        key: license-key

A missing Secret holds the pod at CreateContainerConfigError, with a kubectl describe pod event naming exactly which Secret it wanted, before the container so much as starts. That's the same loud, name-the-variable refusal a Community-licensed process logs on its own stderr, just caught one layer up by the kubelet instead.

Size each pod's CPU request and limit like any single node: a Guaranteed QoS pod with the kubelet's static CPU manager policy pins exclusive cores per replica, and the default policy quota-bounds them instead. See Parallelism for how ScramDB resolves either into a worker count, and for SCRAMDB_MAX_CORES when you bin-pack more than one node onto the same host.

Geo placement across regions and zones​

Every node, in Docker Compose or Kubernetes, can carry an optional region and an optional zone label. A table created through a labelled node homes its voters in that region and commits at region-local quorum latency; ALTER TABLE ... SET (home_region = '<region>') re-homes an existing table live. Nothing about the SQL you write changes either way.

Leaving a label unset is always safe, and so is setting it to an empty string: an empty or whitespace-only value is trimmed and dropped, so the node comes up unlabelled, never a node whose region is literally named "". A compose file or manifest that never mentions REGION/ZONE at all behaves exactly as it always has.

Docker Compose​

docker/cluster-config.template.toml renders REGION/ZONE straight into the node's [cluster] section, next to the fields you've already seen:

[cluster]
node_name = "${NODE_NAME}"
advertise_addr = "${ADVERTISE_ADDR}"
seeds = ${SEEDS}
region = "${REGION}"
zone = "${ZONE}"

Set them per service, the same way you set NODE_NAME. A genuinely multi-region compose file just gives different services different values. replication_factor is fixed at 3 in the demo template, so six nodes, three per region, is the shape that keeps every home-region table's replicas fully inside its own region:

services:
  eu-a:
    image: scramdb/scramdb:latest
    hostname: eu-a
    environment:
      NODE_NAME: eu-a
      ADVERTISE_ADDR: "eu-a:7190"
      SEEDS: '["eu-b:7190", "eu-c:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5432:5432"   # PostgreSQL clients
      - "9090:9090"   # Prometheus metrics
    volumes:
      - eu-a-data:/var/lib/scramdb

  eu-b:
    image: scramdb/scramdb:latest
    hostname: eu-b
    environment:
      NODE_NAME: eu-b
      ADVERTISE_ADDR: "eu-b:7190"
      SEEDS: '["eu-a:7190", "eu-c:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5433:5432"
      - "9091:9090"
    volumes:
      - eu-b-data:/var/lib/scramdb

  eu-c:
    image: scramdb/scramdb:latest
    hostname: eu-c
    environment:
      NODE_NAME: eu-c
      ADVERTISE_ADDR: "eu-c:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "us-a:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: eu-central
      ZONE: eu-central-1a
    networks: [scramdb-cluster]
    ports:
      - "5434:5432"
      - "9092:9090"
    volumes:
      - eu-c-data:/var/lib/scramdb

  us-a:
    image: scramdb/scramdb:latest
    hostname: us-a
    environment:
      NODE_NAME: us-a
      ADVERTISE_ADDR: "us-a:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-b:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5435:5432"
      - "9093:9090"
    volumes:
      - us-a-data:/var/lib/scramdb

  us-b:
    image: scramdb/scramdb:latest
    hostname: us-b
    environment:
      NODE_NAME: us-b
      ADVERTISE_ADDR: "us-b:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-a:7190", "us-c:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5436:5432"
      - "9094:9090"
    volumes:
      - us-b-data:/var/lib/scramdb

  us-c:
    image: scramdb/scramdb:latest
    hostname: us-c
    environment:
      NODE_NAME: us-c
      ADVERTISE_ADDR: "us-c:7190"
      SEEDS: '["eu-a:7190", "eu-b:7190", "eu-c:7190", "us-a:7190", "us-b:7190"]'
      SCRAMDB_LICENSE_KEY: "${SCRAMDB_LICENSE_KEY}"
      REGION: us-east
      ZONE: us-east-1a
    networks: [scramdb-cluster]
    ports:
      - "5437:5432"
      - "9095:9090"
    volumes:
      - us-c-data:/var/lib/scramdb

networks:
  scramdb-cluster:
    driver: bridge

volumes:
  eu-a-data:
  eu-b-data:
  eu-c-data:
  us-a-data:
  us-b-data:
  us-c-data:

Bring it up the same way as the three-node cluster: one exported license key, one docker compose up -d. The per-service settings from that example still apply to every node here, stop_grace_period above all. Each node's own boot log confirms what it rendered:

entrypoint: cluster mode - node_name=eu-a advertise_addr=eu-a:7190 seeds=["eu-b:7190",...] region=eu-central zone=eu-central-1a

On a single bridge network, REGION/ZONE change placement, not physical latency: every node is one Docker hop from every other regardless of label. The labels are real, and a table homed through eu-a genuinely keeps its voters on eu-a/eu-b/eu-c; what a single-host demo can't show you is the cross-ocean round trip a real two-region deployment would.

Kubernetes​

The same idea shaped for Kubernetes is one StatefulSet per region, all sharing one headless Service, forming one flat cluster. In the Helm chart that is a list:

architecture: cluster
cluster:
  replicationFactor: 2
regions:
  - name: eu
    zone: eu-central-1a
    replicas: 2
    nodeSelector:
      topology.kubernetes.io/region: eu-central-1
  - name: us
    zone: us-east-1a
    replicas: 2
    nodeSelector:
      topology.kubernetes.io/region: us-east-1
helm install scramdb scramdb/scramdb -f regions.yaml

Every pool gets its own StatefulSet, its own REGION/ZONE, its own placement, and the same headless Service, so pods from both pools resolve each other by DNS as if they were one StatefulSet. bootstrap_expect is derived from the sum of the pools, which is 3 here (four pods, each waiting for the other three).

replicationFactor: 2 is a deliberate match to two nodes per region: with RF 2, each region can fully supply a home-region table's replicas from its own pool, so a re-home in either direction is a clean move, never a degraded one, the same property the six-node Compose example above gets from three nodes per region instead.

k8s/overlays/regions-2x2 in the engine repository is the hand-written version of this, kept as a kind development loop.

On one kind cluster both pools land on the same nodes, so this still only demonstrates placement, the same caveat as Compose above. On a real multi-region cluster, pin each pool to its region's actual nodes with a nodeSelector or topologySpreadConstraints; the StatefulSet/ConfigMap shape above doesn't change.

Configuration essentials​

Deployments that use a configuration file instead of environment variables turn on cluster mode with a [cluster] section:

[general]
license_key = "<your-enterprise-or-trial-key>"

[cluster]
node_name = "node-a"
advertise_addr = "node-a:7190"
seeds = ["node-b:7190", "node-c:7190"]
replication_factor = 3

The presence of the [cluster] section is what activates multi-node mode, and it's what triggers the license check above. Remove it and the same binary runs as a single node with no license requirement. replication_factor sets how many copies of each piece of data the cluster keeps (default 3); higher means more machine failures tolerated at the cost of more replicated write traffic. [cluster] has several sub-tables ([cluster.swim], [cluster.consensus], [cluster.rebalance], [cluster.dilith], [cluster.distributed_join], [cluster.exchange] and more) for tuning gossip, consensus, rebalancing, distributed-transaction and distributed-query behavior beyond the defaults; they aren't covered in depth on this page. Two plain top-level fields, region and zone, label a node for geo placement; see Geo placement across regions and zones above. bootstrap_expect replaces a hardcoded seeds list for DNS-discovered formation; see Seedless formation under Kubernetes above.

Operating a cluster​

  • Connect to any node. All nodes are equal peers and serve the same consistent database. Put a load balancer in front of the client port if you want a single endpoint.
  • Monitor freshness and health. SHOW FRESHNESS reports replication lag per data group, and every node exposes Prometheus metrics on port 9090 plus a plain /health endpoint. See Observability for the full scramdb_cluster_* reference and which counters are worth alerting on.

Draining a node​

Use ALTER CLUSTER DRAIN '<node-name>' to move a node's data and responsibilities to the rest of the cluster before you shut it down, whether for maintenance or to scale in:

ALTER CLUSTER DRAIN 'node-b';

You can issue it from any node. The membership change it ends with is a consensus operation only the metadata-group leader can propose, and the request is forwarded to that leader in one bounded hop, so there is no leader to find first.

Bucket reassignment converges regardless of which node leads each affected shard group, and regardless of which node you issue the statement against: the reassignment is recorded once, cluster-wide, and each group's own leader drives its own membership change toward it.

This is the on-demand form, not the only one. A node that shuts down gracefully, including a Kubernetes pod being terminated, already runs the same drain against itself first; see Scaling. A node that crashed runs nothing, and ALTER CLUSTER DRAIN against its name is how you reclaim its seat afterwards.

Summary​

A ScramDB cluster is one serializable, highly available database built from the same engine as a single node. Transactions are serializable across every node, distributed transactions commit atomically through a Raft-backed commit protocol, replication keeps one consistent copy rather than stale followers, and row-level security holds no matter how a query is distributed. You get availability and capacity without giving up the guarantees, or the SQL, you rely on, on the Enterprise license tier.