Skip to main content

Forming a cluster

By the end of this page you will have a running three-node ScramDB cluster, verified with a query answered from more than one node, using either Docker Compose, plain processes on your own machines, or Kubernetes.

0. License first​

Multi-node clustering, including a one-node cluster, requires an Enterprise or Trial license. A node with a [cluster] section in its config checks this before it binds any port. If you skip this step, every node logs something like:

cluster mode requires an Enterprise license (multi-node clustering is an
Enterprise-only capability); this node's license resolved to Community -
no license key configured. Remove [cluster] from the config to run
single-node, or configure license_key / SCRAMDB_LICENSE_KEY with a valid
Enterprise license to enable clustering.

and exits. Set your key one of two ways:

  • Environment variable SCRAMDB_LICENSE_KEY (takes precedence if both are set).
  • license_key under [general] in scramdb.toml.

Every node in the cluster needs its own valid key. Keep it out of source control; treat it like any other credential.

1. Start from a single node, then take the smallest cluster step: a cluster-of-one​

If you already run single-node ScramDB (no [cluster] section), nothing about that setup changes yet. The smallest useful step toward a cluster is turning that same node into a one-node cluster, which is a real cluster with zero distributed overhead, not a stub.

Add this to scramdb.toml:

[general]
license_key = "your-enterprise-or-trial-key"

[cluster]
node_name = "node-a"

cluster_listen defaults to 0.0.0.0:7190 (with interactive and bulk traffic on the next two ports, see Ports) and seeds defaults to [] (empty), which is exactly what a cluster-of-one needs. Start the node, with its log at info so the lines below show (the default level, warn, prints only problems):

SCRAMDB_LOG=info scramdb -c scramdb.toml

Expect log lines like:

cluster mode active: node_id=node-a advertise_addr=10.0.0.1:7190 listen=0.0.0.0:7190 seeds=0 replication_factor=3
cluster transport: bound node_id=node-a listen=0.0.0.0:7190 seeds=0 tls=false ...
cluster transport: interactive traffic listening on 0.0.0.0:7191 (queue 16781824 bytes each way)
cluster transport: bulk traffic listening on 0.0.0.0:7192 (queue 150999552 bytes each way)
cluster transport listening on 0.0.0.0:7190

If it fails: no cluster mode active line and instead the licensing error from step 0 means your license key is missing or invalid. Re-check SCRAMDB_LICENSE_KEY or [general] license_key.

This one node is a fully functional cluster you can query normally over its PostgreSQL port. The rest of this page grows it to three nodes so it can actually tolerate a failure; a cluster-of-one has no fault tolerance by itself (see Multi-zone for why).

2. Docker Compose: the fastest way to a three-node cluster​

ScramDB's repository ships a docker-compose.yml that brings up three symmetric peers, node-a, node-b, and node-c, on one host. Inside the compose network every node listens on the same ports (5432 for PostgreSQL clients, 7190 to 7192 for the cluster transport's control, interactive and bulk traffic, 9090 for metrics, 9191 for the Semantic AI MCP server); the file maps each container's client, metrics and MCP ports to the host so you can reach the nodes from your machine, and keeps the cluster transport ports on the compose network. Every service takes SCRAMDB_LICENSE_KEY and SCRAMDB_LOG from your shell, so nothing in the file needs editing. (For the same three nodes with plain docker run, see Quick Start.)

  1. Export your license key, and info logging so the formation lines in step 3 show (the default, warn, prints only problems):

    export SCRAMDB_LICENSE_KEY="your-enterprise-or-trial-key"
    export SCRAMDB_LOG=info

  2. Build and start all three nodes together:

    docker compose up --build -d

    There is deliberately no start-order dependency between the three services; peer leadership is elected at runtime, not by which container comes up first.

  3. Watch the logs:

    docker compose logs -f

    Expect a cluster transport listening line and a swim: member up line for each of the other two peers, once per node.

  4. Connect and query. Every node serves the same consistent database, so any node is a correct endpoint. Because the three containers share one host in this demo, the file maps their PostgreSQL ports to three host ports (node-a at 5432, node-b at 5433, node-c at 5434, with the metrics and MCP ports shifted the same way); that is a single-host convenience, not how a cluster is addressed. In a real deployment every node listens on 5432 and clients reach the cluster through one endpoint: on Kubernetes the chart's client Service, which load balances across all nodes (see Kubernetes), and elsewhere any node or a multi-host connection string (see Connecting to a cluster). Here, connect through node-a:

    psql "host=127.0.0.1 port=5432 user=scramdb dbname=scramdb" -c "select 1;"

    Expect:

     ?column?
    ----------
            1
    (1 row)

  5. Tear down when you're done:

    docker compose down -v

Security note: the released image's default command passes --pg-no-auth, which disables client authentication entirely and binds to every interface. That is fine for this local three-node demo on a network you control; do not run it that way anywhere else.

Shutdown budget: each service also sets stop_grace_period: 120s. A node hands back its bucket ownership and its voter seat before it exits, and that drain, the query drain, and the JIT quiesce each get their own timeout, so a graceful stop needs well over ten seconds. Compose's 10-second default would kill a node mid-drain and leave a configured voter the cluster keeps counting, which costs quorum on the next scale-down. The Helm chart and the Kubernetes manifests in the repository set the same budget (terminationGracePeriodSeconds: 120). This is longer than the single-node figure in Docker Deployment, because a single node has no voter seat or bucket ownership to hand back.

If it fails: a partially-set trio of NODE_NAME/ADVERTISE_ADDR/SEEDS (one or two set, not all three) causes the container to fail hard on startup with a message naming which variables are missing, it will not silently fall back to single-node. Set all three or none.

Tuning beyond the three env vars​

The Docker image's entrypoint only templates five [cluster] keys: node_name, advertise_addr, and seeds through NODE_NAME/ADVERTISE_ADDR/SEEDS, and the optional region and zone through REGION/ZONE. Everything else in the shipped config, the rest of [cluster] (SWIM, Raft, rebalance, transaction, and transport tunables) and all of [storage], [udf], and [gpu], is a fixed value baked into the image. To change any of it, mount your own complete config file over /etc/scramdb/config.toml, or mount it anywhere and name it with -c <path> as the container's command, instead of relying on the three-variable path. A container that names a config file with -c and also sets the three variables refuses to start and says so. Start from the shipped example below rather than writing that file from scratch.

Start from the shipped example​

You don't have to write a cluster config by hand. The image already carries a fully annotated cluster config template at /etc/scramdb/cluster-config.template.toml, the exact file docker/entrypoint.sh renders with envsubst when NODE_NAME, ADVERTISE_ADDR, and SEEDS are all set. It has a comment on nearly every key, and it covers [general], [gpu], [cluster] with its [cluster.consensus] / [cluster.vector] / [cluster.log_compaction] / [cluster.apply] / [cluster.dilith] sub-tables, [storage] with its [storage.memory] / [storage.io] / [storage.cold_config] / [storage.wal.archive] / [storage.wal.checkpoint] / [storage.wal.retention] sub-tables, [udf] with its [udf.thrust] / [udf.registry] sub-tables, and [vector].

  1. Read it straight out of the image, no running server required:

    docker run --rm --entrypoint cat scramdb/scramdb:latest /etc/scramdb/cluster-config.template.toml

    Expect the whole file to print to your terminal: a header comment describing how it's rendered, then each section listed above in order, ending with [vector]. Already have a node running under the name scramdb (as in Quick Start)? Copy straight out of that container instead:

    docker cp scramdb:/etc/scramdb/cluster-config.template.toml ./cluster-config.toml

  2. Save a copy to edit:

    docker run --rm --entrypoint cat scramdb/scramdb:latest /etc/scramdb/cluster-config.template.toml > cluster-config.toml

  3. Edit cluster-config.toml. Mounting your own copy skips the container's envsubst step entirely, so the [cluster] block's placeholders need literal values now, not environment variables:

    [cluster]
    node_name = "node-a"
    cluster_listen = "0.0.0.0:7190"
    cluster_interactive_listen = "0.0.0.0:7191"
    cluster_bulk_listen = "0.0.0.0:7192"
    advertise_addr = "node-a:7190"
    seeds = []
    replication_factor = 3
    region = ""
    zone = ""

    Replace all five placeholders, region and zone included (an empty string means no label); a placeholder left in place is read as literal text. This reproduces the cluster-of-one from step 1 above; for a real multi-node file, set seeds to the node's actual peers, one edited file per node, the same as step 3 below. Change anything else in the file the same way.

  4. Mount the edited copy back in, over the same path the rest of these docs use for a custom config, /etc/scramdb/config.toml:

    docker run -d --rm \
      --name scramdb-from-template \
      -p 5432:5432 \
      -p 9090:9090 \
      -e SCRAMDB_LICENSE_KEY="your-enterprise-or-trial-key" \
      -e SCRAMDB_LOG=info \
      -v scramdb-template-data:/var/lib/scramdb \
      -v $(pwd)/cluster-config.toml:/etc/scramdb/config.toml:ro \
      scramdb/scramdb:latest

    docker logs -f scramdb-from-template

    Expect the same lines as the cluster-of-one in step 1, then Ctrl-C out of the log follow:

    cluster mode active: node_id=node-a advertise_addr=node-a:7190 listen=0.0.0.0:7190 seeds=0 replication_factor=3
    cluster transport: bound node_id=node-a listen=0.0.0.0:7190 seeds=0 tls=false ...
    cluster transport: interactive traffic listening on 0.0.0.0:7191 (queue 16781824 bytes each way)
    cluster transport: bulk traffic listening on 0.0.0.0:7192 (queue 150999552 bytes each way)
    cluster transport listening on 0.0.0.0:7190
Don't set the three env vars on this container too

If NODE_NAME, ADVERTISE_ADDR, and SEEDS are all set, the entrypoint re-renders the image's own template into /etc/scramdb/cluster-config.rendered.toml and runs that instead, no matter what you've mounted at /etc/scramdb/config.toml. Your edits would sit on disk unused, not merged in. Leave all three unset when you're running from a mounted, hand-edited file.

Tear down the demo:

docker stop scramdb-from-template
docker volume rm scramdb-template-data

What the three env vars actually touch. Only these [cluster] keys in the template are placeholders; everything else in the file is a fixed value baked into the image:

PlaceholderEnvironment variableFills in
${NODE_NAME}NODE_NAME[cluster] node_name
${ADVERTISE_ADDR}ADVERTISE_ADDR[cluster] advertise_addr
${SEEDS}SEEDS[cluster] seeds (literal TOML array text, e.g. ["node-b:7190","node-c:7190"])
${REGION}REGION (optional; unset is no region)[cluster] region
${ZONE}ZONE (optional; unset is no zone)[cluster] zone

One thing worth checking before reusing this file across real, separate machines: the shipped [storage.wal.archive] destination points at a plain local path. That's fine for one node, but on a real cluster it needs to name storage every node can actually reach, or point-in-time recovery history fragments across leader failovers; see the [storage.wal.archive] notes in the config file reference.

Need a knob that isn't in this file at all? The template is deliberately a short subset. A much larger, fully annotated cluster-config.example.toml lives at the root of the ScramDB engine's source repository and documents every [cluster.swim] / [cluster.consensus] / [cluster.rebalance] / [cluster.dilith] / [cluster.transport] / [storage.wal] / [storage.gc] / [storage.compaction] knob and its default, one field per line. It isn't shipped inside the container, so docker cp and docker run --entrypoint cat won't reach it there; read it from the repository, then carry the values you want into your own mounted config file.

3. Bare metal or VMs: one TOML file per node​

Without Docker, cluster mode is selected purely by config; there is no env-var path off Docker for the per-field settings. Write one file per node, differing only in node_name, advertise_addr, and seeds. Each file can start from the same shipped template as above instead of a blank file, if you'd rather edit than type one from scratch.

node-a.toml:

[general]
license_key = "your-enterprise-or-trial-key"

[cluster]
node_name = "node-a"
cluster_listen = "0.0.0.0:7190"
advertise_addr = "10.0.0.1:7190"
seeds = ["10.0.0.2:7190", "10.0.0.3:7190"]
replication_factor = 3

node-b.toml and node-c.toml are identical except node_name, advertise_addr is that node's own address, and seeds lists the other two nodes. Keep replication_factor and every other tunable identical across all three files.

Open ports 7190, 7191 and 7192 between every pair of machines, in both directions, and 5432 to your clients (see Ports). Then start each node on its own machine:

SCRAMDB_LOG=info scramdb -c node-a.toml

Expect the same cluster mode active / cluster transport listening lines as step 1, followed by swim: member up for each peer once all three are running.

If it fails: a node that never logs swim: member up for a peer usually means that peer's advertise_addr is not reachable from this node (a firewall or routing issue), since advertise_addr is what peers dial, not cluster_listen.

4. Kubernetes​

ScramDB's repository also ships a headless-service-backed StatefulSet for a local kind cluster. Treat the manifests as a development and demo reference, not a production deployment as-is: the image reference (scramdb:dev) is a placeholder you replace with your own registry image, and every pod reads SCRAMDB_LICENSE_KEY from a Secret named scramdb-license, so create it before applying them (kubectl create secret generic scramdb-license --from-literal=license-key="<your Enterprise or Trial key>"). The pods log at info, so the formation lines below show in kubectl logs.

  1. Build and load the image into a local kind cluster:

    docker build -f docker/Release.x64.Dockerfile -t scramdb:dev .
    kind create cluster --name scramdb-dev
    kind load docker-image scramdb:dev --name scramdb-dev

  2. Apply the three-node overlay:

    kubectl apply -k k8s/overlays/dev-3
    kubectl rollout status statefulset/scramdb --timeout=180s
    kubectl get pods -l app=scramdb -o wide

    Expect three Running pods, scramdb-0, scramdb-1, scramdb-2.

  3. Verify the mesh formed:

    for i in 0 1 2; do
      echo "--- scramdb-$i ---"
      kubectl logs "scramdb-$i" | grep -E "cluster mode active|cluster transport: bound"
    done

    Expect one cluster mode active: node_id=scramdb-<i> ... line and one cluster transport: bound ... line per pod.

  4. Verify SWIM sees every peer:

    for i in 0 1 2; do
      kubectl logs "scramdb-$i" | grep -E "swim: member (up|suspect|down)"
    done

    Expect every pod to log swim: member up for each of the other two node ids within a few seconds of all pods being ready.

  5. Query it:

    kubectl run scramdb-psql --rm -it --restart=Never --image=postgres:16-alpine -- \
      psql "host=scramdb-0.scramdb-headless port=5432 user=scramdb dbname=scramdb" \
      -c "select 1;"

  6. Tear down:

    kubectl delete -k k8s/overlays/dev-3
    kind delete cluster --name scramdb-dev

For a five-node topology, use k8s/overlays/dev-5 in place of dev-3 throughout; only the replica count and bootstrap_expect differ, covered next.

These exact commands are the supported path against the shipped manifests; treat step timings and log text as what to expect on a healthy kind cluster, and adjust probe timeouts for a slower machine.

Seedless formation with bootstrap_expect​

The dev-3 overlay you just ran never sets a seeds list at all. k8s/base/configmap.yaml uses a different mechanism, bootstrap_expect, specifically because a hardcoded seed list is only ever correct at the one replica count it was written for: name two peers for a 3-node cluster and that same file is wrong the moment you scale to five, wrong again if you scale to two, and wrong in the other direction if a node's ordinal changes. A StatefulSet's replica count is a single number you already change in one place (replicas:); a seed list is a second, per-node address list you'd otherwise have to keep in sync with it by hand, at every replica count except the one it was written for.

bootstrap_expect fixes this by moving the founding decision from "here are their addresses" to "here is how many there are." Left unset (the default), nothing changes: seeds.len() decides the count, exactly as every path earlier on this page already does. Set it explicitly instead, and:

  • It replaces the address list, not just the count. Every node runs the identical [cluster] config, no per-pod seed templating, and waits until it has connected to bootstrap_expect peers, discovered through dns or swim rather than a hand-typed list, before founding together with whichever peers it found. This is what k8s/base/configmap.yaml actually does.
  • 0 is the explicit spelling of a cluster-of-one, the same shape seeds = [] already gives you by default, for a config that would rather say what it means than lean on an empty list's implicit behavior.
  • A learner never fills a voter slot. bootstrap_expect counts VOTERS. If your seeds list also names read-replica (learner = true) nodes, set bootstrap_expect to the voter count: seeds naming two voters and one read replica boot with bootstrap_expect = 2. Leaving it unset there means every voter waits for a third voter that does not exist, times out, and the cluster never forms. The timeout error reports how many connected peers were learners, so the misconfiguration names itself.
  • It is refused at boot, loudly, in two cases, rather than silently forming something you didn't ask for: if it is greater than a non-empty seeds list (it cannot expect more founding peers than anything names; below seeds.len() is the voters-plus-learner-seeds case above), and if it's positive with an empty seeds list and no discovery provider that could ever actually supply an address (dns with dns_name set, or swim) is configured, since the node would otherwise wait out the entire group0_bootstrap_timeout for peers that were never coming and only then fail.

A real, complete seedless [cluster] section, the same shape the shipped ConfigMap renders per pod:

[cluster]
node_name = "node-a"                                    # unique per node; falls back to the OS hostname if omitted
advertise_addr = "node-a.scramdb-headless:7190"         # this node's own address, resolvable by every peer

# No `seeds` list: every node runs this identical config and finds its
# peers through DNS instead of a hand-typed address list.
bootstrap_expect = 2                                     # counts the OTHER voters: 3 replicas - 1
discovery = ["dns", "swim"]
dns_name = "scramdb-headless.default.svc.cluster.local"
dns_refresh = "5s"
replication_factor = 3

Every founding node needs the same bootstrap_expect value for this to converge on the one cluster you meant to start, exactly as every node's seeds list has to name the same founding set today; a mismatch doesn't corrupt anything (each side's genesis is fenced by an incarnation check that keeps two independently-formed groups from ever merging or cross-talking) but it does mean you get two small clusters instead of the one you wanted. Scaling k8s/overlays/dev-5 to five nodes in place of dev-3 changes exactly one line from the base config, bootstrap_expect = 4, alongside replicas: 5; there is no seed list to edit because there never was one.

A hardcoded seed list is correct at exactly one replica count: the one it was written for. bootstrap_expect is correct at every replica count, because it never names a replica at all.

5. Verify from any node​

Whichever path you used, confirm peer symmetry by querying more than one node and getting the same answer:

psql "host=<node-a-address> port=5432 user=scramdb dbname=scramdb" -c "select 1;"
psql "host=<node-b-address> port=5432 user=scramdb dbname=scramdb" -c "select 1;"

Both should return the same single row. There is no separate coordinator to connect to; every node answers.

What's next​

You have a working cluster. From here:

  • Multi-zone to spread it across availability zones for real fault tolerance.
  • Multi-region if you need nodes further apart than one region.
  • Failover to know exactly what happens, and what your client sees, when a node dies.
  • Scaling to add or remove nodes from a cluster that's already running.