Skip to main content

Forming a cluster

By the end of this page you will have a running three-node ScramDB cluster, verified with a query answered from more than one node, using either Docker Compose, plain processes on your own machines, or Kubernetes.

0. License first​

Multi-node clustering, including a one-node cluster, requires an Enterprise or Trial license. A node with a [cluster] section in its config checks this before it binds any port. If you skip this step, every node logs something like:

cluster mode requires an Enterprise license (multi-node clustering is an
Enterprise-only capability); this node's license resolved to Community -
no license key configured. Remove [cluster] from the config to run
single-node, or configure license_key / SCRAMDB_LICENSE_KEY with a valid
Enterprise license to enable clustering.

and exits. Set your key one of two ways:

  • Environment variable SCRAMDB_LICENSE_KEY (takes precedence if both are set).
  • license_key under [general] in scramdb.toml.

Every node in the cluster needs its own valid key. Keep it out of source control; treat it like any other credential.

1. Start from a single node, then take the smallest cluster step: a cluster-of-one​

If you already run single-node ScramDB (no [cluster] section), nothing about that setup changes yet. The smallest useful step toward a cluster is turning that same node into a one-node cluster, which is a real cluster with zero distributed overhead, not a stub.

Add this to scramdb.toml:

[general]
license_key = "your-enterprise-or-trial-key"

[cluster]
node_name = "node-a"

cluster_listen defaults to 0.0.0.0:7190 and seeds defaults to [] (empty), which is exactly what a cluster-of-one needs. Start the node:

scramdb -c scramdb.toml

Expect log lines like:

cluster mode active: node_id=node-a seeds=0 replication_factor=3
cluster transport listening on 0.0.0.0:7190

If it fails: no cluster mode active line and instead the licensing error from step 0 means your license key is missing or invalid. Re-check SCRAMDB_LICENSE_KEY or [general] license_key.

This one node is a fully functional cluster you can query normally over its PostgreSQL port. The rest of this page grows it to three nodes so it can actually tolerate a failure; a cluster-of-one has no fault tolerance by itself (see Multi-zone for why).

2. Docker Compose: the fastest way to a three-node cluster​

ScramDB's repository ships a docker-compose.yml that brings up three symmetric peers, node-a, node-b, and node-c, each publishing its PostgreSQL port and its Prometheus metrics port to the host. As shipped, it does not set a license key, so add one to each service before you run it.

  1. Add your license key to each service's environment: block (node-a, node-b, node-c):

    environment:
      NODE_NAME: node-a
      ADVERTISE_ADDR: "node-a:7190"
      SEEDS: '["node-b:7190", "node-c:7190"]'
      SCRAMDB_LOG: info
      SCRAMDB_LICENSE_KEY: "your-enterprise-or-trial-key"

    Repeat for node-b and node-c, keeping each node's own NODE_NAME, ADVERTISE_ADDR, and SEEDS (its two peers) as shipped.

  2. Build and start all three nodes together:

    docker compose up --build -d

    There is deliberately no start-order dependency between the three services; peer leadership is elected at runtime, not by which container comes up first.

  3. Watch the logs:

    docker compose logs -f

    Expect a cluster transport listening line and a swim: member up line for each of the other two peers, once per node.

  4. Connect and query. node-a publishes its PostgreSQL port on the host at 5432:

    psql "host=127.0.0.1 port=5432 user=scramdb dbname=scramdb" -c "select 1;"

    Expect:

     ?column?
    ----------
            1
    (1 row)

  5. Tear down when you're done:

    docker compose down -v

A note on host ports in the shipped file: node-b's compose entry happens to publish its container's PostgreSQL port on host port 7190. That is a coincidence of numbers with the cluster transport port, not the cluster port itself; node-b's real cluster transport port (container-side 7190) is not published to the host at all in the repo's file. Your own compose file is free to pick clearer host ports; only the container-internal ADVERTISE_ADDR/SEEDS values need to say 7190.

Security note: the released image's default command passes --pg-no-auth, which disables client authentication entirely and binds to every interface. That is fine for this local three-node demo on a network you control; do not run it that way anywhere else.

If it fails: a partially-set trio of NODE_NAME/ADVERTISE_ADDR/SEEDS (one or two set, not all three) causes the container to fail hard on startup with a message naming which variables are missing, it will not silently fall back to single-node. Set all three or none.

Tuning beyond the three env vars​

The Docker image's entrypoint only templates three [cluster] keys, node_name, advertise_addr, and seeds, through NODE_NAME/ADVERTISE_ADDR/SEEDS. Everything else in the shipped config, the rest of [cluster] (SWIM, Raft, rebalance, transaction, and transport tunables) and all of [storage], [udf], and [gpu], is a fixed value baked into the image. To change any of it, mount your own complete config file over /etc/scramdb/config.toml, or pass a custom path with -c, instead of relying on the three-variable path. Start from the shipped example below rather than writing that file from scratch.

Start from the shipped example​

You don't have to write a cluster config by hand. The image already carries a fully annotated cluster config template at /etc/scramdb/cluster-config.template.toml, the exact file docker/entrypoint.sh renders with envsubst when NODE_NAME, ADVERTISE_ADDR, and SEEDS are all set. It has a comment on nearly every key, and it covers [general], [gpu], [cluster], [storage] with its [storage.memory] / [storage.io] / [storage.cold_config] / [storage.wal.archive] / [storage.wal.retention] sub-tables, and [udf] with its [udf.thrust] / [udf.registry] sub-tables.

  1. Read it straight out of the image, no running server required:

    docker run --rm --entrypoint cat scramdb/scramdb:latest /etc/scramdb/cluster-config.template.toml

    Expect the whole file to print to your terminal: a header comment describing how it's rendered, then each section listed above in order, ending with [udf.registry]. Already have a node running under the name scramdb (as in Quick Start)? Copy straight out of that container instead:

    docker cp scramdb:/etc/scramdb/cluster-config.template.toml ./cluster-config.toml

  2. Save a copy to edit:

    docker run --rm --entrypoint cat scramdb/scramdb:latest /etc/scramdb/cluster-config.template.toml > cluster-config.toml

  3. Edit cluster-config.toml. Mounting your own copy skips the container's envsubst step entirely, so the [cluster] block's placeholders need literal values now, not environment variables:

    [cluster]
    node_name = "node-a"
    cluster_listen = "0.0.0.0:7190"
    advertise_addr = "node-a:7190"
    seeds = []
    replication_factor = 3

    This reproduces the cluster-of-one from step 1 above; for a real multi-node file, set seeds to the node's actual peers, one edited file per node, the same as step 3 below. Change anything else in the file the same way.

  4. Mount the edited copy back in, over the same path the rest of these docs use for a custom config, /etc/scramdb/config.toml:

    docker run -d --rm \
      --name scramdb-from-template \
      -p 5432:5432 \
      -p 9090:9090 \
      -e SCRAMDB_LICENSE_KEY="your-enterprise-or-trial-key" \
      -v scramdb-template-data:/var/lib/scramdb \
      -v $(pwd)/cluster-config.toml:/etc/scramdb/config.toml:ro \
      scramdb/scramdb:latest

    docker logs -f scramdb-from-template

    Expect the same two lines as the cluster-of-one in step 1, then Ctrl-C out of the log follow:

    cluster mode active: node_id=node-a seeds=0 replication_factor=3
    cluster transport listening on 0.0.0.0:7190
Don't set the three env vars on this container too

If NODE_NAME, ADVERTISE_ADDR, and SEEDS are all set, the entrypoint re-renders the image's own template into /etc/scramdb/cluster-config.rendered.toml and runs that instead, no matter what you've mounted at /etc/scramdb/config.toml. Your edits would sit on disk unused, not merged in. Leave all three unset when you're running from a mounted, hand-edited file.

Tear down the demo:

docker stop scramdb-from-template
docker volume rm scramdb-template-data

What the three env vars actually touch. Only these [cluster] keys in the template are placeholders; everything else in the file is a fixed value baked into the image:

PlaceholderEnvironment variableFills in
${NODE_NAME}NODE_NAME[cluster] node_name
${ADVERTISE_ADDR}ADVERTISE_ADDR[cluster] advertise_addr
${SEEDS}SEEDS[cluster] seeds (literal TOML array text, e.g. ["node-b:7190","node-c:7190"])

One thing worth checking before reusing this file across real, separate machines: the shipped [storage.wal.archive] destination points at a plain local path. That's fine for one node, but on a real cluster it needs to name storage every node can actually reach, or point-in-time recovery history fragments across leader failovers; see the [storage.wal.archive] notes in the config file reference.

Need a knob that isn't in this file at all? The template is deliberately a short subset. A much larger, fully annotated cluster-config.example.toml lives at the root of the ScramDB engine's source repository and documents every [cluster.swim] / [cluster.consensus] / [cluster.rebalance] / [cluster.txn] / [cluster.transport] / [storage.wal] / [storage.gc] / [storage.compaction] knob and its default, one field per line. It isn't shipped inside the container, so docker cp and docker run --entrypoint cat won't reach it there; read it from the repository, then carry the values you want into your own mounted config file.

3. Bare metal or VMs: one TOML file per node​

Without Docker, cluster mode is selected purely by config; there is no env-var path off Docker for the per-field settings. Write one file per node, differing only in node_name, advertise_addr, and seeds. Each file can start from the same shipped template as above instead of a blank file, if you'd rather edit than type one from scratch.

node-a.toml:

[general]
license_key = "your-enterprise-or-trial-key"

[cluster]
node_name = "node-a"
cluster_listen = "0.0.0.0:7190"
advertise_addr = "10.0.0.1:7190"
seeds = ["10.0.0.2:7190", "10.0.0.3:7190"]
replication_factor = 3

node-b.toml and node-c.toml are identical except node_name, advertise_addr is that node's own address, and seeds lists the other two nodes. Keep replication_factor and every other tunable identical across all three files.

Start each node on its own machine:

scramdb -c node-a.toml

Expect the same cluster mode active / cluster transport listening lines as step 1, followed by swim: member up for each peer once all three are running.

If it fails: a node that never logs swim: member up for a peer usually means that peer's advertise_addr is not reachable from this node (a firewall or routing issue), since advertise_addr is what peers dial, not cluster_listen.

4. Kubernetes​

ScramDB's repository also ships a headless-service-backed StatefulSet for a local kind cluster. Treat the manifests as a development and demo reference, not a production deployment as-is: the image reference (scramdb:dev) is a placeholder you replace with your own registry image, and the manifests carry no license key by default either, so add SCRAMDB_LICENSE_KEY to the container's environment (or bake it into the mounted config) before applying them.

  1. Build and load the image into a local kind cluster:

    docker build -f docker/Release.x64.Dockerfile -t scramdb:dev .
    kind create cluster --name scramdb-dev
    kind load docker-image scramdb:dev --name scramdb-dev

  2. Apply the three-node overlay:

    kubectl apply -k k8s/overlays/dev-3
    kubectl rollout status statefulset/scramdb --timeout=180s
    kubectl get pods -l app=scramdb -o wide

    Expect three Running pods, scramdb-0, scramdb-1, scramdb-2.

  3. Verify the mesh formed:

    for i in 0 1 2; do
      echo "--- scramdb-$i ---"
      kubectl logs "scramdb-$i" | grep -E "cluster mode active|cluster transport: bound"
    done

    Expect one cluster mode active: node_id=scramdb-<i> ... line and one cluster transport: bound ... line per pod.

  4. Verify SWIM sees every peer:

    for i in 0 1 2; do
      kubectl logs "scramdb-$i" | grep -E "swim: member (up|suspect|down)"
    done

    Expect every pod to log swim: member up for each of the other two node ids within a few seconds of all pods being ready.

  5. Query it:

    kubectl run scramdb-psql --rm -it --restart=Never --image=postgres:16-alpine -- \
      psql "host=scramdb-0.scramdb-headless port=5432 user=scramdb dbname=scramdb" \
      -c "select 1;"

  6. Tear down:

    kubectl delete -k k8s/overlays/dev-3
    kind delete cluster --name scramdb-dev

For a five-node topology, use k8s/overlays/dev-5 in place of dev-3 throughout; only the replica count and bootstrap_expect differ, covered next.

These exact commands are the supported path against the shipped manifests; treat step timings and log text as what to expect on a healthy kind cluster, and adjust probe timeouts for a slower machine.

Seedless formation with bootstrap_expect​

The dev-3 overlay you just ran never sets a seeds list at all. k8s/base/configmap.yaml uses a different mechanism, bootstrap_expect, specifically because a hardcoded seed list is only ever correct at the one replica count it was written for: name two peers for a 3-node cluster and that same file is wrong the moment you scale to five, wrong again if you scale to two, and wrong in the other direction if a node's ordinal changes. A StatefulSet's replica count is a single number you already change in one place (replicas:); a seed list is a second, per-node address list you'd otherwise have to keep in sync with it by hand, at every replica count except the one it was written for.

bootstrap_expect fixes this by moving the founding decision from "here are their addresses" to "here is how many there are." Left unset (the default), nothing changes: seeds.len() decides the count, exactly as every path earlier on this page already does. Set it explicitly instead, and:

  • It replaces the address list, not just the count. Every node runs the identical [cluster] config, no per-pod seed templating, and waits until it has connected to bootstrap_expect peers, discovered through dns or swim rather than a hand-typed list, before founding together with whichever peers it found. This is what k8s/base/configmap.yaml actually does.
  • 0 is the explicit spelling of a cluster-of-one, the same shape seeds = [] already gives you by default, for a config that would rather say what it means than lean on an empty list's implicit behavior.
  • A learner never fills a voter slot. bootstrap_expect counts VOTERS. If your seeds list also names read-replica (learner = true) nodes, set bootstrap_expect to the voter count: seeds naming two voters and one read replica boot with bootstrap_expect = 2. Leaving it unset there means every voter waits for a third voter that does not exist, times out, and the cluster never forms. The timeout error reports how many connected peers were learners, so the misconfiguration names itself.
  • It is refused at boot, loudly, in two cases, rather than silently forming something you didn't ask for: if it is greater than a non-empty seeds list (it cannot expect more founding peers than anything names; below seeds.len() is the voters-plus-learner-seeds case above), and if it's positive with an empty seeds list and no discovery provider that could ever actually supply an address (dns with dns_name set, or swim) is configured, since the node would otherwise wait out the entire group0_bootstrap_timeout for peers that were never coming and only then fail.

A real, complete seedless [cluster] section, the same shape the shipped ConfigMap renders per pod:

[cluster]
node_name = "node-a"                                    # unique per node; falls back to the OS hostname if omitted
advertise_addr = "node-a.scramdb-headless:7190"         # this node's own address, resolvable by every peer

# No `seeds` list: every node runs this identical config and finds its
# peers through DNS instead of a hand-typed address list.
bootstrap_expect = 2                                     # counts the OTHER voters: 3 replicas - 1
discovery = ["dns", "swim"]
dns_name = "scramdb-headless.default.svc.cluster.local"
dns_refresh = "5s"
replication_factor = 3

Every founding node needs the same bootstrap_expect value for this to converge on the one cluster you meant to start, exactly as every node's seeds list has to name the same founding set today; a mismatch doesn't corrupt anything (each side's genesis is fenced by an incarnation check that keeps two independently-formed groups from ever merging or cross-talking) but it does mean you get two small clusters instead of the one you wanted. Scaling k8s/overlays/dev-5 to five nodes in place of dev-3 changes exactly one line from the base config, bootstrap_expect = 4, alongside replicas: 5; there is no seed list to edit because there never was one.

A hardcoded seed list is correct at exactly one replica count: the one it was written for. bootstrap_expect is correct at every replica count, because it never names a replica at all.

5. Verify from any node​

Whichever path you used, confirm peer symmetry by querying more than one node and getting the same answer:

psql "host=<node-a-address> port=5432 user=scramdb dbname=scramdb" -c "select 1;"
psql "host=<node-b-address> port=5432 user=scramdb dbname=scramdb" -c "select 1;"

Both should return the same single row. There is no separate coordinator to connect to; every node answers.

What's next​

You have a working cluster. From here:

  • Multi-zone to spread it across availability zones for real fault tolerance.
  • Multi-region if you need nodes further apart than one region.
  • Failover to know exactly what happens, and what your client sees, when a node dies.
  • Scaling to add or remove nodes from a cluster that's already running.