Skip to main content

Scaling: adding and removing nodes

By the end of this page you will know exactly what happens, automatically and manually, when you add capacity to a running ScramDB cluster or take a node away, and where the honest limits of that are today.

Adding a node is genuinely automatic​

Bring up a new node whose seeds list points at the existing cluster's peers, and it joins on its own; no admin command is required. Every running node runs an auto-join control loop that, once it discovers a new, live peer, adds it as a non-voting member first. After a short catch-up grace period, it is promoted to a full voting member. Bring up the process, point it at the existing cluster, and wait; the running cluster absorbs it.

This is the accurate way to grow a cluster today: adding capacity does not require you to run any SQL against the cluster to make it happen.

Removing a node, honestly​

ScramDB's engine has a tested mechanism to drain a node, reassign every bucket it owns to the remaining live members, wait for that to converge, and only then remove it from membership, all without ever observing a shard group below its target replication factor mid-transition. That mechanism exists and is exercised by the project's own tests at the engine level.

It is not yet reachable as a SQL statement or a scramdb CLI command. There is currently no client-invocable way to trigger a graceful, pre-drained node removal from a running cluster. If you send SQL that looks like a cluster-admin statement, expect a normal SQL parse error, not a cluster operation.

A node now runs that same mechanism against itself automatically, the moment it shuts down gracefully. Every node's shutdown sequence, the one a normal process stop, docker stop, or a Kubernetes pod termination already triggers, drains its own membership before it exits: it reassigns its buckets to the remaining live members, waits for that to converge, and removes itself, bounded by the same shutdown_drain_timeout_secs deadline (10 seconds by default) the rest of graceful shutdown uses. Two cases are deliberately exempted:

  • A node that isn't currently a member (never joined, or already drained) does nothing; there is nothing to hand back.
  • The cluster's last remaining voter keeps its seat. Removing it would leave a cluster with zero members and no way back short of manual surgery, so it stays a voter through its own shutdown; restarting that same node is how you get both it and the cluster back.

If the deadline passes before the drain converges (a slow peer, a partition), shutdown logs the failure loudly and lets the process exit anyway rather than hang waiting on it; a stuck shutdown is worse than the phantom-voter state described below, and that state is still fully recoverable. This makes the numbered procedure below the fallback path for a crash, an unclean kill, or a drain that ran out of time, not the routine case anymore.

The honest, complete procedure for the cases self-drain doesn't cover is:

  1. The process is already gone. The cluster keeps serving from its remaining replicas, per your configured replication_factor, exactly as described in Failover.
  2. Know that its voter slot is not automatically reclaimed. Per the invariant in Failover, a voter that disappears without going through its own graceful shutdown is never auto-removed from membership; it remains a phantom member, and the shard groups it owned run one replica short of your configured replication factor until it comes back or is manually addressed.
  3. If you need a supported live-removal procedure for your topology, contact ScramDB about your specific plan rather than improvising a workaround; the underlying mechanism exists, it just isn't self-service yet.

Do not expect an ALTER CLUSTER DRAIN, SHOW CLUSTER, or SHOW SHARDS statement to work against a running server; none of the three has a client entry point today, regardless of what you might see referenced elsewhere.

What rebalancing does​

When the set of live members for a bucket changes, whether because a node joined or because the placement ranking is recomputed after a change, ScramDB recomputes which nodes should own each bucket and moves ownership to match. New members are added as non-voting learners of a group before promotion, and a departing member's data is intended to be reassigned before it is dropped from membership, specifically so a group is never observed with fewer live copies than its target replication factor calls for during a planned transition. This is what makes adding a node a safe, non-disruptive operation; it is also exactly the mechanism that is not yet exposed for a manual, on-demand drain (see above).

Kubernetes scale-down​

Scaling a StatefulSet down sends the same SIGTERM any graceful stop does, so the automatic self-drain above applies here too: the terminating pod hands back its bucket ownership and its voter seat before it actually exits. The shipped manifest's preStop hook is unchanged, an explicit two-second placeholder sleep that only buys time for the process's own listeners to shut down cleanly; the drain itself runs afterward, in the shutdown sequence proper, not in preStop. terminationGracePeriodSeconds (30 seconds in the shipped manifest) comfortably covers the preStop sleep plus the 10-second drain deadline plus the rest of graceful shutdown, so a routine scale-down completes the hand-back before Kubernetes escalates to SIGKILL.

This fires on every graceful pod termination, not only an intentional scale-down. The mechanism has no way to tell a pod that's gone for good from one a rolling restart brings back a few seconds later under the same identity; both send the same signal. A restarted pod rejoins the ordinary way, a non-voting learner first, promoted after the catch-up grace period, so the round trip resolves itself automatically, but it is real reconfiguration work on every rolling deploy, not a no-op.

If a pod is force-killed before the drain deadline (an aggressively short terminationGracePeriodSeconds, a node eviction, or the underlying machine disappearing), none of this runs, and that pod's shard groups continue serving from one fewer copy than your configured replication_factor until it comes back or is manually addressed, exactly as in the manual process-stop case above.

Hybrid nodes: analytics without a hop​

Setting columnar_replica = true on a voter turns it into a hybrid node: it keeps its normal bucket ownership exactly like any other voter, and also mirrors every entry committed to the groups it owns into a full local columnar store. A query that would otherwise need to hop to a dedicated analytics replica instead runs whole-statement analytics right there, with zero network hops, on a node that is simultaneously a full write-serving member of the cluster.

The honest recommendation scales with node count. At the documented three-node size, set columnar_replica = true on all three: every node answers analytics locally, and minimum latency wins because there's no smaller topology to dedicate a separate tier out of. At five nodes or more, leave it false and add a learner tier instead: a dedicated node buys you hard resource isolation between the transactional and analytical workloads, where a hybrid node inherently shares one machine's CPU, memory, and disk between both.

The cost is named, not hidden. A hybrid node pays for its zero-hop analytics with one full columnar copy of its owned data resident on that node, plus a second apply per committed entry: every write commits to the row store as usual, and is also applied to the columnar store. Budget for both when sizing a hybrid node's disk and CPU headroom.

columnar_replica and learner are mutually exclusive, and the combination is refused at boot: a learner already owns no buckets, so there is nothing of its own for a columnar mirror to mirror; it is already the analytics tier by definition. Set exactly one of the two per node.

[cluster]
node_name = "node-a"
cluster_listen = "0.0.0.0:7190"
advertise_addr = "10.0.0.1:7190"
seeds = ["10.0.0.2:7190", "10.0.0.3:7190"]
replication_factor = 3
columnar_replica = true   # PERF: this node also mirrors every committed entry into its own columnar store

Why the mirror can never slow down a commit​

The columnar copy is written on its own lane, off the consensus apply path entirely, never inline with it. A transaction commits and is acknowledged the moment the row store, the authoritative replica, durably applies its entry; the columnar write follows afterward, on a background worker that runs at idle IO priority, the same priority class as the segment compactor and auto-analyze, because from the row store's perspective this work is maintenance and nothing waits on it.

That worker's queue is bounded, 4,096 entries deep, on purpose: an unbounded queue would trade commit latency for unbounded memory growth under a sustained write storm, which is just a slower way to fail. When it fills, the mirror drops the entry rather than blocking the row store or the consensus group behind it, and marks its own local copy incomplete; the drop count is tracked, so a node that has silently fallen behind is a distinguishable state, not a wrong answer served with confidence. A mirror failure of any kind (a dropped entry or an apply error) degrades that node's local analytics only. It never fails the write, never rolls back a committed transaction, and never stalls the shard group behind an analytics-only store.

The reasoning is the same either way: writing a wide, column-major analytics record is a fundamentally different disk access pattern than the row store's own write path, and putting it inline between a commit and its acknowledgement would let a slow or busy columnar disk show up directly as your transactional p99. Offloading it keeps that risk contained to the one thing it can actually affect: how current this node's own local analytics are.

Learner nodes: a read-only role​

Setting learner = true on a node's [cluster] config makes it a permanent, non-voting member: it hosts replicated data for local reads but owns no write-serving buckets, and it is never promoted to a voter, even automatically. This is a genuine, tested role you can add to a cluster for local analytical reads that stay off the write path.

A minimal learner config, added as a fourth node alongside an existing three-node cluster:

[cluster]
node_name = "node-d-replica"
cluster_listen = "0.0.0.0:7190"
advertise_addr = "10.0.0.4:7190"
seeds = ["10.0.0.1:7190", "10.0.0.2:7190", "10.0.0.3:7190"]
learner = true

Bring it up the same way as any other node (see Forming a cluster); it joins via the same auto-join path as a voting node, it simply never gets promoted.

What a learner is not: it does not provide geo-partitioning, and it does not give you a separate read-only connection string or automatic query routing that prefers it for reads. A client still connects to whichever node's address it is given, learner or voter, and gets that node's own answer; nothing in ScramDB automatically routes a read to the nearest or least-loaded node.

As a scaling lever, a learner is the correct pattern for adding read-only analytical capacity without growing the size of your write quorum, since it never participates in and never blocks a commit's majority vote.

Next​