Self-driving with Ejder
By the end of this page you will know what Ejder changes on its own, how to turn it off or limit it, how to read what it did, and which metrics to watch.
Ejder is the self-driving loop inside ScramDB. It watches the engine's own counters, then changes settings and runs maintenance when the counters say it helps. It checks every change against a guard and undoes one that makes things worse. It records every decision, so nothing it does is hidden.
It needs no setup
Ejder ships installed. A fresh node starts it and takes its first decision at once. Nothing has to be configured.
To turn it off, set the environment variable SCRAMDB_AUTONOMY=off, or set enabled = false under [autonomy] in the config file. The variable wins over the file. It reads on, off, 1, 0, true and false, and anything else stops the node at start with a message naming the variable.
Every other key is optional. See [autonomy] for the list, the defaults and the bounds.
What it changes
Changes are grouped in classes. Each class has an authority:
observe: Ejder only watches. It records what it saw and changes nothing.advise: Ejder records what it would do and changes nothing. You act on it, or raise the class toauto.auto: Ejder does it, then checks the result.
Both Ejder and the class must allow a change before it runs. If either says observe, nothing runs. If either says advise, it is only advised.
| Class | Default | What it covers |
|---|---|---|
analyze | auto | Refreshing a table's statistics before the next scheduled sweep. |
setpoints | auto | Moving an engine setting inside its bounds: how many heavy queries run at once, the aging bound and morsel threshold, pausing compaction or the statistics sweep, the compaction interval, the cold demotion age, the vector search settings and the fuel and memory caps of a package. |
compaction | auto | Asking for a background merge of one table. |
index | advise | Creating or dropping an index. |
checkpoint | advise | Asking for a checkpoint. |
cluster | advise | Moving a table's shard group to another region and draining a node. |
retention | observe | How long a sealed segment stays local before the cold tier copies it to object storage, when a cold tier is configured. |
An index change runs as an ordinary CREATE INDEX or DROP INDEX, resolved against the catalog first. A table that is quarantined is never touched.
Every change is checked
When Ejder acts, it opens a window with a baseline for the number the change was meant to improve. When the window closes, a change that helped is accepted. A change that hurt is undone, and the undo is recorded as reverted. A change whose window ran out with no clear answer is recorded as expired.
Each setting has its own guard: the number that must not get worse, and the direction of change it watches. Raising the heavy query limit is guarded by the memory left in the pool, and pausing compaction by the count of tables waiting for a merge. A step Ejder takes to protect the node, such as lowering the heavy query limit under memory pressure, is never undone by its guard. A newer change to the same setting replaces the older one's window, which is recorded as expired.
If the node is quarantined from a replication group, Ejder only advises. If a decision fails five times in a row, the loop parks itself and stops deciding until the package is installed again. It reports the reason in scram_autonomy and raises scramdb_autonomy_parked.
In a cluster
Each node decides for itself about itself, at its own cadence. Decisions about the whole cluster are taken only on the leader of the metadata group, at the cluster cadence (cluster_tick_interval).
A cluster change is fenced by the leader's term. If leadership moves between the decision and the change, the change is refused and recorded as such. Ejder acts on a cluster only when it is sure it leads. The cluster class is advise by default, so a re-home or a drain is recorded as a recommendation and runs only once you raise the class to auto. A drain never targets the leader, never takes the cluster below its minimum size, and a node is not drained twice within a cooldown.
See what it is doing
The scram_autonomy view has one row while the loop runs on this node, and no row otherwise.
SELECT * FROM scram_autonomy;
| Column | Meaning |
|---|---|
package | The name the Ejder package is installed under. |
digest | The digest of the installed package. |
state | running, or parked after repeated failures. |
parked_reason | Why it parked. NULL while it runs. |
last_failure | The last decision that failed. NULL when none has. |
ticks | Decisions taken since the node started. |
interval_ms | The interval between decisions now, in milliseconds. |
open_windows | Changes still being checked. |
self_cost_permille | The share of one core the last decision cost, in thousandths. |
Only a superuser or a member of pg_read_all_stats (which pg_monitor includes) sees the row. Any other role sees none.
The decision record
Every decision is written to the autonomy schema. The schema is created at the first decision, so on a node that has just started it appears within a few seconds. Only a superuser or a member of pg_read_all_stats can read it.
Every table except state is kept within two bounds at once: a row older than audit_retention (seven days by default) goes, and so does every row past the newest audit_max_rows, the oldest first. A pass runs when the loop starts, then every hour, and again after each tick interval while rows past a bound remain; the keys are in the [autonomy] section of the configuration file. The scope column is node for a decision about this node and cluster for one about the cluster. tick_seq counts decisions within a scope.
| Table | One row per | Columns |
|---|---|---|
autonomy.ticks | Decision | scope, tick_seq, at_unix_ms, micros, cpu_micros, frame_bytes, interval_ms, failure, watchdog_band, pool_available_permille, admission_parked, explanations_cut |
autonomy.actions | Action proposed | scope, tick_seq, at_unix_ms, action_id, class, object, outcome, reason |
autonomy.setpoints | Setting change or refusal | scope, tick_seq, at_unix_ms, lever, old_value, new_value, outcome, reason |
autonomy.explanations | Reason Ejder gave | scope, tick_seq, at_unix_ms, code, object, message |
autonomy.shadow | Model prediction (only with ml set to shadow or on) | scope, tick_seq, at_unix_ms, model, model_version, features, prediction, classical, diverged, refused |
autonomy.state | Scope | scope, tick_seq, blob, updated_unix_ms |
A decision records at most 64 explanations. The rest are counted in autonomy.ticks.explanations_cut, so a cut is never silent. autonomy.state holds Ejder's memory, so a restart resumes where it stopped.
An action's outcome is one of dispatched, advised, refused, accepted, reverted, expired, completed or failed. A setting change's outcome is applied, advised, refused or reverted.
Once the loop has taken a decision, this reads what it did last:
SELECT at_unix_ms, class, object, outcome, reason
FROM autonomy.actions
ORDER BY at_unix_ms DESC
LIMIT 20;
Recording decisions
With record = true, Ejder also writes each decision's input and answer to a file, for fitting models offline. It is off by default.
The files go to the autonomy folder under the storage base directory. Each start opens a new file named record-<node>-<start time in unix ms>.frames. A file stops growing at record_max_bytes (1 GiB by default). A decision that would pass the bound is dropped whole and counted in scramdb_autonomy_records_dropped_total, never written in part.
Metrics
Every series below is on /metrics whether or not Ejder runs. A node that does not run it reports scramdb_autonomy_running at 0 and zeros. The full list is in the metric reference.
Two are worth an alert: scramdb_autonomy_parked at 1 means the loop stopped deciding, and a rising scramdb_autonomy_audit_failures_total means decisions are made but not recorded. The Helm chart ships both alerts.