Skip to main content

Self-driving with Ejder

By the end of this page you will know what Ejder changes on its own, how to turn it off or limit it, how to read what it did, and which metrics to watch.

Ejder is the self-driving loop inside ScramDB. It watches the engine's own counters, then changes settings and runs maintenance when the counters say it helps. It checks every change against a guard and undoes one that makes things worse. It records every decision, so nothing it does is hidden.

It needs no setup​

Ejder ships installed. A fresh node starts it and takes its first decision at once. Nothing has to be configured.

To turn it off, set the environment variable SCRAMDB_AUTONOMY=off, or set enabled = false under [autonomy] in the config file. The variable wins over the file. It reads on, off, 1, 0, true and false, and anything else stops the node at start with a message naming the variable.

Every other key is optional. See [autonomy] for the list, the defaults and the bounds.

What it changes​

Changes are grouped in classes. Each class has an authority:

  • observe: Ejder only watches. It records what it saw and changes nothing.
  • advise: Ejder records what it would do and changes nothing. You act on it, or raise the class to auto.
  • auto: Ejder does it, then checks the result.

Both Ejder and the class must allow a change before it runs. If either says observe, nothing runs. If either says advise, it is only advised.

ClassDefaultWhat it covers
analyzeautoRefreshing a table's statistics before the next scheduled sweep.
setpointsautoMoving an engine setting inside its bounds: how many heavy queries run at once, the aging bound and morsel threshold, pausing compaction or the statistics sweep, the compaction interval, the cold demotion age, the vector search settings and the fuel and memory caps of a package.
compactionautoAsking for a background merge of one table.
indexadviseCreating or dropping an index.
checkpointadviseAsking for a checkpoint.
clusteradviseMoving a table's shard group to another region and draining a node.
retentionobserveHow long a sealed segment stays local before the cold tier copies it to object storage, when a cold tier is configured.

An index change runs as an ordinary CREATE INDEX or DROP INDEX, resolved against the catalog first. A table that is quarantined is never touched.

Every change is checked​

When Ejder acts, it opens a window with a baseline for the number the change was meant to improve. When the window closes, a change that helped is accepted. A change that hurt is undone, and the undo is recorded as reverted. A change whose window ran out with no clear answer is recorded as expired.

Each setting has its own guard: the number that must not get worse, and the direction of change it watches. Raising the heavy query limit is guarded by the memory left in the pool, and pausing compaction by the count of tables waiting for a merge. A step Ejder takes to protect the node, such as lowering the heavy query limit under memory pressure, is never undone by its guard. A newer change to the same setting replaces the older one's window, which is recorded as expired.

If the node is quarantined from a replication group, Ejder only advises. If a decision fails five times in a row, the loop parks itself and stops deciding until the package is installed again. It reports the reason in scram_autonomy and raises scramdb_autonomy_parked.

In a cluster​

Each node decides for itself about itself, at its own cadence. Decisions about the whole cluster are taken only on the leader of the metadata group, at the cluster cadence (cluster_tick_interval).

A cluster change is fenced by the leader's term. If leadership moves between the decision and the change, the change is refused and recorded as such. Ejder acts on a cluster only when it is sure it leads. The cluster class is advise by default, so a re-home or a drain is recorded as a recommendation and runs only once you raise the class to auto. A drain never targets the leader, never takes the cluster below its minimum size, and a node is not drained twice within a cooldown.

See what it is doing​

The scram_autonomy view has one row while the loop runs on this node, and no row otherwise.

sql
SELECT * FROM scram_autonomy;
ColumnMeaning
packageThe name the Ejder package is installed under.
digestThe digest of the installed package.
staterunning, or parked after repeated failures.
parked_reasonWhy it parked. NULL while it runs.
last_failureThe last decision that failed. NULL when none has.
ticksDecisions taken since the node started.
interval_msThe interval between decisions now, in milliseconds.
open_windowsChanges still being checked.
self_cost_permilleThe share of one core the last decision cost, in thousandths.

Only a superuser or a member of pg_read_all_stats (which pg_monitor includes) sees the row. Any other role sees none.

The decision record​

Every decision is written to the autonomy schema. The schema is created at the first decision, so on a node that has just started it appears within a few seconds. Only a superuser or a member of pg_read_all_stats can read it.

Every table except state is kept within two bounds at once: a row older than audit_retention (seven days by default) goes, and so does every row past the newest audit_max_rows, the oldest first. A pass runs when the loop starts, then every hour, and again after each tick interval while rows past a bound remain; the keys are in the [autonomy] section of the configuration file. The scope column is node for a decision about this node and cluster for one about the cluster. tick_seq counts decisions within a scope.

TableOne row perColumns
autonomy.ticksDecisionscope, tick_seq, at_unix_ms, micros, cpu_micros, frame_bytes, interval_ms, failure, watchdog_band, pool_available_permille, admission_parked, explanations_cut
autonomy.actionsAction proposedscope, tick_seq, at_unix_ms, action_id, class, object, outcome, reason
autonomy.setpointsSetting change or refusalscope, tick_seq, at_unix_ms, lever, old_value, new_value, outcome, reason
autonomy.explanationsReason Ejder gavescope, tick_seq, at_unix_ms, code, object, message
autonomy.shadowModel prediction (only with ml set to shadow or on)scope, tick_seq, at_unix_ms, model, model_version, features, prediction, classical, diverged, refused
autonomy.stateScopescope, tick_seq, blob, updated_unix_ms

A decision records at most 64 explanations. The rest are counted in autonomy.ticks.explanations_cut, so a cut is never silent. autonomy.state holds Ejder's memory, so a restart resumes where it stopped.

An action's outcome is one of dispatched, advised, refused, accepted, reverted, expired, completed or failed. A setting change's outcome is applied, advised, refused or reverted.

Once the loop has taken a decision, this reads what it did last:

text
SELECT at_unix_ms, class, object, outcome, reason
FROM autonomy.actions
ORDER BY at_unix_ms DESC
LIMIT 20;

Recording decisions​

With record = true, Ejder also writes each decision's input and answer to a file, for fitting models offline. It is off by default.

The files go to the autonomy folder under the storage base directory. Each start opens a new file named record-<node>-<start time in unix ms>.frames. A file stops growing at record_max_bytes (1 GiB by default). A decision that would pass the bound is dropped whole and counted in scramdb_autonomy_records_dropped_total, never written in part.

Metrics​

Every series below is on /metrics whether or not Ejder runs. A node that does not run it reports scramdb_autonomy_running at 0 and zeros. The full list is in the metric reference.

Two are worth an alert: scramdb_autonomy_parked at 1 means the loop stopped deciding, and a rising scramdb_autonomy_audit_failures_total means decisions are made but not recorded. The Helm chart ships both alerts.