Skip to main content

Writing a package

Packages are built on afterburner, a polyglot, deterministic WebAssembly runtime with its own package format, registry, and Cargo-style toolchain. You author, test, build, and publish packages with the burn command-line tool.

A package is the right tool when you have more than a few lines of code, want to write in a compiled language, want to share the function with others, or want the code versioned and reviewable independently of the database. For a quick one-off function, skip packages entirely and write CREATE FUNCTION ... LANGUAGE js AS $$...$$ inline; see Program your Database.

Install the toolchain​

Install burn once:

bash
curl -fsSL https://afterburner.sh | sh

On Windows, run iwr -useb https://afterburner.sh | iex in PowerShell. To pin a version, set BURN_VERSION first (for example BURN_VERSION=v0.2.9), or grab a tarball from the releases page.

Scaffold a package​

burn init creates a new package with the manifest, capability grant, and a starter source file already in place. We will follow the built-in embeddings_cosine_similarity package through the rest of this page, so scaffold it here (swap in your own namespace and name for your own packages):

bash
burn init ./embeddings_cosine_similarity --namespace scramdb --name embeddings_cosine_similarity

Develop against it the way you would any project:

bash
burn run # run the package entry, like `cargo run`
burn test # run the tests in tests/, sandboxed

Anatomy of a package​

A package is a directory with three parts: a manifest that describes it, a capability grant that says what it may touch, and your source code.

text
embeddings_cosine_similarity/
afb.toml # the manifest: what this package is
manifold.json # the capability grant: what it may touch (nothing, by default)
source/
main.js # your code: the function body

This three-part layout is the same across every supported language; only the entry file and the language field in afb.toml change (see Writing in another language below).

The manifest (afb.toml)​

afb.toml names the package and points at its entry file. Here is the manifest for one of the built-in starter packages:

toml
[format]
version = "1.0"

[package]
name = "embeddings_cosine_similarity"
namespace = "scramdb"
version = "0.1.0"
language = "js"
entry = "source/main.js"
description = "Cosine similarity of two embedding vectors, each a JSON array of numbers."

[runtime]
min = "0.1.0"

[[metadata.sql.function]]
name = "embeddings_cosine_similarity"
args = "a TEXT, b TEXT"
returns = "DOUBLE PRECISION"

The fields you set are:

FieldMeaning
nameThe package name, and the name you use when you register it in SQL
namespaceYour namespace, for example your team or organization
versionThe package version (semantic versioning)
languageThe source language (js for JavaScript, rust for Rust, and so on)
entryThe source file the package starts from
descriptionA short, human-readable description

The [format] and [runtime] sections are set for you when you scaffold a package and rarely need to change.

Declare the SQL function​

Each [[metadata.sql.function]] entry is a function the package offers to SQL: its name, its args written as in CREATE FUNCTION (the argument names are the column names your code reads from the batch, so they must match), its returns type, and optionally strict = true, which makes a NULL argument answer NULL without calling your code. scram.install registers every entry as a function with that signature, in the language the package is written in, so the function is callable as soon as the install returns. An entry that does not parse, or names a type a function cannot take (see Supported types), fails the install.

A package that declares no function must declare what else it is, or the install is refused naming the missing section: a daemon ([metadata.daemon], see Daemon packages), a library another package bundles ([metadata.library], which must not grant listen), or the self-driving agent ([metadata.autonomy]).

Sealed by default (manifold.json)​

manifold.json is the package's capability grant. When you scaffold a package it grants nothing, so your code starts fully sealed:

json
{
"fs": "None",
"net": "None",
"env": "None",
"crypto": false,
"child_process": false,
"allow_exit": false,
"http_timeout_ms": null,
"listen": "None"
}

Each field is a door: fs is filesystem access, net is outbound network access, env is reading environment variables, crypto is cryptographic APIs, child_process is spawning other processes, allow_exit permits the process to exit on its own, and http_timeout_ms/listen are only meaningful once net/child_process are open.

For a database function you almost always keep every door shut. A function that only computes over its arguments needs nothing else, and sealing is exactly what makes it safe to run inside the database. Whatever a package may touch is declared here, in the open, and reviewable before anyone installs it.

The values each door takes:

DoorValues
fs"None", {"ReadOnly": [paths]}, {"ReadWrite": [paths]}
net"None", {"OutboundHttp": [hosts]}, {"OutboundFull": [hosts]} (null in place of the list means any host)
listen"None", {"Ports": [n]}, {"PortRange": [lo, hi]}, "Any"
env"None", {"AllowList": [names]}, "Full"
crypto, child_process, allow_exittrue or false

A function called from SQL runs fully sealed, regardless of what its package's manifold.json declares. For that path the manifest is bookkeeping and documentation for the reader; the engine forces the module sealed no matter what it asks for, so a function cannot open net, fs, crypto, or child_process at all, even if its manifest claims one of those doors is open. Functions created inline with AS $$...$$ have no manifest and run sealed too.

A daemon package is the exception: one that declares [metadata.daemon] autostart = true runs with its manifold enforced as declared (see Daemon packages). It gets exactly what the manifold grants and nothing more, and the install is refused if the manifold grants no listen.

The function (source/main.js)​

A ScramDB function is handed a whole batch of rows at once and returns one result per row. This batch-at-a-time shape is why user functions keep up with ScramDB's query engine instead of slowing it down. It is the calling convention for JavaScript, whether the source lives inline in a CREATE FUNCTION ... AS $$...$$ statement or in a source-only package's entry file.

The entry file exports one function:

javascript
module.exports = (batch) => {
// read input columns, compute, return an output column
};

The batch your function receives, and the object it returns, follow a simple contract:

  • batch.columns.<name> is the input column for each declared SQL argument: an array with one value per row. A NULL is null, a bytea value is a Uint8Array, and a boolean is 1 or 0, so test it for truth (if (v)) rather than with === true.
  • batch.row_count is how many rows are in the batch.
  • Return { row_count, columns: { <output>: <array> } } with one output column, one value per row. The row count you return must match the row count you were given, or the call fails.

The batch holds only the rows the statement evaluates the call for, as PostgreSQL evaluates a function once per row it reaches: a row a WHERE removed, a row whose CASE branch, later COALESCE argument or AND/OR right side does not reach the call, and, for a function whose rows a LIMIT reads directly, a row past the ones the LIMIT reads are never in it. A function therefore never runs, or raises an error, for a row a WHERE or a branch keeps from it.

Here is the full body of the embeddings_cosine_similarity starter package. It takes two vectors (each a JSON array of numbers, stored as text) and returns their cosine similarity:

javascript
module.exports = (batch) => {
const a = batch.columns.a;
const b = batch.columns.b;
const n = batch.row_count;
const out = new Float64Array(n);

for (let i = 0; i < n; i++) {
const av = JSON.parse(a[i]);
const bv = JSON.parse(b[i]);
if (av.length !== bv.length) {
throw new Error("vector length mismatch (" + av.length + " vs " + bv.length + ")");
}
let dot = 0, na = 0, nb = 0;
for (let j = 0; j < av.length; j++) {
dot += av[j] * bv[j];
na += av[j] * av[j];
nb += bv[j] * bv[j];
}
const denom = Math.sqrt(na) * Math.sqrt(nb);
out[i] = denom !== 0 ? dot / denom : 0;
}

return { row_count: n, columns: { similarity: out } };
};

Reading it top to bottom:

  • a and b are the two input columns, named to match the SQL arguments you will declare (a text, b text).
  • The loop runs once per row, reading a[i] and b[i] and writing out[i].
  • The function returns a single similarity column, which becomes the value of RETURNS double precision when you register it.

Notice there is no I/O, no clock, and no randomness: the same inputs always produce the same result. Throwing an error on bad input (a length mismatch here) fails loudly rather than returning a wrong answer.

Once a package like this is built, it is installed the same way whether it is a starter package or one of your own: scram.install registers the function its manifest declares, and you call it. See Install and use a package.

Inline functions in Python and Ruby​

LANGUAGE python and LANGUAGE ruby take inline source (AS $$...$$ or AS '...') or the name of an installed package written in that language (AS 'namespace/pkg'). Both follow the same convention: a top-level function taking one batch argument and named exactly like the SQL function when it was created, or like the second string of AS $$...$$, 'name' when you give one (the function keeps that entry through a rename, see The entry function of a python or ruby routine). For a package, the manifest's entry file (source/main.py or source/main.rb) holds that function, and one package can define the function of every SQL function bound to it:

sql
-- the package's source/main.py defines def double_it(batch)
CREATE OR REPLACE FUNCTION double_it(x bigint) RETURNS bigint
LANGUAGE python AS 'your_namespace/your_package';

A package whose [[metadata.sql.function]] entries are registered by scram.install binds them in the language the package is written in (language = "python" or "ruby" in afb.toml). A package bound through LANGUAGE python or LANGUAGE ruby carries only its entry source: the vendored pip or gem members of a package are not mounted for it yet.

Python's batch is a plain dict, accessed with [...], not attribute access:

sql
CREATE OR REPLACE FUNCTION double_it(x bigint) RETURNS bigint LANGUAGE python AS $$
def double_it(batch):
xs = batch['columns']['x']
n = batch['row_count']
return {'row_count': n, 'columns': {'r': [v * 2 for v in xs]}}
$$;

Ruby's batch is a Hash with symbol keys at the top level (:columns, :row_count), but the column map inside :columns is keyed by plain strings, the argument name:

sql
CREATE OR REPLACE FUNCTION double_it(x bigint) RETURNS bigint LANGUAGE ruby AS $$
def double_it(batch)
xs = batch[:columns]['x']
n = batch[:row_count]
{ row_count: n, columns: { 'r' => xs.map { |v| v * 2 } } }
end
$$;

Output type inference limit. For both Python and Ruby, the wire type of each output column is inferred from its first value only: a bool becomes boolean, an int becomes bigint, a float becomes double precision, a text string becomes text, and a raw byte string becomes bytea. There is no way today to select a narrower type such as int, real, or date from Python or Ruby. Declare RETURNS to match what the inference actually produces (bigint, double precision, boolean, or text), not the narrower SQL type you might otherwise reach for.

A time limit on the processor. JavaScript UDFs run under the fuel, memory, timeout, and output-size limits described in Install and use a package. The [udf] fuel and memory limits apply to JavaScript functions. [udf] timeout_ms bounds a Python or Ruby call too; it counts the processor time the function's thread spends and not the clock: a call that spends more processor time than the interpreter's own start plus the scaled timeout_ms is ended, and the statement answers UDF "name" exceeded its time budget (timeout_ms=1000; raise [udf].timeout_ms). A busy machine never ends a valid call, and the start of the interpreter is not counted. A statement that is running one also ends at statement_timeout or on a cancel, with SQLSTATE 57014. See also Not yet supported.

Do not time anything in a Python function. Inside a Python function time.monotonic() always reads 0.0 and time.time() is a fixed date. Do not sleep or time anything in a Python function; wait in the caller instead.

Writing in another language​

Only the language field in afb.toml and the entry file change across languages:

Languageafb.toml languageEntry fileRuns as
JavaScriptjavascript (or js)source/main.jsInterpreted by afterburner's own JS engine
TypeScripttypescriptsource/main.tsInterpreted by afterburner's own JS engine, same as JavaScript, once built into a package (see below)
Rustrustsource/main.rsCompiled ahead of time to a native WebAssembly binary
Gogosource/main.goCompiled ahead of time to a native WebAssembly binary
Ccsource/main.cCompiled ahead of time to a native WebAssembly binary
C++cppsource/main.cppCompiled ahead of time to a native WebAssembly binary
Pythonpythonsource/main.pyInterpreted by the embedded Python runtime; the entry function is named like the SQL function
Rubyrubysource/main.rbInterpreted by the embedded Ruby runtime; the entry function is named like the SQL function

Native languages are compiled by their own toolchains into the same sealed .afb artifact, and multi-file projects are fully supported: a package can have several source files with cross-module calls and its own private helpers. ScramDB's built-in starter set includes a compiled Rust package (rust_sha256) right alongside the JavaScript ones.

TypeScript. burn scaffolds, builds, and publishes a TypeScript package exactly like a JavaScript one, and it is installed and called the same way (see Install and use a package); bound by hand it takes LANGUAGE js AS 'namespace/pkg'. Inline source works too: LANGUAGE ts AS $$...$$ strips the types of the body and runs the JavaScript that is left.

Two different calling conventions depending on how the package was built​

This is the part that trips people up, so it gets its own section: which convention your package's .afb uses depends on how it was built, not on what language you typed.

  • A source-only package (JavaScript or TypeScript, kept as interpretable source inside the .afb) is executed the same way inline LANGUAGE js AS $$...$$ is: the entry file exports module.exports = (batch) => {...}, described above under The function.
  • A precompiled package (Rust, Go, C, or C++, compiled ahead of time to a real WebAssembly binary) runs as a plain command: it reads one JSON value from stdin and writes one JSON value to stdout. The input is either a single row object, keyed by your function's own declared SQL argument names, or a JSON array of row objects for a batch call. The output is the matching shape: a single JSON scalar, or a JSON array of scalars. A malformed input must make your program exit non-zero with a message on stderr, never panic.

Concretely, a Rust package's main looks nothing like the JavaScript module.exports(batch) shape. It reads stdin, parses one row (or an array of rows) as JSON, computes a scalar (or array of scalars) per row, and writes that back to stdout as JSON. This is the contract the built-in rust_sha256 package follows, and it is the contract any Rust, Go, C, or C++ package must follow, because those languages are compiled ahead of time rather than interpreted inside afterburner's own JS/TS engine.

Both conventions are installed and called identically from SQL. scram.install registers rust_sha256 as the package declares it; to bind the same package under another name:

sql
CREATE OR REPLACE FUNCTION sha256_with_salt(value text, salt text)
RETURNS text
LANGUAGE js
AS 'scramdb/rust_sha256';

The LANGUAGE js in that statement names the binding form, not the package's implementation language. It is the keyword for packages written in JavaScript, TypeScript, Rust, Go, C, or C++; a package of Python or Ruby source is bound with LANGUAGE python or LANGUAGE ruby (see Install and use a package). scram.install has already registered the function under the package's own name, so this statement is only for another name.

The burn commands​

CommandWhat it does
burn new <name>, burn init [dir]Scaffold a package in a new directory, or in an existing one. --lang picks the language, --template the starter entry
burn run, burn check <file>Run the package entry; or compile a script without running it (exit code 0 on success, 1 on an error), for CI
burn test, burn bench <file>Run every file under tests/; or measure throughput and p50 and p99 latency (--iters, --workers)
burn packageBuild the .afb locally. --compile precompiles it, and --wasm-only ships the precompiled module with no source
burn compileCompile the source ahead of time to a self-contained WebAssembly module and bundle it with the source
burn install, burn add <pkg>Resolve and fetch the dependencies, writing burn.lock; or add one to afb.toml, pinned by digest
burn login, burn publishSign in to a registry, and build and upload the package to it
burn search <query>, burn info <pkg>, burn yank <pkg@version>Search the registry; show a package's versions, capabilities and digest; hide a version from resolution (--undo restores it)

A build is reproducible: identical inputs give a byte-identical archive.

The .afb format​

An .afb is a zstd-compressed tar archive, and it is content-addressed: the SHA-256 of its compressed bytes is the package's identity. It holds afb.toml, manifold.json and the files under source/, and may also hold a precompiled WebAssembly module under precompiled/<target>/main.wasm. A reader accepts the format version it knows and any later minor version of it, and refuses a different major version loudly.

ScramDB checks every archive it installs before it reads it. A compressed archive over 50 MiB, more than 256 MiB once decompressed (checked while streaming, so an archive that expands too far is stopped early), more than 1000 entries, or any single entry over 64 MiB is refused, and so is an archive that holds a path with .., an absolute path, a symbolic link or any entry that is not a regular file.

Once your package is written, the next page shows how to build it, install it, and call it from SQL.