Migrations
Schema evolution the generator way — a version interlock, automatic additive backfill, Auto/Authored hop classification, and an offline transformer bin driven by forgedb migrate create → migrate up, with a per-tenant sweep.
ForgeDB is a code generator, so schema evolution follows the generate-then-compile model: you
edit schema.forge, regenerate, and recompile. What happens to the data on disk depends
on the change:
- Additive changes (a new model, a new nullable field) are preserved automatically — the regenerated app backfills them on reopen. No data step.
- Everything else (a type change, a column/model drop, a nullable→NOT-NULL narrowing, a
&uniqueadd, a required field with no default) rewrites data-at-rest. ForgeDB generates a per-version offline transformer bin that does the rewrite, driven byforgedb migrate.
The transformer is generated code, not a runtime engine
The schema is never a runtime input to a generic engine. The transformer is generated code — one straight-line typed replay per origin→destination version range — not an interpreter that reads your schema at runtime.
The version interlock#
Before touching data, a generated app checks one version number. Every app bakes in the format version its schema expects, and each data directory records the version it was written at. On a mismatch the app refuses to open rather than mis-read stale bytes. A fresh database starts at version 1, and each recorded migration bumps it by one.
The format_version is one opaque integer per data-dir manifest, derived from your migration
lineage. It is the only cross-schema handshake between a generated app and a directory: the
check never reads column names or types to "self-heal." That makes it fail-fast in both
directions. A regenerated app won't silently read a not-yet-migrated directory, and the
transformer won't apply the wrong version range to one.
Additive changes — automatic#
Adding a new model or a new nullable field needs no data step. Record the change, regenerate, restart, and the app backfills the new column on reopen. Existing rows are never rewritten.
# 1. Edit schema.forge — add the new nullable field AT THE END of the model.
# 2. Record the change (baselines the lineage on first run):
forgedb migrate create "add note field" --auto --schema schema.forge
# 3. Regenerate and rebuild:
forgedb generate
# 4. Restart. Existing rows are backfilled with defaults on first open.Two constraints. Append new fields at the end of the model, because columns are addressed
by position, not by name. And a new non-null field backfills to the type's zero value, not its
@default (so prefer nullable when the zero is not meaningful).
On reopen, generated recovery anchors on the tombstone row count (the authoritative committed count) and backfills any column shorter than that anchor. The short column is the new field; every other column already matches the anchor, so existing rows are left untouched. This is why fields must be appended at the end: a mid-model insert would shift every later column off its anchored length.
Data-rewriting changes — hop classification#
Every other change rewrites data at rest — a type change, a dropped column or model, a
nullable→NOT-NULL narrowing, a &unique add, a required field with no default. Running
forgedb migrate create --auto records the change as a versioned hop and classifies it.
Structural changes it can rewrite on its own. For changes where it can't know the new value (a
type re-encode, or filling a new required field), it scaffolds a transform.rs for you to
complete.
migrate create --auto labels each hop Auto or Authored. Auto is a change whose new-row
body the differ can prove on its own: dropping a field or model, a rename, adding a &unique.
Authored is one where the value can't be derived (a type re-encode, a nullable→NOT-NULL fill,
a required-add-without-default), so migrate create scaffolds migrations/<id>/transform.rs.
You fill in every TODO and freeze it. authored_transform(model, row) receives each row as
JSON after the automatic ops and returns it reshaped for the next version.
Lifecycle — migrate create → migrate up#
# 1. Edit schema.forge, then record + classify the change:
forgedb migrate create "qty to string" --auto --schema schema.forge
# → records migrations/<id>_*.json (from_version -> to_version)
# → snapshots migrations/schemas/v<n>.forge
# → for Authored residue, scaffolds migrations/<id>/transform.rs
# 2. If an authored body was scaffolded, fill in every TODO. authored_transform(model, row)
# receives each row as JSON AFTER the automatic ops, and returns it reshaped.
# 3. Regenerate your app (its EXPECTED_FORMAT_VERSION advances):
forgedb generate
# 4. With the app STOPPED, migrate the data in one step:
forgedb migrate up --from 1 --to 2 --src ./data --dest ./data-migrated
# 5. Point the regenerated app at ./data-migrated.migrate up builds the transformer crate for the range, compiles it, and runs it. It writes a
fresh destination and leaves the source untouched, so the original is your rollback.
--from defaults to the version detected from the source manifests, and --to to the
lineage's current version.
For a --from B --to G range, ForgeDB emits a self-contained crate at migrations/transform/:
one typed module per version (vN.rs, each carrying its own version open-guard), any frozen
authored bodies embedded verbatim, and a main.rs that is a fixed straight-line chain of named
transform_vN_to_vM hop functions. There is no runtime step interpreter. Each hop reads every
row through the vN typed structs, applies the baked structural ops then the authored
transform, and writes through vM's insert, which preserves record ids so foreign keys stay
valid. Multi-hop ranges replay through temp dirs and publish with a single atomic rename. The
crate depends only on your app's substrate (storage, types), never on forgedb-parser or
forgedb-migrations, and it never parses a .forge at runtime.
Per-tenant sweep#
Under multi-tenancy, each tenant is an independent data dir
under one root. migrate up sweeps them in one command:
forgedb migrate up --tenant-root ./tenants --to 2
# migrates ./tenants/<t> → ./tenants/<t>-migrated-v2 for every tenant data dirIt builds the transformer once and runs it per tenant independently. A tenant that fails, or is at an unexpected version the open-guard refuses, is reported and skipped with its source unchanged, and the command exits non-zero if any tenant failed.
Limits#
- Offline / exclusive-writer.
migrate upis offline; online (live-writer) migration is not supported. - Additive backfill uses the type zero, not
@default(deferred). compaction_epochverification before apply is deferred — the format-version guard is the interlock today.- Cheap in-place byte-op hops (drop/rename without an O(rows) typed rewrite) are deferred; the transformer does a uniform typed replay.