Migrations
Schema evolution the generator way — a version interlock, additive backfill, Auto/Authored hop classification with the answer captured at create time, and an offline transformer bin driven by forgedb migrate create → build → run.
ForgeDB is a code generator, so schema evolution follows the generate-then-compile model: you
edit schema.forge, regenerate, and recompile. What happens to the data on disk depends
on the change:
- Additive changes (a new model, a new nullable field) are preserved automatically — the regenerated app backfills them on reopen. No data step.
- Everything else (a type change, a column/model drop, a nullable→NOT-NULL narrowing, a
&uniqueadd, a required field with no default) rewrites data-at-rest. ForgeDB generates a per-version offline transformer bin that does the rewrite, driven byforgedb migrate.
The transformer is generated code, not a runtime engine
The schema is never a runtime input to a generic engine. The transformer is generated code — one straight-line typed replay per origin→destination version range — not an interpreter that reads your schema at runtime.
The version interlock#
Before touching data, a generated app checks one version number. Every app bakes in the format version its schema expects, and each data directory records the version it was written at. On a mismatch the app refuses to open rather than mis-read stale bytes. A fresh database starts at version 1, and each recorded migration bumps it by one.
The format_version is one opaque integer per data-dir manifest, derived from your migration
lineage. It is the only cross-schema handshake between a generated app and a directory: the
check never reads column names or types to "self-heal." That makes it fail-fast in both
directions. A regenerated app won't silently read a not-yet-migrated directory, and the
transformer won't apply the wrong version range to one.
Additive changes — automatic#
Adding a new model or a new nullable field needs no data step. Record the change, regenerate, restart, and the app backfills the new column on reopen. Existing rows are never rewritten.
# 1. Edit schema.forge — add the new nullable field AT THE END of the model.
# 2. Record the change (baselines the lineage on first run):
forgedb migrate create "add note field" --schema schema.forge
# 3. Regenerate and rebuild:
forgedb generate
# 4. Build and run the transformer, then point the app at the migrated dir.Two constraints. Append new fields at the end of the model, because columns are addressed
by position, not by name. And a new non-null field backfills to the type's zero value, not its
@default (so prefer nullable when the zero is not meaningful).
On reopen, generated recovery anchors on the tombstone row count (the authoritative committed count) and backfills any column shorter than that anchor. The short column is the new field; every other column already matches the anchor, so existing rows are left untouched. This is why fields must be appended at the end: a mid-model insert would shift every later column off its anchored length.
Data-rewriting changes — hop classification#
Every other change rewrites data at rest — a type change, a dropped column or model, a
nullable→NOT-NULL narrowing, a &unique add, a required field with no default. Running
forgedb migrate create records the change as a versioned hop and classifies it.
Structural changes it can rewrite on its own. For changes where it can't derive the new value (a
type re-encode that isn't a widening, or filling a new required field), it asks you, right
then — a constant, a copy of another field, or "I'll write the transform" — and records your
answer as data in the migration record.
migrate create labels each hop Auto or Authored. Auto is a change whose new-row body
ForgeDB can prove on its own: dropping a field or model, a rename you confirm, adding a
&unique, T → T?, a value-preserving widening (u32→u64, string(N)→string(M) for
M > N, timestamp(s)→timestamp(us)), or an add whose field carries a resolvable @default.
Authored is one where the value can't be derived. ForgeDB asks about it at create time and
records the answer beside the change; migrate build lowers that answer into the generated
transformer, so it is a compile-time input to code generation and never something the
transformer matches on at run time. In a session with no terminal — a CI run, a pipe, or
--no-auto — the first such change is a hard error naming it, and nothing is written.
If you choose to write the transform, it is scaffolded in the language your project already
generates for (derived from [generate].targets), typed against the generated per-version
models, and run on the interpreter you already have — ForgeDB embeds none. Rust is the advanced
escape hatch, not the default path.
Lifecycle — migrate create → build → run#
# 1. Edit schema.forge, then record + classify the change:
forgedb migrate create "qty to string" --schema schema.forge
# → records migrations/<id>_*.json (from_version -> to_version)
# → snapshots migrations/schemas/v<n>.forge
# → asks about anything it can't prove, and records your answer
# → if you chose "I'll write the transform", scaffolds
# migrations/<id>/transform.{ts,py,rs} beside ForgeDB's typed v<n> modules
# 2. If you chose the transform option, write it. transform(model, row) receives
# each row AFTER the automatic ops, and returns it reshaped for the next version.
# 3. Regenerate your app (its EXPECTED_FORMAT_VERSION advances):
forgedb generate
# 4. Build the transformer for the range, then — with the app STOPPED — run it:
forgedb migrate build --schema schema.forge --from 1 --to 2
forgedb migrate run --schema schema.forge --from 1 --to 2 \
--src ./data --dest ./data-migrated
# 5. Point the regenerated app at ./data-migrated.run writes a fresh destination and leaves the source untouched, so the original is your
rollback. build and run name the same range and the same app, which is how run finds
the exact binary build produced.
`migrate up` was removed
It was a wrapper over these two commands, so run the two. What is genuinely lost is the
per-tenant sweep and --from auto-detection; both are tracked for restoration as #373.
Passing forgedb migrate up today errors and names the pair.
For a --from B --to G range, ForgeDB emits a crate named transform-<B>-<G> into the app's
container in the build cache — a member of the project's cargo workspace, sharing its
Cargo.lock and target/, never a [package] dropped into your tree. It holds one typed
module per version (vN.rs, each carrying its own version open-guard), any frozen authored
bodies embedded verbatim, and a main.rs that is a fixed straight-line chain of named
transform_vN_to_vM hop functions. The range is in the package name because one
transform/ per app collided across ranges, and run got whichever built last.
There is no runtime step interpreter. Each hop reads every row through the vN typed
structs, applies the baked structural ops then the authored transform, and writes through
vM's insert, which preserves record ids so foreign keys stay valid. Multi-hop ranges replay through temp dirs and publish with a single atomic rename. The
crate depends only on your app's substrate (storage, types), never on forgedb-parser or
forgedb-migrations, and it never parses a .forge at runtime.
Per-tenant migration#
Under multi-tenancy, each tenant is an independent data dir under one root. Build the transformer once, then run it per tenant:
forgedb migrate build --schema schema.forge --from 1 --to 2
for t in ./tenants/*/; do
forgedb migrate run --schema schema.forge --from 1 --to 2 \
--src "$t" --dest "${t%/}-migrated-v2"
doneEach run is independent: a tenant at an unexpected version is refused by the open-guard with
its source unchanged. The single-command sweep (--tenant-root, with its skip-and-report
behavior and non-zero exit) left with migrate up and is tracked as #373.
Limits#
- Offline / exclusive-writer.
migrate runis offline; online (live-writer) migration is not supported. - Additive backfill uses the type zero, not
@default(deferred). compaction_epochverification before apply is deferred — the format-version guard is the interlock today.- Cheap in-place byte-op hops (drop/rename without an O(rows) typed rewrite) are deferred; the transformer does a uniform typed replay.