Migrations¶

A migration is a data step whose SQL changes the shape of something — a CREATE TABLE, an ALTER, an index. You mark a step as a migration and Data Conductor changes how it runs:

Environment	What happens
PROD	Runs once per version, then never again. Recorded on your own database.
QA and every other env	Runs every time. Nothing is tracked.

That difference is the whole feature, and it is worth being blunt about the second row: outside production your DDL re-executes on every single run, so it has to be safe to repeat. Write IF NOT EXISTS.

A query is something you run. A migration is something you have run — and the record of having run it is the point.

Marking a step as a migration¶

In the SQL editor, set the step's kind to Migration. Two things change immediately:

Execute no longer returns rows. A migration reports what it did, not what it selected.
A History tab appears, listing every run of this migration.

The first PROD run asks permission¶

To run a migration once and not again, Data Conductor has to write down that it ran. It does that in a table on your database, not ours: _data_conductor_migrations.

The first time you apply a migration to PROD on a given connector, it stops and asks — naming the exact table it wants to create, and quoting what your database said when it looked for it. Decline and nothing happens: not the table, not the migration.

After that it never asks again on that connector. The table's existence is the record of consent.

Where that table lives¶

It is created in the database and schema your connector points at, so those have to be set. On Snowflake in particular, a connection opens with no current database and no current schema unless the credential supplies them — set Default Database and Default Schema on the connector, to a schema the connector's role can create tables in.

If the role cannot reach that schema, the migration stops with an error naming the setting to change. It will not offer to create a table it cannot create.

Editing a migration makes a new version¶

A query can be edited in place. A migration cannot, and this is deliberate.

A migration's text is its identity. Once a version has been applied, that version means "this exact SQL ran here". Editing it in place would rewrite history — the record would claim something ran that never did. So saving a change to a migration's SQL forks a new version, every time, and the new version is free to be applied in its own right.

The practical shape of a change is therefore always the same:

Edit the SQL and save → version bumps.
Run the new version on PROD → it applies, because this version has not been applied before.
The previous version's record stays exactly as it was.

If you edit a migration and run the same version again, nothing happens and Data Conductor tells you why: the text recorded as applied no longer matches the text you are running.

Variables lock once applied¶

A migration resolves {{variables}} like any other step. But after a version has been applied to an environment, that environment's values are frozen for that version — the Variables grid shows them read-only, with the reason on hover.

The reason is the same one as above: the record says "this text ran against this target", and the text is a function of the variable values. Letting them change afterwards would make the record a guess.

Editing the SQL forks a version, the new version inherits the values, and they are editable again. The lock is per version, which is what makes it survivable rather than permanent.

Naming another environment's value¶

If a PROD migration's SQL spells out a value that belongs to a different environment, Data Conductor refuses it and asks before anything connects.

This catches a specific and nasty mistake. An environment here is a set of variable values on one connector — there is no separate connection to be wrong about. So a migration run with the selector on PROD, whose SQL hard-codes STAGING.…, will cheerfully alter STAGING and be recorded as a PROD apply. Nothing downstream would contradict it.

The fix is almost always to use the placeholder rather than the literal: {{DATABASE}}.{{SCHEMA}}.MY_TABLE.

History: what this migration has actually done¶

The History tab on a migration step lists every attempt, newest first, across every version and environment:

Column	Why it is there
Env	Decides whether a repeat was expected. Outside PROD, repeats are normal.
Outcome	Applied, skipped, refused, or failed — see below
Target	The database and schema it actually reached
By	You, the agent, or a schedule

Click a row for the SQL as it ran — after variable substitution, with values of sensitive-looking variable names masked.

The outcomes:

Applied — the DDL ran.
Skipped — already applied — this version was already recorded for this environment. Grey, not green: nothing ran.
Skipped — text changed — a different text is recorded as applied here. Save to fork a version.
Refused — names another env — stopped before connecting. See above.
Waiting for consent — the tracking table did not exist and creating it had not been agreed. Nothing was attempted.
Failed — the database rejected it. Expand the row for the message.

History starts from the first run after this feature shipped. Earlier applies live in _data_conductor_migrations on your own database and nowhere else.

Migrations are not pipeline steps¶

A pipeline cannot contain a migration, and attempting to save one is refused.

The reason is mechanical rather than stylistic: a pipeline run does not go through the migration engine. The DDL would execute on every run of the pipeline — nightly, silently — with none of the once-per-environment tracking you would reasonably assume you had. Most DDL is idempotent, so you would not find out until one piece of it was not.

Migrations appear in the pipeline step picker, greyed out, so you can confirm a step's kind without wondering whether it was deleted.

Run migrations from the SQL editor. Attaching them to a pipeline's lifecycle — applied once when the pipeline is activated rather than on every execution — is the right shape for the cases that need it, and is not built yet.

SQL Data Steps — the editor, variables and versions
Monitoring — the SQL Log, every statement across every step
Security — what _data_conductor_migrations holds
Pipelines — what a pipeline step can be