Assertions

An assertion is a data-quality check that runs with a pipeline step. It runs a query, judges the rows against an expectation you set, and applies a policy when the judgement fails.

Or, in one sentence worth memorising because it is the distinction everything here rests on:

A condition decides what runs next. An assertion judges whether the data is right, and applies a policy.

Assertion or condition?

Both look like "a check on a pipeline". They do different jobs and live in different places.

Assertion Condition
Question it answers Is the data right? Should this step run?
Effect Warns, pauses or stops the run Chooses the path through the pipeline
Where it lives Data Tools → Assertions, attached to a step On a pipeline edge
Example "No orders should have a null customer" "Only export if the load wrote rows"

If you find yourself writing an assertion to stop a step from running, you wanted a condition.

Writing one

Data Tools → Assertions → New Assertion. The editor is the SQL editor, with two differences: there is no schedule (an assertion is never run on its own clock — it runs with the step it guards), and under the result there is an expectation.

The SQL should SELECT the rows that represent a problem, or the single row holding a number to compare. Then say what passing means:

Expectation Passes when Use it for
Returns no records The query returns zero rows "Find the broken rows" — the common case
Returns records At least min, at most max "The load should have written at least 100 rows"
A value that… The first row's named column compares eq, neq, lt, lte, gt or gte against your value A single reconciling figure

The picker sits beside the result, not above the editor, because you cannot sensibly say what passing means until you have seen what the query returns.

An assertion with no expectation set can never fail, and the list marks it so.

Attaching it to a step

An assertion does nothing until it is attached. Open the pipeline, then either double-click the edge leading into a step, or double-click the step and choose Edit gates — the second is the only route to the first step in a pipeline, which has no incoming edge.

Two choices per attachment:

Timing — before the step or after it. Before guards the input; after checks the output.

Policy — what failing does.

Policy Effect When
Warn The run continues; the warning is recorded on the step Worth knowing, not worth stopping for
Pause The run stops and waits to be resumed Somebody should look before this continues
Error The run fails The data is wrong and nothing downstream should use it

If several assertions fail on the same step, error outranks pause. They are not really a severity scale — pause means "look at this", error means "this run should not have happened" — and if one check says the run is invalid, pausing for acknowledgement understates it. Every failure is recorded either way.

When the assertion itself cannot run

Two cases, both reported rather than silently skipped:

  • Deleted. Deleting an assertion does not bump the version of every pipeline using it. The attachment stays, the gate stops firing, and every run of that step records "not checked — this assertion no longer exists".
  • Deactivated. Same, with the reason given as deactivated.

In neither case is the attachment's policy applied. A policy says what to do when the data is wrong; here nothing has been learned about the data, only that the check is gone — failing the run would punish you for the cleanup you chose. Clear the stale attachment from Edit gates on the step and the warning stops.

Editing an assertion is safe for a deployed pipeline: an attachment pins a version, so a pipeline keeps running the version it was deployed with.

A worked example

Stark Industries loads sensor readings every hour and wants to know when a line reports a fault it never clears.

  1. New assertion, named unresolved faults older than 6h:

sql SELECT line_id, reading_at, status FROM SENSOR_READINGS WHERE status = 2 AND reading_at < DATEADD(hour, -6, CURRENT_TIMESTAMP)

  1. Expectation: returns no records. The query finds the bad rows, so finding none is the healthy state.
  2. Attach it to the hourly load step, timing after, policy pause.
  3. When a fault persists past six hours the run pauses, the paused step names the assertion, and the rows it found are on the run.

Pause rather than error, because the load itself is fine — the data it landed is telling you something about the factory, and somebody should look at it before the downstream reports are built on it.