PYX data product agentic engineering
← Field notes·06·Migration·6 min

The migration that passes every test and changes every number

Green pipeline, different answer. Both true. Nothing in a normal stack is built to notice.

The pipeline is green. The number is different. Both are true, and nothing in a normal stack is built to notice.

I’ve watched this play out on three migrations now and it goes the same way every time. Row counts match. Schemas match. Every test passes, because every test asserts on structure. Nulls, uniqueness, referential integrity, freshness. Then someone in Finance opens the new report, the exposure figure is 8% lower than last month, and nobody can say whether that’s the migration or the business.

The reason is simple enough to be uncomfortable. Your tests assert on the shape of the data. What changed was the meaning, and no artifact in your stack was holding it. The rule lived in a WHERE clause that got rewritten by someone who didn’t know status IN (‘A’,‘P’) was a regulatory exclusion and not a filter somebody once found convenient.

Late binding is the structural fix

A metric that names a physical table has bet its meaning on that table never moving. In a migration, the table always moves. A metric that names a concept, with the binding as a separate, reviewable, dated fact, survives the move. Better: it makes the move visible. The binding changed, and the binding is an object with a diff.

That’s what turns “what breaks if I change this column?” into a question you can answer before the pull request merges, instead of during the incident review. The column isn’t an anonymous string in a repo. It’s bound to concepts, which carry claims, which carry controls, which hold a release. The blast radius is computable, so it can be refused.

Reconciliation shouldn’t be a spreadsheet that was correct on the day it was signed. It should be a control that runs tonight, and every night after.

Why this got urgent this year

Because the rate of change went up and the rate of review didn’t. When an agent can refactor forty queries in an afternoon, the traditional safeguard, a human noticing something looks off in a diff, stops scaling. It was never a strong control anyway. It was a slow one, and the slowness was doing the work.

The answer isn’t to forbid the agent. It’s to make the guardrail machine-readable, so whatever generates the change hits the same closed door: a contractor, a junior, or a model at three in the morning. An agent that quietly changes a regulated number should fail on a bound control, not on somebody’s vigilance.

The practice

Reconcile against the requirement, not the old table. Matching the legacy output reproduces the legacy bugs. The question is whether the stated rule still holds, on both sides.
Make every exclusion a named claim. If a filter is a policy, it needs a name and an owner, or the next person reads it as a convenience and deletes it.
Run impact before the merge, not after the incident. Dependents warn, holds refuse. If a change touches an asset holding a release, that’s a refusal, with a reason, in the PR, where someone can argue with it.
Keep the reconciliation running after cutover. The parallel-run period is the one time everybody’s watching. Drift shows up four months later, when nobody is. A control doesn’t stop watching.

What this replaces is migration assurance as a consulting deliverable: a beautiful reconciliation pack, correct on the day it was signed, decaying from that afternoon onward. Same rigour, same domain expertise, but generated from the model, versioned with it, and re-runnable every night by nobody at all.

Start a conversation

Have a data problem in mind?