PYX data product agentic engineering
← Field notes·02·Modelling·6 min

Grain is the whole argument, and nobody schedules it

The one decision that determines every number downstream gets made in silence, by whoever types the first GROUP BY.

The one decision that determines every number downstream usually gets made in silence, by whoever types the first GROUP BY.

Ask a utility how many customers are affected by estimated reads. One sentence. At least five defensible answers, because “customer” isn’t a grain. Do you mean the business partner, who might hold nine accounts? The contract account, which is how the invoice runs? The premise, which is how field ops dispatch? The meter, which is the thing that actually failed to read? The installation, which is what the billing engine thinks in?

Every one of those is right. None of them are close to each other. I’ve sat in a room while competent people argued for an hour about a variance between two dashboards that were both correct, and the whole variance was that one counted meters and the other counted contract accounts.

Why it never gets decided on purpose

Because at the moment grain gets fixed, it doesn’t feel like a decision. It feels like plumbing. An engineer needs a key to join on, picks the one that makes the join work, moves on. There’s no artifact anywhere recording that as a choice, so later there’s nothing to disagree with. Only a number. And arguing about a number is a much worse conversation than arguing about a choice.

The traditional fix is a modelling workshop, and it fails for a boring reason: the people who know the answer can’t see the consequence, and the people who can see the consequence don’t know the answer. Finance knows the invoice runs at contract account. They can’t tell you that choosing it drops 22% of the affected meters out of the count.

A model can’t tell you what the grain should be. It can tell you, in seconds, what each candidate would do to the number. That’s the thing that’s always been missing.

Enumeration, not judgement

Give a model a requirement and an estate and it’s very good at listing the grains that sentence could sit at, naming the fan-out between them, and stating what each choice does to the measure. It’s bad, and should be treated as bad, at knowing that your regulator counts affected customers at the business-partner level because that’s what the licence condition says. That isn’t in your schema. It’s in someone’s head, and the job is getting it onto the artifact before the code exists.

The practice

Make grain a stated object. It belongs on the requirement, answered as a blocking question, before anything binds to a table. If it isn’t written down, it wasn’t decided. It was defaulted.
Ask for the exclusions in the same breath. Grain and exclusions are one decision wearing two hats. “Contract account” isn’t an answer until you’ve also said what happens to executed move-outs and flat-rate accounts.
Gate the join. When two concepts meet at different grains, somebody states the path and the cardinality they expect, and that statement blocks the save. Most silent double-counting in a warehouse is an unstated many-to-many nobody was ever asked about.
Keep the concept above the table. A measure should name a concept, not a physical relation. Then grain survives the table moving, and in a migration, the table always moves.

The rule I teach on day one: no measure without a stated grain and a stated exclusion set. It sounds like pedantry for about a week. Then the first reconciliation lands, two teams are 40% apart, both can prove they’re right, and everybody suddenly understands why the rule exists.

Grain hasn’t started mattering more than it used to. What’s changed is the price. The argument used to cost a workshop and a fortnight, so teams skipped it and paid later. Now it costs a page of proposals and an afternoon of disagreement, held while changing your mind is still free.

Start a conversation

Have a data problem in mind?