What nine years of ERP history actually looks like

In the first meeting, someone senior always apologises for the data. “Ours is a mess, just so you know.” Two weeks later, in the readout, we get to say some version of the same thing back: yes, and not in the way you thought, and there is more usable history here than anyone in this room believed. Both halves are almost always true.

These notes are a composite of what we keep finding when we open a mid-size operator's ERP and start profiling. No single company looks exactly like this. Most look uncomfortably close.

The mess is structured

Old operational data doesn't rot randomly. It rots at the boundaries of human process, which means the damage follows patterns you can work with:

Fields change meaning on a date. A “downtime reason” column is trustworthy from 2019 onward, because that's when a plant manager started enforcing it. Before that, one code absorbed everything. The history didn't get worse; it has a birthday, and the profiling finds it.

The catch-all code is doing crimes. Every schema has one: “MISC”, “OTHER”, “999”. It is usually between a fifth and half of the rows, and inside it live the exact events you most want to model. What it contains is recoverable, but from free-text comments, not from the code.

Overrides are invisible. The planner who corrects the system every morning leaves no trace unless the system was built to record it, and it never was. The most important operational signal of all, “a human disagreed here,” has to be reconstructed or, more often, instrumented from the day the new system goes live.

Something upstream is silently stale. A supplier feed, a rate table, a calendar of line speeds that was true four years ago. Nobody noticed because the numbers it feeds still look plausible. Profiling finds these by accident and pays for the diagnostic on its own. We have watched a stale feed explain exceptions an operation had been shrugging at for months.

The data is never a swamp. It's an archaeological site, and the strata have dates.

What survives is enough

Here is the part the apology gets wrong. After the profiling, after the birthday of every field is known and the catch-all is mined and the stale feeds are flagged, what remains is almost always enough to build on: years of order lines, deliveries, downtime events, and quality checks whose core columns (what, when, how much) were kept honest by the accounting system riding on top of them. Money columns get audited. The operational truth clinging to them inherits that honesty.

This is why we rank candidate systems only after the data work, not before. The ranking isn't “what would be nice to have.” It's “what does this specific history, with these specific birthdays and gaps, actually support?” Sometimes the answer kills the project everyone wanted, and funds one nobody had thought to want. A forecast, it turns out, doesn't need the downtime codes to be clean. It needs the order lines, and those are fine.

If you’re about to start

Three things we would tell any operator before their first build, vendor regardless. Profile before you rank: a week of data archaeology reorders every priority list we have ever seen. Date your fields: ask “since when is this column trustworthy?” and write the answers down, because that document outlives any one project. And start logging disagreement now: every place a human corrects a system today is training data and trust measurement tomorrow, and it costs a column.

The apology in the first meeting is always sincere. It's also the wrong worry. The data is a mess in knowable ways, and knowable is the only kind of mess engineering can do something about.