Data access — from database restore to analysable reality
The question was clear, the data was not. Months went into making source systems readable that were never designed for analysis.
After the question was framed, the first hard wall appeared: the source. An operational system from the early days, designed to pass transactions and nothing more. No export layer, no analytical model, no watertight referential integrity. It worked fine for daily operations. For a control agent it was unreadable.
What was needed
We needed a controlled copy we were allowed to query without touching operations. A restore of the source system, refreshed every night, read-only. And on top of it a thin semantic layer translating raw tables into concepts a controller recognises: contract, invoice line, attendance, rate card.
That sounds simple. It took months.
What we ran into
- Missing keys. Contracts referenced customers by a text field, not by an ID. Every name variation was effectively a different customer.
- Implicit rules. Some contracts had no end date in the system — the end date lived in a PDF only the internal domain expert knew.
- Historical mutations. Price agreements were overwritten instead of historised. To reconstruct the tariff at time of invoicing, we had to rebuild the rate card from audit logs.
- Calendar conflicts. Attendance lived in another system with another time zone. Without normalisation every night-shift contract appeared to run an hour too long or too short.
Every finding became a rule in the semantic layer. Not in the model — in code. Reproducible, testable, and visible to anyone reading the answer later.
The real lesson
The biggest resistance was not building agents. It was accepting that without a clean, historised data foundation you never get a reliable answer — no matter how good the model looks. The work at the source feels invisible. Without it, every later answer is a guess.
THE LESSON A model is never better than the data it receives. The invisible work at the source is half the platform.