The Forensic Gap

Mar 23, 2026

I first wrote about the ingestion problem on the Accio blog. This is the longer version. The one where I say the things that are harder to say in a corporate context.

There is a gap in financial data infrastructure that nobody talks about. Not because it is a secret. Because it is embarrassing.

The gap sits between the moment data enters your system and the moment someone discovers it is wrong. I call it the forensic gap. In some institutions, that gap is hours. In others, it is weeks. In a few cases I have seen, it is quarters. And every hour that gap stays open, the cost compounds.

Here is the pattern. A data vendor sends a feed. The feed has an error. Maybe a null character embedded in a numeric field. Maybe an encoding shift that broke a delimiter. Maybe a schema change that nobody communicated. The ETL loads the feed. It does not validate. It does not flag. It does not remember what the feed looked like yesterday. It loads and moves on.

The downstream system receives the data. It calculates on whatever it was given. A corrupted price becomes a corrupted return calculation becomes a corrupted dashboard becomes a corrupted client report becomes a credibility problem. This is the cascade effect. One bad data point at the front door propagates through every system in the building.

Gartner estimates that poor data quality costs organizations an average of $12.9 million annually. Industry analyses suggest the revenue impact ranges from 15 to 25 percent. In financial services, where data errors can cascade into regulatory filings and client reports, these numbers are conservative.

But the dollar figure is not the real cost. The real cost is what I call the Manual Tax.

The Manual Tax is the operational burden of running financial systems on uninspected data. Your most senior analysts are not doing analysis. They are doing forensics. They are validating. Reconciling. Tracing errors backwards through systems that were never designed to tell them what happened. A CrowdFlower study widely cited by Forbes found that data professionals spend roughly 80% of their time on data preparation and cleaning. In financial services, I have watched this number play out in real time. Six days to trace one pricing error. Six days of senior analysts doing work that a properly instrumented ingestion layer would have caught in seconds.

And here is the part that should worry every board member currently asking about AI: AI needs clean data. Not mostly clean. Not probably clean. Clean. If you train a predictive model on data that was never validated at the point of entry, you are not getting AI. You are getting automated bad decisions at scale. The model will do exactly what it was trained to do. If it was trained on garbage, it will predict garbage with extraordinary confidence.

I would personally want 100% clean data going into any AI engine. I know the industry will settle for less. But here is the question nobody is asking: what percentage of your data is clean? Not a guess. An actual measurement. Most institutions cannot answer that question because their ingestion layer does not measure it. It loads.

The forensic gap is not a technology problem. It is a design problem. The systems that ingest financial data were built to move data, not to understand it. ETL was built to extract, transform, and load. Not to interrogate. Not to validate. Not to remember. The ingestion layer was treated as a rigid pipe. And rigid pipes produce rigid data.

The fix is not a better ETL. The fix is rethinking what happens at the point of entry. Validation. Schema checking. Business rule enforcement. Threshold detection. Lineage logging. Correction engines. Deterministic memory that carries institutional knowledge forward instead of losing it every time the batch runs.

When the ingestion layer is instrumented to catch problems at the door, the forensic gap closes. The Manual Tax drops. The cascade effect stops. And the data that reaches your analytics engines, your AI models, and your client reports is data you can actually trust.

I have been building these systems for twenty years. The pattern has not changed. But the cost of ignoring it has.

The Forensic Gap

Ready for more?