Data Quality: Data Cleaning

Back in my Data Quality days, I once told a manager of mine that our whole program had quietly turned into a laundry machine. He laughed, because he knew I was not entirely joking. We were washing data, night after night, rarely asking ourselves whether the clean data actually fit the business.

Data Quality has been living happily inside IT, where it was mostly data testing. We profiled the sources, wrote validation rules, and ran them against the data to catch the breaks, null values, broken formats, but mostly focused on completeness. Then Data Governance arrived, with Data Quality as one of its primary drivers and a seat at last much closer to the business.

At one of the most respected firms I have known, I asked about the business rules that drive our Data Quality. What came back was a long roster of null checks, format checks, length checks, and date checks.

Where We Quietly Placed the Burden

Two beliefs sit underneath all of this, and we rarely say them aloud.

The first is about ownership. When I talk with executives, most of them picture Data Quality as something the data engineering team handles. The pipelines carry the data, so the thinking goes that whoever builds those pipelines must also be the one keeping the data clean and watching it day to day. I understand why it feels that way. Engineers live nearest to the data as it moves, and they are the first to notice when something breaks. So the responsibility drifts toward them and quietly stays.

The second belief is about the work itself. Because Data Quality sits with technology, most of it gets done from the profiling side. We measure completeness, validity, and conformity, the dimensions a profile can read straight off the data. What rarely gets the same attention is the harder ground. Few teams spend time catching anomalies. Business accuracy goes unchecked, because technology was never given the business meaning to check it against. Reference data consistency is rarely tested. A dataset that drifts from its usual profile, a quiet shift no rule was written for, goes unwatched. We take in the day’s data, clean what the profile catches, and call it quality.

None of this came from bad intent. It was the path of least resistance. The data looked clean each morning, and that was usually enough. The trouble is that a clean number is not always a true one. A figure can pass every technical check we own and still tell the business the wrong story. That gap is where the real work of Data Quality waits.

Beyond Clean, Toward Fit for Purpose

Look closely at the tens of thousands of rules we maintain, and a pattern shows itself. In a modern practice, all technical rules based on metadata profiling should be highly automated, with machine learning reading the metadata and bringing data exceptions to the surface. Those exceptions should be assigned, prioritized, and remediated through a workflow that runs with little human effort. None of this should be manual anymore. We need to preserve our resources, our time, money, and effort, for the real task ahead: building AI-ready data.

With the technical checks handled at minimal effort, we partner with the business and inspect data through the lens of data fit for business purpose. The catalog of reports and models lets us carry data concerns and exceptions all the way to these critical endpoints, so a check on a dataset becomes a check on the quality of the endpoint that depends on it. The lineage in the catalog should point us to where an issue was first detected and where it originated, and everyone responsible along that path is notified. That is how we should work, with traceable authoritative sources, reference data, lineage, and endpoints holding together as one collaborative Data Quality chain.

Timeliness deserves its own moment, because it is not easy to measure. A dataset can pass its completeness and accuracy rules and still breach its data freshness SLA. A regulatory report landed after the filing window has already failed. A risk exposure figure refreshed on a nightly batch cannot serve a decision that moves intraday. Latency, refresh frequency, and arrival time are quality dimensions in their own right, yet they rarely carry a rule or a threshold. When data lands past the point of consumption, its quality is already compromised.

The Unstructured Data We Always Set Aside

For years we ran Data Quality on structured data alone and left the unstructured world outside its scope. AI-readiness changes that. The moment an AI use case reaches into a document, that document falls under Data Quality, and the line we draw between structured and unstructured stops making sense. What we need now is one unified practice, where a contract is held to the same standard of trust as a table, and quality is checked across both before any AI agent acts on either.

The same wave of AI that floods us with data also hands us the means to govern it. Language models can read a document and weigh its meaning, its completeness, its consistency with what we already hold. The check we once called impossible is now within reach, and the stack to push Data Quality into unstructured data has arrived. An AI use case runs on data, and a growing share of it is unstructured. We talk a great deal about AI-ready data, yet data is ready only when its holistic quality can be trusted. There is no AI-ready data without unified Data Quality, the kind that checks structured and unstructured data alike.

The Quality AI Will Inherit

We have to change how we handle Data Quality. The function should be owned by business leaders, in true partnership with the technology teams who run the checks. AI-ready data will define the quality dimensions differently, and it will scan across structured and unstructured data alike. Issue management has to be built into the solution from the start. A check means little until we know who fixes the problem, by when, and where.

We are no longer just storing data for people to read. We are handing it to AI agents who will read it, reason over it, and act on it at a speed and scale no team could match. The explosion of data and the pull toward real time consumption leave the business with one real need, to trust the data it runs on.

Data Quality is no longer a laundry room where we wash yesterday’s data.

So here is the question I would leave you with. When your AI agents come looking for data they can trust, will they find quality you built, or only data you cleaned?