- Data
Data lineage: why you cannot decide anything without it
Without lineage, every prioritisation decision is blind and every deletion is frightening. How to build it in three weeks, even on a heterogeneous estate.
· 4 min read · AzerOps
Data lineage is the traceability of a data item from its source to its final use. It is often framed as a governance topic, which makes it sound like a luxury. It is in fact an operational prerequisite: without it, four everyday decisions become impossible.
The four blocked decisions
Deleting a flow. No way to know what breaks. Result: nothing gets deleted.
Changing a source field. A format change in an upstream system can break seven downstream reports. Without lineage you find out the next morning, in production.
Answering a GDPR erasure request. Someone asks for their data to be deleted. Without lineage you do not know how many derived systems it was copied into.
Explaining a figure. A director disputes an amount. Without lineage the answer takes two days of investigation instead of two minutes.
The three levels, in order of cost
Table level. Which table feeds which table. The least precise and fastest to obtain: a few days analysing the flows' code is enough. That is already sufficient to unblock deletion decisions.
Column level. Which source column produces which target column, with the transformation applied. Needed for fine-grained impact analysis and for GDPR. Significantly more expensive, best reserved for sensitive domains.
Usage level. Which report, dashboard or extract consumes which table, and who actually opens it. This is the level everyone forgets and it is the one with the most value, because only it lets you say a flow is no longer used.
Our recommendation: table level across the whole estate, usage level across the whole estate, column level only on domains holding personal or financial data.
How to build it without a dedicated tool
Catalogue tools are useful but take months to roll out and assume a homogeneous estate. On a heterogeneous estate, a hand-rolled approach gives a usable result in three weeks.
- Extract the code of every flow into a text format and version it. That is valuable in itself, independently of lineage.
- Parse the read and write statements to build the table-to-table graph. A parsing script covers the majority of cases; the exceptions are handled by hand.
- Cross-reference with the reporting tool's access logs to get the usage level.
- Publish the graph in a format the teams actually consult, however simple.
The mistake to avoid
Waiting for the perfect tool. An approximate lineage, available today and corrected as you go, immediately unblocks decisions. An exhaustive lineage planned for eighteen months out unblocks nothing, and will be out of date on delivery day anyway if nobody set up the process that keeps it current.
The question that matters is not "how do I get complete lineage" but "which decision am I failing to take for lack of lineage". Answer that one, and you will know what precision level to aim for.