Retail · Data Engineering
Migrating 340 Talend jobs to Snowflake by first deleting the ones nobody read
A 340-job Talend estate, a rising licence bill and dashboards contested at board level. The audit showed that 40% of the flows had no consumer at all.
- Sector
- Retail
- Size
- 220 stores, 3,400 employees
- Service
- Data Engineering
- Model
- Audit then fixed price
- Duration
- 16 weeks
Context
The BI estate fed sales, stock and payroll reporting. Figures diverged between two dashboards supposed to say the same thing, which consumed an entire monthly meeting.
The problem
- 340 Talend jobs, an unknown share of them with no identified owner
- No lineage: impossible to say which report depended on which flow
- Metrics defined differently across reports
- Nightly loads regularly overrunning the store-opening window
- Rising licence cost with no usage review for four years
What we did
- Three-week estate audit: reading the jobs, not the documentation
- Tracing real lineage down to consumed reports through access analysis
- Identification of 137 flows with no consumer at all over twelve months
- Arbitration decision: delete before migrating
- Rebuild of the retained flows in dbt on Snowflake, with freshness and uniqueness tests
- Documented semantic layer: one definition per metric, validated by financial control
- Six-week parallel run with automated reconciliation before cutover
- Formal decommissioning and licence release
Results
- 137
- flows deleted, not migrated
- -58 %
- load time reduction
- 0
- reconciliation gaps
- 16 sem.
- from first line to shutdown
What remains
The nightly load window went from seven hours to under three. The monthly figure-arbitration meeting disappeared from the agenda because every metric now has one written definition. The biggest gain was not technical: it was deleting 40% of the estate before migrating it.
« They started by convincing us to delete a third of the project we wanted to buy from them. That is what made up our minds. »
Stack
- Talend
- dbt
- Snowflake
- Airflow
- Power BI