Migrating a Talend estate: what to decide before writing a line

A Talend estate of several hundred jobs migrates in six to eighteen months. The technical part is not the hardest. The hard part is deciding what deserves to be migrated, and getting that decision accepted.

The first question is not technical

Before choosing dbt, Spark or a managed ETL, you have to answer a governance question: who is entitled to decide that a flow will not be migrated?

Without that answer, the migration becomes a like-for-like transposition. Yet in a legacy estate a significant share of flows has no consumer left. They run every night, produce tables nobody queries, consume a processing window and create dependencies that will block the migration.

Migrating them costs several months of work for no benefit.

Inventory first

Three weeks are usually enough to produce the real inventory: the list of jobs, their frequency, their dependencies, and above all the lineage down to the reports actually consulted.

Lineage is the key point. Until you know which dashboard depends on which flow, nobody will dare delete anything, and the estate can only grow.

To find the real consumers, the most reliable method is not to ask the teams — out of caution they answer that everything is useful — but to analyse the reporting tool's access logs over a rolling twelve months. A report opened zero times in a year has no consumer, whatever the org chart says.

Decide before building

At the end of the inventory, classify each flow into one of four categories:

• Delete. No identified consumer over twelve months. The cheapest gain available.
• Migrate as is. The flow is useful and its logic is sound.
• Rebuild. The flow is useful but its logic has drifted, or it duplicates another flow.
• Defer. The flow is useful but its consumer will disappear within the year.

Have that classification validated in writing by a named person. That validation is what protects you when someone asks, eight months later, for a deleted flow.

The semantic layer solves an older problem

The migration is the opportunity to address the cause of diverging figures between reports. It is almost never a bug: it is two different business definitions of the same metric, each correct in its original context.

Use the migration to have each definition settled by financial control and written into the semantic layer. It is thankless work that produces more perceived value than the technical migration itself.

Parallel run is not optional

Old and new pipelines run in parallel over the same window, with a control job comparing row by row and aggregate by aggregate. Any gap blocks cutover until it is explained.

Expect four to eight weeks of parallel run depending on criticality. It is the only method that gives a real guarantee, and it is also what makes cutover acceptable to the business teams.

The step everyone skips

Decommissioning. Formally switching off the old jobs, archiving, releasing licences, removing access rights. Without that step you pay for two platforms for three years and the "legacy" estate quietly keeps running.

Write decommissioning into the project's contractual scope, with a payment milestone attached. It is the only reliable way to make sure it happens.