Model drift: the invisible cost of an unmonitored model

A deployed model describes a reality at a given moment. That reality changes: buying habits evolve, a competitor enters the market, an upstream system changes a format. The model does not change. The gap between the two is called drift, and it is silent by nature.

The two forms of drift

Data drift. The distribution of input variables changes. A new sales channel brings a different customer base, a product category disappears. The model receives data that no longer resembles its training set.

Concept drift. The relationship between inputs and output changes, even if the inputs look similar. This is the more dangerous one, because no monitoring of inputs alone will detect it.

What to measure

Three levels, from cheapest to most useful.

Inputs. Distribution of each important variable, compared against a reference window. A simple statistical test is enough; the exactness of the method matters less than the regularity of the measurement. Available immediately, without waiting for the actual outcome.

Outputs. Distribution of predictions. If a model that classified 8% of files as high risk suddenly classifies 22%, something is happening, even before ground truth is known.

Real performance. The only measure that counts, but it arrives with the observation lag: for a three-month forecast, truth arrives three months later. That is why the first two levels exist — they are leading indicators.

The point everyone misses: ground truth

Many systems predict without ever recording what actually happened. The model predicts, a decision is taken, and the outcome is stored nowhere usable.

Without that collection it is impossible to measure real performance and impossible to retrain properly. Setting it up is part of industrialisation, not a later phase. If it is missing, it is the first piece of work, before monitoring itself.

The special case of manual overrides

When a user corrects a prediction, that correction is an extremely valuable signal: it marks a case where the model is wrong, identified by someone who knows the business.

Systematically log manual overrides, with their reason where possible. A rise in the override rate is often the first sign of drift, well before a statistical metric moves.

Who receives the alert

A drift alert must reach a named person, with a runbook saying what to do: check this chart, compare this distribution, trigger a retrain or escalate.

An alert sent to a generic distribution list is ignored within three weeks. It is the most common failure mode of monitoring, and it is purely organisational.

A sensible cadence

For most business use cases: daily input and output monitoring, monthly real-performance measurement, quarterly human review, retraining triggered by threshold rather than by calendar.

Retraining monthly on principle consumes resources unnecessarily and introduces risk at every cycle. Retraining when drift crosses a threshold is cheaper and safer.