World Model Readiness Models decay the day they ship

Module · Models decay the day they ship

The Model Risk Management Check

A model that passed its demo can quietly rot in production as the world it learned from moves on. The failure is rarely dramatic: accuracy slips a few points a quarter until the recommendations are wrong more often than right, and nobody notices because nobody is watching. This module checks the five disciplines that keep a live model honest: validation before launch, drift monitoring, performance thresholds with alerts, retraining, and knowing which models you even run.

Question 1 of 5 · Validated before launch

Before a model goes live, does anyone independent check that it actually works?

A model that scores well on the data it was built with can still fail on the cases that matter. Independent pre-deployment validation, on data the builder did not touch, is the difference between a tested system and a hopeful one. The team that trained it is the worst-placed to judge it.

Question 2 of 5 · Drift is watched

Would you notice if a live model slowly got worse over the next six months?

Models degrade as the world drifts away from their training data, and the decline is usually gradual enough to miss. Without monitoring on inputs and outputs, the first signal you get is a customer complaint or a bad quarter. Silence is not the same as everything being fine.

Question 3 of 5 · Thresholds trigger alerts

Is there a performance line a model can cross that actually triggers action?

Monitoring only helps if a number crossing a line does something. A defined threshold, with an owner and an alert, turns 'the model is slipping' into 'someone is now looking at it'. A dashboard nobody watches is monitoring in name only.

Question 4 of 5 · Retraining is deliberate

When a model needs refreshing, does that happen by decision or by accident?

Retraining is where model risk is renewed or reduced. Done on a schedule and revalidated, it keeps the model current; done ad hoc under pressure, it can ship an untested model straight into production. A retrain is a new model and deserves the same gate as the first one.

Question 5 of 5 · You know your models

Could you produce a list of every model making decisions in your company today?

You cannot manage risk in models you cannot name. Between bought tools, embedded features and things a team built quietly, the real inventory is usually longer than anyone expects. The models nobody listed are exactly the ones nobody is monitoring.

For the statistics · one click each

Three questions for the public picture

These do not affect your score. They feed the anonymised, aggregated statistics; groups under 8 respondents are never shown.

How many models are making or shaping decisions in your company right now?

None yet
A handful
Dozens
More than we can easily count
We do not know

Do you monitor live models for performance decline?

No monitoring
Manual checks
Some models monitored
Automated across the board
No models in production

Has a model degrading in production already caused a problem for you?

Not that we know of
We suspect so
Yes, minor
Yes, serious
No models in production

Your context

Used to calibrate the report. Company size and sector remain in the anonymized dataset; your email does not.