Most AI teams still retrain models on a schedule like monthly, quarterly, or \u201cwhenever the pipeline is ready.\u201d It feels disciplined and predictable, but it often misses the real issue: models don\u2019t degrade on calendars. They degrade when the world changes.<\/p>\n\n\n\n
Models don\u2019t expire but they drift<\/p>\n\n\n\n
Once deployed, models operate in a constantly shifting environment. Users change behavior, products evolve, and upstream data pipelines move.<\/p>\n\n\n\n
What looks like \u201cmodel decay\u201d is usually one (or more) of the following:<\/p>\n\n\n\n
– Data drift: input distributions change
– Concept drift: relationships between inputs and outputs change
– Label drift: definitions of ground truth shift
– Business drift: the system being optimized changes<\/p>\n\n\n\n
The result: a model can look stable for months and then fail quickly when conditions shift.<\/p>\n\n\n\n
Why scheduled retraining persists<\/p>\n\n\n\n
Calendar-based retraining survives because it is simple:<\/p>\n\n\n\n
– Easy to communicate and plan
– Convenient for audits and compliance
– Reduces operational ambiguity<\/p>\n\n\n\n
But it replaces a hard question with a weak assumption: that time is a good proxy for model health.<\/p>\n\n\n\n
A better approach: retrain based on signals<\/p>\n\n\n\n
Retraining should be triggered by evidence, not time.<\/p>\n\n\n\n
1. Performance signals
– Accuracy or AUC decline
– Segment-level degradation
– Calibration drift<\/p>\n\n\n\n
2. Data signals
– Feature distribution shifts (PSI, KL divergence)
– New or rare categories appearing
– Changes in missingness patterns<\/p>\n\n\n\n
3. Business signals
– KPI shifts (conversion, churn, fraud loss)
– Product or policy changes
– New user behavior patterns<\/p>\n\n\n\n
4. Human feedback signals
– Rising override or correction rates
– Increased escalations
– Growing disagreement with experts<\/p>\n\n\n\n
When NOT to retrain<\/p>\n\n\n\n
Avoid retraining when:
– Changes are small and within expected variance
– Shifts are seasonal or temporary
– The issue is data quality, not model performance
– Evaluation data is stale or unrepresentative
– Drift has no meaningful business impact<\/p>\n\n\n\n
Not all drift matters.<\/p>\n\n\n\n
Choose the right intervention
\u2022 Recalibrate \u2192 probabilities are off, model still ranks well
\u2022 Fine-tune \u2192 moderate domain shift
\u2022 Retrain \u2192 fundamental distribution change
\u2022 Do nothing \u2192 noise or non-impactful drift
The shift in mindset
Mature MLOps moves from scheduled retraining to continuous evaluation:
\u2022 Monitor performance and drift continuously
\u2022 Tie alerts to business impact
\u2022 Use shadow deployments and A\/B tests
\u2022 Version data and models rigorously<\/p>\n\n\n\n
Retraining becomes a response and not a routine.<\/p>\n\n\n\n
The real question isn\u2019t \u201cWhen do we retrain?\u201d<\/p>\n\n\n\n
It is \u201cWhat changed enough that we can no longer trust the model?\u201d<\/p>","protected":false},"excerpt":{"rendered":"
Most AI teams still retrain models on a schedule like monthly, quarterly, or \u201cwhenever the pipeline is ready.\u201d It feels disciplined and predictable, but it often misses the real issue: models don\u2019t degrade on calendars. They degrade when the world changes. Models don\u2019t expire but they drift Once deployed, models operate in a constantly shifting […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"content-type":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-1009","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\n