Diagnose why a model is performing worse in production
A step-by-step diagnosis separating data pipeline bugs, input drift, concept drift, and label issues, with a recovery plan.
The prompt
Help me diagnose why my model is performing worse in production than it did in testing or than it used to.
Model and use: [what it predicts, how outputs are used]
What I'm seeing: [e.g. "precision dropped from 0.62 to 0.41 over 6 weeks", "complaints from users"]
Monitoring data available: [PASTE input feature stats over time, prediction distribution, actual outcomes, error by segment]
Recent changes: [code releases, upstream data changes, product or policy changes, market events, retraining]
Work through, in order:
1. Pipeline and serving bugs: training-serving skew (features computed differently in production), missing or default-filled features, schema or unit changes, stale feature store, wrong model version deployed.
2. Label issues: are outcomes delayed, defined differently, or incomplete for recent periods, making performance look worse than it is?
3. Data drift: which input features changed distribution, and in which segments? Suggest measures (e.g. population stability index) and thresholds.
4. Concept drift: has the relationship between inputs and outcome changed (new customer behavior, pricing change, seasonality not seen in training)?
5. Feedback loops: is the model's own use changing the data it sees?
Return:
- A diagnostic checklist with the query or check for each step.
- Ranked hypotheses based on my evidence.
- Short-term mitigation options (rollback, threshold adjustment, fallback rules) and longer-term fixes (retraining window, new features, monitoring).
- Monitoring alerts I should add so this is caught earlier next time.
Highlighted [brackets] are blanks to replace with your own details.
Fill in the blanks and use this promptMore Data & Analysis prompts
What to fill in
- what it predicts, how outputs are used
How this prompt is built
This prompt follows the Few-Shot structure (Few-shot prompting). A few-shot prompt includes one or more worked examples of the input and the output you want, so the model can copy the pattern instead of guessing it.
Related prompts
- Identify the leading and lagging indicators for my business or teamSorts your metrics into leading and lagging indicators, links each lead to the outcome it predicts, and fills gaps in early-warning measures.
- Check whether my data shows causation or just correlationSeparates trend, correlation, and possible causation in your analysis and gives you the strongest claim you can honestly make.
- Group students for intervention from screening and assessment dataSorts students into MTSS or RTI tiers and intervention groups from universal screening and diagnostic results.
- Diagnose where our college admissions funnel is losing studentsAnalyzes inquiry-to-enrollment conversion, yield, and summer melt to find where prospective students drop off.