Check my machine learning model for leakage and evaluation mistakes
An audit of how a model was trained and evaluated, catching leakage, bad splits, and misleading metrics before it ships.
The prompt
Audit my machine learning model's training and evaluation setup for mistakes that make results look better than they are.
Problem and target: [what is predicted, unit, prediction time]
Data and split: [date ranges, how train/validation/test were split, row counts, class balance]
Features: [LIST, with how each is calculated]
Preprocessing: [imputation, scaling, encoding, resampling - and whether it was fit before or after the split]
Code (optional): [PASTE key parts]
Results: [metrics on each split]
Check for:
1. Target leakage: features created after the prediction time, proxies for the label, IDs or timestamps that encode the outcome.
2. Train/test contamination: preprocessing or feature selection fit on all data; duplicate or near-duplicate rows across splits; the same customer in both train and test.
3. Wrong split strategy: random splits on time-dependent data; no group split when rows are related.
4. Metric problems: accuracy on imbalanced classes, AUC when the use case needs precision at a threshold, metrics tuned on the test set, threshold chosen on test data.
5. Resampling mistakes: oversampling before splitting.
6. Suspicious results: performance that is too good, big gaps between validation and test, unstable results across folds.
7. Missing checks: comparison to a simple baseline, calibration, performance by segment.
Return a findings table (Issue | Evidence | Severity | Fix), a corrected evaluation plan, and the 3 most important things to re-run. Say clearly if you cannot judge something from what I shared.
Highlighted [brackets] are blanks to replace with your own details.
Fill in the blanks and use this promptMore Data & Analysis prompts
What to fill in
- what is predicted, unit, prediction time
- LIST, with how each is calculated
- PASTE key parts
- metrics on each split
How this prompt is built
This prompt follows the RTF structure (Role, Task, Format). RTF tells the model who to be, what to do and what shape the answer should take. It is the quickest structure that still gives noticeably better results than a bare request.
Related prompts
- Check whether my KPIs measure activity or real value, and fix themSorts your KPIs into activity, output, and outcome measures, flags vanity metrics, and redesigns the set around the outcomes that matter.
- Find where my construction jobs are losing moneyA job cost review comparing estimate to actual by cost code, with over/under billing and profit fade explained and fixes listed.
- Analyze return reasons and cut my product return rateFinds the SKUs and reasons driving costly returns, from fit to bad photos, and plans fixes with estimated savings.
- Turn an analysis into a story people act onShapes analysis into a clear narrative with one message and supporting evidence.