# Check my machine learning model for leakage and evaluation mistakes

An audit of how a model was trained and evaluated, catching leakage, bad splits, and misleading metrics before it ships.

- Topic: Data & Analysis
- Framework: RTF
- Level: Standard
- Tags: Analyze, Code, Fill-in Template
- Source: https://www.thinkwithprompt.com/prompts/check-my-machine-learning-model-for-leakage-and-evaluation-mistakes

## Prompt

```text
Audit my machine learning model's training and evaluation setup for mistakes that make results look better than they are.

**Problem and target:** [what is predicted, unit, prediction time]
**Data and split:** [date ranges, how train/validation/test were split, row counts, class balance]
**Features:** [LIST, with how each is calculated]
**Preprocessing:** [imputation, scaling, encoding, resampling - and whether it was fit before or after the split]
**Code (optional):** [PASTE key parts]
**Results:** [metrics on each split]

Check for:
1. **Target leakage:** features created after the prediction time, proxies for the label, IDs or timestamps that encode the outcome.
2. **Train/test contamination:** preprocessing or feature selection fit on all data; duplicate or near-duplicate rows across splits; the same customer in both train and test.
3. **Wrong split strategy:** random splits on time-dependent data; no group split when rows are related.
4. **Metric problems:** accuracy on imbalanced classes, AUC when the use case needs precision at a threshold, metrics tuned on the test set, threshold chosen on test data.
5. **Resampling mistakes:** oversampling before splitting.
6. **Suspicious results:** performance that is too good, big gaps between validation and test, unstable results across folds.
7. **Missing checks:** comparison to a simple baseline, calibration, performance by segment.

Return a findings table (Issue | Evidence | Severity | Fix), a corrected evaluation plan, and the 3 most important things to re-run. Say clearly if you cannot judge something from what I shared.
```

## What to fill in

- what is predicted, unit, prediction time
- LIST, with how each is calculated
- PASTE key parts
- metrics on each split

Free to copy and use. From ThinkWithPrompt (https://www.thinkwithprompt.com/), a library of 2,663 curated AI prompts.