ThinkWithPrompt by SynotticThinkWithPrompt

Check my machine learning model for leakage and evaluation mistakes

An audit of how a model was trained and evaluated, catching leakage, bad splits, and misleading metrics before it ships.

Data & AnalysisRTFStandardAnalyzeCodeFill-in Template

The prompt

Audit my machine learning model's training and evaluation setup for mistakes that make results look better than they are. Problem and target: [what is predicted, unit, prediction time] Data and split: [date ranges, how train/validation/test were split, row counts, class balance] Features: [LIST, with how each is calculated] Preprocessing: [imputation, scaling, encoding, resampling - and whether it was fit before or after the split] Code (optional): [PASTE key parts] Results: [metrics on each split] Check for: 1. Target leakage: features created after the prediction time, proxies for the label, IDs or timestamps that encode the outcome. 2. Train/test contamination: preprocessing or feature selection fit on all data; duplicate or near-duplicate rows across splits; the same customer in both train and test. 3. Wrong split strategy: random splits on time-dependent data; no group split when rows are related. 4. Metric problems: accuracy on imbalanced classes, AUC when the use case needs precision at a threshold, metrics tuned on the test set, threshold chosen on test data. 5. Resampling mistakes: oversampling before splitting. 6. Suspicious results: performance that is too good, big gaps between validation and test, unstable results across folds. 7. Missing checks: comparison to a simple baseline, calibration, performance by segment. Return a findings table (Issue | Evidence | Severity | Fix), a corrected evaluation plan, and the 3 most important things to re-run. Say clearly if you cannot judge something from what I shared.

Highlighted [brackets] are blanks to replace with your own details.

Fill in the blanks and use this promptMore Data & Analysis prompts

What to fill in

  • what is predicted, unit, prediction time
  • LIST, with how each is calculated
  • PASTE key parts
  • metrics on each split

How this prompt is built

This prompt follows the RTF structure (Role, Task, Format). RTF tells the model who to be, what to do and what shape the answer should take. It is the quickest structure that still gives noticeably better results than a bare request.

Related prompts

Find more prompts like this