# Build an evaluation set and scoring plan for an LLM feature

A test set and scoring approach to measure whether an AI feature's answers are correct, safe, and improving between versions.

- Topic: Working with AI
- Framework: Few-Shot
- Level: Standard
- Tags: AI & Prompting, Summarize, Plan, Fill-in Template
- Source: https://www.thinkwithprompt.com/prompts/build-an-evaluation-set-and-scoring-plan-for-an-llm-feature

## Prompt

```text
Help me build an evaluation set and scoring plan for a feature that uses a large language model.

**What the feature does:** [e.g. "answers customer questions from our help center", "summarizes support tickets", "classifies invoices"]
**Inputs and outputs:** [what goes in, what comes out, format]
**What good looks like:** [accuracy, tone, format, grounded in sources, refusals when appropriate]
**Known failure cases:** [PASTE examples of bad outputs, if any]
**Real examples available:** [e.g. "2,000 historical tickets", "none yet"]

Steps:
1. Define evaluation criteria with clear, testable definitions (e.g. "factually consistent with the provided source", "includes the order number", "does not give legal advice").
2. Design the test set: typical cases, hard cases, edge cases, adversarial or out-of-scope inputs, and cases from known failures. Suggest the size and the mix, and how to sample from real data without bias.
3. Choose scoring methods per criterion: exact match or rules for structured outputs, human ratings with a rubric, or a model-as-judge, and how to check the judge agrees with humans.
4. Write the rubric with scoring levels and examples for each level.
5. Plan how to compare versions (prompt or model changes): same test set, track pass rates per criterion, and look at regressions, not just averages.
6. Plan ongoing monitoring with samples of production outputs.

Return the criteria table, test set design with 10 example test cases, the rubric, the scoring approach, and a results-report template. Remove personal data from any real examples before using them.
```

## What to fill in

- what goes in, what comes out, format
- PASTE examples of bad outputs, if any
- e.g. "2,000 historical tickets", "none yet"

Free to copy and use. From ThinkWithPrompt (https://www.thinkwithprompt.com/), a library of 2,663 curated AI prompts.