Build Data Eval Set – PM Interview Notes

How to Build an
Data Evaluation Set

Product Manager Interview Guide · Core Framework

?? TL;DR · An evaluation set is not for training — it validates real-world model performance. Build from real scenarios, expand wisely, cover edge cases, set clear criteria, and update continuously.

?? What is an Evaluation Set?

An evaluation set (eval set) is used to verify model performance in real-world scenarios, not for learning. It is much smaller than a training set — typically hundreds to 1,000–2,000 samples — because the goal is to cover key scenarios and test generalization, not large-scale learning.

?? Work with algorithm & operations teams to determine the exact size based on your business context.

            200–2,000
            eval samples
            vs. tens of thousands for training · focused on coverage, not volume
        

?? 3 Core Principles

1. Real Scenarios Collect real user queries from business teams,客服 logs, and user feedback — even a few dozen real requests are more valuable than fabricated data. ?? E.g. for an e‑commerce AI, real queries are "write a product description highlighting material and function", not "compose a poem". ?? Stuck with only ~10 samples from stakeholders? Use LLM expansion with careful prompt control — generate diverse tones and details, not just synonyms.
2. Broad Coverage Include both routine and edge cases: incomplete input, typos, out‑of‑scope requests. If your eval set only has standard queries, results will be over‑optimistic. ?? Edge case: "Write a product description for a shoe but make it sound like a poem" — test how the model handles unusual constraints.
3. Clear Standards Define what "high quality" means — grammatical correctness, logical coherence, specific keywords, etc. Align criteria with algorithm and product teams beforehand. ?? Dimensions: relevance, fluency, innovation — each with a scoring rubric.

?? Real‑World Case

?? Content Platform — Eval Set in Practice

Started with 200 real user requests from customer support, then used LLM expansion to reach 1,000 samples across diverse topics and styles.

Evaluated on 3 dimensions:

?? Relevance ?? Fluency ?? Innovation

Human raters scored each sample, creating a robust evaluation system that made product iteration more targeted and efficient.

“An evaluation set is not a one‑time artifact — it must evolve with your product features, scope, and user needs.” ?? Key insight from the field

?? Action Tips

1. Start small, expand smart. Gather 10–30 real queries, then use LLM with prompt engineering to generate 200–1,000 variants.
2. Define scoring criteria early. Align with stakeholders before running evaluations.
3. Treat your eval set as a living asset. Update it as the product grows — schedule quarterly reviews.

?? Summary · The 5‑Step Framework

?? Real Scenarios ?? Reasonable Expansion ?? Cover Boundaries ?? Clear Criteria ?? Dynamic Iteration

?? Interview tip: Lead with the logic — eval is about validation, not training. Then walk through real scenarios → coverage → standards → iteration. Use the case study to show depth.

?? A detailed capability framework & knowledge supplement for PM roles is available separately.

How to Build anData Evaluation Set

How to Build an
Data Evaluation Set