Skip to content

Building evals

Introduces core eval components (code-based grading, human grading, model-based grading) and how they fit into an iteration loop.

  • Upstream notebook: misc/building_evals.ipynb

What to focus on

  • Defining metrics that match product requirements
  • Separating dataset creation from scoring logic
  • Making evals runnable in CI (fast/targeted)

Run locally

make test-notebooks NOTEBOOK=misc/building_evals.ipynb