How do I choose the model that judges a comparison?

Set a task’s judge under Standard → Models, or an organization default under Settings → Models. A judge scores candidate answers against the rubric and cannot be one of the candidates in that comparison. The comparison records which judge it used.

Selection order

The judge is the model that scores answers against the rubric in a comparison. It is never a candidate in the comparison it judges. Which model judges is chosen in this order:

1. the comparison request names one,
2. the task names one, under its Standard → Models section,
3. the organization names one, under Settings → Models,
4. otherwise Orlo picks automatically from the models the organization has a key for.

The judge used is stored with each comparison. A second judge with an agreement threshold flags examples where the two disagree. Judge shift opens a signal when the judge’s view of the same model on the same examples moves between two comparisons.

Reviewer corrections improve the examples the judge scores against.

Compare models on your task. A judge score is evidence to review alongside the returned answers and rule results.