Explains rater effects and grader calibration
Halo, leniency, severity and central tendency; calibration against grades known to be correct, with exact and adjacent agreement (Myford and Wolfe 2003-2004; Standards 2014, ch. 2).
Can AI do this?
AI does this well Most of the tested AI models master this. 3 of 4 models in the current benchmark master it.
| Model | Tier | Probability of mastery | Status |
|---|---|---|---|
| gemini-4-argon | Flagship | 100% | Mastered |
| gemini-3.5-flash-lite | Low cost | 100% | Mastered |
| gpt-6-luna | Low cost | 78% | Not shown |
| claude-haiku-5-5 | Low cost | 100% | Mastered |
Mastered means a probability of at least 90% that the model has this knobit, estimated from its graded answers. “Not shown” means the answers do not settle it either way. How the models are tested.
What it looks like in practice
Expected result: Halo, leniency, severity and central tendency; calibration against grades known to be correct, with exact and adjacent agreement (Myford and Wolfe 2003-2004; Standards 2014, ch. 2).
How it is assessed: Answers a written question on the rater effects correctly
Related knobits
Needed for
- T-303 Recognises a halo in criterion scores
- T-304 Recognises length or confidence rewarded
- T-331 Scores each criterion from its own evidence in the answer
All 16 knobits of Grade answers against a rubric.
Source: Practice the stage 3 reference and review checklist rest on