Explains validity threats in test items
Construct under-representation and construct-irrelevant variance: knobits never exercised, cues in the prompt, points for format or for the reference's wording (Messick 1989; Standards 2014, ch. 1).
Can AI do this?
AI does this well Most of the tested AI models master this. 3 of 4 models in the current benchmark master it.
| Model | Tier | Probability of mastery | Status |
|---|---|---|---|
| gemini-4-argon | Flagship | 100% | Mastered |
| gemini-3.5-flash-lite | Low cost | 93% | Not shown |
| gpt-6-luna | Low cost | 100% | Mastered |
| claude-haiku-5-5 | Low cost | 100% | Mastered |
Mastered means a probability of at least 90% that the model has this knobit, estimated from its graded answers. “Not shown” means the answers do not settle it either way. How the models are tested.
What it looks like in practice
Expected result: Construct under-representation and construct-irrelevant variance: knobits never exercised, cues in the prompt, points for format or for the reference's wording (Messick 1989; Standards 2014, ch. 1).
How it is assessed: Answers a written question on the validity threats correctly
Related knobits
Needed for
- T-204 Recognises an answer given away in the prompt
- T-207 Recognises points out of proportion to the construct
- T-208 Recognises a criterion crediting only the reference's solution
- T-210 Recognises a knobit claimed with no seeded case
- T-232 Seeds a case for each claimed knobit in the task material
All 15 knobits of Write and review test items.
Source: Practice the stage 2 reference and review checklist rest on