Recognises a quality measurement labelled with a characteristic it does not test
A syntactic check reported as accuracy, or any label the measure does not test.
Can AI do this?
People still lead Few or none of the tested AI models showed they master this. 1 of 4 models in the current benchmark master it.
| Model | Tier | Probability of mastery | Status |
|---|---|---|---|
| gemini-4-argon | Flagship | 90% | Mastered |
| gemini-3.5-flash-lite | Low cost | 10% | Not shown |
| gpt-6-luna | Low cost | 10% | Not mastered |
| claude-haiku-5-5 | Low cost | 10% | Not mastered |
Mastered means a probability of at least 90% that the model has this knobit, estimated from its graded answers. “Not shown” means the answers do not settle it either way. How the models are tested.
What it looks like in practice
Expected result: The characteristic the measure tests, and semantic accuracy measured separately against a reference
How it is assessed: Flags the deficient case and passes the acceptable one. Deficient: QM-002 "accuracy 100%" is DQ-014 (not null and >= 0) Acceptable: Syntactic accuracy 100%; semantic accuracy against the repayment schedule reported separately
Also known as: Dimension that does not match what was measured
Related knobits
Builds on
All 10 knobits of Profile and assess data quality.
Source: Review checklist 04-profiling-assessment, row 3 (25012 (syntactic vs semantic accuracy))