Recognises a service level measure unable to detect the element's failure
A critical element scored on a measure its known failure passes.
Can AI do this?
AI does this partly Some of the tested AI models master this; others do not show it. 2 of 4 models in the current benchmark master it.
| Model | Tier | Probability of mastery | Status |
|---|---|---|---|
| gemini-4-argon | Flagship | 90% | Mastered |
| gemini-3.5-flash-lite | Low cost | 90% | Not shown |
| gpt-6-luna | Low cost | 90% | Mastered |
| claude-haiku-5-5 | Low cost | 89% | Not shown |
Mastered means a probability of at least 90% that the model has this knobit, estimated from its graded answers. “Not shown” means the answers do not settle it either way. How the models are tested.
What it looks like in practice
Expected result: A measure the failure mode fails, such as accuracy against a reference
How it is assessed: Flags the deficient case and passes the acceptable one. Deficient: SL-006 unlikely_to_pay completeness 100%; MP-312 defaults it to false Acceptable: Accuracy against collections and forbearance records
Also known as: Measure that cannot detect the element's failure
Related knobits
Builds on
- Q-451 Explains the ISO/IEC 25012 data quality characteristics
- Q-751 Explains data quality service levels
All 10 knobits of Set data quality service levels and the scorecard.
Source: Review checklist 07-sla-scorecard, row 5 (25024; BCBS 239 P3)