Skilirium

Derive data quality rules

Activity 7 of 8 in Data modelling · Occupation: Data modeller

The outcome: derives data quality rules from the data rules and the physical model.

Builds on: Design the physical data model, Document data rules as SBVR fact types and rule statements.

Which AI model is best at derive data quality rules?

Best result for derive data quality rules: gpt-6-astra (94% test score). Best value: gpt-6.1-sol (94%, within 5 points of the best at $0.077 per run instead of $0.45).

ModelProviderTierTest scoreKnobits mastered (expected)Cost per test run
gemini-4-argonGoogleFlagship88% 95%$0.062
gemini-3.1-pro-previewGoogleFlagship71% 86%$0.20
gpt-6-astraOpenAIFlagship94% 98%$0.45
claude-fable-5-1AnthropicFlagship83% 95%$0.85
gemini-3.8-flashGoogleMid tier81% 94%$0.045
gpt-6.1-solOpenAIMid tier94% 98%$0.077
claude-sonnet-5-5AnthropicMid tier90% 92%$0.23
gemini-3.5-flash-liteGoogleLow cost55% 66%$0.008
gpt-6-lunaOpenAILow cost77% 89%$0.005
claude-haiku-5-5AnthropicLow cost84% 95%$0.018

Test score: mean rubric score on the test tasks for this activity. Knobits mastered: expected share of the activity's knobits the model masters. Cost: mean API cost of one test run at list prices. Method and caveats.

The 31 knobits of this activity

18 that AI does well, 6 it does partly, and 3 where people still lead.

IDKnobitTypeAI exposure (models mastering)
K-703Recognises a constraint deferred without a checkSkillAI does this well 8/10
K-704Recognises a derived or copied column without a checkSkillAI does this well 8/10
K-705Recognises history kept without its constraints checkedSkillAI does this well 9/10
K-707Recognises a check with no rule behind itSkillAI does this well 8/10
K-708Recognises a check outliving its ruleSkillAI does this well 8/10
K-709Recognises a check unable to failSkillAI does this well 8/10
K-710Recognises a check on the wrong objectSkillAI does this well 7/10
K-711Recognises a check weaker than its ruleSkillAI does this well 7/10
K-712Recognises a check stronger than its ruleSkillAI does this well 8/10
K-713Recognises a check measuring the wrong populationSkillAI does this well 7/10
K-714Recognises nulls left to chance in a check predicateSkillAI does this partly 6/10
K-715Recognises a past event judged against present dataSkillAI does this partly 6/10
K-716Recognises a set rule written as a row ruleSkillAI does this well 9/10
K-717Recognises a check expression unable to runSkillAI does this well 9/10
K-718Recognises a misfiled data quality dimensionSkillPeople still lead 0/10
K-719Recognises duplicate checks for one ruleSkillPeople still lead 0/10
K-720Recognises a threshold at odds with its ruleSkillAI does this well 9/10
K-721Recognises disagreeing severity, threshold and failure actionSkillAI does this well 9/10
K-722Recognises a rejection destroying more than it protectsSkillNot tested yet
K-723Recognises a check failure with no capable ownerSkillAI does this partly 6/10
K-724Recognises a check scheduled at the wrong momentSkillAI does this partly 6/10
K-725Recognises an aggregate hiding a total failureSkillNot tested yet
K-726Recognises an empty population reported as a passSkillNot tested yet
K-751Explains data quality dimensionsKnowledgePeople still lead 0/10
K-752Explains traceability from data rule to checkKnowledgeAI does this well 10/10
K-753Explains the population, measure and threshold of a checkKnowledgeAI does this well 9/10
K-754Explains row-level and set-level checksKnowledgeAI does this well 9/10
K-755Explains severity, threshold and failure actionKnowledgeAI does this well 9/10
K-756Explains scheduling checks against loads and useKnowledgeAI does this partly 6/10
K-757Explains segmented data quality reportingKnowledgeNot tested yet
K-758Explains ownership of data quality failuresKnowledgeAI does this partly 6/10