Why this matters
As a Data Architect, you define the quality bar and make it visible. Clear metrics and reporting turn vague data issues into measurable, fixable contracts with stakeholders. You will set SLOs, instrument SLIs, track trends, and guide teams toward reliable, trustworthy data products.
- Real tasks: define dataset SLOs (freshness, completeness), review DQ dashboards, decide alert thresholds, and report quality to business owners.
- Outcome: fewer incidents, faster recovery, and higher confidence in analytics, ML features, and downstream services.
Who this is for
- Data Architects and Platform leaders who need dependable, observable data products.
- Senior Data/Analytics Engineers responsible for data contracts and monitoring.
- Data Stewards who own data quality governance and reporting.
Prerequisites
- Basic SQL for aggregations and filters.
- Understanding of batch vs. streaming pipelines and partitions.
- Familiarity with schema management and data lineage concepts.
Concept explained simply
Think of data quality as a scoreboard. Each dataset has a few key scores (SLIs) that show how healthy it is. You set realistic targets (SLOs), measure daily, and report on performance to stakeholders.
Mental model
- SLI: the measurement. Example: p95 freshness delay in minutes, completeness percent, validity percent.
- SLO: the target. Example: p95 freshness <= 15 minutes for 28 of 30 days.
- SLA: the formal agreement (if applicable) tied to business consequences.
- Dashboards and reports: roll SLIs up to teams and executives with green, yellow, red states and trends.
Deeper dive: thresholds and states
A practical scheme is 3-state traffic lights: green meets target, yellow is near miss or at risk, red is breach. Include a short grace window to avoid flapping. Use weekly and monthly rollups for exec reporting; use partition-level views for engineers.
Core metrics to know
- Completeness: non_null_rows divided by expected_non_null_rows. Example: 9,800 of 10,000 rows -> 98.0%.
- Validity: rows passing rules divided by checked_rows. Example rules: type checks, regex, value ranges, reference lookups.
- Accuracy: correct_values divided by sampled_values when ground truth exists (e.g., reconciled totals).
- Uniqueness: 1 minus duplicate_rate for key columns (or percent unique keys).
- Consistency: cross-system match rate (e.g., warehouse vs. source totals).
- Freshness: now minus max(event_time) or now minus max(ingest_time). Choose the definition and be consistent.
- Timeliness compliance: percent of partitions meeting the freshness target.
- Volume anomaly rate: deviations of record counts vs. learned baseline.
- Schema change rate: count of breaking changes per period.
- Reliability: pipeline success rate, incident count, MTTR (mean time to recover), MTBF (mean time between failures).
How to roll up a Data Quality Score
Combine multiple metrics with weights. Example: score = 0.4 completeness + 0.4 validity + 0.2 freshness_score. Invert metrics like duplication_rate into scores (100 minus rate_percent). Keep weights stable and documented.
Designing SLOs and SLAs for data
Example SLO policy
- Freshness: p95 delay <= 30 min for 27 of 30 days. Warn if 25–30 min trend up. Page if > 30 min for 2 consecutive partitions.
- Completeness: >= 98.5% daily. Warn at 98.0–98.5%. Page if < 98.0%.
- Validity: >= 99.0% on key columns. Warn at 98.5–99.0%. Page if < 98.5%.
Reporting patterns that work
By audience
- Executives: monthly trend of green/yellow/red across top datasets, top incidents, time-to-recover improvements.
- Product/Analysts: dataset-level SLIs vs. SLOs, recent regressions, partition heatmaps, rule failures.
- Platform/Eng: partition-level metrics, rule hit counts, lineage impact, incident timeline.
By frequency
- Daily: health summary with exceptions.
- Weekly: trends, root causes, action items.
- Monthly: SLO attainment, incident metrics (MTTR/MTBF), roadmap adjustments.
Must-include sections in a DQ report
- Owner and contacts.
- SLIs, SLOs, current status (traffic light), last 30-day trend.
- Top failed rules and impact estimation.
- Open actions with due dates.
Worked examples
Example 1: Completeness and freshness for daily Orders
Yesterday expected 100,000 rows, ingested 97,800 non-null order_id. Completeness = 97,800 divided by 100,000 = 97.8%.
Freshness target: p95 delay <= 30 minutes. Measured p95 = 25 minutes. Status: green (meets SLO).
Example 2: Email validity and accuracy proxy
Rules: regex for format, domain in allowlist. 198,000 pass of 200,000 checked => validity 99.0%.
Proxy accuracy: compare to bounce signals next day. If 1.2% bounces on previously valid emails, infer an upper bound on true accuracy near 98.8%. Document the proxy.
Example 3: Weighted DQ score
Weights: completeness 35%, validity 35%, duplication (inverted) 10%, freshness 20%.
- Completeness: 97.0 -> 33.95
- Validity: 98.5 -> 34.475
- Duplication rate: 0.7% -> inverted 99.3 -> 9.93
- Freshness within target -> 100 -> 20.00
Total score ≈ 98.36 (green).
Exercises you can do now
These exercises mirror the tasks below. Try them before opening solutions. The quick test is available to everyone; log in to your account if you want your progress saved.
Exercise 1: Calculate a weighted quality score
Given metrics and weights for a daily Orders dataset, compute the overall score and traffic-light status. See the exercise block for details and the solution.
Exercise 2: Design SLOs and alert thresholds
For a Customer 360 dataset, propose SLIs, SLO targets, and alert thresholds. Provide a short weekly report snippet.
Before you submit: checklist
- Weights sum to 100% and are documented.
- Inverted metrics (e.g., duplication_rate) converted to scores first.
- SLOs include a time window (e.g., p95 per day, success for N of last 30 days).
- Alerts include warn vs. page thresholds and hysteresis.
Common mistakes and self-check
- Too many metrics. Fix: pick 2–4 SLIs per dataset tied to business risk.
- Unclear formulas. Fix: define numerator, denominator, and scope (table, column, partition).
- Ignoring evaluation window. Fix: define rolling window and attainment rule.
- Flappy alerts. Fix: add hysteresis and require consecutive breaches for paging.
- Mixing event_time and ingest_time freshness. Fix: choose one definition per dataset and stick to it.
Self-check prompts
- Can a new teammate reproduce each metric from the definition?
- Does your report quantify impact and list owners with due dates?
- Do stakeholders understand why green vs. yellow vs. red?
Practical projects
- Project 1: Build a DQ scorecard for a top-3 dataset. Acceptance: 3 SLIs with SLOs, daily partition view, 30-day trend, traffic-light status.
- Project 2: Freshness SLO for streaming topic. Acceptance: p95 delay chart, breach detection with warn/page thresholds, weekly attainment summary.
- Project 3: Rule coverage report. Acceptance: list of rules, pass/fail counts, top failing columns, and owner-tagged actions.
Learning path
Mini challenge
In one paragraph, define the minimal set of SLIs and SLOs for a Finance Close dataset consumed by executives every morning. Include thresholds and an alert policy.
Next steps
- Take the quick test to validate your understanding. Note: anyone can take it; log in if you want your progress saved.
- Apply these metrics to one live dataset and share the report with its owner.
- Expand to lineage-aware reporting and anomaly detection in your next learning unit.