Approval And Governance Flows

Why this matters

In real teams, models cannot jump from a notebook to production on someone’s enthusiasm. Approval and governance flows ensure every production model is safe, compliant, reproducible, and reversible. As an MLOps Engineer, you translate business, risk, and engineering requirements into clear stages and gates inside the model registry so that launches are predictable and auditable.

• Regulated orgs: document approvals, bias checks, and lineage for audits.
• Consumer products: protect users with rollback plans and drift monitors.
• All teams: reduce outages by enforcing quality gates before promotion.

Who this is for and prerequisites

• Who this is for: MLOps Engineers, ML Engineers, and Tech Leads defining how models move from experimentation to production.
• Prerequisites: Familiarity with model registries (states/versions), CI/CD basics, metrics (accuracy, AUC, latency), and RBAC concepts.

Concept explained simply

An approval and governance flow is the set of steps, checks, and sign-offs required to move a model version from one registry stage (e.g., Draft → Staging → Production) to the next. Each step collects evidence (metrics, tests, documentation) and records who approved what and when. It’s your safety net and your audit trail.

A mental model

Think of a model release like boarding a flight:

• Security gate (automated tests): quality, bias, and security scans.
• Boarding pass check (manual sign-offs): responsible humans approve.
• Flight log (audit trail): immutable record of what flew and why.
• Alternate airport (rollback): a clear plan if conditions degrade.

Core components of an approval flow

• States: Draft/Experiment → Staging → Production → Deprecated/Archived.
• Roles and permissions: Model Owner, Reviewer (Tech Lead), Risk/Compliance (if applicable), MLOps (release), Product Owner (business sign-off).
• Quality gates (automated):
  • Performance: meets baseline or improves by X% with statistical significance.
  • Robustness: adversarial or stress tests; drift baseline saved.
  • Fairness: group metrics within thresholds (e.g., demographic parity diff ≤ 0.05).
  • Latency and cost: p95 latency ≤ target; cost within budget.
  • Security and compliance: dependency/license scan; PII checks; no secrets in artifacts.
  • Reproducibility: code hash, data snapshot ID, and training config pinned.
• Manual approvals: Named approvers sign off with reason, ticket/issue ID, and risk classification.
• Documentation: Model card, evaluation report, monitoring plan, rollback plan.
• Audit trail: Immutable log of who/when/what/why for each transition.
• Change types: Standard (normal), Emergency (hotfix with time-bounded bypass + post-incident review), and Experiment (stays in Staging/canary).

Worked examples

Example 1 — Credit risk model at a bank (regulated)

1. States: Draft → Staging → Production.
2. Automated gates in Staging:
   • KS/AUC not worse than baseline by more than 1% and within 95% CI.
   • Bias: approval rate disparity ratio between protected groups ≥ 0.8.
   • Reproducibility: training run has code hash, data snapshot, seed recorded.
   • Security: SBOM and license scan pass.
3. Manual approvals: Model Owner, Risk Officer, Compliance Officer.
4. Prod promotion: requires monitoring plan (drift, rejection rate) and rollback steps documented.

Example 2 — E-commerce recommender (consumer web)

1. Staging gate: improves CTR ≥ 2% vs. baseline in offline eval; p95 latency ≤ 120 ms.
2. Canary in Prod: 10% traffic for 48 hours with auto-rollback if CTR drops ≥ 1% or error rate ≥ 0.5%.
3. Approvals: Model Owner + Product Owner; Compliance N/A.

Example 3 — Medical imaging triage (safety critical)

1. Two-person rule: Senior Clinician + Model Owner sign-off.
2. Thresholds: sensitivity ≥ 0.95; specificity ≥ 0.9 on locked clinical test set.
3. Human-in-the-loop requirement documented; feature use constraints enforced.

Design your flow (quick, practical)

1. Define states: Draft → Staging → Production → Deprecated.
2. Map roles: Who can request, review, approve, and promote at each state?
3. Set gates:
   • Performance: exact metrics and minimum improvements.
   • Risk: bias thresholds if user-impacting decisions exist.
   • Ops: latency, cost, and error budgets.
4. Evidence pack: model card template, evaluation report, monitoring + rollback plan.
5. Audit policy: capture who/when/what/why and attach ticket IDs.
6. Exceptions: define emergency change path and post-incident review.

Sample RACI (responsible/approver)

• Draft → Staging: Responsible = Model Owner; Approver = Tech Lead.
• Staging → Prod: Responsible = MLOps; Approvers = Model Owner + Product Owner (+ Compliance if regulated).

Exercises

Do these now. They mirror the graded exercises below.

1. Exercise 1: Design an approval flow for a ride-sharing ETA model (medium risk). Specify states, roles, automated gates, manual approvals, and rollback trigger. Write your answer as bullets.
2. Exercise 2: Draft a simple policy (pseudo-YAML) that encodes gates for promotion to Production: performance, fairness, latency, security, and required approvers.

• [ ] I defined clear states.
• [ ] I mapped roles and permissions.
• [ ] My gates are measurable and automatable.
• [ ] I included an audit trail and rollback plan.

Common mistakes and how to self-check

• Vague thresholds: “good performance” is not measurable. Replace with exact numbers and confidence where possible.
• Missing reproducibility: ensure code hash, data snapshot ID, and config are saved.
• No rollback trigger: define objective triggers (e.g., p95 latency > 200 ms for 15 minutes).
• Approver confusion: explicitly list approver roles per transition.
• Skipping fairness on user-impacting models: add at least one relevant bias metric.
• Ignoring cost: include cost per 1k predictions or resource budget.

Self-check: If you removed the names of the people, could another team reproduce the decision from the evidence and thresholds alone? If not, clarify the gates.

Practical projects

• Create a model card and evaluation bundle template your team can reuse.
• Implement a CI job that fails when new model metrics fall below thresholds or when fairness checks breach limits.
• Configure your registry to require two approvals for Production and to store promotion comments with ticket IDs.
• Write an automated rollback policy based on canary KPIs and error budgets.

Learning path

1. Model versioning basics and immutable artifacts.
2. Defining registry stages and stage transitions.
3. Quality gate design: metrics, bias, latency, cost.
4. RBAC and approver workflows.
5. Monitoring, drift, and rollback policies.
6. Audits and documentation: model cards and lineage.

Next steps

• Adapt the sample policies to your org’s risk level and SLAs.
• Automate one gate at a time; start with reproducibility and baseline performance.
• Set up notifications for pending approvals and failed gates.

Mini challenge

Your fraud model is being promoted with +0.5% recall but +0.3% higher false-positive rate, impacting user support costs. Propose a gate adjustment and a canary policy to balance risk and operations. Keep it to 5 bullet points.

Note: The quick test is available to everyone. Only logged-in users will have their progress saved.