Why this matters
As an Applied Scientist, you often propose, design, and ship ML features that depend on multiple teams (product, data engineering, infra, security, legal, SRE). Clear technical specs and RFCs (Request for Comments) are how you align everyone, reduce risk, and get approval to build. You will use them to:
- Define model behavior, data contracts, and API interfaces before coding.
- Explain trade-offs (latency vs. accuracy, cost vs. benefit) to decision-makers.
- Plan experiments, rollout, monitoring, and rollback for safe launches.
- Capture decisions so future teammates understand why choices were made.
Concept explained simply
Think of a technical spec as the blueprint for an implementation, and an RFC as a proposal to change how a system works. Both are structured documents to gather feedback, earn approval, and guide execution.
- Spec: clear problem, goals/non-goals, solution details, APIs, data schema, metrics, risks, rollout, and testing.
- RFC: a decision document proposing options and a recommended direction, inviting comments before committing.
Mental model
Idea → RFC (options and decision) → Technical Spec (how to build) → Implementation → Launch → Monitor → Iterate. Your spec is the contract; your RFC is the debate space.
Step-by-step: write a strong spec/RFC
- Start with context: the problem, who’s affected, urgency, and constraints (privacy, SLAs, budget).
- State goals and non-goals: what success includes and excludes.
- Propose options: 2–3 viable approaches with pros/cons and measurable impact.
- Choose and justify: why the recommended option wins (data, experiments, costs).
- Detail the design: APIs, data flows, models, features, training/inference infra.
- Plan safety: evaluations, guardrails, monitoring, alerts, rollback, and kill-switch.
- Address risk/compliance: PII handling, retention, access controls, and auditing.
- Sequence rollout: milestones, owners, timelines, and communication plan.
Mini task: 5-minute scoping
Write 3 bullets each for problem, goals, and constraints for your current ML task. If it feels fuzzy, you’re not ready to pick a solution yet.
Core components checklist
- Problem statement and context
- Goals and non-goals
- Stakeholders and reviewers
- Assumptions and constraints (latency, cost, privacy, SLAs)
- Solution options with trade-offs
- Chosen approach and rationale
- API and data contracts (schemas, versioning, backward compatibility)
- System design (diagrams described in text), data flows
- Experiment and evaluation plan (metrics, datasets, baselines)
- Risks, mitigations, and open questions
- Security, privacy, and compliance considerations
- Rollout plan (stages, gating, A/B or canary), observability and alerts
- Cost estimate and scaling plan (rough ranges; varies by country/company; treat as rough ranges)
- Timeline, milestones, and owners
- Decision record and follow-ups
Templates you can reuse
RFC template (copy)
Title: [Concise problem & proposed change] Status: Draft | In Review | Accepted | Rejected Owners: [Names] Reviewers: [Eng, Product, Data, Security, Legal, SRE] Date: [YYYY-MM-DD] 1. Context - Problem & impact - Current state & constraints 2. Goals / Non-goals 3. Options Considered - Option A: description, pros/cons, impact - Option B: description, pros/cons, impact - Option C: description, pros/cons, impact 4. Recommendation & Rationale - Why chosen, risks, mitigations 5. High-level Design - Components & interactions - Data and API changes (versioning) 6. Rollout & Success Metrics - Phases, guardrails, rollback - Metrics & targets 7. Risks, Privacy, Security 8. Open Questions & Next Steps Decision Log: - [YYYY-MM-DD] Decision X with reason
Technical spec template (copy)
Title: [Implementation plan for X] Status: Draft | In Review | Approved Owners: [Names] Date: [YYYY-MM-DD] 1. Summary 2. Detailed Requirements - Functional - Non-functional (latency, availability, cost) 3. Architecture & Data Flow - Inference/training paths - Feature and label sources 4. API Contracts - Endpoints, request/response, error codes - Backward compatibility & versioning 5. Data Contracts - Schemas, retention, quality checks 6. Model Details - Algorithm, features, training schedule, evaluation 7. Observability - Metrics, dashboards, alerts, drift detection 8. Rollout Plan - Environments, canary, A/B, kill-switch 9. Risks & Mitigations 10. Testing Plan - Unit, integration, offline/online eval 11. Timeline & Owners 12. Appendix: Glossary, assumptions
Worked examples
Example 1: RFC to add an online feature store for real-time recommendations
Context: Current recommendations use batch features (updated daily), causing stale signals and lower CTR during fast-moving events.
- Goals: Reduce median feature staleness from 12h to <5m; maintain p95 inference latency < 80 ms.
- Options: (A) Online feature store; (B) Real-time inference-only caching; (C) Increase batch frequency to hourly.
- Trade-offs: A improves freshness and consistency with higher infra cost; B limited coverage; C still stale during spikes.
- Recommendation: A. Expected CTR +1.2–1.8pp; cost +$3–5k/mo (rough).
- API: GET /v1/reco?user_id=... returns items with features_freshness_ms.
- Rollout: shadow reads → 5% canary → 50% → 100%; rollback to batch-only path via flag.
Example 2: Spec to switch ranking model from XGBoost to DistilBERT re-ranker
Requirements: Improve NDCG@10 by ≥3% with p95 latency ≤ 120 ms and error rate ≤ 0.1%.
- Design: Two-stage: fast XGBoost candidate generator → DistilBERT re-ranker on top 50 items.
- API: POST /v2/rank with list of item texts; response includes scores, model_version, and inference_time_ms.
- Observability: Track NDCG, latency, GPU utilization, timeouts; alert if p95 > 130 ms for 10m.
- Risks: GPU capacity during peak; mitigation: autoscaling with warm pools and fallback to XGBoost-only.
- Rollout: Train → offline eval → 5% A/B → 50% → 100%; kill-switch env var MODEL_TIER=fast_only.
Example 3: RFC to change data retention for training logs
Context: Keep training logs 180 days; storage costs rising; privacy review suggests 90 days is sufficient.
- Goals: Reduce storage by ~40% while supporting audits and reproducibility.
- Options: (A) 90-day retention + hashed IDs; (B) 180-day with tiered cold storage; (C) 60-day retention.
- Recommendation: A. Keeps compliance needs, reduces cost, minimizes risk.
- Risks: Longer investigations may need older data; mitigation: on-demand legal holds via request process.
- Rollout: New retention policy in pipeline; backfill deletions; update docs and access controls.
Exercises
These exercises mirror the tasks below. Do them in your own editor or notebook. Then compare with the sample solutions.
Exercise 1 (ex1): 1-page RFC for production model monitoring
Scenario: Your model handles 30% of homepage traffic. Incidents show silent performance regressions due to data drift. Propose an RFC to introduce monitoring and alerts.
- Include: context, goals/non-goals, 2–3 options, recommendation, high-level design, rollout, metrics, risks.
Exercise 2 (ex2): API spec for a new inference endpoint version
Scenario: You will launch /v2/infer with a new optional field and stricter error codes while keeping /v1 working.
- Define request/response JSON, versioning rules, backward compatibility plan, rate limits, and error handling.
Self-check checklist
- Problem and goals are measurable.
- At least two credible options considered.
- Chosen approach explains trade-offs and risks.
- API/data contracts are explicit and versioned.
- Monitoring, alerts, and rollback defined.
- Security/privacy constraints addressed.
- Timeline and owners listed.
Common mistakes and how to self-check
- Vague goals: Replace words like "better" with target metrics and thresholds.
- One-option bias: Always document at least one alternative to show you explored trade-offs.
- No rollback: Every risky change needs a quick escape path.
- Unstated constraints: Latency, cost, and privacy must be explicit; don’t assume reviewers know them.
- Ambiguous API: Include types, required/optional fields, error codes, and versioning plan.
- Missing ownership: Name owners for each milestone and dashboard.
- Over-detail too early: Keep RFC high-level; detail the spec after a decision is made.
Quick self-review routine (3 minutes)
- Read only headings: does the story make sense?
- Skim metrics: do they tie to goals?
- Search for the word "rollback": is it present and actionable?
Practical projects
- Write an RFC to add drift detection to an existing model using population stability index (PSI) and model confidence shifts; include a canary plan.
- Create a spec for feature store backfills including data quality checks (missingness, range, uniqueness) and alert thresholds.
- Draft a spec for online evaluation with interleaving or bandits, including guardrails on business KPIs.
Who this is for
- Applied Scientists and ML Engineers who ship models to production.
- Data Scientists moving from research notebooks to collaborative product work.
Prerequisites
- Basic system design literacy (APIs, services, data pipelines).
- Understanding of ML evaluation metrics and A/B testing.
- Awareness of privacy and compliance basics (PII, retention).
Learning path
- Start: Draft simple RFCs for small changes (metrics or monitoring).
- Next: Write full technical specs for API and data changes.
- Advance: Lead cross-team reviews, resolve comments, and record decisions.
- Mastery: Create templates and guidelines for your org; mentor others.
Next steps
- Pick one active project and convert it into a 1–2 page RFC by tomorrow.
- Schedule a 30-minute async review: ask for specific feedback on goals, risks, and rollout.
- After approval, expand into a detailed spec and track the decision log.
Mini challenge
In 10 sentences, write the core of an RFC to deprecate a legacy feature pipeline in favor of a unified feature store. Include goals, two options, your recommendation, and a rollback plan.
Note: The quick test is available to everyone; only logged-in users get saved progress.