Production Collaboration

Why Production Collaboration matters for Applied Scientist

As an Applied Scientist, your models only create impact when they reliably run in production and evolve safely after launch. Production Collaboration is the set of practices that connects research with engineering, product, and stakeholders to ship stable APIs, monitor performance, debug quickly, and communicate clearly. It helps you:

• Turn prototypes into maintainable services with clear interfaces and constraints.
• Align on release scope, SLOs, and risk mitigations with engineering.
• Detect and fix model issues using logs, metrics, and post-launch iteration plans.
• Explain results and decisions to product, business, and leadership.

What you will learn

• Define stable API contracts and non-functional constraints (latency, throughput, cost).
• Create clean prototype-to-production handoffs that engineers love.
• Instrument structured logging and set up monitoring to catch regressions early.
• Run safe experiments, communicate results, and prioritize iterations post-launch.
• Write effective specs and RFCs with crisp acceptance criteria.

Who this is for

• Applied Scientists moving from notebooks to productionized ML/AI systems.
• Data Scientists who partner closely with platform, backend, or ML engineers.
• Researchers who want predictable, low-risk rollouts and measurable impact.

Prerequisites

• Comfortable with Python and basic packaging (virtualenv/conda, requirements).
• Familiar with REST/JSON or similar service patterns.
• Basic understanding of model evaluation and metrics (precision/recall, latency).
• Basic SQL for querying logs and metrics.

Learning path

1. Interface & Constraints: Draft input/output schema, versioning, SLOs, and limits.
2. Handoff Package: Prepare a spec, model card, artifacts, and test dataset.
3. Logging & Debugging: Add structured logs with request IDs; write a triage runbook.
4. Monitoring: Define KPIs; write queries/dashboards; set alert thresholds.
5. Post-launch iteration: Plan A/B tests, guardrails, rollback, and comms cadence.

Worked examples

from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel, Field
from typing import Optional

app = FastAPI(title="sentiment-service", version="2026-01-01")

class PredictIn(BaseModel):
    text: str = Field(min_length=1, max_length=5000)
    lang: Optional[str] = Field(default="en", description="ISO-639-1")

class PredictOut(BaseModel):
    label: str
    score: float
    model_version: str
    request_id: str

SUPPORTED_VERSIONS = {"v1"}
MODEL_VERSION = "2026-01-01"

@app.post("/v1/predict", response_model=PredictOut)
async def predict(payload: PredictIn, x_request_id: Optional[str] = Header(default=None)):
    if x_request_id is None:
        raise HTTPException(400, detail={"code":"missing_request_id","message":"Provide X-Request-Id"})
    # ... call inference ...
    label, score = "positive", 0.91
    return {"label": label, "score": score, "model_version": MODEL_VERSION, "request_id": x_request_id}

handoff/
├─ SPEC.md                 # API contract, constraints, SLOs, limits
├─ RFC.md                  # Rationale, options considered, rollout plan
├─ MODEL_CARD.md           # Data, metrics, bias, limitations, ethics notes
├─ artifacts/
│  ├─ model.bin            # Serialized weights
│  ├─ preprocessor.pkl     # Tokenizer/encoder
│  └─ schema.json          # Feature schema & expected ranges
├─ sample_requests/
│  ├─ happy.json
│  └─ edge_cases.json
├─ tests/
│  └─ contract_tests.py    # Golden inputs/outputs
├─ notebooks/
│  └─ evaluation.ipynb
└─ README.md               # How to run tests and reproduce metrics

import logging, json, time, uuid

logger = logging.getLogger("inference")
h = logging.StreamHandler()
h.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
logger.addHandler(h)
logger.setLevel(logging.INFO)

request_id = str(uuid.uuid4())
start = time.time()
try:
    # pred = model.predict(features)
    latency_ms = int((time.time() - start) * 1000)
    logger.info(json.dumps({
        "event":"predict",
        "request_id":request_id,
        "latency_ms":latency_ms,
        "model_version":"2026-01-01",
        "input_size": 128,
        "result":"positive",
        "score":0.91
    }))
except Exception as e:
    logger.error(json.dumps({
        "event":"predict_error",
        "request_id":request_id,
        "error":str(e)
    }))

-- Requests, latency, and positive rate by day
SELECT
  DATE(timestamp) AS d,
  COUNT(*) AS requests,
  AVG(latency_ms) AS avg_latency,
  APPROX_QUANTILES(latency_ms, 100)[OFFSET(95)] AS p95_latency,
  SUM(CASE WHEN prediction = 'positive' THEN 1 ELSE 0 END) / COUNT(*) AS positive_rate
FROM prediction_logs
WHERE timestamp >= CURRENT_DATE - 7
GROUP BY d
ORDER BY d;

-- Join with ground truth to compute accuracy if labels arrive later
SELECT
  DATE(p.timestamp) AS d,
  AVG(CASE WHEN p.prediction = gt.label THEN 1 ELSE 0 END) AS accuracy
FROM prediction_logs p
JOIN ground_truth gt USING (request_id)
WHERE p.timestamp >= CURRENT_DATE - 30
GROUP BY d
ORDER BY d;

Mini project — Ship a small text-classification service with monitoring

1. Define API: Request schema (text, lang), response (label, score, model_version, request_id). Set SLOs (p95 ≤ 200 ms).
2. Create handoff folder with SPEC.md, MODEL_CARD.md, artifacts, and contract tests.
3. Add structured logging with request_id and latency.
4. Prepare monitoring queries for requests, p95 latency, positive rate, and accuracy with delayed labels.
5. Write an RFC with rollout plan (5% ➜ 25% ➜ 100%) and rollback triggers.
6. Simulate logs for 7 days and generate a short stakeholder update (what changed, results, next steps).

• Acceptance criteria: Contract tests pass; p95 latency under SLO in simulated data; monitoring queries run; RFC and update are clear and concise.

Drills and exercises

• [ ] Write an error response schema with fields: code, message, request_id, retryable.
• [ ] Draft three non-functional constraints (latency, payload size, timeout) and justify each.
• [ ] Instrument a dummy inference function with structured logs and a correlation ID.
• [ ] Write a SQL query to compute a rolling 7-day positive rate and flag a 10% drop.
• [ ] Create a 1-page spec with acceptance criteria that an engineer can implement without meetings.
• [ ] Prepare a 5-slide stakeholder update showing pre/post-launch metrics and a decision.

Common mistakes and debugging tips

Subskills

• Working With Engineering For Deployment — Coordinate packaging, environment, SLOs, and release cadence with engineering.
• Defining Interfaces And Constraints — Design stable APIs, set latency/timeouts, payload limits, and versioning strategy.
• Creating Prototype To Production Handoffs — Deliver specs, artifacts, tests, and documentation that reduce back-and-forth.
• Monitoring And Iterating Post Launch — Set KPIs, build dashboards, run safe experiments, and plan iterations.
• Debugging Model Issues With Logs — Use structured logs, IDs, and triage runbooks to resolve issues quickly.
• Writing Technical Specs And RFCs — Communicate scope, trade-offs, and acceptance criteria.
• Stakeholder Communication Of Results — Share crisp updates, impact, and decisions without jargon.

Practical projects

• Data drift playbook: Build a simulation that gradually shifts input distribution and show how alerts trigger and rollbacks occur.
• Latency budget audit: Profile each step (IO, feature prep, model inference) and reduce p95 by 30% with simple changes.
• Risk review: Produce a one-page risk register (privacy, bias, security) with mitigations and owners.

Next steps

• Pair with an engineer to review your handoff bundle and refine acceptance criteria.
• Run a tabletop incident drill: simulate a spike in errors and practice your on-call runbook.
• Prepare a post-launch update template you can reuse for future releases.

Skill exam

Test your understanding of Production Collaboration. Anyone can take the exam for free. If you are logged in, your progress and results will be saved.