Prompt Management Systems Basics

Why this matters

Worked examples

"Write a product description for the Acme Rocket. Keep it exciting and under 80 words."

{
  "id": "product_description_v1",
  "template": "You are a concise copywriter. Write a vivid, benefit-focused product description for {product_name} aimed at {audience}. Limit: {word_limit} words.",
  "variables": {
    "product_name": {"type":"string", "maxLen":60},
    "audience": {"type":"enum", "values":["shoppers","engineers","executives"]},
    "word_limit": {"type":"integer", "min":20, "max":120}
  },
  "metadata": {
    "purpose":"E-commerce PDP copy",
    "owner":"growth-team",
    "success":"Clarity 4+/5 on rubric; CTR +3%"
  }
}

v1 - Baseline.
v2 - Added audience variable; expectation: +2 points on relevance (offline test). 
Rollback rule: If online CTR drops >1% for 2 hours, revert to v1.

Hands-on: minimal workflow

1. Define the template, variables, and metadata.
2. Create a small offline test set (10–50 examples).
3. Run offline evaluation and record scores.
4. Prepare a release note with hypothesis and rollback triggers.
5. Ship to staging; sanity check logs and safety filters.
6. Canary release to 5–10% traffic.
7. Monitor telemetry; decide promote, iterate, or rollback.
8. Document outcomes and learnings.

Exercises

Complete these tasks. You can compare with the solutions below each exercise card.

Exercise 1: Design a prompt template spec

Create a compact specification for a support-reply prompt. Include id, purpose, template, variables (with types and constraints), safety notes, and success criteria.

• Keep to ~150–250 words or an equivalent JSON/YAML block.
• Include at least one numeric constraint and one enum.
• Add a short note on how you would test it offline.

{
  "id":"support_reply_v1",
  "purpose":"Polite, accurate replies to customer emails",
  "template":"You are a helpful agent. Summarize the user issue from: {email_snippet}. Provide a {tone} reply with up to {sentence_limit} sentences. Include one actionable next step.",
  "variables":{
    "email_snippet":{"type":"string", "maxLen":2000},
    "tone":{"type":"enum", "values":["friendly","formal"]},
    "sentence_limit":{"type":"integer", "min":2, "max":5}
  },
  "safety":{"pii_redaction":true, "avoid_promises":true},
  "success":"Rubric: accuracy ≥4/5; tone ≥4/5; refusal if policy-violating."
}

Exercise 2: Release plan and rollback

Write a short release plan for upgrading v1 to v2. Include hypothesis, metrics, offline baseline, canary size, monitoring window, and rollback trigger.

• Mention at least two metrics (quality and cost or latency).
• State an explicit rollback rule.

Hypothesis: v2 adds tone control; improves user satisfaction.
Offline baseline: Relevance +1.2 points; no cost increase.
Canary: 10% of traffic for 24h.
Metrics: Satisfaction CSAT, response latency, tokens per request.
Promote if CSAT +2% and latency within +5%.
Rollback if CSAT -1% for >2h or latency +15%.

Common mistakes and self-check

• Mistake: No versioning. Fix: Tag each change and log notes.
• Mistake: Vague success criteria. Fix: Define clear rubrics and metrics before release.
• Mistake: Shipping straight to 100%. Fix: Use staging and canary.
• Mistake: Missing telemetry. Fix: Capture prompt id/version, latency, tokens, and key quality signals.
• Mistake: Hardcoding secrets in prompts. Fix: Store creds in secure config; reference variables only.

Who this is for

Prompt Engineers, ML Engineers, and Product-minded researchers who need reliable, auditable prompt workflows from prototype to production.

Prerequisites

• Basic prompt design (system/user/assistant roles, variables).
• Familiarity with evaluation rubrics and A/B testing basics.
• Comfort with reading simple JSON/YAML specs.

Learning path

1. Prompt design fundamentals
2. Prompt management systems (this lesson)
3. Automated evaluation and guardrails
4. Experimentation and canary/A-B
5. Observability and cost optimization

Practical projects

• Build a tiny prompt library: 3 templates, variables, and metadata; add a JSON file of release notes.
• Create an offline test set (20 examples) and a script that scores outputs with a simple rubric.
• Run a simulated canary by routing 10% of test inputs to v2 and compare metrics.

Mini challenge

Pick a prompt you currently use. Turn it into a managed template with variables and metadata, define a two-metric success criterion, and write a one-paragraph release note with rollback. Keep it under 10 minutes.

Next steps

Take the quick test to lock in the core ideas. Note: anyone can take the test; only logged-in users have their progress saved.