Fine Tuning Concepts

Why this matters

As an AI Product Manager, you will often decide whether a base model is good enough with prompting, needs retrieval (RAG), or should be fine-tuned. These choices affect cost, latency, risk, quality, and time-to-market. Typical tasks you will make decisions on include:

• Defining the target behavior (tone, format, policy compliance) and acceptance criteria.
• Choosing between prompting, RAG, parameter-efficient fine-tuning (PEFT), or full fine-tuning.
• Scoping datasets for tuning and evaluation, including labeling guidelines.
• Balancing quality gains against training/inference costs and latency.
• Planning offline and online evaluations (A/B tests, safety checks).

Concept explained simply

Fine-tuning nudges a model to reliably perform a narrow set of tasks in a specific style. Prompting asks the model to do something; fine-tuning teaches the model to default to doing it consistently.

• Supervised fine-tuning (SFT): Train on input-output pairs so the model imitates desired outputs.
• Instruction tuning: A type of SFT focusing on following instructions across many tasks.
• Preference optimization (e.g., RLHF/DPO): Train on human preference data (A vs. B) to align style or helpfulness.
• PEFT (e.g., LoRA/adapters): Update only small additional weights. Cheaper, faster, and easier to roll back than full fine-tunes.
• Alternatives:
  • Prompting for simple changes and small volumes.
  • RAG when knowledge changes frequently (fetch facts at runtime).
  • Embeddings for semantic search, clustering, or deduplication.

Mental model: Think of four knobs—Knowledge, Style, Format, Safety. Use RAG to update Knowledge without retraining; use prompts for light Style/Format tweaks; use fine-tuning to bake in stable Style/Format/Safety behaviors at scale.

When NOT to fine-tune

• Facts or content change weekly—prefer RAG.
• The task is rare or low volume—prompting is usually enough.
• The problem is unclear—first improve task definition and labeling guidelines.
• Budget/latency is tight—measure prompt-only baseline first.

Deciding: fine-tune vs prompt vs RAG

Use this quick checklist. If three or more are true, fine-tuning is likely worth it:

• Stable, repeatable behavior is needed for months.
• Clear input-output schema and labeling criteria.
• High volume or automated pipeline (low tolerance for prompt variance).
• Strict tone/format/policy requirements (e.g., regulated domains).
• Security/privacy requires internalized rules rather than external retrieval.

Common choices:

• Prompt only: One-off tasks, low volume, flexible outputs.
• RAG: Frequently changing or large knowledge bases; needs citations.
• PEFT (LoRA): Stable style/format rules; moderate dataset (500–10k examples); constrained budget.
• Full fine-tune: Specialized tasks at scale where you control infrastructure and can afford higher costs.

Practical workflow (step-by-step)

1. Define target behavior — write a one-page spec: inputs, outputs, constraints, and examples. Include a scoring rubric.
2. Collect data — real logs, synthetic data, or expert-labeled samples. De-identify PII if needed.
3. Label consistently — create guidelines with positive/negative examples. Run a pilot to measure inter-annotator agreement.
4. Split the data — train/dev/test (e.g., 70/15/15). The test set remains untouched until final evaluation.
5. Choose approach — try prompt-only baseline; if insufficient, try PEFT; consider RAG if knowledge updates are frequent.
6. Train small, iterate fast — start with 200–1000 examples. Monitor loss and overfitting; keep a simple changelog.
7. Evaluate offline — use automatic metrics and human review:
   • Classification: accuracy, F1, macro-F1 for imbalance.
   • Generation: exact match when possible; rubric pass rate; pairwise win rate; toxicity and safety scores.
8. Ship safely — A/B test against baseline; add guardrails (max length, content filters); log telemetry for drift.

Common data format (JSONL):

{"instruction": "Rewrite in brand tone: warm, concise.", "input": "We regret to inform you that your order is delayed.", "output": "Thanks for your patience—your order is running a bit late. We’ll keep you posted."}
{"instruction": "Classify intent: refund|status|complaint", "input": "Where’s my package? It’s two days late.", "output": "status"}

Worked examples

Example 1 — Customer support macro classification

• Goal: Map messages to {refund, status, complaint, other} with >95% accuracy.
• Data: 5,000 labeled tickets; imbalanced classes.
• Approach: PEFT (LoRA) SFT on a small instruction-tuned model; class-weighted sampling.
• Why not RAG: Knowledge is stable; we only classify.
• KPIs: Macro-F1 > 0.93; latency < 150 ms; zero toxic outputs.

Example 2 — Brand tone rewriting

• Goal: Enforce a warm, concise tone for outbound emails.
• Data: 2,000 pairs of before/after edits from brand team; style guide.
• Approach: Try prompting with few-shot. If variance persists, PEFT SFT with the pairs.
• Why not RAG: No changing facts to fetch; style is stable.
• KPIs: Human rubric pass rate ≥ 90%; average length within 10% target; no policy violations.

Example 3 — SQL generation from natural language

• Goal: Generate correct SQL over a known schema.
• Data: 1,500 NL→SQL pairs; schema JSON.
• Approach: Start with RAG (include current schema) + constrained decoding; consider PEFT if consistent patterns emerge and volume is high.
• Why: Schema changes occasionally; RAG avoids re-training for each update.
• KPIs: Exact match on hidden queries ≥ 80%; execution success ≥ 95%; latency < 800 ms.

Example 4 — Policy compliance summarizer (regulated)

• Goal: Summaries must include specific disclaimers and sections.
• Data: 3,000 compliant summaries with rubric scores.
• Approach: PEFT SFT to lock format; add a short preference dataset for tie-breaking style via DPO.
• KPIs: Structure exact match ≥ 98%; compliance rubric ≥ 95%.

Evaluation and acceptance

• Define a rubric with unambiguous pass/fail checks (e.g., includes required fields, tone constraints, no PII leakage).
• Use pairwise win rate vs. baseline for subjective quality; sample size 100–300 is often enough to see signal.
• Track cost and latency budgets; compare prompt-only vs. PEFT vs. RAG end-to-end latency.
• Safety gates: toxicity, jailbreak checks, PII redaction tests.

Example acceptance criteria: “On the 300-sample test set, the fine-tuned model must achieve ≥ 90% rubric pass rate, ≤ 5% format errors, latency ≤ 400 ms p95, and pass all safety checks.”

Common mistakes and self-check

• Mistake: Fine-tuning to memorize facts that change monthly. Self-check: Will RAG solve this without retraining?
• Mistake: Vague labels. Self-check: Can two annotators agree ≥ 0.8 Cohen’s kappa on a pilot set?
• Mistake: Overfitting from tiny datasets. Self-check: Is dev performance much higher than test? Add data or regularize.
• Mistake: Ignoring class imbalance. Self-check: Track macro-F1, not just accuracy.
• Mistake: No baseline. Self-check: Did you record prompt-only scores and costs?
• Mistake: Deploying without guardrails. Self-check: Toxicity and policy tests green?

Practical projects

• Project 1: Tone Enforcer
  • Deliverables: data spec, 1k pairs, PEFT model, rubric, offline eval, A/B test plan.
• Project 2: Support Intent Classifier
  • Deliverables: label guide, balanced dataset, macro-F1 ≥ 0.9, latency report, drift monitoring plan.
• Project 3: Policy-Compliant Summarizer
  • Deliverables: structured template, PEFT SFT, safety tests, acceptance criteria doc.

Exercises

These mirror the exercises below. Write answers briefly and check against the solutions. Tip: Keep a simple spreadsheet for your decisions.

1. Exercise 1 — Pick the right approach

   Scenario: You need consistent, on-brand product title normalization for an e-commerce catalog across 20k items/day. Titles should follow a strict format: Brand | Product | Key Attribute | Size. Information changes slowly; style rules are stable.

   • Decide: Prompt-only, RAG, PEFT, or full fine-tune.
   • Outline 3–5 dataset fields you would collect.
   • List two main risks and how you will check them.
   Show solution

   Recommended: PEFT (LoRA) SFT. High volume + stable formatting rules benefit from baked-in behavior. Prompt-only tends to drift; RAG not needed for static rules; full FT is costlier with little added value.

   Dataset fields: original_title, normalized_title (gold), brand, product_category, notes_on_transform (optional).

   Risks and checks:

   • Hallucinated attributes → unit test: any attribute not in source triggers fail.
   • Brand/style drift → rubric check for format and tone; exact format match metric.
2. Exercise 2 — Draft a fine-tuning dataset spec

   Scenario: Extract whether a contract clause includes an auto-renewal condition. Output strictly "yes" or "no".

   • Define labeling rules in 3 bullets.
   • Provide 5 JSONL examples (instruction, input, output).
   • Propose a simple evaluation plan.
   Show solution

   Labeling rules:

   • Answer "yes" only if the clause states renewal without explicit user action.
   • Mentions of "may renew" without default behavior → "no".
   • Conflicting statements → choose "no" and flag for review (not in output).

   Examples (JSONL):

   {"instruction": "Does this clause auto-renew? yes/no", "input": "This Agreement shall automatically renew for successive one-year terms unless either party gives 30 days' notice.", "output": "yes"}
   {"instruction": "Does this clause auto-renew? yes/no", "input": "Upon expiration, parties may negotiate a renewal at their discretion.", "output": "no"}
   {"instruction": "Does this clause auto-renew? yes/no", "input": "The term continues month-to-month until terminated by either party.", "output": "yes"}
   {"instruction": "Does this clause auto-renew? yes/no", "input": "The agreement ends after 12 months unless renewed in writing by both parties.", "output": "no"}
   {"instruction": "Does this clause auto-renew? yes/no", "input": "The subscription renews automatically for another term unless canceled.", "output": "yes"}
   

   Evaluation plan: 300-sample test set; exact match accuracy ≥ 95%, macro-F1 ≥ 0.94; spot-check 50 borderline cases; zero leakage of PII.

• Checklist: Did you choose an approach? Define data fields? Add acceptance criteria? Add a safety check?

Note: The quick test is available to everyone. If you log in, we will save your progress and scores.

Mini challenge (10 minutes)

Choose one of your team’s recurring tasks. In 10 minutes, decide: prompt-only, RAG, PEFT, or full FT. Write 5 acceptance checks (binary) and one latency/cost constraint. Keep it to half a page.

Who this is for

• AI Product Managers and PMs working with LLM features.
• Tech leads and analysts defining data and evaluation plans.

Prerequisites

• Basic familiarity with LLM capabilities and prompting.
• Comfort reading simple JSONL and evaluation metrics (accuracy/F1).

Learning path

• Before: Prompt Engineering Basics; Intro to RAG.
• Now: Fine-tuning concepts (this lesson).
• Next: Evaluation and Safety for LLM Systems; Deployment and Monitoring.

Next steps

• Complete the quick test to confirm understanding.
• Pick one practical project and draft your data spec this week.
• If you log in, your test progress is saved automatically.