Tooling And Deployment

# prompt_template.txt (v1.0.0)
SYSTEM:
You are a helpful assistant that follows company policy.

INSTRUCTIONS:
Summarize the following content for a {audience}.
Tone: {tone}

CONTENT:
{context}

# render.py (Python)
import os
from string import Template

def render(template_path, variables):
    with open(template_path, 'r') as f:
        t = Template(f.read())
    # Convert {var} style to $var for Template
    # or store template as $audience
    text = t.safe_substitute(**variables)
    return text

prompt = render('prompt_template.txt', {
    'audience': 'non-technical stakeholders',
    'tone': 'neutral and concise',
    'context': 'Quarterly report shows 12% growth in Q2 driven by product X.'
})
print(prompt)

Tips: keep a CHANGELOG.md for what changed and why; include a simple semantic version number.

# rag_pipeline.py
from typing import List

class Retriever:
    def query(self, q: str, k: int = 3) -> List[str]:
        # Replace with your vector store or keyword search
        docs = [
            "Policy: Refunds allowed within 30 days.",
            "Policy: Digital goods are non-refundable after download.",
            "Contact: support@example.com"
        ]
        return docs[:k]

retriever = Retriever()

question = "Can I refund a digital item after 2 weeks?"
chunks = retriever.query(question, k=2)

context = "\n\n".join(chunks)
prompt = f"""
SYSTEM:
You are a policy expert.

INSTRUCTIONS:
Answer the user's question strictly using the context. If unknown, say you don't know.

CONTEXT:
{context}

USER:
{question}
""".strip()

print(prompt)

Ensure you handle empty retrieval by instructing the model to say it does not know.

# logging_utils.py
import json
import re

EMAIL = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
PHONE = re.compile(r"\+?[0-9][0-9\-\s]{6,}[0-9]")

def redact(text: str) -> str:
    text = EMAIL.sub("[REDACTED_EMAIL]", text)
    text = PHONE.sub("[REDACTED_PHONE]", text)
    return text

def log_event(event_type: str, data: dict):
    data = {**data}
    if 'prompt' in data:
        data['prompt'] = redact(data['prompt'])
    if 'response' in data:
        data['response'] = redact(data['response'])
    print(json.dumps({
        "type": event_type,
        "data": data
    }))

# usage
log_event("llm_request", {
    "model": "gpt-4o-mini",
    "prompt": "User email john@example.com asked: refund policy?",
    "variables": {"tone": "formal"}
})

Only log what you need. Redact PII before printing or shipping logs.

# resilience.py
import time
import random

class RateLimitError(Exception):
    pass

def call_model(payload, idempotency_key):
    # simulate 429s randomly
    if random.random() < 0.2:
        raise RateLimitError("429 Too Many Requests")
    return {"idempotency_key": idempotency_key, "ok": True}

def request_with_retries(payload, max_retries=5, base=0.5):
    key = payload.get("request_id")  # idempotency key
    for attempt in range(max_retries + 1):
        try:
            return call_model(payload, key)
        except RateLimitError:
            if attempt == max_retries:
                raise
            sleep = base * (2 ** attempt) + random.uniform(0, 0.2)
            time.sleep(sleep)

resp = request_with_retries({"request_id": "abc-123", "text": "hello"})
print(resp)

Use exponential backoff with jitter; ensure retries reuse an idempotency key.

# tests/test_prompt_rules.py
import re

def render(**vars):
    # your real renderer here
    return f"Answer clearly. Tone: {vars['tone']}. Context: {vars['context']}"

def test_required_variables():
    out = render(tone="neutral", context="policy text")
    assert "Tone:" in out
    assert "Context:" in out

def test_no_forbidden_phrases():
    out = render(tone="neutral", context="policy text")
    forbidden = ["as an AI language model", "I cannot"]
    for phrase in forbidden:
        assert phrase not in out.lower()

# .github/workflows/prompt-ci.yml
# name: Prompt CI
# on: [pull_request]
# jobs:
#   test:
#     runs-on: ubuntu-latest
#     steps:
#       - uses: actions/checkout@v4
#       - uses: actions/setup-python@v5
#         with:
#           python-version: '3.11'
#       - run: pip install -r requirements.txt
#       - run: pytest -q

Prefer robust checks (required fields, style, forbidden phrases) over brittle word-for-word comparisons.

LLMs vary phrasing. Test for structure, presence/absence of key facts, or regex patterns. Keep outputs constrained with instructions and examples.

Always redact before logging. Mask emails, phones, and tokens. Restrict log retention and access.

If you do not back off on 429/5xx, you can cause cascading failures. Add jitter and a maximum retry cap with alerts.

Always version prompts and note changes. Use feature flags or traffic splits to safely roll out updates.

Long-running calls can hang threads. Set client and total timeouts; add circuit breakers to protect upstream services.

• Answers only reference provided policy text.
• Returns "Not covered by policy" when retrieval is empty.
• Retries on 429 with exponential backoff.
• Logs are PII-redacted JSON lines.
• Tests run automatically on pull requests.