Model Registry And Artifacts

Why this matters

As an NLP Engineer, you ship models that power user-facing features like search, chat, classification, and content moderation. A model registry and clean artifact management let you:

• Promote models safely from experimentation to production.
• Reproduce any result (same code, data snapshot, tokenizer, and config).
• Rollback fast when metrics drift or bugs appear.
• Track lineage for audits and compliance.

Who this is for

• NLP Engineers moving from notebooks to production.
• Data/ML Engineers building CI/CD for NLP models.
• Scientists who need reliable experiment tracking and reproducibility.

Prerequisites

• Comfort with Python packaging and virtual environments.
• Basic understanding of model training and evaluation for NLP.
• Familiarity with Git and semantic versioning.

Concept explained simply

Think of the model registry as a library catalog for your models. Each "book" (model version) has a unique ID, description, authorship, and location. Artifacts are the files you need to run the model: weights, tokenizer, vocab, configs, label maps, and environment specs.

Mental model

Picture a pipeline with gates:

• Experiment: Many versions appear quickly (v0.1, v0.2...).
• Staging: A few selected versions pass tests and can be A/B tested.
• Production: One or more versions serve traffic (with aliases like "prod" or "current").
• Archive: Old versions, kept immutable for reproducibility and audits.

What to store for NLP models

• Model weights and architecture config.
• Tokenizer + vocab (e.g., BPE merges, SentencePiece model).
• Pre/post-processing code (text normalization, truncation rules, special tokens).
• Label mapping (id→string, string→id) and task schema (e.g., classes, prompts).
• Environment spec (requirements.txt/conda.yaml), Python version, OS/CPU/GPU dependencies.
• Evaluation results (metrics, datasets, slices, confidence intervals).
• Data lineage (dataset version hashes, feature pipeline commit).
• Security and compliance notes (PII handling, forbidden tokens filter if relevant).

Registry structure and naming

Keep versions immutable and human-readable:

• Model name: nlp-sentiment
• Versions: 1, 2, 3 (append build metadata if needed: 3+gpu)
• Aliases: staging, prod, canary, shadow
• Content-addressable artifacts: Store by secure hash to ensure immutability.

Governance and promotion flow

1. Train and log artifacts + metadata.
2. Run automated checks: signature validation, unit tests for preprocess/postprocess.
3. Evaluate on holdout and slice metrics (e.g., short vs. long texts, language variants).
4. Security scan: dependency and license checks.
5. Promote to Staging; run canary/shadow tests.
6. Approve and promote to Production; set alias prod → version N.
7. Monitor; if regressions appear, rollback: prod → version N-1.

Worked examples

Example 1: Register a text classifier

Scenario: You trained nlp-sentiment version 7. You log weights, tokenizer, label map, and metrics. You attach tags language=en, domain=reviews.

• Artifacts: model.safetensors, config.json, tokenizer.json, merges.txt, labels.json
• Metadata: git_commit=ab12cd3, data_version=reviews_v3_hash, framework=torch2.2
• Signature: input: {text: string, max_len: 256}, output: {label: string, score: float}
• Stage: staging

{
  "name": "nlp-sentiment",
  "version": 7,
  "aliases": ["staging"],
  "tags": {"language":"en","domain":"reviews"},
  "artifacts": ["model.safetensors","config.json","tokenizer.json","merges.txt","labels.json"],
  "signature": {
    "inputs": {"text":"string","max_len":"int<=256"},
    "outputs": {"label":"string","score":"float[0,1]"}
  },
  "lineage": {"git_commit":"ab12cd3","data_version":"reviews_v3_hash"},
  "metrics": {"f1":0.915,"latency_p95_ms":18}
}

Example 2: Canary and promotion

Scenario: Compare v7 (candidate) vs v6 (prod). v7 wins on accuracy, slightly slower but within SLO. Promote v7: set alias prod → 7, keep v6 archived with alias prev_prod for quick rollback.

• Before: prod → 6
• After: prod → 7, prev_prod → 6

Example 3: Signature change without breaking clients

Scenario: You add optional field explain=true to request. Keep signature backward compatible: default explain=false. Register as v8; clients that ignore explain continue to work.

• signature_in_v8: inputs: text, max_len, explain?; outputs: label, score, optionally rationale
• Compatibility strategy: optional fields only; avoid changing existing types.

How to build a minimal registry workflow

1. Decide model naming and stages (experiment, staging, prod, archived).
2. Define a signature JSON schema and validate it in CI.
3. Define an artifact manifest (YAML/JSON) listing every file with checksums.
4. Store artifacts immutably (content-addressed paths).
5. Attach metadata: code commit, dataset hash, metrics, tags.
6. Automate promotions with checklists and approvals.

Common mistakes and self-check

• Forgetting tokenizer files → model loads but outputs nonsense. Self-check: verify tokenization round-trip in CI.
• No label map → misaligned outputs. Self-check: assert predicted id maps to expected label names.
• Mutable artifacts → "works on my machine" issues. Self-check: enforce checksums and content-addressable storage.
• Untracked preprocessing → silent accuracy drops. Self-check: unit-test preprocess/postprocess scripts with fixtures.
• Breaking signature changes → client outages. Self-check: contract tests against the signature schema.

Exercises

Do these now to make the ideas stick. Everyone can take the test; only logged-in users get saved progress.

Practical projects

• Package a small text classifier with full manifest, then simulate a promotion to staging.
• Add slice metrics by text length and language variant; store in metrics.json.
• Create a backward-compatible signature update and test it with a dummy client.

Learning path

1. Artifacts and signatures (this lesson)
2. Model evaluation and monitoring
3. CI/CD for training and serving
4. Rollbacks, canaries, and shadow deployments

Next steps

• Automate manifest generation in your training pipeline.
• Add schema validation to CI to prevent breaking changes.
• Adopt aliases for instant promotions and rollbacks.

Mini challenge

Your team wants to add support for multilingual inputs next quarter. Propose three tags and two signature updates that keep current clients working, and list one new slice metric you would add for fairness.