Artifact Storage And Retention

Why this matters

MLOps Engineers keep models, datasets, and training outputs safe, reproducible, and cost-effective. Solid artifact storage and retention policies let teams:

• Reproduce any production model version on demand.
• Control storage costs with lifecycle rules instead of manual cleanup.
• Meet audit, security, and regulatory requirements.
• Prevent accidental deletions and data leaks.
• Speed up delivery by making artifacts easy to find and reuse.

Concept explained simply

Artifacts are the files produced and used by ML workflows: trained models, datasets, feature stores snapshots/exports, metrics, plots, environment files (conda.yaml/requirements.txt), training code bundles, and inference packages.

Artifact storage is where these files live (usually object storage). Retention is the set of rules for how long to keep them and when to archive or delete them.

Core concepts

Worked examples

Example 1: Stage-aware retention policy

Goal: Large experimentation but small budget.

• Dev/Experiment runs: keep 14 days; keep latest 5 versions per model; delete failed runs after 3 days.
• Staging: keep 60 days; keep latest 10 versions per model.
• Production: keep 365 days minimum; keep all promoted releases; archive after 365 days (no delete) to cold storage; enable immutability.

{
  "dev": {"ttl_days": 14, "keep_last_versions": 5, "failed_ttl_days": 3},
  "staging": {"ttl_days": 60, "keep_last_versions": 10},
  "production": {"ttl_days": 365, "archive": true, "immutable": true}
}

Example 2: Object storage lifecycle rules (pseudo)

{
  "rules": [
    {"filter": {"prefix": "ml/dev/"}, "transition": {"days": 14, "storage_class": "COLD"}, "expire": {"days": 30}},
    {"filter": {"prefix": "ml/staging/"}, "transition": {"days": 60, "storage_class": "COLD"}, "expire": {"days": 120}},
    {"filter": {"prefix": "ml/prod/"}, "transition": {"days": 365, "storage_class": "ARCHIVE"}, "lock": {"mode": "WORM", "retain_days": 365}}
  ]
}

Note: Use tags like stage=dev|staging|prod to target rules more flexibly than folder names.

Example 3: Registry cleanup logic

For each model name:

1. Keep all versions with stage=Production.
2. For stage=Staging, keep last 10; delete older if also older than 60 days.
3. For stage=None/Archived, keep last 5 if accessed in past 30 days; otherwise delete.

Always verify that underlying artifact files aren’t referenced by another model/version before deletion.

Example 4: Orphan detection

1. List all artifact files under ml/.
2. Build a set of referenced artifact IDs from registry metadata.
3. Diff to find unreferenced (orphans).
4. Quarantine orphans for 7 days (rename or move), then delete unless referenced.

Step-by-step: Set up storage and retention

1. Define naming: org/project/model/stage/version/run_id/
2. Enable object versioning and default encryption (KMS-managed keys).
3. Decide tags: stage, model, owner, data_sensitivity, ttl_class.
4. Create lifecycle rules for dev/staging/prod, including transitions and expiry.
5. Set IAM roles: ci-writer (write dev/staging), prod-writer (limited), inference-reader (read prod), auditor (read + logs).
6. Implement manifests with checksums (SHA256) and sizes for each artifact package.
7. Schedule pruning jobs and orphan sweeps (e.g., daily).
8. Enable access logs and periodic cost review.

Exercises you can do now

Do these exercises and then check your work below. The Quick Test at the end is available to everyone; only logged-in users will have their progress saved.

1. Exercise 1: Design a stage-aware retention policy YAML for a team with heavy experiments, modest staging, and strict production retention. Include TTL, version limits, and archive rules.
2. Exercise 2: Estimate storage savings when moving 2 TB of staging artifacts from hot to cold storage after 60 days (assume 60% cost reduction), and deleting 40% of old dev runs after 30 days.

Common mistakes and how to self-check

• No versioning: Without versioning, rollbacks are fragile. Self-check: verify older object versions exist for a sample artifact.
• Over-retaining experiments: Costs creep up. Self-check: chart storage by prefix and age; ensure lifecycle rules hit most dev artifacts.
• Deleting referenced files: Breaks reproducibility. Self-check: compare registry references to files before any deletion.
• Weak tagging: Hard to audit or target rules. Self-check: pick 10 artifacts; confirm tags cover stage, model, owner.
• Skipping immutability for releases: Risks tampering. Self-check: attempt to modify a production artifact; it should fail when locked.

Practical projects

• Build a simulated ML project bucket with dev/staging/prod prefixes, tags, manifests, and lifecycle JSON; run a dry-run pruning script.
• Create an artifact packaging template: manifest + checksum + environment files; test validation on CI.
• Implement an orphan quarantine workflow: move, tag as quarantine=true, delete after 7 days if still unreferenced.

Who this is for and prerequisites

Learning path

• Before: Model versioning basics, experiment tracking.
• Now: Artifact storage and retention (this lesson).
• Next: Promotion workflows, model serving packaging, and automated cleanup jobs integrated with your registry.

Mini challenge

In 10 minutes, draft three lifecycle rules: one for dev (short TTL), one for staging (medium TTL + transition), and one for prod (immutability + archive). Add tags you would rely on. Keep it under 20 lines of JSON or YAML.

Next steps

• Convert your chosen examples into your environment’s configuration format.
• Pilot on a non-critical project; review costs after 2 weeks.
• Run the Quick Test below to validate your understanding.

Note: The test is available to everyone. Only logged-in users will have their progress saved.