Who this is for
- Data Platform Engineers who enable product and data teams to create reliable pipelines without direct platform intervention.
- Tech leads designing platform "golden paths" for ingestion, transformation, and backfills.
Prerequisites
- Basic orchestrator knowledge (e.g., DAGs, tasks, schedules, retries).
- Comfort with YAML/JSON, version control, and CI pipelines.
- Understanding of data sources, warehouses/lakes, and data quality checks.
- Familiarity with secrets management and environment promotion (dev/test/prod).
Why this matters
In many organizations, data teams wait days or weeks for platform engineers to create pipelines. Self-serve patterns remove the bottleneck while keeping safety and compliance. You will:
- Reduce lead time: users can create a pipeline in minutes via a template or form.
- Enforce guardrails: policies, validation, and cost limits are applied automatically.
- Standardize: naming, ownership, monitoring, and SLAs are consistent.
- Scale: platform teams focus on platform improvements, not one-off pipeline requests.
Concept explained simply
Self-serve pipeline creation means users describe what they need at a high level (source, destination, schedule, checks), and the platform turns it into a runnable, governed pipeline automatically. Think of it as a "golden path" vending machine that dispenses production-grade pipelines.
Mental model
Picture a vending machine with guardrails. A user selects: source type, destination, transformations, SLA, and tags. The platform validates the selection, assembles a standardized DAG from templates, runs a dry-run, opens a change for review, and then deploys. Ownership, alerts, and costs are wired in by default.
Core self-serve patterns
1) Golden path templates + scaffolding
- Pre-approved templates for common use cases (batch ingestion, CDC, dbt runs, feature pipelines).
- Scaffold code/config and folder structure with sensible defaults and naming.
2) Declarative pipeline spec (YAML/JSON)
- User submits a spec describing source, destination, schedule, owner, and checks.
- Platform compiles spec into orchestrator tasks with consistent conventions.
3) DAG factory / code generation
- Functions that translate spec fields into tasks, dependencies, retries, and resources.
- Versioned to enable safe upgrades and rollbacks.
4) Connector registry
- Catalog of validated sources/destinations with required fields and supported modes.
- Prevents ad-hoc connectors and enforces security patterns.
5) GitOps / PR-based creation
- Specs and generated artifacts live in a repo.
- CI validates, tests, and previews changes before deployment.
6) Guardrails-by-default
- Schema validation (JSON Schema), linting, and policy-as-code.
- Budget and concurrency caps, minimum data quality checks, and required ownership.
7) Metadata-driven scheduling and runtime
- Schedules, SLAs, retries, and resources set via metadata fields.
- Labels/tags feed observability and cost allocation.
8) Environments and promotion
- Automatic dev/test/prod flows with approvals and smoke tests.
- Ephemeral dry-runs or sandbox executions to catch issues early.
9) Versioning and deprecation
- Template/spec version pinning; safe migrations with codemods.
- Rollback paths and deprecation notices with deadlines.
10) Ownership, alerts, and docs
- Owner and on-call contacts required.
- Auto-generated runbooks and dependency diagrams from spec.
Reference flow (step-by-step)
- User picks a golden path and fills a form or YAML/JSON spec.
- Validation: schema + policy checks (naming, SLAs, budgets, secrets references).
- Scaffolding: repo structure, config, and CI pipeline are generated.
- Dry-run: compile DAG and run in a sandbox with sample or limited data.
- PR raised: reviewers check diffs, CI checks pass.
- Deploy on merge: orchestrator registers the pipeline; alerts and dashboards wired.
- Promote: after checks in test, promote to prod with controlled parameters.
- Operate: backfills, replays, and teardown supported via the same spec.
Worked examples
Example 1: Daily batch ingestion (file store to warehouse)
Minimal declarative spec and what it becomes.
version: 1
id: sales_daily_ingestion
owner: team-analytics
source:
type: file
format: parquet
path: s3://raw/sales/dt={{ ds }}
destination:
type: warehouse
dataset: analytics
table: sales_raw
schedule:
cron: "0 2 * * *" # daily at 02:00
runtime:
retries: 2
retry_delay_minutes: 10
quality:
checks:
- type: row_count_greater_than
threshold: 0
observability:
sla_minutes: 60
alerts:
on_failure: team-analytics-oncall
DAG produced: extract_file -> stage -> quality_check -> load_to_warehouse -> mark_sla.
Example 2: Event-driven transform (refresh on upstream change)
version: 1
id: customers_model_refresh
owner: team-analytics
triggers:
on_artifact_update:
artifact: raw.customers
transform:
type: sql_model
refs: [raw.customers]
destination:
type: warehouse
dataset: marts
table: dim_customers
runtime:
resources: { profile: medium }
observability:
alerts: { on_failure: team-analytics-oncall }
DAG: wait_for_artifact -> run_transform -> test_model -> publish -> notify.
Example 3: Backfill with partition parameters
version: 1
id: sales_backfill
owner: team-analytics
mode: backfill
partitions:
field: dt
start: 2024-01-01
end: 2024-01-31
max_concurrency: 4
uses: sales_daily_ingestion # reference existing pipeline
DAG factory generates parameterized runs per date with max 4 concurrent tasks, reusing the same steps as the daily pipeline.
Exercises
Try these. Open the dropdowns to check possible solutions. Your solutions can vary; aim for correctness, safety, and clarity.
Exercise 1: Design a minimal self-serve batch pipeline spec
Create a YAML spec to ingest from a Postgres table to a warehouse daily at 01:00, with a basic row count check. Include: id, owner, source (type, table, connection ref), destination (dataset, table), schedule, runtime retries, quality check, and an alert contact.
Show solution
version: 1
id: pg_orders_daily
owner: team-core-ops
source:
type: postgres
connection: secret://pg-prod
table: public.orders
extract:
mode: full
columns: [order_id, customer_id, amount, created_at]
destination:
type: warehouse
dataset: staging
table: orders_raw
schedule:
cron: "0 1 * * *"
runtime:
retries: 2
retry_delay_minutes: 5
quality:
checks:
- type: row_count_greater_than
threshold: 0
observability:
sla_minutes: 45
alerts:
on_failure: core-ops-oncall
metadata:
tags: [source:postgres, domain:orders]
Exercise 2: Map spec fields to DAG tasks
Using the spec from Exercise 1, list the DAG tasks in order and show key dependencies. Include at least: extract, stage, load, quality check, and notify on failure.
Show solution
- extract_postgres (uses connection ref, selects columns)
- stage_temp (writes raw batch to staging area)
- quality_rowcount (assert > 0 before load)
- load_to_warehouse (upserts into staging.orders_raw)
- mark_sla (records run completion for SLA)
- notify_on_failure (triggered on any upstream failure)
Dependencies: extract_postgres -> stage_temp -> quality_rowcount -> load_to_warehouse -> mark_sla. notify_on_failure is downstream of a failure handler or event from any prior task.
Checklist before you move on
- Spec includes owner, alerts, and SLA minutes.
- Secrets are referenced, not embedded.
- Quality checks exist and run before loads.
- Schedules and retries are explicit.
- Template version or spec version is set.
Common mistakes and self-check
- Missing ownership or on-call contact. Self-check: does every pipeline have a contact and escalation path?
- Embedding secrets in config. Self-check: are all credentials references to a secrets manager?
- No data quality checks. Self-check: is there at least a basic row-count or schema check?
- Unbounded costs (no concurrency or retry caps). Self-check: are budgets/concurrency limited?
- Skipping PR review. Self-check: do all changes go through CI validation and review?
- Template drift. Self-check: are templates versioned and pinned in the spec?
Practical projects
- Project 1: Build a cookiecutter-like scaffold that generates a repo with a pipeline spec, CI config, and a sample DAG factory. Acceptance: running "create" produces a valid repo, CI runs validation, and a sandbox dry-run succeeds.
- Project 2: Write a JSON Schema for your pipeline YAML and a linter that enforces naming, ownership, and check requirements. Acceptance: invalid specs fail CI with clear messages.
- Project 3: Backfill tool. Provide start/end parameters that enqueue partitioned runs via the orchestrator with a max concurrency flag. Acceptance: safe parallelism and resumable runs.
Learning path
- Before this: basics of orchestration, CI, secrets management, and data quality.
- Now: self-serve patterns, guardrails, and golden paths.
- Next: observability (SLIs/SLOs), governance/policy-as-code, cost controls, and multi-tenant isolation.
Next steps
- Introduce template version pinning and a migration playbook.
- Add a dry-run mode that executes a sample partition with synthetic data.
- Publish a user guide: how to choose a template, required fields, and how to request new connectors.
Mini challenge
Draft a template spec for a CDC ingestion golden path. Include: connector reference, change tracking method, ordering/keys, destination merge strategy, schedule or trigger, quality checks, backfill support, resource profile, and owner. Keep it under 60 lines and ensure no secrets are embedded.
Before the quick test
The quick test below is available to everyone. Only logged-in users will have their progress saved.