luvv to helpDiscover the Best Free Online Tools
Topic 3 of 8

Schema Evolution Handling

Learn Schema Evolution Handling for free with explanations, exercises, and a quick test (for Data Engineer).

Published: January 8, 2026 | Updated: January 8, 2026

Table of Contents

Why this matters

Real systems change constantly. Product teams add fields, rename attributes, or move IDs. Without good schema evolution, your pipelines break, dashboards go blank, and backfills become nightmares. As a Data Engineer, you’ll routinely:

  • Introduce a new optional field in a Kafka topic without breaking existing consumers.
  • Backfill historical data to match a new table schema.
  • Adopt a third-party API payload change while keeping downstream jobs stable.
  • Roll out a column rename in a Delta/Iceberg table with zero downtime.
  • Set and enforce compatibility modes in a Schema Registry.
What success looks like
  • New fields appear for consumers that can use them; old consumers keep running.
  • Historical data remains readable by new jobs.
  • You can prove compatibility before deploying.

Concept explained simply

A schema is a contract describing your data’s shape. Schema evolution is how you change that contract safely while old and new data co-exist.

  • Backward compatible: New readers can read old data. (Add field with default; new reader uses default when old data lacks the field.)
  • Forward compatible: Old readers can read new data. (Old reader ignores extra fields it doesn’t know.)
  • Fully compatible: Both backward and forward. Safest for long-lived systems.

Mental model: One stream, many versions

Imagine a mailbox filled with envelopes of slightly different layouts (versions). Your reader must still find the essentials every time. Evolution ensures both new and old envelopes are readable by the same or different readers.

Formats and where evolution happens
  • Avro/Protobuf with Schema Registry: contracts are enforced at write/read.
  • JSON: often enforced by convention; carry a schema_version and validate in code.
  • Parquet/ORC tables (Delta/Iceberg/Hudi): table metadata stores schema; engines manage evolution.

Core rules you’ll use on the job

  • Adding a field: Make it optional and provide a default. This is backward compatible; often forward compatible too.
  • Renaming a field: Prefer add-new-field, dual-write, backfill, then deprecate the old. If using Avro, set aliases; with table formats, use supported RENAME COLUMN to avoid full rewrites.
  • Type changes: Narrowing types is risky (long → int). Some widenings can be safe (int → long) depending on format/tooling.
  • Enums: Adding symbols may be compatible; removing or reusing values is risky.
  • JSON: Without a registry, compatibility is discipline-driven. Include schema_version and validate on read.
  • Tables: Adding nullable columns is easy. Changing partition columns or dropping required columns is breaking.
  • Compatibility modes: BACKWARD, FORWARD, FULL (and transitive variants). FULL is the safest default.

Worked examples

1) Kafka + Avro: add an optional field

Scenario

You have a purchase event. You need to add currency.

Old writer schema (simplified):

{"type":"record","name":"Purchase","fields":[{"name":"purchase_id","type":"string"},{"name":"amount","type":"double"}]}

New writer schema (backward and forward compatible):

{"type":"record","name":"Purchase","fields":[{"name":"purchase_id","type":"string"},{"name":"amount","type":"double"},{"name":"currency","type":["null","string"],"default":null}]}
  • Set registry to FULL (or at least BACKWARD). Old data reads fine (default used). Old consumers ignore the new field.

2) JSON events with versioning

Scenario

You emit JSON without a registry.

Envelope pattern:

{"schema_version":2,"event":{"purchase_id":"p-1","amount":25.0,"currency":"USD"}}
  • Readers switch behavior based on schema_version.
  • Deploy plan: writers start emitting v2; readers support v1 and v2 for a transition period.

3) Delta/Iceberg table: add, rename, and widen

Scenario

You manage a Parquet-backed table with a lakehouse format.

  • Add a nullable column: ALTER TABLE sales ADD COLUMN currency STRING (or equivalent). Existing files remain readable; new files include the column.
  • Rename safely: Use native RENAME COLUMN if supported. Consumers should read by column name, not position.
  • Type widening: INT → BIGINT may be supported; verify engine capabilities and backfill if needed.
  • Avoid breaking ops: Changing partitioning often requires rewrite/backfill.

4) Handling a breaking rename

Playbook
  1. Add new field account_id while keeping user_id.
  2. Dual-write both fields.
  3. Backfill historical data for account_id.
  4. Flip readers to account_id.
  5. Stop writing user_id; later, remove it with a planned deprecation window.

Hands-on exercises

Do these now. They mirror the exercises below this lesson and help lock in the mental model.

  1. Avro evolution drill: Add currency to a Purchase record without breaking old consumers. Specify compatibility and defaults.
  2. Lakehouse rename plan: Rename device_id to client_id on a large table with no downtime. Show steps for readers and writers.
Readiness checklist
  • I can explain backward vs forward vs full compatibility.
  • I know how to add a field safely in Avro/Protobuf/JSON.
  • I can plan a zero-downtime column rename in a Delta/Iceberg table.
  • I can choose a compatibility mode and justify it.

Common mistakes and self-check

  • Assuming JSON is “schemaless.” Fix: Embed schema_version and validate on read.
  • Dropping or renaming fields abruptly. Fix: Dual-write and deprecate with a window; use aliases/rename support.
  • Type narrowing. Fix: Prefer widening or create a new field.
  • Ignoring partitions. Fix: Treat partition changes as breaking; plan rewrites.
  • Not testing compatibility pre-deploy. Fix: Use schema registry checks or offline data validation before rollout.
Self-check prompts
  • Can my newest reader still parse last year’s files?
  • What will an old consumer do when it sees my new field?
  • Do I have a rollback plan if a change turns out breaking?

Practical projects

  • Versioned events sandbox: Produce purchases in v1 and v2 (with a new field). Build a consumer that reads both.
  • Table evolution lab: Start with a small Delta/Iceberg table; add a column, rename one, and widen a type. Validate queries before/after.
  • Deprecation workflow: Simulate dual-write, backfill, then switch readers and remove the old field after a deprecation window.

Mini challenge

Your partner team wants to replace price (double) with amount_cents (long) across stream and table. Design a rollout that keeps dashboards green. Include: dual-write plan, backfill, compatibility mode, and when each consumer flips.

Who this is for

  • Data Engineers building streaming and batch pipelines.
  • Analytics Engineers maintaining stable tables consumed by BI tools.
  • Backend Engineers owning event schemas.

Prerequisites

  • Basic familiarity with Avro/Protobuf/JSON and Parquet.
  • Comfort with SQL and a lakehouse engine (e.g., Spark/Trino or similar).
  • Understanding of producers/consumers and table reads/writes.

Learning path

  1. Review compatibility concepts (backward, forward, full).
  2. Practice safe field additions and type widening.
  3. Learn zero-downtime rename and deprecation flows.
  4. Apply to both streams (Avro/Protobuf/JSON) and tables (Delta/Iceberg/Hudi).
  5. Automate checks in CI (schema validation, contract tests).

Next steps

  • Introduce automated compatibility checks in your pipeline CI.
  • Add data quality assertions to catch unintended breaks early.
  • Document deprecation timelines and communicate widely.

Quick Test: You can take the test below. Anyone can try it; if you’re logged in, your progress will be saved.

Practice Exercises

2 exercises to complete

Instructions

You maintain an Avro-encoded Kafka topic purchase with fields purchase_id (string) and amount (double). Product asks to add currency (string). Make this change without breaking any existing consumer.

  • Give the new schema snippet.
  • Choose a compatibility mode and justify it.
  • Explain what old and new consumers will observe.
Expected Output
A new schema that adds currency as nullable with a default, with FULL (or at least BACKWARD) compatibility and an explanation of consumer behavior.

Schema Evolution Handling — Quick Test

Test your knowledge with 10 questions. Pass with 70% or higher.

10 questions70% to pass
API Ingestion PatternsBatch File Ingestion

Have questions about Schema Evolution Handling?

AI Assistant

Ask questions about this tool