Encryption At Rest And In Transit

Why this matters

As a Data Architect, you design systems that store and move sensitive data. Encryption at rest protects data on disks, backups, and logs if storage is stolen or misused. Encryption in transit protects data moving between services and users from snooping or tampering. Getting both right is essential for compliance, customer trust, and reducing breach impact.

• Real tasks you will face: choosing algorithms and key lifecycles; enforcing TLS; selecting database vs column encryption; configuring cloud KMS; planning key rotation; validating encryption end-to-end.
• Common data estate touchpoints: data lakes, warehouses, streaming platforms, ETL/ELT jobs, backups, snapshots, caches, and admin tooling.

Concept explained simply

Think of data as packages. At rest: packages locked in a safe (disk/database encryption). In transit: packages in a secure truck (TLS). Keys are the safe combinations and truck access cards. Without strong key management, locks don’t matter.

Mental model

• Encrypt everywhere: storage + network.
• Use strong defaults: AES-256 for data; TLS 1.2+ with modern ciphers for transport.
• Manage keys centrally (KMS/HSM), rotate regularly, restrict who/what can use keys.
• Prefer envelope encryption: a small key (DEK) encrypts data; a master key (CMK) in KMS encrypts the DEK.
• Design for provability: logs, metrics, and automated checks to prove encryption is active.

Design decisions you control

• Scope: Full-disk vs database TDE vs column-level. Use column-level for highly sensitive PII; TDE/disk for broad coverage.
• Key ownership: Provider-managed keys (simpler) vs customer-managed keys (control, audit, per-tenant isolation).
• Rotation strategy: Yearly or semi-annual CMK rotation; event-driven rotation after exposure; rewrap vs re-encrypt trade-offs.
• Transport policy: TLS 1.2+ everywhere, disable TLS 1.0/1.1; consider mTLS inside zero-trust networks.
• Performance: AES-GCM is fast with CPU instructions; encrypting columns adds overhead—measure and cache carefully.
• Observability: Prove TLS negotiated versions; verify encryption flags on storage objects; audit KMS key usage.

Worked examples

Example 1: Data lake (object storage)

1. At rest: Enable server-side encryption with KMS-managed keys (envelope encryption). Use per-environment keys (dev/test/prod).
2. Access: Lock down who can use the CMK. Data access roles do not equal key usage roles (separation of duties).
3. In transit: Enforce HTTPS. Block non-TLS endpoints at network and app layers.
4. Validation: Periodic job lists objects and checks encryption flags; alert on unencrypted objects.

Example 2: Warehouse (e.g., PostgreSQL/Snowflake/BigQuery)

1. At rest: Turn on TDE or platform equivalent. For PII (e.g., SSN, card numbers), add column-level encryption using AES-256-GCM via UDFs or native features.
2. Keys: Store DEKs in KMS; application fetches or requests KMS to wrap/unwrap DEKs. Log key usage.
3. In transit: Require TLS to the warehouse and rotate client certificates. Disable plain connections.
4. Query patterns: Avoid indexing encrypted columns directly; use deterministic encryption only when exact-match search is needed (accepting pattern-leak trade-off).

Example 3: Kafka-like streaming

1. Broker to client: TLS 1.2+ for all listeners. Use mTLS for producers/consumers in internal networks.
2. Broker to broker: TLS for inter-broker traffic. Use SASL for auth if supported.
3. At rest: Encrypt broker disks or topic-level storage using OS/disk encryption.
4. End-to-end: For highly sensitive topics, producers envelope-encrypt payloads; consumers decrypt with permissions.

Example 4: Backups and logs

1. Backups: Encrypt with KMS-managed keys. Store key-id alongside metadata and audit every restore test for encryption present.
2. Logs: Don’t log secrets. If logs contain sensitive fields, apply field-level encryption before shipping and ensure TLS to log sinks.
3. Key escrow and deletion: Document break-glass access; support cryptographic deletion by retiring CMKs for specific tenants or datasets when required.

Who this is for

• Data Architects, Platform Engineers, Security-minded Data Engineers, and Tech Leads designing data platforms.

Prerequisites

• Basic networking (TLS/HTTPS), IAM concepts, and familiarity with one cloud KMS.
• SQL and understanding of your primary data stores (object storage, warehouse, stream).

Learning path

1. Start: Encryption basics (symmetric vs asymmetric, AES-GCM, TLS).
2. Then: Key management with KMS/HSM, envelope encryption, rotation.
3. Apply: Storage-level and database-level encryption patterns.
4. Harden: mTLS, zero-trust network design, observability and audits.
5. Operate: Incident playbooks, break-glass, key rotation drills.

Quick setup steps you can adapt

1. Classify data: tag PII, financial, health data.
2. Pick a key hierarchy: one CMK per environment; optional per-tenant CMKs.
3. Enable at rest encryption everywhere: lake, warehouse, caches, snapshots.
4. Enforce TLS 1.2+ and mTLS where services talk internally.
5. Automate checks: periodic scans of storage encryption flags and TLS metrics.
6. Document rotation cadences and test restores with encryption verification.

Common mistakes and self-check

• Relying only on disk encryption: Sensitive columns remain exposed to DB admins; add column-level encryption where needed.
• Forgetting backups and logs: Unencrypted snapshots or log buckets break compliance.
• Weak transport policies: Allowing old TLS versions or mixed-mode ports.
• Poor key hygiene: Shared keys across environments; no rotation; broad key-usage IAM roles.
• No proof: Lack of periodic evidence that encryption and TLS are active.

Exercises

Do these to make the concepts stick. The quick test is available to everyone; only logged-in users get saved progress.

Exercise ex1: Choose the right controls for a mixed data platform

See the Exercises panel for instructions. Produce a short architecture note with chosen controls and key policies.

Exercise ex2: Define a key hierarchy and rotation plan

Design a CMK/DEK plan, rotation cadence, and audit evidence you will collect.

Exercise ex3: TLS hardening checklist

Create a checklist to enforce and verify TLS 1.2+ and mTLS where appropriate.

Practical projects

• Project 1: Build an encryption posture report that lists all buckets/tables/topics, their encryption status, key IDs, and TLS metrics.
• Project 2: Implement column-level encryption for two PII fields in your warehouse, with a secure UDF or app-layer crypto and a masked analytics view.
• Project 3: Create a key rotation runbook and run a non-prod rotation drill, capturing before/after evidence and downtime impacts.

Mini challenge

Your company onboards a high-compliance customer demanding tenant-isolated keys and mTLS internally. Draft a one-paragraph proposal that adds per-tenant CMKs for their data lake prefixes and warehouse schemas, uses mTLS for internal services touching their data, and describes how you will provide monthly evidence (KMS usage logs and TLS version reports).

Next steps

• Automate posture checks to run daily.
• Expand mTLS coverage to all internal service-to-service channels.
• Plan and test key revocation for a single tenant without affecting others.