Who this is for
This lesson is for Data Platform Engineers who design, build, or secure data lakes, warehouses, streaming systems, and orchestration in cloud or hybrid environments.
Prerequisites
- Basic networking: IP/CIDR, routing, DNS.
- Cloud fundamentals: virtual networks/VPCs, subnets, security groups/NSGs, firewalls.
- Data platform basics: object storage, warehouses, orchestration tools.
Why this matters
Real tasks you will face:
- Expose a data warehouse only on private networking, while allowing scheduled jobs to reach it.
- Ingest on-prem data into a cloud data lake over private links with predictable latency.
- Segment dev/stage/prod so a misconfigured job in dev cannot impact prod.
- Restrict egress so batch jobs can only reach approved services, not the public internet.
Concept explained simply
Network segmentation means breaking your network into smaller zones with clear, minimal connections between them. Private connectivity means connecting those zones and services without exposing them to the public internet.
Mental model
Think of your platform as a campus with buildings (subnets) connected by controlled hallways (route tables) and guarded doors (security groups/firewalls). Private connectivity is an internal shuttle system that never drives on public roads.
Core building blocks
Virtual networks and subnets
- Virtual network/VPC: your private IP space in the cloud.
- Subnets: IP ranges for specific tiers (ingest, compute, storage endpoints, admin).
- Route tables: define where traffic can go (local, peering/transit, endpoints, NAT).
Traffic controls
- Security groups/NSGs: stateful, instance-level rules. Allow only what you need.
- Network ACLs: stateless, subnet-level allow/deny lists. Coarse-grained guardrails.
- Firewalls: central policy enforcement, L3/L4/L7 control and egress filtering.
Private service access
- Private service endpoints: expose managed services privately within your network.
- Private DNS: resolves service names to private IPs within your network.
- Service perimeters: restrict access to storage/warehouse APIs to your network.
Network interconnect
- VPC/VNet peering: connect two networks in the same cloud provider.
- Transit hubs/gateways: central hub for many networks, simpler routing and policy.
- Site-to-site VPN: encrypted tunnel over the internet.
- Dedicated private circuits: provider-managed links for stable bandwidth/latency.
Egress control
- NAT gateways: outbound-only internet access. Use sparingly; prefer private endpoints.
- Proxy/firewall: inspect and restrict outbound traffic with allowlists.
- Block default internet egress; allow only required destinations.
Common patterns
- Hub-and-spoke: hub has shared services and controls; spokes host workloads.
- Environment isolation: separate networks for dev/stage/prod; no transitive trust.
- Zero trust: default deny, least privilege, continuous verification.
Worked examples
Example 1: Private data lake and compute
- Create a hub network with firewall and private DNS. Spokes for ingest, compute, and data services.
- Expose object storage and warehouse via private service endpoints in the data services subnet.
- Allow compute subnet to reach endpoints over private IP only. Deny any internet egress from compute.
- Result: jobs run privately, storage and warehouse are never publicly reachable.
Example 2: Hybrid ingest from on‑prem
- Connect on-prem to cloud via site-to-site VPN (start) or dedicated circuit (scale/latency).
- On-prem source subnets route to cloud ingest subnet through the tunnel only.
- Cloud ingest subnet writes to storage via private endpoint; firewall blocks internet egress.
- Result: data flows privately end-to-end with predictable control points.
Example 3: Private access to a managed warehouse
- Enable a private service endpoint in a dedicated subnet for the warehouse account.
- Associate a private DNS zone so warehouse hostname resolves to private IP.
- Restrict security groups to allow only orchestrators and BI tools in approved subnets.
- Result: users and jobs connect privately; public interface disabled or blocked.
Hands-on plan (cloud-agnostic)
- Plan IP space: Pick a CIDR that won’t overlap with on‑prem (e.g., 10.40.0.0/16). Reserve ranges per environment.
- Create subnets: ingest (10.40.1.0/24), compute (10.40.2.0/24), data-services/endpoints (10.40.3.0/24), admin/bastion (10.40.4.0/24).
- Attach route tables: local routes within VPC; routes to private endpoints via the data-services subnet; no default route to the internet for compute/ingest.
- Security groups: allow only required ports from specific subnets (e.g., 443 from compute to data-services).
- Private endpoints: enable private access for storage and warehouse; bind private DNS zones to your VPC.
- Egress policy: deny all; add allow rules only for approved destinations via firewall/proxy. Avoid NAT unless absolutely required.
- Test: From compute, resolve service hostnames (should resolve to private IPs) and connect over 443; confirm public access is blocked.
Troubleshooting tips
- Private DNS misconfigured: names resolve to public IPs; bind private zone to VPC.
- Route gaps: packets take NAT/IGW path; remove default 0.0.0.0/0 routes from restricted subnets.
- Security group asymmetry: stateful rules simplify returns—check both inbound and outbound as needed.
Exercises
Do these to lock in learning. Then check your work with the solutions.
Exercise 1 — Segment your platform
Design dev/stage/prod segmentation for a data platform with object storage, a warehouse, orchestration, and a bastion. Include CIDR ranges, subnets, and allowed flows.
- Deliverable: a short text plan listing subnets, routes, and security rules.
Checklist
- No overlapping CIDRs across environments
- Private endpoints for storage/warehouse
- No internet egress from compute/ingest
- Bastion/admin isolated and audited
Exercise 2 — Pick the right connectivity
Map scenarios to connectivity methods: peering, transit hub, VPN, dedicated circuit, private service endpoint.
- Deliverable: scenario → method, with one-line justification each.
Exercise 3 — Minimal rules
Write a minimal set of security group/NSG rules allowing: orchestrator → warehouse (443), compute → storage (443), deny everything else.
- Deliverable: inbound/outbound rules listing sources, destinations, ports.
Self-check
- Only specific subnets can reach 443
- No broad 0.0.0.0/0 allows
- Return traffic considered (stateful)
Common mistakes and how to self-check
- Overlapping CIDRs: breaks peering and on‑prem routes. Self-check: document all ranges; run an overlap tool or manual review.
- Relying on NAT for everything: increases exposure. Self-check: list all egress; replace with private endpoints where possible.
- Public DNS leakage: resolves to public IPs. Self-check: dig/nslookup from compute; must show private IPs.
- Flat networks: dev can reach prod. Self-check: attempt a ping/curl between envs should fail by default.
- Too-permissive rules: 0.0.0.0/0 source/dest. Self-check: every rule must name a specific subnet or security group.
Practical projects
- Private-only data pipeline: build a small ETL that writes to object storage via a private endpoint; verify no internet path exists.
- Flow logs review: enable flow logs, export to a table, and write a query that flags any non-private egress.
- Hub-and-spoke pilot: create a transit hub with a spoke for compute and a spoke for data services; apply egress policy in the hub.
Learning path
- Review IP planning and subnetting.
- Build a hub-and-spoke skeleton with no internet egress.
- Add private endpoints and private DNS; verify resolution.
- Connect on‑prem via VPN; later evaluate a dedicated circuit.
- Add egress controls and logging; iterate least-privilege rules.
Next steps
- Harden egress: catalog all required destinations; implement explicit allowlists.
- Automate: codify networking as templates to avoid drift.
- Continuously test: add connectivity tests to your CI for each environment.
Mini challenge
Lock down a single job so it can only reach one warehouse endpoint. Provide the exact subnets, ports, and rules you would apply, plus how you would test and monitor it.
Quick Test
Take the quick test below. Note: The test is available to everyone; only logged-in users get saved progress.