Build Versus Buy Decisions

Who this is for

This lesson is for Data Architects, Senior Data/Platform Engineers, and Technical Product Managers who make or influence platform/tooling choices.

Prerequisites

• Basic understanding of data platforms (storage, compute, orchestration, governance)
• Comfort with cost modeling and reading cloud bills
• Awareness of security and compliance needs (PII, retention, encryption)

Why this matters

Choosing whether to build in-house or buy a product affects time-to-value, total cost of ownership (TCO), team focus, risk, and the roadmap. In the Data Architect role, you will:

• Plan platform capabilities (ingestion, transformation, quality, catalog, MDM, observability)
• Estimate cost and effort over 1–3 years and align with budgets
• Reduce delivery risk by selecting sustainable options
• Balance control (build) with speed and support (buy)

Concept explained simply

Build vs Buy is a structured decision: do we create and run a solution ourselves, or purchase/subscribe to a product/service? You compare value, speed, total cost, risk, and control across time.

Mental model: Value–Speed–Control Triangle

• Value: Does this capability differentiate our business or is it a commodity?
• Speed: How fast do we need outcomes? What is the cost of delay?
• Control: How much customization, data residency, and extensibility do we need?

Typically: Buy for commodity needs and speed; Build for differentiating capabilities or when strict control is essential.

A repeatable decision framework

1. Clarify the goal: What problem, who benefits, what outcome metric (e.g., latency, freshness, governance)?
2. Define constraints: Compliance, data residency, SSO, SLAs, budget limits, deadlines, team availability.
3. List options: At least one build path and two buy options (SaaS and/or managed OSS).
4. Score with a simple rubric (1–5: poor to excellent):
   • Time-to-value
   • Requirements fit (features, compliance)
   • Integration complexity
   • Scalability/performance
   • Operational overhead
   • Total cost of ownership (1–3 years)
   • Team capability/availability
   • Vendor risk/lock-in (for buy) or Key-person risk (for build)
   • Strategic alignment (differentiating vs commodity)
5. Estimate TCO: Include licenses/subscriptions, infra, support, training, migration, staffing (FTE), maintenance, and opportunity cost of delay.
6. Decide and document: Chosen option, top risks, mitigation, review date, and exit strategy.

Worked examples (3)

Example 1: Real-time ingestion and stream processing

Scenario: Product analytics needs sub-minute event availability from mobile apps; peak 50k events/sec; go-live in 6 weeks. Options: Build on self-managed Kafka; Buy managed Kafka; Buy a streaming SaaS.

• Time-to-value: Build 2/5, Managed 5/5, SaaS 5/5
• Requirements fit: Build 4/5, Managed 4/5, SaaS 3/5 (limited custom partitioning)
• Integration complexity: Build 2/5, Managed 4/5, SaaS 4/5
• Performance: Build 5/5, Managed 5/5, SaaS 4/5
• Operational overhead: Build 1/5, Managed 4/5, SaaS 5/5
• TCO (3 yrs, rough): Build medium-high, Managed medium, SaaS medium-high
• Team capacity: Build 2/5 (busy), Managed 4/5, SaaS 4/5
• Strategic alignment: Commodity capability → favor buy

Decision: Buy managed Kafka. Rationale: Meets scale, fastest delivery, low ops. Mitigation: Keep events in open formats and enable export to reduce lock-in.

Example 2: Data catalog and governance

Scenario: Regulated enterprise needs lineage, PII tagging, policy-based access, approvals. Options: Build on OSS + custom UI; Buy enterprise catalog; Buy lightweight SaaS catalog.

• Time-to-value: Build 2/5, Enterprise 4/5, Lightweight 4/5
• Requirements fit: Build 3/5, Enterprise 5/5, Lightweight 3/5
• Integration: Build 3/5, Enterprise 4/5, Lightweight 3/5
• Operational overhead: Build 2/5, Enterprise 4/5, Lightweight 4/5
• Risk: Build key-person 2/5, Enterprise vendor 4/5, Lightweight vendor 3/5

Decision: Buy enterprise catalog. Rationale: Strong governance features; low change risk during audits. Mitigation: Contract exit clause and export APIs.

Example 3: Data quality checks for batch pipelines

Scenario: Team needs schema validation, null checks, and SLAs for nightly jobs. Options: Build with open-source library and orchestrator; Buy commercial data quality platform.

• Time-to-value: Build 4/5 (team skilled), Buy 4/5
• Requirements fit: Build 4/5, Buy 5/5 (advanced rules, dashboards)
• TCO (3 yrs): Build low-medium, Buy medium
• Strategic alignment: Quality pipeline is close to core dev workflow → build acceptable

Decision: Build with open-source + minimal hosting. Rationale: Sufficient features, low cost, and skill fit. Mitigation: Document rules and add alerts to reduce key-person risk.

Learning path

1. Learn the Value–Speed–Control triangle and the decision rubric.
2. Practice TCO modeling (1–3 years) including opportunity cost of delay.
3. Run vendor due diligence using the checklist.
4. Pilot one buy and one build in a sandbox; measure time-to-first-value.
5. Standardize decisions using the provided scorecard template.

Exercises

Complete these tasks. A solution is provided in collapsible sections—try first before opening.

Exercise 1 (mirrors ex1): Decision matrix for streaming ingestion

Scenario: Marketing needs real-time event ingestion (up to 15k events/sec), SLA 99.9%, rollout in 8 weeks. Options: Build on self-managed Kafka; Buy managed Kafka.

1. Fill the scorecard (1–5) for: time-to-value, requirements fit, integration complexity, performance, operational overhead, TCO (3 years), team capacity, risk, strategic alignment.
2. Choose Build or Buy and write a 2–3 sentence justification plus top risk and mitigation.

Exercise 2 (mirrors ex2): 3-year TCO mini-calculation

Assume fully-loaded engineer cost = $140k/year (example figure; varies by country/company; treat as rough ranges).

Option A (Buy/SaaS): Subscription $120,000/year; data egress $800/month; implementation 6 weeks at 0.5 FTE for one engineer; delay cost $30,000 per quarter; buy causes 1 quarter delay.

Option B (Build): Infra $700/month; support software $200/month; training (one-time) $6,000; two engineers at 0.3 FTE each for first year; maintenance 0.1 FTE in years 2–3; delay cost $30,000 per quarter; build causes 2 quarters delay.

1. Compute 3-year TCO for each option.
2. Which is cheaper purely on cost?

Exercise completion checklist

• I scored both options using the rubric
• I estimated 3-year TCO including delay cost
• I wrote a concise decision with risks and mitigations

Common mistakes and self-check

• Ignoring opportunity cost of delay. Self-check: Did you monetize delays for each option?
• Comparing year-1 costs only. Self-check: Do you show 1–3 year TCO?
• Underestimating ops burden for build. Self-check: Who will patch/monitor? What FTEs?
• Overlooking vendor egress and per-event overages. Self-check: Did you model volume growth and limits?
• Building differentiating features on top of a weak foundation. Self-check: Can a vendor cover commodity layers so your team focuses on unique logic?
• Not planning an exit strategy. Self-check: How do you export data/configs if you switch?

Practical projects

• Run a 2-week spike: prototype a build option (OSS) and a buy option (trial). Measure time-to-first-event, data freshness, and ops effort.
• Create a 3-year TCO model workbook for one platform area (e.g., quality, catalog, streaming). Include base, best, and worst cases.
• Draft a one-page decision record (ADR) for a recent tooling choice with the scorecard and exit strategy.
• Design a phased approach: buy now for speed, plan a build transition if volumes or customization needs grow.

Next steps

• Apply the rubric to one active decision at work or in a simulated project.
• Review the decision in 3 months: compare predicted vs actual cost and effort; update your scorecard weights if needed.
• Share the template with your team and standardize decision records across the platform domain.

Quick Test

The quick test is available to everyone. If you are logged in, your progress will be saved automatically.

Mini challenge

You have 15 minutes to advise a stakeholder: They need batch-to-near-real-time sync from the data warehouse to CRM for lead scoring within 48 hours, team is at capacity for 2 months. Use the scorecard to decide build vs buy and write 3 sentences explaining the trade-offs and an exit plan.