Technology Selection Criteria

Why this matters

Choosing the right data technology is a high-impact decision. It affects cost, performance, reliability, team productivity, and how quickly the business can launch features. As a Data Architect, you will evaluate options like streaming platforms, warehouses, lakes, orchestration tools, and governance solutions. Clear selection criteria help you make defensible, evidence-based choices that age well.

• Real task: Pick a streaming backbone that meets a 200 ms SLA for product analytics.
• Real task: Migrate from legacy ETL to a modern orchestration tool with minimal downtime.
• Real task: Select a warehouse/lakehouse that balances concurrency, cost, and governance.

Concept explained simply

Technology selection criteria are the questions you use to judge whether a tool fits your business and technical needs. You define what matters, score each option, weigh the scores by importance, and choose the option with the highest evidence-backed value.

Mental model

Think of it as a three-layer fit:

• Business fit: Does it solve the stakeholder problem and unlock measurable value?
• Technical fit: Does it meet functional and non-functional requirements (latency, scale, reliability)?
• Operational fit: Can your team build, run, secure, and govern it cost-effectively?

Core criteria and how to score

Use weights (importance) and scores (1–5) to build a concise, defensible decision.

Common criteria (adapt as needed)

• Business value impact
• Functional fit (features, semantics)
• Performance/latency and throughput
• Scalability and elasticity
• Reliability/availability and recovery
• Security, privacy, and compliance
• Data governance features (lineage, catalog, access control)
• Integration and interoperability (APIs, connectors, formats)
• Operability and support (monitoring, SRE, tooling)
• Team skills and learning curve
• TCO and pricing predictability
• Maturity, ecosystem, community
• Vendor lock-in risk and portability
• Multi-cloud/hybrid fit

Simple scoring framework

1. List criteria and assign weights (1=low, 5=critical).
2. Score each option per criterion (1=poor, 5=excellent) based on evidence.
3. Compute weighted score per option: sum(weight × score).
4. Run a sensitivity check: vary top weights to see if the winner changes.

Keep your matrix small (8–12 criteria) to stay focused.

Worked examples

Example 1 — Real-time analytics backbone

Decision: Choose a managed streaming platform for p95 < 250 ms, 50k events/sec, exactly-once, budget-conscious.

• Weights: Latency 5, Scalability 4, Exactly-once 4, Operability 3, Cost predictability 3, Ecosystem 2
• Options: Managed Kafka, Cloud-native Stream A, Cloud-native Stream B

Sketch scoring (illustrative):

• Managed Kafka: 5,4,5,4,3,4 → weighted sum ≈ 5*5 + 4*4 + 4*5 + 3*4 + 3*3 + 2*4 = 25 + 16 + 20 + 12 + 9 + 8 = 90
• Stream A: 5,4,4,5,4,4 → 25 + 16 + 16 + 15 + 12 + 8 = 92
• Stream B: 4,5,4,4,4,3 → 20 + 20 + 16 + 12 + 12 + 6 = 86

Outcome: Stream A narrowly wins on operability and predictable cost. If latency weight drops to 3 (sensitivity), Managed Kafka may tie. Document your rationale.

Example 2 — Warehouse vs lakehouse on a budget

Decision: Marketing analytics, heavy BI concurrency, cost cap, SQL-first, strong governance.

• Weights: Concurrency 5, Cost 4, Governance 4, Performance 3, Integration 3, Lock-in risk 2
• Options: Warehouse X, Warehouse Y, Lakehouse Z

Sketch scoring:

• Warehouse X: 5,4,4,4,4,2 → 25 + 16 + 16 + 12 + 12 + 4 = 85
• Warehouse Y: 4,5,3,4,4,3 → 20 + 20 + 12 + 12 + 12 + 6 = 82
• Lakehouse Z: 4,4,4,3,5,4 → 20 + 16 + 16 + 9 + 15 + 8 = 84

Outcome: Warehouse X slightly leads for BI concurrency and governance. Lakehouse Z is close for integration/portability; if lock-in weight increases, Z could win.

Example 3 — Feature serving store for ML

Decision: Sub-20 ms reads, 99.9% availability, multi-region DR, simple ops.

• Weights: Latency 5, Availability 4, Operability 4, Cost 3, Integration 3
• Options: Managed in-memory store, Wide-column store, DIY on VMs

Sketch scoring:

• Managed in-memory: 5,4,5,3,4 → 25 + 16 + 20 + 9 + 12 = 82
• Wide-column: 3,4,3,4,3 → 15 + 16 + 12 + 12 + 9 = 64
• DIY VMs: 4,3,2,4,2 → 20 + 12 + 8 + 12 + 6 = 58

Outcome: Managed in-memory store wins due to latency and low ops burden.

Step-by-step selection process

1. Frame the decision: problem statement, scope, success metrics.
2. Capture constraints: SLAs, compliance, regions, budget caps, timelines.
3. Shortlist 2–4 viable options (hard filter on must-haves).
4. Define 8–12 criteria and weights with stakeholders.
5. Collect evidence: docs, pilots, benchmarks, references.
6. Score and compute weighted sums.
7. Run sensitivity and risk analysis (what could go wrong?).
8. Decide, document, and set review checkpoints.

Exercise

Use the scenario below. Then check the solution.

Exercise 1 — Choose a stream processing framework

Decision: Process 15k events/sec with exactly-once semantics, windowed aggregations, p95 < 800 ms end-to-end, minimal ops.

• Options: Engine A (Flink-like), Engine B (Spark Streaming-like), Engine C (Kafka-Streams-like)
• Weights: Exactly-once 5, Latency 4, Operability 4, Stateful windows 4, Ecosystem 3, Cost predictability 3

Task:

• Assign a 1–5 score for each option per criterion based on your assumptions.
• Compute weighted sums and pick a winner.
• Write one-paragraph rationale and one risk to monitor.

Expected output: A chosen engine, weighted table (brief), and rationale.

• I used weights × scores and showed sums
• I wrote a one-paragraph rationale
• I identified a concrete risk and mitigation

Common mistakes and self-check

• Picking by popularity instead of requirements → Self-check: Do you have a written decision statement and criteria?
• Overweighting one benchmark → Self-check: Did you test with your data and workload?
• Ignoring TCO → Self-check: Did you include ops time, training, support, and data egress?
• Underestimating governance → Self-check: Are lineage, PII handling, and audit covered?
• No sensitivity analysis → Self-check: If top two weights change, does the winner flip?
• Skipping runbooks → Self-check: Do you have monitoring, alerts, SLOs, and rollback?

Practical projects

• Build a decision matrix template: parameterize weights, auto-calc totals, and sensitivity.
• Pilot two storage options with a 100 GB dataset; measure ingestion, query latency, and cost after 72 hours.
• Create a governance fit checklist for your org (PII tagging, lineage, access patterns) and test it on a small domain.

Quick test

The quick test below is available to everyone. Sign in to save your progress and track completion.

Who this is for

• Data Architects and Senior Data Engineers making platform/tooling decisions
• Engineering Managers needing to evaluate options with stakeholders
• Analytic Engineers contributing to warehouse/lakehouse choices

Prerequisites

• Working knowledge of data systems: batch vs streaming, storage types, SQL
• Basic understanding of SLAs/SLOs and cost modeling
• Comfort with running small pilots or benchmarks

Learning path

• Before: Business requirements gathering, non-functional requirements, data governance basics
• This: Technology selection criteria and scoring
• After: Reference architecture design, migration planning, SLOs and runbooks

Next steps

• Adopt a standard decision template for your team
• Schedule a 1–2 day pilot for your next upcoming decision
• Run a lightweight postmortem on your last tech choice and refine your criteria

Mini challenge

Timebox 45 minutes: Draft a decision statement and a 10-criterion, weighted scoring sheet for choosing a lakehouse engine for a finance team with fine-grained access control, 30 concurrent BI users, and multi-region DR. Propose at least two risks and mitigations.