Missing Data Handling

Why this matters Real-world data is messy. As a Data Scientist, you will routinely: Clean survey or product logs with blanks, 'N/A', None, and inconsistent placeholders. Prepare training data where missing values can break model training or bias results. Build time-series features where occasional gaps must be forward filled or interpolated. Create dashboards and KPI reports that must handle missing entries clearly and safely. Concept explained simply Mental model Think of missing data as three categories: MCAR (missing completely at random): harmless to drop if sample size is large. MAR (missing at random): imputing using related columns is often reasonable. MNAR (missing not at random): the fact it is missing carries signal; highlight carefully. In pandas/numpy, missing often appears as np.nan, None, or pd.NA. Treat special placeholders like 'NA', '', 'unknown', or infinities as missing explicitly. Core toolkit (pandas/numpy) Detect import numpy as np import pandas as pd # Detect missing mask = df.isna() # same as df.isnull() not_missing = df.notna() # Count per column missing_counts = df.isna().sum() # Include NaN in value_counts vc = df['col'].value_counts(dropna=False) # Detect infinities and convert to NaN df.replace([np.inf, -np.inf], np.nan, inplace=True) Drop # Drop rows where any of subset is missing df2 = df.dropna(subset=['age', 'salary']) # Keep rows with at least thresh non-missing values df3 = df.dropna(thresh=3) Impute # Simple scalars df['age'] = df['age'].fillna(df['age'].median()) df['city'] = df['city'].fillna(df['city'].mode().iloc[0]) # Dict per column df = df.fillna({'age': df['age'].median(), 'city': df['city'].mode().iloc[0]}) # Forward/backward fill (time series) df['price'] = df['price'].ffill(limit=1).bfill(limit=1) # Interpolate numeric s = df['temp'].interpolate(method='linear') Conditional impute # Use where/mask for conditions # Example: fill 'discount' with 0 only for non-null prices df['discount'] = df['discount'].fillna(0).where(df['price'].notna()) # Group-wise median med_by_city = df.groupby('city')['age'].transform('median') df['age'] = df['age'].fillna(med_by_city) Type and coercion # Coerce bad strings to NaN df['amount'] = pd.to_numeric(df['amount'], errors='coerce') # Nullable integer after imputation df['units'] = df['units'].astype('Int64') Worked examples Example 1: Basic cleanup with numeric and categorical import numpy as np import pandas as pd raw = pd.DataFrame({ 'product': ['A', 'B', 'C', 'D'], 'price': [10.0, np.nan, np.inf, 12.0], 'category': ['toy', None, 'toy', ''] }) # Treat blanks and infinities as missing raw['category'].replace({'': np.nan}, inplace=True) raw.replace([np.inf, -np.inf], np.nan, inplace=True) # Impute raw['price'] = raw['price'].fillna(raw['price'].median()) raw['category'] = raw['category'].fillna(raw['category'].mode().iloc[0]) # Optional: price to Int64 after rounding clean = raw.copy() clean['price'] = clean['price'].round().astype('Int64') print(clean) Takeaways: normalize placeholders first, then impute with simple statistics per type. Example 2: Time series with ffill and interpolate dates = pd.date_range('2023-01-01', periods=7) TS = pd.DataFrame({'date': dates, 'temp': [20.0, np.nan, np.nan, 21.5, 22.0, np.nan, 23.5]}) TS.set_index('date', inplace=True) # Limit forward fill to 1 step to avoid long stretches of copying step1 = TS.copy() step1['temp'] = step1['temp'].ffill(limit=1) # Interpolate remaining small gaps linearly step2 = step1.copy() step2['temp'] = step2['temp'].interpolate(method='linear') print(step2) Use a small ffill limit, then interpolate for smoother results in metrics and models. Example 3: Group-wise median and conditional masking df = pd.DataFrame({ 'city': ['NY', 'NY', 'LA', 'LA', 'LA'], 'age': [35, None, 41, -1, None], 'income': [70_000, None, 80_000, 75_000, None] }) # Treat negative age as missing df.loc[df['age'].lt(0), 'age'] = np.nan # Group-wise impute age_med = df.groupby('city')['age'].transform('median') df['age'] = df['age'].fillna(age_med) # Income: per city median, fallback to global median city_income_med = df.groupby('city')['income'].transform('median') global_income_med = df['income'].median() df['income'] = df['income'].fillna(city_income_med).fillna(global_income_med) print(df) Respect local patterns (per group) and use sensible fallbacks. Who this is for Data Scientist and Analyst roles working with pandas and numpy. Anyone preparing datasets for modeling, BI, or experimentation. Prerequisites Comfortable with Python basics and Jupyter/Notebook workflow. Know core pandas operations: selecting, filtering, grouping, type casting. Basic statistics: mean, median, mode, percentiles. Learning path Recognize missing and invalid values in your dataset. Normalize placeholders and infinities into proper NaNs. Choose an imputation strategy per column type and business context. Validate results with summaries and simple plots or checks. Document your choices so downstream users understand the data. Common mistakes and self-check Confusing zero with missing. Self-check: count how many true zeros exist; confirm domain meaning. Forgetting to convert 'NA', '', or 'unknown' into NaN. Self-check: value_counts(dropna=False). Using mean imputation on skewed numeric data. Prefer median for robustness. Forward filling across long gaps in time series. Use limit and sanity checks. Imputing categorical with a frequent class that introduces bias. Consider 'missing' as its own category when appropriate. Not separating train/test when imputing for modeling. For real modeling, fit imputers on train only and apply to test. Practical projects Retail sales cleanup: impute missing prices and categories; ensure types are nullable integers where appropriate. IoT sensor stream: forward fill with limit, then interpolate; flag long gaps for exclusion. HR dataset: group-wise impute salaries by department and seniority; document the rationale. Exercises Complete these in a fresh notebook or REPL. Use the same dataset constructions shown below. Exercise 1: Count, normalize, and impute numeric and categorical safely. Exercise 2: Time series forward fill with limit and interpolation. Exercise 3: Group-wise imputation with fallbacks and conditional masking. Checklist before you run your code Converted placeholders ('', 'NA', 'unknown') and infinities to NaN. Chose median for skewed numeric; mode or explicit 'missing' category for categoricals. Used group-wise statistics when segments differ (city, department, etc.). Set reasonable ffill limit and verified no unrealistic plateaus. Re-checked dtypes after imputation. Mini challenge Given a dataset with columns: user_id, plan (categorical), monthly_fee (float), last_active_at (datetime), churned (bool) with 12% missing in plan and 5% missing in monthly_fee: Propose an imputation strategy for plan and monthly_fee considering churned status. Describe how you would validate that the imputation did not distort churn rate by plan. Thought starter Impute plan with a dedicated 'missing' category or mode per churned group; monthly_fee with median per plan, fallback to global median. Validate with churn rate by plan before/after and distribution overlap (median/IQR compare). Quick Test This test is available to everyone; only logged-in users get saved progress. Answer the questions to check your understanding. Then review mistakes and retry if needed.

Missing Data Handling

Why this matters

Quick Test

Practice Exercises

Normalize placeholders and impute numeric/categorical

Instructions

Expected Output

Time series: ffill with limit and interpolate

Group-wise imputation with fallback and conditional masking

Missing Data Handling — Quick Test

Have questions about Missing Data Handling?

AI Assistant