Deep Learning For Vision Basics

Why this matters

Deep learning powers nearly every modern computer vision system you will work on: product search by image, defect detection on factory lines, medical image triage, face and license-plate redaction, AR effects, and more. As a Computer Vision Engineer, you will train and fine-tune neural networks, choose architectures, prepare datasets, and ship robust models to production. Strong basics help you catch pitfalls early, debug faster, and deliver measurable results.

Concept explained simply

A convolutional neural network (CNN) is a stack of pattern detectors. Each layer uses small learnable filters (kernels) that slide over the image to find edges, textures, and shapes. Early layers detect simple patterns (edges), later layers combine them into higher-level concepts (eyes, wheels). Nonlinear activations (like ReLU) let the model learn flexible boundaries. Pooling (or stride) reduces spatial size, making features more robust and faster to compute.

Mental model

• Local eyes: small filters look at tiny patches.
• Parameter sharing: the same filter scans the whole image, learning a general detector.
• Hierarchy: later layers compose earlier patterns into complex objects.
• Heads: the final layers convert features into predictions (class, box, mask, keypoints).

Core building blocks

• Convolution (Conv): num_filters, kernel_size, stride, padding. Output size per spatial dimension: floor((H + 2P − K)/S) + 1.
• Activation: ReLU is common; others include GELU, LeakyReLU.
• Normalization: BatchNorm/LayerNorm stabilize training.
• Pooling/Stride: downsample features for efficiency.
• Global Average Pooling: converts feature maps to a vector.
• Losses: cross-entropy (classification), L1/L2/GIoU (boxes), Dice/IoU/BCE (segmentation).
• Transfer learning: start from a pretrained backbone; freeze or fine-tune.

Data and labels

• Splits: train/val/test (e.g., 70/15/15). Keep distributions similar.
• Labels:
  • Classification: class id per image (or per crop).
  • Detection: bounding boxes [x_min, y_min, x_max, y_max] + class id.
  • Segmentation: mask per pixel (binary or multi-class).
• Transforms: resize, crop, flip, color jitter, normalization (mean/std) matching the pretrained model’s expectations.
• Quality: consistent class definitions, bounding boxes tightly fit objects, masks are clean.

Training loop in 5 steps

1. Load data: dataset + dataloader with augmentations.
2. Build model: backbone (e.g., ResNet) + task head.
3. Choose loss and metrics: align with task and class balance.
4. Optimize: optimizer (SGD/Adam), learning rate schedule, weight decay.
5. Validate: track val loss/metrics, early stop if overfitting.

Why augmentations help

Augmentations simulate new views of your data, reducing overfitting and improving robustness. Keep them realistic: flips for natural scenes are fine; vertical flips for digits may break labels. Match augmentation to the task: color jitter aids natural images; geometric transforms must also transform labels (boxes/masks).

Worked examples

Example 1 — Binary image classification with transfer learning

1. Task: cat vs dog images, ~2k train, 500 val.
2. Backbone: pretrained ResNet; replace last layer with a 2-class head.
3. Protocol:
   • Freeze backbone for a few epochs to train the new head.
   • Unfreeze top layers and fine-tune with a low learning rate.

# Pseudocode (PyTorch-like)
model = PretrainedResNet()
model.fc = Linear(in_features=model.fc.in_features, out_features=2)
freeze_backbone(model, except_fc=True)
for epoch in range(3):
    train_one_epoch(model, head_only=True, lr=1e-3)
unfreeze_top_layers(model, depth=2)
for epoch in range(7):
    train_one_epoch(model, head_only=False, lr=5e-5, weight_decay=1e-4)

Metric: accuracy; also track precision/recall if classes are imbalanced.

Example 2 — From classification to detection

1. Labels: for each image, a set of boxes + class ids.
2. Model: one-stage detector with a shared backbone and multi-scale heads.
3. Loss: classification loss + box regression loss (e.g., focal + IoU).
4. Tip: start from a pretrained detector; fine-tune on your dataset; ensure geometric augmentations update boxes.

Example 3 — Segmentation basics (U-Net style)

1. Input: N x C x H x W; Output: N x K x H x W (K classes) or N x 1 x H x W (binary).
2. Architecture: encoder-decoder with skip connections to preserve details.
3. Loss: Dice + BCE for binary; Cross-Entropy + Dice for multi-class.
4. Metric: mean IoU; visualize masks to catch label/resize issues.

Shapes and quick math

• Image tensor layout: [Batch, Channels, Height, Width] = [N, C, H, W]
• Conv output per spatial dim: floor((H + 2P − K)/S) + 1
• Conv params: (K_h × K_w × C_in + bias?) × C_out

Mini check: compute output size

Input 128x128, kernel 3, stride 2, padding 1 → (128 + 2 − 3)/2 + 1 = 64. Output 64x64.

Exercises

Do these before the quick test. Aim for clarity, not speed.

Exercise checklist

• I computed spatial sizes with the correct formula.
• I distinguished input/output channels in parameter counts.
• My transfer plan prevents overfitting (freeze, low LR, early stop).
• I included augmentations and proper normalization.
• I kept validation and test sets untouched during training.

Common mistakes and self-check

• Wrong normalization: using raw pixels with a pretrained backbone. Self-check: print batch mean/std; compare to expected values.
• Label drift: class ids inconsistent between train and val. Self-check: print label mappings from both splits.
• Data leakage: augmentations applied after the split is fine; but never tune on test. Self-check: ensure test is only used once.
• Mismatch loss/activation: using sigmoid with softmax cross-entropy. Self-check: verify your final layer and criterion match the task.
• Overfitting: training accuracy high, val low. Self-check: add regularization, more aug, or freeze deeper.
• Eval mode forgotten: BatchNorm/Dropout left in train mode during validation. Self-check: call model.eval() for validation/inference.

Practical projects

• Small classifier: classify recyclable vs non-recyclable items from photos. Goal: >85% val accuracy; include confusion matrix.
• Lightweight detector: fine-tune a pretrained one-stage detector to locate logos on packages. Goal: mAP@0.5 > 0.6 on val.
• Basic segmentation: segment leaves from background in plant images. Goal: mean IoU > 0.7; visualize good and bad cases.

Who this is for, prerequisites, learning path

Who this is for

• Beginners in computer vision who know basic Python.
• Engineers/data scientists moving from classical CV to deep learning.

Prerequisites

• Python basics, arrays/tensors.
• High-school level linear algebra and probability.

Learning path

• Now: Deep Learning for Vision Basics (this page).
• Next: Model evaluation and metrics (precision/recall, mAP, IoU).
• Then: Object detection and segmentation heads.
• Later: Training at scale, experiment tracking, and deployment.

Mini challenge

Build a 5-class image classifier with transfer learning. Constraints: use early stopping, cosine LR schedule, and at least two augmentations. Report accuracy and one failure mode with a hypothesis for improvement.

Next steps

• Run your classifier on a few out-of-distribution images to gauge robustness.
• Visualize model attention (e.g., class activation maps) to sanity-check behavior.
• Move to detection/segmentation to broaden your toolbox.

Ready for the Quick Test?

Take the quick test below to check your understanding. Everyone can take it; logged-in users will have their progress saved.