Computer Vision
Semantic Segmentation — Pixel-Level Understanding of Images
Semantic segmentation assigns a class label to every pixel in an image, enabling fine-grained scene understanding. It is critical for autonomous driving, medical imaging, and robotics.
- U-Net Dominates Medical — Encoder-decoder with skip connections excels on small medical imaging datasets
- Dice + BCE is Standard — Combined loss functions handle class imbalance and optimize overlap directly
- mIoU is the Metric — Mean Intersection over Union across all classes is the primary evaluation measure
Semantic Segmentation — FCN, U-Net, DeepLab and Medical Imaging
What You'll Learn
- 1Formulate segmentation as a per-pixel classification problem and write its cross-entropy loss
- 2Explain how FCN, U-Net and DeepLab upsample features back to input resolution
- 3Combine cross-entropy with dice loss to survive extreme class imbalance
- 4Evaluate a segmentor with class-wise IoU and mean IoU on benchmarks such as Cityscapes
Semantic segmentation assigns a class label to every pixel in an image, enabling fine-grained scene understanding. It is critical for autonomous driving, medical imaging, and robotics. A 512 x 512 Cityscapes frame is 262,144 predictions at once, and the classes are brutally imbalanced: road covers about 30 percent of the pixels while a traffic light covers well under 0.1 percent, so accuracy is a meaningless metric and the loss design matters as much as the architecture.
Segmentation Types
Fully Convolutional Network (FCN)
With a kernel, stride 2 and padding 1 the formula doubles the map exactly, so four such blocks take a bottleneck back to without ever interpolating by hand.
U-Net Architecture
DeepLab
A dilation rate of samples the kernel on a grid spaced pixels apart, so a kernel at rate 4 sees a 9 pixel neighborhood spread over a 25 pixel receptive field without losing resolution:
ASPP stacks rates 1, 6, 12 and 18 in parallel and concatenates the results, which is how DeepLabv3+ recovers both fine boundaries and whole-object context in a single forward pass.
Loss Functions for Segmentation
Every pixel is an independent class prediction, so the segmentation loss is the classification loss summed over all pixels. With the logit for pixel and class and the one hot target:
Unweighted cross-entropy lets road and building dominate, because they hold most of the pixels, so rare classes such as cyclists are frequently predicted as background. Two fixes are standard: inverse frequency class weights, and a loss computed on the overlap of the whole mask.
Dice sees the entire mask at once, so a rare class that covers 400 pixels moves the loss about as much as the 260,000 pixel background, while BCE keeps every pixel's probability calibrated. Medical U-Net training almost always runs on the sum with near 1.0.
Evaluation: IoU
A strong DeepLabv3+ model reaches about 0.83 mIoU on Cityscapes, while the same network scored on raw pixel accuracy reports roughly 0.96 and hides every failure on the rare classes. That gap is the whole argument for mIoU.
Applications
Summary
- Semantic segmentation assigns a class label to every pixel
- FCN introduced fully convolutional architecture for pixel-wise prediction
- U-Net with skip connections dominates medical imaging
- DeepLab uses dilated convolution for multi-scale context
- Dice + BCE loss handles class imbalance
- mIoU is the standard evaluation metric
Next: RNN Deep Dive
Common Mistakes
Reporting pixel accuracy as the main metric
Fix: Report mIoU per class: a net that predicts only road and sky can pass 90 percent accuracy while scoring zero on cyclists.
Training on plain cross-entropy with imbalanced masks
Fix: Add dice loss or inverse frequency class weights so a class covering 0.05 percent of pixels still produces gradient.
Bilinear upsampling with no learned refinement
Fix: Use transposed convolution plus U-Net skip connections; pure interpolation blurs boundaries and costs 1 to 2 mIoU.
Best Practices
- Keep label ids contiguous from 0 to C - 1 and store them as int64 so cross entropy never reads a float mask
- Validate with the official evaluator, such as the Cityscapes or ADE20K scripts, instead of a hand written IoU loop
- Augment with random scale, horizontal flip and crop, and pick crops of at least 512 so small objects survive
Key Takeaways
- Segmentation is per-pixel classification: the loss sums cross-entropy over all H x W pixels
- U-Net skip connections carry full resolution encoder features to the decoder, which is why it dominates medical imaging
- Atrous convolution grows the receptive field without losing resolution, and ASPP runs several dilation rates in parallel
- mIoU averages IoU across classes, so a failure on a rare class stays visible instead of being washed out
Frequently Asked Questions
Why not evaluate segmentation with accuracy?
A 512 x 512 street scene is mostly road, sky and building. Predicting only those three classes can reach 95 percent pixel accuracy while missing every car and pedestrian, so benchmarks report mIoU, which averages IoU per class and therefore gives a traffic light the same weight as the road.
Why does U-Net concatenate encoder features instead of only upsampling?
Pooling throws away exact position: after four 2x pools the decoder knows roughly where an object is but not where its edge is. Skip connections hand the full resolution feature map from the encoder straight to the decoder, so the network combines deep semantics with pixel precision. That is why U-Net still produces sharp boundaries when trained on as few as 30 medical images.
What is the real difference between dice loss and cross-entropy?
Cross-entropy scores each pixel independently and sums the result, so the classes that cover most of the image dominate the gradient. Dice scores the overlap of the entire predicted mask with the ground truth, so a 400 pixel tumor has the same weight as a 260,000 pixel background. Most strong models train on the sum of the two with a mixing weight near 1.0.
Interview Questions
Q1: How do you compute mIoU for a 19 class dataset?
Accumulate a 19 x 19 confusion matrix over the validation set, read TP, FP and FN from each row, compute IoU_c = TP / (TP + FP + FN) per class and average them; classes with no ground truth pixels in the split are skipped by convention.
Q2: Why does DeepLab use dilated convolution instead of more pooling?
Pooling trades resolution for context and segmentation needs both. A dilation rate of r samples a 3 x 3 kernel every r pixels, so the receptive field grows while the feature map keeps its size, giving 8x context at no down-sampling cost.
Q3: What happens if you upsample logits with nearest neighbor before the loss?
Nearest neighbor repeats pixels, keeping boundaries blocky and biasing the gradient toward large blobs. Upsampling logits with bilinear interpolation and computing the loss at full resolution, as modern FCNs do, recovers roughly 1 to 2 mIoU.