Object Detection — YOLO, Faster R-CNN, Anchor Boxes and mAP

Object detection localizes and classifies objects in images, predicting both bounding boxes and class labels. It is one of the most impactful applications of deep learning. A COCO image holds 7.7 objects on average, so the network must emit a variable length set of boxes, classes and confidences rather than a single label, and the training signal must punish localization error, missed objects and false alarms at the same time.

Detection vs. Classification

IoU (Intersection over Union)

IoU is a ratio, so it is invariant to image resolution: a 320 pixel wide box and its 640 pixel copy score the same as long as the shapes match. Detectors are usually scored as correct when IoU reaches 0.5, while the COCO benchmark also counts detections that clear 0.75 for its "primary" subset.

Anchor Boxes

Positions are divided by the anchor size so the network predicts targets of order 1 regardless of object scale, and sizes are logged so that doubling and halving a box produce targets of opposite sign:

Decoding inverts the transform with one exponential per axis, which is why an anchor free head still has to pick a reference scale somewhere:

The regression term itself uses a robust penalty so that a handful of mis-annotated boxes cannot dominate a batch, with smooth L1 that is quadratic below unit error and linear beyond it:

Non-Maximum Suppression (NMS)

Two-Stage: Faster R-CNN

Single-Stage: YOLO

Each grid cell also predicts objectness, and the class score shipped to evaluation multiplies that objectness by the class probability, so a box is only confident if both terms are large:

With 80 COCO classes and 3 anchors per cell, a grid emits raw values before NMS trims them to the handful of boxes that survive.

Two-Stage vs Single-Stage

Evaluation: mAP

COCO averages AP over the nine thresholds 0.50 to 0.95 in steps of 0.05, so the headline number rewards tight boxes as well as correct ones. A YOLOv8-nano scores roughly 55 at [email protected] but about 37 on the averaged metric, and the 18 point gap measures how loose the boxes are.

Summary

• Object detection predicts bounding boxes + class labels + confidence scores
• IoU measures overlap between predicted and ground truth boxes
• Anchor boxes provide reference shapes for regression
• NMS removes duplicate detections
• Two-stage (Faster R-CNN): accurate but slower
• Single-stage (YOLO): faster, real-time capable
• mAP is the standard evaluation metric

Next: Semantic Segmentation

Frequently Asked Questions

Why does non-maximum suppression use an IoU threshold near 0.5?

NMS deletes every box that overlaps a higher scoring box by more than the threshold. At 0.5 the duplicate boxes produced for one object merge into a single detection, while two people standing side by side, whose boxes rarely overlap past 0.2, both survive. Raise the threshold to 0.9 and three or four boxes per object remain; lower it to 0.1 and adjacent real objects are deleted along with the duplicates.

What exactly does [email protected]:0.95 average over?

COCO computes average precision at IoU 0.50, 0.55, 0.60 and so on up to 0.95, then averages those nine AP values into one score. Because half of the thresholds sit above 0.75, the metric punishes loose boxes even when every object is found, which is why a detector can look perfect at [email protected] and mediocre on the full benchmark.

Why integrate precision over recall instead of picking one confidence threshold?

The confidence threshold is a deployment decision that moves with your false positive budget, while AP measures ranking quality: a detector that scores true positives above false positives earns high AP at every operating point. Two models can be identical at 0.5 confidence and still differ by several mAP points because one ranks its detections better.