3D Point Cloud Processing

3D Point Cloud Processing Overview

Point clouds represent 3D data as unordered collections of points with (x, y, z) coordinates, often including additional features like color, normals, or intensity. Unlike regular grids in images, point clouds are irregular and permutation-invariant, requiring specialized architectures that process raw 3D points directly. PointNet pioneered direct point cloud learning, while PointNet++ introduced hierarchical grouping for capturing local geometric structures.

Theory: PointNet++ Architecture

PointNet processes each point independently through shared MLPs, creating per-point features that are aggregated via max pooling to form a global descriptor. While permutation-invariant, this approach misses local geometric structures critical for distinguishing shapes. PointNet++ addresses this by introducing hierarchical set abstraction layers that group nearby points and learn local features at multiple scales.

The set abstraction layer consists of three steps: sampling selects representative points using farthest point sampling (FPS), grouping finds neighboring points using ball query or KNN, and PointNet processes local groups to extract local features. This hierarchical structure captures patterns from small local patches to larger regions, similar to how CNNs build hierarchical representations.

Multi-scale grouping (MSG) and multi-resolution grouping (MRG) handle varying point densities. MSG processes local groups at multiple radius scales, concatenating features for robustness. MRG combines features from different resolutions, using larger neighborhoods in sparse regions and smaller ones in dense areas.

Mathematical Foundations

Farthest point sampling selects centroids iteratively:

Where each parameter means:

• is the set of selected sampling points (centroids)
• is the full set of input points
• is the -th sampled point
• is the Euclidean distance between point and sampled point
• The algorithm maximizes the minimum distance to already selected points

Ball query grouping finds neighbors within radius:

Where each parameter means:

• is the set of neighbors within radius of point
• is the full point set
• is the L2 Euclidean distance
• is the query radius (typically 0.1 to 0.5 of the bounding sphere)

Local feature aggregation in PointNet++ processes groups:

Where each parameter means:

• is the local feature vector for the group
• is the set of points in the local group
• is the centroid point coordinates
• centers the local coordinates relative to the centroid
• MLP applies shared weights to each centered point

Architecture Design

Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F


class SetAbstraction(nn.Module):
    def __init__(self, num_points, radius, num_samples, in_channels, mlp_channels):
        super(SetAbstraction, self).__init__()
        self.num_points = num_points
        self.radius = radius
        self.num_samples = num_samples
        layers = []
        last_ch = in_channels + 3
        for out_ch in mlp_channels:
            layers.extend([
                nn.Conv2d(last_ch, out_ch, 1),
                nn.BatchNorm2d(out_ch),
                nn.ReLU(inplace=True)
            ])
            last_ch = out_ch
        self.mlp = nn.Sequential(*layers)

    def farthest_point_sample(self, xyz, num_points):
        B, N, C = xyz.shape
        centroids = torch.zeros(B, num_points, dtype=torch.long)
        distance = torch.ones(B, N) * 1e10
        farthest = torch.randint(0, N, (B,))
        for i in range(num_points):
            centroids[:, i] = farthest
            centroid = xyz[torch.arange(B), farthest].unsqueeze(1)
            dist = torch.sum((xyz - centroid) ** 2, dim=-1)
            distance = torch.min(distance, dist)
            farthest = torch.max(distance, dim=-1)[1]
        return centroids

    def ball_query(self, xyz, new_xyz, radius, num_samples):
        B, N, _ = xyz.shape
        _, S, _ = new_xyz.shape
        group_idx = torch.zeros(B, S, num_samples).long()
        for i in range(B):
            sqrdists = torch.cdist(new_xyz[i], xyz[i])
            group_idx[i] = torch.topk(sqrdists, num_samples, dim=1, largest=False)[1]
        return group_idx

    def forward(self, xyz, points):
        B, N, C = xyz.shape
        fps_idx = self.farthest_point_sample(xyz, self.num_points)
        new_xyz = torch.gather(xyz, 1, fps_idx.unsqueeze(-1).expand(-1, -1, C))
        idx = self.ball_query(xyz, new_xyz, self.radius, self.num_samples)
        grouped = torch.gather(xyz.unsqueeze(1).expand(-1, self.num_points, N, 3), 2,
                               idx.unsqueeze(-1).expand(-1, -1, -1, 3))
        grouped = grouped - new_xyz.unsqueeze(2)
        if points is not None:
            grouped_points = torch.gather(points.unsqueeze(1).expand(-1, self.num_points, N, -1), 2,
                                         idx.unsqueeze(-1).expand(-1, -1, -1, points.shape[-1]))
            grouped = torch.cat([grouped, grouped_points], dim=-1)
        grouped = grouped.permute(0, 3, 2, 1)
        new_features = self.mlp(grouped)
        new_features = torch.max(new_features, dim=2)[0]
        new_features = new_features.permute(0, 2, 1)
        return new_xyz, new_features


class PointNet2Seg(nn.Module):
    def __init__(self, num_classes=50):
        super(PointNet2Seg, self).__init__()
        self.sa1 = SetAbstraction(512, 0.2, 32, 3, [32, 32, 64])
        self.sa2 = SetAbstraction(128, 0.4, 32, 64, [64, 64, 128])
        self.sa3 = SetAbstraction(1, 0.8, 32, 128, [128, 256, 1024])
        self.fp3 = nn.Sequential(nn.Linear(1024 + 128, 256), nn.ReLU(), nn.Dropout(0.5))
        self.fp2 = nn.Sequential(nn.Linear(256 + 64, 128), nn.ReLU(), nn.Dropout(0.5))
        self.fp1 = nn.Sequential(nn.Linear(128 + 3, 64), nn.ReLU(), nn.Dropout(0.5))
        self.classifier = nn.Sequential(nn.Linear(64, 32), nn.ReLU(), nn.Dropout(0.3),
                                       nn.Linear(32, num_classes))

    def forward(self, xyz):
        l0_points = xyz
        l1_xyz, l1_points = self.sa1(l0_xyz, l0_points)
        l2_xyz, l2_points = self.sa2(l1_xyz, l1_points)
        l3_xyz, l3_points = self.sa3(l2_xyz, l2_points)
        l3_points = l3_points.repeat(1, l2_xyz.shape[1], 1)
        l2_points = self.fp3(torch.cat([l3_points, l2_points], dim=-1))
        l1_points = self.fp2(torch.cat([l2_points, l1_points], dim=-1))
        l0_points = self.fp1(torch.cat([l1_points, l0_points], dim=-1))
        seg = self.classifier(l0_points)
        return seg

Comparison Table

Common Challenges

1. Variable Density: Real-world point clouds have non-uniform density due to sensor distance and occlusion
2. Scale Invariance: Objects appear at different scales requiring multi-scale processing
3. Order Invariance: The model must produce consistent outputs regardless of input point ordering
4. Computational Efficiency: Processing millions of points requires efficient sampling and grouping
5. Noise and Missing Points: Sensor noise and partial scans create incomplete and noisy data

Case Study: S3DIS Indoor Segmentation

The S3DIS dataset contains 6 large-scale indoor scenes with 13 semantic categories. PointNet++ achieves 54.5% mIoU on Area5 using multi-scale grouping with radii [0.1, 0.2, 0.4]. Point Transformer reaches 70.4% mIoU by replacing MLPs with self-attention for better local structure capture. The KPConv method achieves 67.1% using kernel point convolutions that define fixed kernels in continuous space. Training uses Adam optimizer with learning rate 0.001, batch size 16, and input size of 4096 points per scene with random subsampling augmentation.

Key Takeaways

• Point cloud processing requires permutation-invariant operations since points have no inherent order
• Set abstraction layers enable hierarchical feature learning from local to global structures
• Farthest point sampling ensures uniform coverage of the point space
• Ball query grouping captures local geometry with adjustable receptive field size
• Feature propagation through interpolation enables dense per-point predictions
• Multi-scale grouping handles varying point densities across different regions
• Skip connections between encoder and decoder preserve fine-grained spatial details