Grasp Detection and Robotic Manipulation

Module: Computer Vision | Difficulty: Premium

Grasp Configuration

where is position, is angle, is size, is quality.

Grasp Quality Score

Sim-to-Real Transfer

| Model | Cornell | Jaca | Object-wise | Speed | |-------|---------|------|-------------|-------| | GG-CNN | 73.0% | 84.3% | 78.5% | 12 ms | | GraspNet | 92.8% | 94.2% | 88.1% | 25 ms | | Contact-GraspNet | 94.5% | 96.1% | 91.2% | 40 ms | | FoundationGrasp | 96.2% | 97.3% | 93.5% | 30 ms |

import torch
import torch.nn as nn

class GraspDetector(nn.Module):
    def __init__(self, in_channels=4):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(in_channels, 32, 3, stride=2, padding=1),
            nn.BatchNorm2d(32), nn.ReLU(inplace=True),
            nn.Conv2d(32, 64, 3, stride=2, padding=1),
            nn.BatchNorm2d(64), nn.ReLU(inplace=True),
            nn.Conv2d(64, 128, 3, stride=2, padding=1),
            nn.BatchNorm2d(128), nn.ReLU(inplace=True),
        )
        self.angle_head = nn.Conv2d(128, 1, 1)
        self.quality_head = nn.Conv2d(128, 1, 1)
        self.width_head = nn.Conv2d(128, 1, 1)

    def forward(self, x):
        features = self.encoder(x)
        angle = torch.sigmoid(self.angle_head(features)) * 3.14159
        quality = torch.sigmoid(self.quality_head(features))
        width = self.width_head(features)
        return torch.cat([angle, quality, width], dim=1)

class GraspPlanner:
    def __init__(self, detector, robot):
        self.detector = detector
        self.robot = robot

    def plan(self, rgb, depth):
        grasp_map = self.detector(torch.cat([rgb, depth], dim=1))
        angle = grasp_map[:, 0]
        quality = grasp_map[:, 1]
        width = grasp_map[:, 2]
        best_idx = quality.argmax()
        grasp = {
            'position': self._pixel_to_3d(best_idx, depth),
            'angle': angle[best_idx],
            'width': width[best_idx],
            'quality': quality[best_idx],
        }
        return grasp

    def _pixel_to_3d(self, pixel_idx, depth):
        h, w = depth.shape[-2:]
        y, x = pixel_idx // w, pixel_idx % w
        z = depth[0, 0, y, x]
        return torch.tensor([x.item(), y.item(), z.item()])

Research Insight: Robotic grasping has been transformed by learned approaches that predict grasp configurations directly from point clouds. Contact-GraspNet predicted contact points and grasp poses simultaneously, achieving 96% success on the YCB dataset. The key challenge is sim-to-real gap: grasp detectors trained in simulation fail on real objects due to differences in texture, lighting, and physics. Domain randomization (varying textures, lighting, and physics parameters in simulation) has proven effective, with FoundationGrasp achieving 93% real-world success after fine-tuning on just 10 real grasps.