Neural Architecture Search: Methods and Efficiency

Module: Machine Learning | Difficulty: Advanced

NAS Formulation

DARTS (Differentiable NAS)

Weight Sharing

Train a supernet, sample subnets by selecting paths.

One-Shot NAS

import torch
import torch.nn as nn
import torch.nn.functional as F

class DARTS(nn.Module):
    def __init__(self, operations, n_nodes=4):
        super().__init__()
        self.ops = nn.ModuleDict(operations)
        self.n_nodes = n_nodes
        self.alphas = nn.ParameterList([
            nn.Parameter(torch.randn(len(operations)))
            for _ in range(n_nodes * (n_nodes + 1) // 2)
        ])
    def forward(self, x):
        states = [x]
        edge_idx = 0
        for j in range(1, self.n_nodes + 1):
            new_state = 0
            for i in range(j):
                weights = F.softmax(self.alphas[edge_idx], dim=0)
                edge_out = sum(w * op(states[i]) for w, op in zip(weights, self.ops.values()))
                new_state += edge_out
                edge_idx += 1
            states.append(new_state)
        return states[-1]

Research Insight: DARTS discovered architectures that outperform hand-crafted designs, but suffers from performance collapse. The solution is to use fair sampling and architecture regularization.