Skip to content

armgit / ECC

Public Imported from GitHub
0

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

feat(skills): add pytorch-patterns skill (#550)

Muhammad Idrees1b21e082fa9f
diff --git a/skills/pytorch-patterns/SKILL.md b/skills/pytorch-patterns/SKILL.md
new file mode 100644
index 000000000..8b56d8152
--- /dev/null
+++ b/skills/pytorch-patterns/SKILL.md
@@ -0,0 +1,396 @@
+---
+name: pytorch-patterns
+description: PyTorch deep learning patterns and best practices for building robust, efficient, and reproducible training pipelines, model architectures, and data loading.
+origin: ECC
+---
+
+# PyTorch Development Patterns
+
+Idiomatic PyTorch patterns and best practices for building robust, efficient, and reproducible deep learning applications.
+
+## When to Activate
+
+- Writing new PyTorch models or training scripts
+- Reviewing deep learning code
+- Debugging training loops or data pipelines
+- Optimizing GPU memory usage or training speed
+- Setting up reproducible experiments
+
+## Core Principles
+
+### 1. Device-Agnostic Code
+
+Always write code that works on both CPU and GPU without hardcoding devices.
+
+```python
+# Good: Device-agnostic
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+model = MyModel().to(device)
+data = data.to(device)
+
+# Bad: Hardcoded device
+model = MyModel().cuda() # Crashes if no GPU
+data = data.cuda()
+```
+
+### 2. Reproducibility First
+
+Set all random seeds for reproducible results.
+
+```python
+# Good: Full reproducibility setup
+def set_seed(seed: int = 42) -> None:
+ torch.manual_seed(seed)
+ torch.cuda.manual_seed_all(seed)
+ np.random.seed(seed)
+ random.seed(seed)
+ torch.backends.cudnn.deterministic = True
+ torch.backends.cudnn.benchmark = False
+
+# Bad: No seed control
+model = MyModel() # Different weights every run
+```
+
+### 3. Explicit Shape Management
+
+Always document and verify tensor shapes.
+
+```python
+# Good: Shape-annotated forward pass
+def forward(self, x: torch.Tensor) -> torch.Tensor:
+ # x: (batch_size, channels, height, width)
+ x = self.conv1(x) # -> (batch_size, 32, H, W)
+ x = self.pool(x) # -> (batch_size, 32, H//2, W//2)
+ x = x.view(x.size(0), -1) # -> (batch_size, 32*H//2*W//2)
+ return self.fc(x) # -> (batch_size, num_classes)
+
+# Bad: No shape tracking
+def forward(self, x):
+ x = self.conv1(x)
+ x = self.pool(x)
+ x = x.view(x.size(0), -1) # What size is this?
+ return self.fc(x) # Will this even work?
+```
+
+## Model Architecture Patterns
+
+### Clean nn.Module Structure
+
+```python
+# Good: Well-organized module
+class ImageClassifier(nn.Module):
+ def __init__(self, num_classes: int, dropout: float = 0.5) -> None:
+ super().__init__()
+ self.features = nn.Sequential(
+ nn.Conv2d(3, 64, kernel_size=3, padding=1),
+ nn.BatchNorm2d(64),
+ nn.ReLU(inplace=True),
+ nn.MaxPool2d(2),
+ )
+ self.classifier = nn.Sequential(
+ nn.Dropout(dropout),
+ nn.Linear(64 * 16 * 16, num_classes),
+ )
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ x = self.features(x)
+ x = x.view(x.size(0), -1)
+ return self.classifier(x)
+
+# Bad: Everything in forward
+class ImageClassifier(nn.Module):
+ def __init__(self):
+ super().__init__()
+
+ def forward(self, x):
+ x = F.conv2d(x, weight=self.make_weight()) # Creates weight each call!
+ return x
+```
+
+### Proper Weight Initialization
+
+```python
+# Good: Explicit initialization
+def _init_weights(self, module: nn.Module) -> None:
+ if isinstance(module, nn.Linear):
+ nn.init.kaiming_normal_(module.weight, mode="fan_out", nonlinearity="relu")
+ if module.bias is not None:
+ nn.init.zeros_(module.bias)
+ elif isinstance(module, nn.Conv2d):
+ nn.init.kaiming_normal_(module.weight, mode="fan_out", nonlinearity="relu")
+ elif isinstance(module, nn.BatchNorm2d):
+ nn.init.ones_(module.weight)
+ nn.init.zeros_(module.bias)
+
+model = MyModel()
+model.apply(model._init_weights)
+```
+
+## Training Loop Patterns
+
+### Standard Training Loop
+
+```python
+# Good: Complete training loop with best practices
+def train_one_epoch(
+ model: nn.Module,
+ dataloader: DataLoader,
+ optimizer: torch.optim.Optimizer,
+ criterion: nn.Module,
+ device: torch.device,
+ scaler: torch.amp.GradScaler | None = None,
+) -> float:
+ model.train() # Always set train mode
+ total_loss = 0.0
+
+ for batch_idx, (data, target) in enumerate(dataloader):
+ data, target = data.to(device), target.to(device)
+
+ optimizer.zero_grad(set_to_none=True) # More efficient than zero_grad()
+
+ # Mixed precision training
+ with torch.amp.autocast("cuda", enabled=scaler is not None):
+ output = model(data)
+ loss = criterion(output, target)
+
+ if scaler is not None:
+ scaler.scale(loss).backward()
+ scaler.unscale_(optimizer)
+ torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
+ scaler.step(optimizer)
+ scaler.update()
+ else:
+ loss.backward()
+ torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
+ optimizer.step()
+
+ total_loss += loss.item()
+
+ return total_loss / len(dataloader)
+```
+
+### Validation Loop
+
+```python
+# Good: Proper evaluation
[email protected]_grad() # More efficient than wrapping in torch.no_grad() block
+def evaluate(
+ model: nn.Module,
+ dataloader: DataLoader,
+ criterion: nn.Module,
+ device: torch.device,
+) -> tuple[float, float]:
+ model.eval() # Always set eval mode — disables dropout, uses running BN stats
+ total_loss = 0.0
+ correct = 0
+ total = 0
+
+ for data, target in dataloader:
+ data, target = data.to(device), target.to(device)
+ output = model(data)
+ total_loss += criterion(output, target).item()
+ correct += (output.argmax(1) == target).sum().item()
+ total += target.size(0)
+
+ return total_loss / len(dataloader), correct / total
+```
+
+## Data Pipeline Patterns
+
+### Custom Dataset
+
+```python
+# Good: Clean Dataset with type hints
+class ImageDataset(Dataset):
+ def __init__(
+ self,
+ image_dir: str,
+ labels: dict[str, int],
+ transform: transforms.Compose | None = None,
+ ) -> None:
+ self.image_paths = list(Path(image_dir).glob("*.jpg"))
+ self.labels = labels
+ self.transform = transform
+
+ def __len__(self) -> int:
+ return len(self.image_paths)
+
+ def __getitem__(self, idx: int) -> tuple[torch.Tensor, int]:
+ img = Image.open(self.image_paths[idx]).convert("RGB")
+ label = self.labels[self.image_paths[idx].stem]
+
+ if self.transform:
+ img = self.transform(img)
+
+ return img, label
+```
+
+### Efficient DataLoader Configuration
+
+```python
+# Good: Optimized DataLoader
+dataloader = DataLoader(
+ dataset,
+ batch_size=32,
+ shuffle=True, # Shuffle for training
+ num_workers=4, # Parallel data loading
+ pin_memory=True, # Faster CPU->GPU transfer
+ persistent_workers=True, # Keep workers alive between epochs
+ drop_last=True, # Consistent batch sizes for BatchNorm
+)
+
+# Bad: Slow defaults
+dataloader = DataLoader(dataset, batch_size=32) # num_workers=0, no pin_memory
+```
+
+### Custom Collate for Variable-Length Data
+
+```python
+# Good: Pad sequences in collate_fn
+def collate_fn(batch: list[tuple[torch.Tensor, int]]) -> tuple[torch.Tensor, torch.Tensor]:
+ sequences, labels = zip(*batch)
+ # Pad to max length in batch
+ padded = nn.utils.rnn.pad_sequence(sequences, batch_first=True, padding_value=0)
+ return padded, torch.tensor(labels)
+
+dataloader = DataLoader(dataset, batch_size=32, collate_fn=collate_fn)
+```
+
+## Checkpointing Patterns
+
+### Save and Load Checkpoints
+
+```python
+# Good: Complete checkpoint with all training state
+def save_checkpoint(
+ model: nn.Module,
+ optimizer: torch.optim.Optimizer,
+ epoch: int,
+ loss: float,
+ path: str,
+) -> None:
+ torch.save({
+ "epoch": epoch,
+ "model_state_dict": model.state_dict(),
+ "optimizer_state_dict": optimizer.state_dict(),
+ "loss": loss,
+ }, path)
+
+def load_checkpoint(
+ path: str,
+ model: nn.Module,
+ optimizer: torch.optim.Optimizer | None = None,
+) -> dict:
+ checkpoint = torch.load(path, map_location="cpu", weights_only=True)
+ model.load_state_dict(checkpoint["model_state_dict"])
+ if optimizer:
+ optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
+ return checkpoint
+
+# Bad: Only saving model weights (can't resume training)
+torch.save(model.state_dict(), "model.pt")
+```
+
+## Performance Optimization
+
+### Mixed Precision Training
+
+```python
+# Good: AMP with GradScaler
+scaler = torch.amp.GradScaler("cuda")
+for data, target in dataloader:
+ with torch.amp.autocast("cuda"):
+ output = model(data)
+ loss = criterion(output, target)
+ scaler.scale(loss).backward()
+ scaler.step(optimizer)
+ scaler.update()
+ optimizer.zero_grad(set_to_none=True)
+```
+
+### Gradient Checkpointing for Large Models
+
+```python
+# Good: Trade compute for memory
+from torch.utils.checkpoint import checkpoint
+
+class LargeModel(nn.Module):
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ # Recompute activations during backward to save memory
+ x = checkpoint(self.block1, x, use_reentrant=False)
+ x = checkpoint(self.block2, x, use_reentrant=False)
+ return self.head(x)
+```
+
+### torch.compile for Speed
+
+```python
+# Good: Compile the model for faster execution (PyTorch 2.0+)
+model = MyModel().to(device)
+model = torch.compile(model, mode="reduce-overhead")
+
+# Modes: "default" (safe), "reduce-overhead" (faster), "max-autotune" (fastest)
+```
+
+## Quick Reference: PyTorch Idioms
+
+| Idiom | Description |
+|-------|-------------|
+| `model.train()` / `model.eval()` | Always set mode before train/eval |
+| `torch.no_grad()` | Disable gradients for inference |
+| `optimizer.zero_grad(set_to_none=True)` | More efficient gradient clearing |
+| `.to(device)` | Device-agnostic tensor/model placement |
+| `torch.amp.autocast` | Mixed precision for 2x speed |
+| `pin_memory=True` | Faster CPU→GPU data transfer |
+| `torch.compile` | JIT compilation for speed (2.0+) |
+| `weights_only=True` | Secure model loading |
+| `torch.manual_seed` | Reproducible experiments |
+| `gradient_checkpointing` | Trade compute for memory |
+
+## Anti-Patterns to Avoid
+
+```python
+# Bad: Forgetting model.eval() during validation
+model.train()
+with torch.no_grad():
+ output = model(val_data) # Dropout still active! BatchNorm uses batch stats!
+
+# Good: Always set eval mode
+model.eval()
+with torch.no_grad():
+ output = model(val_data)
+
+# Bad: In-place operations breaking autograd
+x = F.relu(x, inplace=True) # Can break gradient computation
+x += residual # In-place add breaks autograd graph
+
+# Good: Out-of-place operations
+x = F.relu(x)
+x = x + residual
+
+# Bad: Moving data to GPU inside the training loop repeatedly
+for data, target in dataloader:
+ model = model.cuda() # Moves model EVERY iteration!
+
+# Good: Move model once before the loop
+model = model.to(device)
+for data, target in dataloader:
+ data, target = data.to(device), target.to(device)
+
+# Bad: Using .item() before backward
+loss = criterion(output, target).item() # Detaches from graph!
+loss.backward() # Error: can't backprop through .item()
+
+# Good: Call .item() only for logging
+loss = criterion(output, target)
+loss.backward()
+print(f"Loss: {loss.item():.4f}") # .item() after backward is fine
+
+# Bad: Not using torch.save properly
+torch.save(model, "model.pt") # Saves entire model (fragile, not portable)
+
+# Good: Save state_dict
+torch.save(model.state_dict(), "model.pt")
+```
+
+__Remember__: PyTorch code should be device-agnostic, reproducible, and memory-conscious. When in doubt, profile with `torch.profiler` and check GPU memory with `torch.cuda.memory_summary()`.