The agent that grows with you
Python aiai-agentai-agentsanthropicchatgptclaudeclaude-codecodexhermeshermes-agentllmnous-researchopenai
chore(skills): drop optional skills that official sources now cover
Teknium9347107cdbc0
diff --git a/optional-skills/creative/audiocraft-audio-generation/SKILL.md b/optional-skills/creative/audiocraft-audio-generation/SKILL.mddeleted file mode 100644index be5075c2f22..00000000000--- a/optional-skills/creative/audiocraft-audio-generation/SKILL.md+++ /dev/null@@ -1,569 +0,0 @@-----name: audiocraft-audio-generation-description: "AudioCraft: MusicGen text-to-music, AudioGen text-to-sound."-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [audiocraft, torch>=2.0.0, transformers>=4.30.0]-platforms: [linux, macos]-metadata:- hermes:- tags: [Multimodal, Audio Generation, Text-to-Music, Text-to-Audio, MusicGen]- related_skills: [heartmula, songwriting-and-ai-music]-------# AudioCraft: Audio Generation--Guide to using Meta's AudioCraft for text-to-music and text-to-audio generation with MusicGen, AudioGen, and EnCodec.--## When to use AudioCraft--**Use AudioCraft when:**-- Need to generate music from text descriptions-- Creating sound effects and environmental audio-- Building music generation applications-- Need melody-conditioned music generation-- Want stereo audio output-- Require controllable music generation with style transfer--**Key features:**-- **MusicGen**: Text-to-music generation with melody conditioning-- **AudioGen**: Text-to-sound effects generation-- **EnCodec**: High-fidelity neural audio codec-- **Multiple model sizes**: Small (300M) to Large (3.3B)-- **Stereo support**: Full stereo audio generation-- **Style conditioning**: MusicGen-Style for reference-based generation--**Use alternatives instead:**-- **Stable Audio**: For longer commercial music generation-- **Bark**: For text-to-speech with music/sound effects-- **Riffusion**: For spectogram-based music generation-- **OpenAI Jukebox**: For raw audio generation with lyrics--## Quick start--### Installation--```bash-# From PyPI-pip install audiocraft--# From GitHub (latest)-pip install git+https://github.com/facebookresearch/audiocraft.git--# Or use HuggingFace Transformers-pip install transformers torch torchaudio-```--### Basic text-to-music (AudioCraft)--```python-import torchaudio-from audiocraft.models import MusicGen--# Load model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Set generation parameters-model.set_generation_params(- duration=8, # seconds- top_k=250,- temperature=1.0-)--# Generate from text-descriptions = ["happy upbeat electronic dance music with synths"]-wav = model.generate(descriptions)--# Save audio-torchaudio.save("output.wav", wav[0].cpu(), sample_rate=32000)-```--### Using HuggingFace Transformers--```python-from transformers import AutoProcessor, MusicgenForConditionalGeneration-import scipy--# Load model and processor-processor = AutoProcessor.from_pretrained("facebook/musicgen-small")-model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")-model.to("cuda")--# Generate music-inputs = processor(- text=["80s pop track with bassy drums and synth"],- padding=True,- return_tensors="pt"-).to("cuda")--audio_values = model.generate(- **inputs,- do_sample=True,- guidance_scale=3,- max_new_tokens=256-)--# Save-sampling_rate = model.config.audio_encoder.sampling_rate-scipy.io.wavfile.write("output.wav", rate=sampling_rate, data=audio_values[0, 0].cpu().numpy())-```--### Text-to-sound with AudioGen--```python-from audiocraft.models import AudioGen--# Load AudioGen-model = AudioGen.get_pretrained('facebook/audiogen-medium')--model.set_generation_params(duration=5)--# Generate sound effects-descriptions = ["dog barking in a park with birds chirping"]-wav = model.generate(descriptions)--torchaudio.save("sound.wav", wav[0].cpu(), sample_rate=16000)-```--## Core concepts--### Architecture overview--```-AudioCraft Architecture:-┌──────────────────────────────────────────────────────────────┐-│ Text Encoder (T5) │-│ │ │-│ Text Embeddings │-└────────────────────────┬─────────────────────────────────────┘- │-┌────────────────────────▼─────────────────────────────────────┐-│ Transformer Decoder (LM) │-│ Auto-regressively generates audio tokens │-│ Using efficient token interleaving patterns │-└────────────────────────┬─────────────────────────────────────┘- │-┌────────────────────────▼─────────────────────────────────────┐-│ EnCodec Audio Decoder │-│ Converts tokens back to audio waveform │-└──────────────────────────────────────────────────────────────┘-```--### Model variants--| Model | Size | Description | Use Case |-|-------|------|-------------|----------|-| `musicgen-small` | 300M | Text-to-music | Quick generation |-| `musicgen-medium` | 1.5B | Text-to-music | Balanced |-| `musicgen-large` | 3.3B | Text-to-music | Best quality |-| `musicgen-melody` | 1.5B | Text + melody | Melody conditioning |-| `musicgen-melody-large` | 3.3B | Text + melody | Best melody |-| `musicgen-stereo-*` | Varies | Stereo output | Stereo generation |-| `musicgen-style` | 1.5B | Style transfer | Reference-based |-| `audiogen-medium` | 1.5B | Text-to-sound | Sound effects |--### Generation parameters--| Parameter | Default | Description |-|-----------|---------|-------------|-| `duration` | 8.0 | Length in seconds (1-120) |-| `top_k` | 250 | Top-k sampling |-| `top_p` | 0.0 | Nucleus sampling (0 = disabled) |-| `temperature` | 1.0 | Sampling temperature |-| `cfg_coef` | 3.0 | Classifier-free guidance |--## MusicGen usage--### Text-to-music generation--```python-from audiocraft.models import MusicGen-import torchaudio--model = MusicGen.get_pretrained('facebook/musicgen-medium')--# Configure generation-model.set_generation_params(- duration=30, # Up to 30 seconds- top_k=250, # Sampling diversity- top_p=0.0, # 0 = use top_k only- temperature=1.0, # Creativity (higher = more varied)- cfg_coef=3.0 # Text adherence (higher = stricter)-)--# Generate multiple samples-descriptions = [- "epic orchestral soundtrack with strings and brass",- "chill lo-fi hip hop beat with jazzy piano",- "energetic rock song with electric guitar"-]--# Generate (returns [batch, channels, samples])-wav = model.generate(descriptions)--# Save each-for i, audio in enumerate(wav):- torchaudio.save(f"music_{i}.wav", audio.cpu(), sample_rate=32000)-```--### Melody-conditioned generation--```python-from audiocraft.models import MusicGen-import torchaudio--# Load melody model-model = MusicGen.get_pretrained('facebook/musicgen-melody')-model.set_generation_params(duration=30)--# Load melody audio-melody, sr = torchaudio.load("melody.wav")--# Generate with melody conditioning-descriptions = ["acoustic guitar folk song"]-wav = model.generate_with_chroma(descriptions, melody, sr)--torchaudio.save("melody_conditioned.wav", wav[0].cpu(), sample_rate=32000)-```--### Stereo generation--```python-from audiocraft.models import MusicGen--# Load stereo model-model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')-model.set_generation_params(duration=15)--descriptions = ["ambient electronic music with wide stereo panning"]-wav = model.generate(descriptions)--# wav shape: [batch, 2, samples] for stereo-print(f"Stereo shape: {wav.shape}") # [1, 2, 480000]-torchaudio.save("stereo.wav", wav[0].cpu(), sample_rate=32000)-```--### Audio continuation--```python-from transformers import AutoProcessor, MusicgenForConditionalGeneration--processor = AutoProcessor.from_pretrained("facebook/musicgen-medium")-model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")--# Load audio to continue-import torchaudio-audio, sr = torchaudio.load("intro.wav")--# Process with text and audio-inputs = processor(- audio=audio.squeeze().numpy(),- sampling_rate=sr,- text=["continue with a epic chorus"],- padding=True,- return_tensors="pt"-)--# Generate continuation-audio_values = model.generate(**inputs, do_sample=True, guidance_scale=3, max_new_tokens=512)-```--## MusicGen-Style usage--### Style-conditioned generation--```python-from audiocraft.models import MusicGen--# Load style model-model = MusicGen.get_pretrained('facebook/musicgen-style')--# Configure generation with style-model.set_generation_params(- duration=30,- cfg_coef=3.0,- cfg_coef_beta=5.0 # Style influence-)--# Configure style conditioner-model.set_style_conditioner_params(- eval_q=3, # RVQ quantizers (1-6)- excerpt_length=3.0 # Style excerpt length-)--# Load style reference-style_audio, sr = torchaudio.load("reference_style.wav")--# Generate with text + style-descriptions = ["upbeat dance track"]-wav = model.generate_with_style(descriptions, style_audio, sr)-```--### Style-only generation (no text)--```python-# Generate matching style without text prompt-model.set_generation_params(- duration=30,- cfg_coef=3.0,- cfg_coef_beta=None # Disable double CFG for style-only-)--wav = model.generate_with_style([None], style_audio, sr)-```--## AudioGen usage--### Sound effect generation--```python-from audiocraft.models import AudioGen-import torchaudio--model = AudioGen.get_pretrained('facebook/audiogen-medium')-model.set_generation_params(duration=10)--# Generate various sounds-descriptions = [- "thunderstorm with heavy rain and lightning",- "busy city traffic with car horns",- "ocean waves crashing on rocks",- "crackling campfire in forest"-]--wav = model.generate(descriptions)--for i, audio in enumerate(wav):- torchaudio.save(f"sound_{i}.wav", audio.cpu(), sample_rate=16000)-```--## EnCodec usage--### Audio compression--```python-from audiocraft.models import CompressionModel-import torch-import torchaudio--# Load EnCodec-model = CompressionModel.get_pretrained('facebook/encodec_32khz')--# Load audio-wav, sr = torchaudio.load("audio.wav")--# Ensure correct sample rate-if sr != 32000:- resampler = torchaudio.transforms.Resample(sr, 32000)- wav = resampler(wav)--# Encode to tokens-with torch.no_grad():- encoded = model.encode(wav.unsqueeze(0))- codes = encoded[0] # Audio codes--# Decode back to audio-with torch.no_grad():- decoded = model.decode(codes)--torchaudio.save("reconstructed.wav", decoded[0].cpu(), sample_rate=32000)-```--## Common workflows--### Workflow 1: Music generation pipeline--```python-import torch-import torchaudio-from audiocraft.models import MusicGen--class MusicGenerator:- def __init__(self, model_name="facebook/musicgen-medium"):- self.model = MusicGen.get_pretrained(model_name)- self.sample_rate = 32000-- def generate(self, prompt, duration=30, temperature=1.0, cfg=3.0):- self.model.set_generation_params(- duration=duration,- top_k=250,- temperature=temperature,- cfg_coef=cfg- )-- with torch.no_grad():- wav = self.model.generate([prompt])-- return wav[0].cpu()-- def generate_batch(self, prompts, duration=30):- self.model.set_generation_params(duration=duration)-- with torch.no_grad():- wav = self.model.generate(prompts)-- return wav.cpu()-- def save(self, audio, path):- torchaudio.save(path, audio, sample_rate=self.sample_rate)--# Usage-generator = MusicGenerator()-audio = generator.generate(- "epic cinematic orchestral music",- duration=30,- temperature=1.0-)-generator.save(audio, "epic_music.wav")-```--### Workflow 2: Sound design batch processing--```python-import json-from pathlib import Path-from audiocraft.models import AudioGen-import torchaudio--def batch_generate_sounds(sound_specs, output_dir):- """- Generate multiple sounds from specifications.-- Args:- sound_specs: list of {"name": str, "description": str, "duration": float}- output_dir: output directory path- """- model = AudioGen.get_pretrained('facebook/audiogen-medium')- output_dir = Path(output_dir)- output_dir.mkdir(exist_ok=True)-- results = []-- for spec in sound_specs:- model.set_generation_params(duration=spec.get("duration", 5))-- wav = model.generate([spec["description"]])-- output_path = output_dir / f"{spec['name']}.wav"- torchaudio.save(str(output_path), wav[0].cpu(), sample_rate=16000)-- results.append({- "name": spec["name"],- "path": str(output_path),- "description": spec["description"]- })-- return results--# Usage-sounds = [- {"name": "explosion", "description": "massive explosion with debris", "duration": 3},- {"name": "footsteps", "description": "footsteps on wooden floor", "duration": 5},- {"name": "door", "description": "wooden door creaking and closing", "duration": 2}-]--results = batch_generate_sounds(sounds, "sound_effects/")-```--### Workflow 3: Gradio demo--```python-import gradio as gr-import torch-import torchaudio-from audiocraft.models import MusicGen--model = MusicGen.get_pretrained('facebook/musicgen-small')--def generate_music(prompt, duration, temperature, cfg_coef):- model.set_generation_params(- duration=duration,- temperature=temperature,- cfg_coef=cfg_coef- )-- with torch.no_grad():- wav = model.generate([prompt])-- # Save to temp file- path = "temp_output.wav"- torchaudio.save(path, wav[0].cpu(), sample_rate=32000)- return path--demo = gr.Interface(- fn=generate_music,- inputs=[- gr.Textbox(label="Music Description", placeholder="upbeat electronic dance music"),- gr.Slider(1, 30, value=8, label="Duration (seconds)"),- gr.Slider(0.5, 2.0, value=1.0, label="Temperature"),- gr.Slider(1.0, 10.0, value=3.0, label="CFG Coefficient")- ],- outputs=gr.Audio(label="Generated Music"),- title="MusicGen Demo"-)--demo.launch()-```--## Performance optimization--### Memory optimization--```python-# Use smaller model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Clear cache between generations-torch.cuda.empty_cache()--# Generate shorter durations-model.set_generation_params(duration=10) # Instead of 30--# Use half precision-model = model.half()-```--### Batch processing efficiency--```python-# Process multiple prompts at once (more efficient)-descriptions = ["prompt1", "prompt2", "prompt3", "prompt4"]-wav = model.generate(descriptions) # Single batch--# Instead of-for desc in descriptions:- wav = model.generate([desc]) # Multiple batches (slower)-```--### GPU memory requirements--| Model | FP32 VRAM | FP16 VRAM |-|-------|-----------|-----------|-| musicgen-small | ~4GB | ~2GB |-| musicgen-medium | ~8GB | ~4GB |-| musicgen-large | ~16GB | ~8GB |--## Common issues--| Issue | Solution |-|-------|----------|-| CUDA OOM | Use smaller model, reduce duration |-| Poor quality | Increase cfg_coef, better prompts |-| Generation too short | Check max duration setting |-| Audio artifacts | Try different temperature |-| Stereo not working | Use stereo model variant |--## References--- **[Advanced Usage](references/advanced-usage.md)** - Training, fine-tuning, deployment-- **[Troubleshooting](references/troubleshooting.md)** - Common issues and solutions--## Resources--- **GitHub**: https://github.com/facebookresearch/audiocraft-- **Paper (MusicGen)**: https://arxiv.org/abs/2306.05284-- **Paper (AudioGen)**: https://arxiv.org/abs/2209.15352-- **HuggingFace**: https://huggingface.co/facebook/musicgen-small-- **Demo**: https://huggingface.co/spaces/facebook/MusicGendiff --git a/optional-skills/creative/audiocraft-audio-generation/references/advanced-usage.md b/optional-skills/creative/audiocraft-audio-generation/references/advanced-usage.mddeleted file mode 100644index 953be2b4a56..00000000000--- a/optional-skills/creative/audiocraft-audio-generation/references/advanced-usage.md+++ /dev/null@@ -1,666 +0,0 @@-# AudioCraft Advanced Usage Guide--## Fine-tuning MusicGen--### Custom dataset preparation--```python-import os-import json-from pathlib import Path-import torchaudio--def prepare_dataset(audio_dir, output_dir, metadata_file):- """- Prepare dataset for MusicGen fine-tuning.-- Directory structure:- output_dir/- ├── audio/- │ ├── 0001.wav- │ ├── 0002.wav- │ └── ...- └── metadata.json- """- output_dir = Path(output_dir)- audio_output = output_dir / "audio"- audio_output.mkdir(parents=True, exist_ok=True)-- # Load metadata (format: {"path": "...", "description": "..."})- with open(metadata_file) as f:- metadata = json.load(f)-- processed = []-- for idx, item in enumerate(metadata):- audio_path = Path(audio_dir) / item["path"]-- # Load and resample to 32kHz- wav, sr = torchaudio.load(str(audio_path))- if sr != 32000:- resampler = torchaudio.transforms.Resample(sr, 32000)- wav = resampler(wav)-- # Convert to mono if stereo- if wav.shape[0] > 1:- wav = wav.mean(dim=0, keepdim=True)-- # Save processed audio- output_path = audio_output / f"{idx:04d}.wav"- torchaudio.save(str(output_path), wav, sample_rate=32000)-- processed.append({- "path": str(output_path.relative_to(output_dir)),- "description": item["description"],- "duration": wav.shape[1] / 32000- })-- # Save processed metadata- with open(output_dir / "metadata.json", "w") as f:- json.dump(processed, f, indent=2)-- print(f"Processed {len(processed)} samples")- return processed-```--### Fine-tuning with dora--```bash-# AudioCraft uses dora for experiment management-# Install dora-pip install dora-search--# Clone AudioCraft-git clone https://github.com/facebookresearch/audiocraft.git-cd audiocraft--# Create config for fine-tuning-cat > config/solver/musicgen/finetune.yaml << 'EOF'-defaults:- - musicgen/musicgen_base- - /model: lm/musicgen_lm- - /conditioner: cond_base--solver: musicgen-autocast: true-autocast_dtype: float16--optim:- epochs: 100- batch_size: 4- lr: 1e-4- ema: 0.999- optimizer: adamw--dataset:- batch_size: 4- num_workers: 4- train:- - dset: your_dataset- root: /path/to/dataset- valid:- - dset: your_dataset- root: /path/to/dataset--checkpoint:- save_every: 10- keep_every_states: null-EOF--# Run fine-tuning-dora run solver=musicgen/finetune-```--### LoRA fine-tuning--```python-from peft import LoraConfig, get_peft_model-from audiocraft.models import MusicGen-import torch--# Load base model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Get the language model component-lm = model.lm--# Configure LoRA-lora_config = LoraConfig(- r=8,- lora_alpha=16,- target_modules=["q_proj", "v_proj", "k_proj", "out_proj"],- lora_dropout=0.05,- bias="none"-)--# Apply LoRA-lm = get_peft_model(lm, lora_config)-lm.print_trainable_parameters()-```--## Multi-GPU Training--### DataParallel--```python-import torch-import torch.nn as nn-from audiocraft.models import MusicGen--model = MusicGen.get_pretrained('facebook/musicgen-small')--# Wrap LM with DataParallel-if torch.cuda.device_count() > 1:- model.lm = nn.DataParallel(model.lm)--model.to("cuda")-```--### DistributedDataParallel--```python-import torch.distributed as dist-from torch.nn.parallel import DistributedDataParallel as DDP--def setup(rank, world_size):- dist.init_process_group("nccl", rank=rank, world_size=world_size)- torch.cuda.set_device(rank)--def train(rank, world_size):- setup(rank, world_size)-- model = MusicGen.get_pretrained('facebook/musicgen-small')- model.lm = model.lm.to(rank)- model.lm = DDP(model.lm, device_ids=[rank])-- # Training loop- # ...-- dist.destroy_process_group()-```--## Custom Conditioning--### Adding new conditioners--```python-from audiocraft.modules.conditioners import BaseConditioner-import torch--class CustomConditioner(BaseConditioner):- """Custom conditioner for additional control signals."""-- def __init__(self, dim, output_dim):- super().__init__(dim, output_dim)- self.embed = torch.nn.Linear(dim, output_dim)-- def forward(self, x):- return self.embed(x)-- def tokenize(self, x):- # Tokenize input for conditioning- return x--# Use with MusicGen-from audiocraft.models.builders import get_lm_model--# Modify model config to include custom conditioner-# This requires editing the model configuration-```--### Melody conditioning internals--```python-from audiocraft.models import MusicGen-from audiocraft.modules.codebooks_patterns import DelayedPatternProvider-import torch--model = MusicGen.get_pretrained('facebook/musicgen-melody')--# Access chroma extractor-chroma_extractor = model.lm.condition_provider.conditioners.get('chroma')--# Manual chroma extraction-def extract_chroma(audio, sr):- """Extract chroma features from audio."""- import librosa-- # Compute chroma- chroma = librosa.feature.chroma_cqt(y=audio.numpy(), sr=sr)-- return torch.from_numpy(chroma).float()--# Use extracted chroma for conditioning-chroma = extract_chroma(melody_audio, sample_rate)-```--## EnCodec Deep Dive--### Custom compression settings--```python-from audiocraft.models import CompressionModel-import torch--# Load EnCodec-encodec = CompressionModel.get_pretrained('facebook/encodec_32khz')--# Access codec parameters-print(f"Sample rate: {encodec.sample_rate}")-print(f"Channels: {encodec.channels}")-print(f"Cardinality: {encodec.cardinality}") # Codebook size-print(f"Num codebooks: {encodec.num_codebooks}")-print(f"Frame rate: {encodec.frame_rate}")--# Encode with specific bandwidth-# Lower bandwidth = more compression, lower quality-encodec.set_target_bandwidth(6.0) # 6 kbps--audio = torch.randn(1, 1, 32000) # 1 second-encoded = encodec.encode(audio)-decoded = encodec.decode(encoded[0])-```--### Streaming encoding--```python-import torch-from audiocraft.models import CompressionModel--encodec = CompressionModel.get_pretrained('facebook/encodec_32khz')--def encode_streaming(audio_stream, chunk_size=32000):- """Encode audio in streaming fashion."""- all_codes = []-- for chunk in audio_stream:- # Ensure chunk is right shape- if chunk.dim() == 1:- chunk = chunk.unsqueeze(0).unsqueeze(0)-- with torch.no_grad():- codes = encodec.encode(chunk)[0]- all_codes.append(codes)-- return torch.cat(all_codes, dim=-1)--def decode_streaming(codes_stream, output_stream):- """Decode codes in streaming fashion."""- for codes in codes_stream:- with torch.no_grad():- audio = encodec.decode(codes)- output_stream.write(audio.cpu().numpy())-```--## MultiBand Diffusion--### Using MBD for enhanced quality--```python-from audiocraft.models import MusicGen, MultiBandDiffusion--# Load MusicGen-model = MusicGen.get_pretrained('facebook/musicgen-medium')--# Load MultiBand Diffusion-mbd = MultiBandDiffusion.get_mbd_musicgen()--model.set_generation_params(duration=10)--# Generate with standard decoder-descriptions = ["epic orchestral music"]-wav_standard = model.generate(descriptions)--# Generate tokens and use MBD decoder-with torch.no_grad():- # Get tokens- gen_tokens = model.generate_tokens(descriptions)-- # Decode with MBD- wav_mbd = mbd.tokens_to_wav(gen_tokens)--# Compare quality-print(f"Standard shape: {wav_standard.shape}")-print(f"MBD shape: {wav_mbd.shape}")-```--## API Server Deployment--### FastAPI server--```python-from fastapi import FastAPI, HTTPException-from pydantic import BaseModel-import torch-import torchaudio-from audiocraft.models import MusicGen-import io-import base64--app = FastAPI()--# Load model at startup-model = None-[email protected]_event("startup")-async def load_model():- global model- model = MusicGen.get_pretrained('facebook/musicgen-small')- model.set_generation_params(duration=10)--class GenerateRequest(BaseModel):- prompt: str- duration: float = 10.0- temperature: float = 1.0- cfg_coef: float = 3.0--class GenerateResponse(BaseModel):- audio_base64: str- sample_rate: int- duration: float-[email protected]("/generate", response_model=GenerateResponse)-async def generate(request: GenerateRequest):- if model is None:- raise HTTPException(status_code=500, detail="Model not loaded")-- try:- model.set_generation_params(- duration=min(request.duration, 30),- temperature=request.temperature,- cfg_coef=request.cfg_coef- )-- with torch.no_grad():- wav = model.generate([request.prompt])-- # Convert to bytes- buffer = io.BytesIO()- torchaudio.save(buffer, wav[0].cpu(), sample_rate=32000, format="wav")- buffer.seek(0)-- audio_base64 = base64.b64encode(buffer.read()).decode()-- return GenerateResponse(- audio_base64=audio_base64,- sample_rate=32000,- duration=wav.shape[-1] / 32000- )-- except Exception as e:- raise HTTPException(status_code=500, detail=str(e))-[email protected]("/health")-async def health():- return {"status": "ok", "model_loaded": model is not None}--# Run: uvicorn server:app --host 0.0.0.0 --port 8000-```--### Batch processing service--```python-import asyncio-from concurrent.futures import ThreadPoolExecutor-import torch-from audiocraft.models import MusicGen--class MusicGenService:- def __init__(self, model_name='facebook/musicgen-small', max_workers=2):- self.model = MusicGen.get_pretrained(model_name)- self.executor = ThreadPoolExecutor(max_workers=max_workers)- self.lock = asyncio.Lock()-- async def generate_async(self, prompt, duration=10):- """Async generation with thread pool."""- loop = asyncio.get_event_loop()-- def _generate():- with torch.no_grad():- self.model.set_generation_params(duration=duration)- return self.model.generate([prompt])-- # Run in thread pool- wav = await loop.run_in_executor(self.executor, _generate)- return wav[0].cpu()-- async def generate_batch_async(self, prompts, duration=10):- """Process multiple prompts concurrently."""- tasks = [self.generate_async(p, duration) for p in prompts]- return await asyncio.gather(*tasks)--# Usage-service = MusicGenService()--async def main():- prompts = ["jazz piano", "rock guitar", "electronic beats"]- results = await service.generate_batch_async(prompts)- return results-```--## Integration Patterns--### LangChain tool--```python-from langchain.tools import BaseTool-import torch-import torchaudio-from audiocraft.models import MusicGen-import tempfile--class MusicGeneratorTool(BaseTool):- name = "music_generator"- description = "Generate music from a text description. Input should be a detailed description of the music style, mood, and instruments."-- def __init__(self):- super().__init__()- self.model = MusicGen.get_pretrained('facebook/musicgen-small')- self.model.set_generation_params(duration=15)-- def _run(self, description: str) -> str:- with torch.no_grad():- wav = self.model.generate([description])-- # Save to temp file- with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as f:- torchaudio.save(f.name, wav[0].cpu(), sample_rate=32000)- return f"Generated music saved to: {f.name}"-- async def _arun(self, description: str) -> str:- return self._run(description)-```--### Gradio with advanced controls--```python-import gradio as gr-import torch-import torchaudio-from audiocraft.models import MusicGen--models = {}--def load_model(model_size):- if model_size not in models:- model_name = f"facebook/musicgen-{model_size}"- models[model_size] = MusicGen.get_pretrained(model_name)- return models[model_size]--def generate(prompt, duration, temperature, cfg_coef, top_k, model_size):- model = load_model(model_size)-- model.set_generation_params(- duration=duration,- temperature=temperature,- cfg_coef=cfg_coef,- top_k=top_k- )-- with torch.no_grad():- wav = model.generate([prompt])-- # Save- path = "output.wav"- torchaudio.save(path, wav[0].cpu(), sample_rate=32000)- return path--demo = gr.Interface(- fn=generate,- inputs=[- gr.Textbox(label="Prompt", lines=3),- gr.Slider(1, 30, value=10, label="Duration (s)"),- gr.Slider(0.1, 2.0, value=1.0, label="Temperature"),- gr.Slider(0.5, 10.0, value=3.0, label="CFG Coefficient"),- gr.Slider(50, 500, value=250, step=50, label="Top-K"),- gr.Dropdown(["small", "medium", "large"], value="small", label="Model Size")- ],- outputs=gr.Audio(label="Generated Music"),- title="MusicGen Advanced",- allow_flagging="never"-)--demo.launch(share=True)-```--## Audio Processing Pipeline--### Post-processing chain--```python-import torch-import torchaudio-import torchaudio.transforms as T-import numpy as np--class AudioPostProcessor:- def __init__(self, sample_rate=32000):- self.sample_rate = sample_rate-- def normalize(self, audio, target_db=-14.0):- """Normalize audio to target loudness."""- rms = torch.sqrt(torch.mean(audio ** 2))- target_rms = 10 ** (target_db / 20)- gain = target_rms / (rms + 1e-8)- return audio * gain-- def fade_in_out(self, audio, fade_duration=0.1):- """Apply fade in/out."""- fade_samples = int(fade_duration * self.sample_rate)-- # Create fade curves- fade_in = torch.linspace(0, 1, fade_samples)- fade_out = torch.linspace(1, 0, fade_samples)-- # Apply fades- audio[..., :fade_samples] *= fade_in- audio[..., -fade_samples:] *= fade_out-- return audio-- def apply_reverb(self, audio, decay=0.5):- """Apply simple reverb effect."""- impulse = torch.zeros(int(self.sample_rate * 0.5))- impulse[0] = 1.0- impulse[int(self.sample_rate * 0.1)] = decay * 0.5- impulse[int(self.sample_rate * 0.2)] = decay * 0.25-- # Convolve- audio = torch.nn.functional.conv1d(- audio.unsqueeze(0),- impulse.unsqueeze(0).unsqueeze(0),- padding=len(impulse) // 2- ).squeeze(0)-- return audio-- def process(self, audio):- """Full processing pipeline."""- audio = self.normalize(audio)- audio = self.fade_in_out(audio)- return audio--# Usage with MusicGen-from audiocraft.models import MusicGen--model = MusicGen.get_pretrained('facebook/musicgen-small')-model.set_generation_params(duration=10)--wav = model.generate(["chill ambient music"])-processor = AudioPostProcessor()-wav_processed = processor.process(wav[0].cpu())--torchaudio.save("processed.wav", wav_processed, sample_rate=32000)-```--## Evaluation--### Audio quality metrics--```python-import torch-from audiocraft.metrics import CLAPTextConsistencyMetric-from audiocraft.data.audio import audio_read--def evaluate_generation(audio_path, text_prompt):- """Evaluate generated audio quality."""- # Load audio- wav, sr = audio_read(audio_path)-- # CLAP consistency (text-audio alignment)- clap_metric = CLAPTextConsistencyMetric()- clap_score = clap_metric.compute(wav, [text_prompt])-- return {- "clap_score": clap_score,- "duration": wav.shape[-1] / sr- }--# Batch evaluation-def evaluate_batch(generations):- """Evaluate multiple generations."""- results = []- for gen in generations:- result = evaluate_generation(gen["path"], gen["prompt"])- result["prompt"] = gen["prompt"]- results.append(result)-- # Aggregate- avg_clap = sum(r["clap_score"] for r in results) / len(results)- return {- "individual": results,- "average_clap": avg_clap- }-```--## Model Comparison--### MusicGen variants benchmark--| Model | CLAP Score | Generation Time (10s) | VRAM |-|-------|------------|----------------------|------|-| musicgen-small | 0.35 | ~5s | 2GB |-| musicgen-medium | 0.42 | ~15s | 4GB |-| musicgen-large | 0.48 | ~30s | 8GB |-| musicgen-melody | 0.45 | ~15s | 4GB |-| musicgen-stereo-medium | 0.41 | ~18s | 5GB |--### Prompt engineering tips--```python-# Good prompts - specific and descriptive-good_prompts = [- "upbeat electronic dance music with synthesizer leads and punchy drums at 128 bpm",- "melancholic piano ballad with strings, slow tempo, emotional and cinematic",- "funky disco groove with slap bass, brass section, and rhythmic guitar"-]--# Bad prompts - too vague-bad_prompts = [- "nice music",- "song",- "good beat"-]--# Structure: [mood] [genre] with [instruments] at [tempo/style]-```diff --git a/optional-skills/creative/audiocraft-audio-generation/references/troubleshooting.md b/optional-skills/creative/audiocraft-audio-generation/references/troubleshooting.mddeleted file mode 100644index 7b83e863d65..00000000000--- a/optional-skills/creative/audiocraft-audio-generation/references/troubleshooting.md+++ /dev/null@@ -1,504 +0,0 @@-# AudioCraft Troubleshooting Guide--## Installation Issues--### Import errors--**Error**: `ModuleNotFoundError: No module named 'audiocraft'`--**Solutions**:-```bash-# Install from PyPI-pip install audiocraft--# Or from GitHub-pip install git+https://github.com/facebookresearch/audiocraft.git--# Verify installation-python -c "from audiocraft.models import MusicGen; print('OK')"-```--### FFmpeg not found--**Error**: `RuntimeError: ffmpeg not found`--**Solutions**:-```bash-# Ubuntu/Debian-sudo apt-get install ffmpeg--# macOS-brew install ffmpeg--# Windows (using conda)-conda install -c conda-forge ffmpeg--# Verify-ffmpeg -version-```--### PyTorch CUDA mismatch--**Error**: `RuntimeError: CUDA error: no kernel image is available`--**Solutions**:-```bash-# Check CUDA version-nvcc --version-python -c "import torch; print(torch.version.cuda)"--# Install matching PyTorch-pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121--# For CUDA 11.8-pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu118-```--### xformers issues--**Error**: `ImportError: xformers` related errors--**Solutions**:-```bash-# Install xformers for memory efficiency-pip install xformers--# Or disable xformers-export AUDIOCRAFT_USE_XFORMERS=0--# In Python-import os-os.environ["AUDIOCRAFT_USE_XFORMERS"] = "0"-from audiocraft.models import MusicGen-```--## Model Loading Issues--### Out of memory during load--**Error**: `torch.cuda.OutOfMemoryError` during model loading--**Solutions**:-```python-# Use smaller model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Force CPU loading first-import torch-device = "cpu"-model = MusicGen.get_pretrained('facebook/musicgen-small', device=device)-model = model.to("cuda")--# Use HuggingFace with device_map-from transformers import MusicgenForConditionalGeneration-model = MusicgenForConditionalGeneration.from_pretrained(- "facebook/musicgen-small",- device_map="auto"-)-```--### Download failures--**Error**: Connection errors or incomplete downloads--**Solutions**:-```python-# Set cache directory-import os-os.environ["AUDIOCRAFT_CACHE_DIR"] = "/path/to/cache"--# Or for HuggingFace-os.environ["HF_HOME"] = "/path/to/hf_cache"--# Resume download-from huggingface_hub import snapshot_download-snapshot_download("facebook/musicgen-small", resume_download=True)--# Use local files-model = MusicGen.get_pretrained('/local/path/to/model')-```--### Wrong model type--**Error**: Loading wrong model for task--**Solutions**:-```python-# For text-to-music: use MusicGen-from audiocraft.models import MusicGen-model = MusicGen.get_pretrained('facebook/musicgen-medium')--# For text-to-sound: use AudioGen-from audiocraft.models import AudioGen-model = AudioGen.get_pretrained('facebook/audiogen-medium')--# For melody conditioning: use melody variant-model = MusicGen.get_pretrained('facebook/musicgen-melody')--# For stereo: use stereo variant-model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')-```--## Generation Issues--### Empty or silent output--**Problem**: Generated audio is silent or very quiet--**Solutions**:-```python-import torch--# Check output-wav = model.generate(["upbeat music"])-print(f"Shape: {wav.shape}")-print(f"Max amplitude: {wav.abs().max().item()}")-print(f"Mean amplitude: {wav.abs().mean().item()}")--# If too quiet, normalize-def normalize_audio(audio, target_db=-14.0):- rms = torch.sqrt(torch.mean(audio ** 2))- target_rms = 10 ** (target_db / 20)- gain = target_rms / (rms + 1e-8)- return audio * gain--wav_normalized = normalize_audio(wav)-```--### Poor quality output--**Problem**: Generated music sounds bad or noisy--**Solutions**:-```python-# Use larger model-model = MusicGen.get_pretrained('facebook/musicgen-large')--# Adjust generation parameters-model.set_generation_params(- duration=15,- top_k=250, # Increase for more diversity- temperature=0.8, # Lower for more focused output- cfg_coef=4.0 # Increase for better text adherence-)--# Use better prompts-# Bad: "music"-# Good: "upbeat electronic dance music with synthesizers and punchy drums"--# Try MultiBand Diffusion-from audiocraft.models import MultiBandDiffusion-mbd = MultiBandDiffusion.get_mbd_musicgen()-tokens = model.generate_tokens(["prompt"])-wav = mbd.tokens_to_wav(tokens)-```--### Generation too short--**Problem**: Audio shorter than expected--**Solutions**:-```python-# Check duration setting-model.set_generation_params(duration=30) # Set before generate--# Verify in generation-print(f"Duration setting: {model.generation_params}")--# Check output shape-wav = model.generate(["prompt"])-actual_duration = wav.shape[-1] / 32000-print(f"Actual duration: {actual_duration}s")--# Note: max duration is typically 30s-```--### Melody conditioning fails--**Error**: Issues with melody-conditioned generation--**Solutions**:-```python-import torchaudio-from audiocraft.models import MusicGen--# Load melody model (not base model)-model = MusicGen.get_pretrained('facebook/musicgen-melody')--# Load and prepare melody-melody, sr = torchaudio.load("melody.wav")--# Resample to model sample rate if needed-if sr != 32000:- resampler = torchaudio.transforms.Resample(sr, 32000)- melody = resampler(melody)--# Ensure correct shape [batch, channels, samples]-if melody.dim() == 1:- melody = melody.unsqueeze(0).unsqueeze(0)-elif melody.dim() == 2:- melody = melody.unsqueeze(0)--# Convert stereo to mono-if melody.shape[1] > 1:- melody = melody.mean(dim=1, keepdim=True)--# Generate with melody-model.set_generation_params(duration=min(melody.shape[-1] / 32000, 30))-wav = model.generate_with_chroma(["piano cover"], melody, 32000)-```--## Memory Issues--### CUDA out of memory--**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory`--**Solutions**:-```python-import torch--# Clear cache before generation-torch.cuda.empty_cache()--# Use smaller model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Reduce duration-model.set_generation_params(duration=10) # Instead of 30--# Generate one at a time-for prompt in prompts:- wav = model.generate([prompt])- save_audio(wav)- torch.cuda.empty_cache()--# Use CPU for very large generations-model = MusicGen.get_pretrained('facebook/musicgen-small', device="cpu")-```--### Memory leak during batch processing--**Problem**: Memory grows over time--**Solutions**:-```python-import gc-import torch--def generate_with_cleanup(model, prompts):- results = []-- for prompt in prompts:- with torch.no_grad():- wav = model.generate([prompt])- results.append(wav.cpu())-- # Cleanup- del wav- gc.collect()- torch.cuda.empty_cache()-- return results--# Use context manager-with torch.inference_mode():- wav = model.generate(["prompt"])-```--## Audio Format Issues--### Wrong sample rate--**Problem**: Audio plays at wrong speed--**Solutions**:-```python-import torchaudio--# MusicGen outputs at 32kHz-sample_rate = 32000--# AudioGen outputs at 16kHz-sample_rate = 16000--# Always use correct rate when saving-torchaudio.save("output.wav", wav[0].cpu(), sample_rate=sample_rate)--# Resample if needed-resampler = torchaudio.transforms.Resample(32000, 44100)-wav_resampled = resampler(wav)-```--### Stereo/mono mismatch--**Problem**: Wrong number of channels--**Solutions**:-```python-# Check model type-print(f"Audio channels: {wav.shape}")-# Mono: [batch, 1, samples]-# Stereo: [batch, 2, samples]--# Convert mono to stereo-if wav.shape[1] == 1:- wav_stereo = wav.repeat(1, 2, 1)--# Convert stereo to mono-if wav.shape[1] == 2:- wav_mono = wav.mean(dim=1, keepdim=True)--# Use stereo model for stereo output-model = MusicGen.get_pretrained('facebook/musicgen-stereo-medium')-```--### Clipping and distortion--**Problem**: Audio has clipping or distortion--**Solutions**:-```python-import torch--# Check for clipping-max_val = wav.abs().max().item()-print(f"Max amplitude: {max_val}")--# Normalize to prevent clipping-if max_val > 1.0:- wav = wav / max_val--# Apply soft clipping-def soft_clip(x, threshold=0.9):- return torch.tanh(x / threshold) * threshold--wav_clipped = soft_clip(wav)--# Lower temperature during generation-model.set_generation_params(temperature=0.7) # More controlled-```--## HuggingFace Transformers Issues--### Processor errors--**Error**: Issues with MusicgenProcessor--**Solutions**:-```python-from transformers import AutoProcessor, MusicgenForConditionalGeneration--# Load matching processor and model-processor = AutoProcessor.from_pretrained("facebook/musicgen-small")-model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")--# Ensure inputs are on same device-inputs = processor(- text=["prompt"],- padding=True,- return_tensors="pt"-).to("cuda")--# Check processor configuration-print(processor.tokenizer)-print(processor.feature_extractor)-```--### Generation parameter errors--**Error**: Invalid generation parameters--**Solutions**:-```python-# HuggingFace uses different parameter names-audio_values = model.generate(- **inputs,- do_sample=True, # Enable sampling- guidance_scale=3.0, # CFG (not cfg_coef)- max_new_tokens=256, # Token limit (not duration)- temperature=1.0-)--# Calculate tokens from duration-# ~50 tokens per second-duration_seconds = 10-max_tokens = duration_seconds * 50-audio_values = model.generate(**inputs, max_new_tokens=max_tokens)-```--## Performance Issues--### Slow generation--**Problem**: Generation takes too long--**Solutions**:-```python-# Use smaller model-model = MusicGen.get_pretrained('facebook/musicgen-small')--# Reduce duration-model.set_generation_params(duration=10)--# Use GPU-model.to("cuda")--# Enable flash attention if available-# (requires compatible hardware)--# Batch multiple prompts-prompts = ["prompt1", "prompt2", "prompt3"]-wav = model.generate(prompts) # Single batch is faster than loop--# Use compile (PyTorch 2.0+)-model.lm = torch.compile(model.lm)-```--### CPU fallback--**Problem**: Generation running on CPU instead of GPU--**Solutions**:-```python-import torch--# Check CUDA availability-print(f"CUDA available: {torch.cuda.is_available()}")-print(f"CUDA device: {torch.cuda.get_device_name(0)}")--# Explicitly move to GPU-model = MusicGen.get_pretrained('facebook/musicgen-small')-model.to("cuda")--# Verify model device-print(f"Model device: {next(model.lm.parameters()).device}")-```--## Common Error Messages--| Error | Cause | Solution |-|-------|-------|----------|-| `CUDA out of memory` | Model too large | Use smaller model, reduce duration |-| `ffmpeg not found` | FFmpeg not installed | Install FFmpeg |-| `No module named 'audiocraft'` | Not installed | `pip install audiocraft` |-| `RuntimeError: Expected 3D tensor` | Wrong input shape | Check tensor dimensions |-| `KeyError: 'melody'` | Wrong model for melody | Use musicgen-melody |-| `Sample rate mismatch` | Wrong audio format | Resample to model rate |--## Getting Help--1. **GitHub Issues**: https://github.com/facebookresearch/audiocraft/issues-2. **HuggingFace Forums**: https://discuss.huggingface.co-3. **Paper**: https://arxiv.org/abs/2306.05284--### Reporting Issues--Include:-- Python version-- PyTorch version-- CUDA version-- AudioCraft version: `pip show audiocraft`-- Full error traceback-- Minimal reproducible code-- Hardware (GPU model, VRAM)diff --git a/optional-skills/creative/comfyui/SKILL.md b/optional-skills/creative/comfyui/SKILL.mdindex 0db5b97cc1c..60b576a3d2d 100644--- a/optional-skills/creative/comfyui/SKILL.md+++ b/optional-skills/creative/comfyui/SKILL.md@@ -23,7 +23,6 @@ metadata:- creative- generative-ai- video-generation- related_skills: [stable-diffusion]category: creative---diff --git a/optional-skills/creative/heartmula/SKILL.md b/optional-skills/creative/heartmula/SKILL.mdindex 9746b9506ba..0e62cd383d2 100644--- a/optional-skills/creative/heartmula/SKILL.md+++ b/optional-skills/creative/heartmula/SKILL.md@@ -8,7 +8,7 @@ platforms: [linux, macos]metadata:hermes:tags: [music, audio, generation, ai, heartmula, heartcodec, lyrics, songs]- related_skills: [audiocraft-audio-generation, songwriting-and-ai-music]+ related_skills: [songwriting-and-ai-music]---# HeartMuLa - Open-Source Music Generationdiff --git a/optional-skills/mlops/clip/SKILL.md b/optional-skills/mlops/clip/SKILL.mddeleted file mode 100644index 750510ec4b0..00000000000--- a/optional-skills/mlops/clip/SKILL.md+++ /dev/null@@ -1,257 +0,0 @@-----name: clip-description: Zero-shot image classification and image-text search.-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [transformers, torch, pillow]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Multimodal, CLIP, Vision-Language, Zero-Shot, Image Classification, OpenAI, Image Search, Cross-Modal Retrieval, Content Moderation]-------# CLIP - Contrastive Language-Image Pre-Training--OpenAI's model that understands images from natural language.--## When to use CLIP--**Use when:**-- Zero-shot image classification (no training data needed)-- Image-text similarity/matching-- Semantic image search-- Content moderation (detect NSFW, violence)-- Visual question answering-- Cross-modal retrieval (image→text, text→image)--**Metrics**:-- **25,300+ GitHub stars**-- Trained on 400M image-text pairs-- Matches ResNet-50 on ImageNet (zero-shot)-- MIT License--**Use alternatives instead**:-- **BLIP-2**: Better captioning-- **LLaVA**: Vision-language chat-- **Segment Anything**: Image segmentation--## Quick start--### Installation--```bash-pip install git+https://github.com/openai/CLIP.git-pip install torch torchvision ftfy regex tqdm-```--### Zero-shot classification--```python-import torch-import clip-from PIL import Image--# Load model-device = "cuda" if torch.cuda.is_available() else "cpu"-model, preprocess = clip.load("ViT-B/32", device=device)--# Load image-image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)--# Define possible labels-text = clip.tokenize(["a dog", "a cat", "a bird", "a car"]).to(device)--# Compute similarity-with torch.no_grad():- image_features = model.encode_image(image)- text_features = model.encode_text(text)-- # Cosine similarity- logits_per_image, logits_per_text = model(image, text)- probs = logits_per_image.softmax(dim=-1).cpu().numpy()--# Print results-labels = ["a dog", "a cat", "a bird", "a car"]-for label, prob in zip(labels, probs[0]):- print(f"{label}: {prob:.2%}")-```--## Available models--```python-# Models (sorted by size)-models = [- "RN50", # ResNet-50- "RN101", # ResNet-101- "ViT-B/32", # Vision Transformer (recommended)- "ViT-B/16", # Better quality, slower- "ViT-L/14", # Best quality, slowest-]--model, preprocess = clip.load("ViT-B/32")-```--| Model | Parameters | Speed | Quality |-|-------|------------|-------|---------|-| RN50 | 102M | Fast | Good |-| ViT-B/32 | 151M | Medium | Better |-| ViT-L/14 | 428M | Slow | Best |--## Image-text similarity--```python-# Compute embeddings-image_features = model.encode_image(image)-text_features = model.encode_text(text)--# Normalize-image_features /= image_features.norm(dim=-1, keepdim=True)-text_features /= text_features.norm(dim=-1, keepdim=True)--# Cosine similarity-similarity = (image_features @ text_features.T).item()-print(f"Similarity: {similarity:.4f}")-```--## Semantic image search--```python-# Index images-image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]-image_embeddings = []--for img_path in image_paths:- image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)- with torch.no_grad():- embedding = model.encode_image(image)- embedding /= embedding.norm(dim=-1, keepdim=True)- image_embeddings.append(embedding)--image_embeddings = torch.cat(image_embeddings)--# Search with text query-query = "a sunset over the ocean"-text_input = clip.tokenize([query]).to(device)-with torch.no_grad():- text_embedding = model.encode_text(text_input)- text_embedding /= text_embedding.norm(dim=-1, keepdim=True)--# Find most similar images-similarities = (text_embedding @ image_embeddings.T).squeeze(0)-top_k = similarities.topk(3)--for idx, score in zip(top_k.indices, top_k.values):- print(f"{image_paths[idx]}: {score:.3f}")-```--## Content moderation--```python-# Define categories-categories = [- "safe for work",- "not safe for work",- "violent content",- "graphic content"-]--text = clip.tokenize(categories).to(device)--# Check image-with torch.no_grad():- logits_per_image, _ = model(image, text)- probs = logits_per_image.softmax(dim=-1)--# Get classification-max_idx = probs.argmax().item()-max_prob = probs[0, max_idx].item()--print(f"Category: {categories[max_idx]} ({max_prob:.2%})")-```--## Batch processing--```python-# Process multiple images-images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]-images = torch.stack(images).to(device)--with torch.no_grad():- image_features = model.encode_image(images)- image_features /= image_features.norm(dim=-1, keepdim=True)--# Batch text-texts = ["a dog", "a cat", "a bird"]-text_tokens = clip.tokenize(texts).to(device)--with torch.no_grad():- text_features = model.encode_text(text_tokens)- text_features /= text_features.norm(dim=-1, keepdim=True)--# Similarity matrix (10 images × 3 texts)-similarities = image_features @ text_features.T-print(similarities.shape) # (10, 3)-```--## Integration with vector databases--```python-# Store CLIP embeddings in Chroma/FAISS-import chromadb--client = chromadb.Client()-collection = client.create_collection("image_embeddings")--# Add image embeddings-for img_path, embedding in zip(image_paths, image_embeddings):- collection.add(- embeddings=[embedding.cpu().numpy().tolist()],- metadatas=[{"path": img_path}],- ids=[img_path]- )--# Query with text-query = "a sunset"-text_embedding = model.encode_text(clip.tokenize([query]))-results = collection.query(- query_embeddings=[text_embedding.cpu().numpy().tolist()],- n_results=5-)-```--## Best practices--1. **Use ViT-B/32 for most cases** - Good balance-2. **Normalize embeddings** - Required for cosine similarity-3. **Batch processing** - More efficient-4. **Cache embeddings** - Expensive to recompute-5. **Use descriptive labels** - Better zero-shot performance-6. **GPU recommended** - 10-50× faster-7. **Preprocess images** - Use provided preprocess function--## Performance--| Operation | CPU | GPU (V100) |-|-----------|-----|------------|-| Image encoding | ~200ms | ~20ms |-| Text encoding | ~50ms | ~5ms |-| Similarity compute | <1ms | <1ms |--## Limitations--1. **Not for fine-grained tasks** - Best for broad categories-2. **Requires descriptive text** - Vague labels perform poorly-3. **Biased on web data** - May have dataset biases-4. **No bounding boxes** - Whole image only-5. **Limited spatial understanding** - Position/counting weak--## Resources--- **GitHub**: https://github.com/openai/CLIP ⭐ 25,300+-- **Paper**: https://arxiv.org/abs/2103.00020-- **Colab**: https://colab.research.google.com/github/openai/clip/-- **License**: MIT--diff --git a/optional-skills/mlops/clip/references/applications.md b/optional-skills/mlops/clip/references/applications.mddeleted file mode 100644index 38e9a0563f1..00000000000--- a/optional-skills/mlops/clip/references/applications.md+++ /dev/null@@ -1,207 +0,0 @@-# CLIP Applications Guide--Practical applications and use cases for CLIP.--## Zero-shot image classification--```python-import torch-import clip-from PIL import Image--model, preprocess = clip.load("ViT-B/32")--# Define categories-categories = [- "a photo of a dog",- "a photo of a cat",- "a photo of a bird",- "a photo of a car",- "a photo of a person"-]--# Prepare image-image = preprocess(Image.open("photo.jpg")).unsqueeze(0)-text = clip.tokenize(categories)--# Classify-with torch.no_grad():- image_features = model.encode_image(image)- text_features = model.encode_text(text)-- logits_per_image, _ = model(image, text)- probs = logits_per_image.softmax(dim=-1).cpu().numpy()--# Print results-for category, prob in zip(categories, probs[0]):- print(f"{category}: {prob:.2%}")-```--## Semantic image search--```python-# Index images-image_database = []-image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]--for img_path in image_paths:- image = preprocess(Image.open(img_path)).unsqueeze(0)- with torch.no_grad():- features = model.encode_image(image)- features /= features.norm(dim=-1, keepdim=True)- image_database.append((img_path, features))--# Search with text-query = "a sunset over mountains"-text_input = clip.tokenize([query])--with torch.no_grad():- text_features = model.encode_text(text_input)- text_features /= text_features.norm(dim=-1, keepdim=True)--# Find matches-similarities = []-for img_path, img_features in image_database:- similarity = (text_features @ img_features.T).item()- similarities.append((img_path, similarity))--# Sort by similarity-similarities.sort(key=lambda x: x[1], reverse=True)-for img_path, score in similarities[:3]:- print(f"{img_path}: {score:.3f}")-```--## Content moderation--```python-# Define safety categories-categories = [- "safe for work content",- "not safe for work content",- "violent or graphic content",- "hate speech or offensive content",- "spam or misleading content"-]--text = clip.tokenize(categories)--# Check image-with torch.no_grad():- logits, _ = model(image, text)- probs = logits.softmax(dim=-1)--# Get classification-max_idx = probs.argmax().item()-confidence = probs[0, max_idx].item()--if confidence > 0.7:- print(f"Classified as: {categories[max_idx]} ({confidence:.2%})")-else:- print(f"Uncertain classification (confidence: {confidence:.2%})")-```--## Image-to-text retrieval--```python-# Text database-captions = [- "A beautiful sunset over the ocean",- "A cute dog playing in the park",- "A modern city skyline at night",- "A delicious pizza with toppings"-]--# Encode captions-caption_features = []-for caption in captions:- text = clip.tokenize([caption])- with torch.no_grad():- features = model.encode_text(text)- features /= features.norm(dim=-1, keepdim=True)- caption_features.append(features)--caption_features = torch.cat(caption_features)--# Find matching captions for image-with torch.no_grad():- image_features = model.encode_image(image)- image_features /= image_features.norm(dim=-1, keepdim=True)--similarities = (image_features @ caption_features.T).squeeze(0)-top_k = similarities.topk(3)--for idx, score in zip(top_k.indices, top_k.values):- print(f"{captions[idx]}: {score:.3f}")-```--## Visual question answering--```python-# Create yes/no questions-image = preprocess(Image.open("photo.jpg")).unsqueeze(0)--questions = [- "a photo showing people",- "a photo showing animals",- "a photo taken indoors",- "a photo taken outdoors",- "a photo taken during daytime",- "a photo taken at night"-]--text = clip.tokenize(questions)--with torch.no_grad():- logits, _ = model(image, text)- probs = logits.softmax(dim=-1)--# Answer questions-for question, prob in zip(questions, probs[0]):- answer = "Yes" if prob > 0.5 else "No"- print(f"{question}: {answer} ({prob:.2%})")-```--## Image deduplication--```python-# Detect duplicate/similar images-def compute_similarity(img1_path, img2_path):- img1 = preprocess(Image.open(img1_path)).unsqueeze(0)- img2 = preprocess(Image.open(img2_path)).unsqueeze(0)-- with torch.no_grad():- feat1 = model.encode_image(img1)- feat2 = model.encode_image(img2)-- feat1 /= feat1.norm(dim=-1, keepdim=True)- feat2 /= feat2.norm(dim=-1, keepdim=True)-- similarity = (feat1 @ feat2.T).item()-- return similarity--# Check for duplicates-threshold = 0.95-image_pairs = [("img1.jpg", "img2.jpg"), ("img1.jpg", "img3.jpg")]--for img1, img2 in image_pairs:- sim = compute_similarity(img1, img2)- if sim > threshold:- print(f"{img1} and {img2} are duplicates (similarity: {sim:.3f})")-```--## Best practices--1. **Use descriptive labels** - "a photo of X" works better than just "X"-2. **Normalize embeddings** - Always normalize for cosine similarity-3. **Batch processing** - Process multiple images/texts together-4. **Cache embeddings** - Expensive to recompute-5. **Set appropriate thresholds** - Test on validation data-6. **Use GPU** - 10-50× faster than CPU-7. **Consider model size** - ViT-B/32 good default, ViT-L/14 for best quality--## Resources--- **Paper**: https://arxiv.org/abs/2103.00020-- **GitHub**: https://github.com/openai/CLIP-- **Colab**: https://colab.research.google.com/github/openai/clip/diff --git a/optional-skills/mlops/evaluation/weights-and-biases/SKILL.md b/optional-skills/mlops/evaluation/weights-and-biases/SKILL.mddeleted file mode 100644index 656cb5de680..00000000000--- a/optional-skills/mlops/evaluation/weights-and-biases/SKILL.md+++ /dev/null@@ -1,598 +0,0 @@-----name: weights-and-biases-description: "W&B: log ML experiments, sweeps, model registry, dashboards."-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [wandb]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [MLOps, Weights And Biases, WandB, Experiment Tracking, Hyperparameter Tuning, Model Registry, Collaboration, Real-Time Visualization, PyTorch, TensorFlow, HuggingFace]-------# Weights & Biases: ML Experiment Tracking & MLOps--## When to Use This Skill--Use Weights & Biases (W&B) when you need to:-- **Track ML experiments** with automatic metric logging-- **Visualize training** in real-time dashboards-- **Compare runs** across hyperparameters and configurations-- **Optimize hyperparameters** with automated sweeps-- **Manage model registry** with versioning and lineage-- **Collaborate on ML projects** with team workspaces-- **Track artifacts** (datasets, models, code) with lineage--**Users**: 200,000+ ML practitioners | **GitHub Stars**: 10.5k+ | **Integrations**: 100+--## Installation--```bash-# Install W&B-pip install wandb--# Login (creates API key)-wandb login--# Or set API key programmatically-export WANDB_API_KEY=your_api_key_here-```--## Quick Start--### Basic Experiment Tracking--```python-import wandb--# Initialize a run-run = wandb.init(- project="my-project",- config={- "learning_rate": 0.001,- "epochs": 10,- "batch_size": 32,- "architecture": "ResNet50"- }-)--# Training loop-for epoch in range(run.config.epochs):- # Your training code- train_loss = train_epoch()- val_loss = validate()-- # Log metrics- wandb.log({- "epoch": epoch,- "train/loss": train_loss,- "val/loss": val_loss,- "train/accuracy": train_acc,- "val/accuracy": val_acc- })--# Finish the run-wandb.finish()-```--### With PyTorch--```python-import torch-import wandb--# Initialize-wandb.init(project="pytorch-demo", config={- "lr": 0.001,- "epochs": 10-})--# Access config-config = wandb.config--# Training loop-for epoch in range(config.epochs):- for batch_idx, (data, target) in enumerate(train_loader):- # Forward pass- output = model(data)- loss = criterion(output, target)-- # Backward pass- optimizer.zero_grad()- loss.backward()- optimizer.step()-- # Log every 100 batches- if batch_idx % 100 == 0:- wandb.log({- "loss": loss.item(),- "epoch": epoch,- "batch": batch_idx- })--# Save model-torch.save(model.state_dict(), "model.pth")-wandb.save("model.pth") # Upload to W&B--wandb.finish()-```--## Core Concepts--### 1. Projects and Runs--**Project**: Collection of related experiments-**Run**: Single execution of your training script--```python-# Create/use project-run = wandb.init(- project="image-classification",- name="resnet50-experiment-1", # Optional run name- tags=["baseline", "resnet"], # Organize with tags- notes="First baseline run" # Add notes-)--# Each run has unique ID-print(f"Run ID: {run.id}")-print(f"Run URL: {run.url}")-```--### 2. Configuration Tracking--Track hyperparameters automatically:--```python-config = {- # Model architecture- "model": "ResNet50",- "pretrained": True,-- # Training params- "learning_rate": 0.001,- "batch_size": 32,- "epochs": 50,- "optimizer": "Adam",-- # Data params- "dataset": "ImageNet",- "augmentation": "standard"-}--wandb.init(project="my-project", config=config)--# Access config during training-lr = wandb.config.learning_rate-batch_size = wandb.config.batch_size-```--### 3. Metric Logging--```python-# Log scalars-wandb.log({"loss": 0.5, "accuracy": 0.92})--# Log multiple metrics-wandb.log({- "train/loss": train_loss,- "train/accuracy": train_acc,- "val/loss": val_loss,- "val/accuracy": val_acc,- "learning_rate": current_lr,- "epoch": epoch-})--# Log with custom x-axis-wandb.log({"loss": loss}, step=global_step)--# Log media (images, audio, video)-wandb.log({"examples": [wandb.Image(img) for img in images]})--# Log histograms-wandb.log({"gradients": wandb.Histogram(gradients)})--# Log tables-table = wandb.Table(columns=["id", "prediction", "ground_truth"])-wandb.log({"predictions": table})-```--### 4. Model Checkpointing--```python-import torch-import wandb--# Save model checkpoint-checkpoint = {- 'epoch': epoch,- 'model_state_dict': model.state_dict(),- 'optimizer_state_dict': optimizer.state_dict(),- 'loss': loss,-}--torch.save(checkpoint, 'checkpoint.pth')--# Upload to W&B-wandb.save('checkpoint.pth')--# Or use Artifacts (recommended)-artifact = wandb.Artifact('model', type='model')-artifact.add_file('checkpoint.pth')-wandb.log_artifact(artifact)-```--## Hyperparameter Sweeps--Automatically search for optimal hyperparameters.--### Define Sweep Configuration--```python-sweep_config = {- 'method': 'bayes', # or 'grid', 'random'- 'metric': {- 'name': 'val/accuracy',- 'goal': 'maximize'- },- 'parameters': {- 'learning_rate': {- 'distribution': 'log_uniform_values',- 'min': 1e-5,- 'max': 1e-1- },- 'batch_size': {- 'values': [16, 32, 64, 128]- },- 'optimizer': {- 'values': ['adam', 'sgd', 'rmsprop']- },- 'dropout': {- 'distribution': 'uniform',- 'min': 0.1,- 'max': 0.5- }- }-}--# Initialize sweep-sweep_id = wandb.sweep(sweep_config, project="my-project")-```--### Define Training Function--```python-def train():- # Initialize run- run = wandb.init()-- # Access sweep parameters- lr = wandb.config.learning_rate- batch_size = wandb.config.batch_size- optimizer_name = wandb.config.optimizer-- # Build model with sweep config- model = build_model(wandb.config)- optimizer = get_optimizer(optimizer_name, lr)-- # Training loop- for epoch in range(NUM_EPOCHS):- train_loss = train_epoch(model, optimizer, batch_size)- val_acc = validate(model)-- # Log metrics- wandb.log({- "train/loss": train_loss,- "val/accuracy": val_acc- })--# Run sweep-wandb.agent(sweep_id, function=train, count=50) # Run 50 trials-```--### Sweep Strategies--```python-# Grid search - exhaustive-sweep_config = {- 'method': 'grid',- 'parameters': {- 'lr': {'values': [0.001, 0.01, 0.1]},- 'batch_size': {'values': [16, 32, 64]}- }-}--# Random search-sweep_config = {- 'method': 'random',- 'parameters': {- 'lr': {'distribution': 'uniform', 'min': 0.0001, 'max': 0.1},- 'dropout': {'distribution': 'uniform', 'min': 0.1, 'max': 0.5}- }-}--# Bayesian optimization (recommended)-sweep_config = {- 'method': 'bayes',- 'metric': {'name': 'val/loss', 'goal': 'minimize'},- 'parameters': {- 'lr': {'distribution': 'log_uniform_values', 'min': 1e-5, 'max': 1e-1}- }-}-```--## Artifacts--Track datasets, models, and other files with lineage.--### Log Artifacts--```python-# Create artifact-artifact = wandb.Artifact(- name='training-dataset',- type='dataset',- description='ImageNet training split',- metadata={'size': '1.2M images', 'split': 'train'}-)--# Add files-artifact.add_file('data/train.csv')-artifact.add_dir('data/images/')--# Log artifact-wandb.log_artifact(artifact)-```--### Use Artifacts--```python-# Download and use artifact-run = wandb.init(project="my-project")--# Download artifact-artifact = run.use_artifact('training-dataset:latest')-artifact_dir = artifact.download()--# Use the data-data = load_data(f"{artifact_dir}/train.csv")-```--### Model Registry--```python-# Log model as artifact-model_artifact = wandb.Artifact(- name='resnet50-model',- type='model',- metadata={'architecture': 'ResNet50', 'accuracy': 0.95}-)--model_artifact.add_file('model.pth')-wandb.log_artifact(model_artifact, aliases=['best', 'production'])--# Link to model registry-run.link_artifact(model_artifact, 'model-registry/production-models')-```--## Integration Examples--### HuggingFace Transformers--```python-from transformers import Trainer, TrainingArguments-import wandb--# Initialize W&B-wandb.init(project="hf-transformers")--# Training arguments with W&B-training_args = TrainingArguments(- output_dir="./results",- report_to="wandb", # Enable W&B logging- run_name="bert-finetuning",- logging_steps=100,- save_steps=500-)--# Trainer automatically logs to W&B-trainer = Trainer(- model=model,- args=training_args,- train_dataset=train_dataset,- eval_dataset=eval_dataset-)--trainer.train()-```--### PyTorch Lightning--```python-from pytorch_lightning import Trainer-from pytorch_lightning.loggers import WandbLogger-import wandb--# Create W&B logger-wandb_logger = WandbLogger(- project="lightning-demo",- log_model=True # Log model checkpoints-)--# Use with Trainer-trainer = Trainer(- logger=wandb_logger,- max_epochs=10-)--trainer.fit(model, datamodule=dm)-```--### Keras/TensorFlow--```python-import wandb-from wandb.integration.keras import WandbMetricsLogger, WandbModelCheckpoint--# Initialize-wandb.init(project="keras-demo")--# Add callbacks (the monolithic WandbCallback was removed;-# use the dedicated callbacks from wandb.integration.keras instead)-model.fit(- x_train, y_train,- validation_data=(x_val, y_val),- epochs=10,- callbacks=[- WandbMetricsLogger(), # Auto-logs metrics- WandbModelCheckpoint("models/model-{epoch}") # Saves checkpoints- ]-)-```--## Visualization & Analysis--### Custom Charts--```python-# Log custom visualizations-import matplotlib.pyplot as plt--fig, ax = plt.subplots()-ax.plot(x, y)-wandb.log({"custom_plot": wandb.Image(fig)})--# Log confusion matrix-wandb.log({"conf_mat": wandb.plot.confusion_matrix(- probs=None,- y_true=ground_truth,- preds=predictions,- class_names=class_names-)})-```--### Reports--Create shareable reports in W&B UI:-- Combine runs, charts, and text-- Markdown support-- Embeddable visualizations-- Team collaboration--## Best Practices--### 1. Organize with Tags and Groups--```python-wandb.init(- project="my-project",- tags=["baseline", "resnet50", "imagenet"],- group="resnet-experiments", # Group related runs- job_type="train" # Type of job-)-```--### 2. Log Everything Relevant--```python-# Log system metrics-wandb.log({- "gpu/util": gpu_utilization,- "gpu/memory": gpu_memory_used,- "cpu/util": cpu_utilization-})--# Log code version-wandb.log({"git_commit": git_commit_hash})--# Log data splits-wandb.log({- "data/train_size": len(train_dataset),- "data/val_size": len(val_dataset)-})-```--### 3. Use Descriptive Names--```python-# ✅ Good: Descriptive run names-wandb.init(- project="nlp-classification",- name="bert-base-lr0.001-bs32-epoch10"-)--# ❌ Bad: Generic names-wandb.init(project="nlp", name="run1")-```--### 4. Save Important Artifacts--```python-# Save final model-artifact = wandb.Artifact('final-model', type='model')-artifact.add_file('model.pth')-wandb.log_artifact(artifact)--# Save predictions for analysis-predictions_table = wandb.Table(- columns=["id", "input", "prediction", "ground_truth"],- data=predictions_data-)-wandb.log({"predictions": predictions_table})-```--### 5. Use Offline Mode for Unstable Connections--```python-import os--# Enable offline mode-os.environ["WANDB_MODE"] = "offline"--wandb.init(project="my-project")-# ... your code ...--# Sync later-# wandb sync <run_directory>-```--## Team Collaboration--### Share Runs--```python-# Runs are automatically shareable via URL-run = wandb.init(project="team-project")-print(f"Share this URL: {run.url}")-```--### Team Projects--- Create team account at wandb.ai-- Add team members-- Set project visibility (private/public)-- Use team-level artifacts and model registry--## Pricing--- **Free**: Unlimited public projects, 100GB storage-- **Academic**: Free for students/researchers-- **Teams**: $50/seat/month, private projects, unlimited storage-- **Enterprise**: Custom pricing, on-prem options--## Resources--- **Documentation**: https://docs.wandb.ai-- **GitHub**: https://github.com/wandb/wandb (10.5k+ stars)-- **Examples**: https://github.com/wandb/examples-- **Community**: https://wandb.ai/community-- **Discord**: https://wandb.me/discord--## See Also--- `references/sweeps.md` - Comprehensive hyperparameter optimization guide-- `references/artifacts.md` - Data and model versioning patterns-- `references/integrations.md` - Framework-specific examples--diff --git a/optional-skills/mlops/evaluation/weights-and-biases/references/artifacts.md b/optional-skills/mlops/evaluation/weights-and-biases/references/artifacts.mddeleted file mode 100644index 2b0f793315e..00000000000--- a/optional-skills/mlops/evaluation/weights-and-biases/references/artifacts.md+++ /dev/null@@ -1,584 +0,0 @@-# Artifacts & Model Registry Guide--Complete guide to data versioning and model management with W&B Artifacts.--## Table of Contents-- What are Artifacts-- Creating Artifacts-- Using Artifacts-- Model Registry-- Versioning & Lineage-- Best Practices--## What are Artifacts--Artifacts are versioned datasets, models, or files tracked with lineage.--**Key Features:**-- Automatic versioning (v0, v1, v2...)-- Lineage tracking (which runs produced/used artifacts)-- Efficient storage (deduplication)-- Collaboration (team-wide access)-- Aliases (latest, best, production)--**Common Use Cases:**-- Dataset versioning-- Model checkpoints-- Preprocessed data-- Evaluation results-- Configuration files--## Creating Artifacts--### Basic Dataset Artifact--```python-import wandb--run = wandb.init(project="my-project")--# Create artifact-dataset = wandb.Artifact(- name='training-data',- type='dataset',- description='ImageNet training split with augmentations',- metadata={- 'size': '1.2M images',- 'format': 'JPEG',- 'resolution': '224x224'- }-)--# Add files-dataset.add_file('data/train.csv') # Single file-dataset.add_dir('data/images') # Entire directory-dataset.add_reference('s3://bucket/data') # Cloud reference--# Log artifact-run.log_artifact(dataset)-wandb.finish()-```--### Model Artifact--```python-import torch-import wandb--run = wandb.init(project="my-project")--# Train model-model = train_model()--# Save model-torch.save(model.state_dict(), 'model.pth')--# Create model artifact-model_artifact = wandb.Artifact(- name='resnet50-classifier',- type='model',- description='ResNet50 trained on ImageNet',- metadata={- 'architecture': 'ResNet50',- 'accuracy': 0.95,- 'loss': 0.15,- 'epochs': 50,- 'framework': 'PyTorch'- }-)--# Add model file-model_artifact.add_file('model.pth')--# Add config-model_artifact.add_file('config.yaml')--# Log with aliases-run.log_artifact(model_artifact, aliases=['latest', 'best'])--wandb.finish()-```--### Preprocessed Data Artifact--```python-import pandas as pd-import wandb--run = wandb.init(project="nlp-project")--# Preprocess data-df = pd.read_csv('raw_data.csv')-df_processed = preprocess(df)-df_processed.to_csv('processed_data.csv', index=False)--# Create artifact-processed_data = wandb.Artifact(- name='processed-text-data',- type='dataset',- metadata={- 'rows': len(df_processed),- 'columns': list(df_processed.columns),- 'preprocessing_steps': ['lowercase', 'remove_stopwords', 'tokenize']- }-)--processed_data.add_file('processed_data.csv')--# Log artifact-run.log_artifact(processed_data)-```--## Using Artifacts--### Download and Use--```python-import wandb--run = wandb.init(project="my-project")--# Download artifact-artifact = run.use_artifact('training-data:latest')-artifact_dir = artifact.download()--# Use files-import pandas as pd-df = pd.read_csv(f'{artifact_dir}/train.csv')--# Train with artifact data-model = train_model(df)-```--### Use Specific Version--```python-# Use specific version-artifact_v2 = run.use_artifact('training-data:v2')--# Use alias-artifact_best = run.use_artifact('model:best')-artifact_prod = run.use_artifact('model:production')--# Use from another project-artifact = run.use_artifact('team/other-project/model:latest')-```--### Check Artifact Metadata--```python-artifact = run.use_artifact('training-data:latest')--# Access metadata-print(artifact.metadata)-print(f"Size: {artifact.metadata['size']}")--# Access version info-print(f"Version: {artifact.version}")-print(f"Created at: {artifact.created_at}")-print(f"Digest: {artifact.digest}")-```--## Model Registry--Link models to a central registry for governance and deployment.--### Create Model Registry--```python-# In W&B UI:-# 1. Go to "Registry" tab-# 2. Create new registry: "production-models"-# 3. Define stages: development, staging, production-```--### Link Model to Registry--```python-import wandb--run = wandb.init(project="training")--# Create model artifact-model_artifact = wandb.Artifact(- name='sentiment-classifier',- type='model',- metadata={'accuracy': 0.94, 'f1': 0.92}-)--model_artifact.add_file('model.pth')--# Log artifact-run.log_artifact(model_artifact)--# Link to registry-run.link_artifact(- model_artifact,- 'model-registry/production-models',- aliases=['staging'] # Deploy to staging-)--wandb.finish()-```--### Promote Model in Registry--```python-# Retrieve model from registry-api = wandb.Api()-artifact = api.artifact('model-registry/production-models/sentiment-classifier:staging')--# Promote to production-artifact.link('model-registry/production-models', aliases=['production'])--# Demote from production-artifact.aliases = ['archived']-artifact.save()-```--### Use Model from Registry--```python-import wandb--run = wandb.init()--# Download production model-model_artifact = run.use_artifact(- 'model-registry/production-models/sentiment-classifier:production'-)--model_dir = model_artifact.download()--# Load and use-import torch-model = torch.load(f'{model_dir}/model.pth')-model.eval()-```--## Versioning & Lineage--### Automatic Versioning--```python-# First log: creates v0-run1 = wandb.init(project="my-project")-dataset_v0 = wandb.Artifact('my-dataset', type='dataset')-dataset_v0.add_file('data_v1.csv')-run1.log_artifact(dataset_v0)--# Second log with same name: creates v1-run2 = wandb.init(project="my-project")-dataset_v1 = wandb.Artifact('my-dataset', type='dataset')-dataset_v1.add_file('data_v2.csv') # Different content-run2.log_artifact(dataset_v1)--# Third log with SAME content as v1: references v1 (no new version)-run3 = wandb.init(project="my-project")-dataset_v1_again = wandb.Artifact('my-dataset', type='dataset')-dataset_v1_again.add_file('data_v2.csv') # Same content as v1-run3.log_artifact(dataset_v1_again) # Still v1, no v2 created-```--### Track Lineage--```python-# Training run-run = wandb.init(project="my-project")--# Use dataset (input)-dataset = run.use_artifact('training-data:v3')-data = load_data(dataset.download())--# Train model-model = train(data)--# Save model (output)-model_artifact = wandb.Artifact('trained-model', type='model')-torch.save(model.state_dict(), 'model.pth')-model_artifact.add_file('model.pth')-run.log_artifact(model_artifact)--# Lineage automatically tracked:-# training-data:v3 --> [run] --> trained-model:v0-```--### View Lineage Graph--```python-# In W&B UI:-# Artifacts → Select artifact → Lineage tab-# Shows:-# - Which runs produced this artifact-# - Which runs used this artifact-# - Parent/child artifacts-```--## Artifact Types--### Dataset Artifacts--```python-# Raw data-raw_data = wandb.Artifact('raw-data', type='dataset')-raw_data.add_dir('raw/')--# Processed data-processed_data = wandb.Artifact('processed-data', type='dataset')-processed_data.add_dir('processed/')--# Train/val/test splits-train_split = wandb.Artifact('train-split', type='dataset')-train_split.add_file('train.csv')--val_split = wandb.Artifact('val-split', type='dataset')-val_split.add_file('val.csv')-```--### Model Artifacts--```python-# Checkpoint during training-checkpoint = wandb.Artifact('checkpoint-epoch-10', type='model')-checkpoint.add_file('checkpoint_epoch_10.pth')--# Final model-final_model = wandb.Artifact('final-model', type='model')-final_model.add_file('model.pth')-final_model.add_file('tokenizer.json')--# Quantized model-quantized = wandb.Artifact('quantized-model', type='model')-quantized.add_file('model_int8.onnx')-```--### Result Artifacts--```python-# Predictions-predictions = wandb.Artifact('test-predictions', type='predictions')-predictions.add_file('predictions.csv')--# Evaluation metrics-eval_results = wandb.Artifact('evaluation', type='evaluation')-eval_results.add_file('metrics.json')-eval_results.add_file('confusion_matrix.png')-```--## Advanced Patterns--### Incremental Artifacts--Add files incrementally without re-uploading.--```python-run = wandb.init(project="my-project")--# Create artifact-dataset = wandb.Artifact('incremental-dataset', type='dataset')--# Add files incrementally-for i in range(100):- filename = f'batch_{i}.csv'- process_batch(i, filename)- dataset.add_file(filename)-- # Log progress- if (i + 1) % 10 == 0:- print(f"Added {i + 1}/100 batches")--# Log complete artifact-run.log_artifact(dataset)-```--### Artifact Tables--Track structured data with W&B Tables.--```python-import wandb--run = wandb.init(project="my-project")--# Create table-table = wandb.Table(columns=["id", "image", "label", "prediction"])--for idx, (img, label, pred) in enumerate(zip(images, labels, predictions)):- table.add_data(- idx,- wandb.Image(img),- label,- pred- )--# Log as artifact-artifact = wandb.Artifact('predictions-table', type='predictions')-artifact.add(table, "predictions")-run.log_artifact(artifact)-```--### Artifact References--Reference external data without copying.--```python-# S3 reference-dataset = wandb.Artifact('s3-dataset', type='dataset')-dataset.add_reference('s3://my-bucket/data/', name='train')-dataset.add_reference('s3://my-bucket/labels/', name='labels')--# GCS reference-dataset.add_reference('gs://my-bucket/data/')--# HTTP reference-dataset.add_reference('https://example.com/data.zip')--# Local filesystem reference (for shared storage)-dataset.add_reference('file:///mnt/shared/data')-```--## Collaboration Patterns--### Team Dataset Sharing--```python-# Data engineer creates dataset-run = wandb.init(project="data-eng", entity="my-team")-dataset = wandb.Artifact('shared-dataset', type='dataset')-dataset.add_dir('data/')-run.log_artifact(dataset, aliases=['latest', 'production'])--# ML engineer uses dataset-run = wandb.init(project="ml-training", entity="my-team")-dataset = run.use_artifact('my-team/data-eng/shared-dataset:production')-data = load_data(dataset.download())-```--### Model Handoff--```python-# Training team-train_run = wandb.init(project="model-training", entity="ml-team")-model = train_model()-model_artifact = wandb.Artifact('nlp-model', type='model')-model_artifact.add_file('model.pth')-train_run.log_artifact(model_artifact)-train_run.link_artifact(model_artifact, 'model-registry/nlp-models', aliases=['candidate'])--# Evaluation team-eval_run = wandb.init(project="model-eval", entity="ml-team")-model_artifact = eval_run.use_artifact('model-registry/nlp-models/nlp-model:candidate')-metrics = evaluate_model(model_artifact)--if metrics['f1'] > 0.9:- # Promote to production- model_artifact.link('model-registry/nlp-models', aliases=['production'])-```--## Best Practices--### 1. Use Descriptive Names--```python-# ✅ Good: Descriptive names-wandb.Artifact('imagenet-train-augmented-v2', type='dataset')-wandb.Artifact('bert-base-sentiment-finetuned', type='model')--# ❌ Bad: Generic names-wandb.Artifact('dataset1', type='dataset')-wandb.Artifact('model', type='model')-```--### 2. Add Comprehensive Metadata--```python-model_artifact = wandb.Artifact(- 'production-model',- type='model',- description='ResNet50 classifier for product categorization',- metadata={- # Model info- 'architecture': 'ResNet50',- 'framework': 'PyTorch 2.0',- 'pretrained': True,-- # Performance- 'accuracy': 0.95,- 'f1_score': 0.93,- 'inference_time_ms': 15,-- # Training- 'epochs': 50,- 'dataset': 'imagenet',- 'num_samples': 1200000,-- # Business context- 'use_case': 'e-commerce product classification',- 'owner': '[email protected]',- 'approved_by': 'data-science-lead'- }-)-```--### 3. Use Aliases for Deployment Stages--```python-# Development-run.log_artifact(model, aliases=['dev', 'latest'])--# Staging-run.log_artifact(model, aliases=['staging'])--# Production-run.log_artifact(model, aliases=['production', 'v1.2.0'])--# Archive old versions-old_artifact = api.artifact('model:production')-old_artifact.aliases = ['archived-v1.1.0']-old_artifact.save()-```--### 4. Track Data Lineage--```python-def create_training_pipeline():- run = wandb.init(project="pipeline")-- # 1. Load raw data- raw_data = run.use_artifact('raw-data:latest')-- # 2. Preprocess- processed = preprocess(raw_data)- processed_artifact = wandb.Artifact('processed-data', type='dataset')- processed_artifact.add_file('processed.csv')- run.log_artifact(processed_artifact)-- # 3. Train model- model = train(processed)- model_artifact = wandb.Artifact('trained-model', type='model')- model_artifact.add_file('model.pth')- run.log_artifact(model_artifact)-- # Lineage: raw-data → processed-data → trained-model-```--### 5. Efficient Storage--```python-# ✅ Good: Reference large files-large_dataset = wandb.Artifact('large-dataset', type='dataset')-large_dataset.add_reference('s3://bucket/huge-file.tar.gz')--# ❌ Bad: Upload giant files-# large_dataset.add_file('huge-file.tar.gz') # Don't do this--# ✅ Good: Upload only metadata-metadata_artifact = wandb.Artifact('dataset-metadata', type='dataset')-metadata_artifact.add_file('metadata.json') # Small file-```--## Resources--- **Artifacts Documentation**: https://docs.wandb.ai/guides/artifacts-- **Model Registry**: https://docs.wandb.ai/guides/model-registry-- **Best Practices**: https://wandb.ai/site/articles/versioning-data-and-models-in-mldiff --git a/optional-skills/mlops/evaluation/weights-and-biases/references/integrations.md b/optional-skills/mlops/evaluation/weights-and-biases/references/integrations.mddeleted file mode 100644index 2a93865b781..00000000000--- a/optional-skills/mlops/evaluation/weights-and-biases/references/integrations.md+++ /dev/null@@ -1,700 +0,0 @@-# Framework Integrations Guide--Complete guide to integrating W&B with popular ML frameworks.--## Table of Contents-- HuggingFace Transformers-- PyTorch Lightning-- Keras/TensorFlow-- Fast.ai-- XGBoost/LightGBM-- PyTorch Native-- Custom Integrations--## HuggingFace Transformers--### Automatic Integration--```python-from transformers import Trainer, TrainingArguments-import wandb--# Initialize W&B-wandb.init(project="hf-transformers", name="bert-finetuning")--# Training arguments with W&B-training_args = TrainingArguments(- output_dir="./results",- report_to="wandb", # Enable W&B logging- run_name="bert-base-finetuning",-- # Training params- num_train_epochs=3,- per_device_train_batch_size=16,- per_device_eval_batch_size=64,- learning_rate=2e-5,-- # Logging- logging_dir="./logs",- logging_steps=100,- logging_first_step=True,-- # Evaluation- evaluation_strategy="steps",- eval_steps=500,- save_steps=500,-- # Other- load_best_model_at_end=True,- metric_for_best_model="eval_accuracy"-)--# Trainer automatically logs to W&B-trainer = Trainer(- model=model,- args=training_args,- train_dataset=train_dataset,- eval_dataset=eval_dataset,- compute_metrics=compute_metrics-)--# Train (metrics logged automatically)-trainer.train()--# Finish W&B run-wandb.finish()-```--### Custom Logging--```python-from transformers import Trainer, TrainingArguments-from transformers.integrations import WandbCallback-import wandb--class CustomWandbCallback(WandbCallback):- def on_evaluate(self, args, state, control, metrics=None, **kwargs):- super().on_evaluate(args, state, control, metrics, **kwargs)-- # Log custom metrics- wandb.log({- "custom/eval_score": metrics["eval_accuracy"] * 100,- "custom/epoch": state.epoch- })--# Use custom callback-trainer = Trainer(- model=model,- args=training_args,- train_dataset=train_dataset,- eval_dataset=eval_dataset,- callbacks=[CustomWandbCallback()]-)-```--### Log Model to Registry--```python-from transformers import Trainer, TrainingArguments--training_args = TrainingArguments(- output_dir="./results",- report_to="wandb",- load_best_model_at_end=True-)--trainer = Trainer(- model=model,- args=training_args,- train_dataset=train_dataset,- eval_dataset=eval_dataset-)--trainer.train()--# Save final model as artifact-model_artifact = wandb.Artifact(- 'hf-bert-model',- type='model',- description='BERT finetuned on sentiment analysis'-)--# Save model files-trainer.save_model("./final_model")-model_artifact.add_dir("./final_model")--# Log artifact-wandb.log_artifact(model_artifact, aliases=['best', 'production'])-wandb.finish()-```--## PyTorch Lightning--### Basic Integration--```python-import pytorch_lightning as pl-from pytorch_lightning.loggers import WandbLogger-import wandb--# Create W&B logger-wandb_logger = WandbLogger(- project="lightning-demo",- name="resnet50-training",- log_model=True, # Log model checkpoints as artifacts- save_code=True # Save code as artifact-)--# Lightning module-class LitModel(pl.LightningModule):- def __init__(self, learning_rate=0.001):- super().__init__()- self.save_hyperparameters()- self.model = create_model()-- def training_step(self, batch, batch_idx):- x, y = batch- y_hat = self.model(x)- loss = F.cross_entropy(y_hat, y)-- # Log metrics (automatically sent to W&B)- self.log('train/loss', loss, on_step=True, on_epoch=True)- self.log('train/accuracy', accuracy(y_hat, y), on_epoch=True)-- return loss-- def validation_step(self, batch, batch_idx):- x, y = batch- y_hat = self.model(x)- loss = F.cross_entropy(y_hat, y)-- self.log('val/loss', loss, on_step=False, on_epoch=True)- self.log('val/accuracy', accuracy(y_hat, y), on_epoch=True)-- return loss-- def configure_optimizers(self):- return torch.optim.Adam(self.parameters(), lr=self.hparams.learning_rate)--# Trainer with W&B logger-trainer = pl.Trainer(- logger=wandb_logger,- max_epochs=10,- accelerator="gpu",- devices=1-)--# Train (metrics logged automatically)-trainer.fit(model, datamodule=dm)--# Finish W&B run-wandb.finish()-```--### Log Media--```python-class LitModel(pl.LightningModule):- def validation_step(self, batch, batch_idx):- x, y = batch- y_hat = self.model(x)-- # Log images (first batch only)- if batch_idx == 0:- self.logger.experiment.log({- "examples": [wandb.Image(img) for img in x[:8]]- })-- return loss-- def on_validation_epoch_end(self):- # Log confusion matrix- cm = compute_confusion_matrix(self.all_preds, self.all_targets)-- self.logger.experiment.log({- "confusion_matrix": wandb.plot.confusion_matrix(- probs=None,- y_true=self.all_targets,- preds=self.all_preds,- class_names=self.class_names- )- })-```--### Hyperparameter Sweeps--```python-import pytorch_lightning as pl-from pytorch_lightning.loggers import WandbLogger-import wandb--# Define sweep-sweep_config = {- 'method': 'bayes',- 'metric': {'name': 'val/accuracy', 'goal': 'maximize'},- 'parameters': {- 'learning_rate': {'min': 1e-5, 'max': 1e-2, 'distribution': 'log_uniform'},- 'batch_size': {'values': [16, 32, 64]},- 'hidden_size': {'values': [128, 256, 512]}- }-}--sweep_id = wandb.sweep(sweep_config, project="lightning-sweeps")--def train():- # Initialize W&B- run = wandb.init()-- # Get hyperparameters- config = wandb.config-- # Create logger- wandb_logger = WandbLogger()-- # Create model with sweep params- model = LitModel(- learning_rate=config.learning_rate,- hidden_size=config.hidden_size- )-- # Create datamodule with sweep batch size- dm = DataModule(batch_size=config.batch_size)-- # Train- trainer = pl.Trainer(logger=wandb_logger, max_epochs=10)- trainer.fit(model, dm)--# Run sweep-wandb.agent(sweep_id, function=train, count=30)-```--## Keras/TensorFlow--### With Callback--```python-import tensorflow as tf-from wandb.keras import WandbCallback-import wandb--# Initialize W&B-wandb.init(- project="keras-demo",- config={- "learning_rate": 0.001,- "epochs": 10,- "batch_size": 32- }-)--config = wandb.config--# Build model-model = tf.keras.Sequential([- tf.keras.layers.Dense(128, activation='relu'),- tf.keras.layers.Dropout(0.2),- tf.keras.layers.Dense(10, activation='softmax')-])--model.compile(- optimizer=tf.keras.optimizers.Adam(config.learning_rate),- loss='sparse_categorical_crossentropy',- metrics=['accuracy']-)--# Train with W&B callback-history = model.fit(- x_train, y_train,- validation_data=(x_val, y_val),- epochs=config.epochs,- batch_size=config.batch_size,- callbacks=[- WandbCallback(- log_weights=True, # Log model weights- log_gradients=True, # Log gradients- training_data=(x_train, y_train),- validation_data=(x_val, y_val),- labels=class_names- )- ]-)--# Save model as artifact-model.save('model.h5')-artifact = wandb.Artifact('keras-model', type='model')-artifact.add_file('model.h5')-wandb.log_artifact(artifact)--wandb.finish()-```--### Custom Training Loop--```python-import tensorflow as tf-import wandb--wandb.init(project="tf-custom-loop")--# Model, optimizer, loss-model = create_model()-optimizer = tf.keras.optimizers.Adam(1e-3)-loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()--# Metrics-train_loss = tf.keras.metrics.Mean(name='train_loss')-train_accuracy = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')--def train_step(x, y):- with tf.GradientTape() as tape:- predictions = model(x, training=True)- loss = loss_fn(y, predictions)-- gradients = tape.gradient(loss, model.trainable_variables)- optimizer.apply_gradients(zip(gradients, model.trainable_variables))-- train_loss(loss)- train_accuracy(y, predictions)--# Training loop-for epoch in range(EPOCHS):- train_loss.reset_states()- train_accuracy.reset_states()-- for step, (x, y) in enumerate(train_dataset):- train_step(x, y)-- # Log every 100 steps- if step % 100 == 0:- wandb.log({- 'train/loss': train_loss.result().numpy(),- 'train/accuracy': train_accuracy.result().numpy(),- 'epoch': epoch,- 'step': step- })-- # Log epoch metrics- wandb.log({- 'epoch/train_loss': train_loss.result().numpy(),- 'epoch/train_accuracy': train_accuracy.result().numpy(),- 'epoch': epoch- })--wandb.finish()-```--## Fast.ai--### With Callback--```python-from fastai.vision.all import *-from fastai.callback.wandb import *-import wandb--# Initialize W&B-wandb.init(project="fastai-demo")--# Create data loaders-dls = ImageDataLoaders.from_folder(- path,- train='train',- valid='valid',- bs=64-)--# Create learner with W&B callback-learn = vision_learner(- dls,- resnet34,- metrics=accuracy,- cbs=WandbCallback(- log_preds=True, # Log predictions- log_model=True, # Log model as artifact- log_dataset=True # Log dataset as artifact- )-)--# Train (metrics logged automatically)-learn.fine_tune(5)--wandb.finish()-```--## XGBoost/LightGBM--### XGBoost--```python-import xgboost as xgb-import wandb--# Initialize W&B-run = wandb.init(project="xgboost-demo", config={- "max_depth": 6,- "learning_rate": 0.1,- "n_estimators": 100-})--config = wandb.config--# Create DMatrix-dtrain = xgb.DMatrix(X_train, label=y_train)-dval = xgb.DMatrix(X_val, label=y_val)--# XGBoost params-params = {- 'max_depth': config.max_depth,- 'learning_rate': config.learning_rate,- 'objective': 'binary:logistic',- 'eval_metric': ['logloss', 'auc']-}--# Custom callback for W&B-def wandb_callback(env):- """Log XGBoost metrics to W&B."""- for metric_name, metric_value in env.evaluation_result_list:- wandb.log({- f"{metric_name}": metric_value,- "iteration": env.iteration- })--# Train with callback-model = xgb.train(- params,- dtrain,- num_boost_round=config.n_estimators,- evals=[(dtrain, 'train'), (dval, 'val')],- callbacks=[wandb_callback],- verbose_eval=10-)--# Save model-model.save_model('xgboost_model.json')-artifact = wandb.Artifact('xgboost-model', type='model')-artifact.add_file('xgboost_model.json')-wandb.log_artifact(artifact)--wandb.finish()-```--### LightGBM--```python-import lightgbm as lgb-import wandb--run = wandb.init(project="lgbm-demo")--# Create datasets-train_data = lgb.Dataset(X_train, label=y_train)-val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)--# Parameters-params = {- 'objective': 'binary',- 'metric': ['binary_logloss', 'auc'],- 'learning_rate': 0.1,- 'num_leaves': 31-}--# Custom callback-def log_to_wandb(env):- """Log LightGBM metrics to W&B."""- for entry in env.evaluation_result_list:- dataset_name, metric_name, metric_value, _ = entry- wandb.log({- f"{dataset_name}/{metric_name}": metric_value,- "iteration": env.iteration- })--# Train-model = lgb.train(- params,- train_data,- num_boost_round=100,- valid_sets=[train_data, val_data],- valid_names=['train', 'val'],- callbacks=[log_to_wandb]-)--# Save model-model.save_model('lgbm_model.txt')-artifact = wandb.Artifact('lgbm-model', type='model')-artifact.add_file('lgbm_model.txt')-wandb.log_artifact(artifact)--wandb.finish()-```--## PyTorch Native--### Training Loop Integration--```python-import torch-import torch.nn as nn-import torch.optim as optim-import wandb--# Initialize W&B-wandb.init(project="pytorch-native", config={- "learning_rate": 0.001,- "epochs": 10,- "batch_size": 32-})--config = wandb.config--# Model, loss, optimizer-model = create_model()-criterion = nn.CrossEntropyLoss()-optimizer = optim.Adam(model.parameters(), lr=config.learning_rate)--# Watch model (logs gradients and parameters)-wandb.watch(model, criterion, log="all", log_freq=100)--# Training loop-for epoch in range(config.epochs):- model.train()- train_loss = 0.0- correct = 0- total = 0-- for batch_idx, (data, target) in enumerate(train_loader):- data, target = data.to(device), target.to(device)-- # Forward pass- optimizer.zero_grad()- output = model(data)- loss = criterion(output, target)-- # Backward pass- loss.backward()- optimizer.step()-- # Track metrics- train_loss += loss.item()- _, predicted = output.max(1)- total += target.size(0)- correct += predicted.eq(target).sum().item()-- # Log every 100 batches- if batch_idx % 100 == 0:- wandb.log({- 'train/loss': loss.item(),- 'train/batch_accuracy': 100. * correct / total,- 'epoch': epoch,- 'batch': batch_idx- })-- # Validation- model.eval()- val_loss = 0.0- val_correct = 0- val_total = 0-- with torch.no_grad():- for data, target in val_loader:- data, target = data.to(device), target.to(device)- output = model(data)- loss = criterion(output, target)-- val_loss += loss.item()- _, predicted = output.max(1)- val_total += target.size(0)- val_correct += predicted.eq(target).sum().item()-- # Log epoch metrics- wandb.log({- 'epoch/train_loss': train_loss / len(train_loader),- 'epoch/train_accuracy': 100. * correct / total,- 'epoch/val_loss': val_loss / len(val_loader),- 'epoch/val_accuracy': 100. * val_correct / val_total,- 'epoch': epoch- })--# Save final model-torch.save(model.state_dict(), 'model.pth')-artifact = wandb.Artifact('final-model', type='model')-artifact.add_file('model.pth')-wandb.log_artifact(artifact)--wandb.finish()-```--## Custom Integrations--### Generic Framework Integration--```python-import wandb--class WandbIntegration:- """Generic W&B integration wrapper."""-- def __init__(self, project, config):- self.run = wandb.init(project=project, config=config)- self.config = wandb.config- self.step = 0-- def log_metrics(self, metrics, step=None):- """Log training metrics."""- if step is None:- step = self.step- self.step += 1-- wandb.log(metrics, step=step)-- def log_images(self, images, caption=""):- """Log images."""- wandb.log({- caption: [wandb.Image(img) for img in images]- })-- def log_table(self, data, columns):- """Log tabular data."""- table = wandb.Table(columns=columns, data=data)- wandb.log({"table": table})-- def save_model(self, model_path, metadata=None):- """Save model as artifact."""- artifact = wandb.Artifact(- 'model',- type='model',- metadata=metadata or {}- )- artifact.add_file(model_path)- self.run.log_artifact(artifact)-- def finish(self):- """Finish W&B run."""- wandb.finish()--# Usage-wb = WandbIntegration(project="my-project", config={"lr": 0.001})--# Training loop-for epoch in range(10):- # Your training code- loss, accuracy = train_epoch()-- # Log metrics- wb.log_metrics({- 'train/loss': loss,- 'train/accuracy': accuracy- })--# Save model-wb.save_model('model.pth', metadata={'accuracy': 0.95})-wb.finish()-```--## Resources--- **Integrations Guide**: https://docs.wandb.ai/guides/integrations-- **HuggingFace**: https://docs.wandb.ai/guides/integrations/huggingface-- **PyTorch Lightning**: https://docs.wandb.ai/guides/integrations/lightning-- **Keras**: https://docs.wandb.ai/guides/integrations/keras-- **Examples**: https://github.com/wandb/examplesdiff --git a/optional-skills/mlops/evaluation/weights-and-biases/references/sweeps.md b/optional-skills/mlops/evaluation/weights-and-biases/references/sweeps.mddeleted file mode 100644index 38d93a2c777..00000000000--- a/optional-skills/mlops/evaluation/weights-and-biases/references/sweeps.md+++ /dev/null@@ -1,847 +0,0 @@-# Comprehensive Hyperparameter Sweeps Guide--Complete guide to hyperparameter optimization with W&B Sweeps.--## Table of Contents-- Sweep Configuration-- Search Strategies-- Parameter Distributions-- Early Termination-- Parallel Execution-- Advanced Patterns-- Real-World Examples--## Sweep Configuration--### Basic Sweep Config--```python-sweep_config = {- 'method': 'bayes', # Search strategy- 'metric': {- 'name': 'val/accuracy',- 'goal': 'maximize' # or 'minimize'- },- 'parameters': {- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- },- 'batch_size': {- 'values': [16, 32, 64, 128]- }- }-}--# Initialize sweep-sweep_id = wandb.sweep(sweep_config, project="my-project")-```--### Complete Config Example--```python-sweep_config = {- # Required: Search method- 'method': 'bayes',-- # Required: Optimization metric- 'metric': {- 'name': 'val/f1_score',- 'goal': 'maximize'- },-- # Required: Parameters to search- 'parameters': {- # Continuous parameter- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- },-- # Discrete values- 'batch_size': {- 'values': [16, 32, 64, 128]- },-- # Categorical- 'optimizer': {- 'values': ['adam', 'sgd', 'rmsprop', 'adamw']- },-- # Uniform distribution- 'dropout': {- 'distribution': 'uniform',- 'min': 0.1,- 'max': 0.5- },-- # Integer range- 'num_layers': {- 'distribution': 'int_uniform',- 'min': 2,- 'max': 10- },-- # Fixed value (constant across runs)- 'epochs': {- 'value': 50- }- },-- # Optional: Early termination- 'early_terminate': {- 'type': 'hyperband',- 'min_iter': 5,- 's': 2,- 'eta': 3,- 'max_iter': 27- }-}-```--## Search Strategies--### 1. Grid Search--Exhaustively search all combinations.--```python-sweep_config = {- 'method': 'grid',- 'parameters': {- 'learning_rate': {- 'values': [0.001, 0.01, 0.1]- },- 'batch_size': {- 'values': [16, 32, 64]- },- 'optimizer': {- 'values': ['adam', 'sgd']- }- }-}--# Total runs: 3 × 3 × 2 = 18 runs-```--**Pros:**-- Comprehensive search-- Reproducible results-- No randomness--**Cons:**-- Exponential growth with parameters-- Inefficient for continuous parameters-- Not scalable beyond 3-4 parameters--**When to use:**-- Few parameters (< 4)-- All discrete values-- Need complete coverage--### 2. Random Search--Randomly sample parameter combinations.--```python-sweep_config = {- 'method': 'random',- 'parameters': {- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- },- 'batch_size': {- 'values': [16, 32, 64, 128, 256]- },- 'dropout': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 0.5- },- 'num_layers': {- 'distribution': 'int_uniform',- 'min': 2,- 'max': 8- }- }-}--# Run 100 random trials-wandb.agent(sweep_id, function=train, count=100)-```--**Pros:**-- Scales to many parameters-- Can run indefinitely-- Often finds good solutions quickly--**Cons:**-- No learning from previous runs-- May miss optimal region-- Results vary with random seed--**When to use:**-- Many parameters (> 4)-- Quick exploration-- Limited budget--### 3. Bayesian Optimization (Recommended)--Learn from previous trials to sample promising regions.--```python-sweep_config = {- 'method': 'bayes',- 'metric': {- 'name': 'val/loss',- 'goal': 'minimize'- },- 'parameters': {- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- },- 'weight_decay': {- 'distribution': 'log_uniform',- 'min': 1e-6,- 'max': 1e-2- },- 'dropout': {- 'distribution': 'uniform',- 'min': 0.1,- 'max': 0.5- },- 'num_layers': {- 'values': [2, 3, 4, 5, 6]- }- }-}-```--**Pros:**-- Most sample-efficient-- Learns from past trials-- Focuses on promising regions--**Cons:**-- Initial random exploration phase-- May get stuck in local optima-- Slower per iteration--**When to use:**-- Expensive training runs-- Need best performance-- Limited compute budget--## Parameter Distributions--### Continuous Distributions--```python-# Log-uniform: Good for learning rates, regularization-'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-6,- 'max': 1e-1-}--# Uniform: Good for dropout, momentum-'dropout': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 0.5-}--# Normal distribution-'parameter': {- 'distribution': 'normal',- 'mu': 0.5,- 'sigma': 0.1-}--# Log-normal distribution-'parameter': {- 'distribution': 'log_normal',- 'mu': 0.0,- 'sigma': 1.0-}-```--### Discrete Distributions--```python-# Fixed values-'batch_size': {- 'values': [16, 32, 64, 128, 256]-}--# Integer uniform-'num_layers': {- 'distribution': 'int_uniform',- 'min': 2,- 'max': 10-}--# Quantized uniform (step size)-'layer_size': {- 'distribution': 'q_uniform',- 'min': 32,- 'max': 512,- 'q': 32 # Step by 32: 32, 64, 96, 128...-}--# Quantized log-uniform-'hidden_size': {- 'distribution': 'q_log_uniform',- 'min': 32,- 'max': 1024,- 'q': 32-}-```--### Categorical Parameters--```python-# Optimizers-'optimizer': {- 'values': ['adam', 'sgd', 'rmsprop', 'adamw']-}--# Model architectures-'model': {- 'values': ['resnet18', 'resnet34', 'resnet50', 'efficientnet_b0']-}--# Activation functions-'activation': {- 'values': ['relu', 'gelu', 'silu', 'leaky_relu']-}-```--## Early Termination--Stop underperforming runs early to save compute.--### Hyperband--```python-sweep_config = {- 'method': 'bayes',- 'metric': {'name': 'val/accuracy', 'goal': 'maximize'},- 'parameters': {...},-- # Hyperband early termination- 'early_terminate': {- 'type': 'hyperband',- 'min_iter': 3, # Minimum iterations before termination- 's': 2, # Bracket count- 'eta': 3, # Downsampling rate- 'max_iter': 27 # Maximum iterations- }-}-```--**How it works:**-- Runs trials in brackets-- Keeps top 1/eta performers each round-- Eliminates bottom performers early--### Custom Termination--```python-def train():- run = wandb.init()-- for epoch in range(MAX_EPOCHS):- loss = train_epoch()- val_acc = validate()-- wandb.log({'val/accuracy': val_acc, 'epoch': epoch})-- # Custom early stopping- if epoch > 5 and val_acc < 0.5:- print("Early stop: Poor performance")- break-- if epoch > 10 and val_acc > best_acc - 0.01:- print("Early stop: No improvement")- break-```--## Training Function--### Basic Template--```python-def train():- # Initialize W&B run- run = wandb.init()-- # Get hyperparameters- config = wandb.config-- # Build model with config- model = build_model(- hidden_size=config.hidden_size,- num_layers=config.num_layers,- dropout=config.dropout- )-- # Create optimizer- optimizer = create_optimizer(- model.parameters(),- name=config.optimizer,- lr=config.learning_rate,- weight_decay=config.weight_decay- )-- # Training loop- for epoch in range(config.epochs):- # Train- train_loss, train_acc = train_epoch(- model, optimizer, train_loader, config.batch_size- )-- # Validate- val_loss, val_acc = validate(model, val_loader)-- # Log metrics- wandb.log({- 'train/loss': train_loss,- 'train/accuracy': train_acc,- 'val/loss': val_loss,- 'val/accuracy': val_acc,- 'epoch': epoch- })-- # Log final model- torch.save(model.state_dict(), 'model.pth')- wandb.save('model.pth')-- # Finish run- wandb.finish()-```--### With PyTorch--```python-import torch-import torch.nn as nn-from torch.utils.data import DataLoader-import wandb--def train():- run = wandb.init()- config = wandb.config-- # Data- train_loader = DataLoader(- train_dataset,- batch_size=config.batch_size,- shuffle=True- )-- # Model- model = ResNet(- num_classes=config.num_classes,- dropout=config.dropout- ).to(device)-- # Optimizer- if config.optimizer == 'adam':- optimizer = torch.optim.Adam(- model.parameters(),- lr=config.learning_rate,- weight_decay=config.weight_decay- )- elif config.optimizer == 'sgd':- optimizer = torch.optim.SGD(- model.parameters(),- lr=config.learning_rate,- momentum=config.momentum,- weight_decay=config.weight_decay- )-- # Scheduler- scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(- optimizer, T_max=config.epochs- )-- # Training- for epoch in range(config.epochs):- model.train()- train_loss = 0.0-- for data, target in train_loader:- data, target = data.to(device), target.to(device)-- optimizer.zero_grad()- output = model(data)- loss = nn.CrossEntropyLoss()(output, target)- loss.backward()- optimizer.step()-- train_loss += loss.item()-- # Validation- model.eval()- val_loss, val_acc = validate(model, val_loader)-- # Step scheduler- scheduler.step()-- # Log- wandb.log({- 'train/loss': train_loss / len(train_loader),- 'val/loss': val_loss,- 'val/accuracy': val_acc,- 'learning_rate': scheduler.get_last_lr()[0],- 'epoch': epoch- })-```--## Parallel Execution--### Multiple Agents--Run sweep agents in parallel to speed up search.--```python-# Initialize sweep once-sweep_id = wandb.sweep(sweep_config, project="my-project")--# Run multiple agents in parallel-# Agent 1 (Terminal 1)-wandb.agent(sweep_id, function=train, count=20)--# Agent 2 (Terminal 2)-wandb.agent(sweep_id, function=train, count=20)--# Agent 3 (Terminal 3)-wandb.agent(sweep_id, function=train, count=20)--# Total: 60 runs across 3 agents-```--### Multi-GPU Execution--```python-import os--def train():- # Get available GPU- gpu_id = os.environ.get('CUDA_VISIBLE_DEVICES', '0')-- run = wandb.init()- config = wandb.config-- # Train on specific GPU- device = torch.device(f'cuda:{gpu_id}')- model = model.to(device)-- # ... rest of training ...--# Run agents on different GPUs-# Terminal 1-# CUDA_VISIBLE_DEVICES=0 wandb agent sweep_id--# Terminal 2-# CUDA_VISIBLE_DEVICES=1 wandb agent sweep_id--# Terminal 3-# CUDA_VISIBLE_DEVICES=2 wandb agent sweep_id-```--## Advanced Patterns--### Nested Parameters--```python-sweep_config = {- 'method': 'bayes',- 'metric': {'name': 'val/accuracy', 'goal': 'maximize'},- 'parameters': {- 'model': {- 'parameters': {- 'type': {- 'values': ['resnet', 'efficientnet']- },- 'size': {- 'values': ['small', 'medium', 'large']- }- }- },- 'optimizer': {- 'parameters': {- 'type': {- 'values': ['adam', 'sgd']- },- 'lr': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- }- }- }- }-}--# Access nested config-def train():- run = wandb.init()- model_type = wandb.config.model.type- model_size = wandb.config.model.size- opt_type = wandb.config.optimizer.type- lr = wandb.config.optimizer.lr-```--### Conditional Parameters--```python-sweep_config = {- 'method': 'bayes',- 'parameters': {- 'optimizer': {- 'values': ['adam', 'sgd']- },- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-1- },- # Only used if optimizer == 'sgd'- 'momentum': {- 'distribution': 'uniform',- 'min': 0.5,- 'max': 0.99- }- }-}--def train():- run = wandb.init()- config = wandb.config-- if config.optimizer == 'adam':- optimizer = torch.optim.Adam(- model.parameters(),- lr=config.learning_rate- )- elif config.optimizer == 'sgd':- optimizer = torch.optim.SGD(- model.parameters(),- lr=config.learning_rate,- momentum=config.momentum # Conditional parameter- )-```--## Real-World Examples--### Image Classification--```python-sweep_config = {- 'method': 'bayes',- 'metric': {- 'name': 'val/top1_accuracy',- 'goal': 'maximize'- },- 'parameters': {- # Model- 'architecture': {- 'values': ['resnet50', 'resnet101', 'efficientnet_b0', 'efficientnet_b3']- },- 'pretrained': {- 'values': [True, False]- },-- # Training- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-5,- 'max': 1e-2- },- 'batch_size': {- 'values': [16, 32, 64, 128]- },- 'optimizer': {- 'values': ['adam', 'sgd', 'adamw']- },- 'weight_decay': {- 'distribution': 'log_uniform',- 'min': 1e-6,- 'max': 1e-2- },-- # Regularization- 'dropout': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 0.5- },- 'label_smoothing': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 0.2- },-- # Data augmentation- 'mixup_alpha': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 1.0- },- 'cutmix_alpha': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 1.0- }- },- 'early_terminate': {- 'type': 'hyperband',- 'min_iter': 5- }-}-```--### NLP Fine-Tuning--```python-sweep_config = {- 'method': 'bayes',- 'metric': {'name': 'eval/f1', 'goal': 'maximize'},- 'parameters': {- # Model- 'model_name': {- 'values': ['bert-base-uncased', 'roberta-base', 'distilbert-base-uncased']- },-- # Training- 'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-6,- 'max': 1e-4- },- 'per_device_train_batch_size': {- 'values': [8, 16, 32]- },- 'num_train_epochs': {- 'values': [3, 4, 5]- },- 'warmup_ratio': {- 'distribution': 'uniform',- 'min': 0.0,- 'max': 0.1- },- 'weight_decay': {- 'distribution': 'log_uniform',- 'min': 1e-4,- 'max': 1e-1- },-- # Optimizer- 'adam_beta1': {- 'distribution': 'uniform',- 'min': 0.8,- 'max': 0.95- },- 'adam_beta2': {- 'distribution': 'uniform',- 'min': 0.95,- 'max': 0.999- }- }-}-```--## Best Practices--### 1. Start Small--```python-# Initial exploration: Random search, 20 runs-sweep_config_v1 = {- 'method': 'random',- 'parameters': {...}-}-wandb.agent(sweep_id_v1, train, count=20)--# Refined search: Bayes, narrow ranges-sweep_config_v2 = {- 'method': 'bayes',- 'parameters': {- 'learning_rate': {- 'min': 5e-5, # Narrowed from 1e-6 to 1e-4- 'max': 1e-4- }- }-}-```--### 2. Use Log Scales--```python-# ✅ Good: Log scale for learning rate-'learning_rate': {- 'distribution': 'log_uniform',- 'min': 1e-6,- 'max': 1e-2-}--# ❌ Bad: Linear scale-'learning_rate': {- 'distribution': 'uniform',- 'min': 0.000001,- 'max': 0.01-}-```--### 3. Set Reasonable Ranges--```python-# Base ranges on prior knowledge-'learning_rate': {'min': 1e-5, 'max': 1e-3}, # Typical for Adam-'batch_size': {'values': [16, 32, 64]}, # GPU memory limits-'dropout': {'min': 0.1, 'max': 0.5} # Too high hurts training-```--### 4. Monitor Resource Usage--```python-def train():- run = wandb.init()-- # Log system metrics- wandb.log({- 'system/gpu_memory_allocated': torch.cuda.memory_allocated(),- 'system/gpu_memory_reserved': torch.cuda.memory_reserved()- })-```--### 5. Save Best Models--```python-def train():- run = wandb.init()- best_acc = 0.0-- for epoch in range(config.epochs):- val_acc = validate(model)-- if val_acc > best_acc:- best_acc = val_acc- # Save best checkpoint- torch.save(model.state_dict(), 'best_model.pth')- wandb.save('best_model.pth')-```--## Resources--- **Sweeps Documentation**: https://docs.wandb.ai/guides/sweeps-- **Configuration Reference**: https://docs.wandb.ai/guides/sweeps/configuration-- **Examples**: https://github.com/wandb/examples/tree/master/examples/wandb-sweepsdiff --git a/optional-skills/mlops/guidance/SKILL.md b/optional-skills/mlops/guidance/SKILL.mddeleted file mode 100644index 157eb98a40f..00000000000--- a/optional-skills/mlops/guidance/SKILL.md+++ /dev/null@@ -1,580 +0,0 @@-----name: guidance-description: Constrain LLM output with grammars; guarantee valid JSON.-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [guidance, transformers]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Prompt Engineering, Guidance, Constrained Generation, Structured Output, JSON Validation, Grammar, Microsoft Research, Format Enforcement, Multi-Step Workflows]-------# Guidance: Constrained LLM Generation--## When to Use This Skill--Use Guidance when you need to:-- **Control LLM output syntax** with regex or grammars-- **Guarantee valid JSON/XML/code** generation-- **Reduce latency** vs traditional prompting approaches-- **Enforce structured formats** (dates, emails, IDs, etc.)-- **Build multi-step workflows** with Pythonic control flow-- **Prevent invalid outputs** through grammatical constraints--**GitHub Stars**: 18,000+ | **From**: Microsoft Research--## Installation--```bash-# Base installation-pip install guidance--# With specific backends-pip install guidance[transformers] # Hugging Face models-pip install guidance[llama_cpp] # llama.cpp models-```--## Quick Start--### Basic Example: Structured Generation--```python-from guidance import models, gen--# Load model (supports OpenAI, Transformers, llama.cpp)-lm = models.OpenAI("gpt-4")--# Generate with constraints-result = lm + "The capital of France is " + gen("capital", max_tokens=5)--print(result["capital"]) # "Paris"-```--### Chat format with a local model--> **Constraint support requires local logit access.** Regex, `select()`, and-> grammar-based constrained generation only work with local backends-> (`Transformers`, `LlamaCpp`). Remote API backends (`OpenAI`, and Azure-> variants) support unconstrained `gen()` / chat only — they cannot enforce-> token-level constraints. guidance 0.3.x has no `models.Anthropic` class.--```python-from guidance import models, gen, system, user, assistant--# Local model (supports constrained generation)-lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# Use context managers for chat format-with system():- lm += "You are a helpful assistant."--with user():- lm += "What is the capital of France?"--with assistant():- lm += gen(max_tokens=20)-```--## Core Concepts--### 1. Context Managers--Guidance uses Pythonic context managers for chat-style interactions.--```python-from guidance import system, user, assistant, gen--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# System message-with system():- lm += "You are a JSON generation expert."--# User message-with user():- lm += "Generate a person object with name and age."--# Assistant response-with assistant():- lm += gen("response", max_tokens=100)--print(lm["response"])-```--**Benefits:**-- Natural chat flow-- Clear role separation-- Easy to read and maintain--### 2. Constrained Generation--Guidance ensures outputs match specified patterns using regex or grammars.--#### Regex Constraints--```python-from guidance import models, gen--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# Constrain to valid email format-lm += "Email: " + gen("email", regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")--# Constrain to date format (YYYY-MM-DD)-lm += "Date: " + gen("date", regex=r"\d{4}-\d{2}-\d{2}")--# Constrain to phone number-lm += "Phone: " + gen("phone", regex=r"\d{3}-\d{3}-\d{4}")--print(lm["email"]) # Guaranteed valid email-print(lm["date"]) # Guaranteed YYYY-MM-DD format-```--**How it works:**-- Regex converted to grammar at token level-- Invalid tokens filtered during generation-- Model can only produce matching outputs--#### Selection Constraints--```python-from guidance import models, gen, select--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# Constrain to specific choices-lm += "Sentiment: " + select(["positive", "negative", "neutral"], name="sentiment")--# Multiple-choice selection-lm += "Best answer: " + select(- ["A) Paris", "B) London", "C) Berlin", "D) Madrid"],- name="answer"-)--print(lm["sentiment"]) # One of: positive, negative, neutral-print(lm["answer"]) # One of: A, B, C, or D-```--### 3. Token Healing--Guidance automatically "heals" token boundaries between prompt and generation.--**Problem:** Tokenization creates unnatural boundaries.--```python-# Without token healing-prompt = "The capital of France is "-# Last token: " is "-# First generated token might be " Par" (with leading space)-# Result: "The capital of France is Paris" (double space!)-```--**Solution:** Guidance backs up one token and regenerates.--```python-from guidance import models, gen--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# Token healing enabled by default-lm += "The capital of France is " + gen("capital", max_tokens=5)-# Result: "The capital of France is Paris" (correct spacing)-```--**Benefits:**-- Natural text boundaries-- No awkward spacing issues-- Better model performance (sees natural token sequences)--### 4. Grammar-Based Generation--Define complex structures by composing grammar functions. The template-string-`grammar=` form is not part of current guidance — build grammars from-composable functions, or use `guidance.json()` for JSON.--```python-from guidance import models, gen-from guidance import json as gen_json-from pydantic import BaseModel, Field--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--# JSON via a Pydantic schema (guidance.json compiles the schema to a grammar)-class Person(BaseModel):- name: str = Field(pattern=r"[A-Za-z ]+")- age: int- email: str = Field(pattern=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")--lm += gen_json(name="person", schema=Person)--print(lm["person"]) # Guaranteed valid JSON matching the schema--# Or compose grammar functions directly:-grammar = "name=" + gen("name", regex=r"[A-Za-z ]+") + " age=" + gen("age", regex=r"[0-9]+")-lm += grammar-```--**Use cases:**-- Complex structured outputs-- Nested data structures-- Programming language syntax-- Domain-specific languages--### 5. Guidance Functions--Create reusable generation patterns with the `@guidance` decorator.--```python-from guidance import guidance, gen, models--@guidance-def generate_person(lm):- """Generate a person with name and age."""- lm += "Name: " + gen("name", max_tokens=20, stop="\n")- lm += "\nAge: " + gen("age", regex=r"[0-9]+", max_tokens=3)- return lm--# Use the function-lm = models.Transformers("microsoft/Phi-4-mini-instruct")-lm = generate_person(lm)--print(lm["name"])-print(lm["age"])-```--**Stateful Functions:**--```python-@guidance(stateless=False)-def react_agent(lm, question, tools, max_rounds=5):- """ReAct agent with tool use."""- lm += f"Question: {question}\n\n"-- for i in range(max_rounds):- # Thought- lm += f"Thought {i+1}: " + gen("thought", stop="\n")-- # Action- lm += "\nAction: " + select(list(tools.keys()), name="action")-- # Execute tool- tool_result = tools[lm["action"]]()- lm += f"\nObservation: {tool_result}\n\n"-- # Check if done- lm += "Done? " + select(["Yes", "No"], name="done")- if lm["done"] == "Yes":- break-- # Final answer- lm += "\nFinal Answer: " + gen("answer", max_tokens=100)- return lm-```--## Backend Configuration--### OpenAI (remote — unconstrained only)--> Remote API backends cannot do constrained generation (regex/select/grammar);-> use them only for plain chat/`gen()`. For constraints, use a local backend.--```python-from guidance import models--lm = models.OpenAI(- model="gpt-4o-mini",- api_key="your-api-key" # Or set OPENAI_API_KEY env var-)-```--### Local Models (Transformers)--```python-from guidance.models import Transformers--lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cuda" # Or "cpu"-)-```--### Local Models (llama.cpp)--```python-from guidance.models import LlamaCpp--lm = LlamaCpp(- model_path="/path/to/model.gguf",- n_ctx=4096,- n_gpu_layers=35-)-```--## Common Patterns--### Pattern 1: JSON Generation--```python-from guidance import models, gen, system, user, assistant--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--with system():- lm += "You generate valid JSON."--with user():- lm += "Generate a user profile with name, age, and email."--with assistant():- lm += """{- "name": """ + gen("name", regex=r'"[A-Za-z ]+"', max_tokens=30) + """,- "age": """ + gen("age", regex=r"[0-9]+", max_tokens=3) + """,- "email": """ + gen("email", regex=r'"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"', max_tokens=50) + """-}"""--print(lm) # Valid JSON guaranteed-```--### Pattern 2: Classification--```python-from guidance import models, gen, select--lm = models.Transformers("microsoft/Phi-4-mini-instruct")--text = "This product is amazing! I love it."--lm += f"Text: {text}\n"-lm += "Sentiment: " + select(["positive", "negative", "neutral"], name="sentiment")-lm += "\nConfidence: " + gen("confidence", regex=r"[0-9]+", max_tokens=3) + "%"--print(f"Sentiment: {lm['sentiment']}")-print(f"Confidence: {lm['confidence']}%")-```--### Pattern 3: Multi-Step Reasoning--```python-from guidance import models, gen, guidance--@guidance-def chain_of_thought(lm, question):- """Generate answer with step-by-step reasoning."""- lm += f"Question: {question}\n\n"-- # Generate multiple reasoning steps- for i in range(3):- lm += f"Step {i+1}: " + gen(f"step_{i+1}", stop="\n", max_tokens=100) + "\n"-- # Final answer- lm += "\nTherefore, the answer is: " + gen("answer", max_tokens=50)-- return lm--lm = models.Transformers("microsoft/Phi-4-mini-instruct")-lm = chain_of_thought(lm, "What is 15% of 200?")--print(lm["answer"])-```--### Pattern 4: ReAct Agent--```python-from guidance import models, gen, select, guidance--@guidance(stateless=False)-def react_agent(lm, question):- """ReAct agent with tool use."""- tools = {- "calculator": lambda expr: eval(expr),- "search": lambda query: f"Search results for: {query}",- }-- lm += f"Question: {question}\n\n"-- for round in range(5):- # Thought- lm += f"Thought: " + gen("thought", stop="\n") + "\n"-- # Action selection- lm += "Action: " + select(["calculator", "search", "answer"], name="action")-- if lm["action"] == "answer":- lm += "\nFinal Answer: " + gen("answer", max_tokens=100)- break-- # Action input- lm += "\nAction Input: " + gen("action_input", stop="\n") + "\n"-- # Execute tool- if lm["action"] in tools:- result = tools[lm["action"]](lm["action_input"])- lm += f"Observation: {result}\n\n"-- return lm--lm = models.Transformers("microsoft/Phi-4-mini-instruct")-lm = react_agent(lm, "What is 25 * 4 + 10?")-print(lm["answer"])-```--### Pattern 5: Data Extraction--```python-from guidance import models, gen, guidance--@guidance-def extract_entities(lm, text):- """Extract structured entities from text."""- lm += f"Text: {text}\n\n"-- # Extract person- lm += "Person: " + gen("person", stop="\n", max_tokens=30) + "\n"-- # Extract organization- lm += "Organization: " + gen("organization", stop="\n", max_tokens=30) + "\n"-- # Extract date- lm += "Date: " + gen("date", regex=r"\d{4}-\d{2}-\d{2}", max_tokens=10) + "\n"-- # Extract location- lm += "Location: " + gen("location", stop="\n", max_tokens=30) + "\n"-- return lm--text = "Tim Cook announced at Apple Park on 2024-09-15 in Cupertino."--lm = models.Transformers("microsoft/Phi-4-mini-instruct")-lm = extract_entities(lm, text)--print(f"Person: {lm['person']}")-print(f"Organization: {lm['organization']}")-print(f"Date: {lm['date']}")-print(f"Location: {lm['location']}")-```--## Best Practices--### 1. Use Regex for Format Validation--```python-# ✅ Good: Regex ensures valid format-lm += "Email: " + gen("email", regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")--# ❌ Bad: Free generation may produce invalid emails-lm += "Email: " + gen("email", max_tokens=50)-```--### 2. Use select() for Fixed Categories--```python-# ✅ Good: Guaranteed valid category-lm += "Status: " + select(["pending", "approved", "rejected"], name="status")--# ❌ Bad: May generate typos or invalid values-lm += "Status: " + gen("status", max_tokens=20)-```--### 3. Leverage Token Healing--```python-# Token healing is enabled by default-# No special action needed - just concatenate naturally-lm += "The capital is " + gen("capital") # Automatic healing-```--### 4. Use stop Sequences--```python-# ✅ Good: Stop at newline for single-line outputs-lm += "Name: " + gen("name", stop="\n")--# ❌ Bad: May generate multiple lines-lm += "Name: " + gen("name", max_tokens=50)-```--### 5. Create Reusable Functions--```python-# ✅ Good: Reusable pattern-@guidance-def generate_person(lm):- lm += "Name: " + gen("name", stop="\n")- lm += "\nAge: " + gen("age", regex=r"[0-9]+")- return lm--# Use multiple times-lm = generate_person(lm)-lm += "\n\n"-lm = generate_person(lm)-```--### 6. Balance Constraints--```python-# ✅ Good: Reasonable constraints-lm += gen("name", regex=r"[A-Za-z ]+", max_tokens=30)--# ❌ Too strict: May fail or be very slow-lm += gen("name", regex=r"^(John|Jane)$", max_tokens=10)-```--## Comparison to Alternatives--| Feature | Guidance | Instructor | Outlines | LMQL |-|---------|----------|------------|----------|------|-| Regex Constraints | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |-| Grammar Support | ✅ CFG | ❌ No | ✅ CFG | ✅ CFG |-| Pydantic Validation | ❌ No | ✅ Yes | ✅ Yes | ❌ No |-| Token Healing | ✅ Yes | ❌ No | ✅ Yes | ❌ No |-| Local Models | ✅ Yes | ⚠️ Limited | ✅ Yes | ✅ Yes |-| API Models | ✅ Yes | ✅ Yes | ⚠️ Limited | ✅ Yes |-| Pythonic Syntax | ✅ Yes | ✅ Yes | ✅ Yes | ❌ SQL-like |-| Learning Curve | Low | Low | Medium | High |--**When to choose Guidance:**-- Need regex/grammar constraints-- Want token healing-- Building complex workflows with control flow-- Using local models (Transformers, llama.cpp)-- Prefer Pythonic syntax--**When to choose alternatives:**-- Instructor: Need Pydantic validation with automatic retrying-- Outlines: Need JSON schema validation-- LMQL: Prefer declarative query syntax--## Performance Characteristics--**Latency Reduction:**-- 30-50% faster than traditional prompting for constrained outputs-- Token healing reduces unnecessary regeneration-- Grammar constraints prevent invalid token generation--**Memory Usage:**-- Minimal overhead vs unconstrained generation-- Grammar compilation cached after first use-- Efficient token filtering at inference time--**Token Efficiency:**-- Prevents wasted tokens on invalid outputs-- No need for retry loops-- Direct path to valid outputs--## Resources--- **Documentation**: https://guidance.readthedocs.io-- **GitHub**: https://github.com/guidance-ai/guidance (18k+ stars)-- **Notebooks**: https://github.com/guidance-ai/guidance/tree/main/notebooks-- **Discord**: Community support available--## See Also--- `references/constraints.md` - Comprehensive regex and grammar patterns-- `references/backends.md` - Backend-specific configuration-- `references/examples.md` - Production-ready examples--diff --git a/optional-skills/mlops/guidance/references/backends.md b/optional-skills/mlops/guidance/references/backends.mddeleted file mode 100644index e1e9c5e4406..00000000000--- a/optional-skills/mlops/guidance/references/backends.md+++ /dev/null@@ -1,554 +0,0 @@-# Backend Configuration Guide--Complete guide to configuring Guidance with different LLM backends.--## Table of Contents-- API-Based Models (Anthropic, OpenAI)-- Local Models (Transformers, llama.cpp)-- Backend Comparison-- Performance Tuning-- Advanced Configuration--## API-Based Models--### Anthropic Claude--#### Basic Setup--```python-from guidance import models--# Using environment variable-lm = models.Anthropic("claude-sonnet-4-5-20250929")-# Reads ANTHROPIC_API_KEY from environment--# Explicit API key-lm = models.Anthropic(- model="claude-sonnet-4-5-20250929",- api_key="your-api-key-here"-)-```--#### Available Models--```python-# Claude 3.5 Sonnet (Latest, recommended)-lm = models.Anthropic("claude-sonnet-4-5-20250929")--# Claude 3.7 Sonnet (Fast, cost-effective)-lm = models.Anthropic("claude-sonnet-3.7-20250219")--# Claude 3 Opus (Most capable)-lm = models.Anthropic("claude-3-opus-20240229")--# Claude 3.5 Haiku (Fastest, cheapest)-lm = models.Anthropic("claude-3-5-haiku-20241022")-```--#### Configuration Options--```python-lm = models.Anthropic(- model="claude-sonnet-4-5-20250929",- api_key="your-api-key",- max_tokens=4096, # Max tokens to generate- temperature=0.7, # Sampling temperature (0-1)- top_p=0.9, # Nucleus sampling- timeout=30, # Request timeout (seconds)- max_retries=3 # Retry failed requests-)-```--#### With Context Managers--```python-from guidance import models, system, user, assistant, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--with system():- lm += "You are a helpful assistant."--with user():- lm += "What is the capital of France?"--with assistant():- lm += gen(max_tokens=50)--print(lm)-```--### OpenAI--#### Basic Setup--```python-from guidance import models--# Using environment variable-lm = models.OpenAI("gpt-4o")-# Reads OPENAI_API_KEY from environment--# Explicit API key-lm = models.OpenAI(- model="gpt-4o",- api_key="your-api-key-here"-)-```--#### Available Models--```python-# GPT-4o (Latest, multimodal)-lm = models.OpenAI("gpt-4o")--# GPT-4o Mini (Fast, cost-effective)-lm = models.OpenAI("gpt-4o-mini")--# GPT-4 Turbo-lm = models.OpenAI("gpt-4-turbo")--# GPT-3.5 Turbo (Cheapest)-lm = models.OpenAI("gpt-3.5-turbo")-```--#### Configuration Options--```python-lm = models.OpenAI(- model="gpt-4o-mini",- api_key="your-api-key",- max_tokens=2048,- temperature=0.7,- top_p=1.0,- frequency_penalty=0.0,- presence_penalty=0.0,- timeout=30-)-```--#### Chat Format--```python-from guidance import models, gen--lm = models.OpenAI("gpt-4o-mini")--# OpenAI uses chat format-lm += [- {"role": "system", "content": "You are a helpful assistant."},- {"role": "user", "content": "What is 2+2?"}-]--# Generate response-lm += gen(max_tokens=50)-```--### Azure OpenAI--```python-from guidance import models--lm = models.AzureOpenAI(- model="gpt-4o",- azure_endpoint="https://your-resource.openai.azure.com/",- api_key="your-azure-api-key",- api_version="2024-02-15-preview",- deployment_name="your-deployment-name"-)-```--## Local Models--### Transformers (Hugging Face)--#### Basic Setup--```python-from guidance.models import Transformers--# Load model from Hugging Face-lm = Transformers("microsoft/Phi-4-mini-instruct")-```--#### GPU Configuration--```python-# Use GPU-lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cuda"-)--# Use specific GPU-lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cuda:0" # GPU 0-)--# Use CPU-lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cpu"-)-```--#### Advanced Configuration--```python-lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cuda",- torch_dtype="float16", # Use FP16 (faster, less memory)- load_in_8bit=True, # 8-bit quantization- max_memory={0: "20GB"}, # GPU memory limit- offload_folder="./offload" # Offload to disk if needed-)-```--#### Popular Models--```python-# Phi-4 (Microsoft)-lm = Transformers("microsoft/Phi-4-mini-instruct")-lm = Transformers("microsoft/Phi-3-medium-4k-instruct")--# Llama 3 (Meta)-lm = Transformers("meta-llama/Llama-3.1-8B-Instruct")-lm = Transformers("meta-llama/Llama-3.1-70B-Instruct")--# Mistral (Mistral AI)-lm = Transformers("mistralai/Mistral-7B-Instruct-v0.3")-lm = Transformers("mistralai/Mixtral-8x7B-Instruct-v0.1")--# Qwen (Alibaba)-lm = Transformers("Qwen/Qwen2.5-7B-Instruct")--# Gemma (Google)-lm = Transformers("google/gemma-2-9b-it")-```--#### Generation Configuration--```python-lm = Transformers(- "microsoft/Phi-4-mini-instruct",- device="cuda"-)--# Configure generation-from guidance import gen--result = lm + gen(- max_tokens=100,- temperature=0.7,- top_p=0.9,- top_k=50,- repetition_penalty=1.1-)-```--### llama.cpp--#### Basic Setup--```python-from guidance.models import LlamaCpp--# Load GGUF model-lm = LlamaCpp(- model_path="/path/to/model.gguf",- n_ctx=4096 # Context window-)-```--#### GPU Configuration--```python-# Use GPU acceleration-lm = LlamaCpp(- model_path="/path/to/model.gguf",- n_ctx=4096,- n_gpu_layers=35, # Offload 35 layers to GPU- n_threads=8 # CPU threads for remaining layers-)--# Full GPU offload-lm = LlamaCpp(- model_path="/path/to/model.gguf",- n_ctx=4096,- n_gpu_layers=-1 # Offload all layers-)-```--#### Advanced Configuration--```python-lm = LlamaCpp(- model_path="/path/to/llama-3.1-8b-instruct.Q4_K_M.gguf",- n_ctx=8192, # Context window (tokens)- n_gpu_layers=35, # GPU layers- n_threads=8, # CPU threads- n_batch=512, # Batch size for prompt processing- use_mmap=True, # Memory-map the model file- use_mlock=False, # Lock model in RAM- seed=42, # Random seed- verbose=False # Suppress verbose output-)-```--#### Quantized Models--```python-# Q4_K_M (4-bit, recommended for most cases)-lm = LlamaCpp("/path/to/model.Q4_K_M.gguf")--# Q5_K_M (5-bit, better quality)-lm = LlamaCpp("/path/to/model.Q5_K_M.gguf")--# Q8_0 (8-bit, high quality)-lm = LlamaCpp("/path/to/model.Q8_0.gguf")--# F16 (16-bit float, highest quality)-lm = LlamaCpp("/path/to/model.F16.gguf")-```--#### Popular GGUF Models--```python-# Llama 3.1-lm = LlamaCpp("llama-3.1-8b-instruct.Q4_K_M.gguf")--# Mistral-lm = LlamaCpp("mistral-7b-instruct-v0.3.Q4_K_M.gguf")--# Phi-4-lm = LlamaCpp("phi-4-mini-instruct.Q4_K_M.gguf")-```--## Backend Comparison--### Feature Matrix--| Feature | Anthropic | OpenAI | Transformers | llama.cpp |-|---------|-----------|--------|--------------|-----------|-| Constrained Generation | ✅ Full | ✅ Full | ✅ Full | ✅ Full |-| Token Healing | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |-| Streaming | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |-| GPU Support | N/A | N/A | ✅ Yes | ✅ Yes |-| Quantization | N/A | N/A | ✅ Yes | ✅ Yes |-| Cost | $$$ | $$$ | Free | Free |-| Latency | Low | Low | Medium | Low |-| Setup Difficulty | Easy | Easy | Medium | Medium |--### Performance Characteristics--**Anthropic Claude:**-- **Latency**: 200-500ms (API call)-- **Throughput**: Limited by API rate limits-- **Cost**: $3-15 per 1M input tokens-- **Best for**: Production systems, high-quality outputs--**OpenAI:**-- **Latency**: 200-400ms (API call)-- **Throughput**: Limited by API rate limits-- **Cost**: $0.15-30 per 1M input tokens-- **Best for**: Cost-sensitive production, gpt-4o-mini--**Transformers:**-- **Latency**: 50-200ms (local inference)-- **Throughput**: GPU-dependent (10-100 tokens/sec)-- **Cost**: Hardware cost only-- **Best for**: Privacy-sensitive, high-volume, experimentation--**llama.cpp:**-- **Latency**: 30-150ms (local inference)-- **Throughput**: Hardware-dependent (20-150 tokens/sec)-- **Cost**: Hardware cost only-- **Best for**: Edge deployment, Apple Silicon, CPU inference--### Memory Requirements--**Transformers (FP16):**-- 7B model: ~14GB GPU VRAM-- 13B model: ~26GB GPU VRAM-- 70B model: ~140GB GPU VRAM (multi-GPU)--**llama.cpp (Q4_K_M):**-- 7B model: ~4.5GB RAM-- 13B model: ~8GB RAM-- 70B model: ~40GB RAM--**Optimization Tips:**-- Use quantized models (Q4_K_M) for lower memory-- Use GPU offloading for faster inference-- Use CPU inference for smaller models (<7B)--## Performance Tuning--### API Models (Anthropic, OpenAI)--#### Reduce Latency--```python-from guidance import models, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--# Use lower max_tokens (faster response)-lm += gen(max_tokens=100) # Instead of 1000--# Use streaming (perceived latency reduction)-for chunk in lm.stream(gen(max_tokens=500)):- print(chunk, end="", flush=True)-```--#### Reduce Cost--```python-# Use cheaper models-lm = models.Anthropic("claude-3-5-haiku-20241022") # vs Sonnet-lm = models.OpenAI("gpt-4o-mini") # vs gpt-4o--# Reduce context size-# - Keep prompts concise-# - Avoid large few-shot examples-# - Use max_tokens limits-```--### Local Models (Transformers, llama.cpp)--#### Optimize GPU Usage--```python-from guidance.models import Transformers--# Use FP16 for 2x speedup-lm = Transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- torch_dtype="float16"-)--# Use 8-bit quantization for 4x memory reduction-lm = Transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- load_in_8bit=True-)--# Use flash attention (requires flash-attn package)-lm = Transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- use_flash_attention_2=True-)-```--#### Optimize llama.cpp--```python-from guidance.models import LlamaCpp--# Maximize GPU layers-lm = LlamaCpp(- model_path="/path/to/model.Q4_K_M.gguf",- n_gpu_layers=-1 # All layers on GPU-)--# Optimize batch size-lm = LlamaCpp(- model_path="/path/to/model.Q4_K_M.gguf",- n_batch=512, # Larger batch = faster prompt processing- n_gpu_layers=-1-)--# Use Metal (Apple Silicon)-lm = LlamaCpp(- model_path="/path/to/model.Q4_K_M.gguf",- n_gpu_layers=-1, # Use Metal GPU acceleration- use_mmap=True-)-```--#### Batch Processing--```python-# Process multiple requests efficiently-requests = [- "What is 2+2?",- "What is the capital of France?",- "What is photosynthesis?"-]--# Bad: Sequential processing-for req in requests:- lm = Transformers("microsoft/Phi-4-mini-instruct")- lm += req + gen(max_tokens=50)--# Good: Reuse loaded model-lm = Transformers("microsoft/Phi-4-mini-instruct")-for req in requests:- lm += req + gen(max_tokens=50)-```--## Advanced Configuration--### Custom Model Configurations--```python-from transformers import AutoTokenizer, AutoModelForCausalLM-from guidance.models import Transformers--# Load custom model-tokenizer = AutoTokenizer.from_pretrained("your-model")-model = AutoModelForCausalLM.from_pretrained(- "your-model",- device_map="auto",- torch_dtype="float16"-)--# Use with Guidance-lm = Transformers(model=model, tokenizer=tokenizer)-```--### Environment Variables--```bash-# API keys-export ANTHROPIC_API_KEY="sk-ant-..."-export OPENAI_API_KEY="sk-..."--# Transformers cache-export HF_HOME="/path/to/cache"-export TRANSFORMERS_CACHE="/path/to/cache"--# GPU selection-export CUDA_VISIBLE_DEVICES=0,1 # Use GPU 0 and 1-```--### Debugging--```python-# Enable verbose logging-import logging-logging.basicConfig(level=logging.DEBUG)--# Check backend info-lm = models.Anthropic("claude-sonnet-4-5-20250929")-print(f"Model: {lm.model_name}")-print(f"Backend: {lm.backend}")--# Check GPU usage (Transformers)-lm = Transformers("microsoft/Phi-4-mini-instruct", device="cuda")-print(f"Device: {lm.device}")-print(f"Memory allocated: {torch.cuda.memory_allocated() / 1e9:.2f} GB")-```--## Resources--- **Anthropic Docs**: https://docs.anthropic.com-- **OpenAI Docs**: https://platform.openai.com/docs-- **Hugging Face Models**: https://huggingface.co/models-- **llama.cpp**: https://github.com/ggerganov/llama.cpp-- **GGUF Models**: https://huggingface.co/models?library=ggufdiff --git a/optional-skills/mlops/guidance/references/constraints.md b/optional-skills/mlops/guidance/references/constraints.mddeleted file mode 100644index 99c81890c91..00000000000--- a/optional-skills/mlops/guidance/references/constraints.md+++ /dev/null@@ -1,674 +0,0 @@-# Comprehensive Constraint Patterns--Guide to regex constraints, grammar-based generation, and token healing in Guidance.--## Table of Contents-- Regex Constraints-- Grammar-Based Generation-- Token Healing-- Selection Constraints-- Complex Patterns-- Performance Optimization--## Regex Constraints--### Basic Patterns--#### Numeric Constraints--```python-from guidance import models, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--# Integer (positive)-lm += "Age: " + gen("age", regex=r"[0-9]+")--# Integer (with negatives)-lm += "Temperature: " + gen("temp", regex=r"-?[0-9]+")--# Float (positive)-lm += "Price: $" + gen("price", regex=r"[0-9]+\.[0-9]{2}")--# Float (with negatives and optional decimals)-lm += "Value: " + gen("value", regex=r"-?[0-9]+(\.[0-9]+)?")--# Percentage (0-100)-lm += "Progress: " + gen("progress", regex=r"(100|[0-9]{1,2})")--# Range (1-5 stars)-lm += "Rating: " + gen("rating", regex=r"[1-5]") + " stars"-```--#### Text Constraints--```python-# Alphabetic only-lm += "Name: " + gen("name", regex=r"[A-Za-z]+")--# Alphabetic with spaces-lm += "Full Name: " + gen("full_name", regex=r"[A-Za-z ]+")--# Alphanumeric-lm += "Username: " + gen("username", regex=r"[A-Za-z0-9_]+")--# Capitalized words-lm += "Title: " + gen("title", regex=r"[A-Z][a-z]+( [A-Z][a-z]+)*")--# Lowercase only-lm += "Code: " + gen("code", regex=r"[a-z0-9-]+")--# Specific length-lm += "ID: " + gen("id", regex=r"[A-Z]{3}-[0-9]{6}") # e.g., "ABC-123456"-```--#### Date and Time Constraints--```python-# Date (YYYY-MM-DD)-lm += "Date: " + gen("date", regex=r"\d{4}-\d{2}-\d{2}")--# Date (MM/DD/YYYY)-lm += "Date: " + gen("date_us", regex=r"\d{2}/\d{2}/\d{4}")--# Time (HH:MM)-lm += "Time: " + gen("time", regex=r"\d{2}:\d{2}")--# Time (HH:MM:SS)-lm += "Time: " + gen("time_full", regex=r"\d{2}:\d{2}:\d{2}")--# ISO 8601 datetime-lm += "Timestamp: " + gen(- "timestamp",- regex=r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z"-)--# Year (YYYY)-lm += "Year: " + gen("year", regex=r"(19|20)\d{2}")--# Month name-lm += "Month: " + gen(- "month",- regex=r"(January|February|March|April|May|June|July|August|September|October|November|December)"-)-```--#### Contact Information--```python-lm += "Email: " + gen(- "email",- regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"-)--# Phone (US format)-lm += "Phone: " + gen("phone", regex=r"\d{3}-\d{3}-\d{4}")--# Phone (international format)-lm += "Phone: " + gen("phone_intl", regex=r"\+[0-9]{1,3}-[0-9]{1,14}")--# ZIP code (US)-lm += "ZIP: " + gen("zip", regex=r"\d{5}(-\d{4})?")--# Postal code (Canada)-lm += "Postal: " + gen("postal", regex=r"[A-Z]\d[A-Z] \d[A-Z]\d")--# URL-lm += "URL: " + gen(- "url",- regex=r"https?://[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}(/[a-zA-Z0-9._~:/?#\[\]@!$&'()*+,;=-]*)?"-)-```--### Advanced Patterns--#### JSON Field Constraints--```python-from guidance import models, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--# String field with quotes-lm += '"name": ' + gen("name", regex=r'"[A-Za-z ]+"')--# Numeric field (no quotes)-lm += '"age": ' + gen("age", regex=r"[0-9]+")--# Boolean field-lm += '"active": ' + gen("active", regex=r"(true|false)")--# Null field-lm += '"optional": ' + gen("optional", regex=r"(null|[0-9]+)")--# Array of strings-lm += '"tags": [' + gen(- "tags",- regex=r'"[a-z]+"(, "[a-z]+")*'-) + ']'--# Complete JSON object-lm += """{- "name": """ + gen("name", regex=r'"[A-Za-z ]+"') + """,- "age": """ + gen("age", regex=r"[0-9]+") + """,- "email": """ + gen(- "email",- regex=r'"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"'- ) + """-}"""-```--#### Code Patterns--```python-# Python variable name-lm += "Variable: " + gen("var", regex=r"[a-z_][a-z0-9_]*")--# Python function name-lm += "Function: " + gen("func", regex=r"[a-z_][a-z0-9_]*")--# Hex color code-lm += "Color: #" + gen("color", regex=r"[0-9A-Fa-f]{6}")--# UUID-lm += "UUID: " + gen(- "uuid",- regex=r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"-)--# Git commit hash (short)-lm += "Commit: " + gen("commit", regex=r"[0-9a-f]{7}")--# Semantic version-lm += "Version: " + gen("version", regex=r"[0-9]+\.[0-9]+\.[0-9]+")--# IP address (IPv4)-lm += "IP: " + gen(- "ip",- regex=r"((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"-)-```--#### Domain-Specific Patterns--```python-# Credit card number-lm += "Card: " + gen("card", regex=r"\d{4}-\d{4}-\d{4}-\d{4}")--# Social Security Number (US)-lm += "SSN: " + gen("ssn", regex=r"\d{3}-\d{2}-\d{4}")--# ISBN-13-lm += "ISBN: " + gen("isbn", regex=r"978-\d{1,5}-\d{1,7}-\d{1,7}-\d")--# License plate (US)-lm += "Plate: " + gen("plate", regex=r"[A-Z]{3}-\d{4}")--# Currency amount-lm += "Amount: $" + gen("amount", regex=r"[0-9]{1,3}(,[0-9]{3})*\.[0-9]{2}")--# Percentage with decimal-lm += "Rate: " + gen("rate", regex=r"[0-9]+\.[0-9]{1,2}%")-```--## Grammar-Based Generation--### JSON Grammar--```python-from guidance import models, gen, guidance--@guidance-def json_object(lm):- """Generate valid JSON object."""- lm += "{\n"-- # Name field (required)- lm += ' "name": ' + gen("name", regex=r'"[A-Za-z ]+"') + ",\n"-- # Age field (required)- lm += ' "age": ' + gen("age", regex=r"[0-9]+") + ",\n"-- # Email field (required)- lm += ' "email": ' + gen(- "email",- regex=r'"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"'- ) + ",\n"-- # Active field (required, boolean)- lm += ' "active": ' + gen("active", regex=r"(true|false)") + "\n"-- lm += "}"- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = json_object(lm)-print(lm) # Valid JSON guaranteed-```--### Nested JSON Grammar--```python-@guidance-def nested_json(lm):- """Generate nested JSON structure."""- lm += "{\n"-- # User object- lm += ' "user": {\n'- lm += ' "name": ' + gen("name", regex=r'"[A-Za-z ]+"') + ",\n"- lm += ' "age": ' + gen("age", regex=r"[0-9]+") + "\n"- lm += " },\n"-- # Address object- lm += ' "address": {\n'- lm += ' "street": ' + gen("street", regex=r'"[A-Za-z0-9 ]+"') + ",\n"- lm += ' "city": ' + gen("city", regex=r'"[A-Za-z ]+"') + ",\n"- lm += ' "zip": ' + gen("zip", regex=r'"\d{5}"') + "\n"- lm += " }\n"-- lm += "}"- return lm-```--### Array Grammar--```python-@guidance-def json_array(lm, count=3):- """Generate JSON array with fixed count."""- lm += "[\n"-- for i in range(count):- lm += " {\n"- lm += ' "id": ' + gen(f"id_{i}", regex=r"[0-9]+") + ",\n"- lm += ' "name": ' + gen(f"name_{i}", regex=r'"[A-Za-z ]+"') + "\n"- lm += " }"- if i < count - 1:- lm += ","- lm += "\n"-- lm += "]"- return lm-```--### XML Grammar--```python-@guidance-def xml_document(lm):- """Generate valid XML document."""- lm += '<?xml version="1.0"?>\n'- lm += "<person>\n"-- # Name element- lm += " <name>" + gen("name", regex=r"[A-Za-z ]+") + "</name>\n"-- # Age element- lm += " <age>" + gen("age", regex=r"[0-9]+") + "</age>\n"-- # Email element- lm += " <email>" + gen(- "email",- regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"- ) + "</email>\n"-- lm += "</person>"- return lm-```--### CSV Grammar--```python-@guidance-def csv_row(lm):- """Generate CSV row."""- lm += gen("name", regex=r"[A-Za-z ]+") + ","- lm += gen("age", regex=r"[0-9]+") + ","- lm += gen("email", regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")- return lm--@guidance-def csv_document(lm, rows=5):- """Generate complete CSV."""- # Header- lm += "Name,Age,Email\n"-- # Rows- for i in range(rows):- lm = csv_row(lm)- if i < rows - 1:- lm += "\n"-- return lm-```--## Token Healing--### How Token Healing Works--**Problem:** Tokenization creates unnatural boundaries.--```python-# Example without token healing-prompt = "The capital of France is "-# Tokenization: ["The", " capital", " of", " France", " is", " "]-# Model sees last token: " "-# First generated token might include leading space: " Paris"-# Result: "The capital of France is Paris" (double space)-```--**Solution:** Guidance backs up and regenerates the last token.--```python-from guidance import models, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--# Token healing enabled by default-lm += "The capital of France is " + gen("capital", max_tokens=5)--# Process:-# 1. Back up to token before " is "-# 2. Regenerate " is" + "capital" together-# 3. Result: "The capital of France is Paris" (correct)-```--### Token Healing Examples--#### Natural Continuations--```python-# Before token healing-lm += "The function name is get" + gen("rest")-# Might generate: "The function name is get User" (space before User)--# With token healing-lm += "The function name is get" + gen("rest")-# Generates: "The function name is getUser" (correct camelCase)-```--#### Code Generation--```python-# Function name completion-lm += "def calculate_" + gen("rest", stop="(")-# Token healing ensures smooth connection: "calculate_total"--# Variable name completion-lm += "my_" + gen("var_name", regex=r"[a-z_]+")-# Token healing ensures: "my_variable_name" (not "my_ variable_name")-```--#### Domain-Specific Terms--```python-# Medical terms-lm += "The patient has hyper" + gen("condition")-# Token healing helps: "hypertension" (not "hyper tension")--# Technical terms-lm += "Using micro" + gen("tech")-# Token healing helps: "microservices" (not "micro services")-```--### Disabling Token Healing--```python-# Disable token healing if needed (rare)-lm += gen("text", token_healing=False)-```--## Selection Constraints--### Basic Selection--```python-from guidance import models, select--lm = models.Anthropic("claude-sonnet-4-5-20250929")--# Simple selection-lm += "Status: " + select(["active", "inactive", "pending"], name="status")--# Boolean selection-lm += "Approved: " + select(["Yes", "No"], name="approved")--# Multiple choice-lm += "Answer: " + select(- ["A) Paris", "B) London", "C) Berlin", "D) Madrid"],- name="answer"-)-```--### Conditional Selection--```python-from guidance import models, select, gen, guidance--@guidance-def conditional_fields(lm):- """Generate fields conditionally based on type."""- lm += "Type: " + select(["person", "company"], name="type")-- if lm["type"] == "person":- lm += "\nName: " + gen("name", regex=r"[A-Za-z ]+")- lm += "\nAge: " + gen("age", regex=r"[0-9]+")- else:- lm += "\nCompany Name: " + gen("company", regex=r"[A-Za-z ]+")- lm += "\nEmployees: " + gen("employees", regex=r"[0-9]+")-- return lm-```--### Repeated Selection--```python-@guidance-def multiple_selections(lm):- """Select multiple items."""- lm += "Select 3 colors:\n"-- colors = ["red", "blue", "green", "yellow", "purple"]-- for i in range(3):- lm += f"{i+1}. " + select(colors, name=f"color_{i}") + "\n"-- return lm-```--## Complex Patterns--### Pattern 1: Structured Forms--```python-@guidance-def user_form(lm):- """Generate structured user form."""- lm += "=== User Registration ===\n\n"-- # Name (alphabetic only)- lm += "Full Name: " + gen("name", regex=r"[A-Za-z ]+", stop="\n") + "\n"-- # Age (numeric)- lm += "Age: " + gen("age", regex=r"[0-9]+", max_tokens=3) + "\n"-- # Email (validated format)- lm += "Email: " + gen(- "email",- regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}",- stop="\n"- ) + "\n"-- # Phone (US format)- lm += "Phone: " + gen("phone", regex=r"\d{3}-\d{3}-\d{4}") + "\n"-- # Account type (selection)- lm += "Account Type: " + select(- ["Standard", "Premium", "Enterprise"],- name="account_type"- ) + "\n"-- # Active status (boolean)- lm += "Active: " + select(["Yes", "No"], name="active") + "\n"-- return lm-```--### Pattern 2: Multi-Entity Extraction--```python-@guidance-def extract_entities(lm, text):- """Extract multiple entities with constraints."""- lm += f"Text: {text}\n\n"-- # Person name (alphabetic)- lm += "Person: " + gen("person", regex=r"[A-Za-z ]+", stop="\n") + "\n"-- # Organization (alphanumeric with spaces)- lm += "Organization: " + gen(- "organization",- regex=r"[A-Za-z0-9 ]+",- stop="\n"- ) + "\n"-- # Date (YYYY-MM-DD format)- lm += "Date: " + gen("date", regex=r"\d{4}-\d{2}-\d{2}") + "\n"-- # Location (alphabetic with spaces)- lm += "Location: " + gen("location", regex=r"[A-Za-z ]+", stop="\n") + "\n"-- # Amount (currency)- lm += "Amount: $" + gen("amount", regex=r"[0-9,]+\.[0-9]{2}") + "\n"-- return lm-```--### Pattern 3: Code Generation--```python-@guidance-def generate_python_function(lm):- """Generate Python function with constraints."""- # Function name (valid Python identifier)- lm += "def " + gen("func_name", regex=r"[a-z_][a-z0-9_]*") + "("-- # Parameter name- lm += gen("param", regex=r"[a-z_][a-z0-9_]*") + "):\n"-- # Docstring- lm += ' """' + gen("docstring", stop='"""', max_tokens=50) + '"""\n'-- # Function body (constrained to valid Python)- lm += " return " + gen("return_value", stop="\n") + "\n"-- return lm-```--### Pattern 4: Hierarchical Data--```python-@guidance-def org_chart(lm):- """Generate organizational chart."""- lm += "Company: " + gen("company", regex=r"[A-Za-z ]+") + "\n\n"-- # CEO- lm += "CEO: " + gen("ceo", regex=r"[A-Za-z ]+") + "\n"-- # Departments- for dept in ["Engineering", "Sales", "Marketing"]:- lm += f"\n{dept} Department:\n"- lm += " Head: " + gen(f"{dept.lower()}_head", regex=r"[A-Za-z ]+") + "\n"- lm += " Size: " + gen(f"{dept.lower()}_size", regex=r"[0-9]+") + " employees\n"-- return lm-```--## Performance Optimization--### Best Practices--#### 1. Use Specific Patterns--```python-# ✅ Good: Specific pattern-lm += gen("age", regex=r"[0-9]{1,3}") # Fast--# ❌ Bad: Overly broad pattern-lm += gen("age", regex=r"[0-9]+") # Slower-```--#### 2. Limit Max Tokens--```python-# ✅ Good: Reasonable limit-lm += gen("name", max_tokens=30)--# ❌ Bad: No limit-lm += gen("name") # May generate forever-```--#### 3. Use stop Sequences--```python-# ✅ Good: Stop at newline-lm += gen("line", stop="\n")--# ❌ Bad: Rely on max_tokens-lm += gen("line", max_tokens=100)-```--#### 4. Cache Compiled Grammars--```python-# Grammars are cached automatically after first use-# No manual caching needed-@guidance-def reusable_pattern(lm):- """This grammar is compiled once and cached."""- lm += gen("email", regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")- return lm--# First call: compiles grammar-lm = reusable_pattern(lm)--# Subsequent calls: uses cached grammar (fast)-lm = reusable_pattern(lm)-```--#### 5. Avoid Overlapping Constraints--```python-# ✅ Good: Clear constraints-lm += gen("age", regex=r"[0-9]+", max_tokens=3)--# ❌ Bad: Conflicting constraints-lm += gen("age", regex=r"[0-9]{2}", max_tokens=10) # max_tokens unnecessary-```--### Performance Benchmarks--**Regex vs Free Generation:**-- Simple regex (digits): ~1.2x slower than free gen-- Complex regex (email): ~1.5x slower than free gen-- Grammar-based: ~2x slower than free gen--**But:**-- 100% valid outputs (vs ~70% with free gen + validation)-- No retry loops needed-- Overall faster end-to-end for structured outputs--**Optimization Tips:**-- Use regex for critical fields only-- Use `select()` for small fixed sets (fastest)-- Use `stop` sequences when possible (faster than max_tokens)-- Cache compiled grammars by reusing functions--## Resources--- **Token Healing Paper**: https://arxiv.org/abs/2306.17648-- **Guidance Docs**: https://guidance.readthedocs.io-- **GitHub**: https://github.com/guidance-ai/guidancediff --git a/optional-skills/mlops/guidance/references/examples.md b/optional-skills/mlops/guidance/references/examples.mddeleted file mode 100644index 31538874828..00000000000--- a/optional-skills/mlops/guidance/references/examples.md+++ /dev/null@@ -1,767 +0,0 @@-# Production-Ready Examples--Real-world examples of using Guidance for structured generation, agents, and workflows.--## Table of Contents-- JSON Generation-- Data Extraction-- Classification Systems-- Agent Systems-- Multi-Step Workflows-- Code Generation-- Production Tips--## JSON Generation--### Basic JSON--```python-from guidance import models, gen, guidance--@guidance-def generate_user(lm):- """Generate valid user JSON."""- lm += "{\n"- lm += ' "name": ' + gen("name", regex=r'"[A-Za-z ]+"') + ",\n"- lm += ' "age": ' + gen("age", regex=r"[0-9]+") + ",\n"- lm += ' "email": ' + gen(- "email",- regex=r'"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"'- ) + "\n"- lm += "}"- return lm--# Use it-lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm += "Generate a user profile:\n"-lm = generate_user(lm)--print(lm)-# Output: Valid JSON guaranteed-```--### Nested JSON--```python-@guidance-def generate_order(lm):- """Generate nested order JSON."""- lm += "{\n"-- # Customer info- lm += ' "customer": {\n'- lm += ' "name": ' + gen("customer_name", regex=r'"[A-Za-z ]+"') + ",\n"- lm += ' "email": ' + gen(- "customer_email",- regex=r'"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"'- ) + "\n"- lm += " },\n"-- # Order details- lm += ' "order": {\n'- lm += ' "id": ' + gen("order_id", regex=r'"ORD-[0-9]{6}"') + ",\n"- lm += ' "date": ' + gen("order_date", regex=r'"\d{4}-\d{2}-\d{2}"') + ",\n"- lm += ' "total": ' + gen("order_total", regex=r"[0-9]+\.[0-9]{2}") + "\n"- lm += " },\n"-- # Status- lm += ' "status": ' + gen(- "status",- regex=r'"(pending|processing|shipped|delivered)"'- ) + "\n"-- lm += "}"- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = generate_order(lm)-```--### JSON Array--```python-@guidance-def generate_user_list(lm, count=3):- """Generate JSON array of users."""- lm += "[\n"-- for i in range(count):- lm += " {\n"- lm += ' "id": ' + gen(f"id_{i}", regex=r"[0-9]+") + ",\n"- lm += ' "name": ' + gen(f"name_{i}", regex=r'"[A-Za-z ]+"') + ",\n"- lm += ' "active": ' + gen(f"active_{i}", regex=r"(true|false)") + "\n"- lm += " }"- if i < count - 1:- lm += ","- lm += "\n"-- lm += "]"- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = generate_user_list(lm, count=5)-```--### Dynamic JSON Schema--```python-import json-from guidance import models, gen, guidance--@guidance-def json_from_schema(lm, schema):- """Generate JSON matching a schema."""- lm += "{\n"-- fields = list(schema["properties"].items())- for i, (field_name, field_schema) in enumerate(fields):- lm += f' "{field_name}": '-- # Handle different types- if field_schema["type"] == "string":- if "pattern" in field_schema:- lm += gen(field_name, regex=f'"{field_schema["pattern"]}"')- else:- lm += gen(field_name, regex=r'"[^"]+"')- elif field_schema["type"] == "number":- lm += gen(field_name, regex=r"[0-9]+(\.[0-9]+)?")- elif field_schema["type"] == "integer":- lm += gen(field_name, regex=r"[0-9]+")- elif field_schema["type"] == "boolean":- lm += gen(field_name, regex=r"(true|false)")-- if i < len(fields) - 1:- lm += ","- lm += "\n"-- lm += "}"- return lm--# Define schema-schema = {- "type": "object",- "properties": {- "name": {"type": "string"},- "age": {"type": "integer"},- "score": {"type": "number"},- "active": {"type": "boolean"}- }-}--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = json_from_schema(lm, schema)-```--## Data Extraction--### Extract from Text--```python-from guidance import models, gen, guidance, system, user, assistant--@guidance-def extract_person_info(lm, text):- """Extract structured info from text."""- lm += f"Text: {text}\n\n"-- with assistant():- lm += "Name: " + gen("name", regex=r"[A-Za-z ]+", stop="\n") + "\n"- lm += "Age: " + gen("age", regex=r"[0-9]+", max_tokens=3) + "\n"- lm += "Occupation: " + gen("occupation", regex=r"[A-Za-z ]+", stop="\n") + "\n"- lm += "Email: " + gen(- "email",- regex=r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}",- stop="\n"- ) + "\n"-- return lm--text = "John Smith is a 35-year-old software engineer. Contact: [email protected]"--lm = models.Anthropic("claude-sonnet-4-5-20250929")--with system():- lm += "You extract structured information from text."--with user():- lm = extract_person_info(lm, text)--print(f"Name: {lm['name']}")-print(f"Age: {lm['age']}")-print(f"Occupation: {lm['occupation']}")-print(f"Email: {lm['email']}")-```--### Multi-Entity Extraction--```python-@guidance-def extract_entities(lm, text):- """Extract multiple entity types."""- lm += f"Analyze: {text}\n\n"-- # Person entities- lm += "People:\n"- for i in range(3): # Up to 3 people- lm += f"- " + gen(f"person_{i}", regex=r"[A-Za-z ]+", stop="\n") + "\n"-- # Organization entities- lm += "\nOrganizations:\n"- for i in range(2): # Up to 2 orgs- lm += f"- " + gen(f"org_{i}", regex=r"[A-Za-z0-9 ]+", stop="\n") + "\n"-- # Dates- lm += "\nDates:\n"- for i in range(2): # Up to 2 dates- lm += f"- " + gen(f"date_{i}", regex=r"\d{4}-\d{2}-\d{2}", stop="\n") + "\n"-- # Locations- lm += "\nLocations:\n"- for i in range(2): # Up to 2 locations- lm += f"- " + gen(f"location_{i}", regex=r"[A-Za-z ]+", stop="\n") + "\n"-- return lm--text = """-Tim Cook and Satya Nadella met at Microsoft headquarters in Redmond on 2024-09-15-to discuss the collaboration between Apple and Microsoft. The meeting continued-in Cupertino on 2024-09-20.-"""--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = extract_entities(lm, text)-```--### Batch Extraction--```python-@guidance-def batch_extract(lm, texts):- """Extract from multiple texts."""- lm += "Batch Extraction Results:\n\n"-- for i, text in enumerate(texts):- lm += f"=== Item {i+1} ===\n"- lm += f"Text: {text}\n"- lm += "Name: " + gen(f"name_{i}", regex=r"[A-Za-z ]+", stop="\n") + "\n"- lm += "Sentiment: " + gen(- f"sentiment_{i}",- regex=r"(positive|negative|neutral)",- stop="\n"- ) + "\n\n"-- return lm--texts = [- "Alice is happy with the product",- "Bob is disappointed with the service",- "Carol has no strong feelings either way"-]--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = batch_extract(lm, texts)-```--## Classification Systems--### Sentiment Analysis--```python-from guidance import models, select, gen--lm = models.Anthropic("claude-sonnet-4-5-20250929")--text = "This product is absolutely amazing! Best purchase ever."--lm += f"Text: {text}\n\n"-lm += "Sentiment: " + select(- ["positive", "negative", "neutral"],- name="sentiment"-)-lm += "\nConfidence: " + gen("confidence", regex=r"[0-9]{1,3}") + "%\n"-lm += "Reasoning: " + gen("reasoning", stop="\n", max_tokens=50)--print(f"Sentiment: {lm['sentiment']}")-print(f"Confidence: {lm['confidence']}%")-print(f"Reasoning: {lm['reasoning']}")-```--### Multi-Label Classification--```python-@guidance-def classify_article(lm, text):- """Classify article with multiple labels."""- lm += f"Article: {text}\n\n"-- # Primary category- lm += "Primary Category: " + select(- ["Technology", "Business", "Science", "Politics", "Entertainment"],- name="primary_category"- ) + "\n"-- # Secondary categories (up to 3)- lm += "\nSecondary Categories:\n"- categories = ["Technology", "Business", "Science", "Politics", "Entertainment"]- for i in range(3):- lm += f"{i+1}. " + select(categories, name=f"secondary_{i}") + "\n"-- # Tags- lm += "\nTags: " + gen("tags", stop="\n", max_tokens=50) + "\n"-- # Target audience- lm += "Target Audience: " + select(- ["General", "Expert", "Beginner"],- name="audience"- )-- return lm--article = """-Apple announced new AI features in iOS 18, leveraging machine learning to improve-battery life and performance. The company's stock rose 5% following the announcement.-"""--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = classify_article(lm, article)-```--### Intent Classification--```python-@guidance-def classify_intent(lm, message):- """Classify user intent."""- lm += f"User Message: {message}\n\n"-- # Intent- lm += "Intent: " + select(- ["question", "complaint", "request", "feedback", "other"],- name="intent"- ) + "\n"-- # Urgency- lm += "Urgency: " + select(- ["low", "medium", "high", "critical"],- name="urgency"- ) + "\n"-- # Department- lm += "Route To: " + select(- ["support", "sales", "billing", "technical"],- name="department"- ) + "\n"-- # Sentiment- lm += "Sentiment: " + select(- ["positive", "neutral", "negative"],- name="sentiment"- )-- return lm--message = "My account was charged twice for the same order. Need help ASAP!"--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = classify_intent(lm, message)--print(f"Intent: {lm['intent']}")-print(f"Urgency: {lm['urgency']}")-print(f"Department: {lm['department']}")-```--## Agent Systems--### ReAct Agent--```python-from guidance import models, gen, select, guidance--@guidance(stateless=False)-def react_agent(lm, question, tools, max_rounds=5):- """ReAct agent with tool use."""- lm += f"Question: {question}\n\n"-- for round in range(max_rounds):- # Thought- lm += f"Thought {round+1}: " + gen("thought", stop="\n", max_tokens=100) + "\n"-- # Action selection- lm += "Action: " + select(- list(tools.keys()) + ["answer"],- name="action"- )-- if lm["action"] == "answer":- lm += "\n\nFinal Answer: " + gen("answer", max_tokens=200)- break-- # Action input- lm += "\nAction Input: " + gen("action_input", stop="\n", max_tokens=100) + "\n"-- # Execute tool- if lm["action"] in tools:- try:- result = tools[lm["action"]](lm["action_input"])- lm += f"Observation: {result}\n\n"- except Exception as e:- lm += f"Observation: Error - {str(e)}\n\n"-- return lm--# Define tools-tools = {- "calculator": lambda expr: eval(expr),- "search": lambda query: f"Search results for '{query}': [Mock results]",- "weather": lambda city: f"Weather in {city}: Sunny, 72°F"-}--# Use agent-lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = react_agent(lm, "What is (25 * 4) + 10?", tools)--print(lm["answer"])-```--### Multi-Agent System--```python-@guidance-def coordinator_agent(lm, task):- """Coordinator that delegates to specialists."""- lm += f"Task: {task}\n\n"-- # Determine which specialist to use- lm += "Specialist: " + select(- ["researcher", "writer", "coder", "analyst"],- name="specialist"- ) + "\n"-- lm += "Reasoning: " + gen("reasoning", stop="\n", max_tokens=100) + "\n"-- return lm--@guidance-def researcher_agent(lm, query):- """Research specialist."""- lm += f"Research Query: {query}\n\n"- lm += "Findings:\n"- for i in range(3):- lm += f"{i+1}. " + gen(f"finding_{i}", stop="\n", max_tokens=100) + "\n"- return lm--@guidance-def writer_agent(lm, topic):- """Writing specialist."""- lm += f"Topic: {topic}\n\n"- lm += "Title: " + gen("title", stop="\n", max_tokens=50) + "\n"- lm += "Content:\n" + gen("content", max_tokens=500)- return lm--# Coordination workflow-task = "Write an article about AI safety"--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = coordinator_agent(lm, task)--specialist = lm["specialist"]-if specialist == "researcher":- lm = researcher_agent(lm, task)-elif specialist == "writer":- lm = writer_agent(lm, task)-```--### Tool Use with Validation--```python-@guidance(stateless=False)-def validated_tool_agent(lm, question):- """Agent with validated tool calls."""- tools = {- "add": lambda a, b: float(a) + float(b),- "multiply": lambda a, b: float(a) * float(b),- "divide": lambda a, b: float(a) / float(b) if float(b) != 0 else "Error: Division by zero"- }-- lm += f"Question: {question}\n\n"-- for i in range(5):- # Select tool- lm += "Tool: " + select(list(tools.keys()) + ["done"], name="tool")-- if lm["tool"] == "done":- lm += "\nAnswer: " + gen("answer", max_tokens=100)- break-- # Get validated numeric arguments- lm += "\nArg1: " + gen("arg1", regex=r"-?[0-9]+(\.[0-9]+)?") + "\n"- lm += "Arg2: " + gen("arg2", regex=r"-?[0-9]+(\.[0-9]+)?") + "\n"-- # Execute- result = tools[lm["tool"]](lm["arg1"], lm["arg2"])- lm += f"Result: {result}\n\n"-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = validated_tool_agent(lm, "What is (10 + 5) * 3?")-```--## Multi-Step Workflows--### Chain of Thought--```python-@guidance-def chain_of_thought(lm, question):- """Multi-step reasoning with CoT."""- lm += f"Question: {question}\n\n"-- # Generate reasoning steps- lm += "Let me think step by step:\n\n"- for i in range(4):- lm += f"Step {i+1}: " + gen(f"step_{i+1}", stop="\n", max_tokens=100) + "\n"-- # Final answer- lm += "\nTherefore, the answer is: " + gen("answer", stop="\n", max_tokens=50)-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = chain_of_thought(lm, "If a train travels 60 mph for 2.5 hours, how far does it go?")--print(lm["answer"])-```--### Self-Consistency--```python-@guidance-def self_consistency(lm, question, num_samples=3):- """Generate multiple reasoning paths and aggregate."""- lm += f"Question: {question}\n\n"-- answers = []- for i in range(num_samples):- lm += f"=== Attempt {i+1} ===\n"- lm += "Reasoning: " + gen(f"reasoning_{i}", stop="\n", max_tokens=100) + "\n"- lm += "Answer: " + gen(f"answer_{i}", stop="\n", max_tokens=50) + "\n\n"- answers.append(lm[f"answer_{i}"])-- # Aggregate (simple majority vote)- from collections import Counter- most_common = Counter(answers).most_common(1)[0][0]-- lm += f"Final Answer (by majority): {most_common}\n"- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = self_consistency(lm, "What is 15% of 200?")-```--### Planning and Execution--```python-@guidance-def plan_and_execute(lm, goal):- """Plan tasks then execute them."""- lm += f"Goal: {goal}\n\n"-- # Planning phase- lm += "Plan:\n"- num_steps = 4- for i in range(num_steps):- lm += f"{i+1}. " + gen(f"plan_step_{i}", stop="\n", max_tokens=100) + "\n"-- # Execution phase- lm += "\nExecution:\n\n"- for i in range(num_steps):- lm += f"Step {i+1}: {lm[f'plan_step_{i}']}\n"- lm += "Status: " + select(["completed", "in-progress", "blocked"], name=f"status_{i}") + "\n"- lm += "Result: " + gen(f"result_{i}", stop="\n", max_tokens=150) + "\n\n"-- # Summary- lm += "Summary: " + gen("summary", max_tokens=200)-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = plan_and_execute(lm, "Build a REST API for a blog platform")-```--## Code Generation--### Python Function--```python-@guidance-def generate_python_function(lm, description):- """Generate Python function from description."""- lm += f"Description: {description}\n\n"-- # Function signature- lm += "def " + gen("func_name", regex=r"[a-z_][a-z0-9_]*") + "("- lm += gen("params", regex=r"[a-z_][a-z0-9_]*(, [a-z_][a-z0-9_]*)*") + "):\n"-- # Docstring- lm += ' """' + gen("docstring", stop='"""', max_tokens=100) + '"""\n'-- # Function body- lm += " " + gen("body", stop="\n", max_tokens=200) + "\n"-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = generate_python_function(lm, "Check if a number is prime")--print(lm)-```--### SQL Query--```python-@guidance-def generate_sql(lm, description):- """Generate SQL query from description."""- lm += f"Description: {description}\n\n"- lm += "SQL Query:\n"-- # SELECT clause- lm += "SELECT " + gen("select_clause", stop=" FROM", max_tokens=100)-- # FROM clause- lm += " FROM " + gen("from_clause", stop=" WHERE", max_tokens=50)-- # WHERE clause (optional)- lm += " WHERE " + gen("where_clause", stop=";", max_tokens=100) + ";"-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = generate_sql(lm, "Get all users who signed up in the last 30 days")-```--### API Endpoint--```python-@guidance-def generate_api_endpoint(lm, description):- """Generate REST API endpoint."""- lm += f"Description: {description}\n\n"-- # HTTP method- lm += "Method: " + select(["GET", "POST", "PUT", "DELETE"], name="method") + "\n"-- # Path- lm += "Path: /" + gen("path", regex=r"[a-z0-9/-]+", stop="\n") + "\n"-- # Request body (if POST/PUT)- if lm["method"] in ["POST", "PUT"]:- lm += "\nRequest Body:\n"- lm += "{\n"- lm += ' "field1": ' + gen("field1", regex=r'"[a-z_]+"') + ",\n"- lm += ' "field2": ' + gen("field2", regex=r'"[a-z_]+"') + "\n"- lm += "}\n"-- # Response- lm += "\nResponse (200 OK):\n"- lm += "{\n"- lm += ' "status": "success",\n'- lm += ' "data": ' + gen("response_data", max_tokens=100) + "\n"- lm += "}\n"-- return lm--lm = models.Anthropic("claude-sonnet-4-5-20250929")-lm = generate_api_endpoint(lm, "Create a new blog post")-```--## Production Tips--### Error Handling--```python-@guidance-def safe_extraction(lm, text):- """Extract with fallback handling."""- try:- lm += f"Text: {text}\n"- lm += "Name: " + gen("name", regex=r"[A-Za-z ]+", stop="\n", max_tokens=30)- return lm- except Exception as e:- # Fallback to less strict extraction- lm += f"Text: {text}\n"- lm += "Name: " + gen("name", stop="\n", max_tokens=30)- return lm-```--### Caching--```python-from functools import lru_cache--@lru_cache(maxsize=100)-def cached_generation(text):- """Cache LLM generations."""- lm = models.Anthropic("claude-sonnet-4-5-20250929")- lm += f"Analyze: {text}\n"- lm += "Sentiment: " + select(["positive", "negative", "neutral"], name="sentiment")- return lm["sentiment"]--# First call: hits LLM-result1 = cached_generation("This is great!")--# Second call: returns cached result-result2 = cached_generation("This is great!") # Instant!-```--### Monitoring--```python-import time--@guidance-def monitored_generation(lm, text):- """Track generation metrics."""- start_time = time.time()-- lm += f"Text: {text}\n"- lm += "Analysis: " + gen("analysis", max_tokens=100)-- elapsed = time.time() - start_time-- # Log metrics- print(f"Generation time: {elapsed:.2f}s")- print(f"Output length: {len(lm['analysis'])} chars")-- return lm-```--### Batch Processing--```python-def batch_process(texts, batch_size=10):- """Process texts in batches."""- lm = models.Anthropic("claude-sonnet-4-5-20250929")- results = []-- for i in range(0, len(texts), batch_size):- batch = texts[i:i+batch_size]-- for text in batch:- lm += f"Text: {text}\n"- lm += "Sentiment: " + select(- ["positive", "negative", "neutral"],- name=f"sentiment_{i}"- ) + "\n\n"-- results.extend([lm[f"sentiment_{i}"] for i in range(len(batch))])-- return results-```--## Resources--- **Guidance Notebooks**: https://github.com/guidance-ai/guidance/tree/main/notebooks-- **Guidance Docs**: https://guidance.readthedocs.io-- **Community Examples**: https://github.com/guidance-ai/guidance/discussionsdiff --git a/optional-skills/mlops/huggingface-tokenizers/SKILL.md b/optional-skills/mlops/huggingface-tokenizers/SKILL.mddeleted file mode 100644index 776b5253e7e..00000000000--- a/optional-skills/mlops/huggingface-tokenizers/SKILL.md+++ /dev/null@@ -1,520 +0,0 @@-----name: huggingface-tokenizers-description: Fast BPE/WordPiece tokenization and custom vocab training.-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [tokenizers, transformers, datasets]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Tokenization, HuggingFace, BPE, WordPiece, Unigram, Fast Tokenization, Rust, Custom Tokenizer, Alignment Tracking, Production]-------# HuggingFace Tokenizers - Fast Tokenization for NLP--Fast, production-ready tokenizers with Rust performance and Python ease-of-use.--## When to use HuggingFace Tokenizers--**Use HuggingFace Tokenizers when:**-- Need extremely fast tokenization (<20s per GB of text)-- Training custom tokenizers from scratch-- Want alignment tracking (token → original text position)-- Building production NLP pipelines-- Need to tokenize large corpora efficiently--**Performance**:-- **Speed**: <20 seconds to tokenize 1GB on CPU-- **Implementation**: Rust core with Python/Node.js bindings-- **Efficiency**: 10-100× faster than pure Python implementations--**Use alternatives instead**:-- **SentencePiece**: Language-independent, used by T5/ALBERT-- **tiktoken**: OpenAI's BPE tokenizer for GPT models-- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)--## Quick start--### Installation--```bash-# Install tokenizers-pip install tokenizers--# With transformers integration-pip install tokenizers transformers-```--### Load pretrained tokenizer--```python-from tokenizers import Tokenizer--# Load from HuggingFace Hub-tokenizer = Tokenizer.from_pretrained("bert-base-uncased")--# Encode text-output = tokenizer.encode("Hello, how are you?")-print(output.tokens) # ['hello', ',', 'how', 'are', 'you', '?']-print(output.ids) # [7592, 1010, 2129, 2024, 2017, 1029]--# Decode back-text = tokenizer.decode(output.ids)-print(text) # "hello, how are you?"-```--### Train custom BPE tokenizer--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from tokenizers.pre_tokenizers import Whitespace--# Initialize tokenizer with BPE model-tokenizer = Tokenizer(BPE(unk_token="[UNK]"))-tokenizer.pre_tokenizer = Whitespace()--# Configure trainer-trainer = BpeTrainer(- vocab_size=30000,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],- min_frequency=2-)--# Train on files-files = ["train.txt", "validation.txt"]-tokenizer.train(files, trainer)--# Save-tokenizer.save("my-tokenizer.json")-```--**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB--### Batch encoding with padding--```python-# Enable padding-tokenizer.enable_padding(pad_id=3, pad_token="[PAD]")--# Encode batch-texts = ["Hello world", "This is a longer sentence"]-encodings = tokenizer.encode_batch(texts)--for encoding in encodings:- print(encoding.ids)-# [101, 7592, 2088, 102, 3, 3, 3]-# [101, 2023, 2003, 1037, 2936, 6251, 102]-```--## Tokenization algorithms--### BPE (Byte-Pair Encoding)--**How it works**:-1. Start with character-level vocabulary-2. Find most frequent character pair-3. Merge into new token, add to vocabulary-4. Repeat until vocabulary size reached--**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from tokenizers.pre_tokenizers import ByteLevel--tokenizer = Tokenizer(BPE(unk_token="<|endoftext|>"))-tokenizer.pre_tokenizer = ByteLevel()--trainer = BpeTrainer(- vocab_size=50257,- special_tokens=["<|endoftext|>"],- min_frequency=2-)--tokenizer.train(files=["data.txt"], trainer=trainer)-```--**Advantages**:-- Handles OOV words well (breaks into subwords)-- Flexible vocabulary size-- Good for morphologically rich languages--**Trade-offs**:-- Tokenization depends on merge order-- May split common words unexpectedly--### WordPiece--**How it works**:-1. Start with character vocabulary-2. Score merge pairs: `frequency(pair) / (frequency(first) × frequency(second))`-3. Merge highest scoring pair-4. Repeat until vocabulary size reached--**Used by**: BERT, DistilBERT, MobileBERT--```python-from tokenizers import Tokenizer-from tokenizers.models import WordPiece-from tokenizers.trainers import WordPieceTrainer-from tokenizers.pre_tokenizers import Whitespace-from tokenizers.normalizers import BertNormalizer--tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))-tokenizer.normalizer = BertNormalizer(lowercase=True)-tokenizer.pre_tokenizer = Whitespace()--trainer = WordPieceTrainer(- vocab_size=30522,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],- continuing_subword_prefix="##"-)--tokenizer.train(files=["corpus.txt"], trainer=trainer)-```--**Advantages**:-- Prioritizes meaningful merges (high score = semantically related)-- Used successfully in BERT (state-of-the-art results)--**Trade-offs**:-- Unknown words become `[UNK]` if no subword match-- Saves vocabulary, not merge rules (larger files)--### Unigram--**How it works**:-1. Start with large vocabulary (all substrings)-2. Compute loss for corpus with current vocabulary-3. Remove tokens with minimal impact on loss-4. Repeat until vocabulary size reached--**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)--```python-from tokenizers import Tokenizer-from tokenizers.models import Unigram-from tokenizers.trainers import UnigramTrainer--tokenizer = Tokenizer(Unigram())--trainer = UnigramTrainer(- vocab_size=8000,- special_tokens=["<unk>", "<s>", "</s>"],- unk_token="<unk>"-)--tokenizer.train(files=["data.txt"], trainer=trainer)-```--**Advantages**:-- Probabilistic (finds most likely tokenization)-- Works well for languages without word boundaries-- Handles diverse linguistic contexts--**Trade-offs**:-- Computationally expensive to train-- More hyperparameters to tune--## Tokenization pipeline--Complete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**--### Normalization--Clean and standardize text:--```python-from tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence--tokenizer.normalizer = Sequence([- NFD(), # Unicode normalization (decompose)- Lowercase(), # Convert to lowercase- StripAccents() # Remove accents-])--# Input: "Héllo WORLD"-# After normalization: "hello world"-```--**Common normalizers**:-- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms-- `Lowercase()` - Convert to lowercase-- `StripAccents()` - Remove accents (é → e)-- `Strip()` - Remove whitespace-- `Replace(pattern, content)` - Regex replacement--### Pre-tokenization--Split text into word-like units:--```python-from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel--# Split on whitespace and punctuation-tokenizer.pre_tokenizer = Sequence([- Whitespace(),- Punctuation()-])--# Input: "Hello, world!"-# After pre-tokenization: ["Hello", ",", "world", "!"]-```--**Common pre-tokenizers**:-- `Whitespace()` - Split on spaces, tabs, newlines-- `ByteLevel()` - GPT-2 style byte-level splitting-- `Punctuation()` - Isolate punctuation-- `Digits(individual_digits=True)` - Split digits individually-- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)--### Post-processing--Add special tokens for model input:--```python-from tokenizers.processors import TemplateProcessing--# BERT-style: [CLS] sentence [SEP]-tokenizer.post_processor = TemplateProcessing(- single="[CLS] $A [SEP]",- pair="[CLS] $A [SEP] $B [SEP]",- special_tokens=[- ("[CLS]", 1),- ("[SEP]", 2),- ],-)-```--**Common patterns**:-```python-# GPT-2: sentence <|endoftext|>-TemplateProcessing(- single="$A <|endoftext|>",- special_tokens=[("<|endoftext|>", 50256)]-)--# RoBERTa: <s> sentence </s>-TemplateProcessing(- single="<s> $A </s>",- pair="<s> $A </s> </s> $B </s>",- special_tokens=[("<s>", 0), ("</s>", 2)]-)-```--## Alignment tracking--Track token positions in original text:--```python-output = tokenizer.encode("Hello, world!")--# Get token offsets-for token, offset in zip(output.tokens, output.offsets):- start, end = offset- print(f"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}")--# Output:-# hello → [ 0, 5): 'Hello'-# , → [ 5, 6): ','-# world → [ 7, 12): 'world'-# ! → [12, 13): '!'-```--**Use cases**:-- Named entity recognition (map predictions back to text)-- Question answering (extract answer spans)-- Token classification (align labels to original positions)--## Integration with transformers--### Load with AutoTokenizer--```python-from transformers import AutoTokenizer--# AutoTokenizer automatically uses fast tokenizers-tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")--# Check if using fast tokenizer-print(tokenizer.is_fast) # True--# Access underlying tokenizers.Tokenizer-fast_tokenizer = tokenizer.backend_tokenizer-print(type(fast_tokenizer)) # <class 'tokenizers.Tokenizer'>-```--### Convert custom tokenizer to transformers--```python-from tokenizers import Tokenizer-from transformers import PreTrainedTokenizerFast--# Train custom tokenizer-tokenizer = Tokenizer(BPE())-# ... train tokenizer ...-tokenizer.save("my-tokenizer.json")--# Wrap for transformers-transformers_tokenizer = PreTrainedTokenizerFast(- tokenizer_file="my-tokenizer.json",- unk_token="[UNK]",- pad_token="[PAD]",- cls_token="[CLS]",- sep_token="[SEP]",- mask_token="[MASK]"-)--# Use like any transformers tokenizer-outputs = transformers_tokenizer(- "Hello world",- padding=True,- truncation=True,- max_length=512,- return_tensors="pt"-)-```--## Common patterns--### Train from iterator (large datasets)--```python-from datasets import load_dataset--# Load dataset-dataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")--# Create batch iterator-def batch_iterator(batch_size=1000):- for i in range(0, len(dataset), batch_size):- yield dataset[i:i + batch_size]["text"]--# Train tokenizer-tokenizer.train_from_iterator(- batch_iterator(),- trainer=trainer,- length=len(dataset) # For progress bar-)-```--**Performance**: Processes 1GB in ~10-20 minutes--### Enable truncation and padding--```python-# Enable truncation-tokenizer.enable_truncation(max_length=512)--# Enable padding-tokenizer.enable_padding(- pad_id=tokenizer.token_to_id("[PAD]"),- pad_token="[PAD]",- length=512 # Fixed length, or None for batch max-)--# Encode with both-output = tokenizer.encode("This is a long sentence that will be truncated...")-print(len(output.ids)) # 512-```--### Multi-processing--```python-from tokenizers import Tokenizer-from multiprocessing import Pool--# Load tokenizer-tokenizer = Tokenizer.from_file("tokenizer.json")--def encode_batch(texts):- return tokenizer.encode_batch(texts)--# Process large corpus in parallel-with Pool(8) as pool:- # Split corpus into chunks- chunk_size = 1000- chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]-- # Encode in parallel- results = pool.map(encode_batch, chunks)-```--**Speedup**: 5-8× with 8 cores--## Performance benchmarks--### Training speed--| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |-|-------------|-----------------|-----------------|--------------|-| 10 MB | 15 sec | 18 sec | 25 sec |-| 100 MB | 1.5 min | 2 min | 4 min |-| 1 GB | 15 min | 20 min | 40 min |--**Hardware**: 16-core CPU, tested on English Wikipedia--### Tokenization speed--| Implementation | 1 GB corpus | Throughput |-|----------------|-------------|---------------|-| Pure Python | ~20 minutes | ~50 MB/min |-| HF Tokenizers | ~15 seconds | ~4 GB/min |-| **Speedup** | **80×** | **80×** |--**Test**: English text, average sentence length 20 words--### Memory usage--| Task | Memory |-|-------------------------|---------|-| Load tokenizer | ~10 MB |-| Train BPE (30k vocab) | ~200 MB |-| Encode 1M sentences | ~500 MB |--## Supported models--Pre-trained tokenizers available via `from_pretrained()`:--**BERT family**:-- `bert-base-uncased`, `bert-large-cased`-- `distilbert-base-uncased`-- `roberta-base`, `roberta-large`--**GPT family**:-- `gpt2`, `gpt2-medium`, `gpt2-large`-- `distilgpt2`--**T5 family**:-- `t5-small`, `t5-base`, `t5-large`-- `google/flan-t5-xxl`--**Other**:-- `facebook/bart-base`, `facebook/mbart-large-cc25`-- `albert-base-v2`, `albert-xlarge-v2`-- `xlm-roberta-base`, `xlm-roberta-large`--Browse all: https://huggingface.co/models?library=tokenizers--## References--- **[Training Guide](references/training.md)** - Train custom tokenizers, configure trainers, handle large datasets-- **[Algorithms Deep Dive](references/algorithms.md)** - BPE, WordPiece, Unigram explained in detail-- **[Pipeline Components](references/pipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders-- **[Transformers Integration](references/integration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens--## Resources--- **Docs**: https://huggingface.co/docs/tokenizers-- **GitHub**: https://github.com/huggingface/tokenizers ⭐ 9,000+-- **Version**: 0.20.0+-- **Course**: https://huggingface.co/learn/nlp-course/chapter6/1-- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)--diff --git a/optional-skills/mlops/huggingface-tokenizers/references/algorithms.md b/optional-skills/mlops/huggingface-tokenizers/references/algorithms.mddeleted file mode 100644index 745bcd909f0..00000000000--- a/optional-skills/mlops/huggingface-tokenizers/references/algorithms.md+++ /dev/null@@ -1,653 +0,0 @@-# Tokenization Algorithms Deep Dive--Comprehensive explanation of BPE, WordPiece, and Unigram algorithms.--## Byte-Pair Encoding (BPE)--### Algorithm overview--BPE iteratively merges the most frequent pair of tokens in a corpus.--**Training process**:-1. Initialize vocabulary with all characters-2. Count frequency of all adjacent token pairs-3. Merge most frequent pair into new token-4. Add new token to vocabulary-5. Update corpus with new token-6. Repeat until vocabulary size reached--### Step-by-step example--**Corpus**:-```-low: 5-lower: 2-newest: 6-widest: 3-```--**Iteration 1**:-```-Count pairs:-'e' + 's': 9 (newest: 6, widest: 3) ← most frequent-'l' + 'o': 7-'o' + 'w': 7-...--Merge: 'e' + 's' → 'es'--Updated corpus:-low: 5-lower: 2-newest: 6 → newes|t: 6-widest: 3 → wides|t: 3--Vocabulary: [a-z] + ['es']-```--**Iteration 2**:-```-Count pairs:-'es' + 't': 9 ← most frequent-'l' + 'o': 7-...--Merge: 'es' + 't' → 'est'--Updated corpus:-low: 5-lower: 2-newest: 6 → new|est: 6-widest: 3 → wid|est: 3--Vocabulary: [a-z] + ['es', 'est']-```--**Continue until desired vocabulary size...**--### Tokenization with trained BPE--Given vocabulary: `['l', 'o', 'w', 'e', 'r', 'n', 's', 't', 'i', 'd', 'es', 'est', 'lo', 'low', 'ne', 'new', 'newest', 'wi', 'wid', 'widest']`--Tokenize "lowest":-```-Step 1: Split into characters-['l', 'o', 'w', 'e', 's', 't']--Step 2: Apply merges in order learned during training-- Merge 'l' + 'o' → 'lo' (if this merge was learned)-- Merge 'lo' + 'w' → 'low' (if learned)-- Merge 'e' + 's' → 'es' (learned)-- Merge 'es' + 't' → 'est' (learned)--Final: ['low', 'est']-```--### Implementation--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from tokenizers.pre_tokenizers import Whitespace--# Initialize-tokenizer = Tokenizer(BPE(unk_token="[UNK]"))-tokenizer.pre_tokenizer = Whitespace()--# Configure trainer-trainer = BpeTrainer(- vocab_size=1000,- min_frequency=2,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]-)--# Train-corpus = [- "This is a sample corpus for BPE training.",- "BPE learns subword units from the training data.",- # ... more sentences-]--tokenizer.train_from_iterator(corpus, trainer=trainer)--# Use-output = tokenizer.encode("This is tokenization")-print(output.tokens) # ['This', 'is', 'token', 'ization']-```--### Byte-level BPE (GPT-2 variant)--**Problem**: Standard BPE has limited character coverage (256+ Unicode chars)--**Solution**: Operate on byte level (256 bytes)--```python-from tokenizers.pre_tokenizers import ByteLevel-from tokenizers.decoders import ByteLevel as ByteLevelDecoder--tokenizer = Tokenizer(BPE())--# Byte-level pre-tokenization-tokenizer.pre_tokenizer = ByteLevel()-tokenizer.decoder = ByteLevelDecoder()--# This handles ALL possible characters, including emojis-text = "Hello 🌍 世界"-tokens = tokenizer.encode(text).tokens-```--**Advantages**:-- Handles any Unicode character (256 byte coverage)-- No unknown tokens (worst case: bytes)-- Used by GPT-2, GPT-3, BART--**Trade-offs**:-- Slightly worse compression (bytes vs characters)-- More tokens for non-ASCII text--### BPE variants--**SentencePiece BPE**:-- Language-independent (no pre-tokenization)-- Treats input as raw byte stream-- Used by T5, ALBERT, XLNet--**Robust BPE**:-- Dropout during training (randomly skip merges)-- More robust tokenization at inference-- Reduces overfitting to training data--## WordPiece--### Algorithm overview--WordPiece is similar to BPE but uses a different merge selection criterion.--**Training process**:-1. Initialize vocabulary with all characters-2. Count frequency of all token pairs-3. Score each pair: `score = freq(pair) / (freq(first) × freq(second))`-4. Merge pair with highest score-5. Repeat until vocabulary size reached--### Why different scoring?--**BPE**: Merges most frequent pairs-- "aa" appears 100 times → high priority-- Even if 'a' appears 1000 times alone--**WordPiece**: Merges pairs that are semantically related-- "aa" appears 100 times, 'a' appears 1000 times → low score (100 / (1000 × 1000))-- "th" appears 50 times, 't' appears 60 times, 'h' appears 55 times → high score (50 / (60 × 55))-- Prioritizes pairs that appear together more than expected--### Step-by-step example--**Corpus**:-```-low: 5-lower: 2-newest: 6-widest: 3-```--**Iteration 1**:-```-Count frequencies:-'e': 11 (lower: 2, newest: 6, widest: 3)-'s': 9-'t': 9-...--Count pairs:-'e' + 's': 9 (newest: 6, widest: 3)-'es' + 't': 9 (newest: 6, widest: 3)-...--Compute scores:-score('e' + 's') = 9 / (11 × 9) = 0.091-score('es' + 't') = 9 / (9 × 9) = 0.111 ← highest score-score('l' + 'o') = 7 / (7 × 9) = 0.111 ← tied--Choose: 'es' + 't' → 'est' (or 'lo' if tied)-```--**Key difference**: WordPiece prioritizes rare combinations over frequent ones.--### Tokenization with WordPiece--Given vocabulary: `['##e', '##s', '##t', 'l', 'o', 'w', 'new', 'est', 'low']`--Tokenize "lowest":-```-Step 1: Find longest matching prefix-'lowest' → 'low' (matches)--Step 2: Find longest match for remainder-'est' → 'est' (matches)--Final: ['low', 'est']-```--**If no match**:-```-Tokenize "unknownword":-'unknownword' → no match-'unknown' → no match-'unkn' → no match-'un' → no match-'u' → no match-→ [UNK]-```--### Implementation--```python-from tokenizers import Tokenizer-from tokenizers.models import WordPiece-from tokenizers.trainers import WordPieceTrainer-from tokenizers.normalizers import BertNormalizer-from tokenizers.pre_tokenizers import BertPreTokenizer--# Initialize BERT-style tokenizer-tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))--# Normalization (lowercase, accent stripping)-tokenizer.normalizer = BertNormalizer(lowercase=True)--# Pre-tokenization (whitespace + punctuation)-tokenizer.pre_tokenizer = BertPreTokenizer()--# Configure trainer-trainer = WordPieceTrainer(- vocab_size=30522, # BERT vocab size- min_frequency=2,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],- continuing_subword_prefix="##" # BERT uses ##-)--# Train-tokenizer.train_from_iterator(corpus, trainer=trainer)--# Use-output = tokenizer.encode("Tokenization works great!")-print(output.tokens) # ['token', '##ization', 'works', 'great', '!']-```--### Subword prefix--**BERT uses `##` prefix**:-```-"unbelievable" → ['un', '##believ', '##able']-```--**Why?**-- Indicates token is a continuation-- Allows reconstruction: remove ##, concatenate-- Helps model distinguish word boundaries--### WordPiece advantages--**Semantic merges**:-- Prioritizes meaningful combinations-- "qu" has high score (always together)-- "qx" has low score (rare combination)--**Better for morphology**:-- Captures affixes: un-, -ing, -ed-- Preserves word stems--**Trade-offs**:-- Slower training than BPE-- More memory (stores vocabulary, not merges)-- Original implementation not open-source (HF reimplementation)--## Unigram--### Algorithm overview--Unigram works backward: start with large vocabulary, remove tokens.--**Training process**:-1. Initialize with large vocabulary (all substrings)-2. Estimate probability of each token (frequency-based)-3. For each token, compute loss increase if removed-4. Remove 10-20% of tokens with lowest loss impact-5. Re-estimate probabilities-6. Repeat until desired vocabulary size--### Probabilistic tokenization--**Unigram assumption**: Each token is independent.--Given vocabulary with probabilities:-```-P('low') = 0.02-P('l') = 0.01-P('o') = 0.015-P('w') = 0.01-P('est') = 0.03-P('e') = 0.02-P('s') = 0.015-P('t') = 0.015-```--Tokenize "lowest":-```-Option 1: ['low', 'est']-P = P('low') × P('est') = 0.02 × 0.03 = 0.0006--Option 2: ['l', 'o', 'w', 'est']-P = 0.01 × 0.015 × 0.01 × 0.03 = 0.000000045--Option 3: ['low', 'e', 's', 't']-P = 0.02 × 0.02 × 0.015 × 0.015 = 0.0000009--Choose option 1 (highest probability)-```--### Viterbi algorithm--Finding best tokenization is expensive (exponential possibilities).--**Viterbi algorithm** (dynamic programming):-```python-def tokenize_viterbi(word, vocab, probs):- n = len(word)- # dp[i] = (best_prob, best_tokens) for word[:i]- dp = [{} for _ in range(n + 1)]- dp[0] = (0.0, []) # log probability-- for i in range(1, n + 1):- best_prob = float('-inf')- best_tokens = []-- # Try all possible last tokens- for j in range(i):- token = word[j:i]- if token in vocab:- prob = dp[j][0] + log(probs[token])- if prob > best_prob:- best_prob = prob- best_tokens = dp[j][1] + [token]-- dp[i] = (best_prob, best_tokens)-- return dp[n][1]-```--**Time complexity**: O(n² × vocab_size) vs O(2^n) brute force--### Implementation--```python-from tokenizers import Tokenizer-from tokenizers.models import Unigram-from tokenizers.trainers import UnigramTrainer--# Initialize-tokenizer = Tokenizer(Unigram())--# Configure trainer-trainer = UnigramTrainer(- vocab_size=8000,- special_tokens=["<unk>", "<s>", "</s>"],- unk_token="<unk>",- max_piece_length=16, # Max token length- n_sub_iterations=2, # EM iterations- shrinking_factor=0.75 # Remove 25% each iteration-)--# Train-tokenizer.train_from_iterator(corpus, trainer=trainer)--# Use-output = tokenizer.encode("Tokenization with Unigram")-print(output.tokens) # ['▁Token', 'ization', '▁with', '▁Un', 'igram']-```--### Unigram advantages--**Probabilistic**:-- Multiple valid tokenizations-- Can sample different tokenizations (data augmentation)--**Subword regularization**:-```python-# Sample different tokenizations-for _ in range(3):- tokens = tokenizer.encode("tokenization", is_pretokenized=False).tokens- print(tokens)--# Output (different each time):-# ['token', 'ization']-# ['tok', 'en', 'ization']-# ['token', 'iz', 'ation']-```--**Language-independent**:-- No word boundaries needed-- Works for CJK languages (Chinese, Japanese, Korean)-- Treats input as character stream--**Trade-offs**:-- Slower training (EM algorithm)-- More hyperparameters-- Larger model (stores probabilities)--## Algorithm comparison--### Training speed--| Algorithm | Small (10MB) | Medium (100MB) | Large (1GB) |-|------------|--------------|----------------|-------------|-| BPE | 10-15 sec | 1-2 min | 10-20 min |-| WordPiece | 15-20 sec | 2-3 min | 15-30 min |-| Unigram | 20-30 sec | 3-5 min | 30-60 min |--**Tested on**: 16-core CPU, 30k vocab--### Tokenization quality--Tested on English Wikipedia (perplexity measurement):--| Algorithm | Vocab Size | Tokens/Word | Unknown Rate |-|------------|------------|-------------|--------------|-| BPE | 30k | 1.3 | 0.5% |-| WordPiece | 30k | 1.2 | 1.2% |-| Unigram | 8k | 1.5 | 0.3% |--**Key observations**:-- WordPiece: Slightly better compression-- BPE: Lower unknown rate-- Unigram: Smallest vocab, good coverage--### Compression ratio--Characters per token (higher = better compression):--| Language | BPE (30k) | WordPiece (30k) | Unigram (8k) |-|----------|-----------|-----------------|--------------|-| English | 4.2 | 4.5 | 3.8 |-| Chinese | 2.1 | 2.3 | 2.5 |-| Arabic | 3.5 | 3.8 | 3.2 |--**Best for each**:-- English: WordPiece-- Chinese: Unigram (language-independent)-- Arabic: WordPiece--### Use case recommendations--**BPE** - Best for:-- English language models-- Code (handles symbols well)-- Fast training needed-- **Models**: GPT-2, GPT-3, RoBERTa, BART--**WordPiece** - Best for:-- Masked language modeling (BERT-style)-- Morphologically rich languages-- Semantic understanding tasks-- **Models**: BERT, DistilBERT, ELECTRA--**Unigram** - Best for:-- Multilingual models-- Languages without word boundaries (CJK)-- Data augmentation via subword regularization-- **Models**: T5, ALBERT, XLNet (via SentencePiece)--## Advanced topics--### Handling rare words--**BPE approach**:-```-"antidisestablishmentarianism"-→ ['anti', 'dis', 'establish', 'ment', 'arian', 'ism']-```--**WordPiece approach**:-```-"antidisestablishmentarianism"-→ ['anti', '##dis', '##establish', '##ment', '##arian', '##ism']-```--**Unigram approach**:-```-"antidisestablishmentarianism"-→ ['▁anti', 'dis', 'establish', 'ment', 'arian', 'ism']-```--### Handling numbers--**Challenge**: Infinite number combinations--**BPE solution**: Byte-level (handles any digit sequence)-```python-tokenizer = Tokenizer(BPE())-tokenizer.pre_tokenizer = ByteLevel()--# Handles any number-"123456789" → byte-level tokens-```--**WordPiece solution**: Digit pre-tokenization-```python-from tokenizers.pre_tokenizers import Digits--# Split digits individually or as groups-tokenizer.pre_tokenizer = Digits(individual_digits=True)--"123" → ['1', '2', '3']-```--**Unigram solution**: Learns common number patterns-```python-# Learns patterns during training-"2023" → ['202', '3'] or ['20', '23']-```--### Handling case sensitivity--**Lowercase (BERT)**:-```python-from tokenizers.normalizers import Lowercase--tokenizer.normalizer = Lowercase()--"Hello WORLD" → "hello world" → ['hello', 'world']-```--**Preserve case (GPT-2)**:-```python-# No case normalization-tokenizer.normalizer = None--"Hello WORLD" → ['Hello', 'WORLD']-```--**Cased tokens (RoBERTa)**:-```python-# Learns separate tokens for different cases-Vocabulary: ['Hello', 'hello', 'HELLO', 'world', 'WORLD']-```--### Handling emojis and special characters--**Byte-level (GPT-2)**:-```python-tokenizer.pre_tokenizer = ByteLevel()--"Hello 🌍 👋" → byte-level representation (always works)-```--**Unicode normalization**:-```python-from tokenizers.normalizers import NFKC--tokenizer.normalizer = NFKC()--"é" (composed) ↔ "é" (decomposed) → normalized to one form-```--## Troubleshooting--### Issue: Poor subword splitting--**Symptom**:-```-"running" → ['r', 'u', 'n', 'n', 'i', 'n', 'g'] (too granular)-```--**Solutions**:-1. Increase vocabulary size-2. Train longer (more merge iterations)-3. Lower `min_frequency` threshold--### Issue: Too many unknown tokens--**Symptom**:-```-5% of tokens are [UNK]-```--**Solutions**:-1. Increase vocabulary size-2. Use byte-level BPE (no UNK possible)-3. Verify training corpus is representative--### Issue: Inconsistent tokenization--**Symptom**:-```-"running" → ['run', 'ning']-"runner" → ['r', 'u', 'n', 'n', 'e', 'r']-```--**Solutions**:-1. Check normalization consistency-2. Ensure pre-tokenization is deterministic-3. Use Unigram for probabilistic variance--## Best practices--1. **Match algorithm to model architecture**:- - BERT-style → WordPiece- - GPT-style → BPE- - T5-style → Unigram--2. **Use byte-level for multilingual**:- - Handles any Unicode- - No unknown tokens--3. **Test on representative data**:- - Measure compression ratio- - Check unknown token rate- - Inspect sample tokenizations--4. **Version control tokenizers**:- - Save with model- - Document special tokens- - Track vocabulary changesdiff --git a/optional-skills/mlops/huggingface-tokenizers/references/integration.md b/optional-skills/mlops/huggingface-tokenizers/references/integration.mddeleted file mode 100644index a5dafec16ff..00000000000--- a/optional-skills/mlops/huggingface-tokenizers/references/integration.md+++ /dev/null@@ -1,637 +0,0 @@-# Transformers Integration--Complete guide to using HuggingFace Tokenizers with the Transformers library.--## AutoTokenizer--The easiest way to load tokenizers.--### Loading pretrained tokenizers--```python-from transformers import AutoTokenizer--# Load from HuggingFace Hub-tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")--# Check if using fast tokenizer (Rust-based)-print(tokenizer.is_fast) # True--# Access underlying tokenizers.Tokenizer-if tokenizer.is_fast:- fast_tokenizer = tokenizer.backend_tokenizer- print(type(fast_tokenizer)) # <class 'tokenizers.Tokenizer'>-```--### Fast vs slow tokenizers--| Feature | Fast (Rust) | Slow (Python) |-|--------------------------|----------------|---------------|-| Speed | 5-10× faster | Baseline |-| Alignment tracking | ✅ Full support | ❌ Limited |-| Batch processing | ✅ Optimized | ⚠️ Slower |-| Offset mapping | ✅ Yes | ❌ No |-| Installation | `tokenizers` | Built-in |--**Always use fast tokenizers when available.**--### Check available tokenizers--```python-from transformers import TOKENIZER_MAPPING--# List all fast tokenizers-for config_class, (slow, fast) in TOKENIZER_MAPPING.items():- if fast is not None:- print(f"{config_class.__name__}: {fast.__name__}")-```--## PreTrainedTokenizerFast--Wrap custom tokenizers for transformers.--### Convert custom tokenizer--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from transformers import PreTrainedTokenizerFast--# Train custom tokenizer-tokenizer = Tokenizer(BPE())-trainer = BpeTrainer(- vocab_size=30000,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]-)-tokenizer.train(files=["corpus.txt"], trainer=trainer)--# Save tokenizer-tokenizer.save("my-tokenizer.json")--# Wrap for transformers-transformers_tokenizer = PreTrainedTokenizerFast(- tokenizer_file="my-tokenizer.json",- unk_token="[UNK]",- sep_token="[SEP]",- pad_token="[PAD]",- cls_token="[CLS]",- mask_token="[MASK]"-)--# Save in transformers format-transformers_tokenizer.save_pretrained("my-tokenizer")-```--**Result**: Directory with `tokenizer.json` + `tokenizer_config.json` + `special_tokens_map.json`--### Use like any transformers tokenizer--```python-# Load-from transformers import AutoTokenizer-tokenizer = AutoTokenizer.from_pretrained("my-tokenizer")--# Encode with all transformers features-outputs = tokenizer(- "Hello world",- padding="max_length",- truncation=True,- max_length=128,- return_tensors="pt"-)--print(outputs.keys())-# dict_keys(['input_ids', 'token_type_ids', 'attention_mask'])-```--## Special tokens--### Default special tokens--| Model Family | CLS/BOS | SEP/EOS | PAD | UNK | MASK |-|--------------|---------|---------------|---------|---------|---------|-| BERT | [CLS] | [SEP] | [PAD] | [UNK] | [MASK] |-| GPT-2 | - | <\|endoftext\|> | <\|endoftext\|> | <\|endoftext\|> | - |-| RoBERTa | <s> | </s> | <pad> | <unk> | <mask> |-| T5 | - | </s> | <pad> | <unk> | - |--### Adding special tokens--```python-# Add new special tokens-special_tokens_dict = {- "additional_special_tokens": ["<|image|>", "<|video|>", "<|audio|>"]-}--num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)-print(f"Added {num_added_tokens} tokens")--# Resize model embeddings-model.resize_token_embeddings(len(tokenizer))--# Use new tokens-text = "This is an image: <|image|>"-tokens = tokenizer.encode(text)-```--### Adding regular tokens--```python-# Add domain-specific tokens-new_tokens = ["COVID-19", "mRNA", "vaccine"]-num_added = tokenizer.add_tokens(new_tokens)--# These are NOT special tokens (can be split if needed)-tokenizer.add_tokens(new_tokens, special_tokens=False)--# These ARE special tokens (never split)-tokenizer.add_tokens(new_tokens, special_tokens=True)-```--## Encoding and decoding--### Basic encoding--```python-# Single sentence-text = "Hello, how are you?"-encoded = tokenizer(text)--print(encoded)-# {'input_ids': [101, 7592, 1010, 2129, 2024, 2017, 1029, 102],-# 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0],-# 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}-```--### Batch encoding--```python-# Multiple sentences-texts = ["Hello world", "How are you?", "I am fine"]-encoded = tokenizer(texts, padding=True, truncation=True, max_length=10)--print(encoded['input_ids'])-# [[101, 7592, 2088, 102, 0, 0, 0, 0, 0, 0],-# [101, 2129, 2024, 2017, 1029, 102, 0, 0, 0, 0],-# [101, 1045, 2572, 2986, 102, 0, 0, 0, 0, 0]]-```--### Return tensors--```python-# Return PyTorch tensors-outputs = tokenizer("Hello world", return_tensors="pt")-print(outputs['input_ids'].shape) # torch.Size([1, 5])--# Return TensorFlow tensors-outputs = tokenizer("Hello world", return_tensors="tf")--# Return NumPy arrays-outputs = tokenizer("Hello world", return_tensors="np")--# Return lists (default)-outputs = tokenizer("Hello world", return_tensors=None)-```--### Decoding--```python-# Decode token IDs-ids = [101, 7592, 2088, 102]-text = tokenizer.decode(ids)-print(text) # "[CLS] hello world [SEP]"--# Skip special tokens-text = tokenizer.decode(ids, skip_special_tokens=True)-print(text) # "hello world"--# Batch decode-batch_ids = [[101, 7592, 102], [101, 2088, 102]]-texts = tokenizer.batch_decode(batch_ids, skip_special_tokens=True)-print(texts) # ["hello", "world"]-```--## Padding and truncation--### Padding strategies--```python-# Pad to max length in batch-tokenizer(texts, padding="longest")--# Pad to model max length-tokenizer(texts, padding="max_length", max_length=128)--# No padding-tokenizer(texts, padding=False)--# Pad to multiple of value (for efficient computation)-tokenizer(texts, padding="max_length", max_length=128, pad_to_multiple_of=8)-# Result: length will be 128 (already multiple of 8)-```--### Truncation strategies--```python-# Truncate to max length-tokenizer(text, truncation=True, max_length=10)--# Only truncate first sequence (for pairs)-tokenizer(text1, text2, truncation="only_first", max_length=20)--# Only truncate second sequence-tokenizer(text1, text2, truncation="only_second", max_length=20)--# Truncate longest first (default for pairs)-tokenizer(text1, text2, truncation="longest_first", max_length=20)--# No truncation (error if too long)-tokenizer(text, truncation=False)-```--### Stride for long documents--```python-# For documents longer than max_length-text = "Very long document " * 1000--# Encode with overlap-encodings = tokenizer(- text,- max_length=512,- stride=128, # Overlap between chunks- truncation=True,- return_overflowing_tokens=True,- return_offsets_mapping=True-)--# Get all chunks-num_chunks = len(encodings['input_ids'])-print(f"Split into {num_chunks} chunks")--# Each chunk overlaps by stride tokens-for i, chunk in enumerate(encodings['input_ids']):- print(f"Chunk {i}: {len(chunk)} tokens")-```--**Use case**: Long document QA, sliding window inference--## Alignment and offsets--### Offset mapping--```python-# Get character offsets for each token-encoded = tokenizer("Hello, world!", return_offsets_mapping=True)--for token, (start, end) in zip(- encoded.tokens(),- encoded['offset_mapping'][0]-):- print(f"{token:10s} → [{start:2d}, {end:2d})")--# Output:-# [CLS] → [ 0, 0)-# Hello → [ 0, 5)-# , → [ 5, 6)-# world → [ 7, 12)-# ! → [12, 13)-# [SEP] → [ 0, 0)-```--### Word IDs--```python-# Get word index for each token-encoded = tokenizer("Hello world", return_offsets_mapping=True)-word_ids = encoded.word_ids()--print(word_ids)-# [None, 0, 1, None]-# None = special token, 0 = first word, 1 = second word-```--**Use case**: Token classification (NER, POS tagging)--### Character to token mapping--```python-text = "Machine learning is awesome"-encoded = tokenizer(text, return_offsets_mapping=True)--# Find token for character position-char_pos = 8 # "l" in "learning"-token_idx = encoded.char_to_token(char_pos)--print(f"Character {char_pos} is in token {token_idx}: {encoded.tokens()[token_idx]}")-# Character 8 is in token 2: learning-```--**Use case**: Question answering (map answer character span to tokens)--### Sequence pairs--```python-# Encode sentence pair-encoded = tokenizer("Question here", "Answer here", return_offsets_mapping=True)--# Get sequence IDs (which sequence each token belongs to)-sequence_ids = encoded.sequence_ids()-print(sequence_ids)-# [None, 0, 0, 0, None, 1, 1, 1, None]-# None = special token, 0 = question, 1 = answer-```--## Model integration--### Use with transformers models--```python-from transformers import AutoModel, AutoTokenizer-import torch--# Load model and tokenizer-model = AutoModel.from_pretrained("bert-base-uncased")-tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")--# Tokenize-text = "Hello world"-inputs = tokenizer(text, return_tensors="pt")--# Forward pass-with torch.no_grad():- outputs = model(**inputs)--# Get embeddings-last_hidden_state = outputs.last_hidden_state-print(last_hidden_state.shape) # [1, seq_len, hidden_size]-```--### Custom model with custom tokenizer--```python-from transformers import BertConfig, BertModel--# Train custom tokenizer-from tokenizers import Tokenizer, models, trainers-tokenizer = Tokenizer(models.BPE())-trainer = trainers.BpeTrainer(vocab_size=30000)-tokenizer.train(files=["data.txt"], trainer=trainer)--# Wrap for transformers-from transformers import PreTrainedTokenizerFast-fast_tokenizer = PreTrainedTokenizerFast(- tokenizer_object=tokenizer,- unk_token="[UNK]",- pad_token="[PAD]"-)--# Create model with custom vocab size-config = BertConfig(vocab_size=30000)-model = BertModel(config)--# Use together-inputs = fast_tokenizer("Hello world", return_tensors="pt")-outputs = model(**inputs)-```--### Save and load together--```python-# Save both-model.save_pretrained("my-model")-tokenizer.save_pretrained("my-model")--# Directory structure:-# my-model/-# ├── config.json-# ├── pytorch_model.bin-# ├── tokenizer.json-# ├── tokenizer_config.json-# └── special_tokens_map.json--# Load both-from transformers import AutoModel, AutoTokenizer--model = AutoModel.from_pretrained("my-model")-tokenizer = AutoTokenizer.from_pretrained("my-model")-```--## Advanced features--### Multimodal tokenization--```python-from transformers import AutoTokenizer--# LLaVA-style (image + text)-tokenizer = AutoTokenizer.from_pretrained("llava-hf/llava-1.5-7b-hf")--# Add image placeholder token-tokenizer.add_special_tokens({"additional_special_tokens": ["<image>"]})--# Use in prompt-text = "Describe this image: <image>"-inputs = tokenizer(text, return_tensors="pt")-```--### Template formatting--```python-# Chat template-messages = [- {"role": "system", "content": "You are a helpful assistant."},- {"role": "user", "content": "Hello!"},- {"role": "assistant", "content": "Hi! How can I help?"},- {"role": "user", "content": "What's the weather?"}-]--# Apply chat template (if tokenizer has one)-if hasattr(tokenizer, "apply_chat_template"):- text = tokenizer.apply_chat_template(messages, tokenize=False)- inputs = tokenizer(text, return_tensors="pt")-```--### Custom template--```python-from transformers import PreTrainedTokenizerFast--tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")--# Define chat template-tokenizer.chat_template = """-{%- for message in messages %}- {%- if message['role'] == 'system' %}- System: {{ message['content'] }}\\n- {%- elif message['role'] == 'user' %}- User: {{ message['content'] }}\\n- {%- elif message['role'] == 'assistant' %}- Assistant: {{ message['content'] }}\\n- {%- endif %}-{%- endfor %}-Assistant:-"""--# Use template-text = tokenizer.apply_chat_template(messages, tokenize=False)-```--## Performance optimization--### Batch processing--```python-# Process large datasets efficiently-from datasets import load_dataset--dataset = load_dataset("imdb", split="train[:1000]")--# Tokenize in batches-def tokenize_function(examples):- return tokenizer(- examples["text"],- padding="max_length",- truncation=True,- max_length=512- )--# Map over dataset (batched)-tokenized_dataset = dataset.map(- tokenize_function,- batched=True,- batch_size=1000,- num_proc=4 # Parallel processing-)-```--### Caching--```python-# Enable caching for repeated tokenization-tokenizer = AutoTokenizer.from_pretrained(- "bert-base-uncased",- use_fast=True,- cache_dir="./cache" # Cache tokenizer files-)--# Tokenize with caching-from functools import lru_cache--@lru_cache(maxsize=10000)-def cached_tokenize(text):- return tuple(tokenizer.encode(text))--# Reuses cached results for repeated inputs-```--### Memory efficiency--```python-# For very large datasets, use streaming-from datasets import load_dataset--dataset = load_dataset("pile", split="train", streaming=True)--def process_batch(batch):- # Tokenize- tokens = tokenizer(batch["text"], truncation=True, max_length=512)-- # Process tokens...-- return tokens--# Process in chunks (memory efficient)-for batch in dataset.batch(batch_size=1000):- processed = process_batch(batch)-```--## Troubleshooting--### Issue: Tokenizer not fast--**Symptom**:-```python-tokenizer.is_fast # False-```--**Solution**: Install tokenizers library-```bash-pip install tokenizers-```--### Issue: Special tokens not working--**Symptom**: Special tokens are split into subwords--**Solution**: Add as special tokens, not regular tokens-```python-# Wrong-tokenizer.add_tokens(["<|image|>"])--# Correct-tokenizer.add_special_tokens({"additional_special_tokens": ["<|image|>"]})-```--### Issue: Offset mapping not available--**Symptom**:-```python-tokenizer("text", return_offsets_mapping=True)-# Error: return_offsets_mapping not supported-```--**Solution**: Use fast tokenizer-```python-from transformers import AutoTokenizer--# Load fast version-tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)-```--### Issue: Padding inconsistent--**Symptom**: Some sequences padded, others not--**Solution**: Specify padding strategy-```python-# Explicit padding-tokenizer(- texts,- padding="max_length", # or "longest"- max_length=128-)-```--## Best practices--1. **Always use fast tokenizers**:- - 5-10× faster- - Full alignment tracking- - Better batch processing--2. **Save tokenizer with model**:- - Ensures reproducibility- - Prevents version mismatches--3. **Use batch processing for datasets**:- - Tokenize with `.map(batched=True)`- - Set `num_proc` for parallelism--4. **Enable caching for repeated inputs**:- - Use `lru_cache` for inference- - Cache tokenizer files with `cache_dir`--5. **Handle special tokens properly**:- - Use `add_special_tokens()` for never-split tokens- - Resize embeddings after adding tokens--6. **Test alignment for downstream tasks**:- - Verify `offset_mapping` is correct- - Test `char_to_token()` on samples--7. **Version control tokenizer config**:- - Save `tokenizer_config.json`- - Document custom templates- - Track vocabulary changesdiff --git a/optional-skills/mlops/huggingface-tokenizers/references/pipeline.md b/optional-skills/mlops/huggingface-tokenizers/references/pipeline.mddeleted file mode 100644index 9efcb48ae37..00000000000--- a/optional-skills/mlops/huggingface-tokenizers/references/pipeline.md+++ /dev/null@@ -1,723 +0,0 @@-# Tokenization Pipeline Components--Complete guide to normalizers, pre-tokenizers, models, post-processors, and decoders.--## Pipeline overview--**Full tokenization pipeline**:-```-Raw Text- ↓-Normalization (cleaning, lowercasing)- ↓-Pre-tokenization (split into words)- ↓-Model (apply BPE/WordPiece/Unigram)- ↓-Post-processing (add special tokens)- ↓-Token IDs-```--**Decoding reverses the process**:-```-Token IDs- ↓-Decoder (handle special encodings)- ↓-Raw Text-```--## Normalizers--Clean and standardize input text.--### Common normalizers--**Lowercase**:-```python-from tokenizers.normalizers import Lowercase--tokenizer.normalizer = Lowercase()--# Input: "Hello WORLD"-# Output: "hello world"-```--**Unicode normalization**:-```python-from tokenizers.normalizers import NFD, NFC, NFKD, NFKC--# NFD: Canonical decomposition-tokenizer.normalizer = NFD()-# "é" → "e" + "́" (separate characters)--# NFC: Canonical composition (default)-tokenizer.normalizer = NFC()-# "e" + "́" → "é" (composed)--# NFKD: Compatibility decomposition-tokenizer.normalizer = NFKD()-# "fi" → "f" + "i"--# NFKC: Compatibility composition-tokenizer.normalizer = NFKC()-# Most aggressive normalization-```--**Strip accents**:-```python-from tokenizers.normalizers import StripAccents--tokenizer.normalizer = StripAccents()--# Input: "café"-# Output: "cafe"-```--**Whitespace handling**:-```python-from tokenizers.normalizers import Strip, StripAccents--# Remove leading/trailing whitespace-tokenizer.normalizer = Strip()--# Input: " hello "-# Output: "hello"-```--**Replace patterns**:-```python-from tokenizers.normalizers import Replace--# Replace newlines with spaces-tokenizer.normalizer = Replace("\\n", " ")--# Input: "hello\\nworld"-# Output: "hello world"-```--### Combining normalizers--```python-from tokenizers.normalizers import Sequence, NFD, Lowercase, StripAccents--# BERT-style normalization-tokenizer.normalizer = Sequence([- NFD(), # Unicode decomposition- Lowercase(), # Convert to lowercase- StripAccents() # Remove accents-])--# Input: "Café au Lait"-# After NFD: "Café au Lait" (e + ́)-# After Lowercase: "café au lait"-# After StripAccents: "cafe au lait"-```--### Use case examples--**Case-insensitive model (BERT)**:-```python-from tokenizers.normalizers import BertNormalizer--# All-in-one BERT normalization-tokenizer.normalizer = BertNormalizer(- clean_text=True, # Remove control characters- handle_chinese_chars=True, # Add spaces around Chinese- strip_accents=True, # Remove accents- lowercase=True # Lowercase-)-```--**Case-sensitive model (GPT-2)**:-```python-# Minimal normalization-tokenizer.normalizer = NFC() # Only normalize Unicode-```--**Multilingual (mBERT)**:-```python-# Preserve scripts, normalize form-tokenizer.normalizer = NFKC()-```--## Pre-tokenizers--Split text into word-like units before tokenization.--### Whitespace splitting--```python-from tokenizers.pre_tokenizers import Whitespace--tokenizer.pre_tokenizer = Whitespace()--# Input: "Hello world! How are you?"-# Output: [("Hello", (0, 5)), ("world!", (6, 12)), ("How", (13, 16)), ("are", (17, 20)), ("you?", (21, 25))]-```--### Punctuation isolation--```python-from tokenizers.pre_tokenizers import Punctuation--tokenizer.pre_tokenizer = Punctuation()--# Input: "Hello, world!"-# Output: [("Hello", ...), (",", ...), ("world", ...), ("!", ...)]-```--### Byte-level (GPT-2)--```python-from tokenizers.pre_tokenizers import ByteLevel--tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=True)--# Input: "Hello world"-# Output: Byte-level tokens with Ġ prefix for spaces-# [("ĠHello", ...), ("Ġworld", ...)]-```--**Key feature**: Handles ALL Unicode characters (256 byte combinations)--### Metaspace (SentencePiece)--```python-from tokenizers.pre_tokenizers import Metaspace--tokenizer.pre_tokenizer = Metaspace(replacement="▁", add_prefix_space=True)--# Input: "Hello world"-# Output: [("▁Hello", ...), ("▁world", ...)]-```--**Used by**: T5, ALBERT (via SentencePiece)--### Digits splitting--```python-from tokenizers.pre_tokenizers import Digits--# Split digits individually-tokenizer.pre_tokenizer = Digits(individual_digits=True)--# Input: "Room 123"-# Output: [("Room", ...), ("1", ...), ("2", ...), ("3", ...)]--# Keep digits together-tokenizer.pre_tokenizer = Digits(individual_digits=False)--# Input: "Room 123"-# Output: [("Room", ...), ("123", ...)]-```--### BERT pre-tokenizer--```python-from tokenizers.pre_tokenizers import BertPreTokenizer--tokenizer.pre_tokenizer = BertPreTokenizer()--# Splits on whitespace and punctuation, preserves CJK-# Input: "Hello, 世界!"-# Output: [("Hello", ...), (",", ...), ("世", ...), ("界", ...), ("!", ...)]-```--### Combining pre-tokenizers--```python-from tokenizers.pre_tokenizers import Sequence, Whitespace, Punctuation--tokenizer.pre_tokenizer = Sequence([- Whitespace(), # Split on whitespace first- Punctuation() # Then isolate punctuation-])--# Input: "Hello, world!"-# After Whitespace: [("Hello,", ...), ("world!", ...)]-# After Punctuation: [("Hello", ...), (",", ...), ("world", ...), ("!", ...)]-```--### Pre-tokenizer comparison--| Pre-tokenizer | Use Case | Example |-|-------------------|---------------------------------|--------------------------------------------|-| Whitespace | Simple English | "Hello world" → ["Hello", "world"] |-| Punctuation | Isolate symbols | "world!" → ["world", "!"] |-| ByteLevel | Multilingual, emojis | "🌍" → byte tokens |-| Metaspace | SentencePiece-style | "Hello" → ["▁Hello"] |-| BertPreTokenizer | BERT-style (CJK aware) | "世界" → ["世", "界"] |-| Digits | Handle numbers | "123" → ["1", "2", "3"] or ["123"] |--## Models--Core tokenization algorithms.--### BPE Model--```python-from tokenizers.models import BPE--model = BPE(- vocab=None, # Or provide pre-built vocab- merges=None, # Or provide merge rules- unk_token="[UNK]", # Unknown token- continuing_subword_prefix="",- end_of_word_suffix="",- fuse_unk=False # Keep unknown tokens separate-)--tokenizer = Tokenizer(model)-```--**Parameters**:-- `vocab`: Dict of token → id-- `merges`: List of merge rules `["a b", "ab c"]`-- `unk_token`: Token for unknown words-- `continuing_subword_prefix`: Prefix for subwords (empty for GPT-2)-- `end_of_word_suffix`: Suffix for last subword (empty for GPT-2)--### WordPiece Model--```python-from tokenizers.models import WordPiece--model = WordPiece(- vocab=None,- unk_token="[UNK]",- max_input_chars_per_word=100, # Max word length- continuing_subword_prefix="##" # BERT-style prefix-)--tokenizer = Tokenizer(model)-```--**Key difference**: Uses `##` prefix for continuing subwords.--### Unigram Model--```python-from tokenizers.models import Unigram--model = Unigram(- vocab=None, # List of (token, score) tuples- unk_id=0, # ID for unknown token- byte_fallback=False # Fall back to bytes if no match-)--tokenizer = Tokenizer(model)-```--**Probabilistic**: Selects tokenization with highest probability.--### WordLevel Model--```python-from tokenizers.models import WordLevel--# Simple word-to-ID mapping (no subwords)-model = WordLevel(- vocab=None,- unk_token="[UNK]"-)--tokenizer = Tokenizer(model)-```--**Warning**: Requires huge vocabulary (one token per word).--## Post-processors--Add special tokens and format output.--### Template processing--**BERT-style** (`[CLS] sentence [SEP]`):-```python-from tokenizers.processors import TemplateProcessing--tokenizer.post_processor = TemplateProcessing(- single="[CLS] $A [SEP]",- pair="[CLS] $A [SEP] $B [SEP]",- special_tokens=[- ("[CLS]", 101),- ("[SEP]", 102),- ],-)--# Single sentence-output = tokenizer.encode("Hello world")-# [101, ..., 102] ([CLS] hello world [SEP])--# Sentence pair-output = tokenizer.encode("Hello", "world")-# [101, ..., 102, ..., 102] ([CLS] hello [SEP] world [SEP])-```--**GPT-2 style** (`sentence <|endoftext|>`):-```python-tokenizer.post_processor = TemplateProcessing(- single="$A <|endoftext|>",- special_tokens=[- ("<|endoftext|>", 50256),- ],-)-```--**RoBERTa style** (`<s> sentence </s>`):-```python-tokenizer.post_processor = TemplateProcessing(- single="<s> $A </s>",- pair="<s> $A </s> </s> $B </s>",- special_tokens=[- ("<s>", 0),- ("</s>", 2),- ],-)-```--**T5 style** (no special tokens):-```python-# T5 doesn't add special tokens via post-processor-tokenizer.post_processor = None-```--### RobertaProcessing--```python-from tokenizers.processors import RobertaProcessing--tokenizer.post_processor = RobertaProcessing(- sep=("</s>", 2),- cls=("<s>", 0),- add_prefix_space=True, # Add space before first token- trim_offsets=True # Trim leading space from offsets-)-```--### ByteLevelProcessing--```python-from tokenizers.processors import ByteLevel as ByteLevelProcessing--tokenizer.post_processor = ByteLevelProcessing(- trim_offsets=True # Remove Ġ from offsets-)-```--## Decoders--Convert token IDs back to text.--### ByteLevel decoder--```python-from tokenizers.decoders import ByteLevel--tokenizer.decoder = ByteLevel()--# Handles byte-level tokens-# ["ĠHello", "Ġworld"] → "Hello world"-```--### WordPiece decoder--```python-from tokenizers.decoders import WordPiece--tokenizer.decoder = WordPiece(prefix="##")--# Removes ## prefix and concatenates-# ["token", "##ization"] → "tokenization"-```--### Metaspace decoder--```python-from tokenizers.decoders import Metaspace--tokenizer.decoder = Metaspace(replacement="▁", add_prefix_space=True)--# Converts ▁ back to spaces-# ["▁Hello", "▁world"] → "Hello world"-```--### BPEDecoder--```python-from tokenizers.decoders import BPEDecoder--tokenizer.decoder = BPEDecoder(suffix="</w>")--# Removes suffix and concatenates-# ["token", "ization</w>"] → "tokenization"-```--### Sequence decoder--```python-from tokenizers.decoders import Sequence, ByteLevel, Strip--tokenizer.decoder = Sequence([- ByteLevel(), # Decode byte-level first- Strip(' ', 1, 1) # Strip leading/trailing spaces-])-```--## Complete pipeline examples--### BERT tokenizer--```python-from tokenizers import Tokenizer-from tokenizers.models import WordPiece-from tokenizers.normalizers import BertNormalizer-from tokenizers.pre_tokenizers import BertPreTokenizer-from tokenizers.processors import TemplateProcessing-from tokenizers.decoders import WordPiece as WordPieceDecoder--# Model-tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))--# Normalization-tokenizer.normalizer = BertNormalizer(lowercase=True)--# Pre-tokenization-tokenizer.pre_tokenizer = BertPreTokenizer()--# Post-processing-tokenizer.post_processor = TemplateProcessing(- single="[CLS] $A [SEP]",- pair="[CLS] $A [SEP] $B [SEP]",- special_tokens=[("[CLS]", 101), ("[SEP]", 102)],-)--# Decoder-tokenizer.decoder = WordPieceDecoder(prefix="##")--# Enable padding-tokenizer.enable_padding(pad_id=0, pad_token="[PAD]")--# Enable truncation-tokenizer.enable_truncation(max_length=512)-```--### GPT-2 tokenizer--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.normalizers import NFC-from tokenizers.pre_tokenizers import ByteLevel-from tokenizers.decoders import ByteLevel as ByteLevelDecoder-from tokenizers.processors import TemplateProcessing--# Model-tokenizer = Tokenizer(BPE())--# Normalization (minimal)-tokenizer.normalizer = NFC()--# Byte-level pre-tokenization-tokenizer.pre_tokenizer = ByteLevel(add_prefix_space=False)--# Post-processing-tokenizer.post_processor = TemplateProcessing(- single="$A <|endoftext|>",- special_tokens=[("<|endoftext|>", 50256)],-)--# Byte-level decoder-tokenizer.decoder = ByteLevelDecoder()-```--### T5 tokenizer (SentencePiece-style)--```python-from tokenizers import Tokenizer-from tokenizers.models import Unigram-from tokenizers.normalizers import NFKC-from tokenizers.pre_tokenizers import Metaspace-from tokenizers.decoders import Metaspace as MetaspaceDecoder--# Model-tokenizer = Tokenizer(Unigram())--# Normalization-tokenizer.normalizer = NFKC()--# Metaspace pre-tokenization-tokenizer.pre_tokenizer = Metaspace(replacement="▁", add_prefix_space=True)--# No post-processing (T5 doesn't add CLS/SEP)-tokenizer.post_processor = None--# Metaspace decoder-tokenizer.decoder = MetaspaceDecoder(replacement="▁", add_prefix_space=True)-```--## Alignment tracking--Track token positions in original text.--### Basic alignment--```python-text = "Hello, world!"-output = tokenizer.encode(text)--for token, (start, end) in zip(output.tokens, output.offsets):- print(f"{token:10s} → [{start:2d}, {end:2d}): {text[start:end]!r}")--# Output:-# [CLS] → [ 0, 0): ''-# hello → [ 0, 5): 'Hello'-# , → [ 5, 6): ','-# world → [ 7, 12): 'world'-# ! → [12, 13): '!'-# [SEP] → [ 0, 0): ''-```--### Word-level alignment--```python-# Get word_ids (which word each token belongs to)-encoding = tokenizer.encode("Hello world")-word_ids = encoding.word_ids--print(word_ids)-# [None, 0, 0, 1, None]-# None = special token, 0 = first word, 1 = second word-```--**Use case**: Token classification (NER)-```python-# Align predictions to words-predictions = ["O", "B-PER", "I-PER", "O", "O"]-word_predictions = {}--for token_idx, word_idx in enumerate(encoding.word_ids):- if word_idx is not None and word_idx not in word_predictions:- word_predictions[word_idx] = predictions[token_idx]--print(word_predictions)-# {0: "B-PER", 1: "O"} # First word is PERSON, second is OTHER-```--### Span alignment--```python-# Find token span for character span-text = "Machine learning is awesome"-char_start, char_end = 8, 16 # "learning"--encoding = tokenizer.encode(text)--# Find token span-token_start = encoding.char_to_token(char_start)-token_end = encoding.char_to_token(char_end - 1) + 1--print(f"Tokens {token_start}:{token_end} = {encoding.tokens[token_start:token_end]}")-# Tokens 2:3 = ['learning']-```--**Use case**: Question answering (extract answer span)--## Custom components--### Custom normalizer--```python-from tokenizers import NormalizedString, Normalizer--class CustomNormalizer:- def normalize(self, normalized: NormalizedString):- # Custom normalization logic- normalized.lowercase()- normalized.replace(" ", " ") # Replace double spaces--# Use custom normalizer-tokenizer.normalizer = CustomNormalizer()-```--### Custom pre-tokenizer--```python-from tokenizers import PreTokenizedString--class CustomPreTokenizer:- def pre_tokenize(self, pretok: PreTokenizedString):- # Custom pre-tokenization logic- pretok.split(lambda i, char: char.isspace())--tokenizer.pre_tokenizer = CustomPreTokenizer()-```--## Troubleshooting--### Issue: Misaligned offsets--**Symptom**: Offsets don't match original text-```python-text = " hello" # Leading spaces-offsets = [(0, 5)] # Expects " hel"-```--**Solution**: Check normalization strips spaces-```python-# Preserve offsets-tokenizer.normalizer = Sequence([- Strip(), # This changes offsets!-])--# Use trim_offsets in post-processor instead-tokenizer.post_processor = ByteLevelProcessing(trim_offsets=True)-```--### Issue: Special tokens not added--**Symptom**: No [CLS] or [SEP] in output--**Solution**: Check post-processor is set-```python-tokenizer.post_processor = TemplateProcessing(- single="[CLS] $A [SEP]",- special_tokens=[("[CLS]", 101), ("[SEP]", 102)],-)-```--### Issue: Incorrect decoding--**Symptom**: Decoded text has ## or ▁--**Solution**: Set correct decoder-```python-# For WordPiece-tokenizer.decoder = WordPieceDecoder(prefix="##")--# For SentencePiece-tokenizer.decoder = MetaspaceDecoder(replacement="▁")-```--## Best practices--1. **Match pipeline to model architecture**:- - BERT → BertNormalizer + BertPreTokenizer + WordPiece- - GPT-2 → NFC + ByteLevel + BPE- - T5 → NFKC + Metaspace + Unigram--2. **Test pipeline on sample inputs**:- - Check normalization doesn't over-normalize- - Verify pre-tokenization splits correctly- - Ensure decoding reconstructs text--3. **Preserve alignment for downstream tasks**:- - Use `trim_offsets` instead of stripping in normalizer- - Test `char_to_token()` on sample spans--4. **Document your pipeline**:- - Save complete tokenizer config- - Document special tokens- - Note any custom componentsdiff --git a/optional-skills/mlops/huggingface-tokenizers/references/training.md b/optional-skills/mlops/huggingface-tokenizers/references/training.mddeleted file mode 100644index 99454a43468..00000000000--- a/optional-skills/mlops/huggingface-tokenizers/references/training.md+++ /dev/null@@ -1,565 +0,0 @@-# Training Custom Tokenizers--Complete guide to training tokenizers from scratch.--## Training workflow--### Step 1: Choose tokenization algorithm--**Decision tree**:-- **GPT-style model** → BPE-- **BERT-style model** → WordPiece-- **Multilingual/No word boundaries** → Unigram--### Step 2: Prepare training data--```python-# Option 1: From files-files = ["train.txt", "validation.txt"]--# Option 2: From Python list-texts = [- "This is the first sentence.",- "This is the second sentence.",- # ... more texts-]--# Option 3: From dataset iterator-from datasets import load_dataset--dataset = load_dataset("wikitext", "wikitext-103-raw-v1", split="train")--def batch_iterator(batch_size=1000):- for i in range(0, len(dataset), batch_size):- yield dataset[i:i + batch_size]["text"]-```--### Step 3: Initialize tokenizer--**BPE example**:-```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from tokenizers.pre_tokenizers import ByteLevel-from tokenizers.decoders import ByteLevel as ByteLevelDecoder--tokenizer = Tokenizer(BPE())-tokenizer.pre_tokenizer = ByteLevel()-tokenizer.decoder = ByteLevelDecoder()--trainer = BpeTrainer(- vocab_size=50000,- min_frequency=2,- special_tokens=["<|endoftext|>", "<|padding|>"],- show_progress=True-)-```--**WordPiece example**:-```python-from tokenizers.models import WordPiece-from tokenizers.trainers import WordPieceTrainer-from tokenizers.normalizers import BertNormalizer-from tokenizers.pre_tokenizers import BertPreTokenizer--tokenizer = Tokenizer(WordPiece(unk_token="[UNK]"))-tokenizer.normalizer = BertNormalizer(lowercase=True)-tokenizer.pre_tokenizer = BertPreTokenizer()--trainer = WordPieceTrainer(- vocab_size=30522,- min_frequency=2,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],- continuing_subword_prefix="##",- show_progress=True-)-```--**Unigram example**:-```python-from tokenizers.models import Unigram-from tokenizers.trainers import UnigramTrainer--tokenizer = Tokenizer(Unigram())--trainer = UnigramTrainer(- vocab_size=8000,- special_tokens=["<unk>", "<s>", "</s>", "<pad>"],- unk_token="<unk>",- show_progress=True-)-```--### Step 4: Train--```python-# From files-tokenizer.train(files=files, trainer=trainer)--# From iterator (recommended for large datasets)-tokenizer.train_from_iterator(- batch_iterator(),- trainer=trainer,- length=len(dataset) # Optional, for progress bar-)-```--**Training time** (30k vocab on 16-core CPU):-- 10 MB: 15-30 seconds-- 100 MB: 1-3 minutes-- 1 GB: 15-30 minutes-- 10 GB: 2-4 hours--### Step 5: Add post-processing--```python-from tokenizers.processors import TemplateProcessing--# BERT-style-tokenizer.post_processor = TemplateProcessing(- single="[CLS] $A [SEP]",- pair="[CLS] $A [SEP] $B [SEP]",- special_tokens=[- ("[CLS]", tokenizer.token_to_id("[CLS]")),- ("[SEP]", tokenizer.token_to_id("[SEP]")),- ],-)--# GPT-2 style-tokenizer.post_processor = TemplateProcessing(- single="$A <|endoftext|>",- special_tokens=[- ("<|endoftext|>", tokenizer.token_to_id("<|endoftext|>")),- ],-)-```--### Step 6: Save--```python-# Save to JSON-tokenizer.save("my-tokenizer.json")--# Save to directory (for transformers)-tokenizer.save("my-tokenizer-dir/tokenizer.json")--# Convert to transformers format-from transformers import PreTrainedTokenizerFast--transformers_tokenizer = PreTrainedTokenizerFast(- tokenizer_object=tokenizer,- unk_token="[UNK]",- pad_token="[PAD]",- cls_token="[CLS]",- sep_token="[SEP]",- mask_token="[MASK]"-)--transformers_tokenizer.save_pretrained("my-tokenizer-dir")-```--## Trainer configuration--### BpeTrainer parameters--```python-from tokenizers.trainers import BpeTrainer--trainer = BpeTrainer(- vocab_size=30000, # Target vocabulary size- min_frequency=2, # Minimum frequency for merges- special_tokens=["[UNK]"], # Special tokens (added first)- limit_alphabet=1000, # Limit initial alphabet size- initial_alphabet=[], # Pre-defined initial characters- show_progress=True, # Show progress bar- continuing_subword_prefix="", # Prefix for continuing subwords- end_of_word_suffix="" # Suffix for end of words-)-```--**Parameter tuning**:-- **vocab_size**: Start with 30k for English, 50k for multilingual-- **min_frequency**: 2-5 for large corpora, 1 for small-- **limit_alphabet**: Reduce for non-English (CJK languages)--### WordPieceTrainer parameters--```python-from tokenizers.trainers import WordPieceTrainer--trainer = WordPieceTrainer(- vocab_size=30522, # BERT uses 30,522- min_frequency=2,- special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"],- limit_alphabet=1000,- continuing_subword_prefix="##", # BERT-style prefix- show_progress=True-)-```--### UnigramTrainer parameters--```python-from tokenizers.trainers import UnigramTrainer--trainer = UnigramTrainer(- vocab_size=8000, # Typically smaller than BPE/WordPiece- special_tokens=["<unk>", "<s>", "</s>"],- unk_token="<unk>",- max_piece_length=16, # Maximum token length- n_sub_iterations=2, # EM algorithm iterations- shrinking_factor=0.75, # Vocabulary reduction rate- show_progress=True-)-```--## Training from large datasets--### Memory-efficient training--```python-from datasets import load_dataset-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer--# Load dataset-dataset = load_dataset("wikipedia", "20220301.en", split="train", streaming=True)--# Create iterator (yields batches)-def batch_iterator(batch_size=1000):- batch = []- for sample in dataset:- batch.append(sample["text"])- if len(batch) >= batch_size:- yield batch- batch = []- if batch:- yield batch--# Initialize tokenizer-tokenizer = Tokenizer(BPE())-trainer = BpeTrainer(vocab_size=50000, special_tokens=["<|endoftext|>"])--# Train (memory efficient - streams data)-tokenizer.train_from_iterator(- batch_iterator(),- trainer=trainer-)-```--**Memory usage**: ~200 MB (vs 10+ GB loading full dataset)--### Multi-file training--```python-import glob--# Find all training files-files = glob.glob("data/train/*.txt")-print(f"Training on {len(files)} files")--# Train on all files-tokenizer.train(files=files, trainer=trainer)-```--### Parallel training (multi-processing)--```python-from multiprocessing import Pool, cpu_count-import os--def train_shard(shard_files):- """Train tokenizer on a shard of files."""- tokenizer = Tokenizer(BPE())- trainer = BpeTrainer(vocab_size=50000)- tokenizer.train(files=shard_files, trainer=trainer)- return tokenizer.get_vocab()--# Split files into shards-num_shards = cpu_count()-file_shards = [files[i::num_shards] for i in range(num_shards)]--# Train shards in parallel-with Pool(num_shards) as pool:- vocab_shards = pool.map(train_shard, file_shards)--# Merge vocabularies (custom logic needed)-# This is a simplified example - real implementation would merge intelligently-final_vocab = {}-for vocab in vocab_shards:- final_vocab.update(vocab)-```--## Domain-specific tokenizers--### Code tokenizer--```python-from tokenizers import Tokenizer-from tokenizers.models import BPE-from tokenizers.trainers import BpeTrainer-from tokenizers.pre_tokenizers import ByteLevel-from tokenizers.normalizers import Sequence, NFC--# Code-optimized configuration-tokenizer = Tokenizer(BPE())--# Minimal normalization (preserve case, whitespace)-tokenizer.normalizer = NFC() # Only normalize Unicode--# Byte-level pre-tokenization (handles all characters)-tokenizer.pre_tokenizer = ByteLevel()--# Train on code corpus-trainer = BpeTrainer(- vocab_size=50000,- special_tokens=["<|endoftext|>", "<|pad|>"],- min_frequency=2-)--tokenizer.train(files=["code_corpus.txt"], trainer=trainer)-```--### Medical/scientific tokenizer--```python-# Preserve case and special characters-from tokenizers.normalizers import NFKC-from tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence--tokenizer = Tokenizer(BPE())--# Minimal normalization-tokenizer.normalizer = NFKC()--# Preserve medical terms-tokenizer.pre_tokenizer = Sequence([- Whitespace(),- Punctuation(behavior="isolated") # Keep punctuation separate-])--trainer = BpeTrainer(- vocab_size=50000,- special_tokens=["[UNK]", "[CLS]", "[SEP]"],- min_frequency=3 # Higher threshold for rare medical terms-)--tokenizer.train(files=["pubmed_corpus.txt"], trainer=trainer)-```--### Multilingual tokenizer--```python-# Handle multiple scripts-from tokenizers.normalizers import NFKC, Lowercase, Sequence--tokenizer = Tokenizer(BPE())--# Normalize but don't lowercase (preserves script differences)-tokenizer.normalizer = NFKC()--# Byte-level handles all Unicode-from tokenizers.pre_tokenizers import ByteLevel-tokenizer.pre_tokenizer = ByteLevel()--trainer = BpeTrainer(- vocab_size=100000, # Larger vocab for multiple languages- special_tokens=["<unk>", "<s>", "</s>"],- limit_alphabet=None # No limit (handles all scripts)-)--# Train on multilingual corpus-tokenizer.train(files=["multilingual_corpus.txt"], trainer=trainer)-```--## Vocabulary size selection--### Guidelines by task--| Task | Recommended Vocab Size | Rationale |-|-----------------------|------------------------|-----------|-| English (monolingual) | 30,000 - 50,000 | Balanced coverage |-| Multilingual | 50,000 - 250,000 | More languages = more tokens |-| Code | 30,000 - 50,000 | Similar to English |-| Domain-specific | 10,000 - 30,000 | Smaller, focused vocabulary |-| Character-level tasks | 1,000 - 5,000 | Only characters + subwords |--### Vocabulary size impact--**Small vocab (10k)**:-- Pros: Faster training, smaller model, less memory-- Cons: More tokens per sentence, worse OOV handling--**Medium vocab (30k-50k)**:-- Pros: Good balance, standard choice-- Cons: None (recommended default)--**Large vocab (100k+)**:-- Pros: Fewer tokens per sentence, better OOV-- Cons: Slower training, larger embedding table--### Empirical testing--```python-# Train multiple tokenizers with different vocab sizes-vocab_sizes = [10000, 30000, 50000, 100000]--for vocab_size in vocab_sizes:- tokenizer = Tokenizer(BPE())- trainer = BpeTrainer(vocab_size=vocab_size)- tokenizer.train(files=["sample.txt"], trainer=trainer)-- # Evaluate on test set- test_text = "Test sentence for evaluation..."- tokens = tokenizer.encode(test_text).ids-- print(f"Vocab: {vocab_size:6d} | Tokens: {len(tokens):3d} | Avg: {len(test_text)/len(tokens):.2f} chars/token")--# Example output:-# Vocab: 10000 | Tokens: 12 | Avg: 2.33 chars/token-# Vocab: 30000 | Tokens: 8 | Avg: 3.50 chars/token-# Vocab: 50000 | Tokens: 7 | Avg: 4.00 chars/token-# Vocab: 100000 | Tokens: 6 | Avg: 4.67 chars/token-```--## Testing tokenizer quality--### Coverage test--```python-# Test on held-out data-test_corpus = load_dataset("wikitext", "wikitext-103-raw-v1", split="test")--total_tokens = 0-unk_tokens = 0-unk_id = tokenizer.token_to_id("[UNK]")--for text in test_corpus["text"]:- if text.strip():- encoding = tokenizer.encode(text)- total_tokens += len(encoding.ids)- unk_tokens += encoding.ids.count(unk_id)--unk_rate = unk_tokens / total_tokens-print(f"Unknown token rate: {unk_rate:.2%}")--# Good quality: <1% unknown tokens-# Acceptable: 1-5%-# Poor: >5%-```--### Compression test--```python-# Measure tokenization efficiency-import numpy as np--token_lengths = []--for text in test_corpus["text"][:1000]:- if text.strip():- encoding = tokenizer.encode(text)- chars_per_token = len(text) / len(encoding.ids)- token_lengths.append(chars_per_token)--avg_chars_per_token = np.mean(token_lengths)-print(f"Average characters per token: {avg_chars_per_token:.2f}")--# Good: 4-6 chars/token (English)-# Acceptable: 3-4 chars/token-# Poor: <3 chars/token (under-compression)-```--### Semantic test--```python-# Manually inspect tokenization of common words/phrases-test_phrases = [- "tokenization",- "machine learning",- "artificial intelligence",- "preprocessing",- "hello world"-]--for phrase in test_phrases:- tokens = tokenizer.encode(phrase).tokens- print(f"{phrase:25s} → {tokens}")--# Good tokenization:-# tokenization → ['token', 'ization']-# machine learning → ['machine', 'learning']-# artificial intelligence → ['artificial', 'intelligence']-```--## Troubleshooting--### Issue: Training too slow--**Solutions**:-1. Reduce vocabulary size-2. Increase `min_frequency`-3. Use `limit_alphabet` to reduce initial alphabet-4. Train on subset first--```python-# Fast training configuration-trainer = BpeTrainer(- vocab_size=20000, # Smaller vocab- min_frequency=5, # Higher threshold- limit_alphabet=500, # Limit alphabet- show_progress=True-)-```--### Issue: High unknown token rate--**Solutions**:-1. Increase vocabulary size-2. Decrease `min_frequency`-3. Check normalization (might be too aggressive)--```python-# Better coverage configuration-trainer = BpeTrainer(- vocab_size=50000, # Larger vocab- min_frequency=1, # Lower threshold-)-```--### Issue: Poor quality tokenization--**Solutions**:-1. Verify normalization matches your use case-2. Check pre-tokenization splits correctly-3. Ensure training data is representative-4. Try different algorithm (BPE vs WordPiece vs Unigram)--```python-# Debug tokenization pipeline-text = "Sample text to debug"--# Check normalization-normalized = tokenizer.normalizer.normalize_str(text)-print(f"Normalized: {normalized}")--# Check pre-tokenization-pre_tokens = tokenizer.pre_tokenizer.pre_tokenize_str(text)-print(f"Pre-tokens: {pre_tokens}")--# Check final tokenization-tokens = tokenizer.encode(text).tokens-print(f"Tokens: {tokens}")-```--## Best practices--1. **Use representative training data** - Match your target domain-2. **Start with standard configs** - BERT WordPiece or GPT-2 BPE-3. **Test on held-out data** - Measure unknown token rate-4. **Iterate on vocabulary size** - Test 30k, 50k, 100k-5. **Save tokenizer with model** - Ensure reproducibility-6. **Version your tokenizers** - Track changes for reproducibility-7. **Document special tokens** - Critical for model trainingdiff --git a/optional-skills/mlops/inference/outlines/SKILL.md b/optional-skills/mlops/inference/outlines/SKILL.mddeleted file mode 100644index 16744e9a61c..00000000000--- a/optional-skills/mlops/inference/outlines/SKILL.md+++ /dev/null@@ -1,662 +0,0 @@-----name: outlines-description: "Outlines: structured JSON/regex/Pydantic LLM generation."-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [outlines, transformers, vllm, pydantic]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Prompt Engineering, Outlines, Structured Generation, JSON Schema, Pydantic, Local Models, Grammar-Based Generation, vLLM, Transformers, Type Safety]-------# Outlines: Structured Text Generation--## When to Use This Skill--Use Outlines when you need to:-- **Guarantee valid JSON/XML/code** structure during generation-- **Use Pydantic models** for type-safe outputs-- **Support local models** (Transformers, llama.cpp, vLLM)-- **Maximize inference speed** with zero-overhead structured generation-- **Generate against JSON schemas** automatically-- **Control token sampling** at the grammar level--**GitHub Stars**: 12,000+ | **From**: dottxt.ai (formerly .txt)--> **API note (Outlines 1.x):** This skill targets the current v1 API.-> The pre-1.0 helpers (`outlines.models.transformers(...)`,-> `outlines.generate.json/choice/regex/...`) have been **removed**. In v1 you-> create a model with `outlines.from_transformers(...)` (or `from_vllm`,-> `from_llamacpp`, `from_openai`) and then **call the model directly** with an-> output type: `model(prompt, output_type)`. JSON/Pydantic outputs are returned-> as a **JSON string** — validate with `YourModel.model_validate_json(result)`.--## Installation--```bash-# Base installation-pip install outlines--# With specific backends-pip install outlines transformers # Hugging Face models-pip install outlines llama-cpp-python # llama.cpp-pip install outlines vllm # vLLM for high-throughput-```--## Quick Start--### Basic Example: Classification--```python-import outlines-from typing import Literal-from transformers import AutoModelForCausalLM, AutoTokenizer--MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"--# v1: wrap a Transformers model + tokenizer-model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),- AutoTokenizer.from_pretrained(MODEL_NAME),-)--# Call the model directly with an output type-prompt = "Sentiment of 'This product is amazing!': "-sentiment = model(prompt, Literal["positive", "negative", "neutral"])--print(sentiment) # "positive" (guaranteed one of these)-```--### With Pydantic Models--```python-from pydantic import BaseModel-import outlines-from transformers import AutoModelForCausalLM, AutoTokenizer--class User(BaseModel):- name: str- age: int- email: str--MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"-model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),- AutoTokenizer.from_pretrained(MODEL_NAME),-)--# Generate structured output (returns a JSON string)-prompt = "Extract user: John Doe, 30 years old, [email protected]"-result = model(prompt, User, max_new_tokens=200)--user = User.model_validate_json(result) # parse into the Pydantic model-print(user.name) # "John Doe"-print(user.age) # 30-print(user.email) # "[email protected]"-```--## Core Concepts--### 1. Constrained Token Sampling--Outlines constrains token generation at the logit level using a compiled-automaton derived from your output type.--**How it works:**-1. Convert the output type (JSON/Pydantic/regex/`Literal`) to a schema/grammar-2. Compile the grammar into a token-level automaton-3. Filter invalid tokens at each step during generation-4. Fast-forward when only one valid token exists--**Benefits:**-- **Zero overhead**: Filtering happens at token level-- **Speed improvement**: Fast-forward through deterministic paths-- **Guaranteed validity**: Invalid outputs impossible--```python-import outlines-from pydantic import BaseModel-from transformers import AutoModelForCausalLM, AutoTokenizer--class Person(BaseModel):- name: str- age: int--model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),- AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),-)--result = model("Generate person: Alice, 25", Person)-person = Person.model_validate_json(result)-```--### 2. Output Types--In v1 you pass the desired **output type** directly as the second argument.--#### Multiple choice (`Literal`)--```python-from typing import Literal--sentiment = model("Review: This is great!", Literal["positive", "negative", "neutral"])-# Result: one of the three choices-```--#### JSON via Pydantic--```python-from pydantic import BaseModel--class Product(BaseModel):- name: str- price: float- in_stock: bool--result = model("Extract: iPhone 15, $999, available", Product)-product = Product.model_validate_json(result) # valid Product instance-```--#### Regex (pass a regex string)--```python-# Generate text matching a regex pattern-phone = model("Generate phone number:", r"[0-9]{3}-[0-9]{3}-[0-9]{4}")-# Result: "555-123-4567" (guaranteed to match the pattern)-```--#### Numeric types--```python-# Pass the Python type directly-age = model("Person's age:", int) # guaranteed integer-price = model("Product price:", float) # guaranteed float-```--### 3. Model Backends--Outlines supports multiple local and API-based backends via `from_*` factories.--#### Transformers (Hugging Face)--```python-import outlines-from transformers import AutoModelForCausalLM, AutoTokenizer--model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),- AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),-)--result = model(prompt, YourModel)-```--#### llama.cpp--```python-import outlines-from llama_cpp import Llama--llm = Llama("./models/llama-3.1-8b-instruct.Q4_K_M.gguf", n_gpu_layers=35, n_ctx=4096)-model = outlines.from_llamacpp(llm)--result = model(prompt, YourModel)-```--#### vLLM (High Throughput)--```python-import outlines-from vllm import LLM--llm = LLM("meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=2)-model = outlines.from_vllm(llm)--result = model(prompt, YourModel)-```--#### OpenAI (server-side constrained JSON)--```python-import outlines-from openai import OpenAI--client = OpenAI()-model = outlines.from_openai(client, "gpt-4o-mini")--# API backends support JSON-schema style structured output-result = model(prompt, YourModel)-```--### 4. Pydantic Integration--Outlines has first-class Pydantic support with automatic schema translation.-Generation returns a JSON string; call `model_validate_json` to get an instance.--#### Basic Models--```python-from pydantic import BaseModel, Field--class Article(BaseModel):- title: str = Field(description="Article title")- author: str = Field(description="Author name")- word_count: int = Field(description="Number of words", gt=0)- tags: list[str] = Field(description="List of tags")--result = model("Generate article about AI", Article, max_new_tokens=300)-article = Article.model_validate_json(result)-print(article.title)-print(article.word_count) # Guaranteed > 0-```--#### Nested Models--```python-class Address(BaseModel):- street: str- city: str- country: str--class Person(BaseModel):- name: str- age: int- address: Address # Nested model--result = model("Generate person in New York", Person)-person = Person.model_validate_json(result)-print(person.address.city) # "New York"-```--#### Enums and Literals--```python-from enum import Enum-from typing import Literal--class Status(str, Enum):- PENDING = "pending"- APPROVED = "approved"- REJECTED = "rejected"--class Application(BaseModel):- applicant: str- status: Status # Must be one of enum values- priority: Literal["low", "medium", "high"] # Must be one of literals--result = model("Generate application", Application)-app = Application.model_validate_json(result)-print(app.status) # Status.PENDING (or APPROVED/REJECTED)-```--## Common Patterns--### Pattern 1: Data Extraction--```python-from pydantic import BaseModel-import outlines-from transformers import AutoModelForCausalLM, AutoTokenizer--class CompanyInfo(BaseModel):- name: str- founded_year: int- industry: str- employees: int--model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),- AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),-)--text = """-Apple Inc. was founded in 1976 in the technology industry.-The company employs approximately 164,000 people worldwide.-"""--prompt = f"Extract company information:\n{text}\n\nCompany:"-company = CompanyInfo.model_validate_json(model(prompt, CompanyInfo, max_new_tokens=200))--print(f"Name: {company.name}")-print(f"Founded: {company.founded_year}")-print(f"Industry: {company.industry}")-print(f"Employees: {company.employees}")-```--### Pattern 2: Classification--```python-from typing import Literal-from pydantic import BaseModel--# Binary classification-result = model("Email: Buy now! 50% off!", Literal["spam", "not_spam"])--# Multi-class classification-category = model(- "Article: Apple announces new iPhone...",- Literal["technology", "business", "sports", "entertainment"],-)--# With confidence-class Classification(BaseModel):- label: Literal["positive", "negative", "neutral"]- confidence: float--out = model("Review: This product is okay, nothing special", Classification)-result = Classification.model_validate_json(out)-```--### Pattern 3: Structured Forms--```python-class UserProfile(BaseModel):- full_name: str- age: int- email: str- phone: str- country: str- interests: list[str]--prompt = """-Extract user profile from:-Name: Alice Johnson-Age: 28-Email: [email protected]-Phone: 555-0123-Country: USA-Interests: hiking, photography, cooking-"""--profile = UserProfile.model_validate_json(model(prompt, UserProfile, max_new_tokens=250))-print(profile.full_name)-print(profile.interests) # ["hiking", "photography", "cooking"]-```--### Pattern 4: Multi-Entity Extraction--```python-from typing import Literal--class Entity(BaseModel):- name: str- type: Literal["PERSON", "ORGANIZATION", "LOCATION"]--class DocumentEntities(BaseModel):- entities: list[Entity]--text = "Tim Cook met with Satya Nadella at Microsoft headquarters in Redmond."-prompt = f"Extract entities from: {text}"--result = DocumentEntities.model_validate_json(model(prompt, DocumentEntities, max_new_tokens=300))-for entity in result.entities:- print(f"{entity.name} ({entity.type})")-```--### Pattern 5: Code Generation--```python-class PythonFunction(BaseModel):- function_name: str- parameters: list[str]- docstring: str- body: str--prompt = "Generate a Python function to calculate factorial"-func = PythonFunction.model_validate_json(model(prompt, PythonFunction, max_new_tokens=300))--print(f"def {func.function_name}({', '.join(func.parameters)}):")-print(f' """{func.docstring}"""')-print(f" {func.body}")-```--### Pattern 6: Batch Processing--```python-import outlines-from transformers import AutoModelForCausalLM, AutoTokenizer-from pydantic import BaseModel--class Person(BaseModel):- name: str- age: int--model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-4k-instruct", device_map="auto"),- AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct"),-)--texts = [- "John is 30 years old",- "Alice is 25 years old",- "Bob is 40 years old",-]--# v1 accepts a list of prompts for batched generation-prompts = [f"Extract from: {t}" for t in texts]-outputs = model(prompts, Person, max_new_tokens=100)-people = [Person.model_validate_json(o) for o in outputs]-for person in people:- print(f"{person.name}: {person.age}")-```--## Backend Configuration--### Transformers--```python-import outlines-from transformers import AutoModelForCausalLM, AutoTokenizer--MODEL_NAME = "microsoft/Phi-3-mini-4k-instruct"--# Basic usage-model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),- AutoTokenizer.from_pretrained(MODEL_NAME),-)--# GPU + dtype configuration is set on the HF model itself-import torch-model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="cuda", torch_dtype=torch.float16),- AutoTokenizer.from_pretrained(MODEL_NAME),-)--# Popular models-for name in [- "meta-llama/Llama-3.1-8B-Instruct",- "mistralai/Mistral-7B-Instruct-v0.3",- "Qwen/Qwen2.5-7B-Instruct",-]:- model = outlines.from_transformers(- AutoModelForCausalLM.from_pretrained(name, device_map="auto"),- AutoTokenizer.from_pretrained(name),- )-```--### llama.cpp--```python-import outlines-from llama_cpp import Llama--# Load GGUF model-llm = Llama(- "./models/llama-3.1-8b.Q4_K_M.gguf",- n_ctx=4096, # Context window- n_gpu_layers=35, # GPU layers- n_threads=8, # CPU threads-)-model = outlines.from_llamacpp(llm)--# Full GPU offload: set n_gpu_layers=-1 on the Llama object-```--### vLLM (Production)--```python-import outlines-from vllm import LLM--# Single GPU-model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct"))--# Multi-GPU-model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-70B-Instruct", tensor_parallel_size=4))--# With quantization-model = outlines.from_vllm(LLM("meta-llama/Llama-3.1-8B-Instruct", quantization="awq"))-```--## Best Practices--### 1. Use Specific Types--```python-# ✅ Good: Specific types-class Product(BaseModel):- name: str- price: float # Not str- quantity: int # Not str- in_stock: bool # Not str--# ❌ Bad: Everything as string-class Product(BaseModel):- name: str- price: str # Should be float- quantity: str # Should be int-```--### 2. Add Constraints--```python-from pydantic import Field--# ✅ Good: With constraints-class User(BaseModel):- name: str = Field(min_length=1, max_length=100)- age: int = Field(ge=0, le=120)- email: str = Field(pattern=r"^[\w\.-]+@[\w\.-]+\.\w+$")--# ❌ Bad: No constraints-class User(BaseModel):- name: str- age: int- email: str-```--### 3. Use Enums for Categories--```python-# ✅ Good: Enum for fixed set-class Priority(str, Enum):- LOW = "low"- MEDIUM = "medium"- HIGH = "high"--class Task(BaseModel):- title: str- priority: Priority--# ❌ Bad: Free-form string-class Task(BaseModel):- title: str- priority: str # Can be anything-```--### 4. Provide Context in Prompts--```python-# ✅ Good: Clear context-prompt = """-Extract product information from the following text.-Text: iPhone 15 Pro costs $999 and is currently in stock.-Product:-"""--# ❌ Bad: Minimal context-prompt = "iPhone 15 Pro costs $999 and is currently in stock."-```--### 5. Handle Optional Fields--```python-from typing import Optional--# ✅ Good: Optional fields for incomplete data-class Article(BaseModel):- title: str # Required- author: Optional[str] = None # Optional- date: Optional[str] = None # Optional- tags: list[str] = [] # Default empty list--# Can succeed even if author/date missing-```--### 6. Always Validate JSON Output--```python-# v1 returns a JSON string for Pydantic/JSON output types.-result = model(prompt, Article) # str-article = Article.model_validate_json(result) # Article instance-```--## Comparison to Alternatives--| Feature | Outlines | Instructor | Guidance | LMQL |-|---------|----------|------------|----------|------|-| Pydantic Support | ✅ Native | ✅ Native | ✅ Yes | ❌ No |-| JSON Schema | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |-| Regex Constraints | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |-| Local Models | ✅ Full | ⚠️ Limited | ✅ Full | ✅ Full |-| API Models | ✅ Yes | ✅ Full | ✅ Yes | ✅ Full |-| Zero Overhead | ✅ Yes | ❌ No | ⚠️ Partial | ✅ Yes |-| Automatic Retrying | ❌ No | ✅ Yes | ❌ No | ❌ No |-| Learning Curve | Low | Low | Low | High |--**When to choose Outlines:**-- Using local models (Transformers, llama.cpp, vLLM)-- Need maximum inference speed-- Want Pydantic model support-- Require zero-overhead structured generation-- Control token sampling process--**When to choose alternatives:**-- Instructor: Need API models with automatic retrying-- Guidance: Need token healing and complex workflows-- LMQL: Prefer declarative query syntax--## Performance Characteristics--**Speed:**-- **Zero overhead**: Structured generation as fast as unconstrained-- **Fast-forward optimization**: Skips deterministic tokens-- **1.2-2x faster** than post-generation validation approaches--**Memory:**-- Automaton compiled once per output type (cached)-- Minimal runtime overhead-- Efficient with vLLM for high throughput--**Accuracy:**-- **100% valid outputs** (guaranteed by the constrained automaton)-- No retry loops needed-- Deterministic token filtering--## Resources--- **Documentation**: https://dottxt-ai.github.io/outlines/-- **GitHub**: https://github.com/dottxt-ai/outlines (12k+ stars)-- **Discord**: https://discord.gg/R9DSu34mGd-- **Blog**: https://blog.dottxt.co--## See Also--- `references/json_generation.md` - Comprehensive JSON and Pydantic patterns-- `references/backends.md` - Backend-specific configuration-- `references/examples.md` - Production-ready examplesdiff --git a/optional-skills/mlops/inference/outlines/references/backends.md b/optional-skills/mlops/inference/outlines/references/backends.mddeleted file mode 100644index f019f121437..00000000000--- a/optional-skills/mlops/inference/outlines/references/backends.md+++ /dev/null@@ -1,615 +0,0 @@-# Backend Configuration Guide--Complete guide to configuring Outlines with different model backends.--## Table of Contents-- Local Models (Transformers, llama.cpp, vLLM)-- API Models (OpenAI)-- Performance Comparison-- Configuration Examples-- Production Deployment--## Transformers (Hugging Face)--### Basic Setup--```python-import outlines--# Load model from Hugging Face-model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")--# Use with generator-generator = outlines.generate.json(model, YourModel)-result = generator("Your prompt")-```--### GPU Configuration--```python-# Use CUDA GPU-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="cuda"-)--# Use specific GPU-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="cuda:0" # GPU 0-)--# Use CPU-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="cpu"-)--# Use Apple Silicon MPS-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="mps"-)-```--### Advanced Configuration--```python-# FP16 for faster inference-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="cuda",- model_kwargs={- "torch_dtype": "float16"- }-)--# 8-bit quantization (less memory)-model = outlines.models.transformers(- "microsoft/Phi-3-mini-4k-instruct",- device="cuda",- model_kwargs={- "load_in_8bit": True,- "device_map": "auto"- }-)--# 4-bit quantization (even less memory)-model = outlines.models.transformers(- "meta-llama/Llama-3.1-70B-Instruct",- device="cuda",- model_kwargs={- "load_in_4bit": True,- "device_map": "auto",- "bnb_4bit_compute_dtype": "float16"- }-)--# Multi-GPU-model = outlines.models.transformers(- "meta-llama/Llama-3.1-70B-Instruct",- device="cuda",- model_kwargs={- "device_map": "auto", # Automatic GPU distribution- "max_memory": {0: "40GB", 1: "40GB"} # Per-GPU limits- }-)-```--### Popular Models--```python-# Phi-4 (Microsoft)-model = outlines.models.transformers("microsoft/Phi-4-mini-instruct")-model = outlines.models.transformers("microsoft/Phi-3-medium-4k-instruct")--# Llama 3.1 (Meta)-model = outlines.models.transformers("meta-llama/Llama-3.1-8B-Instruct")-model = outlines.models.transformers("meta-llama/Llama-3.1-70B-Instruct")-model = outlines.models.transformers("meta-llama/Llama-3.1-405B-Instruct")--# Mistral (Mistral AI)-model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.3")-model = outlines.models.transformers("mistralai/Mixtral-8x7B-Instruct-v0.1")-model = outlines.models.transformers("mistralai/Mixtral-8x22B-Instruct-v0.1")--# Qwen (Alibaba)-model = outlines.models.transformers("Qwen/Qwen2.5-7B-Instruct")-model = outlines.models.transformers("Qwen/Qwen2.5-14B-Instruct")-model = outlines.models.transformers("Qwen/Qwen2.5-72B-Instruct")--# Gemma (Google)-model = outlines.models.transformers("google/gemma-2-9b-it")-model = outlines.models.transformers("google/gemma-2-27b-it")--# Llava (Vision)-model = outlines.models.transformers("llava-hf/llava-v1.6-mistral-7b-hf")-```--### Custom Model Loading--```python-from transformers import AutoTokenizer, AutoModelForCausalLM-import outlines--# Load model manually-tokenizer = AutoTokenizer.from_pretrained("your-model")-model_hf = AutoModelForCausalLM.from_pretrained(- "your-model",- device_map="auto",- torch_dtype="float16"-)--# Use with Outlines-model = outlines.models.transformers(- model=model_hf,- tokenizer=tokenizer-)-```--## llama.cpp--### Basic Setup--```python-import outlines--# Load GGUF model-model = outlines.models.llamacpp(- "./models/llama-3.1-8b-instruct.Q4_K_M.gguf",- n_ctx=4096 # Context window-)--# Use with generator-generator = outlines.generate.json(model, YourModel)-```--### GPU Configuration--```python-# CPU only-model = outlines.models.llamacpp(- "./models/model.gguf",- n_ctx=4096,- n_threads=8 # Use 8 CPU threads-)--# GPU offload (partial)-model = outlines.models.llamacpp(- "./models/model.gguf",- n_ctx=4096,- n_gpu_layers=35, # Offload 35 layers to GPU- n_threads=4 # CPU threads for remaining layers-)--# Full GPU offload-model = outlines.models.llamacpp(- "./models/model.gguf",- n_ctx=8192,- n_gpu_layers=-1 # All layers on GPU-)-```--### Advanced Configuration--```python-model = outlines.models.llamacpp(- "./models/llama-3.1-8b.Q4_K_M.gguf",- n_ctx=8192, # Context window (tokens)- n_gpu_layers=35, # GPU layers- n_threads=8, # CPU threads- n_batch=512, # Batch size for prompt processing- use_mmap=True, # Memory-map model file (faster loading)- use_mlock=False, # Lock model in RAM (prevents swapping)- seed=42, # Random seed for reproducibility- verbose=False # Suppress verbose output-)-```--### Quantization Formats--```python-# Q4_K_M (4-bit, recommended for most cases)-# - Size: ~4.5GB for 7B model-# - Quality: Good-# - Speed: Fast-model = outlines.models.llamacpp("./models/model.Q4_K_M.gguf")--# Q5_K_M (5-bit, better quality)-# - Size: ~5.5GB for 7B model-# - Quality: Very good-# - Speed: Slightly slower than Q4-model = outlines.models.llamacpp("./models/model.Q5_K_M.gguf")--# Q6_K (6-bit, high quality)-# - Size: ~6.5GB for 7B model-# - Quality: Excellent-# - Speed: Slower than Q5-model = outlines.models.llamacpp("./models/model.Q6_K.gguf")--# Q8_0 (8-bit, near-original quality)-# - Size: ~8GB for 7B model-# - Quality: Near FP16-# - Speed: Slower than Q6-model = outlines.models.llamacpp("./models/model.Q8_0.gguf")--# F16 (16-bit float, original quality)-# - Size: ~14GB for 7B model-# - Quality: Original-# - Speed: Slowest-model = outlines.models.llamacpp("./models/model.F16.gguf")-```--### Popular GGUF Models--```python-# Llama 3.1-model = outlines.models.llamacpp("llama-3.1-8b-instruct.Q4_K_M.gguf")-model = outlines.models.llamacpp("llama-3.1-70b-instruct.Q4_K_M.gguf")--# Mistral-model = outlines.models.llamacpp("mistral-7b-instruct-v0.3.Q4_K_M.gguf")--# Phi-4-model = outlines.models.llamacpp("phi-4-mini-instruct.Q4_K_M.gguf")--# Qwen-model = outlines.models.llamacpp("qwen2.5-7b-instruct.Q4_K_M.gguf")-```--### Apple Silicon Optimization--```python-# Optimized for M1/M2/M3 Macs-model = outlines.models.llamacpp(- "./models/llama-3.1-8b.Q4_K_M.gguf",- n_ctx=4096,- n_gpu_layers=-1, # Use Metal GPU acceleration- use_mmap=True, # Efficient memory mapping- n_threads=8 # Use performance cores-)-```--## vLLM (Production)--### Basic Setup--```python-import outlines--# Load model with vLLM-model = outlines.models.vllm("meta-llama/Llama-3.1-8B-Instruct")--# Use with generator-generator = outlines.generate.json(model, YourModel)-```--### Single GPU--```python-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- gpu_memory_utilization=0.9, # Use 90% of GPU memory- max_model_len=4096 # Max sequence length-)-```--### Multi-GPU--```python-# Tensor parallelism (split model across GPUs)-model = outlines.models.vllm(- "meta-llama/Llama-3.1-70B-Instruct",- tensor_parallel_size=4, # Use 4 GPUs- gpu_memory_utilization=0.9-)--# Pipeline parallelism (rare, for very large models)-model = outlines.models.vllm(- "meta-llama/Llama-3.1-405B-Instruct",- pipeline_parallel_size=8, # 8-GPU pipeline- tensor_parallel_size=4 # 4-GPU tensor split- # Total: 32 GPUs-)-```--### Quantization--```python-# AWQ quantization (4-bit)-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- quantization="awq",- dtype="float16"-)--# GPTQ quantization (4-bit)-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- quantization="gptq"-)--# SqueezeLLM quantization-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- quantization="squeezellm"-)-```--### Advanced Configuration--```python-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- tensor_parallel_size=1,- gpu_memory_utilization=0.9,- max_model_len=8192,- max_num_seqs=256, # Max concurrent sequences- max_num_batched_tokens=8192, # Max tokens per batch- dtype="float16",- trust_remote_code=True,- enforce_eager=False, # Use CUDA graphs (faster)- swap_space=4 # CPU swap space (GB)-)-```--### Batch Processing--```python-# vLLM optimized for high-throughput batch processing-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- max_num_seqs=128 # Process 128 sequences in parallel-)--generator = outlines.generate.json(model, YourModel)--# Process many prompts efficiently-prompts = ["prompt1", "prompt2", ..., "prompt100"]-results = [generator(p) for p in prompts]-# vLLM automatically batches and optimizes-```--## OpenAI (Limited Support)--### Basic Setup--```python-import outlines--# Basic OpenAI support-model = outlines.models.openai("gpt-4o-mini", api_key="your-api-key")--# Use with generator-generator = outlines.generate.json(model, YourModel)-result = generator("Your prompt")-```--### Configuration--```python-model = outlines.models.openai(- "gpt-4o-mini",- api_key="your-api-key", # Or set OPENAI_API_KEY env var- max_tokens=2048,- temperature=0.7-)-```--### Available Models--```python-# GPT-4o (latest)-model = outlines.models.openai("gpt-4o")--# GPT-4o Mini (cost-effective)-model = outlines.models.openai("gpt-4o-mini")--# GPT-4 Turbo-model = outlines.models.openai("gpt-4-turbo")--# GPT-3.5 Turbo-model = outlines.models.openai("gpt-3.5-turbo")-```--**Note**: OpenAI support is limited compared to local models. Some advanced features may not work.--## Backend Comparison--### Feature Matrix--| Feature | Transformers | llama.cpp | vLLM | OpenAI |-|---------|-------------|-----------|------|--------|-| Structured Generation | ✅ Full | ✅ Full | ✅ Full | ⚠️ Limited |-| FSM Optimization | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No |-| GPU Support | ✅ Yes | ✅ Yes | ✅ Yes | N/A |-| Multi-GPU | ✅ Yes | ✅ Yes | ✅ Yes | N/A |-| Quantization | ✅ Yes | ✅ Yes | ✅ Yes | N/A |-| High Throughput | ⚠️ Medium | ⚠️ Medium | ✅ Excellent | ⚠️ API-limited |-| Setup Difficulty | Easy | Medium | Medium | Easy |-| Cost | Hardware | Hardware | Hardware | API usage |--### Performance Characteristics--**Transformers:**-- **Latency**: 50-200ms (single request, GPU)-- **Throughput**: 10-50 tokens/sec (depends on hardware)-- **Memory**: 2-4GB per 1B parameters (FP16)-- **Best for**: Development, small-scale deployment, flexibility--**llama.cpp:**-- **Latency**: 30-150ms (single request)-- **Throughput**: 20-150 tokens/sec (depends on quantization)-- **Memory**: 0.5-2GB per 1B parameters (Q4-Q8)-- **Best for**: CPU inference, Apple Silicon, edge deployment, low memory--**vLLM:**-- **Latency**: 30-100ms (single request)-- **Throughput**: 100-1000+ tokens/sec (batch processing)-- **Memory**: 2-4GB per 1B parameters (FP16)-- **Best for**: Production, high-throughput, batch processing, serving--**OpenAI:**-- **Latency**: 200-500ms (API call)-- **Throughput**: API rate limits-- **Memory**: N/A (cloud-based)-- **Best for**: Quick prototyping, no infrastructure--### Memory Requirements--**7B Model:**-- FP16: ~14GB-- 8-bit: ~7GB-- 4-bit: ~4GB-- Q4_K_M (GGUF): ~4.5GB--**13B Model:**-- FP16: ~26GB-- 8-bit: ~13GB-- 4-bit: ~7GB-- Q4_K_M (GGUF): ~8GB--**70B Model:**-- FP16: ~140GB (multi-GPU)-- 8-bit: ~70GB (multi-GPU)-- 4-bit: ~35GB (single A100/H100)-- Q4_K_M (GGUF): ~40GB--## Performance Tuning--### Transformers Optimization--```python-# Use FP16-model = outlines.models.transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- model_kwargs={"torch_dtype": "float16"}-)--# Use flash attention (2-4x faster)-model = outlines.models.transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- model_kwargs={- "torch_dtype": "float16",- "use_flash_attention_2": True- }-)--# Use 8-bit quantization (2x less memory)-model = outlines.models.transformers(- "meta-llama/Llama-3.1-8B-Instruct",- device="cuda",- model_kwargs={- "load_in_8bit": True,- "device_map": "auto"- }-)-```--### llama.cpp Optimization--```python-# Maximize GPU usage-model = outlines.models.llamacpp(- "./models/model.Q4_K_M.gguf",- n_gpu_layers=-1, # All layers on GPU- n_ctx=8192,- n_batch=512 # Larger batch = faster-)--# Optimize for CPU (Apple Silicon)-model = outlines.models.llamacpp(- "./models/model.Q4_K_M.gguf",- n_ctx=4096,- n_threads=8, # Use all performance cores- use_mmap=True-)-```--### vLLM Optimization--```python-# High throughput-model = outlines.models.vllm(- "meta-llama/Llama-3.1-8B-Instruct",- gpu_memory_utilization=0.95, # Use 95% of GPU- max_num_seqs=256, # High concurrency- enforce_eager=False # Use CUDA graphs-)--# Multi-GPU-model = outlines.models.vllm(- "meta-llama/Llama-3.1-70B-Instruct",- tensor_parallel_size=4, # 4 GPUs- gpu_memory_utilization=0.9-)-```--## Production Deployment--### Docker with vLLM--```dockerfile-FROM vllm/vllm-openai:latest--# Install outlines-RUN pip install outlines--# Copy your code-COPY app.py /app/--# Run-CMD ["python", "/app/app.py"]-```--### Environment Variables--```bash-# Transformers cache-export HF_HOME="/path/to/cache"-export TRANSFORMERS_CACHE="/path/to/cache"--# GPU selection-export CUDA_VISIBLE_DEVICES=0,1,2,3--# OpenAI API key-export OPENAI_API_KEY="sk-..."--# Disable tokenizers parallelism warning-export TOKENIZERS_PARALLELISM=false-```--### Model Serving--```python-# Simple HTTP server with vLLM-import outlines-from fastapi import FastAPI-from pydantic import BaseModel--app = FastAPI()--# Load model once at startup-model = outlines.models.vllm("meta-llama/Llama-3.1-8B-Instruct")--class User(BaseModel):- name: str- age: int- email: str--generator = outlines.generate.json(model, User)-[email protected]("/extract")-def extract(text: str):- result = generator(f"Extract user from: {text}")- return result.model_dump()-```--## Resources--- **Transformers**: https://huggingface.co/docs/transformers-- **llama.cpp**: https://github.com/ggerganov/llama.cpp-- **vLLM**: https://docs.vllm.ai-- **Outlines**: https://github.com/outlines-dev/outlinesdiff --git a/optional-skills/mlops/inference/outlines/references/examples.md b/optional-skills/mlops/inference/outlines/references/examples.mddeleted file mode 100644index c32ecdfcc7e..00000000000--- a/optional-skills/mlops/inference/outlines/references/examples.md+++ /dev/null@@ -1,773 +0,0 @@-# Production-Ready Examples--Real-world examples of using Outlines for structured generation in production systems.--## Table of Contents-- Data Extraction-- Classification Systems-- Form Processing-- Multi-Entity Extraction-- Code Generation-- Batch Processing-- Production Patterns--## Data Extraction--### Basic Information Extraction--```python-from pydantic import BaseModel, Field-import outlines--class PersonInfo(BaseModel):- name: str = Field(description="Full name")- age: int = Field(ge=0, le=120)- occupation: str- email: str = Field(pattern=r"^[\w\.-]+@[\w\.-]+\.\w+$")- location: str--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, PersonInfo)--text = """-Dr. Sarah Johnson is a 42-year-old research scientist at MIT.-She can be reached at [email protected] and currently lives in Cambridge, MA.-"""--prompt = f"Extract person information from:\n{text}\n\nPerson:"-person = generator(prompt)--print(f"Name: {person.name}")-print(f"Age: {person.age}")-print(f"Occupation: {person.occupation}")-print(f"Email: {person.email}")-print(f"Location: {person.location}")-```--### Company Information--```python-class CompanyInfo(BaseModel):- name: str- founded_year: int = Field(ge=1800, le=2025)- industry: str- headquarters: str- employees: int = Field(gt=0)- revenue: Optional[str] = None--model = outlines.models.transformers("meta-llama/Llama-3.1-8B-Instruct")-generator = outlines.generate.json(model, CompanyInfo)--text = """-Tesla, Inc. was founded in 2003 and operates primarily in the automotive-and energy industries. The company is headquartered in Austin, Texas,-and employs approximately 140,000 people worldwide.-"""--company = generator(f"Extract company information:\n{text}\n\nCompany:")--print(f"Company: {company.name}")-print(f"Founded: {company.founded_year}")-print(f"Industry: {company.industry}")-print(f"HQ: {company.headquarters}")-print(f"Employees: {company.employees:,}")-```--### Product Specifications--```python-class ProductSpec(BaseModel):- name: str- brand: str- price: float = Field(gt=0)- dimensions: str- weight: str- features: list[str]- rating: Optional[float] = Field(None, ge=0, le=5)--generator = outlines.generate.json(model, ProductSpec)--text = """-The Apple iPhone 15 Pro is priced at $999. It measures 146.6 x 70.6 x 8.25 mm-and weighs 187 grams. Key features include the A17 Pro chip, titanium design,-action button, and USB-C port. It has an average customer rating of 4.5 stars.-"""--product = generator(f"Extract product specifications:\n{text}\n\nProduct:")--print(f"Product: {product.brand} {product.name}")-print(f"Price: ${product.price}")-print(f"Features: {', '.join(product.features)}")-```--## Classification Systems--### Sentiment Analysis--```python-from typing import Literal-from enum import Enum--class Sentiment(str, Enum):- VERY_POSITIVE = "very_positive"- POSITIVE = "positive"- NEUTRAL = "neutral"- NEGATIVE = "negative"- VERY_NEGATIVE = "very_negative"--class SentimentAnalysis(BaseModel):- text: str- sentiment: Sentiment- confidence: float = Field(ge=0.0, le=1.0)- aspects: list[str] # What aspects were mentioned- reasoning: str--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, SentimentAnalysis)--review = """-This product completely exceeded my expectations! The build quality is-outstanding, and customer service was incredibly helpful. My only minor-complaint is the packaging could be better.-"""--result = generator(f"Analyze sentiment:\n{review}\n\nAnalysis:")--print(f"Sentiment: {result.sentiment.value}")-print(f"Confidence: {result.confidence:.2%}")-print(f"Aspects: {', '.join(result.aspects)}")-print(f"Reasoning: {result.reasoning}")-```--### Content Classification--```python-class Category(str, Enum):- TECHNOLOGY = "technology"- BUSINESS = "business"- SCIENCE = "science"- POLITICS = "politics"- ENTERTAINMENT = "entertainment"- SPORTS = "sports"- HEALTH = "health"--class ArticleClassification(BaseModel):- primary_category: Category- secondary_categories: list[Category]- keywords: list[str] = Field(min_items=3, max_items=10)- target_audience: Literal["general", "expert", "beginner"]- reading_level: Literal["elementary", "intermediate", "advanced"]--generator = outlines.generate.json(model, ArticleClassification)--article = """-Apple announced groundbreaking advancements in its AI capabilities with the-release of iOS 18. The new features leverage machine learning to significantly-improve battery life and overall device performance. Industry analysts predict-this will strengthen Apple's position in the competitive smartphone market.-"""--classification = generator(f"Classify article:\n{article}\n\nClassification:")--print(f"Primary: {classification.primary_category.value}")-print(f"Secondary: {[c.value for c in classification.secondary_categories]}")-print(f"Keywords: {classification.keywords}")-print(f"Audience: {classification.target_audience}")-```--### Intent Recognition--```python-class Intent(str, Enum):- QUESTION = "question"- COMPLAINT = "complaint"- REQUEST = "request"- FEEDBACK = "feedback"- CANCEL = "cancel"- UPGRADE = "upgrade"--class UserMessage(BaseModel):- original_message: str- intent: Intent- urgency: Literal["low", "medium", "high", "critical"]- department: Literal["support", "sales", "billing", "technical"]- sentiment: Literal["positive", "neutral", "negative"]- action_required: bool- summary: str--generator = outlines.generate.json(model, UserMessage)--message = """-I've been charged twice for my subscription this month! This is the third-time this has happened. I need someone to fix this immediately and refund-the extra charge. Very disappointed with this service.-"""--result = generator(f"Analyze message:\n{message}\n\nAnalysis:")--print(f"Intent: {result.intent.value}")-print(f"Urgency: {result.urgency}")-print(f"Route to: {result.department}")-print(f"Action required: {result.action_required}")-print(f"Summary: {result.summary}")-```--## Form Processing--### Job Application--```python-class Education(BaseModel):- degree: str- field: str- institution: str- year: int--class Experience(BaseModel):- title: str- company: str- duration: str- responsibilities: list[str]--class JobApplication(BaseModel):- full_name: str- email: str- phone: str- education: list[Education]- experience: list[Experience]- skills: list[str]- availability: str--model = outlines.models.transformers("meta-llama/Llama-3.1-8B-Instruct")-generator = outlines.generate.json(model, JobApplication)--resume_text = """-John Smith-Email: [email protected] | Phone: 555-0123--EDUCATION-- BS in Computer Science, MIT, 2018-- MS in Artificial Intelligence, Stanford, 2020--EXPERIENCE-Software Engineer, Google (2020-2023)-- Developed ML pipelines for search ranking-- Led team of 5 engineers-- Improved search quality by 15%--SKILLS: Python, Machine Learning, TensorFlow, System Design--AVAILABILITY: Immediate-"""--application = generator(f"Extract job application:\n{resume_text}\n\nApplication:")--print(f"Applicant: {application.full_name}")-print(f"Email: {application.email}")-print(f"Education: {len(application.education)} degrees")-for edu in application.education:- print(f" - {edu.degree} in {edu.field}, {edu.institution} ({edu.year})")-print(f"Experience: {len(application.experience)} positions")-```--### Invoice Processing--```python-class InvoiceItem(BaseModel):- description: str- quantity: int = Field(gt=0)- unit_price: float = Field(gt=0)- total: float = Field(gt=0)--class Invoice(BaseModel):- invoice_number: str- date: str = Field(pattern=r"\d{4}-\d{2}-\d{2}")- vendor: str- customer: str- items: list[InvoiceItem]- subtotal: float = Field(gt=0)- tax: float = Field(ge=0)- total: float = Field(gt=0)--generator = outlines.generate.json(model, Invoice)--invoice_text = """-INVOICE #INV-2024-001-Date: 2024-01-15--From: Acme Corp-To: Smith & Co--Items:-- Widget A: 10 units @ $50.00 = $500.00-- Widget B: 5 units @ $75.00 = $375.00-- Service Fee: 1 @ $100.00 = $100.00--Subtotal: $975.00-Tax (8%): $78.00-TOTAL: $1,053.00-"""--invoice = generator(f"Extract invoice:\n{invoice_text}\n\nInvoice:")--print(f"Invoice: {invoice.invoice_number}")-print(f"From: {invoice.vendor} → To: {invoice.customer}")-print(f"Items: {len(invoice.items)}")-for item in invoice.items:- print(f" - {item.description}: {item.quantity} × ${item.unit_price} = ${item.total}")-print(f"Total: ${invoice.total}")-```--### Survey Responses--```python-class SurveyResponse(BaseModel):- respondent_id: str- completion_date: str- satisfaction: Literal[1, 2, 3, 4, 5]- would_recommend: bool- favorite_features: list[str]- improvement_areas: list[str]- additional_comments: Optional[str] = None--generator = outlines.generate.json(model, SurveyResponse)--survey_text = """-Survey ID: RESP-12345-Completed: 2024-01-20--How satisfied are you with our product? 4 out of 5--Would you recommend to a friend? Yes--What features do you like most?-- Fast performance-- Easy to use-- Great customer support--What could we improve?-- Better documentation-- More integrations--Additional feedback: Overall great product, keep up the good work!-"""--response = generator(f"Extract survey response:\n{survey_text}\n\nResponse:")--print(f"Respondent: {response.respondent_id}")-print(f"Satisfaction: {response.satisfaction}/5")-print(f"Would recommend: {response.would_recommend}")-print(f"Favorite features: {response.favorite_features}")-print(f"Improvement areas: {response.improvement_areas}")-```--## Multi-Entity Extraction--### News Article Entities--```python-class Person(BaseModel):- name: str- role: Optional[str] = None- affiliation: Optional[str] = None--class Organization(BaseModel):- name: str- type: Optional[str] = None--class Location(BaseModel):- name: str- type: Literal["city", "state", "country", "region"]--class Event(BaseModel):- name: str- date: Optional[str] = None- location: Optional[str] = None--class ArticleEntities(BaseModel):- people: list[Person]- organizations: list[Organization]- locations: list[Location]- events: list[Event]- dates: list[str]--model = outlines.models.transformers("meta-llama/Llama-3.1-8B-Instruct")-generator = outlines.generate.json(model, ArticleEntities)--article = """-Apple CEO Tim Cook met with Microsoft CEO Satya Nadella at Microsoft-headquarters in Redmond, Washington on September 15, 2024, to discuss-potential collaboration opportunities. The meeting was attended by executives-from both companies and focused on AI integration strategies. Apple's-Cupertino offices will host a follow-up meeting on October 20, 2024.-"""--entities = generator(f"Extract all entities:\n{article}\n\nEntities:")--print("People:")-for person in entities.people:- print(f" - {person.name} ({person.role}) @ {person.affiliation}")--print("\nOrganizations:")-for org in entities.organizations:- print(f" - {org.name} ({org.type})")--print("\nLocations:")-for loc in entities.locations:- print(f" - {loc.name} ({loc.type})")--print("\nEvents:")-for event in entities.events:- print(f" - {event.name} on {event.date}")-```--### Document Metadata--```python-class Author(BaseModel):- name: str- email: Optional[str] = None- affiliation: Optional[str] = None--class Reference(BaseModel):- title: str- authors: list[str]- year: int- source: str--class DocumentMetadata(BaseModel):- title: str- authors: list[Author]- abstract: str- keywords: list[str]- publication_date: str- journal: str- doi: Optional[str] = None- references: list[Reference]--generator = outlines.generate.json(model, DocumentMetadata)--paper = """-Title: Advances in Neural Machine Translation--Authors:-- Dr. Jane Smith ([email protected]), MIT-- Prof. John Doe ([email protected]), Stanford University--Abstract: This paper presents novel approaches to neural machine translation-using transformer architectures. We demonstrate significant improvements in-translation quality across multiple language pairs.--Keywords: Neural Networks, Machine Translation, Transformers, NLP--Published: Journal of AI Research, 2024-03-15-DOI: 10.1234/jair.2024.001--References:-1. "Attention Is All You Need" by Vaswani et al., 2017, NeurIPS-2. "BERT: Pre-training of Deep Bidirectional Transformers" by Devlin et al., 2019, NAACL-"""--metadata = generator(f"Extract document metadata:\n{paper}\n\nMetadata:")--print(f"Title: {metadata.title}")-print(f"Authors: {', '.join(a.name for a in metadata.authors)}")-print(f"Keywords: {', '.join(metadata.keywords)}")-print(f"References: {len(metadata.references)}")-```--## Code Generation--### Python Function Generation--```python-class Parameter(BaseModel):- name: str = Field(pattern=r"^[a-z_][a-z0-9_]*$")- type_hint: str- default: Optional[str] = None--class PythonFunction(BaseModel):- function_name: str = Field(pattern=r"^[a-z_][a-z0-9_]*$")- parameters: list[Parameter]- return_type: str- docstring: str- body: list[str] # Lines of code--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, PythonFunction)--spec = "Create a function to calculate the factorial of a number"--func = generator(f"Generate Python function:\n{spec}\n\nFunction:")--print(f"def {func.function_name}(", end="")-print(", ".join(f"{p.name}: {p.type_hint}" for p in func.parameters), end="")-print(f") -> {func.return_type}:")-print(f' """{func.docstring}"""')-for line in func.body:- print(f" {line}")-```--### SQL Query Generation--```python-class SQLQuery(BaseModel):- query_type: Literal["SELECT", "INSERT", "UPDATE", "DELETE"]- select_columns: Optional[list[str]] = None- from_tables: list[str]- joins: Optional[list[str]] = None- where_conditions: Optional[list[str]] = None- group_by: Optional[list[str]] = None- order_by: Optional[list[str]] = None- limit: Optional[int] = None--generator = outlines.generate.json(model, SQLQuery)--request = "Get top 10 users who made purchases in the last 30 days, ordered by total spent"--sql = generator(f"Generate SQL query:\n{request}\n\nQuery:")--print(f"Query type: {sql.query_type}")-print(f"SELECT {', '.join(sql.select_columns)}")-print(f"FROM {', '.join(sql.from_tables)}")-if sql.joins:- for join in sql.joins:- print(f" {join}")-if sql.where_conditions:- print(f"WHERE {' AND '.join(sql.where_conditions)}")-if sql.order_by:- print(f"ORDER BY {', '.join(sql.order_by)}")-if sql.limit:- print(f"LIMIT {sql.limit}")-```--### API Endpoint Spec--```python-class Parameter(BaseModel):- name: str- type: str- required: bool- description: str--class APIEndpoint(BaseModel):- method: Literal["GET", "POST", "PUT", "DELETE", "PATCH"]- path: str- description: str- parameters: list[Parameter]- request_body: Optional[dict] = None- response_schema: dict- status_codes: dict[int, str]--generator = outlines.generate.json(model, APIEndpoint)--spec = "Create user endpoint"--endpoint = generator(f"Generate API endpoint:\n{spec}\n\nEndpoint:")--print(f"{endpoint.method} {endpoint.path}")-print(f"Description: {endpoint.description}")-print("\nParameters:")-for param in endpoint.parameters:- req = "required" if param.required else "optional"- print(f" - {param.name} ({param.type}, {req}): {param.description}")-```--## Batch Processing--### Parallel Extraction--```python-def batch_extract(texts: list[str], schema: type[BaseModel], model_name: str):- """Extract structured data from multiple texts."""- model = outlines.models.transformers(model_name)- generator = outlines.generate.json(model, schema)-- results = []- for i, text in enumerate(texts):- print(f"Processing {i+1}/{len(texts)}...", end="\r")- result = generator(f"Extract:\n{text}\n\nData:")- results.append(result)-- return results--class Product(BaseModel):- name: str- price: float- category: str--texts = [- "iPhone 15 Pro costs $999 in Electronics",- "Running Shoes are $89.99 in Sports",- "Coffee Maker priced at $49.99 in Home & Kitchen"-]--products = batch_extract(texts, Product, "microsoft/Phi-3-mini-4k-instruct")--for product in products:- print(f"{product.name}: ${product.price} ({product.category})")-```--### CSV Processing--```python-import csv--def process_csv(csv_file: str, schema: type[BaseModel]):- """Process CSV file and extract structured data."""- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)-- results = []- with open(csv_file, 'r') as f:- reader = csv.DictReader(f)- for row in reader:- text = " | ".join(f"{k}: {v}" for k, v in row.items())- result = generator(f"Extract:\n{text}\n\nData:")- results.append(result)-- return results--class Customer(BaseModel):- name: str- email: str- tier: Literal["basic", "premium", "enterprise"]- mrr: float--# customers = process_csv("customers.csv", Customer)-```--## Production Patterns--### Error Handling--```python-from pydantic import ValidationError--def safe_extract(text: str, schema: type[BaseModel], retries: int = 3):- """Extract with error handling and retries."""- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)-- for attempt in range(retries):- try:- result = generator(f"Extract:\n{text}\n\nData:")- return result- except ValidationError as e:- print(f"Attempt {attempt + 1} failed: {e}")- if attempt == retries - 1:- raise- except Exception as e:- print(f"Unexpected error: {e}")- if attempt == retries - 1:- raise-- return None-```--### Caching--```python-from functools import lru_cache-import hashlib--@lru_cache(maxsize=1000)-def cached_extract(text_hash: str, schema_name: str):- """Cache extraction results."""- # This would be called with actual extraction logic- pass--def extract_with_cache(text: str, schema: type[BaseModel]):- """Extract with caching."""- text_hash = hashlib.md5(text.encode()).hexdigest()- schema_name = schema.__name__-- cached_result = cached_extract(text_hash, schema_name)- if cached_result:- return cached_result-- # Perform actual extraction- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)- result = generator(f"Extract:\n{text}\n\nData:")-- return result-```--### Monitoring--```python-import time-import logging--logging.basicConfig(level=logging.INFO)-logger = logging.getLogger(__name__)--def monitored_extract(text: str, schema: type[BaseModel]):- """Extract with monitoring and logging."""- start_time = time.time()-- try:- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)-- result = generator(f"Extract:\n{text}\n\nData:")-- elapsed = time.time() - start_time- logger.info(f"Extraction succeeded in {elapsed:.2f}s")- logger.info(f"Input length: {len(text)} chars")-- return result-- except Exception as e:- elapsed = time.time() - start_time- logger.error(f"Extraction failed after {elapsed:.2f}s: {e}")- raise-```--### Rate Limiting--```python-import time-from threading import Lock--class RateLimiter:- def __init__(self, max_requests: int, time_window: int):- self.max_requests = max_requests- self.time_window = time_window- self.requests = []- self.lock = Lock()-- def wait_if_needed(self):- with self.lock:- now = time.time()- # Remove old requests- self.requests = [r for r in self.requests if now - r < self.time_window]-- if len(self.requests) >= self.max_requests:- sleep_time = self.time_window - (now - self.requests[0])- time.sleep(sleep_time)- self.requests = []-- self.requests.append(now)--def rate_limited_extract(texts: list[str], schema: type[BaseModel]):- """Extract with rate limiting."""- limiter = RateLimiter(max_requests=10, time_window=60) # 10 req/min- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)-- results = []- for text in texts:- limiter.wait_if_needed()- result = generator(f"Extract:\n{text}\n\nData:")- results.append(result)-- return results-```--## Resources--- **Outlines Documentation**: https://outlines-dev.github.io/outlines-- **Pydantic Documentation**: https://docs.pydantic.dev-- **GitHub Examples**: https://github.com/outlines-dev/outlines/tree/main/examplesdiff --git a/optional-skills/mlops/inference/outlines/references/json_generation.md b/optional-skills/mlops/inference/outlines/references/json_generation.mddeleted file mode 100644index 20cee9fc824..00000000000--- a/optional-skills/mlops/inference/outlines/references/json_generation.md+++ /dev/null@@ -1,652 +0,0 @@-# Comprehensive JSON Generation Guide--Complete guide to JSON generation with Outlines using Pydantic models and JSON schemas.--## Table of Contents-- Pydantic Models-- JSON Schema Support-- Advanced Patterns-- Nested Structures-- Complex Types-- Validation-- Performance Optimization--## Pydantic Models--### Basic Models--```python-from pydantic import BaseModel-import outlines--class User(BaseModel):- name: str- age: int- email: str--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, User)--user = generator("Generate user: Alice, 25, [email protected]")-print(user.name) # "Alice"-print(user.age) # 25-print(user.email) # "[email protected]"-```--###-- Field Constraints--```python-from pydantic import BaseModel, Field--class Product(BaseModel):- name: str = Field(min_length=1, max_length=100)- price: float = Field(gt=0, description="Price in USD")- discount: float = Field(ge=0, le=100, description="Discount percentage")- quantity: int = Field(ge=0, description="Available quantity")- sku: str = Field(pattern=r"^[A-Z]{3}-\d{6}$")--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, Product)--product = generator("Generate product: iPhone 15, $999")-# All fields guaranteed to meet constraints-```--**Available Constraints:**-- `min_length`, `max_length`: String length-- `gt`, `ge`, `lt`, `le`: Numeric comparisons-- `multiple_of`: Number must be multiple of value-- `pattern`: Regex pattern for strings-- `min_items`, `max_items`: List length--### Optional Fields--```python-from typing import Optional--class Article(BaseModel):- title: str # Required- author: Optional[str] = None # Optional- published_date: Optional[str] = None # Optional- tags: list[str] = [] # Default empty list- view_count: int = 0 # Default value--generator = outlines.generate.json(model, Article)--# Can generate even if optional fields missing-article = generator("Title: Introduction to AI")-print(article.author) # None (not provided)-print(article.tags) # [] (default)-```--### Default Values--```python-class Config(BaseModel):- debug: bool = False- max_retries: int = 3- timeout: float = 30.0- log_level: str = "INFO"--# Generator uses defaults when not specified-generator = outlines.generate.json(model, Config)-config = generator("Generate config with debug enabled")-print(config.debug) # True (from prompt)-print(config.timeout) # 30.0 (default)-```--## Enums and Literals--### Enum Fields--```python-from enum import Enum--class Status(str, Enum):- PENDING = "pending"- APPROVED = "approved"- REJECTED = "rejected"- CANCELLED = "cancelled"--class Application(BaseModel):- applicant_name: str- status: Status # Must be one of enum values- submitted_date: str--generator = outlines.generate.json(model, Application)-app = generator("Generate application for John Doe")--print(app.status) # Status.PENDING (or one of the enum values)-print(type(app.status)) # <enum 'Status'>-```--### Literal Types--```python-from typing import Literal--class Task(BaseModel):- title: str- priority: Literal["low", "medium", "high", "critical"]- status: Literal["todo", "in_progress", "done"]- assigned_to: str--generator = outlines.generate.json(model, Task)-task = generator("Create high priority task: Fix bug")--print(task.priority) # One of: "low", "medium", "high", "critical"-```--### Multiple Choice Fields--```python-class Survey(BaseModel):- question: str- answer: Literal["strongly_disagree", "disagree", "neutral", "agree", "strongly_agree"]- confidence: Literal["low", "medium", "high"]--generator = outlines.generate.json(model, Survey)-survey = generator("Rate: 'I enjoy using this product'")-```--## Nested Structures--### Nested Models--```python-class Address(BaseModel):- street: str- city: str- state: str- zip_code: str- country: str = "USA"--class Person(BaseModel):- name: str- age: int- email: str- address: Address # Nested model--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, Person)--prompt = """-Extract person:-Name: Alice Johnson-Age: 28-Email: [email protected]-Address: 123 Main St, Boston, MA, 02101-"""--person = generator(prompt)-print(person.name) # "Alice Johnson"-print(person.address.city) # "Boston"-print(person.address.state) # "MA"-```--### Deep Nesting--```python-class Coordinates(BaseModel):- latitude: float- longitude: float--class Location(BaseModel):- name: str- coordinates: Coordinates--class Event(BaseModel):- title: str- date: str- location: Location--generator = outlines.generate.json(model, Event)-event = generator("Generate event: Tech Conference in San Francisco")--print(event.title) # "Tech Conference"-print(event.location.name) # "San Francisco"-print(event.location.coordinates.latitude) # 37.7749-```--### Lists of Nested Models--```python-class Item(BaseModel):- name: str- quantity: int- price: float--class Order(BaseModel):- order_id: str- customer: str- items: list[Item] # List of nested models- total: float--generator = outlines.generate.json(model, Order)--prompt = """-Generate order for John:-- 2x Widget ($10 each)-- 3x Gadget ($15 each)-Order ID: ORD-001-"""--order = generator(prompt)-print(f"Order ID: {order.order_id}")-for item in order.items:- print(f"- {item.quantity}x {item.name} @ ${item.price}")-print(f"Total: ${order.total}")-```--## Complex Types--### Union Types--```python-from typing import Union--class TextContent(BaseModel):- type: Literal["text"]- content: str--class ImageContent(BaseModel):- type: Literal["image"]- url: str- caption: str--class Post(BaseModel):- title: str- content: Union[TextContent, ImageContent] # Either type--generator = outlines.generate.json(model, Post)--# Can generate either text or image content-post = generator("Generate blog post with image")-if post.content.type == "text":- print(post.content.content)-elif post.content.type == "image":- print(post.content.url)-```--### Lists and Arrays--```python-class Article(BaseModel):- title: str- authors: list[str] # List of strings- tags: list[str]- sections: list[dict[str, str]] # List of dicts- related_ids: list[int]--generator = outlines.generate.json(model, Article)-article = generator("Generate article about AI")--print(article.authors) # ["Alice", "Bob"]-print(article.tags) # ["AI", "Machine Learning", "Technology"]-```--### Dictionaries--```python-class Metadata(BaseModel):- title: str- properties: dict[str, str] # String keys and values- counts: dict[str, int] # String keys, int values- settings: dict[str, Union[str, int, bool]] # Mixed value types--generator = outlines.generate.json(model, Metadata)-meta = generator("Generate metadata")--print(meta.properties) # {"author": "Alice", "version": "1.0"}-print(meta.counts) # {"views": 1000, "likes": 50}-```--### Any Type (Use Sparingly)--```python-from typing import Any--class FlexibleData(BaseModel):- name: str- structured_field: str- flexible_field: Any # Can be anything--# Note: Any reduces type safety, use only when necessary-generator = outlines.generate.json(model, FlexibleData)-```--## JSON Schema Support--### Direct Schema Usage--```python-import outlines--model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")--# Define JSON schema-schema = {- "type": "object",- "properties": {- "name": {"type": "string"},- "age": {"type": "integer", "minimum": 0, "maximum": 120},- "email": {"type": "string", "format": "email"}- },- "required": ["name", "age", "email"]-}--# Generate from schema-generator = outlines.generate.json(model, schema)-result = generator("Generate person: Alice, 25, [email protected]")--print(result) # Valid JSON matching schema-```--### Schema from Pydantic--```python-class User(BaseModel):- name: str- age: int- email: str--# Get JSON schema from Pydantic model-schema = User.model_json_schema()-print(schema)-# {-# "type": "object",-# "properties": {-# "name": {"type": "string"},-# "age": {"type": "integer"},-# "email": {"type": "string"}-# },-# "required": ["name", "age", "email"]-# }--# Both approaches equivalent:-generator1 = outlines.generate.json(model, User)-generator2 = outlines.generate.json(model, schema)-```--## Advanced Patterns--### Conditional Fields--```python-class Order(BaseModel):- order_type: Literal["standard", "express"]- delivery_date: str- express_fee: Optional[float] = None # Only for express orders--generator = outlines.generate.json(model, Order)--# Express order-order1 = generator("Create express order for tomorrow")-print(order1.express_fee) # 25.0--# Standard order-order2 = generator("Create standard order")-print(order2.express_fee) # None-```--### Recursive Models--```python-from typing import Optional, List--class TreeNode(BaseModel):- value: str- children: Optional[List['TreeNode']] = None--# Enable forward references-TreeNode.model_rebuild()--generator = outlines.generate.json(model, TreeNode)-tree = generator("Generate file tree with subdirectories")--print(tree.value) # "root"-print(tree.children[0].value) # "subdir1"-```--### Model with Validation--```python-from pydantic import field_validator--class DateRange(BaseModel):- start_date: str- end_date: str-- @field_validator('end_date')- def end_after_start(cls, v, info):- """Ensure end_date is after start_date."""- if 'start_date' in info.data:- from datetime import datetime- start = datetime.strptime(info.data['start_date'], '%Y-%m-%d')- end = datetime.strptime(v, '%Y-%m-%d')- if end < start:- raise ValueError('end_date must be after start_date')- return v--generator = outlines.generate.json(model, DateRange)-# Validation happens after generation-```--## Multiple Objects--### Generate List of Objects--```python-class Person(BaseModel):- name: str- age: int--class Team(BaseModel):- team_name: str- members: list[Person]--generator = outlines.generate.json(model, Team)--team = generator("Generate engineering team with 5 members")-print(f"Team: {team.team_name}")-for member in team.members:- print(f"- {member.name}, {member.age}")-```--### Batch Generation--```python-def generate_batch(prompts: list[str], schema: type[BaseModel]):- """Generate structured outputs for multiple prompts."""- model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")- generator = outlines.generate.json(model, schema)-- results = []- for prompt in prompts:- result = generator(prompt)- results.append(result)-- return results--class Product(BaseModel):- name: str- price: float--prompts = [- "Product: iPhone 15, $999",- "Product: MacBook Pro, $2499",- "Product: AirPods, $179"-]--products = generate_batch(prompts, Product)-for product in products:- print(f"{product.name}: ${product.price}")-```--## Performance Optimization--### Caching Generators--```python-from functools import lru_cache--@lru_cache(maxsize=10)-def get_generator(model_name: str, schema_hash: int):- """Cache generators for reuse."""- model = outlines.models.transformers(model_name)- return outlines.generate.json(model, schema)--# First call: creates generator-gen1 = get_generator("microsoft/Phi-3-mini-4k-instruct", hash(User))--# Second call: returns cached generator (fast!)-gen2 = get_generator("microsoft/Phi-3-mini-4k-instruct", hash(User))-```--### Batch Processing--```python-# Process multiple items efficiently-model = outlines.models.transformers("microsoft/Phi-3-mini-4k-instruct")-generator = outlines.generate.json(model, User)--texts = ["User: Alice, 25", "User: Bob, 30", "User: Carol, 35"]--# Reuse generator (model stays loaded)-users = [generator(text) for text in texts]-```--### Minimize Schema Complexity--```python-# ✅ Good: Simple, flat structure (faster)-class SimplePerson(BaseModel):- name: str- age: int- city: str--# ⚠️ Slower: Deep nesting-class ComplexPerson(BaseModel):- personal_info: PersonalInfo- address: Address- employment: Employment- # ... many nested levels-```--## Error Handling--### Handle Missing Fields--```python-from pydantic import ValidationError--class User(BaseModel):- name: str- age: int- email: str--try:- user = generator("Generate user") # May not include all fields-except ValidationError as e:- print(f"Validation error: {e}")- # Handle gracefully-```--### Fallback with Optional Fields--```python-class RobustUser(BaseModel):- name: str # Required- age: Optional[int] = None # Optional- email: Optional[str] = None # Optional--# More likely to succeed even with incomplete data-user = generator("Generate user: Alice")-print(user.name) # "Alice"-print(user.age) # None (not provided)-```--## Best Practices--### 1. Use Specific Types--```python-# ✅ Good: Specific types-class Product(BaseModel):- name: str- price: float # Not Any or str- quantity: int # Not str- in_stock: bool # Not int--# ❌ Bad: Generic types-class Product(BaseModel):- name: Any- price: str # Should be float- quantity: str # Should be int-```--### 2. Add Descriptions--```python-# ✅ Good: Clear descriptions-class Article(BaseModel):- title: str = Field(description="Article title, 10-100 characters")- content: str = Field(description="Main article content in paragraphs")- tags: list[str] = Field(description="List of relevant topic tags")--# Descriptions help the model understand expected output-```--### 3. Use Constraints--```python-# ✅ Good: With constraints-class Age(BaseModel):- value: int = Field(ge=0, le=120, description="Age in years")--# ❌ Bad: No constraints-class Age(BaseModel):- value: int # Could be negative or > 120-```--### 4. Prefer Enums Over Strings--```python-# ✅ Good: Enum for fixed set-class Priority(str, Enum):- LOW = "low"- MEDIUM = "medium"- HIGH = "high"--class Task(BaseModel):- priority: Priority # Guaranteed valid--# ❌ Bad: Free-form string-class Task(BaseModel):- priority: str # Could be "urgent", "ASAP", "!!", etc.-```--### 5. Test Your Models--```python-# Test models work as expected-def test_product_model():- product = Product(- name="Test Product",- price=19.99,- quantity=10,- in_stock=True- )- assert product.price == 19.99- assert isinstance(product, Product)--# Run tests before using in production-```--## Resources--- **Pydantic Docs**: https://docs.pydantic.dev-- **JSON Schema**: https://json-schema.org-- **Outlines GitHub**: https://github.com/outlines-dev/outlinesdiff --git a/optional-skills/mlops/instructor/SKILL.md b/optional-skills/mlops/instructor/SKILL.mddeleted file mode 100644index 7dbcc7de59f..00000000000--- a/optional-skills/mlops/instructor/SKILL.md+++ /dev/null@@ -1,744 +0,0 @@-----name: instructor-description: Structured LLM outputs validated with Pydantic.-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [instructor, pydantic, openai, anthropic]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Prompt Engineering, Instructor, Structured Output, Pydantic, Data Extraction, JSON Parsing, Type Safety, Validation, Streaming, OpenAI, Anthropic]-------# Instructor: Structured LLM Outputs--## When to Use This Skill--Use Instructor when you need to:-- **Extract structured data** from LLM responses reliably-- **Validate outputs** against Pydantic schemas automatically-- **Retry failed extractions** with automatic error handling-- **Parse complex JSON** with type safety and validation-- **Stream partial results** for real-time processing-- **Support multiple LLM providers** with consistent API--**GitHub Stars**: 15,000+ | **Battle-tested**: 100,000+ developers--## Installation--```bash-# Base installation-pip install instructor--# With specific providers-pip install "instructor[anthropic]" # Anthropic Claude-pip install "instructor[openai]" # OpenAI-pip install "instructor[all]" # All providers-```--## Quick Start--### Basic Example: Extract User Data--```python-import instructor-from pydantic import BaseModel-from anthropic import Anthropic--# Define output structure-class User(BaseModel):- name: str- age: int- email: str--# Create instructor client-client = instructor.from_anthropic(Anthropic())--# Extract structured data-user = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "John Doe is 30 years old. His email is [email protected]"- }],- response_model=User-)--print(user.name) # "John Doe"-print(user.age) # 30-print(user.email) # "[email protected]"-```--### With OpenAI--```python-from openai import OpenAI--client = instructor.from_openai(OpenAI())--user = client.chat.completions.create(- model="gpt-4o-mini",- response_model=User,- messages=[{"role": "user", "content": "Extract: Alice, 25, [email protected]"}]-)-```--## Core Concepts--### 1. Response Models (Pydantic)--Response models define the structure and validation rules for LLM outputs.--#### Basic Model--```python-from pydantic import BaseModel, Field--class Article(BaseModel):- title: str = Field(description="Article title")- author: str = Field(description="Author name")- word_count: int = Field(description="Number of words", gt=0)- tags: list[str] = Field(description="List of relevant tags")--article = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Analyze this article: [article text]"- }],- response_model=Article-)-```--**Benefits:**-- Type safety with Python type hints-- Automatic validation (word_count > 0)-- Self-documenting with Field descriptions-- IDE autocomplete support--#### Nested Models--```python-class Address(BaseModel):- street: str- city: str- country: str--class Person(BaseModel):- name: str- age: int- address: Address # Nested model--person = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "John lives at 123 Main St, Boston, USA"- }],- response_model=Person-)--print(person.address.city) # "Boston"-```--#### Optional Fields--```python-from typing import Optional--class Product(BaseModel):- name: str- price: float- discount: Optional[float] = None # Optional- description: str = Field(default="No description") # Default value--# LLM doesn't need to provide discount or description-```--#### Enums for Constraints--```python-from enum import Enum--class Sentiment(str, Enum):- POSITIVE = "positive"- NEGATIVE = "negative"- NEUTRAL = "neutral"--class Review(BaseModel):- text: str- sentiment: Sentiment # Only these 3 values allowed--review = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "This product is amazing!"- }],- response_model=Review-)--print(review.sentiment) # Sentiment.POSITIVE-```--### 2. Validation--Pydantic validates LLM outputs automatically. If validation fails, Instructor retries.--#### Built-in Validators--```python-from pydantic import Field, EmailStr, HttpUrl--class Contact(BaseModel):- name: str = Field(min_length=2, max_length=100)- age: int = Field(ge=0, le=120) # 0 <= age <= 120- email: EmailStr # Validates email format- website: HttpUrl # Validates URL format--# If LLM provides invalid data, Instructor retries automatically-```--#### Custom Validators--```python-from pydantic import field_validator--class Event(BaseModel):- name: str- date: str- attendees: int-- @field_validator('date')- def validate_date(cls, v):- """Ensure date is in YYYY-MM-DD format."""- import re- if not re.match(r'\d{4}-\d{2}-\d{2}', v):- raise ValueError('Date must be YYYY-MM-DD format')- return v-- @field_validator('attendees')- def validate_attendees(cls, v):- """Ensure positive attendees."""- if v < 1:- raise ValueError('Must have at least 1 attendee')- return v-```--#### Model-Level Validation--```python-from pydantic import model_validator--class DateRange(BaseModel):- start_date: str- end_date: str-- @model_validator(mode='after')- def check_dates(self):- """Ensure end_date is after start_date."""- from datetime import datetime- start = datetime.strptime(self.start_date, '%Y-%m-%d')- end = datetime.strptime(self.end_date, '%Y-%m-%d')-- if end < start:- raise ValueError('end_date must be after start_date')- return self-```--### 3. Automatic Retrying--Instructor retries automatically when validation fails, providing error feedback to the LLM.--```python-# Retries up to 3 times if validation fails-user = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Extract user from: John, age unknown"- }],- response_model=User,- max_retries=3 # Default is 3-)--# If age can't be extracted, Instructor tells the LLM:-# "Validation error: age - field required"-# LLM tries again with better extraction-```--**How it works:**-1. LLM generates output-2. Pydantic validates-3. If invalid: Error message sent back to LLM-4. LLM tries again with error feedback-5. Repeats up to max_retries--### 4. Streaming--Stream partial results for real-time processing.--#### Streaming Partial Objects--```python-from instructor import Partial--class Story(BaseModel):- title: str- content: str- tags: list[str]--# Stream partial updates as LLM generates-for partial_story in client.messages.create_partial(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Write a short sci-fi story"- }],- response_model=Story-):- print(f"Title: {partial_story.title}")- print(f"Content so far: {partial_story.content[:100]}...")- # Update UI in real-time-```--#### Streaming Iterables--```python-class Task(BaseModel):- title: str- priority: str--# Stream list items as they're generated-tasks = client.messages.create_iterable(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Generate 10 project tasks"- }],- response_model=Task-)--for task in tasks:- print(f"- {task.title} ({task.priority})")- # Process each task as it arrives-```--## Provider Configuration--### Anthropic Claude--```python-import instructor-from anthropic import Anthropic--client = instructor.from_anthropic(- Anthropic(api_key="your-api-key")-)--# Use with Claude models-response = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[...],- response_model=YourModel-)-```--### OpenAI--```python-from openai import OpenAI--client = instructor.from_openai(- OpenAI(api_key="your-api-key")-)--response = client.chat.completions.create(- model="gpt-4o-mini",- response_model=YourModel,- messages=[...]-)-```--### Local Models (Ollama)--```python-from openai import OpenAI--# Point to local Ollama server-client = instructor.from_openai(- OpenAI(- base_url="http://localhost:11434/v1",- api_key="ollama" # Required but ignored- ),- mode=instructor.Mode.JSON-)--response = client.chat.completions.create(- model="llama3.1",- response_model=YourModel,- messages=[...]-)-```--## Common Patterns--### Pattern 1: Data Extraction from Text--```python-class CompanyInfo(BaseModel):- name: str- founded_year: int- industry: str- employees: int- headquarters: str--text = """-Tesla, Inc. was founded in 2003. It operates in the automotive and energy-industry with approximately 140,000 employees. The company is headquartered-in Austin, Texas.-"""--company = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": f"Extract company information from: {text}"- }],- response_model=CompanyInfo-)-```--### Pattern 2: Classification--```python-class Category(str, Enum):- TECHNOLOGY = "technology"- FINANCE = "finance"- HEALTHCARE = "healthcare"- EDUCATION = "education"- OTHER = "other"--class ArticleClassification(BaseModel):- category: Category- confidence: float = Field(ge=0.0, le=1.0)- keywords: list[str]--classification = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Classify this article: [article text]"- }],- response_model=ArticleClassification-)-```--### Pattern 3: Multi-Entity Extraction--```python-class Person(BaseModel):- name: str- role: str--class Organization(BaseModel):- name: str- industry: str--class Entities(BaseModel):- people: list[Person]- organizations: list[Organization]- locations: list[str]--text = "Tim Cook, CEO of Apple, announced at the event in Cupertino..."--entities = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": f"Extract all entities from: {text}"- }],- response_model=Entities-)--for person in entities.people:- print(f"{person.name} - {person.role}")-```--### Pattern 4: Structured Analysis--```python-class SentimentAnalysis(BaseModel):- overall_sentiment: Sentiment- positive_aspects: list[str]- negative_aspects: list[str]- suggestions: list[str]- score: float = Field(ge=-1.0, le=1.0)--review = "The product works well but setup was confusing..."--analysis = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": f"Analyze this review: {review}"- }],- response_model=SentimentAnalysis-)-```--### Pattern 5: Batch Processing--```python-def extract_person(text: str) -> Person:- return client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": f"Extract person from: {text}"- }],- response_model=Person- )--texts = [- "John Doe is a 30-year-old engineer",- "Jane Smith, 25, works in marketing",- "Bob Johnson, age 40, software developer"-]--people = [extract_person(text) for text in texts]-```--## Advanced Features--### Union Types--```python-from typing import Union--class TextContent(BaseModel):- type: str = "text"- content: str--class ImageContent(BaseModel):- type: str = "image"- url: HttpUrl- caption: str--class Post(BaseModel):- title: str- content: Union[TextContent, ImageContent] # Either type--# LLM chooses appropriate type based on content-```--### Dynamic Models--```python-from pydantic import create_model--# Create model at runtime-DynamicUser = create_model(- 'User',- name=(str, ...),- age=(int, Field(ge=0)),- email=(EmailStr, ...)-)--user = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[...],- response_model=DynamicUser-)-```--### Custom Modes--```python-# For providers without native structured outputs-client = instructor.from_anthropic(- Anthropic(),- mode=instructor.Mode.JSON # JSON mode-)--# Available modes:-# - Mode.ANTHROPIC_TOOLS (recommended for Claude)-# - Mode.JSON (fallback)-# - Mode.TOOLS (OpenAI tools)-```--### Context Management--```python-# Single-use client-with instructor.from_anthropic(Anthropic()) as client:- result = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[...],- response_model=YourModel- )- # Client closed automatically-```--## Error Handling--### Handling Validation Errors--```python-from pydantic import ValidationError--try:- user = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[...],- response_model=User,- max_retries=3- )-except ValidationError as e:- print(f"Failed after retries: {e}")- # Handle gracefully--except Exception as e:- print(f"API error: {e}")-```--### Custom Error Messages--```python-class ValidatedUser(BaseModel):- name: str = Field(description="Full name, 2-100 characters")- age: int = Field(description="Age between 0 and 120", ge=0, le=120)- email: EmailStr = Field(description="Valid email address")-- class Config:- # Custom error messages- json_schema_extra = {- "examples": [- {- "name": "John Doe",- "age": 30,- "email": "[email protected]"- }- ]- }-```--## Best Practices--### 1. Clear Field Descriptions--```python-# ❌ Bad: Vague-class Product(BaseModel):- name: str- price: float--# ✅ Good: Descriptive-class Product(BaseModel):- name: str = Field(description="Product name from the text")- price: float = Field(description="Price in USD, without currency symbol")-```--### 2. Use Appropriate Validation--```python-# ✅ Good: Constrain values-class Rating(BaseModel):- score: int = Field(ge=1, le=5, description="Rating from 1 to 5 stars")- review: str = Field(min_length=10, description="Review text, at least 10 chars")-```--### 3. Provide Examples in Prompts--```python-messages = [{- "role": "user",- "content": """Extract person info from: "John, 30, engineer"--Example format:-{- "name": "John Doe",- "age": 30,- "occupation": "engineer"-}"""-}]-```--### 4. Use Enums for Fixed Categories--```python-# ✅ Good: Enum ensures valid values-class Status(str, Enum):- PENDING = "pending"- APPROVED = "approved"- REJECTED = "rejected"--class Application(BaseModel):- status: Status # LLM must choose from enum-```--### 5. Handle Missing Data Gracefully--```python-class PartialData(BaseModel):- required_field: str- optional_field: Optional[str] = None- default_field: str = "default_value"--# LLM only needs to provide required_field-```--## Comparison to Alternatives--| Feature | Instructor | Manual JSON | LangChain | DSPy |-|---------|------------|-------------|-----------|------|-| Type Safety | ✅ Yes | ❌ No | ⚠️ Partial | ✅ Yes |-| Auto Validation | ✅ Yes | ❌ No | ❌ No | ⚠️ Limited |-| Auto Retry | ✅ Yes | ❌ No | ❌ No | ✅ Yes |-| Streaming | ✅ Yes | ❌ No | ✅ Yes | ❌ No |-| Multi-Provider | ✅ Yes | ⚠️ Manual | ✅ Yes | ✅ Yes |-| Learning Curve | Low | Low | Medium | High |--**When to choose Instructor:**-- Need structured, validated outputs-- Want type safety and IDE support-- Require automatic retries-- Building data extraction systems--**When to choose alternatives:**-- DSPy: Need prompt optimization-- LangChain: Building complex chains-- Manual: Simple, one-off extractions--## Resources--- **Documentation**: https://python.useinstructor.com-- **GitHub**: https://github.com/jxnl/instructor (15k+ stars)-- **Cookbook**: https://python.useinstructor.com/examples-- **Discord**: Community support available--## See Also--- `references/validation.md` - Advanced validation patterns-- `references/providers.md` - Provider-specific configuration-- `references/examples.md` - Real-world use cases--diff --git a/optional-skills/mlops/instructor/references/examples.md b/optional-skills/mlops/instructor/references/examples.mddeleted file mode 100644index e114835234f..00000000000--- a/optional-skills/mlops/instructor/references/examples.md+++ /dev/null@@ -1,107 +0,0 @@-# Real-World Examples--Practical examples of using Instructor for structured data extraction.--## Data Extraction--```python-class CompanyInfo(BaseModel):- name: str- founded: int- industry: str- employees: int--text = "Apple was founded in 1976 in the technology industry with 164,000 employees."--company = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": f"Extract: {text}"}],- response_model=CompanyInfo-)-```--## Classification--```python-class Sentiment(str, Enum):- POSITIVE = "positive"- NEGATIVE = "negative"- NEUTRAL = "neutral"--class Review(BaseModel):- sentiment: Sentiment- confidence: float = Field(ge=0.0, le=1.0)--review = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": "This product is amazing!"}],- response_model=Review-)-```--## Multi-Entity Extraction--```python-class Person(BaseModel):- name: str- role: str--class Entities(BaseModel):- people: list[Person]- organizations: list[str]- locations: list[str]--entities = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": "Tim Cook, CEO of Apple, spoke in Cupertino..."}],- response_model=Entities-)-```--## Structured Analysis--```python-class Analysis(BaseModel):- summary: str- key_points: list[str]- sentiment: Sentiment- actionable_items: list[str]--analysis = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": "Analyze: [long text]"}],- response_model=Analysis-)-```--## Batch Processing--```python-texts = ["text1", "text2", "text3"]-results = [- client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": text}],- response_model=YourModel- )- for text in texts-]-```--## Streaming--```python-for partial in client.messages.create_partial(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": "Generate report..."}],- response_model=Report-):- print(f"Progress: {partial.title}")- # Update UI in real-time-```diff --git a/optional-skills/mlops/instructor/references/providers.md b/optional-skills/mlops/instructor/references/providers.mddeleted file mode 100644index 1f5975ef6ae..00000000000--- a/optional-skills/mlops/instructor/references/providers.md+++ /dev/null@@ -1,70 +0,0 @@-# Provider Configuration--Guide to using Instructor with different LLM providers.--## Anthropic Claude--```python-import instructor-from anthropic import Anthropic--# Basic setup-client = instructor.from_anthropic(Anthropic())--# With API key-client = instructor.from_anthropic(- Anthropic(api_key="your-api-key")-)--# Recommended mode-client = instructor.from_anthropic(- Anthropic(),- mode=instructor.Mode.ANTHROPIC_TOOLS-)--# Usage-result = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": "..."}],- response_model=YourModel-)-```--## OpenAI--```python-from openai import OpenAI--client = instructor.from_openai(OpenAI())--result = client.chat.completions.create(- model="gpt-4o-mini",- response_model=YourModel,- messages=[{"role": "user", "content": "..."}]-)-```--## Local Models (Ollama)--```python-client = instructor.from_openai(- OpenAI(- base_url="http://localhost:11434/v1",- api_key="ollama"- ),- mode=instructor.Mode.JSON-)--result = client.chat.completions.create(- model="llama3.1",- response_model=YourModel,- messages=[...]-)-```--## Modes--- `Mode.ANTHROPIC_TOOLS`: Recommended for Claude-- `Mode.TOOLS`: OpenAI function calling-- `Mode.JSON`: Fallback for unsupported providersdiff --git a/optional-skills/mlops/instructor/references/validation.md b/optional-skills/mlops/instructor/references/validation.mddeleted file mode 100644index 790c4867153..00000000000--- a/optional-skills/mlops/instructor/references/validation.md+++ /dev/null@@ -1,606 +0,0 @@-# Advanced Validation Patterns--Complete guide to validation in Instructor using Pydantic.--## Table of Contents-- Built-in Validators-- Custom Field Validators-- Model-Level Validation-- Complex Validation Patterns-- Error Handling--## Built-in Validators--### Numeric Constraints--```python-from pydantic import BaseModel, Field--class Product(BaseModel):- price: float = Field(gt=0, description="Price must be positive")- discount: float = Field(ge=0, le=100, description="Discount 0-100%")- quantity: int = Field(ge=1, description="At least 1 item")- rating: float = Field(ge=0.0, le=5.0, description="Rating 0-5 stars")--# If LLM provides invalid values, automatic retry with error feedback-```--**Available constraints:**-- `gt`: Greater than-- `ge`: Greater than or equal-- `lt`: Less than-- `le`: Less than or equal-- `multiple_of`: Must be multiple of this number--### String Constraints--```python-class User(BaseModel):- username: str = Field(- min_length=3,- max_length=20,- pattern=r'^[a-zA-Z0-9_]+$',- description="3-20 alphanumeric characters"- )- bio: str = Field(max_length=500, description="Bio up to 500 chars")- status: str = Field(pattern=r'^(active|inactive|pending)$')--# pattern validates against regex-```--### Email and URL Validation--```python-from pydantic import EmailStr, HttpUrl, AnyUrl--class Contact(BaseModel):- email: EmailStr # Validates email format- website: HttpUrl # Validates HTTP/HTTPS URLs- portfolio: AnyUrl # Any valid URL scheme--contact = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{- "role": "user",- "content": "Extract: [email protected], https://example.com"- }],- response_model=Contact-)-```--### Date and DateTime Validation--```python-from datetime import date, datetime-from pydantic import Field, field_validator--class Event(BaseModel):- event_date: date # Validates date format- created_at: datetime # Validates datetime format- year: int = Field(ge=1900, le=2100)-- @field_validator('event_date')- def future_date(cls, v):- """Ensure event is in the future."""- if v < date.today():- raise ValueError('Event must be in the future')- return v-```--### List and Dict Validation--```python-class Document(BaseModel):- tags: list[str] = Field(min_length=1, max_length=10)- keywords: list[str] = Field(min_length=3, description="At least 3 keywords")- metadata: dict[str, str] = Field(description="String key-value pairs")-- @field_validator('tags')- def unique_tags(cls, v):- """Ensure tags are unique."""- if len(v) != len(set(v)):- raise ValueError('Tags must be unique')- return v-```--## Custom Field Validators--### Basic Field Validator--```python-from pydantic import field_validator--class Person(BaseModel):- name: str- age: int-- @field_validator('name')- def name_must_not_be_empty(cls, v):- """Validate name is not empty or just whitespace."""- if not v or not v.strip():- raise ValueError('Name cannot be empty')- return v.strip()-- @field_validator('age')- def age_must_be_reasonable(cls, v):- """Validate age is between 0 and 120."""- if v < 0 or v > 120:- raise ValueError('Age must be between 0 and 120')- return v-```--### Validator with Field Info--```python-from pydantic import ValidationInfo--class Article(BaseModel):- title: str- content: str-- @field_validator('content')- def content_length(cls, v, info: ValidationInfo):- """Validate content is longer than title."""- if 'title' in info.data:- title_len = len(info.data['title'])- if len(v) < title_len * 2:- raise ValueError('Content should be at least 2x title length')- return v-```--### Multiple Fields Validation--```python-class TimeRange(BaseModel):- start_time: str- end_time: str-- @field_validator('start_time', 'end_time')- def valid_time_format(cls, v):- """Validate both times are in HH:MM format."""- import re- if not re.match(r'^\d{2}:\d{2}$', v):- raise ValueError('Time must be in HH:MM format')- return v-```--### Transform and Validate--```python-class URL(BaseModel):- url: str-- @field_validator('url')- def normalize_url(cls, v):- """Add https:// if missing."""- if not v.startswith(('http://', 'https://')):- v = f'https://{v}'- return v-```--## Model-Level Validation--### Cross-Field Validation--```python-from pydantic import model_validator--class DateRange(BaseModel):- start_date: str- end_date: str-- @model_validator(mode='after')- def check_dates(self):- """Ensure end_date is after start_date."""- from datetime import datetime- start = datetime.strptime(self.start_date, '%Y-%m-%d')- end = datetime.strptime(self.end_date, '%Y-%m-%d')-- if end < start:- raise ValueError('end_date must be after start_date')- return self--class PriceRange(BaseModel):- min_price: float- max_price: float-- @model_validator(mode='after')- def check_price_range(self):- """Ensure max > min."""- if self.max_price <= self.min_price:- raise ValueError('max_price must be greater than min_price')- return self-```--### Conditional Validation--```python-class Order(BaseModel):- order_type: str # "standard" or "express"- delivery_date: str- delivery_time: Optional[str] = None-- @model_validator(mode='after')- def check_delivery_time(self):- """Express orders need delivery time."""- if self.order_type == "express" and not self.delivery_time:- raise ValueError('Express orders require delivery_time')- return self-```--### Complex Business Logic--```python-class Discount(BaseModel):- code: str- percentage: float = Field(ge=0, le=100)- min_purchase: float = Field(ge=0)- max_discount: float = Field(ge=0)-- @model_validator(mode='after')- def validate_discount(self):- """Ensure discount logic is sound."""- # Max discount can't exceed percentage of min_purchase- theoretical_max = (self.percentage / 100) * self.min_purchase- if self.max_discount > theoretical_max:- self.max_discount = theoretical_max- return self-```--## Complex Validation Patterns--### Nested Model Validation--```python-class Address(BaseModel):- street: str- city: str- country: str- postal_code: str-- @field_validator('postal_code')- def validate_postal_code(cls, v, info: ValidationInfo):- """Validate postal code format based on country."""- if 'country' in info.data:- country = info.data['country']- if country == "USA":- import re- if not re.match(r'^\d{5}(-\d{4})?$', v):- raise ValueError('Invalid US postal code')- elif country == "Canada":- if not re.match(r'^[A-Z]\d[A-Z] \d[A-Z]\d$', v):- raise ValueError('Invalid Canadian postal code')- return v--class Person(BaseModel):- name: str- address: Address--# Nested validation runs automatically-```--### List of Models--```python-class Task(BaseModel):- title: str = Field(min_length=1)- priority: int = Field(ge=1, le=5)--class Project(BaseModel):- name: str- tasks: list[Task] = Field(min_length=1, description="At least 1 task")-- @field_validator('tasks')- def at_least_one_high_priority(cls, v):- """Ensure at least one task has priority >= 4."""- if not any(task.priority >= 4 for task in v):- raise ValueError('Project needs at least one high-priority task')- return v-```--### Union Type Validation--```python-from typing import Union--class TextBlock(BaseModel):- type: str = "text"- content: str = Field(min_length=1)--class ImageBlock(BaseModel):- type: str = "image"- url: HttpUrl- alt_text: str--class Page(BaseModel):- title: str- blocks: list[Union[TextBlock, ImageBlock]]-- @field_validator('blocks')- def validate_block_types(cls, v):- """Ensure first block is TextBlock."""- if v and not isinstance(v[0], TextBlock):- raise ValueError('First block must be text')- return v-```--### Dependent Fields--```python-class Subscription(BaseModel):- plan: str # "free", "pro", "enterprise"- max_users: int- features: list[str]-- @model_validator(mode='after')- def validate_plan_limits(self):- """Enforce plan-specific limits."""- limits = {- "free": {"max_users": 1, "required_features": ["basic"]},- "pro": {"max_users": 10, "required_features": ["basic", "advanced"]},- "enterprise": {"max_users": 999, "required_features": ["basic", "advanced", "premium"]}- }-- if self.plan in limits:- limit = limits[self.plan]-- if self.max_users > limit["max_users"]:- raise ValueError(f'{self.plan} plan limited to {limit["max_users"]} users')-- for feature in limit["required_features"]:- if feature not in self.features:- raise ValueError(f'{self.plan} plan requires {feature} feature')-- return self-```--## Error Handling--### Graceful Degradation--```python-class OptionalExtraction(BaseModel):- # Required fields- title: str-- # Optional fields with defaults- author: Optional[str] = None- date: Optional[str] = None- tags: list[str] = Field(default_factory=list)--# LLM can succeed even if it can't extract everything-```--### Partial Validation--```python-from pydantic import ValidationError--def extract_with_fallback(text: str):- """Try full extraction, fall back to partial."""- try:- # Try full extraction- return client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": text}],- response_model=FullModel- )- except ValidationError:- # Fall back to partial model- return client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[{"role": "user", "content": text}],- response_model=PartialModel- )-```--### Validation Error Inspection--```python-from pydantic import ValidationError--try:- result = client.messages.create(- model="claude-sonnet-4-5-20250929",- max_tokens=1024,- messages=[...],- response_model=MyModel,- max_retries=3- )-except ValidationError as e:- # Inspect specific errors- for error in e.errors():- field = error['loc'][0]- message = error['msg']- print(f"Field '{field}' failed: {message}")-- # Custom handling per field- if field == 'email':- # Handle email validation failure- pass-```--### Custom Error Messages--```python-class DetailedModel(BaseModel):- name: str = Field(- min_length=2,- max_length=100,- description="Name between 2-100 characters"- )- age: int = Field(- ge=0,- le=120,- description="Age between 0 and 120 years"- )-- @field_validator('name')- def validate_name(cls, v):- """Provide helpful error message."""- if not v.strip():- raise ValueError(- 'Name cannot be empty. '- 'Please provide a valid name from the text.'- )- return v--# When validation fails, LLM sees these helpful messages-```--## Validation Best Practices--### 1. Be Specific--```python-# ❌ Bad: Vague validation-class Item(BaseModel):- name: str--# ✅ Good: Specific constraints-class Item(BaseModel):- name: str = Field(- min_length=1,- max_length=200,- description="Item name, 1-200 characters"- )-```--### 2. Provide Context--```python-# ✅ Good: Explain why validation failed-@field_validator('price')-def validate_price(cls, v):- if v <= 0:- raise ValueError(- 'Price must be positive. '- 'Extract numeric price from text without currency symbols.'- )- return v-```--### 3. Use Enums for Fixed Sets--```python-# ❌ Bad: String validation-status: str--@field_validator('status')-def validate_status(cls, v):- if v not in ['active', 'inactive', 'pending']:- raise ValueError('Invalid status')- return v--# ✅ Good: Enum-class Status(str, Enum):- ACTIVE = "active"- INACTIVE = "inactive"- PENDING = "pending"--status: Status # Validation automatic-```--### 4. Balance Strictness--```python-# Too strict: May fail unnecessarily-class StrictModel(BaseModel):- date: str = Field(pattern=r'^\d{4}-\d{2}-\d{2}$')- # Fails if LLM uses "2024-1-5" instead of "2024-01-05"--# Better: Normalize in validator-class FlexibleModel(BaseModel):- date: str-- @field_validator('date')- def normalize_date(cls, v):- from datetime import datetime- # Parse flexible formats- for fmt in ['%Y-%m-%d', '%Y/%m/%d', '%m/%d/%Y']:- try:- dt = datetime.strptime(v, fmt)- return dt.strftime('%Y-%m-%d') # Normalize- except ValueError:- continue- raise ValueError('Invalid date format')-```--### 5. Test Validation--```python-# Test your validators with edge cases-def test_validation():- # Should succeed- valid = MyModel(field="valid_value")-- # Should fail- try:- invalid = MyModel(field="invalid")- assert False, "Should have raised ValidationError"- except ValidationError:- pass # Expected--# Run tests before using in production-```--## Advanced Techniques--### Conditional Required Fields--```python-from typing import Optional--class ConditionalModel(BaseModel):- type: str- detail_a: Optional[str] = None- detail_b: Optional[str] = None-- @model_validator(mode='after')- def check_required_details(self):- """Require different fields based on type."""- if self.type == "type_a" and not self.detail_a:- raise ValueError('type_a requires detail_a')- if self.type == "type_b" and not self.detail_b:- raise ValueError('type_b requires detail_b')- return self-```--### Validation with External Data--```python-class Product(BaseModel):- sku: str- name: str-- @field_validator('sku')- def validate_sku(cls, v):- """Check SKU exists in database."""- # Query database or API- if not database.sku_exists(v):- raise ValueError(f'SKU {v} not found in catalog')- return v-```--### Progressive Validation--```python-# Start with loose validation-class Stage1(BaseModel):- data: str # Any string--# Then strict validation-class Stage2(BaseModel):- data: str = Field(pattern=r'^[A-Z]{3}-\d{6}$')--# Use Stage1 for initial extraction-# Use Stage2 for final validation-```--## Resources--- **Pydantic Docs**: https://docs.pydantic.dev/latest/concepts/validators/-- **Instructor Examples**: https://python.useinstructor.com/examplesdiff --git a/optional-skills/mlops/llava/SKILL.md b/optional-skills/mlops/llava/SKILL.mddeleted file mode 100644index 3327b024018..00000000000--- a/optional-skills/mlops/llava/SKILL.md+++ /dev/null@@ -1,308 +0,0 @@-----name: llava-description: "Vision-language chat: VQA, captioning, image dialogue."-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [transformers, torch, pillow]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [LLaVA, Vision-Language, Multimodal, Visual Question Answering, Image Chat, CLIP, Vicuna, Conversational AI, Instruction Tuning, VQA]-------# LLaVA - Large Language and Vision Assistant--Open-source vision-language model for conversational image understanding.--## When to use LLaVA--**Use when:**-- Building vision-language chatbots-- Visual question answering (VQA)-- Image description and captioning-- Multi-turn image conversations-- Visual instruction following-- Document understanding with images--**Metrics**:-- **23,000+ GitHub stars**-- GPT-4V level capabilities (targeted)-- Apache 2.0 License-- Multiple model sizes (7B-34B params)--**Use alternatives instead**:-- **GPT-4V**: Highest quality, API-based-- **CLIP**: Simple zero-shot classification-- **BLIP-2**: Better for captioning only-- **Flamingo**: Research, not open-source--## Quick start--### Installation--```bash-# Clone repository-git clone https://github.com/haotian-liu/LLaVA-cd LLaVA--# Install-pip install -e .-```--### Basic usage--```python-from llava.model.builder import load_pretrained_model-from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token-from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN-from llava.conversation import conv_templates-from PIL import Image-import torch--# Load model-model_path = "liuhaotian/llava-v1.5-7b"-tokenizer, model, image_processor, context_len = load_pretrained_model(- model_path=model_path,- model_base=None,- model_name=get_model_name_from_path(model_path)-)--# Load image-image = Image.open("image.jpg")-image_tensor = process_images([image], image_processor, model.config)-image_tensor = image_tensor.to(model.device, dtype=torch.float16)--# Create conversation-conv = conv_templates["llava_v1"].copy()-conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")-conv.append_message(conv.roles[1], None)-prompt = conv.get_prompt()--# Generate response-input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)--with torch.inference_mode():- output_ids = model.generate(- input_ids,- images=image_tensor,- do_sample=True,- temperature=0.2,- max_new_tokens=512- )--response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()-print(response)-```--## Available models--| Model | Parameters | VRAM | Quality |-|-------|------------|------|---------|-| LLaVA-v1.5-7B | 7B | ~14 GB | Good |-| LLaVA-v1.5-13B | 13B | ~28 GB | Better |-| LLaVA-v1.6-34B | 34B | ~70 GB | Best |--```python-# Load different models-model_7b = "liuhaotian/llava-v1.5-7b"-model_13b = "liuhaotian/llava-v1.5-13b"-model_34b = "liuhaotian/llava-v1.6-34b"--# 4-bit quantization for lower VRAM-load_4bit = True # Reduces VRAM by ~4×-```--## CLI usage--```bash-# Single image query-python -m llava.serve.cli \- --model-path liuhaotian/llava-v1.5-7b \- --image-file image.jpg \- --query "What is in this image?"--# Multi-turn conversation-python -m llava.serve.cli \- --model-path liuhaotian/llava-v1.5-7b \- --image-file image.jpg-# Then type questions interactively-```--## Web UI (Gradio)--```bash-# Launch Gradio interface-python -m llava.serve.gradio_web_server \- --model-path liuhaotian/llava-v1.5-7b \- --load-4bit # Optional: reduce VRAM--# Access at http://localhost:7860-```--## Multi-turn conversations--```python-# Initialize conversation-conv = conv_templates["llava_v1"].copy()--# Turn 1-conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")-conv.append_message(conv.roles[1], None)-response1 = generate(conv, model, image) # "A dog playing in a park"--# Turn 2-conv.messages[-1][1] = response1 # Add previous response-conv.append_message(conv.roles[0], "What breed is the dog?")-conv.append_message(conv.roles[1], None)-response2 = generate(conv, model, image) # "Golden Retriever"--# Turn 3-conv.messages[-1][1] = response2-conv.append_message(conv.roles[0], "What time of day is it?")-conv.append_message(conv.roles[1], None)-response3 = generate(conv, model, image)-```--## Common tasks--### Image captioning--```python-question = "Describe this image in detail."-response = ask(model, image, question)-```--### Visual question answering--```python-question = "How many people are in the image?"-response = ask(model, image, question)-```--### Object detection (textual)--```python-question = "List all the objects you can see in this image."-response = ask(model, image, question)-```--### Scene understanding--```python-question = "What is happening in this scene?"-response = ask(model, image, question)-```--### Document understanding--```python-question = "What is the main topic of this document?"-response = ask(model, document_image, question)-```--## Training custom model--```bash-# Stage 1: Feature alignment (558K image-caption pairs)-bash scripts/v1_5/pretrain.sh--# Stage 2: Visual instruction tuning (150K instruction data)-bash scripts/v1_5/finetune.sh-```--## Quantization (reduce VRAM)--```python-# 4-bit quantization-tokenizer, model, image_processor, context_len = load_pretrained_model(- model_path="liuhaotian/llava-v1.5-13b",- model_base=None,- model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),- load_4bit=True # Reduces VRAM ~4×-)--# 8-bit quantization-load_8bit=True # Reduces VRAM ~2×-```--## Best practices--1. **Start with 7B model** - Good quality, manageable VRAM-2. **Use 4-bit quantization** - Reduces VRAM significantly-3. **GPU required** - CPU inference extremely slow-4. **Clear prompts** - Specific questions get better answers-5. **Multi-turn conversations** - Maintain conversation context-6. **Temperature 0.2-0.7** - Balance creativity/consistency-7. **max_new_tokens 512-1024** - For detailed responses-8. **Batch processing** - Process multiple images sequentially--## Performance--| Model | VRAM (FP16) | VRAM (4-bit) | Speed (tokens/s) |-|-------|-------------|--------------|------------------|-| 7B | ~14 GB | ~4 GB | ~20 |-| 13B | ~28 GB | ~8 GB | ~12 |-| 34B | ~70 GB | ~18 GB | ~5 |--*On A100 GPU*--## Benchmarks--LLaVA achieves competitive scores on:-- **VQAv2**: 78.5%-- **GQA**: 62.0%-- **MM-Vet**: 35.4%-- **MMBench**: 64.3%--## Limitations--1. **Hallucinations** - May describe things not in image-2. **Spatial reasoning** - Struggles with precise locations-3. **Small text** - Difficulty reading fine print-4. **Object counting** - Imprecise for many objects-5. **VRAM requirements** - Need powerful GPU-6. **Inference speed** - Slower than CLIP--## Integration with frameworks--### LangChain--```python-from langchain.llms.base import LLM--class LLaVALLM(LLM):- def _call(self, prompt, stop=None):- # Custom LLaVA inference- return response--llm = LLaVALLM()-```--### Gradio App--```python-import gradio as gr--def chat(image, text, history):- response = ask_llava(model, image, text)- return response--demo = gr.ChatInterface(- chat,- additional_inputs=[gr.Image(type="pil")],- title="LLaVA Chat"-)-demo.launch()-```--## Resources--- **GitHub**: https://github.com/haotian-liu/LLaVA ⭐ 23,000+-- **Paper**: https://arxiv.org/abs/2304.08485-- **Demo**: https://llava.hliu.cc-- **Models**: https://huggingface.co/liuhaotian-- **License**: Apache 2.0--diff --git a/optional-skills/mlops/llava/references/training.md b/optional-skills/mlops/llava/references/training.mddeleted file mode 100644index a4c83e31eb4..00000000000--- a/optional-skills/mlops/llava/references/training.md+++ /dev/null@@ -1,197 +0,0 @@-# LLaVA Training Guide--Guide to training and fine-tuning LLaVA models.--## Training stages--### Stage 1: Feature alignment (Pretraining)--**Purpose**: Align vision encoder with language model--**Data**: 558K image-caption pairs (CC3M subset)--```bash-# Download pretrained projector or train from scratch-bash scripts/v1_5/pretrain.sh-```--**Configuration:**-- Base model: Vicuna-7B or LLaMA-2-7B-- Vision encoder: CLIP ViT-L/14-- Training time: ~20 hours on 8× A100--### Stage 2: Visual instruction tuning--**Purpose**: Teach model to follow visual instructions--**Data**: 150K GPT-generated multimodal instruction data--```bash-# Fine-tune with instruction data-bash scripts/v1_5/finetune.sh-```--**Configuration:**-- Epochs: 1-- Batch size: 128 (across 8 GPUs)-- Learning rate: 2e-5-- Training time: ~24 hours on 8× A100--## Data format--### Instruction data format--```json-[- {- "id": "001",- "image": "path/to/image.jpg",- "conversations": [- {- "from": "human",- "value": "<image>\nWhat is in this image?"- },- {- "from": "gpt",- "value": "The image shows a dog playing in a park."- },- {- "from": "human",- "value": "What breed is the dog?"- },- {- "from": "gpt",- "value": "It appears to be a Golden Retriever."- }- ]- }-]-```--## Fine-tuning on custom data--### Prepare your data--```python-import json--# Create instruction data-data = []-for image_path, qa_pairs in your_dataset:- conversations = []- for q, a in qa_pairs:- conversations.append({"from": "human", "value": f"<image>\n{q}"})- conversations.append({"from": "gpt", "value": a})-- data.append({- "id": str(len(data)),- "image": image_path,- "conversations": conversations- })--# Save-with open("custom_data.json", "w") as f:- json.dump(data, f, indent=2)-```--### Fine-tune script--```bash-#!/usr/bin/env bash--# Set paths-DATA_PATH="custom_data.json"-IMAGE_FOLDER="path/to/images"-MODEL_PATH="liuhaotian/llava-v1.5-7b"-OUTPUT_DIR="./checkpoints/llava-custom"--# Fine-tune-deepspeed llava/train/train_mem.py \- --deepspeed ./scripts/zero2.json \- --model_name_or_path $MODEL_PATH \- --version v1 \- --data_path $DATA_PATH \- --image_folder $IMAGE_FOLDER \- --vision_tower openai/clip-vit-large-patch14-336 \- --mm_projector_type mlp2x_gelu \- --mm_vision_select_layer -2 \- --mm_use_im_start_end False \- --mm_use_im_patch_token False \- --image_aspect_ratio pad \- --group_by_modality_length True \- --bf16 True \- --output_dir $OUTPUT_DIR \- --num_train_epochs 1 \- --per_device_train_batch_size 16 \- --per_device_eval_batch_size 4 \- --gradient_accumulation_steps 1 \- --evaluation_strategy "no" \- --save_strategy "steps" \- --save_steps 50000 \- --save_total_limit 1 \- --learning_rate 2e-5 \- --weight_decay 0. \- --warmup_ratio 0.03 \- --lr_scheduler_type "cosine" \- --logging_steps 1 \- --tf32 True \- --model_max_length 2048 \- --gradient_checkpointing True \- --dataloader_num_workers 4 \- --lazy_preprocess True \- --report_to wandb-```--## LoRA fine-tuning (memory efficient)--```python-from peft import LoraConfig, get_peft_model--# LoRA config-lora_config = LoraConfig(- r=8, # LoRA rank- lora_alpha=16,- target_modules=["q_proj", "v_proj"],- lora_dropout=0.05,- bias="none",- task_type="CAUSAL_LM"-)--# Apply LoRA-model = get_peft_model(base_model, lora_config)--# Train with much lower memory-```--## Hardware requirements--### Full fine-tuning--- **7B model**: 8× A100 (40GB)-- **13B model**: 8× A100 (80GB)-- **Training time**: 20-48 hours--### LoRA fine-tuning--- **7B model**: 1× A100 (40GB)-- **13B model**: 2× A100 (40GB)-- **Training time**: 10-24 hours--## Best practices--1. **Start with pretrained** - Don't train from scratch-2. **Use LoRA for efficiency** - 10× less memory-3. **Quality over quantity** - 1K high-quality > 10K low-quality-4. **Multi-turn conversations** - More engaging than single Q&A-5. **Diverse images** - Cover different scenarios-6. **Clear instructions** - Specific questions get better answers-7. **Monitor loss** - Should decrease smoothly-8. **Save checkpoints** - Training can fail-9. **Test regularly** - Validate on held-out set-10. **Use DeepSpeed** - For multi-GPU training--## Resources--- **Training script**: https://github.com/haotian-liu/LLaVA/tree/main/scripts-- **Data format**: https://github.com/haotian-liu/LLaVA/blob/main/docs/Data.md-- **Paper**: https://arxiv.org/abs/2304.08485diff --git a/optional-skills/mlops/models/huggingface-hub/SKILL.md b/optional-skills/mlops/models/huggingface-hub/SKILL.mddeleted file mode 100644index bb82ac1c040..00000000000--- a/optional-skills/mlops/models/huggingface-hub/SKILL.md+++ /dev/null@@ -1,81 +0,0 @@-----name: huggingface-hub-description: "HuggingFace hf CLI: search/download/upload models, datasets."-version: 1.0.1-author: Hugging Face-license: MIT-tags: [huggingface, hf, models, datasets, hub, mlops]-platforms: [linux, macos, windows]------# Hugging Face CLI (`hf`) Reference Guide--The `hf` command is the modern command-line interface for interacting with the Hugging Face Hub, providing tools to manage repositories, models, datasets, and Spaces.--> **IMPORTANT:** The `hf` command replaces the now deprecated `huggingface-cli` command.--## Quick Start-* **Installation:** `curl -LsSf https://hf.co/cli/install.sh | bash -s`-* **Help:** Use `hf --help` to view all available functions and real-world examples.-* **Authentication:** Recommended via `HF_TOKEN` environment variable or the `--token` flag.-------## Core Commands--### General Operations-* `hf download REPO_ID`: Download files from the Hub.-* `hf upload REPO_ID`: Upload files/folders (recommended for single-commit; also handles resumable uploads of large directories).-* `hf upload-large-folder REPO_ID LOCAL_PATH`: **[Deprecated]** — use `hf upload` instead.-* `hf sync`: Sync files between a local directory and a bucket.-* `hf env` / `hf version`: View environment and version details.--### Authentication (`hf auth`)-* `login` / `logout`: Manage sessions using tokens from [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).-* `list` / `switch`: Manage and toggle between multiple stored access tokens.-* `whoami`: Identify the currently logged-in account.--### Repository Management (`hf repos`)-* `create` / `delete`: Create or permanently remove repositories.-* `duplicate`: Clone a model, dataset, or Space to a new ID.-* `move`: Transfer a repository between namespaces.-* `branch` / `tag`: Manage Git-like references.-* `delete-files`: Remove specific files using patterns.-------## Specialized Hub Interactions--### Datasets & Models-* **Datasets:** `hf datasets list`, `info`, and `parquet` (list parquet URLs).-* **SQL Queries:** `hf datasets sql SQL` — Execute raw SQL via DuckDB against dataset parquet URLs.-* **Models:** `hf models list` and `info`.-* **Papers:** `hf papers ls` — View daily papers.--### Discussions & Pull Requests (`hf discussions`)-* Manage the lifecycle of Hub contributions: `list`, `create`, `info`, `comment`, `close`, `reopen`, and `rename`.-* `diff`: View changes in a PR.-* `merge`: Finalize pull requests.--### Infrastructure & Compute-* **Endpoints:** Deploy and manage Inference Endpoints (`deploy`, `pause`, `resume`, `scale-to-zero`, `catalog`).-* **Jobs:** Run compute tasks on HF infrastructure. Includes `hf jobs uv` for running Python scripts with inline dependencies and `stats` for resource monitoring.-* **Spaces:** Manage interactive apps. Includes `dev-mode` and `hot-reload` for Python files without full restarts.--### Storage & Automation-* **Buckets:** Full S3-like bucket management (`create`, `cp`, `mv`, `rm`, `sync`).-* **Cache:** Manage local storage with `list`, `prune` (remove detached revisions), and `verify` (checksum checks).-* **Webhooks:** Automate workflows by managing Hub webhooks (`create`, `watch`, `enable`/`disable`).-* **Collections:** Organize Hub items into collections (`add-item`, `update`, `list`).-------## Advanced Usage & Tips--### Global Flags-* `--format json`: Produces machine-readable output for automation.-* `-q` / `--quiet`: Limits output to IDs only.--### Extensions & Skills-* **Extensions:** Extend CLI functionality via GitHub repositories using `hf extensions install REPO_ID`.-* **Skills:** Manage AI assistant skills with `hf skills add`.diff --git a/optional-skills/mlops/obliteratus/SKILL.md b/optional-skills/mlops/obliteratus/SKILL.mdindex bea9cfe298f..6464aab427d 100644--- a/optional-skills/mlops/obliteratus/SKILL.md+++ b/optional-skills/mlops/obliteratus/SKILL.md@@ -9,7 +9,7 @@ platforms: [linux, macos]metadata:hermes:tags: [Abliteration, Uncensoring, Refusal-Removal, LLM, Weight-Projection, SVD, Mechanistic-Interpretability, HuggingFace, Model-Surgery]- related_skills: [serving-llms-vllm, llama-cpp, huggingface-tokenizers]+ related_skills: [serving-llms-vllm, llama-cpp]---# OBLITERATUS Skilldiff --git a/optional-skills/mlops/pinecone/SKILL.md b/optional-skills/mlops/pinecone/SKILL.mddeleted file mode 100644index b32dcddadfb..00000000000--- a/optional-skills/mlops/pinecone/SKILL.md+++ /dev/null@@ -1,381 +0,0 @@-----name: pinecone-description: Managed vector DB for production RAG and search.-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [pinecone]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [RAG, Pinecone, Vector Database, Managed Service, Serverless, Hybrid Search, Production, Auto-Scaling, Low Latency, Recommendations]-------# Pinecone - Managed Vector Database--The vector database for production AI applications.--## When to use Pinecone--**Use when:**-- Need managed, serverless vector database-- Production RAG applications-- Auto-scaling required-- Low latency critical (<100ms)-- Don't want to manage infrastructure-- Need hybrid search (dense + sparse vectors)--**Metrics**:-- Fully managed SaaS-- Auto-scales to billions of vectors-- **p95 latency <100ms**-- 99.9% uptime SLA--**Use alternatives instead**:-- **Chroma**: Self-hosted, open-source-- **FAISS**: Offline, pure similarity search-- **Weaviate**: Self-hosted with more features--## Quick start--### Installation--```bash-pip install pinecone-```--> Note: the old `pinecone-client` package is deprecated. Install `pinecone` (v5+; current 9.x). The import stays `from pinecone import Pinecone`.--### Basic usage--```python-from pinecone import Pinecone, ServerlessSpec--# Initialize-pc = Pinecone(api_key="your-api-key")--# Create index-pc.create_index(- name="my-index",- dimension=1536, # Must match embedding dimension- metric="cosine", # or "euclidean", "dotproduct"- spec=ServerlessSpec(cloud="aws", region="us-east-1")-)--# Connect to index-index = pc.Index("my-index")--# Upsert vectors-index.upsert(vectors=[- {"id": "vec1", "values": [0.1, 0.2, ...], "metadata": {"category": "A"}},- {"id": "vec2", "values": [0.3, 0.4, ...], "metadata": {"category": "B"}}-])--# Query-results = index.query(- vector=[0.1, 0.2, ...],- top_k=5,- include_metadata=True-)--print(results["matches"])-```--## Core operations--### Create index--```python-# Serverless (recommended)-pc.create_index(- name="my-index",- dimension=1536,- metric="cosine",- spec=ServerlessSpec(- cloud="aws", # or "gcp", "azure"- region="us-east-1"- )-)--# Pod-based (for consistent performance)-from pinecone import PodSpec--pc.create_index(- name="my-index",- dimension=1536,- metric="cosine",- spec=PodSpec(- environment="us-east1-gcp",- pod_type="p1.x1"- )-)-```--### Upsert vectors--```python-# Single upsert-index.upsert(vectors=[- {- "id": "doc1",- "values": [0.1, 0.2, ...], # 1536 dimensions- "metadata": {- "text": "Document content",- "category": "tutorial",- "timestamp": "2025-01-01"- }- }-])--# Batch upsert (recommended)-vectors = [- {"id": f"vec{i}", "values": embedding, "metadata": metadata}- for i, (embedding, metadata) in enumerate(zip(embeddings, metadatas))-]--index.upsert(vectors=vectors, batch_size=100)-```--### Query vectors--```python-# Basic query-results = index.query(- vector=[0.1, 0.2, ...],- top_k=10,- include_metadata=True,- include_values=False-)--# With metadata filtering-results = index.query(- vector=[0.1, 0.2, ...],- top_k=5,- filter={"category": {"$eq": "tutorial"}}-)--# Namespace query-results = index.query(- vector=[0.1, 0.2, ...],- top_k=5,- namespace="production"-)--# Access results-for match in results["matches"]:- print(f"ID: {match['id']}")- print(f"Score: {match['score']}")- print(f"Metadata: {match['metadata']}")-```--### Metadata filtering--```python-# Exact match-filter = {"category": "tutorial"}--# Comparison-filter = {"price": {"$gte": 100}} # $gt, $gte, $lt, $lte, $ne--# Logical operators-filter = {- "$and": [- {"category": "tutorial"},- {"difficulty": {"$lte": 3}}- ]-} # Also: $or--# In operator-filter = {"tags": {"$in": ["python", "ml"]}}-```--## Namespaces--```python-# Partition data by namespace-index.upsert(- vectors=[{"id": "vec1", "values": [...]}],- namespace="user-123"-)--# Query specific namespace-results = index.query(- vector=[...],- namespace="user-123",- top_k=5-)--# List namespaces-stats = index.describe_index_stats()-print(stats['namespaces'])-```--## Hybrid search (dense + sparse)--```python-# Upsert with sparse vectors-index.upsert(vectors=[- {- "id": "doc1",- "values": [0.1, 0.2, ...], # Dense vector- "sparse_values": {- "indices": [10, 45, 123], # Token IDs- "values": [0.5, 0.3, 0.8] # TF-IDF scores- },- "metadata": {"text": "..."}- }-])--# Hybrid query-# NOTE: index.query() does NOT accept an `alpha` kwarg. Pinecone stores a-# single sparse-dense vector, so weighting must be applied by pre-scaling the-# query vectors before sending them. Use the hybrid_score_norm helper below-# (alpha * dense + (1 - alpha) * sparse; alpha=1 → pure dense, 0 → pure sparse).--def hybrid_score_norm(dense, sparse, alpha: float):- """Scale dense/sparse query vectors for weighted hybrid search."""- if not 0 <= alpha <= 1:- raise ValueError("alpha must be between 0 and 1")- scaled_sparse = {- "indices": sparse["indices"],- "values": [v * (1 - alpha) for v in sparse["values"]],- }- return [v * alpha for v in dense], scaled_sparse--hdense, hsparse = hybrid_score_norm(- dense=[0.1, 0.2, ...],- sparse={"indices": [10, 45], "values": [0.5, 0.3]},- alpha=0.5, # 0=sparse, 1=dense, 0.5=balanced-)--results = index.query(- vector=hdense,- sparse_vector=hsparse,- top_k=5,-)-```--## LangChain integration--```python-from langchain_pinecone import PineconeVectorStore-from langchain_openai import OpenAIEmbeddings--# Create vector store-vectorstore = PineconeVectorStore.from_documents(- documents=docs,- embedding=OpenAIEmbeddings(),- index_name="my-index"-)--# Query-results = vectorstore.similarity_search("query", k=5)--# With metadata filter-results = vectorstore.similarity_search(- "query",- k=5,- filter={"category": "tutorial"}-)--# As retriever-retriever = vectorstore.as_retriever(search_kwargs={"k": 10})-```--## LlamaIndex integration--```python-from llama_index.vector_stores.pinecone import PineconeVectorStore--# Connect to Pinecone-pc = Pinecone(api_key="your-key")-pinecone_index = pc.Index("my-index")--# Create vector store-vector_store = PineconeVectorStore(pinecone_index=pinecone_index)--# Use in LlamaIndex-from llama_index.core import StorageContext, VectorStoreIndex--storage_context = StorageContext.from_defaults(vector_store=vector_store)-index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)-```--## Index management--```python-# List indices-indexes = pc.list_indexes()--# Describe index-index_info = pc.describe_index("my-index")-print(index_info)--# Get index stats-stats = index.describe_index_stats()-print(f"Total vectors: {stats['total_vector_count']}")-print(f"Namespaces: {stats['namespaces']}")--# Delete index-pc.delete_index("my-index")-```--## Delete vectors--```python-# Delete by ID-index.delete(ids=["vec1", "vec2"])--# Delete by filter-index.delete(filter={"category": "old"})--# Delete all in namespace-index.delete(delete_all=True, namespace="test")--# Delete entire index-index.delete(delete_all=True)-```--## Best practices--1. **Use serverless** - Auto-scaling, cost-effective-2. **Batch upserts** - More efficient (100-200 per batch)-3. **Add metadata** - Enable filtering-4. **Use namespaces** - Isolate data by user/tenant-5. **Monitor usage** - Check Pinecone dashboard-6. **Optimize filters** - Index frequently filtered fields-7. **Test with free tier** - 1 index, 100K vectors free-8. **Use hybrid search** - Better quality-9. **Set appropriate dimensions** - Match embedding model-10. **Regular backups** - Export important data--## Performance--| Operation | Latency | Notes |-|-----------|---------|-------|-| Upsert | ~50-100ms | Per batch |-| Query (p50) | ~50ms | Depends on index size |-| Query (p95) | ~100ms | SLA target |-| Metadata filter | ~+10-20ms | Additional overhead |--## Pricing (as of 2025)--**Serverless**:-- $0.096 per million read units-- $0.06 per million write units-- $0.06 per GB storage/month--**Free tier**:-- 1 serverless index-- 100K vectors (1536 dimensions)-- Great for prototyping--## Resources--- **Website**: https://www.pinecone.io-- **Docs**: https://docs.pinecone.io-- **Console**: https://app.pinecone.io-- **Pricing**: https://www.pinecone.io/pricing--diff --git a/optional-skills/mlops/pinecone/references/deployment.md b/optional-skills/mlops/pinecone/references/deployment.mddeleted file mode 100644index 0f32988c6b1..00000000000--- a/optional-skills/mlops/pinecone/references/deployment.md+++ /dev/null@@ -1,181 +0,0 @@-# Pinecone Deployment Guide--Production deployment patterns for Pinecone.--## Serverless vs Pod-based--### Serverless (Recommended)--```python-from pinecone import Pinecone, ServerlessSpec--pc = Pinecone(api_key="your-key")--# Create serverless index-pc.create_index(- name="my-index",- dimension=1536,- metric="cosine",- spec=ServerlessSpec(- cloud="aws", # or "gcp", "azure"- region="us-east-1"- )-)-```--**Benefits:**-- Auto-scaling-- Pay per usage-- No infrastructure management-- Cost-effective for variable load--**Use when:**-- Variable traffic-- Cost optimization important-- Don't need consistent latency--### Pod-based--```python-from pinecone import PodSpec--pc.create_index(- name="my-index",- dimension=1536,- metric="cosine",- spec=PodSpec(- environment="us-east1-gcp",- pod_type="p1.x1", # or p1.x2, p1.x4, p1.x8- pods=2, # Number of pods- replicas=2 # High availability- )-)-```--**Benefits:**-- Consistent performance-- Predictable latency-- Higher throughput-- Dedicated resources--**Use when:**-- Production workloads-- Need consistent p95 latency-- High throughput required--## Hybrid search--### Dense + Sparse vectors--```python-# Upsert with both dense and sparse vectors-index.upsert(vectors=[- {- "id": "doc1",- "values": [0.1, 0.2, ...], # Dense (semantic)- "sparse_values": {- "indices": [10, 45, 123], # Token IDs- "values": [0.5, 0.3, 0.8] # TF-IDF/BM25 scores- },- "metadata": {"text": "..."}- }-])--# Hybrid query-results = index.query(- vector=[0.1, 0.2, ...], # Dense query- sparse_vector={- "indices": [10, 45],- "values": [0.5, 0.3]- },- top_k=10,- alpha=0.5 # 0=sparse only, 1=dense only, 0.5=balanced-)-```--**Benefits:**-- Best of both worlds-- Semantic + keyword matching-- Better recall than either alone--## Namespaces for multi-tenancy--```python-# Separate data by user/tenant-index.upsert(- vectors=[{"id": "doc1", "values": [...]}],- namespace="user-123"-)--# Query specific namespace-results = index.query(- vector=[...],- namespace="user-123",- top_k=5-)--# List namespaces-stats = index.describe_index_stats()-print(stats['namespaces'])-```--**Use cases:**-- Multi-tenant SaaS-- User-specific data isolation-- A/B testing (prod/staging namespaces)--## Metadata filtering--### Exact match--```python-results = index.query(- vector=[...],- filter={"category": "tutorial"},- top_k=5-)-```--### Range queries--```python-results = index.query(- vector=[...],- filter={"price": {"$gte": 100, "$lte": 500}},- top_k=5-)-```--### Complex filters--```python-results = index.query(- vector=[...],- filter={- "$and": [- {"category": {"$in": ["tutorial", "guide"]}},- {"difficulty": {"$lte": 3}},- {"published": {"$gte": "2024-01-01"}}- ]- },- top_k=5-)-```--## Best practices--1. **Use serverless for development** - Cost-effective-2. **Switch to pods for production** - Consistent performance-3. **Implement namespaces** - Multi-tenancy-4. **Add metadata strategically** - Enable filtering-5. **Use hybrid search** - Better quality-6. **Batch upserts** - 100-200 vectors per batch-7. **Monitor usage** - Check Pinecone dashboard-8. **Set up alerts** - Usage/cost thresholds-9. **Regular backups** - Export important data-10. **Test filters** - Verify performance--## Resources--- **Docs**: https://docs.pinecone.io-- **Console**: https://app.pinecone.iodiff --git a/optional-skills/mlops/qdrant/SKILL.md b/optional-skills/mlops/qdrant/SKILL.mddeleted file mode 100644index cbea02af7c4..00000000000--- a/optional-skills/mlops/qdrant/SKILL.md+++ /dev/null@@ -1,505 +0,0 @@-----name: qdrant-description: Vector search engine for production RAG systems.-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [qdrant-client>=1.14.0]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [RAG, Vector Search, Qdrant, Semantic Search, Embeddings, Similarity Search, HNSW, Production, Distributed]-------# Qdrant - Vector Similarity Search Engine--High-performance vector database written in Rust for production RAG and semantic search.--## When to use Qdrant--**Use Qdrant when:**-- Building production RAG systems requiring low latency-- Need hybrid search (vectors + metadata filtering)-- Require horizontal scaling with sharding/replication-- Want on-premise deployment with full data control-- Need multi-vector storage per record (dense + sparse)-- Building real-time recommendation systems--**Key features:**-- **Rust-powered**: Memory-safe, high performance-- **Rich filtering**: Filter by any payload field during search-- **Multiple vectors**: Dense, sparse, multi-dense per point-- **Quantization**: Scalar, product, binary for memory efficiency-- **Distributed**: Raft consensus, sharding, replication-- **REST + gRPC**: Both APIs with full feature parity--**Use alternatives instead:**-- **Chroma**: Simpler setup, embedded use cases-- **FAISS**: Maximum raw speed, research/batch processing-- **Pinecone**: Fully managed, zero ops preferred-- **Weaviate**: GraphQL preference, built-in vectorizers--## Quick start--### Installation--```bash-# Python client-pip install qdrant-client--# Docker (recommended for development)-docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant--# Docker with persistent storage-docker run -p 6333:6333 -p 6334:6334 \- -v $(pwd)/qdrant_storage:/qdrant/storage \- qdrant/qdrant-```--### Basic usage--```python-from qdrant_client import QdrantClient-from qdrant_client.models import Distance, VectorParams, PointStruct--# Connect to Qdrant-client = QdrantClient(host="localhost", port=6333)--# Create collection-client.create_collection(- collection_name="documents",- vectors_config=VectorParams(size=384, distance=Distance.COSINE)-)--# Insert vectors with payload-client.upsert(- collection_name="documents",- points=[- PointStruct(- id=1,- vector=[0.1, 0.2, ...], # 384-dim vector- payload={"title": "Doc 1", "category": "tech"}- ),- PointStruct(- id=2,- vector=[0.3, 0.4, ...],- payload={"title": "Doc 2", "category": "science"}- )- ]-)--# Search with filtering (query_points is the current API; client.search is removed in qdrant-client 1.14+)-response = client.query_points(- collection_name="documents",- query=[0.15, 0.25, ...],- query_filter={- "must": [{"key": "category", "match": {"value": "tech"}}]- },- limit=10-)--for point in response.points:- print(f"ID: {point.id}, Score: {point.score}, Payload: {point.payload}")-```--## Core concepts--### Points - Basic data unit--```python-from qdrant_client.models import PointStruct--# Point = ID + Vector(s) + Payload-point = PointStruct(- id=123, # Integer or UUID string- vector=[0.1, 0.2, 0.3, ...], # Dense vector- payload={ # Arbitrary JSON metadata- "title": "Document title",- "category": "tech",- "timestamp": 1699900000,- "tags": ["python", "ml"]- }-)--# Batch upsert (recommended)-client.upsert(- collection_name="documents",- points=[point1, point2, point3],- wait=True # Wait for indexing-)-```--### Collections - Vector containers--```python-from qdrant_client.models import VectorParams, Distance, HnswConfigDiff--# Create with HNSW configuration-client.create_collection(- collection_name="documents",- vectors_config=VectorParams(- size=384, # Vector dimensions- distance=Distance.COSINE # COSINE, EUCLID, DOT, MANHATTAN- ),- hnsw_config=HnswConfigDiff(- m=16, # Connections per node (default 16)- ef_construct=100, # Build-time accuracy (default 100)- full_scan_threshold=10000 # Switch to brute force below this- ),- on_disk_payload=True # Store payload on disk-)--# Collection info-info = client.get_collection("documents")-print(f"Points: {info.points_count}, Vectors: {info.vectors_count}")-```--### Distance metrics--| Metric | Use Case | Range |-|--------|----------|-------|-| `COSINE` | Text embeddings, normalized vectors | 0 to 2 |-| `EUCLID` | Spatial data, image features | 0 to ∞ |-| `DOT` | Recommendations, unnormalized | -∞ to ∞ |-| `MANHATTAN` | Sparse features, discrete data | 0 to ∞ |--## Search operations--### Basic search--```python-# Simple nearest neighbor search (returns a QueryResponse; use .points)-response = client.query_points(- collection_name="documents",- query=[0.1, 0.2, ...],- limit=10,- with_payload=True,- with_vectors=False # Don't return vectors (faster)-)-results = response.points-```--### Filtered search--```python-from qdrant_client.models import Filter, FieldCondition, MatchValue, Range--# Complex filtering-response = client.query_points(- collection_name="documents",- query=query_embedding,- query_filter=Filter(- must=[- FieldCondition(key="category", match=MatchValue(value="tech")),- FieldCondition(key="timestamp", range=Range(gte=1699000000))- ],- must_not=[- FieldCondition(key="status", match=MatchValue(value="archived"))- ]- ),- limit=10-).points--# Shorthand filter syntax-response = client.query_points(- collection_name="documents",- query=query_embedding,- query_filter={- "must": [- {"key": "category", "match": {"value": "tech"}},- {"key": "price", "range": {"gte": 10, "lte": 100}}- ]- },- limit=10-).points-```--### Batch search--```python-from qdrant_client.models import QueryRequest--# Multiple queries in one request (search_batch is replaced by query_batch_points)-responses = client.query_batch_points(- collection_name="documents",- requests=[- QueryRequest(query=[0.1, ...], limit=5),- QueryRequest(query=[0.2, ...], limit=5, filter={"must": [...]}),- QueryRequest(query=[0.3, ...], limit=10)- ]-)-# Each element is a QueryResponse; use .points-for resp in responses:- for point in resp.points:- print(point.id, point.score)-```--## RAG integration--### With sentence-transformers--```python-from sentence_transformers import SentenceTransformer-from qdrant_client import QdrantClient-from qdrant_client.models import VectorParams, Distance, PointStruct--# Initialize-encoder = SentenceTransformer("all-MiniLM-L6-v2")-client = QdrantClient(host="localhost", port=6333)--# Create collection-client.create_collection(- collection_name="knowledge_base",- vectors_config=VectorParams(size=384, distance=Distance.COSINE)-)--# Index documents-documents = [- {"id": 1, "text": "Python is a programming language", "source": "wiki"},- {"id": 2, "text": "Machine learning uses algorithms", "source": "textbook"},-]--points = [- PointStruct(- id=doc["id"],- vector=encoder.encode(doc["text"]).tolist(),- payload={"text": doc["text"], "source": doc["source"]}- )- for doc in documents-]-client.upsert(collection_name="knowledge_base", points=points)--# RAG retrieval-def retrieve(query: str, top_k: int = 5) -> list[dict]:- query_vector = encoder.encode(query).tolist()- response = client.query_points(- collection_name="knowledge_base",- query=query_vector,- limit=top_k- )- return [{"text": r.payload["text"], "score": r.score} for r in response.points]--# Use in RAG pipeline-context = retrieve("What is Python?")-prompt = f"Context: {context}\n\nQuestion: What is Python?"-```--### With LangChain--```python-from langchain_community.vectorstores import Qdrant-from langchain_community.embeddings import HuggingFaceEmbeddings--embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")-vectorstore = Qdrant.from_documents(documents, embeddings, url="http://localhost:6333", collection_name="docs")-retriever = vectorstore.as_retriever(search_kwargs={"k": 5})-```--### With LlamaIndex--```python-from llama_index.vector_stores.qdrant import QdrantVectorStore-from llama_index.core import VectorStoreIndex, StorageContext--vector_store = QdrantVectorStore(client=client, collection_name="llama_docs")-storage_context = StorageContext.from_defaults(vector_store=vector_store)-index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)-query_engine = index.as_query_engine()-```--## Multi-vector support--### Named vectors (different embedding models)--```python-from qdrant_client.models import VectorParams, Distance--# Collection with multiple vector types-client.create_collection(- collection_name="hybrid_search",- vectors_config={- "dense": VectorParams(size=384, distance=Distance.COSINE),- "sparse": VectorParams(size=30000, distance=Distance.DOT)- }-)--# Insert with named vectors-client.upsert(- collection_name="hybrid_search",- points=[- PointStruct(- id=1,- vector={- "dense": dense_embedding,- "sparse": sparse_embedding- },- payload={"text": "document text"}- )- ]-)--# Search specific named vector (pass the vector name via `using`)-response = client.query_points(- collection_name="hybrid_search",- query=query_dense,- using="dense", # Specify which named vector to search- limit=10-)-results = response.points-```--### Sparse vectors (BM25, SPLADE)--```python-from qdrant_client.models import SparseVectorParams, SparseIndexParams, SparseVector--# Collection with sparse vectors-client.create_collection(- collection_name="sparse_search",- vectors_config={},- sparse_vectors_config={"text": SparseVectorParams(index=SparseIndexParams(on_disk=False))}-)--# Insert sparse vector-client.upsert(- collection_name="sparse_search",- points=[PointStruct(id=1, vector={"text": SparseVector(indices=[1, 5, 100], values=[0.5, 0.8, 0.2])}, payload={"text": "document"})]-)-```--## Quantization (memory optimization)--```python-from qdrant_client.models import ScalarQuantization, ScalarQuantizationConfig, ScalarType--# Scalar quantization (4x memory reduction)-client.create_collection(- collection_name="quantized",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- quantization_config=ScalarQuantization(- scalar=ScalarQuantizationConfig(- type=ScalarType.INT8,- quantile=0.99, # Clip outliers- always_ram=True # Keep quantized in RAM- )- )-)--# Search with rescoring-response = client.query_points(- collection_name="quantized",- query=query,- search_params={"quantization": {"rescore": True}}, # Rescore top results- limit=10-)-results = response.points-```--## Payload indexing--```python-from qdrant_client.models import PayloadSchemaType--# Create payload index for faster filtering-client.create_payload_index(- collection_name="documents",- field_name="category",- field_schema=PayloadSchemaType.KEYWORD-)--client.create_payload_index(- collection_name="documents",- field_name="timestamp",- field_schema=PayloadSchemaType.INTEGER-)--# Index types: KEYWORD, INTEGER, FLOAT, GEO, TEXT (full-text), BOOL-```--## Production deployment--### Qdrant Cloud--```python-from qdrant_client import QdrantClient--# Connect to Qdrant Cloud-client = QdrantClient(- url="https://your-cluster.cloud.qdrant.io",- api_key="your-api-key"-)-```--### Performance tuning--```python-# Optimize for search speed (higher recall)-client.update_collection(- collection_name="documents",- hnsw_config=HnswConfigDiff(ef_construct=200, m=32)-)--# Optimize for indexing speed (bulk loads)-client.update_collection(- collection_name="documents",- optimizer_config={"indexing_threshold": 20000}-)-```--## Best practices--1. **Batch operations** - Use batch upsert/search for efficiency-2. **Payload indexing** - Index fields used in filters-3. **Quantization** - Enable for large collections (>1M vectors)-4. **Sharding** - Use for collections >10M vectors-5. **On-disk storage** - Enable `on_disk_payload` for large payloads-6. **Connection pooling** - Reuse client instances--## Common issues--**Slow search with filters:**-```python-# Create payload index for filtered fields-client.create_payload_index(- collection_name="docs",- field_name="category",- field_schema=PayloadSchemaType.KEYWORD-)-```--**Out of memory:**-```python-# Enable quantization and on-disk storage-client.create_collection(- collection_name="large_collection",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- quantization_config=ScalarQuantization(...),- on_disk_payload=True-)-```--**Connection issues:**-```python-# Use timeout and retry-client = QdrantClient(- host="localhost",- port=6333,- timeout=30,- prefer_grpc=True # gRPC for better performance-)-```--## References--- **[Advanced Usage](references/advanced-usage.md)** - Distributed mode, hybrid search, recommendations-- **[Troubleshooting](references/troubleshooting.md)** - Common issues, debugging, performance tuning--## Resources--- **GitHub**: https://github.com/qdrant/qdrant (22k+ stars)-- **Docs**: https://qdrant.tech/documentation/-- **Python Client**: https://github.com/qdrant/qdrant-client-- **Cloud**: https://cloud.qdrant.io-- **Version**: 1.14.0+-- **License**: Apache 2.0diff --git a/optional-skills/mlops/qdrant/references/advanced-usage.md b/optional-skills/mlops/qdrant/references/advanced-usage.mddeleted file mode 100644index 54a8b25d11d..00000000000--- a/optional-skills/mlops/qdrant/references/advanced-usage.md+++ /dev/null@@ -1,648 +0,0 @@-# Qdrant Advanced Usage Guide--## Distributed Deployment--### Cluster Setup--Qdrant uses Raft consensus for distributed coordination.--```yaml-# docker-compose.yml for 3-node cluster-version: '3.8'-services:- qdrant-node-1:- image: qdrant/qdrant:latest- ports:- - "6333:6333"- - "6334:6334"- - "6335:6335"- volumes:- - ./node1_storage:/qdrant/storage- environment:- - QDRANT__CLUSTER__ENABLED=true- - QDRANT__CLUSTER__P2P__PORT=6335- - QDRANT__SERVICE__HTTP_PORT=6333- - QDRANT__SERVICE__GRPC_PORT=6334-- qdrant-node-2:- image: qdrant/qdrant:latest- ports:- - "6343:6333"- - "6344:6334"- - "6345:6335"- volumes:- - ./node2_storage:/qdrant/storage- environment:- - QDRANT__CLUSTER__ENABLED=true- - QDRANT__CLUSTER__P2P__PORT=6335- - QDRANT__CLUSTER__BOOTSTRAP=http://qdrant-node-1:6335- depends_on:- - qdrant-node-1-- qdrant-node-3:- image: qdrant/qdrant:latest- ports:- - "6353:6333"- - "6354:6334"- - "6355:6335"- volumes:- - ./node3_storage:/qdrant/storage- environment:- - QDRANT__CLUSTER__ENABLED=true- - QDRANT__CLUSTER__P2P__PORT=6335- - QDRANT__CLUSTER__BOOTSTRAP=http://qdrant-node-1:6335- depends_on:- - qdrant-node-1-```--### Sharding Configuration--```python-from qdrant_client import QdrantClient-from qdrant_client.models import VectorParams, Distance, ShardingMethod--client = QdrantClient(host="localhost", port=6333)--# Create sharded collection-client.create_collection(- collection_name="large_collection",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- shard_number=6, # Number of shards- replication_factor=2, # Replicas per shard- write_consistency_factor=1 # Required acks for write-)--# Check cluster status-cluster_info = client.get_cluster_info()-print(f"Peers: {cluster_info.peers}")-print(f"Raft state: {cluster_info.raft_info}")-```--### Replication and Consistency--```python-from qdrant_client.models import WriteOrdering--# Strong consistency write-client.upsert(- collection_name="critical_data",- points=points,- ordering=WriteOrdering.STRONG # Wait for all replicas-)--# Eventual consistency (faster)-client.upsert(- collection_name="logs",- points=points,- ordering=WriteOrdering.WEAK # Return after primary ack-)--# Read from specific shard-results = client.search(- collection_name="documents",- query_vector=query,- consistency="majority" # Read from majority of replicas-)-```--## Hybrid Search--### Dense + Sparse Vectors--Combine semantic (dense) and keyword (sparse) search:--```python-from qdrant_client.models import (- VectorParams, SparseVectorParams, SparseIndexParams,- Distance, PointStruct, SparseVector, Prefetch, Query-)--# Create hybrid collection-client.create_collection(- collection_name="hybrid",- vectors_config={- "dense": VectorParams(size=384, distance=Distance.COSINE)- },- sparse_vectors_config={- "sparse": SparseVectorParams(- index=SparseIndexParams(on_disk=False)- )- }-)--# Insert with both vector types-def encode_sparse(text: str) -> SparseVector:- """Simple BM25-like sparse encoding"""- from collections import Counter- tokens = text.lower().split()- counts = Counter(tokens)- # Map tokens to indices (use vocabulary in production)- indices = [hash(t) % 30000 for t in counts.keys()]- values = list(counts.values())- return SparseVector(indices=indices, values=values)--client.upsert(- collection_name="hybrid",- points=[- PointStruct(- id=1,- vector={- "dense": dense_encoder.encode("Python programming").tolist(),- "sparse": encode_sparse("Python programming language code")- },- payload={"text": "Python programming language code"}- )- ]-)--# Hybrid search with Reciprocal Rank Fusion (RRF)-from qdrant_client.models import FusionQuery--results = client.query_points(- collection_name="hybrid",- prefetch=[- Prefetch(query=dense_query, using="dense", limit=20),- Prefetch(query=sparse_query, using="sparse", limit=20)- ],- query=FusionQuery(fusion="rrf"), # Combine results- limit=10-)-```--### Multi-Stage Search--```python-from qdrant_client.models import Prefetch, Query--# Two-stage retrieval: coarse then fine-results = client.query_points(- collection_name="documents",- prefetch=[- Prefetch(- query=query_vector,- limit=100, # Broad first stage- params={"quantization": {"rescore": False}} # Fast, approximate- )- ],- query=Query(nearest=query_vector),- limit=10,- params={"quantization": {"rescore": True}} # Accurate reranking-)-```--## Recommendations--### Item-to-Item Recommendations--```python-# Find similar items-recommendations = client.recommend(- collection_name="products",- positive=[1, 2, 3], # IDs user liked- negative=[4], # IDs user disliked- limit=10-)--# With filtering-recommendations = client.recommend(- collection_name="products",- positive=[1, 2],- query_filter={- "must": [- {"key": "category", "match": {"value": "electronics"}},- {"key": "in_stock", "match": {"value": True}}- ]- },- limit=10-)-```--### Lookup from Another Collection--```python-from qdrant_client.models import RecommendStrategy, LookupLocation--# Recommend using vectors from another collection-results = client.recommend(- collection_name="products",- positive=[- LookupLocation(- collection_name="user_history",- id="user_123"- )- ],- strategy=RecommendStrategy.AVERAGE_VECTOR,- limit=10-)-```--## Advanced Filtering--### Nested Payload Filtering--```python-from qdrant_client.models import Filter, FieldCondition, MatchValue, NestedCondition--# Filter on nested objects-results = client.search(- collection_name="documents",- query_vector=query,- query_filter=Filter(- must=[- NestedCondition(- key="metadata",- filter=Filter(- must=[- FieldCondition(- key="author.name",- match=MatchValue(value="John")- )- ]- )- )- ]- ),- limit=10-)-```--### Geo Filtering--```python-from qdrant_client.models import FieldCondition, GeoRadius, GeoPoint--# Find within radius-results = client.search(- collection_name="locations",- query_vector=query,- query_filter=Filter(- must=[- FieldCondition(- key="location",- geo_radius=GeoRadius(- center=GeoPoint(lat=40.7128, lon=-74.0060),- radius=5000 # meters- )- )- ]- ),- limit=10-)--# Geo bounding box-from qdrant_client.models import GeoBoundingBox--results = client.search(- collection_name="locations",- query_vector=query,- query_filter=Filter(- must=[- FieldCondition(- key="location",- geo_bounding_box=GeoBoundingBox(- top_left=GeoPoint(lat=40.8, lon=-74.1),- bottom_right=GeoPoint(lat=40.6, lon=-73.9)- )- )- ]- ),- limit=10-)-```--### Full-Text Search--```python-from qdrant_client.models import TextIndexParams, TokenizerType--# Create text index-client.create_payload_index(- collection_name="documents",- field_name="content",- field_schema=TextIndexParams(- type="text",- tokenizer=TokenizerType.WORD,- min_token_len=2,- max_token_len=15,- lowercase=True- )-)--# Full-text filter-from qdrant_client.models import MatchText--results = client.search(- collection_name="documents",- query_vector=query,- query_filter=Filter(- must=[- FieldCondition(- key="content",- match=MatchText(text="machine learning")- )- ]- ),- limit=10-)-```--## Quantization Strategies--### Scalar Quantization (INT8)--```python-from qdrant_client.models import ScalarQuantization, ScalarQuantizationConfig, ScalarType--# ~4x memory reduction, minimal accuracy loss-client.create_collection(- collection_name="scalar_quantized",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- quantization_config=ScalarQuantization(- scalar=ScalarQuantizationConfig(- type=ScalarType.INT8,- quantile=0.99, # Clip extreme values- always_ram=True # Keep quantized vectors in RAM- )- )-)-```--### Product Quantization--```python-from qdrant_client.models import ProductQuantization, ProductQuantizationConfig, CompressionRatio--# ~16x memory reduction, some accuracy loss-client.create_collection(- collection_name="product_quantized",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- quantization_config=ProductQuantization(- product=ProductQuantizationConfig(- compression=CompressionRatio.X16,- always_ram=True- )- )-)-```--### Binary Quantization--```python-from qdrant_client.models import BinaryQuantization, BinaryQuantizationConfig--# ~32x memory reduction, requires oversampling-client.create_collection(- collection_name="binary_quantized",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- quantization_config=BinaryQuantization(- binary=BinaryQuantizationConfig(always_ram=True)- )-)--# Search with oversampling-results = client.search(- collection_name="binary_quantized",- query_vector=query,- search_params={- "quantization": {- "rescore": True,- "oversampling": 2.0 # Retrieve 2x candidates, rescore- }- },- limit=10-)-```--## Snapshots and Backups--### Create Snapshot--```python-# Create collection snapshot-snapshot_info = client.create_snapshot(collection_name="documents")-print(f"Snapshot: {snapshot_info.name}")--# List snapshots-snapshots = client.list_snapshots(collection_name="documents")-for s in snapshots:- print(f"{s.name}: {s.size} bytes")--# Full storage snapshot-full_snapshot = client.create_full_snapshot()-```--### Restore from Snapshot--```python-# Download snapshot-client.download_snapshot(- collection_name="documents",- snapshot_name="documents-2024-01-01.snapshot",- target_path="./backup/"-)--# Restore (via REST API)-import requests--response = requests.put(- "http://localhost:6333/collections/documents/snapshots/recover",- json={"location": "file:///backup/documents-2024-01-01.snapshot"}-)-```--## Collection Aliases--```python-# Create alias-client.update_collection_aliases(- change_aliases_operations=[- {"create_alias": {"alias_name": "production", "collection_name": "documents_v2"}}- ]-)--# Blue-green deployment-# 1. Create new collection with updates-client.create_collection(collection_name="documents_v3", ...)--# 2. Populate new collection-client.upsert(collection_name="documents_v3", points=new_points)--# 3. Atomic switch-client.update_collection_aliases(- change_aliases_operations=[- {"delete_alias": {"alias_name": "production"}},- {"create_alias": {"alias_name": "production", "collection_name": "documents_v3"}}- ]-)--# Search via alias-results = client.search(collection_name="production", query_vector=query, limit=10)-```--## Scroll and Iteration--### Scroll Through All Points--```python-# Paginated iteration-offset = None-all_points = []--while True:- results, offset = client.scroll(- collection_name="documents",- limit=100,- offset=offset,- with_payload=True,- with_vectors=False- )- all_points.extend(results)-- if offset is None:- break--print(f"Total points: {len(all_points)}")-```--### Filtered Scroll--```python-# Scroll with filter-results, _ = client.scroll(- collection_name="documents",- scroll_filter=Filter(- must=[- FieldCondition(key="status", match=MatchValue(value="active"))- ]- ),- limit=1000-)-```--## Async Client--```python-import asyncio-from qdrant_client import AsyncQdrantClient--async def main():- client = AsyncQdrantClient(host="localhost", port=6333)-- # Async operations- await client.create_collection(- collection_name="async_docs",- vectors_config=VectorParams(size=384, distance=Distance.COSINE)- )-- await client.upsert(- collection_name="async_docs",- points=points- )-- results = await client.search(- collection_name="async_docs",- query_vector=query,- limit=10- )-- return results--results = asyncio.run(main())-```--## gRPC Client--```python-from qdrant_client import QdrantClient--# Prefer gRPC for better performance-client = QdrantClient(- host="localhost",- port=6333,- grpc_port=6334,- prefer_grpc=True # Use gRPC when available-)--# gRPC-only client-from qdrant_client import QdrantClient--client = QdrantClient(- host="localhost",- grpc_port=6334,- prefer_grpc=True,- https=False-)-```--## Multitenancy--### Payload-Based Isolation--```python-# Single collection, filter by tenant-client.upsert(- collection_name="multi_tenant",- points=[- PointStruct(- id=1,- vector=embedding,- payload={"tenant_id": "tenant_a", "text": "..."}- )- ]-)--# Search within tenant-results = client.search(- collection_name="multi_tenant",- query_vector=query,- query_filter=Filter(- must=[FieldCondition(key="tenant_id", match=MatchValue(value="tenant_a"))]- ),- limit=10-)-```--### Collection-Per-Tenant--```python-# Create tenant collection-def create_tenant_collection(tenant_id: str):- client.create_collection(- collection_name=f"tenant_{tenant_id}",- vectors_config=VectorParams(size=384, distance=Distance.COSINE)- )--# Search tenant collection-def search_tenant(tenant_id: str, query_vector: list, limit: int = 10):- return client.search(- collection_name=f"tenant_{tenant_id}",- query_vector=query_vector,- limit=limit- )-```--## Performance Monitoring--### Collection Statistics--```python-# Collection info-info = client.get_collection("documents")-print(f"Points: {info.points_count}")-print(f"Indexed vectors: {info.indexed_vectors_count}")-print(f"Segments: {len(info.segments)}")-print(f"Status: {info.status}")--# Detailed segment info-for i, segment in enumerate(info.segments):- print(f"Segment {i}: {segment}")-```--### Telemetry--```python-# Get telemetry data-telemetry = client.get_telemetry()-print(f"Collections: {telemetry.collections}")-print(f"Operations: {telemetry.operations}")-```diff --git a/optional-skills/mlops/qdrant/references/troubleshooting.md b/optional-skills/mlops/qdrant/references/troubleshooting.mddeleted file mode 100644index 219f281bd0a..00000000000--- a/optional-skills/mlops/qdrant/references/troubleshooting.md+++ /dev/null@@ -1,631 +0,0 @@-# Qdrant Troubleshooting Guide--## Installation Issues--### Docker Issues--**Error**: `Cannot connect to Docker daemon`--**Fix**:-```bash-# Start Docker daemon-sudo systemctl start docker--# Or use Docker Desktop on Mac/Windows-open -a Docker-```--**Error**: `Port 6333 already in use`--**Fix**:-```bash-# Find process using port-lsof -i :6333--# Kill process or use different port-docker run -p 6334:6333 qdrant/qdrant-```--### Python Client Issues--**Error**: `ModuleNotFoundError: No module named 'qdrant_client'`--**Fix**:-```bash-pip install qdrant-client--# With specific version-pip install qdrant-client>=1.12.0-```--**Error**: `grpc._channel._InactiveRpcError`--**Fix**:-```bash-# Install with gRPC support-pip install 'qdrant-client[grpc]'--# Or disable gRPC-client = QdrantClient(host="localhost", port=6333, prefer_grpc=False)-```--## Connection Issues--### Cannot Connect to Server--**Error**: `ConnectionRefusedError: [Errno 111] Connection refused`--**Solutions**:--1. **Check server is running**:-```bash-docker ps | grep qdrant-curl http://localhost:6333/healthz-```--2. **Verify port binding**:-```bash-# Check listening ports-netstat -tlnp | grep 6333--# Docker port mapping-docker port <container_id>-```--3. **Use correct host**:-```python-# Docker on Linux-client = QdrantClient(host="localhost", port=6333)--# Docker on Mac/Windows with networking issues-client = QdrantClient(host="127.0.0.1", port=6333)--# Inside Docker network-client = QdrantClient(host="qdrant", port=6333)-```--### Timeout Errors--**Error**: `TimeoutError: Connection timed out`--**Fix**:-```python-# Increase timeout-client = QdrantClient(- host="localhost",- port=6333,- timeout=60 # seconds-)--# For large operations-client.upsert(- collection_name="documents",- points=large_batch,- wait=False # Don't wait for indexing-)-```--### SSL/TLS Errors--**Error**: `ssl.SSLCertVerificationError`--**Fix**:-```python-# Qdrant Cloud-client = QdrantClient(- url="https://cluster.cloud.qdrant.io",- api_key="your-api-key"-)--# Self-signed certificate-client = QdrantClient(- host="localhost",- port=6333,- https=True,- verify=False # Disable verification (not recommended for production)-)-```--## Collection Issues--### Collection Already Exists--**Error**: `ValueError: Collection 'documents' already exists`--**Fix**:-```python-# Check before creating-collections = client.get_collections().collections-names = [c.name for c in collections]--if "documents" not in names:- client.create_collection(...)--# Or recreate-client.recreate_collection(- collection_name="documents",- vectors_config=VectorParams(size=384, distance=Distance.COSINE)-)-```--### Collection Not Found--**Error**: `NotFoundException: Collection 'docs' not found`--**Fix**:-```python-# List available collections-collections = client.get_collections()-print([c.name for c in collections.collections])--# Check exact name (case-sensitive)-try:- info = client.get_collection("documents")-except Exception as e:- print(f"Collection not found: {e}")-```--### Vector Dimension Mismatch--**Error**: `ValueError: Vector dimension mismatch. Expected 384, got 768`--**Fix**:-```python-# Check collection config-info = client.get_collection("documents")-print(f"Expected dimension: {info.config.params.vectors.size}")--# Recreate with correct dimension-client.recreate_collection(- collection_name="documents",- vectors_config=VectorParams(size=768, distance=Distance.COSINE) # Match your embeddings-)-```--## Search Issues--### Empty Search Results--**Problem**: Search returns empty results.--**Solutions**:--1. **Verify data exists**:-```python-info = client.get_collection("documents")-print(f"Points: {info.points_count}")--# Scroll to check data-points, _ = client.scroll(- collection_name="documents",- limit=10,- with_payload=True-)-print(points)-```--2. **Check vector format**:-```python-# Must be list of floats-query_vector = embedding.tolist() # Convert numpy to list--# Check dimensions-print(f"Query dimension: {len(query_vector)}")-```--3. **Verify filter conditions**:-```python-# Test without filter first-results = client.search(- collection_name="documents",- query_vector=query,- limit=10- # No filter-)--# Then add filter incrementally-```--### Slow Search Performance--**Problem**: Search takes too long.--**Solutions**:--1. **Create payload indexes**:-```python-# Index fields used in filters-client.create_payload_index(- collection_name="documents",- field_name="category",- field_schema="keyword"-)-```--2. **Enable quantization**:-```python-client.update_collection(- collection_name="documents",- quantization_config=ScalarQuantization(- scalar=ScalarQuantizationConfig(type=ScalarType.INT8)- )-)-```--3. **Tune HNSW parameters**:-```python-# Faster search (less accurate)-client.update_collection(- collection_name="documents",- hnsw_config=HnswConfigDiff(ef_construct=64, m=8)-)--# Use ef search parameter-results = client.search(- collection_name="documents",- query_vector=query,- search_params={"hnsw_ef": 64}, # Lower = faster- limit=10-)-```--4. **Use gRPC**:-```python-client = QdrantClient(- host="localhost",- port=6333,- grpc_port=6334,- prefer_grpc=True-)-```--### Inconsistent Results--**Problem**: Same query returns different results.--**Solutions**:--1. **Wait for indexing**:-```python-client.upsert(- collection_name="documents",- points=points,- wait=True # Wait for index update-)-```--2. **Check replication consistency**:-```python-# Strong consistency read-results = client.search(- collection_name="documents",- query_vector=query,- consistency="all" # Read from all replicas-)-```--## Upsert Issues--### Batch Upsert Fails--**Error**: `PayloadError: Payload too large`--**Fix**:-```python-# Split into smaller batches-def batch_upsert(client, collection, points, batch_size=100):- for i in range(0, len(points), batch_size):- batch = points[i:i + batch_size]- client.upsert(- collection_name=collection,- points=batch,- wait=True- )--batch_upsert(client, "documents", large_points_list)-```--### Invalid Point ID--**Error**: `ValueError: Invalid point ID`--**Fix**:-```python-# Valid ID types: int or UUID string-from uuid import uuid4--# Integer ID-PointStruct(id=123, vector=vec, payload={})--# UUID string-PointStruct(id=str(uuid4()), vector=vec, payload={})--# NOT valid-PointStruct(id="custom-string-123", ...) # Use UUID format-```--### Payload Validation Errors--**Error**: `ValidationError: Invalid payload`--**Fix**:-```python-# Ensure JSON-serializable payload-import json--payload = {- "title": "Document",- "count": 42,- "tags": ["a", "b"],- "nested": {"key": "value"}-}--# Validate before upsert-json.dumps(payload) # Should not raise--# Avoid non-serializable types-# NOT valid: datetime, numpy arrays, custom objects-payload = {- "timestamp": datetime.now().isoformat(), # Convert to string- "vector": embedding.tolist() # Convert numpy to list-}-```--## Memory Issues--### Out of Memory--**Error**: `MemoryError` or container killed--**Solutions**:--1. **Enable on-disk storage**:-```python-client.create_collection(- collection_name="large_collection",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- on_disk_payload=True, # Store payloads on disk- hnsw_config=HnswConfigDiff(on_disk=True) # Store HNSW on disk-)-```--2. **Use quantization**:-```python-# 4x memory reduction-client.update_collection(- collection_name="large_collection",- quantization_config=ScalarQuantization(- scalar=ScalarQuantizationConfig(- type=ScalarType.INT8,- always_ram=False # Keep on disk- )- )-)-```--3. **Increase Docker memory**:-```bash-docker run -m 8g -p 6333:6333 qdrant/qdrant-```--4. **Configure Qdrant storage**:-```yaml-# config.yaml-storage:- performance:- max_search_threads: 2- optimizers:- memmap_threshold_kb: 20000-```--### High Memory Usage During Indexing--**Fix**:-```python-# Increase indexing threshold for bulk loads-client.update_collection(- collection_name="documents",- optimizer_config={- "indexing_threshold": 50000 # Delay indexing- }-)--# Bulk insert-client.upsert(collection_name="documents", points=all_points, wait=False)--# Then optimize-client.update_collection(- collection_name="documents",- optimizer_config={- "indexing_threshold": 10000 # Resume normal indexing- }-)-```--## Cluster Issues--### Node Not Joining Cluster--**Problem**: New node fails to join cluster.--**Fix**:-```bash-# Check network connectivity-docker exec qdrant-node-2 ping qdrant-node-1--# Verify bootstrap URL-docker logs qdrant-node-2 | grep bootstrap--# Check Raft state-curl http://localhost:6333/cluster-```--### Split Brain--**Problem**: Cluster has inconsistent state.--**Fix**:-```bash-# Force leader election-curl -X POST http://localhost:6333/cluster/recover--# Or restart minority nodes-docker restart qdrant-node-2 qdrant-node-3-```--### Replication Lag--**Problem**: Replicas fall behind.--**Fix**:-```python-# Check collection status-info = client.get_collection("documents")-print(f"Status: {info.status}")--# Use strong consistency for critical writes-client.upsert(- collection_name="documents",- points=points,- ordering=WriteOrdering.STRONG-)-```--## Performance Tuning--### Benchmark Configuration--```python-import time-import numpy as np--def benchmark_search(client, collection, n_queries=100, dimension=384):- # Generate random queries- queries = [np.random.rand(dimension).tolist() for _ in range(n_queries)]-- # Warmup- for q in queries[:10]:- client.search(collection_name=collection, query_vector=q, limit=10)-- # Benchmark- start = time.perf_counter()- for q in queries:- client.search(collection_name=collection, query_vector=q, limit=10)- elapsed = time.perf_counter() - start-- print(f"QPS: {n_queries / elapsed:.2f}")- print(f"Latency: {elapsed / n_queries * 1000:.2f}ms")--benchmark_search(client, "documents")-```--### Optimal HNSW Parameters--```python-# High recall (slower)-client.create_collection(- collection_name="high_recall",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- hnsw_config=HnswConfigDiff(- m=32, # More connections- ef_construct=200 # Higher build quality- )-)--# High speed (lower recall)-client.create_collection(- collection_name="high_speed",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- hnsw_config=HnswConfigDiff(- m=8, # Fewer connections- ef_construct=64 # Lower build quality- )-)--# Balanced-client.create_collection(- collection_name="balanced",- vectors_config=VectorParams(size=384, distance=Distance.COSINE),- hnsw_config=HnswConfigDiff(- m=16, # Default- ef_construct=100 # Default- )-)-```--## Debugging Tips--### Enable Verbose Logging--```python-import logging--logging.basicConfig(level=logging.DEBUG)-logging.getLogger("qdrant_client").setLevel(logging.DEBUG)-```--### Check Server Logs--```bash-# Docker logs-docker logs -f qdrant--# With timestamps-docker logs --timestamps qdrant--# Last 100 lines-docker logs --tail 100 qdrant-```--### Inspect Collection State--```python-# Collection info-info = client.get_collection("documents")-print(f"Status: {info.status}")-print(f"Points: {info.points_count}")-print(f"Segments: {len(info.segments)}")-print(f"Config: {info.config}")--# Sample points-points, _ = client.scroll(- collection_name="documents",- limit=5,- with_payload=True,- with_vectors=True-)-for p in points:- print(f"ID: {p.id}, Payload: {p.payload}")-```--### Test Connection--```python-def test_connection(host="localhost", port=6333):- try:- client = QdrantClient(host=host, port=port, timeout=5)- collections = client.get_collections()- print(f"Connected! Collections: {len(collections.collections)}")- return True- except Exception as e:- print(f"Connection failed: {e}")- return False--test_connection()-```--## Getting Help--1. **Documentation**: https://qdrant.tech/documentation/-2. **GitHub Issues**: https://github.com/qdrant/qdrant/issues-3. **Discord**: https://discord.gg/qdrant-4. **Stack Overflow**: Tag `qdrant`--### Reporting Issues--Include:-- Qdrant version: `curl http://localhost:6333/`-- Python client version: `pip show qdrant-client`-- Full error traceback-- Minimal reproducible code-- Collection configurationdiff --git a/optional-skills/mlops/stable-diffusion/SKILL.md b/optional-skills/mlops/stable-diffusion/SKILL.mddeleted file mode 100644index 0a94bb2a73a..00000000000--- a/optional-skills/mlops/stable-diffusion/SKILL.md+++ /dev/null@@ -1,523 +0,0 @@-----name: stable-diffusion-description: Text-to-image generation, inpainting, and img2img.-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [diffusers>=0.30.0, transformers>=4.41.0, accelerate>=0.31.0, torch>=2.0.0]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Image Generation, Stable Diffusion, Diffusers, Text-to-Image, Multimodal, Computer Vision]-------# Stable Diffusion Image Generation--Guide to generating images with Stable Diffusion using the HuggingFace Diffusers library.--## When to use Stable Diffusion--**Use Stable Diffusion when:**-- Generating images from text descriptions-- Performing image-to-image translation (style transfer, enhancement)-- Inpainting (filling in masked regions)-- Outpainting (extending images beyond boundaries)-- Creating variations of existing images-- Building custom image generation workflows--**Key features:**-- **Text-to-Image**: Generate images from natural language prompts-- **Image-to-Image**: Transform existing images with text guidance-- **Inpainting**: Fill masked regions with context-aware content-- **ControlNet**: Add spatial conditioning (edges, poses, depth)-- **LoRA Support**: Efficient fine-tuning and style adaptation-- **Multiple Models**: SD 1.5, SDXL, SD 3.0, Flux support--**Use alternatives instead:**-- **DALL-E 3**: For API-based generation without GPU-- **Midjourney**: For artistic, stylized outputs-- **Imagen**: For Google Cloud integration-- **Leonardo.ai**: For web-based creative workflows--## Quick start--### Installation--```bash-pip install diffusers transformers accelerate torch-pip install xformers # Optional: memory-efficient attention-```--### Basic text-to-image--```python-from diffusers import DiffusionPipeline-import torch--# Load pipeline (auto-detects model type)-pipe = DiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-)-pipe.to("cuda")--# Generate image-image = pipe(- "A serene mountain landscape at sunset, highly detailed",- num_inference_steps=50,- guidance_scale=7.5-).images[0]--image.save("output.png")-```--### Using SDXL (higher quality)--```python-from diffusers import AutoPipelineForText2Image-import torch--pipe = AutoPipelineForText2Image.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- torch_dtype=torch.float16,- variant="fp16"-)-pipe.to("cuda")--# Enable memory optimization-pipe.enable_model_cpu_offload()--image = pipe(- prompt="A futuristic city with flying cars, cinematic lighting",- height=1024,- width=1024,- num_inference_steps=30-).images[0]-```--## Architecture overview--### Three-pillar design--Diffusers is built around three core components:--```-Pipeline (orchestration)-├── Model (neural networks)-│ ├── UNet / Transformer (noise prediction)-│ ├── VAE (latent encoding/decoding)-│ └── Text Encoder (CLIP/T5)-└── Scheduler (denoising algorithm)-```--### Pipeline inference flow--```-Text Prompt → Text Encoder → Text Embeddings- ↓-Random Noise → [Denoising Loop] ← Scheduler- ↓- Predicted Noise- ↓- VAE Decoder → Final Image-```--## Core concepts--### Pipelines--Pipelines orchestrate complete workflows:--| Pipeline | Purpose |-|----------|---------|-| `StableDiffusionPipeline` | Text-to-image (SD 1.x/2.x) |-| `StableDiffusionXLPipeline` | Text-to-image (SDXL) |-| `StableDiffusion3Pipeline` | Text-to-image (SD 3.0) |-| `FluxPipeline` | Text-to-image (Flux models) |-| `StableDiffusionImg2ImgPipeline` | Image-to-image |-| `StableDiffusionInpaintPipeline` | Inpainting |--### Schedulers--Schedulers control the denoising process:--| Scheduler | Steps | Quality | Use Case |-|-----------|-------|---------|----------|-| `EulerDiscreteScheduler` | 20-50 | Good | Default choice |-| `EulerAncestralDiscreteScheduler` | 20-50 | Good | More variation |-| `DPMSolverMultistepScheduler` | 15-25 | Excellent | Fast, high quality |-| `DDIMScheduler` | 50-100 | Good | Deterministic |-| `LCMScheduler` | 4-8 | Good | Very fast |-| `UniPCMultistepScheduler` | 15-25 | Excellent | Fast convergence |--### Swapping schedulers--```python-from diffusers import DPMSolverMultistepScheduler--# Swap for faster generation-pipe.scheduler = DPMSolverMultistepScheduler.from_config(- pipe.scheduler.config-)--# Now generate with fewer steps-image = pipe(prompt, num_inference_steps=20).images[0]-```--## Generation parameters--### Key parameters--| Parameter | Default | Description |-|-----------|---------|-------------|-| `prompt` | Required | Text description of desired image |-| `negative_prompt` | None | What to avoid in the image |-| `num_inference_steps` | 50 | Denoising steps (more = better quality) |-| `guidance_scale` | 7.5 | Prompt adherence (7-12 typical) |-| `height`, `width` | 512/1024 | Output dimensions (multiples of 8) |-| `generator` | None | Torch generator for reproducibility |-| `num_images_per_prompt` | 1 | Batch size |--### Reproducible generation--```python-import torch--generator = torch.Generator(device="cuda").manual_seed(42)--image = pipe(- prompt="A cat wearing a top hat",- generator=generator,- num_inference_steps=50-).images[0]-```--### Negative prompts--```python-image = pipe(- prompt="Professional photo of a dog in a garden",- negative_prompt="blurry, low quality, distorted, ugly, bad anatomy",- guidance_scale=7.5-).images[0]-```--## Image-to-image--Transform existing images with text guidance:--```python-from diffusers import AutoPipelineForImage2Image-from PIL import Image--pipe = AutoPipelineForImage2Image.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-).to("cuda")--init_image = Image.open("input.jpg").resize((512, 512))--image = pipe(- prompt="A watercolor painting of the scene",- image=init_image,- strength=0.75, # How much to transform (0-1)- num_inference_steps=50-).images[0]-```--## Inpainting--Fill masked regions:--```python-from diffusers import AutoPipelineForInpainting-from PIL import Image--pipe = AutoPipelineForInpainting.from_pretrained(- "runwayml/stable-diffusion-inpainting",- torch_dtype=torch.float16-).to("cuda")--image = Image.open("photo.jpg")-mask = Image.open("mask.png") # White = inpaint region--result = pipe(- prompt="A red car parked on the street",- image=image,- mask_image=mask,- num_inference_steps=50-).images[0]-```--## ControlNet--Add spatial conditioning for precise control:--```python-from diffusers import StableDiffusionControlNetPipeline, ControlNetModel-import torch--# Load ControlNet for edge conditioning-controlnet = ControlNetModel.from_pretrained(- "lllyasviel/control_v11p_sd15_canny",- torch_dtype=torch.float16-)--pipe = StableDiffusionControlNetPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- controlnet=controlnet,- torch_dtype=torch.float16-).to("cuda")--# Use Canny edge image as control-control_image = get_canny_image(input_image)--image = pipe(- prompt="A beautiful house in the style of Van Gogh",- image=control_image,- num_inference_steps=30-).images[0]-```--### Available ControlNets--| ControlNet | Input Type | Use Case |-|------------|------------|----------|-| `canny` | Edge maps | Preserve structure |-| `openpose` | Pose skeletons | Human poses |-| `depth` | Depth maps | 3D-aware generation |-| `normal` | Normal maps | Surface details |-| `mlsd` | Line segments | Architectural lines |-| `scribble` | Rough sketches | Sketch-to-image |--## LoRA adapters--Load fine-tuned style adapters:--```python-from diffusers import DiffusionPipeline--pipe = DiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-).to("cuda")--# Load LoRA weights-pipe.load_lora_weights("path/to/lora", weight_name="style.safetensors")--# Generate with LoRA style-image = pipe("A portrait in the trained style").images[0]--# Adjust LoRA strength-pipe.fuse_lora(lora_scale=0.8)--# Unload LoRA-pipe.unload_lora_weights()-```--### Multiple LoRAs--```python-# Load multiple LoRAs-pipe.load_lora_weights("lora1", adapter_name="style")-pipe.load_lora_weights("lora2", adapter_name="character")--# Set weights for each-pipe.set_adapters(["style", "character"], adapter_weights=[0.7, 0.5])--image = pipe("A portrait").images[0]-```--## Memory optimization--### Enable CPU offloading--```python-# Model CPU offload - moves models to CPU when not in use-pipe.enable_model_cpu_offload()--# Sequential CPU offload - more aggressive, slower-pipe.enable_sequential_cpu_offload()-```--### Attention slicing--```python-# Reduce memory by computing attention in chunks-pipe.enable_attention_slicing()--# Or specific chunk size-pipe.enable_attention_slicing("max")-```--### xFormers memory-efficient attention--```python-# Requires xformers package-pipe.enable_xformers_memory_efficient_attention()-```--### VAE slicing for large images--```python-# Decode latents in tiles for large images-pipe.enable_vae_slicing()-pipe.enable_vae_tiling()-```--## Model variants--### Loading different precisions--```python-# FP16 (recommended for GPU)-pipe = DiffusionPipeline.from_pretrained(- "model-id",- torch_dtype=torch.float16,- variant="fp16"-)--# BF16 (better precision, requires Ampere+ GPU)-pipe = DiffusionPipeline.from_pretrained(- "model-id",- torch_dtype=torch.bfloat16-)-```--### Loading specific components--```python-from diffusers import UNet2DConditionModel, AutoencoderKL--# Load custom VAE-vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")--# Use with pipeline-pipe = DiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- vae=vae,- torch_dtype=torch.float16-)-```--## Batch generation--Generate multiple images efficiently:--```python-# Multiple prompts-prompts = [- "A cat playing piano",- "A dog reading a book",- "A bird painting a picture"-]--images = pipe(prompts, num_inference_steps=30).images--# Multiple images per prompt-images = pipe(- "A beautiful sunset",- num_images_per_prompt=4,- num_inference_steps=30-).images-```--## Common workflows--### Workflow 1: High-quality generation--```python-from diffusers import StableDiffusionXLPipeline, DPMSolverMultistepScheduler-import torch--# 1. Load SDXL with optimizations-pipe = StableDiffusionXLPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- torch_dtype=torch.float16,- variant="fp16"-)-pipe.to("cuda")-pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)-pipe.enable_model_cpu_offload()--# 2. Generate with quality settings-image = pipe(- prompt="A majestic lion in the savanna, golden hour lighting, 8k, detailed fur",- negative_prompt="blurry, low quality, cartoon, anime, sketch",- num_inference_steps=30,- guidance_scale=7.5,- height=1024,- width=1024-).images[0]-```--### Workflow 2: Fast prototyping--```python-from diffusers import AutoPipelineForText2Image, LCMScheduler-import torch--# Use LCM for 4-8 step generation-pipe = AutoPipelineForText2Image.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- torch_dtype=torch.float16-).to("cuda")--# Load LCM LoRA for fast generation-pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")-pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)-pipe.fuse_lora()--# Generate in ~1 second-image = pipe(- "A beautiful landscape",- num_inference_steps=4,- guidance_scale=1.0-).images[0]-```--## Common issues--**CUDA out of memory:**-```python-# Enable memory optimizations-pipe.enable_model_cpu_offload()-pipe.enable_attention_slicing()-pipe.enable_vae_slicing()--# Or use lower precision-pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)-```--**Black/noise images:**-```python-# Check VAE configuration-# Use safety checker bypass if needed-pipe.safety_checker = None--# Ensure proper dtype consistency-pipe = pipe.to(dtype=torch.float16)-```--**Slow generation:**-```python-# Use faster scheduler-from diffusers import DPMSolverMultistepScheduler-pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)--# Reduce steps-image = pipe(prompt, num_inference_steps=20).images[0]-```--## References--- **[Advanced Usage](references/advanced-usage.md)** - Custom pipelines, fine-tuning, deployment-- **[Troubleshooting](references/troubleshooting.md)** - Common issues and solutions--## Resources--- **Documentation**: https://huggingface.co/docs/diffusers-- **Repository**: https://github.com/huggingface/diffusers-- **Model Hub**: https://huggingface.co/models?library=diffusers-- **Discord**: https://discord.gg/diffusersdiff --git a/optional-skills/mlops/stable-diffusion/references/advanced-usage.md b/optional-skills/mlops/stable-diffusion/references/advanced-usage.mddeleted file mode 100644index 2384715f949..00000000000--- a/optional-skills/mlops/stable-diffusion/references/advanced-usage.md+++ /dev/null@@ -1,716 +0,0 @@-# Stable Diffusion Advanced Usage Guide--## Custom Pipelines--### Building from components--```python-from diffusers import (- UNet2DConditionModel,- AutoencoderKL,- DDPMScheduler,- StableDiffusionPipeline-)-from transformers import CLIPTextModel, CLIPTokenizer-import torch--# Load components individually-unet = UNet2DConditionModel.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- subfolder="unet"-)-vae = AutoencoderKL.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- subfolder="vae"-)-text_encoder = CLIPTextModel.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- subfolder="text_encoder"-)-tokenizer = CLIPTokenizer.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- subfolder="tokenizer"-)-scheduler = DDPMScheduler.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- subfolder="scheduler"-)--# Assemble pipeline-pipe = StableDiffusionPipeline(- unet=unet,- vae=vae,- text_encoder=text_encoder,- tokenizer=tokenizer,- scheduler=scheduler,- safety_checker=None,- feature_extractor=None,- requires_safety_checker=False-)-```--### Custom denoising loop--```python-from diffusers import DDIMScheduler, AutoencoderKL, UNet2DConditionModel-from transformers import CLIPTextModel, CLIPTokenizer-import torch--def custom_generate(- prompt: str,- num_steps: int = 50,- guidance_scale: float = 7.5,- height: int = 512,- width: int = 512-):- # Load components- tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-large-patch14")- text_encoder = CLIPTextModel.from_pretrained("openai/clip-vit-large-patch14")- unet = UNet2DConditionModel.from_pretrained("sd-model", subfolder="unet")- vae = AutoencoderKL.from_pretrained("sd-model", subfolder="vae")- scheduler = DDIMScheduler.from_pretrained("sd-model", subfolder="scheduler")-- device = "cuda"- text_encoder.to(device)- unet.to(device)- vae.to(device)-- # Encode prompt- text_input = tokenizer(- prompt,- padding="max_length",- max_length=77,- truncation=True,- return_tensors="pt"- )- text_embeddings = text_encoder(text_input.input_ids.to(device))[0]-- # Unconditional embeddings for classifier-free guidance- uncond_input = tokenizer(- "",- padding="max_length",- max_length=77,- return_tensors="pt"- )- uncond_embeddings = text_encoder(uncond_input.input_ids.to(device))[0]-- # Concatenate for batch processing- text_embeddings = torch.cat([uncond_embeddings, text_embeddings])-- # Initialize latents- latents = torch.randn(- (1, 4, height // 8, width // 8),- device=device- )- latents = latents * scheduler.init_noise_sigma-- # Denoising loop- scheduler.set_timesteps(num_steps)- for t in scheduler.timesteps:- latent_model_input = torch.cat([latents] * 2)- latent_model_input = scheduler.scale_model_input(latent_model_input, t)-- # Predict noise- with torch.no_grad():- noise_pred = unet(- latent_model_input,- t,- encoder_hidden_states=text_embeddings- ).sample-- # Classifier-free guidance- noise_pred_uncond, noise_pred_cond = noise_pred.chunk(2)- noise_pred = noise_pred_uncond + guidance_scale * (- noise_pred_cond - noise_pred_uncond- )-- # Update latents- latents = scheduler.step(noise_pred, t, latents).prev_sample-- # Decode latents- latents = latents / vae.config.scaling_factor- with torch.no_grad():- image = vae.decode(latents).sample-- # Convert to PIL- image = (image / 2 + 0.5).clamp(0, 1)- image = image.cpu().permute(0, 2, 3, 1).numpy()- image = (image * 255).round().astype("uint8")[0]-- return Image.fromarray(image)-```--## IP-Adapter--Use image prompts alongside text:--```python-from diffusers import StableDiffusionPipeline-from diffusers.utils import load_image-import torch--pipe = StableDiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-).to("cuda")--# Load IP-Adapter-pipe.load_ip_adapter(- "h94/IP-Adapter",- subfolder="models",- weight_name="ip-adapter_sd15.bin"-)--# Set IP-Adapter scale-pipe.set_ip_adapter_scale(0.6)--# Load reference image-ip_image = load_image("reference_style.jpg")--# Generate with image + text prompt-image = pipe(- prompt="A portrait in a garden",- ip_adapter_image=ip_image,- num_inference_steps=50-).images[0]-```--### Multiple IP-Adapter images--```python-# Use multiple reference images-pipe.set_ip_adapter_scale([0.5, 0.7])--images = [- load_image("style_reference.jpg"),- load_image("composition_reference.jpg")-]--result = pipe(- prompt="A landscape painting",- ip_adapter_image=images,- num_inference_steps=50-).images[0]-```--## SDXL Refiner--Two-stage generation for higher quality:--```python-from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline-import torch--# Load base model-base = StableDiffusionXLPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- torch_dtype=torch.float16,- variant="fp16"-).to("cuda")--# Load refiner-refiner = StableDiffusionXLImg2ImgPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-refiner-1.0",- torch_dtype=torch.float16,- variant="fp16"-).to("cuda")--# Generate with base (partial denoising)-image = base(- prompt="A majestic eagle soaring over mountains",- num_inference_steps=40,- denoising_end=0.8,- output_type="latent"-).images--# Refine with refiner-refined = refiner(- prompt="A majestic eagle soaring over mountains",- image=image,- num_inference_steps=40,- denoising_start=0.8-).images[0]-```--## T2I-Adapter--Lightweight conditioning without full ControlNet:--```python-from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter-import torch--# Load adapter-adapter = T2IAdapter.from_pretrained(- "TencentARC/t2i-adapter-canny-sdxl-1.0",- torch_dtype=torch.float16-)--pipe = StableDiffusionXLAdapterPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- adapter=adapter,- torch_dtype=torch.float16-).to("cuda")--# Get canny edges-canny_image = get_canny_image(input_image)--image = pipe(- prompt="A colorful anime character",- image=canny_image,- num_inference_steps=30,- adapter_conditioning_scale=0.8-).images[0]-```--## Fine-tuning with DreamBooth--Train on custom subjects:--```python-from diffusers import StableDiffusionPipeline, DDPMScheduler-from diffusers.optimization import get_scheduler-import torch-from torch.utils.data import Dataset, DataLoader-from PIL import Image-import os--class DreamBoothDataset(Dataset):- def __init__(self, instance_images_path, instance_prompt, tokenizer, size=512):- self.instance_images_path = instance_images_path- self.instance_prompt = instance_prompt- self.tokenizer = tokenizer- self.size = size-- self.instance_images = [- os.path.join(instance_images_path, f)- for f in os.listdir(instance_images_path)- if f.endswith(('.png', '.jpg', '.jpeg'))- ]-- def __len__(self):- return len(self.instance_images)-- def __getitem__(self, idx):- image = Image.open(self.instance_images[idx]).convert("RGB")- image = image.resize((self.size, self.size))- image = torch.tensor(np.array(image)).permute(2, 0, 1) / 127.5 - 1.0-- tokens = self.tokenizer(- self.instance_prompt,- padding="max_length",- max_length=77,- truncation=True,- return_tensors="pt"- )-- return {"image": image, "input_ids": tokens.input_ids.squeeze()}--def train_dreambooth(- pretrained_model: str,- instance_data_dir: str,- instance_prompt: str,- output_dir: str,- learning_rate: float = 5e-6,- max_train_steps: int = 800,- train_batch_size: int = 1-):- # Load pipeline- pipe = StableDiffusionPipeline.from_pretrained(pretrained_model)-- unet = pipe.unet- vae = pipe.vae- text_encoder = pipe.text_encoder- tokenizer = pipe.tokenizer- noise_scheduler = DDPMScheduler.from_pretrained(pretrained_model, subfolder="scheduler")-- # Freeze VAE and text encoder- vae.requires_grad_(False)- text_encoder.requires_grad_(False)-- # Create dataset- dataset = DreamBoothDataset(- instance_data_dir, instance_prompt, tokenizer- )- dataloader = DataLoader(dataset, batch_size=train_batch_size, shuffle=True)-- # Setup optimizer- optimizer = torch.optim.AdamW(unet.parameters(), lr=learning_rate)- lr_scheduler = get_scheduler(- "constant",- optimizer=optimizer,- num_warmup_steps=0,- num_training_steps=max_train_steps- )-- # Training loop- unet.train()- device = "cuda"- unet.to(device)- vae.to(device)- text_encoder.to(device)-- global_step = 0- for epoch in range(max_train_steps // len(dataloader) + 1):- for batch in dataloader:- if global_step >= max_train_steps:- break-- # Encode images to latents- latents = vae.encode(batch["image"].to(device)).latent_dist.sample()- latents = latents * vae.config.scaling_factor-- # Sample noise- noise = torch.randn_like(latents)- timesteps = torch.randint(0, noise_scheduler.num_train_timesteps, (latents.shape[0],))- timesteps = timesteps.to(device)-- # Add noise- noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps)-- # Get text embeddings- encoder_hidden_states = text_encoder(batch["input_ids"].to(device))[0]-- # Predict noise- noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample-- # Compute loss- loss = torch.nn.functional.mse_loss(noise_pred, noise)-- # Backprop- loss.backward()- optimizer.step()- lr_scheduler.step()- optimizer.zero_grad()-- global_step += 1-- if global_step % 100 == 0:- print(f"Step {global_step}, Loss: {loss.item():.4f}")-- # Save model- pipe.unet = unet- pipe.save_pretrained(output_dir)-```--## LoRA Training--Efficient fine-tuning with Low-Rank Adaptation:--```python-from peft import LoraConfig, get_peft_model-from diffusers import StableDiffusionPipeline-import torch--def train_lora(- base_model: str,- train_dataset,- output_dir: str,- lora_rank: int = 4,- learning_rate: float = 1e-4,- max_train_steps: int = 1000-):- pipe = StableDiffusionPipeline.from_pretrained(base_model)- unet = pipe.unet-- # Configure LoRA- lora_config = LoraConfig(- r=lora_rank,- lora_alpha=lora_rank,- target_modules=["to_q", "to_v", "to_k", "to_out.0"],- lora_dropout=0.1- )-- # Apply LoRA to UNet- unet = get_peft_model(unet, lora_config)- unet.print_trainable_parameters() # Shows ~0.1% trainable-- # Train (similar to DreamBooth but only LoRA params)- optimizer = torch.optim.AdamW(- unet.parameters(),- lr=learning_rate- )-- # ... training loop ...-- # Save LoRA weights only- unet.save_pretrained(output_dir)-```--## Textual Inversion--Learn new concepts through embeddings:--```python-from diffusers import StableDiffusionPipeline-import torch--# Load with textual inversion-pipe = StableDiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-).to("cuda")--# Load learned embedding-pipe.load_textual_inversion(- "sd-concepts-library/cat-toy",- token="<cat-toy>"-)--# Use in prompts-image = pipe("A photo of <cat-toy> on a beach").images[0]-```--## Quantization--Reduce memory with quantization:--```python-from diffusers import BitsAndBytesConfig, StableDiffusionXLPipeline-import torch--# 8-bit quantization-quantization_config = BitsAndBytesConfig(load_in_8bit=True)--pipe = StableDiffusionXLPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- quantization_config=quantization_config,- torch_dtype=torch.float16-)-```--### NF4 quantization (4-bit)--```python-quantization_config = BitsAndBytesConfig(- load_in_4bit=True,- bnb_4bit_quant_type="nf4",- bnb_4bit_compute_dtype=torch.float16-)--pipe = StableDiffusionXLPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- quantization_config=quantization_config-)-```--## Production Deployment--### FastAPI server--```python-from fastapi import FastAPI, HTTPException-from pydantic import BaseModel-from diffusers import DiffusionPipeline-import torch-import base64-from io import BytesIO--app = FastAPI()--# Load model at startup-pipe = DiffusionPipeline.from_pretrained(- "stable-diffusion-v1-5/stable-diffusion-v1-5",- torch_dtype=torch.float16-).to("cuda")-pipe.enable_model_cpu_offload()--class GenerationRequest(BaseModel):- prompt: str- negative_prompt: str = ""- num_inference_steps: int = 30- guidance_scale: float = 7.5- width: int = 512- height: int = 512- seed: int = None--class GenerationResponse(BaseModel):- image_base64: str- seed: int-[email protected]("/generate", response_model=GenerationResponse)-async def generate(request: GenerationRequest):- try:- generator = None- seed = request.seed or torch.randint(0, 2**32, (1,)).item()- generator = torch.Generator("cuda").manual_seed(seed)-- image = pipe(- prompt=request.prompt,- negative_prompt=request.negative_prompt,- num_inference_steps=request.num_inference_steps,- guidance_scale=request.guidance_scale,- width=request.width,- height=request.height,- generator=generator- ).images[0]-- # Convert to base64- buffer = BytesIO()- image.save(buffer, format="PNG")- image_base64 = base64.b64encode(buffer.getvalue()).decode()-- return GenerationResponse(image_base64=image_base64, seed=seed)-- except Exception as e:- raise HTTPException(status_code=500, detail=str(e))-[email protected]("/health")-async def health():- return {"status": "healthy"}-```--### Docker deployment--```dockerfile-FROM nvidia/cuda:12.1-runtime-ubuntu22.04--RUN apt-get update && apt-get install -y python3 python3-pip--WORKDIR /app--COPY requirements.txt .-RUN pip3 install -r requirements.txt--COPY . .--# Pre-download model-RUN python3 -c "from diffusers import DiffusionPipeline; DiffusionPipeline.from_pretrained('stable-diffusion-v1-5/stable-diffusion-v1-5')"--EXPOSE 8000-CMD ["uvicorn", "server:app", "--host", "0.0.0.0", "--port", "8000"]-```--### Kubernetes deployment--```yaml-apiVersion: apps/v1-kind: Deployment-metadata:- name: stable-diffusion-spec:- replicas: 2- selector:- matchLabels:- app: stable-diffusion- template:- metadata:- labels:- app: stable-diffusion- spec:- containers:- - name: sd- image: your-registry/stable-diffusion:latest- ports:- - containerPort: 8000- resources:- limits:- nvidia.com/gpu: 1- memory: "16Gi"- requests:- nvidia.com/gpu: 1- memory: "8Gi"- env:- - name: TRANSFORMERS_CACHE- value: "/cache/huggingface"- volumeMounts:- - name: model-cache- mountPath: /cache- volumes:- - name: model-cache- persistentVolumeClaim:- claimName: model-cache-pvc-----apiVersion: v1-kind: Service-metadata:- name: stable-diffusion-spec:- selector:- app: stable-diffusion- ports:- - port: 80- targetPort: 8000- type: LoadBalancer-```--## Callback System--Monitor and modify generation:--```python-from diffusers import StableDiffusionPipeline-from diffusers.callbacks import PipelineCallback-import torch--class ProgressCallback(PipelineCallback):- def __init__(self):- self.progress = []-- def callback_fn(self, pipe, step_index, timestep, callback_kwargs):- self.progress.append({- "step": step_index,- "timestep": timestep.item()- })-- # Optionally modify latents- latents = callback_kwargs["latents"]-- return callback_kwargs--# Use callback-callback = ProgressCallback()--image = pipe(- prompt="A sunset",- callback_on_step_end=callback.callback_fn,- callback_on_step_end_tensor_inputs=["latents"]-).images[0]--print(f"Generation completed in {len(callback.progress)} steps")-```--### Early stopping--```python-def early_stop_callback(pipe, step_index, timestep, callback_kwargs):- # Stop after 20 steps- if step_index >= 20:- pipe._interrupt = True- return callback_kwargs--image = pipe(- prompt="A landscape",- num_inference_steps=50,- callback_on_step_end=early_stop_callback-).images[0]-```--## Multi-GPU Inference--### Device map auto--```python-from diffusers import StableDiffusionXLPipeline--pipe = StableDiffusionXLPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0",- device_map="auto", # Automatically distribute across GPUs- torch_dtype=torch.float16-)-```--### Manual distribution--```python-from accelerate import infer_auto_device_map, dispatch_model--# Create device map-device_map = infer_auto_device_map(- pipe.unet,- max_memory={0: "10GiB", 1: "10GiB"}-)--# Dispatch model-pipe.unet = dispatch_model(pipe.unet, device_map=device_map)-```diff --git a/optional-skills/mlops/stable-diffusion/references/troubleshooting.md b/optional-skills/mlops/stable-diffusion/references/troubleshooting.mddeleted file mode 100644index f358643b628..00000000000--- a/optional-skills/mlops/stable-diffusion/references/troubleshooting.md+++ /dev/null@@ -1,555 +0,0 @@-# Stable Diffusion Troubleshooting Guide--## Installation Issues--### Package conflicts--**Error**: `ImportError: cannot import name 'cached_download' from 'huggingface_hub'`--**Fix**:-```bash-# Update huggingface_hub-pip install --upgrade huggingface_hub--# Reinstall diffusers-pip install --upgrade diffusers-```--### xFormers installation fails--**Error**: `RuntimeError: CUDA error: no kernel image is available for execution`--**Fix**:-```bash-# Check CUDA version-nvcc --version--# Install matching xformers-pip install xformers --index-url https://download.pytorch.org/whl/cu121 # For CUDA 12.1--# Or build from source-pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers-```--### Torch/CUDA mismatch--**Error**: `RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED`--**Fix**:-```bash-# Check versions-python -c "import torch; print(torch.__version__, torch.cuda.is_available())"--# Reinstall PyTorch with correct CUDA-pip uninstall torch torchvision-pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121-```--## Memory Issues--### CUDA out of memory--**Error**: `torch.cuda.OutOfMemoryError: CUDA out of memory`--**Solutions**:--```python-# Solution 1: Enable CPU offloading-pipe.enable_model_cpu_offload()--# Solution 2: Sequential CPU offload (more aggressive)-pipe.enable_sequential_cpu_offload()--# Solution 3: Attention slicing-pipe.enable_attention_slicing()--# Solution 4: VAE slicing for large images-pipe.enable_vae_slicing()--# Solution 5: Use lower precision-pipe = DiffusionPipeline.from_pretrained(- "model-id",- torch_dtype=torch.float16 # or torch.bfloat16-)--# Solution 6: Reduce batch size-image = pipe(prompt, num_images_per_prompt=1).images[0]--# Solution 7: Generate smaller images-image = pipe(prompt, height=512, width=512).images[0]--# Solution 8: Clear cache between generations-import gc-torch.cuda.empty_cache()-gc.collect()-```--### Memory grows over time--**Problem**: Memory usage increases with each generation--**Fix**:-```python-import gc-import torch--def generate_with_cleanup(pipe, prompt, **kwargs):- try:- image = pipe(prompt, **kwargs).images[0]- return image- finally:- # Clear cache after generation- if torch.cuda.is_available():- torch.cuda.empty_cache()- gc.collect()-```--### Large model loading fails--**Error**: `RuntimeError: Unable to load model weights`--**Fix**:-```python-# Use low CPU memory mode-pipe = DiffusionPipeline.from_pretrained(- "large-model-id",- low_cpu_mem_usage=True,- torch_dtype=torch.float16-)-```--## Generation Issues--### Black images--**Problem**: Output images are completely black--**Solutions**:-```python-# Solution 1: Disable safety checker-pipe.safety_checker = None--# Solution 2: Check VAE scaling-# The issue might be with VAE encoding/decoding-latents = latents / pipe.vae.config.scaling_factor # Before decode--# Solution 3: Ensure proper dtype-pipe = pipe.to(dtype=torch.float16)-pipe.vae = pipe.vae.to(dtype=torch.float32) # VAE often needs fp32--# Solution 4: Check guidance scale-# Too high can cause issues-image = pipe(prompt, guidance_scale=7.5).images[0] # Not 20+-```--### Noise/static images--**Problem**: Output looks like random noise--**Solutions**:-```python-# Solution 1: Increase inference steps-image = pipe(prompt, num_inference_steps=50).images[0]--# Solution 2: Check scheduler configuration-pipe.scheduler = pipe.scheduler.from_config(pipe.scheduler.config)--# Solution 3: Verify model was loaded correctly-print(pipe.unet) # Should show model architecture-```--### Blurry images--**Problem**: Output images are low quality or blurry--**Solutions**:-```python-# Solution 1: Use more steps-image = pipe(prompt, num_inference_steps=50).images[0]--# Solution 2: Use better VAE-from diffusers import AutoencoderKL-vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse")-pipe.vae = vae--# Solution 3: Use SDXL or refiner-pipe = DiffusionPipeline.from_pretrained(- "stabilityai/stable-diffusion-xl-base-1.0"-)--# Solution 4: Upscale with img2img-upscale_pipe = StableDiffusionImg2ImgPipeline.from_pretrained(...)-upscaled = upscale_pipe(- prompt=prompt,- image=image.resize((1024, 1024)),- strength=0.3-).images[0]-```--### Prompt not being followed--**Problem**: Generated image doesn't match the prompt--**Solutions**:-```python-# Solution 1: Increase guidance scale-image = pipe(prompt, guidance_scale=10.0).images[0]--# Solution 2: Use negative prompts-image = pipe(- prompt="A red car",- negative_prompt="blue, green, yellow, wrong color",- guidance_scale=7.5-).images[0]--# Solution 3: Use prompt weighting-# Emphasize important words-prompt = "A (red:1.5) car on a street"--# Solution 4: Use longer, more detailed prompts-prompt = """-A bright red sports car, ferrari style, parked on a city street,-photorealistic, high detail, 8k, professional photography-"""-```--### Distorted faces/hands--**Problem**: Faces and hands look deformed--**Solutions**:-```python-# Solution 1: Use negative prompts-negative_prompt = """-bad hands, bad anatomy, deformed, ugly, blurry,-extra fingers, mutated hands, poorly drawn hands,-poorly drawn face, mutation, deformed face-"""--# Solution 2: Use face-specific models-# ADetailer or similar post-processing--# Solution 3: Use ControlNet for poses-# Load pose estimation and condition generation--# Solution 4: Inpaint problematic areas-mask = create_face_mask(image)-fixed = inpaint_pipe(- prompt="beautiful detailed face",- image=image,- mask_image=mask-).images[0]-```--## Scheduler Issues--### Scheduler not compatible--**Error**: `ValueError: Scheduler ... is not compatible with pipeline`--**Fix**:-```python-from diffusers import EulerDiscreteScheduler--# Create scheduler from config-pipe.scheduler = EulerDiscreteScheduler.from_config(- pipe.scheduler.config-)--# Check compatible schedulers-print(pipe.scheduler.compatibles)-```--### Wrong number of steps--**Problem**: Model generates different quality with same steps--**Fix**:-```python-# Reset timesteps explicitly-pipe.scheduler.set_timesteps(num_inference_steps)--# Check scheduler's step count-print(len(pipe.scheduler.timesteps))-```--## LoRA Issues--### LoRA weights not loading--**Error**: `RuntimeError: Error(s) in loading state_dict for UNet2DConditionModel`--**Fix**:-```python-# Check weight file format-# Should be .safetensors or .bin--# Load with correct key prefix-pipe.load_lora_weights(- "path/to/lora",- weight_name="lora.safetensors"-)--# Try loading into specific component-pipe.unet.load_attn_procs("path/to/lora")-```--### LoRA not affecting output--**Problem**: Generated images look the same with/without LoRA--**Fix**:-```python-# Fuse LoRA weights-pipe.fuse_lora(lora_scale=1.0)--# Or set scale explicitly-pipe.set_adapters(["lora_name"], adapter_weights=[1.0])--# Verify LoRA is loaded-print(list(pipe.unet.attn_processors.keys()))-```--### Multiple LoRAs conflict--**Problem**: Multiple LoRAs produce artifacts--**Fix**:-```python-# Load with different adapter names-pipe.load_lora_weights("lora1", adapter_name="style")-pipe.load_lora_weights("lora2", adapter_name="subject")--# Balance weights-pipe.set_adapters(- ["style", "subject"],- adapter_weights=[0.5, 0.5] # Lower weights-)--# Or use LoRA merge before loading-# Merge LoRAs offline with appropriate ratios-```--## ControlNet Issues--### ControlNet not conditioning--**Problem**: ControlNet has no effect on output--**Fix**:-```python-# Check control image format-# Should be RGB, matching generation size-control_image = control_image.resize((512, 512))--# Increase conditioning scale-image = pipe(- prompt=prompt,- image=control_image,- controlnet_conditioning_scale=1.0, # Try 0.5-1.5- num_inference_steps=30-).images[0]--# Verify ControlNet is loaded-print(pipe.controlnet)-```--### Control image preprocessing--**Fix**:-```python-from controlnet_aux import CannyDetector--# Proper preprocessing-canny = CannyDetector()-control_image = canny(input_image)--# Ensure correct format-control_image = control_image.convert("RGB")-control_image = control_image.resize((512, 512))-```--## Hub/Download Issues--### Model download fails--**Error**: `requests.exceptions.ConnectionError`--**Fix**:-```bash-# Set longer timeout-export HF_HUB_DOWNLOAD_TIMEOUT=600--# Use mirror if available-export HF_ENDPOINT=https://hf-mirror.com--# Or download manually-huggingface-cli download stable-diffusion-v1-5/stable-diffusion-v1-5-```--### Cache issues--**Error**: `OSError: Can't load model from cache`--**Fix**:-```bash-# Clear cache-rm -rf ~/.cache/huggingface/hub--# Or set different cache location-export HF_HOME=/path/to/cache--# Force re-download-pipe = DiffusionPipeline.from_pretrained(- "model-id",- force_download=True-)-```--### Access denied for gated models--**Error**: `401 Client Error: Unauthorized`--**Fix**:-```bash-# Login to Hugging Face-huggingface-cli login--# Or use token-pipe = DiffusionPipeline.from_pretrained(- "model-id",- token="hf_xxxxx"-)--# Accept model license on Hub website first-```--## Performance Issues--### Slow generation--**Problem**: Generation takes too long--**Solutions**:-```python-# Solution 1: Use faster scheduler-from diffusers import DPMSolverMultistepScheduler-pipe.scheduler = DPMSolverMultistepScheduler.from_config(- pipe.scheduler.config-)--# Solution 2: Reduce steps-image = pipe(prompt, num_inference_steps=20).images[0]--# Solution 3: Use LCM-from diffusers import LCMScheduler-pipe.load_lora_weights("latent-consistency/lcm-lora-sdxl")-pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)-image = pipe(prompt, num_inference_steps=4, guidance_scale=1.0).images[0]--# Solution 4: Enable xFormers-pipe.enable_xformers_memory_efficient_attention()--# Solution 5: Compile model-pipe.unet = torch.compile(pipe.unet, mode="reduce-overhead", fullgraph=True)-```--### First generation is slow--**Problem**: First image takes much longer--**Fix**:-```python-# Warm up the model-_ = pipe("warmup", num_inference_steps=1)--# Then run actual generation-image = pipe(prompt, num_inference_steps=50).images[0]--# Compile for faster subsequent runs-pipe.unet = torch.compile(pipe.unet)-```--## Debugging Tips--### Enable debug logging--```python-import logging-logging.basicConfig(level=logging.DEBUG)--# Or for specific modules-logging.getLogger("diffusers").setLevel(logging.DEBUG)-logging.getLogger("transformers").setLevel(logging.DEBUG)-```--### Check model components--```python-# Print pipeline components-print(pipe.components)--# Check model config-print(pipe.unet.config)-print(pipe.vae.config)-print(pipe.scheduler.config)--# Verify device placement-print(pipe.device)-for name, module in pipe.components.items():- if hasattr(module, 'device'):- print(f"{name}: {module.device}")-```--### Validate inputs--```python-# Check image dimensions-print(f"Height: {height}, Width: {width}")-assert height % 8 == 0, "Height must be divisible by 8"-assert width % 8 == 0, "Width must be divisible by 8"--# Check prompt tokenization-tokens = pipe.tokenizer(prompt, return_tensors="pt")-print(f"Token count: {tokens.input_ids.shape[1]}") # Max 77 for SD-```--### Save intermediate results--```python-def save_latents_callback(pipe, step_index, timestep, callback_kwargs):- latents = callback_kwargs["latents"]-- # Decode and save intermediate- with torch.no_grad():- image = pipe.vae.decode(latents / pipe.vae.config.scaling_factor).sample- image = (image / 2 + 0.5).clamp(0, 1)- image = image.cpu().permute(0, 2, 3, 1).numpy()[0]- Image.fromarray((image * 255).astype("uint8")).save(f"step_{step_index}.png")-- return callback_kwargs--image = pipe(- prompt,- callback_on_step_end=save_latents_callback,- callback_on_step_end_tensor_inputs=["latents"]-).images[0]-```--## Getting Help--1. **Documentation**: https://huggingface.co/docs/diffusers-2. **GitHub Issues**: https://github.com/huggingface/diffusers/issues-3. **Discord**: https://discord.gg/diffusers-4. **Forum**: https://discuss.huggingface.co--### Reporting Issues--Include:-- Diffusers version: `pip show diffusers`-- PyTorch version: `python -c "import torch; print(torch.__version__)"`-- CUDA version: `nvcc --version`-- GPU model: `nvidia-smi`-- Full error traceback-- Minimal reproducible code-- Model name/ID useddiff --git a/optional-skills/mlops/training/trl-fine-tuning/SKILL.md b/optional-skills/mlops/training/trl-fine-tuning/SKILL.mddeleted file mode 100644index 7e250364eb4..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/SKILL.md+++ /dev/null@@ -1,498 +0,0 @@-----name: trl-fine-tuning-description: "TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF."-version: 1.0.1-author: Orchestra Research-license: MIT-dependencies: [trl, transformers, datasets, peft, accelerate, torch]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, GRPO, RLOO, RLHF, Preference Alignment, HuggingFace]-------# TRL - Transformer Reinforcement Learning--## Quick start--TRL provides post-training methods for aligning language models with human preferences.--**Installation**:-```bash-pip install trl transformers datasets peft accelerate-```--**Supervised Fine-Tuning** (instruction tuning):-```python-from trl import SFTTrainer--trainer = SFTTrainer(- model="Qwen/Qwen2.5-0.5B",- train_dataset=dataset, # Prompt-completion pairs-)-trainer.train()-```--**DPO** (align with preferences):-```python-from trl import DPOTrainer, DPOConfig--config = DPOConfig(output_dir="model-dpo", beta=0.1)-trainer = DPOTrainer(- model=model,- args=config,- train_dataset=preference_dataset, # chosen/rejected pairs- processing_class=tokenizer-)-trainer.train()-```--## Common workflows--### Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO)--Complete pipeline from base model to human-aligned model.--> **Note (TRL 1.x):** PPO has been **removed** from TRL — `PPOTrainer`, `PPOConfig`, and-> `python -m trl.scripts.ppo` no longer exist. Use an online-RL trainer TRL still ships:-> **RLOO** (`RLOOTrainer` / `trl rloo`) is the closest drop-in for a reward-model-driven-> RLHF pipeline, and **GRPO** (`GRPOTrainer` / `trl grpo`, see Workflow 3) is the-> memory-efficient alternative. The step below uses RLOO.--Copy this checklist:--```-RLHF Training:-- [ ] Step 1: Supervised fine-tuning (SFT)-- [ ] Step 2: Train reward model-- [ ] Step 3: RLOO reinforcement learning-- [ ] Step 4: Evaluate aligned model-```--**Step 1: Supervised fine-tuning**--Train base model on instruction-following data:--```python-from transformers import AutoModelForCausalLM, AutoTokenizer-from trl import SFTTrainer, SFTConfig-from datasets import load_dataset--# Load model-model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")--# Load instruction dataset-dataset = load_dataset("trl-lib/Capybara", split="train")--# Configure training-training_args = SFTConfig(- output_dir="Qwen2.5-0.5B-SFT",- per_device_train_batch_size=4,- num_train_epochs=1,- learning_rate=2e-5,- logging_steps=10,- save_strategy="epoch"-)--# Train-trainer = SFTTrainer(- model=model,- args=training_args,- train_dataset=dataset,- processing_class=tokenizer-)-trainer.train()-trainer.save_model()-```--**Step 2: Train reward model**--Train model to predict human preferences:--```python-from transformers import AutoModelForSequenceClassification-from trl import RewardTrainer, RewardConfig--# Load SFT model as base-model = AutoModelForSequenceClassification.from_pretrained(- "Qwen2.5-0.5B-SFT",- num_labels=1 # Single reward score-)-tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")--# Load preference data (chosen/rejected pairs)-dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")--# Configure training-training_args = RewardConfig(- output_dir="Qwen2.5-0.5B-Reward",- per_device_train_batch_size=2,- num_train_epochs=1,- learning_rate=1e-5-)--# Train reward model-trainer = RewardTrainer(- model=model,- args=training_args,- processing_class=tokenizer,- train_dataset=dataset-)-trainer.train()-trainer.save_model()-```--**Step 3: RLOO reinforcement learning**--Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI-(`trl rloo`) with the trained reward model passed via `--reward_model_name_or_path`:--```bash-trl rloo \- --model_name_or_path Qwen2.5-0.5B-SFT \- --reward_model_name_or_path Qwen2.5-0.5B-Reward \- --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \- --output_dir Qwen2.5-0.5B-RLOO \- --learning_rate 3e-6 \- --per_device_train_batch_size 64 \- --num_generations 4-```--Equivalent Python (`RLOOTrainer` / `RLOOConfig`):-```python-from trl import RLOOTrainer, RLOOConfig-from transformers import AutoModelForSequenceClassification, AutoTokenizer--reward_model = AutoModelForSequenceClassification.from_pretrained(- "Qwen2.5-0.5B-Reward", num_labels=1-)--config = RLOOConfig(- output_dir="Qwen2.5-0.5B-RLOO",- per_device_train_batch_size=64,- learning_rate=3e-6,- num_generations=4,-)--trainer = RLOOTrainer(- model="Qwen2.5-0.5B-SFT",- reward_funcs=reward_model, # a reward model (or a callable reward function)- args=config,- train_dataset=dataset, # prompt-only dataset- processing_class=tokenizer,-)-trainer.train()-```--**Step 4: Evaluate**--```python-from transformers import pipeline--# Load aligned model-generator = pipeline("text-generation", model="Qwen2.5-0.5B-RLOO")--# Test-prompt = "Explain quantum computing to a 10-year-old"-output = generator(prompt, max_length=200)[0]["generated_text"]-print(output)-```--### Workflow 2: Simple preference alignment with DPO--Align model with preferences without reward model.--Copy this checklist:--```-DPO Training:-- [ ] Step 1: Prepare preference dataset-- [ ] Step 2: Configure DPO-- [ ] Step 3: Train with DPOTrainer-- [ ] Step 4: Evaluate alignment-```--**Step 1: Prepare preference dataset**--Dataset format:-```json-{- "prompt": "What is the capital of France?",- "chosen": "The capital of France is Paris.",- "rejected": "I don't know."-}-```--Load dataset:-```python-from datasets import load_dataset--dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")-# Or load your own-# dataset = load_dataset("json", data_files="preferences.json")-```--**Step 2: Configure DPO**--```python-from trl import DPOConfig--config = DPOConfig(- output_dir="Qwen2.5-0.5B-DPO",- per_device_train_batch_size=4,- num_train_epochs=1,- learning_rate=5e-7,- beta=0.1, # KL penalty strength- max_prompt_length=512,- max_length=1024,- logging_steps=10-)-```--**Step 3: Train with DPOTrainer**--```python-from transformers import AutoModelForCausalLM, AutoTokenizer-from trl import DPOTrainer--model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")--trainer = DPOTrainer(- model=model,- args=config,- train_dataset=dataset,- processing_class=tokenizer-)--trainer.train()-trainer.save_model()-```--**CLI alternative**:-```bash-trl dpo \- --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \- --dataset_name argilla/Capybara-Preferences \- --output_dir Qwen2.5-0.5B-DPO \- --per_device_train_batch_size 4 \- --learning_rate 5e-7 \- --beta 0.1-```--### Workflow 3: Memory-efficient online RL with GRPO--Train with reinforcement learning using minimal memory.--For in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references/grpo-training.md](references/grpo-training.md)**. A production-ready training script is in **[templates/basic_grpo_training.py](templates/basic_grpo_training.py)**.--Copy this checklist:--```-GRPO Training:-- [ ] Step 1: Define reward function-- [ ] Step 2: Configure GRPO-- [ ] Step 3: Train with GRPOTrainer-```--**Step 1: Define reward function**--```python-def reward_function(completions, **kwargs):- """- Compute rewards for completions.-- Args:- completions: List of generated texts-- Returns:- List of reward scores (floats)- """- rewards = []- for completion in completions:- # Example: reward based on length and unique words- score = len(completion.split()) # Favor longer responses- score += len(set(completion.lower().split())) # Reward unique words- rewards.append(score)- return rewards-```--Or use a reward model:-```python-from transformers import pipeline--reward_model = pipeline("text-classification", model="reward-model-path")--def reward_from_model(completions, prompts, **kwargs):- # Combine prompt + completion- full_texts = [p + c for p, c in zip(prompts, completions)]- # Get reward scores- results = reward_model(full_texts)- return [r["score"] for r in results]-```--**Step 2: Configure GRPO**--```python-from trl import GRPOConfig--config = GRPOConfig(- output_dir="Qwen2-GRPO",- per_device_train_batch_size=4,- num_train_epochs=1,- learning_rate=1e-5,- num_generations=4, # Generate 4 completions per prompt- max_new_tokens=128-)-```--**Step 3: Train with GRPOTrainer**--```python-from datasets import load_dataset-from trl import GRPOTrainer--# Load prompt-only dataset-dataset = load_dataset("trl-lib/tldr", split="train")--trainer = GRPOTrainer(- model="Qwen/Qwen2-0.5B-Instruct",- reward_funcs=reward_function, # Your reward function- args=config,- train_dataset=dataset-)--trainer.train()-```--**CLI**:-```bash-trl grpo \- --model_name_or_path Qwen/Qwen2-0.5B-Instruct \- --dataset_name trl-lib/tldr \- --output_dir Qwen2-GRPO \- --num_generations 4-```--## When to use vs alternatives--**Use TRL when:**-- Need to align model with human preferences-- Have preference data (chosen/rejected pairs)-- Want to use reinforcement learning (RLOO, GRPO)-- Need reward model training-- Doing RLHF (full pipeline)--**Method selection**:-- **SFT**: Have prompt-completion pairs, want basic instruction following-- **DPO**: Have preferences, want simple alignment (no reward model needed)-- **RLOO**: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)-- **GRPO**: Memory-constrained, want online RL with reward functions-- **Reward Model**: Building RLHF pipeline, need to score generations--**Use alternatives instead:**-- **HuggingFace Trainer**: Basic fine-tuning without RL-- **Axolotl**: YAML-based training configuration-- **LitGPT**: Educational, minimal fine-tuning-- **Unsloth**: Fast LoRA training--## Common issues--**Issue: OOM during DPO training**--Reduce batch size and sequence length:-```python-config = DPOConfig(- per_device_train_batch_size=1, # Reduce from 4- max_length=512, # Reduce from 1024- gradient_accumulation_steps=8 # Maintain effective batch-)-```--Or use gradient checkpointing:-```python-model.gradient_checkpointing_enable()-```--**Issue: Poor alignment quality**--Tune beta parameter:-```python-# Higher beta = more conservative (stays closer to reference)-config = DPOConfig(beta=0.5) # Default 0.1--# Lower beta = more aggressive alignment-config = DPOConfig(beta=0.01)-```--**Issue: Reward model not learning**--Check loss type and learning rate:-```python-config = RewardConfig(- learning_rate=1e-5, # Try different LR- num_train_epochs=3 # Train longer-)-```--Ensure preference dataset has clear winners:-```python-# Verify dataset-print(dataset[0])-# Should have clear chosen > rejected-```--**Issue: Online RL (RLOO/GRPO) training unstable**--Adjust the KL/beta regularization toward the reference policy:-```python-from trl import RLOOConfig--config = RLOOConfig(- beta=0.05, # KL coefficient toward the reference model (increase for stability)- num_generations=4, # more samples per prompt = lower-variance advantage estimates-)-```--## Advanced topics--**SFT training guide**: See [references/sft-training.md](references/sft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.--**DPO variants**: See [references/dpo-variants.md](references/dpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.--**Reward modeling**: See [references/reward-modeling.md](references/reward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.--**Online RL methods**: See [references/online-rl.md](references/online-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.--**GRPO deep dive**: See [references/grpo-training.md](references/grpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates/basic_grpo_training.py](templates/basic_grpo_training.py).--## Hardware requirements--- **GPU**: NVIDIA (CUDA required)-- **VRAM**: Depends on model and method- - SFT 7B: 16GB (with LoRA)- - DPO 7B: 24GB (stores reference model)- - RLOO 7B: 40GB (policy + reward model)- - GRPO 7B: 24GB (more memory efficient)-- **Multi-GPU**: Supported via `accelerate`-- **Mixed precision**: BF16 recommended (A100/H100)--**Memory optimization**:-- Use LoRA/QLoRA for all methods-- Enable gradient checkpointing-- Use smaller batch sizes with gradient accumulation--## Resources--- Docs: https://huggingface.co/docs/trl/-- GitHub: https://github.com/huggingface/trl-- Papers:- - "Training language models to follow instructions with human feedback" (InstructGPT, 2022)- - "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO, 2023)- - "Group Relative Policy Optimization" (GRPO, 2024)-- Examples: https://github.com/huggingface/trl/tree/main/examples/scripts---diff --git a/optional-skills/mlops/training/trl-fine-tuning/references/dpo-variants.md b/optional-skills/mlops/training/trl-fine-tuning/references/dpo-variants.mddeleted file mode 100644index 5623b9ab847..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/references/dpo-variants.md+++ /dev/null@@ -1,227 +0,0 @@-# DPO Variants--Complete guide to Direct Preference Optimization loss variants in TRL.--## Overview--DPO optimizes models using preference data (chosen/rejected pairs). TRL supports 10+ loss variants for different scenarios.--## Loss Types--### 1. Sigmoid (Standard DPO)--**Formula**: `-log(sigmoid(β * logits))`--**When to use**: Default choice, general preference alignment--**Config**:-```python-DPOConfig(- loss_type="sigmoid",- beta=0.1, # KL penalty- per_device_train_batch_size=64,- learning_rate=1e-6-)-```--### 2. IPO (Identity Policy Optimization)--**Formula**: `(logits - 1/(2β))²`--**When to use**: Better theoretical foundation, reduce overfitting--**Config**:-```python-DPOConfig(- loss_type="ipo",- beta=0.1,- per_device_train_batch_size=90,- learning_rate=1e-2-)-```--### 3. Hinge (SLiC)--**Formula**: `ReLU(1 - β * logits)`--**When to use**: Margin-based objective--**Config**:-```python-DPOConfig(- loss_type="hinge",- beta=0.1,- per_device_train_batch_size=512,- learning_rate=1e-4-)-```--### 4. Robust DPO--**Formula**: Sigmoid with label smoothing for noise robustness--**When to use**: Noisy preference labels--**Config**:-```python-DPOConfig(- loss_type="robust",- beta=0.01,- label_smoothing=0.1, # Noise probability- per_device_train_batch_size=16,- learning_rate=1e-3,- max_prompt_length=128,- max_length=512-)-```--### 5. BCO Pair (Binary Classification)--**Formula**: Train binary classifier (chosen=1, rejected=0)--**When to use**: Pairwise preference data--**Config**:-```python-DPOConfig(- loss_type="bco_pair",- beta=0.01,- per_device_train_batch_size=128,- learning_rate=5e-7,- max_prompt_length=1536,- max_completion_length=512-)-```--### 6. SPPO Hard--**Formula**: Push chosen→0.5, rejected→-0.5--**When to use**: Nash equilibrium, sparse data--**Config**:-```python-DPOConfig(- loss_type="sppo_hard",- beta=0.1-)-```--### 7. DiscoPOP--**Formula**: Log-Ratio Modulated Loss--**When to use**: Automated loss discovery--**Config**:-```python-DPOConfig(- loss_type="discopop",- beta=0.05,- discopop_tau=0.05,- per_device_train_batch_size=64,- learning_rate=5e-7-)-```--### 8. APO Zero--**Formula**: Increase chosen, decrease rejected likelihood--**When to use**: Model worse than winning outputs--**Config**:-```python-DPOConfig(- loss_type="apo_zero",- beta=0.1,- per_device_train_batch_size=64,- learning_rate=2e-7,- max_prompt_length=512,- max_completion_length=512-)-```--### 9. APO Down--**Formula**: Decrease both, emphasize rejected reduction--**When to use**: Model better than winning outputs--**Config**:-```python-DPOConfig(- loss_type="apo_down",- beta=0.1,- # Same hyperparameters as apo_zero-)-```--### 10. AOT & AOT Pair--**Formula**: Distributional alignment via stochastic dominance--**When to use**:-- `aot_pair`: Paired preference data-- `aot`: Unpaired data--**Config**:-```python-DPOConfig(- loss_type="aot_pair", # or "aot"- beta=0.1,- label_smoothing=0.0-)-```--## Multi-Loss Training--Combine multiple losses:--```python-DPOConfig(- loss_type=["sigmoid", "ipo"],- loss_weights=[0.7, 0.3], # Weighted combination- beta=0.1-)-```--## Key Parameters--### Beta (β)--Controls deviation from reference model:-- **Higher** (0.5): More conservative, stays close to reference-- **Lower** (0.01): More aggressive alignment-- **Default**: 0.1--### Label Smoothing--For robust DPO:-- **0.0**: No smoothing (default)-- **0.1-0.3**: Moderate noise robustness-- **0.5**: Maximum noise tolerance--### Max Lengths--- `max_prompt_length`: 128-1536-- `max_completion_length`: 128-512-- `max_length`: Total sequence (1024-2048)--## Comparison Table--| Loss | Speed | Stability | Best For |-|------|-------|-----------|----------|-| Sigmoid | Fast | Good | **General use** |-| IPO | Fast | Better | Overfitting issues |-| Hinge | Fast | Good | Margin objectives |-| Robust | Fast | Best | Noisy data |-| BCO | Medium | Good | Binary classification |-| DiscoPOP | Fast | Good | New architectures |-| APO | Fast | Good | Model quality matching |--## References--- DPO paper: https://arxiv.org/abs/2305.18290-- IPO paper: https://arxiv.org/abs/2310.12036-- TRL docs: https://huggingface.co/docs/trl/dpo_trainerdiff --git a/optional-skills/mlops/training/trl-fine-tuning/references/grpo-training.md b/optional-skills/mlops/training/trl-fine-tuning/references/grpo-training.mddeleted file mode 100644index a22bd40945d..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/references/grpo-training.md+++ /dev/null@@ -1,504 +0,0 @@-# GRPO (Group Relative Policy Optimization) — Deep Guide--Expert-level patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions using TRL's `GRPOTrainer`. This is the deep reference for the GRPO workflow summarized in the main skill.--## When to use GRPO--Use GRPO when you need to:-- **Enforce specific output formats** (XML tags, JSON, structured reasoning)-- **Teach verifiable tasks** with objective correctness metrics (math, coding, fact-checking)-- **Improve reasoning capabilities** by rewarding chain-of-thought patterns-- **Align models to domain-specific behaviors** without labeled preference data-- **Optimize for multiple objectives** simultaneously (format + correctness + style)--**Do NOT use GRPO for:**-- Simple supervised fine-tuning tasks → use SFT-- Tasks without clear reward signals-- When you already have high-quality preference pairs → use DPO/PPO--## Core concepts--### 1. GRPO algorithm fundamentals--**Key mechanism:**-- Generates **multiple completions** per prompt (group size: 4–16)-- Compares completions within each group using reward functions-- Updates policy to favor higher-rewarded responses relative to the group--**Critical differences from PPO:**-- No separate reward model needed-- More sample-efficient (learns from within-group comparisons)-- Simpler to implement and debug--**Mathematical intuition:**-```-For each prompt p:- 1. Generate N completions: {c₁, c₂, ..., cₙ}- 2. Compute rewards: {r₁, r₂, ..., rₙ}- 3. Learn to increase probability of high-reward completions- relative to low-reward ones in the same group-```--### 2. Reward function design philosophy--**Golden rules:**-1. **Compose multiple reward functions** — each handles one aspect (format, correctness, style)-2. **Scale rewards appropriately** — higher weight = stronger signal-3. **Use incremental rewards** — partial credit for partial compliance-4. **Test rewards independently** — debug each reward function in isolation--**Reward function types:**--| Type | Use Case | Example Weight |-|------|----------|----------------|-| **Correctness** | Verifiable tasks (math, code) | 2.0 (highest) |-| **Format** | Strict structure enforcement | 0.5–1.0 |-| **Length** | Encourage verbosity/conciseness | 0.1–0.5 |-| **Style** | Penalize unwanted patterns | −0.5 to 0.5 |--## Implementation workflow--### Step 1: Dataset preparation--**Critical requirements:**-- Prompts in chat format (list of dicts with `role` and `content`)-- Include system prompts to set expectations-- For verifiable tasks, include ground truth answers as additional columns--```python-from datasets import load_dataset, Dataset--SYSTEM_PROMPT = """-Respond in the following format:-<reasoning>-[Your step-by-step thinking]-</reasoning>-<answer>-[Final answer]-</answer>-"""--def prepare_dataset(raw_data):- """Transform raw data into GRPO-compatible format.-- Returns: Dataset with columns:- - 'prompt': List[Dict] with role/content (system + user messages)- - 'answer': str (ground truth, optional but recommended)- """- return raw_data.map(lambda x: {- 'prompt': [- {'role': 'system', 'content': SYSTEM_PROMPT},- {'role': 'user', 'content': x['question']}- ],- 'answer': extract_answer(x['raw_answer'])- })-```--**Pro tips:**-- Use one-shot or few-shot examples in the system prompt for complex formats-- Keep prompts concise (max_prompt_length: 256–512 tokens)-- Validate data quality before training (garbage in = garbage out)--### Step 2: Reward function implementation--**Template structure:**-```python-def reward_function_name(- prompts, # List[List[Dict]]: Original prompts- completions, # List[List[Dict]]: Model generations- answer=None, # Optional: Ground truth from dataset- **kwargs # Additional dataset columns-) -> list[float]:- """Evaluate completions and return rewards (one per completion)."""- responses = [comp[0]['content'] for comp in completions]- rewards = []- for response in responses:- score = compute_score(response)- rewards.append(score)- return rewards-```--**Example 1: correctness reward (math/coding)**-```python-def correctness_reward(prompts, completions, answer, **kwargs):- """Reward correct answers with high score."""- responses = [comp[0]['content'] for comp in completions]- extracted = [extract_final_answer(r) for r in responses]- return [2.0 if ans == gt else 0.0- for ans, gt in zip(extracted, answer)]-```--**Example 2: format reward (structured output)**-```python-import re--def format_reward(completions, **kwargs):- """Reward XML-like structured format."""- pattern = r'<reasoning>.*?</reasoning>\s*<answer>.*?</answer>'- responses = [comp[0]['content'] for comp in completions]- return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0- for r in responses]-```--**Example 3: incremental format reward (partial credit)**-```python-def incremental_format_reward(completions, **kwargs):- """Award partial credit for format compliance."""- responses = [comp[0]['content'] for comp in completions]- rewards = []-- for r in responses:- score = 0.0- if '<reasoning>' in r: score += 0.25- if '</reasoning>' in r: score += 0.25- if '<answer>' in r: score += 0.25- if '</answer>' in r: score += 0.25- # Penalize extra text after closing tag- if r.count('</answer>') == 1:- extra_text = r.split('</answer>')[-1].strip()- score -= len(extra_text) * 0.001- rewards.append(score)-- return rewards-```--**Critical insight:** Combine 3–5 reward functions for robust training. Order matters less than diversity of signals.--### Step 3: Training configuration--**Memory-optimized config (small GPU)**-```python-from trl import GRPOConfig--training_args = GRPOConfig(- output_dir="outputs/grpo-model",-- # Learning rate- learning_rate=5e-6, # Lower = more stable- adam_beta1=0.9,- adam_beta2=0.99,- weight_decay=0.1,- warmup_ratio=0.1,- lr_scheduler_type='cosine',-- # Batch settings- per_device_train_batch_size=1,- gradient_accumulation_steps=4, # Effective batch = 4-- # GRPO-specific- num_generations=8, # Group size: 8–16 recommended- max_prompt_length=256,- max_completion_length=512,-- # Training duration- num_train_epochs=1,- max_steps=None,-- # Optimization- bf16=True, # Faster on A100/H100- optim="adamw_8bit", # Memory-efficient optimizer- max_grad_norm=0.1,-- # Logging- logging_steps=1,- save_steps=100,- report_to="wandb",-)-```--**High-performance config (large GPU)**-```python-training_args = GRPOConfig(- output_dir="outputs/grpo-model",- learning_rate=1e-5,- per_device_train_batch_size=4,- gradient_accumulation_steps=2,- num_generations=16, # Larger groups = better signal- max_prompt_length=512,- max_completion_length=1024,- num_train_epochs=1,- bf16=True,- use_vllm=True, # Fast generation with vLLM- logging_steps=10,-)-```--**Critical hyperparameters:**--| Parameter | Impact | Tuning Advice |-|-----------|--------|---------------|-| `num_generations` | Group size for comparison | Start 8, increase to 16 if GPU allows |-| `learning_rate` | Convergence speed/stability | 5e-6 (safe), 1e-5 (faster, riskier) |-| `max_completion_length` | Output verbosity | Match your task (512 reasoning, 256 short answers) |-| `gradient_accumulation_steps` | Effective batch size | Increase if GPU memory limited |--### Step 4: Model setup and training--**Standard setup (Transformers + TRL)**-```python-import torch-from transformers import AutoModelForCausalLM, AutoTokenizer-from peft import LoraConfig-from trl import GRPOTrainer--model_name = "Qwen/Qwen2.5-1.5B-Instruct"-model = AutoModelForCausalLM.from_pretrained(- model_name,- torch_dtype=torch.bfloat16,- attn_implementation="flash_attention_2", # 2–3× faster- device_map="auto",-)--tokenizer = AutoTokenizer.from_pretrained(model_name)-tokenizer.pad_token = tokenizer.eos_token--# Optional: LoRA for parameter-efficient training-peft_config = LoraConfig(- r=16,- lora_alpha=32,- target_modules=[- "q_proj", "k_proj", "v_proj", "o_proj",- "gate_proj", "up_proj", "down_proj",- ],- task_type="CAUSAL_LM",- lora_dropout=0.05,-)--trainer = GRPOTrainer(- model=model,- processing_class=tokenizer,- reward_funcs=[- incremental_format_reward,- format_reward,- correctness_reward,- ],- args=training_args,- train_dataset=dataset,- peft_config=peft_config, # Remove for full fine-tuning-)--trainer.train()-trainer.save_model("final_model")-```--**Unsloth setup (2–3× faster)**-```python-from unsloth import FastLanguageModel--model, tokenizer = FastLanguageModel.from_pretrained(- model_name="google/gemma-3-1b-it",- max_seq_length=1024,- load_in_4bit=True,- fast_inference=True,- max_lora_rank=32,-)--model = FastLanguageModel.get_peft_model(- model,- r=32,- target_modules=["q_proj", "k_proj", "v_proj", "o_proj",- "gate_proj", "up_proj", "down_proj"],- lora_alpha=32,- use_gradient_checkpointing="unsloth",-)--# Rest is identical to the standard setup-trainer = GRPOTrainer(model=model, ...)-trainer.train()-```--## Critical training insights--### 1. Loss behavior (EXPECTED pattern)-- **Loss starts near 0 and INCREASES during training** — this is CORRECT-- Loss measures KL divergence from initial policy; the model is learning (diverging from original behavior to optimize rewards)-- **Monitor reward metrics, not loss, for progress**--### 2. Reward tracking--Key metrics to watch:-- `reward` — average across all completions-- `reward_std` — diversity within groups (should remain > 0)-- `kl` — KL divergence from reference (should grow moderately)--**Healthy pattern:**-```-Step Reward Reward_Std KL-100 0.5 0.3 0.02-200 0.8 0.25 0.05-300 1.2 0.2 0.08 ← Good progression-400 1.5 0.15 0.12-```--**Warning signs:**-- `reward_std` → 0 (model collapsing to a single response)-- `kl` exploding (> 0.5) — diverging too much, reduce LR-- Reward stuck — reward functions too harsh or model capacity issue--### 3. Common pitfalls and solutions--| Problem | Symptom | Solution |-|---------|---------|----------|-| **Mode collapse** | All completions identical | Increase `num_generations`, add diversity penalty |-| **No learning** | Flat rewards | Check reward function logic, increase LR |-| **OOM errors** | GPU memory exceeded | Reduce `num_generations`, enable gradient checkpointing |-| **Slow training** | < 1 it/s | Enable `use_vllm=True`, use Unsloth, reduce seq length |-| **Format ignored** | Model doesn't follow structure | Increase format reward weight, add incremental rewards |--## Advanced patterns--### 1. Multi-stage training--For complex tasks, train in stages:--```python-# Stage 1: Format compliance-trainer_stage1 = GRPOTrainer(- model=model,- reward_funcs=[incremental_format_reward, format_reward],- ...-)-trainer_stage1.train()--# Stage 2: Correctness-trainer_stage2 = GRPOTrainer(- model=model,- reward_funcs=[format_reward, correctness_reward],- ...-)-trainer_stage2.train()-```--### 2. Adaptive reward scaling--```python-class AdaptiveReward:- def __init__(self, base_reward_func, initial_weight=1.0):- self.func = base_reward_func- self.weight = initial_weight-- def __call__(self, *args, **kwargs):- rewards = self.func(*args, **kwargs)- return [r * self.weight for r in rewards]-- def adjust_weight(self, success_rate):- """Increase weight if model struggling, decrease if succeeding."""- if success_rate < 0.3:- self.weight *= 1.2- elif success_rate > 0.8:- self.weight *= 0.9-```--### 3. Custom dataset integration--```python-def load_custom_knowledge_base(csv_path):- import pandas as pd- df = pd.read_csv(csv_path)- return Dataset.from_pandas(df).map(lambda x: {- 'prompt': [- {'role': 'system', 'content': CUSTOM_SYSTEM_PROMPT},- {'role': 'user', 'content': x['question']}- ],- 'answer': x['expert_answer']- })-```--## Deployment and inference--### Save and merge LoRA-```python-if hasattr(trainer.model, 'merge_and_unload'):- merged_model = trainer.model.merge_and_unload()- merged_model.save_pretrained("production_model")- tokenizer.save_pretrained("production_model")-```--### Inference-```python-from transformers import pipeline--generator = pipeline("text-generation", model="production_model", tokenizer=tokenizer)--result = generator(- [- {'role': 'system', 'content': SYSTEM_PROMPT},- {'role': 'user', 'content': "What is 15 + 27?"},- ],- max_new_tokens=256,- do_sample=True,- temperature=0.7,- top_p=0.9,-)-print(result[0]['generated_text'])-```--## Best practices checklist--**Before training:**-- [ ] Validate dataset format (prompts as List[Dict])-- [ ] Test reward functions on sample data-- [ ] Calculate expected `max_prompt_length` from data-- [ ] Choose `num_generations` based on GPU memory-- [ ] Set up logging (wandb recommended)--**During training:**-- [ ] Monitor reward progression (should increase)-- [ ] Check `reward_std` (should stay > 0.1)-- [ ] Watch for OOM errors (reduce batch size if needed)-- [ ] Sample generations every 50–100 steps-- [ ] Validate format compliance on holdout set--**After training:**-- [ ] Merge LoRA weights if using PEFT-- [ ] Test on diverse prompts-- [ ] Compare to baseline model-- [ ] Document reward weights and hyperparameters-- [ ] Save reproducibility config--## Troubleshooting--### Debugging workflow-1. **Isolate reward functions** — test each independently-2. **Check data distribution** — ensure diversity in prompts-3. **Reduce complexity** — start with single reward, add gradually-4. **Monitor generations** — print samples every N steps-5. **Validate extraction logic** — ensure answer parsing works--### Quick debug reward-```python-def debug_reward(completions, **kwargs):- responses = [comp[0]['content'] for comp in completions]- for i, r in enumerate(responses[:2]):- print(f"Response {i}: {r[:200]}...")- return [1.0] * len(responses)--# Test without training-trainer = GRPOTrainer(..., reward_funcs=[debug_reward])-trainer.generate_completions(dataset[:1])-```--## Template--A production-ready training script lives at **`../templates/basic_grpo_training.py`**. It uses Qwen 2.5-1.5B-Instruct with LoRA and three reward functions (incremental format, strict format, correctness) on GSM8K. Copy and adapt:-1. `get_dataset()` — swap in your data loader-2. Reward functions — tune to your task-3. `SYSTEM_PROMPT` — match your output format-4. `GRPOConfig` — adjust hyperparameters for your GPU--## References and resources--- TRL GRPO Trainer: https://huggingface.co/docs/trl/grpo_trainer-- GRPO paper (DeepSeek): https://arxiv.org/abs/2402.03300-- DeepSeek R1 paper: https://arxiv.org/abs/2501.12948-- Open R1 implementation: https://github.com/huggingface/open-r1-- TRL examples: https://github.com/huggingface/trl/tree/main/examples-- Unsloth (faster training): https://docs.unsloth.ai/--## Critical reminders--- **Loss goes UP during training** — this is normal (it's KL divergence)-- **Use 3–5 reward functions** — single rewards often fail-- **Test rewards before training** — debug each function independently-- **Monitor `reward_std`** — should stay > 0.1 (avoid mode collapse)-- **Start with `num_generations=4–8`** — scale up if GPU allowsdiff --git a/optional-skills/mlops/training/trl-fine-tuning/references/online-rl.md b/optional-skills/mlops/training/trl-fine-tuning/references/online-rl.mddeleted file mode 100644index 87f46e91fb2..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/references/online-rl.md+++ /dev/null@@ -1,82 +0,0 @@-# Online RL Methods--Guide to online reinforcement learning with PPO, GRPO, RLOO, and OnlineDPO.--## Overview--Online RL generates completions during training and optimizes based on rewards.--## PPO (Proximal Policy Optimization)--Classic RL algorithm for LLM alignment.--### Basic Usage--```bash-python -m trl.scripts.ppo \- --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \- --reward_model_path reward-model \- --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \- --output_dir model-ppo \- --learning_rate 3e-6 \- --per_device_train_batch_size 64 \- --total_episodes 10000 \- --num_ppo_epochs 4 \- --kl_coef 0.05-```--### Key Parameters--- `kl_coef`: KL penalty (0.05-0.2)-- `num_ppo_epochs`: Epochs per batch (2-4)-- `cliprange`: PPO clip (0.1-0.3)-- `vf_coef`: Value function coef (0.1)--## GRPO (Group Relative Policy Optimization)--Memory-efficient online RL.--### Basic Usage--```python-from trl import GRPOTrainer, GRPOConfig-from datasets import load_dataset--# Define reward function-def reward_func(completions, **kwargs):- return [len(set(c.split())) for c in completions]--config = GRPOConfig(- output_dir="model-grpo",- num_generations=4, # Completions per prompt- max_new_tokens=128-)--trainer = GRPOTrainer(- model="Qwen/Qwen2-0.5B-Instruct",- reward_funcs=reward_func,- args=config,- train_dataset=load_dataset("trl-lib/tldr", split="train")-)-trainer.train()-```--### Key Parameters--- `num_generations`: 2-8 completions-- `max_new_tokens`: 64-256-- Learning rate: 1e-5 to 1e-4--## Memory Comparison--| Method | Memory (7B) | Speed | Use Case |-|--------|-------------|-------|----------|-| PPO | 40GB | Medium | Maximum control |-| GRPO | 24GB | Fast | **Memory-constrained** |-| OnlineDPO | 28GB | Fast | No reward model |--## References--- PPO paper: https://arxiv.org/abs/1707.06347-- GRPO paper: https://arxiv.org/abs/2402.03300-- TRL docs: https://huggingface.co/docs/trl/diff --git a/optional-skills/mlops/training/trl-fine-tuning/references/reward-modeling.md b/optional-skills/mlops/training/trl-fine-tuning/references/reward-modeling.mddeleted file mode 100644index 3b59695b19d..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/references/reward-modeling.md+++ /dev/null@@ -1,122 +0,0 @@-# Reward Modeling--Guide to training reward models with TRL for RLHF pipelines.--## Overview--Reward models score completions based on human preferences. Used in:-- PPO training (RL feedback)-- GRPO online RL-- Completion ranking--## Basic Training--```python-from transformers import AutoModelForSequenceClassification, AutoTokenizer-from trl import RewardTrainer, RewardConfig-from datasets import load_dataset--# Load model (num_labels=1 for single reward score)-model = AutoModelForSequenceClassification.from_pretrained(- "Qwen/Qwen2.5-0.5B-Instruct",- num_labels=1-)-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")--# Load preference dataset (chosen/rejected pairs)-dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")--# Configure-config = RewardConfig(- output_dir="Qwen2.5-Reward",- per_device_train_batch_size=2,- num_train_epochs=1,- learning_rate=1e-5-)--# Train-trainer = RewardTrainer(- model=model,- args=config,- processing_class=tokenizer,- train_dataset=dataset-)-trainer.train()-```--## Dataset Format--Required fields:-```json-{- "prompt": "Question or instruction",- "chosen": "Better response",- "rejected": "Worse response"-}-```--## Bradley-Terry Loss--Default loss function:-```-loss = -log(sigmoid(reward_chosen - reward_rejected))-```--Learns to score chosen > rejected.--## Using Reward Models--### Inference--```python-from transformers import pipeline--# Load trained reward model-reward_pipe = pipeline("text-classification", model="Qwen2.5-Reward")--# Score completions-texts = ["Good answer", "Bad answer"]-scores = reward_pipe(texts)-print(scores) # Higher score = better-```--### In PPO--```python-from trl import PPOTrainer, PPOConfig--config = PPOConfig(- reward_model_path="Qwen2.5-Reward" # Use trained reward model-)--trainer = PPOTrainer(- model=policy_model,- config=config,- # Reward model loaded automatically-)-```--## Hyperparameters--| Model Size | Learning Rate | Batch Size | Epochs |-|------------|---------------|------------|--------|-| <1B | 2e-5 | 4-8 | 1-2 |-| 1-7B | 1e-5 | 2-4 | 1 |-| 7-13B | 5e-6 | 1-2 | 1 |--## Evaluation--Check reward separation:-```python-# Chosen should score higher than rejected-chosen_rewards = model(**chosen_inputs).logits-rejected_rewards = model(**rejected_inputs).logits--accuracy = (chosen_rewards > rejected_rewards).float().mean()-print(f"Accuracy: {accuracy:.2%}") # Target: >80%-```--## References--- InstructGPT paper: https://arxiv.org/abs/2203.02155-- TRL docs: https://huggingface.co/docs/trl/reward_trainerdiff --git a/optional-skills/mlops/training/trl-fine-tuning/references/sft-training.md b/optional-skills/mlops/training/trl-fine-tuning/references/sft-training.mddeleted file mode 100644index cd4294c63b6..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/references/sft-training.md+++ /dev/null@@ -1,168 +0,0 @@-# SFT Training Guide--Complete guide to Supervised Fine-Tuning (SFT) with TRL for instruction tuning and task-specific fine-tuning.--## Overview--SFT trains models on input-output pairs to minimize cross-entropy loss. Use for:-- Instruction following-- Task-specific fine-tuning-- Chatbot training-- Domain adaptation--## Dataset Formats--### Format 1: Prompt-Completion--```json-[- {- "prompt": "What is the capital of France?",- "completion": "The capital of France is Paris."- }-]-```--### Format 2: Conversational (ChatML)--```json-[- {- "messages": [- {"role": "user", "content": "What is Python?"},- {"role": "assistant", "content": "Python is a programming language."}- ]- }-]-```--### Format 3: Text-only--```json-[- {"text": "User: Hello\nAssistant: Hi! How can I help?"}-]-```--## Basic Training--```python-from trl import SFTTrainer, SFTConfig-from transformers import AutoModelForCausalLM, AutoTokenizer-from datasets import load_dataset--# Load model-model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")-tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")--# Load dataset-dataset = load_dataset("trl-lib/Capybara", split="train")--# Configure-config = SFTConfig(- output_dir="Qwen2.5-SFT",- per_device_train_batch_size=4,- num_train_epochs=1,- learning_rate=2e-5,- save_strategy="epoch"-)--# Train-trainer = SFTTrainer(- model=model,- args=config,- train_dataset=dataset,- tokenizer=tokenizer-)-trainer.train()-```--## Chat Templates--Apply chat templates automatically:--```python-trainer = SFTTrainer(- model=model,- args=config,- train_dataset=dataset, # Messages format- tokenizer=tokenizer- # Chat template applied automatically-)-```--Or manually:-```python-def format_chat(example):- messages = example["messages"]- text = tokenizer.apply_chat_template(messages, tokenize=False)- return {"text": text}--dataset = dataset.map(format_chat)-```--## Packing for Efficiency--Pack multiple sequences into one to maximize GPU utilization:--```python-config = SFTConfig(- packing=True, # Enable packing- max_seq_length=2048,- dataset_text_field="text"-)-```--**Benefits**: 2-3× faster training-**Trade-off**: Slightly more complex batching--## Multi-GPU Training--```bash-accelerate launch --num_processes 4 train_sft.py-```--Or with config:-```python-config = SFTConfig(- output_dir="model-sft",- per_device_train_batch_size=4,- gradient_accumulation_steps=4,- num_train_epochs=1-)-```--## LoRA Fine-Tuning--```python-from peft import LoraConfig--lora_config = LoraConfig(- r=16,- lora_alpha=32,- target_modules="all-linear",- lora_dropout=0.05,- task_type="CAUSAL_LM"-)--trainer = SFTTrainer(- model=model,- args=config,- train_dataset=dataset,- peft_config=lora_config # Add LoRA-)-```--## Hyperparameters--| Model Size | Learning Rate | Batch Size | Epochs |-|------------|---------------|------------|--------|-| <1B | 5e-5 | 8-16 | 1-3 |-| 1-7B | 2e-5 | 4-8 | 1-2 |-| 7-13B | 1e-5 | 2-4 | 1 |-| 13B+ | 5e-6 | 1-2 | 1 |--## References--- TRL docs: https://huggingface.co/docs/trl/sft_trainer-- Examples: https://github.com/huggingface/trl/tree/main/examples/scriptsdiff --git a/optional-skills/mlops/training/trl-fine-tuning/templates/basic_grpo_training.py b/optional-skills/mlops/training/trl-fine-tuning/templates/basic_grpo_training.pydeleted file mode 100644index 8ad45dfcf4e..00000000000--- a/optional-skills/mlops/training/trl-fine-tuning/templates/basic_grpo_training.py+++ /dev/null@@ -1,228 +0,0 @@-"""-Basic GRPO Training Template-=============================--A minimal, production-ready template for GRPO training with TRL.-Adapt this for your specific task by modifying:-1. Dataset loading (get_dataset function)-2. Reward functions (reward_*_func)-3. System prompt (SYSTEM_PROMPT)-4. Hyperparameters (GRPOConfig)-"""--import torch-import re-from datasets import load_dataset-from transformers import AutoModelForCausalLM, AutoTokenizer-from peft import LoraConfig-from trl import GRPOTrainer, GRPOConfig--# ==================== CONFIGURATION ====================--MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"-OUTPUT_DIR = "outputs/grpo-model"-MAX_PROMPT_LENGTH = 256-MAX_COMPLETION_LENGTH = 512--SYSTEM_PROMPT = """-Respond in the following format:-<reasoning>-[Your step-by-step thinking]-</reasoning>-<answer>-[Final answer]-</answer>-"""--# ==================== DATASET ====================--def get_dataset(split="train"):- """- Load and prepare your dataset.-- Returns: Dataset with columns:- - 'prompt': List[Dict] with role/content- - 'answer': str (ground truth, optional)- """- # Example: GSM8K math dataset- data = load_dataset('openai/gsm8k', 'main')[split]-- def process_example(x):- # Extract ground truth answer- answer = x['answer'].split('####')[1].strip() if '####' in x['answer'] else None-- return {- 'prompt': [- {'role': 'system', 'content': SYSTEM_PROMPT},- {'role': 'user', 'content': x['question']}- ],- 'answer': answer- }-- return data.map(process_example)--# ==================== HELPER FUNCTIONS ====================--def extract_xml_tag(text: str, tag: str) -> str:- """Extract content between XML tags."""- pattern = f'<{tag}>(.*?)</{tag}>'- match = re.search(pattern, text, re.DOTALL)- return match.group(1).strip() if match else ""--def extract_answer(text: str) -> str:- """Extract the final answer from structured output."""- return extract_xml_tag(text, 'answer')--# ==================== REWARD FUNCTIONS ====================--def correctness_reward_func(prompts, completions, answer, **kwargs):- """- Reward correct answers.- Weight: 2.0 (highest priority)- """- responses = [comp[0]['content'] for comp in completions]- extracted = [extract_answer(r) for r in responses]- return [2.0 if ans == gt else 0.0 for ans, gt in zip(extracted, answer)]--def format_reward_func(completions, **kwargs):- """- Reward proper XML format.- Weight: 0.5- """- pattern = r'<reasoning>.*?</reasoning>\s*<answer>.*?</answer>'- responses = [comp[0]['content'] for comp in completions]- return [0.5 if re.search(pattern, r, re.DOTALL) else 0.0 for r in responses]--def incremental_format_reward_func(completions, **kwargs):- """- Incremental reward for partial format compliance.- Weight: up to 0.5- """- responses = [comp[0]['content'] for comp in completions]- rewards = []-- for r in responses:- score = 0.0- if '<reasoning>' in r:- score += 0.125- if '</reasoning>' in r:- score += 0.125- if '<answer>' in r:- score += 0.125- if '</answer>' in r:- score += 0.125-- # Penalize extra content after closing tag- if '</answer>' in r:- extra = r.split('</answer>')[-1].strip()- score -= len(extra) * 0.001-- rewards.append(score)-- return rewards--# ==================== MODEL SETUP ====================--def setup_model_and_tokenizer():- """Load model and tokenizer with optimizations."""- model = AutoModelForCausalLM.from_pretrained(- MODEL_NAME,- torch_dtype=torch.bfloat16,- attn_implementation="flash_attention_2",- device_map="auto"- )-- tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)- tokenizer.pad_token = tokenizer.eos_token-- return model, tokenizer--def get_peft_config():- """LoRA configuration for parameter-efficient training."""- return LoraConfig(- r=16,- lora_alpha=32,- target_modules=[- "q_proj", "k_proj", "v_proj", "o_proj",- "gate_proj", "up_proj", "down_proj"- ],- task_type="CAUSAL_LM",- lora_dropout=0.05,- )--# ==================== TRAINING ====================--def main():- """Main training function."""-- # Load data- print("Loading dataset...")- dataset = get_dataset()- print(f"Dataset size: {len(dataset)}")-- # Setup model- print("Loading model...")- model, tokenizer = setup_model_and_tokenizer()-- # Training configuration- training_args = GRPOConfig(- output_dir=OUTPUT_DIR,- run_name="grpo-training",-- # Learning rate- learning_rate=5e-6,- adam_beta1=0.9,- adam_beta2=0.99,- weight_decay=0.1,- warmup_ratio=0.1,- lr_scheduler_type='cosine',-- # Batch settings- per_device_train_batch_size=1,- gradient_accumulation_steps=4,-- # GRPO specific- num_generations=8,- max_prompt_length=MAX_PROMPT_LENGTH,- max_completion_length=MAX_COMPLETION_LENGTH,-- # Training duration- num_train_epochs=1,-- # Optimization- bf16=True,- optim="adamw_8bit",- max_grad_norm=0.1,-- # Logging- logging_steps=1,- save_steps=100,- report_to="wandb", # Change to "none" to disable logging- )-- # Initialize trainer- trainer = GRPOTrainer(- model=model,- processing_class=tokenizer,- reward_funcs=[- incremental_format_reward_func,- format_reward_func,- correctness_reward_func,- ],- args=training_args,- train_dataset=dataset,- peft_config=get_peft_config(),- )-- # Train- print("Starting training...")- trainer.train()-- # Save final model- print(f"Saving model to {OUTPUT_DIR}/final")- trainer.save_model(f"{OUTPUT_DIR}/final")-- print("Training complete!")--if __name__ == "__main__":- main()diff --git a/optional-skills/mlops/whisper/SKILL.md b/optional-skills/mlops/whisper/SKILL.mddeleted file mode 100644index e5d1da634d7..00000000000--- a/optional-skills/mlops/whisper/SKILL.md+++ /dev/null@@ -1,321 +0,0 @@-----name: whisper-description: Transcribe and translate speech in 99 languages.-version: 1.0.0-author: Orchestra Research-license: MIT-dependencies: [openai-whisper, transformers, torch]-platforms: [linux, macos]-metadata:- hermes:- tags: [Whisper, Speech Recognition, ASR, Multimodal, Multilingual, OpenAI, Speech-To-Text, Transcription, Translation, Audio Processing]-------# Whisper - Robust Speech Recognition--OpenAI's multilingual speech recognition model.--## When to use Whisper--**Use when:**-- Speech-to-text transcription (99 languages)-- Podcast/video transcription-- Meeting notes automation-- Translation to English-- Noisy audio transcription-- Multilingual audio processing--**Metrics**:-- **72,900+ GitHub stars**-- 99 languages supported-- Trained on 680,000 hours of audio-- MIT License--**Use alternatives instead**:-- **AssemblyAI**: Managed API, speaker diarization-- **Deepgram**: Real-time streaming ASR-- **Google Speech-to-Text**: Cloud-based--## Quick start--### Installation--```bash-# Requires Python 3.8-3.11-pip install -U openai-whisper--# Requires ffmpeg-# macOS: brew install ffmpeg-# Ubuntu: sudo apt install ffmpeg-# Windows: choco install ffmpeg-```--### Basic transcription--```python-import whisper--# Load model-model = whisper.load_model("base")--# Transcribe-result = model.transcribe("audio.mp3")--# Print text-print(result["text"])--# Access segments-for segment in result["segments"]:- print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")-```--## Model sizes--```python-# Available models-models = ["tiny", "base", "small", "medium", "large", "turbo"]--# Load specific model-model = whisper.load_model("turbo") # Fastest, good quality-```--| Model | Parameters | English-only | Multilingual | Speed | VRAM |-|-------|------------|--------------|--------------|-------|------|-| tiny | 39M | ✓ | ✓ | ~32x | ~1 GB |-| base | 74M | ✓ | ✓ | ~16x | ~1 GB |-| small | 244M | ✓ | ✓ | ~6x | ~2 GB |-| medium | 769M | ✓ | ✓ | ~2x | ~5 GB |-| large | 1550M | ✗ | ✓ | 1x | ~10 GB |-| turbo | 809M | ✗ | ✓ | ~8x | ~6 GB |--**Recommendation**: Use `turbo` for best speed/quality, `base` for prototyping--## Transcription options--### Language specification--```python-# Auto-detect language-result = model.transcribe("audio.mp3")--# Specify language (faster)-result = model.transcribe("audio.mp3", language="en")--# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more-```--### Task selection--```python-# Transcription (default)-result = model.transcribe("audio.mp3", task="transcribe")--# Translation to English-result = model.transcribe("spanish.mp3", task="translate")-# Input: Spanish audio → Output: English text-```--### Initial prompt--```python-# Improve accuracy with context-result = model.transcribe(- "audio.mp3",- initial_prompt="This is a technical podcast about machine learning and AI."-)--# Helps with:-# - Technical terms-# - Proper nouns-# - Domain-specific vocabulary-```--### Timestamps--```python-# Word-level timestamps-result = model.transcribe("audio.mp3", word_timestamps=True)--for segment in result["segments"]:- for word in segment["words"]:- print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")-```--### Temperature fallback--```python-# Retry with different temperatures if confidence low-result = model.transcribe(- "audio.mp3",- temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)-)-```--## Command line usage--```bash-# Basic transcription-whisper audio.mp3--# Specify model-whisper audio.mp3 --model turbo--# Output formats-whisper audio.mp3 --output_format txt # Plain text-whisper audio.mp3 --output_format srt # Subtitles-whisper audio.mp3 --output_format vtt # WebVTT-whisper audio.mp3 --output_format json # JSON with timestamps--# Language-whisper audio.mp3 --language Spanish--# Translation-whisper spanish.mp3 --task translate-```--## Batch processing--```python-import os--audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]--for audio_file in audio_files:- print(f"Transcribing {audio_file}...")- result = model.transcribe(audio_file)-- # Save to file- output_file = audio_file.replace(".mp3", ".txt")- with open(output_file, "w") as f:- f.write(result["text"])-```--## Real-time transcription--```python-# For streaming audio, use faster-whisper-# pip install faster-whisper--from faster_whisper import WhisperModel--model = WhisperModel("base", device="cuda", compute_type="float16")--# Transcribe with streaming-segments, info = model.transcribe("audio.mp3", beam_size=5)--for segment in segments:- print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")-```--## GPU acceleration--```python-import whisper--# Automatically uses GPU if available-model = whisper.load_model("turbo")--# Force CPU-model = whisper.load_model("turbo", device="cpu")--# Force GPU-model = whisper.load_model("turbo", device="cuda")--# 10-20× faster on GPU-```--## Integration with other tools--### Subtitle generation--```bash-# Generate SRT subtitles-whisper video.mp4 --output_format srt --language English--# Output: video.srt-```--### With LangChain--```python-from langchain.document_loaders import WhisperTranscriptionLoader--loader = WhisperTranscriptionLoader(file_path="audio.mp3")-docs = loader.load()--# Use transcription in RAG-from langchain_chroma import Chroma-from langchain_openai import OpenAIEmbeddings--vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())-```--### Extract audio from video--```bash-# Use ffmpeg to extract audio-ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav--# Then transcribe-whisper audio.wav-```--## Best practices--1. **Use turbo model** - Best speed/quality for English-2. **Specify language** - Faster than auto-detect-3. **Add initial prompt** - Improves technical terms-4. **Use GPU** - 10-20× faster-5. **Batch process** - More efficient-6. **Convert to WAV** - Better compatibility-7. **Split long audio** - <30 min chunks-8. **Check language support** - Quality varies by language-9. **Use faster-whisper** - 4× faster than openai-whisper-10. **Monitor VRAM** - Scale model size to hardware--## Performance--| Model | Real-time factor (CPU) | Real-time factor (GPU) |-|-------|------------------------|------------------------|-| tiny | ~0.32 | ~0.01 |-| base | ~0.16 | ~0.01 |-| turbo | ~0.08 | ~0.01 |-| large | ~1.0 | ~0.05 |--*Real-time factor: 0.1 = 10× faster than real-time*--## Language support--Top-supported languages:-- English (en)-- Spanish (es)-- French (fr)-- German (de)-- Italian (it)-- Portuguese (pt)-- Russian (ru)-- Japanese (ja)-- Korean (ko)-- Chinese (zh)--Full list: 99 languages total--## Limitations--1. **Hallucinations** - May repeat or invent text-2. **Long-form accuracy** - Degrades on >30 min audio-3. **Speaker identification** - No diarization-4. **Accents** - Quality varies-5. **Background noise** - Can affect accuracy-6. **Real-time latency** - Not suitable for live captioning--## Resources--- **GitHub**: https://github.com/openai/whisper ⭐ 72,900+-- **Paper**: https://arxiv.org/abs/2212.04356-- **Model Card**: https://github.com/openai/whisper/blob/main/model-card.md-- **Colab**: Available in repo-- **License**: MIT--diff --git a/optional-skills/mlops/whisper/references/languages.md b/optional-skills/mlops/whisper/references/languages.mddeleted file mode 100644index dd17e123a08..00000000000--- a/optional-skills/mlops/whisper/references/languages.md+++ /dev/null@@ -1,189 +0,0 @@-# Whisper Language Support Guide--Complete guide to Whisper's multilingual capabilities.--## Supported languages (99 total)--### Top-tier support (WER < 10%)--- English (en)-- Spanish (es)-- French (fr)-- German (de)-- Italian (it)-- Portuguese (pt)-- Dutch (nl)-- Polish (pl)-- Russian (ru)-- Japanese (ja)-- Korean (ko)-- Chinese (zh)--### Good support (WER 10-20%)--- Arabic (ar)-- Turkish (tr)-- Vietnamese (vi)-- Swedish (sv)-- Finnish (fi)-- Czech (cs)-- Romanian (ro)-- Hungarian (hu)-- Danish (da)-- Norwegian (no)-- Thai (th)-- Hebrew (he)-- Greek (el)-- Indonesian (id)-- Malay (ms)--### Full list (99 languages)--Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Azerbaijani, Bashkir, Basque, Belarusian, Bengali, Bosnian, Breton, Bulgarian, Burmese, Cantonese, Catalan, Chinese, Croatian, Czech, Danish, Dutch, English, Estonian, Faroese, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Lao, Latin, Latvian, Lingala, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Moldavian, Mongolian, Myanmar, Nepali, Norwegian, Nynorsk, Occitan, Pashto, Persian, Polish, Portuguese, Punjabi, Pushto, Romanian, Russian, Sanskrit, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Spanish, Sundanese, Swahili, Swedish, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkish, Turkmen, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, Yiddish, Yoruba--## Usage examples--### Auto-detect language--```python-import whisper--model = whisper.load_model("turbo")--# Auto-detect language-result = model.transcribe("audio.mp3")--print(f"Detected language: {result['language']}")-print(f"Text: {result['text']}")-```--### Specify language (faster)--```python-# Specify language for faster transcription-result = model.transcribe("audio.mp3", language="es") # Spanish-result = model.transcribe("audio.mp3", language="fr") # French-result = model.transcribe("audio.mp3", language="ja") # Japanese-```--### Translation to English--```python-# Translate any language to English-result = model.transcribe(- "spanish_audio.mp3",- task="translate" # Translates to English-)--print(f"Original language: {result['language']}")-print(f"English translation: {result['text']}")-```--## Language-specific tips--### Chinese--```python-# Chinese works well with larger models-model = whisper.load_model("large")--result = model.transcribe(- "chinese_audio.mp3",- language="zh",- initial_prompt="这是一段关于技术的讨论" # Context helps-)-```--### Japanese--```python-# Japanese benefits from initial prompt-result = model.transcribe(- "japanese_audio.mp3",- language="ja",- initial_prompt="これは技術的な会議の録音です"-)-```--### Arabic--```python-# Arabic: Use large model for best results-model = whisper.load_model("large")--result = model.transcribe(- "arabic_audio.mp3",- language="ar"-)-```--## Model size recommendations--| Language Tier | Recommended Model | WER |-|---------------|-------------------|-----|-| Top-tier (en, es, fr, de) | base/turbo | < 10% |-| Good (ar, tr, vi) | medium/large | 10-20% |-| Lower-resource | large | 20-30% |--## Performance by language--### English--- **tiny**: WER ~15%-- **base**: WER ~8%-- **small**: WER ~5%-- **medium**: WER ~4%-- **large**: WER ~3%-- **turbo**: WER ~3.5%--### Spanish--- **tiny**: WER ~20%-- **base**: WER ~12%-- **medium**: WER ~6%-- **large**: WER ~4%--### Chinese--- **small**: WER ~15%-- **medium**: WER ~8%-- **large**: WER ~5%--## Best practices--1. **Use English-only models** - Better for small models (tiny/base)-2. **Specify language** - Faster than auto-detect-3. **Add initial prompt** - Improves accuracy for technical terms-4. **Use larger models** - For low-resource languages-5. **Test on sample** - Quality varies by accent/dialect-6. **Consider audio quality** - Clear audio = better results-7. **Check language codes** - Use ISO 639-1 codes (2 letters)--## Language detection--```python-# Detect language only (no transcription)-import whisper--model = whisper.load_model("base")--# Load audio-audio = whisper.load_audio("audio.mp3")-audio = whisper.pad_or_trim(audio)--# Make log-Mel spectrogram-mel = whisper.log_mel_spectrogram(audio).to(model.device)--# Detect language-_, probs = model.detect_language(mel)-detected_language = max(probs, key=probs.get)--print(f"Detected language: {detected_language}")-print(f"Confidence: {probs[detected_language]:.2%}")-```--## Resources--- **Paper**: https://arxiv.org/abs/2212.04356-- **GitHub**: https://github.com/openai/whisper-- **Model Card**: https://github.com/openai/whisper/blob/main/model-card.mddiff --git a/optional-skills/research/pinecone-research/SKILL.md b/optional-skills/research/pinecone-research/SKILL.mddeleted file mode 100644index 8be3c285ad1..00000000000--- a/optional-skills/research/pinecone-research/SKILL.md+++ /dev/null@@ -1,108 +0,0 @@-----name: pinecone-research-description: Agent RAG and long-term memory with Pinecone.-version: 1.0.0-author: immuhammadfurqan-license: MIT-dependencies: [pinecone-client, langchain-pinecone]-platforms: [linux, macos, windows]-metadata:- hermes:- tags: [RAG, Pinecone, Memory, Research, Vector Database, Agent, Retrieval]-------# Pinecone Research — Agent RAG & Long-Term Memory--Use Pinecone as a retrieval-augmented generation (RAG) backend for agent-conversations: persist embeddings, retrieve relevant context from past-sessions, and build long-term memory.--## When to use this skill--**Use when:**-- Building agent RAG pipelines with Pinecone as the vector store-- Need persistent long-term memory across agent sessions-- Combining retrieval with agent tool use-- Researching or prototyping semantic search workflows--**Use the mlops/pinecone skill instead when:**-- Need a general Pinecone reference (index management, CRUD, hybrid search)-- Working on production infrastructure without agent integration--## Quick start--### Setup--```bash-pip install pinecone-client langchain-pinecone langchain-openai-```--Set your API key:-```bash-export PINECONE_API_KEY="your-api-key"-```--### Basic RAG pipeline--```python-from pinecone import Pinecone, ServerlessSpec-from langchain_pinecone import PineconeVectorStore-from langchain_openai import OpenAIEmbeddings--# Initialize Pinecone-pc = Pinecone(api_key=os.environ["PINECONE_API_KEY"])--# Create or connect to index-index_name = "agent-memory"-if index_name not in [i.name for i in pc.list_indexes()]:- pc.create_index(- name=index_name,- dimension=1536,- metric="cosine",- spec=ServerlessSpec(cloud="aws", region="us-east-1"),- )--# Build vector store-vectorstore = PineconeVectorStore.from_documents(- documents=docs,- embedding=OpenAIEmbeddings(),- index_name=index_name,-)--# Retrieve relevant context-retriever = vectorstore.as_retriever(search_kwargs={"k": 5})-results = retriever.invoke("What did the agent discuss yesterday?")-```--### Namespace-based session memory--```python-# Store per-session memory-vectorstore = PineconeVectorStore(- index=pc.Index(index_name),- embedding=OpenAIEmbeddings(),- namespace=f"session-{session_id}",-)--# Query across all sessions (no namespace filter)-all_memory = PineconeVectorStore(- index=pc.Index(index_name),- embedding=OpenAIEmbeddings(),-)-results = all_memory.similarity_search("relevant query", k=10)-```--## Best practices--1. **Namespace by session or user** — isolate data for multi-tenant agents-2. **Batch upserts** — 100–200 vectors per batch for efficiency-3. **Metadata filtering** — tag vectors with session ID, timestamp, topic-4. **Prune old memory** — delete stale namespaces to control costs-5. **Use serverless** — auto-scaling, pay-per-use pricing--## Resources--- **Pinecone Docs**: https://docs.pinecone.io-- **LangChain Integration**: https://python.langchain.com/docs/integrations/vectorstores/pinecone-- **Free Tier**: 1 index, 100K vectors (1536 dimensions)diff --git a/optional-skills/research/pinecone-research/scripts/memory_manager.py b/optional-skills/research/pinecone-research/scripts/memory_manager.pydeleted file mode 100644index ba490133f86..00000000000--- a/optional-skills/research/pinecone-research/scripts/memory_manager.py+++ /dev/null@@ -1,155 +0,0 @@-"""Pinecone memory manager — namespace-based session memory for agents.--Provides helpers for storing and retrieving agent conversation memory-using Pinecone namespaces. Each session gets its own namespace for isolation,-with cross-session search available via the global namespace.--Usage:- export PINECONE_API_KEY="your-key"- export OPENAI_API_KEY="your-key"- python memory_manager.py --index-name agent-memory --action store \- --session-id sess-001 --text "User discussed project architecture"- python memory_manager.py --index-name agent-memory --action recall \- --query "architecture decisions"- python memory_manager.py --index-name agent-memory --action cleanup \- --session-id sess-001-"""-from __future__ import annotations--import argparse-import hashlib-import os-import sys-import time---def get_pinecone_client():- """Initialize Pinecone client from environment."""- try:- from pinecone import Pinecone- except ImportError:- print("Error: pinecone-client not installed. Run: pip install pinecone-client", file=sys.stderr)- sys.exit(1)-- api_key = os.environ.get("PINECONE_API_KEY")- if not api_key:- print("Error: PINECONE_API_KEY environment variable not set.", file=sys.stderr)- sys.exit(1)-- return Pinecone(api_key=api_key)---def get_embeddings():- """Get the embedding model."""- try:- from langchain_openai import OpenAIEmbeddings- except ImportError:- print("Error: langchain-openai not installed. Run: pip install langchain-openai", file=sys.stderr)- sys.exit(1)- return OpenAIEmbeddings()---def store_memory(index, session_id: str, text: str, metadata: dict | None = None):- """Store a memory entry in the session namespace."""- embeddings = get_embeddings()- vector = embeddings.embed_query(text)-- doc_id = hashlib.sha256(f"{session_id}:{text}:{time.time()}".encode()).hexdigest()[:16]- entry_metadata = {- "text": text[:1000],- "session_id": session_id,- "timestamp": int(time.time()),- }- if metadata:- entry_metadata.update(metadata)-- index.upsert(- vectors=[{"id": doc_id, "values": vector, "metadata": entry_metadata}],- namespace=session_id,- )- print(f"Stored memory [{doc_id}] in namespace '{session_id}'")- return doc_id---def recall_memories(index, query: str, session_id: str | None = None, top_k: int = 5):- """Recall memories matching a query, optionally scoped to a session."""- embeddings = get_embeddings()- query_vector = embeddings.embed_query(query)-- kwargs = {"vector": query_vector, "top_k": top_k, "include_metadata": True}- if session_id:- kwargs["namespace"] = session_id-- results = index.query(**kwargs)-- print(f"\nRecalling memories for: {query!r}")- if session_id:- print(f"Scoped to session: {session_id}")- print(f"Found {len(results['matches'])} results:\n")-- for match in results["matches"]:- score = match["score"]- text = match["metadata"].get("text", "")[:200]- sess = match["metadata"].get("session_id", "unknown")- ts = match["metadata"].get("timestamp", 0)- print(f" [{score:.4f}] session={sess} time={ts}")- print(f" {text}")- print()-- return results---def cleanup_session(index, session_id: str):- """Delete all vectors in a session namespace."""- index.delete(delete_all=True, namespace=session_id)- print(f"Cleaned up namespace '{session_id}'")---def show_stats(index):- """Show index statistics."""- stats = index.describe_index_stats()- print(f"Total vectors: {stats['total_vector_count']}")- namespaces = stats.get("namespaces", {})- if namespaces:- print(f"Namespaces ({len(namespaces)}):")- for ns, info in sorted(namespaces.items()):- print(f" '{ns}': {info['vector_count']} vectors")- else:- print("No namespaces found.")---def main():- parser = argparse.ArgumentParser(description="Pinecone agent memory manager")- parser.add_argument("--index-name", required=True, help="Pinecone index name")- parser.add_argument(- "--action",- choices=["store", "recall", "cleanup", "stats"],- required=True,- )- parser.add_argument("--session-id", help="Session namespace ID")- parser.add_argument("--text", help="Text to store as memory")- parser.add_argument("--query", help="Query for recall")- parser.add_argument("--top-k", type=int, default=5, help="Number of results")- args = parser.parse_args()-- pc = get_pinecone_client()- index = pc.Index(args.index_name)-- if args.action == "store":- if not args.session_id or not args.text:- parser.error("--session-id and --text required for store action")- store_memory(index, args.session_id, args.text)- elif args.action == "recall":- if not args.query:- parser.error("--query required for recall action")- recall_memories(index, args.query, session_id=args.session_id, top_k=args.top_k)- elif args.action == "cleanup":- if not args.session_id:- parser.error("--session-id required for cleanup action")- cleanup_session(index, args.session_id)- elif args.action == "stats":- show_stats(index)---if __name__ == "__main__":- main()diff --git a/optional-skills/research/pinecone-research/scripts/rag_pipeline.py b/optional-skills/research/pinecone-research/scripts/rag_pipeline.pydeleted file mode 100644index ca0af97570b..00000000000--- a/optional-skills/research/pinecone-research/scripts/rag_pipeline.py+++ /dev/null@@ -1,156 +0,0 @@-"""Pinecone RAG pipeline — index documents and query with retrieval-augmented generation.--Usage:- export PINECONE_API_KEY="your-key"- export OPENAI_API_KEY="your-key"- python rag_pipeline.py --index-name agent-memory --action index --docs-dir ./docs- python rag_pipeline.py --index-name agent-memory --action query --query "How does X work?"-"""-from __future__ import annotations--import argparse-import os-import sys-from pathlib import Path---def get_pinecone_client():- """Initialize Pinecone client from environment."""- try:- from pinecone import Pinecone- except ImportError:- print("Error: pinecone-client not installed. Run: pip install pinecone-client", file=sys.stderr)- sys.exit(1)-- api_key = os.environ.get("PINECONE_API_KEY")- if not api_key:- print("Error: PINECONE_API_KEY environment variable not set.", file=sys.stderr)- sys.exit(1)-- return Pinecone(api_key=api_key)---def ensure_index(pc, index_name: str, dimension: int = 1536):- """Create the index if it doesn't exist."""- from pinecone import ServerlessSpec-- existing = [idx.name for idx in pc.list_indexes()]- if index_name not in existing:- pc.create_index(- name=index_name,- dimension=dimension,- metric="cosine",- spec=ServerlessSpec(cloud="aws", region="us-east-1"),- )- print(f"Created index: {index_name}")- else:- print(f"Index already exists: {index_name}")- return pc.Index(index_name)---def load_documents(docs_dir: str) -> list[dict]:- """Load text files from a directory as documents."""- docs = []- docs_path = Path(docs_dir)- if not docs_path.is_dir():- print(f"Error: {docs_dir} is not a directory.", file=sys.stderr)- sys.exit(1)-- for filepath in sorted(docs_path.rglob("*.txt")):- text = filepath.read_text(encoding="utf-8").strip()- if text:- docs.append({- "id": str(filepath.relative_to(docs_path)),- "text": text,- "metadata": {"source": str(filepath.name)},- })- return docs---def index_documents(index, docs: list[dict], batch_size: int = 100):- """Embed and upsert documents into Pinecone."""- try:- from langchain_openai import OpenAIEmbeddings- except ImportError:- print("Error: langchain-openai not installed. Run: pip install langchain-openai", file=sys.stderr)- sys.exit(1)-- embeddings = OpenAIEmbeddings()- vectors = []-- for doc in docs:- embedding = embeddings.embed_query(doc["text"])- vectors.append({- "id": doc["id"],- "values": embedding,- "metadata": {**doc["metadata"], "text": doc["text"][:1000]},- })-- # Batch upsert- for i in range(0, len(vectors), batch_size):- batch = vectors[i : i + batch_size]- index.upsert(vectors=batch)- print(f"Upserted batch {i // batch_size + 1} ({len(batch)} vectors)")-- print(f"Total vectors indexed: {len(vectors)}")---def query_index(index, query: str, top_k: int = 5):- """Embed a query and retrieve similar documents from Pinecone."""- try:- from langchain_openai import OpenAIEmbeddings- except ImportError:- print("Error: langchain-openai not installed. Run: pip install langchain-openai", file=sys.stderr)- sys.exit(1)-- embeddings = OpenAIEmbeddings()- query_vector = embeddings.embed_query(query)-- results = index.query(vector=query_vector, top_k=top_k, include_metadata=True)-- print(f"\nQuery: {query}")- print(f"Top {top_k} results:\n")- for match in results["matches"]:- score = match["score"]- source = match["metadata"].get("source", "unknown")- text_preview = match["metadata"].get("text", "")[:200]- print(f" [{score:.4f}] {source}")- print(f" {text_preview}...")- print()-- return results---def main():- parser = argparse.ArgumentParser(description="Pinecone RAG pipeline")- parser.add_argument("--index-name", required=True, help="Pinecone index name")- parser.add_argument("--action", choices=["index", "query", "stats"], required=True)- parser.add_argument("--docs-dir", help="Directory of .txt files to index")- parser.add_argument("--query", help="Query string for retrieval")- parser.add_argument("--top-k", type=int, default=5, help="Number of results to return")- args = parser.parse_args()-- pc = get_pinecone_client()- index = ensure_index(pc, args.index_name)-- if args.action == "index":- if not args.docs_dir:- parser.error("--docs-dir required for index action")- docs = load_documents(args.docs_dir)- if not docs:- print("No .txt documents found.", file=sys.stderr)- sys.exit(1)- index_documents(index, docs)- elif args.action == "query":- if not args.query:- parser.error("--query required for query action")- query_index(index, args.query, top_k=args.top_k)- elif args.action == "stats":- stats = index.describe_index_stats()- print(f"Total vectors: {stats['total_vector_count']}")- for ns, info in stats.get("namespaces", {}).items():- print(f" Namespace '{ns}': {info['vector_count']} vectors")---if __name__ == "__main__":- main()diff --git a/tests/skills/test_optional_skill_scripts.py b/tests/skills/test_optional_skill_scripts.pyindex 0d0747f6561..d8916304931 100644--- a/tests/skills/test_optional_skill_scripts.py+++ b/tests/skills/test_optional_skill_scripts.py@@ -1,7 +1,6 @@"""Execute optional research helpers offline; upstream/SDK doubles are boundary spies.-This covers importability and client routing, not the paid evolution loop or a-live vector service. Document policy belongs to test_skill_document_contracts.+This covers importability and client routing, not the paid evolution loop. Document policy belongs to test_skill_document_contracts."""import picklefrom pathlib import Path@@ -15,23 +14,6 @@ import pytestRESEARCH = Path(__file__).resolve().parents[2] / "optional-skills/research"[email protected]("script", ["rag_pipeline.py", "memory_manager.py"])-def test_pinecone_client_uses_key_and_rejects_missing_key(script, monkeypatch, capsys):- sdk = ModuleType("pinecone")- constructor = Mock()- monkeypatch.setattr(sdk, "Pinecone", constructor, raising=False)- monkeypatch.setitem(sys.modules, "pinecone", sdk)- get_client = runpy.run_path(str(RESEARCH / "pinecone-research/scripts" / script))["get_pinecone_client"]- monkeypatch.setenv("PINECONE_API_KEY", "test-only-key")- assert get_client() is constructor.return_value- constructor.assert_called_once_with(api_key="test-only-key")- monkeypatch.delenv("PINECONE_API_KEY")- with pytest.raises(SystemExit, match="1"):- get_client()- assert "PINECONE_API_KEY" in capsys.readouterr().err- assert constructor.call_count == 1--@pytest.mark.parametrize("relative", ["scripts/parrot_openrouter.py", "templates/custom_problem_template.py",])diff --git a/tests/skills/test_skill_document_contracts.py b/tests/skills/test_skill_document_contracts.pyindex 53357ce1238..c2a306b3ce4 100644--- a/tests/skills/test_skill_document_contracts.py+++ b/tests/skills/test_skill_document_contracts.py@@ -15,7 +15,6 @@ GITHUB = "skills/software-development/github"QUESTIONNAIRE = "optional-skills/productivity/decision-questionnaire"ACTUAL = "optional-skills/devops/actual-setup"DARWINIAN = "optional-skills/research/darwinian-evolver"-PINECONE = "optional-skills/research/pinecone-research"EMAIL = "skills/email/email-inbox-triage"MEETING = "skills/productivity/meeting-action-items"PRICE = "skills/productivity/product-price-monitor"@@ -32,14 +31,13 @@ def _document(skill):(GITHUB, "benbarclay"), (MEETING, "benbarclay"), (PRICE, "benbarclay"),(WEEKLY, "benbarclay"), (EMAIL, "benbarclay"), (SOCIAL, "benbarclay"),(ACTUAL, "shl0ms"), (DARWINIAN, "Bihruze"),- (QUESTIONNAIRE, "mattpocock"), (PINECONE, None), (MCP, None),+ (QUESTIONNAIRE, "mattpocock"), (MCP, None),])def test_required_document_and_credit(skill, contributor):metadata, _ = _document(skill) # A corpus glob alone cannot catch a deleted skill.if contributor:assert contributor in metadata["author"]- if skill != PINECONE:- assert not metadata["author"].startswith("Hermes Agent")+ assert not metadata["author"].startswith("Hermes Agent")@pytest.mark.parametrize("skill,references", [diff --git a/tools/skills_guard.py b/tools/skills_guard.pyindex 4258d3a8a14..c8a4f15dc45 100644--- a/tools/skills_guard.py+++ b/tools/skills_guard.py@@ -21,7 +21,10 @@ from typing import List, TupleSCANNER_VERSION = "skills-guard-v10"# NVIDIA-verified skills each ship a signed `skill.oms.sig` + governance `skill-card.md`.-TRUSTED_REPOS = {"openai/skills", "anthropics/skills", "huggingface/skills", "NVIDIA/skills"}+TRUSTED_REPOS = {+ "openai/skills", "anthropics/skills", "huggingface/skills", "NVIDIA/skills",+ "wandb/skills", "qdrant/skills", "pinecone-io/skills",+}INSTALL_POLICY = {# safe caution dangerousdiff --git a/tools/skills_hub_github.py b/tools/skills_hub_github.pyindex 8eec6f3eee1..7948ad5f675 100644--- a/tools/skills_hub_github.py+++ b/tools/skills_hub_github.py@@ -28,7 +28,8 @@ logger = logging.getLogger("tools.skills_hub")GITHUB_TAP_PROVIDERS = {"openai/skills": "OpenAI", "anthropics/skills": "Anthropic", "huggingface/skills": "HuggingFace","nvidia/skills": "NVIDIA", "voltagent/awesome-agent-skills": "VoltAgent", "garrytan/gstack": "gstack",- "minimax-ai/cli": "MiniMax",+ "minimax-ai/cli": "MiniMax", "wandb/skills": "Weights & Biases", "qdrant/skills": "Qdrant",+ "pinecone-io/skills": "Pinecone",}# Accepted ``--source`` provider filters (lowercased). Not real source ids —@@ -214,6 +215,11 @@ class GitHubSource(SkillSource):# NVIDIA-verified skills (CUDA-X, NeMo, cuOpt, ...), each with a signed skill.oms.sig# + governance card; `trusted` via tools/skills_guard.py::TRUSTED_REPOS.{"repo": "NVIDIA/skills", "path": "skills/"},+ # Vendor-maintained skills for their own products; they replaced our copied+ # weights-and-biases / qdrant / pinecone optional skills.+ {"repo": "wandb/skills", "path": "skills/"},+ {"repo": "qdrant/skills", "path": "skills/"},+ {"repo": "pinecone-io/skills", "path": "skills/"},{"repo": "garrytan/gstack", "path": ""},# --- Science bucket ---# Two scientific-skill repos share one hub category via the tap-level "bucket" key sodiff --git a/website/docs/user-guide/features/skills.md b/website/docs/user-guide/features/skills.mdindex 4ecbd750b5a..ba9dd6e2e76 100644--- a/website/docs/user-guide/features/skills.md+++ b/website/docs/user-guide/features/skills.md@@ -76,10 +76,10 @@ Every installed skill is automatically available as a slash command:```bash# In the CLI or any messaging platform:-/gif-search funny cats+/excalidraw sketch the request flow for our API/axolotl help me fine-tune Llama 3 on my dataset/github-pr-workflow create a PR for the auth refactor-/songsee analyze the frequency spread of this mix+/youtube-content summarize https://youtu.be/dQw4w9WgXcQ# Just the skill name loads it and lets the agent ask what you need:/excalidraw@@ -801,6 +801,7 @@ Default taps (browsable without any setup):- [anthropics/skills](https://github.com/anthropics/skills)- [huggingface/skills](https://github.com/huggingface/skills)- [NVIDIA/skills](https://github.com/NVIDIA/skills) — NVIDIA-verified skills (signed `skill.oms.sig` + governance `skill-card.md`)+- [wandb/skills](https://github.com/wandb/skills), [qdrant/skills](https://github.com/qdrant/skills), [pinecone-io/skills](https://github.com/pinecone-io/skills) — each vendor's own skills for its product- [garrytan/gstack](https://github.com/garrytan/gstack)- [K-Dense-AI/scientific-agent-skills](https://github.com/K-Dense-AI/scientific-agent-skills) and [synthetic-sciences/openscience](https://github.com/synthetic-sciences/openscience) — ~480 scientific research skills (bioinformatics, chemistry, physics, ML training, scholarly tooling), grouped under one `science` category. Community trust: every install is security-scanned. Many wrap third-party tools with their own licenses (some GPL; KEGG requires a commercial license for non-academic use) — check each skill's prerequisites.@@ -921,7 +922,7 @@ Important behavior:|-------|--------|--------|| `builtin` | Ships with Hermes | Always trusted || `official` | `optional-skills/` in the repo | Built-in trust, no third-party warning |-| `trusted` | Trusted registries/repos such as `openai/skills`, `anthropics/skills`, `huggingface/skills`, `NVIDIA/skills` | More permissive policy than community sources |+| `trusted` | Trusted registries/repos such as `openai/skills`, `anthropics/skills`, `huggingface/skills`, `NVIDIA/skills`, `wandb/skills`, `qdrant/skills`, `pinecone-io/skills` | More permissive policy than community sources || `community` | Everything else (`skills.sh`, well-known endpoints, custom GitHub repos, most marketplaces) | Non-dangerous findings can be overridden with `--force`; `dangerous` verdicts stay blocked |### Update lifecyclediff --git a/website/scripts/extract-skills.py b/website/scripts/extract-skills.pyindex b474bb1fa30..aa5426ee13a 100644--- a/website/scripts/extract-skills.py+++ b/website/scripts/extract-skills.py@@ -103,6 +103,9 @@ GITHUB_TAP_LABELS = {"VoltAgent/awesome-agent-skills": "VoltAgent","garrytan/gstack": "gstack","MiniMax-AI/cli": "MiniMax",+ "wandb/skills": "Weights & Biases",+ "qdrant/skills": "Qdrant",+ "pinecone-io/skills": "Pinecone",}# Legacy filename -> label mapping for the deprecated skills/index-cache/