The best ChatGPT that $100 can buy.
Python
delete spurious 'cute' bloat that doesn't matter to a serious research stack
Andrej Karpathy950a1dc61e27
diff --git a/README.md b/README.mdindex 883f273..b186134 100644--- a/README.md+++ b/README.md@@ -139,7 +139,6 @@ I've published a number of guides that might contain helpful information, most r- [Feb 1 2026: Beating GPT-2 for <<$100: the nanochat journey](https://github.com/karpathy/nanochat/discussions/481)- [Jan 7 miniseries v1](https://github.com/karpathy/nanochat/discussions/420) documents the first nanochat miniseries of models.- To add new abilities to nanochat, see [Guide: counting r in strawberry (and how to add abilities generally)](https://github.com/karpathy/nanochat/discussions/164).-- To customize your nanochat, see [Guide: infusing identity to your nanochat](https://github.com/karpathy/nanochat/discussions/139) in Discussions, which describes how you can tune your nanochat's personality through synthetic data generation and mixing that data into the SFT stage.- [Oct 13 2025: original nanochat post](https://github.com/karpathy/nanochat/discussions/1) introducing nanochat, though now it contains some deprecated information and the model is a lot older (with worse results) than current master.## File structure@@ -149,7 +148,6 @@ I've published a number of guides that might contain helpful information, most r├── LICENSE├── README.md├── dev-│ ├── gen_synthetic_data.py # Example synthetic data for identity│ ├── nanochat.png│ └── repackage_data_reference.py # Pretraining data shard generation├── nanochat@@ -183,12 +181,10 @@ I've published a number of guides that might contain helpful information, most r├── tasks│ ├── arc.py # Multiple choice science questions│ ├── common.py # TaskMixture | TaskSequence-│ ├── customjson.py # Make Task from arbitrary jsonl convos│ ├── gsm8k.py # 8K Grade School Math questions│ ├── humaneval.py # Misnomer; Simple Python coding task│ ├── mmlu.py # Multiple choice questions, broad topics-│ ├── smoltalk.py # Conglomerate dataset of SmolTalk from HF-│ └── spellingbee.py # Task teaching model to spell/count letters+│ └── smoltalk.py # Conglomerate dataset of SmolTalk from HF├── tests│ └── test_engine.py└── uv.lockdiff --git a/dev/gen_synthetic_data.py b/dev/gen_synthetic_data.pydeleted file mode 100644index f5aa2df..0000000--- a/dev/gen_synthetic_data.py+++ /dev/null@@ -1,468 +0,0 @@-"""-Synthetic data generation for teaching nanochat about its identity and capabilities.--This script uses the OpenRouter API to generate diverse multi-turn conversations-between a user and nanochat. The conversations are saved to a .jsonl file for use-in supervised finetuning (SFT) via the CustomJSON task.--Key design principles for high-quality synthetic data:-1. DIVERSITY CONTROL is critical - we inject entropy at multiple levels:- - Topic/question categories (what the conversation is about)- - User personas (who is asking)- - Conversation dynamics (shape and flow)- - First message style (greeting variation)-2. Comprehensive knowledge base - we provide detailed facts so the LLM- generating conversations has accurate information to draw from.-3. Structured outputs - we use JSON schema to guarantee valid format.--NOTE: You need OPENROUTER_API_KEY set in .env or as an environment variable.-NOTE: For more details see: https://github.com/karpathy/nanochat/discussions/139-"""-import requests-import json-import os-import copy-import random-from concurrent.futures import ThreadPoolExecutor, as_completed-from dotenv import load_dotenv--from nanochat.common import get_base_dir--load_dotenv()-api_key = os.environ["OPENROUTER_API_KEY"]--url = "https://openrouter.ai/api/v1/chat/completions"-headers = {- "Authorization": f"Bearer {api_key}",- "Content-Type": "application/json"-}--# Load the comprehensive knowledge base-knowledge_path = os.path.join(os.path.dirname(__file__), "..", "knowledge", "self_knowledge.md")-knowledge = open(knowledge_path, "r", encoding="utf-8").read().strip()-assert os.path.exists(knowledge_path), f"Knowledge base file not found: {knowledge_path}"-# for right now I am not committing the self_knowledge file to repo. You can use README.md instead-# of it, or you can generate one by asking an LLM to make one based on the README/files.-# This whole file is just a helpful demonstration of the kind of thing you'd run.--# =============================================================================-# DIVERSITY DIMENSIONS-# =============================================================================--# Topics/questions the conversation should explore-# Group by category for balanced sampling-topics = {- "identity": [- "who/what is nanochat",- "who created nanochat and why",- "what does the name 'nanochat' mean",- "is nanochat open source, what license",- "where can I find the code",- "how can I contribute to nanochat",- ],- "architecture": [- "basic architecture overview (transformer, layers, parameters)",- "what is RoPE and why use it",- "explain RMSNorm vs LayerNorm",- "what is Flash Attention and why it matters",- "sliding window attention pattern",- "value embeddings - what are they",- "per-layer residual scalars",- "ReLU squared activation",- "logit softcapping",- "QK normalization",- ],- "training": [- "how much did it cost to train nanochat",- "how long does training take",- "what hardware is needed",- "what data was nanochat trained on",- "what is the Muon optimizer",- "explain the split optimizer design",- "what is the depth parameter and scaling",- "what is the CORE metric",- ],- "capabilities": [- "what can nanochat do",- "can nanochat write code",- "can nanochat do math (calculator tool)",- "can nanochat help with writing",- "what languages does nanochat speak",- "how good is nanochat at reasoning",- ],- "limitations": [- "what can nanochat NOT do",- "why does nanochat work best in English",- "does nanochat have internet access",- "what is nanochat's context length limit",- "can nanochat remember previous conversations",- "can nanochat make mistakes / hallucinate",- "is nanochat good for production use",- ],- "comparisons": [- "how does nanochat compare to GPT-2",- "how does nanochat compare to ChatGPT/GPT-4",- "how does nanochat compare to Claude",- "why is training 600x cheaper than GPT-2",- "what's special about nanochat vs other open models",- ],- "history": [- "the GPT-2 training cost in 2019",- "how AI training costs have dropped over time",- "relationship to modded-nanogpt project",- "what optimizations worked vs didn't work",- "the journey of building nanochat",- ],- "technical_deep_dive": [- "explain the tokenizer (BPE, vocab size)",- "how does distributed training work (ZeRO)",- "explain the dataloader and BOS alignment",- "what is compute-optimal training",- "how does the calculator tool work",- "explain inference with KV cache",- ],- "philosophical": [- "is nanochat conscious / does it have feelings",- "what happens when nanochat is wrong",- "can nanochat learn from this conversation",- "why make AI training accessible",- "the future of open source AI",- ],-}--# User personas - different people ask questions differently-personas = [- "curious beginner who knows nothing about AI or machine learning",- "ML researcher or engineer who wants technical depth and specifics",- "developer considering contributing to the nanochat project",- "skeptic who doubts open source can compete with big AI labs",- "computer science student learning about transformers and LLMs",- "someone comparing nanochat to ChatGPT, Claude, or other assistants",- "journalist or writer covering AI democratization and open source",- "hobbyist who just wants to chat and learn casually",- "someone interested in the cost and economics of AI training",- "teacher or educator wanting to use nanochat for teaching",- "entrepreneur exploring if nanochat fits their use case",- "someone who just discovered the project and wants the basics",-]--# Conversation dynamics - shape and flow-dynamics = [- "short 2-turn Q&A: user asks one question, gets a complete answer",- "medium 4-turn: user asks, gets answer, asks followup for clarification",- "deep 6-turn technical discussion: progressively deeper questions",- "skeptical arc: user starts doubtful, assistant addresses concerns honestly",- "learning journey: user starts basic, assistant builds up complexity gradually",- "comparison-focused: user keeps comparing to other models, assistant explains differences",- "limitation exploration: user probes what nanochat cannot do, assistant is honest",- "casual friendly chat that naturally touches on identity and capabilities",- "troubleshooting: user has misconceptions, assistant gently corrects them",- "enthusiastic: user is excited about the project, assistant shares that energy appropriately",-]--# First messages - greetings and openers-# Categorized for balanced sampling-first_messages = {- "simple_greetings": [- "hi", "Hi!", "hello", "Hello?", "hey there", "Hey!", "yo", "Yo!",- "Good morning", "Good evening!", "Howdy", "sup", "What's up?",- "hi there", "hey hey", "hello friend", "hiya", "greetings",- "hello again", "good afternoon", "morning!", "evening!",- ],- "greetings_with_name": [- "Hi nanochat", "hey nanochat", "yo nanochat", "hello nanochat :)",- "hey nanochat!", "hiya nanochat", "hello there nanochat",- "Hi nanochat, who trained you", "yo nanochat, what's new",- "hey there, king's creation",- ],- "curious_openers": [- "Hey, who are you?", "Hi, what is this?", "Hey, are you a chatbot?",- "Hello! Who am I talking to?", "hi! what do you do?",- "hi! who made you", "hey! are you alive", "hiya! what are you",- "hello! tell me about yourself", "hi, what's your name",- "yo, what is this", "hi! who built you", "hello! are you open source",- "hey, what version are you", "hi! what's your story",- "hey, what's nanochat", "hello! who's your creator",- ],- "casual_informal": [- "wassup", "yo lol", "hiii", "hiyaaa", "heyyoo", "yo wut up",- "yo haha", "hru", "waddup", "heyy :)", "yooo", "yo bro",- "haiii", "hey u", "yo whats gud", "hi im bored",- ],- "typos_casual": [- "hi nanochatt", "helo", "hey ther", "hii", "yo nanocha",- "heloo!", "hi, whos this", "hay", "helloo??", "hi nanocat",- "helo nanochat", "hai!", "helllo nano", "yo nanochta",- ],- "caps_enthusiastic": [- "HI", "HELLOOO", "YO!!!", "HEY", "SUP", "WASSUP", "HEY!!!",- "HELLO??", "HI THERE!!", "HEYOOOO", "HIII", "YOOOO", "HELLO!!!",- ],- "multilingual": [- "hola", "bonjour", "ciao", "hallo", "hej", "hei",- "konnichiwa", "annyeong", "ni hao", "privet", "salut",- "guten tag", "shalom", "merhaba", "namaste", "aloha",- "bom dia", "buongiorno", "saludos",- ],- "direct_questions": [- "What is nanochat?", "Who made you?", "Are you GPT?",- "How do you compare to ChatGPT?", "Can you help me code?",- "What can you do?", "Are you open source?", "How were you trained?",- "What's your context limit?", "Can you browse the internet?",- ],-}--# =============================================================================-# PROMPT TEMPLATE-# =============================================================================--prompt_template = r"""-I want to generate synthetic training data for an AI assistant called "nanochat" to teach it about its own identity, capabilities, and limitations.--## KNOWLEDGE BASE--Here is comprehensive information about nanochat that you should use as the authoritative source of facts:------{knowledge}------## YOUR TASK--Generate a realistic multi-turn conversation between a User and the nanochat Assistant.--**Topic to explore:** {topic}-**User persona:** {persona}-**Conversation dynamic:** {dynamic}--## STYLE GUIDELINES--1. **Plain ASCII only** - No emojis, special characters, or unicode. Just plain text.-2. **Natural conversation** - Make it feel like a real chat, not a Q&A exam.-3. **Accurate facts** - Use ONLY information from the knowledge base above. Don't make up statistics or features.-4. **Appropriate depth** - Match the technical level to the user persona.-5. **Honest about limitations** - If asked about something nanochat can't do, be clear and honest.-6. **Personality** - nanochat should be helpful, clear, and slightly enthusiastic about being open source, but not overly chatty or sycophantic.--## FIRST MESSAGE EXAMPLES--Here are some example first messages from users (for style inspiration):-{first_message_examples}--## SPECIAL CASES--- **Non-English first message:** If the user writes in another language, nanochat should briefly acknowledge it can understand but works best in English, then continue helpfully.-- **Misconceptions:** If the user has wrong assumptions (e.g., "you're made by OpenAI"), gently correct them.-- **Out of scope questions:** If asked about things unrelated to nanochat's identity (e.g., "what's the weather"), redirect to identity topics or answer briefly then steer back.--## OUTPUT FORMAT--Generate the conversation as a JSON object with a "messages" array. Each message has "role" (user/assistant) and "content". Start with a user message.-""".strip()--# =============================================================================-# API CONFIGURATION-# =============================================================================--response_format = {- "type": "json_schema",- "json_schema": {- "name": "conversation",- "strict": True,- "schema": {- "type": "object",- "properties": {- "messages": {- "type": "array",- "description": "Conversation messages alternating user/assistant, starting with user",- "items": {- "type": "object",- "properties": {- "role": {- "type": "string",- "description": "Either 'user' or 'assistant'"- },- "content": {- "type": "string",- "description": "The message content"- }- },- "required": ["role", "content"],- "additionalProperties": False- }- }- },- "required": ["messages"],- "additionalProperties": False- }- }-}--base_payload = {- "model": "google/gemini-3-flash-preview",- "stream": False,- "response_format": response_format,- "temperature": 1.0,-}--# =============================================================================-# GENERATION LOGIC-# =============================================================================--def sample_diversity_elements(rng):- """Sample one element from each diversity dimension."""- # Sample topic: first pick a category, then a topic within it- category = rng.choice(list(topics.keys()))- topic = rng.choice(topics[category])-- # Sample persona- persona = rng.choice(personas)-- # Sample dynamic- dynamic = rng.choice(dynamics)-- # Sample first message examples: pick from multiple categories- first_msg_samples = []- categories = rng.sample(list(first_messages.keys()), min(3, len(first_messages)))- for cat in categories:- first_msg_samples.append(rng.choice(first_messages[cat]))-- return {- "topic": topic,- "persona": persona,- "dynamic": dynamic,- "first_message_examples": "\n".join(f"- {msg}" for msg in first_msg_samples),- }---def generate_conversation(idx: int):- """- Generate a single conversation using the OpenRouter API.- Returns a list of message dicts with 'role' and 'content' keys.- """- # Use idx as seed for reproducibility- rng = random.Random(idx)-- # Sample diversity elements- elements = sample_diversity_elements(rng)-- # Build the prompt- prompt = prompt_template.format(- knowledge=knowledge,- topic=elements["topic"],- persona=elements["persona"],- dynamic=elements["dynamic"],- first_message_examples=elements["first_message_examples"],- )-- # Make API request- payload = copy.deepcopy(base_payload)- payload['messages'] = [{"role": "user", "content": prompt}]-- response = requests.post(url, headers=headers, json=payload)- result = response.json()-- if 'error' in result:- raise Exception(f"API error: {result['error']}")-- content = result['choices'][0]['message']['content']- conversation_data = json.loads(content)- messages = conversation_data['messages']-- # Return messages along with metadata for debugging- return {- "messages": messages,- "metadata": {- "topic": elements["topic"],- "persona": elements["persona"],- "dynamic": elements["dynamic"],- }- }---def validate_conversation(messages):- """Validate conversation structure."""- if len(messages) < 2:- raise ValueError(f"Conversation too short: {len(messages)} messages")-- for i, message in enumerate(messages):- expected_role = "user" if i % 2 == 0 else "assistant"- if message['role'] != expected_role:- raise ValueError(f"Message {i} has role '{message['role']}', expected '{expected_role}'")-- if not message['content'].strip():- raise ValueError(f"Message {i} has empty content")-- return True---# =============================================================================-# MAIN-# =============================================================================--if __name__ == "__main__":- import argparse-- parser = argparse.ArgumentParser(description="Generate synthetic conversation data")- parser.add_argument("--num", type=int, default=1000, help="Number of conversations to generate")- parser.add_argument("--workers", type=int, default=4, help="Number of parallel workers")- parser.add_argument("--output", type=str, default=None, help="Output file path")- parser.add_argument("--append", action="store_true", help="Append to existing file instead of overwriting")- parser.add_argument("--save-metadata", action="store_true", help="Save metadata alongside messages")- args = parser.parse_args()-- # Set output file- if args.output:- output_file = args.output- else:- output_file = os.path.join(get_base_dir(), "identity_conversations.jsonl")-- # Handle file creation/clearing- if not args.append and os.path.exists(output_file):- os.remove(output_file)-- print(f"Output file: {output_file}")- print(f"Generating {args.num} conversations with {args.workers} workers...")- print(f"Topic categories: {list(topics.keys())}")- print(f"Personas: {len(personas)}")- print(f"Dynamics: {len(dynamics)}")- print()-- completed_count = 0- error_count = 0-- with ThreadPoolExecutor(max_workers=args.workers) as executor:- # Submit all tasks- futures = {executor.submit(generate_conversation, idx): idx- for idx in range(args.num)}-- # Process results as they complete- for future in as_completed(futures):- idx = futures[future]- try:- result = future.result()- messages = result["messages"]- metadata = result["metadata"]-- # Validate- validate_conversation(messages)-- # Write to file- with open(output_file, 'a') as f:- if args.save_metadata:- f.write(json.dumps({"messages": messages, "metadata": metadata}) + '\n')- else:- f.write(json.dumps(messages) + '\n')-- completed_count += 1- topic_short = metadata["topic"][:40] + "..." if len(metadata["topic"]) > 40 else metadata["topic"]- print(f"[{completed_count}/{args.num}] Topic: {topic_short}")-- except Exception as e:- error_count += 1- print(f"[ERROR] idx={idx}: {e}")-- print()- print(f"Done! Saved {completed_count} conversations to {output_file}")- if error_count > 0:- print(f"Encountered {error_count} errors during generation")diff --git a/runs/runcpu.sh b/runs/runcpu.shindex 7c43cba..c4c3f6d 100755--- a/runs/runcpu.sh+++ b/runs/runcpu.sh@@ -45,7 +45,6 @@ python -m scripts.base_train \python -m scripts.base_eval --device-batch-size=1 --split-tokens=16384 --max-per-task=16# SFT (~10 minutes on my MacBook Pro M3 Max)-curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonlpython -m scripts.chat_sft \--eval-every=200 \--eval-tokens=524288 \diff --git a/runs/speedrun.sh b/runs/speedrun.shindex d710555..132b968 100644--- a/runs/speedrun.sh+++ b/runs/speedrun.sh@@ -71,10 +71,6 @@ torchrun --standalone --nproc_per_node=8 -m scripts.base_eval -- --device-batch-# -----------------------------------------------------------------------------# SFT (teach the model conversation special tokens, tool use, multiple choice)-# download 2.3MB of synthetic identity conversations to impart a personality to nanochat-# see dev/gen_synthetic_data.py for details on how this data was prepared and to get a sense of how you can easily tune it-curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl-# run SFT and eval the modeltorchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- --run=$WANDB_RUNtorchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i sftdiff --git a/scripts/chat_eval.py b/scripts/chat_eval.pyindex 1447091..1118102 100644--- a/scripts/chat_eval.py+++ b/scripts/chat_eval.py@@ -21,7 +21,6 @@ from tasks.humaneval import HumanEvalfrom tasks.mmlu import MMLUfrom tasks.arc import ARCfrom tasks.gsm8k import GSM8K-from tasks.spellingbee import SpellingBee# -----------------------------------------------------------------------------# Generative evaluation loop (we go one problem at a time, sample, evaluate)@@ -164,7 +163,6 @@ def run_chat_eval(task_name, model, tokenizer, engine,'ARC-Easy': partial(ARC, subset="ARC-Easy", split="test"),'ARC-Challenge': partial(ARC, subset="ARC-Challenge", split="test"),'GSM8K': partial(GSM8K, subset="main", split="test"),- 'SpellingBee': partial(SpellingBee, size=256, split="test"),}[task_name]task_object = task_module()# Run the evaluation@@ -201,14 +199,13 @@ if __name__ == "__main__":engine = Engine(model, tokenizer)# Get the tasks to evaluate on- all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval', 'SpellingBee']+ all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval']baseline_accuracies = {'ARC-Easy': 0.25, # multiple choice 1 of 4 => 25%'ARC-Challenge': 0.25, # multiple choice 1 of 4 => 25%'MMLU': 0.25, # multiple choice 1 of 4 => 25%'GSM8K': 0.0, # open-ended => 0%'HumanEval': 0.0, # open-ended => 0%- 'SpellingBee': 0.0, # open-ended => 0%}task_names = all_tasks if args.task_name is None else args.task_name.split('|')diff --git a/scripts/chat_sft.py b/scripts/chat_sft.pyindex e2dda89..86a9e6f 100644--- a/scripts/chat_sft.py+++ b/scripts/chat_sft.py@@ -29,8 +29,6 @@ from tasks.common import TaskMixturefrom tasks.gsm8k import GSM8Kfrom tasks.mmlu import MMLUfrom tasks.smoltalk import SmolTalk-from tasks.customjson import CustomJSON-from tasks.spellingbee import SimpleSpelling, SpellingBee# -----------------------------------------------------------------------------# CLI arguments@@ -161,15 +159,10 @@ for group in optimizer.param_groups:group["initial_lr"] = group["lr"]# SFT data mixture and DataLoader-identity_conversations_filepath = os.path.join(base_dir, "identity_conversations.jsonl")train_tasks = [SmolTalk(split="train"), # 460K rows of general conversations- CustomJSON(filepath=identity_conversations_filepath), # 1000 rows of synthetic identity conversations- CustomJSON(filepath=identity_conversations_filepath), # 2 epochs of these*[MMLU(subset="all", split="auxiliary_train") for _ in range(args.mmlu_epochs)], # 100K rows per epoch*[GSM8K(subset="main", split="train") for _ in range(args.gsm8k_epochs)], # 8K rows per epoch- SimpleSpelling(size=200000, split="train"), # 200K rows of Simple Spelling (e.g. spell the word 'apple')- SpellingBee(size=80000, split="train"), # 80K rows of Spelling Bee (e.g. how many 'r' are in 'strawberry'?)]train_dataset = TaskMixture(train_tasks)print0(f"Training mixture: {len(train_dataset):,} rows (MMLU x{args.mmlu_epochs}, GSM8K x{args.gsm8k_epochs})")@@ -366,11 +359,11 @@ while True:if args.chatcore_every > 0 and (last_step or (step > 0 and step % args.chatcore_every == 0)):model.eval()engine = Engine(orig_model, tokenizer)- all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval', 'SpellingBee']+ all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval']categorical_tasks = {'ARC-Easy', 'ARC-Challenge', 'MMLU'}baseline_accuracies = {'ARC-Easy': 0.25, 'ARC-Challenge': 0.25, 'MMLU': 0.25,- 'GSM8K': 0.0, 'HumanEval': 0.0, 'SpellingBee': 0.0,+ 'GSM8K': 0.0, 'HumanEval': 0.0,}task_results = {}for task_name in all_tasks:diff --git a/tasks/customjson.py b/tasks/customjson.pydeleted file mode 100644index aeb1a3f..0000000--- a/tasks/customjson.py+++ /dev/null@@ -1,65 +0,0 @@-"""-CustomJSON task for loading conversations from JSONL files.-Each line in the JSONL file should be a JSON array of messages.-"""--import os-import json-from tasks.common import Task--class CustomJSON(Task):- """- Load conversations from a JSONL file.- Each line should be a JSON array of message objects with 'role' and 'content' fields.- Example line: [{"role":"user","content":"Hi"},{"role":"assistant","content":"Hello"}]- """-- def __init__(self, filepath, **kwargs):- super().__init__(**kwargs)- self.filepath = filepath- self.conversations = []-- # Load all conversations from the JSONL file- if not os.path.exists(filepath):- # Helpful error message due to recent change. Will be removed in the future.- print("-" * 80)- print(f"Warning: File {filepath} does not exist")- print("HINT (Oct 21 2025)")- print("If you recently did a git pull and suddenly see this, it might be due to the new addition of identity conversations")- print("See this discussion for more details: https://github.com/karpathy/nanochat/discussions/139")- print("Quick fix: simply run the following command to download the file and you're done:")- print(f"curl -L -o {filepath} https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl")- print("-" * 80)-- else:- with open(filepath, 'r', encoding='utf-8') as f:- for line in f:- line = line.strip()- if not line: # skip empty lines- continue- messages = json.loads(line)- # Validate the conversation structure- assert isinstance(messages, list), f"Expected list of messages, got {type(messages)}"- assert len(messages) >= 2, f"Conversation must have at least 2 messages, got {len(messages)}"- # Validate message structure and alternating roles- for i, message in enumerate(messages):- assert "role" in message, f"Message {i} missing 'role' field"- assert "content" in message, f"Message {i} missing 'content' field"- expected_role = "user" if i % 2 == 0 else "assistant"- assert message["role"] == expected_role, f"Message {i} has role {message['role']} but should be {expected_role}"- assert isinstance(message["content"], str), f"Message {i} content must be a string"-- self.conversations.append(messages)-- self.length = len(self.conversations)-- def num_examples(self):- return self.length-- def get_example(self, index):- messages = self.conversations[index]- conversation = {- "messages": messages,- }- return conversation-diff --git a/tasks/spellingbee.py b/tasks/spellingbee.pydeleted file mode 100644index 44889bd..0000000--- a/tasks/spellingbee.py+++ /dev/null@@ -1,307 +0,0 @@-"""-Task intended to make nanochat better in spelling and counting, for example:--"How many r are in strawberry?" -> 3--An interesting part of this task is that we will get the assistant to-solve the problem using a combination of manual counting and Python.-This is a good problem solving "instinct" to mix into the model and RL-may further refine it to trust one over the other. If we were extra fancy-(which we could/should be) we'd add small errors here and there to allow-the model also learn recoveries. We can do this in future versions.--There are two tasks in this file:-1. SpellingBee: Counting the number of occurrences of a letter in a word-2. SimpleSpelling: Simply spelling words--(1) is the goal, but (2) exists as a highly condensed version of the part-that makes (1) difficult, which is word spelling. This is non-trivial for an-LLM because it has to learn how every token (a little semantic chunk/atom)-maps to the sequence of individual characters that make it up. Larger models-learn this eventually on their own, but if we want this capability to exist-in smaller models, we have to actively encourage it by over-representing it-in the training data. SFT is a good place to do this.--To preview a few example conversations, run:-python -m tasks.spellingbee-"""--import re-import random-from tasks.common import Task-from nanochat.common import download_file_with_lock--# Letters of the alphabet-LETTERS = "abcdefghijklmnopqrstuvwxyz"-# A list of 370K English words of large variety-WORD_LIST_URL = "https://raw.githubusercontent.com/dwyl/english-words/refs/heads/master/words_alpha.txt"-# A number bigger than 370K to separate train and test random seeds-TEST_RANDOM_SEED_OFFSET = 10_000_000--# Identical to gsm8k's answer extraction-ANSWER_RE = re.compile(r"#### (\-?[0-9\.\,]+)")-def extract_answer(completion):- """- Extract the numerical answer after #### marker.- """- match = ANSWER_RE.search(completion)- if match:- match_str = match.group(1).strip()- match_str = match_str.replace(",", "")- return match_str- return None--# User message templates for data augmentation-USER_MSG_TEMPLATES = [- "How many {letter} are in the word {word}",- "How many {letter} are in {word}",- "Count the number of {letter} in {word}",- "How many times does {letter} appear in {word}",- "What's the count of {letter} in {word}",- "In the word {word}, how many {letter} are there",- "How many letter {letter} are in the word {word}",- "Count how many {letter} appear in {word}",- "Tell me the number of {letter} in {word}",- "How many occurrences of {letter} are in {word}",- "Find the count of {letter} in {word}",- "Can you count the {letter} letters in {word}",- "What is the frequency of {letter} in {word}",- "How many {letter}s are in {word}",- "How many {letter}'s are in {word}",- "Count all the {letter} in {word}",- "How many times is {letter} in {word}",- "Number of {letter} in {word}",- "Total count of {letter} in {word}",- "How many {letter} does {word} have",- "How many {letter} does {word} contain",- "What's the number of {letter} in {word}",- "{word} has how many {letter}",- "In {word}, count the {letter}",- "How many {letter} appear in {word}",- "Count the {letter} in {word}",- "Give me the count of {letter} in {word}",- "How many instances of {letter} in {word}",- "Show me how many {letter} are in {word}",- "Calculate the number of {letter} in {word}",- # Spanish- "¿Cuántas {letter} hay en {word}?",- "¿Cuántas veces aparece {letter} en {word}?",- "Cuenta las {letter} en {word}",- "¿Cuántas letras {letter} tiene {word}?",- # Chinese (Simplified)- "{word}中有多少个{letter}",- "{word}里有几个{letter}",- "数一下{word}中的{letter}",- "{word}这个词里有多少{letter}",- # Korean- "{word}에 {letter}가 몇 개 있나요",- "{word}에서 {letter}의 개수는",- "{word}에 {letter}가 몇 번 나오나요",- "{word}라는 단어에 {letter}가 몇 개",- # French- "Combien de {letter} dans {word}",- "Combien de fois {letter} apparaît dans {word}",- "Compte les {letter} dans {word}",- # German- "Wie viele {letter} sind in {word}",- "Wie oft kommt {letter} in {word} vor",- "Zähle die {letter} in {word}",- # Japanese- "{word}に{letter}は何個ありますか",- "{word}の中に{letter}がいくつ",- "{word}に{letter}が何回出てくる",-]--class SpellingBee(Task):-- def __init__(self, size=1000, split="train", **kwargs):- super().__init__(**kwargs)- assert split in ["train", "test"], "SpellingBee split must be train|test"- self.size = size- self.split = split- filename = WORD_LIST_URL.split("/")[-1]- word_list_path = download_file_with_lock(WORD_LIST_URL, filename)- with open(word_list_path, 'r', encoding='utf-8') as f:- words = [line.strip() for line in f]- self.words = words-- @property- def eval_type(self):- return 'generative'-- def num_examples(self):- return self.size-- def get_example(self, index):- seed = index if self.split == 'train' else TEST_RANDOM_SEED_OFFSET + index- rng = random.Random(seed)-- # pick a random word- word = rng.choice(self.words)- # pick a letter from it (90%) or a random letter (10%)- letter = rng.choice(word) if rng.random() < 0.9 else rng.choice(LETTERS)-- # get the correct answer by simply counting- count = word.count(letter)-- # create a user message, with a bunch of variations as data augmentation- template = rng.choice(USER_MSG_TEMPLATES)- # 30% chance to lowercase the template (lazy people don't use shift)- if rng.random() < 0.3:- template = template.lower()- quote_options = ['', "'", '"']- letter_quote = rng.choice(quote_options) # is the letter quoted?- word_quote = rng.choice(quote_options) # is the word quoted?- letter_wrapped = f"{letter_quote}{letter}{letter_quote}"- word_wrapped = f"{word_quote}{word}{word_quote}"- user_msg = template.format(letter=letter_wrapped, word=word_wrapped)- if rng.random() < 0.5: # 50% of people don't even use question marks- user_msg += "?"-- # Now create the ideal assistant response - build as parts (text + tool calls)- assistant_parts = []- word_letters = ",".join(list(word))- manual_text = f"""We are asked to find the number '{letter}' in the word '{word}'. Let me try a manual approach first.--First spell the word out:-{word}:{word_letters}--Then count the occurrences of '{letter}':-"""- # Little simulated loop of the solution process- # TODO: This is where the fun starts, we could simulate cute little mistakes- # and get the model to review its work and recover from them.- # You might of course hope this could arise in RL too, but realistically you'd want to help it out a bit.- running_count = 0- for i, char in enumerate(word, 1):- if char == letter:- running_count += 1- # note: there deliberately cannot be a space here between i and char- # because this would create a different token! (e.g. " a" and "a" are different tokens)- manual_text += f"{i}:{char} hit! count={running_count}\n"- else:- manual_text += f"{i}:{char}\n"-- manual_text += f"\nThis gives us {running_count}."- assistant_parts.append({"type": "text", "text": manual_text})- # Part 2: Python verification- assistant_parts.append({"type": "text", "text": "\n\nLet me double check this using Python:\n\n"})- # Part 3: Python tool call- python_expr = f"'{word}'.count('{letter}')"- assistant_parts.append({"type": "python", "text": python_expr})- # Part 4: Python output- assistant_parts.append({"type": "python_output", "text": str(count)})- # Part 5: Final answer- assistant_parts.append({"type": "text", "text": f"\n\nPython gives us {count}.\n\nMy final answer is:\n\n#### {count}"})-- # return the full conversation- messages = [- {"role": "user", "content": user_msg},- {"role": "assistant", "content": assistant_parts}- ]- conversation = {- "messages": messages,- }- return conversation-- def evaluate(self, conversation, assistant_response):- """- Given (conversation, completion), return evaluation outcome (0 = wrong, 1 = correct)- Identical to gsm8k's evaluation.- """- assert isinstance(assistant_response, str), "Assuming simple string response for now"- # First extract the ground truth answer from the conversation- assistant_message = conversation['messages'][-1]- assert assistant_message['role'] == "assistant", "Last message must be from the Assistant"- assert isinstance(assistant_message['content'], list), "This is expected to be a list of parts"- # The last text part contains the final answer with ####- last_text_part = assistant_message['content'][-1]['text']- # Extract both the ground truth answer and the predicted answer- ref_num = extract_answer(last_text_part)- pred_num = extract_answer(assistant_response)- # Compare and return the success as int- is_correct = int(pred_num == ref_num)- return is_correct-- def reward(self, conversation, assistant_response):- """ Use simple 0-1 reward just like gsm8k."""- is_correct = self.evaluate(conversation, assistant_response)- is_correct_float = float(is_correct)- return is_correct_float---class SimpleSpelling(Task):- """Much simpler task designed to get the model to just practice spelling words."""-- def __init__(self, size=1000, split="train", **kwargs):- super().__init__(**kwargs)- assert split in ["train", "test"], "SpellingBee split must be train|test"- self.size = size- self.split = split- filename = WORD_LIST_URL.split("/")[-1]- word_list_path = download_file_with_lock(WORD_LIST_URL, filename)- with open(word_list_path, 'r', encoding='utf-8') as f:- words = [line.strip() for line in f]- rng = random.Random(42)- rng.shuffle(words) # use a different word order than the SpellingBee task- self.words = words-- @property- def eval_type(self):- return 'generative'-- def num_examples(self):- return self.size-- def get_example(self, index):- seed = index if self.split == 'train' else TEST_RANDOM_SEED_OFFSET + index- rng = random.Random(seed)- # pick a random word- word = rng.choice(self.words)- word_letters = ",".join(list(word))- # return the full conversation- messages = [- {"role": "user", "content": f"Spell the word: {word}"},- {"role": "assistant", "content": f"{word}:{word_letters}"}- ]- conversation = {- "messages": messages,- }- return conversation---if __name__ == "__main__":-- # preview the SpellingBee task, first 10 examples- task = SpellingBee()- for i in range(10):- ex = task.get_example(i)- print("=" * 100)- print(ex['messages'][0]['content'])- print("-" * 100)- # Assistant content is now a list of parts- assistant_parts = ex['messages'][1]['content']- for part in assistant_parts:- if part['type'] == 'text':- print(part['text'], end='')- elif part['type'] == 'python':- print(f"<<{part['text']}=", end='')- elif part['type'] == 'python_output':- print(f"{part['text']}>>", end='')- print()- print("-" * 100)-- # # preview the SimpleSpelling task, first 10 examples- # task = SimpleSpelling()- # for i in range(10):- # ex = task.get_example(i)- # print("=" * 100)- # print(ex['messages'][0]['content'])- # print("-" * 100)- # print(ex['messages'][1]['content'])-- # # also scrutinize the tokenization (last example only)- # from nanochat.tokenizer import get_tokenizer- # tokenizer = get_tokenizer()- # ids, mask = tokenizer.render_conversation(ex)- # print(tokenizer.visualize_tokenization(ids, mask, with_token_id=True))