Skip to content

The best ChatGPT that $100 can buy.

Python

delete spurious 'cute' bloat that doesn't matter to a serious research stack

Andrej Karpathy950a1dc61e27
diff --git a/README.md b/README.md
index 883f273..b186134 100644
--- a/README.md
+++ b/README.md
@@ -139,7 +139,6 @@ I've published a number of guides that might contain helpful information, most r
- [Feb 1 2026: Beating GPT-2 for <<$100: the nanochat journey](https://github.com/karpathy/nanochat/discussions/481)
- [Jan 7 miniseries v1](https://github.com/karpathy/nanochat/discussions/420) documents the first nanochat miniseries of models.
- To add new abilities to nanochat, see [Guide: counting r in strawberry (and how to add abilities generally)](https://github.com/karpathy/nanochat/discussions/164).
-- To customize your nanochat, see [Guide: infusing identity to your nanochat](https://github.com/karpathy/nanochat/discussions/139) in Discussions, which describes how you can tune your nanochat's personality through synthetic data generation and mixing that data into the SFT stage.
- [Oct 13 2025: original nanochat post](https://github.com/karpathy/nanochat/discussions/1) introducing nanochat, though now it contains some deprecated information and the model is a lot older (with worse results) than current master.
## File structure
@@ -149,7 +148,6 @@ I've published a number of guides that might contain helpful information, most r
├── LICENSE
├── README.md
├── dev
-│ ├── gen_synthetic_data.py # Example synthetic data for identity
│ ├── nanochat.png
│ └── repackage_data_reference.py # Pretraining data shard generation
├── nanochat
@@ -183,12 +181,10 @@ I've published a number of guides that might contain helpful information, most r
├── tasks
│ ├── arc.py # Multiple choice science questions
│ ├── common.py # TaskMixture | TaskSequence
-│ ├── customjson.py # Make Task from arbitrary jsonl convos
│ ├── gsm8k.py # 8K Grade School Math questions
│ ├── humaneval.py # Misnomer; Simple Python coding task
│ ├── mmlu.py # Multiple choice questions, broad topics
-│ ├── smoltalk.py # Conglomerate dataset of SmolTalk from HF
-│ └── spellingbee.py # Task teaching model to spell/count letters
+│ └── smoltalk.py # Conglomerate dataset of SmolTalk from HF
├── tests
│ └── test_engine.py
└── uv.lock
diff --git a/dev/gen_synthetic_data.py b/dev/gen_synthetic_data.py
deleted file mode 100644
index f5aa2df..0000000
--- a/dev/gen_synthetic_data.py
+++ /dev/null
@@ -1,468 +0,0 @@
-"""
-Synthetic data generation for teaching nanochat about its identity and capabilities.
-
-This script uses the OpenRouter API to generate diverse multi-turn conversations
-between a user and nanochat. The conversations are saved to a .jsonl file for use
-in supervised finetuning (SFT) via the CustomJSON task.
-
-Key design principles for high-quality synthetic data:
-1. DIVERSITY CONTROL is critical - we inject entropy at multiple levels:
- - Topic/question categories (what the conversation is about)
- - User personas (who is asking)
- - Conversation dynamics (shape and flow)
- - First message style (greeting variation)
-2. Comprehensive knowledge base - we provide detailed facts so the LLM
- generating conversations has accurate information to draw from.
-3. Structured outputs - we use JSON schema to guarantee valid format.
-
-NOTE: You need OPENROUTER_API_KEY set in .env or as an environment variable.
-NOTE: For more details see: https://github.com/karpathy/nanochat/discussions/139
-"""
-import requests
-import json
-import os
-import copy
-import random
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from dotenv import load_dotenv
-
-from nanochat.common import get_base_dir
-
-load_dotenv()
-api_key = os.environ["OPENROUTER_API_KEY"]
-
-url = "https://openrouter.ai/api/v1/chat/completions"
-headers = {
- "Authorization": f"Bearer {api_key}",
- "Content-Type": "application/json"
-}
-
-# Load the comprehensive knowledge base
-knowledge_path = os.path.join(os.path.dirname(__file__), "..", "knowledge", "self_knowledge.md")
-knowledge = open(knowledge_path, "r", encoding="utf-8").read().strip()
-assert os.path.exists(knowledge_path), f"Knowledge base file not found: {knowledge_path}"
-# for right now I am not committing the self_knowledge file to repo. You can use README.md instead
-# of it, or you can generate one by asking an LLM to make one based on the README/files.
-# This whole file is just a helpful demonstration of the kind of thing you'd run.
-
-# =============================================================================
-# DIVERSITY DIMENSIONS
-# =============================================================================
-
-# Topics/questions the conversation should explore
-# Group by category for balanced sampling
-topics = {
- "identity": [
- "who/what is nanochat",
- "who created nanochat and why",
- "what does the name 'nanochat' mean",
- "is nanochat open source, what license",
- "where can I find the code",
- "how can I contribute to nanochat",
- ],
- "architecture": [
- "basic architecture overview (transformer, layers, parameters)",
- "what is RoPE and why use it",
- "explain RMSNorm vs LayerNorm",
- "what is Flash Attention and why it matters",
- "sliding window attention pattern",
- "value embeddings - what are they",
- "per-layer residual scalars",
- "ReLU squared activation",
- "logit softcapping",
- "QK normalization",
- ],
- "training": [
- "how much did it cost to train nanochat",
- "how long does training take",
- "what hardware is needed",
- "what data was nanochat trained on",
- "what is the Muon optimizer",
- "explain the split optimizer design",
- "what is the depth parameter and scaling",
- "what is the CORE metric",
- ],
- "capabilities": [
- "what can nanochat do",
- "can nanochat write code",
- "can nanochat do math (calculator tool)",
- "can nanochat help with writing",
- "what languages does nanochat speak",
- "how good is nanochat at reasoning",
- ],
- "limitations": [
- "what can nanochat NOT do",
- "why does nanochat work best in English",
- "does nanochat have internet access",
- "what is nanochat's context length limit",
- "can nanochat remember previous conversations",
- "can nanochat make mistakes / hallucinate",
- "is nanochat good for production use",
- ],
- "comparisons": [
- "how does nanochat compare to GPT-2",
- "how does nanochat compare to ChatGPT/GPT-4",
- "how does nanochat compare to Claude",
- "why is training 600x cheaper than GPT-2",
- "what's special about nanochat vs other open models",
- ],
- "history": [
- "the GPT-2 training cost in 2019",
- "how AI training costs have dropped over time",
- "relationship to modded-nanogpt project",
- "what optimizations worked vs didn't work",
- "the journey of building nanochat",
- ],
- "technical_deep_dive": [
- "explain the tokenizer (BPE, vocab size)",
- "how does distributed training work (ZeRO)",
- "explain the dataloader and BOS alignment",
- "what is compute-optimal training",
- "how does the calculator tool work",
- "explain inference with KV cache",
- ],
- "philosophical": [
- "is nanochat conscious / does it have feelings",
- "what happens when nanochat is wrong",
- "can nanochat learn from this conversation",
- "why make AI training accessible",
- "the future of open source AI",
- ],
-}
-
-# User personas - different people ask questions differently
-personas = [
- "curious beginner who knows nothing about AI or machine learning",
- "ML researcher or engineer who wants technical depth and specifics",
- "developer considering contributing to the nanochat project",
- "skeptic who doubts open source can compete with big AI labs",
- "computer science student learning about transformers and LLMs",
- "someone comparing nanochat to ChatGPT, Claude, or other assistants",
- "journalist or writer covering AI democratization and open source",
- "hobbyist who just wants to chat and learn casually",
- "someone interested in the cost and economics of AI training",
- "teacher or educator wanting to use nanochat for teaching",
- "entrepreneur exploring if nanochat fits their use case",
- "someone who just discovered the project and wants the basics",
-]
-
-# Conversation dynamics - shape and flow
-dynamics = [
- "short 2-turn Q&A: user asks one question, gets a complete answer",
- "medium 4-turn: user asks, gets answer, asks followup for clarification",
- "deep 6-turn technical discussion: progressively deeper questions",
- "skeptical arc: user starts doubtful, assistant addresses concerns honestly",
- "learning journey: user starts basic, assistant builds up complexity gradually",
- "comparison-focused: user keeps comparing to other models, assistant explains differences",
- "limitation exploration: user probes what nanochat cannot do, assistant is honest",
- "casual friendly chat that naturally touches on identity and capabilities",
- "troubleshooting: user has misconceptions, assistant gently corrects them",
- "enthusiastic: user is excited about the project, assistant shares that energy appropriately",
-]
-
-# First messages - greetings and openers
-# Categorized for balanced sampling
-first_messages = {
- "simple_greetings": [
- "hi", "Hi!", "hello", "Hello?", "hey there", "Hey!", "yo", "Yo!",
- "Good morning", "Good evening!", "Howdy", "sup", "What's up?",
- "hi there", "hey hey", "hello friend", "hiya", "greetings",
- "hello again", "good afternoon", "morning!", "evening!",
- ],
- "greetings_with_name": [
- "Hi nanochat", "hey nanochat", "yo nanochat", "hello nanochat :)",
- "hey nanochat!", "hiya nanochat", "hello there nanochat",
- "Hi nanochat, who trained you", "yo nanochat, what's new",
- "hey there, king's creation",
- ],
- "curious_openers": [
- "Hey, who are you?", "Hi, what is this?", "Hey, are you a chatbot?",
- "Hello! Who am I talking to?", "hi! what do you do?",
- "hi! who made you", "hey! are you alive", "hiya! what are you",
- "hello! tell me about yourself", "hi, what's your name",
- "yo, what is this", "hi! who built you", "hello! are you open source",
- "hey, what version are you", "hi! what's your story",
- "hey, what's nanochat", "hello! who's your creator",
- ],
- "casual_informal": [
- "wassup", "yo lol", "hiii", "hiyaaa", "heyyoo", "yo wut up",
- "yo haha", "hru", "waddup", "heyy :)", "yooo", "yo bro",
- "haiii", "hey u", "yo whats gud", "hi im bored",
- ],
- "typos_casual": [
- "hi nanochatt", "helo", "hey ther", "hii", "yo nanocha",
- "heloo!", "hi, whos this", "hay", "helloo??", "hi nanocat",
- "helo nanochat", "hai!", "helllo nano", "yo nanochta",
- ],
- "caps_enthusiastic": [
- "HI", "HELLOOO", "YO!!!", "HEY", "SUP", "WASSUP", "HEY!!!",
- "HELLO??", "HI THERE!!", "HEYOOOO", "HIII", "YOOOO", "HELLO!!!",
- ],
- "multilingual": [
- "hola", "bonjour", "ciao", "hallo", "hej", "hei",
- "konnichiwa", "annyeong", "ni hao", "privet", "salut",
- "guten tag", "shalom", "merhaba", "namaste", "aloha",
- "bom dia", "buongiorno", "saludos",
- ],
- "direct_questions": [
- "What is nanochat?", "Who made you?", "Are you GPT?",
- "How do you compare to ChatGPT?", "Can you help me code?",
- "What can you do?", "Are you open source?", "How were you trained?",
- "What's your context limit?", "Can you browse the internet?",
- ],
-}
-
-# =============================================================================
-# PROMPT TEMPLATE
-# =============================================================================
-
-prompt_template = r"""
-I want to generate synthetic training data for an AI assistant called "nanochat" to teach it about its own identity, capabilities, and limitations.
-
-## KNOWLEDGE BASE
-
-Here is comprehensive information about nanochat that you should use as the authoritative source of facts:
-
----
-{knowledge}
----
-
-## YOUR TASK
-
-Generate a realistic multi-turn conversation between a User and the nanochat Assistant.
-
-**Topic to explore:** {topic}
-**User persona:** {persona}
-**Conversation dynamic:** {dynamic}
-
-## STYLE GUIDELINES
-
-1. **Plain ASCII only** - No emojis, special characters, or unicode. Just plain text.
-2. **Natural conversation** - Make it feel like a real chat, not a Q&A exam.
-3. **Accurate facts** - Use ONLY information from the knowledge base above. Don't make up statistics or features.
-4. **Appropriate depth** - Match the technical level to the user persona.
-5. **Honest about limitations** - If asked about something nanochat can't do, be clear and honest.
-6. **Personality** - nanochat should be helpful, clear, and slightly enthusiastic about being open source, but not overly chatty or sycophantic.
-
-## FIRST MESSAGE EXAMPLES
-
-Here are some example first messages from users (for style inspiration):
-{first_message_examples}
-
-## SPECIAL CASES
-
-- **Non-English first message:** If the user writes in another language, nanochat should briefly acknowledge it can understand but works best in English, then continue helpfully.
-- **Misconceptions:** If the user has wrong assumptions (e.g., "you're made by OpenAI"), gently correct them.
-- **Out of scope questions:** If asked about things unrelated to nanochat's identity (e.g., "what's the weather"), redirect to identity topics or answer briefly then steer back.
-
-## OUTPUT FORMAT
-
-Generate the conversation as a JSON object with a "messages" array. Each message has "role" (user/assistant) and "content". Start with a user message.
-""".strip()
-
-# =============================================================================
-# API CONFIGURATION
-# =============================================================================
-
-response_format = {
- "type": "json_schema",
- "json_schema": {
- "name": "conversation",
- "strict": True,
- "schema": {
- "type": "object",
- "properties": {
- "messages": {
- "type": "array",
- "description": "Conversation messages alternating user/assistant, starting with user",
- "items": {
- "type": "object",
- "properties": {
- "role": {
- "type": "string",
- "description": "Either 'user' or 'assistant'"
- },
- "content": {
- "type": "string",
- "description": "The message content"
- }
- },
- "required": ["role", "content"],
- "additionalProperties": False
- }
- }
- },
- "required": ["messages"],
- "additionalProperties": False
- }
- }
-}
-
-base_payload = {
- "model": "google/gemini-3-flash-preview",
- "stream": False,
- "response_format": response_format,
- "temperature": 1.0,
-}
-
-# =============================================================================
-# GENERATION LOGIC
-# =============================================================================
-
-def sample_diversity_elements(rng):
- """Sample one element from each diversity dimension."""
- # Sample topic: first pick a category, then a topic within it
- category = rng.choice(list(topics.keys()))
- topic = rng.choice(topics[category])
-
- # Sample persona
- persona = rng.choice(personas)
-
- # Sample dynamic
- dynamic = rng.choice(dynamics)
-
- # Sample first message examples: pick from multiple categories
- first_msg_samples = []
- categories = rng.sample(list(first_messages.keys()), min(3, len(first_messages)))
- for cat in categories:
- first_msg_samples.append(rng.choice(first_messages[cat]))
-
- return {
- "topic": topic,
- "persona": persona,
- "dynamic": dynamic,
- "first_message_examples": "\n".join(f"- {msg}" for msg in first_msg_samples),
- }
-
-
-def generate_conversation(idx: int):
- """
- Generate a single conversation using the OpenRouter API.
- Returns a list of message dicts with 'role' and 'content' keys.
- """
- # Use idx as seed for reproducibility
- rng = random.Random(idx)
-
- # Sample diversity elements
- elements = sample_diversity_elements(rng)
-
- # Build the prompt
- prompt = prompt_template.format(
- knowledge=knowledge,
- topic=elements["topic"],
- persona=elements["persona"],
- dynamic=elements["dynamic"],
- first_message_examples=elements["first_message_examples"],
- )
-
- # Make API request
- payload = copy.deepcopy(base_payload)
- payload['messages'] = [{"role": "user", "content": prompt}]
-
- response = requests.post(url, headers=headers, json=payload)
- result = response.json()
-
- if 'error' in result:
- raise Exception(f"API error: {result['error']}")
-
- content = result['choices'][0]['message']['content']
- conversation_data = json.loads(content)
- messages = conversation_data['messages']
-
- # Return messages along with metadata for debugging
- return {
- "messages": messages,
- "metadata": {
- "topic": elements["topic"],
- "persona": elements["persona"],
- "dynamic": elements["dynamic"],
- }
- }
-
-
-def validate_conversation(messages):
- """Validate conversation structure."""
- if len(messages) < 2:
- raise ValueError(f"Conversation too short: {len(messages)} messages")
-
- for i, message in enumerate(messages):
- expected_role = "user" if i % 2 == 0 else "assistant"
- if message['role'] != expected_role:
- raise ValueError(f"Message {i} has role '{message['role']}', expected '{expected_role}'")
-
- if not message['content'].strip():
- raise ValueError(f"Message {i} has empty content")
-
- return True
-
-
-# =============================================================================
-# MAIN
-# =============================================================================
-
-if __name__ == "__main__":
- import argparse
-
- parser = argparse.ArgumentParser(description="Generate synthetic conversation data")
- parser.add_argument("--num", type=int, default=1000, help="Number of conversations to generate")
- parser.add_argument("--workers", type=int, default=4, help="Number of parallel workers")
- parser.add_argument("--output", type=str, default=None, help="Output file path")
- parser.add_argument("--append", action="store_true", help="Append to existing file instead of overwriting")
- parser.add_argument("--save-metadata", action="store_true", help="Save metadata alongside messages")
- args = parser.parse_args()
-
- # Set output file
- if args.output:
- output_file = args.output
- else:
- output_file = os.path.join(get_base_dir(), "identity_conversations.jsonl")
-
- # Handle file creation/clearing
- if not args.append and os.path.exists(output_file):
- os.remove(output_file)
-
- print(f"Output file: {output_file}")
- print(f"Generating {args.num} conversations with {args.workers} workers...")
- print(f"Topic categories: {list(topics.keys())}")
- print(f"Personas: {len(personas)}")
- print(f"Dynamics: {len(dynamics)}")
- print()
-
- completed_count = 0
- error_count = 0
-
- with ThreadPoolExecutor(max_workers=args.workers) as executor:
- # Submit all tasks
- futures = {executor.submit(generate_conversation, idx): idx
- for idx in range(args.num)}
-
- # Process results as they complete
- for future in as_completed(futures):
- idx = futures[future]
- try:
- result = future.result()
- messages = result["messages"]
- metadata = result["metadata"]
-
- # Validate
- validate_conversation(messages)
-
- # Write to file
- with open(output_file, 'a') as f:
- if args.save_metadata:
- f.write(json.dumps({"messages": messages, "metadata": metadata}) + '\n')
- else:
- f.write(json.dumps(messages) + '\n')
-
- completed_count += 1
- topic_short = metadata["topic"][:40] + "..." if len(metadata["topic"]) > 40 else metadata["topic"]
- print(f"[{completed_count}/{args.num}] Topic: {topic_short}")
-
- except Exception as e:
- error_count += 1
- print(f"[ERROR] idx={idx}: {e}")
-
- print()
- print(f"Done! Saved {completed_count} conversations to {output_file}")
- if error_count > 0:
- print(f"Encountered {error_count} errors during generation")
diff --git a/runs/runcpu.sh b/runs/runcpu.sh
index 7c43cba..c4c3f6d 100755
--- a/runs/runcpu.sh
+++ b/runs/runcpu.sh
@@ -45,7 +45,6 @@ python -m scripts.base_train \
python -m scripts.base_eval --device-batch-size=1 --split-tokens=16384 --max-per-task=16
# SFT (~10 minutes on my MacBook Pro M3 Max)
-curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl
python -m scripts.chat_sft \
--eval-every=200 \
--eval-tokens=524288 \
diff --git a/runs/speedrun.sh b/runs/speedrun.sh
index d710555..132b968 100644
--- a/runs/speedrun.sh
+++ b/runs/speedrun.sh
@@ -71,10 +71,6 @@ torchrun --standalone --nproc_per_node=8 -m scripts.base_eval -- --device-batch-
# -----------------------------------------------------------------------------
# SFT (teach the model conversation special tokens, tool use, multiple choice)
-# download 2.3MB of synthetic identity conversations to impart a personality to nanochat
-# see dev/gen_synthetic_data.py for details on how this data was prepared and to get a sense of how you can easily tune it
-curl -L -o $NANOCHAT_BASE_DIR/identity_conversations.jsonl https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl
-
# run SFT and eval the model
torchrun --standalone --nproc_per_node=8 -m scripts.chat_sft -- --run=$WANDB_RUN
torchrun --standalone --nproc_per_node=8 -m scripts.chat_eval -- -i sft
diff --git a/scripts/chat_eval.py b/scripts/chat_eval.py
index 1447091..1118102 100644
--- a/scripts/chat_eval.py
+++ b/scripts/chat_eval.py
@@ -21,7 +21,6 @@ from tasks.humaneval import HumanEval
from tasks.mmlu import MMLU
from tasks.arc import ARC
from tasks.gsm8k import GSM8K
-from tasks.spellingbee import SpellingBee
# -----------------------------------------------------------------------------
# Generative evaluation loop (we go one problem at a time, sample, evaluate)
@@ -164,7 +163,6 @@ def run_chat_eval(task_name, model, tokenizer, engine,
'ARC-Easy': partial(ARC, subset="ARC-Easy", split="test"),
'ARC-Challenge': partial(ARC, subset="ARC-Challenge", split="test"),
'GSM8K': partial(GSM8K, subset="main", split="test"),
- 'SpellingBee': partial(SpellingBee, size=256, split="test"),
}[task_name]
task_object = task_module()
# Run the evaluation
@@ -201,14 +199,13 @@ if __name__ == "__main__":
engine = Engine(model, tokenizer)
# Get the tasks to evaluate on
- all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval', 'SpellingBee']
+ all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval']
baseline_accuracies = {
'ARC-Easy': 0.25, # multiple choice 1 of 4 => 25%
'ARC-Challenge': 0.25, # multiple choice 1 of 4 => 25%
'MMLU': 0.25, # multiple choice 1 of 4 => 25%
'GSM8K': 0.0, # open-ended => 0%
'HumanEval': 0.0, # open-ended => 0%
- 'SpellingBee': 0.0, # open-ended => 0%
}
task_names = all_tasks if args.task_name is None else args.task_name.split('|')
diff --git a/scripts/chat_sft.py b/scripts/chat_sft.py
index e2dda89..86a9e6f 100644
--- a/scripts/chat_sft.py
+++ b/scripts/chat_sft.py
@@ -29,8 +29,6 @@ from tasks.common import TaskMixture
from tasks.gsm8k import GSM8K
from tasks.mmlu import MMLU
from tasks.smoltalk import SmolTalk
-from tasks.customjson import CustomJSON
-from tasks.spellingbee import SimpleSpelling, SpellingBee
# -----------------------------------------------------------------------------
# CLI arguments
@@ -161,15 +159,10 @@ for group in optimizer.param_groups:
group["initial_lr"] = group["lr"]
# SFT data mixture and DataLoader
-identity_conversations_filepath = os.path.join(base_dir, "identity_conversations.jsonl")
train_tasks = [
SmolTalk(split="train"), # 460K rows of general conversations
- CustomJSON(filepath=identity_conversations_filepath), # 1000 rows of synthetic identity conversations
- CustomJSON(filepath=identity_conversations_filepath), # 2 epochs of these
*[MMLU(subset="all", split="auxiliary_train") for _ in range(args.mmlu_epochs)], # 100K rows per epoch
*[GSM8K(subset="main", split="train") for _ in range(args.gsm8k_epochs)], # 8K rows per epoch
- SimpleSpelling(size=200000, split="train"), # 200K rows of Simple Spelling (e.g. spell the word 'apple')
- SpellingBee(size=80000, split="train"), # 80K rows of Spelling Bee (e.g. how many 'r' are in 'strawberry'?)
]
train_dataset = TaskMixture(train_tasks)
print0(f"Training mixture: {len(train_dataset):,} rows (MMLU x{args.mmlu_epochs}, GSM8K x{args.gsm8k_epochs})")
@@ -366,11 +359,11 @@ while True:
if args.chatcore_every > 0 and (last_step or (step > 0 and step % args.chatcore_every == 0)):
model.eval()
engine = Engine(orig_model, tokenizer)
- all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval', 'SpellingBee']
+ all_tasks = ['ARC-Easy', 'ARC-Challenge', 'MMLU', 'GSM8K', 'HumanEval']
categorical_tasks = {'ARC-Easy', 'ARC-Challenge', 'MMLU'}
baseline_accuracies = {
'ARC-Easy': 0.25, 'ARC-Challenge': 0.25, 'MMLU': 0.25,
- 'GSM8K': 0.0, 'HumanEval': 0.0, 'SpellingBee': 0.0,
+ 'GSM8K': 0.0, 'HumanEval': 0.0,
}
task_results = {}
for task_name in all_tasks:
diff --git a/tasks/customjson.py b/tasks/customjson.py
deleted file mode 100644
index aeb1a3f..0000000
--- a/tasks/customjson.py
+++ /dev/null
@@ -1,65 +0,0 @@
-"""
-CustomJSON task for loading conversations from JSONL files.
-Each line in the JSONL file should be a JSON array of messages.
-"""
-
-import os
-import json
-from tasks.common import Task
-
-class CustomJSON(Task):
- """
- Load conversations from a JSONL file.
- Each line should be a JSON array of message objects with 'role' and 'content' fields.
- Example line: [{"role":"user","content":"Hi"},{"role":"assistant","content":"Hello"}]
- """
-
- def __init__(self, filepath, **kwargs):
- super().__init__(**kwargs)
- self.filepath = filepath
- self.conversations = []
-
- # Load all conversations from the JSONL file
- if not os.path.exists(filepath):
- # Helpful error message due to recent change. Will be removed in the future.
- print("-" * 80)
- print(f"Warning: File {filepath} does not exist")
- print("HINT (Oct 21 2025)")
- print("If you recently did a git pull and suddenly see this, it might be due to the new addition of identity conversations")
- print("See this discussion for more details: https://github.com/karpathy/nanochat/discussions/139")
- print("Quick fix: simply run the following command to download the file and you're done:")
- print(f"curl -L -o {filepath} https://karpathy-public.s3.us-west-2.amazonaws.com/identity_conversations.jsonl")
- print("-" * 80)
-
- else:
- with open(filepath, 'r', encoding='utf-8') as f:
- for line in f:
- line = line.strip()
- if not line: # skip empty lines
- continue
- messages = json.loads(line)
- # Validate the conversation structure
- assert isinstance(messages, list), f"Expected list of messages, got {type(messages)}"
- assert len(messages) >= 2, f"Conversation must have at least 2 messages, got {len(messages)}"
- # Validate message structure and alternating roles
- for i, message in enumerate(messages):
- assert "role" in message, f"Message {i} missing 'role' field"
- assert "content" in message, f"Message {i} missing 'content' field"
- expected_role = "user" if i % 2 == 0 else "assistant"
- assert message["role"] == expected_role, f"Message {i} has role {message['role']} but should be {expected_role}"
- assert isinstance(message["content"], str), f"Message {i} content must be a string"
-
- self.conversations.append(messages)
-
- self.length = len(self.conversations)
-
- def num_examples(self):
- return self.length
-
- def get_example(self, index):
- messages = self.conversations[index]
- conversation = {
- "messages": messages,
- }
- return conversation
-
diff --git a/tasks/spellingbee.py b/tasks/spellingbee.py
deleted file mode 100644
index 44889bd..0000000
--- a/tasks/spellingbee.py
+++ /dev/null
@@ -1,307 +0,0 @@
-"""
-Task intended to make nanochat better in spelling and counting, for example:
-
-"How many r are in strawberry?" -> 3
-
-An interesting part of this task is that we will get the assistant to
-solve the problem using a combination of manual counting and Python.
-This is a good problem solving "instinct" to mix into the model and RL
-may further refine it to trust one over the other. If we were extra fancy
-(which we could/should be) we'd add small errors here and there to allow
-the model also learn recoveries. We can do this in future versions.
-
-There are two tasks in this file:
-1. SpellingBee: Counting the number of occurrences of a letter in a word
-2. SimpleSpelling: Simply spelling words
-
-(1) is the goal, but (2) exists as a highly condensed version of the part
-that makes (1) difficult, which is word spelling. This is non-trivial for an
-LLM because it has to learn how every token (a little semantic chunk/atom)
-maps to the sequence of individual characters that make it up. Larger models
-learn this eventually on their own, but if we want this capability to exist
-in smaller models, we have to actively encourage it by over-representing it
-in the training data. SFT is a good place to do this.
-
-To preview a few example conversations, run:
-python -m tasks.spellingbee
-"""
-
-import re
-import random
-from tasks.common import Task
-from nanochat.common import download_file_with_lock
-
-# Letters of the alphabet
-LETTERS = "abcdefghijklmnopqrstuvwxyz"
-# A list of 370K English words of large variety
-WORD_LIST_URL = "https://raw.githubusercontent.com/dwyl/english-words/refs/heads/master/words_alpha.txt"
-# A number bigger than 370K to separate train and test random seeds
-TEST_RANDOM_SEED_OFFSET = 10_000_000
-
-# Identical to gsm8k's answer extraction
-ANSWER_RE = re.compile(r"#### (\-?[0-9\.\,]+)")
-def extract_answer(completion):
- """
- Extract the numerical answer after #### marker.
- """
- match = ANSWER_RE.search(completion)
- if match:
- match_str = match.group(1).strip()
- match_str = match_str.replace(",", "")
- return match_str
- return None
-
-# User message templates for data augmentation
-USER_MSG_TEMPLATES = [
- "How many {letter} are in the word {word}",
- "How many {letter} are in {word}",
- "Count the number of {letter} in {word}",
- "How many times does {letter} appear in {word}",
- "What's the count of {letter} in {word}",
- "In the word {word}, how many {letter} are there",
- "How many letter {letter} are in the word {word}",
- "Count how many {letter} appear in {word}",
- "Tell me the number of {letter} in {word}",
- "How many occurrences of {letter} are in {word}",
- "Find the count of {letter} in {word}",
- "Can you count the {letter} letters in {word}",
- "What is the frequency of {letter} in {word}",
- "How many {letter}s are in {word}",
- "How many {letter}'s are in {word}",
- "Count all the {letter} in {word}",
- "How many times is {letter} in {word}",
- "Number of {letter} in {word}",
- "Total count of {letter} in {word}",
- "How many {letter} does {word} have",
- "How many {letter} does {word} contain",
- "What's the number of {letter} in {word}",
- "{word} has how many {letter}",
- "In {word}, count the {letter}",
- "How many {letter} appear in {word}",
- "Count the {letter} in {word}",
- "Give me the count of {letter} in {word}",
- "How many instances of {letter} in {word}",
- "Show me how many {letter} are in {word}",
- "Calculate the number of {letter} in {word}",
- # Spanish
- "¿Cuántas {letter} hay en {word}?",
- "¿Cuántas veces aparece {letter} en {word}?",
- "Cuenta las {letter} en {word}",
- "¿Cuántas letras {letter} tiene {word}?",
- # Chinese (Simplified)
- "{word}中有多少个{letter}",
- "{word}里有几个{letter}",
- "数一下{word}中的{letter}",
- "{word}这个词里有多少{letter}",
- # Korean
- "{word}에 {letter}가 몇 개 있나요",
- "{word}에서 {letter}의 개수는",
- "{word}에 {letter}가 몇 번 나오나요",
- "{word}라는 단어에 {letter}가 몇 개",
- # French
- "Combien de {letter} dans {word}",
- "Combien de fois {letter} apparaît dans {word}",
- "Compte les {letter} dans {word}",
- # German
- "Wie viele {letter} sind in {word}",
- "Wie oft kommt {letter} in {word} vor",
- "Zähle die {letter} in {word}",
- # Japanese
- "{word}に{letter}は何個ありますか",
- "{word}の中に{letter}がいくつ",
- "{word}に{letter}が何回出てくる",
-]
-
-class SpellingBee(Task):
-
- def __init__(self, size=1000, split="train", **kwargs):
- super().__init__(**kwargs)
- assert split in ["train", "test"], "SpellingBee split must be train|test"
- self.size = size
- self.split = split
- filename = WORD_LIST_URL.split("/")[-1]
- word_list_path = download_file_with_lock(WORD_LIST_URL, filename)
- with open(word_list_path, 'r', encoding='utf-8') as f:
- words = [line.strip() for line in f]
- self.words = words
-
- @property
- def eval_type(self):
- return 'generative'
-
- def num_examples(self):
- return self.size
-
- def get_example(self, index):
- seed = index if self.split == 'train' else TEST_RANDOM_SEED_OFFSET + index
- rng = random.Random(seed)
-
- # pick a random word
- word = rng.choice(self.words)
- # pick a letter from it (90%) or a random letter (10%)
- letter = rng.choice(word) if rng.random() < 0.9 else rng.choice(LETTERS)
-
- # get the correct answer by simply counting
- count = word.count(letter)
-
- # create a user message, with a bunch of variations as data augmentation
- template = rng.choice(USER_MSG_TEMPLATES)
- # 30% chance to lowercase the template (lazy people don't use shift)
- if rng.random() < 0.3:
- template = template.lower()
- quote_options = ['', "'", '"']
- letter_quote = rng.choice(quote_options) # is the letter quoted?
- word_quote = rng.choice(quote_options) # is the word quoted?
- letter_wrapped = f"{letter_quote}{letter}{letter_quote}"
- word_wrapped = f"{word_quote}{word}{word_quote}"
- user_msg = template.format(letter=letter_wrapped, word=word_wrapped)
- if rng.random() < 0.5: # 50% of people don't even use question marks
- user_msg += "?"
-
- # Now create the ideal assistant response - build as parts (text + tool calls)
- assistant_parts = []
- word_letters = ",".join(list(word))
- manual_text = f"""We are asked to find the number '{letter}' in the word '{word}'. Let me try a manual approach first.
-
-First spell the word out:
-{word}:{word_letters}
-
-Then count the occurrences of '{letter}':
-"""
- # Little simulated loop of the solution process
- # TODO: This is where the fun starts, we could simulate cute little mistakes
- # and get the model to review its work and recover from them.
- # You might of course hope this could arise in RL too, but realistically you'd want to help it out a bit.
- running_count = 0
- for i, char in enumerate(word, 1):
- if char == letter:
- running_count += 1
- # note: there deliberately cannot be a space here between i and char
- # because this would create a different token! (e.g. " a" and "a" are different tokens)
- manual_text += f"{i}:{char} hit! count={running_count}\n"
- else:
- manual_text += f"{i}:{char}\n"
-
- manual_text += f"\nThis gives us {running_count}."
- assistant_parts.append({"type": "text", "text": manual_text})
- # Part 2: Python verification
- assistant_parts.append({"type": "text", "text": "\n\nLet me double check this using Python:\n\n"})
- # Part 3: Python tool call
- python_expr = f"'{word}'.count('{letter}')"
- assistant_parts.append({"type": "python", "text": python_expr})
- # Part 4: Python output
- assistant_parts.append({"type": "python_output", "text": str(count)})
- # Part 5: Final answer
- assistant_parts.append({"type": "text", "text": f"\n\nPython gives us {count}.\n\nMy final answer is:\n\n#### {count}"})
-
- # return the full conversation
- messages = [
- {"role": "user", "content": user_msg},
- {"role": "assistant", "content": assistant_parts}
- ]
- conversation = {
- "messages": messages,
- }
- return conversation
-
- def evaluate(self, conversation, assistant_response):
- """
- Given (conversation, completion), return evaluation outcome (0 = wrong, 1 = correct)
- Identical to gsm8k's evaluation.
- """
- assert isinstance(assistant_response, str), "Assuming simple string response for now"
- # First extract the ground truth answer from the conversation
- assistant_message = conversation['messages'][-1]
- assert assistant_message['role'] == "assistant", "Last message must be from the Assistant"
- assert isinstance(assistant_message['content'], list), "This is expected to be a list of parts"
- # The last text part contains the final answer with ####
- last_text_part = assistant_message['content'][-1]['text']
- # Extract both the ground truth answer and the predicted answer
- ref_num = extract_answer(last_text_part)
- pred_num = extract_answer(assistant_response)
- # Compare and return the success as int
- is_correct = int(pred_num == ref_num)
- return is_correct
-
- def reward(self, conversation, assistant_response):
- """ Use simple 0-1 reward just like gsm8k."""
- is_correct = self.evaluate(conversation, assistant_response)
- is_correct_float = float(is_correct)
- return is_correct_float
-
-
-class SimpleSpelling(Task):
- """Much simpler task designed to get the model to just practice spelling words."""
-
- def __init__(self, size=1000, split="train", **kwargs):
- super().__init__(**kwargs)
- assert split in ["train", "test"], "SpellingBee split must be train|test"
- self.size = size
- self.split = split
- filename = WORD_LIST_URL.split("/")[-1]
- word_list_path = download_file_with_lock(WORD_LIST_URL, filename)
- with open(word_list_path, 'r', encoding='utf-8') as f:
- words = [line.strip() for line in f]
- rng = random.Random(42)
- rng.shuffle(words) # use a different word order than the SpellingBee task
- self.words = words
-
- @property
- def eval_type(self):
- return 'generative'
-
- def num_examples(self):
- return self.size
-
- def get_example(self, index):
- seed = index if self.split == 'train' else TEST_RANDOM_SEED_OFFSET + index
- rng = random.Random(seed)
- # pick a random word
- word = rng.choice(self.words)
- word_letters = ",".join(list(word))
- # return the full conversation
- messages = [
- {"role": "user", "content": f"Spell the word: {word}"},
- {"role": "assistant", "content": f"{word}:{word_letters}"}
- ]
- conversation = {
- "messages": messages,
- }
- return conversation
-
-
-if __name__ == "__main__":
-
- # preview the SpellingBee task, first 10 examples
- task = SpellingBee()
- for i in range(10):
- ex = task.get_example(i)
- print("=" * 100)
- print(ex['messages'][0]['content'])
- print("-" * 100)
- # Assistant content is now a list of parts
- assistant_parts = ex['messages'][1]['content']
- for part in assistant_parts:
- if part['type'] == 'text':
- print(part['text'], end='')
- elif part['type'] == 'python':
- print(f"<<{part['text']}=", end='')
- elif part['type'] == 'python_output':
- print(f"{part['text']}>>", end='')
- print()
- print("-" * 100)
-
- # # preview the SimpleSpelling task, first 10 examples
- # task = SimpleSpelling()
- # for i in range(10):
- # ex = task.get_example(i)
- # print("=" * 100)
- # print(ex['messages'][0]['content'])
- # print("-" * 100)
- # print(ex['messages'][1]['content'])
-
- # # also scrutinize the tokenization (last example only)
- # from nanochat.tokenizer import get_tokenizer
- # tokenizer = get_tokenizer()
- # ids, mask = tokenizer.render_conversation(ex)
- # print(tokenizer.visualize_tokenization(ids, mask, with_token_id=True))