Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
updates
HenryNdubuaku2bc4e4952665
diff --git a/dataset_readme.md b/dataset_readme.mdnew file mode 100644index 0000000..5c1a23e--- /dev/null+++ b/dataset_readme.md@@ -0,0 +1,211 @@+---+dataset_info:+ features:+ - name: query+ dtype: string+ - name: answers+ dtype: string+ - name: tools+ dtype: string+ - name: source+ dtype: string+ - name: model+ dtype: string+ splits:+ - name: train+ num_examples: 2996257+ - name: validation+ num_examples: 10000+ download_size: 2160782760+configs:+- config_name: default+ data_files:+ - split: train+ path: data/train-*+ - split: validation+ path: data/validation-*+license: apache-2.0+task_categories:+- text-generation+language:+- en+tags:+- tool-calling+- function-calling+- on-device+- assistant+- synthetic+pretty_name: On-Device Tool Calls+size_categories:+- 1M<n<10M+---++# On-Device Tool Calls++A large-scale synthetic dataset for training compact tool-calling models. Designed for on-device assistants that run on phones, wearables, smart home hubs, and laptops — where latency and model size matter.++Built for [Needle](https://github.com/Cactus-Compute/needle), a 32M-parameter encoder-decoder transformer that achieves 92% single-call F1 and 78% multi-call F1 on tool-call generation.++## Overview++| | |+|---|---|+| **Examples** | 2,996,257 train / 10,000 validation |+| **Unique tool names** | ~1.75M (232 curated + millions from open-source sources) |+| **Tool categories** | 23 on-device domains |+| **Languages** | English (primary), some multilingual queries |+| **License** | Apache 2.0 |++## Schema++Each example has 5 fields:++| Field | Type | Description |+|---|---|---|+| `query` | string | Natural language user request (e.g. *"Set a timer for 10 minutes"*) |+| `tools` | string | JSON array of available tool definitions with names, descriptions, and parameter schemas |+| `answers` | string | JSON array of tool calls the assistant should make (e.g. `[{"name": "set_timer", "arguments": {"time_human": "10 minutes"}}]`) |+| `source` | string | Data source identifier |+| `model` | string | Model used for generation |++### Example++```json+{+ "query": "Turn off the bedroom lights and lock the front door",+ "tools": "[{\"name\":\"control_lights\",\"description\":\"Turn lights on or off in a specific room.\",\"parameters\":{\"room\":{\"type\":\"string\"},\"action\":{\"type\":\"string\"}}},{\"name\":\"lock_door\",\"description\":\"Lock or unlock a smart door lock.\",\"parameters\":{\"door\":{\"type\":\"string\"},\"action\":{\"type\":\"string\"}}}]",+ "answers": "[{\"name\":\"control_lights\",\"arguments\":{\"room\":\"bedroom\",\"action\":\"off\"}},{\"name\":\"lock_door\",\"arguments\":{\"door\":\"front door\",\"action\":\"lock\"}}]",+ "source": "synth-gemini-assistant",+ "model": "gemini-3.1-flash-lite-preview"+}+```++## Data Sources++The dataset is a blend of four sources, each contributing different strengths:++### synth-gemini-assistant (1.10M examples, 36.8%)++Custom synthetic data generated with Gemini Flash Lite using a curated pool of **232 tools across 23 on-device categories**. Each batch is generated with a randomized subset of tools and a specified call type (single, multi, or no-call). Queries span ultra-terse commands (*"timer 5"*), conversational requests, indirect intent (*"it's dark in here"*), and multilingual input.++**Tool categories (232 tools):**++| Category | Tools | Examples |+|---|---|---|+| Time & Productivity | 15 | `set_timer`, `set_alarm`, `create_reminder`, `create_calendar_event` |+| Lists & Notes | 21 | `create_list_item`, `create_note`, `mark_list_item_done`, `pin_note` |+| Messaging | 10 | `send_instant_message`, `make_phone_call`, `send_email`, `send_sms` |+| Device Control | 14 | `set_brightness`, `set_volume`, `toggle_wifi`, `toggle_bluetooth` |+| Media | 14 | `play_music`, `pause_media`, `play_podcast`, `cast_media` |+| Navigation | 9 | `get_directions`, `find_nearby`, `share_location`, `start_navigation` |+| Smart Home | 12 | `set_thermostat`, `control_lights`, `lock_door`, `start_robot_vacuum` |+| Utility | 15 | `evaluate_js`, `web_search`, `translate_text`, `get_weather` |+| Camera & Photos | 10 | `take_photo`, `record_video`, `share_photo`, `search_photos` |+| Fitness & Health | 10 | `start_workout`, `log_water_intake`, `get_step_count`, `log_meal` |+| System | 7 | `toggle_airplane_mode`, `clear_notifications`, `check_battery` |+| Finance | 10 | `send_payment`, `check_balance`, `convert_currency`, `get_stock_price` |+| Reading & News | 10 | `get_news_headlines`, `read_aloud`, `summarize_page`, `open_ebook` |+| Accessibility | 4 | `set_font_size`, `toggle_screen_reader`, `toggle_magnifier` |+| Shopping | 3 | `search_product`, `add_to_cart`, `check_order_status` |+| Social | 10 | `post_status_update`, `like_post`, `comment_on_post`, `follow_user` |+| Ride & Delivery | 6 | `request_ride`, `order_food`, `track_food_delivery` |+| File Management | 7 | `open_file`, `share_file`, `move_file`, `compress_files` |+| Wearable | 10 | `check_heart_rate`, `find_my_phone`, `change_watch_face` |+| Desktop | 13 | `split_screen`, `toggle_vpn`, `print_document`, `kill_process` |+| Browser | 6 | `open_tab`, `bookmark_page`, `clear_browsing_data` |+| Emergency & Safety | 8 | `call_emergency_services`, `share_emergency_location` |+| Digital Wellbeing | 8 | `check_screen_time`, `set_app_timer`, `toggle_bedtime_mode` |++### synth-pleias (1.00M examples, 33.5%)++Reprompted from [Pleias](https://huggingface.co/Pleias) open-source tool-calling data. Original examples were reprompted through Gemini to normalize tool schemas and query styles to match the on-device assistant domain. Provides diverse query patterns and broader tool coverage.++### synth-gcs (832K examples, 27.8%)++Reprompted from [Toucan](https://huggingface.co/datasets/Salesforce/Toucan) (Salesforce) tool-calling data via Gemini. Contributes high multi-call density (68.8% multi-call) and diverse tool combinations from the Salesforce ecosystem.++### xlam-function-calling-60k (58K examples, 1.9%)++Human-curated examples from [xLAM](https://huggingface.co/datasets/Salesforce/xlam-function-calling-60k) (Salesforce). High-quality ground truth with verified tool calls. Provides a quality anchor for the synthetic majority.++## Distribution++### Call Type++```+Single-call (1 tool): 1,064,063 (35.5%)+Multi-call (2+ tools): 1,761,740 (58.8%)+No-call (empty []): 170,454 ( 5.7%)+```++### Calls Per Example++```+0 calls: 170,454 ( 5.7%) ░░░░░░+1 call: 1,064,063 (35.5%) ████████████████████████████████████+2 calls: 1,445,299 (48.2%) ████████████████████████████████████████████████+3 calls: 261,474 ( 8.7%) █████████+4 calls: 38,057 ( 1.3%) █+5+ calls: 16,910 ( 0.6%) ░+```++### Available Tools Per Example++The tool count distribution has been rebalanced from its original skewed form (28% at 3 tools, 27% at 10 tools) to a near-uniform distribution across 1-10 tools. This was done by trimming uncalled tools from over-represented bins while preserving all called tools and answer correctness.++```+ 0 tools: 101,974 ( 3.4%) ███+ 1 tool: 334,077 (11.1%) ███████████+ 2 tools: 361,916 (12.1%) ████████████+ 3 tools: 280,541 ( 9.4%) █████████+ 4 tools: 234,639 ( 7.8%) ████████+ 5 tools: 280,186 ( 9.4%) █████████+ 6 tools: 284,218 ( 9.5%) █████████+ 7 tools: 292,442 ( 9.8%) ██████████+ 8 tools: 281,952 ( 9.4%) █████████+ 9 tools: 278,085 ( 9.3%) █████████+10 tools: 266,224 ( 8.9%) █████████+```++### Call Type by Source++| Source | Single | Multi | No-call |+|---|---|---|---|+| synth-gemini-assistant | 31.2% | 60.0% | 8.8% |+| synth-pleias | 43.8% | 49.4% | 6.8% |+| synth-gcs | 30.6% | 68.8% | 0.6% |+| xlam | 46.0% | 54.0% | 0.0% |++### Field Lengths (characters)++| Field | Mean | Median | P95 |+|---|---|---|---|+| query | 210 | 121 | 670 |+| answers | 210 | 161 | 510 |+| tools | 1,936 | 1,884 | 3,611 |++## Intended Use++This dataset is designed for training **small, fast tool-calling models** (10M-200M parameters) that run on-device. The model receives a user query and a set of available tool definitions, then generates the appropriate JSON tool calls.++**Target architecture:** Encoder-decoder transformer where the encoder processes `[query <tools> tool_definitions]` and the decoder generates `[<tool_call> JSON_tool_calls]`.++**Not intended for:** General-purpose language modeling, chat, or instruction following.++## Known Limitations++- **Synthetic data:** ~98% of examples are LLM-generated. While filtered for schema validity, some argument values may be unrealistic.+- **English-centric:** Queries are predominantly English with occasional multilingual examples.+- **Multi-call with few tools:** The hardest setting (1-3 tools, 2+ calls with different argument values) is underrepresented relative to its difficulty. Targeted generation of these examples is ongoing.+- **No audio:** The dataset is text-only. Audio-to-tool-call requires separate speech data.++## Citation++```bibtex+@misc{cactus-compute-tool-calls-2025,+ title={On-Device Tool Calls: A Large-Scale Synthetic Dataset for Compact Tool-Calling Models},+ author={Cactus Compute},+ year={2025},+ url={https://huggingface.co/datasets/Cactus-Compute/tool-calls}+}+```diff --git a/scripts/rebalance_tools.py b/scripts/rebalance_tools.pynew file mode 100644index 0000000..6e597dd--- /dev/null+++ b/scripts/rebalance_tools.py@@ -0,0 +1,189 @@+#!/usr/bin/env python3+"""Rebalance tool count distribution by trimming over-represented bins.++- 2/3 of 3-tool examples → trimmed to 1 or 2 tools+- 2/3 of 10-tool examples → trimmed to 5-9 tools++Keeps called tools + random subset of uncalled tools to reach target count.+Uploads the rebalanced dataset back to HuggingFace.++Usage:+ python scripts/rebalance_tools.py+ python scripts/rebalance_tools.py --dry-run+"""++import argparse+import json+import random+from collections import Counter++from datasets import Dataset, concatenate_datasets, load_dataset+from tqdm import tqdm++HF_REPO = "Cactus-Compute/tool-calls"+++def _get_called_names(answers_str):+ """Extract set of called tool names from answers JSON."""+ try:+ calls = json.loads(answers_str)+ except (ValueError, TypeError):+ return set()+ if not isinstance(calls, list):+ return set()+ return {c["name"] for c in calls if isinstance(c, dict) and "name" in c}+++def _trim_tools(tools_str, called_names, target_n, rng):+ """Trim tools list to target_n, keeping all called tools + random fillers."""+ try:+ tools = json.loads(tools_str)+ except (ValueError, TypeError):+ return tools_str+ if not isinstance(tools, list):+ return tools_str++ # Separate called vs uncalled+ called = [t for t in tools if isinstance(t, dict) and t.get("name") in called_names]+ uncalled = [t for t in tools if isinstance(t, dict) and t.get("name") not in called_names]++ # If we need fewer tools than called tools, can't trim — skip+ if target_n < len(called):+ return None++ # Pick random uncalled fillers to reach target+ n_fillers = target_n - len(called)+ if n_fillers > len(uncalled):+ return None # not enough uncalled tools++ fillers = rng.sample(uncalled, n_fillers)+ new_tools = called + fillers+ rng.shuffle(new_tools)+ return json.dumps(new_tools, separators=(",", ":"))+++def rebalance(dry_run=False):+ print(f"Loading dataset from {HF_REPO}...")+ ds = load_dataset(HF_REPO, split="train", token=True)+ print(f"Loaded {len(ds):,} examples")++ rng = random.Random(42)++ # Categorize by tool count+ indices_by_n = {}+ for i, ex in enumerate(tqdm(ds, desc="Categorizing")):+ try:+ tools = json.loads(ex["tools"])+ n = len(tools) if isinstance(tools, list) else 0+ except (ValueError, TypeError):+ n = 0+ indices_by_n.setdefault(n, []).append(i)++ print("\nBefore rebalancing:")+ for n in sorted(indices_by_n.keys()):+ if n <= 10:+ print(f" {n:>2} tools: {len(indices_by_n[n]):>10,}")++ # Plan: which indices to modify+ modify_3 = [] # (index, target_n)+ modify_10 = [] # (index, target_n)++ # 2/3 of 3-tool → trim to 1 or 2+ if 3 in indices_by_n:+ pool = list(indices_by_n[3])+ rng.shuffle(pool)+ n_modify = len(pool) * 2 // 3+ for idx in pool[:n_modify]:+ target = rng.choice([1, 2])+ modify_3.append((idx, target))++ # 2/3 of 10-tool → trim to 5-9+ if 10 in indices_by_n:+ pool = list(indices_by_n[10])+ rng.shuffle(pool)+ n_modify = len(pool) * 2 // 3+ for idx in pool[:n_modify]:+ target = rng.randint(5, 9)+ modify_10.append((idx, target))++ all_modifications = modify_3 + modify_10+ print(f"\nModifications planned:")+ print(f" 3-tool → 1 or 2: {len(modify_3):,}")+ print(f" 10-tool → 5-9: {len(modify_10):,}")+ print(f" Total: {len(all_modifications):,}")++ if dry_run:+ # Preview target distribution+ target_counts = Counter()+ for n in sorted(indices_by_n.keys()):+ target_counts[n] = len(indices_by_n[n])+ for idx, target in all_modifications:+ ex = ds[idx]+ try:+ tools = json.loads(ex["tools"])+ orig_n = len(tools)+ except:+ continue+ target_counts[orig_n] -= 1+ target_counts[target] += 1++ print("\nProjected distribution:")+ for n in sorted(target_counts.keys()):+ if n <= 10 and target_counts[n] > 0:+ total = sum(target_counts.values())+ print(f" {n:>2} tools: {target_counts[n]:>10,} ({target_counts[n]/total*100:5.1f}%)")+ return++ # Apply modifications+ mod_map = {idx: target for idx, target in all_modifications}+ modified_indices = set(mod_map.keys())++ new_tools_col = []+ skipped = 0+ for i in tqdm(range(len(ds)), desc="Applying"):+ ex = ds[i]+ if i in modified_indices:+ called = _get_called_names(ex["answers"])+ new_tools = _trim_tools(ex["tools"], called, mod_map[i], rng)+ if new_tools is None:+ new_tools_col.append(ex["tools"]) # keep original+ skipped += 1+ else:+ new_tools_col.append(new_tools)+ else:+ new_tools_col.append(ex["tools"])++ if skipped:+ print(f" Skipped {skipped} examples (couldn't trim)")++ # Build new dataset+ new_ds = ds.remove_columns(["tools"]).add_column("tools", new_tools_col)+ # Reorder columns to match original+ new_ds = new_ds.select_columns(ds.column_names)++ # Verify+ print("\nAfter rebalancing:")+ verify_counts = Counter()+ for ex in tqdm(new_ds, desc="Verifying"):+ try:+ tools = json.loads(ex["tools"])+ n = len(tools) if isinstance(tools, list) else 0+ except:+ n = 0+ verify_counts[n] += 1+ for n in sorted(verify_counts.keys()):+ if n <= 10:+ total = sum(verify_counts.values())+ print(f" {n:>2} tools: {verify_counts[n]:>10,} ({verify_counts[n]/total*100:5.1f}%)")++ # Upload+ print(f"\nUploading to {HF_REPO} (train split)...")+ new_ds.push_to_hub(HF_REPO, split="train", token=True)+ print("Done.")+++if __name__ == "__main__":+ parser = argparse.ArgumentParser()+ parser.add_argument("--dry-run", action="store_true", help="Preview without modifying")+ args = parser.parse_args()+ rebalance(dry_run=args.dry_run)