Skip to content

An agent organization structured as a company — 15+ departments, 125+ skills, each independently installable, citing the standards and regulators that settle the question. Runs in Claude Code and ChatGPT.

21.8 KB
Raw Blame
#!/usr/bin/env python3
"""Emit a standalone vertical repository from the core tree plus a per-vertical config.

D8 resolved the vertical architecture to template-plus-generator rather than overlays, because
productization is planned (D7c) and migrating from overlays to a generator later is worse than
paying for the generator now. What that commits to is one rule, and the rule is the whole cost:

    Generation is one-way. Generated output is never hand-edited.

An edit made downstream is lost on the next emit, silently, which is worse than a fork because the
divergence is invisible. Everything a vertical changes lives in `verticals/<slug>/` or in the core.

Output goes to `dist/headcount-<slug>/`, which is gitignored. Committing the emitted tree would
put a hand-editable copy of it in front of every contributor, which is the one thing the rule above
forbids; the guarantee that the output is sound comes from `--verify`, which emits into a temporary
directory and runs the emitted repository's own checks against it (D36).

What a vertical config may do:

  * add skills           `verticals/<slug>/skills/<dept>/<skill>/SKILL.md`, discovered on disk
  * extend a core skill  `verticals/<slug>/context/<dept>/<skill>.md`, spliced in ahead of `## Never`
  * drop a department    `[emit] exclude_departments`

It may not edit a core skill in place. A core skill that is wrong for every vertical is wrong in the
core, and belongs in a core fix.

Usage:
    build-vertical.py industrial            emit to dist/headcount-industrial/
    build-vertical.py industrial --verify   emit to a temp dir and run its checks; emit nothing
    build-vertical.py --all --verify        the same for every configured vertical
"""
import argparse
import glob
import json
import os
import re
import re
import shutil
import subprocess
import sys
import tempfile
import tomllib

ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
VERTICALS = os.path.join(ROOT, "verticals")

# Copied into the emitted repository verbatim. The three document generators are deliberately not
# among them: the emitted README is written by this script, so a "README is current" check
# downstream would be verifying the wrong generator. Freshness is guaranteed here instead.
EMIT_SCRIPTS = [
    "validate-skills.py",
    "check-provenance.py",
    "check-skill-refs.py",
    "check-us-english.py",
    "check-never-blocks.py",
]

CHECK_ALL = """#!/usr/bin/env bash
# Generated by headcount's build-vertical.py. Do not edit — see README.
# The checks that still mean something in a generated repository. The three that verify generated
# documents are upstream, where the generator that writes them lives.
set -uo pipefail
cd "$(dirname "$0")/.."

fail=0
run() {
  printf '\\n\\033[1m%s\\033[0m\\n' "$1"; shift
  if "$@"; then :; else fail=1; printf '  FAILED\\n'; fi
}

run "Surface map is coherent" \\
  node plugins/executive/skills/agent-hierarchy/scripts/agent-guard.mjs check
run "Skill frontmatter is valid" \\
  python3 scripts/validate-skills.py
run "No third-party license text" \\
  python3 scripts/check-provenance.py
run "Skill references resolve" \\
  python3 scripts/check-skill-refs.py
run "US English spelling" \\
  python3 scripts/check-us-english.py
run "Never blocks are internally consistent" \\
  python3 scripts/check-never-blocks.py
run "Manifests parse" \\
  python3 -c "
import json,glob,sys
bad=[]
for f in ['.claude-plugin/marketplace.json']+glob.glob('plugins/*/.claude-plugin/plugin.json'):
    try: json.load(open(f))
    except Exception as e: bad.append(f'{f}: {e}')
for b in bad: print(' ',b)
print(f'manifests: {len(bad)} problems')
sys.exit(1 if bad else 0)
"

printf '\\n'
if [ "$fail" -eq 0 ]; then printf '\\033[32mAll checks passed.\\033[0m\\n'; else printf '\\033[31mChecks failed.\\033[0m\\n'; fi
exit "$fail"
"""

BANNER = (
    "> **This repository is generated.** It is emitted from "
    "[`{upstream_slug}`]({upstream}) by `scripts/build-vertical.py {slug}`, and every file in it "
    "is overwritten on the next emit. Edits made here are lost silently. Fix the core upstream, or "
    "the vertical's own content under `verticals/{slug}/` upstream, and re-emit."
)

CONTRIBUTING = """# Contributing

{banner}

Nothing is merged into this repository directly. Where a change belongs depends on what is wrong:

| What you found | Where it is fixed |
|---|---|
| A skill that is wrong for every industry | The core, in [`{upstream_slug}`]({upstream}) |
| A skill that is wrong only for {title} | `verticals/{slug}/` upstream |
| A missing {title} capability | A new skill under `verticals/{slug}/skills/` upstream |
| Anything about the generator itself | `scripts/build-vertical.py` upstream |

Open the issue or pull request against the upstream repository. A pull request here cannot be
merged — the next emit would erase it.

## Running the checks

```
./scripts/check-all.sh
```

These are the upstream checks that still mean something downstream. The three that verify generated
documents run upstream, where the generator that writes them lives.
"""


def die(message):
    sys.exit(f"build-vertical: {message}")


def load_config(slug):
    path = os.path.join(VERTICALS, slug, "vertical.toml")
    if not os.path.exists(path):
        die(f"no config at verticals/{slug}/vertical.toml")
    with open(path, "rb") as handle:
        config = tomllib.load(handle)
    for section, key in (("vertical", "slug"), ("vertical", "title"), ("repo", "url")):
        if key not in config.get(section, {}):
            die(f"verticals/{slug}/vertical.toml is missing [{section}] {key}")
    if config["vertical"]["slug"] != slug:
        die(f"verticals/{slug}/vertical.toml declares slug "
            f"{config['vertical']['slug']!r}, which is not its directory name")
    return config


def configured_verticals():
    return sorted(
        os.path.basename(os.path.dirname(p))
        for p in glob.glob(os.path.join(VERTICALS, "*", "vertical.toml"))
    )


def core_departments():
    """Departments on disk, in the marketplace manifest's order.

    The manifest already lists them in reporting order — the chief executive first, then the
    functions beneath — so ordering by it keeps the emitted README reading like the core's without
    a second copy of that order for someone to forget to update. A department on disk but absent
    from the manifest sorts last rather than disappearing.
    """
    found = sorted(
        os.path.basename(os.path.dirname(os.path.dirname(m)))
        for m in glob.glob(os.path.join(ROOT, "plugins", "*", ".claude-plugin", "plugin.json"))
    )
    manifest = json.load(open(os.path.join(ROOT, ".claude-plugin", "marketplace.json"), encoding="utf-8"))
    order = {p["name"]: n for n, p in enumerate(manifest["plugins"])}
    return sorted(found, key=lambda d: (order.get(d, len(order)), d))


def splice(core_text, fragment):
    """Put a vertical's extra material into a core skill ahead of its `## Never` block.

    Appending at the end would land it after the Never list, which every skill closes with, and the
    result reads as though the rules were followed by an afterthought. Ahead of the rules is where
    a section of the same kind would have been written by hand.
    """
    fragment = fragment.strip() + "\n"
    marker = "\n## Never\n"
    at = core_text.find(marker)
    if at == -1:
        return core_text.rstrip("\n") + "\n\n" + fragment
    return core_text[:at] + "\n" + fragment + core_text[at:]


def collect(slug):
    """What the vertical adds, discovered on disk rather than listed in the config.

    Listing them would be a second place to forget: a skill added to the directory and not to the
    list would be silently dropped from every emit, which is the failure `build-readme.py` already
    guards against for departments.
    """
    base = os.path.join(VERTICALS, slug)
    skills = {}
    for path in sorted(glob.glob(os.path.join(base, "skills", "*", "*", "SKILL.md"))):
        skill = os.path.basename(os.path.dirname(path))
        dept = os.path.basename(os.path.dirname(os.path.dirname(path)))
        skills[(dept, skill)] = path
    context = {}
    for path in sorted(glob.glob(os.path.join(base, "context", "*", "*.md"))):
        skill = os.path.basename(path)[: -len(".md")]
        dept = os.path.basename(os.path.dirname(path))
        context[(dept, skill)] = path
    return skills, context


def charter(entry, vertical_title):
    """The agent charter for a department a vertical brings with it."""
    name, title = entry["name"], entry["title"]
    return f"""---
name: {name}
description: {title}. Owns plugins/{name}/** and nothing else. Delegate work in this department's remit here.
---

# {title}

## Why this agent exists

The single owner of `plugins/{name}/**`. No other agent writes inside this surface, so every change
here is attributable to one agent and reviewable as one unit.

This department does not exist in the cross-industry core. It is brought by the {vertical_title}
vertical, because its remit is specific to that industry rather than a thinner version of something
every company has.

## Surface

Writes: `plugins/{name}/**`.
Reads: anything. Commits: nothing; the orchestrator is the sole committer.

## Standard

Skills in this department follow the conventions in `technology:skill-authoring`: the frontmatter
`name` equals the directory name, and the description carries both what the skill does and when to
reach for it.

## Verification this surface implies

- `python3 scripts/validate-skills.py` passes.
- `python3 scripts/check-provenance.py` passes — all content here is original.
- No change outside `plugins/{name}/**`. Needing one means coordinating with that surface's owner.
"""


def emit(slug, config, out):
    vertical, repo = config["vertical"], config["repo"]
    title = vertical["title"]
    name = vertical.get("name", f"headcount-{slug}")
    excluded = set(config.get("emit", {}).get("exclude_departments", []))
    departments = [d for d in core_departments() if d not in excluded]
    unknown = excluded - set(core_departments())
    if unknown:
        die("exclude_departments names department(s) that are not on disk: " + ", ".join(sorted(unknown)))

    # A vertical may bring a department the core has no reason to carry. Industrial did not need
    # this — its skills belong to operations and people, which already exist. Education does: a
    # curriculum function is not a thinner version of anything in a cross-industry core, and
    # filing it under an existing department to avoid the feature would misroute every request.
    added = config.get("department", [])
    for entry in added:
        for field in ("name", "title", "description"):
            if not entry.get(field):
                die(f"verticals/{slug}: a [[department]] entry is missing {field}")
        if entry["name"] in departments:
            die(f"verticals/{slug}: department {entry['name']!r} already exists in the core — "
                f"add skills to it instead of declaring it")
    departments += [e["name"] for e in added]
    new_departments = {e["name"]: e for e in added}

    skills, context = collect(slug)
    for (dept, skill) in list(skills) + list(context):
        if dept not in departments:
            die(f"verticals/{slug} targets department {dept!r}, which is not emitted")
    for (dept, skill) in skills:
        if os.path.exists(os.path.join(ROOT, "plugins", dept, "skills", skill, "SKILL.md")):
            die(f"verticals/{slug}/skills/{dept}/{skill} would shadow a core skill of the same "
                f"name — a vertical adds skills, it does not replace them")
    for (dept, skill) in context:
        if not os.path.exists(os.path.join(ROOT, "plugins", dept, "skills", skill, "SKILL.md")):
            die(f"verticals/{slug}/context/{dept}/{skill}.md extends a skill that does not exist")

    shutil.rmtree(out, ignore_errors=True)
    os.makedirs(out)

    banner = BANNER.format(
        slug=slug, upstream=repo["upstream"],
        upstream_slug=repo["upstream"].rstrip("/").split("github.com/")[-1],
    )

    # Departments, with the vertical's material folded in.
    counts = {}
    for dept in departments:
        dst = os.path.join(out, "plugins", dept)
        if dept in new_departments:
            entry = new_departments[dept]
            os.makedirs(os.path.join(dst, "skills"))
            os.makedirs(os.path.join(dst, ".claude-plugin"))
            with open(os.path.join(dst, ".claude-plugin", "plugin.json"), "w",
                      encoding="utf-8") as handle:
                json.dump({
                    "name": dept,
                    "description": entry["description"],
                    "version": "1.0.0",
                    "author": {"name": vertical.get("author", "Chris Brock")},
                    "repository": repo["url"],
                    "keywords": entry.get("keywords", []),
                }, handle, indent=2)
                handle.write("\n")
        else:
            shutil.copytree(os.path.join(ROOT, "plugins", dept), dst)
        for path in glob.glob(os.path.join(dst, "skills", "*", "SKILL.md")):
            skill = os.path.basename(os.path.dirname(path))
            fragment = context.get((dept, skill))
            if fragment:
                text = open(path, encoding="utf-8").read()
                with open(path, "w", encoding="utf-8") as handle:
                    handle.write(splice(text, open(fragment, encoding="utf-8").read()))
        for (d, skill), path in skills.items():
            if d != dept:
                continue
            # The whole directory, not just SKILL.md — a skill's `references/` is where its
            # sources live, and copying the file alone would ship the pointer without the target.
            shutil.copytree(os.path.dirname(path), os.path.join(dst, "skills", skill))
        manifest_path = os.path.join(dst, ".claude-plugin", "plugin.json")
        manifest = json.load(open(manifest_path, encoding="utf-8"))
        manifest["repository"] = repo["url"]
        manifest["keywords"] = sorted(set(manifest.get("keywords", [])) | set(vertical.get("keywords", [])))
        json.dump(manifest, open(manifest_path, "w", encoding="utf-8"), indent=2)
        open(manifest_path, "a", encoding="utf-8").write("\n")
        counts[dept] = len(glob.glob(os.path.join(dst, "skills", "*", "SKILL.md")))

    # Marketplace manifest, renamed and re-sourced.
    marketplace = json.load(open(os.path.join(ROOT, ".claude-plugin", "marketplace.json"), encoding="utf-8"))
    marketplace["name"] = name
    marketplace["metadata"]["description"] = vertical.get("description", marketplace["metadata"]["description"])
    marketplace["plugins"] = [p for p in marketplace["plugins"] if p["name"] in departments]
    marketplace["plugins"] += [
        {
            "name": e["name"],
            "source": f"./plugins/{e['name']}",
            "description": e["description"],
            "version": "1.0.0",
            "author": {"name": vertical.get("author", "Chris Brock")},
            "keywords": e.get("keywords", []),
            "category": e.get("category", e["name"]),
        }
        for e in added
    ]
    os.makedirs(os.path.join(out, ".claude-plugin"))
    with open(os.path.join(out, ".claude-plugin", "marketplace.json"), "w", encoding="utf-8") as handle:
        json.dump(marketplace, handle, indent=2)
        handle.write("\n")

    # The surface map, charters, and the checks that still apply downstream.
    os.makedirs(os.path.join(out, "docs"))
    surfaces = open(os.path.join(ROOT, "docs", "AGENT-SURFACES.md"), encoding="utf-8").read()
    shutil.copytree(os.path.join(ROOT, ".claude"), os.path.join(out, ".claude"))
    for entry in added:
        name = entry["name"]
        # A roster row, a surface block and a charter, in the same change — the rule the map states
        # for a new department, applied by the generator because nobody is there to apply it by hand.
        surfaces = surfaces.replace(
            "verticals            builder    installed  proposes\n",
            f"{name:<20} builder    installed  autonomous\n", 1)
        surfaces = surfaces.replace(
            "```surface:repo-meta",
            f"```surface:{name}\nplugins/{name}/**\n```\n\n```surface:repo-meta", 1)
        with open(os.path.join(out, ".claude", "agents", f"{name}.md"), "w",
                  encoding="utf-8") as handle:
            handle.write(charter(entry, title))
    # `verticals` and `sources` own nothing in an emitted repository — their inputs stay upstream —
    # so their rows would claim paths that are not here. Dropping them keeps the emitted map true.
    for gone in ("verticals", "sources"):
        surfaces = re.sub(rf"^{gone}\s+builder\s+installed\s+\w+\n", "", surfaces, flags=re.M)
        surfaces = re.sub(rf"```surface:{gone}\n.*?```\n\n", "", surfaces, flags=re.S)
        os.remove(os.path.join(out, ".claude", "agents", f"{gone}.md"))
    open(os.path.join(out, "docs", "AGENT-SURFACES.md"), "w", encoding="utf-8").write(surfaces)
    os.makedirs(os.path.join(out, "scripts"))
    for script in EMIT_SCRIPTS:
        shutil.copy2(os.path.join(ROOT, "scripts", script), os.path.join(out, "scripts"))
    check_all = os.path.join(out, "scripts", "check-all.sh")
    open(check_all, "w", encoding="utf-8").write(CHECK_ALL)
    os.chmod(check_all, 0o755)
    shutil.copytree(os.path.join(ROOT, ".github"), os.path.join(out, ".github"))
    for name_ in ("LICENSE", ".gitattributes", ".gitignore"):
        shutil.copy2(os.path.join(ROOT, name_), os.path.join(out, name_))

    open(os.path.join(out, "CONTRIBUTING.md"), "w", encoding="utf-8").write(
        CONTRIBUTING.format(
            banner=banner, slug=slug, title=title,
            upstream=repo["upstream"],
            upstream_slug=repo["upstream"].rstrip("/").split("github.com/")[-1],
        )
    )
    open(os.path.join(out, "README.md"), "w", encoding="utf-8").write(
        readme(config, departments, counts, banner, name)
    )
    return sum(counts.values()), len(departments)


def readme(config, departments, counts, banner, name):
    vertical, repo = config["vertical"], config["repo"]
    total = sum(counts.values())
    added = config.get("_added", [])
    lines = [
        f"# {name}",
        "",
        vertical.get("tagline", ""),
        "",
        banner,
        "",
        f"{len(departments)} departments, {total} skills. Every department installs on its own, so "
        "only the function you are asking about loads.",
        "",
        "```",
        f"/plugin marketplace add {repo['url'].rstrip('/').split('github.com/')[-1]}",
        f"/plugin install operations@{name}",
        "```",
        "",
        "## What this adds over the core",
        "",
        vertical.get("rationale", "").strip(),
        "",
    ]
    if added:
        lines += ["| Skill | What it covers |", "|---|---|"]
        lines += [f"| `{dept}:{skill}` | {summary} |" for dept, skill, summary in added]
        lines += [""]
    lines += [
        "## Departments",
        "",
        "| Department | Skills |",
        "|---|---|",
    ]
    lines += [f"| `{d}` | {counts[d]} |" for d in departments]
    lines += [
        "",
        "## License",
        "",
        "MIT, the same as the core. See `LICENSE`.",
        "",
    ]
    return "\n".join(lines)


def describe(path):
    front = re.match(r"^---\s*\n(.*?)\n---", open(path, encoding="utf-8").read(), re.S)
    desc = re.search(r"^description:\s*(.*)$", front.group(1), re.M).group(1).strip()
    cut = re.split(r"(?:\.\s+)(?:Also )?[Uu]se (?:this|it)\b", desc)[0].rstrip(" .,—-")
    return cut


def build(slug, verify):
    config = load_config(slug)
    skills, _ = collect(slug)
    config["_added"] = sorted(
        (dept, skill, describe(path)) for (dept, skill), path in skills.items()
    )
    name = config["vertical"].get("name", f"headcount-{slug}")
    out = tempfile.mkdtemp(prefix=f"{name}-") if verify else os.path.join(ROOT, "dist", name)
    try:
        total, departments = emit(slug, config, out)
        if not verify:
            print(f"{name}: {departments} departments, {total} skills → dist/{name}/")
            return 0
        # The emitted surface map governs the emitted tree, and `agent-guard check` reads
        # `git ls-files` to know what that tree contains. A consumer gets this repository by
        # cloning it, so verify it the same way: initialize and stage, or the check either errors
        # out (no repository) or passes vacuously against zero tracked files.
        for command in (["git", "init", "-q", "."], ["git", "add", "-A"]):
            subprocess.run(command, cwd=out, check=True, capture_output=True)
        result = subprocess.run(["./scripts/check-all.sh"], cwd=out, capture_output=True, text=True)
        if result.returncode != 0:
            print(f"  {name}: emitted repository fails its own checks")
            for line in (result.stdout + result.stderr).splitlines():
                if line.strip():
                    print(f"    {line}")
            return 1
        print(f"  {name}: {departments} departments, {total} skills, emitted repository passes its own checks")
        return 0
    finally:
        if verify:
            shutil.rmtree(out, ignore_errors=True)


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("slug", nargs="?", help="vertical to emit")
    parser.add_argument("--all", action="store_true", help="every configured vertical")
    parser.add_argument("--verify", action="store_true",
                        help="emit to a temporary directory and run the emitted checks; write nothing")
    args = parser.parse_args()

    slugs = configured_verticals() if args.all else ([args.slug] if args.slug else [])
    if not slugs:
        die("name a vertical, or pass --all. Configured: " + ", ".join(configured_verticals()))

    failures = sum(build(slug, args.verify) for slug in slugs)
    if args.verify:
        print(f"verticals: {len(slugs)} checked, {failures} problems")
    return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(main())