AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
ci(platform): keep the gitbook branch a one-way copy of master's docs (#15165)
Nicholas Tindle28d65d4a211f
diff --git a/.github/rulesets/gitbook-no-force-push-or-delete.json b/.github/rulesets/gitbook-no-force-push-or-delete.jsonnew file mode 100644index 0000000000..ae5b067eca--- /dev/null+++ b/.github/rulesets/gitbook-no-force-push-or-delete.json@@ -0,0 +1,13 @@+{+ "name": "gitbook: no force-push or delete",+ "target": "branch",+ "enforcement": "active",+ "conditions": {+ "ref_name": {+ "include": ["refs/heads/gitbook"],+ "exclude": []+ }+ },+ "rules": [{ "type": "deletion" }, { "type": "non_fast_forward" }],+ "bypass_actors": []+}diff --git a/.github/rulesets/gitbook-sync-only.json b/.github/rulesets/gitbook-sync-only.jsonnew file mode 100644index 0000000000..11e7946546--- /dev/null+++ b/.github/rulesets/gitbook-sync-only.json@@ -0,0 +1,19 @@+{+ "name": "gitbook: docs sync only",+ "target": "branch",+ "enforcement": "active",+ "conditions": {+ "ref_name": {+ "include": ["refs/heads/gitbook"],+ "exclude": []+ }+ },+ "rules": [{ "type": "creation" }, { "type": "update" }],+ "bypass_actors": [+ {+ "actor_id": 4260672,+ "actor_type": "Integration",+ "bypass_mode": "always"+ }+ ]+}diff --git a/.github/workflows/docs-gitbook-sync.yml b/.github/workflows/docs-gitbook-sync.ymlnew file mode 100644index 0000000000..1f978c2a3b--- /dev/null+++ b/.github/workflows/docs-gitbook-sync.yml@@ -0,0 +1,58 @@+name: Docs - Sync master to gitbook++# GitBook publishes agpt.co/docs from the `gitbook` branch. This keeps that+# branch a one-way, docs-only copy of master's docs/, so the published docs+# match what is deployed and nothing flows back from GitBook into the repo.+#+# Runs as the batch-bot App: the rulesets in .github/rulesets/gitbook-*.json+# restrict updates to `gitbook` to that App (4260672 there is its App ID), and+# GITHUB_TOKEN cannot be put on a ruleset's bypass list. Those files are the+# record of the rulesets; an admin applies them. The daily run is there to+# notice a commit that reached `gitbook` some other way between pushes to master.+on:+ push:+ branches: [master]+ schedule:+ - cron: "23 5 * * *"+ workflow_dispatch:++permissions:+ contents: read++jobs:+ sync:+ # Only master's copy of this workflow may write to gitbook. Without this,+ # a manual run from any other branch would run that branch's script with+ # the App's credentials.+ if: github.repository == 'Significant-Gravitas/AutoGPT' && github.ref == 'refs/heads/master'+ runs-on: ubuntu-latest+ timeout-minutes: 10+ # On the job, not the workflow: a run this job skips must not take the+ # group and displace a pending sync.+ concurrency:+ group: docs-gitbook-sync+ cancel-in-progress: false+ steps:+ - uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3.2.0+ id: app-token+ with:+ app-id: ${{ vars.BATCH_BOT_APP_ID }}+ private-key: ${{ secrets.BATCH_BOT_PRIVATE_KEY }}+ permission-contents: write # fast-forward gitbook+ permission-issues: write # report commits the sync did not write+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ sparse-checkout: .github/workflows/scripts+ persist-credentials: false+ - name: Sync docs/ to gitbook+ run: python3 .github/workflows/scripts/gitbook_sync.py+ env:+ GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}+ SYNC_ACTOR: ${{ steps.app-token.outputs.app-slug }}[bot]+ SOURCE_BRANCH: master+ TARGET_BRANCH: gitbook+ # Mentioned on the report. A team mention only notifies if the App+ # can see the organization's teams, which it cannot today, so the+ # default names a person as well. Set the repository variable to+ # change who is told.+ NOTIFY: ${{ vars.GITBOOK_SYNC_NOTIFY || '@Significant-Gravitas/maintainers @ntindle' }}diff --git a/.github/workflows/scripts/gitbook_sync.py b/.github/workflows/scripts/gitbook_sync.pynew file mode 100644index 0000000000..a098296f5b--- /dev/null+++ b/.github/workflows/scripts/gitbook_sync.py@@ -0,0 +1,482 @@+"""Keep the `gitbook` branch a one-way, docs-only copy of master's docs/.++GitBook publishes agpt.co/docs from the `gitbook` branch. Nothing else should+write to that branch: a ruleset restricts updates to the App this runs as. Each+run that finds a difference writes one commit on top of `gitbook` whose tree is+exactly master's docs/ directory, so the branch only ever fast-forwards.++A commit this script wrote names the master commit it copied in a+`Synced-From:` trailer. If the branch head is anything else, something other+than this script wrote to the branch. The run reports those commits in an+issue, then writes a fresh snapshot on top, which puts master's docs back.+The foreign commits stay in the branch history; nothing is force-pushed.+"""++import hashlib+import http.client+import json+import os+import re+import sys+import time+import urllib.error+import urllib.parse+import urllib.request+from typing import Any, Dict, List, NamedTuple, Optional, Tuple++DOCS_PATH = "docs"+SYNC_TRAILER = "Synced-From"+TRAILER_PATTERN = re.compile(rf"^{SYNC_TRAILER}: ([0-9a-f]{{40}})$", re.MULTILINE)+# How far back to look for the last snapshot when the head is not one.+HISTORY_LIMIT = 100+MAX_REPORTED = 20+ISSUE_TITLE = "Docs sync: the `gitbook` branch needs attention"+ATTEMPTS = 3+RETRIED_STATUSES = {429, 500, 502, 503, 504}+CONTROL_CHARACTERS = re.compile(r"[\x00-\x1f\x7f-\x9f]")+++class SyncError(Exception):+ """The sync cannot go on. Reported in the issue, then the run fails."""+++class ApiError(SyncError):+ def __init__(self, method: str, path: str, status: int, body: str) -> None:+ super().__init__(+ f"{method} {path} -> HTTP {status}: {' '.join(body.split())[:500]}"+ )+ self.status = status+++class GitHub:+ retry_delay = 2.0++ def __init__(self, api_url: str, repo: str, token: str, actor: str) -> None:+ self.api_url = api_url.rstrip("/")+ self.repo = repo+ self.token = token+ # The login the token acts as, e.g. "autogpt-batch-bot[bot]".+ self.actor = actor++ def call(+ self,+ method: str,+ path: str,+ body: Optional[Dict[str, Any]] = None,+ retry: bool = True,+ ) -> Any:+ """Call the API. `retry=False` is for calls that are not safe to+ repeat: a retried POST that had in fact succeeded posts twice."""+ request = urllib.request.Request(+ f"{self.api_url}/repos/{self.repo}{path}",+ data=json.dumps(body).encode() if body is not None else None,+ method=method,+ headers={+ "Authorization": f"Bearer {self.token}",+ "Accept": "application/vnd.github+json",+ "X-GitHub-Api-Version": "2022-11-28",+ "Content-Type": "application/json",+ "User-Agent": "autogpt-gitbook-sync",+ },+ )+ attempts = ATTEMPTS if retry else 1+ for attempt in range(1, attempts + 1):+ try:+ with urllib.request.urlopen(request, timeout=30) as response:+ return json.load(response)+ except urllib.error.HTTPError as e:+ error = ApiError(+ method, path, e.code, e.read().decode(errors="replace")+ )+ if e.code not in RETRIED_STATUSES:+ raise error from e+ except (OSError, http.client.HTTPException, ValueError) as e:+ # Connection errors, timeouts, and replies cut short or not JSON.+ error = ApiError(method, path, 0, f"{type(e).__name__}: {e}")+ if attempt < attempts:+ time.sleep(self.retry_delay * attempt)+ raise error+++class Snapshot(NamedTuple):+ """What a commit holds, as far as the sync cares."""++ sha: str+ docs_tree: Optional[str]+ docs_only: bool+ synced_from: Optional[str]+++def synced_from(message: str) -> Optional[str]:+ """Return the master SHA a snapshot commit says it copied, if any."""+ matches = TRAILER_PATTERN.findall(message)+ return matches[-1] if matches else None+++def branch_head(gh: GitHub, branch: str) -> str:+ return gh.call("GET", f"/git/ref/heads/{branch}")["object"]["sha"]+++def read_snapshot(gh: GitHub, sha: str) -> Snapshot:+ commit = gh.call("GET", f"/git/commits/{sha}")+ entries = gh.call("GET", f"/git/trees/{commit['tree']['sha']}")["tree"]+ docs = [e for e in entries if e["path"] == DOCS_PATH and e["type"] == "tree"]+ return Snapshot(+ sha=sha,+ docs_tree=docs[0]["sha"] if docs else None,+ docs_only=len(entries) == 1 and bool(docs),+ synced_from=synced_from(commit["message"]),+ )+++def is_own_snapshot(gh: GitHub, target: Snapshot) -> bool:+ """True if `target` is exactly the docs/ of the commit its trailer names.++ The trailer alone is not enough: an amended or cherry-picked snapshot+ keeps the trailer while carrying different content.+ """+ if not (target.synced_from and target.docs_only):+ return False+ try:+ named = read_snapshot(gh, target.synced_from)+ except ApiError as e:+ if e.status in (404, 422):+ return False+ raise+ return named.docs_tree == target.docs_tree+++def foreign_commits(gh: GitHub, head: str) -> Tuple[List[Dict[str, Any]], bool]:+ """First-parent commits on top of the last snapshot, newest first.++ `head` is known not to be a snapshot, so it is foreign whatever its+ message says. Following first parents keeps a merge to one entry instead+ of every commit it brought in.++ The second value says whether a snapshot was reached. If not, the list is+ only the newest HISTORY_LIMIT commits and the branch may never have been+ synced, or may have been rewritten since.+ """+ foreign: List[Dict[str, Any]] = []+ sha = head+ for _ in range(HISTORY_LIMIT):+ commit = gh.call("GET", f"/git/commits/{sha}")+ if (+ foreign+ and synced_from(commit["message"])+ and is_own_snapshot(gh, read_snapshot(gh, sha))+ ):+ return foreign, True+ foreign.append(commit)+ if not commit["parents"]:+ break+ sha = commit["parents"][0]["sha"]+ return foreign, False+++def has_written_before(gh: GitHub, branch: str) -> bool:+ """Whether the account this runs as has ever updated the branch.++ Tells the first sync, where the branch simply predates the sync, from a+ branch that lost its snapshots by being rewritten or recreated.+ """+ actor = urllib.parse.quote(gh.actor)+ try:+ return bool(+ gh.call(+ "GET", f"/activity?ref=refs/heads/{branch}&actor={actor}&per_page=1"+ )+ )+ except ApiError as e:+ # Unknown is treated as yes: better a needless report than a silent+ # overwrite of somebody's commits.+ print(f"Could not read {branch} activity: {e}")+ return True+++def ref_updates(gh: GitHub, branch: str, foreign: List[Dict[str, Any]]) -> List[str]:+ """Who moved the branch since the sync last wrote it, as GitHub recorded.++ Commits name their author; this names the account that pushed or merged+ them, which is what shows how they got past the ruleset. Best effort: the+ report goes out without it if the activity log cannot be read.+ """+ try:+ updates = gh.call("GET", f"/activity?ref=refs/heads/{branch}&per_page=100")+ except ApiError as e:+ print(f"Could not read {branch} activity: {e}")+ return []+ logins = [(u.get("actor") or {}).get("login", "unknown") for u in updates]+ if gh.actor in logins:+ since = range(logins.index(gh.actor))+ else:+ shas = {c["sha"] for c in foreign}+ since = [i for i, u in enumerate(updates) if u.get("after") in shas]+ return [+ f"{updates[i]['timestamp'][:10]} {updates[i]['activity_type']} "+ f"by {inert(logins[i])}"+ for i in list(since)[:MAX_REPORTED]+ ]+++def inert(text: str) -> str:+ """Quote text from a commit so it cannot mention, link or format.++ Control characters go too: a carriage return in an author name would end+ the quoted span in the issue and start a new line in the job log, where+ the runner reads a line beginning with "::" as a command.+ """+ flat = " ".join(CONTROL_CHARACTERS.sub(" ", text).split())+ return "`" + flat.replace("`", "'")[:200] + "`"+++def describe(commit: Dict[str, Any]) -> str:+ subject = commit["message"].splitlines()[0] if commit["message"] else ""+ author = commit["author"]["name"]+ committer = commit["committer"]["name"]+ who = author if author == committer else f"{author}, committed by {committer}"+ return (+ f"{commit['sha']} {inert(subject)} "+ f"({inert(who)}, {commit['committer']['date'][:10]})"+ )+++def snapshot_message(source_branch: str, source_sha: str) -> str:+ return (+ f"docs: sync {DOCS_PATH}/ from {source_branch}@{source_sha[:10]}\n\n"+ f"One-way copy for GitBook, written by docs-gitbook-sync.yml.\n"+ f"Do not commit to this branch: change {DOCS_PATH}/ through a PR to dev.\n\n"+ f"{SYNC_TRAILER}: {source_sha}"+ )+++def write_snapshot(+ gh: GitHub, branch: str, parent: str, source_branch: str, source: Snapshot+) -> str:+ tree = gh.call(+ "POST",+ "/git/trees",+ {+ "tree": [+ {+ "path": DOCS_PATH,+ "mode": "040000",+ "type": "tree",+ "sha": source.docs_tree,+ }+ ]+ },+ )+ # No author or committer: GitHub then signs the commit as the App.+ commit = gh.call(+ "POST",+ "/git/commits",+ {+ "message": snapshot_message(source_branch, source.sha),+ "tree": tree["sha"],+ "parents": [parent],+ },+ )+ # Never forced: if the branch moved since `parent` was read, this fails and+ # the next run starts over from the new head.+ gh.call(+ "PATCH", f"/git/refs/heads/{branch}", {"sha": commit["sha"], "force": False}+ )+ return commit["sha"]+++def foreign_section(+ gh: GitHub,+ branch: str,+ source_branch: str,+ foreign: List[Dict[str, Any]],+ reached_snapshot: bool,+) -> List[str]:+ lines = [+ f"`{branch}` is a one-way copy of `{DOCS_PATH}/` on `{source_branch}`, "+ "written only by `.github/workflows/docs-gitbook-sync.yml`. "+ "These commits were written to it by something else:",+ "",+ *(f"- {describe(c)}" for c in foreign[:MAX_REPORTED]),+ ]+ if not reached_snapshot:+ lines.append(+ "- ...and the history below them. None of the sync's own commits "+ f"is within {HISTORY_LIMIT} commits of the head, so the branch was "+ "rewritten, recreated, or had another branch pushed over it."+ )+ elif len(foreign) > MAX_REPORTED:+ lines.append(f"- ...and {len(foreign) - MAX_REPORTED} more")+ updates = ref_updates(gh, branch, foreign)+ if updates:+ lines += ["", "How they got there:", "", *(f"- {u}" for u in updates)]+ return lines + [+ "",+ f"This run now puts `{source_branch}`'s docs back on top of them, which "+ "takes these changes off the published site. The commits stay in the "+ "branch history. If that write fails, a second report follows here.",+ "",+ "To keep a change, open a PR against `dev` with it. Then check how the "+ f"commit got past the ruleset that restricts updates to `{branch}`.",+ ]+++def failed_section(branch: str, source_branch: str, error: SyncError) -> List[str]:+ run = os.environ.get("GITHUB_RUN_ID")+ where = (+ f" ([run]({os.environ.get('GITHUB_SERVER_URL', 'https://github.com')}/"+ f"{os.environ.get('GITHUB_REPOSITORY')}/actions/runs/{run}))"+ if run+ else ""+ )+ return [+ f"The docs sync could not update `{branch}`{where}:",+ "",+ f" {error}",+ "",+ f"The published docs will not follow `{source_branch}` until this is fixed.",+ ]+++def report(gh: GitHub, lines: List[str], notify: str, key: str) -> str:+ """File the report on the sync's open issue, or open one.++ Only an issue this account opened is reused: the repository is public, so+ anyone can open one with the same title.++ `key` names what is being reported. The sync runs daily and on every push+ to master, so a problem that persists would otherwise be posted again by+ every run. A report whose key is already on the open issue is not repeated.+ """+ marker = f"<!-- gitbook-sync:{key} -->"+ body = "\n".join(lines + (["", f"cc {notify}"] if notify else []) + ["", marker])+ creator = urllib.parse.quote(gh.actor)+ mine = gh.call("GET", f"/issues?state=open&creator={creator}&per_page=100")+ existing = [+ i for i in mine if i["title"] == ISSUE_TITLE and "pull_request" not in i+ ]+ if not existing:+ created = gh.call(+ "POST", "/issues", {"title": ISSUE_TITLE, "body": body}, retry=False+ )+ return created["html_url"]+ issue = existing[0]+ comments = gh.call("GET", f"/issues/{issue['number']}/comments?per_page=100")+ said = [issue.get("body") or ""] + [c.get("body") or "" for c in comments]+ if any(marker in text for text in said):+ print(f"Already reported on {issue['html_url']}; not repeating it.")+ else:+ gh.call(+ "POST", f"/issues/{issue['number']}/comments", {"body": body}, retry=False+ )+ return issue["html_url"]+++def summarize(line: str) -> None:+ print(line)+ summary_path = os.environ.get("GITHUB_STEP_SUMMARY")+ if summary_path:+ with open(summary_path, "a", encoding="utf-8") as f:+ f.write(line + "\n")+++def sync(+ gh: GitHub, source_branch: str, target_branch: str, notify: str, dry_run: bool+) -> int:+ source = read_snapshot(gh, branch_head(gh, source_branch))+ if not source.docs_tree:+ raise SyncError(+ f"{source_branch}@{source.sha} has no {DOCS_PATH}/ directory to copy"+ )++ target = read_snapshot(gh, branch_head(gh, target_branch))+ own = is_own_snapshot(gh, target)+ if own and target.docs_tree == source.docs_tree:+ summarize(+ f"`{target_branch}` already holds `{DOCS_PATH}/` as of "+ f"`{source_branch}@{source.sha[:10]}`. Nothing to do."+ )+ return 0++ found: List[str] = []+ if not own:+ foreign, reached_snapshot = foreign_commits(gh, target.sha)+ if reached_snapshot or has_written_before(gh, target_branch):+ for commit in foreign[:MAX_REPORTED]:+ print(f"::warning::Not written by the sync: {describe(commit)}")+ found = foreign_section(+ gh, target_branch, source_branch, foreign, reached_snapshot+ )+ else:+ print(f"The sync has never written {target_branch}: first sync.")++ if dry_run:+ summarize(+ f"Dry run: would write `{DOCS_PATH}/` from `{source_branch}@"+ f"{source.sha[:10]}` on top of `{target_branch}@{target.sha[:10]}`"+ + (" and report commits the sync did not write." if found else ".")+ )+ return 0++ if found:+ # Reported before the branch moves. Once the snapshot is on top, the+ # next run sees nothing wrong, so a report that failed after the write+ # would never be sent.+ url = report(gh, found, notify, key=f"foreign:{target.sha}")+ summarize(f"Reported commits the sync did not write: {url}")+ written = write_snapshot(gh, target_branch, target.sha, source_branch, source)+ summarize(+ f"Wrote `{DOCS_PATH}/` from `{source_branch}@{source.sha[:10]}` to "+ f"`{target_branch}` as `{written[:10]}`."+ )+ if found:+ print("Failing the run so that the foreign commits are noticed.")+ return 1+ return 0+++def run(+ gh: GitHub, source_branch: str, target_branch: str, notify: str, dry_run: bool+) -> int:+ try:+ return sync(gh, source_branch, target_branch, notify, dry_run)+ except SyncError as e:+ print(f"Error: {e}")+ if dry_run:+ return 1+ # A sync that cannot run leaves the site stale with nobody watching,+ # so say so where the foreign commits are reported.+ try:+ report(+ gh,+ failed_section(target_branch, source_branch, e),+ notify,+ key="failed:" + hashlib.sha1(str(e).encode()).hexdigest()[:12],+ )+ except SyncError as report_error:+ print(f"Could not report the failure: {report_error}")+ return 1+++def main() -> int:+ try:+ gh = GitHub(+ os.environ.get("GITHUB_API_URL", "https://api.github.com"),+ os.environ["GITHUB_REPOSITORY"],+ os.environ["GITHUB_TOKEN"],+ os.environ["SYNC_ACTOR"],+ )+ except KeyError as e:+ print(f"Error: Missing required environment variable: {e}")+ return 1+ return run(+ gh,+ source_branch=os.environ.get("SOURCE_BRANCH", "master"),+ target_branch=os.environ.get("TARGET_BRANCH", "gitbook"),+ notify=os.environ.get("NOTIFY", ""),+ dry_run=os.environ.get("DRY_RUN", "").lower() in ("1", "true"),+ )+++if __name__ == "__main__":+ sys.exit(main())diff --git a/.github/workflows/scripts/test_gitbook_sync.py b/.github/workflows/scripts/test_gitbook_sync.pynew file mode 100644index 0000000000..1dfbf01eb7--- /dev/null+++ b/.github/workflows/scripts/test_gitbook_sync.py@@ -0,0 +1,606 @@+import contextlib+import io+import json+import os+import re+import unittest+import urllib.error+import urllib.parse+from typing import Any, Dict, List, Optional+from unittest.mock import MagicMock, patch++import gitbook_sync+from gitbook_sync import ISSUE_TITLE, ApiError, GitHub, snapshot_message, synced_from++BOT = "docs-sync[bot]"+PERSON = {+ "name": "Some Editor",+ "email": "[email protected]",+ "date": "2026-10-03T00:00:00Z",+}+GITBOOK_BOT = {+ "name": "gitbook-bot",+ "email": "[email protected]",+ "date": "2026-10-03T00:00:00Z",+}+++class FakeGitHub:+ """The slice of the GitHub API the sync uses, over in-memory objects."""++ repo = "example/repo"+ actor = BOT++ def __init__(self) -> None:+ self.commits: Dict[str, Dict[str, Any]] = {}+ self.trees: Dict[str, List[Dict[str, str]]] = {}+ self.refs: Dict[str, str] = {}+ self.issues: List[Dict[str, Any]] = []+ self.comments: List[Dict[str, Any]] = []+ self.activity: List[Dict[str, Any]] = []+ self.writes: List[str] = []+ self.reject_ref_update = False+ self.reject_issue_writes = False+ self._next = 0++ def _sha(self) -> str:+ self._next += 1+ return f"{self._next:040x}"++ def tree(self, **entries: str) -> str:+ """Add a root tree; `docs="x"` makes a docs/ directory with content x."""+ sha = self._sha()+ self.trees[sha] = [+ {"path": path, "type": "tree", "sha": f"{content:0>40}"}+ for path, content in entries.items()+ ]+ return sha++ def commit(+ self,+ tree: str,+ parents: List[str],+ message: str = "a change",+ committer: Optional[Dict[str, str]] = None,+ ) -> str:+ sha = self._sha()+ self.commits[sha] = {+ "sha": sha,+ "message": message,+ "tree": {"sha": tree},+ "parents": [{"sha": p} for p in parents],+ "author": PERSON,+ "committer": committer or PERSON,+ }+ return sha++ def move(+ self, branch: str, sha: str, login: Optional[str], kind: str = "push"+ ) -> None:+ """Point a branch at a commit the way GitHub records it."""+ self.refs[branch] = sha+ self.activity.insert(+ 0,+ {+ "timestamp": "2026-10-03T00:00:00Z",+ "activity_type": kind,+ "actor": {"login": login} if login else None,+ "ref": f"refs/heads/{branch}",+ "after": sha,+ },+ )++ def open_issue(self, title: str, body: str, login: str) -> Dict[str, Any]:+ issue = {+ "number": len(self.issues) + 1,+ "title": title,+ "body": body,+ "state": "open",+ "user": {"login": login},+ "html_url": f"https://example.com/issues/{len(self.issues) + 1}",+ }+ self.issues.append(issue)+ return issue++ def call(+ self,+ method: str,+ path: str,+ body: Optional[Dict[str, Any]] = None,+ retry: bool = True,+ ) -> Any:+ if method != "GET":+ self.writes.append(f"{method} {path}")+ path, _, query = path.partition("?")+ params = dict(urllib.parse.parse_qsl(query))+ if m := re.fullmatch(r"/git/ref/heads/(.+)", path):+ if m[1] not in self.refs:+ raise ApiError(method, path, 404, "Not Found")+ return {"object": {"sha": self.refs[m[1]]}}+ if m := re.fullmatch(r"/git/commits/(.+)", path):+ if m[1] not in self.commits:+ raise ApiError(method, path, 404, "Not Found")+ return self.commits[m[1]]+ if m := re.fullmatch(r"/git/trees/(.+)", path):+ return {"tree": self.trees[m[1]]}+ if (method, path) == ("POST", "/git/trees"):+ sha = self._sha()+ self.trees[sha] = [+ {"path": e["path"], "type": e["type"], "sha": e["sha"]}+ for e in body["tree"]+ ]+ return {"sha": sha}+ if (method, path) == ("POST", "/git/commits"):+ return {"sha": self.commit(body["tree"], body["parents"], body["message"])}+ if m := re.fullmatch(r"/git/refs/heads/(.+)", path):+ if self.reject_ref_update or body["force"]:+ raise ApiError(method, path, 422, "Update is not a fast forward")+ self.move(m[1], body["sha"], self.actor)+ return {}+ if path == "/activity":+ return [+ a+ for a in self.activity+ if a["ref"] == params["ref"]+ and (+ "actor" not in params+ or (a["actor"] or {}).get("login") == params["actor"]+ )+ ]+ if (method, path) == ("GET", "/issues"):+ return [+ i+ for i in self.issues+ if i["state"] == "open" and i["user"]["login"] == params["creator"]+ ]+ if self.reject_issue_writes and path.startswith("/issues"):+ raise ApiError(method, path, 503, "Service Unavailable")+ if (method, path) == ("POST", "/issues"):+ return self.open_issue(body["title"], body["body"], self.actor)+ if m := re.fullmatch(r"/issues/(\d+)/comments", path):+ if method == "GET":+ return [c for c in self.comments if c["issue"] == int(m[1])]+ self.comments.append({"issue": int(m[1]), "body": body["body"]})+ return {}+ raise AssertionError(f"unexpected call: {method} {path}")+++def run(+ gh: FakeGitHub, dry_run: bool = False, output: Optional[io.StringIO] = None+) -> int:+ # Keep the script's workflow commands and job summary out of the CI job+ # that runs these tests.+ quiet = {k: v for k, v in os.environ.items() if not k.startswith("GITHUB_")}+ with patch.dict(os.environ, quiet, clear=True):+ with contextlib.redirect_stdout(output or io.StringIO()):+ return gitbook_sync.run(+ gh, "master", "gitbook", notify="@example/team", dry_run=dry_run+ )+++class SyncedFromTests(unittest.TestCase):+ def test_reads_trailer(self):+ sha = "a" * 40+ self.assertEqual(synced_from(snapshot_message("master", sha)), sha)++ def test_ignores_mentions_that_are_not_a_trailer_line(self):+ self.assertIsNone(synced_from(f"Revert the commit Synced-From: {'a' * 40}"))+ self.assertIsNone(synced_from("Synced-From: not-a-sha"))+++class SyncTests(unittest.TestCase):+ def setUp(self):+ self.gh = FakeGitHub()+ # master carries code and docs; gitbook starts as an old full copy+ # with a few commits of its own.+ self.master = self.gh.commit(self.gh.tree(docs="1", autogpt_platform="c"), [])+ self.gh.move("master", self.master, "releaser")+ older = self.gh.commit(self.gh.tree(docs="0", autogpt_platform="a"), [])+ self.legacy = self.gh.commit(+ self.gh.tree(docs="0", autogpt_platform="b"), [older]+ )+ self.gh.move("gitbook", self.legacy, "someone")++ def head(self) -> Dict[str, Any]:+ return self.gh.commits[self.gh.refs["gitbook"]]++ def advance_master(self, docs: str) -> str:+ self.master = self.gh.commit(+ self.gh.tree(docs=docs, autogpt_platform="c"), [self.master]+ )+ self.gh.move("master", self.master, "releaser")+ return self.master++ def stray(self, message: str = "a stray edit", docs: str = "edited") -> str:+ """Somebody else commits on top of gitbook."""+ sha = self.gh.commit(+ self.gh.tree(docs=docs), [self.gh.refs["gitbook"]], message+ )+ self.gh.move("gitbook", sha, "someone")+ return sha++ def test_first_sync_writes_docs_only_snapshot_without_reporting(self):+ self.assertEqual(run(self.gh), 0)++ head = self.head()+ self.assertEqual([p["sha"] for p in head["parents"]], [self.legacy])+ self.assertEqual(synced_from(head["message"]), self.master)+ self.assertEqual(+ self.gh.trees[head["tree"]["sha"]],+ [{"path": "docs", "type": "tree", "sha": f"{'1':0>40}"}],+ )+ self.assertEqual(self.gh.issues, [])++ def test_nothing_to_do_when_snapshot_is_current(self):+ run(self.gh)+ self.gh.writes.clear()++ self.assertEqual(run(self.gh), 0)++ self.assertEqual(self.gh.writes, [])++ def test_master_moving_without_docs_changes_writes_nothing(self):+ run(self.gh)+ self.advance_master(docs="1")+ self.gh.writes.clear()++ self.assertEqual(run(self.gh), 0)++ self.assertEqual(self.gh.writes, [])++ def test_docs_change_on_master_fast_forwards_gitbook(self):+ run(self.gh)+ previous = self.gh.refs["gitbook"]+ self.advance_master(docs="2")++ self.assertEqual(run(self.gh), 0)++ head = self.head()+ self.assertEqual([p["sha"] for p in head["parents"]], [previous])+ self.assertEqual(synced_from(head["message"]), self.master)+ self.assertEqual(self.gh.issues, [])++ def test_foreign_commit_is_reported_and_docs_are_restored(self):+ run(self.gh)+ snapshot = self.gh.refs["gitbook"]+ foreign = self.gh.commit(+ self.gh.tree(docs="edited"),+ [snapshot],+ "GITBOOK-101: tweak install steps",+ committer=GITBOOK_BOT,+ )+ self.gh.move("gitbook", foreign, "gitbook-com[bot]")++ self.assertEqual(run(self.gh), 1)++ head = self.head()+ self.assertEqual([p["sha"] for p in head["parents"]], [foreign])+ self.assertEqual(self.gh.trees[head["tree"]["sha"]][0]["sha"], f"{'1':0>40}")+ self.assertEqual(len(self.gh.issues), 1)+ issue = self.gh.issues[0]+ self.assertEqual(issue["title"], ISSUE_TITLE)+ self.assertIn(foreign, issue["body"])+ self.assertIn("GITBOOK-101: tweak install steps", issue["body"])+ self.assertIn("committed by gitbook-bot", issue["body"])+ self.assertIn("push by `gitbook-com[bot]`", issue["body"])+ self.assertIn("cc @example/team", issue["body"])+ self.assertNotIn(snapshot, issue["body"])+ self.assertNotIn(BOT, issue["body"])++ def test_report_is_filed_before_the_branch_moves(self):+ """Otherwise a failed report could never be retried."""+ run(self.gh)+ stray = self.stray()+ self.gh.reject_issue_writes = True++ self.assertEqual(run(self.gh), 1)+ self.assertEqual(self.gh.refs["gitbook"], stray)++ self.gh.reject_issue_writes = False+ self.assertEqual(run(self.gh), 1)+ self.assertNotEqual(self.gh.refs["gitbook"], stray)+ self.assertIn("a stray edit", self.gh.issues[0]["body"])++ def test_second_report_comments_on_the_open_issue(self):+ run(self.gh)+ for subject in ("first stray", "second stray"):+ self.stray(subject)+ run(self.gh)++ self.assertEqual(len(self.gh.issues), 1)+ self.assertEqual(len(self.gh.comments), 1)+ self.assertIn("second stray", self.gh.comments[0]["body"])+ self.assertNotIn("first stray", self.gh.comments[0]["body"])++ def test_issue_opened_by_someone_else_is_not_reused(self):+ run(self.gh)+ self.gh.open_issue(ISSUE_TITLE, "not from the sync", "an-outsider")+ self.stray()++ run(self.gh)++ self.assertEqual(self.gh.comments, [])+ self.assertEqual(+ [i["user"]["login"] for i in self.gh.issues], ["an-outsider", BOT]+ )++ def test_merge_is_reported_as_one_commit(self):+ run(self.gh)+ snapshot = self.gh.refs["gitbook"]+ merged_in = self.gh.commit(self.gh.tree(docs="dev"), [], "work from dev")+ merge = self.gh.commit(+ self.gh.tree(docs="dev", autogpt_platform="d"),+ [snapshot, merged_in],+ "Merge branch 'dev' into gitbook",+ )+ self.gh.move("gitbook", merge, "someone", "pr_merge")++ run(self.gh)++ body = self.gh.issues[0]["body"]+ self.assertIn(merge, body)+ self.assertNotIn(merged_in, body)+ self.assertIn("pr_merge by `someone`", body)++ def test_snapshot_with_changed_content_is_foreign(self):+ """An amended snapshot keeps the trailer but not the content."""+ run(self.gh)+ amended = self.stray(snapshot_message("master", self.master))++ run(self.gh)++ self.assertIn(amended, self.gh.issues[0]["body"])++ def test_snapshot_naming_a_missing_commit_is_foreign(self):+ run(self.gh)+ forged = self.stray(snapshot_message("master", "f" * 40), docs="1")++ run(self.gh)++ self.assertIn(forged, self.gh.issues[0]["body"])++ def test_trailer_below_the_head_does_not_hide_a_foreign_commit(self):+ run(self.gh)+ amended = self.stray(snapshot_message("master", self.master))+ on_top = self.stray("on top of it")++ run(self.gh)++ body = self.gh.issues[0]["body"]+ self.assertIn(on_top, body)+ self.assertIn(amended, body)++ def test_branch_replaced_after_a_sync_is_reported(self):+ """Recreated or force-pushed: no snapshot left to compare against."""+ run(self.gh)+ self.gh.move("gitbook", self.master, "an-admin", "force_push")++ self.assertEqual(run(self.gh), 1)++ body = self.gh.issues[0]["body"]+ self.assertIn(self.master, body)+ self.assertIn("rewritten, recreated", body)+ self.assertIn("force_push by `an-admin`", body)+ self.assertEqual(synced_from(self.head()["message"]), self.master)++ def test_more_foreign_commits_than_the_history_limit_are_reported(self):+ run(self.gh)+ for n in range(5):+ self.stray(f"stray {n}")++ with patch.object(gitbook_sync, "HISTORY_LIMIT", 3):+ self.assertEqual(run(self.gh), 1)++ body = self.gh.issues[0]["body"]+ self.assertIn("stray 4", body)+ self.assertIn("rewritten, recreated", body)++ def test_commit_text_cannot_mention_or_format(self):+ run(self.gh)+ self.stray("fix `x` @someone [link](https://example.com)")++ run(self.gh)++ self.assertIn(+ "`fix 'x' @someone [link](https://example.com)`", self.gh.issues[0]["body"]+ )++ def test_control_characters_in_a_name_cannot_start_a_new_line(self):+ """A carriage return would begin a workflow command in the job log."""+ run(self.gh)+ stray = self.gh.commit(+ self.gh.tree(docs="edited"), [self.gh.refs["gitbook"]], "a stray edit"+ )+ self.gh.commits[stray]["author"] = dict(+ PERSON, name="Eve\r::error::forged\r\r@victim [l](https://example.com)"+ )+ self.gh.move("gitbook", stray, "someone")++ output = io.StringIO()+ run(self.gh, output=output)++ self.assertNotIn("\r", output.getvalue())+ self.assertNotIn("\r", self.gh.issues[0]["body"])+ self.assertIn(+ "`Eve ::error::forged @victim [l](https://example.com), "+ "committed by Some Editor`",+ self.gh.issues[0]["body"],+ )++ def test_bot_activity_on_another_branch_is_not_a_previous_sync(self):+ self.gh.move("batch/rollup", self.master, BOT, "force_push")++ self.assertEqual(run(self.gh), 0)++ self.assertEqual(self.gh.issues, [])++ def test_pull_request_with_the_report_title_is_not_reused(self):+ """The issues endpoint lists pull requests too."""+ run(self.gh)+ self.gh.open_issue(ISSUE_TITLE, "a pull request", BOT)["pull_request"] = {}+ self.stray()++ run(self.gh)++ self.assertEqual(self.gh.comments, [])+ self.assertEqual(len(self.gh.issues), 2)++ def test_persistent_failure_is_reported_once(self):+ self.gh.reject_ref_update = True++ for _ in range(5):+ self.assertEqual(run(self.gh), 1)++ self.assertEqual(len(self.gh.issues), 1)+ self.assertEqual(self.gh.comments, [])++ def test_foreign_commits_with_a_failing_write_are_reported_once(self):+ run(self.gh)+ self.stray()+ self.gh.reject_ref_update = True++ for _ in range(5):+ self.assertEqual(run(self.gh), 1)++ self.assertEqual(len(self.gh.issues), 1)+ self.assertIn("a stray edit", self.gh.issues[0]["body"])+ self.assertEqual(len(self.gh.comments), 1)+ self.assertIn("could not update `gitbook`", self.gh.comments[0]["body"])++ def test_issue_posts_are_not_retried(self):+ """A retried POST that had succeeded would open a second issue."""+ run(self.gh)+ self.stray()+ calls = []+ real_call = self.gh.call++ def recording(method, path, body=None, retry=True):+ calls.append((method, path.partition("?")[0], retry))+ return real_call(method, path, body, retry)++ self.gh.call = recording+ run(self.gh)++ self.assertIn(("POST", "/issues", False), calls)+ self.assertIn(("PATCH", "/git/refs/heads/gitbook", True), calls)++ def test_activity_without_an_actor_does_not_stop_the_restore(self):+ run(self.gh)+ stray = self.stray()+ self.gh.activity[0]["actor"] = None++ self.assertEqual(run(self.gh), 1)++ self.assertNotEqual(self.gh.refs["gitbook"], stray)+ self.assertIn("push by `unknown`", self.gh.issues[0]["body"])++ def test_dry_run_writes_nothing(self):+ self.assertEqual(run(self.gh, dry_run=True), 0)++ self.assertEqual(self.gh.writes, [])+ self.assertEqual(self.gh.refs["gitbook"], self.legacy)++ def test_source_without_docs_is_reported_and_fails(self):+ self.gh.move(+ "master",+ self.gh.commit(self.gh.tree(autogpt_platform="c"), [self.master]),+ "releaser",+ )++ self.assertEqual(run(self.gh), 1)++ self.assertEqual(self.gh.refs["gitbook"], self.legacy)+ self.assertIn("has no docs/ directory", self.gh.issues[0]["body"])++ def test_missing_target_branch_is_reported_and_fails(self):+ del self.gh.refs["gitbook"]++ self.assertEqual(run(self.gh), 1)++ self.assertIn("could not update `gitbook`", self.gh.issues[0]["body"])+ self.assertIn("HTTP 404", self.gh.issues[0]["body"])++ def test_rejected_ref_update_is_reported_and_fails(self):+ """A sync that cannot write must not just leave the site stale."""+ self.gh.reject_ref_update = True++ self.assertEqual(run(self.gh), 1)++ self.assertEqual(self.gh.refs["gitbook"], self.legacy)+ self.assertEqual(len(self.gh.issues), 1)+ self.assertIn("could not update `gitbook`", self.gh.issues[0]["body"])+ self.assertIn("Update is not a fast forward", self.gh.issues[0]["body"])+++def response(payload: Any) -> MagicMock:+ opened = MagicMock()+ opened.__enter__.return_value = io.BytesIO(json.dumps(payload).encode())+ return opened+++def http_error(status: int) -> urllib.error.HTTPError:+ return urllib.error.HTTPError(+ "url", status, "error", {}, io.BytesIO(b"upstream\nbroke")+ )+++class GitHubCallTests(unittest.TestCase):+ def setUp(self):+ self.gh = GitHub("https://api.example.com", "example/repo", "token", BOT)+ self.gh.retry_delay = 0++ def test_retries_a_server_error(self):+ with patch("urllib.request.urlopen") as urlopen:+ urlopen.side_effect = [http_error(502), response({"ok": True})]++ self.assertEqual(self.gh.call("GET", "/git/ref/heads/master"), {"ok": True})++ self.assertEqual(urlopen.call_count, 2)+ self.assertEqual(+ urlopen.call_args.args[0].full_url,+ "https://api.example.com/repos/example/repo/git/ref/heads/master",+ )++ def test_does_not_retry_a_rejection(self):+ with patch("urllib.request.urlopen") as urlopen:+ urlopen.side_effect = [http_error(422), response({})]++ with self.assertRaises(ApiError) as raised:+ self.gh.call("PATCH", "/git/refs/heads/gitbook", {"sha": "x"})++ self.assertEqual(urlopen.call_count, 1)+ self.assertEqual(raised.exception.status, 422)+ self.assertIn("upstream broke", str(raised.exception))++ def test_reply_that_is_not_json_becomes_an_api_error(self):+ broken = MagicMock()+ broken.__enter__.return_value = io.BytesIO(b"")+ with patch("urllib.request.urlopen") as urlopen:+ urlopen.return_value = broken++ with self.assertRaises(ApiError) as raised:+ self.gh.call("GET", "/git/ref/heads/master")++ self.assertEqual(raised.exception.status, 0)++ def test_unrepeatable_call_is_sent_once(self):+ with patch("urllib.request.urlopen") as urlopen:+ urlopen.side_effect = [http_error(502), response({})]++ with self.assertRaises(ApiError):+ self.gh.call("POST", "/issues", {"title": "t"}, retry=False)++ self.assertEqual(urlopen.call_count, 1)++ def test_network_failure_becomes_an_api_error(self):+ with patch("urllib.request.urlopen") as urlopen:+ urlopen.side_effect = urllib.error.URLError("connection reset")++ with self.assertRaises(ApiError):+ self.gh.call("GET", "/git/ref/heads/master")++ self.assertEqual(urlopen.call_count, gitbook_sync.ATTEMPTS)+++if __name__ == "__main__":+ unittest.main()diff --git a/docs/AGENTS.md b/docs/AGENTS.mdindex 919bc33683..61c4732f84 100644--- a/docs/AGENTS.md+++ b/docs/AGENTS.md@@ -9,6 +9,8 @@When unsure, write the document in `docs/engineering/`. Listing a page in a `SUMMARY.md` publishes it, so do that only for a page meant for the site.+The site follows `master`. `.github/workflows/docs-gitbook-sync.yml` copies `master`'s `docs/` to the `gitbook` branch, and GitBook publishes that branch. Change docs through a pull request to `dev`. Never commit to `gitbook`, open a pull request against it, or edit pages in GitBook: the next copy overwrites anything that isn't on `master`.+Every Markdown page under `docs/platform/` must be listed in `docs/platform/SUMMARY.md`: every `.md` file other than `SUMMARY.md` itself and GitBook's own files under a `.gitbook/` directory. A `SUMMARY.md` may list only pages inside its own folder. `.github/workflows/scripts/test_docs_layout.py` checks those two things on every pull request. It cannot tell whether a listed page is meant for the public; that is for the author and the reviewer to judge.## Block Documentation Manual Sectionsdiff --git a/docs/content/contribute/index.md b/docs/content/contribute/index.mdindex 3b458a04df..42f2b7fdc3 100644--- a/docs/content/contribute/index.md+++ b/docs/content/contribute/index.md@@ -1,35 +1,33 @@# Contributing to the Docs-We welcome contributions to our documentation! Our docs are hosted on GitBook and synced with GitHub.+We welcome contributions to our documentation! Our docs are hosted on GitBook and published from GitHub.## How It Works-- Documentation lives in the `docs/` directory on the `gitbook` branch-- GitBook automatically syncs changes from GitHub-- You can edit docs directly on GitHub or locally+- Documentation lives in the `docs/` directory and changes like any other code: through a pull request to `dev`+- The published docs match `master`. When a release reaches `master`, its `docs/` directory is copied to the `gitbook` branch, and GitBook publishes that+- The `gitbook` branch belongs to that copy. Don't open pull requests against it, and don't edit pages in GitBook: the next copy overwrites anything that isn't in `master`## Editing Docs Locally-1. Clone the repository and switch to the gitbook branch:+1. Clone the repository and create a branch from `dev`:```shellgit clone https://github.com/Significant-Gravitas/AutoGPT.gitcd AutoGPT- git checkout gitbook+ git checkout -b my-docs-change origin/dev```2. Make your changes to markdown files in `docs/`-3. Preview changes:- - Push to a branch and create a PR - GitBook will generate a preview- - Or use any markdown preview tool locally+3. Preview changes with any markdown preview tool## Adding a New Page1. Create a new markdown file in the appropriate `docs/` subdirectory2. Add the new page to the relevant `SUMMARY.md` file to include it in the navigation-3. Submit a pull request to the `gitbook` branch+3. Submit a pull request to the `dev` branch## Submitting a Pull Request-When you're ready to submit your changes, create a pull request targeting the `gitbook` branch. We will review your changes and merge them if appropriate.+When you're ready to submit your changes, create a pull request targeting the `dev` branch. We will review your changes and merge them if appropriate. They appear on the published docs with the next release.