Open-source AI coworkers that each get a computer of their own: a browser, files and tools, with every action decided before it happens and recorded after. Bring any AG-UI agent.
Publish an image for every service Compose can build, on both architectures (#365)
David McKayff00ada1f803
diff --git a/.github/published-images.json b/.github/published-images.jsonnew file mode 100644index 00000000..f5fab2fb--- /dev/null+++ b/.github/published-images.json@@ -0,0 +1 @@+["agent-computer", "supervisor", "agent-bot", "agent-langgraph", "server"]diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex 4d734395..899a2f96 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -358,13 +358,63 @@ jobs:- if: always()run: docker rm -f openbot-ci >/dev/null 2>&1 || true+ # The five images the release publishes alongside the one above. Nothing else here builds them:+ # they are built by `docker compose up --build`, which only the smoke journey runs and which+ # cannot run in CI. So a broken `supervisor/Dockerfile` used to surface during a release, after+ # the `openbot` image had already been pushed, which is the one failure this repository calls out+ # as leaving a published image with nothing pointing at it.+ #+ # Built, not pushed, and one architecture: this answers whether the Dockerfile resolves. Whether+ # it resolves on arm64 too is answered at release time, on an arm64 runner.+ #+ # No registry cache, so a release builds these a second time. `type=gha` needs ACTIONS_* in the+ # environment, which `build-push-action` injects and a plain `run:` step does not have, and+ # reaching for another action to export them would buy minutes in a job that runs rarely at the+ # price of a dependency in the one workflow whose inputs are a security boundary.+ component-dockerfiles:+ name: component dockerfiles+ runs-on: ubuntu-latest+ # The same cost gate as `image` above, and for the same reason: `agent-computer` pulls+ # Playwright's base and `server` builds the app, so this is minutes rather than seconds. On main+ # and through the release's workflow_call it always runs.+ if: >-+ github.event_name != 'pull_request' ||+ contains(github.event.pull_request.labels.*.name, 'full-ci')+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ persist-credentials: false+ - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0+ # A Dockerfile the release does not publish is one a machine would build from source, so the+ # list and the tree have to agree. Checked here rather than trusted, because the failure is+ # silent: the release succeeds and a laptop somewhere compiles the app.+ - name: Every buildable service is published+ run: |+ set -euo pipefail+ found="$(find . -mindepth 2 -maxdepth 2 -name Dockerfile -not -path './node_modules/*' \+ | sed -e 's|^\./||' -e 's|/Dockerfile$||' | sort)"+ published="$(jq -r '.[]' .github/published-images.json | sort)"+ if [ "$found" != "$published" ]; then+ echo "::error::.github/published-images.json does not match the Dockerfiles in the tree."+ diff <(echo "$published") <(echo "$found") || true+ exit 1+ fi+ - name: They build+ run: |+ set -euo pipefail+ while read -r image; do+ echo "::group::$image"+ docker buildx build --file "$image/Dockerfile" --platform linux/amd64 .+ echo "::endgroup::"+ done < <(jq -r '.[]' .github/published-images.json)+# One check for branch protection to require. A new job above is covered by this without anybody# remembering to add it to a list, and a job that was skipped for the wrong reason is not a pass.verify:name: verifyruns-on: ubuntu-latestif: always()- needs: [static, deployables, chart, test, build, migrations, image]+ needs: [static, deployables, chart, test, build, migrations, image, component-dockerfiles]steps:- name: Require every checkenv:diff --git a/.github/workflows/publish-release.yml b/.github/workflows/publish-release.ymlindex 4d079c0b..031bd405 100644--- a/.github/workflows/publish-release.yml+++ b/.github/workflows/publish-release.yml@@ -77,6 +77,8 @@ jobs:needs: metadataif: needs.metadata.outputs.is_release == 'true'runs-on: ubuntu-latest+ outputs:+ components: ${{ steps.components.outputs.components }}steps:- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1with:@@ -94,6 +96,11 @@ jobs:echo "::error::CHANGELOG.md has no section for ${VERSION#v}."exit 1}+ # The services published alongside the image above. Read from the tree rather than written+ # here, because CI checks the same file when it proves those Dockerfiles still build, so the+ # set that is tested and the set that is published cannot drift apart.+ - id: components+ run: echo "components=$(jq -c . .github/published-images.json)" >> "$GITHUB_OUTPUT"# The same checks CI runs, against the commit being published. This is the gate: nothing is built# or tagged unless they pass here, on this exact tree.@@ -156,12 +163,167 @@ jobs:subject-digest: ${{ steps.push.outputs.digest }}push-to-registry: true+ # The images the installer pulls instead of building. `docker-compose.yml` builds these from+ # source on every machine, which needs a toolchain and several minutes a desktop install does not+ # have. Published here, from the same commit as the image above, so a deployment and a laptop run+ # the same code.+ #+ # Two architectures, because the machines are laptops: arm64 Macs and amd64 everything else. Built+ # on native runners rather than under QEMU. Emulated `bun install` and `vite build` are a known+ # source of release-day flakiness, and arm64 runners are free to a public repository, so emulation+ # would be the slower and less reliable option at no saving.+ component-images:+ name: ${{ matrix.image }} ${{ matrix.platform.arch }}+ needs: [metadata, verify, checks]+ if: needs.metadata.outputs.is_release == 'true'+ runs-on: ${{ matrix.platform.runner }}+ permissions:+ contents: read+ packages: write+ strategy:+ # One image failing should not hide whether the others build, and a half-finished run leaves+ # nothing deployable: these builds are pushed untagged, and the tags are written by the job+ # below only once both architectures of an image exist.+ fail-fast: false+ matrix:+ image: ${{ fromJSON(needs.verify.outputs.components) }}+ platform:+ - arch: amd64+ runner: ubuntu-latest+ - arch: arm64+ runner: ubuntu-24.04-arm+ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1+ with:+ ref: ${{ github.sha }}+ persist-credentials: false+ - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0+ - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0+ with:+ registry: ghcr.io+ username: ${{ github.actor }}+ password: ${{ secrets.GITHUB_TOKEN }}+ # Pushed by digest and deliberately untagged. A tag written here would name one architecture,+ # and the two jobs for one image would race to own it, so the last to finish would decide what+ # the tag meant.+ - id: push+ uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0+ with:+ context: .+ file: ${{ matrix.image }}/Dockerfile+ platforms: linux/${{ matrix.platform.arch }}+ outputs: type=image,name=ghcr.io/copilotkit/openbot-${{ matrix.image }},push-by-digest=true,name-canonical=true,push=true+ # Scoped per image and per architecture. One shared scope would have ten builds+ # overwriting each other's cache and none of them reading their own.+ cache-from: type=gha,scope=${{ matrix.image }}-${{ matrix.platform.arch }}+ cache-to: type=gha,mode=max,scope=${{ matrix.image }}-${{ matrix.platform.arch }}+ provenance: true+ sbom: true+ # A matrix job's outputs are not addressable by the jobs that consume them, so the digest+ # travels as a file. One artifact per image and architecture, because same-named artifacts+ # from different matrix legs collide.+ - name: Record the digest+ env:+ DIGEST: ${{ steps.push.outputs.digest }}+ ARCH: ${{ matrix.platform.arch }}+ run: |+ set -euo pipefail+ [[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]+ mkdir -p digests+ echo "$DIGEST" > "digests/$ARCH"+ - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: digest-${{ matrix.image }}-${{ matrix.platform.arch }}+ path: digests/+ retention-days: 1++ # Two per-architecture images become one reference. Whatever pulls it, a laptop or a cluster, names+ # the manifest list and gets its own architecture without being told which one it is.+ component-manifests:+ name: ${{ matrix.image }} manifest+ needs: [metadata, verify, checks, component-images]+ if: needs.metadata.outputs.is_release == 'true'+ runs-on: ubuntu-latest+ permissions:+ contents: read+ packages: write+ # Same identity and the same reason as the image above: the attestation says which workflow,+ # repository and commit produced this, and there is no key to hold.+ id-token: write+ attestations: write+ strategy:+ fail-fast: false+ matrix:+ image: ${{ fromJSON(needs.verify.outputs.components) }}+ steps:+ - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4.3.0+ - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0+ with:+ registry: ghcr.io+ username: ${{ github.actor }}+ password: ${{ secrets.GITHUB_TOKEN }}+ - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1+ with:+ pattern: digest-${{ matrix.image }}-*+ path: digests+ merge-multiple: true+ - id: merge+ name: Write the tags over both architectures+ env:+ REPOSITORY: ghcr.io/copilotkit/openbot-${{ matrix.image }}+ VERSION: ${{ needs.metadata.outputs.version }}+ COMMIT: ${{ github.sha }}+ run: |+ set -euo pipefail+ # Both architectures or neither. A manifest list holding one of them installs on half the+ # machines and looks exactly like one holding both until somebody's laptop says+ # "no matching manifest".+ refs=()+ for arch in amd64 arm64; do+ digest="$(cat "digests/$arch")"+ [[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]+ refs+=("$REPOSITORY@$digest")+ done+ docker buildx imagetools create \+ --tag "$REPOSITORY:$VERSION" \+ --tag "$REPOSITORY:$COMMIT" \+ --tag "$REPOSITORY:latest" \+ "${refs[@]}"+ # The list's own digest, which is what anything downstream pins. Read back from the+ # registry rather than derived here, and checked, so a template that stops returning a+ # digest fails now instead of writing something unusable into the release.+ digest="$(docker buildx imagetools inspect "$REPOSITORY:$VERSION" --format '{{.Manifest.Digest}}')"+ [[ "$digest" =~ ^sha256:[0-9a-f]{64}$ ]]+ echo "digest=$digest" >> "$GITHUB_OUTPUT"+ - uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2+ with:+ subject-name: ghcr.io/copilotkit/openbot-${{ matrix.image }}+ subject-digest: ${{ steps.merge.outputs.digest }}+ push-to-registry: true+ # For the same reason the digests above travel as files: a matrix cannot hand a value to a+ # later job.+ - name: Record the manifest+ env:+ NAME: ${{ matrix.image }}+ REPOSITORY: ghcr.io/copilotkit/openbot-${{ matrix.image }}+ DIGEST: ${{ steps.merge.outputs.digest }}+ run: |+ set -euo pipefail+ mkdir -p manifests+ jq -n --arg name "$NAME" --arg repository "$REPOSITORY" --arg digest "$DIGEST" \+ '{name: $name, repository: $repository, digest: $digest}' > "manifests/$NAME.json"+ - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1+ with:+ name: manifest-${{ matrix.image }}+ path: manifests/+ retention-days: 1+# The tag and the release, last, so nothing is announced that was not built. The manifest is the# useful artefact: it pins the digest, so a deploy or a rollback names an exact image rather than a# tag somebody could move.github-release:name: tag and release- needs: [metadata, verify, checks, image]+ needs: [metadata, verify, checks, image, component-manifests]if: needs.metadata.outputs.is_release == 'true'runs-on: ubuntu-latestpermissions:@@ -173,14 +335,36 @@ jobs:with:ref: ${{ github.sha }}persist-credentials: false+ - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1+ with:+ pattern: manifest-*+ path: manifests+ merge-multiple: true- name: Write the image manifestenv:VERSION: ${{ needs.metadata.outputs.version }}DIGEST: ${{ needs.image.outputs.digest }}COMMIT: ${{ github.sha }}+ COMPONENTS: ${{ needs.verify.outputs.components }}run: |set -euo pipefail[[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]+ # Every image this release published, or the file is not written. A missing entry is a+ # service whatever reads this would quietly build from source instead, which is the thing+ # publishing them was for, and it would only be noticed on somebody's laptop.+ expected="$(jq -r 'length' <<< "$COMPONENTS")"+ found="$(find manifests -name '*.json' -type f | wc -l | tr -d ' ')"+ if [ "$found" != "$expected" ]; then+ echo "::error::Expected $expected component manifests, found $found."+ exit 1+ fi+ components="$(jq -s 'map({+ (.name): {+ repository: .repository,+ digest: .digest,+ reference: (.repository + "@" + .digest),+ }+ }) | add' manifests/*.json)"# `jq`, not `bun`: this job deliberately checks out without credentials and installs no# toolchain, so reaching for the repository's runtime here is a step that was never taken.# It was, and the tag was never cut: the manifest step died on `bun: command not found`@@ -191,16 +375,17 @@ jobs:--arg digest "$DIGEST" \--arg commit "$COMMIT" \--arg repository "ghcr.io/copilotkit/openbot" \+ --argjson components "$components" \'{version: $version,commit: $commit,- images: {+ images: ({openbot: {repository: $repository,digest: $digest,reference: ($repository + "@" + $digest),},- },+ } + $components),}' > container-images.jsoncat container-images.json- name: Tag and publishdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 5b4e5d4a..c6832671 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -8,6 +8,26 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.## Unreleased+### A release publishes every service's image, not just the one++`ghcr.io/copilotkit/openbot` was the only image a release produced, so anything running the Compose+stack built `agent-computer`, the supervisor, both Bots and the migration image from source on+every machine. That needs a toolchain and it needs several minutes, most of them Chromium, and it+is the difference between a deployment and a laptop being able to start at all.++Each of those is now published beside it, at `ghcr.io/copilotkit/openbot-<service>`, carrying both+`linux/amd64` and `linux/arm64` so one reference works on a server and on an arm64 laptop. Every+image gets its own build provenance attestation, and `container-images.json` on the release pins a+digest for all of them rather than for one.++Nothing changes for a checkout: unset, `COMPUTER_IMAGE`, `SUPERVISOR_IMAGE`, `BOT_IMAGE`,+`LANGGRAPH_IMAGE` and `SERVER_IMAGE` name local tags and Compose builds them as before. Point them+at published digests and set `IMAGE_PULL_POLICY=missing` to pull instead. `docs/releasing.md` shows+reading the references out of `container-images.json`; `docs/configuration.md` lists the settings.++CI now builds those five Dockerfiles too. Nothing did before, because they are built by+`docker compose up --build`, which only the smoke journey runs and which cannot run in CI, so a+broken one surfaced during a release after the first image had already been pushed.### A skill can be written in the conversation instead of retyped into a formA skill is four fields and an instruction a Bot follows, and the instruction is the one that decidesdiff --git a/docker-compose.yml b/docker-compose.ymlindex cdfe4e35..710ecb6c 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -39,6 +39,18 @@ services:build:context: .dockerfile: server/Dockerfile+ # Named, and pullable, for the same reason as agent-computer below: a release publishes this+ # image, and a machine that can name it does not need a toolchain to build it. `pull_policy`+ # is what decides between the two, because a service carrying a `build` section builds by+ # default however its image is named. IMAGE_PULL_POLICY=missing makes it pull instead.+ #+ # Not a guarantee that nothing is built: a pull that fails falls back to building, silently,+ # which is right for a developer and wrong for an installed OpenBot, where there is no+ # toolchain and the honest answer is that the image could not be fetched. Anything shipping+ # these images to a machine that cannot build them overrides the `build` section away rather+ # than relying on this variable.+ image: ${SERVER_IMAGE:-openbot-server:latest}+ pull_policy: ${IMAGE_PULL_POLICY:-build}command: ["bun", "x", "drizzle-kit", "migrate", "--config=drizzle.config.ts"]environment:DATABASE_URL: postgres://openbot:openbot@postgres:5432/openbot@@ -57,6 +69,7 @@ services:# Tagged explicitly because the supervisor starts one computer per Bot from this image by name,# and a Compose-derived name would change with the project.image: ${COMPUTER_IMAGE:-openbot-agent-computer:latest}+ pull_policy: ${IMAGE_PULL_POLICY:-build}ports:# Loopback only. This process drives a browser holding real logins; COMPUTER_TOKEN is the# request control, and loopback keeps the surface off routed networks.@@ -158,6 +171,9 @@ services:build:context: .dockerfile: supervisor/Dockerfile+ # Published by a release; see migrate above for what pull_policy decides.+ image: ${SUPERVISOR_IMAGE:-openbot-supervisor:latest}+ pull_policy: ${IMAGE_PULL_POLICY:-build}# The same file, because this process does not read these itself: it forwards every EGRESS_PROXY# key out of its own environment into each computer it creates, so it has to be given them first.env_file:@@ -218,6 +234,9 @@ services:build:context: .dockerfile: agent-bot/Dockerfile+ # Published by a release; see migrate above for what pull_policy decides.+ image: ${BOT_IMAGE:-openbot-agent-bot:latest}+ pull_policy: ${IMAGE_PULL_POLICY:-build}ports:# Loopback, not every interface. The token is the boundary; this means an attacker needs to be# on the machine before they can even try it. Nothing legitimate reaches a Bot from another@@ -245,6 +264,9 @@ services:build:context: .dockerfile: agent-langgraph/Dockerfile+ # Published by a release; see migrate above for what pull_policy decides.+ image: ${LANGGRAPH_IMAGE:-openbot-agent-langgraph:latest}+ pull_policy: ${IMAGE_PULL_POLICY:-build}ports:# Loopback, for the same reason as agent-bot above.- "127.0.0.1:${LANGGRAPH_PORT:-4201}:4201"diff --git a/docs/configuration.md b/docs/configuration.mdindex 96ca38bc..782622fe 100644--- a/docs/configuration.md+++ b/docs/configuration.md@@ -326,6 +326,32 @@ When optional SPIRE services are used:- computers read `SPIFFE_ENDPOINT_SOCKET`;- Compose also uses `SPIRE_JOIN_TOKEN` and `COMPOSE_PROJECT_NAME`.+## Images++Every service `docker-compose.yml` can build is published by a release, so a machine can run the+stack without a toolchain and without waiting for Chromium to build.++| Service | Setting | Published image |+| ----------------- | ------------------- | ------------------------------------------ |+| `agent-computer` | `COMPUTER_IMAGE` | `ghcr.io/copilotkit/openbot-agent-computer` |+| `supervisor` | `SUPERVISOR_IMAGE` | `ghcr.io/copilotkit/openbot-supervisor` |+| `agent-bot` | `BOT_IMAGE` | `ghcr.io/copilotkit/openbot-agent-bot` |+| `agent-langgraph` | `LANGGRAPH_IMAGE` | `ghcr.io/copilotkit/openbot-agent-langgraph`|+| `migrate` | `SERVER_IMAGE` | `ghcr.io/copilotkit/openbot-server` |++Unset, each names a local tag and Compose builds it, which is what a checkout of this repository+does. Set to a published reference, pinned by digest, together with `IMAGE_PULL_POLICY=missing`,+Compose pulls instead. Both architectures are in every image, so the same reference works on an+arm64 laptop and an amd64 server.++`IMAGE_PULL_POLICY` is needed because a service carrying a `build` section builds by default+however its image is named. It is also not a promise that nothing is built: a pull that fails falls+back to building, which suits a developer and does not suit a machine with no toolchain, where the+useful answer is that the image could not be fetched. Somewhere that must never build, override the+`build` sections away instead.++`docs/releasing.md` shows reading the digests straight out of a release's `container-images.json`.+## Ports| Service | Default port | Setting |diff --git a/docs/releasing.md b/docs/releasing.mdindex 884ea2ce..b646f688 100644--- a/docs/releasing.md+++ b/docs/releasing.md@@ -29,13 +29,19 @@ Then, in order:checked for a section with that number- one image is built and pushed to `ghcr.io/copilotkit/openbot`, tagged with the version, the commitand `latest`-- a build provenance attestation is signed with the workflow's OIDC identity and pushed alongside it+- the services `docker-compose.yml` can build are published too, one image each, at+ `ghcr.io/copilotkit/openbot-<service>`. Those are `linux/amd64` and `linux/arm64`, built on native+ runners of each architecture and joined into one manifest list, because the machines pulling them+ are laptops as well as servers. `.github/published-images.json` is the list, and CI fails if it+ stops matching the Dockerfiles in the tree+- a build provenance attestation is signed with the workflow's OIDC identity and pushed alongside+ every one of them- the commit is tagged and a GitHub Release is created, carrying the changelog section as its notesand `container-images.json` as an asset## Deploying a release-`container-images.json` pins the digest. Deploy that, not a tag:+`container-images.json` pins a digest for every image the release published. Deploy those, not tags:```shgh release download v0.1.0 --pattern container-images.json@@ -43,13 +49,33 @@ docker run -p 3001:3001 --env-file .env \"$(jq -r .images.openbot.reference container-images.json)"```+The same file is how a machine runs the stack without building any of it. Each key under `images`+is a service, so the references can be read straight out of it:++```sh+export COMPUTER_IMAGE="$(jq -r '.images["agent-computer"].reference' container-images.json)"+export SUPERVISOR_IMAGE="$(jq -r .images.supervisor.reference container-images.json)"+export BOT_IMAGE="$(jq -r '.images["agent-bot"].reference' container-images.json)"+export LANGGRAPH_IMAGE="$(jq -r '.images["agent-langgraph"].reference' container-images.json)"+export SERVER_IMAGE="$(jq -r .images.server.reference container-images.json)"+export IMAGE_PULL_POLICY=missing+docker compose up -d+```++`IMAGE_PULL_POLICY=missing` is what turns those names into pulls; without it every one of those+services builds from source, because a service with a `build` section builds by default whatever+its image is called. A pull that fails still falls back to building, so a machine with no toolchain+wants the `build` sections overridden away rather than this variable alone.+A tag can be moved to point at a different image; a digest cannot. The same digest that CI smoketested is the one that runs, and rolling back is the same command with an earlier version.-Before deploying, you can check the image is the one this repository built:+Before deploying, you can check an image is the one this repository built. Every published image+carries its own attestation:```shgh attestation verify oci://ghcr.io/copilotkit/openbot:v0.1.0 -R CopilotKit/OpenBot+gh attestation verify oci://ghcr.io/copilotkit/openbot-supervisor:v0.1.0 -R CopilotKit/OpenBot```## What has to be green@@ -64,6 +90,7 @@ A job added to `ci.yml` is covered by it without anybody updating a list.| `build` | the app not compiling || `migrations` | a schema change with no migration, or a snapshot that has drifted || `image` | an image that builds but does not boot, or a supervised service that respawns |+| `component dockerfiles` | a Dockerfile a release would publish that no longer builds, or one the publish list has stopped covering |`image` matters more than its position suggests. Everything above it can pass on a tree whose imagenever starts, because nothing else here runs the thing it ships. It builds the container, boots it