Skip to content

The open-source app everyone uses to manage agents at work

docs(skills): distinguish Paperclip verification targets (#16145)

Dottafd61dc1dd538
diff --git a/.agents/skills/verify-paperclip/SKILL.md b/.agents/skills/verify-paperclip/SKILL.md
index b16467140a..9e10d7c2f9 100644
--- a/.agents/skills/verify-paperclip/SKILL.md
+++ b/.agents/skills/verify-paperclip/SKILL.md
@@ -1,6 +1,6 @@
---
name: verify-paperclip
-description: Launch an isolated Paperclip instance, drive its real web UI or CLI using the existing feature-map recipes, capture evidence, and clean up owned processes. Use to verify user-facing Paperclip behavior.
+description: Verify Paperclip from a source checkout, published npm install, or existing staging or production stack. Select the environment, data state, version, and platform; drive real UI or CLI journeys and capture evidence. Route deployment to the private Cloud skills.
---
# Verify Paperclip
@@ -15,7 +15,66 @@ recipe or access drift with evidence; map updates belong to
[maintain-verification-skill](../maintain-verification-skill/SKILL.md).
Read [development instructions](../../../doc/DEVELOPING.md) for current commands.
-## Launch
+## Select the verification target
+
+State what is being tested before starting. Use the requested target; for ordinary
+feature development, default to the current checkout on the current host with
+isolated data. Expand into cross-platform or release qualification only when
+requested or needed to reproduce the problem.
+
+Record these independent dimensions together:
+
+- **Artifact:** source checkout (path, commit, dirty state), published npm package
+ (requested channel and resolved exact version), or deployed image/release.
+- **Environment:** local, staging, or production; target URL and intended company
+ or workspace; board/user/agent identity and authentication mode.
+- **Starting data:** fresh and unonboarded, freshly seeded, cloned, or reused;
+ embedded versus external PostgreSQL; configuration/data location or a redacted
+ database identifier. Never record credentials or connection strings.
+- **Platform:** server OS/architecture, Node/npm versions, and browser/client OS
+ when relevant. A Mac browser against a Linux server is Linux server evidence;
+ WSL is Linux evidence, not native Windows evidence.
+- **Journey:** first installation/onboarding, an existing-user feature, upgrade,
+ or a live agent run; expected outcome and real versus simulated dependencies.
+
+| Target | Use it to prove | Procedure |
+| --- | --- | --- |
+| Checkout dev server, existing or cloned data | Source behavior against existing state | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) |
+| Checkout with fresh empty data | First-run onboarding and subsequent features | [Launch](#launch-an-empty-checkout-instance) below |
+| Checkout test drive, fresh seed or reused data | Features with a configured company/CEO | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) |
+| Published npm canary | End-user installation of a specific prerelease | [npm verification](references/npm-install.md) |
+| Published npm stable (`latest`) | End-user installation of the released package | [npm verification](references/npm-install.md) |
+| Existing staging stack | Deployed build, hosted auth, flags, integrations | [Hosted verification](references/verification-targets.md#staging-and-production) |
+| Existing production stack | Current production journey within the authorized scope | [Hosted verification](references/verification-targets.md#staging-and-production) |
+
+Fresh data does not imply a fresh install. A seeded test drive does not prove
+onboarding. A source build does not prove npm packaging. A passing staging check
+does not prove production, and a production Cloud stack does not prove the stable
+npm install. Report only the combinations actually exercised.
+
+## Deployment handoff
+
+When a requested revision must first be built or deployed, use the skills in the
+private `paperclip-cloud` repo:
+
+- `deploy-paperclip-stack` at `.agents/skills/deploy-paperclip-stack/SKILL.md`
+ for a tenant workspace or an explicitly scoped fleet rollout.
+- `deploy-cloud-harness` at `.agents/skills/deploy-cloud-harness/SKILL.md`
+ for the Cloud control plane itself.
+
+Locate the authorized Cloud checkout or installed skill by name; do not assume a
+sibling directory exists. Those skills own build preparation, deployment,
+migrations, and rollout recovery. Return here with environment, target, resolved
+source SHA, release/image identity, deployment result, and evidence link. If
+private access is unavailable, report that prerequisite as blocked and continue
+checks possible on the supplied target without inventing deploy commands.
+
+A verification request alone does not authorize deployment, npm publication,
+fleet-default changes, or a production rollout. Preserve explicit deployment
+instructions already given. Publishing npm packages belongs to the existing
+[release skill](../release/SKILL.md); installing one does not require publishing it.
+
+## Launch an empty checkout instance
Run from the repository root with Node 24.11+, Python 3, `lsof`, and dependencies
installed with `pnpm install --frozen-lockfile`. Browser driving uses the existing
@@ -53,7 +112,11 @@ For Paperclip-managed reusable services, follow
instead of launching a second server. Its cloned-data/login requirements differ
from this empty local instance.
-## Doctor
+## Doctor (owned checkout helper only)
+
+The doctor/stop commands require the real `launch.json` created by `launch.py`.
+Do not fabricate one for dev servers, npm installs, test drives, or hosted stacks.
+Use their own readiness and lifecycle procedures instead.
After startup, and again after any surprising behavior, run:
@@ -98,14 +161,16 @@ guessing resource URLs. The unprefixed `/onboarding` route starts naming; use th
company-prefixed onboarding route to resume its agent setup. Check relevant
experimental gates and actor permissions.
-For CLI checks, discover exact flags with
+For source-checkout CLI checks, discover exact flags with
`node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts <command> --help`, then
-select the disposable data directory/API/company explicitly. API reads can
+select the disposable data directory/API/company explicitly. For published npm
+checks, use the installed version’s CLI and help instead. API reads can
corroborate UI writes; API-created state does not prove a creation UI.
## Evidence
-Record feature file, sub-feature and entry-point ID, commit/dirty state, owned URL,
+Include the target dimensions above, then record feature file, sub-feature and
+entry-point ID, commit/dirty state, target URL,
company/user, flags, runtime/harness, live versus simulated dependencies, command,
exit code, expected and observed behavior, and evidence links. Report each entry
as passed, failed, blocked with its prerequisite, or not run.
@@ -126,7 +191,10 @@ documentation coverage rather than live test health.
## Cleanup
-Close the browser context you created (the smoke does this in `finally`). Stop the owned runtime with:
+Close the browser context you created (the smoke does this in `finally`). Leave
+shared dev, staging, and production services running. The remaining cleanup steps
+in this section apply only to `launch.py` instances; use the selected target’s
+procedure for other launchers. Stop the helper-owned runtime with:
```sh
python3 .agents/skills/verify-paperclip/scripts/stop.py \
diff --git a/.agents/skills/verify-paperclip/references/npm-install.md b/.agents/skills/verify-paperclip/references/npm-install.md
new file mode 100644
index 0000000000..fe4491948d
--- /dev/null
+++ b/.agents/skills/verify-paperclip/references/npm-install.md
@@ -0,0 +1,78 @@
+# Verify the published npm experience
+
+Use this route for “can a user install and run this package?” Source startup,
+workspace links, `pnpm build`, and local tarballs do not answer that question.
+
+## Resolve the exact artifact
+
+Read [.github/workflows/release.yml](../../../../.github/workflows/release.yml)
+for publication behavior. Canary publication follows verified pushes to `master`;
+an arbitrary feature-branch SHA need not have a published canary. Publication can
+fail or still be running, so prove registry availability before claiming coverage.
+
+Resolve the requested channel once against the public registry:
+
+```sh
+npm view paperclipai@canary version --registry=https://registry.npmjs.org
+npm view paperclipai@latest version --registry=https://registry.npmjs.org
+```
+
+Use the one requested by the task: `canary` for prerelease verification, `latest`
+for stable production npm verification, or an explicit version. Record resolved
+version, registry, dist integrity, and source provenance from release metadata,
+workflow, and tags. Pin that version for all later commands. If testing a SHA,
+confirm the package maps to it; never substitute the current canary silently.
+Missing publication is a blocked check, not a reason to publish during verification.
+
+## Install like a new user
+
+Use a disposable directory outside the checkout and fresh npm cache,
+instance/config, and user-config paths. Use an isolated OS user/container when
+proving absence of all pre-existing user state. Inspect inherited routing and
+registry overrides without exposing secrets. Avoid workspace dependencies, local
+`file:` packages, preinstalled Paperclip binaries, and cached `npx` installs.
+
+Run the documented end-user entry point with the pinned package, for example
+`npx --yes --registry=https://registry.npmjs.org paperclipai@<exact-version>`.
+Read that version's `--help` before selecting `onboard`, `install`, or service
+flags. For a foreground disposable check use `onboard --yes --bind loopback
+--no-install-service --run --data-dir <fresh-directory>` when supported by that
+version; keep ordinary interactive onboarding when that is the journey requested.
+A managed CLI/service installation is separate from an ephemeral `npx` launch;
+test it explicitly if requested.
+
+Prove each relevant stage separately: registry download/install, CLI execution,
+server readiness, visible onboarding, saved company/setup state, restart/reload,
+and a first agent task when authorized. Successful `--version` alone does not
+prove a usable install. For upgrades, first create state with the previous exact
+version, then install the target against that same disposable data and verify
+state survives; report this separately from a fresh install.
+
+## Existing automation and its limits
+
+The repo has a published-package onboarding suite:
+
+```sh
+PAPERCLIPAI_VERSION=<exact-published-canary-version> pnpm test:canary-onboarding-smoke
+```
+
+Read [its configuration](../../../../tests/canary-onboarding/playwright.config.ts)
+and [launcher](../../../../tests/canary-onboarding/start-published-canary.mjs)
+before running. It installs the published package into isolated temporary state,
+uses a fresh npm cache and loopback server, and retains failure diagnostics.
+It uses checkout Playwright tooling to drive a published runtime; record both
+revisions. Its name and automated release wiring target canaries. For stable
+verification, inspect compatibility with the selected version or use the manual
+pinned-package path above; do not relabel a canary result as stable proof.
+
+The Docker release-smoke suite is separate image evidence. Likewise,
+`scripts/clean-install-npm.sh` defaults to source, and its published mode uses a
+moving version and source-driven uninstall; it does not by itself prove the
+whole lifecycle of a specific published version.
+
+Capture OS/architecture, Node/npm versions, exact commands and exit codes,
+installed version, serving revision when available, initial data state, screenshots,
+logs, and observed outcomes. Redact credentials. Stop owned foreground processes
+or uninstall an owned test service with the tested version's supported command;
+retain evidence outside disposable data. Never run checkout ownership helpers
+with invented metadata to stop a package process.
diff --git a/.agents/skills/verify-paperclip/references/verification-targets.md b/.agents/skills/verify-paperclip/references/verification-targets.md
new file mode 100644
index 0000000000..837f3803a1
--- /dev/null
+++ b/.agents/skills/verify-paperclip/references/verification-targets.md
@@ -0,0 +1,87 @@
+# Checkout, data, and hosted targets
+
+## Checkout and data modes
+
+Read [DEVELOPING.md](../../../../doc/DEVELOPING.md) and
+[DATABASE.md](../../../../doc/DATABASE.md) for the current startup contract.
+
+- **Existing dev server:** use `pnpm dev` (watch) or `pnpm dev:once` from the
+ intended checkout. Inspect its instance routing and data configuration first;
+ do not assume the default port belongs to this checkout. Record the actual
+ served commit and dirty state. Reuse a matching running server when appropriate.
+- **Fresh empty instance:** use the skill's `launch.py` helper with a new data
+ directory. Complete the UI or CLI onboarding journey under test. This is the
+ credential-free route; a live model run needs separate provider setup.
+- **Seeded test drive:** run the documented CLI `test-drive --no-browser` with
+ an authorized environment-backed provider credential. It creates a temporary
+ embedded instance, completes setup, and seeds Test Company and a CEO. It does
+ not execute the first task. Verify that separately when required.
+- **Reused test drive:** pass `--data-dir <existing-test-directory>`. If it already
+ contains a company, bootstrap flags do not replace companies, agents, or
+ secrets. Record existing state; do not label the result fresh onboarding.
+ Test drive rejects external PostgreSQL configuration and database URL overrides.
+- **Managed or cloned worktree:** use
+ [dev-workspace run/verify/fix](../../paperclip-dev-workspace-run-verify-fix/SKILL.md).
+ Verify seed completion, instance routing, and login handoff before driving it.
+ Cloned data is existing-user evidence even when the checkout is new.
+
+`--data-dir` alone is not proof of a fresh database: check effective config and
+whether `DATABASE_URL` routes to an external database. Do not print its value.
+For ordinary dev, inspect both environment overrides and the active database
+config: with `DATABASE_URL` unset, `database.mode: postgres` and a saved
+`connectionString` still select external PostgreSQL. Embedded PostgreSQL is the
+default only when neither an environment override nor the active config selects
+an external database; its instance directory may already hold data. Do not reset
+shared data to make a test pass. Choose separate disposable data for first-run
+checks and verify the effective backend before calling it isolated.
+
+Record runtime PID, URL, port owner, checkout, and data location for a process you
+start. Check `/api/health`, expected served revision, bootstrap/auth readiness,
+and the actual page. Capture starting state and verify persistence after reload.
+Use that launcher's documented stop procedure for an owned process; leave a
+server you borrowed running.
+
+## Platform scope
+
+Ordinary development covers the current host unless the request or failure calls
+for more. For a platform issue or release matrix, list required combinations:
+macOS, Linux, native Windows, architecture, runtime, and install path. Mark each
+combination passed, failed, blocked, or not run separately.
+
+The local helper needs POSIX tools such as `lsof`; do not assume it qualifies
+native Windows. Use the supported CLI and native shell for that platform, with
+isolated instance/config/cache paths and platform-appropriate process cleanup.
+A container proves its guest OS and selected architecture, not the host's native
+installer, service manager, file permissions, or path behavior. Keep browser and
+server platforms distinct when testing a remote Linux box from a Mac.
+
+## Staging and production
+
+Use the supplied tenant URL and intended identity. Determine whether the target
+is the Paperclip tenant application or the Cloud control plane. Confirm the
+environment from the operator's deployment record and authenticated routing;
+a page title or local Git checkout is not evidence of the serving revision.
+
+If a build/deploy is needed, follow the private skill handoff in `SKILL.md` first.
+Record the exact serving SHA and release/image identity after deployment, not
+just the requested branch, CI result, or fleet default. Recheck after a long run:
+a staging auto-roll or another deployment can change the serving version. Mark
+unknown or changing provenance as inconclusive for a claim about a specific SHA.
+
+For staging, use a designated test workspace and its real hosted sign-in/entry
+flow. Record fresh versus existing workspace state, experimental flags, relevant
+provider connections, and whether dependencies are live. Check tenant readiness
+and then the feature journey, including persistence or continuation. A control
+plane `/healthz` response or tenant HTML 200 is not feature acceptance.
+
+For production, keep verification within the requested stack, identity, and
+journey. Prefer observation of existing state unless writes are authorized; use
+an agreed test workspace for mutating checks. Do not run the empty-instance
+onboarding smoke, reset data, change flags, or trigger paid agent work merely to
+make a production check pass. If a necessary write is outside the request,
+report that missing authorization and complete the read-only checks.
+
+Never use local ownership helpers against a remote server. Close only browser
+sessions you created and clean up only test records whose removal is authorized.
+Keep shared services running. Report production Cloud verification separately
+from stable npm verification; the shipped artifacts and configuration differ.