Skip to content

πŸ™Œ OpenHands: AI-Driven Development

Add files via upload (#1187)

Ash Clarke78f96f49eda8
diff --git a/docs/TESTING_MATRIX.md b/docs/TESTING_MATRIX.md
new file mode 100644
index 0000000000..3ef39c5b2d
--- /dev/null
+++ b/docs/TESTING_MATRIX.md
@@ -0,0 +1,98 @@
+# Agent Canvas β€” Testing Matrix
+
+**Priority key:** P0 = must pass before any release Β· P1 = must pass before GA Β· P2 = best-effort
+
+---
+
+## Install Γ— OS Γ— Agent
+
+Each cell = smoke test: install β†’ onboard β†’ start conversation β†’ agent replies.
+
+| | **macOS** | **Linux** | **Windows** |
+|---|:---:|:---:|:---:|
+| **npm β€” OpenHands** | ☐ | ☐ | ☐ |
+| **npm β€” Claude Code** | ☐ | ☐ | ☐ |
+| **npm β€” Codex** | ☐ | ☐ | ☐ |
+| **npm β€” Gemini CLI** | ☐ | ☐ | ☐ |
+| **npm β€” Custom ACP** | ☐ | ☐ | ☐ |
+| **Docker β€” OpenHands** | ☐ | ☐ | ☐ |
+| **Docker β€” Claude Code** | ☐ | ☐ | ☐ |
+| **Docker β€” Codex** | ☐ | ☐ | ☐ |
+| **Docker β€” Gemini CLI** | ☐ | ☐ | ☐ |
+| **Docker β€” Custom ACP** | ☐ | ☐ | ☐ |
+
+---
+
+## Automations Γ— Install Γ— Agent
+
+Requires full stack (automation backend running).
+
+| | **npm** | **Docker** |
+|---|:---:|:---:|
+| **OpenHands** | βœ… P0 | βœ… P0 |
+| **Claude Code** | βœ… P1 | βœ… P1 |
+| **Codex** | βœ… P1 | βœ… P1 |
+| **Gemini CLI** | βœ… P2 | βœ… P2 |
+
+Each cell = create automation β†’ dispatch run β†’ run reaches COMPLETED β†’ conversation link works.
+
+---
+
+## Auth Modes
+
+| | **npm** | **Docker** |
+|---|:---:|:---:|
+| **Local (auto-generated key)** | βœ… P0 | βœ… P0 |
+| **Public (`--public` + user key)** | βœ… P1 | βœ… P1 |
+
+---
+
+## Feature Checklist
+
+### npm
+
+| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
+|---|:---:|:---:|:---:|:---:|
+| Onboarding | ☐ | ☐ | ☐ | ☐ |
+| Conversation β€” start, resume, history | ☐ | ☐ | ☐ | ☐ |
+| Terminal tool | ☐ | ☐ | ☐ | ☐ |
+| File editor tool | ☐ | ☐ | ☐ | ☐ |
+| Browser tool | ☐ | ☐ | ☐ | ☐ |
+| LLM profiles β€” create / switch | ☐ | β€” | β€” | β€” |
+| Secrets β€” add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
+| Automations β€” create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
+| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
+| MCP server install | ☐ | ☐ | ☐ | ☐ |
+| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
+| Key rotation | ☐ | ☐ | ☐ | ☐ |
+
+### Docker
+
+| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
+|---|:---:|:---:|:---:|:---:|
+| Onboarding | ☐ | ☐ | ☐ | ☐ |
+| Conversation β€” start, resume, history | ☐ | ☐ | ☐ | ☐ |
+| Terminal tool | ☐ | ☐ | ☐ | ☐ |
+| File editor tool | ☐ | ☐ | ☐ | ☐ |
+| Browser tool | ☐ | ☐ | ☐ | ☐ |
+| LLM profiles β€” create / switch | ☐ | β€” | β€” | β€” |
+| Secrets β€” add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
+| Automations β€” create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
+| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
+| MCP server install | ☐ | ☐ | ☐ | ☐ |
+| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
+| Key rotation | ☐ | ☐ | ☐ | ☐ |
+
+---
+
+## Automated Coverage
+
+| Suite | Install | OS | Agents | Automations |
+|---|---|---|---|---|
+| `vitest` (unit) | β€” | Linux | β€” | partial |
+| `test:e2e` (snapshots) | npm (mock) | Linux | β€” | partial |
+| `test:e2e:mock-llm` | npm | Linux | OpenHands, ACP (mock) | βœ… full |
+| `test:e2e:mock-llm:docker` | Docker | Linux | OpenHands, ACP (mock) | βœ… full |
+| `test:e2e:live` | npm | Linux | OpenHands | ❌ |
+
+**Not yet covered by CI:** real ACP credentials (Claude Code / Codex / Gemini), macOS, public auth mode, subscription login paths, Windows.