The agent that turns websites into APIs!
release: v0.2.5 - fix OpenCodeEngineer initialization and improve README
kalil0321e6eaf7a65c3b
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a4e8d58..5372d08 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -5,6 +5,13 @@ All notable changes to this project will be documented in this file.The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).+## [0.2.5] - 2025-12-27++### Fixed+- **OpenCodeEngineer initialization**: Refactored to properly pop specific kwargs (`opencode_provider` and `opencode_model`) before passing to parent class+ - Ensures only relevant arguments are sent to BaseEngineer+ - Improves initialization logic clarity and prevents unintended argument passing+## [0.2.4] - 2025-12-27### Fixeddiff --git a/README.md b/README.mdindex c9f6ef7..0a9d974 100644--- a/README.md+++ b/README.md@@ -12,39 +12,43 @@ CLI tool that captures browser traffic and automatically generates production-re+## Table of Contents++- [Features](#-features)+- [Installation](#-installation)+- [Quick Start](#-quick-start)+- [Usage Modes](#-usage-modes)+ - [Manual Mode](#manual-mode)+ - [Engineer Mode](#engineer-mode)+ - [Agent Mode](#agent-mode)+- [Configuration](#-configuration)+ - [Model Selection](#model-selection)+ - [Agent Configuration](#agent-configuration)+ - [SDK Selection](#sdk-selection)+- [CLI Commands](#-cli-commands)+- [Examples](#-examples)+- [Development](#-development)+- [Contributing](#-contributing)+## β¨ Features- π **Browser Automation**: Built on Playwright with stealth mode for realistic browsing- π€ **Autonomous Agent Mode**: Fully automated browser interaction using AI agents (browser-use, stagehand)- π **HAR Recording**: Captures all network traffic in HTTP Archive format- π§ **AI-Powered Generation**: Uses Claude 4.5 to analyze traffic and generate clean Python code-- π **OpenCode SDK Support**: Native integration with OpenCode SDK for more flexibility+- π **Multi-SDK Support**: Native integration with Claude and OpenCode SDKs- π» **Interactive CLI**: Minimalist terminal interface with mode cycling (Shift+Tab)- π¦ **Production Ready**: Generated scripts include error handling, type hints, and documentation- πΎ **Session History**: All runs saved locally with full message logs- π° **Cost Tracking**: Detailed token usage and cost estimation with cache support-- π **Multi-Provider Support**: Supports Browser-Use and Stagehand agent providers with multiple LLM options--## Limitations-- This tool executes code locally using Claude Code please monitor output.+### Limitations-- Some websites employ advanced bot-detection and anti-automation protections, which may limit capture or require additional manual interaction.+- This tool executes code locally using Claude Codeβplease monitor output+- Some websites employ advanced bot-detection that may limit capture or require manual interaction## π Installation-### Using pip-```bash-# Basic installation (manual and engineer modes)-pip install reverse-api-engineer--# With agent mode support-pip install 'reverse-api-engineer[agent]'--# Install browser-use with HAR recording support (required for agent mode)-pip install git+https://github.com/browser-use/browser-use.git@49a345fb19e9f12befc5cc1658e0033873892455-```-### Using uv (recommended)```bash# Basic installation@@ -54,11 +58,14 @@ uv tool install reverse-api-engineeruv tool install 'reverse-api-engineer[agent]' --with 'browser-use @ git+https://github.com/browser-use/browser-use.git@49a345fb19e9f12befc5cc1658e0033873892455'```-### From source+### Using pip```bash-git clone https://github.com/kalil0321/reverse-api-engineer.git-cd reverse-api-engineer-uv sync+# Basic installation+pip install reverse-api-engineer++# With agent mode support+pip install 'reverse-api-engineer[agent]'+pip install git+https://github.com/browser-use/browser-use.git@49a345fb19e9f12befc5cc1658e0033873892455```### Post-installation@@ -67,9 +74,7 @@ Install Playwright browsers:playwright install chromium```-## π Usage--### Quick Start+## π Quick StartLaunch the interactive CLI:```bash@@ -77,87 +82,65 @@ reverse-api-engineer```The CLI has three modes (cycle with **Shift+Tab**):-- **manual**: Full pipeline (browser capture + AI generation)-- **engineer**: Reverse engineer from existing run_id-- **agent**: Autonomous browser agent + capture (fully automated)+- **manual**: Browser capture + AI generation+- **engineer**: Re-process existing captures+- **agent**: Autonomous AI browser agent++Example workflow:+```bash+$ reverse-api-engineer+> fetch all apple jobs from their careers page-### Manual Mode (Full Pipeline)+# Browser opens, navigate and interact+# Close browser when done+# AI generates production-ready API client++# Scripts saved to: ./scripts/apple_jobs_api/+```++## π Usage Modes++### Manual Mode++Full pipeline with manual browser interaction:1. Start the CLI: `reverse-api-engineer`-2. Enter your task description (e.g., "Fetch Apple job listings")-3. Optionally provide a starting URL+2. Enter task description (e.g., "Fetch Apple job listings")+3. Optionally provide starting URL4. Browse and interact with the website-5. Close the browser when done+5. Close browser when done6. AI automatically generates the API client-Generated scripts are saved to:+**Output locations:**- `~/.reverse-api/runs/scripts/{run_id}/` (permanent storage)- `./scripts/{descriptive_name}/` (local copy with readable name)-### Engineer Mode (Re-process existing captures)+### Engineer ModeRe-run AI generation on a previous capture:```bash-# In CLI, switch to engineer mode (Shift+Tab) and enter run_id+# Switch to engineer mode (Shift+Tab) and enter run_id# Or use command line:reverse-api-engineer engineer <run_id>```-### Agent Mode (Autonomous Browser Agent)+### Agent ModeFully automated browser interaction using AI agents:-1. Install with agent support (see installation instructions above for pip or uv)-2. Start the CLI: `reverse-api-engineer`-3. Switch to agent mode (Shift+Tab until you see `[agent]`)-4. Enter your task description (e.g., "Click on the first job listing")-5. Optionally provide a starting URL-6. The agent automatically navigates and interacts with the website-7. HAR is captured automatically-8. Optionally run reverse engineering to generate API client--**Important:** Browser-use must be installed from the specific git commit shown in the installation instructions above. This version includes HAR recording support which is not yet available in the PyPI release.--**Agent Provider Configuration:**-- **Browser-Use** (default): Supports Browser-Use LLM, OpenAI, and Google models-- **Stagehand**: Supports OpenAI and Anthropic Computer Use models--**Agent Model Configuration:**-- **Browser-Use Provider**:- - **Browser-Use LLM**: Requires `BROWSER_USE_API_KEY`- - **OpenAI Models**: Format `openai/{model}` (e.g., `openai/gpt-4`), requires `OPENAI_API_KEY`- - **Google Models**: Format `google/{model}` (e.g., `google/gemini-pro`), requires `GOOGLE_API_KEY`-- **Stagehand Provider**:- - **OpenAI Computer Use Models**: Format `openai/{cua-model}` (e.g., `openai/computer-use-preview-2025-03-11`), requires `OPENAI_API_KEY`- - **Anthropic Computer Use Models**: Format `anthropic/{cua-model}` (e.g., `anthropic/claude-sonnet-4-5-20250929`), requires `ANTHROPIC_API_KEY`- - Note: Stagehand only supports Computer Use Agents (CUA) models--Configure agent provider and model in `/settings` β "agent provider" and "agent model"--### CLI Commands--While in the CLI, use these slash commands:-- `/settings` - Configure model, agent model, SDK, and output directory-- `/history` - View past runs with costs-- `/messages <run_id>` - View detailed message logs-- `/help` - Show all commands-- `/exit` - Quit--### Model Selection+1. Install with agent support (see [Installation](#-installation))+2. Start CLI and switch to agent mode (Shift+Tab)+3. Enter task description (e.g., "Click on the first job listing")+4. Optionally provide starting URL+5. Agent automatically navigates and interacts+6. HAR captured automatically+7. Optionally generate API client-Choose from Claude 4.5 models:-- **Sonnet 4.5** (default): Balanced performance and cost-- **Opus 4.5**: Maximum capability for complex APIs-- **Haiku 4.5**: Fastest and most economical--Change model in `/settings` or via CLI:-```bash-reverse-api-engineer manual --model claude-sonnet-4-5-```+**Note:** Requires browser-use from the specific git commit shown in installation instructions (includes HAR recording support).## π§ Configuration-Settings are stored in `~/.reverse-api/config.json`:+Settings stored in `~/.reverse-api/config.json`:```json{"model": "claude-sonnet-4-5",@@ -168,71 +151,64 @@ Settings are stored in `~/.reverse-api/config.json`:}```-### Agent Provider Configuration+### Model Selection-The `agent_provider` setting controls which browser automation library is used:-- **`browser-use`** (default): Browser-Use library with support for multiple LLM providers-- **`stagehand`**: Stagehand library with OpenAI and Anthropic Computer Use models+Choose from Claude 4.5 models for API generation:+- **Sonnet 4.5** (default): Balanced performance and cost+- **Opus 4.5**: Maximum capability for complex APIs+- **Haiku 4.5**: Fastest and most economical-### Agent Model Configuration+Change in `/settings` or via CLI:+```bash+reverse-api-engineer manual --model claude-sonnet-4-5+```-The `agent_model` setting controls which LLM is used for autonomous browser agents:+### Agent Configuration-**For Browser-Use Provider:**-- **`bu-llm`** (default): Browser-Use's own LLM- - Requires: `BROWSER_USE_API_KEY` environment variable-- **`openai/{model}`**: OpenAI models (e.g., `openai/gpt-4`, `openai/gpt-3.5-turbo`)- - Requires: `OPENAI_API_KEY` environment variable- - Optional: Install `langchain-openai` for additional model support-- **`google/{model}`**: Google models (e.g., `google/gemini-pro`, `google/gemini-1.5-pro`)- - Requires: `GOOGLE_API_KEY` environment variable- - Optional: Install `langchain-google-genai` for additional model support+Configure AI agents for autonomous browser automation.-**For Stagehand Provider:**-- **`openai/{cua-model}`**: OpenAI Computer Use models (e.g., `openai/computer-use-preview-2025-03-11`)- - Requires: `OPENAI_API_KEY` environment variable- - Supported models: `computer-use-preview-2025-03-11`-- **`anthropic/{cua-model}`**: Anthropic Computer Use models (e.g., `anthropic/claude-sonnet-4-5-20250929`)- - Requires: `ANTHROPIC_API_KEY` environment variable- - Supported models: `claude-sonnet-4-5-20250929`, `claude-haiku-4-5-20251001`, `claude-opus-4-5-20251101`-- Note: Stagehand's `agent.execute()` mode only supports Computer Use Agents (CUA)+**Agent Providers:**+- **browser-use** (default): Supports Browser-Use LLM, OpenAI, and Google models+- **stagehand**: Supports OpenAI and Anthropic Computer Use models-**Setting API Keys:**-```bash-# Browser-Use (default)-export BROWSER_USE_API_KEY="your-api-key"+**Agent Models:**-# OpenAI (required for OpenAI models and Stagehand)-export OPENAI_API_KEY="your-api-key"+**Browser-Use Provider:**+- `bu-llm` (default) - Requires `BROWSER_USE_API_KEY`+- `openai/gpt-4`, `openai/gpt-3.5-turbo` - Requires `OPENAI_API_KEY`+- `google/gemini-pro`, `google/gemini-1.5-pro` - Requires `GOOGLE_API_KEY`-# Anthropic (required for Anthropic models with Stagehand)-export ANTHROPIC_API_KEY="your-api-key"+**Stagehand Provider (Computer Use only):**+- `openai/computer-use-preview-2025-03-11` - Requires `OPENAI_API_KEY`+- `anthropic/claude-sonnet-4-5-20250929` - Requires `ANTHROPIC_API_KEY`+- `anthropic/claude-haiku-4-5-20251001` - Requires `ANTHROPIC_API_KEY`+- `anthropic/claude-opus-4-5-20251101` - Requires `ANTHROPIC_API_KEY`-# Google (for Browser-Use only)-export GOOGLE_API_KEY="your-api-key"+**Setting API Keys:**+```bash+export BROWSER_USE_API_KEY="your-api-key" # For Browser-Use+export OPENAI_API_KEY="your-api-key" # For OpenAI models+export ANTHROPIC_API_KEY="your-api-key" # For Anthropic models+export GOOGLE_API_KEY="your-api-key" # For Google models```-Change agent provider and model in `/settings` β "agent provider" and "agent model" or edit `config.json` directly.+Change in `/settings` β "agent provider" and "agent model"### SDK Selection-Choose between two SDKs:-- **OpenCode**: Uses OpenCode SDK for AI-powered reverse engineering. Requires OpenCode to be running locally.-- **Claude** (default): Direct integration with Anthropic's Claude API.+- **Claude** (default): Direct integration with Anthropic's Claude API+- **OpenCode**: Uses OpenCode SDK (requires OpenCode running locally)-Change SDK in `/settings` or edit `config.json` directly. When using OpenCode SDK, ensure OpenCode is running (`opencode` command).+Change in `/settings` or edit `config.json` directly.-## π Project Structure+## π» CLI Commands-```-~/.reverse-api/-βββ config.json # User settings-βββ history.json # Run history with metadata-βββ runs/- βββ har/{run_id}/ # Captured HAR files- βββ scripts/{run_id}/ # Generated API clients- βββ messages/{run_id}.jsonl # Full message logs-```+Use these slash commands while in the CLI:+- `/settings` - Configure model, agent, SDK, and output directory+- `/history` - View past runs with costs+- `/messages <run_id>` - View detailed message logs+- `/help` - Show all commands+- `/exit` - Quit## π‘ Examples@@ -260,15 +236,6 @@ Generated `api_client.py` includes:- Error handling- Production-ready code-## πΊοΈ Roadmap--### SDK Support-- β **Claude** - Integration with Claude Code-- β **OpenCode** - Integration with OpenCode-- π **Codex** - Codex SDK support---## π οΈ Development### Setup@@ -285,15 +252,21 @@ uv run reverse-api-engineer### Build```bash-uv build+./scripts/clean_build.sh```+## πΊοΈ Roadmap++- β **Claude SDK** - Integration with Claude Code+- β **OpenCode SDK** - Integration with OpenCode+- π **Codex SDK** - Codex SDK support+## π Requirements- Python 3.11+- Claude Code / OpenCode (for reverse engineering)- Playwright browsers installed-- API key for agent mode (see [Agent Model Configuration](#agent-model-configuration))+- API key for agent mode (see [Agent Configuration](#agent-configuration))## π€ Contributingdiff --git a/pyproject.toml b/pyproject.tomlindex 2086fd0..4b16a22 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -1,6 +1,6 @@[project]name = "reverse-api-engineer"-version = "0.2.4"+version = "0.2.5"description = "A tool to capture browser traffic for API reverse engineering"readme = "README.md"requires-python = ">=3.11"diff --git a/src/reverse_api/__init__.py b/src/reverse_api/__init__.pyindex 0eed36f..99b2c4c 100644--- a/src/reverse_api/__init__.py+++ b/src/reverse_api/__init__.py@@ -1,3 +1,3 @@"""Reverse API - Browser traffic capture for API reverse engineering."""-__version__ = "0.2.4"+__version__ = "0.2.5"