The agent that turns websites into APIs!
feat: scripts for plugin
kalil0321865aa1d6fb70
diff --git a/plugins/reverse-api-engineer/commands/agent.md b/plugins/reverse-api-engineer/commands/agent.mdindex 7b49746..1461671 100644--- a/plugins/reverse-api-engineer/commands/agent.md+++ b/plugins/reverse-api-engineer/commands/agent.md@@ -19,6 +19,26 @@ Ensure the `reverse-engineering-api` skill is loaded for guidance on HAR analysiIf arguments are not provided, prompt the user interactively.+## Task Tracking++Before starting the workflow, create a todo list to track progress:++```python+TodoWrite([+ {"content": "Launch browser with HAR recording", "status": "pending", "activeForm": "Launching browser"},+ {"content": "Navigate autonomously to complete task", "status": "pending", "activeForm": "Navigating autonomously"},+ {"content": "Verify HAR capture", "status": "pending", "activeForm": "Verifying HAR file"},+ {"content": "Filter HAR using har_filter.py", "status": "pending", "activeForm": "Filtering HAR"},+ {"content": "Analyze HAR using har_analyze.py", "status": "pending", "activeForm": "Analyzing endpoints"},+ {"content": "Generate API client code", "status": "pending", "activeForm": "Generating code"},+ {"content": "Validate code using har_validate.py", "status": "pending", "activeForm": "Validating code"},+ {"content": "Test API client implementation", "status": "pending", "activeForm": "Testing implementation"},+ {"content": "Generate documentation", "status": "pending", "activeForm": "Writing README"}+])+```++**CRITICAL:** Mark each task as `in_progress` when starting, `completed` when done. NEVER skip tasks - complete all 9 steps.+## Workflow### Step 1: Gather Task Information@@ -148,31 +168,62 @@ Analyzing for API endpoints...### Step 6: Analyze HAR File-Use the `reverse-engineering-api` skill guidance to:+**Mark todo in_progress: "Filter HAR using har_filter.py"**-1. **Filter relevant entries**: Exclude static assets, analytics, ads, CDN resources-2. **Identify API endpoints**: Look for `/api/`, `/v1/`, `/graphql`, XHR/Fetch requests-3. **Extract patterns**:- - URL patterns and query parameters- - HTTP methods (GET, POST, PUT, DELETE)- - Request/response headers- - Authentication mechanisms- - Request body schemas- - Response structures+Filter the HAR file to remove static assets, analytics, and CDN resources:-Summarize findings:+```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py ~/.reverse-api/runs/har/{run_id}/recording.har --output {output_dir}/filtered.har --stats```-Analysis complete!-Found {count} relevant API endpoints:-- {method} {endpoint} ({description})-- {method} {endpoint} ({description})-...+Read and display the filtering statistics:+```+Filtering complete!+Total entries: {total}+API endpoints found: {filtered_entries}+Removed:+- Static assets: {removed_static}+- Analytics/tracking: {removed_analytics}+- CDN resources: {removed_cdn}++API patterns detected: {api_patterns_found}+```++**Mark todo completed**++**Mark todo in_progress: "Analyze HAR using har_analyze.py"**++Extract structured endpoint information from filtered HAR:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.py {output_dir}/filtered.har --output {output_dir}/analysis.json+```++Read analysis.json and summarize for the user:++```bash+cat {output_dir}/analysis.json+```++Display summary:+```+Analysis complete!-Authentication detected: {auth_type} (Bearer token / API key / None)Base URL: {base_url}+Authentication: {authentication.type} ({authentication.description})+Unique endpoints: {unique_endpoints}+Pagination: {pagination.type if pagination.detected else "Not detected"}++Endpoints found:+{for each endpoint in endpoints:}+- {endpoint.methods} {endpoint.pattern}+ Calls observed: {endpoint.calls_observed}+ Auth required: {endpoint.requires_auth}+ Query params: {endpoint.query_params.required + endpoint.query_params.optional}```+**Mark todo completed**+### Step 7: Generate API ClientGenerate a descriptive name for the script based on the task:@@ -323,6 +374,58 @@ The agent autonomously performed these actions:- Test thoroughly before production use```+### Step 7.5: Validate Generated Code++**Mark todo in_progress: "Validate code using har_validate.py"**++Validate the generated API client against the HAR analysis:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.py {output_dir}/api_client.py {output_dir}/analysis.json+```++Check the validation score and issues:++**If validation score < 90:**+1. Read the validation issues carefully+2. Fix each issue in api_client.py:+ - Missing endpoints: Add methods for each missing endpoint+ - Missing auth: Implement authentication from analysis.json+ - Missing error handling: Add try-except blocks and custom exceptions+ - Missing type hints: Add type hints to all methods+3. Save the updated api_client.py+4. Run validation again+5. Repeat until score >= 90++**Example validation loop:**+```+Attempt 1:+- Score: 75+- Issues: 2 missing endpoints, no auth implementation+- Fix: Add missing endpoints, implement bearer token auth+- Regenerate api_client.py++Attempt 2:+- Score: 92+- Issues: 1 info (minor type hint improvement)+- Result: PASS (score >= 90)+```++Display validation result:+```+Validation complete!+Score: {score}/100+Coverage: {coverage.percentage}% ({coverage.endpoints_covered}/{coverage.endpoints_total} endpoints)+Issues: {summary.errors} errors, {summary.warnings} warnings, {summary.info} info++{if score >= 90:}+✓ Code validation passed!+{else:}+✗ Code validation failed - fixing issues and regenerating...+```++**Mark todo completed** (only when score >= 90)+### Step 8: Save Run HistoryUpdate `~/.reverse-api/history.json` with run metadata:diff --git a/plugins/reverse-api-engineer/commands/capture.md b/plugins/reverse-api-engineer/commands/capture.mdnew file mode 100644index 0000000..7160215--- /dev/null+++ b/plugins/reverse-api-engineer/commands/capture.md@@ -0,0 +1,191 @@+---+description: Launch browser with HAR recording for manual traffic capture+argument-hint: "[url]"+allowed-tools: ["MCP:rae-playwright-mcp", "Bash", "Read"]+---++# Capture Mode - Browser HAR Recording Only++Launch a browser with HAR (HTTP Archive) recording enabled. Navigate manually, then close the browser to save the HAR file. This command does NOT generate code - use `/reverse-api-engineer:engineer <run_id>` afterward to analyze the captured traffic and generate an API client.++## Purpose++Capture mode is useful when you want to:+- Record multiple browsing sessions before analysis+- Separate capture from code generation+- Share HAR files with others for analysis+- Capture traffic first, decide on implementation later++## Workflow++### Step 1: Generate Run ID++Create a unique run ID for this capture session:++```bash+run_id=$(uuidgen | tr '[:upper:]' '[:lower:]' | tr -d '-' | cut -c1-12)+echo "Starting capture session: ${run_id}"+```++Setup HAR file path:+```bash+har_dir="${HOME}/.reverse-api/runs/har/${run_id}"+mkdir -p "${har_dir}"+har_path="${har_dir}/recording.har"+```++Display to user:+```+Capture session started!+Run ID: {run_id}+HAR file: ~/.reverse-api/runs/har/{run_id}/recording.har+```++### Step 2: Launch Browser++Launch browser with HAR recording enabled using Playwright MCP:++```+Call MCP tool: playwright_navigate+Parameters:+ - url: {url} (if provided) or "about:blank"+ - options:+ - record_har: true+ - har_path: ~/.reverse-api/runs/har/{run_id}/recording.har+ - headless: false+```++Display instructions to user:+```+Browser launched with HAR recording enabled.++Instructions:+1. Navigate to the website you want to capture+2. Perform the actions you want to reverse engineer:+ - Log in (to capture authentication)+ - Browse pages (to capture GET requests)+ - Submit forms (to capture POST/PUT requests)+ - Trigger any API calls you need+3. Close the browser when done++The HAR file will be automatically saved when you close the browser.+```++### Step 3: Wait for Browser Close++Monitor the browser session. When the browser window closes, proceed to verification.++### Step 4: Verify HAR Capture++Check that the HAR file was created and contains data:++```bash+if [ -f ~/.reverse-api/runs/har/{run_id}/recording.har ]; then+ size=$(ls -lh ~/.reverse-api/runs/har/{run_id}/recording.har | awk '{print $5}')+ echo "✓ HAR file saved: $size"+else+ echo "✗ Error: HAR file not found"+ exit 1+fi+```++Count network requests in the HAR file:++```python+import json+from pathlib import Path++har_path = Path.home() / ".reverse-api" / "runs" / "har" / "{run_id}" / "recording.har"++with open(har_path, 'r') as f:+ har_data = json.load(f)+ entry_count = len(har_data['log']['entries'])++print(f"Captured {entry_count} network requests")+```++### Step 5: Save Capture Metadata++Save metadata about this capture session:++```bash+cat > ~/.reverse-api/runs/har/{run_id}/metadata.json <<EOF+{+ "run_id": "{run_id}",+ "mode": "capture",+ "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",+ "har_path": "~/.reverse-api/runs/har/{run_id}/recording.har",+ "url": "{url or 'manual navigation'}",+ "entry_count": {entry_count}+}+EOF+```++### Step 6: Display Summary++Present completion summary to user:++```+Capture complete!++Run ID: {run_id}+HAR file: ~/.reverse-api/runs/har/{run_id}/recording.har+File size: {size}+Network requests captured: {entry_count}++Next steps:+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━++Option 1: Analyze and generate API client+ /reverse-api-engineer:engineer {run_id}++Option 2: View captured HAR file+ cat ~/.reverse-api/runs/har/{run_id}/recording.har | jq++Option 3: Filter HAR to see API calls only+ python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py \\+ ~/.reverse-api/runs/har/{run_id}/recording.har --stats++Tip: You can capture multiple times and analyze later!+```++## Advantages Over /manual Mode++- **Faster**: No analysis step during capture+- **Flexible**: Capture once, analyze multiple times with different approaches+- **Shareable**: HAR files can be shared with team members+- **Clean separation**: Capture and analysis are independent steps++## Example Usage++```bash+# Capture with starting URL+/reverse-api-engineer:capture https://api.example.com++# Capture with manual navigation (no starting URL)+/reverse-api-engineer:capture++# After capture, analyze whenever ready+/reverse-api-engineer:engineer abc123def456+```++## Tips++1. **Capture thoroughly**: Include all API calls you want to reverse engineer+2. **Authentication**: If the site requires login, make sure to log in during capture+3. **Pagination**: If listing data, navigate through pages to capture pagination patterns+4. **Multiple captures**: You can run multiple capture sessions and analyze them separately++## Error Handling++**Browser doesn't launch:**+- Ensure Playwright MCP is properly configured+- Check that the browser binary is installed++**HAR file is empty:**+- Make sure you performed actions in the browser+- Some sites may have very few network requests++**HAR file is too large:**+- This is normal for sites with many static assets+- Use har_filter.py to reduce it before analysisdiff --git a/plugins/reverse-api-engineer/commands/engineer.md b/plugins/reverse-api-engineer/commands/engineer.mdindex dcab1a7..305e4c7 100644--- a/plugins/reverse-api-engineer/commands/engineer.md+++ b/plugins/reverse-api-engineer/commands/engineer.md@@ -16,6 +16,24 @@ Ensure the `reverse-engineering-api` skill is loaded for guidance on HAR analysi- `run_id_or_path` (required): Either a run ID (e.g., "abc-123-def") or direct path to HAR file+## Task Tracking++Before starting the workflow, create a todo list to track progress:++```python+TodoWrite([+ {"content": "Verify HAR file exists", "status": "pending", "activeForm": "Verifying HAR file"},+ {"content": "Filter HAR using har_filter.py", "status": "pending", "activeForm": "Filtering HAR"},+ {"content": "Analyze HAR using har_analyze.py", "status": "pending", "activeForm": "Analyzing endpoints"},+ {"content": "Generate API client code", "status": "pending", "activeForm": "Generating code"},+ {"content": "Validate code using har_validate.py", "status": "pending", "activeForm": "Validating code"},+ {"content": "Test API client implementation", "status": "pending", "activeForm": "Testing implementation"},+ {"content": "Generate documentation", "status": "pending", "activeForm": "Writing README"}+])+```++**CRITICAL:** Mark each task as `in_progress` when starting, `completed` when done. NEVER skip tasks - complete all 7 steps.+## Workflow### Step 1: Parse Input Argument@@ -122,40 +140,62 @@ If user provides new description, use it. Otherwise, use original task.### Step 6: Analyze HAR File-Read and parse the HAR file:+**Mark todo in_progress: "Filter HAR using har_filter.py"**-```python-import json-with open(har_path, "r") as f:- har_data = json.load(f)+Filter the HAR file to remove static assets, analytics, and CDN resources:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py {har_path} --output {output_dir}/filtered.har --stats```-Use the `reverse-engineering-api` skill guidance to:+Read and display the filtering statistics:+```+Filtering HAR file: {har_path}+Total entries: {total}+API endpoints found: {filtered_entries}+Removed:+- Static assets: {removed_static}+- Analytics/tracking: {removed_analytics}+- CDN resources: {removed_cdn}++API patterns detected: {api_patterns_found}+```++**Mark todo completed**-1. **Filter relevant entries**: Exclude static assets, analytics, ads, CDN resources-2. **Identify API endpoints**: Look for `/api/`, `/v1/`, `/graphql`, XHR/Fetch requests-3. **Extract patterns**:- - URL patterns and query parameters- - HTTP methods (GET, POST, PUT, DELETE)- - Request/response headers- - Authentication mechanisms- - Request body schemas- - Response structures+**Mark todo in_progress: "Analyze HAR using har_analyze.py"**-Summarize findings:+Extract structured endpoint information from filtered HAR:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.py {output_dir}/filtered.har --output {output_dir}/analysis.json```-Analyzing HAR file: {har_path}-Total entries: {total_count}-Found {count} relevant API endpoints:-- {method} {endpoint} ({description})-- {method} {endpoint} ({description})-...+Read analysis.json and summarize for the user:++```bash+cat {output_dir}/analysis.json+```++Display summary:+```+Analysis complete!-Authentication detected: {auth_type} (Bearer token / API key / None)Base URL: {base_url}+Authentication: {authentication.type} ({authentication.description})+Unique endpoints: {unique_endpoints}+Pagination: {pagination.type if pagination.detected else "Not detected"}++Endpoints found:+{for each endpoint in endpoints:}+- {endpoint.methods} {endpoint.pattern}+ Calls observed: {endpoint.calls_observed}+ Auth required: {endpoint.requires_auth}+ Query params: {endpoint.query_params.required + endpoint.query_params.optional}```+**Mark todo completed**+### Step 7: Generate New Run IDGenerate a new run ID for this regeneration:@@ -312,6 +352,58 @@ print(result)- Test thoroughly before production use```+### Step 8.5: Validate Generated Code++**Mark todo in_progress: "Validate code using har_validate.py"**++Validate the generated API client against the HAR analysis:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.py {output_dir}/api_client.py {output_dir}/analysis.json+```++Check the validation score and issues:++**If validation score < 90:**+1. Read the validation issues carefully+2. Fix each issue in api_client.py:+ - Missing endpoints: Add methods for each missing endpoint+ - Missing auth: Implement authentication from analysis.json+ - Missing error handling: Add try-except blocks and custom exceptions+ - Missing type hints: Add type hints to all methods+3. Save the updated api_client.py+4. Run validation again+5. Repeat until score >= 90++**Example validation loop:**+```+Attempt 1:+- Score: 75+- Issues: 2 missing endpoints, no auth implementation+- Fix: Add missing endpoints, implement bearer token auth+- Regenerate api_client.py++Attempt 2:+- Score: 92+- Issues: 1 info (minor type hint improvement)+- Result: PASS (score >= 90)+```++Display validation result:+```+Validation complete!+Score: {score}/100+Coverage: {coverage.percentage}% ({coverage.endpoints_covered}/{coverage.endpoints_total} endpoints)+Issues: {summary.errors} errors, {summary.warnings} warnings, {summary.info} info++{if score >= 90:}+✓ Code validation passed!+{else:}+✗ Code validation failed - fixing issues and regenerating...+```++**Mark todo completed** (only when score >= 90)+### Step 9: Save Run HistoryUpdate `~/.reverse-api/history.json` with run metadata:diff --git a/plugins/reverse-api-engineer/commands/manual.md b/plugins/reverse-api-engineer/commands/manual.mdindex d35f6df..6a99d97 100644--- a/plugins/reverse-api-engineer/commands/manual.md+++ b/plugins/reverse-api-engineer/commands/manual.md@@ -19,6 +19,26 @@ Ensure the `reverse-engineering-api` skill is loaded for guidance on HAR analysiIf arguments are not provided, prompt the user interactively.+## Task Tracking++Before starting the workflow, create a todo list to track progress:++```python+TodoWrite([+ {"content": "Launch browser with HAR recording", "status": "pending", "activeForm": "Launching browser"},+ {"content": "Wait for user to complete manual navigation", "status": "pending", "activeForm": "Recording traffic"},+ {"content": "Verify HAR capture", "status": "pending", "activeForm": "Verifying HAR file"},+ {"content": "Filter HAR using har_filter.py", "status": "pending", "activeForm": "Filtering HAR"},+ {"content": "Analyze HAR using har_analyze.py", "status": "pending", "activeForm": "Analyzing endpoints"},+ {"content": "Generate API client", "status": "pending", "activeForm": "Generating code"},+ {"content": "Validate code using har_validate.py", "status": "pending", "activeForm": "Validating code"},+ {"content": "Test implementation", "status": "pending", "activeForm": "Testing API client"},+ {"content": "Generate documentation", "status": "pending", "activeForm": "Writing README"}+])+```++**CRITICAL:** Mark each task as `in_progress` when starting, `completed` when done. NEVER skip tasks - complete all 9 steps.+## Workflow### Step 1: Gather Task Information@@ -90,36 +110,62 @@ Exit with error### Step 5: Analyze HAR File-Read and parse the HAR file:-```python-import json-with open("~/.reverse-api/runs/har/{run_id}/recording.har", "r") as f:- har_data = json.load(f)+**Mark todo in_progress: "Filter HAR using har_filter.py"**++Filter the HAR file to remove static assets, analytics, and CDN resources:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py ~/.reverse-api/runs/har/{run_id}/recording.har --output {output_dir}/filtered.har --stats```-Use the `reverse-engineering-api` skill guidance to:+Read and display the filtering statistics:+```+Filtering complete!+Total entries: {total}+API endpoints found: {filtered_entries}+Removed:+- Static assets: {removed_static}+- Analytics/tracking: {removed_analytics}+- CDN resources: {removed_cdn}++API patterns detected: {api_patterns_found}+```++**Mark todo completed**-1. **Filter relevant entries**: Exclude static assets, analytics, ads, CDN resources-2. **Identify API endpoints**: Look for `/api/`, `/v1/`, `/graphql`, XHR/Fetch requests-3. **Extract patterns**:- - URL patterns and query parameters- - HTTP methods (GET, POST, PUT, DELETE)- - Request/response headers- - Authentication mechanisms- - Request body schemas- - Response structures+**Mark todo in_progress: "Analyze HAR using har_analyze.py"**-Summarize findings:+Extract structured endpoint information from filtered HAR:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.py {output_dir}/filtered.har --output {output_dir}/analysis.json+```++Read analysis.json and summarize for the user:++```bash+cat {output_dir}/analysis.json```-Found {count} relevant API endpoints:-- {method} {endpoint} ({description})-- {method} {endpoint} ({description})-...-Authentication detected: {auth_type} (Bearer token / API key / None)+Display summary:+```+Analysis complete!+Base URL: {base_url}+Authentication: {authentication.type} ({authentication.description})+Unique endpoints: {unique_endpoints}+Pagination: {pagination.type if pagination.detected else "Not detected"}++Endpoints found:+{for each endpoint in endpoints:}+- {endpoint.methods} {endpoint.pattern}+ Calls observed: {endpoint.calls_observed}+ Auth required: {endpoint.requires_auth}+ Query params: {endpoint.query_params.required + endpoint.query_params.optional}```+**Mark todo completed**+### Step 6: Generate API ClientGenerate a descriptive name for the script based on the task:@@ -261,6 +307,58 @@ print(result)- Test thoroughly before production use```+### Step 6.5: Validate Generated Code++**Mark todo in_progress: "Validate code using har_validate.py"**++Validate the generated API client against the HAR analysis:++```bash+python plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.py {output_dir}/api_client.py {output_dir}/analysis.json+```++Check the validation score and issues:++**If validation score < 90:**+1. Read the validation issues carefully+2. Fix each issue in api_client.py:+ - Missing endpoints: Add methods for each missing endpoint+ - Missing auth: Implement authentication from analysis.json+ - Missing error handling: Add try-except blocks and custom exceptions+ - Missing type hints: Add type hints to all methods+3. Save the updated api_client.py+4. Run validation again+5. Repeat until score >= 90++**Example validation loop:**+```+Attempt 1:+- Score: 75+- Issues: 2 missing endpoints, no auth implementation+- Fix: Add missing endpoints, implement bearer token auth+- Regenerate api_client.py++Attempt 2:+- Score: 92+- Issues: 1 info (minor type hint improvement)+- Result: PASS (score >= 90)+```++Display validation result:+```+Validation complete!+Score: {score}/100+Coverage: {coverage.percentage}% ({coverage.endpoints_covered}/{coverage.endpoints_total} endpoints)+Issues: {summary.errors} errors, {summary.warnings} warnings, {summary.info} info++{if score >= 90:}+✓ Code validation passed!+{else:}+✗ Code validation failed - fixing issues and regenerating...+```++**Mark todo completed** (only when score >= 90)+### Step 7: Save Run HistoryUpdate `~/.reverse-api/history.json` with run metadata:diff --git a/plugins/reverse-api-engineer/skills/reverse-engineering-api/SKILL.md b/plugins/reverse-api-engineer/skills/reverse-engineering-api/SKILL.mdindex a8c79ec..be80b96 100644--- a/plugins/reverse-api-engineer/skills/reverse-engineering-api/SKILL.md+++ b/plugins/reverse-api-engineer/skills/reverse-engineering-api/SKILL.md@@ -27,6 +27,82 @@ This skill enables you to reverse engineer web APIs by:[User Task] -> [Browser Capture] -> [HAR Analysis] -> [API Client Generation] -> [Testing & Refinement]```+## Phase 0: Preparation (Using HAR Helper Scripts)++### Available Helper Scripts++This skill provides Python utilities for HAR analysis located at:++**Script Directory:** `plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/`++**Available Scripts:**+- `har_filter.py` - Filter HAR files to API endpoints only+- `har_analyze.py` - Extract structured endpoint information+- `har_validate.py` - Validate generated code against HAR analysis+- `har_utils.py` - Shared utility functions++### Script Usage Pattern++Use these scripts in sequence for optimal code generation:++```bash+# 1. Filter HAR to remove noise (static assets, analytics, CDN)+python {SKILL_DIR}/scripts/har_filter.py {har_path} --output filtered.har --stats++# 2. Analyze endpoints and extract patterns+python {SKILL_DIR}/scripts/har_analyze.py filtered.har --output analysis.json++# 3. Read analysis for code generation guidance+cat analysis.json++# 4. Generate API client code based on analysis++# 5. Validate generated code+python {SKILL_DIR}/scripts/har_validate.py api_client.py analysis.json+```++### Why Use These Scripts?++**har_filter.py benefits:**+- Reduces HAR file size by 80-90% (removes noise)+- Focuses analysis on actual API calls+- Significantly improves code generation quality+- Outputs statistics showing what was filtered++**har_analyze.py benefits:**+- Provides structured endpoint information+- Detects authentication patterns automatically+- Identifies pagination mechanisms+- Extracts request/response schemas+- Groups endpoints by pattern++**har_validate.py benefits:**+- Ensures all endpoints are implemented+- Validates authentication handling+- Checks for proper error handling+- Calculates coverage score (must be >= 90)+- Identifies missing features++### Task Tracking++Use TodoWrite to track workflow progress:+- Mark tasks as `pending`, `in_progress`, or `completed`+- Only ONE task should be `in_progress` at a time+- Complete ALL tasks - never stop early++**Example TodoWrite usage:**+```python+TodoWrite([+ {"content": "Filter HAR using har_filter.py", "status": "in_progress", "activeForm": "Filtering HAR"},+ {"content": "Analyze HAR using har_analyze.py", "status": "pending", "activeForm": "Analyzing endpoints"},+ {"content": "Generate API client", "status": "pending", "activeForm": "Generating code"},+ {"content": "Validate using har_validate.py", "status": "pending", "activeForm": "Validating code"},+ {"content": "Test implementation", "status": "pending", "activeForm": "Testing API client"}+])+```++**CRITICAL:** Task tracking ensures complete workflow execution. Never skip tasks or stop early.+## Phase 1: Browser Capture with HAR Recording### Starting the Browserdiff --git a/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.py b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.pynew file mode 100644index 0000000..d5b6113--- /dev/null+++ b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_analyze.py@@ -0,0 +1,554 @@+#!/usr/bin/env python3+"""+HAR Analyzer - Extract structured endpoint information from HAR files.++This script analyzes filtered HAR files to extract:+- API endpoints with path parameters+- Authentication patterns+- Request/response schemas+- Pagination patterns+- Query parameters++Usage:+ python har_analyze.py <filtered.har> --output <analysis.json>+"""++import argparse+import json+import sys+from collections import defaultdict+from pathlib import Path+from typing import Dict, List, Any, Optional, Set, Tuple+from urllib.parse import urlparse, parse_qs++from har_utils import (+ load_har,+ save_json,+ extract_url_parts,+ get_base_url,+ is_path_parameter,+ get_request_body,+ get_response_body,+ get_request_headers,+ get_response_headers,+)+++def detect_auth(entries: List[Dict[str, Any]]) -> Dict[str, Any]:+ """+ Detect authentication patterns from HAR entries.++ Args:+ entries: List of HAR entries++ Returns:+ Dictionary describing authentication mechanism+ """+ auth_headers = defaultdict(int)+ auth_cookies = defaultdict(int)+ auth_query_params = defaultdict(int)++ for entry in entries:+ request_headers = get_request_headers(entry)+ request = entry.get('request', {})+ url = request.get('url', '')+ parts = extract_url_parts(url)+ query_params = parts.get('query_params', {})++ # Check for Authorization header+ for header_name, header_value in request_headers.items():+ header_lower = header_name.lower()++ if header_lower == 'authorization':+ # Determine auth type from value+ value_lower = header_value.lower()+ if value_lower.startswith('bearer '):+ auth_headers['bearer_token'] += 1+ elif value_lower.startswith('basic '):+ auth_headers['basic_auth'] += 1+ elif 'oauth' in value_lower:+ auth_headers['oauth'] += 1+ else:+ auth_headers['unknown_token'] += 1++ # Check for X-API-Key or similar+ elif 'api' in header_lower and 'key' in header_lower:+ auth_headers['api_key_header'] += 1++ # Check for CSRF tokens+ elif 'csrf' in header_lower or 'xsrf' in header_lower:+ auth_headers['csrf_token'] += 1++ # Check for session tokens+ elif 'token' in header_lower:+ auth_headers['custom_token'] += 1++ # Check for cookies+ elif header_lower == 'cookie':+ if 'session' in header_value.lower():+ auth_cookies['session_cookie'] += 1+ elif 'token' in header_value.lower():+ auth_cookies['token_cookie'] += 1+ else:+ auth_cookies['generic_cookie'] += 1++ # Check for API keys in query parameters+ for param_name in query_params.keys():+ param_lower = param_name.lower()+ if 'api' in param_lower and 'key' in param_lower:+ auth_query_params['api_key_query'] += 1+ elif 'token' in param_lower or 'auth' in param_lower:+ auth_query_params['token_query'] += 1++ # Determine primary auth method+ all_auth_methods = {}+ all_auth_methods.update(auth_headers)+ all_auth_methods.update(auth_cookies)+ all_auth_methods.update(auth_query_params)++ if not all_auth_methods:+ return {+ 'type': 'none',+ 'location': None,+ 'header_name': None,+ 'pattern': None,+ 'description': 'No authentication detected (public API)',+ }++ # Find most common auth method+ primary_method = max(all_auth_methods.items(), key=lambda x: x[1])[0]++ # Map to structured auth info+ auth_info = {+ 'type': primary_method,+ 'detected_methods': list(all_auth_methods.keys()),+ }++ # Add details based on type+ if primary_method == 'bearer_token':+ auth_info.update({+ 'location': 'header',+ 'header_name': 'Authorization',+ 'pattern': 'Bearer {token}',+ 'description': 'JWT or Bearer token authentication',+ })+ elif primary_method == 'api_key_header':+ auth_info.update({+ 'location': 'header',+ 'header_name': 'X-API-Key', # Common convention+ 'pattern': '{api_key}',+ 'description': 'API key in header',+ })+ elif primary_method == 'api_key_query':+ auth_info.update({+ 'location': 'query',+ 'header_name': None,+ 'pattern': '?api_key={key}',+ 'description': 'API key in query parameter',+ })+ elif 'session_cookie' in primary_method or 'cookie' in primary_method:+ auth_info.update({+ 'location': 'cookie',+ 'header_name': 'Cookie',+ 'pattern': 'session={session_id}',+ 'description': 'Session-based authentication with cookies',+ })+ elif primary_method == 'basic_auth':+ auth_info.update({+ 'location': 'header',+ 'header_name': 'Authorization',+ 'pattern': 'Basic {base64_credentials}',+ 'description': 'HTTP Basic Authentication',+ })+ else:+ auth_info.update({+ 'location': 'header',+ 'header_name': 'Authorization',+ 'pattern': '{token}',+ 'description': f'Custom authentication: {primary_method}',+ })++ return auth_info+++def infer_path_params(paths: List[str]) -> str:+ """+ Infer path parameter pattern from multiple paths.++ Example:+ ["/api/users/123", "/api/users/456"] -> "/api/users/{id}"++ Args:+ paths: List of URL paths++ Returns:+ Path pattern with {param} placeholders+ """+ if not paths:+ return ''++ # Split all paths into segments+ all_segments = [path.strip('/').split('/') for path in paths]++ # Find common length+ min_length = min(len(segments) for segments in all_segments)++ # Build pattern+ pattern_segments = []+ for i in range(min_length):+ segment_values = [segments[i] for segments in all_segments]++ # Check if all values are the same (static segment)+ if len(set(segment_values)) == 1:+ pattern_segments.append(segment_values[0])+ else:+ # Values differ - likely a parameter+ # Check what type of parameter+ if all(is_path_parameter(val) for val in segment_values):+ # Determine parameter name+ if all(val.isdigit() for val in segment_values):+ param_name = 'id'+ else:+ param_name = 'uuid' # or 'id' for UUIDs+ pattern_segments.append(f'{{{param_name}}}')+ else:+ # Mixed values - keep first as example+ pattern_segments.append(segment_values[0])++ pattern = '/' + '/'.join(pattern_segments)+ return pattern+++def group_endpoints(entries: List[Dict[str, Any]]) -> List[Dict[str, Any]]:+ """+ Group HAR entries by endpoint pattern.++ Args:+ entries: List of HAR entries++ Returns:+ List of endpoint dictionaries with patterns and metadata+ """+ # Group by method + base pattern+ groups = defaultdict(lambda: {+ 'urls': [],+ 'entries': [],+ 'query_params': defaultdict(int),+ 'request_bodies': [],+ 'response_bodies': [],+ })++ for entry in entries:+ request = entry.get('request', {})+ method = request.get('method', 'GET')+ url = request.get('url', '')++ parts = extract_url_parts(url)+ path = parts.get('path', '')+ query_params = parts.get('query_params', {})++ # Create a preliminary grouping key+ # We'll refine this later to detect path parameters+ key = (method, path)+ groups[key]['urls'].append(url)+ groups[key]['entries'].append(entry)++ # Track query parameters+ for param_name in query_params.keys():+ groups[key]['query_params'][param_name] += 1++ # Collect request/response bodies+ req_body = get_request_body(entry)+ if req_body:+ groups[key]['request_bodies'].append(req_body)++ resp_body = get_response_body(entry)+ if resp_body:+ groups[key]['response_bodies'].append(resp_body)++ # Now refine groups to detect path parameters+ refined_endpoints = []++ for (method, path), group_data in groups.items():+ # If multiple URLs with same method+path structure, infer pattern+ unique_paths = list(set(parts.get('path', '') for parts in+ [extract_url_parts(url) for url in group_data['urls']]))++ if len(unique_paths) > 1:+ # Multiple different paths - infer pattern+ pattern = infer_path_params(unique_paths)+ else:+ pattern = path++ # Determine required vs optional query params+ total_calls = len(group_data['entries'])+ required_params = []+ optional_params = []++ for param_name, count in group_data['query_params'].items():+ if count == total_calls:+ required_params.append(param_name)+ else:+ optional_params.append(param_name)++ # Merge request/response bodies to infer schema+ request_schema = merge_schemas(group_data['request_bodies'])+ response_schema = merge_schemas(group_data['response_bodies'])++ # Check if auth required (heuristic: has auth headers)+ has_auth = any(+ 'authorization' in get_request_headers(entry).keys() or+ 'cookie' in (h.lower() for h in get_request_headers(entry).keys())+ for entry in group_data['entries']+ )++ endpoint_info = {+ 'pattern': pattern,+ 'methods': [method],+ 'calls_observed': total_calls,+ 'query_params': {+ 'required': sorted(required_params),+ 'optional': sorted(optional_params),+ },+ 'request_body_schema': request_schema,+ 'response_schema': response_schema,+ 'requires_auth': has_auth,+ }++ refined_endpoints.append(endpoint_info)++ # Merge endpoints with same pattern but different methods+ pattern_groups = defaultdict(list)+ for endpoint in refined_endpoints:+ pattern_groups[endpoint['pattern']].append(endpoint)++ final_endpoints = []+ for pattern, endpoints in pattern_groups.items():+ if len(endpoints) == 1:+ final_endpoints.append(endpoints[0])+ else:+ # Merge multiple methods for same pattern+ merged = {+ 'pattern': pattern,+ 'methods': sorted(set(method for ep in endpoints for method in ep['methods'])),+ 'calls_observed': sum(ep['calls_observed'] for ep in endpoints),+ 'query_params': endpoints[0]['query_params'], # Use first as representative+ 'request_body_schema': merge_schemas([ep['request_body_schema'] for ep in endpoints if ep['request_body_schema']]),+ 'response_schema': merge_schemas([ep['response_schema'] for ep in endpoints if ep['response_schema']]),+ 'requires_auth': any(ep['requires_auth'] for ep in endpoints),+ }+ final_endpoints.append(merged)++ # Sort by pattern for consistency+ final_endpoints.sort(key=lambda x: x['pattern'])++ return final_endpoints+++def merge_schemas(bodies: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]:+ """+ Merge multiple JSON bodies to infer a common schema.++ Args:+ bodies: List of JSON bodies++ Returns:+ Inferred schema or None+ """+ if not bodies:+ return None++ # Simple schema inference: collect all keys and their types+ schema = {}++ for body in bodies:+ if not isinstance(body, dict):+ continue++ for key, value in body.items():+ if key not in schema:+ schema[key] = type(value).__name__++ return schema if schema else None+++def detect_pagination(entries: List[Dict[str, Any]]) -> Dict[str, Any]:+ """+ Detect pagination patterns from HAR entries.++ Args:+ entries: List of HAR entries++ Returns:+ Dictionary describing pagination pattern+ """+ pagination_params = defaultdict(int)++ for entry in entries:+ request = entry.get('request', {})+ url = request.get('url', '')+ parts = extract_url_parts(url)+ query_params = parts.get('query_params', {})++ # Check for common pagination parameters+ for param_name in query_params.keys():+ param_lower = param_name.lower()++ if param_lower in ['page', 'p']:+ pagination_params['page'] += 1+ elif param_lower in ['limit', 'per_page', 'page_size', 'size']:+ pagination_params['limit'] += 1+ elif param_lower in ['offset', 'skip']:+ pagination_params['offset'] += 1+ elif param_lower in ['cursor', 'next', 'continuation']:+ pagination_params['cursor'] += 1++ if not pagination_params:+ return {+ 'detected': False,+ 'type': None,+ 'params': [],+ }++ # Determine pagination type+ if 'page' in pagination_params:+ return {+ 'detected': True,+ 'type': 'page',+ 'params': ['page', 'limit'] if 'limit' in pagination_params else ['page'],+ 'description': 'Page-based pagination',+ }+ elif 'offset' in pagination_params:+ return {+ 'detected': True,+ 'type': 'offset',+ 'params': ['offset', 'limit'] if 'limit' in pagination_params else ['offset'],+ 'description': 'Offset-based pagination',+ }+ elif 'cursor' in pagination_params:+ return {+ 'detected': True,+ 'type': 'cursor',+ 'params': ['cursor', 'limit'] if 'limit' in pagination_params else ['cursor'],+ 'description': 'Cursor-based pagination',+ }+ else:+ return {+ 'detected': True,+ 'type': 'unknown',+ 'params': list(pagination_params.keys()),+ 'description': 'Custom pagination pattern',+ }+++def analyze_har(har: Dict[str, Any]) -> Dict[str, Any]:+ """+ Analyze HAR file and extract structured information.++ Args:+ har: Parsed HAR data++ Returns:+ Analysis dictionary with endpoints, auth, pagination, etc.+ """+ entries = har['log']['entries']++ if not entries:+ return {+ 'error': 'No entries found in HAR file',+ 'base_url': None,+ 'authentication': None,+ 'endpoints': [],+ 'pagination': None,+ }++ # Determine base URL (most common)+ base_urls = defaultdict(int)+ for entry in entries:+ request = entry.get('request', {})+ url = request.get('url', '')+ base = get_base_url(url)+ if base:+ base_urls[base] += 1++ base_url = max(base_urls.items(), key=lambda x: x[1])[0] if base_urls else None++ # Detect authentication+ authentication = detect_auth(entries)++ # Group and analyze endpoints+ endpoints = group_endpoints(entries)++ # Detect pagination+ pagination = detect_pagination(entries)++ # Build analysis result+ analysis = {+ 'base_url': base_url,+ 'authentication': authentication,+ 'endpoints': endpoints,+ 'pagination': pagination,+ 'total_entries_analyzed': len(entries),+ 'unique_endpoints': len(endpoints),+ }++ return analysis+++def main():+ """Main entry point for CLI."""+ parser = argparse.ArgumentParser(+ description='Analyze HAR files to extract endpoint information',+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ # Analyze HAR and save to JSON+ python har_analyze.py filtered.har --output analysis.json++ # Print analysis to stdout+ python har_analyze.py filtered.har+ """+ )++ parser.add_argument(+ 'input',+ type=str,+ help='Input HAR file path (filtered recommended)'+ )++ parser.add_argument(+ '--output',+ type=str,+ help='Output analysis JSON file path'+ )++ args = parser.parse_args()++ # Validate input file+ input_path = Path(args.input).expanduser().resolve()+ if not input_path.exists():+ print(f"Error: Input file not found: {input_path}", file=sys.stderr)+ sys.exit(1)++ try:+ # Load and analyze HAR+ har_data = load_har(input_path)+ analysis = analyze_har(har_data)++ # Save or print analysis+ if args.output:+ output_path = Path(args.output).expanduser().resolve()+ save_json(analysis, output_path, pretty=True)+ print(f"Analysis saved to: {output_path}")+ else:+ print(json.dumps(analysis, indent=2))++ except Exception as e:+ print(f"Error: {e}", file=sys.stderr)+ import traceback+ traceback.print_exc()+ sys.exit(1)+++if __name__ == '__main__':+ main()diff --git a/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.pynew file mode 100644index 0000000..a7f2642--- /dev/null+++ b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_filter.py@@ -0,0 +1,408 @@+#!/usr/bin/env python3+"""+HAR Filter - Filter HAR files to API-relevant entries only.++This script removes static assets, analytics, tracking, and CDN resources+from HAR files, keeping only API-relevant requests.++Usage:+ python har_filter.py <input.har> --output <filtered.har> [--stats]+"""++import argparse+import json+import sys+from pathlib import Path+from typing import Dict, List, Tuple, Any+from urllib.parse import urlparse++from har_utils import load_har, save_json, get_content_type, extract_url_parts+++# File extensions to skip (static assets)+SKIP_EXTENSIONS = {+ # JavaScript/CSS+ '.js', '.jsx', '.ts', '.tsx', '.css', '.scss', '.sass', '.less',+ # Images+ '.png', '.jpg', '.jpeg', '.gif', '.webp', '.svg', '.ico', '.bmp', '.tiff',+ # Fonts+ '.woff', '.woff2', '.ttf', '.otf', '.eot',+ # Media+ '.mp4', '.webm', '.ogg', '.mp3', '.wav', '.flac',+ # Documents (usually not API responses)+ '.pdf', '.doc', '.docx', '.xls', '.xlsx', '.zip', '.tar', '.gz',+ # Other static files+ '.xml', '.txt', '.csv', '.html', '.htm',+ # Source maps and manifests+ '.map', '.manifest',+}++# Domain patterns to skip (analytics, tracking, CDN)+SKIP_DOMAIN_PATTERNS = [+ # Analytics+ 'google-analytics.com',+ 'analytics.google.com',+ 'googletagmanager.com',+ 'doubleclick.net',+ 'facebook.com/tr',+ 'facebook.net',+ 'connect.facebook.net',+ 'mixpanel.com',+ 'segment.com',+ 'segment.io',+ 'amplitude.com',+ 'hotjar.com',+ 'fullstory.com',+ 'heap.io',+ 'clarity.ms',+ # Ads+ 'googlesyndication.com',+ 'adservice.google.com',+ 'advertising.com',+ # CDN patterns+ 'cloudfront.net',+ 'fastly.net',+ 'akamai.net',+ 'akamaihd.net',+ 'cdn.jsdelivr.net',+ 'unpkg.com',+ 'cdnjs.cloudflare.com',+ # Social media tracking+ 'twitter.com/i/',+ 'linkedin.com/px/',+ 'snapchat.com/tr',+ # Other tracking+ 'bugsnag.com',+ 'sentry.io',+ 'newrelic.com',+]++# URL path patterns that indicate API endpoints+API_PATH_PATTERNS = [+ '/api/',+ '/v1/',+ '/v2/',+ '/v3/',+ '/v4/',+ '/rest/',+ '/graphql',+ '/query',+ '/mutation',+ '/rpc/',+ '/_api/',+]+++def get_skip_extensions() -> set:+ """Get set of file extensions to skip."""+ return SKIP_EXTENSIONS+++def get_skip_domain_patterns() -> list:+ """Get list of domain patterns to skip."""+ return SKIP_DOMAIN_PATTERNS+++def should_skip_domain(hostname: str) -> bool:+ """+ Check if a domain should be skipped.++ Args:+ hostname: Hostname from URL++ Returns:+ True if domain should be skipped+ """+ if not hostname:+ return False++ hostname_lower = hostname.lower()++ for pattern in SKIP_DOMAIN_PATTERNS:+ if pattern in hostname_lower:+ return True++ # Skip common CDN patterns like cdn.*, static.*, assets.*+ if hostname_lower.startswith(('cdn.', 'static.', 'assets.', 'media.')):+ return True++ return False+++def should_skip_extension(path: str) -> bool:+ """+ Check if file extension should be skipped.++ Args:+ path: URL path++ Returns:+ True if extension should be skipped+ """+ path_lower = path.lower()++ for ext in SKIP_EXTENSIONS:+ if path_lower.endswith(ext):+ return True++ return False+++def has_api_pattern(path: str) -> bool:+ """+ Check if path contains API patterns.++ Args:+ path: URL path++ Returns:+ True if path looks like an API endpoint+ """+ path_lower = path.lower()++ for pattern in API_PATH_PATTERNS:+ if pattern in path_lower:+ return True++ return False+++def is_xhr_fetch(entry: Dict[str, Any]) -> bool:+ """+ Check if request is XHR/Fetch (likely an API call).++ Args:+ entry: HAR entry++ Returns:+ True if request appears to be XHR/Fetch+ """+ request = entry.get('request', {})+ headers = request.get('headers', [])++ # Check for XHR/Fetch indicators in headers+ for header in headers:+ if not isinstance(header, dict):+ continue++ name = header.get('name', '').lower()+ value = header.get('value', '').lower()++ # X-Requested-With: XMLHttpRequest+ if name == 'x-requested-with' and 'xmlhttprequest' in value:+ return True++ # Fetch API usually sets specific Accept headers+ if name == 'accept' and ('application/json' in value or 'application/*' in value):+ return True++ return False+++def is_json_response(entry: Dict[str, Any]) -> bool:+ """+ Check if response is JSON.++ Args:+ entry: HAR entry++ Returns:+ True if response content-type is JSON+ """+ content_type = get_content_type(entry)+ if content_type:+ return 'json' in content_type.lower()+ return False+++def categorize_entry(entry: Dict[str, Any]) -> str:+ """+ Categorize a HAR entry.++ Args:+ entry: HAR entry++ Returns:+ Category: "api", "static", "analytics", "cdn", or "other"+ """+ request = entry.get('request', {})+ url = request.get('url', '')++ parts = extract_url_parts(url)+ hostname = parts.get('hostname', '')+ path = parts.get('path', '')++ # Check domain-based skips first+ if should_skip_domain(hostname):+ # Categorize analytics vs CDN+ hostname_lower = hostname.lower()+ if any(term in hostname_lower for term in ['analytics', 'tracking', 'ads', 'doubleclick', 'facebook.com/tr']):+ return 'analytics'+ return 'cdn'++ # Check file extension+ if should_skip_extension(path):+ return 'static'++ # Check for API patterns+ if has_api_pattern(path):+ return 'api'++ # Check if XHR/Fetch+ if is_xhr_fetch(entry):+ return 'api'++ # Check if JSON response+ if is_json_response(entry):+ return 'api'++ return 'other'+++def is_api_endpoint(entry: Dict[str, Any]) -> bool:+ """+ Determine if a HAR entry is an API endpoint.++ Args:+ entry: HAR entry++ Returns:+ True if entry should be kept as API endpoint+ """+ category = categorize_entry(entry)+ return category == 'api'+++def filter_har(har_path: Path) -> Tuple[Dict[str, Any], Dict[str, Any]]:+ """+ Filter HAR file to API-relevant entries only.++ Args:+ har_path: Path to input HAR file++ Returns:+ Tuple of (filtered_har, statistics)+ """+ har_data = load_har(har_path)++ entries = har_data['log']['entries']+ total_entries = len(entries)++ # Categorize all entries+ categorized = {+ 'api': [],+ 'static': [],+ 'analytics': [],+ 'cdn': [],+ 'other': [],+ }++ for entry in entries:+ category = categorize_entry(entry)+ categorized[category].append(entry)++ # Build filtered HAR with only API entries+ filtered_har = {+ 'log': {+ 'version': har_data['log'].get('version', '1.2'),+ 'creator': har_data['log'].get('creator', {}),+ 'pages': har_data['log'].get('pages', []),+ 'entries': categorized['api'],+ }+ }++ # Collect API patterns found+ api_patterns = set()+ for entry in categorized['api']:+ request = entry.get('request', {})+ url = request.get('url', '')+ parts = extract_url_parts(url)+ path = parts.get('path', '')++ for pattern in API_PATH_PATTERNS:+ if pattern in path.lower():+ api_patterns.add(pattern)++ # Build statistics+ stats = {+ 'total_entries': total_entries,+ 'filtered_entries': len(categorized['api']),+ 'removed_static': len(categorized['static']),+ 'removed_analytics': len(categorized['analytics']),+ 'removed_cdn': len(categorized['cdn']),+ 'removed_other': len(categorized['other']),+ 'api_patterns_found': sorted(list(api_patterns)),+ 'filter_ratio': round(len(categorized['api']) / total_entries * 100, 1) if total_entries > 0 else 0,+ }++ return filtered_har, stats+++def main():+ """Main entry point for CLI."""+ parser = argparse.ArgumentParser(+ description='Filter HAR files to API-relevant entries only',+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ # Filter HAR and save to new file+ python har_filter.py recording.har --output filtered.har++ # Filter and display stats+ python har_filter.py recording.har --output filtered.har --stats++ # Stats only (no output file)+ python har_filter.py recording.har --stats+ """+ )++ parser.add_argument(+ 'input',+ type=str,+ help='Input HAR file path'+ )++ parser.add_argument(+ '--output',+ type=str,+ help='Output filtered HAR file path'+ )++ parser.add_argument(+ '--stats',+ action='store_true',+ help='Print filtering statistics'+ )++ args = parser.parse_args()++ # Validate input file+ input_path = Path(args.input).expanduser().resolve()+ if not input_path.exists():+ print(f"Error: Input file not found: {input_path}", file=sys.stderr)+ sys.exit(1)++ try:+ # Filter HAR+ filtered_har, stats = filter_har(input_path)++ # Save filtered HAR if output specified+ if args.output:+ output_path = Path(args.output).expanduser().resolve()+ save_json(filtered_har, output_path, pretty=True)+ stats['filtered_har_path'] = str(output_path)++ # Print stats if requested+ if args.stats:+ print(json.dumps(stats, indent=2))+ elif not args.output:+ # If neither output nor stats requested, print stats by default+ print(json.dumps(stats, indent=2))++ except Exception as e:+ print(f"Error: {e}", file=sys.stderr)+ sys.exit(1)+++if __name__ == '__main__':+ main()diff --git a/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_utils.py b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_utils.pynew file mode 100644index 0000000..a96b2aa--- /dev/null+++ b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_utils.py@@ -0,0 +1,405 @@+#!/usr/bin/env python3+"""+Shared utilities for HAR analysis scripts.++This module provides common functions used across har_filter.py,+har_analyze.py, and har_validate.py.+"""++import json+import re+from pathlib import Path+from typing import Any, Dict, Optional+from urllib.parse import urlparse, parse_qs+++def load_har(har_path: Path) -> Dict[str, Any]:+ """+ Load and parse a HAR file.++ Args:+ har_path: Path to the HAR file++ Returns:+ Parsed HAR data as dictionary++ Raises:+ FileNotFoundError: If HAR file doesn't exist+ json.JSONDecodeError: If HAR file is invalid JSON+ ValueError: If HAR structure is invalid+ """+ if not har_path.exists():+ raise FileNotFoundError(f"HAR file not found: {har_path}")++ with open(har_path, 'r', encoding='utf-8') as f:+ har_data = json.load(f)++ if not validate_har(har_data):+ raise ValueError("Invalid HAR structure")++ return har_data+++def validate_har(har: Dict[str, Any]) -> bool:+ """+ Validate HAR structure.++ Args:+ har: HAR data dictionary++ Returns:+ True if valid, False otherwise+ """+ if not isinstance(har, dict):+ return False++ if 'log' not in har:+ return False++ log = har['log']+ if not isinstance(log, dict):+ return False++ if 'entries' not in log:+ return False++ return isinstance(log['entries'], list)+++def get_content_type(entry: Dict[str, Any]) -> Optional[str]:+ """+ Extract content-type from HAR entry.++ Args:+ entry: HAR entry dictionary++ Returns:+ Content-type string or None if not found+ """+ response = entry.get('response', {})+ headers = response.get('headers', [])++ for header in headers:+ if isinstance(header, dict):+ name = header.get('name', '').lower()+ if name == 'content-type':+ return header.get('value', '')++ return None+++def parse_json_safe(text: str) -> Optional[Dict[str, Any]]:+ """+ Safely parse JSON text.++ Args:+ text: JSON string to parse++ Returns:+ Parsed dictionary or None if parsing fails+ """+ if not text or not isinstance(text, str):+ return None++ try:+ return json.loads(text)+ except (json.JSONDecodeError, ValueError):+ return None+++def is_uuid(s: str) -> bool:+ """+ Check if a string is a UUID.++ Args:+ s: String to check++ Returns:+ True if string matches UUID pattern+ """+ uuid_pattern = re.compile(+ r'^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$',+ re.IGNORECASE+ )+ return bool(uuid_pattern.match(s))+++def is_numeric_id(s: str) -> bool:+ """+ Check if a string is a numeric ID.++ Args:+ s: String to check++ Returns:+ True if string is all digits+ """+ return s.isdigit()+++def is_path_parameter(s: str) -> bool:+ """+ Check if a URL path segment is likely a parameter.++ Args:+ s: Path segment to check++ Returns:+ True if likely a parameter (UUID, numeric ID, etc.)+ """+ if not s or s.startswith('.'):+ return False++ # UUID+ if is_uuid(s):+ return True++ # Numeric ID (but not too short, could be version number)+ if is_numeric_id(s) and len(s) >= 3:+ return True++ # Long alphanumeric strings (likely IDs)+ if len(s) > 16 and s.isalnum():+ return True++ # Base64-like strings+ if len(s) > 20 and re.match(r'^[A-Za-z0-9_-]+$', s):+ return True++ return False+++def sanitize_method_name(endpoint: str, method: str = 'GET') -> str:+ """+ Convert an endpoint pattern to a Python method name.++ Examples:+ "/api/users/{id}" -> "get_user"+ "/api/v1/products/{id}/reviews" -> "get_product_reviews"+ "/graphql" -> "post_graphql"++ Args:+ endpoint: URL path pattern (e.g., "/api/users/{id}")+ method: HTTP method (e.g., "GET", "POST")++ Returns:+ Python method name (snake_case)+ """+ # Remove leading/trailing slashes+ path = endpoint.strip('/')++ # Remove common prefixes+ for prefix in ['api/', 'v1/', 'v2/', 'v3/', 'api/v1/', 'api/v2/', 'api/v3/']:+ if path.startswith(prefix):+ path = path[len(prefix):]++ # Split by slashes and filter out parameter placeholders+ parts = []+ for segment in path.split('/'):+ # Skip parameter placeholders like {id}, {uuid}+ if segment.startswith('{') and segment.endswith('}'):+ continue+ # Skip numeric segments (version numbers)+ if segment.isdigit():+ continue+ # Add the segment+ if segment:+ parts.append(segment)++ # Join parts with underscores+ if not parts:+ # Fallback for paths like /api/{id}+ parts = ['item']++ base_name = '_'.join(parts)++ # Singularize if method suggests single item+ if method in ['GET', 'PUT', 'PATCH', 'DELETE']:+ # Check if path has parameter (suggesting single item)+ if '{id}' in endpoint or '{uuid}' in endpoint:+ # Simple singularization (remove trailing 's')+ if base_name.endswith('ies'):+ base_name = base_name[:-3] + 'y'+ elif base_name.endswith('s') and not base_name.endswith('ss'):+ base_name = base_name[:-1]++ # Add method prefix+ method_prefix = method.lower()+ if method_prefix == 'delete':+ method_prefix = 'remove'++ method_name = f"{method_prefix}_{base_name}"++ # Clean up any double underscores+ method_name = re.sub(r'_+', '_', method_name)++ # Ensure it's a valid Python identifier+ method_name = re.sub(r'[^a-z0-9_]', '_', method_name)++ return method_name+++def extract_url_parts(url: str) -> Dict[str, Any]:+ """+ Extract components from a URL.++ Args:+ url: Full URL string++ Returns:+ Dictionary with scheme, netloc, path, query_params+ """+ parsed = urlparse(url)+ query_params = parse_qs(parsed.query)++ return {+ 'scheme': parsed.scheme,+ 'netloc': parsed.netloc,+ 'hostname': parsed.hostname,+ 'port': parsed.port,+ 'path': parsed.path,+ 'query': parsed.query,+ 'query_params': query_params,+ 'fragment': parsed.fragment,+ }+++def get_base_url(url: str) -> str:+ """+ Extract base URL (scheme + netloc) from full URL.++ Args:+ url: Full URL string++ Returns:+ Base URL (e.g., "https://api.example.com")+ """+ parsed = urlparse(url)+ return f"{parsed.scheme}://{parsed.netloc}"+++def normalize_path(path: str) -> str:+ """+ Normalize a URL path by removing trailing slashes and lowercasing.++ Args:+ path: URL path++ Returns:+ Normalized path+ """+ path = path.strip()+ # Remove trailing slash unless it's the root path+ if len(path) > 1 and path.endswith('/'):+ path = path[:-1]+ return path+++def get_request_body(entry: Dict[str, Any]) -> Optional[Dict[str, Any]]:+ """+ Extract and parse request body from HAR entry.++ Args:+ entry: HAR entry dictionary++ Returns:+ Parsed request body as dict, or None if not JSON+ """+ request = entry.get('request', {})+ post_data = request.get('postData', {})++ mime_type = post_data.get('mimeType', '')+ text = post_data.get('text', '')++ if 'json' in mime_type.lower():+ return parse_json_safe(text)++ return None+++def get_response_body(entry: Dict[str, Any]) -> Optional[Dict[str, Any]]:+ """+ Extract and parse response body from HAR entry.++ Args:+ entry: HAR entry dictionary++ Returns:+ Parsed response body as dict, or None if not JSON+ """+ response = entry.get('response', {})+ content = response.get('content', {})++ mime_type = content.get('mimeType', '')+ text = content.get('text', '')++ if 'json' in mime_type.lower():+ return parse_json_safe(text)++ return None+++def get_request_headers(entry: Dict[str, Any]) -> Dict[str, str]:+ """+ Extract request headers as a dictionary.++ Args:+ entry: HAR entry dictionary++ Returns:+ Dictionary of header name -> value+ """+ request = entry.get('request', {})+ headers = request.get('headers', [])++ result = {}+ for header in headers:+ if isinstance(header, dict):+ name = header.get('name', '')+ value = header.get('value', '')+ if name:+ result[name] = value++ return result+++def get_response_headers(entry: Dict[str, Any]) -> Dict[str, str]:+ """+ Extract response headers as a dictionary.++ Args:+ entry: HAR entry dictionary++ Returns:+ Dictionary of header name -> value+ """+ response = entry.get('response', {})+ headers = response.get('headers', [])++ result = {}+ for header in headers:+ if isinstance(header, dict):+ name = header.get('name', '')+ value = header.get('value', '')+ if name:+ result[name] = value++ return result+++def save_json(data: Dict[str, Any], output_path: Path, pretty: bool = True) -> None:+ """+ Save data as JSON file.++ Args:+ data: Data to save+ output_path: Output file path+ pretty: Whether to pretty-print the JSON+ """+ output_path.parent.mkdir(parents=True, exist_ok=True)++ with open(output_path, 'w', encoding='utf-8') as f:+ if pretty:+ json.dump(data, f, indent=2, ensure_ascii=False)+ else:+ json.dump(data, f, ensure_ascii=False)diff --git a/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.py b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.pynew file mode 100644index 0000000..b609a72--- /dev/null+++ b/plugins/reverse-api-engineer/skills/reverse-engineering-api/scripts/har_validate.py@@ -0,0 +1,457 @@+#!/usr/bin/env python3+"""+HAR Validator - Validate generated API client against HAR analysis.++This script validates that a generated API client properly implements+all endpoints found in the HAR analysis.++Usage:+ python har_validate.py <api_client.py> <analysis.json>+"""++import argparse+import ast+import json+import re+import sys+from pathlib import Path+from typing import Dict, List, Any, Set, Tuple+++def load_analysis(analysis_path: Path) -> Dict[str, Any]:+ """Load analysis JSON file."""+ if not analysis_path.exists():+ raise FileNotFoundError(f"Analysis file not found: {analysis_path}")++ with open(analysis_path, 'r', encoding='utf-8') as f:+ return json.load(f)+++def load_client_code(client_path: Path) -> str:+ """Load API client Python code."""+ if not client_path.exists():+ raise FileNotFoundError(f"Client file not found: {client_path}")++ with open(client_path, 'r', encoding='utf-8') as f:+ return f.read()+++def parse_client_methods(client_code: str) -> Set[str]:+ """+ Extract method names from API client code.++ Args:+ client_code: Python source code++ Returns:+ Set of method names (excluding private methods and __init__)+ """+ methods = set()++ try:+ tree = ast.parse(client_code)++ for node in ast.walk(tree):+ if isinstance(node, ast.ClassDef):+ for item in node.body:+ if isinstance(item, ast.FunctionDef):+ method_name = item.name+ # Skip private methods and __init__+ if not method_name.startswith('_'):+ methods.add(method_name)++ except SyntaxError:+ # If AST parsing fails, fall back to regex+ pattern = r'def\s+([a-z_][a-z0-9_]*)\s*\('+ matches = re.findall(pattern, client_code, re.MULTILINE)+ methods = set(m for m in matches if not m.startswith('_'))++ return methods+++def endpoint_to_method_names(pattern: str, methods: List[str]) -> List[str]:+ """+ Generate expected method names from endpoint pattern.++ Args:+ pattern: Endpoint pattern (e.g., "/api/users/{id}")+ methods: HTTP methods (e.g., ["GET", "POST"])++ Returns:+ List of expected Python method names+ """+ # Remove leading/trailing slashes+ path = pattern.strip('/')++ # Remove common prefixes+ for prefix in ['api/', 'v1/', 'v2/', 'v3/', 'api/v1/', 'api/v2/', 'api/v3/']:+ if path.startswith(prefix):+ path = path[len(prefix):]++ # Split and filter+ parts = []+ for segment in path.split('/'):+ # Skip parameter placeholders+ if segment.startswith('{') and segment.endswith('}'):+ continue+ # Skip numeric segments+ if segment.isdigit():+ continue+ if segment:+ parts.append(segment)++ if not parts:+ parts = ['item']++ base_name = '_'.join(parts)++ # Singularize if needed (simple approach)+ method_names = []+ for method in methods:+ name = base_name+ if method in ['GET', 'PUT', 'PATCH', 'DELETE']:+ # Check if pattern has parameter (suggests single item)+ if '{id}' in pattern or '{uuid}' in pattern:+ # Singularize+ if name.endswith('ies'):+ name = name[:-3] + 'y'+ elif name.endswith('s') and not name.endswith('ss'):+ name = name[:-1]++ # Add method prefix+ prefix = method.lower()+ if prefix == 'delete':+ prefix = 'remove'++ method_name = f"{prefix}_{name}"+ method_name = re.sub(r'_+', '_', method_name)+ method_name = re.sub(r'[^a-z0-9_]', '_', method_name)++ method_names.append(method_name)++ return method_names+++def check_endpoint_coverage(+ client_code: str,+ client_methods: Set[str],+ endpoints: List[Dict[str, Any]]+) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:+ """+ Check if all endpoints from analysis are implemented.++ Args:+ client_code: Python source code+ client_methods: Set of implemented method names+ endpoints: List of endpoint dictionaries from analysis++ Returns:+ Tuple of (issues, coverage_stats)+ """+ issues = []+ endpoints_covered = 0+ endpoints_total = 0++ for endpoint in endpoints:+ pattern = endpoint['pattern']+ methods = endpoint['methods']++ # Generate expected method names+ expected_names = endpoint_to_method_names(pattern, methods)++ # Check if any expected name is implemented+ found = any(name in client_methods for name in expected_names)++ if found:+ endpoints_covered += 1+ else:+ # Not implemented+ for method in methods:+ issues.append({+ 'severity': 'error',+ 'category': 'missing_endpoint',+ 'message': f"Endpoint {method} {pattern} not implemented (expected method: {expected_names[0]})",+ 'line': None,+ })++ endpoints_total += 1++ coverage = {+ 'endpoints_covered': endpoints_covered,+ 'endpoints_total': endpoints_total,+ 'percentage': round(endpoints_covered / endpoints_total * 100, 1) if endpoints_total > 0 else 0,+ }++ return issues, coverage+++def check_auth_implementation(+ client_code: str,+ auth: Dict[str, Any]+) -> List[Dict[str, Any]]:+ """+ Check if authentication is properly implemented.++ Args:+ client_code: Python source code+ auth: Authentication dict from analysis++ Returns:+ List of issues+ """+ issues = []+ auth_type = auth.get('type', 'none')++ if auth_type == 'none':+ # No auth required+ return issues++ # Check for auth implementation+ auth_indicators = {+ 'bearer_token': ['Authorization', 'Bearer', 'token'],+ 'api_key_header': ['X-API-Key', 'api_key', 'apikey'],+ 'api_key_query': ['api_key', 'apikey', 'params'],+ 'session_cookie': ['cookie', 'session'],+ 'basic_auth': ['Authorization', 'Basic', 'credentials'],+ }++ expected_indicators = auth_indicators.get(auth_type, [])+ found_any = any(indicator.lower() in client_code.lower() for indicator in expected_indicators)++ if not found_any:+ issues.append({+ 'severity': 'warning',+ 'category': 'missing_auth',+ 'message': f"Authentication type '{auth_type}' not implemented (expected {auth['header_name'] or 'session handling'})",+ 'line': None,+ })++ return issues+++def check_error_handling(client_code: str) -> List[Dict[str, Any]]:+ """+ Check if proper error handling is implemented.++ Args:+ client_code: Python source code++ Returns:+ List of issues+ """+ issues = []++ # Check for try-except blocks+ has_try_except = 'try:' in client_code and 'except' in client_code++ # Check for custom exceptions+ has_custom_exceptions = 'class' in client_code and 'Error' in client_code and 'Exception' in client_code++ # Check for response validation+ has_response_check = 'raise_for_status' in client_code or 'status_code' in client_code++ if not has_try_except:+ issues.append({+ 'severity': 'warning',+ 'category': 'missing_error_handling',+ 'message': 'No try-except blocks found for error handling',+ 'line': None,+ })++ if not has_custom_exceptions:+ issues.append({+ 'severity': 'info',+ 'category': 'missing_custom_exceptions',+ 'message': 'No custom exception classes defined',+ 'line': None,+ })++ if not has_response_check:+ issues.append({+ 'severity': 'warning',+ 'category': 'missing_response_validation',+ 'message': 'No response status validation found',+ 'line': None,+ })++ return issues+++def check_type_hints(client_code: str) -> List[Dict[str, Any]]:+ """+ Check if type hints are used.++ Args:+ client_code: Python source code++ Returns:+ List of issues+ """+ issues = []++ # Check for typing imports+ has_typing_import = 'from typing import' in client_code or 'import typing' in client_code++ # Check for type hints in code (look for -> or :)+ has_type_hints = ' -> ' in client_code or ': Dict' in client_code or ': List' in client_code++ if not has_typing_import or not has_type_hints:+ issues.append({+ 'severity': 'info',+ 'category': 'missing_type_hints',+ 'message': 'Type hints not fully implemented',+ 'line': None,+ })++ return issues+++def validate_client(+ client_path: Path,+ analysis: Dict[str, Any]+) -> Dict[str, Any]:+ """+ Validate API client against HAR analysis.++ Args:+ client_path: Path to API client Python file+ analysis: HAR analysis dictionary++ Returns:+ Validation report with score, issues, coverage+ """+ client_code = load_client_code(client_path)+ client_methods = parse_client_methods(client_code)++ all_issues = []++ # Check endpoint coverage+ endpoints = analysis.get('endpoints', [])+ endpoint_issues, coverage = check_endpoint_coverage(client_code, client_methods, endpoints)+ all_issues.extend(endpoint_issues)++ # Check authentication+ auth = analysis.get('authentication', {})+ auth_issues = check_auth_implementation(client_code, auth)+ all_issues.extend(auth_issues)++ # Check error handling+ error_issues = check_error_handling(client_code)+ all_issues.extend(error_issues)++ # Check type hints+ type_issues = check_type_hints(client_code)+ all_issues.extend(type_issues)++ # Calculate score+ # Base score starts at 100+ score = 100++ # Deduct points for errors+ error_count = len([i for i in all_issues if i['severity'] == 'error'])+ warning_count = len([i for i in all_issues if i['severity'] == 'warning'])+ info_count = len([i for i in all_issues if i['severity'] == 'info'])++ score -= error_count * 15 # -15 points per error+ score -= warning_count * 5 # -5 points per warning+ score -= info_count * 2 # -2 points per info++ # Bonus points for high coverage+ if coverage['percentage'] == 100:+ score += 10++ # Ensure score is between 0 and 100+ score = max(0, min(100, score))++ report = {+ 'score': score,+ 'issues': all_issues,+ 'coverage': coverage,+ 'summary': {+ 'errors': error_count,+ 'warnings': warning_count,+ 'info': info_count,+ 'total_issues': len(all_issues),+ },+ }++ return report+++def main():+ """Main entry point for CLI."""+ parser = argparse.ArgumentParser(+ description='Validate generated API client against HAR analysis',+ formatter_class=argparse.RawDescriptionHelpFormatter,+ epilog="""+Examples:+ # Validate API client+ python har_validate.py api_client.py analysis.json++ # Save validation report to file+ python har_validate.py api_client.py analysis.json --output report.json+ """+ )++ parser.add_argument(+ 'client',+ type=str,+ help='API client Python file path'+ )++ parser.add_argument(+ 'analysis',+ type=str,+ help='HAR analysis JSON file path'+ )++ parser.add_argument(+ '--output',+ type=str,+ help='Output validation report JSON file path'+ )++ args = parser.parse_args()++ # Validate input files+ client_path = Path(args.client).expanduser().resolve()+ analysis_path = Path(args.analysis).expanduser().resolve()++ if not client_path.exists():+ print(f"Error: Client file not found: {client_path}", file=sys.stderr)+ sys.exit(1)++ if not analysis_path.exists():+ print(f"Error: Analysis file not found: {analysis_path}", file=sys.stderr)+ sys.exit(1)++ try:+ # Load analysis+ analysis = load_analysis(analysis_path)++ # Validate client+ report = validate_client(client_path, analysis)++ # Save or print report+ if args.output:+ output_path = Path(args.output).expanduser().resolve()+ output_path.parent.mkdir(parents=True, exist_ok=True)+ with open(output_path, 'w', encoding='utf-8') as f:+ json.dump(report, f, indent=2)+ print(f"Validation report saved to: {output_path}")+ else:+ print(json.dumps(report, indent=2))++ # Exit with error code if score < 90+ if report['score'] < 90:+ sys.exit(1)++ except Exception as e:+ print(f"Error: {e}", file=sys.stderr)+ import traceback+ traceback.print_exc()+ sys.exit(1)+++if __name__ == '__main__':+ main()