LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
[MISC] Add worktree setup script for automated env file copying (#1756)
Chandrasekharan M9e2f01c81bb9
diff --git a/.claude/skills/worktree/SKILL.md b/.claude/skills/worktree/SKILL.mdnew file mode 100644index 000000000..50f8fb5e5--- /dev/null+++ b/.claude/skills/worktree/SKILL.md@@ -0,0 +1,96 @@+---+name: worktree+description: Creates git worktrees for isolated development. Use when user wants to "create a worktree", "start a new feature branch", "work on a fix in isolation", or needs a parallel development environment.+---++# Git Worktree Skill++Create isolated worktrees for parallel development. Ends by providing commands to continue in the new worktree.++## Workflow++1. **Detect repo info**+ ```bash+ REPO_NAME=$(basename -s .git $(git config --get remote.origin.url) 2>/dev/null || basename $(pwd))+ SOURCE_REPO=$(git rev-parse --show-toplevel)+ ```++2. **Parse user description**+ - Type: `fix` (bug, patch, fix) | `feat` (feature, add, implement) | `misc` (default)+ - Branch: `{type}/{kebab-description}` (max 30 chars)+ - Folder: `{type}-{kebab-description}`+ - Path: `{SOURCE_REPO}/../{REPO_NAME}-worktrees/{folder}/`+ - Version: `{kebab-description}` (for docker image tag)++3. **Create worktree**+ ```bash+ git fetch origin main+ mkdir -p "$(dirname "$WORKTREE_PATH")"+ git worktree add -b "{branch}" "$WORKTREE_PATH" origin/main+ ```++4. **Copy config files**+ - **unstract repo:** Run `"$SOURCE_REPO/.claude/skills/worktree/worktree-setup.sh" "$WORKTREE_PATH" "$SOURCE_REPO"`+ - **Other repos:** Ask user which config files to copy, then copy them++5. **Detect services from request** (unstract only)++ Match keywords in user's request to services:++ | Keywords | Services |+ | ------------------------------ | -------------------------- |+ | frontend, UI, react, component | frontend |+ | backend, API, django | backend |+ | platform, adapter, LLM | platform-service |+ | prompt, studio | prompt-service |+ | worker, celery, task | backend (includes workers) |+ | x2text, extraction | x2text-service |++ Default if unclear: `backend frontend`++6. **Print summary with commands**++ ```text+ Worktree created!++ Branch: {branch}+ Path: {worktree_path}++ ─────────────────────────────────────────────────────────+ Continue in new worktree and paste your prompt:++ cd {worktree_path} && claude++ {user_request_slightly_made_better_for_new_agent_to_understand}+ ─────────────────────────────────────────────────────────+ Build & run services:++ cd {worktree_path}/docker && \+ VERSION={version} docker compose -f docker-compose.yaml -f compose.override.yaml build {services} && \+ VERSION={version} docker compose -f docker-compose.yaml -f compose.override.yaml watch {services}+ ─────────────────────────────────────────────────────────++ Cleanup later: git worktree remove {worktree_path}+ ```++ IMPORTANT: Escape any quotes in `{user_request}` for shell safety.++## Examples++**User:** "worktree for fixing login validation in frontend"+- Branch: `fix/login-validation`+- Version: `login-validation`+- Services: `frontend`++**User:** "create worktree to add workflow export API"+- Branch: `feat/workflow-export-api`+- Version: `workflow-export-api`+- Services: `backend`++## Reference++```bash+git worktree list # List worktrees+git worktree remove {path} # Remove worktree+git worktree prune # Prune stale entries+```diff --git a/.claude/skills/worktree/worktree-setup.sh b/.claude/skills/worktree/worktree-setup.shnew file mode 100755index 000000000..db51c4e58--- /dev/null+++ b/.claude/skills/worktree/worktree-setup.sh@@ -0,0 +1,194 @@+#!/bin/bash+#+# Worktree Setup Script for Unstract+# Copies necessary config files from source repo to a new worktree+#+# Usage: ./worktree-setup.sh <target_worktree_path> [source_repo_path]+#++set -euo pipefail++show_usage() {+ echo "Usage: $0 <target_worktree_path> [source_repo_path]"+ echo ""+ echo "Arguments:"+ echo " target_worktree_path Path to the new worktree"+ echo " source_repo_path Path to source repo (defaults to script's parent dir)"+ exit "${1:-0}"+}++# Handle help flag+if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then+ show_usage 0+fi++TARGET_PATH="$1"+SOURCE_PATH="${2:-$(cd "$(dirname "$0")/../../.." && pwd)}"++if [ -z "$TARGET_PATH" ]; then+ show_usage 1+fi++# Convert to absolute paths (handle non-existent target)+if [ -d "$TARGET_PATH" ]; then+ TARGET_PATH=$(cd "$TARGET_PATH" && pwd)+else+ # Portable: construct absolute path relative to CWD without requiring realpath+ TARGET_PATH="$(cd "$(dirname "$TARGET_PATH")" 2>/dev/null && pwd)/$(basename "$TARGET_PATH")" \+ || TARGET_PATH="$(pwd)/$TARGET_PATH"+fi+if [ ! -d "$SOURCE_PATH" ]; then+ echo "Error: source repo path does not exist: $SOURCE_PATH"+ show_usage 1+fi+SOURCE_PATH=$(cd "$SOURCE_PATH" && pwd)++echo "Copying config files from unstract repo..."+echo " Source: $SOURCE_PATH"+echo " Target: $TARGET_PATH"+echo ""++# Files to copy (relative to repo root)+CONFIG_FILES=(+ # Service .env files+ "backend/.env"+ "frontend/.env"+ "platform-service/.env"+ "prompt-service/.env"+ "runner/.env"+ "workers/.env"+ "x2text-service/.env"+ # Docker config files+ "docker/.env"+ "docker/essentials.env"+ "docker/compose.override.yaml"+ "docker/proxy_overrides.yaml"+)++copied=0+skipped=0+missing=0++for file in "${CONFIG_FILES[@]}"; do+ src="$SOURCE_PATH/$file"+ dest="$TARGET_PATH/$file"++ if [ -f "$src" ]; then+ # Create directory if needed+ mkdir -p "$(dirname "$dest")"++ if [ -f "$dest" ]; then+ # File exists, check if different+ if ! diff -q "$src" "$dest" > /dev/null 2>&1; then+ cp "$src" "$dest"+ echo " [updated] $file"+ copied=$((copied + 1))+ else+ echo " [skipped] $file (unchanged)"+ skipped=$((skipped + 1))+ fi+ else+ cp "$src" "$dest"+ echo " [copied] $file"+ copied=$((copied + 1))+ fi+ else+ echo " [missing] $file (not found in source)"+ missing=$((missing + 1))+ fi+done++echo ""+echo "Env files: Copied=$copied, Skipped=$skipped, Missing=$missing"+echo ""++# ============================================================+# Step 2: Copy gitignored backend settings+# ============================================================+echo "Copying gitignored backend settings..."++SETTINGS_DIR="backend/backend/settings"+TRACKED_SETTINGS=("__init__.py" "base.py" "dev.py" "test.py")++settings_copied=0+settings_skipped=0++if [ -d "$SOURCE_PATH/$SETTINGS_DIR" ]; then+ for file in "$SOURCE_PATH/$SETTINGS_DIR"/*.py; do+ [ -f "$file" ] || continue+ filename=$(basename "$file")++ # Skip tracked files (these come from git)+ skip=false+ for tracked in "${TRACKED_SETTINGS[@]}"; do+ if [ "$filename" = "$tracked" ]; then+ skip=true+ break+ fi+ done+ $skip && continue++ # Copy gitignored settings file+ dest="$TARGET_PATH/$SETTINGS_DIR/$filename"+ mkdir -p "$(dirname "$dest")"++ if [ -f "$dest" ]; then+ if ! diff -q "$file" "$dest" > /dev/null 2>&1; then+ cp "$file" "$dest"+ echo " [updated] $filename"+ settings_copied=$((settings_copied + 1))+ else+ echo " [skipped] $filename (unchanged)"+ settings_skipped=$((settings_skipped + 1))+ fi+ else+ cp "$file" "$dest"+ echo " [copied] $filename"+ settings_copied=$((settings_copied + 1))+ fi+ done+else+ echo " [warning] $SETTINGS_DIR not found in source"+fi++echo ""+echo "Settings: Copied=$settings_copied, Skipped=$settings_skipped"+echo ""++# ============================================================+# Step 3: Copy project-local Claude skills (gitignored ones)+# ============================================================+SKILLS_DIR=".claude/skills"+skills_copied=0+skills_skipped=0++if [ -d "$SOURCE_PATH/$SKILLS_DIR" ]; then+ echo "Copying project-local Claude skills..."++ for skill_dir in "$SOURCE_PATH/$SKILLS_DIR"/*/; do+ [ -d "$skill_dir" ] || continue+ skill_name=$(basename "$skill_dir")+ dest_dir="$TARGET_PATH/$SKILLS_DIR/$skill_name"++ # Skip if already present and identical+ if [ -d "$dest_dir" ] && diff -rq "$skill_dir" "$dest_dir" > /dev/null 2>&1; then+ echo " [skipped] $skill_name/ (unchanged)"+ skills_skipped=$((skills_skipped + 1))+ continue+ fi++ mkdir -p "$dest_dir"+ # Trailing-dot trick: copies all contents including hidden files, but not . or ..+ cp -r "$skill_dir." "$dest_dir/" 2>/dev/null || true+ echo " [copied] $skill_name/"+ skills_copied=$((skills_copied + 1))+ done++ echo ""+ echo "Skills: Copied=$skills_copied, Skipped=$skills_skipped"+else+ echo "No $SKILLS_DIR directory found in source, skipping."+fi++echo ""+echo "Done!"diff --git a/.gitignore b/.gitignoreindex 80c568a0f..c96f27b53 100644--- a/.gitignore+++ b/.gitignore@@ -8,6 +8,7 @@!workers/run-worker.sh!workers/run-worker-docker.sh!workers/log_consumer/scheduler.sh+!.claude/skills/worktree/worktree-setup.sh### Django ###*.log@@ -688,6 +689,7 @@ prompting/# ClaudeCLAUDE.md.claude/*+!.claude/skills/CONTRIBUTION_GUIDE.md.mcp.json