LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
[MISC] Fix adapter-ops skill cost-tracking guidance and stale paths (#2163)
Chandrasekharan M9609ec0e4926
diff --git a/.claude/skills/adapter-ops/SKILL.md b/.claude/skills/adapter-ops/SKILL.mdindex 521111db3..417355b03 100644--- a/.claude/skills/adapter-ops/SKILL.md+++ b/.claude/skills/adapter-ops/SKILL.md@@ -56,7 +56,7 @@ LiteLLM requires provider prefixes on model names:1. **Run initialization script**:```bash- python .claude/skills/unstract-adapter-extension/scripts/init_llm_adapter.py \+ python .claude/skills/adapter-ops/scripts/init_llm_adapter.py \--provider newprovider \--name "New Provider" \--description "New Provider LLM adapter" \@@ -122,7 +122,7 @@ LiteLLM requires provider prefixes on model names:1. **Run initialization script**:```bash- python .claude/skills/unstract-adapter-extension/scripts/init_embedding_adapter.py \+ python .claude/skills/adapter-ops/scripts/init_embedding_adapter.py \--provider newprovider \--name "New Provider" \--description "New Provider embedding adapter" \@@ -190,10 +190,10 @@ LiteLLM requires provider prefixes on model names:3. **Run management script** for automated updates:```bash- python .claude/skills/unstract-adapter-extension/scripts/manage_models.py \+ python .claude/skills/adapter-ops/scripts/manage_models.py \--adapter llm \--provider openai \- --action add \+ --action add-enum \--models "gpt-4-turbo,gpt-4o-mini"```@@ -238,17 +238,17 @@ Compare existing adapter schemas against known LiteLLM features to identify pote1. **Run the update checker**:```bash# Check all adapters- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py# Check specific adapter type- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py --adapter llm- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py --adapter embedding+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py --adapter llm+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py --adapter embedding# Check specific provider- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py --provider openai+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py --provider openai# Output as JSON- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py --json+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py --json```2. **Review the report**:@@ -278,7 +278,8 @@ Before submitting adapter changes:- [ ] Adapter class inherits from correct parameter class AND `BaseAdapter`- [ ] `get_id()` returns unique `{provider}|{uuid}` format- [ ] `get_metadata()` returns dict with `name`, `version`, `adapter`, `description`, `is_active`-- [ ] **CRITICAL: `get_provider()` returns value matching `litellm_provider` in LiteLLM pricing data** (see below)+- [ ] `get_provider()` matches the static JSON filename (`static/{get_provider()}.json`)+- [ ] **CRITICAL: the model string produced by `validate_model()` resolves in LiteLLM's cost map** (see below)- [ ] `get_adapter_type()` returns correct `AdapterTypes.LLM` or `AdapterTypes.EMBEDDING`- [ ] JSON schema has `adapter_name` as required field- [ ] `validate()` method adds correct model prefix@@ -286,43 +287,86 @@ Before submitting adapter changes:- [ ] All static methods decorated with `@staticmethod`- [ ] Icon path follows pattern `/icons/adapter-icons/{Name}.png`-### Provider Name Verification (MANDATORY)+### Model Prefix Verification (MANDATORY)-The `get_provider()` return value is used for **cost calculation**. It MUST match the `litellm_provider` field in LiteLLM's pricing data, otherwise costs will show as $0.+Cost is looked up from the **validated model string**, not from `get_provider()`. The string+that `validate_model()` produces (e.g. `mistral/mistral-embed`) is passed straight to+`litellm.cost_per_token()`. If LiteLLM's cost map has no entry for it, the lookup raises, the+exception is swallowed, and usage records **$0**.-**Before implementing any adapter, verify the provider name:**+The lookup sites:-```bash-# Fetch LiteLLM pricing data and check provider name-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.value.litellm_provider != null)) |- map({key: .key, provider: .value.litellm_provider}) |- unique_by(.provider) |- sort_by(.provider) |- .[].provider' | sort -u-```+| Path | Site |+|------|------|+| LLM | `unstract/sdk1/src/unstract/sdk1/audit.py` — `cost_per_token(model=model_name)` |+| Embedding | `unstract/sdk1/src/unstract/sdk1/usage_handler.py` — `litellm.cost_per_token(...)` |++`model_name` is `self._cost_model or self.kwargs["model"]` — the prefixed string. Both call+sites catch every exception and fall back to `0.0`, so a miss is **silent**. It will not fail a+test, a build, or a run. The only symptom is revenue-affecting: zero-cost usage rows.++**Before implementing any adapter, verify the prefix resolves.** Use the pinned LiteLLM in the+sdk1 venv rather than curling upstream JSON — the pinned version is what actually runs:-**Common provider name mappings:**-| Display Name | `get_provider()` Value | LiteLLM Provider |-|--------------|------------------------|------------------|-| OpenAI | `openai` | `openai` |-| Anthropic | `anthropic` | `anthropic` |-| Azure OpenAI | `azure` | `azure` |-| Azure AI Foundry | `azure_ai` | `azure_ai` |-| AWS Bedrock | `bedrock` | `bedrock` |-| Google VertexAI | `vertex_ai` | `vertex_ai` |-| Mistral | `mistral` | `mistral` |-| Ollama | `ollama` | `ollama` |--**Example verification for Azure AI Foundry:**```bash-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.key | startswith("azure_ai"))) | .[0].value.litellm_provider'-# Output: "azure_ai" (NOT "azure_ai_foundry")+cd unstract/sdk1 && uv run python -c "+import litellm+from litellm import cost_per_token++# 1. Does LiteLLM have a native provider for this vendor?+print([p for p in litellm.provider_list if 'YOUR_VENDOR' in str(p).lower()])++# 2. Which of its models are priced?+print([k for k in litellm.model_cost if k.lower().startswith('YOUR_PROVIDER/')])++# 3. Does the string your validate_model() emits actually resolve?+for m in ['YOUR_PROVIDER/some-model', 'custom_openai/some-model']:+ try:+ print(m, cost_per_token(model=m, prompt_tokens=1_000_000, completion_tokens=1_000_000))+ except Exception as e:+ print(m, 'RAISES', type(e).__name__, '=> cost silently recorded as 0.0')+"```-**Why this matters:**-The cost calculation in `platform-service/src/unstract/platform_service/helper/cost_calculation.py` filters models by checking if the provider string is contained in `litellm_provider`. A mismatch (e.g., returning `"azure_ai_foundry"` when LiteLLM uses `"azure_ai"`) causes the cost lookup to fail silently, returning $0.+**Branded OpenAI-compatible adapters: pick the right base class.**++Many vendors speak the OpenAI wire protocol, which tempts you to subclass+`OpenAICompatibleLLMParameters` and just pin `api_base`. But its `validate_model()`+unconditionally prepends `custom_openai/`, and **nothing** in LiteLLM's cost map is keyed that+way. That silently forfeits cost tracking.++| If… | Then | Cost tracking |+|-----|------|---------------|+| LiteLLM has a native provider for the vendor | Extend `BaseChatCompletionParameters`, emit `{provider}/{model}` — follow `OpenRouterLLMParameters` | ✅ resolves |+| LiteLLM has **no** priced models for the vendor | Extend `OpenAICompatibleLLMParameters`, pin `api_base` — follow `NvidiaBuildLLMParameters` | ⚠️ $0 either way, nothing forfeited |++`NvidiaBuildLLMParameters` is only a safe template because LiteLLM prices zero `nvidia_nim/`+chat models — there is no cost to lose. Do not copy that shape for a vendor LiteLLM *does*+price. Check first with the snippet above.++**What `get_provider()` is actually for:**++1. Resolving the static schema path — `{type}1/static/{get_provider()}.json` (case-sensitive `open()`).+2. The `provider` column on the usage row (`audit.py`) — metadata only, not used for pricing.++Keeping it equal to LiteLLM's `litellm_provider` value is still good hygiene, and matters when+you route natively (the prefix and the provider name coincide). But a correct `get_provider()`+does **not** on its own guarantee cost resolution — a branded adapter can return `"minimax"`,+match `litellm_provider` exactly, and still bill $0 because its model string carries the+`custom_openai/` prefix.++**Common provider names:**++| Display Name | `get_provider()` Value |+|--------------|------------------------|+| OpenAI | `openai` |+| Anthropic | `anthropic` |+| Azure OpenAI | `azure` |+| Azure AI Foundry | `azure_ai` |+| AWS Bedrock | `bedrock` |+| Google VertexAI | `vertex_ai` |+| Mistral | `mistral` |+| Ollama | `ollama` |## Maintenance Workflow@@ -332,7 +376,7 @@ Periodic maintenance to keep adapters current with LiteLLM features:1. **Run the update checker**:```bash- python .claude/skills/unstract-adapter-extension/scripts/check_adapter_updates.py+ python .claude/skills/adapter-ops/scripts/check_adapter_updates.py```2. **Review LiteLLM changelog** for new provider features:diff --git a/.claude/skills/adapter-ops/references/adapter_patterns.md b/.claude/skills/adapter-ops/references/adapter_patterns.mdindex a3b200335..9bab1d5ea 100644--- a/.claude/skills/adapter-ops/references/adapter_patterns.md+++ b/.claude/skills/adapter-ops/references/adapter_patterns.md@@ -568,48 +568,87 @@ print(schema)4. **Missing `is_active: True`** - Adapter won't be registered without it5. **Wrong inheritance order** - Parameter class must come before BaseAdapter6. **Incorrect provider in get_provider()** - Must match filename and schema path-7. **Provider name not matching LiteLLM pricing data** - Causes $0 cost calculation (see below)+7. **Model prefix not resolving in LiteLLM's cost map** - Causes silent $0 cost calculation (see below)-## Provider Name Verification (CRITICAL)+## Model Prefix Verification (CRITICAL)-The `get_provider()` return value MUST match the `litellm_provider` field in LiteLLM's pricing JSON. A mismatch causes cost calculation to silently fail, returning $0.+Cost is derived from the **validated model string**, not from `get_provider()`. Whatever+`validate_model()` emits is handed to `litellm.cost_per_token()`. No cost-map entry means the+call raises, the exception is swallowed, and the usage row records $0.++### Cost Calculation Flow++1. **Usage recording**: `LLM._record_usage()` logs tokens (no cost math here).+2. **Cost lookup**:+ - LLM → `Audit.push_usage_data()` → `cost_per_token(model=model_name)` in `audit.py`+ - Embedding → `UsageHandler` → `litellm.cost_per_token(...)` in `usage_handler.py`+ - `model_name` is `self._cost_model or self.kwargs["model"]` — the **prefixed** string.+3. **Failure mode**: both call sites wrap the lookup in a bare `except` and fall back to+ `cost_in_dollars = 0.0`, logged at `debug`/`warning`.++The `provider` value travels alongside the usage payload and lands in the `provider` DB column.+It is **not** consulted for pricing.++Because the failure is swallowed, nothing breaks loudly. Tests pass. Runs succeed. Usage rows+just quietly say $0.### How to Verify-**Before implementing any new adapter:**+Query the pinned LiteLLM in the sdk1 venv — that's the version that actually runs:```bash-# Step 1: Identify the correct provider name from LiteLLM pricing data-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.key | startswith("YOUR_PROVIDER"))) | .[0].value.litellm_provider'--# Step 2: Use that exact value in get_provider()+cd unstract/sdk1 && uv run python -c "+import litellm+from litellm import cost_per_token++print([p for p in litellm.provider_list if 'YOUR_VENDOR' in str(p).lower()])+print([k for k in litellm.model_cost if k.lower().startswith('YOUR_PROVIDER/')])++for m in ['YOUR_PROVIDER/some-model', 'custom_openai/some-model']:+ try:+ print(m, cost_per_token(model=m, prompt_tokens=1_000_000, completion_tokens=1_000_000))+ except Exception:+ print(m, 'RAISES => cost silently recorded as 0.0')+"```-### Real Example: Azure AI Foundry Bug+### Real Example: the `custom_openai/` trap-**Wrong implementation (caused $0 costs):**-```python-@staticmethod-def get_provider() -> str:- return "azure_ai_foundry" # WRONG - doesn't match litellm_provider-```+A vendor speaking the OpenAI wire protocol invites subclassing+`OpenAICompatibleLLMParameters` and pinning `api_base`. That class's `validate_model()`+unconditionally prepends `custom_openai/`, and **no** LiteLLM cost-map key uses that prefix.-**Correct implementation:**```python-@staticmethod-def get_provider() -> str:- return "azure_ai" # CORRECT - matches litellm_provider in pricing data+# Vendor has a native LiteLLM provider AND priced models.+# WRONG - emits "custom_openai/MiniMax-M3", which is not in the cost map -> $0+class MiniMaxLLMParameters(OpenAICompatibleLLMParameters):+ api_base: str = "https://api.minimax.io/v1"++# CORRECT - emits "minimax/MiniMax-M3", priced at $0.30 / $1.20 per 1M tokens+class MiniMaxLLMParameters(BaseChatCompletionParameters):+ ... # follow OpenRouterLLMParameters```-**How the bug was found:**-```bash-# Check what LiteLLM actually uses for azure_ai models-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.key | startswith("azure_ai"))) | .[0]'+Note that `get_provider()` returns `"minimax"` in *both* cases and matches `litellm_provider`+exactly. The provider string was never the problem — the model prefix was.-# Output shows: "litellm_provider": "azure_ai" (NOT "azure_ai_foundry")-```+### Choosing a base class++| If… | Then | Cost tracking |+|-----|------|---------------|+| LiteLLM has a native provider for the vendor | `BaseChatCompletionParameters`, emit `{provider}/{model}` — see `OpenRouterLLMParameters` | ✅ resolves |+| LiteLLM has **no** priced models for the vendor | `OpenAICompatibleLLMParameters`, pin `api_base` — see `NvidiaBuildLLMParameters` | ⚠️ $0 regardless, nothing forfeited |++`NvidiaBuildLLMParameters` is a safe template *only because* LiteLLM prices zero `nvidia_nim/`+chat models. Don't copy its shape for a vendor LiteLLM prices — verify first.++### What `get_provider()` is for++1. Resolving the static schema path: `{type}1/static/{get_provider()}.json` (case-sensitive).+2. The `provider` column on the usage row — metadata only.++Match it to LiteLLM's `litellm_provider` for hygiene and because it coincides with the prefix+when routing natively. But it does not, by itself, guarantee cost resolution.### Provider Name Reference@@ -623,14 +662,3 @@ curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_| Google VertexAI | `vertex_ai` || Mistral | `mistral` || Ollama | `ollama` |--### Cost Calculation Flow--Understanding why this matters:--1. **Usage Recording**: `LLM._record_usage()` → `Audit.push_usage_data(provider=get_provider())`-2. **Platform Service**: Receives provider name with usage data-3. **Cost Lookup**: `CostCalculationHelper` checks `provider in model_info.get("litellm_provider", "")`-4. **Result**: If check fails, returns `cost = 0`--The check `"azure_ai_foundry" in "azure_ai"` returns `False`, causing silent failure.diff --git a/.claude/skills/adapter-ops/references/provider_capabilities.md b/.claude/skills/adapter-ops/references/provider_capabilities.mdindex 4a004abae..5a27a5705 100644--- a/.claude/skills/adapter-ops/references/provider_capabilities.md+++ b/.claude/skills/adapter-ops/references/provider_capabilities.md@@ -132,30 +132,39 @@ api_key, api_base, model, max_tokens, max_retries, timeout### Verification Command-Always verify the provider name before implementing an adapter:+Always verify the provider name before implementing an adapter — against the **pinned** LiteLLM+in the sdk1 venv, not upstream `main`, since `main` may price models the pinned version doesn't:```bash-# Check all unique litellm_provider values-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.value.litellm_provider != null)) |- map(.value.litellm_provider) | unique | sort'--# Check specific provider (e.g., for azure_ai models)-curl -s https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json | \- jq 'to_entries | map(select(.key | startswith("azure_ai"))) | .[0].value.litellm_provider'+cd unstract/sdk1 && uv run python -c "+import litellm+# All unique litellm_provider values in the pinned cost map+print(sorted({v['litellm_provider'] for v in litellm.model_cost.values()+ if isinstance(v, dict) and v.get('litellm_provider')}))+# Provider for a specific model-key prefix (e.g. azure_ai)+print([v['litellm_provider'] for k, v in litellm.model_cost.items()+ if k.startswith('azure_ai')][:1])+"```### Why This MattersThe cost calculation flow:-1. `LLM._record_usage()` calls `Audit.push_usage_data()` with provider from `get_provider()`-2. Platform service receives usage data with provider name-3. `CostCalculationHelper.calculate_cost()` filters models where `provider in litellm_provider`-4. If no match found, cost = $0--**Example bug**: If `get_provider()` returns `"azure_ai_foundry"` but LiteLLM uses `"azure_ai"`:-- Check: `"azure_ai_foundry" in "azure_ai"` = `False`-- Result: Cost calculation returns $0+1. `LLM._record_usage()` logs tokens — no cost math.+2. `Audit.push_usage_data()` (LLM) / `UsageHandler` (embedding) calls+ `litellm.cost_per_token(model=model_name)`, where `model_name` is the **prefixed** model+ string emitted by `validate_model()`.+3. No cost-map entry for that string → the call raises → the bare `except` records $0.++The `provider` value from `get_provider()` rides along into the usage row's `provider` column+but is **not** used for pricing.++**Example bug**: a branded OpenAI-compatible adapter emits `custom_openai/MiniMax-M3`. LiteLLM+prices `minimax/MiniMax-M3` but has no `custom_openai/` keys, so cost silently resolves to $0 —+even though `get_provider()` correctly returns `"minimax"`.++See `references/adapter_patterns.md` → *Model Prefix Verification* for how to choose a base+class so the prefix resolves.## Models Supporting Advanced Featuresdiff --git a/.claude/skills/adapter-ops/scripts/check_adapter_updates.py b/.claude/skills/adapter-ops/scripts/check_adapter_updates.pyindex ec50ed7f7..418046524 100755--- a/.claude/skills/adapter-ops/scripts/check_adapter_updates.py+++ b/.claude/skills/adapter-ops/scripts/check_adapter_updates.py@@ -440,7 +440,8 @@ def main():args = parser.parse_args()- print(f"SDK1 Adapters Path: {SDK1_ADAPTERS}")+ # stderr so --json output on stdout stays machine-parseable+ print(f"SDK1 Adapters Path: {SDK1_ADAPTERS}", file=sys.stderr)adapter_types = ["llm", "embedding"] if args.adapter == "all" else [args.adapter]results = []diff --git a/.claude/skills/adapter-ops/scripts/init_embedding_adapter.py b/.claude/skills/adapter-ops/scripts/init_embedding_adapter.pyindex 0122c39ed..9835d7923 100755--- a/.claude/skills/adapter-ops/scripts/init_embedding_adapter.py+++ b/.claude/skills/adapter-ops/scripts/init_embedding_adapter.py@@ -25,9 +25,7 @@ from urllib.request import Request, urlopenSCRIPT_DIR = Path(__file__).parentSKILL_DIR = SCRIPT_DIR.parent# Find the sdk1 adapters directory relative to the repo root-REPO_ROOT = (- SKILL_DIR.parent.parent.parent-) # .claude/skills/unstract-adapter-extension -> repo root+REPO_ROOT = SKILL_DIR.parent.parent.parent # .claude/skills/adapter-ops -> repo rootSDK1_ADAPTERS = REPO_ROOT / "unstract" / "sdk1" / "src" / "unstract" / "sdk1" / "adapters"ICONS_DIR = REPO_ROOT / "frontend" / "public" / "icons" / "adapter-icons"diff --git a/.claude/skills/adapter-ops/scripts/init_llm_adapter.py b/.claude/skills/adapter-ops/scripts/init_llm_adapter.pyindex 8226f69ee..828e04c3d 100755--- a/.claude/skills/adapter-ops/scripts/init_llm_adapter.py+++ b/.claude/skills/adapter-ops/scripts/init_llm_adapter.py@@ -26,9 +26,7 @@ from urllib.request import Request, urlopenSCRIPT_DIR = Path(__file__).parentSKILL_DIR = SCRIPT_DIR.parent# Find the sdk1 adapters directory relative to the repo root-REPO_ROOT = (- SKILL_DIR.parent.parent.parent-) # .claude/skills/unstract-adapter-extension -> repo root+REPO_ROOT = SKILL_DIR.parent.parent.parent # .claude/skills/adapter-ops -> repo rootSDK1_ADAPTERS = REPO_ROOT / "unstract" / "sdk1" / "src" / "unstract" / "sdk1" / "adapters"ICONS_DIR = REPO_ROOT / "frontend" / "public" / "icons" / "adapter-icons"