Skip to content

Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)

Python

Refactor smoke testing framework and enhance provider configurations

Alishahryar1751694a5dadd
diff --git a/pyproject.toml b/pyproject.toml
index d9a3e6dc..7b145bb7 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -104,9 +104,9 @@ markers = [
"provider: live provider checks",
"messaging: live messaging platform checks",
"cli: CLI integration checks",
- "vscode: VS Code or IDE client compatibility checks",
+ "clients: client compatibility checks",
"voice: voice transcription checks",
- "contract: deterministic smoke contract checks",
+ "contract: deterministic feature contract checks",
"smoke_target(name): route a smoke test behind FCC_SMOKE_TARGETS",
]
diff --git a/smoke/README.md b/smoke/README.md
index 894dfdfc..fedad56f 100644
--- a/smoke/README.md
+++ b/smoke/README.md
@@ -1,7 +1,7 @@
# Local Live Smoke Tests
-These tests are for maintainers running against their own `.env`. They are not
-part of CI and are not collected by plain `uv run pytest`.
+`smoke/` is for live/process/service checks only. Hermetic feature contracts
+belong under `tests/` and run with plain `uv run pytest`.
## Safe Default Run
@@ -12,23 +12,54 @@ uv run pytest smoke -n 0 -m live -s --tb=short
`-n 0` is recommended because the normal project pytest config enables xdist.
The smoke suite can run with workers, but one process gives clearer logs when a
-real provider or bot fails.
+real provider, bot, or local model server fails.
+
+## Targets
+
+Default targets do not require bot tokens or voice backends:
+
+| Target | What it checks | Required environment |
+| --- | --- | --- |
+| `api` | live uvicorn routes, probes, token count, optimizations, `/stop` fallback | none |
+| `auth` | Anthropic auth header variants on a live server | none; test sets an isolated token |
+| `cli` | `fcc-init`, `free-claude-code`, optional Claude CLI prompt | `claude` and provider config only for the prompt check |
+| `clients` | VS Code and JetBrains-shaped requests | none |
+| `providers` | configured provider model streams | provider keys or local provider URLs |
+| `tools` | configured model emits live tool use | tool-capable configured provider |
+| `rate_limit` | client disconnect does not crash a provider stream | configured provider |
+| `lmstudio` | LM Studio `/models` endpoint is reachable | running LM Studio server |
+| `llamacpp` | llama.cpp `/models` endpoint is reachable | running `llama-server` |
+
+Side-effectful targets are opt-in:
+
+| Target | What it checks | Required environment |
+| --- | --- | --- |
+| `telegram` | bot API getMe, send, edit, delete | `TELEGRAM_BOT_TOKEN` and chat/user ID |
+| `discord` | channel access, send, edit, delete | `DISCORD_BOT_TOKEN` and channel ID |
+| `voice` | configured transcription backend | `VOICE_NOTE_ENABLED=true` and `FCC_SMOKE_RUN_VOICE=1` |
+
+Use `FCC_SMOKE_TARGETS=all` to include Telegram, Discord, and voice checks.
## Targeted Runs
```powershell
$env:FCC_LIVE_SMOKE = "1"
-$env:FCC_SMOKE_TARGETS = "api,providers,thinking,tools"
+$env:FCC_SMOKE_TARGETS = "api,providers,tools"
uv run pytest smoke -n 0 -m live -s --tb=short
```
-Use `FCC_SMOKE_TARGETS=all` to include Telegram, Discord, and voice checks.
-The default target set intentionally excludes those side-effectful integrations.
+```powershell
+$env:FCC_LIVE_SMOKE = "1"
+$env:FCC_SMOKE_TARGETS = "telegram,discord,voice"
+$env:FCC_SMOKE_RUN_VOICE = "1"
+uv run pytest smoke -n 0 -m live -s --tb=short
+```
## Environment
- `FCC_ENV_FILE`: optional explicit dotenv path. The app still uses its normal
env-file precedence.
+- `FCC_SMOKE_TARGETS`: comma-separated targets, or `all`.
- `FCC_SMOKE_PROVIDER_MATRIX`: comma-separated provider prefixes to test.
- `FCC_SMOKE_TIMEOUT_S`: per-request/subprocess timeout, default `45`.
- `FCC_SMOKE_CLAUDE_BIN`: Claude CLI executable name, default `claude`.
@@ -37,18 +68,24 @@ The default target set intentionally excludes those side-effectful integrations.
- `FCC_SMOKE_INTERACTIVE=1`: enables manual inbound messaging checks.
- `FCC_SMOKE_RUN_VOICE=1`: allows the voice transcription backend to load/run.
-## Results
+## Feature Inventory
+
+The public feature inventory lives in `smoke/features.py`. Each entry records
+whether the feature is covered by hermetic pytest, live smoke, or both, plus the
+owning tests and skip policy. The manifest intentionally excludes `claude-pick`.
+
+## Results And Failure Classes
Smoke artifacts are written to `.smoke-results/` and ignored by git. Reports and
logs redact env values whose names contain `KEY`, `TOKEN`, `SECRET`, `WEBHOOK`,
or `AUTH`.
-## How To Read Failures
+Report classifications:
-- `missing_env`: configure the required key, token, channel, or local base URL.
-- `upstream_unavailable`: the provider/local model/bot API is not reachable.
-- `product_failure`: the app returned the wrong shape or crashed.
-- `harness_bug`: the smoke test itself made an invalid assumption.
+- `missing_env`: a credential, opt-in flag, binary, channel, or model config is absent.
+- `upstream_unavailable`: a real provider, bot API, or local model server is not reachable.
+- `product_failure`: the app returned the wrong shape, crashed, or violated a contract.
+- `harness_bug`: the smoke test made an invalid assumption.
-The first real run is expected to find product failures. Fix those separately
-from harness problems so the suite becomes a reliable regression signal.
+`missing_env` and `upstream_unavailable` should skip. `product_failure` and
+`harness_bug` should be fixed before relying on the smoke baseline.
diff --git a/smoke/conftest.py b/smoke/conftest.py
index 7ebb91eb..a10e413e 100644
--- a/smoke/conftest.py
+++ b/smoke/conftest.py
@@ -34,7 +34,9 @@ def pytest_runtest_setup(item: pytest.Item) -> None:
def pytest_runtest_logreport(report: pytest.TestReport) -> None:
- if report.when != "call":
+ if report.when == "setup" and not report.skipped:
+ return
+ if report.when == "teardown" and not report.failed:
return
if _REPORT is None:
return
diff --git a/smoke/features.py b/smoke/features.py
index 3f7e06e2..7e11552e 100644
--- a/smoke/features.py
+++ b/smoke/features.py
@@ -1,20 +1,38 @@
-"""Feature inventory used by the local smoke suite.
+"""Public feature inventory for pytest contracts and live smoke checks.
-This file is intentionally explicit: advertised features should not exist only
-in README prose without at least one smoke check or a documented manual gap.
+The inventory is intentionally explicit. README features and exposed public
+surface area should have either hermetic pytest coverage, live smoke coverage,
+or both. Private helpers are not listed here.
"""
from __future__ import annotations
from dataclasses import dataclass
+from typing import Literal
+
+FeatureSource = Literal["readme", "public_surface"]
+CoverageKind = Literal["pytest", "live_smoke", "both"]
@dataclass(frozen=True, slots=True)
-class FeatureSmoke:
+class FeatureCoverage:
feature_id: str
title: str
- mode: str
- checks: tuple[str, ...]
+ source: FeatureSource
+ coverage: CoverageKind
+ pytest_tests: tuple[str, ...]
+ smoke_tests: tuple[str, ...]
+ smoke_targets: tuple[str, ...]
+ required_env: tuple[str, ...]
+ skip_policy: str
+
+ @property
+ def has_pytest_coverage(self) -> bool:
+ return self.coverage in {"pytest", "both"}
+
+ @property
+ def has_smoke_coverage(self) -> bool:
+ return self.coverage in {"live_smoke", "both"}
README_FEATURES: tuple[str, ...] = (
@@ -36,112 +54,408 @@ README_FEATURES: tuple[str, ...] = (
)
-FEATURE_SMOKES: tuple[FeatureSmoke, ...] = (
- FeatureSmoke(
+FEATURE_INVENTORY: tuple[FeatureCoverage, ...] = (
+ FeatureCoverage(
"zero_cost_provider_access",
"Configured provider accepts a real prompt",
- "live",
+ "readme",
+ "both",
+ ("tests/api/test_dependencies.py", "tests/providers/test_open_router.py"),
("test_configured_provider_models_stream_successfully",),
+ ("providers",),
+ ("NVIDIA_NIM_API_KEY|OPENROUTER_API_KEY|DEEPSEEK_API_KEY|local provider",),
+ "skip when no usable provider credentials or local provider endpoint exists",
),
- FeatureSmoke(
+ FeatureCoverage(
"drop_in_claude_code_replacement",
"Claude-compatible routes and CLI environment work",
- "live",
+ "readme",
+ "both",
+ ("tests/api/test_api.py", "tests/cli/test_cli.py"),
(
"test_probe_and_models_routes",
"test_claude_cli_prompt_when_available",
"test_vscode_and_jetbrains_shaped_requests",
),
+ ("api", "cli", "clients"),
+ ("FCC_SMOKE_CLAUDE_BIN", "configured provider for Claude CLI"),
+ "skip Claude CLI path when binary or provider config is unavailable",
),
- FeatureSmoke(
+ FeatureCoverage(
"provider_matrix",
"All configured provider prefixes can be exercised",
- "live",
+ "readme",
+ "both",
+ ("tests/api/test_dependencies.py", "tests/providers/"),
("test_configured_provider_models_stream_successfully",),
+ ("providers",),
+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),
+ "skip when no configured provider has usable credentials/base URL",
),
- FeatureSmoke(
+ FeatureCoverage(
"per_model_mapping",
"Opus, Sonnet, Haiku, and fallback mappings are visible",
- "live",
+ "readme",
+ "both",
+ ("tests/api/test_models_validators.py", "tests/config/test_config.py"),
("test_model_mapping_configuration_is_consistent",),
+ ("providers",),
+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),
+ "skip live provider execution when no provider config is usable",
),
- FeatureSmoke(
+ FeatureCoverage(
+ "mixed_provider_mapping",
+ "Model-specific overrides can route to different providers",
+ "public_surface",
+ "both",
+ ("tests/config/test_config.py",),
+ ("test_mixed_provider_model_mapping_when_configured",),
+ ("providers",),
+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),
+ "skip when fewer than two configured provider prefixes are present",
+ ),
+ FeatureCoverage(
"thinking_token_support",
"Thinking streams and suppression are contract-tested",
- "contract",
+ "readme",
+ "pytest",
(
"test_interleaved_thinking_text_blocks_are_valid",
"test_split_think_tags_preserve_text_and_thinking",
"test_enable_thinking_false_suppresses_reasoning_only",
),
+ (),
+ (),
+ (),
+ "hermetic contract coverage",
),
- FeatureSmoke(
+ FeatureCoverage(
"heuristic_tool_parser",
"Text tool calls become structured tool_use blocks",
- "contract",
- ("test_thinking_tool_text_and_transcript_order_contract",),
+ "readme",
+ "both",
+ (
+ "tests/providers/test_parsers.py",
+ "test_thinking_tool_text_and_transcript_order_contract",
+ ),
+ ("test_live_tool_use_when_configured_model_supports_tools",),
+ ("tools",),
+ ("configured tool-capable provider",),
+ "skip live tool-use when no configured provider model is available",
),
- FeatureSmoke(
+ FeatureCoverage(
"request_optimization",
"Fast-path local optimizations respond without providers",
- "live",
+ "readme",
+ "both",
+ (
+ "tests/api/test_optimization_handlers.py",
+ "tests/api/test_routes_optimizations.py",
+ ),
("test_optimization_fast_paths_do_not_need_provider",),
+ ("api",),
+ (),
+ "always runnable once live server starts",
),
- FeatureSmoke(
+ FeatureCoverage(
"smart_rate_limiting",
- "Concurrency/disconnect and retry-sensitive paths are checked",
- "live",
+ "Provider limiter covers proactive throttling, 429 retry, and disconnects",
+ "readme",
+ "both",
+ ("tests/providers/test_provider_rate_limit.py",),
("test_client_disconnect_mid_stream_does_not_crash_server",),
+ ("rate_limit", "providers"),
+ ("configured provider",),
+ "skip live disconnect path when no configured provider model is available",
),
- FeatureSmoke(
+ FeatureCoverage(
"discord_telegram_bot",
"Messaging credentials, send/edit/delete, and transcript behavior",
- "live_or_interactive",
+ "readme",
+ "both",
+ (
+ "tests/messaging/test_discord_platform.py",
+ "tests/messaging/test_telegram.py",
+ ),
(
"test_telegram_bot_api_permissions",
"test_discord_bot_api_permissions",
- "test_thinking_tool_text_and_transcript_order_contract",
+ "test_interactive_inbound_messaging_requires_explicit_mode",
+ ),
+ ("telegram", "discord"),
+ (
+ "TELEGRAM_BOT_TOKEN",
+ "ALLOWED_TELEGRAM_USER_ID|FCC_SMOKE_TELEGRAM_CHAT_ID",
+ "DISCORD_BOT_TOKEN",
+ "ALLOWED_DISCORD_CHANNELS|FCC_SMOKE_DISCORD_CHANNEL_ID",
),
+ "skip when bot tokens/channels are absent; inbound checks require interactive mode",
),
- FeatureSmoke(
+ FeatureCoverage(
"subagent_control",
"Task tool calls do not run in the background",
- "contract",
- ("test_task_tool_arguments_force_foreground_execution",),
+ "readme",
+ "pytest",
+ (
+ "tests/providers/test_subagent_interception.py",
+ "test_thinking_tool_text_and_transcript_order_contract",
+ ),
+ (),
+ (),
+ (),
+ "hermetic contract coverage",
),
- FeatureSmoke(
+ FeatureCoverage(
"extensible_provider_platform_abcs",
"Provider/platform registries expose expected built-ins",
- "contract",
+ "readme",
+ "pytest",
("test_provider_and_platform_registries_include_advertised_builtins",),
+ (),
+ (),
+ (),
+ "hermetic contract coverage",
),
- FeatureSmoke(
+ FeatureCoverage(
"optional_authentication",
"Anthropic-style auth headers are accepted and enforced",
- "live",
+ "readme",
+ "both",
+ ("tests/api/test_auth.py",),
("test_auth_token_is_enforced_for_all_supported_header_shapes",),
+ ("auth",),
+ ("ANTHROPIC_AUTH_TOKEN",),
+ "live test starts an isolated server with its own token",
),
- FeatureSmoke(
+ FeatureCoverage(
"vscode_extension",
"VS Code-shaped beta requests work against the proxy",
- "live",
+ "readme",
+ "live_smoke",
+ (),
("test_vscode_and_jetbrains_shaped_requests",),
+ ("clients",),
+ (),
+ "always runnable once live server starts",
),
- FeatureSmoke(
+ FeatureCoverage(
"intellij_extension",
"JetBrains/ACP-shaped environment requests work against the proxy",
- "live",
+ "readme",
+ "live_smoke",
+ (),
("test_vscode_and_jetbrains_shaped_requests",),
+ ("clients",),
+ (),
+ "always runnable once live server starts",
),
- FeatureSmoke(
+ FeatureCoverage(
"voice_notes",
"Configured transcription backend can process an audio fixture",
- "live_or_skip",
+ "readme",
+ "both",
+ (
+ "tests/messaging/test_voice_handlers.py",
+ "tests/messaging/test_transcription.py",
+ ),
("test_voice_transcription_backend_when_explicitly_enabled",),
+ ("voice",),
+ ("VOICE_NOTE_ENABLED", "FCC_SMOKE_RUN_VOICE", "WHISPER_DEVICE"),
+ "skip unless voice is explicitly enabled for local smoke",
+ ),
+ FeatureCoverage(
+ "anthropic_api_routes",
+ "Root, health, models, messages, count_tokens, and stop routes respond",
+ "public_surface",
+ "both",
+ ("tests/api/test_api.py",),
+ ("test_probe_and_models_routes", "test_stop_endpoint_reports_no_messaging"),
+ ("api",),
+ (),
+ "always runnable once live server starts",
+ ),
+ FeatureCoverage(
+ "probe_routes",
+ "HEAD and OPTIONS compatibility probes return 204 and Allow headers",
+ "public_surface",
+ "both",
+ ("tests/api/test_api.py::test_probe_endpoints_return_204_with_allow_headers",),
+ ("test_probe_and_models_routes",),
+ ("api",),
+ (),
+ "always runnable once live server starts",
+ ),
+ FeatureCoverage(
+ "count_tokens_contract",
+ "Token counting accepts text, thinking, tools, tool results, and images",
+ "public_surface",
+ "both",
+ ("tests/api/test_request_utils.py",),
+ ("test_count_tokens_accepts_thinking_tools_and_results",),
+ ("api",),
+ (),
+ "always runnable once live server starts",
+ ),
+ FeatureCoverage(
+ "provider_proxy_timeout_config",
+ "Provider proxies and HTTP timeouts are passed into clients",
+ "public_surface",
+ "pytest",
+ ("tests/api/test_dependencies.py",),
+ (),
+ (),
+ (),
+ "hermetic config coverage",
+ ),
+ FeatureCoverage(
+ "lmstudio_endpoint",
+ "Configured LM Studio endpoint exposes an OpenAI-compatible models route",
+ "public_surface",
+ "both",
+ ("tests/providers/test_lmstudio.py",),
+ ("test_lmstudio_models_endpoint_when_available",),
+ ("lmstudio",),
+ ("LM_STUDIO_BASE_URL",),
+ "skip when the local LM Studio server is not reachable",
+ ),
+ FeatureCoverage(
+ "llamacpp_endpoint",
+ "Configured llama.cpp endpoint exposes an OpenAI-compatible models route",
+ "public_surface",
+ "both",
+ ("tests/providers/test_llamacpp.py",),
+ ("test_llamacpp_models_endpoint_when_available",),
+ ("llamacpp",),
+ ("LLAMACPP_BASE_URL",),
+ "skip when the local llama.cpp server is not reachable",
+ ),
+ FeatureCoverage(
+ "package_cli_entrypoints",
+ "Installed package scripts scaffold config and start the server",
+ "public_surface",
+ "both",
+ ("tests/cli/test_entrypoints.py",),
+ (
+ "test_fcc_init_scaffolds_user_config",
+ "test_free_claude_code_entrypoint_starts_server",
+ ),
+ ("cli",),
+ (),
+ "always runnable once uv project dependencies are available",
+ ),
+ FeatureCoverage(
+ "claude_cli_drop_in",
+ "Claude CLI can send a prompt through the proxy when installed",
+ "public_surface",
+ "live_smoke",
+ (),
+ ("test_claude_cli_prompt_when_available",),
+ ("cli",),
+ ("FCC_SMOKE_CLAUDE_BIN", "configured provider"),
+ "skip when Claude CLI or provider config is unavailable",
+ ),
+ FeatureCoverage(
+ "messaging_commands",
+ "Messaging /stop, /clear, and /stats commands use platform queues safely",
+ "public_surface",
+ "pytest",
+ (
+ "tests/messaging/test_handler.py",
+ "tests/messaging/test_handler_integration.py",
+ ),
+ (),
+ (),
+ (),
+ "hermetic messaging handler coverage",
+ ),
+ FeatureCoverage(
+ "tree_threading",
+ "Reply-based message trees support queueing, branching, and cancellation",
+ "public_surface",
+ "pytest",
+ (
+ "tests/messaging/test_tree_queue.py",
+ "tests/messaging/test_tree_concurrency.py",
+ ),
+ (),
+ (),
+ (),
+ "hermetic tree queue coverage",
+ ),
+ FeatureCoverage(
+ "restart_restore",
+ "Persisted tree state is restored after restart for reply routing",
+ "public_surface",
+ "pytest",
+ ("tests/messaging/test_restart_reply_restore.py",),
+ (),
+ (),
+ (),
+ "hermetic persistence coverage",
+ ),
+ FeatureCoverage(
+ "session_persistence",
+ "SessionStore persists trees, node mappings, and message logs",
+ "public_surface",
+ "pytest",
+ ("tests/messaging/test_session_store_edge_cases.py",),
+ (),
+ (),
+ (),
+ "hermetic persistence coverage",
+ ),
+ FeatureCoverage(
+ "config_env_precedence",
+ "Environment and dotenv precedence are deterministic",
+ "public_surface",
+ "pytest",
+ ("tests/config/test_config.py",),
+ (),
+ (),
+ (),
+ "hermetic config coverage",
+ ),
+ FeatureCoverage(
+ "removed_env_migration",
+ "Removed NIM_ENABLE_THINKING env var fails fast with migration guidance",
+ "public_surface",
+ "pytest",
+ ("tests/config/test_config.py",),
+ (),
+ (),
+ (),
+ "hermetic config coverage",
+ ),
+ FeatureCoverage(
+ "streaming_error_mapping",
+ "Provider and streaming errors map to Anthropic-compatible error shapes",
+ "public_surface",
+ "pytest",
+ (
+ "tests/providers/test_streaming_errors.py",
+ "tests/providers/test_error_mapping.py",
+ ),
+ (),
+ (),
+ (),
+ "hermetic provider coverage",
),
)
-def smoke_ids() -> set[str]:
- """Return feature IDs covered by the smoke manifest."""
- return {feature.feature_id for feature in FEATURE_SMOKES}
+def feature_ids(*, source: FeatureSource | None = None) -> set[str]:
+ """Return feature IDs covered by the inventory."""
+ return {
+ feature.feature_id
+ for feature in FEATURE_INVENTORY
+ if source is None or feature.source == source
+ }
+
+
+def smoke_feature_ids() -> set[str]:
+ """Return feature IDs with at least one live smoke owner."""
+ return {
+ feature.feature_id
+ for feature in FEATURE_INVENTORY
+ if feature.has_smoke_coverage
+ }
diff --git a/smoke/lib/config.py b/smoke/lib/config.py
index 73da08f2..3862b651 100644
--- a/smoke/lib/config.py
+++ b/smoke/lib/config.py
@@ -14,18 +14,47 @@ DEFAULT_TARGETS = frozenset(
"api",
"auth",
"cli",
- "contract",
- "optimizations",
+ "clients",
+ "llamacpp",
+ "lmstudio",
"providers",
- "thinking",
+ "rate_limit",
"tools",
- "vscode",
}
)
-ALL_TARGETS = DEFAULT_TARGETS | frozenset({"discord", "telegram", "voice"})
+SIDE_EFFECT_TARGETS = frozenset({"discord", "telegram", "voice"})
+ALL_TARGETS = DEFAULT_TARGETS | SIDE_EFFECT_TARGETS
+TARGET_ALIASES = {
+ "contract": "api",
+ "optimizations": "api",
+ "thinking": "providers",
+ "vscode": "clients",
+}
SECRET_KEY_PARTS = ("KEY", "TOKEN", "SECRET", "WEBHOOK", "AUTH")
+TARGET_REQUIRED_ENV: dict[str, tuple[str, ...]] = {
+ "api": (),
+ "auth": (),
+ "cli": ("FCC_SMOKE_CLAUDE_BIN", "configured provider for Claude CLI prompt"),
+ "clients": (),
+ "providers": ("MODEL or MODEL_* with usable provider configuration",),
+ "rate_limit": ("configured provider model",),
+ "tools": ("configured tool-capable provider model",),
+ "lmstudio": ("LM_STUDIO_BASE_URL with a running LM Studio server",),
+ "llamacpp": ("LLAMACPP_BASE_URL with a running llama-server",),
+ "telegram": (
+ "TELEGRAM_BOT_TOKEN",
+ "ALLOWED_TELEGRAM_USER_ID or FCC_SMOKE_TELEGRAM_CHAT_ID",
+ ),
+ "discord": (
+ "DISCORD_BOT_TOKEN",
+ "ALLOWED_DISCORD_CHANNELS or FCC_SMOKE_DISCORD_CHANNEL_ID",
+ ),
+ "voice": ("VOICE_NOTE_ENABLED=true", "FCC_SMOKE_RUN_VOICE=1"),
+}
+
+
@dataclass(frozen=True, slots=True)
class ProviderModel:
provider: str
@@ -122,7 +151,7 @@ def _parse_targets(raw: str | None) -> frozenset[str]:
parsed = _parse_csv(raw)
if "all" in parsed:
return ALL_TARGETS
- return parsed
+ return frozenset(TARGET_ALIASES.get(target, target) for target in parsed)
def auth_headers(token: str | None = None) -> dict[str, str]:
diff --git a/smoke/lib/report.py b/smoke/lib/report.py
index 7a54597c..448b1d44 100644
--- a/smoke/lib/report.py
+++ b/smoke/lib/report.py
@@ -13,6 +13,7 @@ from .config import SmokeConfig, redacted
class SmokeOutcome:
nodeid: str
outcome: str
+ classification: str
duration_s: float
markers: list[str]
detail: str
@@ -37,6 +38,9 @@ class SmokeReport:
SmokeOutcome(
nodeid=nodeid,
outcome=outcome,
+ classification=classify_outcome(
+ nodeid=nodeid, outcome=outcome, detail=detail
+ ),
duration_s=duration_s,
markers=markers,
detail=redacted(detail),
@@ -56,3 +60,42 @@ class SmokeReport:
"outcomes": [asdict(outcome) for outcome in self.outcomes],
}
path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")
+
+
+def classify_outcome(*, nodeid: str, outcome: str, detail: str) -> str:
+ """Classify smoke outcomes for triage reports."""
+ if outcome == "passed":
+ return "passed"
+
+ text = f"{nodeid}\n{detail}".lower()
+ if outcome == "skipped":
+ if any(
+ marker in text
+ for marker in (
+ "upstream_unavailable",
+ "connection refused",
+ "connecterror",
+ "readtimeout",
+ "timed out",
+ "not reachable",
+ )
+ ):
+ return "upstream_unavailable"
+ return "missing_env"
+
+ if "harness_bug" in text:
+ return "harness_bug"
+ if any(
+ marker in text
+ for marker in (
+ "upstream_unavailable",
+ "connection refused",
+ "connecterror",
+ "readtimeout",
+ "timed out",
+ "not reachable",
+ "upstream provider",
+ )
+ ):
+ return "upstream_unavailable"
+ return "product_failure"
diff --git a/smoke/lib/skips.py b/smoke/lib/skips.py
new file mode 100644
index 00000000..8ce03edd
--- /dev/null
+++ b/smoke/lib/skips.py
@@ -0,0 +1,61 @@
+"""Skip helpers for expected live-smoke environment gaps."""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+
+from smoke.lib.sse import SSEEvent
+
+UPSTREAM_UNAVAILABLE_MARKERS = (
+ "connection refused",
+ "connecterror",
+ "connect timeout",
+ "readtimeout",
+ "server disconnected",
+ "service unavailable",
+ "temporary failure",
+ "timed out",
+ "upstream provider",
+)
+
+
+def is_upstream_unavailable_text(text: str) -> bool:
+ normalized = text.lower()
+ return any(marker in normalized for marker in UPSTREAM_UNAVAILABLE_MARKERS)
+
+
+def skip_upstream_unavailable(reason: str) -> None:
+ pytest.skip(f"upstream_unavailable: {reason}")
+
+
+def skip_if_upstream_unavailable_exception(exc: Exception) -> None:
+ if isinstance(
+ exc,
+ (
+ httpx.ConnectError,
+ httpx.ConnectTimeout,
+ httpx.ReadTimeout,
+ httpx.RemoteProtocolError,
+ httpx.ProxyError,
+ ),
+ ):
+ skip_upstream_unavailable(f"{type(exc).__name__}: {exc}")
+ if is_upstream_unavailable_text(f"{type(exc).__name__}: {exc}"):
+ skip_upstream_unavailable(f"{type(exc).__name__}: {exc}")
+
+
+def skip_if_upstream_unavailable_events(events: list[SSEEvent]) -> None:
+ for event in events:
+ if getattr(event, "event", None) != "error":
+ continue
+ data = getattr(event, "data", {})
+ message = ""
+ if isinstance(data, dict):
+ error = data.get("error")
+ if isinstance(error, dict):
+ message = str(error.get("message", ""))
+ else:
+ message = str(data)
+ if is_upstream_unavailable_text(message):
+ skip_upstream_unavailable(message[:500])
diff --git a/smoke/test_api_live.py b/smoke/test_api_live.py
index ffffa204..7f488fca 100644
--- a/smoke/test_api_live.py
+++ b/smoke/test_api_live.py
@@ -173,3 +173,15 @@ def test_invalid_messages_returns_anthropic_error(
payload = response.json()
assert payload["type"] == "error"
assert payload["error"]["type"] == "invalid_request_error"
+
+
+def test_stop_endpoint_reports_no_messaging(
+ smoke_server: RunningServer, smoke_headers: dict[str, str]
+) -> None:
+ response = httpx.post(
+ f"{smoke_server.base_url}/stop",
+ headers=smoke_headers,
+ timeout=5,
+ )
+ assert response.status_code == 503
+ assert response.json()["detail"] == "Messaging system not initialized"
diff --git a/smoke/test_auth_live.py b/smoke/test_auth_live.py
index c6390b98..0062f90f 100644
--- a/smoke/test_auth_live.py
+++ b/smoke/test_auth_live.py
@@ -1,5 +1,7 @@
from __future__ import annotations
+from pathlib import Path
+
import httpx
import pytest
@@ -10,12 +12,19 @@ pytestmark = [pytest.mark.live, pytest.mark.smoke_target("auth")]
def test_auth_token_is_enforced_for_all_supported_header_shapes(
- smoke_config: SmokeConfig,
+ smoke_config: SmokeConfig, tmp_path: Path
) -> None:
token = "fcc-smoke-token"
+ env_file = tmp_path / "auth.env"
+ env_file.write_text(f'ANTHROPIC_AUTH_TOKEN="{token}"\n', encoding="utf-8")
+
with start_server(
smoke_config,
- env_overrides={"ANTHROPIC_AUTH_TOKEN": token, "MESSAGING_PLATFORM": "none"},
+ env_overrides={
+ "ANTHROPIC_AUTH_TOKEN": token,
+ "FCC_ENV_FILE": str(env_file),
+ "MESSAGING_PLATFORM": "none",
+ },
name="auth",
) as server:
assert httpx.get(f"{server.base_url}/").status_code == 401
diff --git a/smoke/test_client_shapes_live.py b/smoke/test_client_shapes_live.py
index 47d3b924..47a94592 100644
--- a/smoke/test_client_shapes_live.py
+++ b/smoke/test_client_shapes_live.py
@@ -6,7 +6,7 @@ from smoke.lib.config import SmokeConfig, auth_headers
from smoke.lib.http import message_payload, post_json
from smoke.lib.server import RunningServer
-pytestmark = [pytest.mark.live, pytest.mark.smoke_target("vscode")]
+pytestmark = [pytest.mark.live, pytest.mark.smoke_target("clients")]
def test_vscode_and_jetbrains_shaped_requests(
diff --git a/smoke/test_feature_manifest.py b/smoke/test_feature_manifest.py
deleted file mode 100644
index a72b651b..00000000
--- a/smoke/test_feature_manifest.py
+++ /dev/null
@@ -1,48 +0,0 @@
-from __future__ import annotations
-
-import pytest
-
-from messaging.platforms.factory import create_messaging_platform
-from providers.base import BaseProvider
-from providers.deepseek import DeepSeekProvider
-from providers.llamacpp import LlamaCppProvider
-from providers.lmstudio import LMStudioProvider
-from providers.nvidia_nim import NvidiaNimProvider
-from providers.open_router import OpenRouterProvider
-from smoke.features import FEATURE_SMOKES, README_FEATURES, smoke_ids
-
-pytestmark = [pytest.mark.live, pytest.mark.smoke_target("contract")]
-
-
-def test_every_advertised_feature_has_a_smoke_entry() -> None:
- missing = sorted(set(README_FEATURES) - smoke_ids())
- extra = sorted(smoke_ids() - set(README_FEATURES))
- assert not missing, f"README features missing smoke entries: {missing}"
- assert not extra, f"smoke entries not tied to README features: {extra}"
-
-
-def test_smoke_manifest_has_unique_ids_and_checks() -> None:
- ids = [feature.feature_id for feature in FEATURE_SMOKES]
- assert len(ids) == len(set(ids))
- for feature in FEATURE_SMOKES:
- assert feature.checks, feature
- assert feature.mode in {
- "live",
- "contract",
- "live_or_interactive",
- "live_or_skip",
- }
-
-
-def test_provider_and_platform_registries_include_advertised_builtins() -> None:
- provider_classes = {
- "nvidia_nim": NvidiaNimProvider,
- "open_router": OpenRouterProvider,
- "deepseek": DeepSeekProvider,
- "lmstudio": LMStudioProvider,
- "llamacpp": LlamaCppProvider,
- }
- for provider_class in provider_classes.values():
- assert issubclass(provider_class, BaseProvider)
-
- assert create_messaging_platform("not-a-platform") is None
diff --git a/smoke/test_local_provider_endpoints_live.py b/smoke/test_local_provider_endpoints_live.py
new file mode 100644
index 00000000..ab9652ed
--- /dev/null
+++ b/smoke/test_local_provider_endpoints_live.py
@@ -0,0 +1,48 @@
+from __future__ import annotations
+
+import httpx
+import pytest
+
+from smoke.lib.config import SmokeConfig
+from smoke.lib.skips import skip_if_upstream_unavailable_exception
+
+
[email protected]_target("lmstudio")
+def test_lmstudio_models_endpoint_when_available(smoke_config: SmokeConfig) -> None:
+ _assert_models_endpoint(
+ smoke_config.settings.lm_studio_base_url,
+ timeout_s=smoke_config.timeout_s,
+ provider_name="LM Studio",
+ )
+
+
[email protected]_target("llamacpp")
+def test_llamacpp_models_endpoint_when_available(smoke_config: SmokeConfig) -> None:
+ _assert_models_endpoint(
+ smoke_config.settings.llamacpp_base_url,
+ timeout_s=smoke_config.timeout_s,
+ provider_name="llama.cpp",
+ )
+
+
+def _assert_models_endpoint(
+ base_url: str, *, timeout_s: float, provider_name: str
+) -> None:
+ url = f"{base_url.rstrip('/')}/models"
+ try:
+ response = httpx.get(url, timeout=timeout_s)
+ except Exception as exc:
+ skip_if_upstream_unavailable_exception(exc)
+ raise
+
+ if response.status_code in {404, 405, 502, 503}:
+ pytest.skip(
+ f"upstream_unavailable: {provider_name} models endpoint "
+ f"{url} returned HTTP {response.status_code}"
+ )
+
+ assert response.status_code == 200, response.text
+ payload = response.json()
+ assert isinstance(payload.get("data"), list), payload
diff --git a/smoke/test_provider_live.py b/smoke/test_provider_live.py
index c139e0a8..8731278a 100644
--- a/smoke/test_provider_live.py
+++ b/smoke/test_provider_live.py
@@ -8,6 +8,10 @@ import pytest
from smoke.lib.config import SmokeConfig, auth_headers
from smoke.lib.http import collect_message_stream, message_payload
from smoke.lib.server import start_server
+from smoke.lib.skips import (
+ skip_if_upstream_unavailable_events,
+ skip_if_upstream_unavailable_exception,
+)
from smoke.lib.sse import assert_anthropic_stream_contract, text_content
pytestmark = [pytest.mark.live, pytest.mark.smoke_target("providers")]
@@ -22,6 +26,19 @@ def test_model_mapping_configuration_is_consistent(smoke_config: SmokeConfig) ->
assert provider_model.model_name
+def test_mixed_provider_model_mapping_when_configured(
+ smoke_config: SmokeConfig,
+) -> None:
+ models = smoke_config.provider_models()
+ providers = {provider_model.provider for provider_model in models}
+ if len(providers) < 2:
+ pytest.skip("configure MODEL_* with at least two provider prefixes")
+
+ sources = {provider_model.source for provider_model in models}
+ assert sources <= {"MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"}
+ assert len(providers) >= 2
+
+
def test_configured_provider_models_stream_successfully(
smoke_config: SmokeConfig,
) -> None:
@@ -45,9 +62,11 @@ def test_configured_provider_models_stream_successfully(
message_payload(smoke_config.prompt, model="fcc-smoke-default"),
smoke_config,
)
+ skip_if_upstream_unavailable_events(events)
assert_anthropic_stream_contract(events)
assert text_content(events).strip(), "provider returned no text"
except Exception as exc:
+ skip_if_upstream_unavailable_exception(exc)
failures.append(
f"{provider_model.source}={provider_model.full_model}: "
f"{type(exc).__name__}: {exc}"
@@ -56,6 +75,7 @@ def test_configured_provider_models_stream_successfully(
assert not failures, "\n".join(failures)
[email protected]_target("rate_limit")
def test_client_disconnect_mid_stream_does_not_crash_server(
smoke_config: SmokeConfig,
) -> None:
diff --git a/smoke/test_tools_live.py b/smoke/test_tools_live.py
index b4824109..824197be 100644
--- a/smoke/test_tools_live.py
+++ b/smoke/test_tools_live.py
@@ -5,6 +5,7 @@ import pytest
from smoke.lib.config import SmokeConfig
from smoke.lib.http import collect_message_stream, message_payload
from smoke.lib.server import start_server
+from smoke.lib.skips import skip_if_upstream_unavailable_events
from smoke.lib.sse import assert_anthropic_stream_contract, has_tool_use
pytestmark = [pytest.mark.live, pytest.mark.smoke_target("tools")]
@@ -47,5 +48,6 @@ def test_live_tool_use_when_configured_model_supports_tools(
name="tools",
) as server:
events = collect_message_stream(server, payload, smoke_config)
+ skip_if_upstream_unavailable_events(events)
assert_anthropic_stream_contract(events)
assert has_tool_use(events), "model did not emit a tool_use block"
diff --git a/tests/contracts/test_feature_manifest.py b/tests/contracts/test_feature_manifest.py
new file mode 100644
index 00000000..71d3deee
--- /dev/null
+++ b/tests/contracts/test_feature_manifest.py
@@ -0,0 +1,88 @@
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+from messaging.platforms.factory import create_messaging_platform
+from providers.base import BaseProvider
+from providers.deepseek import DeepSeekProvider
+from providers.llamacpp import LlamaCppProvider
+from providers.lmstudio import LMStudioProvider
+from providers.nvidia_nim import NvidiaNimProvider
+from providers.open_router import OpenRouterProvider
+from smoke.features import FEATURE_INVENTORY, README_FEATURES, feature_ids
+
+VALID_COVERAGE = {"pytest", "live_smoke", "both"}
+VALID_SOURCE = {"readme", "public_surface"}
+
+
+def test_every_readme_feature_has_inventory_entry() -> None:
+ missing = sorted(set(README_FEATURES) - feature_ids(source="readme"))
+ extra_readme = sorted(feature_ids(source="readme") - set(README_FEATURES))
+ assert not missing, f"README features missing inventory entries: {missing}"
+ assert not extra_readme, (
+ f"README inventory entries not in README_FEATURES: {extra_readme}"
+ )
+
+
+def test_feature_inventory_is_unique_and_decision_complete() -> None:
+ ids = [feature.feature_id for feature in FEATURE_INVENTORY]
+ assert len(ids) == len(set(ids))
+ assert "claude_pick" not in ids
+
+ for feature in FEATURE_INVENTORY:
+ assert feature.source in VALID_SOURCE, feature
+ assert feature.coverage in VALID_COVERAGE, feature
+ assert feature.title.strip(), feature
+ assert feature.skip_policy.strip(), feature
+ if feature.has_pytest_coverage:
+ assert feature.pytest_tests, feature
+ if feature.has_smoke_coverage:
+ assert feature.smoke_tests, feature
+ assert feature.smoke_targets, feature
+
+
+def test_feature_inventory_test_owners_exist() -> None:
+ repo_root = Path(__file__).resolve().parents[2]
+ pytest_names = _collect_test_names(repo_root / "tests")
+ smoke_names = _collect_test_names(repo_root / "smoke")
+
+ for feature in FEATURE_INVENTORY:
+ for owner in feature.pytest_tests:
+ _assert_owner_exists(owner, repo_root, pytest_names)
+ for owner in feature.smoke_tests:
+ assert owner in smoke_names or owner in pytest_names, (feature, owner)
+
+
+def test_provider_and_platform_registries_include_advertised_builtins() -> None:
+ provider_classes = {
+ "nvidia_nim": NvidiaNimProvider,
+ "open_router": OpenRouterProvider,
+ "deepseek": DeepSeekProvider,
+ "lmstudio": LMStudioProvider,
+ "llamacpp": LlamaCppProvider,
+ }
+ for provider_class in provider_classes.values():
+ assert issubclass(provider_class, BaseProvider)
+
+ assert create_messaging_platform("not-a-platform") is None
+
+
+def _collect_test_names(root: Path) -> set[str]:
+ names: set[str] = set()
+ for path in root.rglob("test_*.py"):
+ text = path.read_text(encoding="utf-8")
+ names.update(re.findall(r"^\s*(?:async\s+)?def (test_[^(]+)", text, re.M))
+ return names
+
+
+def _assert_owner_exists(owner: str, repo_root: Path, test_names: set[str]) -> None:
+ if owner.startswith("test_"):
+ assert owner in test_names, owner
+ return
+
+ path_part, _, node_name = owner.partition("::")
+ path = repo_root / path_part
+ assert path.exists(), owner
+ if node_name:
+ assert node_name in test_names, owner
diff --git a/smoke/test_stream_contracts.py b/tests/contracts/test_stream_contracts.py
similarity index 94%
rename from smoke/test_stream_contracts.py
rename to tests/contracts/test_stream_contracts.py
index 7f4ee6e0..cff9eb33 100644
--- a/smoke/test_stream_contracts.py
+++ b/tests/contracts/test_stream_contracts.py
@@ -2,8 +2,6 @@ from __future__ import annotations
from collections.abc import Iterable
-import pytest
-
from messaging.event_parser import parse_cli_event
from messaging.transcript import RenderCtx, TranscriptBuffer
from providers.common import (
@@ -21,12 +19,6 @@ from smoke.lib.sse import (
thinking_content,
)
-pytestmark = [
- pytest.mark.live,
- pytest.mark.smoke_target("contract"),
- pytest.mark.smoke_target("thinking"),
-]
-
def test_interleaved_thinking_text_blocks_are_valid() -> None:
events = _parse_builder_events(
@@ -50,7 +42,7 @@ def test_split_think_tags_preserve_text_and_thinking() -> None:
def test_mixed_reasoning_content_and_think_tags_keep_order() -> None:
- builder = SSEBuilder("msg_smoke", "smoke-model")
+ builder = SSEBuilder("msg_contract", "contract-model")
chunks = [builder.message_start()]
chunks.extend(builder.ensure_thinking_block())
chunks.append(builder.emit_thinking_delta("reasoning field"))
@@ -68,7 +60,7 @@ def test_mixed_reasoning_content_and_think_tags_keep_order() -> None:
def test_thinking_tool_text_and_transcript_order_contract() -> None:
- builder = SSEBuilder("msg_smoke", "smoke-model")
+ builder = SSEBuilder("msg_contract", "contract-model")
chunks = [builder.message_start()]
chunks.extend(builder.ensure_thinking_block())
chunks.append(builder.emit_thinking_delta("inspect first"))
@@ -136,7 +128,7 @@ def test_task_tool_arguments_force_foreground_execution() -> None:
def _interleaved_thinking_text_events(
parts: tuple[str, str, str, str],
) -> Iterable[str]:
- builder = SSEBuilder("msg_smoke", "smoke-model")
+ builder = SSEBuilder("msg_contract", "contract-model")
yield builder.message_start()
yield from builder.ensure_thinking_block()
yield builder.emit_thinking_delta(parts[0])
@@ -157,7 +149,7 @@ def _events_from_text_chunks(
*,
enable_thinking: bool = True,
) -> list[str]:
- sse = builder or SSEBuilder("msg_smoke", "smoke-model")
+ sse = builder or SSEBuilder("msg_contract", "contract-model")
out: list[str] = [] if builder else [sse.message_start()]
parser = ThinkTagParser()