Use Claude Code, Codex, VSCode, Pi, and OpenCode (and 6 other harnesses) for free (1.3B+ free tokens) from your terminal, app, IDE, or phone, and now from the browser with native browser sessions (multi-harness + multi-model) like OpenClaw (voice supported + ToS friendly)
Python
Refactor smoke testing framework and enhance provider configurations
Alishahryar1751694a5dadd
diff --git a/pyproject.toml b/pyproject.tomlindex d9a3e6dc..7b145bb7 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -104,9 +104,9 @@ markers = ["provider: live provider checks","messaging: live messaging platform checks","cli: CLI integration checks",- "vscode: VS Code or IDE client compatibility checks",+ "clients: client compatibility checks","voice: voice transcription checks",- "contract: deterministic smoke contract checks",+ "contract: deterministic feature contract checks","smoke_target(name): route a smoke test behind FCC_SMOKE_TARGETS",]diff --git a/smoke/README.md b/smoke/README.mdindex 894dfdfc..fedad56f 100644--- a/smoke/README.md+++ b/smoke/README.md@@ -1,7 +1,7 @@# Local Live Smoke Tests-These tests are for maintainers running against their own `.env`. They are not-part of CI and are not collected by plain `uv run pytest`.+`smoke/` is for live/process/service checks only. Hermetic feature contracts+belong under `tests/` and run with plain `uv run pytest`.## Safe Default Run@@ -12,23 +12,54 @@ uv run pytest smoke -n 0 -m live -s --tb=short`-n 0` is recommended because the normal project pytest config enables xdist.The smoke suite can run with workers, but one process gives clearer logs when a-real provider or bot fails.+real provider, bot, or local model server fails.++## Targets++Default targets do not require bot tokens or voice backends:++| Target | What it checks | Required environment |+| --- | --- | --- |+| `api` | live uvicorn routes, probes, token count, optimizations, `/stop` fallback | none |+| `auth` | Anthropic auth header variants on a live server | none; test sets an isolated token |+| `cli` | `fcc-init`, `free-claude-code`, optional Claude CLI prompt | `claude` and provider config only for the prompt check |+| `clients` | VS Code and JetBrains-shaped requests | none |+| `providers` | configured provider model streams | provider keys or local provider URLs |+| `tools` | configured model emits live tool use | tool-capable configured provider |+| `rate_limit` | client disconnect does not crash a provider stream | configured provider |+| `lmstudio` | LM Studio `/models` endpoint is reachable | running LM Studio server |+| `llamacpp` | llama.cpp `/models` endpoint is reachable | running `llama-server` |++Side-effectful targets are opt-in:++| Target | What it checks | Required environment |+| --- | --- | --- |+| `telegram` | bot API getMe, send, edit, delete | `TELEGRAM_BOT_TOKEN` and chat/user ID |+| `discord` | channel access, send, edit, delete | `DISCORD_BOT_TOKEN` and channel ID |+| `voice` | configured transcription backend | `VOICE_NOTE_ENABLED=true` and `FCC_SMOKE_RUN_VOICE=1` |++Use `FCC_SMOKE_TARGETS=all` to include Telegram, Discord, and voice checks.## Targeted Runs```powershell$env:FCC_LIVE_SMOKE = "1"-$env:FCC_SMOKE_TARGETS = "api,providers,thinking,tools"+$env:FCC_SMOKE_TARGETS = "api,providers,tools"uv run pytest smoke -n 0 -m live -s --tb=short```-Use `FCC_SMOKE_TARGETS=all` to include Telegram, Discord, and voice checks.-The default target set intentionally excludes those side-effectful integrations.+```powershell+$env:FCC_LIVE_SMOKE = "1"+$env:FCC_SMOKE_TARGETS = "telegram,discord,voice"+$env:FCC_SMOKE_RUN_VOICE = "1"+uv run pytest smoke -n 0 -m live -s --tb=short+```## Environment- `FCC_ENV_FILE`: optional explicit dotenv path. The app still uses its normalenv-file precedence.+- `FCC_SMOKE_TARGETS`: comma-separated targets, or `all`.- `FCC_SMOKE_PROVIDER_MATRIX`: comma-separated provider prefixes to test.- `FCC_SMOKE_TIMEOUT_S`: per-request/subprocess timeout, default `45`.- `FCC_SMOKE_CLAUDE_BIN`: Claude CLI executable name, default `claude`.@@ -37,18 +68,24 @@ The default target set intentionally excludes those side-effectful integrations.- `FCC_SMOKE_INTERACTIVE=1`: enables manual inbound messaging checks.- `FCC_SMOKE_RUN_VOICE=1`: allows the voice transcription backend to load/run.-## Results+## Feature Inventory++The public feature inventory lives in `smoke/features.py`. Each entry records+whether the feature is covered by hermetic pytest, live smoke, or both, plus the+owning tests and skip policy. The manifest intentionally excludes `claude-pick`.++## Results And Failure ClassesSmoke artifacts are written to `.smoke-results/` and ignored by git. Reports andlogs redact env values whose names contain `KEY`, `TOKEN`, `SECRET`, `WEBHOOK`,or `AUTH`.-## How To Read Failures+Report classifications:-- `missing_env`: configure the required key, token, channel, or local base URL.-- `upstream_unavailable`: the provider/local model/bot API is not reachable.-- `product_failure`: the app returned the wrong shape or crashed.-- `harness_bug`: the smoke test itself made an invalid assumption.+- `missing_env`: a credential, opt-in flag, binary, channel, or model config is absent.+- `upstream_unavailable`: a real provider, bot API, or local model server is not reachable.+- `product_failure`: the app returned the wrong shape, crashed, or violated a contract.+- `harness_bug`: the smoke test made an invalid assumption.-The first real run is expected to find product failures. Fix those separately-from harness problems so the suite becomes a reliable regression signal.+`missing_env` and `upstream_unavailable` should skip. `product_failure` and+`harness_bug` should be fixed before relying on the smoke baseline.diff --git a/smoke/conftest.py b/smoke/conftest.pyindex 7ebb91eb..a10e413e 100644--- a/smoke/conftest.py+++ b/smoke/conftest.py@@ -34,7 +34,9 @@ def pytest_runtest_setup(item: pytest.Item) -> None:def pytest_runtest_logreport(report: pytest.TestReport) -> None:- if report.when != "call":+ if report.when == "setup" and not report.skipped:+ return+ if report.when == "teardown" and not report.failed:returnif _REPORT is None:returndiff --git a/smoke/features.py b/smoke/features.pyindex 3f7e06e2..7e11552e 100644--- a/smoke/features.py+++ b/smoke/features.py@@ -1,20 +1,38 @@-"""Feature inventory used by the local smoke suite.+"""Public feature inventory for pytest contracts and live smoke checks.-This file is intentionally explicit: advertised features should not exist only-in README prose without at least one smoke check or a documented manual gap.+The inventory is intentionally explicit. README features and exposed public+surface area should have either hermetic pytest coverage, live smoke coverage,+or both. Private helpers are not listed here."""from __future__ import annotationsfrom dataclasses import dataclass+from typing import Literal++FeatureSource = Literal["readme", "public_surface"]+CoverageKind = Literal["pytest", "live_smoke", "both"]@dataclass(frozen=True, slots=True)-class FeatureSmoke:+class FeatureCoverage:feature_id: strtitle: str- mode: str- checks: tuple[str, ...]+ source: FeatureSource+ coverage: CoverageKind+ pytest_tests: tuple[str, ...]+ smoke_tests: tuple[str, ...]+ smoke_targets: tuple[str, ...]+ required_env: tuple[str, ...]+ skip_policy: str++ @property+ def has_pytest_coverage(self) -> bool:+ return self.coverage in {"pytest", "both"}++ @property+ def has_smoke_coverage(self) -> bool:+ return self.coverage in {"live_smoke", "both"}README_FEATURES: tuple[str, ...] = (@@ -36,112 +54,408 @@ README_FEATURES: tuple[str, ...] = ()-FEATURE_SMOKES: tuple[FeatureSmoke, ...] = (- FeatureSmoke(+FEATURE_INVENTORY: tuple[FeatureCoverage, ...] = (+ FeatureCoverage("zero_cost_provider_access","Configured provider accepts a real prompt",- "live",+ "readme",+ "both",+ ("tests/api/test_dependencies.py", "tests/providers/test_open_router.py"),("test_configured_provider_models_stream_successfully",),+ ("providers",),+ ("NVIDIA_NIM_API_KEY|OPENROUTER_API_KEY|DEEPSEEK_API_KEY|local provider",),+ "skip when no usable provider credentials or local provider endpoint exists",),- FeatureSmoke(+ FeatureCoverage("drop_in_claude_code_replacement","Claude-compatible routes and CLI environment work",- "live",+ "readme",+ "both",+ ("tests/api/test_api.py", "tests/cli/test_cli.py"),("test_probe_and_models_routes","test_claude_cli_prompt_when_available","test_vscode_and_jetbrains_shaped_requests",),+ ("api", "cli", "clients"),+ ("FCC_SMOKE_CLAUDE_BIN", "configured provider for Claude CLI"),+ "skip Claude CLI path when binary or provider config is unavailable",),- FeatureSmoke(+ FeatureCoverage("provider_matrix","All configured provider prefixes can be exercised",- "live",+ "readme",+ "both",+ ("tests/api/test_dependencies.py", "tests/providers/"),("test_configured_provider_models_stream_successfully",),+ ("providers",),+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),+ "skip when no configured provider has usable credentials/base URL",),- FeatureSmoke(+ FeatureCoverage("per_model_mapping","Opus, Sonnet, Haiku, and fallback mappings are visible",- "live",+ "readme",+ "both",+ ("tests/api/test_models_validators.py", "tests/config/test_config.py"),("test_model_mapping_configuration_is_consistent",),+ ("providers",),+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),+ "skip live provider execution when no provider config is usable",),- FeatureSmoke(+ FeatureCoverage(+ "mixed_provider_mapping",+ "Model-specific overrides can route to different providers",+ "public_surface",+ "both",+ ("tests/config/test_config.py",),+ ("test_mixed_provider_model_mapping_when_configured",),+ ("providers",),+ ("MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"),+ "skip when fewer than two configured provider prefixes are present",+ ),+ FeatureCoverage("thinking_token_support","Thinking streams and suppression are contract-tested",- "contract",+ "readme",+ "pytest",("test_interleaved_thinking_text_blocks_are_valid","test_split_think_tags_preserve_text_and_thinking","test_enable_thinking_false_suppresses_reasoning_only",),+ (),+ (),+ (),+ "hermetic contract coverage",),- FeatureSmoke(+ FeatureCoverage("heuristic_tool_parser","Text tool calls become structured tool_use blocks",- "contract",- ("test_thinking_tool_text_and_transcript_order_contract",),+ "readme",+ "both",+ (+ "tests/providers/test_parsers.py",+ "test_thinking_tool_text_and_transcript_order_contract",+ ),+ ("test_live_tool_use_when_configured_model_supports_tools",),+ ("tools",),+ ("configured tool-capable provider",),+ "skip live tool-use when no configured provider model is available",),- FeatureSmoke(+ FeatureCoverage("request_optimization","Fast-path local optimizations respond without providers",- "live",+ "readme",+ "both",+ (+ "tests/api/test_optimization_handlers.py",+ "tests/api/test_routes_optimizations.py",+ ),("test_optimization_fast_paths_do_not_need_provider",),+ ("api",),+ (),+ "always runnable once live server starts",),- FeatureSmoke(+ FeatureCoverage("smart_rate_limiting",- "Concurrency/disconnect and retry-sensitive paths are checked",- "live",+ "Provider limiter covers proactive throttling, 429 retry, and disconnects",+ "readme",+ "both",+ ("tests/providers/test_provider_rate_limit.py",),("test_client_disconnect_mid_stream_does_not_crash_server",),+ ("rate_limit", "providers"),+ ("configured provider",),+ "skip live disconnect path when no configured provider model is available",),- FeatureSmoke(+ FeatureCoverage("discord_telegram_bot","Messaging credentials, send/edit/delete, and transcript behavior",- "live_or_interactive",+ "readme",+ "both",+ (+ "tests/messaging/test_discord_platform.py",+ "tests/messaging/test_telegram.py",+ ),("test_telegram_bot_api_permissions","test_discord_bot_api_permissions",- "test_thinking_tool_text_and_transcript_order_contract",+ "test_interactive_inbound_messaging_requires_explicit_mode",+ ),+ ("telegram", "discord"),+ (+ "TELEGRAM_BOT_TOKEN",+ "ALLOWED_TELEGRAM_USER_ID|FCC_SMOKE_TELEGRAM_CHAT_ID",+ "DISCORD_BOT_TOKEN",+ "ALLOWED_DISCORD_CHANNELS|FCC_SMOKE_DISCORD_CHANNEL_ID",),+ "skip when bot tokens/channels are absent; inbound checks require interactive mode",),- FeatureSmoke(+ FeatureCoverage("subagent_control","Task tool calls do not run in the background",- "contract",- ("test_task_tool_arguments_force_foreground_execution",),+ "readme",+ "pytest",+ (+ "tests/providers/test_subagent_interception.py",+ "test_thinking_tool_text_and_transcript_order_contract",+ ),+ (),+ (),+ (),+ "hermetic contract coverage",),- FeatureSmoke(+ FeatureCoverage("extensible_provider_platform_abcs","Provider/platform registries expose expected built-ins",- "contract",+ "readme",+ "pytest",("test_provider_and_platform_registries_include_advertised_builtins",),+ (),+ (),+ (),+ "hermetic contract coverage",),- FeatureSmoke(+ FeatureCoverage("optional_authentication","Anthropic-style auth headers are accepted and enforced",- "live",+ "readme",+ "both",+ ("tests/api/test_auth.py",),("test_auth_token_is_enforced_for_all_supported_header_shapes",),+ ("auth",),+ ("ANTHROPIC_AUTH_TOKEN",),+ "live test starts an isolated server with its own token",),- FeatureSmoke(+ FeatureCoverage("vscode_extension","VS Code-shaped beta requests work against the proxy",- "live",+ "readme",+ "live_smoke",+ (),("test_vscode_and_jetbrains_shaped_requests",),+ ("clients",),+ (),+ "always runnable once live server starts",),- FeatureSmoke(+ FeatureCoverage("intellij_extension","JetBrains/ACP-shaped environment requests work against the proxy",- "live",+ "readme",+ "live_smoke",+ (),("test_vscode_and_jetbrains_shaped_requests",),+ ("clients",),+ (),+ "always runnable once live server starts",),- FeatureSmoke(+ FeatureCoverage("voice_notes","Configured transcription backend can process an audio fixture",- "live_or_skip",+ "readme",+ "both",+ (+ "tests/messaging/test_voice_handlers.py",+ "tests/messaging/test_transcription.py",+ ),("test_voice_transcription_backend_when_explicitly_enabled",),+ ("voice",),+ ("VOICE_NOTE_ENABLED", "FCC_SMOKE_RUN_VOICE", "WHISPER_DEVICE"),+ "skip unless voice is explicitly enabled for local smoke",+ ),+ FeatureCoverage(+ "anthropic_api_routes",+ "Root, health, models, messages, count_tokens, and stop routes respond",+ "public_surface",+ "both",+ ("tests/api/test_api.py",),+ ("test_probe_and_models_routes", "test_stop_endpoint_reports_no_messaging"),+ ("api",),+ (),+ "always runnable once live server starts",+ ),+ FeatureCoverage(+ "probe_routes",+ "HEAD and OPTIONS compatibility probes return 204 and Allow headers",+ "public_surface",+ "both",+ ("tests/api/test_api.py::test_probe_endpoints_return_204_with_allow_headers",),+ ("test_probe_and_models_routes",),+ ("api",),+ (),+ "always runnable once live server starts",+ ),+ FeatureCoverage(+ "count_tokens_contract",+ "Token counting accepts text, thinking, tools, tool results, and images",+ "public_surface",+ "both",+ ("tests/api/test_request_utils.py",),+ ("test_count_tokens_accepts_thinking_tools_and_results",),+ ("api",),+ (),+ "always runnable once live server starts",+ ),+ FeatureCoverage(+ "provider_proxy_timeout_config",+ "Provider proxies and HTTP timeouts are passed into clients",+ "public_surface",+ "pytest",+ ("tests/api/test_dependencies.py",),+ (),+ (),+ (),+ "hermetic config coverage",+ ),+ FeatureCoverage(+ "lmstudio_endpoint",+ "Configured LM Studio endpoint exposes an OpenAI-compatible models route",+ "public_surface",+ "both",+ ("tests/providers/test_lmstudio.py",),+ ("test_lmstudio_models_endpoint_when_available",),+ ("lmstudio",),+ ("LM_STUDIO_BASE_URL",),+ "skip when the local LM Studio server is not reachable",+ ),+ FeatureCoverage(+ "llamacpp_endpoint",+ "Configured llama.cpp endpoint exposes an OpenAI-compatible models route",+ "public_surface",+ "both",+ ("tests/providers/test_llamacpp.py",),+ ("test_llamacpp_models_endpoint_when_available",),+ ("llamacpp",),+ ("LLAMACPP_BASE_URL",),+ "skip when the local llama.cpp server is not reachable",+ ),+ FeatureCoverage(+ "package_cli_entrypoints",+ "Installed package scripts scaffold config and start the server",+ "public_surface",+ "both",+ ("tests/cli/test_entrypoints.py",),+ (+ "test_fcc_init_scaffolds_user_config",+ "test_free_claude_code_entrypoint_starts_server",+ ),+ ("cli",),+ (),+ "always runnable once uv project dependencies are available",+ ),+ FeatureCoverage(+ "claude_cli_drop_in",+ "Claude CLI can send a prompt through the proxy when installed",+ "public_surface",+ "live_smoke",+ (),+ ("test_claude_cli_prompt_when_available",),+ ("cli",),+ ("FCC_SMOKE_CLAUDE_BIN", "configured provider"),+ "skip when Claude CLI or provider config is unavailable",+ ),+ FeatureCoverage(+ "messaging_commands",+ "Messaging /stop, /clear, and /stats commands use platform queues safely",+ "public_surface",+ "pytest",+ (+ "tests/messaging/test_handler.py",+ "tests/messaging/test_handler_integration.py",+ ),+ (),+ (),+ (),+ "hermetic messaging handler coverage",+ ),+ FeatureCoverage(+ "tree_threading",+ "Reply-based message trees support queueing, branching, and cancellation",+ "public_surface",+ "pytest",+ (+ "tests/messaging/test_tree_queue.py",+ "tests/messaging/test_tree_concurrency.py",+ ),+ (),+ (),+ (),+ "hermetic tree queue coverage",+ ),+ FeatureCoverage(+ "restart_restore",+ "Persisted tree state is restored after restart for reply routing",+ "public_surface",+ "pytest",+ ("tests/messaging/test_restart_reply_restore.py",),+ (),+ (),+ (),+ "hermetic persistence coverage",+ ),+ FeatureCoverage(+ "session_persistence",+ "SessionStore persists trees, node mappings, and message logs",+ "public_surface",+ "pytest",+ ("tests/messaging/test_session_store_edge_cases.py",),+ (),+ (),+ (),+ "hermetic persistence coverage",+ ),+ FeatureCoverage(+ "config_env_precedence",+ "Environment and dotenv precedence are deterministic",+ "public_surface",+ "pytest",+ ("tests/config/test_config.py",),+ (),+ (),+ (),+ "hermetic config coverage",+ ),+ FeatureCoverage(+ "removed_env_migration",+ "Removed NIM_ENABLE_THINKING env var fails fast with migration guidance",+ "public_surface",+ "pytest",+ ("tests/config/test_config.py",),+ (),+ (),+ (),+ "hermetic config coverage",+ ),+ FeatureCoverage(+ "streaming_error_mapping",+ "Provider and streaming errors map to Anthropic-compatible error shapes",+ "public_surface",+ "pytest",+ (+ "tests/providers/test_streaming_errors.py",+ "tests/providers/test_error_mapping.py",+ ),+ (),+ (),+ (),+ "hermetic provider coverage",),)-def smoke_ids() -> set[str]:- """Return feature IDs covered by the smoke manifest."""- return {feature.feature_id for feature in FEATURE_SMOKES}+def feature_ids(*, source: FeatureSource | None = None) -> set[str]:+ """Return feature IDs covered by the inventory."""+ return {+ feature.feature_id+ for feature in FEATURE_INVENTORY+ if source is None or feature.source == source+ }+++def smoke_feature_ids() -> set[str]:+ """Return feature IDs with at least one live smoke owner."""+ return {+ feature.feature_id+ for feature in FEATURE_INVENTORY+ if feature.has_smoke_coverage+ }diff --git a/smoke/lib/config.py b/smoke/lib/config.pyindex 73da08f2..3862b651 100644--- a/smoke/lib/config.py+++ b/smoke/lib/config.py@@ -14,18 +14,47 @@ DEFAULT_TARGETS = frozenset("api","auth","cli",- "contract",- "optimizations",+ "clients",+ "llamacpp",+ "lmstudio","providers",- "thinking",+ "rate_limit","tools",- "vscode",})-ALL_TARGETS = DEFAULT_TARGETS | frozenset({"discord", "telegram", "voice"})+SIDE_EFFECT_TARGETS = frozenset({"discord", "telegram", "voice"})+ALL_TARGETS = DEFAULT_TARGETS | SIDE_EFFECT_TARGETS+TARGET_ALIASES = {+ "contract": "api",+ "optimizations": "api",+ "thinking": "providers",+ "vscode": "clients",+}SECRET_KEY_PARTS = ("KEY", "TOKEN", "SECRET", "WEBHOOK", "AUTH")+TARGET_REQUIRED_ENV: dict[str, tuple[str, ...]] = {+ "api": (),+ "auth": (),+ "cli": ("FCC_SMOKE_CLAUDE_BIN", "configured provider for Claude CLI prompt"),+ "clients": (),+ "providers": ("MODEL or MODEL_* with usable provider configuration",),+ "rate_limit": ("configured provider model",),+ "tools": ("configured tool-capable provider model",),+ "lmstudio": ("LM_STUDIO_BASE_URL with a running LM Studio server",),+ "llamacpp": ("LLAMACPP_BASE_URL with a running llama-server",),+ "telegram": (+ "TELEGRAM_BOT_TOKEN",+ "ALLOWED_TELEGRAM_USER_ID or FCC_SMOKE_TELEGRAM_CHAT_ID",+ ),+ "discord": (+ "DISCORD_BOT_TOKEN",+ "ALLOWED_DISCORD_CHANNELS or FCC_SMOKE_DISCORD_CHANNEL_ID",+ ),+ "voice": ("VOICE_NOTE_ENABLED=true", "FCC_SMOKE_RUN_VOICE=1"),+}++@dataclass(frozen=True, slots=True)class ProviderModel:provider: str@@ -122,7 +151,7 @@ def _parse_targets(raw: str | None) -> frozenset[str]:parsed = _parse_csv(raw)if "all" in parsed:return ALL_TARGETS- return parsed+ return frozenset(TARGET_ALIASES.get(target, target) for target in parsed)def auth_headers(token: str | None = None) -> dict[str, str]:diff --git a/smoke/lib/report.py b/smoke/lib/report.pyindex 7a54597c..448b1d44 100644--- a/smoke/lib/report.py+++ b/smoke/lib/report.py@@ -13,6 +13,7 @@ from .config import SmokeConfig, redactedclass SmokeOutcome:nodeid: stroutcome: str+ classification: strduration_s: floatmarkers: list[str]detail: str@@ -37,6 +38,9 @@ class SmokeReport:SmokeOutcome(nodeid=nodeid,outcome=outcome,+ classification=classify_outcome(+ nodeid=nodeid, outcome=outcome, detail=detail+ ),duration_s=duration_s,markers=markers,detail=redacted(detail),@@ -56,3 +60,42 @@ class SmokeReport:"outcomes": [asdict(outcome) for outcome in self.outcomes],}path.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8")+++def classify_outcome(*, nodeid: str, outcome: str, detail: str) -> str:+ """Classify smoke outcomes for triage reports."""+ if outcome == "passed":+ return "passed"++ text = f"{nodeid}\n{detail}".lower()+ if outcome == "skipped":+ if any(+ marker in text+ for marker in (+ "upstream_unavailable",+ "connection refused",+ "connecterror",+ "readtimeout",+ "timed out",+ "not reachable",+ )+ ):+ return "upstream_unavailable"+ return "missing_env"++ if "harness_bug" in text:+ return "harness_bug"+ if any(+ marker in text+ for marker in (+ "upstream_unavailable",+ "connection refused",+ "connecterror",+ "readtimeout",+ "timed out",+ "not reachable",+ "upstream provider",+ )+ ):+ return "upstream_unavailable"+ return "product_failure"diff --git a/smoke/lib/skips.py b/smoke/lib/skips.pynew file mode 100644index 00000000..8ce03edd--- /dev/null+++ b/smoke/lib/skips.py@@ -0,0 +1,61 @@+"""Skip helpers for expected live-smoke environment gaps."""++from __future__ import annotations++import httpx+import pytest++from smoke.lib.sse import SSEEvent++UPSTREAM_UNAVAILABLE_MARKERS = (+ "connection refused",+ "connecterror",+ "connect timeout",+ "readtimeout",+ "server disconnected",+ "service unavailable",+ "temporary failure",+ "timed out",+ "upstream provider",+)+++def is_upstream_unavailable_text(text: str) -> bool:+ normalized = text.lower()+ return any(marker in normalized for marker in UPSTREAM_UNAVAILABLE_MARKERS)+++def skip_upstream_unavailable(reason: str) -> None:+ pytest.skip(f"upstream_unavailable: {reason}")+++def skip_if_upstream_unavailable_exception(exc: Exception) -> None:+ if isinstance(+ exc,+ (+ httpx.ConnectError,+ httpx.ConnectTimeout,+ httpx.ReadTimeout,+ httpx.RemoteProtocolError,+ httpx.ProxyError,+ ),+ ):+ skip_upstream_unavailable(f"{type(exc).__name__}: {exc}")+ if is_upstream_unavailable_text(f"{type(exc).__name__}: {exc}"):+ skip_upstream_unavailable(f"{type(exc).__name__}: {exc}")+++def skip_if_upstream_unavailable_events(events: list[SSEEvent]) -> None:+ for event in events:+ if getattr(event, "event", None) != "error":+ continue+ data = getattr(event, "data", {})+ message = ""+ if isinstance(data, dict):+ error = data.get("error")+ if isinstance(error, dict):+ message = str(error.get("message", ""))+ else:+ message = str(data)+ if is_upstream_unavailable_text(message):+ skip_upstream_unavailable(message[:500])diff --git a/smoke/test_api_live.py b/smoke/test_api_live.pyindex ffffa204..7f488fca 100644--- a/smoke/test_api_live.py+++ b/smoke/test_api_live.py@@ -173,3 +173,15 @@ def test_invalid_messages_returns_anthropic_error(payload = response.json()assert payload["type"] == "error"assert payload["error"]["type"] == "invalid_request_error"+++def test_stop_endpoint_reports_no_messaging(+ smoke_server: RunningServer, smoke_headers: dict[str, str]+) -> None:+ response = httpx.post(+ f"{smoke_server.base_url}/stop",+ headers=smoke_headers,+ timeout=5,+ )+ assert response.status_code == 503+ assert response.json()["detail"] == "Messaging system not initialized"diff --git a/smoke/test_auth_live.py b/smoke/test_auth_live.pyindex c6390b98..0062f90f 100644--- a/smoke/test_auth_live.py+++ b/smoke/test_auth_live.py@@ -1,5 +1,7 @@from __future__ import annotations+from pathlib import Path+import httpximport pytest@@ -10,12 +12,19 @@ pytestmark = [pytest.mark.live, pytest.mark.smoke_target("auth")]def test_auth_token_is_enforced_for_all_supported_header_shapes(- smoke_config: SmokeConfig,+ smoke_config: SmokeConfig, tmp_path: Path) -> None:token = "fcc-smoke-token"+ env_file = tmp_path / "auth.env"+ env_file.write_text(f'ANTHROPIC_AUTH_TOKEN="{token}"\n', encoding="utf-8")+with start_server(smoke_config,- env_overrides={"ANTHROPIC_AUTH_TOKEN": token, "MESSAGING_PLATFORM": "none"},+ env_overrides={+ "ANTHROPIC_AUTH_TOKEN": token,+ "FCC_ENV_FILE": str(env_file),+ "MESSAGING_PLATFORM": "none",+ },name="auth",) as server:assert httpx.get(f"{server.base_url}/").status_code == 401diff --git a/smoke/test_client_shapes_live.py b/smoke/test_client_shapes_live.pyindex 47d3b924..47a94592 100644--- a/smoke/test_client_shapes_live.py+++ b/smoke/test_client_shapes_live.py@@ -6,7 +6,7 @@ from smoke.lib.config import SmokeConfig, auth_headersfrom smoke.lib.http import message_payload, post_jsonfrom smoke.lib.server import RunningServer-pytestmark = [pytest.mark.live, pytest.mark.smoke_target("vscode")]+pytestmark = [pytest.mark.live, pytest.mark.smoke_target("clients")]def test_vscode_and_jetbrains_shaped_requests(diff --git a/smoke/test_feature_manifest.py b/smoke/test_feature_manifest.pydeleted file mode 100644index a72b651b..00000000--- a/smoke/test_feature_manifest.py+++ /dev/null@@ -1,48 +0,0 @@-from __future__ import annotations--import pytest--from messaging.platforms.factory import create_messaging_platform-from providers.base import BaseProvider-from providers.deepseek import DeepSeekProvider-from providers.llamacpp import LlamaCppProvider-from providers.lmstudio import LMStudioProvider-from providers.nvidia_nim import NvidiaNimProvider-from providers.open_router import OpenRouterProvider-from smoke.features import FEATURE_SMOKES, README_FEATURES, smoke_ids--pytestmark = [pytest.mark.live, pytest.mark.smoke_target("contract")]---def test_every_advertised_feature_has_a_smoke_entry() -> None:- missing = sorted(set(README_FEATURES) - smoke_ids())- extra = sorted(smoke_ids() - set(README_FEATURES))- assert not missing, f"README features missing smoke entries: {missing}"- assert not extra, f"smoke entries not tied to README features: {extra}"---def test_smoke_manifest_has_unique_ids_and_checks() -> None:- ids = [feature.feature_id for feature in FEATURE_SMOKES]- assert len(ids) == len(set(ids))- for feature in FEATURE_SMOKES:- assert feature.checks, feature- assert feature.mode in {- "live",- "contract",- "live_or_interactive",- "live_or_skip",- }---def test_provider_and_platform_registries_include_advertised_builtins() -> None:- provider_classes = {- "nvidia_nim": NvidiaNimProvider,- "open_router": OpenRouterProvider,- "deepseek": DeepSeekProvider,- "lmstudio": LMStudioProvider,- "llamacpp": LlamaCppProvider,- }- for provider_class in provider_classes.values():- assert issubclass(provider_class, BaseProvider)-- assert create_messaging_platform("not-a-platform") is Nonediff --git a/smoke/test_local_provider_endpoints_live.py b/smoke/test_local_provider_endpoints_live.pynew file mode 100644index 00000000..ab9652ed--- /dev/null+++ b/smoke/test_local_provider_endpoints_live.py@@ -0,0 +1,48 @@+from __future__ import annotations++import httpx+import pytest++from smoke.lib.config import SmokeConfig+from smoke.lib.skips import skip_if_upstream_unavailable_exception++[email protected]_target("lmstudio")+def test_lmstudio_models_endpoint_when_available(smoke_config: SmokeConfig) -> None:+ _assert_models_endpoint(+ smoke_config.settings.lm_studio_base_url,+ timeout_s=smoke_config.timeout_s,+ provider_name="LM Studio",+ )++[email protected]_target("llamacpp")+def test_llamacpp_models_endpoint_when_available(smoke_config: SmokeConfig) -> None:+ _assert_models_endpoint(+ smoke_config.settings.llamacpp_base_url,+ timeout_s=smoke_config.timeout_s,+ provider_name="llama.cpp",+ )+++def _assert_models_endpoint(+ base_url: str, *, timeout_s: float, provider_name: str+) -> None:+ url = f"{base_url.rstrip('/')}/models"+ try:+ response = httpx.get(url, timeout=timeout_s)+ except Exception as exc:+ skip_if_upstream_unavailable_exception(exc)+ raise++ if response.status_code in {404, 405, 502, 503}:+ pytest.skip(+ f"upstream_unavailable: {provider_name} models endpoint "+ f"{url} returned HTTP {response.status_code}"+ )++ assert response.status_code == 200, response.text+ payload = response.json()+ assert isinstance(payload.get("data"), list), payloaddiff --git a/smoke/test_provider_live.py b/smoke/test_provider_live.pyindex c139e0a8..8731278a 100644--- a/smoke/test_provider_live.py+++ b/smoke/test_provider_live.py@@ -8,6 +8,10 @@ import pytestfrom smoke.lib.config import SmokeConfig, auth_headersfrom smoke.lib.http import collect_message_stream, message_payloadfrom smoke.lib.server import start_server+from smoke.lib.skips import (+ skip_if_upstream_unavailable_events,+ skip_if_upstream_unavailable_exception,+)from smoke.lib.sse import assert_anthropic_stream_contract, text_contentpytestmark = [pytest.mark.live, pytest.mark.smoke_target("providers")]@@ -22,6 +26,19 @@ def test_model_mapping_configuration_is_consistent(smoke_config: SmokeConfig) ->assert provider_model.model_name+def test_mixed_provider_model_mapping_when_configured(+ smoke_config: SmokeConfig,+) -> None:+ models = smoke_config.provider_models()+ providers = {provider_model.provider for provider_model in models}+ if len(providers) < 2:+ pytest.skip("configure MODEL_* with at least two provider prefixes")++ sources = {provider_model.source for provider_model in models}+ assert sources <= {"MODEL", "MODEL_OPUS", "MODEL_SONNET", "MODEL_HAIKU"}+ assert len(providers) >= 2++def test_configured_provider_models_stream_successfully(smoke_config: SmokeConfig,) -> None:@@ -45,9 +62,11 @@ def test_configured_provider_models_stream_successfully(message_payload(smoke_config.prompt, model="fcc-smoke-default"),smoke_config,)+ skip_if_upstream_unavailable_events(events)assert_anthropic_stream_contract(events)assert text_content(events).strip(), "provider returned no text"except Exception as exc:+ skip_if_upstream_unavailable_exception(exc)failures.append(f"{provider_model.source}={provider_model.full_model}: "f"{type(exc).__name__}: {exc}"@@ -56,6 +75,7 @@ def test_configured_provider_models_stream_successfully(assert not failures, "\n".join(failures)[email protected]_target("rate_limit")def test_client_disconnect_mid_stream_does_not_crash_server(smoke_config: SmokeConfig,) -> None:diff --git a/smoke/test_tools_live.py b/smoke/test_tools_live.pyindex b4824109..824197be 100644--- a/smoke/test_tools_live.py+++ b/smoke/test_tools_live.py@@ -5,6 +5,7 @@ import pytestfrom smoke.lib.config import SmokeConfigfrom smoke.lib.http import collect_message_stream, message_payloadfrom smoke.lib.server import start_server+from smoke.lib.skips import skip_if_upstream_unavailable_eventsfrom smoke.lib.sse import assert_anthropic_stream_contract, has_tool_usepytestmark = [pytest.mark.live, pytest.mark.smoke_target("tools")]@@ -47,5 +48,6 @@ def test_live_tool_use_when_configured_model_supports_tools(name="tools",) as server:events = collect_message_stream(server, payload, smoke_config)+ skip_if_upstream_unavailable_events(events)assert_anthropic_stream_contract(events)assert has_tool_use(events), "model did not emit a tool_use block"diff --git a/tests/contracts/test_feature_manifest.py b/tests/contracts/test_feature_manifest.pynew file mode 100644index 00000000..71d3deee--- /dev/null+++ b/tests/contracts/test_feature_manifest.py@@ -0,0 +1,88 @@+from __future__ import annotations++import re+from pathlib import Path++from messaging.platforms.factory import create_messaging_platform+from providers.base import BaseProvider+from providers.deepseek import DeepSeekProvider+from providers.llamacpp import LlamaCppProvider+from providers.lmstudio import LMStudioProvider+from providers.nvidia_nim import NvidiaNimProvider+from providers.open_router import OpenRouterProvider+from smoke.features import FEATURE_INVENTORY, README_FEATURES, feature_ids++VALID_COVERAGE = {"pytest", "live_smoke", "both"}+VALID_SOURCE = {"readme", "public_surface"}+++def test_every_readme_feature_has_inventory_entry() -> None:+ missing = sorted(set(README_FEATURES) - feature_ids(source="readme"))+ extra_readme = sorted(feature_ids(source="readme") - set(README_FEATURES))+ assert not missing, f"README features missing inventory entries: {missing}"+ assert not extra_readme, (+ f"README inventory entries not in README_FEATURES: {extra_readme}"+ )+++def test_feature_inventory_is_unique_and_decision_complete() -> None:+ ids = [feature.feature_id for feature in FEATURE_INVENTORY]+ assert len(ids) == len(set(ids))+ assert "claude_pick" not in ids++ for feature in FEATURE_INVENTORY:+ assert feature.source in VALID_SOURCE, feature+ assert feature.coverage in VALID_COVERAGE, feature+ assert feature.title.strip(), feature+ assert feature.skip_policy.strip(), feature+ if feature.has_pytest_coverage:+ assert feature.pytest_tests, feature+ if feature.has_smoke_coverage:+ assert feature.smoke_tests, feature+ assert feature.smoke_targets, feature+++def test_feature_inventory_test_owners_exist() -> None:+ repo_root = Path(__file__).resolve().parents[2]+ pytest_names = _collect_test_names(repo_root / "tests")+ smoke_names = _collect_test_names(repo_root / "smoke")++ for feature in FEATURE_INVENTORY:+ for owner in feature.pytest_tests:+ _assert_owner_exists(owner, repo_root, pytest_names)+ for owner in feature.smoke_tests:+ assert owner in smoke_names or owner in pytest_names, (feature, owner)+++def test_provider_and_platform_registries_include_advertised_builtins() -> None:+ provider_classes = {+ "nvidia_nim": NvidiaNimProvider,+ "open_router": OpenRouterProvider,+ "deepseek": DeepSeekProvider,+ "lmstudio": LMStudioProvider,+ "llamacpp": LlamaCppProvider,+ }+ for provider_class in provider_classes.values():+ assert issubclass(provider_class, BaseProvider)++ assert create_messaging_platform("not-a-platform") is None+++def _collect_test_names(root: Path) -> set[str]:+ names: set[str] = set()+ for path in root.rglob("test_*.py"):+ text = path.read_text(encoding="utf-8")+ names.update(re.findall(r"^\s*(?:async\s+)?def (test_[^(]+)", text, re.M))+ return names+++def _assert_owner_exists(owner: str, repo_root: Path, test_names: set[str]) -> None:+ if owner.startswith("test_"):+ assert owner in test_names, owner+ return++ path_part, _, node_name = owner.partition("::")+ path = repo_root / path_part+ assert path.exists(), owner+ if node_name:+ assert node_name in test_names, ownerdiff --git a/smoke/test_stream_contracts.py b/tests/contracts/test_stream_contracts.pysimilarity index 94%rename from smoke/test_stream_contracts.pyrename to tests/contracts/test_stream_contracts.pyindex 7f4ee6e0..cff9eb33 100644--- a/smoke/test_stream_contracts.py+++ b/tests/contracts/test_stream_contracts.py@@ -2,8 +2,6 @@ from __future__ import annotationsfrom collections.abc import Iterable-import pytest-from messaging.event_parser import parse_cli_eventfrom messaging.transcript import RenderCtx, TranscriptBufferfrom providers.common import (@@ -21,12 +19,6 @@ from smoke.lib.sse import (thinking_content,)-pytestmark = [- pytest.mark.live,- pytest.mark.smoke_target("contract"),- pytest.mark.smoke_target("thinking"),-]-def test_interleaved_thinking_text_blocks_are_valid() -> None:events = _parse_builder_events(@@ -50,7 +42,7 @@ def test_split_think_tags_preserve_text_and_thinking() -> None:def test_mixed_reasoning_content_and_think_tags_keep_order() -> None:- builder = SSEBuilder("msg_smoke", "smoke-model")+ builder = SSEBuilder("msg_contract", "contract-model")chunks = [builder.message_start()]chunks.extend(builder.ensure_thinking_block())chunks.append(builder.emit_thinking_delta("reasoning field"))@@ -68,7 +60,7 @@ def test_mixed_reasoning_content_and_think_tags_keep_order() -> None:def test_thinking_tool_text_and_transcript_order_contract() -> None:- builder = SSEBuilder("msg_smoke", "smoke-model")+ builder = SSEBuilder("msg_contract", "contract-model")chunks = [builder.message_start()]chunks.extend(builder.ensure_thinking_block())chunks.append(builder.emit_thinking_delta("inspect first"))@@ -136,7 +128,7 @@ def test_task_tool_arguments_force_foreground_execution() -> None:def _interleaved_thinking_text_events(parts: tuple[str, str, str, str],) -> Iterable[str]:- builder = SSEBuilder("msg_smoke", "smoke-model")+ builder = SSEBuilder("msg_contract", "contract-model")yield builder.message_start()yield from builder.ensure_thinking_block()yield builder.emit_thinking_delta(parts[0])@@ -157,7 +149,7 @@ def _events_from_text_chunks(*,enable_thinking: bool = True,) -> list[str]:- sse = builder or SSEBuilder("msg_smoke", "smoke-model")+ sse = builder or SSEBuilder("msg_contract", "contract-model")out: list[str] = [] if builder else [sse.message_start()]parser = ThinkTagParser()