Open-source AI penetration testing tool to find and fix your app’s vulnerabilities.
Python agentsai-hackingai-penetration-testingai-pentestingai-securityartificial-intelligencebug-bountycode-qualityctf-toolscybersecuritycybersecurity-toolsethical-hackinghackingllm-securityoffensive-securitypenetration-testingpentesting-toolsred-teamingsecuritysecurity-automation
fix: calibrate vulnerability severity to demonstrated impact
bearsyankeesa51ca1866654
diff --git a/strix/agents/prompts/system_prompt.jinja b/strix/agents/prompts/system_prompt.jinjaindex 97448a3b..0be45595 100644--- a/strix/agents/prompts/system_prompt.jinja+++ b/strix/agents/prompts/system_prompt.jinja@@ -207,6 +207,9 @@ VALIDATION REQUIREMENTS:- Full validation required - no assumptions- Demonstrate concrete impact with evidence- Consider business context for severity assessment — check whether the target is a demo/sandbox environment or content meant to be public, and factor that in+- Score only the security impact demonstrated by the proof of concept. Reachability, missing authentication, scanner labels, and theoretical follow-on attacks do not by themselves justify non-None CVSS impact metrics+- Treat public metadata, internal-looking identifiers, source maps without secrets, and transport/configuration hygiene as observations unless validation proves unauthorized restricted-data access, modification, or service disruption+- Every non-None Confidentiality, Integrity, or Availability metric must map to explicit evidence in the report; use Scope Changed only for a demonstrated crossing of security authorities- Independent verification through subagent- Document complete attack chain- Keep going until you find something that mattersdiff --git a/strix/skills/vulnerabilities/information_disclosure.md b/strix/skills/vulnerabilities/information_disclosure.mdindex 583756b4..5c09f3ab 100644--- a/strix/skills/vulnerabilities/information_disclosure.md+++ b/strix/skills/vulnerabilities/information_disclosure.md@@ -113,10 +113,13 @@ Information leaks accelerate exploitation by revealing code, configuration, iden## Triage Rubric-- **Critical**: Credentials/keys; signed URL secrets; config dumps; unrestricted admin/observability panels-- **High**: Versions with reachable CVEs; cross-tenant data; caches serving cross-user content-- **Medium**: Internal paths/hosts enabling LFI/SSRF pivots; source maps revealing hidden endpoints-- **Low**: Generic headers, marketing versions, intended documentation without exploit path+- **Critical**: Direct disclosure of secrets that provide broad privileged control, with successful use validated where safe+- **High**: Direct disclosure of highly sensitive data, cross-tenant data, or credentials with serious demonstrated access+- **Medium**: Unauthorized access to a limited set of genuinely restricted data, or a fully validated chain from the disclosure to a concrete security consequence+- **Low**: Limited unauthorized disclosure with modest sensitivity and no serious direct consequence+- **Informational / no report**: Public or intended client data, generic headers, internal names or private addresses, versions without a reachable exploit, and source maps or diagnostics that expose no secrets or restricted source++Do not assign Confidentiality Low merely because information helps reconnaissance. CVSS C:L requires actual access to restricted information. If a path, hostname, version, source map, schema, or debug value only suggests a possible second vulnerability, either validate that complete chain and score its demonstrated outcome or leave C:N and omit the vulnerability report.## Exploitation Chains@@ -152,6 +155,7 @@ Information leaks accelerate exploitation by revealing code, configuration, iden3. Attempt minimal, reversible exploitation or present a concrete step-by-step chain4. Show reproducibility and minimal request set5. Bound scope (user, tenant, environment) and data sensitivity classification+6. Map each non-None CVSS impact metric to evidence of actual restricted disclosure, modification, or service interruption## False Positivesdiff --git a/strix/tools/reporting/tool.py b/strix/tools/reporting/tool.pyindex b7bb1885..598def57 100644--- a/strix/tools/reporting/tool.py+++ b/strix/tools/reporting/tool.py@@ -126,23 +126,25 @@ def _validate_cwe(cwe: str) -> str | None:def _calculate_cvss(breakdown: dict[str, str]) -> tuple[float, str, str]:+ from cvss import CVSS3++ vector = (+ f"CVSS:3.1/AV:{breakdown['attack_vector']}/AC:{breakdown['attack_complexity']}/"+ f"PR:{breakdown['privileges_required']}/UI:{breakdown['user_interaction']}/"+ f"S:{breakdown['scope']}/C:{breakdown['confidentiality']}/"+ f"I:{breakdown['integrity']}/A:{breakdown['availability']}"+ )+try:- from cvss import CVSS3+ cvss = CVSS3(vector)+ score = cvss.scores()[0]+ base_severity = cvss.severities()[0].lower()+ except Exception as exc:+ msg = f"Failed to calculate CVSS for validated vector: {vector}"+ raise ValueError(msg) from exc- vector = (- f"CVSS:3.1/AV:{breakdown['attack_vector']}/AC:{breakdown['attack_complexity']}/"- f"PR:{breakdown['privileges_required']}/UI:{breakdown['user_interaction']}/"- f"S:{breakdown['scope']}/C:{breakdown['confidentiality']}/"- f"I:{breakdown['integrity']}/A:{breakdown['availability']}"- )- c = CVSS3(vector)- score = c.scores()[0]- severity = c.severities()[0].lower()- except Exception:- logger.exception("Failed to calculate CVSS")- return 7.5, "high", ""- else:- return score, severity, vector+ severity = "info" if base_severity == "none" else base_severity+ return score, severity, vector_REQUIRED_FIELDS = {@@ -233,7 +235,10 @@ async def _do_create( # noqa: PLR0912if errors:return {"success": False, "error": "Validation failed", "errors": errors}- cvss_score, severity, _vector = _calculate_cvss(cvss_breakdown)+ try:+ cvss_score, severity, _vector = _calculate_cvss(cvss_breakdown)+ except ValueError as exc:+ return {"success": False, "error": "Validation failed", "errors": [str(exc)]}try:from strix.report.state import get_global_report_state@@ -377,6 +382,30 @@ async def create_vulnerability_report(lockfile/manifest that matches a published advisory. File thosewith ``create_dependency_report`` instead, never with this tool.+ **Reporting and severity gate**:++ - A reachable endpoint, unusual response, weak configuration, or+ reconnaissance artifact is not by itself a vulnerability. File a+ report only when the PoC demonstrates an unauthorized security+ consequence or a realistic, fully validated path to one.+ - Score only the reasonable final impact supported by the PoC. Do+ not score speculative pivots or consequences that require another+ unverified vulnerability.+ - Network reachability and missing authentication affect+ exploitability; neither creates Confidentiality, Integrity, or+ Availability impact by itself.+ - Public metadata, internal-looking names or addresses, software+ versions, intended client-side code, and source maps without+ secrets or restricted source normally have ``C:N``.+ - Configuration and transport observations require a realistic+ attacker-controlled exploit and direct security impact. Client+ errors, compatibility issues, fingerprinting, and attack-surface+ discovery alone should not be filed as vulnerabilities.+ - Before filing, verify that the impact narrative, PoC, and every+ non-None CVSS impact metric describe the same demonstrated+ consequence. When evidence is incomplete, lower the metric or+ continue validation; never choose a higher value "to be safe."+Automatic LLM-based **deduplication** rejects reports that describethe same root cause on the same asset as an existing report. If youget a ``duplicate_of`` response, do NOT retry — move on to other@@ -430,6 +459,23 @@ async def create_vulnerability_report(- ``confidentiality`` / ``integrity`` / ``availability``: ``N`` /``L`` / ``H``+ Derive the vector from the demonstrated attack, not the finding+ category or a scanner/template severity:++ - ``C:L`` requires actual access to some restricted information.+ Reconnaissance value alone is ``C:N``. ``C:H`` requires total+ disclosure or limited disclosure with a direct serious impact,+ such as a usable administrator credential or private key.+ - ``I:L`` requires demonstrated unauthorized, limited modification;+ ``I:H`` requires total or directly serious modification. Otherwise+ use ``I:N``.+ - ``A:L`` requires demonstrated performance degradation or service+ interruption; ``A:H`` requires complete or directly serious+ denial of the affected service. Otherwise use ``A:N``.+ - Use ``S:C`` only when exploitation demonstrably crosses into a+ component governed by a different security authority. A separate+ backend, downstream effect, or third-party name is insufficient.+Example::{@@ -502,7 +548,10 @@ async def create_vulnerability_report((1-3 sentences) — it appears first in the report. Deeptechnical detail and root-cause analysis belong in``technical_analysis``, not here.- impact: What an attacker achieves; business risk; data at risk.+ impact: The unauthorized result demonstrated by the PoC, the+ affected data or operation, and its scope. Keep plausible+ but unverified follow-on risks separate; do not use them to+ set CVSS metrics.target: Affected URL / domain / repository.technical_analysis: The mechanism and root cause.poc_description: Step-by-step reproduction (steps only, no code).diff --git a/tests/test_reporting_tool.py b/tests/test_reporting_tool.pynew file mode 100644index 00000000..55122334--- /dev/null+++ b/tests/test_reporting_tool.py@@ -0,0 +1,48 @@+from __future__ import annotations++import pytest++from strix.tools.reporting.tool import _calculate_cvss+++def test_cvss_without_demonstrated_impact_is_informational() -> None:+ score, severity, vector = _calculate_cvss(+ {+ "attack_vector": "N",+ "attack_complexity": "L",+ "privileges_required": "N",+ "user_interaction": "N",+ "scope": "U",+ "confidentiality": "N",+ "integrity": "N",+ "availability": "N",+ }+ )++ assert score == 0.0+ assert severity == "info"+ assert vector == "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:N"+++def test_cvss_calculation_does_not_fabricate_high_severity(+ monkeypatch: pytest.MonkeyPatch,+) -> None:+ class BrokenCVSS:+ def __init__(self, vector: str) -> None:+ raise RuntimeError(vector)++ monkeypatch.setattr("cvss.CVSS3", BrokenCVSS)++ with pytest.raises(ValueError, match="Failed to calculate CVSS"):+ _calculate_cvss(+ {+ "attack_vector": "N",+ "attack_complexity": "L",+ "privileges_required": "N",+ "user_interaction": "N",+ "scope": "U",+ "confidentiality": "L",+ "integrity": "N",+ "availability": "N",+ }+ )