Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse_sources): support qualys vmdr api version 6.0 (#126953)

Tom Owers77b4959d94b5
diff --git a/.agents/skills/warehouse-source-new-version/SKILL.md b/.agents/skills/warehouse-source-new-version/SKILL.md
index ddbc9ea51ea6..1d58b044015e 100644
--- a/.agents/skills/warehouse-source-new-version/SKILL.md
+++ b/.agents/skills/warehouse-source-new-version/SKILL.md
@@ -53,7 +53,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP
## Adding a new version, step by step
-1. **Read the vendor's changelog** (the source's `api_docs_url`) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. When the vendor's docs page renders client-side and comes back as an empty shell, read its machine-readable mirror (an `llms.txt`, a `sitemap.md`, or the same path with a `.md` suffix) rather than treating the version as undocumented. A Postman-published docs site carries `ownerId` and `publishedId` meta tags; `https://documenter.gw.postman.com/api/collections/<ownerId>/<publishedId>?versionTag=latest` returns the whole collection as JSON, changelog included in `info.description`. The reverse also happens: a `.md` mirror can be a stub that links to a spec without the endpoint, while the rendered HTML page carries the full request line and schema, so read the HTML before concluding the docs are missing. When the vendor publishes only one version-blind spec for the current version and no changelog, diff it against an archived copy (the Wayback Machine's CDX index lists snapshots; the `id_` form returns the raw file, sometimes still gzip-encoded) and check which label that copy names: differences it shows span every version between that copy and now, so do not attribute them to the new label alone. For a GraphQL vendor that serves a schema per version, validate the source's exact query strings against each version's SDL (for example with `graphql-core`'s `validate`): it names every removed field a query selects, and it settles changelog entries that only say a removal is "planned". This is also the evidence the gate above runs on.
+1. **Read the vendor's changelog** (the source's `api_docs_url`) and list what changed between the currently supported version(s) and the new one: renamed/removed fields, changed pagination, new required headers, changed webhook payloads, or a field the source reads becoming opt-in behind a new query parameter (a field returned by default in the old version now empty unless requested — restore it by adding that parameter on the new version's request path). Verification is docs-only — there are no stored credentials and no live-sync harness, so the docs are the sole source of truth for what each version serves. When the vendor's docs page renders client-side and comes back as an empty shell, read its machine-readable mirror (an `llms.txt`, a `sitemap.md`, or the same path with a `.md` suffix) rather than treating the version as undocumented. A Postman-published docs site carries `ownerId` and `publishedId` meta tags; `https://documenter.gw.postman.com/api/collections/<ownerId>/<publishedId>?versionTag=latest` returns the whole collection as JSON, changelog included in `info.description`. The reverse also happens: a `.md` mirror can be a stub that links to a spec without the endpoint, while the rendered HTML page carries the full request line and schema, so read the HTML before concluding the docs are missing. When the vendor publishes only one version-blind spec for the current version and no changelog, diff it against an archived copy (the Wayback Machine's CDX index lists snapshots; the `id_` form returns the raw file, sometimes still gzip-encoded) and check which label that copy names: differences it shows span every version between that copy and now, so do not attribute them to the new label alone. For a GraphQL vendor that serves a schema per version, validate the source's exact query strings against each version's SDL (for example with `graphql-core`'s `validate`): it names every removed field a query selects, and it settles changelog entries that only say a removal is "planned". For an XML vendor that publishes a DTD or XSD per version (often unauthenticated on the API host itself), diff those files: they list every added, removed or renamed element, which sample payloads do not. This is also the evidence the gate above runs on.
2. **Declare the version** (only once the gate says the version has to exist): add the new label to `supported_versions` and flip `default_version` to it — new sources always start on the newest stable version. A pinned row's **sync** path is unaffected by a default flip (that is the point of pinning), but two things still follow the new default: discovery/`get_schemas` if the pin isn't threaded there (step 3), and any row whose `api_version` is NULL. Reference the request layer's version constants instead of duplicating string literals.
3. **Dispatch on `SourceInputs.api_version`** at the request layer:
- Keep it minimal. If the version is just a header/URL segment and response shapes are compatible, thread the version string down to where the client/URL is built (see Stripe: `StripeSource.source_for_pipeline` passes `self.resolve_api_version(inputs.api_version)` → `stripe_source(...)` → `StripeClient(stripe_version=...)`). Resolve through `resolve_api_version` at the source class — never hardcode a fallback version in the request layer.
@@ -109,6 +109,7 @@ The opposite gap also occurs: a reference rendered from one version-blind OpenAP
- A new version can bring endpoints from a newer API generation of the same vendor (a `/v2/` namespace) with a different list envelope. Check the request layer's response-shape guards (list markers, truncation or corrupt-body retries) against that envelope: a guard keyed on the old marker classifies every new response as corrupt and repeats each request.
- When the vendor rejects a retired version with a generic status (a plain `404`), its signature also matches every other 404 from that host. Add it only when every URL the source requests is static (list endpoints, no per-id or user-supplied paths), so a 404 can only mean the version, and say so in a comment next to the entry.
- A vendor that falls forward on a retired pin — serving the oldest accessible version instead of rejecting the request — has no rejection signature to add. Its sunset still needs the repin migration: the pin silently stops being honored, with no failed sync to surface it.
+- The samples on one version's reference page can mix auth schemes (Basic in one example, Bearer in the next), so a sample header is not a requirement. Before you add a credential for the new version, find the vendor's general authentication statement: a credential the version does not need moves every row lacking it out of the scripted repin.
- A version bump can change the auth scheme, not just the wire format. Then the source config needs both credential shapes as optional fields, auth construction dispatches on the resolved version, and `validate_credentials` enforces the pair that version needs — form-level `required` can't express "depends on the pin".
- When the new auth scheme is OAuth client credentials, build the sync auth from the framework's `oauth2` auth config, which re-mints an expired token mid-sync. Vendor tokens can live only minutes, so a token minted once per sync expires during a long one.
- When the docs say a missing version header resolves to "latest" but print no accepted header value anywhere, do not guess one: a rejected guess fails every request on the new default. Send no header and flag the unpinned drift in the PR for a reviewer to resolve.
diff --git a/products/warehouse_sources/backend/migrations/0191_repin_qualys_vmdr_api_version_6_0.py b/products/warehouse_sources/backend/migrations/0191_repin_qualys_vmdr_api_version_6_0.py
new file mode 100644
index 000000000000..22e80de80b58
--- /dev/null
+++ b/products/warehouse_sources/backend/migrations/0191_repin_qualys_vmdr_api_version_6_0.py
@@ -0,0 +1,38 @@
+from django.db import migrations
+
+# Qualys retires the Host List and VM Scan 2.0 routes in December 2026 (End of Life), and source-level
+# "4.0" pins still call them, so those pins move to "6.0". The whole cohort is safe to script: on 6.0
+# the hosts and host_list_detection tables call /api/6.0/, and the scans and appliances tables call
+# /api/3.0/. All four keep basic auth with the stored credentials, keep their primary keys, and only
+# gain optional XML elements, which land as new columns. knowledge_base already uses the 4.0 JWT
+# route on both pins, so no source needs a credential it doesn't already have.
+#
+# Only the source-level pin is touched. Schema-level `ExternalDataSchema.api_version` overrides are
+# user-managed by design (a customer intentionally pinned that schema) and are left alone — the
+# schema-level deprecation warning prompts the user to migrate those.
+QUALYS_VMDR_SOURCE_TYPE = "QualysVmdr"
+QUALYS_VMDR_API_VERSION_4_0 = "4.0"
+QUALYS_VMDR_API_VERSION_6_0 = "6.0"
+
+
+def repin_qualys_vmdr_4_0_to_6_0(apps, schema_editor):
+ ExternalDataSource = apps.get_model("warehouse_sources", "ExternalDataSource")
+
+ # Idempotent: matches only explicit "4.0" pins, so a second run finds none. NULL pins already
+ # resolve to the "6.0" default, and "2.0" pins wait on the gateway URL manual path, so neither
+ # is touched.
+ ExternalDataSource.objects.filter(
+ source_type=QUALYS_VMDR_SOURCE_TYPE, api_version=QUALYS_VMDR_API_VERSION_4_0
+ ).update(api_version=QUALYS_VMDR_API_VERSION_6_0)
+
+
+class Migration(migrations.Migration):
+ dependencies = [
+ ("warehouse_sources", "0190_repin_profound_api_version_v2"),
+ ]
+
+ operations = [
+ # Reverse is a no-op: once repinned, "6.0" rows are indistinguishable from natively-created
+ # ones, so a blanket downgrade would clobber legitimate "6.0" pins.
+ migrations.RunPython(repin_qualys_vmdr_4_0_to_6_0, migrations.RunPython.noop, elidable=False),
+ ]
diff --git a/products/warehouse_sources/backend/migrations/max_migration.txt b/products/warehouse_sources/backend/migrations/max_migration.txt
index 396f6348cfae..8be49032f927 100644
--- a/products/warehouse_sources/backend/migrations/max_migration.txt
+++ b/products/warehouse_sources/backend/migrations/max_migration.txt
@@ -1 +1 @@
-0190_repin_profound_api_version_v2
+0191_repin_qualys_vmdr_api_version_6_0
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/qualys_vmdr.py b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/qualys_vmdr.py
index 424269c33c20..ae453acd736d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/qualys_vmdr.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/qualys_vmdr.py
@@ -52,9 +52,9 @@ MAX_JWT_DOWNLOAD_SECONDS = 30
RESPONSE_TOO_LARGE_ERROR = "Qualys API response body was too large"
RESPONSE_TOO_SLOW_ERROR = "Qualys API response download was too slow"
-# Raised when a JWT-authenticated endpoint (KnowledgeBase on 4.0) is synced but no gateway URL is
-# configured — the token can't be minted without it. Non-retryable: retrying can't create the field.
-MISSING_GATEWAY_ERROR = "Qualys API gateway URL is required for API version 4.0"
+# Raised when a JWT-authenticated endpoint (KnowledgeBase on 4.0 and later) is synced but no gateway URL
+# is configured — the token can't be minted without it. Non-retryable: retrying can't create the field.
+MISSING_GATEWAY_ERROR = "Qualys API gateway URL is required to sync the KnowledgeBase API"
# Qualys rejects any request without an X-Requested-With header.
_BASE_HEADERS = {"X-Requested-With": "PostHog Data Warehouse"}
@@ -319,7 +319,8 @@ def _extract_rows(root: Element, config: QualysVmdrEndpointConfig) -> Iterator[d
# One row per (host, detection): host scalars prefixed host_* merged with the
# detection's own fields. `unique_vuln_id` is the subscription-wide detection id.
host_fields = {
- f"host_{key}": value for key, value in _element_to_row(item, skip_tags={"DETECTION_LIST"}).items()
+ f"host_{key}": value
+ for key, value in _element_to_row(item, skip_tags={"DETECTION_LIST", *config.drop_tags}).items()
}
for detection in item.iter("DETECTION"):
yield {**host_fields, **_element_to_row(detection)}
@@ -340,6 +341,17 @@ def _next_batch_url(root: Element, base_url: str) -> str | None:
return urlunsplit((base_parts.scheme, base_parts.netloc, next_parts.path, next_parts.query, ""))
+def _resume_url(saved_url: str, base_url: str, path: str) -> str:
+ """Rebuild a saved next-batch URL on the pin's current route.
+
+ A checkpoint saved before a repin carries the old version's path, which Qualys may have retired.
+ The truncation query (filters plus the id_min cursor) is the same on every version, so only the
+ path changes.
+ """
+ base_parts = urlsplit(base_url)
+ return urlunsplit((base_parts.scheme, base_parts.netloc, path, urlsplit(saved_url).query, ""))
+
+
def _build_initial_url(
base_url: str,
config: QualysVmdrEndpointConfig,
@@ -382,7 +394,7 @@ def get_rows(
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
if resume is not None and resume.next_url:
- url = resume.next_url
+ url = _resume_url(resume.next_url, base_url, config.resolve_path(api_version))
logger.debug(f"Qualys VMDR: resuming from URL: {url}")
else:
url = _build_initial_url(
@@ -406,8 +418,8 @@ def get_rows(
url = next_url
-def validate_credentials(api_server: str, username: str, password: str) -> tuple[bool, str | None]:
- """Cheap create-time probe: one host-list request capped at a single record.
+def validate_credentials(api_server: str, username: str, password: str, api_version: str) -> tuple[bool, str | None]:
+ """Cheap create-time probe: one host-list request capped at a single record, on the pin's route.
403 is accepted — the credential authenticated but lacks the asset-listing scope; per-table
scope problems surface at sync time. 409 (rate/concurrency limited) also proves the
@@ -418,7 +430,8 @@ def validate_credentials(api_server: str, username: str, password: str) -> tuple
if not allowed:
return False, f"Qualys API server URL is not allowed: {reason}"
- url = f"{base_url}/api/2.0/fo/asset/host/?{urlencode({'action': 'list', 'truncation_limit': 1})}"
+ path = QUALYS_VMDR_ENDPOINTS["hosts"].resolve_path(api_version)
+ url = f"{base_url}{path}?{urlencode({'action': 'list', 'truncation_limit': 1})}"
try:
# stream=True so a user-controlled host can't force us to buffer an unbounded probe body:
# we only ever inspect the status line here, never the body.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/settings.py
index b4fc09d88ac9..f549fc79dbca 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/settings.py
@@ -5,16 +5,20 @@ from posthog.dataclasses import frozen
from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
-# Opaque vendor version labels — never parsed or ordered. Only the KnowledgeBase Vulnerability API
-# is versioned past 2.0: Qualys is retiring its 2.0 (EOS Dec 2025) in favor of 4.0, which serves the
-# same feed at `/api/4.0/fo/knowledge_base/vuln/` but requires JWT Bearer auth instead of basic auth.
-# The FO Host List / Host List Detection / VM Scan APIs are a separate versioning track and stay on
-# `/api/2.0/` with basic auth under both source-level pins.
+# Opaque vendor version labels — never parsed or ordered. Qualys versions each FO API family on its
+# own track, so a source-level label names a set of per-endpoint routes rather than one URL segment:
+# - "4.0" moves the KnowledgeBase Vulnerability API off 2.0 (EOS Dec 2025) to
+# `/api/4.0/fo/knowledge_base/vuln/`, which requires JWT Bearer auth instead of basic auth. The
+# other FO APIs stay on `/api/2.0/` with basic auth.
+# - "6.0" also moves Host List and Host List Detection to `/api/6.0/`, and the VM Scan and Scanner
+# Appliance lists to `/api/3.0/`, because Qualys retires their 2.0 routes. Those routes keep basic
+# auth and only add optional elements to the same XML output.
QUALYS_VMDR_API_VERSION_2_0 = "2.0"
QUALYS_VMDR_API_VERSION_4_0 = "4.0"
+QUALYS_VMDR_API_VERSION_6_0 = "6.0"
# Declared oldest -> newest; the default is always the last entry.
-QUALYS_VMDR_SUPPORTED_VERSIONS = (QUALYS_VMDR_API_VERSION_2_0, QUALYS_VMDR_API_VERSION_4_0)
-QUALYS_VMDR_DEFAULT_VERSION = QUALYS_VMDR_API_VERSION_4_0
+QUALYS_VMDR_SUPPORTED_VERSIONS = (QUALYS_VMDR_API_VERSION_2_0, QUALYS_VMDR_API_VERSION_4_0, QUALYS_VMDR_API_VERSION_6_0)
+QUALYS_VMDR_DEFAULT_VERSION = QUALYS_VMDR_API_VERSION_6_0
@frozen
@@ -41,8 +45,7 @@ class QualysVmdrEndpointConfig:
drop_tags: frozenset[str] = frozenset()
should_sync_default: bool = True
# Per-version path overrides for endpoints whose route moved between vendor versions. Absent
- # keys fall through to `path` — that is the correct version-independent route, not silent drift,
- # since only the KnowledgeBase endpoint diverges.
+ # keys fall through to `path`, the route every other version still serves.
path_overrides: dict[str, str] = field(default_factory=dict)
# Source-level versions on which this endpoint authenticates with a JWT Bearer token minted at
# the Qualys gateway, rather than HTTP basic auth. Empty = basic auth on every version.
@@ -75,6 +78,9 @@ QUALYS_VMDR_ENDPOINTS: dict[str, QualysVmdrEndpointConfig] = {
],
primary_keys=["id"],
truncation_limit=1000,
+ # The 6.0 host DTD adds the cloud agent's registration credential.
+ drop_tags=frozenset({"AGENT_ACTIVATION_KEY"}),
+ path_overrides={QUALYS_VMDR_API_VERSION_6_0: "/api/6.0/fo/asset/host/"},
),
# The core VMDR feed: one row per (host, detection). `detection_updated_since` is a
# server-side filter on the detection's last update, so incremental syncs re-pull rows whose
@@ -98,6 +104,8 @@ QUALYS_VMDR_ENDPOINTS: dict[str, QualysVmdrEndpointConfig] = {
partition_key="first_found_datetime",
truncation_limit=500,
flatten_host_detections=True,
+ drop_tags=frozenset({"AGENT_ACTIVATION_KEY"}),
+ path_overrides={QUALYS_VMDR_API_VERSION_6_0: "/api/6.0/fo/asset/host/vm/detection/"},
),
# VM scan history. The scan list endpoint has no truncation pagination — it returns every
# scan matching the filters in one response — so incremental filtering on launch datetime
@@ -119,6 +127,7 @@ QUALYS_VMDR_ENDPOINTS: dict[str, QualysVmdrEndpointConfig] = {
primary_keys=["ref"],
partition_key="launch_datetime",
truncation_limit=None,
+ path_overrides={QUALYS_VMDR_API_VERSION_6_0: "/api/3.0/fo/scan/"},
),
# Vulnerability definitions (QIDs). Requires the "KnowledgeBase download" option enabled on
# the Qualys subscription, so it's off by default. `details=Basic` keeps rows compact
@@ -141,8 +150,11 @@ QUALYS_VMDR_ENDPOINTS: dict[str, QualysVmdrEndpointConfig] = {
truncation_limit=None,
should_sync_default=False,
# Qualys is sunsetting KnowledgeBase 2.0; 4.0 serves the same feed but only over JWT Bearer.
- path_overrides={QUALYS_VMDR_API_VERSION_4_0: "/api/4.0/fo/knowledge_base/vuln/"},
- jwt_versions=frozenset({QUALYS_VMDR_API_VERSION_4_0}),
+ path_overrides={
+ QUALYS_VMDR_API_VERSION_4_0: "/api/4.0/fo/knowledge_base/vuln/",
+ QUALYS_VMDR_API_VERSION_6_0: "/api/4.0/fo/knowledge_base/vuln/",
+ },
+ jwt_versions=frozenset({QUALYS_VMDR_API_VERSION_4_0, QUALYS_VMDR_API_VERSION_6_0}),
),
# Asset groups: the grouping dimension for hosts and reports. No server-side "modified since"
# filter exists, so this is full refresh. `show_attributes=ALL` adds the owner, last update,
@@ -178,6 +190,7 @@ QUALYS_VMDR_ENDPOINTS: dict[str, QualysVmdrEndpointConfig] = {
truncation_limit=None,
# The activation code registers a scanner to the subscription, so it stays out of the warehouse.
drop_tags=frozenset({"ACTIVATION_CODE"}),
+ path_overrides={QUALYS_VMDR_API_VERSION_6_0: "/api/3.0/fo/appliance/"},
),
# The subscription's tracked IP inventory, as the single IPs and ranges Qualys returns. The
# entries carry no ID, so the entry text is the key. The list is not paginated.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/source.py
index 16f16498a17c..bb2cb48cdc69 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/source.py
@@ -34,6 +34,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.qualys_vmd
ENDPOINTS,
INCREMENTAL_FIELDS,
QUALYS_VMDR_API_VERSION_2_0,
+ QUALYS_VMDR_API_VERSION_4_0,
QUALYS_VMDR_DEFAULT_VERSION,
QUALYS_VMDR_ENDPOINTS,
QUALYS_VMDR_SUPPORTED_VERSIONS,
@@ -52,6 +53,9 @@ class QualysVmdrSource(ResumableSource[QualysVmdrSourceConfig, QualysVmdrResumeC
# sunset date is the End of Life, when Qualys stops serving the version.
deprecated_versions = (
VersionDeprecation(version=QUALYS_VMDR_API_VERSION_2_0, sunset_at=datetime.date(2026, 6, 30)),
+ # Host List and VM Scan 2.0, which 4.0 still calls, reach End of Life in December 2026 with no
+ # day given, so the sunset is the first day of that month.
+ VersionDeprecation(version=QUALYS_VMDR_API_VERSION_4_0, sunset_at=datetime.date(2026, 12, 1)),
)
api_docs_url = "https://docs.qualys.com/en/vm/api/index.htm"
@@ -77,7 +81,7 @@ class QualysVmdrSource(ResumableSource[QualysVmdrSourceConfig, QualysVmdrResumeC
Use your account's regional API server URL (for example `qualysapi.qualys.com`, `qualysapi.qg2.apps.qualys.com`, or `qualysapi.qualys.eu`) — you can find it under **Help > About** in the Qualys UI. The user needs API access enabled (a Manager role, or a role granted API access).
-The `knowledge_base` table additionally requires the KnowledgeBase download option to be enabled on your Qualys subscription. On API version 4.0 it also needs your account's gateway URL (for example `gateway.qg2.apps.qualys.com`), which you can find under **Help > About** in the Qualys UI. Leave the gateway URL blank if you do not sync the `knowledge_base` table.""",
+The `knowledge_base` table additionally requires the KnowledgeBase download option to be enabled on your Qualys subscription. On API version 4.0 and later it also needs your account's gateway URL (for example `gateway.qg2.apps.qualys.com`), which you can find under **Help > About** in the Qualys UI. Leave the gateway URL blank if you do not sync the `knowledge_base` table.""",
iconPath="/static/services/qualys_vmdr.png",
docsUrl="https://posthog.com/docs/cdp/sources/qualys-vmdr",
fields=cast(
@@ -107,7 +111,7 @@ The `knowledge_base` table additionally requires the KnowledgeBase download opti
placeholder="",
secret=True,
),
- # Optional: only the KnowledgeBase table on API version 4.0 needs it, so form-level
+ # Optional: only the KnowledgeBase table on API version 4.0 and later needs it, so form-level
# `required` can't express the dependency — it's enforced at sync time instead.
SourceFieldInputConfig(
name="gateway_server",
@@ -137,7 +141,7 @@ The `knowledge_base` table additionally requires the KnowledgeBase download opti
"Qualys API server URL is not allowed": "The configured API server URL points at a blocked or internal address. Set it to your Qualys account's regional API server and reconnect.",
"Qualys API response body was too large": "The Qualys API server returned an unexpectedly large response. Confirm the configured API server URL points at your Qualys account's regional API server and reconnect.",
"Qualys API response download was too slow": "The Qualys API server was too slow to return a response. Confirm the configured API server URL points at your Qualys account's regional API server and reconnect.",
- MISSING_GATEWAY_ERROR: "Syncing the knowledge_base table on API version 4.0 needs your Qualys gateway URL. Add it to the source configuration and reconnect.",
+ MISSING_GATEWAY_ERROR: "Syncing the knowledge_base table on API version 4.0 and later needs your Qualys gateway URL. Add it to the source configuration and reconnect.",
}
def get_schemas(
@@ -181,7 +185,9 @@ The `knowledge_base` table additionally requires the KnowledgeBase download opti
schema_name: Optional[str] = None,
api_version: str | None = None,
) -> tuple[bool, str | None]:
- return validate_qualys_vmdr_credentials(config.api_server, config.username, config.password)
+ return validate_qualys_vmdr_credentials(
+ config.api_server, config.username, config.password, self.resolve_api_version(api_version)
+ )
def get_resumable_source_manager(self, inputs: SourceInputs) -> ResumableSourceManager[QualysVmdrResumeConfig]:
return ResumableSourceManager[QualysVmdrResumeConfig](inputs, QualysVmdrResumeConfig)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/tests/test_qualys_vmdr.py b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/tests/test_qualys_vmdr.py
index 94ff1fa5f972..81084ea0e9fa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/tests/test_qualys_vmdr.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/qualys_vmdr/tests/test_qualys_vmdr.py
@@ -90,6 +90,7 @@ DETECTION_PAGE = """<?xml version="1.0" encoding="UTF-8" ?>
<HOST>
<ID>1001</ID>
<IP>10.0.0.1</IP>
+ <AGENT_ACTIVATION_KEY>00000000-0000-0000-0000-000000000000</AGENT_ACTIVATION_KEY>
<DETECTION_LIST>
<DETECTION>
<UNIQUE_VULN_ID>111</UNIQUE_VULN_ID>
@@ -264,6 +265,17 @@ class TestQualysVmdr:
assert rows[0]["last_update_datetime"] == "2026-07-01T00:00:00Z"
# The host's DETECTION_LIST must not leak into the flattened row
assert "host_detection_list" not in rows[0]
+ assert "host_agent_activation_key" not in rows[0]
+
+ def test_extract_host_rows_drop_agent_activation_key(self):
+ root = _parse_xml(
+ "<HOST_LIST_OUTPUT><RESPONSE><HOST_LIST><HOST><ID>1001</ID>"
+ "<AGENT_ACTIVATION_KEY>00000000-0000-0000-0000-000000000000</AGENT_ACTIVATION_KEY>"
+ "</HOST></HOST_LIST></RESPONSE></HOST_LIST_OUTPUT>"
+ )
+ rows = list(_extract_rows(root, QUALYS_VMDR_ENDPOINTS["hosts"]))
+
+ assert rows == [{"id": "1001"}]
def test_get_rows_paginates_and_saves_state_after_yield(self):
result = _drive("hosts", [_xml(HOST_LIST_PAGE_1), _xml(HOST_LIST_PAGE_2)], api_version="2.0")
@@ -347,17 +359,45 @@ class TestQualysVmdr:
assert result.rows == expected_rows
assert result.requests[0].headers["authorization"].startswith("Basic ")
- def test_get_rows_resumes_from_saved_state(self):
+ @pytest.mark.parametrize(
+ "endpoint,api_version,expected_path",
+ [
+ ("hosts", "4.0", "/api/2.0/fo/asset/host/"),
+ ("hosts", "6.0", "/api/6.0/fo/asset/host/"),
+ ("host_list_detection", "4.0", "/api/2.0/fo/asset/host/vm/detection/"),
+ ("host_list_detection", "6.0", "/api/6.0/fo/asset/host/vm/detection/"),
+ ("scans", "4.0", "/api/2.0/fo/scan/"),
+ ("scans", "6.0", "/api/3.0/fo/scan/"),
+ ("appliances", "4.0", "/api/2.0/fo/appliance/"),
+ ("appliances", "6.0", "/api/3.0/fo/appliance/"),
+ ],
+ )
+ def test_get_rows_routes_each_pin_to_its_fo_api_version_with_basic_auth(self, endpoint, api_version, expected_path):
+ result = _drive(endpoint, [_xml("<RESPONSE/>")], api_version=api_version)
+
+ assert result.raised is None
+ assert result.paths == [expected_path]
+ assert result.requests[0].headers["authorization"].startswith("Basic ")
+
+ @pytest.mark.parametrize(
+ "api_version,expected_url",
+ [
+ ("4.0", "https://qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list&id_min=1002"),
+ ("6.0", "https://qualysapi.qualys.com/api/6.0/fo/asset/host/?action=list&id_min=1002"),
+ ],
+ )
+ def test_get_rows_resumes_from_saved_state_on_the_pins_route(self, api_version, expected_url):
+ # A checkpoint saved before a 4.0 -> 6.0 repin still carries the retired 2.0 path
resume_url = "https://qualysapi.qualys.com/api/2.0/fo/asset/host/?action=list&id_min=1002"
result = _drive(
"hosts",
[_xml(HOST_LIST_PAGE_2)],
- api_version="2.0",
+ api_version=api_version,
resume_state=QualysVmdrResumeConfig(next_url=resume_url),
)
- assert result.urls == [resume_url]
+ assert result.urls == [expected_url]
assert [row["id"] for row in result.rows] == ["1002"]
def test_get_rows_rejects_disallowed_api_server_before_any_request(self):
@@ -402,29 +442,33 @@ class TestQualysVmdr:
(409, True), # rate limited — the credential authenticated
],
)
- def test_validate_credentials_status_mapping(self, status_code, expected):
+ @pytest.mark.parametrize(
+ "api_version,expected_path", [("4.0", "/api/2.0/fo/asset/host/"), ("6.0", "/api/6.0/fo/asset/host/")]
+ )
+ def test_validate_credentials_status_mapping(self, status_code, expected, api_version, expected_path):
with scripted_network(always(_xml("<x/>", status=status_code))) as network:
- ok, error = validate_credentials("qualysapi.qualys.com", "user", "pass")
+ ok, error = validate_credentials("qualysapi.qualys.com", "user", "pass", api_version)
assert ok is expected
assert (error is None) is expected
- assert network.requests_log[0].path == "/api/2.0/fo/asset/host/"
+ assert network.requests_log[0].path == expected_path
assert network.requests_log[0].param("truncation_limit") == "1"
def test_validate_credentials_rejects_disallowed_api_server_before_any_request(self):
with scripted_network([]) as network:
- ok, error = validate_credentials("localhost", "user", "pass")
+ ok, error = validate_credentials("localhost", "user", "pass", "6.0")
assert ok is False
assert error is not None and "not allowed" in error
assert network.requests_log == []
- def test_get_rows_knowledge_base_4_0_mints_jwt_and_uses_bearer(self):
+ @pytest.mark.parametrize("api_version", ["4.0", "6.0"])
+ def test_get_rows_knowledge_base_mints_jwt_and_uses_bearer(self, api_version):
result = _drive(
"knowledge_base",
route({"/auth": [_xml("jwt-token")], "/api/4.0/fo/knowledge_base/vuln/": [_xml(KNOWLEDGE_BASE_PAGE)]}),
config=_config(gateway_server="gateway.qg2.apps.qualys.com"),
- api_version="4.0",
+ api_version=api_version,
)
# The JWT is minted at the gateway host, then threaded into the request session as a Bearer token