:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 5 missing tables to the Secureframe source (#126730)
Tom Owers4168cd20712d
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex dca0c5c37a07..a0f5bc677230 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10664,10 +10664,10 @@ Diffed against: <https://developer.secureframe.com/>- [x] `framework_asset_scopes (GET /devices/{id}/framework_asset_scopes, /cloud_resources/{id}/…, /repositories/{id}/…)` — the join that says which framework each already-synced asset is in scope for; without it assets can't be filtered by framework (high)- [x] `ssp_policies (GET /ssp_policies)` — the policy inventory, a compliance object with no equivalent in the current tables (medium)- [x] `comments (GET /comments)` — remediation discussion attached to controls, tests and risks — the activity trail (medium)-- [ ] `trust_center_requests (GET /trust_center_requests)` — inbound access requests to the trust center, a demand signal worth trending (medium)-- [ ] `ssp_report_assessment_objectives (GET /ssp_report_assessment_objectives)` — per-objective assessment results underpinning control status (medium)-- [ ] `ssp_reports (GET /ssp_reports)` — System Security Plan report headers, the parent for all ssp\_\* sections (low)-- [ ] `ssp_report_sections and ssp_report_section_blocks` — section-level SSP content for report reconstruction (low)+- [x] `trust_center_requests (GET /trust_center_requests)` — inbound access requests to the trust center, a demand signal worth trending (medium)+- [x] `ssp_report_assessment_objectives (GET /ssp_report_assessment_objectives)` — per-objective assessment results underpinning control status (medium)+- [x] `ssp_reports (GET /ssp_reports)` — System Security Plan report headers, the parent for all ssp\_\* sections (low)+- [x] `ssp_report_sections and ssp_report_section_blocks` — section-level SSP content for report reconstruction (low)- [ ] `ssp_roles / ssp_duties / ssp_duty_roles` — role and duty assignment lookups referenced from SSP reports (low)- [ ] `ssp_vendors (GET /ssp_vendors)` — vendors as recorded in the SSP, distinct from the vendors and tprm/vendors tables (low)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.pyindex ddbe3c0cd1bf..6d361345ab1d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py@@ -359,6 +359,80 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"updated_at": "The date when this comment was updated.",},},+ "trust_center_requests": {+ "description": "A request from an outside party to get access to documents in the trust center.",+ "docs_url": "https://developer.secureframe.com/",+ "columns": {+ "id": "The identifier for this trust center request.",+ "company_name": "The company name of the requester.",+ "created_at": "The date this trust center request was created.",+ "document_security": "The document security level for this trust center request.",+ "email": "The email address of the requester.",+ "job_title": "The job title of the requester.",+ "reason": "The reason for this trust center request.",+ "requester_name": "The full name of the requester.",+ "resources": "The names of the resources requested.",+ "reviewed": "True if this trust center request was reviewed.",+ "trust_center_resource_requests": "The trust center resource requests associated with this trust center request.",+ "updated_at": "The date this trust center request was updated.",+ },+ },+ "ssp_reports": {+ "description": "A System Security Plan (SSP) report for one framework. It is the parent of the SSP sections, policies and assessment objectives.",+ "docs_url": "https://developer.secureframe.com/",+ "columns": {+ "id": "The identifier for this SSP report.",+ "completion_progress_percentage": "The completion progress percentage of the SSP report.",+ "created_at": "The date the SSP report was created.",+ "framework_id": "The identifier of the associated framework.",+ "name": "The name of the SSP report.",+ "sprs_score": "The SPRS score of the SSP report.",+ "updated_at": "The date the SSP report was last updated.",+ },+ },+ "ssp_report_sections": {+ "description": "A section of a System Security Plan (SSP) report.",+ "docs_url": "https://developer.secureframe.com/",+ "columns": {+ "id": "The identifier for this SSP report section.",+ "created_at": "The date when this SSP report section was created.",+ "description": "The description of the SSP report section.",+ "position": "The position of the SSP report section.",+ "section_key": "The section key of the SSP report section.",+ "ssp_report_id": "The identifier for the associated SSP report.",+ "status": "The status of the SSP report section.",+ "title": "The title of the SSP report section.",+ "updated_at": "The date when this SSP report section was last updated.",+ },+ },+ "ssp_report_section_blocks": {+ "description": "A content block of a System Security Plan (SSP) report section, such as an input field or a contact list.",+ "docs_url": "https://developer.secureframe.com/",+ "columns": {+ "id": "The identifier for this SSP report section block.",+ "block_key": "The semantic key identifying what this block represents.",+ "block_type": "The type of the block (e.g. input, textarea, select, multiselect, contacts).",+ "child_block_ids": "The ordered IDs of this block's child blocks.",+ "required": "Whether this block is required for section completion.",+ "ssp_report_id": "The identifier for the associated SSP report.",+ "ssp_report_section_id": "The identifier for the associated SSP report section.",+ "value": "The current value of the block.",+ },+ },+ "ssp_report_assessment_objectives": {+ "description": "The assessment result of one objective of a framework requirement in a System Security Plan (SSP) report.",+ "docs_url": "https://developer.secureframe.com/",+ "columns": {+ "id": "The identifier for this SSP report assessment objective.",+ "created_at": "The date this assessment objective was created.",+ "framework_requirement_id": "The identifier for the framework requirement.",+ "implementation_statement": "The implementation statement for this assessment objective.",+ "response": "The response status for this assessment objective.",+ "ssp_report_id": "The identifier for the associated SSP report.",+ "ssp_report_requirement_id": "The identifier for the SSP report requirement.",+ "updated_at": "The date this assessment objective was last updated.",+ },+ },"framework_asset_scopes": {"description": "The scope of one asset (a device, cloud resource or repository) for one compliance framework.","docs_url": "https://developer.secureframe.com/",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.pyindex 0bd8fefb0704..ae7c441250a9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py@@ -103,6 +103,31 @@ SECUREFRAME_ENDPOINTS: dict[str, SecureframeEndpointConfig] = {path="/comments",partition_key="created_at",),+ "trust_center_requests": SecureframeEndpointConfig(+ name="trust_center_requests",+ path="/trust_center_requests",+ partition_key="created_at",+ ),+ "ssp_reports": SecureframeEndpointConfig(+ name="ssp_reports",+ path="/ssp_reports",+ partition_key="created_at",+ ),+ "ssp_report_sections": SecureframeEndpointConfig(+ name="ssp_report_sections",+ path="/ssp_report_sections",+ partition_key="created_at",+ ),+ # The section block schema has no created_at, so no partitioning.+ "ssp_report_section_blocks": SecureframeEndpointConfig(+ name="ssp_report_section_blocks",+ path="/ssp_report_section_blocks",+ ),+ "ssp_report_assessment_objectives": SecureframeEndpointConfig(+ name="ssp_report_assessment_objectives",+ path="/ssp_report_assessment_objectives",+ partition_key="created_at",+ ),# Secureframe lists framework scopes only per asset, so this fans out over every device, cloud# resource and repository. Each row carries its own asset_type and asset_id."framework_asset_scopes": SecureframeEndpointConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.pyindex 72691c7558d6..0b4d259022e1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py@@ -74,16 +74,30 @@ class TestExtractRows:class TestPagination:- def test_top_level_array_envelopes_are_flattened(self):+ @pytest.mark.parametrize(+ "endpoint, expected_path",+ [+ ("controls", "/controls"),+ ("trust_center_requests", "/trust_center_requests"),+ ("ssp_reports", "/ssp_reports"),+ ("ssp_report_sections", "/ssp_report_sections"),+ ("ssp_report_section_blocks", "/ssp_report_section_blocks"),+ ("ssp_report_assessment_objectives", "/ssp_report_assessment_objectives"),+ ],+ )+ def test_top_level_array_envelopes_are_flattened(self, endpoint, expected_path):# The declared shape: a top-level array of per-item JSON:API envelopes.result = _driver().run(- "controls",+ endpoint,[- _response([{"data": {"id": "a", "type": "control", "attributes": {"name": "x"}}}]),+ _response([{"data": {"id": "a", "type": "x", "attributes": {"name": "x"}}}]),+ _response([{"data": {"id": "b", "type": "x", "attributes": {"name": "y"}}}]),_response([]),],)- assert result.rows == [{"id": "a", "name": "x"}]+ assert result.rows == [{"id": "a", "name": "x"}, {"id": "b", "name": "y"}]+ assert result.paths == [expected_path] * 3+ assert result.params("page") == ["1", "2", "3"]def test_resumes_from_saved_page(self):result = _driver().run(