Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 5 missing tables to the Secureframe source (#126730)

Tom Owers4168cd20712d
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index dca0c5c37a07..a0f5bc677230 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10664,10 +10664,10 @@ Diffed against: <https://developer.secureframe.com/>
- [x] `framework_asset_scopes (GET /devices/{id}/framework_asset_scopes, /cloud_resources/{id}/…, /repositories/{id}/…)` — the join that says which framework each already-synced asset is in scope for; without it assets can't be filtered by framework (high)
- [x] `ssp_policies (GET /ssp_policies)` — the policy inventory, a compliance object with no equivalent in the current tables (medium)
- [x] `comments (GET /comments)` — remediation discussion attached to controls, tests and risks — the activity trail (medium)
-- [ ] `trust_center_requests (GET /trust_center_requests)` — inbound access requests to the trust center, a demand signal worth trending (medium)
-- [ ] `ssp_report_assessment_objectives (GET /ssp_report_assessment_objectives)` — per-objective assessment results underpinning control status (medium)
-- [ ] `ssp_reports (GET /ssp_reports)` — System Security Plan report headers, the parent for all ssp\_\* sections (low)
-- [ ] `ssp_report_sections and ssp_report_section_blocks` — section-level SSP content for report reconstruction (low)
+- [x] `trust_center_requests (GET /trust_center_requests)` — inbound access requests to the trust center, a demand signal worth trending (medium)
+- [x] `ssp_report_assessment_objectives (GET /ssp_report_assessment_objectives)` — per-objective assessment results underpinning control status (medium)
+- [x] `ssp_reports (GET /ssp_reports)` — System Security Plan report headers, the parent for all ssp\_\* sections (low)
+- [x] `ssp_report_sections and ssp_report_section_blocks` — section-level SSP content for report reconstruction (low)
- [ ] `ssp_roles / ssp_duties / ssp_duty_roles` — role and duty assignment lookups referenced from SSP reports (low)
- [ ] `ssp_vendors (GET /ssp_vendors)` — vendors as recorded in the SSP, distinct from the vendors and tprm/vendors tables (low)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py
index ddbe3c0cd1bf..6d361345ab1d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/canonical_descriptions.py
@@ -359,6 +359,80 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"updated_at": "The date when this comment was updated.",
},
},
+ "trust_center_requests": {
+ "description": "A request from an outside party to get access to documents in the trust center.",
+ "docs_url": "https://developer.secureframe.com/",
+ "columns": {
+ "id": "The identifier for this trust center request.",
+ "company_name": "The company name of the requester.",
+ "created_at": "The date this trust center request was created.",
+ "document_security": "The document security level for this trust center request.",
+ "email": "The email address of the requester.",
+ "job_title": "The job title of the requester.",
+ "reason": "The reason for this trust center request.",
+ "requester_name": "The full name of the requester.",
+ "resources": "The names of the resources requested.",
+ "reviewed": "True if this trust center request was reviewed.",
+ "trust_center_resource_requests": "The trust center resource requests associated with this trust center request.",
+ "updated_at": "The date this trust center request was updated.",
+ },
+ },
+ "ssp_reports": {
+ "description": "A System Security Plan (SSP) report for one framework. It is the parent of the SSP sections, policies and assessment objectives.",
+ "docs_url": "https://developer.secureframe.com/",
+ "columns": {
+ "id": "The identifier for this SSP report.",
+ "completion_progress_percentage": "The completion progress percentage of the SSP report.",
+ "created_at": "The date the SSP report was created.",
+ "framework_id": "The identifier of the associated framework.",
+ "name": "The name of the SSP report.",
+ "sprs_score": "The SPRS score of the SSP report.",
+ "updated_at": "The date the SSP report was last updated.",
+ },
+ },
+ "ssp_report_sections": {
+ "description": "A section of a System Security Plan (SSP) report.",
+ "docs_url": "https://developer.secureframe.com/",
+ "columns": {
+ "id": "The identifier for this SSP report section.",
+ "created_at": "The date when this SSP report section was created.",
+ "description": "The description of the SSP report section.",
+ "position": "The position of the SSP report section.",
+ "section_key": "The section key of the SSP report section.",
+ "ssp_report_id": "The identifier for the associated SSP report.",
+ "status": "The status of the SSP report section.",
+ "title": "The title of the SSP report section.",
+ "updated_at": "The date when this SSP report section was last updated.",
+ },
+ },
+ "ssp_report_section_blocks": {
+ "description": "A content block of a System Security Plan (SSP) report section, such as an input field or a contact list.",
+ "docs_url": "https://developer.secureframe.com/",
+ "columns": {
+ "id": "The identifier for this SSP report section block.",
+ "block_key": "The semantic key identifying what this block represents.",
+ "block_type": "The type of the block (e.g. input, textarea, select, multiselect, contacts).",
+ "child_block_ids": "The ordered IDs of this block's child blocks.",
+ "required": "Whether this block is required for section completion.",
+ "ssp_report_id": "The identifier for the associated SSP report.",
+ "ssp_report_section_id": "The identifier for the associated SSP report section.",
+ "value": "The current value of the block.",
+ },
+ },
+ "ssp_report_assessment_objectives": {
+ "description": "The assessment result of one objective of a framework requirement in a System Security Plan (SSP) report.",
+ "docs_url": "https://developer.secureframe.com/",
+ "columns": {
+ "id": "The identifier for this SSP report assessment objective.",
+ "created_at": "The date this assessment objective was created.",
+ "framework_requirement_id": "The identifier for the framework requirement.",
+ "implementation_statement": "The implementation statement for this assessment objective.",
+ "response": "The response status for this assessment objective.",
+ "ssp_report_id": "The identifier for the associated SSP report.",
+ "ssp_report_requirement_id": "The identifier for the SSP report requirement.",
+ "updated_at": "The date this assessment objective was last updated.",
+ },
+ },
"framework_asset_scopes": {
"description": "The scope of one asset (a device, cloud resource or repository) for one compliance framework.",
"docs_url": "https://developer.secureframe.com/",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py
index 0bd8fefb0704..ae7c441250a9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/settings.py
@@ -103,6 +103,31 @@ SECUREFRAME_ENDPOINTS: dict[str, SecureframeEndpointConfig] = {
path="/comments",
partition_key="created_at",
),
+ "trust_center_requests": SecureframeEndpointConfig(
+ name="trust_center_requests",
+ path="/trust_center_requests",
+ partition_key="created_at",
+ ),
+ "ssp_reports": SecureframeEndpointConfig(
+ name="ssp_reports",
+ path="/ssp_reports",
+ partition_key="created_at",
+ ),
+ "ssp_report_sections": SecureframeEndpointConfig(
+ name="ssp_report_sections",
+ path="/ssp_report_sections",
+ partition_key="created_at",
+ ),
+ # The section block schema has no created_at, so no partitioning.
+ "ssp_report_section_blocks": SecureframeEndpointConfig(
+ name="ssp_report_section_blocks",
+ path="/ssp_report_section_blocks",
+ ),
+ "ssp_report_assessment_objectives": SecureframeEndpointConfig(
+ name="ssp_report_assessment_objectives",
+ path="/ssp_report_assessment_objectives",
+ partition_key="created_at",
+ ),
# Secureframe lists framework scopes only per asset, so this fans out over every device, cloud
# resource and repository. Each row carries its own asset_type and asset_id.
"framework_asset_scopes": SecureframeEndpointConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py
index 72691c7558d6..0b4d259022e1 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/secureframe/tests/test_secureframe.py
@@ -74,16 +74,30 @@ class TestExtractRows:
class TestPagination:
- def test_top_level_array_envelopes_are_flattened(self):
+ @pytest.mark.parametrize(
+ "endpoint, expected_path",
+ [
+ ("controls", "/controls"),
+ ("trust_center_requests", "/trust_center_requests"),
+ ("ssp_reports", "/ssp_reports"),
+ ("ssp_report_sections", "/ssp_report_sections"),
+ ("ssp_report_section_blocks", "/ssp_report_section_blocks"),
+ ("ssp_report_assessment_objectives", "/ssp_report_assessment_objectives"),
+ ],
+ )
+ def test_top_level_array_envelopes_are_flattened(self, endpoint, expected_path):
# The declared shape: a top-level array of per-item JSON:API envelopes.
result = _driver().run(
- "controls",
+ endpoint,
[
- _response([{"data": {"id": "a", "type": "control", "attributes": {"name": "x"}}}]),
+ _response([{"data": {"id": "a", "type": "x", "attributes": {"name": "x"}}}]),
+ _response([{"data": {"id": "b", "type": "x", "attributes": {"name": "y"}}}]),
_response([]),
],
)
- assert result.rows == [{"id": "a", "name": "x"}]
+ assert result.rows == [{"id": "a", "name": "x"}, {"id": "b", "name": "y"}]
+ assert result.paths == [expected_path] * 3
+ assert result.params("page") == ["1", "2", "3"]
def test_resumes_from_saved_page(self):
result = _driver().run(