:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Picqer source (#125425)
Tom Owers1cfcb6285cf4
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 9f2aaeb80d42..452b1777d545 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9401,13 +9401,13 @@ Diffed against: <https://picqer.com/en/api>- [x] `/shipments (also /picklists/{idpicklist}/shipments)` — actual outbound shipments with carrier and tracking; picklists alone can't answer ship-rate or carrier-mix questions (high)- [x] `/warehouses/{idwarehouse}/stock and /products/{idproduct}/stock` — current stock level per product per warehouse; the products table has no per-warehouse quantities (high)- [x] `/return_reasons` — lookup resolving the reason ID carried on the returns we already sync (high)-- [ ] `/return_statuses` — lookup resolving the status ID on synced returns (high)-- [ ] `/backorders` — unfulfillable order lines - the standard stockout/backorder analysis (medium)-- [ ] `/picklists/batches` — batch-picking grouping over picklists; needed for picker throughput and batch efficiency (medium)+- [x] `/return_statuses` — lookup resolving the status ID on synced returns (high)+- [x] `/backorders` — unfulfillable order lines - the standard stockout/backorder analysis (medium)+- [x] `/picklists/batches` — batch-picking grouping over picklists; needed for picker throughput and batch efficiency (medium)- [ ] `/webshoporders` — import status per external webshop order, including failures that never became a Picqer order (medium)- [ ] `/comments` — cross-entity comment log (orders, picklists, returns, purchase orders) usable as an operational activity feed (medium)- [ ] `/shippingproviders` — lookup resolving shipping provider IDs on shipments and orders (medium)-- [ ] `/pricelists` — price list definitions behind customer-specific pricing on orders (medium)+- [x] `/pricelists` — price list definitions behind customer-specific pricing on orders (medium)- [ ] `/productfields, /orderfields, /customerfields` — custom field definitions that name the free-field values embedded in synced products, orders and customers (medium)Note: Full vendor resource list read from the docs navigation plus each resource page. Also present but lower value and not listed above: /tasks, /packagings, /location_types, /templates, /picking-containers, /stats, /products/{id}/pricehistory, /locations/{id}/products.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/canonical_descriptions.pyindex 8e6a6b0f43de..a4b92a5f0048 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/canonical_descriptions.py@@ -37,6 +37,46 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"created": "Datetime the picklist was created.",},},+ "picklist_batches": {+ "description": "A picklist batch, which collects several picklists into a single run through the warehouse.",+ "docs_url": f"{_DOCS}/picklist-batches",+ "columns": {+ "idpicklist_batch": "Unique Picqer reference for the picklist batch.",+ "picklist_batchid": "Per-account batch ID.",+ "title": "Title set for the batch, or null when no title was set.",+ "display_title": "Title as shown in the Picqer interface.",+ "idwarehouse": "Reference to the warehouse the batch is picked in.",+ "idpicklist_batch_preset": "Reference to the batch preset the batch was created from, or null.",+ "type": "Type of batch: singles or normal.",+ "status": "Status of the batch: open or completed.",+ "assigned_to": "The user the batch is assigned to (iduser, full_name, username).",+ "completed_by": "The user who completed the batch (iduser, full_name, username).",+ "total_products": "Total quantity of products in the batch.",+ "total_picklists": "Number of picklists in the batch.",+ "completed_at": "Datetime the batch was completed.",+ "created_at": "Datetime the batch was created.",+ "updated_at": "Datetime the batch was last changed.",+ },+ },+ "backorders": {+ "description": "An order line that cannot be picked yet because there is not enough stock.",+ "docs_url": f"{_DOCS}/backorders",+ "columns": {+ "idbackorder": "Unique Picqer reference for the backorder.",+ "idorder_product": "Reference to the order line in backorder.",+ "idorder": "Reference to the order the backorder belongs to.",+ "idreturn": "Reference to the linked return, if any.",+ "idproduct": "Reference to the product in backorder.",+ "idcustomer": "Reference to the customer of the order.",+ "idwarehouse": "Reference to the warehouse the backorder is waiting on.",+ "amount": "Quantity of the product in backorder.",+ "amount_available": "Quantity in stock that is reserved for this backorder.",+ "priority": "Priority of the backorder. Stock goes to a lower number first.",+ "has_parts": "Whether the backorder is part of a composition.",+ "part_of_idbackorder": "Backorder of the parent product when this backorder is part of a composition.",+ "created_at": "Datetime the backorder was created.",+ },+ },"purchaseorders": {"description": "A purchase order sent to a supplier to replenish stock.","docs_url": f"{_DOCS}/purchaseorders",@@ -222,4 +262,26 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"deleted_at": "Datetime the return reason was deleted, or null.",},},+ "return_statuses": {+ "description": "A status a return can have. Resolves `idreturn_status` on returns.",+ "docs_url": f"{_DOCS}/returns",+ "columns": {+ "idreturn_status": "Unique Picqer reference for the return status.",+ "name": "Name of the return status.",+ "default": "Whether this is the default return status.",+ "completed": "Whether this status marks a return as completed.",+ "color": "Display color of the status, as a hex code.",+ "created_at": "Datetime the return status was created.",+ "updated_at": "Datetime the return status was last changed.",+ "deleted_at": "Datetime the return status was deleted, or null.",+ },+ },+ "pricelists": {+ "description": "A price list, which gives products a different price for specific orders and customers.",+ "docs_url": f"{_DOCS}/pricelists",+ "columns": {+ "idpricelist": "Unique Picqer reference for the price list.",+ "name": "Name of the price list.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/settings.pyindex 1ecd068b5334..7da4d40fded4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/settings.py@@ -79,6 +79,21 @@ PICQER_ENDPOINTS: dict[str, PicqerEndpointConfig] = {primary_keys=["idpicklist"],partition_key="created",),+ # Batches change status as picking completes and expose no time filter.+ "picklist_batches": PicqerEndpointConfig(+ name="picklist_batches",+ path="/picklists/batches",+ primary_keys=["idpicklist_batch"],+ partition_key="created_at",+ ),+ # Backorders change as stock is reserved and disappear once processed, so only a full refresh+ # reflects the current set. `sincedate` filters by creation and would miss both.+ "backorders": PicqerEndpointConfig(+ name="backorders",+ path="/backorders",+ primary_keys=["idbackorder"],+ partition_key="created_at",+ ),"purchaseorders": PicqerEndpointConfig(name="purchaseorders",path="/purchaseorders",@@ -176,6 +191,17 @@ PICQER_ENDPOINTS: dict[str, PicqerEndpointConfig] = {primary_keys=["idreturn_reason"],paginated=False,),+ "return_statuses": PicqerEndpointConfig(+ name="return_statuses",+ path="/return_statuses",+ primary_keys=["idreturn_status"],+ paginated=False,+ ),+ "pricelists": PicqerEndpointConfig(+ name="pricelists",+ path="/pricelists",+ primary_keys=["idpricelist"],+ ),}ENDPOINTS = tuple(PICQER_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/tests/test_picqer.py b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/tests/test_picqer.pyindex e42a0b2f931a..883bd882e607 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/tests/test_picqer.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/picqer/tests/test_picqer.py@@ -108,15 +108,22 @@ class TestPagination:assert result.params(filter_param) == ["2020-01-02 03:04:05"] * 2assert result.params("offset") == ["0", str(PAGE_SIZE)]- def test_unpaginated_endpoint_stops_after_one_full_page(self) -> None:- # return_reasons is not documented as paginated; an offset loop would refetch the same full+ @pytest.mark.parametrize(+ "endpoint,primary_key",+ [+ ("return_reasons", "idreturn_reason"),+ ("return_statuses", "idreturn_status"),+ ],+ )+ def test_unpaginated_endpoint_stops_after_one_full_page(self, endpoint: str, primary_key: str) -> None:+ # These lookups are not documented as paginated; an offset loop would refetch the same full# page forever if the endpoint ignores `offset`.- reasons = [{"idreturn_reason": i, "name": f"Reason {i}"} for i in range(PAGE_SIZE)]- result = _driver().run("return_reasons", route({"/return_reasons": [ScriptedResponse(json=reasons)]}))+ rows = [{primary_key: i, "name": f"Row {i}"} for i in range(PAGE_SIZE)]+ result = _driver().run(endpoint, route({f"/{endpoint}": [ScriptedResponse(json=rows)]}))assert result.raised is Noneassert len(result.rows) == PAGE_SIZE- assert result.paths == ["/api/v1/return_reasons"]+ assert result.paths == [f"/api/v1/{endpoint}"]assert result.params("offset") == [None]