:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(integrations): add the integrations SDK with three auth types and a model-access ratchet (#124844)
Tom Owers005d709492f9
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.mdindex 2980a4fc9a33..017cef242467 100644--- a/.agents/skills/implementing-warehouse-sources/SKILL.md+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md@@ -401,19 +401,37 @@ Rules:- When you are not sure, keep the default (connection). A setting stored as a connection field still works. A credential marked as a sync setting stays in plain source config.- A field in `connection_host_fields` must stay a connection field. The invariant test fails otherwise.- Never set `sync_setting=True` on a field that is `secret=True`, a password or a file upload. The invariant test fails otherwise.-- Every new source and every field change needs the audit CSV regenerated (step 6 of the workflow). The command also rewrites `facade/credential_kinds.py`.+- Every new source and every field change needs the audit CSV regenerated (step 6 of the workflow). The command also rewrites `facade/credential_kinds.py` and `facade/credential_auth_types.py`.- A source that has an OAuth field and an account id field must mark the account id field `account_selector=True`. A test flags the unmarked ones by name.#### Credential kind-Each source has a credential kind: the `Integration` kind that holds its typed-in credentials. It is at most 32 characters. `sources/common/credential_kind.py` derives it in this order:+Each source has a credential kind: the `Integration` kind that holds its typed-in credentials. It is at most 32 characters. The source gives it with the `credential_kind` property on its own class. The default on `BaseSource` is:1. The kind of the source's OAuth field (`SourceFieldOauthConfig.kind`), at any depth.-2. `EXISTING_KIND_OVERRIDES`, for sources that share the shape of an existing kind (`Postgres` is `postgresql`, `AmazonS3` is `aws-s3`).-3. `BRAND_CASING_OVERRIDES`, for one-word brands that kebab-case splits (`MySQL` is `mysql`, not `my-sql`).-4. The kebab-case source type (`AwsCloudTrail` is `aws-cloud-trail`).+2. Else the kebab-case source type (`AwsCloudTrail` is `aws-cloud-trail`).-The result is frozen in the generated file `facade/credential_kinds.py` (`CREDENTIAL_KINDS`), because a kind is stored data. A test fails when the file is out of date, when a kind is longer than 32 characters, or when two source types share a kind. Add an override when a new source needs one of the cases above. The Custom source has no credential kind.+Override `credential_kind` in your `source.py` when the default is wrong:++- The source has the credential shape of an existing `Integration` kind (`PostgresSource` returns `postgresql`, `AmazonS3Source` returns `aws-s3`).+- Kebab-case splits a one-word brand name (`MySQLSource` returns `mysql`, not `my-sql`).+- The source has no credential kind. Return `None` (`CustomSource` does this).++A subclass inherits the override of its parent, so a subclass of an overriding source sets its own kind (`NeonSource` returns `neon`, not the `postgresql` of `PostgresSource`).++The result is frozen in the generated file `facade/credential_kinds.py` (`CREDENTIAL_KINDS`), because a kind is stored data. Never change the kind of a released source. A test fails when the file is out of date, when a kind is longer than 32 characters, or when two source types share a kind.++Each credential kind also has a default auth type: the `credential_auth_type` property on the source class. The default on `BaseSource` (`derive_credential_auth_type` in `sources/common/credential_auth_type.py`) reads the connection fields:++- `oauth`: the source has an OAuth field and no typed-in connection field.+- `token`: the source has exactly one typed-in field, and it is a secret.+- `credentials`: every other source.++Override `credential_auth_type` in your `source.py` only when this rule gives the wrong type for the source.++The result is frozen in `facade/credential_auth_types.py` (`CREDENTIAL_AUTH_TYPES`), because a row stores its auth type only when it is not the default. The product registers these kinds with the integrations SDK at startup. Read and write their rows with `IntegrationManager.source(kind)` from `posthog.integrations`. A kind that is also in the integrations catalogue (`stripe`, `postgresql`) keeps the default of the catalogue.++Both generated files are append-only. A new source adds its entry. A removed source keeps its entry, because rows of its kind can exist. When a change to a source would change a committed kind or default auth type, `generate_source_field_classification` fails and writes nothing. Keep the old value with an override in the source class. If the change is necessary, write a data migration for the stored rows first, then rerun the command with `--allow-changed-defaults`.#### Reading and writing `job_inputs`@@ -809,6 +827,9 @@ If new:2. **Integration kind**.- Add to the `IntegrationKind` enum in `posthog/models/integration/model.py`.+ - Add the kind to `KINDS` in `posthog/integrations/catalogue.py`, with `default=_OAUTH`, then run `hogli build:projections --only integration-kinds`.+ Read and write its rows through the integrations SDK, for example `IntegrationManager.your_source.oauth.get(team_id, integration_id)`.+ See `posthog/integrations/README.md`.- Add to `OauthIntegration.supported_kinds` in `posthog/models/integration/oauth.py`.- Add an `elif kind == "your-source": return OauthConfig(...)` branch in `oauth_config_for_kind()`, also in `oauth.py`.Raise `NotImplementedError("<Source> app not configured")` when the env vars are empty — that's thediff --git a/.github/workflows/ci-python.yml b/.github/workflows/ci-python.ymlindex 689f3cba1482..38c96772b9d3 100644--- a/.github/workflows/ci-python.yml+++ b/.github/workflows/ci-python.yml@@ -509,6 +509,12 @@ jobs:[ "$RUN_BIN_SCRIPTS" = "true" ] || { echo "bin/ unchanged — skipping"; exit 0; }pytest bin/test/test_find_python_dependencies.py -v --junitxml=junit-dependency-detection.xml+ - name: Run semgrep ratchet script tests+ shell: bash+ run: |+ [ "$RUN_BIN_SCRIPTS" = "true" ] || { echo "bin/ unchanged — skipping"; exit 0; }+ pytest bin/test/test_semgrep_ratchet.py -v+- name: Run granian shared socket testsshell: bash# The bin_scripts filter includes uv.lock, so every granian upgrade runs this.diff --git a/.github/workflows/ci-security.yaml b/.github/workflows/ci-security.yamlindex fa15071100b3..11182df34b52 100644--- a/.github/workflows/ci-security.yaml+++ b/.github/workflows/ci-security.yaml@@ -486,6 +486,30 @@ jobs:--timeout 300 \bin/ common/ docs/onboarding/ ee/ frontend/ packages/ posthog/ products/ services/ tools/+ # Per-file ratchet: the INFO rule integration-model-through-sdk has a backlog, and its+ # baseline file lists the allowed count per file. More findings than the baseline fails, and+ # so do fewer, so each migration lowers the baseline. See .semgrep/rules/devex/README.md.+ # Skip on unrebased branches: the rule may not exist there yet.+ - name: Integration model ratchet (blocking)+ run: |+ if [ ! -f .semgrep/rules/devex/integration-model-through-sdk.yaml ]; then+ echo "integration-model-through-sdk not present on this branch yet, skipping (rebase to pick it up)"+ exit 0+ fi+ docker run --rm -v "${{ github.workspace }}:/src" -w /src \+ -e SEMGREP_ENABLE_VERSION_CHECK=false \+ ${{ env.SEMGREP_IMAGE }} \+ semgrep \+ --config ".semgrep/rules/devex/integration-model-through-sdk.yaml" \+ --json \+ --output semgrep-integration-model.json \+ --metrics=off \+ --timeout 300 \+ common/ ee/ posthog/ products/+ python3 bin/semgrep_ratchet.py \+ --findings semgrep-integration-model.json \+ --baseline posthog/integrations/model_access_baseline.txt+# Pass 3 (PRs only): fail on newly-introduced WARNING findings.# --baseline-commit diffs against the PR base (first parent of the# merge commit), so the pass-1 backlog is grandfathered while newdiff --git a/.semgrep/rules/devex/README.md b/.semgrep/rules/devex/README.mdindex ea5e1b97a48e..1a9830900138 100644--- a/.semgrep/rules/devex/README.md+++ b/.semgrep/rules/devex/README.md@@ -25,3 +25,18 @@ A WARNING rule typically gets flipped to ERROR once its backlog hits zero. The r- Include the migration playbook (or a pointer to a skill) in the rule's `message:` field — that's what reviewers and authors see when the rule fires.- Scope with `paths.include:` rather than relying only on `paths.exclude:` — narrower scope means faster scans and clearer intent.- For legitimate exceptions, prefer `paths.exclude:` for categorical cases (whole dirs/files) and `# nosemgrep: <rule-id>` for case-by-case exemptions at the call site.++## Per-file baseline ratchets++A WARNING rule grandfathers its backlog through `--baseline-commit`, so the backlog has no count that a migration can lower, and a moved file looks new.+For a migration that runs over many PRs, use a per-file baseline instead:++1. Give the rule `severity: INFO`. The three passes above select WARNING and ERROR only, so they do not run it.+2. Check in a baseline file that lists `<path> <count>` for each file with findings. `bin/semgrep_ratchet.py --update` writes it from `semgrep --json` output.+3. Add a step to the `semgrep-devex` job that runs the rule with `--json` and then `bin/semgrep_ratchet.py --findings <json> --baseline <file>`.++The ratchet fails when a file has more findings than its baseline count, and also when it has fewer. So each migration PR lowers the count or removes the line.++| Rule | Baseline |+| ------------------------------- | ------------------------------------------------ |+| `integration-model-through-sdk` | `posthog/integrations/model_access_baseline.txt` |diff --git a/.semgrep/rules/devex/github-installation-first-team.py b/.semgrep/rules/devex/github-installation-first-team.pyindex 7ccaa52e4211..cf52e1434f7c 100644--- a/.semgrep/rules/devex/github-installation-first-team.py+++ b/.semgrep/rules/devex/github-installation-first-team.py@@ -101,3 +101,43 @@ def every_integration(installation_id):def justified_exception(installation_id):# nosemgrep: github-installation-first-team -- the installation has one project by construction herereturn Integration.objects.filter(kind="github", integration_id=installation_id).first()+++def first_sdk_row(installation_id):+ # ruleid: github-installation-first-team+ return IntegrationManager.github.oauth.list_by_external_id(str(installation_id))[0]+++def first_sdk_row_through_next(installation_id):+ # ruleid: github-installation-first-team+ return next(iter(IntegrationManager.kinds("github").list_by_external_id(str(installation_id))), None)+++def first_sdk_row_through_variable(installation_id):+ rows = IntegrationManager.github.oauth.list_by_external_id(str(installation_id))+ if not rows:+ return None+ # ruleid: github-installation-first-team+ return rows[0]+++def first_sdk_row_of_some_teams(installation_id, team_ids):+ # ok: github-installation-first-team+ return IntegrationManager.github.oauth.list_by_external_id(str(installation_id), team_ids=team_ids)[0]+++def first_sdk_row_of_some_teams_through_variable(installation_id, team_ids):+ rows = IntegrationManager.github.oauth.list_by_external_id(str(installation_id), team_ids=team_ids)+ # ok: github-installation-first-team+ return next(iter(rows), None)+++def every_sdk_row(installation_id):+ # ok: github-installation-first-team+ for row in IntegrationManager.github.oauth.list_by_external_id(str(installation_id)):+ notify(row.team_id)+++def first_slack_sdk_row(slack_team_id):+ # ok: github-installation-first-team+ return IntegrationManager.slack.oauth.list_by_external_id(slack_team_id)[0]diff --git a/.semgrep/rules/devex/github-installation-first-team.yaml b/.semgrep/rules/devex/github-installation-first-team.yamlindex b12ec7b33cd6..db2ab0e1d3da 100644--- a/.semgrep/rules/devex/github-installation-first-team.yaml+++ b/.semgrep/rules/devex/github-installation-first-team.yaml@@ -52,6 +52,26 @@ rules:- pattern-not: Integration.objects. ... .get(..., team=$TEAM, ...)- pattern-not: Integration.objects. ... .filter(..., team_id=$TEAM, ...). ... [$INDEX]- pattern-not: Integration.objects. ... .filter(..., team=$TEAM, ...). ... [$INDEX]+ # One GitHub row out of an SDK installation lookup that no project filter narrows.+ - patterns:+ - pattern-either:+ - pattern: $ACCESSOR.list_by_external_id(...)[$INDEX]+ - pattern: next(iter($ACCESSOR.list_by_external_id(...)), ...)+ - patterns:+ - pattern-inside: |+ $ROWS = $ACCESSOR.list_by_external_id(...)+ ...+ - pattern-not-inside: |+ $ROWS = $ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)+ ...+ - pattern-either:+ - pattern: $ROWS[$INDEX]+ - pattern: next(iter($ROWS), ...)+ - metavariable-regex:+ metavariable: $ACCESSOR+ regex: (?i).*github.*+ - pattern-not: $ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)[$INDEX]+ - pattern-not: next(iter($ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)), ...)paths:include:- ee/**/*.pydiff --git a/.semgrep/rules/devex/integration-model-through-sdk.py b/.semgrep/rules/devex/integration-model-through-sdk.pynew file mode 100644index 000000000000..c814f771ff38--- /dev/null+++ b/.semgrep/rules/devex/integration-model-through-sdk.py@@ -0,0 +1,192 @@+# Test cases for integration-model-through-sdk.+# ruff: noqa++# ruleid: integration-model-through-sdk+from posthog.models import Integration++# ruleid: integration-model-through-sdk+from posthog.models import Integration, Team++# ruleid: integration-model-through-sdk+from posthog.models.integration import Integration as TeamIntegration++# ruleid: integration-model-through-sdk+from posthog.models.integration.model import Integration++# ruleid: integration-model-through-sdk+import posthog.models.integration.model++# ok: integration-model-through-sdk+from posthog.models.integration import SlackIntegration++# ok: integration-model-through-sdk+from posthog.integrations import IntegrationManager, IntegrationRecord++from django.apps import apps+from django.db import models+from django.db.models import F, Q+from django.db.models import ForeignKey++team_id = 1++# ruleid: integration-model-through-sdk+Integration.objects.filter(team_id=team_id, kind="github").first()++# ruleid: integration-model-through-sdk+TeamIntegration.objects.get(id=1)++# ruleid: integration-model-through-sdk+manager = posthog.models.integration.model.Integration.objects++# ruleid: integration-model-through-sdk+IntegrationRecord.objects.filter(team_id=team_id)++# ruleid: integration-model-through-sdk+facade_api.IntegrationRecord.objects.get(id=1)++# ruleid: integration-model-through-sdk+Integration._base_manager.filter(id=1)++# ruleid: integration-model-through-sdk+IntegrationRecord._default_manager.all()++# ruleid: integration-model-through-sdk+posthog.models.Integration._default_manager.all()++# ruleid: integration-model-through-sdk+TeamIntegration._base_manager.get(id=1)++# ok: integration-model-through-sdk+SlackThread._default_manager.all()++# ruleid: integration-model-through-sdk+model = apps.get_model("posthog", "Integration")++# ruleid: integration-model-through-sdk+model = apps.get_model("posthog.Integration")++# ruleid: integration-model-through-sdk+model = apps.get_model("posthog.integration")++# ok: integration-model-through-sdk+model = apps.get_model("posthog", "Team")++# ok: integration-model-through-sdk+model = apps.get_model("posthog", "UserIntegration")+++class SlackThread(models.Model):+ # ruleid: integration-model-through-sdk+ integration = models.ForeignKey(Integration, on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ other = models.ForeignKey("posthog.Integration", on_delete=models.SET_NULL, null=True)+ # ruleid: integration-model-through-sdk+ third = models.ForeignKey(to="posthog.Integration", on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ fourth = models.OneToOneField("posthog.Integration", on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ fifth = models.ForeignKey(IntegrationRecord, on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ sixth = models.ForeignKey("posthog.integration", on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ seventh = models.ManyToManyField("posthog.Integration", related_name="+")+ # ruleid: integration-model-through-sdk+ eighth = models.ManyToManyField(to=Integration, related_name="+")+ # ruleid: integration-model-through-sdk+ ninth = models.OneToOneField(to=IntegrationRecord, on_delete=models.CASCADE)+ # ruleid: integration-model-through-sdk+ tenth = ForeignKey("POSTHOG.INTEGRATION", on_delete=models.CASCADE)+ # ok: integration-model-through-sdk+ user_link = models.ForeignKey("posthog.UserIntegration", on_delete=models.CASCADE)+ # ok: integration-model-through-sdk+ org_links = models.ManyToManyField("posthog.OrganizationIntegration")+ # ok: integration-model-through-sdk+ integration_id_only = models.BigIntegerField(null=True)+ # ok: integration-model-through-sdk+ team = models.ForeignKey("posthog.Team", on_delete=models.CASCADE)+++# ruleid: integration-model-through-sdk+SlackThread.objects.select_related("integration").all()++# ruleid: integration-model-through-sdk+SlackThread.objects.select_related("team", "slack_integration").all()++# ruleid: integration-model-through-sdk+SlackThread.objects.prefetch_related("integration__created_by").all()++# ok: integration-model-through-sdk+SlackThread.objects.select_related("team", "created_by").all()++# ok: integration-model-through-sdk+SlackThread.objects.select_related("github_user_integration").all()++# ruleid: integration-model-through-sdk+SlackThread.objects.filter(integration__kind="slack")++# ruleid: integration-model-through-sdk+SlackThread.objects.filter(team_id=team_id, slack_integration__config__team_id="T1")++# ruleid: integration-model-through-sdk+SlackThread.objects.exclude(Q(thread__integration__kind="slack"))++# ok: integration-model-through-sdk+SlackThread.objects.filter(integration_id=3)++# ok: integration-model-through-sdk+SlackThread.objects.filter(team_id=team_id)++# ok: integration-model-through-sdk+SlackThread.objects.filter(github_user_integration__isnull=True)++# ok: integration-model-through-sdk+SlackThread.objects.filter(organization_integration__kind="vercel")++# ruleid: integration-model-through-sdk+SlackThread.objects.values("id", "integration__kind")++# ruleid: integration-model-through-sdk+SlackThread.objects.values_list("slack_integration__config", flat=True)++# ruleid: integration-model-through-sdk+SlackThread.objects.order_by("-integration__created_at")++# ruleid: integration-model-through-sdk+SlackThread.objects.annotate(kind=F("thread__integration__kind"))++# ruleid: integration-model-through-sdk+SlackThread.objects.filter(**{"integration__kind": "slack"})++# ruleid: integration-model-through-sdk+SlackThread.objects.exclude(Q(**{"integration__config__team_id": "T1"}))++# ok: integration-model-through-sdk+SlackThread.objects.values("integration_id", "team__name")++# ok: integration-model-through-sdk+SlackThread.objects.order_by("-user_integration__created_at")++# ok: integration-model-through-sdk+SlackThread.objects.values_list("organization_integration__kind", flat=True)++# ok: integration-model-through-sdk+SlackThread.objects.filter(**{"integration_id": 3})+++def handler(self, user, thread, integration, slack_integration, org_integration):+ # ruleid: integration-model-through-sdk+ user.integration_set.all()+ # ruleid: integration-model-through-sdk+ integration.save()+ # ruleid: integration-model-through-sdk+ thread.integration.save(update_fields=["config"])+ # ruleid: integration-model-through-sdk+ slack_integration.save()+ # ok: integration-model-through-sdk+ thread.save()+ # ok: integration-model-through-sdk+ org_integration.save()+ # ok: integration-model-through-sdk+ self.integration.save(update_fields=["config"])+ # ok: integration-model-through-sdk+ IntegrationManager.github.oauth.update(team_id, 1, config_patch={"a": 1})diff --git a/.semgrep/rules/devex/integration-model-through-sdk.yaml b/.semgrep/rules/devex/integration-model-through-sdk.yamlnew file mode 100644index 000000000000..c63c5705cd90--- /dev/null+++ b/.semgrep/rules/devex/integration-model-through-sdk.yaml@@ -0,0 +1,130 @@+rules:+ # Severity INFO keeps this rule out of the three semgrep-devex passes, which select WARNING and+ # ERROR. The "Integration model ratchet" step of that job runs it on its own and compares the+ # findings per file with posthog/integrations/model_access_baseline.txt.+ - id: integration-model-through-sdk+ message: |+ Read and write integration rows through the integrations SDK, not through the `Integration` model:++ from posthog.integrations import IntegrationManager, IntegrationRecord++ IntegrationManager.github.oauth.get(team_id, integration_id)+ IntegrationManager.stripe.any.list(team_id)+ IntegrationManager.any.get(team_id, integration_id)++ The integration rows move to a separate service. The SDK is the closed set of operations that+ keeps working after the move, so code outside it must not import the model, use one of its+ managers (`Integration.objects`, `_default_manager`, `_base_manager`), load it with+ `apps.get_model`, add a relation field to it, join it with `select_related`, `prefetch_related`+ or an `integration__` lookup, or call `.save()` on a row. For a DRF field, use+ `IntegrationReferenceField` from `posthog/integrations/fields.py`.+ See `posthog/integrations/README.md`.++ After you move a file to the SDK, lower its count in+ `posthog/integrations/model_access_baseline.txt`, or remove the line at zero.+ severity: INFO+ languages: [python]+ paths:+ include:+ - 'common/**/*.py'+ - 'ee/**/*.py'+ - 'posthog/**/*.py'+ - 'products/**/*.py'+ exclude:+ - 'products/desktop/**'+ - '**/test/**'+ - '**/tests/**'+ - '**/test_*.py'+ - '**/*_test.py'+ - '**/conftest*.py'+ - '**/migrations/**'+ # The model, its per-kind adapters and the SDK are the implementation.+ - 'posthog/models/integration/**'+ - 'posthog/models/github_integration_base.py'+ - 'posthog/integrations/**'+ # Django admin works on the model by design.+ - 'posthog/admin/**'+ - 'products/*/backend/admin.py'+ - 'products/*/backend/admin/**'+ pattern-either:+ # The model class, imported by name or reached through its module.+ - pattern: from posthog.models import Integration+ - pattern: from posthog.models.integration import Integration+ - pattern: from posthog.models.integration.model import Integration+ - pattern: import posthog.models.integration.model+ # A manager of the model. `IntegrationRecord` is the same class while the SDK returns models.+ # `$MODULE.Integration` also matches an import alias such as `Integration as TeamIntegration`,+ # because semgrep resolves the alias to its module path.+ - pattern: Integration.objects+ - pattern: $MODULE.Integration.objects+ - pattern: $MODULE.Integration._base_manager+ - pattern: $MODULE.Integration._default_manager+ - patterns:+ - pattern-either:+ - pattern: $MODEL.objects+ - pattern: $MODEL._base_manager+ - pattern: $MODEL._default_manager+ - metavariable-regex:+ metavariable: $MODEL+ regex: ^(.*\.)?(Integration|IntegrationRecord)$+ # The model class looked up by its label.+ - patterns:+ - pattern-either:+ - pattern: $APPS.get_model($APP, "$NAME")+ - pattern: $APPS.get_model("$NAME")+ - metavariable-regex:+ metavariable: $NAME+ regex: ^(?i:(posthog\.)?integration)$+ # A relation field to the model: the class, or its label in any letter case.+ - patterns:+ - pattern-either:+ - pattern: $FIELD($TARGET, ...)+ - pattern: $FIELD(..., to=$TARGET, ...)+ - metavariable-regex:+ metavariable: $FIELD+ regex: ^(\w+\.)*(ForeignKey|OneToOneField|ManyToManyField)$+ - metavariable-regex:+ metavariable: $TARGET+ regex: ^((\w+\.)*(Integration|IntegrationRecord)|["'](?i:posthog\.integration)["'])$+ # A join through an integration relation.+ - patterns:+ - pattern-either:+ - pattern: $QUERYSET.select_related(..., "$RELATION", ...)+ - pattern: $QUERYSET.prefetch_related(..., "$RELATION", ...)+ # `user_integration` and `organization_integration` name other models.+ - metavariable-regex:+ metavariable: $RELATION+ regex: ^(?!(\w+_)?(user|org|organization)_integration)(\w+_)?integration(_set)?(__.*)?$+ # A lookup through an integration relation, such as `filter(integration__kind="slack")`.+ - patterns:+ - pattern-either:+ - pattern: $QUERYSET.filter(...)+ - pattern: $QUERYSET.exclude(...)+ - pattern: $QUERYSET.get(...)+ - pattern: $QUERYSET.update(...)+ - pattern: Q(...)+ - pattern-regex: \b(?!\w*(user|org|organization)_integration__)(\w+__)?(\w+_)?integration__\w+\s*=+ # The same lookup as a string, such as `values("integration__kind")` or `filter(**{"integration__kind": k})`.+ - patterns:+ - pattern-either:+ - pattern: $QUERYSET.filter(...)+ - pattern: $QUERYSET.exclude(...)+ - pattern: $QUERYSET.get(...)+ - pattern: $QUERYSET.values(...)+ - pattern: $QUERYSET.values_list(...)+ - pattern: $QUERYSET.order_by(...)+ - pattern: F(...)+ - pattern: Q(...)+ - pattern-regex: '["'']-?(\w+__)*(?!(\w+_)?(user|org|organization)_integration__)(\w+_)?integration__\w+["'']'+ - pattern: $OBJECT.integration_set+ # A write to a row outside the SDK. Names only: `self.integration` and `org_integration` are+ # often a UserIntegration or an OrganizationIntegration.+ - patterns:+ - pattern: $ROW.save(...)+ - metavariable-regex:+ metavariable: $ROW+ regex: ^(?!self\.)(?!(.*\.)?(user|org|organization)_integration$)(.*\.)?(\w+_)?integration$+ metadata:+ category: best-practice+ subcategory:+ - auditdiff --git a/bin/semgrep_ratchet.py b/bin/semgrep_ratchet.pynew file mode 100755index 000000000000..f939a3226795--- /dev/null+++ b/bin/semgrep_ratchet.py@@ -0,0 +1,140 @@+#!/usr/bin/env python3+# ruff: noqa: T201 allow print statements+"""Hold the findings of one semgrep rule to a checked-in, per-file baseline.++The baseline lists each file that still has findings, with its finding count. The check fails+when a file has more findings than its baseline count (new debt), and also when it has fewer+(the baseline is stale). So the baseline only goes down, and each migration lowers it.++A file that semgrep reports an error for (a timeout, a parse error) can have missing findings, so+the check does not call its line stale, and `--update` does not lower its line.++ semgrep --config <rule>.yaml --json --output findings.json <paths>+ bin/semgrep_ratchet.py --findings findings.json --baseline <baseline>.txt+ bin/semgrep_ratchet.py --findings findings.json --baseline <baseline>.txt --update+"""++import sys+import json+import argparse+from collections import Counter+from dataclasses import dataclass, field+from pathlib import Path+++@dataclass(frozen=True)+class RatchetResult:+ over: dict[str, tuple[int, int]] = field(default_factory=dict)+ stale: dict[str, tuple[int, int]] = field(default_factory=dict)+ # Fewer findings than the baseline in a file with a semgrep error. Not a failure.+ unchecked: dict[str, tuple[int, int]] = field(default_factory=dict)++ @property+ def ok(self) -> bool:+ return not self.over and not self.stale+++def count_findings(semgrep_output: dict) -> Counter[str]:+ return Counter(result["path"] for result in semgrep_output.get("results", []))+++def error_paths(semgrep_output: dict) -> frozenset[str]:+ paths: set[str] = set()+ for error in semgrep_output.get("errors", []):+ if error.get("path"):+ paths.add(error["path"])+ paths.update(span["file"] for span in error.get("spans") or [] if span.get("file"))+ return frozenset(paths)+++def parse_baseline(text: str) -> dict[str, int]:+ baseline: dict[str, int] = {}+ for line_number, raw_line in enumerate(text.splitlines(), start=1):+ line = raw_line.strip()+ if not line or line.startswith("#"):+ continue+ path, _, count = line.rpartition(" ")+ if not path or not count.isdigit() or int(count) < 1:+ raise ValueError(f"Baseline line {line_number} is not '<path> <count>': {raw_line!r}")+ if path in baseline:+ raise ValueError(f"Baseline line {line_number} repeats {path}")+ baseline[path] = int(count)+ return baseline+++def compare(+ findings: Counter[str], baseline: dict[str, int], incomplete: frozenset[str] = frozenset()+) -> RatchetResult:+ over: dict[str, tuple[int, int]] = {}+ stale: dict[str, tuple[int, int]] = {}+ unchecked: dict[str, tuple[int, int]] = {}+ for path in sorted(set(findings) | set(baseline)):+ found, allowed = findings.get(path, 0), baseline.get(path, 0)+ if found > allowed:+ over[path] = (found, allowed)+ elif found < allowed:+ (unchecked if path in incomplete else stale)[path] = (found, allowed)+ return RatchetResult(over=over, stale=stale, unchecked=unchecked)+++def updated_findings(findings: Counter[str], baseline: dict[str, int], incomplete: frozenset[str]) -> Counter[str]:+ result = Counter(findings)+ for path in incomplete:+ if baseline.get(path, 0) > result.get(path, 0):+ result[path] = baseline[path]+ return result+++def render_baseline(header: str, findings: Counter[str]) -> str:+ lines = [line for line in header.splitlines() if line.startswith("#")]+ lines += [f"{path} {count}" for path, count in sorted(findings.items())]+ return "\n".join(lines) + "\n"+++def main(argv: list[str] | None = None) -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--findings", type=Path, required=True, help="semgrep --json output")+ parser.add_argument("--baseline", type=Path, required=True, help="the checked-in baseline file")+ parser.add_argument("--update", action="store_true", help="rewrite the baseline from the findings")+ args = parser.parse_args(argv)++ semgrep_output = json.loads(args.findings.read_text())+ incomplete = error_paths(semgrep_output)+ if semgrep_output.get("errors"):+ print(f"semgrep reported {len(semgrep_output['errors'])} errors. The counts can be incomplete.")+ findings = count_findings(semgrep_output)+ baseline_text = args.baseline.read_text() if args.baseline.exists() else ""+ baseline = parse_baseline(baseline_text)++ if args.update:+ findings = updated_findings(findings, baseline, incomplete)+ args.baseline.write_text(render_baseline(baseline_text, findings))+ print(f"Wrote {sum(findings.values())} findings in {len(findings)} files to {args.baseline}")+ return 0++ result = compare(findings, baseline, incomplete)+ for path, (found, allowed) in result.unchecked.items():+ print(+ f"Warning: semgrep reported an error for {path}, so its {found} findings (baseline: {allowed}) are not checked."+ )+ if result.ok:+ print(f"{sum(findings.values())} findings in {len(findings)} files. They match {args.baseline}.")+ return 0++ if result.over:+ print(f"New findings. Fix them, or add `# nosemgrep: <rule-id>` with a reason. Do not raise {args.baseline}.")+ new_paths = set(result.over)+ for item in semgrep_output.get("results", []):+ if item["path"] in new_paths:+ print(f" {item['path']}:{item['start']['line']}: {item.get('check_id', '')}")+ for path, (found, allowed) in result.over.items():+ print(f" {path}: {found} findings, baseline allows {allowed}")+ if result.stale:+ print(f"Fewer findings than {args.baseline} allows. Lower these lines (remove a line at 0):")+ for path, (found, allowed) in result.stale.items():+ print(f" {path} {found} (baseline: {allowed})")+ return 1+++if __name__ == "__main__":+ sys.exit(main())diff --git a/bin/test/test_semgrep_ratchet.py b/bin/test/test_semgrep_ratchet.pynew file mode 100644index 000000000000..0ccca56f3c03--- /dev/null+++ b/bin/test/test_semgrep_ratchet.py@@ -0,0 +1,147 @@+#!/usr/bin/env python3++import json+import tempfile+from collections import Counter+from pathlib import Path++import unittest++from parameterized import parameterized++from bin.semgrep_ratchet import compare, error_paths, main, parse_baseline, render_baseline+++def semgrep_output(*paths: str, errors: list[dict] | None = None) -> dict:+ return {+ "results": [{"path": path, "check_id": "rule", "start": {"line": 1}} for path in paths],+ "errors": errors or [],+ }+++TIMEOUT = {"type": "Timeout", "path": "a.py"}+++class TestSemgrepRatchet(unittest.TestCase):+ @parameterized.expand(+ [+ ("matches", Counter({"a.py": 2}), {"a.py": 2}, {}, {}),+ ("no_findings_no_baseline", Counter(), {}, {}, {}),+ ("new_file", Counter({"a.py": 1, "b.py": 1}), {"a.py": 1}, {"b.py": (1, 0)}, {}),+ ("more_in_a_listed_file", Counter({"a.py": 3}), {"a.py": 2}, {"a.py": (3, 2)}, {}),+ ("fewer_in_a_listed_file", Counter({"a.py": 1}), {"a.py": 2}, {}, {"a.py": (1, 2)}),+ ("file_fully_migrated", Counter(), {"a.py": 2}, {}, {"a.py": (0, 2)}),+ (+ "moved_file",+ Counter({"new/a.py": 2}),+ {"old/a.py": 2},+ {"new/a.py": (2, 0)},+ {"old/a.py": (0, 2)},+ ),+ ]+ )+ def test_compare(+ self,+ _name: str,+ findings: Counter[str],+ baseline: dict[str, int],+ expected_over: dict[str, tuple[int, int]],+ expected_stale: dict[str, tuple[int, int]],+ ) -> None:+ result = compare(findings, baseline)++ assert (result.over, result.stale, result.unchecked) == (expected_over, expected_stale, {})++ @parameterized.expand(+ [+ ("fewer", Counter({"a.py": 1}), {"a.py": 2}, {}, {}, {"a.py": (1, 2)}),+ ("none", Counter(), {"a.py": 2}, {}, {}, {"a.py": (0, 2)}),+ ("more_still_fails", Counter({"a.py": 3}), {"a.py": 2}, {"a.py": (3, 2)}, {}, {}),+ ("other_file_still_stale", Counter({"a.py": 2}), {"a.py": 2, "b.py": 2}, {}, {"b.py": (0, 2)}, {}),+ ]+ )+ def test_compare_with_an_error_in_a_file(+ self,+ _name: str,+ findings: Counter[str],+ baseline: dict[str, int],+ expected_over: dict[str, tuple[int, int]],+ expected_stale: dict[str, tuple[int, int]],+ expected_unchecked: dict[str, tuple[int, int]],+ ) -> None:+ result = compare(findings, baseline, incomplete=frozenset({"a.py"}))++ assert (result.over, result.stale, result.unchecked) == (expected_over, expected_stale, expected_unchecked)++ def test_error_paths_reads_the_path_and_the_spans(self) -> None:+ output = semgrep_output(+ errors=[TIMEOUT, {"type": "Syntax error", "spans": [{"file": "b.py"}, {"file": "c.py"}]}, {"type": "x"}]+ )++ assert error_paths(output) == frozenset({"a.py", "b.py", "c.py"})++ @parameterized.expand(+ [+ ("no_count", "a.py\n"),+ ("zero_count", "a.py 0\n"),+ ("not_a_number", "a.py two\n"),+ ("repeated_path", "a.py 1\na.py 2\n"),+ ]+ )+ def test_parse_baseline_rejects_bad_lines(self, _name: str, text: str) -> None:+ with self.assertRaises(ValueError):+ parse_baseline(text)++ def test_render_keeps_the_header_and_round_trips(self) -> None:+ text = render_baseline("# header\n#\nold.py 9\n", Counter({"b.py": 1, "a b/c.py": 2}))++ assert text == "# header\n#\na b/c.py 2\nb.py 1\n"+ assert parse_baseline(text) == {"a b/c.py": 2, "b.py": 1}++ @parameterized.expand(+ [+ ("matches", ("a.py", "a.py"), "a.py 2\n", 0),+ ("new_finding", ("a.py", "a.py", "b.py"), "a.py 2\n", 1),+ ("stale_entry", ("a.py",), "a.py 2\n", 1),+ ("stale_entry_with_an_error", ("a.py",), "a.py 2\n", 0, [TIMEOUT]),+ ("new_finding_with_an_error", ("a.py", "a.py", "a.py"), "a.py 2\n", 1, [TIMEOUT]),+ ]+ )+ def test_main_exit_code(+ self, _name: str, paths: tuple[str, ...], baseline: str, expected: int, errors: list[dict] | None = None+ ) -> None:+ with tempfile.TemporaryDirectory() as directory:+ findings_file = Path(directory) / "findings.json"+ baseline_file = Path(directory) / "baseline.txt"+ findings_file.write_text(json.dumps(semgrep_output(*paths, errors=errors)))+ baseline_file.write_text(baseline)++ exit_code = main(["--findings", str(findings_file), "--baseline", str(baseline_file)])++ assert exit_code == expected++ def test_main_update_writes_a_baseline_that_then_passes(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ findings_file = Path(directory) / "findings.json"+ baseline_file = Path(directory) / "baseline.txt"+ findings_file.write_text(json.dumps(semgrep_output("a.py", "b.py", "a.py")))+ baseline_file.write_text("# header\nold.py 3\n")+ args = ["--findings", str(findings_file), "--baseline", str(baseline_file)]++ update_exit_code = main([*args, "--update"])+ check_exit_code = main(args)+ written = baseline_file.read_text()++ assert (update_exit_code, check_exit_code, written) == (0, 0, "# header\na.py 2\nb.py 1\n")++ def test_main_update_keeps_the_count_of_a_file_with_an_error(self) -> None:+ with tempfile.TemporaryDirectory() as directory:+ findings_file = Path(directory) / "findings.json"+ baseline_file = Path(directory) / "baseline.txt"+ findings_file.write_text(json.dumps(semgrep_output("b.py", errors=[TIMEOUT])))+ baseline_file.write_text("a.py 3\nb.py 2\n")++ main(["--findings", str(findings_file), "--baseline", str(baseline_file), "--update"])+ written = baseline_file.read_text()++ assert written == "a.py 3\nb.py 1\n"diff --git a/hogli.yaml b/hogli.yamlindex 474be0401b02..20ba6062933f 100644--- a/hogli.yaml+++ b/hogli.yaml@@ -1303,3 +1303,7 @@ environment:bin_script: rasterize-recording-filedescription: 'Render a recording JSONL file to video with the rasterizer'hidden: true+ lint:semgrep-ratchet:+ bin_script: semgrep_ratchet.py+ description: 'Compare semgrep findings per file with a checked-in baseline'+ hidden: truediff --git a/posthog/admin/admins/integration_admin.py b/posthog/admin/admins/integration_admin.pyindex d2f77dcf4962..13166cdf4b44 100644--- a/posthog/admin/admins/integration_admin.py+++ b/posthog/admin/admins/integration_admin.py@@ -1,7 +1,9 @@from django.contrib import admin+from django.db.models import QuerySet+from django.http import HttpRequestfrom django.utils.html import format_html-from posthog.models.integration import Integration+from posthog.models.integration import ANY, Integrationfrom posthog.models.organization_integration import OrganizationIntegrationfrom posthog.models.user_integration import UserIntegration@@ -9,7 +11,7 @@ from posthog.models.user_integration import UserIntegration@admin.register(Integration)class IntegrationAdmin(admin.ModelAdmin):list_select_related = ("team", "created_by")- list_display = ("id", "kind", "integration_id", "team_link", "created_by", "created_at")+ list_display = ("id", "kind", "auth_type", "integration_id", "team_link", "created_by", "created_at")list_display_links = ("id",)list_filter = ("kind",@@ -34,6 +36,12 @@ class IntegrationAdmin(admin.ModelAdmin):),]+ def get_queryset(self, request: HttpRequest) -> QuerySet[Integration]:+ # The default manager holds only rows with the default auth type of their kind.+ queryset = Integration.objects._with_auth(ANY)+ ordering = self.get_ordering(request)+ return queryset.order_by(*ordering) if ordering else queryset+@admin.display(description="Team")def team_link(self, obj: Integration) -> str:return format_html("{} ({})", obj.team.name, obj.team_id)diff --git a/posthog/api/integration.py b/posthog/api/integration.pyindex eea14de8a700..3017847779c0 100644--- a/posthog/api/integration.py+++ b/posthog/api/integration.py@@ -77,6 +77,8 @@ from posthog.models.integration import (ANTHROPIC_DEFAULT_INTEGRATION_ID_PREFIX,ANTHROPIC_MANAGED_AGENT_LIST_PAGE_LIMIT,ANTHROPIC_WORKSPACE_LABEL_MAX_LENGTH,+ ANY,+ DEFAULT,ERROR_TOKEN_REFRESH_FAILED,GITHUB_REPOSITORY_REFRESH_COOLDOWN_SECONDS,POSTHOG_CONNECT_ALLOWED_REGIONS,@@ -107,7 +109,9 @@ from posthog.models.integration import (GoogleCloudIntegration,GoogleCloudServiceAccountIntegration,Integration,+ IntegrationAuthType,IntegrationError,+ IntegrationQuerySet,JiraIntegration,LinearIntegration,LinkedInAdsIntegration,@@ -124,6 +128,7 @@ from posthog.models.integration import (defer_repository_cache_fields,resolve_aliased_oauth_kind,)+from posthog.models.integration.auth_types import rows_without_source_marker_qfrom posthog.models.integration.github_audit import GitHubAuditfrom posthog.models.integration.stripe import stripe_signing_secretfrom posthog.models.integration.twitter_ads import TwitterAdsIntegration@@ -769,6 +774,15 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial"""Standard Integration serializer."""created_by = UserBasicSerializer(read_only=True)+ auth_type = serializers.ChoiceField(+ choices=IntegrationAuthType.choices,+ read_only=True,+ help_text=(+ "How this integration authenticates with the provider: `oauth` for a connection the provider manages "+ "(an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several "+ "typed-in fields. A kind can have several auth types."+ ),+ )files_write_requestable = serializers.SerializerMethodField(help_text="Slack only: whether reconnecting can request the files:write scope.")@@ -791,6 +805,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerialfields = ["id","kind",+ "auth_type","config","created_at","created_by",@@ -802,6 +817,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial]read_only_fields = ["id",+ "auth_type","created_at","created_by","errors",@@ -813,11 +829,15 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial@extend_schema_field(serializers.BooleanField())def get_files_write_requestable(self, obj: Integration) -> bool:- return obj.kind == "slack" and "files:write" in POSTHOG_SLACK_SCOPE.split(",")+ return (+ obj.kind == "slack"+ and obj.auth_type == IntegrationAuthType.OAUTH+ and "files:write" in POSTHOG_SLACK_SCOPE.split(",")+ )@extend_schema_field(serializers.BooleanField(allow_null=True))def get_installation_shared(self, obj: Integration) -> bool | None:- if obj.kind != "github" or not obj.integration_id:+ if obj.kind != "github" or obj.auth_type != IntegrationAuthType.OAUTH or not obj.integration_id:return None# Mirrors the check in IntegrationViewSet.perform_destroy: only other *team* rows keep the App# installed on GitHub. Personal rows don't count because destroy deletes them along with it.@@ -829,7 +849,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial@extend_schema_field(serializers.ChoiceField(choices=GITHUB_INSTALLATION_STATUS_CHOICES, allow_null=True))def get_installation_status(self, obj: Integration) -> str | None:- if obj.kind != "github":+ if obj.kind != "github" or obj.auth_type != IntegrationAuthType.OAUTH:return Nonereturn ("unavailable"@@ -1557,7 +1577,8 @@ class IntegrationViewSet(# page boundary. Clients page this list to find one kind, so a dropped row reads as# "not configured". Order oldest-first: several clients take the first row of a kind as their# default connection.- queryset = defer_repository_cache_fields(Integration.objects.all()).order_by("created_at", "id")+ # Lists every auth type, so typed-in credentials show next to installations and OAuth grants.+ queryset = defer_repository_cache_fields(Integration.objects._with_auth(ANY)).order_by("created_at", "id")serializer_class = IntegrationSerializerfilter_backends = [DjangoFilterBackend]filterset_fields = ["kind"]@@ -1568,7 +1589,14 @@ class IntegrationViewSet(# creator. Unlike team-shared integrations, don't expose other members' connections in# list/retrieve — scope `posthog` rows to the requesting user. Other kinds stay team-visible.user_id = getattr(self.request.user, "id", None)- return queryset.exclude(Q(kind=POSTHOG_CONNECT_KIND) & ~Q(created_by_id=user_id))+ queryset = queryset.exclude(Q(kind=POSTHOG_CONNECT_KIND) & ~Q(created_by_id=user_id))+ # The other actions (destroy, channels, github_repos, ...) call provider code that reads the+ # row in the default shape of its kind, so they keep to rows with the default auth type, and+ # skip warehouse source rows, the same as the implicit filter of `Integration.objects`.+ if self.action not in ("list", "retrieve"):+ assert isinstance(queryset, IntegrationQuerySet)+ queryset = queryset._with_auth(DEFAULT).filter(rows_without_source_marker_q())+ return querysetdef handle_exception(self, exc: Exception) -> Response:# GitHub rate limits surface from any GitHub-backed action (teams, repos, branches, refresh);@@ -2974,17 +3002,18 @@ class IntegrationViewSet(queryset = self.filter_queryset(self.get_queryset())page = self.paginate_queryset(queryset)instances = page if page is not None else list(queryset)- for instance in instances:- if instance.kind == "github":- GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()+ # Only App installations have an installation id to heal from. A typed-in token row does not.+ github_apps = [+ instance+ for instance in instances+ if instance.kind == "github" and instance.auth_type == IntegrationAuthType.OAUTH+ ]+ for instance in github_apps:+ GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()# One grouped lookup for the whole page instead of an `exists()` per row, so# `installation_shared` stays a fixed one query however many GitHub rows there are.reference_counts = GitHubIntegration.installation_reference_counts(- {- instance.integration_id- for instance in instances- if instance.kind == "github" and instance.integration_id- },+ {instance.integration_id for instance in github_apps if instance.integration_id},include_personal=False,)serializer = self.get_serializer(@@ -3000,6 +3029,6 @@ class IntegrationViewSet(# Inlined from RetrieveModelMixin too: deferring to it would re-run `get_object`, paying a# second query and permission check for the row already in hand.instance = self.get_object()- if instance.kind == "github":+ if instance.kind == "github" and instance.auth_type == IntegrationAuthType.OAUTH:GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()return Response(self.get_serializer(instance).data)diff --git a/posthog/api/test/test_integration.py b/posthog/api/test/test_integration.pyindex fb9b916f7fc2..93d291f70c29 100644--- a/posthog/api/test/test_integration.py+++ b/posthog/api/test/test_integration.py@@ -3272,6 +3272,39 @@ class TestIntegrationAPIKeyAccess:assert "github" in kindsassert "twilio" in kinds+ def test_list_and_retrieve_show_every_auth_type(self, client: HttpClient):+ client.force_login(self.user)+ token_row = Integration.objects._with_auth(Integration.AuthType.TOKEN).create(+ team=self.team, kind="github", integration_id="typed-in", sensitive_config={"token": "secret"}+ )+ source_row = Integration.objects._with_auth(Integration.AuthType.CREDENTIALS).create(+ team=self.team, kind="postgresql", integration_id="source", config={"warehouse_source_type": "Postgres"}+ )+ base_url = f"/api/environments/{self.team.pk}/integrations"++ with patch.object(GitHubIntegration, "ensure_account_name") as heal:+ listed = client.get(f"{base_url}/").json()["results"]+ retrieved = client.get(f"{base_url}/{token_row.id}/")+ repos = client.get(f"{base_url}/{token_row.id}/github_repos/")++ assert {(row["id"], row["auth_type"]) for row in listed} == {+ (self.github_integration.id, "oauth"),+ (self.twilio_integration.id, "token"),+ (token_row.id, "token"),+ (source_row.id, "credentials"),+ }+ # Code that does not state its auth types does not see the warehouse source row.+ assert not Integration.objects.filter(team=self.team, kind="postgresql").exists()+ assert retrieved.status_code == status.HTTP_200_OK+ assert (retrieved.json()["auth_type"], retrieved.json()["installation_status"]) == ("token", None)+ assert repos.status_code == status.HTTP_404_NOT_FOUND+ # Only the App row of the list heals its account name. The token row has no installation.+ assert heal.call_count == 1+ # Actions other than list and retrieve do not reach the warehouse source row.+ deleted = client.delete(f"{base_url}/{source_row.id}/")+ assert deleted.status_code == status.HTTP_404_NOT_FOUND+ assert Integration._base_manager.filter(id=source_row.id).exists()+def test_list_integrations_filtered_by_kind(self, client: HttpClient):client.force_login(self.user)diff --git a/posthog/integrations/AGENTS.md b/posthog/integrations/AGENTS.mdnew file mode 100644index 000000000000..4c1cb0443230--- /dev/null+++ b/posthog/integrations/AGENTS.md@@ -0,0 +1,13 @@+# Integrations SDK: rules for callers++Read `README.md` before you change the SDK or add an operation.++- Read and write `Integration` rows only through `IntegrationManager` from `posthog.integrations`. Never import the `Integration` model, its managers or its relations. The semgrep ratchet `integration-model-through-sdk` fails CI on new uses.+- Pick the accessor by kind and auth type: `IntegrationManager.<kind>.<oauth|token|credentials>` for code that reads one config shape. Use `.any` or `IntegrationManager.kinds(...)` only for code that handles every auth type the same way, such as a picker or a delete.+- Rows are read-only `IntegrationRecord`s. Do not call `.save()`, `.delete()` or `.refresh_from_db()` on a row, and do not write to `row.config`. Write with `update` or `upsert`, and delete with `delete`.+- Do not read related rows (`row.team`, `row.created_by`). Use `row.team_id` and `row.created_by_id`. Set a foreign key with its id column (`integration_id=row.id`).+- Warehouse source rows (`config["warehouse_source_type"]`) are hidden by default. Add `.with_source_rows()` only in code that handles every row the same way, such as the integrations list and delete API. Use `IntegrationManager.source(kind)` for the source credentials themselves.+- Refresh an OAuth token with `IntegrationManager.<kind>.oauth.refresh_access_token(team_id, id, stale_access_token=...)`. Do not lock a row yourself.+- When an operation is missing, add a small named operation to the SDK. Do not widen `find(where=...)`, which compares named config keys for equality only, and never return a queryset.+- The model moves behind an HTTP API in the credentials service. Every operation must work over that transport: no callbacks into caller code inside a transaction, no querysets, no model instances that callers write.+- When you move a file to the SDK, lower its count in `model_access_baseline.txt`, or remove the line at zero.diff --git a/posthog/integrations/CLAUDE.md b/posthog/integrations/CLAUDE.mdnew file mode 120000index 000000000000..47dc3e3d863c--- /dev/null+++ b/posthog/integrations/CLAUDE.md@@ -0,0 +1 @@+AGENTS.md\ No newline at end of filediff --git a/posthog/integrations/README.md b/posthog/integrations/README.mdnew file mode 100644index 000000000000..6a2c9c3c0215--- /dev/null+++ b/posthog/integrations/README.md@@ -0,0 +1,171 @@+# Integrations SDK++The integrations SDK is the only way to read and write `Integration` rows (`posthog/models/integration/model.py`).+The rows move to a separate credentials service soon.+The SDK is a small, closed set of operations, so only its transport changes after the move.+No operation returns a queryset, and no operation runs caller code inside its transaction.++```python+from posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationRecord+from posthog.integrations.fields import IntegrationReferenceField+```++The rules for callers are in `AGENTS.md`.++| Module | Holds |+| -------------------- | -------------------------------------------------------------------------- |+| `__init__.py` | The public names |+| `accessors.py` | The operations |+| `catalogue.py` | The kinds and their auth types (hand-written) |+| `kinds_generated.py` | One accessor class per kind, generated from the catalogue |+| `contracts.py` | `IntegrationRecord`, `IntegrationPatch`, `IntegrationUsage` and the errors |+| `source_kinds.py` | The registry of warehouse source kinds |+| `usages.py` | The registry of usage providers |+| `fields.py` | `IntegrationReferenceField` for DRF serializers |++The model module imports `catalogue.py`, `contracts.py` and `enums.py`. These modules must not import the model.+`__init__.py` loads `IntegrationManager` on first use for the same reason.++## Read rows++An accessor fixes the kind, the auth type, or both. Each operation takes `team_id` first and returns rows in the scope of the accessor only.++| Accessor | Rows | Operations |+| ---------------------------------------- | ------------------------------------------------------ | ---------------- |+| `IntegrationManager.github.oauth` | One kind with one auth type | Reads and writes |+| `IntegrationManager.stripe.any` | One kind with every auth type | Reads and delete |+| `IntegrationManager.kinds("slack", ...)` | Several kinds, with the default auth type of each kind | Reads and delete |+| `IntegrationManager.any` | Every kind with every auth type | Reads and delete |+| `IntegrationManager.source(kind).token` | Warehouse source rows of one kind with one auth type | Reads and writes |+| `IntegrationManager.source(kind).any` | Warehouse source rows of one kind with every auth type | Reads and delete |++Only `IntegrationManager.source(kind)` reads warehouse source rows (rows with `config["warehouse_source_type"]`).+The other accessors skip them, because their config has the shape of the warehouse source, not of the kind.+Code that handles every row the same way, such as the integrations list and delete API, adds them with `with_source_rows()`:++```python+IntegrationManager.any.with_source_rows().delete(team_id, integration_id)+IntegrationManager.slack.any.with_source_rows().list(team_id)+IntegrationManager.kinds("github", "stripe", auth_types="any").with_source_rows().get_many(team_id, ids)+```++Reads:++- `get(team_id, id)` returns the row, or raises `IntegrationNotFound` when the row is not in the scope.+- `get_many(team_id, ids)` returns `{id: row}` for the ids in the scope.+- `find(team_id, external_id=..., where={...})` returns the oldest matching row or `None`.+- `list(team_id, external_id=..., where={...}, newest_first=False)` and `exists(...)`.+- `find` and `exists` also take `team_ids=[...]` instead of `team_id`, for example the teams of an organization. Pass exactly one of the two. An empty `team_ids` matches no row.+- `usages(team_id, id)` returns the PostHog objects that use the row.++These reads go across teams, for example to the teams that share one GitHub installation:++- `list_by_external_id(external_id, team_ids=None)` returns the rows with this external id.+- `get_by_external_id(external_id, id)` returns the row with this id and this external id, or raises `IntegrationNotFound`.+- `count_by_external_id(external_id)` and `exists_by_external_id(external_id)`.++They need an accessor with a kind: on `IntegrationManager.any` they raise `ValueError`, because external ids are unique only per provider.+Do not take the first row of `list_by_external_id` without `team_ids`: the rule `github-installation-first-team` flags it for GitHub.++`IntegrationManager.kinds(...)` takes `auth_types`: `"default"` (the default auth type of each kind), `"any"` for every auth type, or a list of auth types.++Reads do not load the GitHub repository cache columns (`repository_cache`, `repository_cache_updated_at`). Django loads them with one more query when code reads them.++`where` compares named config keys for equality, for example `where={"region": "eu"}`.+Equality also compares the JSON type, so `where={"installation_id": "123"}` misses a row that stores the number `123`.+For a GitHub installation, use the external id, which always holds the installation id as a string: `IntegrationManager.github.oauth.find(team_id, external_id=str(installation_id))`.+It does not accept lookups (`a__b`), nested values or the keys that the SDK writes (`auth_type`, `warehouse_source_type`), and raises `UnsupportedIntegrationFilter`.+When you need more, add a named method to the SDK.++Each operation has an async form with an `a` prefix: `aget`, `alist`, `aupdate`, and so on.++## Write rows++Only an accessor with one auth type can write, because a new row has exactly one auth type.++- `create(team_id, external_id=..., config=..., sensitive_config=..., created_by_id=...)`. It raises `IntegrationConflict` when the team already has a row of the kind with this external id, of any auth type.+- `upsert(team_id, external_id, config=..., ...)` returns `(row, created)`. It replaces the config of an existing row. It raises `IntegrationConflict` when the existing row is not in the scope, for example a row with another auth type.+- `update(team_id, id, config_patch=..., config_remove=..., sensitive_config_patch=..., errors=...)` merges a patch. `errors=None` keeps the errors.+- `update_by_external_id(external_id, patch)` patches the rows of every team that are in the scope. It writes only the rows that the patch changes, in one `bulk_update`, and returns their count. A `bulk_update` sends no `post_save` signal.++Delete rows with `delete(team_id, id)`. It raises `IntegrationInUse` with the usages when other objects use the row. Pass `ignore_usages=True` to delete it anyway.+`delete_by_external_id(external_id)` deletes the rows of every team in one transaction. It deletes each row on its own, so the delete signals fire. Like `list_by_external_id`, it needs an accessor with a kind.++An OAuth accessor (`IntegrationManager.<kind>.oauth`) also refreshes access tokens:++- `refresh_access_token(team_id, id, stale_access_token=None)` locks the row, refreshes the token when it expires soon, saves the new tokens and returns the row.+ Pass the token that the provider refused as `stale_access_token`, to refresh also when it is still the stored token.+ When another worker already stored a newer token, the row comes back without a refresh.+ The lock stops two workers from spending the same refresh token, which matters for providers that rotate refresh tokens (Resend, Jira).+ A refresh that fails does not raise: the row comes back with `errors` set.+ It raises `ValueError` for a kind that `OauthIntegration` cannot refresh.++## Rows are read-only++The SDK returns rows typed as `IntegrationRecord`, a read-only `Protocol`:+`id`, `team_id`, `kind`, `integration_id`, `config`, `sensitive_config`, `errors`, `created_at`, `created_by_id`, `auth_type`, `access_token`, `refresh_token` and `can_be_managed_by_creator(user_id)`.+`config` and `sensitive_config` are `Mapping`s, so mypy rejects `row.config["x"] = 1`.+Mypy also rejects `.save()`, `.delete()`, `.refresh_from_db()`, managers and related rows such as `row.team`.+Use `row.team_id` and `row.created_by_id`, and load the team or the user from their own models.+Write with `update` or `upsert`. Read the row again with `get` after a change.+To set a foreign key to a row, set the id column: `task.github_integration_id = row.id`.++At runtime the rows are still model instances. A later change returns frozen records with the same reads, and callers do not change.+The per-kind adapters next to the model (`SlackIntegration`, `GitHubIntegration`, `OauthIntegration`, ...) take an `IntegrationRecord` and narrow it to the model with `integration_model`, because they still write the row.++## Pick an auth type++Each kind supports some of three auth types, and has one default:++| Auth type | Use it for |+| ------------- | ----------------------------------------------------------------------------------------------- |+| `oauth` | A connection that the provider manages: an OAuth grant or an app installation |+| `token` | One static secret: an API key, a personal access token, a bot token or a webhook secret |+| `credentials` | Several typed-in fields: host, user and password, an access key and a secret, a service account |++Use the auth type of the rows that your code can handle.+A kind class has an attribute only for the auth types that the kind supports, so mypy rejects `IntegrationManager.github.credentials`.+Use `.any` only when your code handles every auth type of the kind the same way, for example a picker or a delete.++A row stores its auth type in `config["auth_type"]` only when it is not the default of its kind. The SDK writes that key. Do not write it yourself.++## Add a kind++1. Add the kind to `Integration.IntegrationKind` in `posthog/models/integration/model.py`.+2. Add the kind to `KINDS` in `catalogue.py`, with its default auth type and its other auth types.+3. Run `hogli build:projections --only integration-kinds` and commit `kinds_generated.py`.++Use `IntegrationKind` from `posthog.integrations` for kind values in code, not `Integration.IntegrationKind`.++The default of a kind is stored data. Do not change the default of a kind that has rows.++Warehouse source kinds are not in the catalogue. The warehouse product registers them at startup with `register_source_kinds`, and callers reach their rows through `IntegrationManager.source(kind)`.+A warehouse source row has `config["warehouse_source_type"]`. Every other accessor skips these rows unless you call `with_source_rows()`, and `IntegrationManager.source(kind)` reads only these rows.++## Report usages++A product that uses integration rows reports them from `products/<product>/backend/integration_usages.py`:++```python+def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]: ...+```++The SDK finds the module on the first call to `usages` or `delete`. Code outside a product calls `register_usage_provider` instead.++## Reference a row from a serializer++Use `IntegrationReferenceField` from `posthog.integrations.fields` instead of a `PrimaryKeyRelatedField` on `Integration`.+It reads the team from the serializer context, so a row of another team does not exist for the field.+Pass `kinds=[...]` to accept only rows of these kinds.+With `kinds`, the field accepts only the default auth type of each kind, because code that reads a kind expects its usual config shape.+Pass `auth_types=[AuthType.TOKEN]` to accept other auth types, or `auth_types="any"` to accept every auth type.+The field never accepts a warehouse source row.++## Why direct model use fails CI++The semgrep rule `.semgrep/rules/devex/integration-model-through-sdk.yaml` finds uses of the model outside the SDK: imports of `Integration`, its managers (`objects`, `_default_manager`, `_base_manager`, also through `IntegrationRecord`), `apps.get_model` lookups of it, relation fields to it, `select_related` and `prefetch_related` on an integration relation, `integration__` lookups (as keyword arguments or as strings), `integration_set`, and `.save()` on a row.+The "Integration model ratchet" step in `.github/workflows/ci-security.yaml` compares the findings per file with `model_access_baseline.txt`.+It fails when a file has more findings than its baseline count, and also when it has fewer.+A file that semgrep reports an error for (a timeout or a parse error) is not checked for fewer findings, and `--update` keeps its count.+When you move a file to the SDK, lower its count in the baseline, or remove the line at zero.+Tests, migrations, Django admin, the model and its per-kind adapters (`posthog/models/integration/`) are outside the rule.diff --git a/posthog/integrations/__init__.py b/posthog/integrations/__init__.pynew file mode 100644index 000000000000..a2d7fc1c878f--- /dev/null+++ b/posthog/integrations/__init__.py@@ -0,0 +1,92 @@+"""The integrations SDK: the only way to read and write `Integration` rows. See README.md and AGENTS.md.++from posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationRecord+from posthog.integrations.fields import IntegrationReferenceField+"""++import importlib+from typing import TYPE_CHECKING, Any++from posthog.integrations.contracts import (+ SOURCE_MARKER_CONFIG_KEY,+ ConfigValue,+ IntegrationConflict,+ IntegrationInUse,+ IntegrationNotFound,+ IntegrationPatch,+ IntegrationRecord,+ IntegrationUsage,+ KindSpec,+ UnknownIntegrationKind,+ UnsupportedAuthType,+ UnsupportedIntegrationFilter,+)+from posthog.integrations.enums import AuthType, IntegrationAuthType++if TYPE_CHECKING:+ from posthog.integrations.kinds_generated import IntegrationKind+ from posthog.integrations.manager import (+ AuthAccessor,+ IntegrationManager,+ IntegrationReader,+ OAuthAccessor,+ SourceKindAccessor,+ UsageProvider,+ register_source_kinds,+ register_usage_provider,+ )++# Loaded on first use (PEP 562). The model module imports the catalogue from this package, and+# importing a submodule runs this file first, so an eager import of the accessors (which import+# the model) would be a circular import.+_LAZY_NAMES = {+ "IntegrationKind": "posthog.integrations.kinds_generated",+ **dict.fromkeys(+ (+ "AuthAccessor",+ "IntegrationManager",+ "IntegrationReader",+ "OAuthAccessor",+ "SourceKindAccessor",+ "UsageProvider",+ "register_source_kinds",+ "register_usage_provider",+ ),+ "posthog.integrations.manager",+ ),+}+++def __getattr__(name: str) -> Any:+ if name not in _LAZY_NAMES:+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")+ value = getattr(importlib.import_module(_LAZY_NAMES[name]), name)+ globals()[name] = value+ return value+++__all__ = [+ "SOURCE_MARKER_CONFIG_KEY",+ "AuthAccessor",+ "AuthType",+ "ConfigValue",+ "IntegrationAuthType",+ "IntegrationConflict",+ "IntegrationInUse",+ "IntegrationKind",+ "IntegrationManager",+ "IntegrationNotFound",+ "IntegrationPatch",+ "IntegrationReader",+ "IntegrationRecord",+ "IntegrationUsage",+ "KindSpec",+ "OAuthAccessor",+ "SourceKindAccessor",+ "UnknownIntegrationKind",+ "UnsupportedAuthType",+ "UnsupportedIntegrationFilter",+ "UsageProvider",+ "register_source_kinds",+ "register_usage_provider",+]diff --git a/posthog/integrations/accessors.py b/posthog/integrations/accessors.pynew file mode 100644index 000000000000..e113cef98834--- /dev/null+++ b/posthog/integrations/accessors.py@@ -0,0 +1,676 @@+"""The operations of the integrations SDK.++An accessor holds a scope: the kinds, the auth type, and whether it sees warehouse source rows. Every+operation reads or writes only rows in that scope. No accessor sees warehouse source rows unless the+caller asks for them with `with_source_rows()` or uses `IntegrationManager.source(kind)`.+`IntegrationReader` has the reads and the deletes. `AuthAccessor` adds the writes, and exists only for+one kind with one auth type, because a new row has exactly one auth type. `OAuthAccessor` adds the+token refresh.++No operation returns a queryset, and no operation runs caller code inside its transaction. The+integration rows move to a separate service later, so the operations here are the whole contract, and+only their transport changes. Rows come back typed as the read-only `IntegrationRecord`.+"""++import re+import json+import builtins+from collections.abc import Iterable, Mapping+from enum import Enum+from typing import ClassVar, Literal++from django.db import IntegrityError, transaction+from django.db.models import Q++from posthog.integrations.catalogue import KINDS+from posthog.integrations.contracts import (+ SOURCE_MARKER_CONFIG_KEY,+ ConfigValue,+ IntegrationConflict,+ IntegrationInUse,+ IntegrationNotFound,+ IntegrationPatch,+ IntegrationRecord,+ IntegrationUsage,+ UnknownIntegrationKind,+ UnsupportedAuthType,+ UnsupportedIntegrationFilter,+)+from posthog.integrations.enums import AuthType+from posthog.integrations.source_kinds import is_source_kind+from posthog.integrations.usages import integration_usages+from posthog.models.integration.auth_types import ANY, AUTH_TYPE_CONFIG_KEY, default_auth_type, kind_auth_types+from posthog.models.integration.model import Integration, IntegrationQuerySet, defer_repository_cache_fields+from posthog.models.integration.oauth import OauthIntegration+from posthog.sync import database_sync_to_async++# The SDK writes these keys itself. A caller cannot set, remove or filter on them.+RESERVED_CONFIG_KEYS = frozenset({AUTH_TYPE_CONFIG_KEY, SOURCE_MARKER_CONFIG_KEY})++_CONFIG_KEY = re.compile(r"^[A-Za-z][A-Za-z0-9_]*$")+# The unique key of a row: (team, kind, external id).+_UNIQUE_KEY_CONSTRAINT = "posthog_integration_kind_id_unique"+_SCALAR_TYPES = (str, int, float, bool, type(None))+++class SourceRows(Enum):+ # Named accessors skip warehouse source rows, so code that reads a kind in its usual shape never+ # gets a row with typed-in source credentials.+ EXCLUDE = "exclude"+ ONLY = "only"+ INCLUDE = "include"+++def _rows() -> IntegrationQuerySet:+ # The GitHub repository cache can be large. Django loads a deferred column when code reads it.+ return defer_repository_cache_fields(Integration.objects._with_auth(ANY))+++# The auth types an accessor reads: one auth type, several, or the default auth type of each kind.+AuthTypes = AuthType | frozenset[AuthType] | Literal["default"]+++def _auth_type_q(kind: str, auth_type: AuthType) -> Q:+ stored = Q(config__auth_type=auth_type.value)+ if auth_type == default_auth_type(kind):+ # A JSON null counts as unset, the same as in `auth_types.rows_with_default_auth_type_q`. A+ # stored default is possible too: `QuerySet.update()` skips the `normalized_config` of `save()`.+ return Q(config__auth_type__isnull=True) | Q(config__auth_type=None) | stored+ return stored+++def _where_q(where: Mapping[str, ConfigValue] | None) -> Q:+ q = Q()+ for key, value in (where or {}).items():+ if not isinstance(key, str) or not _CONFIG_KEY.match(key) or "__" in key or key in RESERVED_CONFIG_KEYS:+ raise UnsupportedIntegrationFilter(f"Cannot filter integrations on the config key {key!r}")+ if not isinstance(value, _SCALAR_TYPES):+ raise UnsupportedIntegrationFilter(f"Cannot compare the config key {key!r} with a {type(value).__name__}")+ # `__exact`, so a key named like a lookup (`contains`, `regex`) still reads as a key.+ # nosemgrep: orm-field-injection - the key is one checked JSON key segment, and `__exact` fixes the lookup+ q &= Q(**{f"config__{key}__exact": value})+ return q+++def _is_unique_key_violation(error: IntegrityError) -> bool:+ constraint = getattr(getattr(error.__cause__, "diag", None), "constraint_name", None)+ return constraint == _UNIQUE_KEY_CONSTRAINT or _UNIQUE_KEY_CONSTRAINT in str(error)+++def _checked_config_keys(keys: Iterable[str]) -> None:+ reserved = sorted(RESERVED_CONFIG_KEYS.intersection(keys))+ if reserved:+ raise ValueError(f"The integrations SDK writes {reserved} itself")+++def _patched(+ current: Mapping[str, object] | None, patch: Mapping[str, object], remove: Iterable[str]+) -> dict[str, object]:+ result = dict(current or {})+ result.update(patch)+ for key in remove:+ result.pop(key, None)+ return result+++class IntegrationReader:+ """Reads and deletes of the integration rows in one scope."""++ def __init__(+ self,+ *,+ kinds: frozenset[str] | None,+ auth_type: AuthTypes | None = None,+ source_rows: SourceRows = SourceRows.EXCLUDE,+ ) -> None:+ """`auth_type=None` reads every auth type. `"default"` reads the default auth type of each kind."""+ if auth_type is not None and not kinds:+ raise ValueError("An accessor with an auth type needs its kinds")+ self._kinds = kinds+ self._auth_type = auth_type+ self._source_rows = source_rows++ def _auth_types_of(self, kind: str) -> builtins.list[AuthType]:+ if self._auth_type == "default":+ return [default_auth_type(kind)]+ if isinstance(self._auth_type, AuthType):+ return [self._auth_type]+ if isinstance(self._auth_type, frozenset):+ return sorted(self._auth_type)+ return []++ def __repr__(self) -> str:+ kinds = sorted(self._kinds) if self._kinds is not None else "every kind"+ return f"{type(self).__name__}({kinds}, {self._auth_type or 'any auth type'}, {self._source_rows.value})"++ def _scope_q(self) -> Q:+ q = Q()+ if self._kinds is not None:+ if self._auth_type is None:+ q &= Q(kind__in=sorted(self._kinds))+ else:+ # An empty Q() in an OR matches every row, so start from a clause that matches none.+ by_kind = Q(pk__in=[])+ for kind in sorted(self._kinds):+ for auth_type in self._auth_types_of(kind):+ by_kind |= Q(kind=kind) & _auth_type_q(kind, auth_type)+ q &= by_kind+ if self._source_rows == SourceRows.EXCLUDE:+ q &= ~Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)+ elif self._source_rows == SourceRows.ONLY:+ q &= Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)+ return q++ def _cross_team_rows(self, operation: str) -> IntegrationQuerySet:+ # A cross-team operation without a kind would reach every row with the external id, of any+ # provider, in every team. External ids are only unique per provider.+ if self._kinds is None:+ raise ValueError(f"{operation} needs an accessor with a kind, such as IntegrationManager.github.oauth")+ return _rows().filter(self._scope_q())++ def _team_rows(self, team_id: int) -> IntegrationQuerySet:+ return _rows().filter(self._scope_q(), team_id=team_id)++ def _rows_of_teams(self, team_id: int | None, team_ids: Iterable[int] | None) -> IntegrationQuerySet:+ if (team_id is None) == (team_ids is None):+ raise ValueError("Pass exactly one of team_id and team_ids")+ if team_id is not None:+ return self._team_rows(team_id)+ # An empty list matches no row.+ return _rows().filter(self._scope_q(), team_id__in=builtins.list(team_ids or ()))++ def _matching(+ self,+ team_id: int | None,+ external_id: str | None,+ where: Mapping[str, ConfigValue] | None,+ team_ids: Iterable[int] | None = None,+ ) -> IntegrationQuerySet:+ rows = self._rows_of_teams(team_id, team_ids).filter(_where_q(where))+ return rows if external_id is None else rows.filter(integration_id=external_id)++ def with_source_rows(self) -> "IntegrationReader":+ """A reader of the same scope that also sees warehouse source rows.++ Use it only for code that handles every row the same way, such as the integrations list and+ delete API. Code that reads the config of a kind must not see source rows: their config has the+ shape of the warehouse source, not of the kind.+ """+ if self._source_rows == SourceRows.ONLY:+ raise ValueError("This accessor reads only warehouse source rows")+ return IntegrationReader(kinds=self._kinds, auth_type=self._auth_type, source_rows=SourceRows.INCLUDE)++ def _row(self, team_id: int, id: int, *, lock: bool = False) -> Integration:+ rows = self._team_rows(team_id)+ try:+ return (rows.select_for_update() if lock else rows).get(id=id)+ except (Integration.DoesNotExist, ValueError, TypeError) as error:+ raise IntegrationNotFound(f"No integration {id!r} in {self!r} for team {team_id}") from error++ def get(self, team_id: int, id: int) -> IntegrationRecord:+ """The row with this id. Raises `IntegrationNotFound` when the row is not in the scope."""+ return self._row(team_id, id)++ def get_many(self, team_id: int, ids: Iterable[int]) -> dict[int, IntegrationRecord]:+ """The rows with these ids, keyed by id. Ids that are not in the scope are missing from the result."""+ return {row.id: row for row in self._team_rows(team_id).filter(id__in=list(ids))}++ def find(+ self,+ team_id: int | None = None,+ *,+ team_ids: Iterable[int] | None = None,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ ) -> IntegrationRecord | None:+ """The oldest row with this external id and these config values, or None.++ Pass `team_id` or `team_ids`, not both. An empty `team_ids` finds nothing.+ """+ return self._matching(team_id, external_id, where, team_ids).order_by("created_at", "id").first()++ def list(+ self,+ team_id: int,+ *,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ newest_first: bool = False,+ ) -> builtins.list[IntegrationRecord]:+ order = ("-created_at", "-id") if newest_first else ("created_at", "id")+ return list(self._matching(team_id, external_id, where).order_by(*order))++ def exists(+ self,+ team_id: int | None = None,+ *,+ team_ids: Iterable[int] | None = None,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ ) -> bool:+ """Pass `team_id` or `team_ids`, not both. An empty `team_ids` is False."""+ return self._matching(team_id, external_id, where, team_ids).exists()++ def list_by_external_id(+ self, external_id: str, *, team_ids: Iterable[int] | None = None+ ) -> builtins.list[IntegrationRecord]:+ """Rows of every team with this external id, for example the teams that share one GitHub installation.++ Raises `ValueError` on `IntegrationManager.any`, because the result would mix providers.+ """+ rows = self._cross_team_rows("list_by_external_id").filter(integration_id=external_id)+ if team_ids is not None:+ rows = rows.filter(team_id__in=list(team_ids))+ return list(rows.order_by("created_at", "id"))++ def get_by_external_id(self, external_id: str, id: int) -> IntegrationRecord:+ """The row with this id and this external id, in any team.++ For a callback that names both, such as a GitHub installation and one of its rows. Raises+ `IntegrationNotFound` when the row has another external id or is not in the scope.+ """+ rows = self._cross_team_rows("get_by_external_id")+ try:+ return rows.get(integration_id=external_id, id=id)+ except (Integration.DoesNotExist, ValueError, TypeError) as error:+ raise IntegrationNotFound(f"No integration {id!r} with external id {external_id!r} in {self!r}") from error++ def count_by_external_id(self, external_id: str) -> int:+ """The number of rows of every team with this external id."""+ return self._cross_team_rows("count_by_external_id").filter(integration_id=external_id).count()++ def exists_by_external_id(self, external_id: str) -> bool:+ """True when a row of any team has this external id."""+ return self._cross_team_rows("exists_by_external_id").filter(integration_id=external_id).exists()++ def usages(self, team_id: int, id: int) -> builtins.list[IntegrationUsage]:+ """The PostHog objects of the team that use the row. See `usages.py`."""+ row = self.get(team_id, id)+ return integration_usages(team_id, row.id)++ def delete(self, team_id: int, id: int, *, ignore_usages: bool = False) -> None:+ """Delete the row. Raises `IntegrationInUse` when other objects use it, unless `ignore_usages`."""+ row = self._row(team_id, id)+ if not ignore_usages:+ usages = integration_usages(team_id, row.id)+ if usages:+ raise IntegrationInUse(usages)+ row.delete()++ def delete_by_external_id(self, external_id: str) -> int:+ """Delete the rows of every team with this external id, for example after an app is uninstalled.++ Raises `ValueError` on `IntegrationManager.any`, because it would delete rows of other providers.+ All the deletes commit together or not at all. Each row is deleted on its own, so the delete+ signals fire.+ """+ deleted = 0+ with transaction.atomic():+ for row in self._cross_team_rows("delete_by_external_id").filter(integration_id=external_id):+ row.delete()+ deleted += 1+ return deleted++ async def aget(self, team_id: int, id: int) -> IntegrationRecord:+ return await database_sync_to_async(self.get)(team_id, id)++ async def aget_many(self, team_id: int, ids: Iterable[int]) -> dict[int, IntegrationRecord]:+ return await database_sync_to_async(self.get_many)(team_id, list(ids))++ async def afind(+ self,+ team_id: int | None = None,+ *,+ team_ids: Iterable[int] | None = None,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ ) -> IntegrationRecord | None:+ return await database_sync_to_async(self.find)(+ team_id, team_ids=None if team_ids is None else list(team_ids), external_id=external_id, where=where+ )++ async def alist(+ self,+ team_id: int,+ *,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ newest_first: bool = False,+ ) -> builtins.list[IntegrationRecord]:+ return await database_sync_to_async(self.list)(+ team_id, external_id=external_id, where=where, newest_first=newest_first+ )++ async def aexists(+ self,+ team_id: int | None = None,+ *,+ team_ids: Iterable[int] | None = None,+ external_id: str | None = None,+ where: Mapping[str, ConfigValue] | None = None,+ ) -> bool:+ return await database_sync_to_async(self.exists)(+ team_id, team_ids=None if team_ids is None else list(team_ids), external_id=external_id, where=where+ )++ async def ausages(self, team_id: int, id: int) -> builtins.list[IntegrationUsage]:+ return await database_sync_to_async(self.usages)(team_id, id)++ async def adelete(self, team_id: int, id: int, *, ignore_usages: bool = False) -> None:+ await database_sync_to_async(self.delete)(team_id, id, ignore_usages=ignore_usages)+++class AuthAccessor(IntegrationReader):+ """Reads, deletes and writes of the rows of one kind with one auth type."""++ def __init__(self, *, kind: str, auth_type: AuthType, source_rows: SourceRows = SourceRows.EXCLUDE) -> None:+ if source_rows == SourceRows.INCLUDE:+ raise ValueError("A write accessor writes either named rows or warehouse source rows")+ super().__init__(kinds=frozenset({kind}), auth_type=auth_type, source_rows=source_rows)+ self._kind = kind+ self._write_auth_type = auth_type++ def _config_for_write(self, config: Mapping[str, object]) -> dict[str, object]:+ if self._write_auth_type not in kind_auth_types(self._kind).supported:+ raise UnsupportedAuthType(f"Integration kind {self._kind!r} does not support {self._write_auth_type}")+ if AUTH_TYPE_CONFIG_KEY in config:+ raise ValueError("The integrations SDK writes the auth type itself")+ has_marker = SOURCE_MARKER_CONFIG_KEY in config+ if self._source_rows == SourceRows.ONLY and not has_marker:+ raise ValueError(f"A warehouse source row needs config[{SOURCE_MARKER_CONFIG_KEY!r}]")+ if self._source_rows == SourceRows.EXCLUDE and has_marker:+ raise ValueError("Write warehouse source rows through IntegrationManager.source(kind)")+ result = dict(config)+ if self._write_auth_type != default_auth_type(self._kind):+ result[AUTH_TYPE_CONFIG_KEY] = self._write_auth_type.value+ return result++ def _in_scope(self, row: Integration) -> bool:+ return self._team_rows(row.team_id).filter(id=row.id).exists()++ def create(+ self,+ team_id: int,+ *,+ external_id: str | None,+ config: Mapping[str, object],+ sensitive_config: Mapping[str, object] | None = None,+ created_by_id: int | None = None,+ ) -> IntegrationRecord:+ """Create a row. The SDK writes `config["auth_type"]` when the auth type is not the kind's default.++ Raises `IntegrationConflict` when the team already has a row of the kind with this external id.+ """+ new_config = self._config_for_write(config)+ try:+ with transaction.atomic():+ return _rows().create(+ team_id=team_id,+ kind=self._kind,+ integration_id=external_id,+ config=new_config,+ sensitive_config=dict(sensitive_config or {}),+ created_by_id=created_by_id,+ )+ except IntegrityError as error:+ if not _is_unique_key_violation(error):+ raise+ raise IntegrationConflict(+ f"Team {team_id} already has a {self._kind!r} integration with external id {external_id!r}"+ ) from error++ def upsert(+ self,+ team_id: int,+ external_id: str,+ *,+ config: Mapping[str, object],+ sensitive_config: Mapping[str, object] | None = None,+ created_by_id: int | None = None,+ ) -> tuple[IntegrationRecord, bool]:+ """Create the row with this external id, or replace the config of the existing one.++ Returns the row and True when it is new. Raises `IntegrationConflict` when a row with this+ external id exists outside the scope, for example with another auth type.+ """+ new_config = self._config_for_write(config)+ new_sensitive_config = dict(sensitive_config or {})+ key = {"team_id": team_id, "kind": self._kind, "integration_id": external_id}+ with transaction.atomic():+ row = _rows().select_for_update().filter(**key).first()+ if row is None:+ try:+ with transaction.atomic():+ created = _rows().create(+ **key, config=new_config, sensitive_config=new_sensitive_config, created_by_id=created_by_id+ )+ return created, True+ except IntegrityError as error:+ if not _is_unique_key_violation(error):+ raise+ row = _rows().select_for_update().get(**key)+ if not self._in_scope(row):+ raise IntegrationConflict(f"Integration {row.id} with external id {external_id!r} is not in {self!r}")+ row.config = new_config+ row.sensitive_config = new_sensitive_config+ update_fields = ["config", "sensitive_config"]+ if created_by_id is not None:+ row.created_by_id = created_by_id+ update_fields.append("created_by")+ row.save(update_fields=update_fields)+ return row, False++ def update(+ self,+ team_id: int,+ id: int,+ *,+ config_patch: Mapping[str, object] | None = None,+ config_remove: Iterable[str] = (),+ sensitive_config_patch: Mapping[str, object] | None = None,+ sensitive_config_remove: Iterable[str] = (),+ errors: str | None = None,+ ) -> IntegrationRecord:+ """Merge `config_patch` into the config and drop the keys in `config_remove`. `errors=None` keeps the errors."""+ patch = IntegrationPatch(+ config_patch=dict(config_patch or {}),+ config_remove=tuple(config_remove),+ sensitive_config_patch=dict(sensitive_config_patch or {}),+ sensitive_config_remove=tuple(sensitive_config_remove),+ errors=errors,+ )+ _checked_config_keys([*patch.config_patch, *patch.config_remove])+ with transaction.atomic():+ row = self._row(team_id, id, lock=True)+ update_fields = _patch_row(row, patch)+ if update_fields:+ row.save(update_fields=update_fields)+ return row++ def update_by_external_id(self, external_id: str, patch: IntegrationPatch) -> int:+ """Apply `patch` to the rows of every team with this external id. Returns the number of rows that changed.++ One `bulk_update` writes the changed rows, so no `post_save` signal fires. Rows that the patch+ does not change are not written.+ """+ _checked_config_keys([*patch.config_patch, *patch.config_remove])+ with transaction.atomic():+ rows = (+ self._cross_team_rows("update_by_external_id")+ .select_for_update()+ .filter(integration_id=external_id)+ .order_by("id")+ )+ changed: builtins.list[Integration] = []+ fields: set[str] = set()+ for row in rows:+ row_fields = _patch_row(row, patch)+ if row_fields:+ changed.append(row)+ fields.update(row_fields)+ if changed:+ _rows().bulk_update(changed, sorted(fields))+ return len(changed)++ async def acreate(+ self,+ team_id: int,+ *,+ external_id: str | None,+ config: Mapping[str, object],+ sensitive_config: Mapping[str, object] | None = None,+ created_by_id: int | None = None,+ ) -> IntegrationRecord:+ return await database_sync_to_async(self.create)(+ team_id,+ external_id=external_id,+ config=config,+ sensitive_config=sensitive_config,+ created_by_id=created_by_id,+ )++ async def aupsert(+ self,+ team_id: int,+ external_id: str,+ *,+ config: Mapping[str, object],+ sensitive_config: Mapping[str, object] | None = None,+ created_by_id: int | None = None,+ ) -> tuple[IntegrationRecord, bool]:+ return await database_sync_to_async(self.upsert)(+ team_id, external_id, config=config, sensitive_config=sensitive_config, created_by_id=created_by_id+ )++ async def aupdate(+ self,+ team_id: int,+ id: int,+ *,+ config_patch: Mapping[str, object] | None = None,+ config_remove: Iterable[str] = (),+ sensitive_config_patch: Mapping[str, object] | None = None,+ sensitive_config_remove: Iterable[str] = (),+ errors: str | None = None,+ ) -> IntegrationRecord:+ return await database_sync_to_async(self.update)(+ team_id,+ id,+ config_patch=config_patch,+ config_remove=tuple(config_remove),+ sensitive_config_patch=sensitive_config_patch,+ sensitive_config_remove=tuple(sensitive_config_remove),+ errors=errors,+ )+++def _same_json(a: object, b: object) -> bool:+ # Python equality treats `1`, `1.0` and `True` as equal, but JSON stores them as different values.+ return json.dumps(a, sort_keys=True, default=str) == json.dumps(b, sort_keys=True, default=str)+++def _patch_row(row: Integration, patch: IntegrationPatch) -> list[str]:+ """Apply `patch` to the row in memory. Returns the fields whose value changed."""+ changed: list[str] = []+ config = _patched(row.config, patch.config_patch, patch.config_remove)+ if not _same_json(config, row.config or {}):+ row.config = config+ changed.append("config")+ sensitive_config = _patched(row.sensitive_config, patch.sensitive_config_patch, patch.sensitive_config_remove)+ if not _same_json(sensitive_config, row.sensitive_config or {}):+ row.sensitive_config = sensitive_config+ changed.append("sensitive_config")+ if patch.errors is not None and patch.errors != row.errors:+ row.errors = patch.errors+ changed.append("errors")+ return changed+++class OAuthAccessor(AuthAccessor):+ """Reads and writes of the OAuth rows of one kind, and the refresh of their access tokens."""++ def __init__(self, *, kind: str, source_rows: SourceRows = SourceRows.EXCLUDE) -> None:+ super().__init__(kind=kind, auth_type=AuthType.OAUTH, source_rows=source_rows)++ def refresh_access_token(+ self, team_id: int, id: int, *, stale_access_token: str | None = None+ ) -> IntegrationRecord:+ """Refresh the access token of the row when it expires soon, and return the row.++ The refresh holds a lock on the row. Some providers rotate the refresh token on each refresh+ and revoke the grant when a token is spent twice, so two workers must not refresh one row at+ the same time. The second worker waits for the lock, and then gets the token that the first+ worker stored.++ Pass `stale_access_token` to refresh also when that token is still the stored access token,+ for example after the provider refused it. When the row has another token, the row comes back+ without a refresh. A refresh that fails does not raise: the row comes back with `errors` set.+ Raises `ValueError` when the kind has no OAuth token refresh.+ """+ if self._kind not in OauthIntegration.supported_kinds:+ raise ValueError(f"Integration kind {self._kind!r} has no OAuth token refresh")+ with transaction.atomic():+ row = self._row(team_id, id, lock=True)+ oauth = OauthIntegration(row)+ stale = stale_access_token is not None and row.access_token == stale_access_token+ if stale or oauth.access_token_expired():+ oauth.refresh_access_token()+ return row++ async def arefresh_access_token(+ self, team_id: int, id: int, *, stale_access_token: str | None = None+ ) -> IntegrationRecord:+ # Not thread-sensitive: the provider call can be slow, and it must not hold the shared thread+ # that every other sync database call of this event loop waits for. The row lock still applies.+ return await database_sync_to_async(self.refresh_access_token, thread_sensitive=False)(+ team_id, id, stale_access_token=stale_access_token+ )+++class KindAccessor:+ """The base of the generated accessor of one kind. Each subclass has one attribute per supported auth type."""++ kind: ClassVar[str]+ any: IntegrationReader+++class SourceKindAccessor:+ """The rows of one warehouse source kind. Only rows with `config["warehouse_source_type"]` are in scope."""++ def __init__(self, kind: str) -> None:+ self.kind = kind+ self.oauth = AuthAccessor(kind=kind, auth_type=AuthType.OAUTH, source_rows=SourceRows.ONLY)+ self.token = AuthAccessor(kind=kind, auth_type=AuthType.TOKEN, source_rows=SourceRows.ONLY)+ self.credentials = AuthAccessor(kind=kind, auth_type=AuthType.CREDENTIALS, source_rows=SourceRows.ONLY)+ self.any = IntegrationReader(kinds=frozenset({kind}), source_rows=SourceRows.ONLY)+++class IntegrationManagerBase:+ """The hand-written part of `IntegrationManager`. The generated subclass adds one attribute per kind."""++ # Every row of every kind and every auth type, without warehouse source rows. For reads by id with a+ # kind that the caller does not know. `IntegrationManager.any.with_source_rows()` adds the source rows.+ any = IntegrationReader(kinds=None)++ def source(self, kind: str) -> SourceKindAccessor:+ """The warehouse source rows of a registered source kind. Raises `UnknownIntegrationKind`."""+ if not is_source_kind(kind):+ raise UnknownIntegrationKind(f"No warehouse source registered the integration kind {kind!r}")+ return SourceKindAccessor(kind)++ def kinds(+ self, *kinds: str, auth_types: Iterable[AuthType] | Literal["any", "default"] = "default"+ ) -> IntegrationReader:+ """Reads of the named rows of several catalogue kinds.++ By default it reads the default auth type of each kind, because code that names kinds expects+ their usual config shape. `auth_types="any"` reads every auth type. A list reads those auth types.+ """+ unknown = sorted(set(kinds) - set(KINDS))+ if not kinds or unknown:+ raise UnknownIntegrationKind(f"Unknown integration kinds: {unknown or 'none given'}")+ if auth_types == "any":+ return IntegrationReader(kinds=frozenset(kinds))+ if auth_types == "default":+ return IntegrationReader(kinds=frozenset(kinds), auth_type="default")+ return IntegrationReader(kinds=frozenset(kinds), auth_type=frozenset(AuthType(t) for t in auth_types))diff --git a/posthog/integrations/catalogue.py b/posthog/integrations/catalogue.pynew file mode 100644index 000000000000..c2252bc0777d--- /dev/null+++ b/posthog/integrations/catalogue.py@@ -0,0 +1,74 @@+"""The integration kinds and the auth types that each kind supports.++This is the hand-written source of `kinds_generated.py`. After a change here, run+`hogli build:projections --only integration-kinds` and commit the result. A kind also needs a value+in `Integration.IntegrationKind` (`posthog/models/integration/model.py`); a test compares the two.++A row stores its auth type in `config["auth_type"]` only when it is not the default of its kind, so+the default of a kind is stored data. Do not change the default of a kind that has rows.++Warehouse source kinds are not in this file. The warehouse product registers them at startup with+`register_source_kinds`.+"""++from collections.abc import Mapping++from posthog.integrations.contracts import KindSpec+from posthog.integrations.enums import AuthType++_OAUTH = AuthType.OAUTH+_TOKEN = AuthType.TOKEN+_CREDENTIALS = AuthType.CREDENTIALS++KINDS: Mapping[str, KindSpec] = {+ "anthropic": KindSpec(label="Anthropic", default=_TOKEN),+ "apns": KindSpec(label="Apple Push Notification service", default=_CREDENTIALS),+ "apple-ads": KindSpec(label="Apple Ads", default=_OAUTH, others=frozenset({_CREDENTIALS})),+ "aws-redshift": KindSpec(label="Amazon Redshift", default=_CREDENTIALS),+ "aws-s3": KindSpec(label="Amazon S3", default=_CREDENTIALS),+ "azure-blob": KindSpec(label="Azure Blob Storage", default=_CREDENTIALS),+ "bing-ads": KindSpec(label="Bing Ads", default=_OAUTH),+ "clickup": KindSpec(label="ClickUp", default=_OAUTH, others=frozenset({_TOKEN})),+ "customerio-app": KindSpec(label="Customer.io App API", default=_TOKEN),+ "customerio-track": KindSpec(label="Customer.io Track API", default=_TOKEN),+ "customerio-webhook": KindSpec(label="Customer.io webhook", default=_TOKEN),+ "databricks": KindSpec(label="Databricks", default=_CREDENTIALS, others=frozenset({_TOKEN})),+ "email": KindSpec(label="Email", default=_CREDENTIALS),+ "firebase": KindSpec(label="Firebase", default=_CREDENTIALS),+ "github": KindSpec(label="GitHub", default=_OAUTH, others=frozenset({_TOKEN})),+ "gitlab": KindSpec(label="GitLab", default=_TOKEN),+ "google-ads": KindSpec(label="Google Ads", default=_OAUTH),+ "google-analytics": KindSpec(label="Google Analytics", default=_OAUTH),+ "google-calendar": KindSpec(label="Google Calendar", default=_OAUTH),+ "google-cloud-service-account": KindSpec(label="Google Cloud service account", default=_CREDENTIALS),+ "google-cloud-storage": KindSpec(label="Google Cloud Storage", default=_CREDENTIALS),+ "google-pubsub": KindSpec(label="Google Pub/Sub", default=_CREDENTIALS),+ "google-search-console": KindSpec(label="Google Search Console", default=_OAUTH),+ "google-sheets": KindSpec(label="Google Sheets", default=_OAUTH, others=frozenset({_CREDENTIALS})),+ "helpscout": KindSpec(label="Help Scout", default=_OAUTH),+ "hubspot": KindSpec(label="HubSpot", default=_OAUTH),+ "instagram": KindSpec(label="Instagram", default=_OAUTH),+ "intercom": KindSpec(label="Intercom", default=_OAUTH),+ "jira": KindSpec(label="Jira", default=_OAUTH, others=frozenset({_TOKEN})),+ "linear": KindSpec(label="Linear", default=_OAUTH),+ "linkedin-ads": KindSpec(label="LinkedIn Ads", default=_OAUTH),+ "meta-ads": KindSpec(label="Meta Ads", default=_OAUTH),+ "pardot": KindSpec(label="Pardot", default=_OAUTH),+ "pinterest-ads": KindSpec(label="Pinterest Ads", default=_OAUTH),+ "postgresql": KindSpec(label="PostgreSQL", default=_CREDENTIALS),+ "posthog": KindSpec(label="PostHog", default=_OAUTH),+ "reddit-ads": KindSpec(label="Reddit Ads", default=_OAUTH),+ "resend": KindSpec(label="Resend", default=_OAUTH, others=frozenset({_TOKEN})),+ "s3-compatible": KindSpec(label="S3-compatible storage", default=_CREDENTIALS),+ "salesforce": KindSpec(label="Salesforce", default=_OAUTH),+ "slack": KindSpec(label="Slack", default=_OAUTH, others=frozenset({_TOKEN})),+ "slack-posthog-code": KindSpec(label="Slack (PostHog Code)", default=_OAUTH),+ "snapchat": KindSpec(label="Snapchat", default=_OAUTH),+ "snowflake": KindSpec(label="Snowflake", default=_CREDENTIALS),+ "stripe": KindSpec(label="Stripe", default=_OAUTH, others=frozenset({_TOKEN})),+ "tiktok-ads": KindSpec(label="TikTok Ads", default=_OAUTH),+ "twilio": KindSpec(label="Twilio", default=_TOKEN),+ "twitter-ads": KindSpec(label="X Ads", default=_OAUTH),+ "vercel": KindSpec(label="Vercel", default=_OAUTH, others=frozenset({_TOKEN})),+ "youtube-analytics": KindSpec(label="YouTube Analytics", default=_OAUTH),+}diff --git a/posthog/integrations/contracts.py b/posthog/integrations/contracts.pynew file mode 100644index 000000000000..21cce799df3d--- /dev/null+++ b/posthog/integrations/contracts.py@@ -0,0 +1,142 @@+"""Data shapes and errors of the integrations SDK."""++import dataclasses+from collections.abc import Mapping+from datetime import datetime+from typing import Any, Protocol++from posthog.dataclasses import frozen+from posthog.integrations.enums import AuthType++# A value that `find`, `list` and `exists` can compare a config key with. Equality only.+ConfigValue = str | int | float | bool | None++# A row that holds the typed-in credentials of a warehouse source stores the source type under this+# config key. Only `IntegrationManager.source(kind)` and a reader from `with_source_rows()` read these rows.+SOURCE_MARKER_CONFIG_KEY = "warehouse_source_type"+++class IntegrationRecord(Protocol):+ """One integration row, as the SDK returns it. Read-only.++ At runtime the SDK returns the Django model, but callers see only these reads. So mypy rejects+ `.save()`, `.delete()`, managers and writes to `config`. Write through the SDK, for example with+ `update` or `upsert`. Later the SDK returns a frozen record with the same reads, so callers do not+ change. Add a read here only when callers need it, and never a related row: use `team_id` and+ `created_by_id`, and load the team or the user from their own models.+ """++ @property+ def id(self) -> int: ...++ @property+ def team_id(self) -> int: ...++ @property+ def kind(self) -> str: ...++ # The id of the connection in the external system, for example a GitHub installation id.+ @property+ def integration_id(self) -> str | None: ...++ @property+ def config(self) -> Mapping[str, Any]: ...++ @property+ def sensitive_config(self) -> Mapping[str, Any]: ...++ @property+ def errors(self) -> str: ...++ @property+ def created_at(self) -> datetime: ...++ @property+ def created_by_id(self) -> int | None: ...++ # `config["auth_type"]`, or the default auth type of the kind when the row does not store one.+ @property+ def auth_type(self) -> str: ...++ # The decrypted `sensitive_config["access_token"]`. Raises `UndecryptedIntegrationSecretError`+ # when the stored value cannot be decrypted.+ @property+ def access_token(self) -> str | None: ...++ @property+ def refresh_token(self) -> str | None: ...++ def can_be_managed_by_creator(self, user_id: int | None) -> bool: ...+++@frozen+class KindSpec:+ """The auth types of one integration kind. A row without `config["auth_type"]` has the default."""++ label: str+ default: AuthType+ others: frozenset[AuthType] = frozenset()++ @property+ def supported(self) -> frozenset[AuthType]:+ return self.others | {self.default}+++@frozen+class IntegrationPatch:+ """A change to one row. `update` and `update_by_external_id` apply it.++ `errors=None` keeps the stored errors. Pass `""` to clear them.+ """++ config_patch: Mapping[str, object] = dataclasses.field(default_factory=dict)+ config_remove: tuple[str, ...] = ()+ sensitive_config_patch: Mapping[str, object] = dataclasses.field(default_factory=dict)+ sensitive_config_remove: tuple[str, ...] = ()+ errors: str | None = None+++@frozen+class IntegrationUsage:+ """One PostHog object that uses an integration row, such as a warehouse source or a workflow."""++ # The kind of object, for example "warehouse_source". The product that reports it chooses the value.+ type: str+ id: str+ name: str+++class IntegrationNotFound(LookupError):+ """The row does not exist, belongs to another team, or is not in the scope of the accessor."""+++class UnknownIntegrationKind(ValueError):+ """The kind is not in the catalogue, and no product registered it."""+++class UnsupportedIntegrationFilter(ValueError):+ """`where` holds a key or a value that the SDK cannot compare.++ The SDK compares named config keys for equality. Anything more needs its own SDK method.+ """+++class UnsupportedAuthType(ValueError):+ """The kind does not support the auth type of the write."""+++class IntegrationConflict(ValueError):+ """A write hit the unique key (team, kind, external id) of a row.++ `create` raises it for any row with the same key. `upsert` raises it for a row with the same key+ that the accessor cannot write, for example a row with another auth type, or a warehouse source+ row found through a named accessor.+ """+++class IntegrationInUse(Exception):+ """`delete` found PostHog objects that use the row."""++ def __init__(self, usages: list[IntegrationUsage]) -> None:+ super().__init__(f"The integration has {len(usages)} usages")+ self.usages = usagesdiff --git a/posthog/integrations/enums.py b/posthog/integrations/enums.pynew file mode 100644index 000000000000..b24488121cbc--- /dev/null+++ b/posthog/integrations/enums.py@@ -0,0 +1,21 @@+"""Enums of the integrations SDK."""++from posthog.enums import LabeledStrEnum+++class IntegrationAuthType(LabeledStrEnum):+ """How an integration row authenticates with its provider.++ A kind names the provider. One kind can support several auth types, and has one default.+ """++ # A connection that the provider manages: an OAuth grant or an app installation. It has scopes or permissions.+ OAUTH = "oauth", "OAuth"+ # One static secret string: an API key, a personal access token, a bot token or a webhook secret.+ TOKEN = "token", "Token"+ # Several typed-in fields: a host, a user and a password, an access key and a secret, or a service account.+ CREDENTIALS = "credentials", "Credentials"+++# The short name for callers. The class keeps the long name, because the OpenAPI enum name comes from it.+AuthType = IntegrationAuthTypediff --git a/posthog/integrations/fields.py b/posthog/integrations/fields.pynew file mode 100644index 000000000000..2e20c741bdfc--- /dev/null+++ b/posthog/integrations/fields.py@@ -0,0 +1,98 @@+"""DRF fields that reference an integration row. Not in `posthog.integrations`, so the SDK does not load DRF."""++from collections.abc import Iterable+from typing import Any, Literal++from drf_spectacular.types import OpenApiTypes+from drf_spectacular.utils import extend_schema_field+from rest_framework import serializers+from rest_framework.fields import empty, get_attribute+from rest_framework.relations import PKOnlyObject++from posthog.integrations.contracts import IntegrationNotFound, IntegrationRecord+from posthog.integrations.enums import AuthType+from posthog.integrations.manager import IntegrationManager+++@extend_schema_field(OpenApiTypes.INT)+class IntegrationReferenceField(serializers.Field):+ """An integration id of the request's team. Use it instead of a `PrimaryKeyRelatedField` on `Integration`.++ It reads the team from the serializer context (`team_id`, or `get_team`), so a row of another team+ is "does not exist". Pass `kinds` to accept only rows of these catalogue kinds. The field refuses+ warehouse source rows, the same as `IntegrationManager.any` and `IntegrationManager.kinds(...)`.++ With `kinds`, `auth_types` selects the auth types the field accepts. The default (None) accepts only+ the default auth type of each kind, because code that reads a kind expects its usual config shape.+ Pass a list of auth types, or `"any"` for every auth type. Without `kinds`, the field accepts every row.+ """++ default_error_messages = {+ "does_not_exist": 'Invalid pk "{pk_value}" - object does not exist.',+ "incorrect_type": "Incorrect type. Expected pk value, received {data_type}.",+ "no_team": "The serializer context has no team.",+ }++ def __init__(+ self,+ *,+ kinds: Iterable[str] = (),+ auth_types: Iterable[AuthType] | Literal["any"] | None = None,+ **kwargs: Any,+ ) -> None:+ self.kinds = frozenset(kinds)+ if not self.kinds:+ if auth_types is not None:+ raise ValueError("IntegrationReferenceField needs kinds to filter by auth type")+ self._rows = IntegrationManager.any+ else:+ self._rows = IntegrationManager.kinds(+ *sorted(self.kinds), auth_types="default" if auth_types is None else auth_types+ )+ super().__init__(**kwargs)++ def _team_id(self) -> int:+ team_id = self.context.get("team_id")+ if team_id is None and "get_team" in self.context:+ team_id = self.context["get_team"]().id+ if team_id is None:+ self.fail("no_team")+ return int(team_id)++ def run_validation(self, data: object = empty) -> object:+ # Same as DRF's `RelatedField`: an empty string means no row, so `allow_null` and `required` decide.+ if data == "":+ data = None+ return super().run_validation(data)++ def to_internal_value(self, data: object) -> IntegrationRecord:+ if isinstance(data, bool) or not isinstance(data, int | str):+ self.fail("incorrect_type", data_type=type(data).__name__)+ try:+ integration_id = int(data)+ except ValueError:+ self.fail("incorrect_type", data_type=type(data).__name__)+ try:+ return self._rows.get(self._team_id(), integration_id)+ except IntegrationNotFound:+ self.fail("does_not_exist", pk_value=data)++ def get_attribute(self, instance: object) -> object:+ # Read the foreign key column instead of the related row, so a list does not query once per row.+ if self.source_attrs:+ try:+ parent = get_attribute(instance, self.source_attrs[:-1])+ value = parent.serializable_value(self.source_attrs[-1])+ except AttributeError:+ pass+ else:+ # `serializable_value` returns the attribute itself when it is not a model field.+ return PKOnlyObject(pk=getattr(value, "pk", value))+ return super().get_attribute(instance)++ def to_representation(self, value: IntegrationRecord | PKOnlyObject | int) -> int:+ if isinstance(value, int):+ return value+ if isinstance(value, PKOnlyObject):+ return value.pk+ return value.iddiff --git a/posthog/integrations/kinds_generated.py b/posthog/integrations/kinds_generated.pynew file mode 100644index 000000000000..7253484d034c--- /dev/null+++ b/posthog/integrations/kinds_generated.py@@ -0,0 +1,432 @@+"""Generated from `posthog/integrations/catalogue.py`. Do not edit by hand.++Run `hogli build:projections --only integration-kinds` after a change to the catalogue. Import+`IntegrationManager` from `posthog.integrations`, not the classes here.+"""++from enum import StrEnum++from posthog.integrations.accessors import (+ AuthAccessor,+ IntegrationManagerBase,+ IntegrationReader,+ KindAccessor,+ OAuthAccessor,+)+from posthog.integrations.enums import AuthType+++class IntegrationKind(StrEnum):+ ANTHROPIC = "anthropic"+ APNS = "apns"+ APPLE_ADS = "apple-ads"+ AWS_REDSHIFT = "aws-redshift"+ AWS_S3 = "aws-s3"+ AZURE_BLOB = "azure-blob"+ BING_ADS = "bing-ads"+ CLICKUP = "clickup"+ CUSTOMERIO_APP = "customerio-app"+ CUSTOMERIO_TRACK = "customerio-track"+ CUSTOMERIO_WEBHOOK = "customerio-webhook"+ DATABRICKS = "databricks"+ EMAIL = "email"+ FIREBASE = "firebase"+ GITHUB = "github"+ GITLAB = "gitlab"+ GOOGLE_ADS = "google-ads"+ GOOGLE_ANALYTICS = "google-analytics"+ GOOGLE_CALENDAR = "google-calendar"+ GOOGLE_CLOUD_SERVICE_ACCOUNT = "google-cloud-service-account"+ GOOGLE_CLOUD_STORAGE = "google-cloud-storage"+ GOOGLE_PUBSUB = "google-pubsub"+ GOOGLE_SEARCH_CONSOLE = "google-search-console"+ GOOGLE_SHEETS = "google-sheets"+ HELPSCOUT = "helpscout"+ HUBSPOT = "hubspot"+ INSTAGRAM = "instagram"+ INTERCOM = "intercom"+ JIRA = "jira"+ LINEAR = "linear"+ LINKEDIN_ADS = "linkedin-ads"+ META_ADS = "meta-ads"+ PARDOT = "pardot"+ PINTEREST_ADS = "pinterest-ads"+ POSTGRESQL = "postgresql"+ POSTHOG = "posthog"+ REDDIT_ADS = "reddit-ads"+ RESEND = "resend"+ S3_COMPATIBLE = "s3-compatible"+ SALESFORCE = "salesforce"+ SLACK = "slack"+ SLACK_POSTHOG_CODE = "slack-posthog-code"+ SNAPCHAT = "snapchat"+ SNOWFLAKE = "snowflake"+ STRIPE = "stripe"+ TIKTOK_ADS = "tiktok-ads"+ TWILIO = "twilio"+ TWITTER_ADS = "twitter-ads"+ VERCEL = "vercel"+ YOUTUBE_ANALYTICS = "youtube-analytics"+++class AnthropicKind(KindAccessor):+ kind = "anthropic"+ token = AuthAccessor(kind="anthropic", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"anthropic"}))+++class ApnsKind(KindAccessor):+ kind = "apns"+ credentials = AuthAccessor(kind="apns", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"apns"}))+++class AppleAdsKind(KindAccessor):+ kind = "apple-ads"+ oauth = OAuthAccessor(kind="apple-ads") # the default+ credentials = AuthAccessor(kind="apple-ads", auth_type=AuthType.CREDENTIALS)+ any = IntegrationReader(kinds=frozenset({"apple-ads"}))+++class AwsRedshiftKind(KindAccessor):+ kind = "aws-redshift"+ credentials = AuthAccessor(kind="aws-redshift", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"aws-redshift"}))+++class AwsS3Kind(KindAccessor):+ kind = "aws-s3"+ credentials = AuthAccessor(kind="aws-s3", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"aws-s3"}))+++class AzureBlobKind(KindAccessor):+ kind = "azure-blob"+ credentials = AuthAccessor(kind="azure-blob", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"azure-blob"}))+++class BingAdsKind(KindAccessor):+ kind = "bing-ads"+ oauth = OAuthAccessor(kind="bing-ads") # the default+ any = IntegrationReader(kinds=frozenset({"bing-ads"}))+++class ClickupKind(KindAccessor):+ kind = "clickup"+ oauth = OAuthAccessor(kind="clickup") # the default+ token = AuthAccessor(kind="clickup", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"clickup"}))+++class CustomerioAppKind(KindAccessor):+ kind = "customerio-app"+ token = AuthAccessor(kind="customerio-app", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"customerio-app"}))+++class CustomerioTrackKind(KindAccessor):+ kind = "customerio-track"+ token = AuthAccessor(kind="customerio-track", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"customerio-track"}))+++class CustomerioWebhookKind(KindAccessor):+ kind = "customerio-webhook"+ token = AuthAccessor(kind="customerio-webhook", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"customerio-webhook"}))+++class DatabricksKind(KindAccessor):+ kind = "databricks"+ token = AuthAccessor(kind="databricks", auth_type=AuthType.TOKEN)+ credentials = AuthAccessor(kind="databricks", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"databricks"}))+++class EmailKind(KindAccessor):+ kind = "email"+ credentials = AuthAccessor(kind="email", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"email"}))+++class FirebaseKind(KindAccessor):+ kind = "firebase"+ credentials = AuthAccessor(kind="firebase", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"firebase"}))+++class GithubKind(KindAccessor):+ kind = "github"+ oauth = OAuthAccessor(kind="github") # the default+ token = AuthAccessor(kind="github", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"github"}))+++class GitlabKind(KindAccessor):+ kind = "gitlab"+ token = AuthAccessor(kind="gitlab", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"gitlab"}))+++class GoogleAdsKind(KindAccessor):+ kind = "google-ads"+ oauth = OAuthAccessor(kind="google-ads") # the default+ any = IntegrationReader(kinds=frozenset({"google-ads"}))+++class GoogleAnalyticsKind(KindAccessor):+ kind = "google-analytics"+ oauth = OAuthAccessor(kind="google-analytics") # the default+ any = IntegrationReader(kinds=frozenset({"google-analytics"}))+++class GoogleCalendarKind(KindAccessor):+ kind = "google-calendar"+ oauth = OAuthAccessor(kind="google-calendar") # the default+ any = IntegrationReader(kinds=frozenset({"google-calendar"}))+++class GoogleCloudServiceAccountKind(KindAccessor):+ kind = "google-cloud-service-account"+ credentials = AuthAccessor(kind="google-cloud-service-account", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"google-cloud-service-account"}))+++class GoogleCloudStorageKind(KindAccessor):+ kind = "google-cloud-storage"+ credentials = AuthAccessor(kind="google-cloud-storage", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"google-cloud-storage"}))+++class GooglePubsubKind(KindAccessor):+ kind = "google-pubsub"+ credentials = AuthAccessor(kind="google-pubsub", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"google-pubsub"}))+++class GoogleSearchConsoleKind(KindAccessor):+ kind = "google-search-console"+ oauth = OAuthAccessor(kind="google-search-console") # the default+ any = IntegrationReader(kinds=frozenset({"google-search-console"}))+++class GoogleSheetsKind(KindAccessor):+ kind = "google-sheets"+ oauth = OAuthAccessor(kind="google-sheets") # the default+ credentials = AuthAccessor(kind="google-sheets", auth_type=AuthType.CREDENTIALS)+ any = IntegrationReader(kinds=frozenset({"google-sheets"}))+++class HelpscoutKind(KindAccessor):+ kind = "helpscout"+ oauth = OAuthAccessor(kind="helpscout") # the default+ any = IntegrationReader(kinds=frozenset({"helpscout"}))+++class HubspotKind(KindAccessor):+ kind = "hubspot"+ oauth = OAuthAccessor(kind="hubspot") # the default+ any = IntegrationReader(kinds=frozenset({"hubspot"}))+++class InstagramKind(KindAccessor):+ kind = "instagram"+ oauth = OAuthAccessor(kind="instagram") # the default+ any = IntegrationReader(kinds=frozenset({"instagram"}))+++class IntercomKind(KindAccessor):+ kind = "intercom"+ oauth = OAuthAccessor(kind="intercom") # the default+ any = IntegrationReader(kinds=frozenset({"intercom"}))+++class JiraKind(KindAccessor):+ kind = "jira"+ oauth = OAuthAccessor(kind="jira") # the default+ token = AuthAccessor(kind="jira", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"jira"}))+++class LinearKind(KindAccessor):+ kind = "linear"+ oauth = OAuthAccessor(kind="linear") # the default+ any = IntegrationReader(kinds=frozenset({"linear"}))+++class LinkedinAdsKind(KindAccessor):+ kind = "linkedin-ads"+ oauth = OAuthAccessor(kind="linkedin-ads") # the default+ any = IntegrationReader(kinds=frozenset({"linkedin-ads"}))+++class MetaAdsKind(KindAccessor):+ kind = "meta-ads"+ oauth = OAuthAccessor(kind="meta-ads") # the default+ any = IntegrationReader(kinds=frozenset({"meta-ads"}))+++class PardotKind(KindAccessor):+ kind = "pardot"+ oauth = OAuthAccessor(kind="pardot") # the default+ any = IntegrationReader(kinds=frozenset({"pardot"}))+++class PinterestAdsKind(KindAccessor):+ kind = "pinterest-ads"+ oauth = OAuthAccessor(kind="pinterest-ads") # the default+ any = IntegrationReader(kinds=frozenset({"pinterest-ads"}))+++class PostgresqlKind(KindAccessor):+ kind = "postgresql"+ credentials = AuthAccessor(kind="postgresql", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"postgresql"}))+++class PosthogKind(KindAccessor):+ kind = "posthog"+ oauth = OAuthAccessor(kind="posthog") # the default+ any = IntegrationReader(kinds=frozenset({"posthog"}))+++class RedditAdsKind(KindAccessor):+ kind = "reddit-ads"+ oauth = OAuthAccessor(kind="reddit-ads") # the default+ any = IntegrationReader(kinds=frozenset({"reddit-ads"}))+++class ResendKind(KindAccessor):+ kind = "resend"+ oauth = OAuthAccessor(kind="resend") # the default+ token = AuthAccessor(kind="resend", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"resend"}))+++class S3CompatibleKind(KindAccessor):+ kind = "s3-compatible"+ credentials = AuthAccessor(kind="s3-compatible", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"s3-compatible"}))+++class SalesforceKind(KindAccessor):+ kind = "salesforce"+ oauth = OAuthAccessor(kind="salesforce") # the default+ any = IntegrationReader(kinds=frozenset({"salesforce"}))+++class SlackKind(KindAccessor):+ kind = "slack"+ oauth = OAuthAccessor(kind="slack") # the default+ token = AuthAccessor(kind="slack", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"slack"}))+++class SlackPosthogCodeKind(KindAccessor):+ kind = "slack-posthog-code"+ oauth = OAuthAccessor(kind="slack-posthog-code") # the default+ any = IntegrationReader(kinds=frozenset({"slack-posthog-code"}))+++class SnapchatKind(KindAccessor):+ kind = "snapchat"+ oauth = OAuthAccessor(kind="snapchat") # the default+ any = IntegrationReader(kinds=frozenset({"snapchat"}))+++class SnowflakeKind(KindAccessor):+ kind = "snowflake"+ credentials = AuthAccessor(kind="snowflake", auth_type=AuthType.CREDENTIALS) # the default+ any = IntegrationReader(kinds=frozenset({"snowflake"}))+++class StripeKind(KindAccessor):+ kind = "stripe"+ oauth = OAuthAccessor(kind="stripe") # the default+ token = AuthAccessor(kind="stripe", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"stripe"}))+++class TiktokAdsKind(KindAccessor):+ kind = "tiktok-ads"+ oauth = OAuthAccessor(kind="tiktok-ads") # the default+ any = IntegrationReader(kinds=frozenset({"tiktok-ads"}))+++class TwilioKind(KindAccessor):+ kind = "twilio"+ token = AuthAccessor(kind="twilio", auth_type=AuthType.TOKEN) # the default+ any = IntegrationReader(kinds=frozenset({"twilio"}))+++class TwitterAdsKind(KindAccessor):+ kind = "twitter-ads"+ oauth = OAuthAccessor(kind="twitter-ads") # the default+ any = IntegrationReader(kinds=frozenset({"twitter-ads"}))+++class VercelKind(KindAccessor):+ kind = "vercel"+ oauth = OAuthAccessor(kind="vercel") # the default+ token = AuthAccessor(kind="vercel", auth_type=AuthType.TOKEN)+ any = IntegrationReader(kinds=frozenset({"vercel"}))+++class YoutubeAnalyticsKind(KindAccessor):+ kind = "youtube-analytics"+ oauth = OAuthAccessor(kind="youtube-analytics") # the default+ any = IntegrationReader(kinds=frozenset({"youtube-analytics"}))+++class GeneratedIntegrationManager(IntegrationManagerBase):+ anthropic = AnthropicKind()+ apns = ApnsKind()+ apple_ads = AppleAdsKind()+ aws_redshift = AwsRedshiftKind()+ aws_s3 = AwsS3Kind()+ azure_blob = AzureBlobKind()+ bing_ads = BingAdsKind()+ clickup = ClickupKind()+ customerio_app = CustomerioAppKind()+ customerio_track = CustomerioTrackKind()+ customerio_webhook = CustomerioWebhookKind()+ databricks = DatabricksKind()+ email = EmailKind()+ firebase = FirebaseKind()+ github = GithubKind()+ gitlab = GitlabKind()+ google_ads = GoogleAdsKind()+ google_analytics = GoogleAnalyticsKind()+ google_calendar = GoogleCalendarKind()+ google_cloud_service_account = GoogleCloudServiceAccountKind()+ google_cloud_storage = GoogleCloudStorageKind()+ google_pubsub = GooglePubsubKind()+ google_search_console = GoogleSearchConsoleKind()+ google_sheets = GoogleSheetsKind()+ helpscout = HelpscoutKind()+ hubspot = HubspotKind()+ instagram = InstagramKind()+ intercom = IntercomKind()+ jira = JiraKind()+ linear = LinearKind()+ linkedin_ads = LinkedinAdsKind()+ meta_ads = MetaAdsKind()+ pardot = PardotKind()+ pinterest_ads = PinterestAdsKind()+ postgresql = PostgresqlKind()+ posthog = PosthogKind()+ reddit_ads = RedditAdsKind()+ resend = ResendKind()+ s3_compatible = S3CompatibleKind()+ salesforce = SalesforceKind()+ slack = SlackKind()+ slack_posthog_code = SlackPosthogCodeKind()+ snapchat = SnapchatKind()+ snowflake = SnowflakeKind()+ stripe = StripeKind()+ tiktok_ads = TiktokAdsKind()+ twilio = TwilioKind()+ twitter_ads = TwitterAdsKind()+ vercel = VercelKind()+ youtube_analytics = YoutubeAnalyticsKind()diff --git a/posthog/integrations/kinds_projection.py b/posthog/integrations/kinds_projection.pynew file mode 100644index 000000000000..9be86b95bf50--- /dev/null+++ b/posthog/integrations/kinds_projection.py@@ -0,0 +1,73 @@+"""Render `kinds_generated.py`, the typed accessors of the integrations SDK, from `catalogue.py`.++A renderer in the projection registry (`tools/hogli-commands/hogli_commands/projections.py`). Run+`hogli build:projections --only integration-kinds` after a change to the catalogue.++Each kind gets a class with one attribute per auth type that it supports, plus `any`. So mypy rejects+an auth type that a kind does not support, for example `IntegrationManager.github.credentials`.+"""++from posthog.integrations.catalogue import KINDS+from posthog.integrations.contracts import KindSpec+from posthog.integrations.enums import AuthType++OUTPUT = "posthog/integrations/kinds_generated.py"++HEADER = '''"""Generated from `posthog/integrations/catalogue.py`. Do not edit by hand.++Run `hogli build:projections --only integration-kinds` after a change to the catalogue. Import+`IntegrationManager` from `posthog.integrations`, not the classes here.+"""++from enum import StrEnum++from posthog.integrations.accessors import (+ AuthAccessor,+ IntegrationManagerBase,+ IntegrationReader,+ KindAccessor,+ OAuthAccessor,+)+from posthog.integrations.enums import AuthType+'''+++def _attribute(kind: str) -> str:+ return kind.replace("-", "_")+++def _class_name(kind: str) -> str:+ return "".join(part.capitalize() for part in kind.split("-")) + "Kind"+++def _kind_class(kind: str, spec: KindSpec) -> list[str]:+ lines = ["", "", f"class {_class_name(kind)}(KindAccessor):", f' kind = "{kind}"']+ for auth_type in AuthType:+ if auth_type not in spec.supported:+ continue+ note = " # the default" if auth_type == spec.default else ""+ if auth_type == AuthType.OAUTH:+ lines.append(f' oauth = OAuthAccessor(kind="{kind}"){note}')+ else:+ lines.append(+ f' {auth_type.value} = AuthAccessor(kind="{kind}", auth_type=AuthType.{auth_type.name}){note}'+ )+ lines.append(f' any = IntegrationReader(kinds=frozenset({{"{kind}"}}))')+ return lines+++def render_kinds() -> str:+ kinds = sorted(KINDS.items())+ # A plain StrEnum: the OpenAPI enum names come from the model's `Integration.IntegrationKind`, and a+ # second labeled class with the same name would make both names ambiguous.+ lines = [HEADER, "", "class IntegrationKind(StrEnum):"]+ lines += [f' {_attribute(kind).upper()} = "{kind}"' for kind, _ in kinds]+ for kind, spec in kinds:+ lines += _kind_class(kind, spec)+ lines += ["", "", "class GeneratedIntegrationManager(IntegrationManagerBase):"]+ lines += [f" {_attribute(kind)} = {_class_name(kind)}()" for kind, _ in kinds]+ return "\n".join(lines) + "\n"+++def render() -> dict[str, str]:+ return {OUTPUT: render_kinds()}diff --git a/posthog/integrations/manager.py b/posthog/integrations/manager.pynew file mode 100644index 000000000000..45839fa1c0c7--- /dev/null+++ b/posthog/integrations/manager.py@@ -0,0 +1,38 @@+"""The entry point of the integrations SDK. Import the names from `posthog.integrations`, not from here.++ from posthog.integrations import IntegrationManager++ IntegrationManager.github.oauth.get(team_id, integration_id) # one kind, one auth type: reads and writes+ IntegrationManager.stripe.any.list(team_id) # one kind, every auth type: reads and deletes+ IntegrationManager.any.get(team_id, integration_id) # every kind: reads and deletes+ IntegrationManager.any.with_source_rows().delete(team_id, id) # every kind, warehouse source rows too+ IntegrationManager.source("ab-tasty").token.create(...) # warehouse source rows of a registered kind++A kind class has an attribute only for the auth types that the kind supports, so mypy rejects+`IntegrationManager.github.credentials`. `.any` has no writes, because a new row has exactly one auth+type. See `posthog/integrations/README.md`.+"""++from collections.abc import Mapping++from posthog.integrations import accessors, source_kinds, usages+from posthog.integrations.enums import AuthType+from posthog.integrations.kinds_generated import GeneratedIntegrationManager++IntegrationReader = accessors.IntegrationReader+AuthAccessor = accessors.AuthAccessor+OAuthAccessor = accessors.OAuthAccessor+SourceKindAccessor = accessors.SourceKindAccessor+UsageProvider = usages.UsageProvider++IntegrationManager = GeneratedIntegrationManager()+++def register_source_kinds(kinds: Mapping[str, AuthType]) -> None:+ """Register warehouse source kinds and the default auth type of each. Call it from `AppConfig.ready`."""+ source_kinds.register_source_kinds(kinds)+++def register_usage_provider(provider: UsageProvider) -> None:+ """Add a provider for `usages`. A product adds `backend/integration_usages.py` instead."""+ usages.register_usage_provider(provider)diff --git a/posthog/integrations/model_access_baseline.txt b/posthog/integrations/model_access_baseline.txtnew file mode 100644index 000000000000..ea3da4b65320--- /dev/null+++ b/posthog/integrations/model_access_baseline.txt@@ -0,0 +1,241 @@+# Files that use the `Integration` model outside the integrations SDK, and how many findings each+# file has. Rule: .semgrep/rules/devex/integration-model-through-sdk.yaml. The count only goes down:+# after you move a file to the SDK, lower its count or remove its line.+ee/api/integration.py 2+ee/api/vercel/tasks.py 3+ee/api/vercel/vercel_connect.py 5+ee/api/vercel/vercel_sso.py 2+ee/management/commands/backfill_vercel_secrets.py 2+ee/tasks/slack.py 2+ee/vercel/integration.py 6+posthog/api/comments.py 2+posthog/api/github_callback/installation_events.py 5+posthog/api/github_callback/personal_finish.py 3+posthog/api/github_callback/team_services.py 11+posthog/api/integration.py 7+posthog/api/integration_push_tokens.py 1+posthog/api/organization_integration.py 5+posthog/api/posthog_connection.py 2+posthog/api/team/team_config.py 2+posthog/api/user_integration.py 5+posthog/domain_connect.py 3+posthog/github/attribution.py 1+posthog/github/installations.py 2+posthog/management/commands/flag_legacy_oauth_integrations.py 3+posthog/management/commands/migrate_ses_tenants.py 2+posthog/management/commands/normalize_apns_integrations.py 3+posthog/models/comment/slack_thread.py 1+posthog/models/integration_repository_cache.py 2+posthog/models/user.py 1+posthog/models/user_integration.py 1+posthog/ownership/github_files.py 2+posthog/session_recordings/models/session_recording_external_reference.py 2+posthog/session_recordings/session_recording_api.py 1+posthog/session_recordings/session_recording_external_reference_api.py 2+posthog/slack/channels.py 1+posthog/tasks/comment_slack_sync.py 3+posthog/tasks/integrations.py 11+posthog/temporal/ai_observability/eval_reports/delivery.py 2+products/ai_observability/backend/api/evaluation_reports.py 2+products/alerts/backend/presentation/views/alert.py 2+products/alerts_platform/backend/delivery/slack.py 2+products/batch_exports/backend/debug/debugger.py 1+products/batch_exports/backend/debug/start_session.py 1+products/batch_exports/backend/facade/api.py 2+products/batch_exports/backend/logic/exports.py 3+products/batch_exports/backend/models/batch_export.py 1+products/batch_exports/backend/presentation/views/batch_export/destinations.py 3+products/batch_exports/backend/presentation/views/batch_export/exports.py 1+products/batch_exports/backend/temporal/destinations/azure_blob_batch_export.py 2+products/batch_exports/backend/temporal/destinations/bigquery_batch_export.py 2+products/batch_exports/backend/temporal/destinations/databricks_batch_export.py 2+products/batch_exports/backend/temporal/destinations/postgres_batch_export.py 2+products/batch_exports/backend/temporal/destinations/redshift_batch_export.py 3+products/batch_exports/backend/temporal/destinations/s3_batch_export.py 2+products/batch_exports/backend/temporal/destinations/snowflake_batch_export.py 2+products/billing_alerts/backend/facade/api.py 2+products/business_knowledge/backend/github_repos.py 2+products/canvas/backend/welcome.py 2+products/cdp/backend/validation.py 3+products/commands/backend/channels/slack/mentions.py 3+products/commands/backend/logic/github.py 2+products/commands/backend/logic/subjects.py 2+products/conversations/backend/api/github_setup.py 4+products/conversations/backend/facade/api.py 2+products/conversations/backend/services/gmail_sync.py 7+products/conversations/backend/tasks/github.py 2+products/customer_analytics/backend/facade/api.py 5+products/customer_analytics/backend/logic/calendar_sync.py 5+products/customer_analytics/backend/logic/event_stream_destination.py 2+products/customer_analytics/backend/logic/feature_request_github.py 2+products/customer_analytics/backend/logic/feature_requests.py 2+products/customer_analytics/backend/models/event_stream.py 1+products/customer_analytics/backend/models/feature_request.py 1+products/customer_analytics/backend/temporal/calendar_sync.py 2+products/data_modeling/backend/models/github_sync_config.py 1+products/engineering_analytics/backend/logic/job_log_insights.py 2+products/engineering_analytics/backend/logic/job_logs/activity.py 2+products/engineering_analytics/backend/logic/ownership_files.py 2+products/engineering_analytics/backend/logic/quarantine.py 2+products/error_tracking/backend/logic/__init__.py 1+products/error_tracking/backend/logic/alerts.py 2+products/error_tracking/backend/logic/external_references.py 4+products/error_tracking/backend/logic/github_external_references.py 2+products/error_tracking/backend/models.py 3+products/error_tracking/backend/presentation/views/git_provider_file_link_resolver.py 3+products/error_tracking/backend/temporal/alerts/delivery.py 2+products/growth/backend/product_push/surfaces.py 2+products/messaging/backend/models/optout_sync_config.py 3+products/messaging/backend/models/provider_import_run.py 1+products/messaging/backend/presentation/views/customerio_webhook.py 2+products/messaging/backend/presentation/views/message_templates.py 2+products/messaging/backend/remote_config.py 2+products/messaging/backend/services/customerio_config.py 13+products/messaging/backend/services/customerio_sync_service.py 1+products/messaging/backend/services/customerio_webhook.py 1+products/messaging/backend/services/transactional_email.py 2+products/review_hog/backend/activity_logging.py 2+products/review_hog/backend/automatic_reviews.py 3+products/review_hog/backend/label_reviews.py 2+products/review_hog/backend/ownership.py 2+products/review_hog/backend/repository_settings.py 2+products/review_hog/backend/reviewer/status_comment.py 2+products/review_hog/backend/temporal/activities.py 2+products/review_hog/backend/temporal/resolution.py 2+products/signals/backend/github_optional.py 2+products/signals/backend/inbox_summary.py 1+products/signals/backend/models.py 4+products/signals/backend/onboarding_completion.py 2+products/signals/backend/report_generation/resolve_reviewers.py 3+products/signals/backend/scout_harness/profile/builders.py 2+products/signals/backend/scout_harness/serializers.py 2+products/signals/backend/scout_harness/slack_delivery.py 2+products/signals/backend/serializers.py 2+products/signals/backend/slack_inbox_notifications.py 4+products/signals/backend/slack_notification_targets.py 2+products/signals/backend/temporal/agentic/select_repository.py 2+products/signals/backend/tracker_issues.py 4+products/signals/backend/views.py 2+products/skills/backend/api/community_publish_services.py 1+products/slack_app/backend/analytics.py 1+products/slack_app/backend/api.py 28+products/slack_app/backend/discussion_replies.py 1+products/slack_app/backend/facade/api.py 3+products/slack_app/backend/feature_flags.py 1+products/slack_app/backend/inbox_channel.py 1+products/slack_app/backend/logic/feature_tips/catalog.py 3+products/slack_app/backend/logic/feature_tips/interactivity.py 2+products/slack_app/backend/logic/feature_tips/offer.py 1+products/slack_app/backend/logic/interview_recruitment.py 2+products/slack_app/backend/logic/link_unfurl_charts.py 2+products/slack_app/backend/logic/thread_join.py 1+products/slack_app/backend/models.py 3+products/slack_app/backend/onboarding.py 2+products/slack_app/backend/services/commands.py 3+products/slack_app/backend/services/followup_invite.py 1+products/slack_app/backend/services/inbox_interactivity.py 2+products/slack_app/backend/services/integration_resolver.py 6+products/slack_app/backend/services/project_picker.py 1+products/slack_app/backend/services/slack_app_home.py 6+products/slack_app/backend/services/slack_app_home_stats.py 1+products/slack_app/backend/services/slack_auth.py 1+products/slack_app/backend/services/slack_messages.py 2+products/slack_app/backend/services/slack_scopes.py 1+products/slack_app/backend/services/slack_settings.py 1+products/slack_app/backend/services/slack_user_info.py 3+products/slack_app/backend/services/slack_welcome_messages.py 1+products/slack_app/backend/services/turn_feedback.py 2+products/slack_app/backend/signals.py 2+products/slack_app/backend/slack_link_unfurl.py 1+products/slack_app/backend/slack_thread.py 2+products/slack_app/backend/slack_workflow_events.py 2+products/slack_app/backend/tasks.py 2+products/slack_app/backend/temporal/activities/billing.py 2+products/slack_app/backend/temporal/activities/classifiers.py 6+products/slack_app/backend/temporal/activities/messaging.py 9+products/slack_app/backend/temporal/activities/model_router.py 2+products/slack_app/backend/temporal/activities/onboarding.py 2+products/slack_app/backend/temporal/activities/repo_selection.py 4+products/slack_app/backend/temporal/activities/rules.py 2+products/slack_app/backend/temporal/activities/task_creation.py 4+products/slack_app/backend/temporal/activities/thread.py 2+products/slack_app/backend/temporal/activities/untagged_question.py 2+products/slack_app/backend/temporal/activities/user_resolution.py 2+products/slack_app/backend/temporal/attachments.py 1+products/slack_app/backend/views/slack_user_link.py 2+products/slack_app/evals/eval_project_classifier.py 1+products/stamphog/backend/logic/channel_resolution.py 2+products/stamphog/backend/logic/slack_digest.py 3+products/subscriptions/backend/logic/slack_subscriptions.py 2+products/subscriptions/backend/logic/subscription_writes.py 2+products/subscriptions/backend/models/subscription.py 1+products/subscriptions/backend/temporal/activities.py 2+products/subscriptions/backend/temporal/ai_subscription/delivery.py 1+products/subscriptions/backend/temporal/delivery_common.py 1+products/tasks/backend/facade/api.py 15+products/tasks/backend/github_repository_access.py 2+products/tasks/backend/logic/pull_request_titles.py 2+products/tasks/backend/logic/repo_selection/agent.py 3+products/tasks/backend/logic/services/comment_slack_dm.py 2+products/tasks/backend/logic/services/dev_sandbox_context.py 2+products/tasks/backend/logic/services/pr_reconciliation.py 1+products/tasks/backend/logic/services/workflow_tasks.py 3+products/tasks/backend/logic/task_review.py 3+products/tasks/backend/max_tools.py 4+products/tasks/backend/models.py 5+products/tasks/backend/presentation/serializers.py 6+products/tasks/backend/repository_readiness.py 2+products/tasks/backend/repository_resolver.py 4+products/tasks/backend/temporal/build_image/activities.py 2+products/tasks/backend/temporal/create_snapshot/activities/create_sandbox.py 2+products/tasks/backend/temporal/create_snapshot/activities/get_snapshot_context.py 2+products/tasks/backend/temporal/create_snapshot/utils.py 2+products/tasks/backend/temporal/process_task/activities/create_sandbox_from_snapshot.py 1+products/tasks/backend/temporal/process_task/activities/feature_flags.py 2+products/tasks/backend/temporal/process_task/activities/get_pr_context.py 2+products/tasks/backend/temporal/process_task/activities/get_task_processing_context.py 1+products/tasks/backend/temporal/process_task/activities/provision_sandbox.py 1+products/tasks/backend/temporal/process_task/activities/refresh_sandbox_credentials.py 1+products/tasks/backend/temporal/process_task/activities/send_followup_to_sandbox.py 1+products/tasks/backend/temporal/process_task/activities/start_agent_server.py 2+products/tasks/backend/temporal/process_task/sandbox_credentials.py 1+products/tasks/backend/temporal/process_task/utils.py 3+products/tasks/scripts/run_agent_in_docker.py 2+products/visual_review/backend/logic/github_api.py 2+products/visual_review/backend/logic/team_channels.py 2+products/warehouse_sources/backend/models/external_data_destination.py 1+products/warehouse_sources/backend/presentation/destination_connection_check.py 1+products/warehouse_sources/backend/presentation/destination_connection_testing.py 1+products/warehouse_sources/backend/presentation/views/external_data_destination.py 2+products/warehouse_sources/backend/presentation/views/external_data_source/oauth_accounts.py 2+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/bigquery.py 2+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/databricks.py 2+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/postgres.py 2+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/redshift.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/oauth.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/bigquery.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/common/mixins.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/google_ads.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/google_analytics.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/google_search_console.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/helpscout/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/linkedin_ads/linkedin_ads.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/linkedin_ads/source.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/meta_ads.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/resend/oauth.py 2+products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py 1+products/warehouse_sources/backend/temporal/data_imports/sources/twitter_ads/twitter_ads.py 1+products/wizard/backend/logic/workers/repository_publisher.py 3+products/workflows/backend/management/commands/delete_stale_ses_identity.py 2+products/workflows/backend/management/commands/disable_ses_feedback_forwarding.py 2+products/workflows/backend/services/customerio_import/content.py 3+products/workflows/backend/services/data_suggestions/team_context.py 5+products/workflows/backend/services/email_health.py 3+products/workflows/backend/services/setup_status.py 2+products/workflows/backend/services/transactional_email_workflows.py 2+products/workflows/backend/tasks/ses_tenant_state.py 2diff --git a/posthog/integrations/source_kinds.py b/posthog/integrations/source_kinds.pynew file mode 100644index 000000000000..668805882548--- /dev/null+++ b/posthog/integrations/source_kinds.py@@ -0,0 +1,46 @@+"""Kinds that hold the typed-in credentials of warehouse sources.++There are about a thousand of these kinds, so they get no generated accessor. The warehouse product+registers them at startup, and callers reach their rows through `IntegrationManager.source(kind)`.+A row of a source kind carries the source type in `config["warehouse_source_type"]`.+"""++import threading+from collections.abc import Mapping++from posthog.integrations.catalogue import KINDS+from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY, KindSpec+from posthog.integrations.enums import AuthType+from posthog.models.integration.auth_types import register_kind_auth_types++__all__ = ["SOURCE_MARKER_CONFIG_KEY", "is_source_kind", "register_source_kinds"]++_lock = threading.Lock()+_source_kinds: set[str] = set()+++def register_source_kinds(kinds: Mapping[str, AuthType]) -> None:+ """Register warehouse source kinds with the default auth type of each.++ A kind that is also in the catalogue, such as `stripe` or `postgresql`, keeps its catalogue default.+ The auth type given here only joins the auth types the kind supports, so its source rows can be written.+ """+ with _lock:+ for kind, auth_type in kinds.items():+ catalogued = KINDS.get(kind)+ if catalogued is None:+ register_kind_auth_types(kind, KindSpec(label=kind, default=AuthType(auth_type)))+ elif AuthType(auth_type) not in catalogued.supported:+ register_kind_auth_types(+ kind,+ KindSpec(+ label=catalogued.label,+ default=catalogued.default,+ others=catalogued.others | {AuthType(auth_type)},+ ),+ )+ _source_kinds.add(kind)+++def is_source_kind(kind: str) -> bool:+ return kind in _source_kindsdiff --git a/posthog/integrations/test/__init__.py b/posthog/integrations/test/__init__.pynew file mode 100644index 000000000000..e69de29bb2d1diff --git a/posthog/integrations/test/test_accessors.py b/posthog/integrations/test/test_accessors.pynew file mode 100644index 000000000000..4569fc96fdef--- /dev/null+++ b/posthog/integrations/test/test_accessors.py@@ -0,0 +1,872 @@+import time+import threading+from collections.abc import Callable, Iterable, Iterator+from contextlib import contextmanager+from types import SimpleNamespace+from typing import Any++from posthog.test.base import BaseTest+from unittest.mock import patch++from django.db import connection+from django.test.utils import CaptureQueriesContext++from asgiref.sync import async_to_sync+from parameterized import parameterized++from posthog.integrations import IntegrationManager, register_source_kinds, register_usage_provider, usages+from posthog.integrations.accessors import AuthAccessor, IntegrationReader, OAuthAccessor, SourceRows+from posthog.integrations.contracts import (+ IntegrationConflict,+ IntegrationInUse,+ IntegrationNotFound,+ IntegrationPatch,+ IntegrationRecord,+ IntegrationUsage,+ UnknownIntegrationKind,+ UnsupportedAuthType,+ UnsupportedIntegrationFilter,+)+from posthog.integrations.enums import AuthType+from posthog.models import Team+from posthog.models.integration import ANY, Integration, auth_types+from posthog.models.integration.auth_types import default_auth_type, kind_auth_types+from posthog.models.integration.model import IntegrationQuerySet+from posthog.models.integration.oauth import OauthIntegration++MARKER = "warehouse_source_type"++# name -> (kind, config). The name is also the external id.+ROWS: dict[str, tuple[str, dict[str, Any]]] = {+ "github_oauth": ("github", {"account": {"name": "a"}}),+ "github_token": ("github", {"auth_type": "token", "account": {"name": "a"}}),+ "slack_oauth": ("slack", {}),+ "stripe_oauth": ("stripe", {}),+ "stripe_token": ("stripe", {"auth_type": "token"}),+ "stripe_source": ("stripe", {"auth_type": "token", MARKER: "Stripe"}),+ "postgres": ("postgresql", {"host": "db.example.com"}),+ "postgres_source": ("postgresql", {"host": "db.example.com", MARKER: "Postgres"}),+ "ably_source": ("ably", {MARKER: "Ably"}),+}+SOURCE_ROWS = {name for name, (_kind, config) in ROWS.items() if MARKER in config}+++@contextmanager+def _usage_providers(*providers: usages.UsageProvider) -> Iterator[None]:+ with patch.object(usages, "_registered", []), patch.object(usages, "_product_providers", list(providers)):+ yield+++class TestIntegrationAccessors(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.rows = {+ name: Integration.objects._with_auth(ANY).create(+ team=self.team, kind=kind, integration_id=name, config=config+ )+ for name, (kind, config) in ROWS.items()+ }+ self.other_team = Team.objects.create(organization=self.organization, name="Other")+ self.other_row = Integration.objects._with_auth(ANY).create(+ team=self.other_team, kind="github", integration_id="github_oauth", config={}+ )++ def _names(self, rows: Iterable[IntegrationRecord]) -> set[str]:+ return {row.integration_id for row in rows if row.team_id == self.team.id and row.integration_id}++ @parameterized.expand(+ [+ ("single_auth_default", lambda: IntegrationManager.github.oauth, {"github_oauth"}),+ ("single_auth_non_default", lambda: IntegrationManager.github.token, {"github_token"}),+ ("kind_any", lambda: IntegrationManager.github.any, {"github_oauth", "github_token"}),+ ("skips_source_rows", lambda: IntegrationManager.stripe.token, {"stripe_token"}),+ ("kind_any_skips_source_rows", lambda: IntegrationManager.stripe.any, {"stripe_oauth", "stripe_token"}),+ ("credentials_default", lambda: IntegrationManager.postgresql.credentials, {"postgres"}),+ ("everything_but_source_rows", lambda: IntegrationManager.any, set(ROWS) - SOURCE_ROWS),+ ("everything_with_source_rows", lambda: IntegrationManager.any.with_source_rows(), set(ROWS)),+ (+ "kind_any_with_source_rows",+ lambda: IntegrationManager.stripe.any.with_source_rows(),+ {"stripe_oauth", "stripe_token", "stripe_source"},+ ),+ ("source_of_a_catalogue_kind", lambda: IntegrationManager.source("stripe").token, {"stripe_source"}),+ ("source_any", lambda: IntegrationManager.source("stripe").any, {"stripe_source"}),+ ("source_of_a_registered_kind", lambda: IntegrationManager.source("ably").token, {"ably_source"}),+ ("source_other_auth_type", lambda: IntegrationManager.source("ably").credentials, set()),+ (+ "several_kinds_default_auth_type_by_default",+ lambda: IntegrationManager.kinds("github", "slack"),+ {"github_oauth", "slack_oauth"},+ ),+ (+ "several_kinds_any_auth_type",+ lambda: IntegrationManager.kinds("github", "slack", auth_types="any"),+ {"github_oauth", "github_token", "slack_oauth"},+ ),+ # The source rows join the scope, and the auth type filter still applies to them.+ (+ "several_kinds_default_auth_type_with_source_rows",+ lambda: IntegrationManager.kinds("stripe", "postgresql").with_source_rows(),+ {"stripe_oauth", "postgres", "postgres_source"},+ ),+ (+ "several_kinds_any_auth_type_with_source_rows",+ lambda: IntegrationManager.kinds("stripe", auth_types="any").with_source_rows(),+ {"stripe_oauth", "stripe_token", "stripe_source"},+ ),+ (+ "several_kinds_default_auth_type",+ lambda: IntegrationManager.kinds("github", "stripe", "postgresql", auth_types="default"),+ {"github_oauth", "stripe_oauth", "postgres"},+ ),+ (+ "several_kinds_listed_auth_types",+ lambda: IntegrationManager.kinds("github", "stripe", "postgresql", auth_types=[AuthType.TOKEN]),+ {"github_token", "stripe_token"},+ ),+ ]+ )+ def test_list_returns_only_rows_in_scope(+ self, _name: str, accessor: Callable[[], IntegrationReader], expected: set[str]+ ) -> None:+ assert self._names(accessor().list(self.team.id)) == expected++ @parameterized.expand(+ [+ ("other_auth_type", lambda: IntegrationManager.github.oauth, "github_token"),+ ("source_row_through_named_accessor", lambda: IntegrationManager.stripe.any, "stripe_source"),+ ("named_row_through_source_accessor", lambda: IntegrationManager.source("stripe").token, "stripe_token"),+ ("other_kind", lambda: IntegrationManager.slack.oauth, "github_oauth"),+ ("source_row_through_any", lambda: IntegrationManager.any, "stripe_source"),+ ("other_auth_type_through_kinds", lambda: IntegrationManager.kinds("github"), "github_token"),+ ]+ )+ def test_get_raises_for_a_row_out_of_scope(+ self, _name: str, accessor: Callable[[], IntegrationReader], row_name: str+ ) -> None:+ with self.assertRaises(IntegrationNotFound):+ accessor().get(self.team.id, self.rows[row_name].id)++ @parameterized.expand([("missing_id", 10**9), ("not_a_number", "abc"), ("none", None)])+ def test_get_raises_for_a_bad_id(self, _name: str, row_id: Any) -> None:+ with self.assertRaises(IntegrationNotFound):+ IntegrationManager.any.get(self.team.id, row_id)++ def test_a_stored_default_auth_type_reads_as_the_default(self) -> None:+ row = self.rows["github_oauth"]+ # `QuerySet.update()` skips `save()`, so it can store the default auth type of the kind.+ Integration._base_manager.filter(id=row.id).update(config={"auth_type": "oauth"})++ assert IntegrationManager.github.oauth.get(self.team.id, row.id).id == row.id+ assert self._names(IntegrationManager.kinds("github", auth_types="default").list(self.team.id)) == {+ "github_oauth"+ }+ with self.assertRaises(IntegrationNotFound):+ IntegrationManager.github.token.get(self.team.id, row.id)++ def test_get_does_not_cross_teams(self) -> None:+ with self.assertRaises(IntegrationNotFound):+ IntegrationManager.github.oauth.get(self.team.id, self.other_row.id)++ assert IntegrationManager.github.oauth.get(self.other_team.id, self.other_row.id) == self.other_row++ def test_get_many_drops_ids_out_of_scope(self) -> None:+ ids = [self.rows["github_oauth"].id, self.rows["github_token"].id, self.other_row.id, 10**9]++ assert IntegrationManager.github.oauth.get_many(self.team.id, ids) == {+ self.rows["github_oauth"].id: self.rows["github_oauth"]+ }++ @parameterized.expand(+ [+ ("external_id", {"external_id": "github_token"}, "github_token"),+ ("nested_config_value", {"where": {"account": {"name": "a"}}}, None),+ ("scalar_config_value", {"where": {"auth_type_hint": None}}, None),+ ("oldest_without_filters", {}, "github_oauth"),+ ("no_match", {"external_id": "missing"}, None),+ ]+ )+ def test_find(self, _name: str, kwargs: dict[str, Any], expected: str | None) -> None:+ if isinstance(next(iter(kwargs.get("where", {}).values()), None), dict):+ with self.assertRaises(UnsupportedIntegrationFilter):+ IntegrationManager.github.any.find(self.team.id, **kwargs)+ return+ row = IntegrationManager.github.any.find(self.team.id, **kwargs)++ assert (row.integration_id if row else None) == expected++ @parameterized.expand(+ [+ ("string", {"host": "db.example.com"}, True),+ ("string_mismatch", {"host": "other.example.com"}, False),+ ("missing_key_matches_none", {"port": None}, False),+ ("key_named_like_a_lookup", {"contains": "x"}, False),+ ]+ )+ def test_exists_compares_config_keys_for_equality(self, _name: str, where: dict[str, Any], expected: bool) -> None:+ assert IntegrationManager.postgresql.credentials.exists(self.team.id, where=where) is expected++ def test_find_returns_the_oldest_row_and_list_can_reverse(self) -> None:+ newer = IntegrationManager.github.oauth.create(self.team.id, external_id="newer", config={})++ assert IntegrationManager.github.oauth.find(self.team.id) == self.rows["github_oauth"]+ assert IntegrationManager.github.oauth.list(self.team.id, newest_first=True)[0] == newer++ @parameterized.expand(+ [+ ("every_team", None, {"github_oauth": 2}),+ ("one_team", "other", {"github_oauth": 1}),+ ]+ )+ def test_list_by_external_id(self, _name: str, team_ids: str | None, expected: dict[str, int]) -> None:+ rows = IntegrationManager.github.oauth.list_by_external_id(+ "github_oauth", team_ids=[self.other_team.id] if team_ids else None+ )++ assert {"github_oauth": len(rows)} == expected+ assert all(row.integration_id == "github_oauth" for row in rows)++ @parameterized.expand(+ [+ ("both_teams", ["own", "other"], "github_oauth"),+ ("other_team_only", ["other"], "other"),+ ("empty", [], None),+ ("unknown_team", [10**9], None),+ ]+ )+ def test_find_and_exists_with_team_ids(self, _name: str, teams: list[Any], expected: str | None) -> None:+ by_name = {"own": self.team.id, "other": self.other_team.id}+ team_ids = [by_name.get(team, team) for team in teams]+ expected_row = {"github_oauth": self.rows["github_oauth"], "other": self.other_row}.get(expected or "")++ row = IntegrationManager.github.oauth.find(team_ids=team_ids, external_id="github_oauth")++ assert row == expected_row+ assert IntegrationManager.github.oauth.exists(team_ids=iter(team_ids)) is (expected_row is not None)+ assert async_to_sync(IntegrationManager.github.oauth.afind)(team_ids=team_ids) == expected_row+ assert async_to_sync(IntegrationManager.github.oauth.aexists)(team_ids=team_ids) is (expected_row is not None)++ @parameterized.expand(+ [+ ("neither", {}),+ ("both", {"team_id": 1, "team_ids": [1]}),+ ]+ )+ def test_find_and_exists_need_exactly_one_team_argument(self, _name: str, kwargs: dict[str, Any]) -> None:+ for read in (IntegrationManager.github.oauth.find, IntegrationManager.github.oauth.exists):+ with self.assertRaises(ValueError):+ read(**kwargs)++ @parameterized.expand(+ [+ ("own_row", "github_oauth", lambda self: self.rows["github_oauth"], True),+ ("row_of_another_team", "github_oauth", lambda self: self.other_row, True),+ ("row_with_another_external_id", "github_oauth", lambda self: self.rows["github_token"], False),+ ("another_workspace_id_for_the_external_id", "github_token", lambda self: self.other_row, False),+ ("missing_id", "github_oauth", lambda self: SimpleNamespace(id=10**9), False),+ ("not_a_number", "github_oauth", lambda self: SimpleNamespace(id="abc"), False),+ ]+ )+ def test_get_by_external_id(self, _name: str, external_id: str, row: Callable[[Any], Any], found: bool) -> None:+ expected = row(self)++ if found:+ assert IntegrationManager.github.oauth.get_by_external_id(external_id, expected.id) == expected+ else:+ with self.assertRaises(IntegrationNotFound):+ IntegrationManager.github.oauth.get_by_external_id(external_id, expected.id)++ @parameterized.expand(+ [+ ("every_team", lambda: IntegrationManager.github.oauth, "github_oauth", 2),+ ("other_auth_type_is_out_of_scope", lambda: IntegrationManager.github.oauth, "github_token", 0),+ ("kind_any", lambda: IntegrationManager.github.any, "github_token", 1),+ ("source_row_is_out_of_scope", lambda: IntegrationManager.stripe.any, "stripe_source", 0),+ ("missing", lambda: IntegrationManager.github.oauth, "missing", 0),+ ]+ )+ def test_count_and_exists_by_external_id(+ self, _name: str, accessor: Callable[[], IntegrationReader], external_id: str, expected: int+ ) -> None:+ assert accessor().count_by_external_id(external_id) == expected+ assert accessor().exists_by_external_id(external_id) is (expected > 0)++ @parameterized.expand(+ [+ ("get", lambda self, a: [a.get(self.team.id, self.rows["github_oauth"].id)]),+ ("get_many", lambda self, a: list(a.get_many(self.team.id, [self.rows["github_oauth"].id]).values())),+ ("find", lambda self, a: [a.find(self.team.id)]),+ ("list", lambda self, a: a.list(self.team.id)),+ ("list_by_external_id", lambda self, a: a.list_by_external_id("github_oauth")),+ ("get_by_external_id", lambda self, a: [a.get_by_external_id("github_oauth", self.other_row.id)]),+ ("update", lambda self, a: [a.update(self.team.id, self.rows["github_oauth"].id, errors="")]),+ ("upsert_of_an_existing_row", lambda self, a: [a.upsert(self.team.id, "github_oauth", config={})[0]]),+ ]+ )+ def test_reads_defer_the_repository_cache(+ self, _name: str, read: Callable[[Any, AuthAccessor], list[Integration]]+ ) -> None:+ Integration._base_manager.filter(integration_id="github_oauth").update(repository_cache=[{"name": "repo"}])++ rows = read(self, IntegrationManager.github.oauth)++ assert rows+ for row in rows:+ assert {"repository_cache", "repository_cache_updated_at"} <= row.get_deferred_fields()+ assert row.repository_cache == [{"name": "repo"}]++ def test_with_source_rows_refuses_a_source_only_accessor(self) -> None:+ with self.assertRaises(ValueError):+ IntegrationManager.source("stripe").any.with_source_rows()++ @parameterized.expand(+ [+ ("list_by_external_id", lambda: IntegrationManager.any.list_by_external_id("github_oauth")),+ ("delete_by_external_id", lambda: IntegrationManager.any.delete_by_external_id("github_oauth")),+ ("get_by_external_id", lambda: IntegrationManager.any.get_by_external_id("github_oauth", 1)),+ ("count_by_external_id", lambda: IntegrationManager.any.count_by_external_id("github_oauth")),+ ("exists_by_external_id", lambda: IntegrationManager.any.exists_by_external_id("github_oauth")),+ ]+ )+ def test_cross_team_operations_need_a_kind(self, _name: str, call: Callable[[], Any]) -> None:+ with self.assertRaises(ValueError):+ call()++ assert Integration._base_manager.filter(integration_id="github_oauth").count() == 2+ assert len(IntegrationManager.kinds("github").list_by_external_id("github_oauth")) == 2++ @parameterized.expand(+ [+ ("default_auth_type_stores_nothing", lambda: IntegrationManager.github.oauth, {}),+ ("non_default_auth_type_is_stored", lambda: IntegrationManager.github.token, {"auth_type": "token"}),+ (+ "token_on_a_credentials_kind",+ lambda: IntegrationManager.databricks.token,+ {"auth_type": "token"},+ ),+ ("credentials_default", lambda: IntegrationManager.databricks.credentials, {}),+ ("source_kind_default", lambda: IntegrationManager.source("ably").token, {MARKER: "Ably"}),+ (+ "source_row_of_a_catalogue_kind",+ lambda: IntegrationManager.source("stripe").token,+ {MARKER: "Stripe", "auth_type": "token"},+ ),+ ]+ )+ def test_create_stores_the_auth_type_only_when_it_is_not_the_default(+ self, _name: str, accessor: Callable[[], Any], expected_config: dict[str, Any]+ ) -> None:+ config = {MARKER: expected_config[MARKER]} if MARKER in expected_config else {}++ row = accessor().create(+ self.team.id, external_id="new", config=config, sensitive_config={"secret": "s"}, created_by_id=self.user.id+ )++ stored = Integration._base_manager.get(id=row.id)+ assert (stored.config, stored.sensitive_config, stored.created_by_id) == (+ expected_config,+ {"secret": "s"},+ self.user.id,+ )+ assert accessor().get(self.team.id, row.id) == row++ @parameterized.expand(+ [+ ("auth_type_in_config", lambda: IntegrationManager.github.token, {"auth_type": "token"}, ValueError),+ ("marker_through_named_accessor", lambda: IntegrationManager.stripe.token, {MARKER: "Stripe"}, ValueError),+ ("source_row_without_marker", lambda: IntegrationManager.source("ably").token, {}, ValueError),+ (+ "auth_type_the_kind_does_not_support",+ lambda: IntegrationManager.source("postgresql").token,+ {MARKER: "Postgres"},+ UnsupportedAuthType,+ ),+ ]+ )+ def test_create_rejects(+ self, _name: str, accessor: Callable[[], Any], config: dict[str, Any], error: type[Exception]+ ) -> None:+ with self.assertRaises(error):+ accessor().create(self.team.id, external_id="new", config=config)++ @parameterized.expand(+ [+ ("same_scope", lambda: IntegrationManager.github.oauth, "github_oauth", {}),+ ("other_auth_type", lambda: IntegrationManager.github.token, "github_oauth", {}),+ ("source_row_through_named_accessor", lambda: IntegrationManager.stripe.oauth, "stripe_source", {}),+ (+ "named_row_through_source_accessor",+ lambda: IntegrationManager.source("stripe").token,+ "stripe_token",+ {MARKER: "Stripe"},+ ),+ ]+ )+ def test_create_raises_conflict_on_the_unique_key(+ self, _name: str, accessor: Callable[[], Any], external_id: str, config: dict[str, Any]+ ) -> None:+ with self.assertRaises(IntegrationConflict):+ accessor().create(self.team.id, external_id=external_id, config=config)++ # The failed insert rolls back to a savepoint, so the transaction stays usable.+ assert IntegrationManager.any.with_source_rows().list(self.team.id, external_id=external_id) == [+ self.rows[external_id]+ ]++ def test_upsert_creates_then_replaces_the_config(self) -> None:+ row, created = IntegrationManager.github.token.upsert(self.team.id, "new", config={"a": 1})+ again, created_again = IntegrationManager.github.token.upsert(+ self.team.id, "new", config={"b": 2}, sensitive_config={"token": "t"}, created_by_id=self.user.id+ )++ again = Integration._base_manager.get(id=again.id)+ assert (created, created_again, again.id) == (True, False, row.id)+ assert (again.config, again.sensitive_config, again.created_by_id) == (+ {"b": 2, "auth_type": "token"},+ {"token": "t"},+ self.user.id,+ )++ @parameterized.expand(+ [+ ("other_auth_type", lambda: IntegrationManager.github.token, "github_oauth", {}),+ ("source_row", lambda: IntegrationManager.stripe.token, "stripe_source", {}),+ (+ "named_row_through_source",+ lambda: IntegrationManager.source("stripe").token,+ "stripe_token",+ {MARKER: "Stripe"},+ ),+ ]+ )+ def test_upsert_raises_on_a_row_out_of_scope(+ self, _name: str, accessor: Callable[[], Any], external_id: str, config: dict[str, Any]+ ) -> None:+ before = Integration._base_manager.get(id=self.rows[external_id].id).config++ with self.assertRaises(IntegrationConflict):+ accessor().upsert(self.team.id, external_id, config=config)++ assert Integration._base_manager.get(id=self.rows[external_id].id).config == before++ def test_update_merges_and_removes_keys(self) -> None:+ row = self.rows["github_token"]++ IntegrationManager.github.token.update(+ self.team.id,+ row.id,+ config_patch={"login": "octocat"},+ config_remove=["account", "missing"],+ sensitive_config_patch={"token": "t"},+ errors="TOKEN_REFRESH_FAILED",+ )++ stored = Integration._base_manager.get(id=row.id)+ assert (stored.config, stored.sensitive_config, stored.errors) == (+ {"auth_type": "token", "login": "octocat"},+ {"token": "t"},+ "TOKEN_REFRESH_FAILED",+ )++ @parameterized.expand(+ [+ ("patch_auth_type", {"config_patch": {"auth_type": "oauth"}}, ValueError),+ ("remove_auth_type", {"config_remove": ["auth_type"]}, ValueError),+ ("patch_marker", {"config_patch": {MARKER: "Stripe"}}, ValueError),+ ]+ )+ def test_update_rejects_reserved_keys(self, _name: str, kwargs: dict[str, Any], error: type[Exception]) -> None:+ with self.assertRaises(error):+ IntegrationManager.github.token.update(self.team.id, self.rows["github_token"].id, **kwargs)++ def test_upsert_without_created_by_keeps_the_creator(self) -> None:+ row = IntegrationManager.github.token.create(+ self.team.id, external_id="new", config={}, created_by_id=self.user.id+ )++ again, created = IntegrationManager.github.token.upsert(self.team.id, "new", config={"b": 2})++ assert (created, again.id) == (False, row.id)+ assert Integration._base_manager.get(id=row.id).created_by_id == self.user.id++ def test_upsert_updates_the_row_that_a_concurrent_insert_won(self) -> None:+ original_first = IntegrationQuerySet.first+ reads: list[int] = []++ def first_after_a_competing_insert(queryset: Any) -> Integration | None:+ result = original_first(queryset)+ reads.append(1)+ if len(reads) == 1:+ assert result is None+ Integration.objects._with_auth(ANY).create(+ team=self.team, kind="github", integration_id="raced", config={"auth_type": "token", "old": True}+ )+ return result++ with patch.object(IntegrationQuerySet, "first", autospec=True, side_effect=first_after_a_competing_insert):+ row, created = IntegrationManager.github.token.upsert(+ self.team.id, "raced", config={"new": True}, created_by_id=self.user.id+ )++ stored = Integration._base_manager.filter(team=self.team, kind="github", integration_id="raced")+ saved = stored.get()+ assert (created, stored.count(), saved.id) == (False, 1, row.id)+ assert (saved.config, saved.created_by_id) == ({"new": True, "auth_type": "token"}, self.user.id)++ @parameterized.expand(+ [+ ("nothing", {}),+ ("empty_collections", {"config_patch": {}, "config_remove": (), "sensitive_config_patch": {}}),+ ]+ )+ def test_an_empty_patch_writes_nothing(self, _name: str, kwargs: dict[str, Any]) -> None:+ row = self.rows["github_oauth"]+ before = Integration._base_manager.get(id=row.id)++ with patch.object(Integration, "save") as save:+ updated = IntegrationManager.github.oauth.update(self.team.id, row.id, **kwargs)++ save.assert_not_called()+ assert (updated.config, updated.sensitive_config, updated.errors) == (+ before.config,+ before.sensitive_config,+ before.errors,+ )++ @parameterized.expand(+ [+ ("int_to_bool", 1, True),+ ("bool_to_int", True, 1),+ ("int_to_float", 1, 1.0),+ ("zero_to_false", 0, False),+ ("nested_int_to_bool", {"a": 1}, {"a": True}),+ ]+ )+ def test_update_writes_a_value_of_another_json_type(self, _name: str, stored: object, new: object) -> None:+ row = self.rows["github_oauth"]+ IntegrationManager.github.oauth.update(self.team.id, row.id, config_patch={"flag": stored})++ IntegrationManager.github.oauth.update(self.team.id, row.id, config_patch={"flag": new})++ saved = Integration._base_manager.get(id=row.id).config["flag"]+ assert (saved, type(saved)) == (new, type(new))++ @parameterized.expand(+ [+ ("auth_type_without_kinds", {"kinds": None, "auth_type": AuthType.TOKEN}),+ ("auth_type_with_empty_kinds", {"kinds": frozenset(), "auth_type": AuthType.TOKEN}),+ ]+ )+ def test_a_reader_with_an_auth_type_needs_kinds(self, _name: str, kwargs: dict[str, Any]) -> None:+ with self.assertRaises(ValueError):+ IntegrationReader(**kwargs)++ def test_a_write_accessor_refuses_to_include_source_rows(self) -> None:+ with self.assertRaises(ValueError):+ AuthAccessor(kind="github", auth_type=AuthType.OAUTH, source_rows=SourceRows.INCLUDE)++ def test_update_raises_for_a_row_out_of_scope(self) -> None:+ with self.assertRaises(IntegrationNotFound):+ IntegrationManager.github.oauth.update(self.team.id, self.rows["github_token"].id, errors="")++ def _resend_row(self, *, refreshed_ago: int) -> Integration:+ return Integration.objects._with_auth(ANY).create(+ team=self.team,+ kind="resend",+ integration_id="resend-account",+ config={"expires_in": 900, "refreshed_at": int(time.time()) - refreshed_ago},+ sensitive_config={"access_token": "old", "refresh_token": "r1"},+ )++ @parameterized.expand(+ [+ ("expired", 800, None, True),+ ("fresh", 10, None, False),+ ("fresh_but_the_caller_holds_the_stored_token", 10, "old", True),+ ("fresh_and_another_worker_rotated_the_token", 10, "older", False),+ ("expired_and_another_worker_rotated_the_token", 800, "older", True),+ ]+ )+ def test_refresh_access_token_refreshes_only_when_needed(+ self, _name: str, refreshed_ago: int, stale_access_token: str | None, refreshes: bool+ ) -> None:+ row = self._resend_row(refreshed_ago=refreshed_ago)++ def rotate(oauth: OauthIntegration) -> None:+ oauth.integration.sensitive_config = {"access_token": "new", "refresh_token": "r2"}+ oauth.integration.save()++ with (+ patch.object(OauthIntegration, "refresh_access_token", autospec=True, side_effect=rotate) as refresh,+ CaptureQueriesContext(connection) as queries,+ ):+ result = IntegrationManager.resend.oauth.refresh_access_token(+ self.team.id, row.id, stale_access_token=stale_access_token+ )++ expected_token = "new" if refreshes else "old"+ assert refresh.call_count == (1 if refreshes else 0)+ assert (result.id, result.sensitive_config["access_token"]) == (row.id, expected_token)+ assert Integration._base_manager.get(id=row.id).sensitive_config["access_token"] == expected_token+ assert any("FOR UPDATE" in query["sql"] for query in queries.captured_queries)++ @parameterized.expand(+ [+ ("kind_without_a_refresh", lambda: IntegrationManager.github.oauth, "github_oauth", ValueError),+ ("row_of_another_kind", lambda: IntegrationManager.stripe.oauth, "github_oauth", IntegrationNotFound),+ (+ "row_with_another_auth_type",+ lambda: IntegrationManager.stripe.oauth,+ "stripe_token",+ IntegrationNotFound,+ ),+ ]+ )+ def test_refresh_access_token_raises(+ self, _name: str, accessor: Callable[[], Any], row_name: str, error: type[Exception]+ ) -> None:+ with patch.object(OauthIntegration, "refresh_access_token") as refresh:+ with self.assertRaises(error):+ accessor().refresh_access_token(self.team.id, self.rows[row_name].id)++ refresh.assert_not_called()++ @parameterized.expand(+ [+ # Queries: the savepoint, the locked read, the one bulk update, the savepoint release.+ ("changes_every_team", [], IntegrationPatch(config_patch={"suspended": True}), 2, 4),+ ("skips_a_row_that_already_matches", ["other"], IntegrationPatch(config_patch={"suspended": True}), 1, 4),+ (+ "writes_nothing_when_nothing_changes",+ ["own", "other"],+ IntegrationPatch(config_patch={"suspended": True}),+ 0,+ 3,+ ),+ ("removes_a_key", ["own"], IntegrationPatch(config_remove=("suspended",)), 1, 4),+ ("errors_only", [], IntegrationPatch(errors="SUSPENDED"), 2, 4),+ ("empty_patch", [], IntegrationPatch(), 0, 3),+ ]+ )+ def test_update_by_external_id_reaches_every_team(+ self, _name: str, already_suspended: list[str], patch: IntegrationPatch, changed: int, queries: int+ ) -> None:+ by_name = {"own": self.rows["github_oauth"], "other": self.other_row}+ for name in already_suspended:+ Integration._base_manager.filter(id=by_name[name].id).update(config={"suspended": True})++ with self.assertNumQueries(queries):+ count = IntegrationManager.github.oauth.update_by_external_id("github_oauth", patch)++ stored = Integration._base_manager.filter(integration_id="github_oauth")+ assert count == changed+ if patch.config_patch:+ assert {row.config.get("suspended") for row in stored} == {True}+ if patch.config_remove:+ assert {row.config.get("suspended") for row in stored} == {None}+ if patch.errors:+ assert {row.errors for row in stored} == {"SUSPENDED"}+ assert "suspended" not in Integration._base_manager.get(id=self.rows["github_token"].id).config++ def test_update_by_external_id_rejects_reserved_keys(self) -> None:+ with self.assertRaises(ValueError):+ IntegrationManager.github.oauth.update_by_external_id(+ "github_oauth", IntegrationPatch(config_patch={MARKER: "Github"})+ )++ def test_delete_checks_usages(self) -> None:+ row = self.rows["github_oauth"]++ def provider(team_id: int, integration_id: int) -> list[IntegrationUsage]:+ return [IntegrationUsage(type="workflow", id="1", name="Flow")] if integration_id == row.id else []++ with _usage_providers(provider):+ with self.assertRaises(IntegrationInUse) as raised:+ IntegrationManager.github.oauth.delete(self.team.id, row.id)+ assert raised.exception.usages == [IntegrationUsage(type="workflow", id="1", name="Flow")]++ IntegrationManager.github.oauth.delete(self.team.id, row.id, ignore_usages=True)+ IntegrationManager.github.token.delete(self.team.id, self.rows["github_token"].id)++ assert not Integration._base_manager.filter(id__in=[row.id, self.rows["github_token"].id]).exists()++ def test_delete_raises_for_a_row_out_of_scope(self) -> None:+ with _usage_providers(), self.assertRaises(IntegrationNotFound):+ IntegrationManager.github.oauth.delete(self.team.id, self.rows["github_token"].id)++ def test_delete_by_external_id_reaches_every_team(self) -> None:+ assert IntegrationManager.github.oauth.delete_by_external_id("github_oauth") == 2+ assert not Integration._base_manager.filter(integration_id="github_oauth").exists()+ assert Integration._base_manager.filter(id=self.rows["github_token"].id).exists()++ def test_delete_by_external_id_rolls_back_every_row_when_one_delete_fails(self) -> None:+ original_delete = Integration.delete+ calls: list[int] = []++ def delete_then_fail_on_the_second_row(row: Integration, *args: Any, **kwargs: Any) -> Any:+ calls.append(row.id)+ if len(calls) == 2:+ raise RuntimeError("delete failed")+ return original_delete(row, *args, **kwargs)++ with patch.object(Integration, "delete", autospec=True, side_effect=delete_then_fail_on_the_second_row):+ with self.assertRaises(RuntimeError):+ IntegrationManager.github.oauth.delete_by_external_id("github_oauth")++ assert len(calls) == 2+ assert Integration._base_manager.filter(integration_id="github_oauth").count() == 2++ def test_usages_reads_registered_and_product_providers_in_order(self) -> None:+ row = self.rows["postgres"]+ product_module = SimpleNamespace(+ integration_usages=lambda team_id, integration_id: [+ IntegrationUsage(type="warehouse_source", id="2", name="b"),+ IntegrationUsage(type="warehouse_source", id="1", name="a"),+ ]+ )++ with (+ patch.object(usages, "_registered", []),+ patch.object(usages, "_product_providers", None),+ patch.object(usages, "load_product_modules", return_value=iter([product_module])) as load,+ ):+ register_usage_provider(+ lambda team_id, integration_id: [IntegrationUsage(type="batch_export", id="9", name="z")]+ )+ first = IntegrationManager.postgresql.credentials.usages(self.team.id, row.id)+ IntegrationManager.postgresql.credentials.usages(self.team.id, row.id)++ assert [(usage.type, usage.name) for usage in first] == [+ ("batch_export", "z"),+ ("warehouse_source", "a"),+ ("warehouse_source", "b"),+ ]+ load.assert_called_once_with("integration_usages")++ def test_a_provider_module_that_registers_at_import_does_not_deadlock(self) -> None:+ def import_a_module_that_registers(_name: str) -> Iterable[SimpleNamespace]:+ register_usage_provider(lambda team_id, integration_id: [])+ return [SimpleNamespace(integration_usages=lambda team_id, integration_id: [])]++ with (+ patch.object(usages, "_registered", []),+ patch.object(usages, "_product_providers", None),+ patch.object(usages, "load_product_modules", side_effect=import_a_module_that_registers),+ ):+ # A daemon thread, so a deadlock fails the test instead of hanging the run.+ loader = threading.Thread(target=usages.integration_usages, args=(self.team.id, 1), daemon=True)+ loader.start()+ loader.join(timeout=10)++ assert not loader.is_alive()+ assert len(usages._registered) == 1++ @parameterized.expand(+ [+ ("aget", lambda t, r: ("get", (t, r), {})),+ ("aget_many", lambda t, r: ("get_many", (t, [r]), {})),+ ("afind", lambda t, r: ("find", (t,), {"external_id": "github_oauth"})),+ ("alist", lambda t, r: ("list", (t,), {})),+ ("aexists", lambda t, r: ("exists", (t,), {"external_id": "github_oauth"})),+ ("ausages", lambda t, r: ("usages", (t, r), {})),+ ]+ )+ def test_async_reads_match_the_sync_reads(+ self, name: str, args: Callable[[int, int], tuple[str, tuple[Any, ...], dict[str, Any]]]+ ) -> None:+ accessor = IntegrationManager.github.oauth+ sync_name, call_args, call_kwargs = args(self.team.id, self.rows["github_oauth"].id)++ with _usage_providers(+ lambda team_id, integration_id: [IntegrationUsage(type="warehouse_source", id="1", name="a")]+ ):+ expected = getattr(accessor, sync_name)(*call_args, **call_kwargs)+ actual = async_to_sync(getattr(accessor, name))(*call_args, **call_kwargs)++ assert actual == expected+ assert actual not in (None, [], {}, False)++ def test_async_delete_deletes_and_honors_usages(self) -> None:+ accessor = IntegrationManager.github.oauth+ row = self.rows["github_oauth"]++ with _usage_providers(+ lambda team_id, integration_id: [IntegrationUsage(type="warehouse_source", id="1", name="a")]+ ):+ with self.assertRaises(IntegrationInUse):+ async_to_sync(accessor.adelete)(self.team.id, row.id)+ assert Integration._base_manager.filter(id=row.id).exists()+ async_to_sync(accessor.adelete)(self.team.id, row.id, ignore_usages=True)++ assert not Integration._base_manager.filter(id=row.id).exists()++ def test_async_writes_match_the_sync_writes(self) -> None:+ accessor = IntegrationManager.github.token+ row = self.rows["github_token"]++ created = async_to_sync(accessor.acreate)(+ self.team.id, external_id="made", config={"a": 1}, sensitive_config={"s": 1}, created_by_id=self.user.id+ )+ upserted, was_created = async_to_sync(accessor.aupsert)(self.team.id, "made", config={"b": 2})+ updated = async_to_sync(accessor.aupdate)(self.team.id, row.id, config_patch={"x": 1}, errors="boom")++ assert (created.config, created.sensitive_config, created.created_by_id) == (+ {"a": 1, "auth_type": "token"},+ {"s": 1},+ self.user.id,+ )+ assert (upserted.id, was_created, upserted.config) == (created.id, False, {"b": 2, "auth_type": "token"})+ assert (updated.config["x"], updated.errors) == (1, "boom")+ assert Integration._base_manager.get(id=row.id).config["x"] == 1++ def test_async_refresh_delegates_to_the_sync_refresh(self) -> None:+ # The async form runs in its own thread with its own connection, which cannot see the rows+ # of this test's transaction, so the sync refresh is replaced here.+ accessor = IntegrationManager.resend.oauth+ row = self._resend_row(refreshed_ago=800)++ with patch.object(OAuthAccessor, "refresh_access_token", autospec=True, return_value=row) as refresh:+ refreshed = async_to_sync(accessor.arefresh_access_token)(self.team.id, row.id, stale_access_token="old")++ assert refreshed is row+ refresh.assert_called_once_with(accessor, self.team.id, row.id, stale_access_token="old")+++class TestRegistration(BaseTest):+ @parameterized.expand(+ [+ ("unknown_source_kind", lambda: IntegrationManager.source("acme-crm-unregistered")),+ ("unknown_kind", lambda: IntegrationManager.kinds("github", "acme-crm")),+ ("no_kinds", lambda: IntegrationManager.kinds()),+ ]+ )+ def test_unknown_kinds_raise(self, _name: str, build: Callable[[], Any]) -> None:+ with self.assertRaises(UnknownIntegrationKind):+ build()++ def test_register_source_kinds_sets_the_default_of_a_new_kind_only(self) -> None:+ with patch.dict(auth_types._KIND_AUTH_TYPES):+ register_source_kinds({"acme-sdk-test": AuthType.TOKEN, "stripe": AuthType.CREDENTIALS})++ assert default_auth_type("acme-sdk-test") == AuthType.TOKEN+ assert default_auth_type("stripe") == AuthType.OAUTH+ assert AuthType.CREDENTIALS in kind_auth_types("stripe").supported+ assert IntegrationManager.source("acme-sdk-test").token.list(self.team.id) == []++ def test_source_rows_of_a_catalogue_kind_use_the_warehouse_auth_type(self) -> None:+ row = IntegrationManager.source("clickup").credentials.create(+ self.team.id, external_id="new", config={MARKER: "ClickUp"}+ )++ assert default_auth_type("clickup") == AuthType.OAUTH+ assert row.config == {MARKER: "ClickUp", "auth_type": "credentials"}++ def test_register_source_kinds_rejects_a_changed_default(self) -> None:+ register_source_kinds({"acme-sdk-test-2": AuthType.TOKEN})++ with self.assertRaises(ValueError):+ register_source_kinds({"acme-sdk-test-2": AuthType.CREDENTIALS})++ def test_warehouse_sources_register_their_kinds_at_startup(self) -> None:+ assert IntegrationManager.source("ably").token is not None+ assert default_auth_type("ably") == AuthType.TOKENdiff --git a/posthog/integrations/test/test_catalogue.py b/posthog/integrations/test/test_catalogue.pynew file mode 100644index 000000000000..50211284d466--- /dev/null+++ b/posthog/integrations/test/test_catalogue.py@@ -0,0 +1,222 @@+import shutil+import textwrap+from pathlib import Path+from typing import Any++import pytest++from django.test import SimpleTestCase++import mypy.api+from hogli_commands.projections import PROJECTIONS, OutputFormatter, ProjectionRunner+from parameterized import parameterized++from posthog.integrations import IntegrationManager+from posthog.integrations.accessors import AuthAccessor, IntegrationReader, OAuthAccessor+from posthog.integrations.catalogue import KINDS+from posthog.integrations.contracts import UnsupportedIntegrationFilter+from posthog.integrations.enums import AuthType+from posthog.integrations.kinds_generated import IntegrationKind+from posthog.models.integration import Integration++REPO_ROOT = Path(__file__).resolve().parents[3]+SDK_DIR = REPO_ROOT / "posthog/integrations"+++# The default auth type of a kind is stored data: a row without `config["auth_type"]` has it.+# A change here needs a data migration for the rows of the kind, and a deliberate edit of this mapping.+FROZEN_KIND_DEFAULTS: dict[str, str] = {+ "anthropic": "token",+ "apns": "credentials",+ "apple-ads": "oauth",+ "aws-redshift": "credentials",+ "aws-s3": "credentials",+ "azure-blob": "credentials",+ "bing-ads": "oauth",+ "clickup": "oauth",+ "customerio-app": "token",+ "customerio-track": "token",+ "customerio-webhook": "token",+ "databricks": "credentials",+ "email": "credentials",+ "firebase": "credentials",+ "github": "oauth",+ "gitlab": "token",+ "google-ads": "oauth",+ "google-analytics": "oauth",+ "google-calendar": "oauth",+ "google-cloud-service-account": "credentials",+ "google-cloud-storage": "credentials",+ "google-pubsub": "credentials",+ "google-search-console": "oauth",+ "google-sheets": "oauth",+ "helpscout": "oauth",+ "hubspot": "oauth",+ "instagram": "oauth",+ "intercom": "oauth",+ "jira": "oauth",+ "linear": "oauth",+ "linkedin-ads": "oauth",+ "meta-ads": "oauth",+ "pardot": "oauth",+ "pinterest-ads": "oauth",+ "postgresql": "credentials",+ "posthog": "oauth",+ "reddit-ads": "oauth",+ "resend": "oauth",+ "s3-compatible": "credentials",+ "salesforce": "oauth",+ "slack": "oauth",+ "slack-posthog-code": "oauth",+ "snapchat": "oauth",+ "snowflake": "credentials",+ "stripe": "oauth",+ "tiktok-ads": "oauth",+ "twilio": "token",+ "twitter-ads": "oauth",+ "vercel": "oauth",+ "youtube-analytics": "oauth",+}+++class TestCatalogue(SimpleTestCase):+ def test_kinds_match_the_model_choices(self) -> None:+ assert set(KINDS) == set(Integration.IntegrationKind.values) == {kind.value for kind in IntegrationKind}++ def test_kind_defaults_do_not_change(self) -> None:+ assert {kind: spec.default.value for kind, spec in KINDS.items()} == FROZEN_KIND_DEFAULTS++ def test_generated_file_is_in_sync_with_the_catalogue(self) -> None:+ [projection] = [projection for projection in PROJECTIONS if projection.name == "integration-kinds"]+ runner = ProjectionRunner(REPO_ROOT, [projection], OutputFormatter(REPO_ROOT))++ assert runner.stale(projection) == [], "Run `hogli build:projections --only integration-kinds`."++ @parameterized.expand([(kind,) for kind in sorted(KINDS)])+ def test_each_kind_has_exactly_its_supported_auth_types(self, kind: str) -> None:+ accessor = getattr(IntegrationManager, kind.replace("-", "_"))+ present = {auth_type for auth_type in AuthType if hasattr(accessor, auth_type.value)}++ assert present == KINDS[kind].supported+ expected = {AuthType.OAUTH: OAuthAccessor, AuthType.TOKEN: AuthAccessor, AuthType.CREDENTIALS: AuthAccessor}+ assert all(type(getattr(accessor, auth_type.value)) is expected[auth_type] for auth_type in present)+ assert type(accessor.any) is IntegrationReader++ @parameterized.expand(+ [+ ("create",),+ ("upsert",),+ ("update",),+ ("refresh_access_token",),+ ("update_by_external_id",),+ ("acreate",),+ ("aupdate",),+ ]+ )+ def test_any_readers_have_no_writes(self, method: str) -> None:+ assert not hasattr(IntegrationManager.any, method)+ assert not hasattr(IntegrationManager.stripe.any, method)+ assert hasattr(IntegrationManager.stripe.oauth, method)++ @parameterized.expand(+ [+ ("lookup_in_key", {"account__name": "a"}),+ ("auth_type_key", {"auth_type": "token"}),+ ("marker_key", {"warehouse_source_type": "Stripe"}),+ ("empty_key", {"": "a"}),+ ("key_with_a_dash", {"a-b": "a"}),+ ("key_starts_with_a_digit", {"1a": "a"}),+ ("dict_value", {"account": {"name": "a"}}),+ ("list_value", {"ids": [1, 2]}),+ ("tuple_value", {"ids": (1,)}),+ ]+ )+ def test_find_rejects_unsupported_filters(self, _name: str, where: dict[str, Any]) -> None:+ with self.assertRaises(UnsupportedIntegrationFilter):+ IntegrationManager.github.oauth.find(1, where=where)+++# The SDK modules are copied into an empty directory, so mypy reads them without the Django plugin+# and without following imports into the rest of the repo. Imports it cannot find become `Any`.+SDK_FILES = (+ "__init__.py",+ "accessors.py",+ "catalogue.py",+ "contracts.py",+ "enums.py",+ "kinds_generated.py",+ "manager.py",+ "source_kinds.py",+ "usages.py",+)++SNIPPET = """+from posthog.integrations import IntegrationManager++IntegrationManager.github.oauth.create(1, external_id="x", config={})+IntegrationManager.github.token.upsert(1, "x", config={})+IntegrationManager.source("ably").token.create(1, external_id="x", config={})+IntegrationManager.stripe.any.get(1, 2)+IntegrationManager.any.list(1)+IntegrationManager.stripe.any.create(1, external_id="x", config={})+IntegrationManager.github.credentials+IntegrationManager.any.update(1, 2, errors="")+IntegrationManager.postgresql.oauth+IntegrationManager.stripe.any.upsert(1, "x", config={})+row = IntegrationManager.github.oauth.get(1, 2)+row.save()+row.delete()+row.config["x"] = 1+row.objects+IntegrationManager.github.token.refresh_access_token(1, 2)+row.config.get("x")+IntegrationManager.resend.oauth.refresh_access_token(1, 2, stale_access_token="t").sensitive_config+"""++[email protected](300)+def test_mypy_rejects_unsupported_auth_types_writes_through_any_and_row_writes(+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch+) -> None:+ # mypy also searches the working directory, which would pull in the whole repo.+ monkeypatch.chdir(tmp_path)+ root = tmp_path / "src"+ for relative in SDK_FILES:+ target = root / "posthog/integrations" / relative+ target.parent.mkdir(parents=True, exist_ok=True)+ shutil.copy(SDK_DIR / relative, target)+ (root / "posthog/__init__.py").touch()+ (root / "snippet.py").write_text(textwrap.dedent(SNIPPET))+ (tmp_path / "mypy.ini").write_text("[mypy]\n")++ stdout, stderr, _ = mypy.api.run(+ [+ "--config-file",+ str(tmp_path / "mypy.ini"),+ "--cache-dir",+ str(tmp_path / "cache"),+ "--no-site-packages",+ "--ignore-missing-imports",+ "--no-error-summary",+ "--hide-error-context",+ str(root / "snippet.py"),+ ]+ )++ errors = sorted(+ (int(line.split(":")[1]), line.split(": error: ")[1].split(" [")[0])+ for line in stdout.splitlines()+ if "snippet.py" in line and ": error: " in line+ )+ assert errors == [+ (9, '"IntegrationReader" has no attribute "create"'),+ (10, '"GithubKind" has no attribute "credentials"'),+ (11, '"IntegrationReader" has no attribute "update"'),+ (12, '"PostgresqlKind" has no attribute "oauth"'),+ (13, '"IntegrationReader" has no attribute "upsert"'),+ (15, '"IntegrationRecord" has no attribute "save"'),+ (16, '"IntegrationRecord" has no attribute "delete"'),+ (17, 'Unsupported target for indexed assignment ("Mapping[str, Any]")'),+ (18, '"IntegrationRecord" has no attribute "objects"'),+ (19, '"AuthAccessor" has no attribute "refresh_access_token"'),+ ], stdout + stderrdiff --git a/posthog/integrations/test/test_fields.py b/posthog/integrations/test/test_fields.pynew file mode 100644index 000000000000..8abaaa7fbe6c--- /dev/null+++ b/posthog/integrations/test/test_fields.py@@ -0,0 +1,183 @@+from typing import Any++from posthog.test.base import BaseTest++from parameterized import parameterized+from rest_framework import serializers++from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY+from posthog.integrations.enums import AuthType+from posthog.integrations.fields import IntegrationReferenceField+from posthog.models import Team+from posthog.models.comment.slack_thread import CommentSlackThread+from posthog.models.integration import ANY, Integration+++class _Serializer(serializers.Serializer):+ integration = IntegrationReferenceField()+ github = IntegrationReferenceField(kinds=["github"], required=False, allow_null=True)+ github_any = IntegrationReferenceField(kinds=["github"], auth_types="any", required=False)+ github_token = IntegrationReferenceField(kinds=["github"], auth_types=[AuthType.TOKEN], required=False)+++class TestIntegrationReferenceField(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.slack = Integration.objects._with_auth(ANY).create(team=self.team, kind="slack", integration_id="s")+ self.github_oauth = Integration.objects._with_auth(ANY).create(+ team=self.team, kind="github", integration_id="o"+ )+ self.github = Integration.objects._with_auth(ANY).create(+ team=self.team, kind="github", integration_id="g", config={"auth_type": "token"}+ )+ self.github_source_row = Integration.objects._with_auth(ANY).create(+ team=self.team,+ kind="github",+ integration_id="gs",+ config={"auth_type": "token", SOURCE_MARKER_CONFIG_KEY: "Github"},+ )+ other_team = Team.objects.create(organization=self.organization, name="Other")+ self.other = Integration.objects._with_auth(ANY).create(team=other_team, kind="slack", integration_id="s")++ def _validate(self, data: dict[str, Any], context: dict[str, Any] | None = None) -> _Serializer:+ serializer = _Serializer(data=data, context={"team_id": self.team.id} if context is None else context)+ serializer.is_valid()+ return serializer++ @parameterized.expand(+ [+ ("int", lambda self: self.slack.id),+ ("string", lambda self: str(self.slack.id)),+ ]+ )+ def test_accepts_an_id_of_the_team(self, _name: str, value: Any) -> None:+ serializer = self._validate({"integration": value(self), "github": self.github_oauth.id})++ assert serializer.errors == {}+ assert serializer.validated_data == {"integration": self.slack, "github": self.github_oauth}++ @parameterized.expand(+ [+ ("no_kinds_accepts_every_auth_type", "integration", "github", None),+ ("default_accepts_the_kind_default", "github", "github_oauth", None),+ ("default_refuses_another_auth_type", "github", "github", "does_not_exist"),+ ("any_accepts_the_kind_default", "github_any", "github_oauth", None),+ ("any_accepts_another_auth_type", "github_any", "github", None),+ ("any_refuses_a_source_row", "github_any", "github_source_row", "does_not_exist"),+ ("listed_accepts_its_auth_type", "github_token", "github", None),+ ("listed_refuses_the_kind_default", "github_token", "github_oauth", "does_not_exist"),+ ]+ )+ def test_auth_types(self, _name: str, field: str, row: str, error_code: str | None) -> None:+ serializer = self._validate({"integration": self.slack.id, field: getattr(self, row).id})++ if error_code is None:+ assert serializer.validated_data[field] == getattr(self, row)+ else:+ assert [error.code for error in serializer.errors[field]] == [error_code]++ @parameterized.expand(+ [+ ("allow_null", "github", None),+ ("not_allow_null", "github_any", "null"),+ ("not_allow_null_without_kinds", "integration", "null"),+ ]+ )+ def test_an_empty_string_means_null(self, _name: str, field: str, error_code: str | None) -> None:+ serializer = self._validate({"integration": self.slack.id, field: ""})++ if error_code is None:+ assert serializer.errors == {}+ assert serializer.validated_data[field] is None+ else:+ assert [error.code for error in serializer.errors[field]] == [error_code]++ def test_auth_types_without_kinds_raises(self) -> None:+ with self.assertRaises(ValueError):+ IntegrationReferenceField(auth_types="any")++ def test_reads_the_team_from_get_team(self) -> None:+ serializer = self._validate({"integration": self.slack.id}, context={"get_team": lambda: self.team})++ assert serializer.validated_data == {"integration": self.slack}++ @parameterized.expand(+ [+ ("other_team", lambda self: {"integration": self.other.id}, "integration", "does_not_exist"),+ ("missing", lambda self: {"integration": 10**9}, "integration", "does_not_exist"),+ (+ "other_kind",+ lambda self: {"integration": self.slack.id, "github": self.slack.id},+ "github",+ "does_not_exist",+ ),+ # The field reads only named rows, so a warehouse source row does not exist for it.+ (+ "source_row_without_kinds",+ lambda self: {"integration": self.github_source_row.id},+ "integration",+ "does_not_exist",+ ),+ (+ "source_row_of_the_kind",+ lambda self: {"integration": self.slack.id, "github": self.github_source_row.id},+ "github",+ "does_not_exist",+ ),+ ("not_a_number", lambda self: {"integration": "abc"}, "integration", "incorrect_type"),+ ("bool", lambda self: {"integration": True}, "integration", "incorrect_type"),+ ("list", lambda self: {"integration": [1]}, "integration", "incorrect_type"),+ ]+ )+ def test_rejects(self, _name: str, data: Any, field: str, code: str) -> None:+ serializer = self._validate(data(self))++ assert [error.code for error in serializer.errors[field]] == [code]++ def test_needs_a_team_in_the_context(self) -> None:+ serializer = self._validate({"integration": self.slack.id}, context={})++ assert [error.code for error in serializer.errors["integration"]] == ["no_team"]++ def test_represents_the_id(self) -> None:+ field = IntegrationReferenceField()++ assert (field.to_representation(self.slack), field.to_representation(self.slack.id)) == (+ self.slack.id,+ self.slack.id,+ )+++class _ReadSerializer(serializers.Serializer):+ integration = IntegrationReferenceField(allow_null=True)+++class _Holder:+ def __init__(self, integration: Integration | None) -> None:+ self.integration = integration+++class TestIntegrationReferenceFieldRead(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.slack = Integration.objects._with_auth(ANY).create(team=self.team, kind="slack", integration_id="s")++ def test_reads_the_foreign_key_column_without_a_query(self) -> None:+ thread = CommentSlackThread(team=self.team, integration_id=self.slack.id)++ with self.assertNumQueries(0):+ data = _ReadSerializer(thread).data++ assert data == {"integration": self.slack.id}++ @parameterized.expand(+ [+ ("model_field_null", lambda self: CommentSlackThread(team=self.team, integration_id=None), None),+ ("plain_attribute", lambda self: _Holder(self.slack), "slack"),+ ("plain_attribute_null", lambda self: _Holder(None), None),+ ]+ )+ def test_represents(self, _name: str, build: Any, expected: str | None) -> None:+ data = _ReadSerializer(build(self)).data++ assert data == {"integration": None if expected is None else getattr(self, expected).id}diff --git a/posthog/integrations/usages.py b/posthog/integrations/usages.pynew file mode 100644index 000000000000..3b30ea843b9a--- /dev/null+++ b/posthog/integrations/usages.py@@ -0,0 +1,42 @@+"""The registry of the products that report which PostHog objects use an integration row.++A product reports its usages from `products/<product>/backend/integration_usages.py`, with a function+`integration_usages(team_id: int, integration_id: int) -> Iterable[IntegrationUsage]`. The SDK finds+these modules on the first call, so no core module has to import a product. Code outside a product+calls `register_usage_provider` instead.+"""++import threading+from collections.abc import Callable, Iterable++from posthog.integrations.contracts import IntegrationUsage+from posthog.products import load_product_modules++UsageProvider = Callable[[int, int], Iterable[IntegrationUsage]]++PRODUCT_MODULE = "integration_usages"+PRODUCT_FUNCTION = "integration_usages"++# Reentrant: a provider module can call `register_usage_provider` while `_providers` imports it.+_lock = threading.RLock()+_registered: list[UsageProvider] = []+_product_providers: list[UsageProvider] | None = None+++def register_usage_provider(provider: UsageProvider) -> None:+ with _lock:+ if provider not in _registered:+ _registered.append(provider)+++def _providers() -> list[UsageProvider]:+ global _product_providers+ with _lock:+ if _product_providers is None:+ _product_providers = [getattr(module, PRODUCT_FUNCTION) for module in load_product_modules(PRODUCT_MODULE)]+ return [*_product_providers, *_registered]+++def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]:+ usages = [usage for provider in _providers() for usage in provider(team_id, integration_id)]+ return sorted(usages, key=lambda usage: (usage.type, usage.name, usage.id))diff --git a/posthog/models/integration/__init__.py b/posthog/models/integration/__init__.pyindex eb0922152a84..efa63c38c926 100644--- a/posthog/models/integration/__init__.py+++ b/posthog/models/integration/__init__.py@@ -16,6 +16,15 @@ from .anthropic import (AnthropicIntegrationError,)from .assignees import Assignee, AssigneeLookupFailed, ReconnectRequired+from .auth_types import (+ ANY,+ DEFAULT,+ AuthRequirement,+ AuthTypeSelector,+ IntegrationAuthType,+ KindAuthTypes,+ register_kind_auth_types,+)from .aws import (AWSCredentialsIntegration,AWSRedshiftIntegration,@@ -73,7 +82,7 @@ from .linkedin_ads import LinkedInAdsIntegrationfrom .meta import InstagramIntegration, MetaAdsIntegration, MetaGraphIntegrationfrom .model import (Integration,- IntegrationManager,+ IntegrationModelManager,IntegrationQuerySet,UndecryptedIntegrationSecretError,aget_integration_by_id,@@ -150,10 +159,17 @@ from .stripe import STRIPE_POSTHOG_SECRET_NAMES, StripeIntegrationfrom .twilio import TwilioIntegration__all__ = [+ "ANY",+ "DEFAULT",+ "AuthRequirement",+ "AuthTypeSelector",+ "IntegrationAuthType",+ "KindAuthTypes",+ "register_kind_auth_types","UndecryptedIntegrationSecretError","integration_secret_decrypt_counter","IntegrationQuerySet",- "IntegrationManager",+ "IntegrationModelManager","Integration","defer_repository_cache_fields","aget_integration_by_id",diff --git a/posthog/models/integration/anthropic.py b/posthog/models/integration/anthropic.pyindex c9da24e6b05a..9229bef3c920 100644--- a/posthog/models/integration/anthropic.py+++ b/posthog/models/integration/anthropic.py@@ -10,6 +10,7 @@ from django.db import transactionimport structlogfrom rest_framework.exceptions import ValidationError+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.models.utils import IntegrityError@@ -48,13 +49,13 @@ class AnthropicIntegration:integration: model.Integration_client: "Anthropic"- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != model.Integration.IntegrationKind.ANTHROPIC.value:# Internal misuse, not a user-input issue.raise AnthropicIntegrationError(f"AnthropicIntegration init called with Integration kind={integration.kind!r}")- self.integration = integration+ self.integration = model.integration_model(integration)api_key = integration.sensitive_config.get("api_key")if not api_key:raise AnthropicIntegrationError(f"Anthropic integration {integration.id} is missing 'api_key'")diff --git a/posthog/models/integration/auth_types.py b/posthog/models/integration/auth_types.pynew file mode 100644index 000000000000..3ad606cede29--- /dev/null+++ b/posthog/models/integration/auth_types.py@@ -0,0 +1,229 @@+"""Auth types of `Integration` rows, and the catalogue of the auth types that each kind supports.++A kind names the provider (`github`, `stripe`, `postgresql`). An auth type names how the row+authenticates with that provider: `oauth` (a connection the provider manages), `token` (one static+secret) or `credentials` (several typed-in fields). A kind can have several auth types. Each kind+has one default auth type: the meaning the kind had before auth types existed. A row stores its auth+type in `config["auth_type"]` only when it is not the default. A row without that key has the+default auth type of its kind, so rows written before auth types existed need no migration.++Code outside the integrations SDK (`posthog/integrations/`) does not query the model. The SDK+states the auth types of each query with the internal `IntegrationQuerySet._with_auth`.++This module must not import `model.py`: `model.py` imports it to build the queryset filters. It+imports only the light SDK modules (catalogue, contracts, enums), which do not import the model.+"""++from collections.abc import Iterable, Mapping+from enum import Enum+from typing import Any, Self++from django.db.models import Q++from posthog.dataclasses import frozen+from posthog.integrations.catalogue import KINDS+from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY, KindSpec+from posthog.integrations.enums import IntegrationAuthType++AUTH_TYPE_CONFIG_KEY = "auth_type"+++# The auth types and the kind catalogue belong to the integrations SDK. The model reads them here for+# the storage rule and the implicit default filter of the model manager.+KindAuthTypes = KindSpec+++class AuthTypeSelector(Enum):+ # Every auth type. For code that handles each row the same way, such as the integrations list.+ ANY = "any"+ # The default auth type of each row's kind. For code that handles several kinds by their usual meaning.+ DEFAULT = "default"+++ANY = AuthTypeSelector.ANY+DEFAULT = AuthTypeSelector.DEFAULT++# Kinds that are not in the catalogue get this entry until they register their own with+# `register_kind_auth_types`.+UNKNOWN_KIND_AUTH_TYPES = KindSpec(label="Unknown", default=IntegrationAuthType.CREDENTIALS)++_KIND_AUTH_TYPES: dict[str, KindSpec] = dict(KINDS)+++def register_kind_auth_types(kind: str, auth_types: KindSpec) -> None:+ """Add a kind to the catalogue, before the first query of the kind runs. `register_source_kinds` calls it."""+ existing = _KIND_AUTH_TYPES.get(kind)+ # Adding supported auth types is safe. Changing the default is not: rows store only a non-default.+ widens = (+ existing is not None and existing.default == auth_types.default and existing.supported <= auth_types.supported+ )+ if existing is not None and existing != auth_types and not widens:+ raise ValueError(f"Integration kind {kind!r} already has auth types {existing!r}")+ _KIND_AUTH_TYPES[kind] = auth_types+++def kind_auth_types(kind: str) -> KindSpec:+ return _KIND_AUTH_TYPES.get(kind, UNKNOWN_KIND_AUTH_TYPES)+++def catalogued_kinds() -> Mapping[str, KindSpec]:+ return dict(_KIND_AUTH_TYPES)+++def default_auth_type(kind: str) -> IntegrationAuthType:+ return kind_auth_types(kind).default+++def stored_auth_type(config: Any) -> str | None:+ if not isinstance(config, dict):+ return None+ return config.get(AUTH_TYPE_CONFIG_KEY)+++def effective_auth_type(kind: str, config: Any) -> str:+ stored = stored_auth_type(config)+ return stored if stored is not None else default_auth_type(kind).value+++def _unset_q() -> Q:+ # `isnull=True` on a JSON key matches a missing key only. A JSON null counts as unset too,+ # the same as `config.get("auth_type")` in Python.+ return Q(config__auth_type__isnull=True) | Q(config__auth_type=None)+++def _kinds_with_default_q(auth_type: IntegrationAuthType) -> Q | None:+ kinds = [kind for kind, entry in _KIND_AUTH_TYPES.items() if entry.default == auth_type]+ clauses: list[Q] = []+ if kinds:+ clauses.append(Q(kind__in=kinds))+ if UNKNOWN_KIND_AUTH_TYPES.default == auth_type:+ clauses.append(~Q(kind__in=list(_KIND_AUTH_TYPES)))+ return _any_of(clauses)+++def _any_of(clauses: Iterable[Q]) -> Q | None:+ result: Q | None = None+ for clause in clauses:+ result = clause if result is None else result | clause+ return result+++def rows_with_auth_type_q(auth_type: IntegrationAuthType) -> Q:+ explicit = Q(config__auth_type=auth_type.value)+ kinds = _kinds_with_default_q(auth_type)+ return explicit if kinds is None else explicit | (_unset_q() & kinds)+++def rows_with_default_auth_type_q() -> Q:+ # Correct because `normalized_config` never lets a row store the default auth type of its kind.+ return _unset_q()+++def rows_without_source_marker_q() -> Q:+ return ~Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)+++def normalized_config(kind: Any, config: Any) -> Any:+ """Drop a stored auth type that is null or the default of the kind.++ So a row has `config["auth_type"]` only when its auth type is not the default, and the filter for+ default rows stays one short clause. `Integration.save` and `bulk_create` call it.+ """+ if not isinstance(config, dict) or AUTH_TYPE_CONFIG_KEY not in config or not isinstance(kind, str):+ return config+ stored = config[AUTH_TYPE_CONFIG_KEY]+ if stored is not None and stored != default_auth_type(kind):+ return config+ return {key: value for key, value in config.items() if key != AUTH_TYPE_CONFIG_KEY}+++@frozen+class AuthRequirement:+ """The auth types that a caller can handle.++ It is the input of `_with_auth`. Later it can also carry the scopes that a caller needs, so that+ one check compares them with the scopes that a row grants.+ """++ any: bool = False+ kind_default: bool = False+ auth_types: frozenset[IntegrationAuthType] = frozenset()++ @classmethod+ def of(cls, *selectors: "IntegrationAuthType | AuthTypeSelector | str") -> Self:+ if not selectors:+ raise ValueError("_with_auth() needs at least one auth type, DEFAULT or ANY")+ if ANY in selectors:+ if len(selectors) > 1:+ raise ValueError("ANY already includes every auth type, so pass it alone")+ return cls(any=True)+ auth_types: set[IntegrationAuthType] = set()+ for selector in selectors:+ if selector is DEFAULT:+ continue+ if isinstance(selector, AuthTypeSelector) or selector not in IntegrationAuthType.values:+ raise ValueError(f"Unknown integration auth type: {selector!r}")+ auth_types.add(IntegrationAuthType(selector))+ return cls(kind_default=DEFAULT in selectors, auth_types=frozenset(auth_types))++ @classmethod+ def kind_default_only(cls) -> Self:+ return cls(kind_default=True)++ def q(self) -> Q:+ if self.any:+ return Q()+ clauses = [rows_with_auth_type_q(auth_type) for auth_type in sorted(self.auth_types)]+ if self.kind_default:+ clauses.append(rows_with_default_auth_type_q())+ result = _any_of(clauses)+ assert result is not None+ return result++ def accepts(self, kind: str | None, auth_type: str) -> bool:+ if self.any or auth_type in self.auth_types:+ return True+ return self.kind_default and kind is not None and auth_type == default_auth_type(kind)++ @property+ def single_auth_type(self) -> IntegrationAuthType | None:+ if self.any or self.kind_default or len(self.auth_types) != 1:+ return None+ return next(iter(self.auth_types))+++def config_for_write(kind: Any, config: Any, requirement: AuthRequirement) -> Any:+ """Return the config to write for a new or updated row under `requirement`.++ A requirement with one auth type writes it into the config when it is not the kind's default.+ A requirement that does not cover the row's auth type raises, so a write never makes a row that+ the same query cannot read back. `kind` is None when the caller does not pass it.+ """+ if config is None:+ config = {}+ if not isinstance(config, dict) or not (kind is None or isinstance(kind, str)):+ return config++ stored = stored_auth_type(config)+ if stored is not None:+ if kind is not None and stored not in kind_auth_types(kind).supported:+ raise ValueError(f"Integration kind {kind!r} does not support auth type {stored!r}")+ if not requirement.accepts(kind, stored):+ raise ValueError(f"Auth type {stored!r} is not in the _with_auth() of this query")+ return config++ target = requirement.single_auth_type+ if target is not None:+ if kind is None:+ if any(entry.default == target for entry in _KIND_AUTH_TYPES.values()):+ raise ValueError(f"Pass the kind, so the write can tell if {target.value!r} is its default")+ return {**config, AUTH_TYPE_CONFIG_KEY: target.value}+ if target not in kind_auth_types(kind).supported:+ raise ValueError(f"Integration kind {kind!r} does not support auth type {target.value!r}")+ if target == default_auth_type(kind):+ return config+ return {**config, AUTH_TYPE_CONFIG_KEY: target.value}++ if kind is not None and not requirement.accepts(kind, default_auth_type(kind).value):+ raise ValueError("A write through _with_auth() needs exactly one auth type, or one that is the kind's default")+ return configdiff --git a/posthog/models/integration/aws.py b/posthog/models/integration/aws.pyindex 92092e958016..baf0a3f89d14 100644--- a/posthog/models/integration/aws.py+++ b/posthog/models/integration/aws.py@@ -14,6 +14,7 @@ from rest_framework.exceptions import ValidationErrorfrom posthog.credentials import AWSKeyPairfrom posthog.credentials_service.callers import CredentialsCallerfrom posthog.credentials_service.settings_fallback import get_or_settings+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.models.utils import IntegrityErrorfrom posthog.security.url_validation import is_url_allowed@@ -258,13 +259,13 @@ class AWSRoleBasedIntegration:integration_kind: ClassVar[_AWSKindType]- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != self.integration_kind:raise common.IntegrationError("Integration provided is not the expected AWS integration "f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')")- self.integration = integration+ self.integration = model.integration_model(integration)try:self.aws_role_arn = integration.config["aws_role_arn"]except KeyError:@@ -413,14 +414,14 @@ class AWSCredentialsIntegration:integration_kind: ClassVar[_AWSKindType]- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != self.integration_kind:raise common.IntegrationError("Integration provided is not the expected AWS integration "f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')")- self.integration = integration- self.credentials = _read_aws_credentials(integration)+ self.integration = model.integration_model(integration)+ self.credentials = _read_aws_credentials(self.integration)@propertydef aws_account_id(self) -> str | None:@@ -540,13 +541,13 @@ class S3CompatibleIntegration:credentials: AWSKeyPairendpoint_url: str- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != model.Integration.IntegrationKind.S3_COMPATIBLE:raise common.IntegrationError(f"Integration provided is not an S3-compatible integration (got kind='{integration.kind}')")- self.integration = integration- self.credentials = _read_aws_credentials(integration)+ self.integration = model.integration_model(integration)+ self.credentials = _read_aws_credentials(self.integration)try:self.endpoint_url = integration.config["endpoint_url"]except KeyError:diff --git a/posthog/models/integration/azure_blob.py b/posthog/models/integration/azure_blob.pyindex 4b1fc1dc6a01..0aebb9bb65a1 100644--- a/posthog/models/integration/azure_blob.py+++ b/posthog/models/integration/azure_blob.py@@ -6,6 +6,7 @@ from itertools import groupbyfrom operator import itemgetterfrom urllib.parse import urlsplit+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.security.url_validation import (ResolvedIPs,@@ -33,12 +34,12 @@ class AzureBlobIntegration:integration: model.Integrationconnection_string: str- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != model.Integration.IntegrationKind.AZURE_BLOB.value:raise AzureBlobIntegrationError(f"Integration provided is not an Azure Blob integration (got kind='{integration.kind}')")- self.integration = integration+ self.integration = model.integration_model(integration)try:self.connection_string = self.integration.sensitive_config["connection_string"]diff --git a/posthog/models/integration/clickup.py b/posthog/models/integration/clickup.pyindex d7133ec3cb2d..121827c942e7 100644--- a/posthog/models/integration/clickup.py+++ b/posthog/models/integration/clickup.py@@ -5,6 +5,7 @@ import structlogfrom rest_framework.exceptions import ValidationErrorfrom posthog.exceptions_capture import capture_exception+from posthog.integrations.contracts import IntegrationRecordfrom . import common, model@@ -14,11 +15,11 @@ logger = structlog.get_logger(__name__)class ClickUpIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "clickup":raise Exception("ClickUpIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)def _check_auth_error(self, response: requests.Response, context: str) -> None:if response.status_code == 401:diff --git a/posthog/models/integration/databricks.py b/posthog/models/integration/databricks.pyindex b49a0363279a..71fe5b740882 100644--- a/posthog/models/integration/databricks.py+++ b/posthog/models/integration/databricks.py@@ -2,6 +2,7 @@import re+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.security.url_validation import is_url_allowed@@ -61,10 +62,10 @@ class DatabricksIntegration:client_id: strclient_secret: str- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != model.Integration.IntegrationKind.DATABRICKS.value:raise DatabricksIntegrationError("Integration provided is not a Databricks integration")- self.integration = integration+ self.integration = model.integration_model(integration)try:self.server_hostname = self.integration.config["server_hostname"].lower()diff --git a/posthog/models/integration/email.py b/posthog/models/integration/email.pyindex 132309a4899d..9a25cf0b274f 100644--- a/posthog/models/integration/email.py+++ b/posthog/models/integration/email.py@@ -18,6 +18,7 @@ from rest_framework.exceptions import ValidationErrorfrom posthog.dataclasses import frozenfrom posthog.exceptions_capture import capture_exception+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.team.team import Teamfrom posthog.models.user import Userfrom posthog.plugins.plugin_server_api import reload_integrations_on_workers@@ -42,10 +43,10 @@ class EmailDomainStatusReport:class EmailIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "email":raise Exception("EmailIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@classmethoddef create_native_integration(diff --git a/posthog/models/integration/github.py b/posthog/models/integration/github.pyindex 11cc061ca850..b07afc04ec09 100644--- a/posthog/models/integration/github.py+++ b/posthog/models/integration/github.py@@ -16,6 +16,7 @@ import structlogfrom posthog.egress.github.transport import GitHubRateLimitError, github_requestfrom posthog.egress.limiter.policies import Priorityfrom posthog.egress.transport.transport import EgressBudgetExhausted+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.github_integration_base import (GitHubIntegrationBase,GitHubIntegrationError,@@ -368,10 +369,10 @@ class GitHubIntegration(GitHubIntegrationBase):raise exhaustedreturn None- def __init__(self, integration: model.Integration, *, source: str, priority: Priority) -> None:+ def __init__(self, integration: IntegrationRecord, *, source: str, priority: Priority) -> None:if integration.kind != "github":raise Exception("GitHubIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)self.source = sourceself.priority = prioritydiff --git a/posthog/models/integration/gitlab.py b/posthog/models/integration/gitlab.pyindex e4fef7d0233a..c72f894c7392 100644--- a/posthog/models/integration/gitlab.py+++ b/posthog/models/integration/gitlab.py@@ -6,6 +6,7 @@ from urllib.parse import urlencode, urlparseimport requests+from posthog.integrations.contracts import IntegrationRecordfrom posthog.security.url_validation import is_url_allowedfrom . import common, model@@ -77,10 +78,10 @@ class GitLabIntegration:return integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "gitlab":raise Exception("GitLabIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@propertydef project_path(self) -> str:diff --git a/posthog/models/integration/google_ads.py b/posthog/models/integration/google_ads.pyindex 44fbe16eb90c..403da540156d 100644--- a/posthog/models/integration/google_ads.py+++ b/posthog/models/integration/google_ads.py@@ -11,6 +11,7 @@ from posthog.credentials_service.callers import CredentialsCallerfrom posthog.credentials_service.settings_fallback import get_or_settingsfrom posthog.egress.slack.client import SlackWebClient as WebClientfrom posthog.exceptions_capture import capture_exception+from posthog.integrations.contracts import IntegrationRecordfrom . import common, model@@ -51,11 +52,11 @@ def _google_ads_request(method: str, url: str, **kwargs) -> requests.Response:class GoogleAdsIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "google-ads":raise Exception("GoogleAdsIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@propertydef client(self) -> WebClient:diff --git a/posthog/models/integration/google_cloud.py b/posthog/models/integration/google_cloud.pyindex 0cbb015b78b5..4b8a9d94dc79 100644--- a/posthog/models/integration/google_cloud.py+++ b/posthog/models/integration/google_cloud.py@@ -8,6 +8,7 @@ from google.auth.transport.requests import Request as GoogleRequestfrom google.oauth2 import service_accountfrom rest_framework.exceptions import ValidationError+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.plugins.plugin_server_api import reload_integrations_on_workers@@ -71,8 +72,8 @@ def is_unique_service_account_by_organization_id(service_account_email: str, orgclass GoogleCloudServiceAccountIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:- self.integration = integration+ def __init__(self, integration: IntegrationRecord) -> None:+ self.integration = model.integration_model(integration)@classmethoddef integration_from_service_account(@@ -158,8 +159,8 @@ class GoogleCloudIntegration:supported_kinds = ["google-pubsub", "google-cloud-storage"]integration: model.Integration- def __init__(self, integration: model.Integration) -> None:- self.integration = integration+ def __init__(self, integration: IntegrationRecord) -> None:+ self.integration = model.integration_model(integration)@classmethoddef integration_from_key(diff --git a/posthog/models/integration/jira.py b/posthog/models/integration/jira.pyindex 83b89d4a2fd6..08bff68e9889 100644--- a/posthog/models/integration/jira.py+++ b/posthog/models/integration/jira.py@@ -10,6 +10,7 @@ import structlogfrom rest_framework.exceptions import ValidationErrorfrom posthog.exceptions_capture import capture_exception+from posthog.integrations.contracts import IntegrationRecordfrom . import common, model, oauthfrom .assignees import MAX_ASSIGNEES, Assignee, AssigneeLookupFailed, ReconnectRequired@@ -71,11 +72,11 @@ def description_to_adf(description: str) -> dict[str, Any]:class JiraIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "jira":raise Exception("JiraIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)def cloud_id(self) -> str | None:"""Get the Atlassian cloud ID from the integration config"""diff --git a/posthog/models/integration/linear.py b/posthog/models/integration/linear.pyindex e2c38a60549d..73c3c5840a9b 100644--- a/posthog/models/integration/linear.py+++ b/posthog/models/integration/linear.py@@ -6,6 +6,8 @@ import requestsimport structlogfrom rest_framework.exceptions import ValidationError+from posthog.integrations.contracts import IntegrationRecord+from . import common, modelfrom .assignees import MAX_ASSIGNEES, Assignee, AssigneeLookupFailed@@ -15,11 +17,11 @@ logger = structlog.get_logger(__name__)class LinearIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "linear":raise Exception("LinearIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)def url_key(self) -> str:return common.dot_get(self.integration.config, "data.viewer.organization.urlKey")diff --git a/posthog/models/integration/linkedin_ads.py b/posthog/models/integration/linkedin_ads.pyindex 3615cb6490dc..213fac4529ad 100644--- a/posthog/models/integration/linkedin_ads.py+++ b/posthog/models/integration/linkedin_ads.py@@ -5,6 +5,7 @@ import structlogfrom rest_framework.exceptions import ValidationErrorfrom posthog.egress.slack.client import SlackWebClient as WebClient+from posthog.integrations.contracts import IntegrationRecordfrom . import common, model@@ -14,11 +15,11 @@ logger = structlog.get_logger(__name__)class LinkedInAdsIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "linkedin-ads":raise Exception("LinkedInAdsIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@propertydef client(self) -> WebClient:diff --git a/posthog/models/integration/meta.py b/posthog/models/integration/meta.pyindex b49bacff4aff..d5242a7a0ac6 100644--- a/posthog/models/integration/meta.py+++ b/posthog/models/integration/meta.py@@ -5,6 +5,8 @@ import timeimport requestsimport structlog+from posthog.integrations.contracts import IntegrationRecord+from . import common, model, oauth, refresh_trackinglogger = structlog.get_logger(__name__)@@ -22,10 +24,10 @@ class MetaGraphIntegration:kind: str = ""api_version: str = common.META_GRAPH_API_VERSION- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != self.kind:raise Exception(f"{type(self).__name__} init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)def refresh_access_token(self):oauth_config = oauth.OauthIntegration.oauth_config_for_kind(self.integration.kind)diff --git a/posthog/models/integration/model.py b/posthog/models/integration/model.pyindex 2e5e0b2cb7ec..46568df96364 100644--- a/posthog/models/integration/model.py+++ b/posthog/models/integration/model.py@@ -1,20 +1,34 @@"""The Integration Django model, its manager, and generic secret-decryption helpers."""+from collections.abc import Iterable, Mappingfrom typing import Any, Self, castfrom django.db import modelsfrom django.db.models import Q+from django.db.models.manager import ManagerDescriptorfrom django.utils.functional import Promiseimport structlogfrom prometheus_client import Counterfrom posthog.helpers.encrypted_fields import FERNET_TOKEN_PREFIX, EncryptedJSONField+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.rbac.decorators import field_access_controlfrom posthog.sync import database_sync_to_asyncfrom . import common+from .auth_types import (+ AuthRequirement,+ AuthTypeSelector,+ IntegrationAuthType,+ config_for_write,+ default_auth_type,+ effective_auth_type,+ normalized_config,+ rows_with_default_auth_type_q,+ rows_without_source_marker_q,+)logger = structlog.get_logger(__name__)@@ -87,24 +101,120 @@ def _decrypted_sensitive_value(integration: "Integration", field_name: str) -> sclass IntegrationQuerySet(models.QuerySet["Integration"]):+ # Set by `_with_auth`. None means the query did not state its auth types.+ _auth_requirement: AuthRequirement | None = None+ # True on the queryset that `Integration.objects` returns without `_with_auth`. It holds only+ # rows with the default auth type of their kind and without the warehouse source marker, so code+ # that does not state its auth types never gets a row it cannot handle. A source row can have+ # the default auth type of its kind and still hold a different config shape.+ _implicit_default_auth: bool = False++ def _clone(self) -> Self:+ clone = super()._clone() # type: ignore[misc]+ clone._auth_requirement = self._auth_requirement+ clone._implicit_default_auth = self._implicit_default_auth+ return clone++ def _with_auth(self, *auth_types: IntegrationAuthType | AuthTypeSelector | str) -> Self:+ """Keep only the rows whose auth type is in `auth_types`. Internal to the integrations SDK.++ Pass `DEFAULT` for the default auth type of each row's kind, or `ANY` for every row. A row+ without `config["auth_type"]` has the default auth type of its kind. A create through a+ `_with_auth` with one non-default auth type writes it into `config["auth_type"]`.+ """+ if self._implicit_default_auth:+ raise ValueError(+ "Call _with_auth() directly on Integration.objects. The queryset without it holds only rows "+ "with the default auth type of their kind."+ )+ if self._auth_requirement is not None and not self._auth_requirement.any:+ # A second selector would filter by both but write by the last, so a create could make a+ # row that this queryset cannot read back. Narrow only from ANY.+ raise ValueError("This queryset already has _with_auth(). Narrow only from _with_auth(ANY).")+ requirement = AuthRequirement.of(*auth_types)+ queryset = self._chain() if requirement.any else self.filter(requirement.q()) # type: ignore[attr-defined]+ queryset._auth_requirement = requirement+ return queryset++ def _with_implicit_default_auth(self) -> Self:+ queryset = self.filter(rows_with_default_auth_type_q(), rows_without_source_marker_q())+ queryset._implicit_default_auth = True+ return queryset++ def _write_requirement(self) -> AuthRequirement | None:+ if self._auth_requirement is not None:+ return self._auth_requirement+ return AuthRequirement.kind_default_only() if self._implicit_default_auth else None++ def _with_config_for_write(self, values: dict[str, Any], kind: Any) -> dict[str, Any]:+ requirement = self._write_requirement()+ if requirement is None:+ return values+ return {**values, "config": config_for_write(kind, values.get("config"), requirement)}++ # The django-stubs plugin rebinds this subclass to its own TypeVar, so a concrete signature cannot match.+ def create(self, **kwargs: Any) -> "Integration": # type: ignore[override]+ return super().create(**self._with_config_for_write(kwargs, kwargs.get("kind")))++ def bulk_create(self, objs: Iterable["Integration"], *args: Any, **kwargs: Any) -> list["Integration"]: # type: ignore[override]+ objs = list(objs)+ requirement = self._write_requirement()+ for obj in objs:+ if requirement is not None:+ obj.config = config_for_write(obj.kind, obj.config, requirement)+ # bulk_create does not call save(), so normalize here too.+ obj.config = normalized_config(obj.kind, obj.config)+ return cast(list[Integration], super().bulk_create(cast(Iterable[Any], objs), *args, **kwargs))++ def update_or_create( # type: ignore[override]+ self,+ defaults: Mapping[str, Any] | None = None,+ create_defaults: Mapping[str, Any] | None = None,+ **kwargs: Any,+ ) -> tuple["Integration", bool]:+ # The update path sets `config` from `defaults` on the row it found. Without this, a new+ # config without the key would turn a typed-in row back into a default row.+ kind = kwargs.get("kind") or (defaults or {}).get("kind") or (create_defaults or {}).get("kind")+ if defaults is not None and "config" in defaults:+ defaults = self._with_config_for_write(dict(defaults), kind)+ return super().update_or_create(defaults=defaults, create_defaults=create_defaults, **kwargs)+def for_github_installation_id(self, installation_id: str | int) -> Self:return self.filter(Q(config__installation_id=str(installation_id)) | Q(config__installation_id=int(installation_id)))-class IntegrationManager(models.Manager["Integration"]):+class IntegrationModelManager(models.Manager["Integration"]):_queryset_class = IntegrationQuerySet+ # False on `_AllRowsManager`, the default manager that Django internals use.+ _filters_to_default_auth: bool = Truedef get_queryset(self) -> IntegrationQuerySet:- return IntegrationQuerySet(self.model, using=self._db)+ queryset = IntegrationQuerySet(self.model, using=self._db, hints=self._hints) # type: ignore[attr-defined]+ if self._is_related_manager() or not self._filters_to_default_auth:+ return queryset+ return queryset._with_implicit_default_auth()++ def _with_auth(self, *auth_types: IntegrationAuthType | AuthTypeSelector | str) -> IntegrationQuerySet:+ if self._is_related_manager():+ return self.get_queryset()._with_auth(*auth_types)+ queryset = IntegrationQuerySet(self.model, using=self._db, hints=self._hints) # type: ignore[attr-defined]+ return queryset._with_auth(*auth_types)def filter(self, *args: Any, **kwargs: Any) -> IntegrationQuerySet:return self.get_queryset().filter(*args, **kwargs)+ def _is_related_manager(self) -> bool:+ # Django builds related managers such as `team.integration_set` as subclasses of this class,+ # and only they have `core_filters`. They stay unfiltered, the same as the base manager that+ # cascades, `save()` and `refresh_from_db()` use.+ return hasattr(self, "core_filters")+def first_github_for_team_installation(self, team_id: int, installation_id: str) -> "Integration | None":return (- self.filter(team_id=team_id, kind=Integration.IntegrationKind.GITHUB)+ self._with_auth(IntegrationAuthType.OAUTH)+ .filter(team_id=team_id, kind=Integration.IntegrationKind.GITHUB).for_github_installation_id(installation_id).first())@@ -112,18 +222,25 @@ class IntegrationManager(models.Manager["Integration"]):def first_github_for_user_installation(self, user: User, installation_id: str) -> "Integration | None":user_team_ids = user.teams.values_list("id", flat=True)return (- self.filter(team_id__in=user_team_ids, kind=Integration.IntegrationKind.GITHUB)+ self._with_auth(IntegrationAuthType.OAUTH)+ .filter(team_id__in=user_team_ids, kind=Integration.IntegrationKind.GITHUB).for_github_installation_id(installation_id).first())+class _AllRowsManager(IntegrationModelManager):+ _filters_to_default_auth = False++def integration_kind_choices() -> list[tuple[str, str | Promise]]:# Callable so growing the enum doesn't generate a no-op migration.return list(Integration.IntegrationKind.choices)class Integration(models.Model):+ AuthType = IntegrationAuthType+class IntegrationKind(models.TextChoices):ANTHROPIC = "anthropic"APPLE_ADS = "apple-ads"@@ -178,7 +295,11 @@ class Integration(models.Model):VERCEL = "vercel"YOUTUBE_ANALYTICS = "youtube-analytics"+ # Declared for type checkers that do not read Django fields: ty refuses the model as an+ # `IntegrationRecord` without them.+ id: intteam = models.ForeignKey("Team", on_delete=models.CASCADE)+ team_id: int# The integration type identifierkind = field_access_control(models.CharField(max_length=32, choices=integration_kind_choices), "project", "admin")@@ -202,8 +323,14 @@ class Integration(models.Model):# Metacreated_at = models.DateTimeField(auto_now_add=True, blank=True)created_by = models.ForeignKey("User", on_delete=models.SET_NULL, null=True, blank=True)+ created_by_id: int | None- objects: IntegrationManager = IntegrationManager()+ # Django's default manager, because it is declared first. Admin widgets, DRF relation fields,+ # `validate_constraints` and `get_object_or_404` read through it, so it must see every row. After+ # the class, `Integration.objects` points to `_implicit_default_auth_objects` instead. The Django+ # name stays `objects`, so the migration state does not change.+ objects: IntegrationModelManager = _AllRowsManager()+ _implicit_default_auth_objects: IntegrationModelManager = IntegrationModelManager()class Meta:constraints = [@@ -213,8 +340,20 @@ class Integration(models.Model):]indexes = [models.Index(fields=["kind", "integration_id"], name="posthog_integration_kind_ext")]+ def save(self, *args: Any, **kwargs: Any) -> None:+ self.config = normalized_config(self.kind, self.config)+ super().save(*args, **kwargs)++ @property+ def auth_type(self) -> str:+ """The stored `config["auth_type"]`, or the default auth type of the kind when the key is missing."""+ return effective_auth_type(self.kind, self.config)+@propertydef display_name(self) -> str:+ if self.auth_type != default_auth_type(self.kind):+ # The branches below read the config shape of the default auth type.+ return self.config.get("display_name") or f"ID: {self.integration_id}"if self.kind == "twitter-ads":return self.config.get("screen_name") or self.integration_idif self.kind == "pinterest-ads":@@ -311,7 +450,20 @@ class Integration(models.Model):return _decrypted_sensitive_value(self, "refresh_token")-def defer_repository_cache_fields(queryset: models.QuerySet[Integration]) -> models.QuerySet[Integration]:+# `setattr`, because the type of `objects` is the manager, not its descriptor.+setattr(Integration, "objects", ManagerDescriptor(Integration._implicit_default_auth_objects)) # noqa: B010+++def integration_model(record: IntegrationRecord) -> Integration:+ """The model behind a row of the integrations SDK.++ The SDK returns its rows as the read-only `IntegrationRecord`, and they are model instances at+ runtime. Only the per-kind adapters next to the model use it, because they still write the row.+ """+ return cast(Integration, record)+++def defer_repository_cache_fields[QuerySetT: models.QuerySet[Integration]](queryset: QuerySetT) -> QuerySetT:return queryset.defer("repository_cache", "repository_cache_updated_at")diff --git a/posthog/models/integration/oauth.py b/posthog/models/integration/oauth.pyindex 144016df4a3f..117714c49d4e 100644--- a/posthog/models/integration/oauth.py+++ b/posthog/models/integration/oauth.py@@ -23,6 +23,7 @@ from posthog.credentials_service.instance_settings import get_many_or_instance_sfrom posthog.credentials_service.settings_fallback import get_many_or_settingsfrom posthog.dataclasses import frozenfrom posthog.egress.slack.observability import record_slack_api_response+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.plugins.plugin_server_api import reload_integrations_on_workersfrom posthog.schema_enums import SlackIntegrationScope@@ -555,8 +556,8 @@ class OauthIntegration:def __str__(self) -> str:return f"OauthIntegration(integration={self.integration.id}, kind={self.integration.kind}, team={self.integration.team_id})"- def __init__(self, integration: model.Integration) -> None:- self.integration = integration+ def __init__(self, integration: IntegrationRecord) -> None:+ self.integration = model.integration_model(integration)@classmethod@cache_for(timedelta(minutes=5))diff --git a/posthog/models/integration/postgres.py b/posthog/models/integration/postgres.pyindex b8a45edfcb43..90ba9835278c 100644--- a/posthog/models/integration/postgres.py+++ b/posthog/models/integration/postgres.py@@ -2,6 +2,7 @@from typing import ClassVar, Literal, NamedTuple+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.psycopg_helpers import has_ipv6_routefrom posthog.security.postgres_hosts import ipv6_only_host_message@@ -68,14 +69,14 @@ class PostgreSQLServerIntegration:integration: model.Integrationintegration_kind: ClassVar[_PostgreSQLServerKindType]- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != self.integration_kind:raise common.IntegrationError("Integration provided is not the expected PostgreSQL server integration "f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')")- self.integration = integration+ self.integration = model.integration_model(integration)@classmethoddef integration_from_config(diff --git a/posthog/models/integration/push.py b/posthog/models/integration/push.pyindex 0085b4afb213..cb0aaec7c0b7 100644--- a/posthog/models/integration/push.py+++ b/posthog/models/integration/push.py@@ -12,6 +12,7 @@ from google.auth.transport.requests import Request as GoogleRequestfrom google.oauth2 import service_accountfrom rest_framework.exceptions import ValidationError+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.plugins.plugin_server_api import reload_integrations_on_workers@@ -124,10 +125,10 @@ def preserved_push_config(class FirebaseIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "firebase":raise Exception("FirebaseIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@classmethoddef integration_from_key(@@ -278,10 +279,10 @@ class ApplePushIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "apns":raise Exception("ApplePushIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)@classmethoddef integration_from_key(diff --git a/posthog/models/integration/slack.py b/posthog/models/integration/slack.pyindex 3273a167dcf2..3bc3aae0d1c2 100644--- a/posthog/models/integration/slack.py+++ b/posthog/models/integration/slack.py@@ -28,6 +28,7 @@ from posthog.credentials_service.settings_fallback import get_many_or_settingsfrom posthog.egress.slack.client import SlackWebClient as WebClientfrom posthog.ingress.slack.provider import build_slack_signature_schemefrom posthog.ingress.verify.schemes import VerificationOutcome, hmac_sha256_signature+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.instance_setting import get_instance_settingsfrom . import model@@ -91,11 +92,11 @@ SLACK_MEMBERS_MAX_REQUESTS = 10class SlackIntegration:integration: model.Integration- def __init__(self, integration: model.Integration, *, source: str = "integration") -> None:+ def __init__(self, integration: IntegrationRecord, *, source: str = "integration") -> None:if integration.kind not in SLACK_INTEGRATION_KINDS:raise Exception("SlackIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)# The egress `source` label. A flow that posts on its own names itself here, so its Slack# calls and 429s can be told apart from every other caller of the same integration.self.source = sourcediff --git a/posthog/models/integration/snowflake.py b/posthog/models/integration/snowflake.pyindex 927f2b467ade..3220f2b8ac23 100644--- a/posthog/models/integration/snowflake.py+++ b/posthog/models/integration/snowflake.py@@ -4,6 +4,7 @@ import refrom django.db import transaction+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.user import Userfrom posthog.models.utils import IntegrityError@@ -46,12 +47,12 @@ class SnowflakeIntegration:user: strauthentication_type: str- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != model.Integration.IntegrationKind.SNOWFLAKE:raise SnowflakeIntegrationError(f"Integration provided is not a Snowflake integration (got kind='{integration.kind}')")- self.integration = integration+ self.integration = model.integration_model(integration)try:self.account = integration.config["account"]self.user = integration.config["user"]diff --git a/posthog/models/integration/stripe.py b/posthog/models/integration/stripe.pyindex 5160a5c7e141..a6794f032b73 100644--- a/posthog/models/integration/stripe.py+++ b/posthog/models/integration/stripe.py@@ -17,6 +17,7 @@ from posthog.credentials_service.errors import CredentialsServiceFailurefrom posthog.credentials_service.settings_fallback import get_or_settingsfrom posthog.dataclasses import frozenfrom posthog.exceptions_capture import capture_exception+from posthog.integrations.contracts import IntegrationRecordfrom posthog.models.oauth import OAuthAccessToken, OAuthApplication, OAuthRefreshToken, lock_oauth_connectionfrom posthog.models.user import Userfrom posthog.models.utils import generate_random_oauth_access_token, generate_random_oauth_refresh_token@@ -129,10 +130,10 @@ class StripeIntegration:])- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "stripe":raise ValueError(f"Expected stripe integration, got {integration.kind}")- self.integration = integration+ self.integration = model.integration_model(integration)def _stripe_client(self) -> "StripeClient | None":# Returns None when the required env vars are missing so callers can skip Stripediff --git a/posthog/models/integration/twilio.py b/posthog/models/integration/twilio.pyindex 2be240e2b9e4..9c11de19a2d0 100644--- a/posthog/models/integration/twilio.py+++ b/posthog/models/integration/twilio.py@@ -4,6 +4,8 @@ from typing import TYPE_CHECKINGfrom rest_framework.exceptions import ValidationError+from posthog.integrations.contracts import IntegrationRecord+from . import modelif TYPE_CHECKING:@@ -13,10 +15,10 @@ if TYPE_CHECKING:class TwilioIntegration:integration: model.Integration- def __init__(self, integration: model.Integration) -> None:+ def __init__(self, integration: IntegrationRecord) -> None:if integration.kind != "twilio":raise Exception("TwilioIntegration init called with Integration with wrong 'kind'")- self.integration = integration+ self.integration = model.integration_model(integration)self._account_sid = self.integration.config["account_sid"]self._auth_token = self.integration.sensitive_config["auth_token"]diff --git a/posthog/models/test/test_integration_auth_types.py b/posthog/models/test/test_integration_auth_types.pynew file mode 100644index 000000000000..527bdd189fd6--- /dev/null+++ b/posthog/models/test/test_integration_auth_types.py@@ -0,0 +1,299 @@+from collections.abc import Callable+from typing import Any++from posthog.test.base import BaseTest++from django.db import models+from django.db.migrations.state import ModelState+from django.test import SimpleTestCase++from parameterized import parameterized++from posthog.integrations.catalogue import KINDS+from posthog.models import Team, User+from posthog.models.integration import ANY, DEFAULT, AuthRequirement, Integration, IntegrationAuthType+from posthog.models.integration.auth_types import config_for_write++AuthType = IntegrationAuthType+++class TestIntegrationAuthTypeCatalogue(SimpleTestCase):+ def test_every_integration_kind_has_a_catalogue_entry(self) -> None:+ assert set(KINDS) == set(Integration.IntegrationKind.values)++ @parameterized.expand(+ [+ ("nothing", ()),+ ("any_with_a_type", (ANY, AuthType.OAUTH)),+ ("any_with_default", (ANY, DEFAULT)),+ ("unknown_string", ("bogus",)),+ ]+ )+ def test_requirement_rejects_invalid_selectors(self, _name: str, selectors: tuple[Any, ...]) -> None:+ with self.assertRaises(ValueError):+ AuthRequirement.of(*selectors)++ @parameterized.expand(+ [+ ("non_default_type_is_written", "github", {}, (AuthType.TOKEN,), {"auth_type": "token"}),+ ("plain_string_type_is_written", "github", {"a": 1}, ("token",), {"a": 1, "auth_type": "token"}),+ ("missing_config_gets_the_type", "slack", None, (AuthType.TOKEN,), {"auth_type": "token"}),+ ("token_on_a_credentials_kind_is_written", "databricks", {}, (AuthType.TOKEN,), {"auth_type": "token"}),+ ("default_type_is_not_written", "github", {}, (AuthType.OAUTH,), {}),+ ("stored_type_in_requirement_is_kept", "github", {"auth_type": "token"}, (AuthType.TOKEN,), None),+ ("several_types_with_the_default", "github", {}, (AuthType.OAUTH, AuthType.TOKEN), {}),+ ("default_selector", "github", {}, (DEFAULT,), {}),+ ("any_keeps_config", "github", {"a": 1}, (ANY,), {"a": 1}),+ ("any_keeps_stored_type", "github", {"auth_type": "token"}, (ANY,), None),+ ("unknown_kind_default", "acme-crm", {}, (AuthType.CREDENTIALS,), {}),+ ]+ )+ def test_config_for_write(+ self,+ _name: str,+ kind: str | None,+ config: dict | None,+ selectors: tuple[Any, ...],+ expected: dict | None,+ ) -> None:+ result = config_for_write(kind, config, AuthRequirement.of(*selectors))++ assert result == (config if expected is None else expected)++ @parameterized.expand(+ [+ ("type_the_kind_does_not_support", "postgresql", {}, (AuthType.TOKEN,)),+ ("stored_type_the_kind_does_not_support", "postgresql", {"auth_type": "token"}, (ANY,)),+ ("stored_type_outside_requirement", "github", {"auth_type": "token"}, (AuthType.OAUTH,)),+ ("several_types_without_the_default", "github", {}, (AuthType.TOKEN, AuthType.CREDENTIALS)),+ ("unknown_kind_with_a_specific_type", "acme-crm", {}, (AuthType.TOKEN,)),+ # Every auth type is the default of some kind, so a write without the kind cannot tell.+ ("no_kind_with_oauth", None, {}, (AuthType.OAUTH,)),+ ("no_kind_with_token", None, {}, (AuthType.TOKEN,)),+ ]+ )+ def test_config_for_write_rejects(+ self, _name: str, kind: str | None, config: dict, selectors: tuple[Any, ...]+ ) -> None:+ with self.assertRaises(ValueError):+ config_for_write(kind, config, AuthRequirement.of(*selectors))++ @parameterized.expand(+ [+ ("missing_key", {}, None),+ ("stored_default", {"auth_type": "oauth"}, None),+ ("stored_non_default", {"auth_type": "token"}, ValueError),+ ]+ )+ def test_config_for_write_without_with_auth(+ self, _name: str, config: dict, expected_error: type[Exception] | None+ ) -> None:+ requirement = AuthRequirement.kind_default_only()++ if expected_error is None:+ assert config_for_write("github", config, requirement) == config+ else:+ with self.assertRaises(expected_error):+ config_for_write("github", config, requirement)+++ROWS: dict[str, tuple[str, dict]] = {+ "github_oauth": ("github", {}),+ "github_oauth_explicit": ("github", {"auth_type": "oauth"}),+ "github_null": ("github", {"auth_type": None}),+ "github_token": ("github", {"auth_type": "token"}),+ "slack_oauth": ("slack", {}),+ "slack_token": ("slack", {"auth_type": "token"}),+ "postgres": ("postgresql", {}),+ "postgres_source": ("postgresql", {"warehouse_source_type": "Postgres"}),+ "unknown_kind": ("acme-crm", {}),+}+DEFAULT_ROWS = {+ "github_oauth",+ "github_oauth_explicit",+ "github_null",+ "slack_oauth",+ "postgres",+ "postgres_source",+ "unknown_kind",+}+# An unqualified query also skips warehouse source rows, whose config has another shape.+IMPLICIT_ROWS = DEFAULT_ROWS - {"postgres_source"}+GITHUB_OAUTH_ROWS = {"github_oauth", "github_oauth_explicit", "github_null"}+++class TestIntegrationWithAuth(BaseTest):+ def setUp(self) -> None:+ super().setUp()+ self.rows = {+ name: Integration.objects._with_auth(ANY).create(+ team=self.team, kind=kind, integration_id=name, config=config+ )+ for name, (kind, config) in ROWS.items()+ }++ def _names(self, queryset: models.QuerySet[Integration]) -> set[str]:+ return {row.integration_id for row in queryset.filter(team=self.team) if row.integration_id}++ @parameterized.expand(+ [+ ("oauth", lambda: Integration.objects._with_auth(AuthType.OAUTH), GITHUB_OAUTH_ROWS | {"slack_oauth"}),+ (+ "token_across_kinds",+ lambda: Integration.objects._with_auth(AuthType.TOKEN),+ {"github_token", "slack_token"},+ ),+ (+ "token_with_kind_in",+ lambda: Integration.objects._with_auth(AuthType.TOKEN).filter(kind__in=["slack", "postgresql"]),+ {"slack_token"},+ ),+ (+ "credentials_with_the_unknown_kind_default",+ lambda: Integration.objects._with_auth(AuthType.CREDENTIALS),+ {"postgres", "postgres_source", "unknown_kind"},+ ),+ (+ "oauth_with_kind",+ lambda: Integration.objects._with_auth(AuthType.OAUTH).filter(kind="github"),+ GITHUB_OAUTH_ROWS,+ ),+ ("default_selector", lambda: Integration.objects._with_auth(DEFAULT), DEFAULT_ROWS),+ ("default_and_token", lambda: Integration.objects._with_auth(DEFAULT, AuthType.TOKEN), set(ROWS)),+ ("any", lambda: Integration.objects._with_auth(ANY), set(ROWS)),+ ("unqualified_query", lambda: Integration.objects.all(), IMPLICIT_ROWS),+ ("unqualified_filter", lambda: Integration.objects.filter(kind="github"), GITHUB_OAUTH_ROWS),+ (+ "unqualified_filter_skips_source_rows",+ lambda: Integration.objects.filter(kind="postgresql"),+ {"postgres"},+ ),+ ("default_manager", lambda: Integration._default_manager.all(), set(ROWS)),+ (+ "narrowed_twice",+ lambda: Integration.objects._with_auth(ANY)._with_auth(AuthType.TOKEN),+ {"github_token", "slack_token"},+ ),+ ]+ )+ def test_with_auth_filters_rows(+ self, _name: str, queryset: Callable[[], models.QuerySet[Integration]], expected: set[str]+ ) -> None:+ assert self._names(queryset()) == expected++ def test_auth_type_property_reads_config_or_kind_default(self) -> None:+ assert {+ row.integration_id: row.auth_type for row in Integration.objects._with_auth(ANY).filter(team=self.team)+ } == {+ "github_oauth": "oauth",+ "github_oauth_explicit": "oauth",+ "github_null": "oauth",+ "github_token": "token",+ "slack_oauth": "oauth",+ "slack_token": "token",+ "postgres": "credentials",+ "postgres_source": "credentials",+ "unknown_kind": "credentials",+ }++ def test_save_drops_a_stored_auth_type_that_is_null_or_the_kind_default(self) -> None:+ token_row = self.rows["github_token"]+ token_row.config = {"auth_type": "oauth", "a": 1}+ token_row.save()++ stored = {+ row.integration_id: row.config+ for row in Integration._base_manager.filter(+ team=self.team,+ integration_id__in=["github_oauth_explicit", "github_null", "github_token", "slack_token"],+ )+ }+ assert stored == {+ "github_oauth_explicit": {},+ "github_null": {},+ "github_token": {"a": 1},+ "slack_token": {"auth_type": "token"},+ }++ def test_unqualified_queries_do_not_touch_non_default_rows(self) -> None:+ updated = Integration.objects.filter(team=self.team, kind="github").update(errors="x")+ deleted, _ = Integration.objects.filter(team=self.team, kind="slack").delete()++ assert (updated, deleted) == (3, 1)+ assert Integration.objects._with_auth(AuthType.TOKEN).filter(team=self.team).count() == 2++ @parameterized.expand(+ [+ ("create", lambda qs, **kw: qs.create(**kw)),+ ("get_or_create", lambda qs, **kw: qs.get_or_create(**kw)[0]),+ ("update_or_create", lambda qs, **kw: qs.update_or_create(**kw)[0]),+ ("bulk_create", lambda qs, **kw: qs.bulk_create([Integration(**kw)])[0]),+ ]+ )+ def test_writes_through_one_non_default_type_store_it(self, _name: str, write: Callable[..., Integration]) -> None:+ row = write(Integration.objects._with_auth(AuthType.TOKEN), team=self.team, kind="github", integration_id="new")++ assert Integration._base_manager.get(team=self.team, integration_id="new").config == {"auth_type": "token"}+ assert row.auth_type == "token"+ assert not Integration.objects.filter(team=self.team, integration_id="new").exists()++ @parameterized.expand([("create",), ("get_or_create",), ("update_or_create",)])+ def test_writes_through_the_default_type_store_nothing(self, method: str) -> None:+ getattr(Integration.objects._with_auth(AuthType.OAUTH), method)(+ team=self.team, kind="github", integration_id="new"+ )++ row = Integration.objects.get(team=self.team, integration_id="new")+ assert (row.config, row.auth_type) == ({}, "oauth")++ def test_update_or_create_keeps_the_auth_type_of_an_existing_row(self) -> None:+ row, created = Integration.objects._with_auth(AuthType.TOKEN).update_or_create(+ team=self.team, kind="github", integration_id="github_token", defaults={"config": {"account": "a"}}+ )++ row.refresh_from_db()+ assert (created, row.config) == (False, {"account": "a", "auth_type": "token"})++ def test_unqualified_create_rejects_a_non_default_type(self) -> None:+ with self.assertRaises(ValueError):+ Integration.objects.create(+ team=self.team, kind="github", integration_id="new", config={"auth_type": "token"}+ )++ @parameterized.expand(+ [+ ("all", lambda: Integration.objects.all()._with_auth(ANY)), # type: ignore[attr-defined]+ ("filter", lambda: Integration.objects.filter(kind="github")._with_auth(AuthType.TOKEN)),+ (+ "chained_auth_types",+ lambda: Integration.objects._with_auth(AuthType.OAUTH)._with_auth(AuthType.TOKEN),+ ),+ ]+ )+ def test_with_auth_after_an_unqualified_query_raises(self, _name: str, build: Callable[[], Any]) -> None:+ with self.assertRaises(ValueError):+ build()++ def test_related_and_base_managers_are_not_filtered(self) -> None:+ token_row = self.rows["github_token"]+ prefetched_team = Team.objects.prefetch_related("integration_set").get(pk=self.team.pk)++ token_row.refresh_from_db()+ assert {row.integration_id for row in self.team.integration_set.all()} == set(ROWS)+ assert {row.integration_id for row in prefetched_team.integration_set.all()} == set(ROWS)+ assert self._names(Integration._base_manager.all()) == set(ROWS)+ assert self._names(self.team.integration_set._with_auth(AuthType.TOKEN)) == {"github_token", "slack_token"}++ def test_the_unfiltered_default_manager_does_not_change_the_migration_state(self) -> None:+ # A default manager that is not named `objects` would need a migration.+ assert (Integration._default_manager.name, ModelState.from_model(Integration).managers) == ("objects", [])+ assert Integration.objects is not Integration._default_manager+ assert self._names(Integration._default_manager._with_auth(AuthType.TOKEN)) == {"github_token", "slack_token"}++ def test_deleting_the_creator_reaches_non_default_rows(self) -> None:+ creator = User.objects.create_and_join(self.organization, "[email protected]", None)+ Integration._base_manager.filter(team=self.team).update(created_by=creator)++ creator.delete()++ assert not Integration._base_manager.filter(team=self.team, created_by__isnull=False).exists()diff --git a/posthog/owners.yaml b/posthog/owners.yamlindex 2de43abd6d05..8217231dd6ba 100644--- a/posthog/owners.yaml+++ b/posthog/owners.yaml@@ -19,6 +19,8 @@ rules:owners: team-web-analytics- match: '/hogql/'owners: team-data-tools+ - match: '/integrations/'+ owners: team-warehouse-sources- match: '/management/migration_analysis/'owners: team-devex# Each line acknowledges a risky migration, so one added line lets that migration skip the check.diff --git a/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr b/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambrindex 256d1841491a..262e99026415 100644--- a/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr+++ b/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr@@ -38,7 +38,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -290,7 +293,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -1196,7 +1202,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.tsindex bc567b1f6bff..fb60fcdbd537 100644--- a/products/desktop/packages/api-client/src/generated.ts+++ b/products/desktop/packages/api-client/src/generated.ts@@ -16391,6 +16391,12 @@ export namespace Schemas {* * `unavailable` - unavailable*/export type InstallationStatusEnum = "connected" | "unavailable";+ /**+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials+ */+ export type IntegrationAuthTypeEnum = "oauth" | "token" | "credentials";/*** * `anthropic` - Anthropic* * `apple-ads` - Apple Ads@@ -16500,6 +16506,14 @@ export namespace Schemas {export type IntegrationConfig = {id: number;kind: IntegrationKindEnum;+ /**+ * How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.+ *+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials+ */+ auth_type: IntegrationAuthTypeEnum & unknown;/*** Standard Integration serializer.*/diff --git a/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr b/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambrindex d6286d5b8ad7..78aef96b3c42 100644--- a/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr+++ b/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr@@ -38,7 +38,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -287,7 +290,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -452,7 +458,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -878,7 +887,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''@@ -1108,7 +1120,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''diff --git a/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr b/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambrindex a6d93d281e93..e95941f512d8 100644--- a/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr+++ b/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr@@ -380,7 +380,10 @@"posthog_integration"."created_at","posthog_integration"."created_by_id"FROM "posthog_integration"- WHERE ("posthog_integration"."kind" = 'vercel'+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')+ AND "posthog_integration"."kind" = 'vercel'AND "posthog_integration"."team_id" = 99999)LIMIT 21'''diff --git a/products/integrations/frontend/generated/api.schemas.ts b/products/integrations/frontend/generated/api.schemas.tsindex e07e9a291b51..713480dfb364 100644--- a/products/integrations/frontend/generated/api.schemas.ts+++ b/products/integrations/frontend/generated/api.schemas.ts@@ -257,6 +257,19 @@ export const IntegrationKindEnumApi = {YoutubeAnalytics: 'youtube-analytics',} as const+/**+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials+ */+export type IntegrationAuthTypeEnumApi = (typeof IntegrationAuthTypeEnumApi)[keyof typeof IntegrationAuthTypeEnumApi]++export const IntegrationAuthTypeEnumApi = {+ Oauth: 'oauth',+ Token: 'token',+ Credentials: 'credentials',+} as const+/*** * `connected` - connected* * `unavailable` - unavailable@@ -274,6 +287,12 @@ export const InstallationStatusEnumApi = {export interface IntegrationConfigApi {readonly id: numberkind: IntegrationKindEnumApi+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.+ *+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials */+ readonly auth_type: IntegrationAuthTypeEnumApiconfig?: unknownreadonly created_at: stringreadonly created_by: UserBasicApi@@ -332,6 +351,12 @@ export interface SlackChannelsResponseApi {export interface PatchedIntegrationConfigApi {readonly id?: numberkind?: IntegrationKindEnumApi+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.+ *+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials */+ readonly auth_type?: IntegrationAuthTypeEnumApiconfig?: unknownreadonly created_at?: stringreadonly created_by?: UserBasicApidiff --git a/products/warehouse_sources/backend/apps.py b/products/warehouse_sources/backend/apps.pyindex 7f1ae3c23c44..2854587ea072 100644--- a/products/warehouse_sources/backend/apps.py+++ b/products/warehouse_sources/backend/apps.py@@ -20,3 +20,13 @@ class WarehouseSourcesConfig(AppConfig):from products.warehouse_sources.backend.sync_alerts.registration import register # noqa: PLC0415register()++ # Source kinds have no generated accessor. Callers reach their rows through `IntegrationManager.source(kind)`.+ # The SDK imports the Integration model, which needs the app registry.+ from posthog.integrations import AuthType, register_source_kinds # noqa: PLC0415++ from products.warehouse_sources.backend.facade.credential_auth_types import ( # noqa: PLC0415+ CREDENTIAL_AUTH_TYPES,+ )++ register_source_kinds({kind: AuthType(auth_type) for kind, auth_type in CREDENTIAL_AUTH_TYPES.items()})diff --git a/products/warehouse_sources/backend/facade/credential_auth_types.py b/products/warehouse_sources/backend/facade/credential_auth_types.pynew file mode 100644index 000000000000..c66cf7a1ab73--- /dev/null+++ b/products/warehouse_sources/backend/facade/credential_auth_types.py@@ -0,0 +1,1425 @@+"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.++The default auth type of each warehouse source credential kind, keyed by kind. A row stores its auth+type only when it is not the default, so a change to an existing value needs a data migration. See+`sources/common/credential_auth_type.py` for the derivation. A kind in the integrations catalogue, such+as `stripe`, keeps the default of the catalogue. The file is append-only: a removed kind keeps its+entry.+"""++from collections.abc import Mapping++CREDENTIAL_AUTH_TYPES: Mapping[str, str] = {+ "ab-tasty": "credentials",+ "ably": "token",+ "abnormal-security": "credentials",+ "acast": "token",+ "acculynx": "token",+ "actionstep": "credentials",+ "active-campaign": "credentials",+ "acuity-scheduling": "credentials",+ "ad-roll": "credentials",+ "adapty": "token",+ "adjust": "token",+ "adobe-analytics": "credentials",+ "adobe-commerce": "credentials",+ "adp-workforce-now": "credentials",+ "adyen": "credentials",+ "aftership": "token",+ "agile-crm": "credentials",+ "aha": "credentials",+ "aha-ideas": "credentials",+ "ahrefs": "credentials",+ "aikido-security": "credentials",+ "air-ops": "token",+ "airbrake": "token",+ "airbridge": "credentials",+ "airbyte": "credentials",+ "aircall": "credentials",+ "airship": "credentials",+ "airtable": "token",+ "airwallex": "credentials",+ "aiven": "token",+ "akamai-reporting": "credentials",+ "akeneo": "credentials",+ "alation": "credentials",+ "alegra": "credentials",+ "algolia": "credentials",+ "alguna": "token",+ "allegro": "credentials",+ "alpaca-broker-api": "credentials",+ "alpha-vantage": "token",+ "amazon-ads": "credentials",+ "amazon-cloud-watch": "credentials",+ "amazon-event-bridge": "credentials",+ "amazon-kinesis": "credentials",+ "amazon-selling-partner": "credentials",+ "amazon-sns": "credentials",+ "amazon-sqs": "credentials",+ "amplemarket": "token",+ "amplitude": "credentials",+ "anodot-cost": "credentials",+ "anomalo": "credentials",+ "anthropic": "token",+ "anvil": "token",+ "apaleo": "credentials",+ "apify-dataset": "credentials",+ "apitally": "token",+ "apollo": "token",+ "app-lovin": "credentials",+ "app-store-connect": "credentials",+ "appcues": "credentials",+ "appdirect": "credentials",+ "appdynamics": "credentials",+ "appfigures": "token",+ "appfolio": "credentials",+ "appfollow": "token",+ "apple-ads": "credentials",+ "apps-flyer": "credentials",+ "appsignal": "credentials",+ "appstack": "token",+ "apptivo": "credentials",+ "appwrite": "credentials",+ "arcade": "credentials",+ "argocd": "credentials",+ "arxiv": "credentials",+ "asaas": "credentials",+ "asana": "token",+ "ashby": "token",+ "asknicely": "credentials",+ "assembly-ai": "credentials",+ "astronomer": "credentials",+ "athenahealth": "credentials",+ "atlan": "credentials",+ "attentive": "token",+ "attio": "token",+ "audio-go": "credentials",+ "auth0": "credentials",+ "autodesk-construction-cloud": "credentials",+ "automox": "credentials",+ "autumn": "token",+ "avalara": "credentials",+ "aviationstack": "credentials",+ "aviator": "token",+ "awin": "credentials",+ "aws-athena": "credentials",+ "aws-batch": "credentials",+ "aws-budgets": "credentials",+ "aws-cloud-trail": "credentials",+ "aws-cloudformation": "credentials",+ "aws-compute-optimizer": "credentials",+ "aws-config": "credentials",+ "aws-connect": "credentials",+ "aws-cost-and-usage-report": "credentials",+ "aws-cost-anomaly-detection": "credentials",+ "aws-cost-explorer": "credentials",+ "aws-documentdb": "credentials",+ "aws-glue-data-catalog": "credentials",+ "aws-guardduty": "credentials",+ "aws-health": "credentials",+ "aws-iam-access-analyzer": "credentials",+ "aws-inspector": "credentials",+ "aws-macie": "credentials",+ "aws-organizations": "credentials",+ "aws-rds-performance-insights": "credentials",+ "aws-redshift": "credentials",+ "aws-s3": "credentials",+ "aws-s3-tables": "credentials",+ "aws-sagemaker": "credentials",+ "aws-savings-plans": "credentials",+ "aws-security-hub": "credentials",+ "aws-ses": "credentials",+ "aws-step-functions": "credentials",+ "aws-support": "credentials",+ "aws-systems-manager": "credentials",+ "aws-trusted-advisor": "credentials",+ "aws-waf": "credentials",+ "aws-xray": "credentials",+ "axiom": "credentials",+ "azure-activity-log": "credentials",+ "azure-advisor": "credentials",+ "azure-api-management": "credentials",+ "azure-application-insights": "credentials",+ "azure-blob": "credentials",+ "azure-cost-management": "credentials",+ "azure-data-explorer": "credentials",+ "azure-data-factory": "credentials",+ "azure-dev-ops": "credentials",+ "azure-log-analytics": "credentials",+ "azure-monitor-alerts": "credentials",+ "azure-monitor-metrics": "credentials",+ "azure-openai-usage": "credentials",+ "azure-policy-insights": "credentials",+ "azure-reservations": "credentials",+ "azure-resource-graph": "credentials",+ "azure-resource-health": "credentials",+ "azure-service-health": "credentials",+ "azure-synapse": "credentials",+ "azure-table-storage": "credentials",+ "babelforce": "credentials",+ "back-market": "token",+ "backblaze": "credentials",+ "bamboo-hr": "credentials",+ "basecamp": "credentials",+ "baserow": "credentials",+ "baseten": "token",+ "bcms": "credentials",+ "beacons": "credentials",+ "beamer": "token",+ "beehiiv": "credentials",+ "better-stack": "token",+ "bettermode": "credentials",+ "bexio": "credentials",+ "big-commerce": "credentials",+ "big-mailer": "token",+ "bigeye": "credentials",+ "bill-com": "credentials",+ "billit": "credentials",+ "billomat": "credentials",+ "bing-ads": "oauth",+ "bing-webmaster-tools": "credentials",+ "bitbucket": "credentials",+ "bitly": "credentials",+ "bitrise": "token",+ "bitwarden": "credentials",+ "blackbaud-raisers-edge-nxt": "credentials",+ "blackboard-learn": "credentials",+ "bland-ai": "token",+ "bling": "credentials",+ "blogger": "credentials",+ "bloomerang": "token",+ "blueshift": "credentials",+ "bluesky": "credentials",+ "bluetally": "credentials",+ "bol-retailer": "credentials",+ "bold-sign": "credentials",+ "boulevard": "credentials",+ "box": "credentials",+ "braintree": "credentials",+ "braintrust": "credentials",+ "branch": "credentials",+ "braze": "credentials",+ "breezometer": "token",+ "breezy-hr": "credentials",+ "brevo": "token",+ "brex": "token",+ "browse-ai": "token",+ "browser-use": "token",+ "browserbase": "token",+ "buffer": "credentials",+ "bugherd": "token",+ "bugsnag": "token",+ "build-better": "token",+ "buildium": "credentials",+ "buildkite": "credentials",+ "bunny": "token",+ "buttondown": "token",+ "buy-me-a-coffee": "token",+ "buzzsprout": "credentials",+ "cal-com": "credentials",+ "calendarific": "token",+ "calendly": "token",+ "calibre": "credentials",+ "call-rail": "credentials",+ "campaign-manager360": "credentials",+ "campaign-monitor": "credentials",+ "campayn": "credentials",+ "campfire": "token",+ "canny": "token",+ "canvas-lms": "credentials",+ "capsule-crm": "token",+ "captain-data": "credentials",+ "capterra": "credentials",+ "captivate": "credentials",+ "care-quality-commission": "credentials",+ "cart-com": "credentials",+ "cashfree": "credentials",+ "cast-ai": "token",+ "castor-edc": "credentials",+ "catchpoint": "token",+ "cdc-open-data": "token",+ "census": "credentials",+ "chameleon": "token",+ "channel-io": "credentials",+ "chargeback-stop": "credentials",+ "chargebee": "credentials",+ "chargedesk": "token",+ "chargeflow": "credentials",+ "chargify": "credentials",+ "chart-hop": "credentials",+ "chart-mogul": "token",+ "chatwoot": "credentials",+ "check-hq": "credentials",+ "checkly": "credentials",+ "checkmarx": "credentials",+ "checkout-com": "credentials",+ "chess-com": "credentials",+ "chift": "credentials",+ "chorus": "credentials",+ "churnkey": "credentials",+ "cimis": "token",+ "cin7": "credentials",+ "circle-ci": "credentials",+ "circle-so": "credentials",+ "circleci-insights": "credentials",+ "cisco-duo": "credentials",+ "cisco-meraki": "credentials",+ "clari": "credentials",+ "clarifai": "credentials",+ "clarify": "credentials",+ "classy": "credentials",+ "clay": "token",+ "clazar": "credentials",+ "cleartax": "credentials",+ "clerk": "token",+ "clever": "token",+ "clevertap": "credentials",+ "clickhouse": "credentials",+ "clickhouse-cloud": "credentials",+ "clickup": "credentials",+ "cliniko": "token",+ "clio": "credentials",+ "clip": "credentials",+ "clockify": "token",+ "clockodo": "credentials",+ "close": "token",+ "cloudability": "credentials",+ "cloudbeds": "credentials",+ "cloudflare": "token",+ "cloudflare-d1": "credentials",+ "cloudflare-r2": "credentials",+ "cloudinary": "credentials",+ "cloudsmith": "credentials",+ "cloudzero": "token",+ "clover": "credentials",+ "coassemble": "credentials",+ "cockroachdb": "credentials",+ "coda": "token",+ "codacy": "credentials",+ "codecov": "credentials",+ "codefresh": "token",+ "codemagic": "token",+ "codescene": "credentials",+ "cody": "credentials",+ "cohere": "token",+ "coin-api": "token",+ "coin-gecko": "credentials",+ "coin-market-cap": "token",+ "collibra": "credentials",+ "commercetools": "credentials",+ "commission-junction": "credentials",+ "commslayer": "credentials",+ "companycam": "token",+ "concord": "credentials",+ "conecto-crm": "credentials",+ "conekta": "token",+ "config-cat": "credentials",+ "confluence": "credentials",+ "confluent-cloud": "credentials",+ "constant-contact": "credentials",+ "conta-azul": "credentials",+ "contentsquare": "credentials",+ "convert-kit": "token",+ "convex": "credentials",+ "convonite": "credentials",+ "coolify": "credentials",+ "copper": "credentials",+ "coralogix": "credentials",+ "cortex": "token",+ "cosmosdb": "credentials",+ "couchbase": "credentials",+ "coupa": "credentials",+ "courier": "token",+ "coveralls": "credentials",+ "crates-io": "credentials",+ "creem": "credentials",+ "crisp": "credentials",+ "criteo": "credentials",+ "cronitor": "token",+ "crossref": "credentials",+ "crowdstrike-falcon": "credentials",+ "crunchbase": "token",+ "cube-cloud": "credentials",+ "culture-amp": "credentials",+ "cursor": "token",+ "curve": "credentials",+ "customer-io": "credentials",+ "customerly": "token",+ "cybersource": "credentials",+ "d2l-brightspace": "credentials",+ "dagster-cloud": "credentials",+ "data-for-seo": "credentials",+ "databricks": "credentials",+ "datadog": "credentials",+ "datahub": "credentials",+ "datascope": "credentials",+ "dato-cms": "credentials",+ "datorama": "credentials",+ "dayforce": "credentials",+ "db2": "credentials",+ "dbt": "credentials",+ "debugbear": "token",+ "decagon": "token",+ "deel": "token",+ "deel-flows": "credentials",+ "deepgram": "token",+ "deepsource": "credentials",+ "demodesk": "token",+ "deno-deploy": "token",+ "depot": "token",+ "deputy": "credentials",+ "descope": "credentials",+ "develocity": "credentials",+ "devin-ai": "credentials",+ "dialpad": "credentials",+ "digital-ocean": "token",+ "ding-connect": "token",+ "directus": "credentials",+ "discord": "credentials",+ "discourse": "credentials",+ "display-video360": "credentials",+ "dixa": "token",+ "do-it": "token",+ "dockerhub": "credentials",+ "docuseal": "credentials",+ "docusign": "credentials",+ "dodo-payments": "credentials",+ "dokploy": "credentials",+ "dolibarr": "credentials",+ "donorbox": "credentials",+ "doorloop": "credentials",+ "doppler": "token",+ "dovetail": "token",+ "dragonboat": "credentials",+ "drata": "credentials",+ "drchrono": "credentials",+ "dreamdata": "credentials",+ "dremio": "credentials",+ "drip": "credentials",+ "dripify": "credentials",+ "dropbox": "credentials",+ "dropbox-sign": "token",+ "dub": "token",+ "dubsado": "credentials",+ "duck-lake": "credentials",+ "dwolla": "credentials",+ "dynamics365": "credentials",+ "dynamics365-business-central": "credentials",+ "dynamodb": "credentials",+ "dynatrace": "credentials",+ "e-conomic": "credentials",+ "e2b": "credentials",+ "easybill": "token",+ "easypost": "token",+ "easypromos": "token",+ "ebay": "credentials",+ "ecb-data-portal": "credentials",+ "elasticemail": "token",+ "elasticsearch": "credentials",+ "electricity-maps": "token",+ "eleven-labs": "token",+ "eloqua": "credentials",+ "email-octopus": "token",+ "emarsys": "credentials",+ "embrace": "credentials",+ "employment-hero": "credentials",+ "encharge": "credentials",+ "entsoe": "credentials",+ "env0": "credentials",+ "eppo": "token",+ "etsy": "credentials",+ "eurostat": "credentials",+ "eventbrite": "token",+ "eventee": "token",+ "eventzilla": "token",+ "everflow": "credentials",+ "everhour": "token",+ "exact-online": "credentials",+ "exchange-rates-api": "token",+ "expensify": "credentials",+ "expo": "credentials",+ "ez-office-inventory": "credentials",+ "facebook-pages": "credentials",+ "factorial": "token",+ "faire": "token",+ "faros-ai": "credentials",+ "fastbill": "credentials",+ "fastly": "token",+ "fauna": "credentials",+ "featurebase": "token",+ "feishu": "credentials",+ "fieldpulse": "token",+ "fieldwire": "credentials",+ "file-upload": "credentials",+ "filevine": "credentials",+ "fillout": "credentials",+ "finage": "token",+ "financial-modelling": "token",+ "finnhub": "token",+ "finnworlds": "token",+ "finout": "credentials",+ "fintoc": "credentials",+ "fire-hydrant": "credentials",+ "firebase": "credentials",+ "firebolt": "credentials",+ "firecrawl": "token",+ "fireworks-ai": "credentials",+ "first-promoter": "credentials",+ "five9": "credentials",+ "flagsmith": "credentials",+ "fleetio": "credentials",+ "flexera-cloud-cost": "credentials",+ "flexmail": "credentials",+ "flexport": "credentials",+ "float-app": "token",+ "flowlu": "credentials",+ "flutterwave": "credentials",+ "fly-io": "credentials",+ "folk": "token",+ "formbricks": "credentials",+ "fortnox": "credentials",+ "fourthwall": "credentials",+ "framer": "credentials",+ "fred": "token",+ "free-agent": "credentials",+ "freightview": "credentials",+ "fresh-books": "credentials",+ "freshcaller": "credentials",+ "freshchat": "credentials",+ "freshdesk": "credentials",+ "freshsales": "credentials",+ "freshservice": "credentials",+ "frill": "token",+ "front": "token",+ "frontegg": "credentials",+ "fulcrum": "token",+ "full-story": "token",+ "fusion-auth": "credentials",+ "g2": "credentials",+ "gainsight-cs": "credentials",+ "gainsight-px": "credentials",+ "gcore": "token",+ "gcp-apigee": "credentials",+ "gcp-artifact-registry": "credentials",+ "gcp-bigtable": "credentials",+ "gcp-chronicle": "credentials",+ "gcp-cloud-asset-inventory": "credentials",+ "gcp-cloud-billing": "credentials",+ "gcp-cloud-build": "credentials",+ "gcp-cloud-deploy": "credentials",+ "gcp-cloud-dns": "credentials",+ "gcp-cloud-functions": "credentials",+ "gcp-cloud-logging": "credentials",+ "gcp-cloud-monitoring": "credentials",+ "gcp-cloud-run": "credentials",+ "gcp-cloud-spanner": "credentials",+ "gcp-cloud-sql": "credentials",+ "gcp-cloud-trace": "credentials",+ "gcp-cloud-workflows": "credentials",+ "gcp-compute-engine": "credentials",+ "gcp-container-analysis": "credentials",+ "gcp-dataflow": "credentials",+ "gcp-dataplex": "credentials",+ "gcp-dataproc": "credentials",+ "gcp-error-reporting": "credentials",+ "gcp-gke": "credentials",+ "gcp-pubsub": "credentials",+ "gcp-recaptcha-enterprise": "credentials",+ "gcp-recommender": "credentials",+ "gcp-security-command-center": "credentials",+ "gdelt": "credentials",+ "gem": "credentials",+ "genesys-cloud": "credentials",+ "gerrit": "credentials",+ "get-stream": "credentials",+ "getdx": "token",+ "ghost": "credentials",+ "giphy": "token",+ "git-book": "token",+ "gitea": "credentials",+ "gitguardian": "credentials",+ "github": "token",+ "gitlab": "credentials",+ "givebutter": "token",+ "gladly": "credentials",+ "glassfrog": "token",+ "gleif": "credentials",+ "gmail": "credentials",+ "gnews": "token",+ "go-cardless": "credentials",+ "go-login": "token",+ "gojiberry": "credentials",+ "goldcast": "token",+ "gong": "credentials",+ "google-ad-manager": "credentials",+ "google-ad-mob": "credentials",+ "google-ad-sense": "credentials",+ "google-ads": "oauth",+ "google-analytics": "oauth",+ "google-business-profile": "credentials",+ "google-calendar": "credentials",+ "google-chat": "credentials",+ "google-classroom": "credentials",+ "google-cloud-service-account": "credentials",+ "google-cloud-storage": "credentials",+ "google-directory": "credentials",+ "google-drive": "credentials",+ "google-forms": "credentials",+ "google-merchant-center": "credentials",+ "google-page-speed-insights": "token",+ "google-play-console": "credentials",+ "google-postmaster-tools": "credentials",+ "google-search-console": "oauth",+ "google-sheets": "credentials",+ "google-tasks": "credentials",+ "google-webfonts": "token",+ "google-workspace-admin-reports": "credentials",+ "gorgias": "credentials",+ "grafana": "credentials",+ "grafana-irm": "credentials",+ "granola": "token",+ "greenhouse": "credentials",+ "greyt-hr": "credentials",+ "gridly": "credentials",+ "groq": "token",+ "growi": "credentials",+ "growth-book": "credentials",+ "guardian": "token",+ "guesty": "credentials",+ "gumloop": "credentials",+ "gumroad": "token",+ "guru": "credentials",+ "gusto": "credentials",+ "harness": "credentials",+ "harness-ccm": "credentials",+ "harness-sei": "credentials",+ "harvest": "credentials",+ "harvey": "credentials",+ "hatchet": "credentials",+ "healthchecks": "credentials",+ "healthie": "credentials",+ "heap": "credentials",+ "height": "token",+ "helicone": "credentials",+ "hellobaton": "credentials",+ "helpscout": "oauth",+ "heroku": "token",+ "hetzner": "token",+ "hex": "credentials",+ "hey-gen": "token",+ "hey-reach": "token",+ "hi-bob": "credentials",+ "high-level": "credentials",+ "hightouch": "token",+ "hitpay": "credentials",+ "hivebrite": "credentials",+ "holded": "credentials",+ "honeybadger": "token",+ "honeycomb": "credentials",+ "hookdeck": "token",+ "hooray-hr": "token",+ "hootsuite": "credentials",+ "hostaway": "credentials",+ "housecall-pro": "token",+ "hubplanner": "token",+ "hubspot": "oauth",+ "hugging-face": "credentials",+ "humanitec": "credentials",+ "humanitix": "token",+ "huntr": "token",+ "hyperspell": "credentials",+ "hyros": "token",+ "ikas": "credentials",+ "illumina-basespace": "credentials",+ "imagga": "credentials",+ "imf-data": "credentials",+ "impact": "credentials",+ "impact-partner": "credentials",+ "imperva": "credentials",+ "incident-io": "token",+ "infisical": "credentials",+ "inflowinventory": "credentials",+ "influxdb-cloud": "credentials",+ "infor-nexus": "credentials",+ "inngest": "credentials",+ "insightful": "credentials",+ "insightly": "credentials",+ "instagram": "oauth",+ "instana": "credentials",+ "instantly": "token",+ "instatus": "token",+ "intercom": "oauth",+ "interzoid": "credentials",+ "inth": "credentials",+ "intruder": "token",+ "invoiced": "token",+ "invoiceninja": "credentials",+ "ip2-whois": "token",+ "iron-source-ads": "credentials",+ "iterable": "credentials",+ "iyzico": "credentials",+ "jamf-pro": "credentials",+ "jellyfish": "token",+ "jenkins": "credentials",+ "jfrog-artifactory": "credentials",+ "jira": "credentials",+ "job-nimbus": "token",+ "jobber": "credentials",+ "jobtread": "credentials",+ "jotform": "credentials",+ "judge-me-reviews": "credentials",+ "jumpcloud": "credentials",+ "just-call": "credentials",+ "just-sift": "token",+ "k6-cloud": "credentials",+ "kafka": "credentials",+ "kajabi": "credentials",+ "kalshi": "credentials",+ "kameleoon": "credentials",+ "kandji": "credentials",+ "kapa-ai": "credentials",+ "katana": "token",+ "kaufland-marketplace": "credentials",+ "keka": "credentials",+ "kernel": "token",+ "kestra": "credentials",+ "kick": "credentials",+ "kickscale": "credentials",+ "kickstarter": "credentials",+ "kinde": "credentials",+ "kion": "credentials",+ "kisi": "token",+ "kissmetrics": "credentials",+ "klar": "credentials",+ "klarna": "credentials",+ "klaus": "credentials",+ "klaviyo": "token",+ "knock": "token",+ "knowbe4": "credentials",+ "kommo": "credentials",+ "komodor": "credentials",+ "kong-konnect": "credentials",+ "koyeb": "token",+ "kubecost": "credentials",+ "kustomer": "credentials",+ "kyve": "credentials",+ "labelbox": "credentials",+ "lacework": "credentials",+ "lago": "credentials",+ "lambda-labs": "token",+ "lang-smith": "credentials",+ "langfuse": "credentials",+ "laravel-cloud": "credentials",+ "latitude": "credentials",+ "lattice": "credentials",+ "launch-darkly": "token",+ "lawmatics": "credentials",+ "leadfeeder": "token",+ "leantime": "credentials",+ "learnworlds": "credentials",+ "ledyer": "credentials",+ "leexi": "credentials",+ "lemlist": "token",+ "lemon-squeezy": "token",+ "less-annoying-crm": "token",+ "lettr-labs": "credentials",+ "lever": "token",+ "lexware-office": "token",+ "liana": "credentials",+ "lightdash": "credentials",+ "lightfield": "token",+ "lightspeed-retail": "credentials",+ "linear": "oauth",+ "linearb": "token",+ "lingo-dev": "token",+ "linkedin-ads": "oauth",+ "linkedin-pages": "credentials",+ "linkrunner": "token",+ "linnworks": "credentials",+ "linode": "token",+ "listmonk": "credentials",+ "lite-llm": "credentials",+ "liveblocks": "credentials",+ "livestorm": "token",+ "llama-cloud": "credentials",+ "lob": "token",+ "lodgify": "token",+ "logicmonitor": "credentials",+ "logrocket": "credentials",+ "logz-io": "credentials",+ "lokalise": "token",+ "looker": "credentials",+ "loom": "credentials",+ "loop-returns": "token",+ "loops": "token",+ "lovable": "token",+ "luma": "token",+ "m3ter": "credentials",+ "mailchimp": "token",+ "mailer-lite": "token",+ "mailer-send": "token",+ "mailgun": "credentials",+ "mailjet": "credentials",+ "mailosaur": "token",+ "mailtrap": "token",+ "mantle": "credentials",+ "manychat": "credentials",+ "marketo": "credentials",+ "marketstack": "token",+ "mastodon": "credentials",+ "matomo": "credentials",+ "maxio": "credentials",+ "medusa": "credentials",+ "meetup": "credentials",+ "megaphone": "credentials",+ "meltwater": "credentials",+ "mem0": "credentials",+ "memberful": "credentials",+ "membrain": "credentials",+ "mendeley": "credentials",+ "mention": "token",+ "mercado-ads": "credentials",+ "mercado-pago": "token",+ "mercury": "token",+ "merge": "credentials",+ "meta-ads": "oauth",+ "metabase": "credentials",+ "metaplane": "token",+ "meteostat": "token",+ "metorial": "token",+ "metricool": "credentials",+ "metriport": "credentials",+ "metronome": "token",+ "mews": "credentials",+ "mezmo": "credentials",+ "microsoft-advertising": "credentials",+ "microsoft-clarity": "token",+ "microsoft-dataverse": "credentials",+ "microsoft-defender-cloud-apps": "credentials",+ "microsoft-defender-endpoint": "credentials",+ "microsoft-defender-for-cloud": "credentials",+ "microsoft-entra-id": "credentials",+ "microsoft-excel": "credentials",+ "microsoft-intune": "credentials",+ "microsoft-lists": "credentials",+ "microsoft-purview": "credentials",+ "microsoft-purview-audit": "credentials",+ "microsoft-sentinel": "credentials",+ "microsoft-teams": "credentials",+ "microsoft-teams-call-records": "credentials",+ "microsoft365-usage-reports": "credentials",+ "midtrans": "credentials",+ "mighty-networks": "credentials",+ "mindbody": "credentials",+ "mintlify": "credentials",+ "mirakl": "credentials",+ "miro": "credentials",+ "missive": "credentials",+ "mistral-ai": "token",+ "mix-max": "token",+ "mixpanel": "credentials",+ "mo-engage": "credentials",+ "modal": "credentials",+ "mode": "credentials",+ "moesif": "credentials",+ "mollie": "token",+ "monaco": "credentials",+ "monday": "token",+ "moneybird": "credentials",+ "mongodb": "credentials",+ "mono": "token",+ "monte-carlo": "credentials",+ "moodle": "credentials",+ "motherduck": "credentials",+ "motion": "token",+ "moxie": "credentials",+ "msg91": "credentials",+ "mssql": "credentials",+ "mux": "credentials",+ "my-hours": "token",+ "mycase": "credentials",+ "mysql": "credentials",+ "n8n": "credentials",+ "nager-date": "credentials",+ "nango": "credentials",+ "nasa": "credentials",+ "nation-builder": "credentials",+ "navan": "credentials",+ "naver-ads": "credentials",+ "nebius-ai": "token",+ "neo4j": "credentials",+ "neon": "credentials",+ "neon-crm": "credentials",+ "net-suite": "credentials",+ "netlify": "token",+ "new-relic": "credentials",+ "new-york-times": "token",+ "news-api": "token",+ "news-data": "token",+ "nexhealth": "credentials",+ "nexiopay": "credentials",+ "nextdoor-ads": "credentials",+ "ninja-one-rmm": "credentials",+ "no-crm": "credentials",+ "noaa-cdo": "token",+ "nobl9": "credentials",+ "nolt": "credentials",+ "nops": "credentials",+ "northbeam": "credentials",+ "northflank": "token",+ "northpass-lms": "token",+ "notion": "credentials",+ "npm-registry": "credentials",+ "nuget": "credentials",+ "nuntly": "token",+ "nutshell": "credentials",+ "nylas": "credentials",+ "octolens": "token",+ "octopus-deploy": "credentials",+ "odoo": "credentials",+ "oecd": "credentials",+ "okendo": "credentials",+ "okta": "credentials",+ "omni": "credentials",+ "omnisend": "token",+ "oncehub": "token",+ "one-drive": "credentials",+ "one-hundred-ms": "credentials",+ "one-password": "credentials",+ "one-signal": "credentials",+ "oneleet": "credentials",+ "onelogin": "credentials",+ "onepagecrm": "credentials",+ "onfleet": "token",+ "open-ai": "credentials",+ "open-ai-ads": "token",+ "open-aq": "token",+ "open-data-dc": "credentials",+ "open-dental": "credentials",+ "open-exchange-rates": "token",+ "open-fda": "token",+ "open-meteo": "token",+ "open-router": "token",+ "open-weather": "token",+ "openalex": "token",+ "opencorporates": "token",+ "openfec": "credentials",+ "opinion-stage": "token",+ "opn-payments": "token",+ "opsgenie": "credentials",+ "opslevel": "credentials",+ "optimizely": "token",+ "opus-watch": "token",+ "oracle": "credentials",+ "oracle-ebs": "credentials",+ "oracle-fusion": "credentials",+ "orb": "token",+ "orbit": "credentials",+ "orca-security": "credentials",+ "ortto": "credentials",+ "otto-market": "credentials",+ "oura": "token",+ "outbrain": "credentials",+ "outlook": "credentials",+ "outreach": "credentials",+ "oveit": "credentials",+ "ownerrez": "credentials",+ "pabbly-subscriptions-billing": "credentials",+ "packagist": "credentials",+ "paddle": "token",+ "pagbank": "credentials",+ "pager-duty": "token",+ "panda-doc": "token",+ "paperform": "token",+ "papersign": "token",+ "pardot": "credentials",+ "particle": "credentials",+ "partner-stack": "credentials",+ "partnerize": "credentials",+ "patreon": "credentials",+ "pax8": "credentials",+ "pay-fit": "token",+ "pay-pal": "credentials",+ "paychex": "credentials",+ "paylocity": "credentials",+ "paymob": "credentials",+ "paymongo": "token",+ "paystack": "token",+ "peec-ai": "credentials",+ "pendo": "credentials",+ "pennylane": "credentials",+ "perigon": "token",+ "perk": "credentials",+ "persist-iq": "token",+ "persona": "token",+ "personio": "credentials",+ "pexels": "token",+ "pg-analyze": "credentials",+ "phonepe": "credentials",+ "phyllo": "credentials",+ "picqer": "credentials",+ "pike13": "credentials",+ "pinecone": "token",+ "pingdom": "token",+ "pingone": "credentials",+ "pinterest-ads": "oauth",+ "pinterest-organic": "credentials",+ "pipedrive": "credentials",+ "pipeliner": "credentials",+ "pivotal-tracker": "credentials",+ "piwik": "credentials",+ "plaid": "credentials",+ "plain": "token",+ "plane": "credentials",+ "planetscale-mysql": "credentials",+ "planetscale-postgres": "credentials",+ "planhat": "token",+ "planning-center": "credentials",+ "platform-sh": "credentials",+ "plausible": "credentials",+ "plivo": "credentials",+ "plunk": "credentials",+ "pluralsight-flow": "credentials",+ "pocket": "credentials",+ "podbean": "credentials",+ "podium": "credentials",+ "polar": "token",+ "polygon": "credentials",+ "polymarket": "credentials",+ "poplar": "token",+ "post-nord": "credentials",+ "postgresql": "credentials",+ "postmark": "token",+ "postscript": "token",+ "power-bi-admin": "credentials",+ "practicepanther": "credentials",+ "prefect-cloud": "credentials",+ "preset": "credentials",+ "presta-shop": "credentials",+ "pretix": "credentials",+ "primetric": "credentials",+ "printavo": "credentials",+ "printify": "token",+ "procore": "credentials",+ "productboard": "token",+ "productiv": "credentials",+ "productive": "credentials",+ "profound": "token",+ "prompt-watch": "credentials",+ "prompting-company": "credentials",+ "proofpoint-tap": "credentials",+ "propertyware": "credentials",+ "pubnub": "credentials",+ "pulumi-cloud": "credentials",+ "pylon": "token",+ "pypi": "credentials",+ "qdrant": "credentials",+ "qonto": "credentials",+ "qualaroo": "credentials",+ "qualtrics": "credentials",+ "qualys-vmdr": "credentials",+ "quay": "credentials",+ "quick-books": "credentials",+ "quo": "token",+ "railway": "token",+ "railz": "credentials",+ "rainforest-pay": "credentials",+ "raisely": "credentials",+ "raken": "credentials",+ "rakuten-advertising": "credentials",+ "ramp": "credentials",+ "rapid7-insightvm": "credentials",+ "raygun": "token",+ "razorpay": "credentials",+ "rb2b": "credentials",+ "rd-station-marketing": "credentials",+ "recall-ai": "credentials",+ "recharge": "token",+ "recreation": "token",+ "recruitee": "credentials",+ "recurly": "credentials",+ "reddit": "credentials",+ "reddit-ads": "oauth",+ "redis": "credentials",+ "redpanda-cloud": "credentials",+ "referral-hero": "token",+ "render": "credentials",+ "rent-cast": "token",+ "rent-manager": "credentials",+ "repairshopr": "credentials",+ "replicate": "token",+ "reply-io": "token",+ "resend": "token",+ "retail-express": "credentials",+ "retell-ai": "token",+ "retently": "token",+ "retreaver": "credentials",+ "revenue-cat": "credentials",+ "reverb": "token",+ "revolut-merchant": "credentials",+ "rewardful": "token",+ "ring-central": "credentials",+ "rippling": "token",+ "rki-covid": "credentials",+ "roark": "token",+ "rocket-chat": "credentials",+ "rocket-matter": "credentials",+ "rocketlane": "token",+ "rokt-ads": "credentials",+ "rollbar": "token",+ "rootly": "token",+ "rss": "token",+ "rubygems": "credentials",+ "rudder-stack": "credentials",+ "ruddr": "token",+ "run-pod": "token",+ "safety-culture": "token",+ "sage-hr": "credentials",+ "sage-intacct": "credentials",+ "sailthru": "credentials",+ "sales-loft": "token",+ "salesflare": "token",+ "salesforce": "oauth",+ "salesforce-marketing-cloud": "credentials",+ "salestrics": "credentials",+ "sam-cart": "credentials",+ "sanity": "credentials",+ "sap-concur": "credentials",+ "sap-erp": "credentials",+ "sap-fieldglass": "credentials",+ "sap-hana": "credentials",+ "sap-success-factors": "credentials",+ "savvy-cal": "token",+ "scale-ai": "token",+ "scaleway": "credentials",+ "scalr": "credentials",+ "scarf": "credentials",+ "schematic": "token",+ "scrunch": "token",+ "search-ads360": "credentials",+ "sec-edgar": "credentials",+ "secoda": "token",+ "secureframe": "credentials",+ "segment": "credentials",+ "select-star": "token",+ "semantic-scholar": "token",+ "semaphore": "credentials",+ "semgrep": "token",+ "semrush": "credentials",+ "send-grid": "token",+ "send-pulse": "credentials",+ "sendowl": "credentials",+ "senseforce": "credentials",+ "sentinelone": "credentials",+ "sentry": "credentials",+ "sequenzy": "credentials",+ "serpstat": "credentials",+ "service-fusion": "credentials",+ "service-now": "credentials",+ "servicem8": "token",+ "servicetitan": "credentials",+ "servicetrade": "credentials",+ "sevalla": "credentials",+ "sevdesk": "token",+ "seven-shifts": "credentials",+ "sftp": "credentials",+ "sharepoint": "credentials",+ "sharetribe": "credentials",+ "ship-station": "credentials",+ "shipmail": "token",+ "shippo": "token",+ "shop-wired": "credentials",+ "shopify": "credentials",+ "shopware": "credentials",+ "shortcut": "token",+ "shortio": "token",+ "shutterstock": "credentials",+ "side-shift": "credentials",+ "sig-noz": "credentials",+ "sigma-computing": "credentials",+ "sign-now": "credentials",+ "sim": "credentials",+ "sim-fin": "token",+ "similarweb": "token",+ "simon-data": "credentials",+ "simple-cast": "token",+ "simplesat": "token",+ "simpro": "credentials",+ "sinch": "credentials",+ "singlestore": "token",+ "singular": "token",+ "site24x7": "credentials",+ "skio": "token",+ "skyvern": "credentials",+ "slack": "token",+ "slash": "credentials",+ "sleekplan": "token",+ "sleuth": "credentials",+ "smaily": "credentials",+ "smart-engage": "token",+ "smartlead": "token",+ "smartlook": "credentials",+ "smartreach": "credentials",+ "smartrecruiters": "credentials",+ "smartsheet": "token",+ "smartwaiver": "token",+ "smokeball": "credentials",+ "snapchat": "oauth",+ "snovio": "credentials",+ "snowflake": "credentials",+ "snowplow": "credentials",+ "snyk": "credentials",+ "social-pilot": "credentials",+ "soda-cloud": "credentials",+ "solarwinds-service-desk": "credentials",+ "sonar-cloud": "credentials",+ "sonarqube": "credentials",+ "sonatype-nexus": "credentials",+ "soniox": "credentials",+ "sourcegraph": "credentials",+ "spacelift": "credentials",+ "spark-post": "credentials",+ "speedcurve": "token",+ "split-io": "token",+ "splunk-observability-cloud": "credentials",+ "spot-io": "credentials",+ "spotify-ads": "credentials",+ "spotler-crm": "token",+ "sprig": "token",+ "sprinklr": "credentials",+ "sprinto": "credentials",+ "sprout-social": "credentials",+ "sqlite": "credentials",+ "squadcast": "credentials",+ "square": "credentials",+ "squarespace": "token",+ "stack-adapt": "credentials",+ "stack-overflow-for-teams": "credentials",+ "starburst": "credentials",+ "statsig": "token",+ "statuscake": "token",+ "statuspage": "token",+ "steam": "token",+ "stigg": "token",+ "stock-data": "token",+ "stockx": "credentials",+ "strato": "token",+ "strava": "credentials",+ "stream-elements": "token",+ "streamlabs": "credentials",+ "stripe": "token",+ "stytch": "credentials",+ "substack": "credentials",+ "sumo-logic": "credentials",+ "sumsub": "credentials",+ "supabase": "credentials",+ "supermetrics": "credentials",+ "superwall": "credentials",+ "survey-monkey": "credentials",+ "survey-sparrow": "credentials",+ "survicate": "token",+ "svix": "token",+ "swan": "credentials",+ "swarmia": "token",+ "swonkie": "credentials",+ "synthesia": "token",+ "systeme": "token",+ "taboola": "credentials",+ "tackle-io": "credentials",+ "tailscale": "credentials",+ "talkdesk": "credentials",+ "talkwalker": "credentials",+ "tally": "token",+ "tana": "credentials",+ "tavus": "token",+ "tawk-to": "credentials",+ "teachable": "token",+ "teamcity": "credentials",+ "teamtailor": "token",+ "teamup-fitness": "credentials",+ "teamwork": "credentials",+ "tebra": "credentials",+ "telli": "token",+ "telnyx": "token",+ "tempo": "token",+ "temporal-io": "credentials",+ "tenable-vulnerability-management": "credentials",+ "tenjin": "token",+ "tera-box": "credentials",+ "ternary": "credentials",+ "terra-api": "credentials",+ "terraform-cloud": "credentials",+ "tessitura": "credentials",+ "test-dino": "credentials",+ "testomatio": "credentials",+ "testrail": "credentials",+ "thinkific": "credentials",+ "thinkific-courses": "credentials",+ "thoughtspot": "credentials",+ "thousandeyes": "credentials",+ "threads": "credentials",+ "thrive-learning": "credentials",+ "tick-tick": "credentials",+ "ticket-tailor": "token",+ "ticketmaster": "token",+ "tiktok-ads": "oauth",+ "tiktok-shop": "credentials",+ "tile38": "credentials",+ "timely": "credentials",+ "tiny-erp": "credentials",+ "tinybird": "credentials",+ "tinyemail": "token",+ "tipalti": "credentials",+ "tmdb": "token",+ "toast": "credentials",+ "todoist": "token",+ "together-ai": "token",+ "toggl": "credentials",+ "torii": "token",+ "track-pms": "credentials",+ "tracklution": "credentials",+ "tradable-bits": "credentials",+ "traffic-stars": "token",+ "transistor": "token",+ "travis-ci": "token",+ "trello": "credentials",+ "tremendous": "credentials",+ "trigger-dev": "credentials",+ "trino": "credentials",+ "triple-whale": "credentials",+ "trunk-io": "credentials",+ "trust-pilot": "credentials",+ "trustradius": "token",+ "turso": "credentials",+ "tv-maze": "credentials",+ "twelve-data": "token",+ "twelve-labs": "token",+ "twenty": "credentials",+ "twilio": "credentials",+ "twitch": "credentials",+ "twitter": "credentials",+ "twitter-ads": "oauth",+ "two-c2p": "credentials",+ "tyntec-sms": "token",+ "typeform": "credentials",+ "typesense": "credentials",+ "ubidots": "credentials",+ "uk-companies-house": "credentials",+ "uk-ons": "credentials",+ "umami": "credentials",+ "un-comtrade": "credentials",+ "unity-ads": "credentials",+ "unleash": "credentials",+ "unstructured": "credentials",+ "up-promote": "token",+ "uploadcare": "credentials",+ "upstash": "credentials",+ "uptick": "credentials",+ "uptimerobot": "token",+ "us-bea": "token",+ "us-bls": "credentials",+ "us-census": "token",+ "us-eia": "token",+ "us-treasury-fiscal-data": "credentials",+ "user-com": "credentials",+ "userback": "credentials",+ "usersnap": "credentials",+ "uservoice": "credentials",+ "vanta": "credentials",+ "vantage": "token",+ "vapi": "token",+ "veeqo": "token",+ "vellum": "token",+ "vendr": "token",+ "veracode": "credentials",+ "vercel": "credentials",+ "vespa": "credentials",+ "vimeo": "token",+ "virtuous": "credentials",+ "visma-economic": "credentials",+ "vitally": "credentials",+ "vonage": "credentials",+ "vturb": "credentials",+ "vultr": "token",+ "vwo": "credentials",+ "waiteraid": "credentials",+ "walmart-marketplace": "credentials",+ "wasabi": "token",+ "watchmode": "token",+ "waydev": "token",+ "wayfair": "credentials",+ "webflow": "credentials",+ "weights-and-biases": "credentials",+ "whatsapp-business-management": "credentials",+ "when-i-work": "credentials",+ "whmcs": "credentials",+ "who-gho": "credentials",+ "whop": "credentials",+ "wikipedia-pageviews": "credentials",+ "windmill": "credentials",+ "windsor-ai": "credentials",+ "wise": "credentials",+ "wispr-flow": "credentials",+ "wistia": "credentials",+ "wix": "credentials",+ "wiz": "credentials",+ "wompi": "credentials",+ "woo-commerce": "credentials",+ "woo-payments": "credentials",+ "wordpress": "credentials",+ "work-os": "token",+ "workable": "credentials",+ "workato": "credentials",+ "workday": "credentials",+ "workflowmax": "credentials",+ "workiz": "token",+ "workramp": "credentials",+ "world-bank": "credentials",+ "wps-office": "credentials",+ "wrike": "credentials",+ "writesonic": "credentials",+ "wufoo": "credentials",+ "xendit": "credentials",+ "xero": "credentials",+ "xmatters": "credentials",+ "xsolla": "credentials",+ "yahoo-finance": "credentials",+ "yandex-metrica": "credentials",+ "ynab": "token",+ "yoco": "token",+ "yotpo": "credentials",+ "you-sign": "credentials",+ "younium": "credentials",+ "youtube-analytics": "oauth",+ "youtube-data": "credentials",+ "zalando-zdirect": "credentials",+ "zap-sign": "credentials",+ "zapier-supported-storage": "token",+ "zellify": "credentials",+ "zenchef": "credentials",+ "zendesk": "credentials",+ "zendesk-sell": "token",+ "zendesk-sunshine": "credentials",+ "zenduty": "token",+ "zenefits": "credentials",+ "zenloop": "token",+ "zep": "token",+ "zero": "token",+ "zitadel": "credentials",+ "zluri": "credentials",+ "zoho-analytics": "credentials",+ "zoho-bigin": "credentials",+ "zoho-billing": "credentials",+ "zoho-books": "credentials",+ "zoho-campaign": "credentials",+ "zoho-crm": "credentials",+ "zoho-desk": "credentials",+ "zoho-expense": "credentials",+ "zoho-inventory": "credentials",+ "zoho-invoice": "credentials",+ "zonka-feedback": "credentials",+ "zoom": "credentials",+ "zoom-info": "credentials",+ "zuora": "credentials",+ "zylo": "credentials",+}diff --git a/products/warehouse_sources/backend/facade/credential_kinds.py b/products/warehouse_sources/backend/facade/credential_kinds.pyindex a14adde5c75f..f01e59362cbf 100644--- a/products/warehouse_sources/backend/facade/credential_kinds.py+++ b/products/warehouse_sources/backend/facade/credential_kinds.py@@ -2,7 +2,7 @@The `Integration` kind that holds the typed-in credentials of each warehouse source type, keyed bysource type value. A kind is stored data, so a change to an existing value needs a data migration.-See `sources/common/credential_kind.py` for the derivation.+The file is append-only: a removed source keeps its entry. See `sources/common/credential_kind.py`."""from collections.abc import Mappingdiff --git a/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.pyindex 32a01901d0d5..7a55231d6f8b 100644--- a/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py+++ b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py@@ -1,9 +1,14 @@from typing import Any-from django.core.management.base import BaseCommand+from django.core.management.base import BaseCommand, CommandError, CommandParser+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import (+ CREDENTIAL_AUTH_TYPES_PATH,+ render_credential_auth_types,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (CREDENTIAL_KINDS_PATH,+ ChangedFrozenValueError,render_credential_kinds,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (@@ -16,11 +21,30 @@ class Command(BaseCommand):help = ("Write the audit table of which job_inputs keys of each warehouse source are connection fields, ""references, selectors, sync settings or runtime state, and write the frozen credential kind of "- "each source. A test fails when either committed file is out of date."+ "each source and the default auth type of each kind. A test fails when a committed file is out of date.")+ def add_arguments(self, parser: CommandParser) -> None:+ parser.add_argument(+ "--allow-changed-defaults",+ action="store_true",+ help=(+ "Write a changed credential kind or default auth type for an existing entry. Integration rows "+ "store these values, so the change needs a data migration."+ ),+ )+def handle(self, *args: Any, **options: Any) -> None:- CREDENTIAL_KINDS_PATH.write_text(render_credential_kinds())+ allow_changed = options["allow_changed_defaults"]+ try:+ # Render both before writing either, so a refused change writes nothing.+ kinds = render_credential_kinds(allow_changed=allow_changed)+ auth_types = render_credential_auth_types(allow_changed=allow_changed)+ except ChangedFrozenValueError as error:+ raise CommandError(str(error)) from error+ CREDENTIAL_KINDS_PATH.write_text(kinds)self.stdout.write(self.style.SUCCESS(f"Wrote {CREDENTIAL_KINDS_PATH}"))+ CREDENTIAL_AUTH_TYPES_PATH.write_text(auth_types)+ self.stdout.write(self.style.SUCCESS(f"Wrote {CREDENTIAL_AUTH_TYPES_PATH}"))CLASSIFICATION_AUDIT_PATH.write_text(render_classification_audit())self.stdout.write(self.style.SUCCESS(f"Wrote {CLASSIFICATION_AUDIT_PATH}"))diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.pyindex 80aff19b1aaa..f8bacacd1d26 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py@@ -63,6 +63,10 @@ class AmazonS3Source(ObjectStorageSource[AmazonS3SourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.AMAZONS3+ @property+ def credential_kind(self) -> str | None:+ return "aws-s3"+@propertydef connection_host_fields(self) -> list[str]:# The URL picks the bucket and region the stored key is sent to, so changing it must re-require the key.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.pyindex 9c46e424a4a5..1a53bcf88c61 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py@@ -90,6 +90,10 @@ class AppleSearchAdsSource(def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.APPLESEARCHADS+ @property+ def credential_kind(self) -> str | None:+ return "apple-ads"+@propertydef connection_host_fields(self) -> list[str]:# The stored private key is sent against whichever ad account (Platform API) ordiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.pyindex 1d78cb197a39..4834058fb512 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py@@ -39,6 +39,11 @@ class AwsDocumentDBSource(MongoDBSource):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.AWSDOCUMENTDB+ @property+ def credential_kind(self) -> str | None:+ # Without this, the source inherits the kind of `MongoDBSource`.+ return "aws-documentdb"+@staticmethoddef _adjust_field(field: FieldType) -> FieldType:if isinstance(field, SourceFieldInputConfig) and field.name == "connection_string":diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.pyindex 4325d596e00d..b4d1ea2e88b3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py@@ -251,6 +251,10 @@ class BigQuerySource(SQLSource[BigQuerySourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.BIGQUERY+ @property+ def credential_kind(self) -> str | None:+ return "google-cloud-service-account"+def get_retryable_errors(self) -> set[str]:# The client-side limits on a query job and on the wait for one page of rows. BigQuery was# reachable and then gave no result in time, so a later attempt can succeed.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.pyindex 5f05bca1e1c9..4e440a52251a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py@@ -240,6 +240,10 @@ class ClickHouseSource(SimpleSource[ClickHouseSourceConfig], SSHTunnelMixin, Valdef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.CLICKHOUSE+ @property+ def credential_kind(self) -> str | None:+ return "clickhouse"+def _bypass_env_proxy(self, config: ClickHouseSourceConfig, team_id: int) -> BypassEnvProxy:"""Why this connection may skip the egress proxy, or None to stay proxied.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.pyindex 41b9ee0dee19..837fcac97747 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py@@ -49,6 +49,10 @@ class ClickUpSource(ResumableSource[ClickUpSourceConfig, ClickUpResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.CLICKUP+ @property+ def credential_kind(self) -> str | None:+ return "clickup"+@propertydef connection_host_fields(self) -> list[str]:# `workspace_id` selects which ClickUp workspace the stored API token is used against.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.pyindex 2ad35e70a75f..4b42d552a54c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py@@ -15,6 +15,10 @@ class CockroachDBSource(SimpleSource[CockroachDBSourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.COCKROACHDB+ @property+ def credential_kind(self) -> str | None:+ return "cockroachdb"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.pyindex 8c9a9b92f2a0..7af300b187c6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py@@ -33,6 +33,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.canCanonicalDescriptions,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.config import Config+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (+ default_credential_kind,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchemafrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import (@@ -518,6 +521,29 @@ class _BaseSource(ABC, Generic[ConfigType]):a differently named field (e.g. Okta's ``okta_domain``) should list it here."""return []+ @property+ def credential_kind(self) -> str | None:+ """The ``Integration`` kind that holds this source's typed-in credentials, or None when the source has none.++ The default is the kind of the source's OAuth field, else the kebab-case source type. Override it when the+ source has the credential shape of an existing kind (Postgres is ``postgresql``), or when kebab-case splits+ a brand name (MySQL is ``mysql``). The kind is stored data and `facade/credential_kinds.py` freezes it, so+ never change the kind of a released source. A subclass inherits an override, so it must set its own kind."""+ return default_credential_kind(self.source_type, self.get_source_config)++ @property+ def credential_auth_type(self) -> str:+ """The default auth type (``oauth``, ``token`` or ``credentials``) of rows of ``credential_kind``.++ The default comes from the connection fields. It is stored data and `facade/credential_auth_types.py`+ freezes it, so never change it for a released source."""+ # The derivation module imports this module.+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import ( # noqa: PLC0415+ derive_credential_auth_type,+ )++ return derive_credential_auth_type(self.get_source_config)+def is_unreachable_validation_error(self, error: str) -> bool:"""Whether a ``validate_credentials`` message means only that the host could not be reachedover the network. The update serializer may then save the change with a warning for a teamdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.pynew file mode 100644index 000000000000..5a5cb5ff7aea--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.py@@ -0,0 +1,119 @@+"""The default auth type of each warehouse source credential kind.++A row of a credential kind stores its auth type in `config["auth_type"]` only when it is not the default+of the kind, so the default is stored data. The result is frozen in the generated file+`facade/credential_auth_types.py`, and the warehouse product registers it with the integrations SDK at+startup. `generate_source_field_classification` writes that file, and a test fails when it is out of date.++Each source gives the auth type of its kind with `credential_auth_type` on its class. The default,+`derive_credential_auth_type`, comes from the source's connection fields, the same split as+`job_input_classification`:++- `oauth`: the source has an OAuth field and no typed-in connection field.+- `token`: the source has exactly one typed-in field, and it is a secret.+- `credentials`: every other source.+"""++from collections.abc import Iterator+from pathlib import Path++from products.warehouse_sources.backend.facade.credential_auth_types import CREDENTIAL_AUTH_TYPES+from products.warehouse_sources.backend.facade.source_config import (+ SourceConfig,+ SourceFieldInputConfig,+ SourceFieldSelectConfig,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import merge_append_only+from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (+ JobInputKind,+ field_kind,+ reference_option_values,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry++CREDENTIAL_AUTH_TYPES_PATH = Path(__file__).resolve().parents[4] / "facade" / "credential_auth_types.py"++OAUTH = "oauth"+TOKEN = "token"+CREDENTIALS = "credentials"+++class _Unclear(Exception):+ pass+++def _typed_in_fields(field: FieldType) -> Iterator[FieldType]:+ """The fields a user types in for one connection field. Raises `_Unclear` when the shape has a choice."""+ if isinstance(field, SourceFieldSelectConfig):+ typed_in_options = [+ option for option in field.options if option.fields and option.value not in reference_option_values(field)+ ]+ if not any(option.fields for option in field.options):+ # A plain choice, such as a region. It is a typed-in field itself.+ yield field+ elif len(typed_in_options) == 1:+ for nested in typed_in_options[0].fields or []:+ yield from _typed_in_fields(nested)+ elif typed_in_options:+ # Several typed-in shapes, such as an API key or a key pair.+ raise _Unclear+ return+ yield field+++def derive_credential_auth_type(source_config: SourceConfig) -> str:+ fields = list(source_config.fields)+ has_oauth = any(field_kind(field) == JobInputKind.REFERENCE or reference_option_values(field) for field in fields)+ try:+ typed_in = [+ leaf for field in fields if field_kind(field) == JobInputKind.CONNECTION for leaf in _typed_in_fields(field)+ ]+ except _Unclear:+ return CREDENTIALS+ if not typed_in:+ return OAUTH if has_oauth else CREDENTIALS+ if len(typed_in) == 1 and isinstance(typed_in[0], SourceFieldInputConfig) and typed_in[0].secret:+ return TOKEN+ return CREDENTIALS+++def compute_credential_auth_types() -> dict[str, str]:+ """The default auth type of each credential kind, keyed by kind."""+ auth_types: dict[str, str] = {}+ for source in SourceRegistry.get_all_sources().values():+ kind = source.credential_kind+ if kind is not None:+ auth_types[kind] = source.credential_auth_type+ return dict(sorted(auth_types.items()))+++def frozen_credential_auth_types(*, allow_changed: bool = False) -> dict[str, str]:+ """The committed default auth types plus the ones of new kinds. See `merge_append_only`."""+ return merge_append_only(+ CREDENTIAL_AUTH_TYPES,+ compute_credential_auth_types(),+ name="default auth types",+ allow_changed=allow_changed,+ )+++def render_credential_auth_types(*, allow_changed: bool = False) -> str:+ lines = [+ '"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.',+ "",+ "The default auth type of each warehouse source credential kind, keyed by kind. A row stores its auth",+ "type only when it is not the default, so a change to an existing value needs a data migration. See",+ "`sources/common/credential_auth_type.py` for the derivation. A kind in the integrations catalogue, such",+ "as `stripe`, keeps the default of the catalogue. The file is append-only: a removed kind keeps its",+ "entry.",+ '"""',+ "",+ "from collections.abc import Mapping",+ "",+ "CREDENTIAL_AUTH_TYPES: Mapping[str, str] = {",+ ]+ auth_types = frozen_credential_auth_types(allow_changed=allow_changed)+ lines += [f' "{kind}": "{auth_type}",' for kind, auth_type in auth_types.items()]+ lines += ["}", ""]+ return "\n".join(lines)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.pyindex 000fed4e47e7..2c5a35bde14c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py@@ -1,16 +1,21 @@"""The `Integration` kind that holds a warehouse source's typed-in credentials.-The kind comes from `derive_credential_kind`. The result is frozen in the generated file-`facade/credential_kinds.py`, because a kind is stored data: renaming a kind after the first row-exists needs a data migration. `generate_source_field_classification` writes that file, and a test-fails when it is out of date.+Each source gives its kind with `credential_kind` on its class. The default comes from+`default_credential_kind`. The result is frozen in the generated file `facade/credential_kinds.py`,+because a kind is stored data: renaming a kind after the first row exists needs a data migration.+`generate_source_field_classification` writes that file, and a test fails when it is out of date.++The generated files are append-only: a new source adds its entry, a removed source keeps its entry,+and a changed value fails the generation unless `--allow-changed-defaults` is passed."""import refrom collections.abc import Iterator, Mappingfrom pathlib import Path+from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDSfrom products.warehouse_sources.backend.facade.source_config import (+ SourceConfig,SourceFieldConfig,SourceFieldOauthConfig,SourceFieldSelectConfig,@@ -24,43 +29,6 @@ CREDENTIAL_KINDS_PATH = Path(__file__).resolve().parents[4] / "facade" / "creden# `Integration.kind` is `CharField(max_length=32)`.CREDENTIAL_KIND_MAX_LENGTH = 32-# Sources that have no credential kind. Custom keeps its own `job_inputs` and its OAuth rows are not `Integration` rows.-CREDENTIAL_KIND_EXCLUDED: frozenset[ExternalDataSourceType] = frozenset({ExternalDataSourceType.CUSTOM})--# A source whose credentials have the same shape as an existing `Integration` kind uses that kind.-EXISTING_KIND_OVERRIDES: Mapping[ExternalDataSourceType, str] = {- ExternalDataSourceType.POSTGRES: "postgresql",- ExternalDataSourceType.REDSHIFT: "aws-redshift",- ExternalDataSourceType.AMAZONS3: "aws-s3",- ExternalDataSourceType.BIGQUERY: "google-cloud-service-account",- ExternalDataSourceType.GITLAB: "gitlab",- ExternalDataSourceType.APPLESEARCHADS: "apple-ads",- ExternalDataSourceType.CUSTOMERIO: "customer-io",- ExternalDataSourceType.CLICKUP: "clickup",-}--# The kebab-case rule splits a one-word brand name at its inner capitals. These names keep the brand spelling.-BRAND_CASING_OVERRIDES: Mapping[ExternalDataSourceType, str] = {- ExternalDataSourceType.MYSQL: "mysql",- ExternalDataSourceType.MONGODB: "mongodb",- ExternalDataSourceType.CLICKHOUSE: "clickhouse",- ExternalDataSourceType.SHAREPOINT: "sharepoint",- ExternalDataSourceType.TIKTOKADS: "tiktok-ads",- ExternalDataSourceType.YOUTUBEANALYTICS: "youtube-analytics",- ExternalDataSourceType.DYNAMODB: "dynamodb",- ExternalDataSourceType.COCKROACHDB: "cockroachdb",- ExternalDataSourceType.COSMOSDB: "cosmosdb",- ExternalDataSourceType.PLANETSCALEMYSQL: "planetscale-mysql",- ExternalDataSourceType.PLANETSCALEPOSTGRES: "planetscale-postgres",- ExternalDataSourceType.AWSDOCUMENTDB: "aws-documentdb",- ExternalDataSourceType.SQLITE: "sqlite",- ExternalDataSourceType.GNEWS: "gnews",- ExternalDataSourceType.PYPI: "pypi",- ExternalDataSourceType.TMDB: "tmdb",- ExternalDataSourceType.RB2B: "rb2b",- ExternalDataSourceType.E2B: "e2b",-}-_WORD_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])")@@ -68,17 +36,6 @@ def kebab_case(name: str) -> str:return _WORD_BOUNDARY.sub("-", name).lower()-def derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind: str | None = None) -> str:- """The kind for a source. A source with an OAuth field uses the kind that field names, because its rows exist already."""- if oauth_kind:- return oauth_kind- if source_type in EXISTING_KIND_OVERRIDES:- return EXISTING_KIND_OVERRIDES[source_type]- if source_type in BRAND_CASING_OVERRIDES:- return BRAND_CASING_OVERRIDES[source_type]- return kebab_case(source_type.value)--def _walk(fields: list[SourceFieldConfig]) -> Iterator[SourceFieldConfig]:for field in fields:yield field@@ -89,36 +46,73 @@ def _walk(fields: list[SourceFieldConfig]) -> Iterator[SourceFieldConfig]:yield from _walk(option.fields or [])-def oauth_kind_of(source_type: ExternalDataSourceType) -> str | None:+def oauth_kind_of(source_config: SourceConfig) -> str | None:"""The `Integration` kind of the source's OAuth field, at any depth. None when the source has no such field."""- for field in _walk(list(SourceRegistry.get_source(source_type).get_source_config.fields)):+ for field in _walk(list(source_config.fields)):if isinstance(field, SourceFieldOauthConfig):return field.kindreturn None+def default_credential_kind(source_type: ExternalDataSourceType, source_config: SourceConfig) -> str:+ """A source with an OAuth field uses the kind that field names, because its rows exist already."""+ return oauth_kind_of(source_config) or kebab_case(source_type.value)+++class ChangedFrozenValueError(ValueError):+ """The generation would change a value that rows already store."""+++def merge_append_only(+ committed: Mapping[str, str], computed: Mapping[str, str], *, name: str, allow_changed: bool = False+) -> dict[str, str]:+ """`committed` with the new keys of `computed`. A key that is gone from `computed` stays.++ Raises `ChangedFrozenValueError` when `computed` changes a committed value, unless `allow_changed`.+ """+ changed = {+ key: (committed[key], value) for key, value in computed.items() if key in committed and committed[key] != value+ }+ if changed and not allow_changed:+ lines = "\n".join(f" {key}: {old!r} -> {new!r}" for key, (old, new) in sorted(changed.items()))+ raise ChangedFrozenValueError(+ f"The generation changes committed {name}, which integration rows store:\n{lines}\n"+ "Keep the old value in the source class, or write a data migration for the stored rows and "+ "rerun with --allow-changed-defaults."+ )+ return dict(sorted({**committed, **computed}.items()))++def compute_credential_kinds() -> dict[str, str]:sources = SourceRegistry.get_all_sources()- return {- source_type.value: derive_credential_kind(source_type, oauth_kind_of(source_type))+ kinds = {+ source_type.value: sources[source_type].credential_kindfor source_type in sorted(sources, key=lambda source_type: source_type.value)- if source_type not in CREDENTIAL_KIND_EXCLUDED}+ return {source_type: kind for source_type, kind in kinds.items() if kind is not None}+++def frozen_credential_kinds(*, allow_changed: bool = False) -> dict[str, str]:+ """The committed kinds plus the kinds of new sources. See `merge_append_only`."""+ return merge_append_only(+ CREDENTIAL_KINDS, compute_credential_kinds(), name="credential kinds", allow_changed=allow_changed+ )-def render_credential_kinds() -> str:+def render_credential_kinds(*, allow_changed: bool = False) -> str:lines = ['"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.',"","The `Integration` kind that holds the typed-in credentials of each warehouse source type, keyed by","source type value. A kind is stored data, so a change to an existing value needs a data migration.",- "See `sources/common/credential_kind.py` for the derivation.",+ "The file is append-only: a removed source keeps its entry. See `sources/common/credential_kind.py`.",'"""',"","from collections.abc import Mapping","","CREDENTIAL_KINDS: Mapping[str, str] = {",]- lines += [f' "{source_type}": "{kind}",' for source_type, kind in compute_credential_kinds().items()]+ kinds = frozen_credential_kinds(allow_changed=allow_changed)+ lines += [f' "{source_type}": "{kind}",' for source_type, kind in kinds.items()]lines += ["}", ""]return "\n".join(lines)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.pynew file mode 100644index 000000000000..f863a30b74e3--- /dev/null+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.py@@ -0,0 +1,92 @@+from pathlib import Path++import pytest+from unittest.mock import PropertyMock, patch++from django.core.management import CommandError, call_command++import products.warehouse_sources.backend.temporal.data_imports.sources._load_all # noqa: F401+from products.warehouse_sources.backend.facade.credential_auth_types import CREDENTIAL_AUTH_TYPES+from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS+from products.warehouse_sources.backend.management.commands import (+ generate_source_field_classification as command_module,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import (+ CREDENTIAL_AUTH_TYPES_PATH,+ compute_credential_auth_types,+ derive_credential_auth_type,+ frozen_credential_auth_types,+ render_credential_auth_types,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (+ ChangedFrozenValueError,+)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry+from products.warehouse_sources.backend.types import ExternalDataSourceType+++ "source_type,expected",+ [+ # One secret input.+ (ExternalDataSourceType.ABLY, "token"),+ (ExternalDataSourceType.KLAVIYO, "token"),+ # An auth method select with an OAuth option and one typed-in option that holds one secret.+ (ExternalDataSourceType.STRIPE, "token"),+ (ExternalDataSourceType.GITHUB, "token"),+ # Only an OAuth field.+ (ExternalDataSourceType.HUBSPOT, "oauth"),+ (ExternalDataSourceType.GOOGLEADS, "oauth"),+ # Several typed-in fields.+ (ExternalDataSourceType.POSTGRES, "credentials"),+ (ExternalDataSourceType.MYSQL, "credentials"),+ (ExternalDataSourceType.ZENDESK, "credentials"),+ ],+)+def test_credential_auth_type(source_type: ExternalDataSourceType, expected: str) -> None:+ source = SourceRegistry.get_source(source_type)++ assert source.credential_auth_type == expected+ assert derive_credential_auth_type(source.get_source_config) == expected+++def test_compute_reads_the_auth_type_from_the_source_class() -> None:+ zendesk = type(SourceRegistry.get_source(ExternalDataSourceType.ZENDESK))+ with patch.object(zendesk, "credential_auth_type", new_callable=PropertyMock, return_value="token"):+ auth_types = compute_credential_auth_types()++ assert auth_types[CREDENTIAL_KINDS[ExternalDataSourceType.ZENDESK.value]] == "token"+++def test_every_credential_kind_has_an_auth_type() -> None:+ # A kind that a source stopped using keeps its entry, because rows of the kind can exist.+ assert set(CREDENTIAL_KINDS.values()) <= set(CREDENTIAL_AUTH_TYPES)+ assert set(CREDENTIAL_AUTH_TYPES.values()) <= {"oauth", "token", "credentials"}+++def test_a_changed_default_auth_type_fails_the_generation(tmp_path: Path) -> None:+ zendesk = type(SourceRegistry.get_source(ExternalDataSourceType.ZENDESK))+ kind = CREDENTIAL_KINDS[ExternalDataSourceType.ZENDESK.value]+ with (+ patch.object(zendesk, "credential_auth_type", new_callable=PropertyMock, return_value="token"),+ patch.object(command_module, "CREDENTIAL_KINDS_PATH", tmp_path / "kinds.py"),+ patch.object(command_module, "CREDENTIAL_AUTH_TYPES_PATH", tmp_path / "auth_types.py"),+ patch.object(command_module, "CLASSIFICATION_AUDIT_PATH", tmp_path / "audit.csv"),+ ):+ with pytest.raises(ChangedFrozenValueError, match=f"{kind}: 'credentials' -> 'token'"):+ render_credential_auth_types()+ with pytest.raises(CommandError):+ call_command("generate_source_field_classification")+ written_after_refusal = sorted(path.name for path in tmp_path.iterdir())+ call_command("generate_source_field_classification", "--allow-changed-defaults")++ assert written_after_refusal == []+ assert f' "{kind}": "token",' in (tmp_path / "auth_types.py").read_text()+++def test_credential_auth_types_file_is_up_to_date() -> None:+ assert dict(CREDENTIAL_AUTH_TYPES) == frozen_credential_auth_types()+ assert CREDENTIAL_AUTH_TYPES_PATH.read_text() == render_credential_auth_types(), (+ "The credential auth types are out of date. "+ "Run `python manage.py generate_source_field_classification` and commit the result."+ )diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.pyindex 1d16551de311..0b99c0b9954d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py@@ -1,14 +1,17 @@import pytest+from unittest.mock import PropertyMock, patchimport products.warehouse_sources.backend.temporal.data_imports.sources._load_all # noqa: F401from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDSfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (- CREDENTIAL_KIND_EXCLUDED,CREDENTIAL_KIND_MAX_LENGTH,CREDENTIAL_KINDS_PATH,+ ChangedFrozenValueError,compute_credential_kinds,- derive_credential_kind,+ default_credential_kind,+ frozen_credential_kinds,kebab_case,+ merge_append_only,oauth_kind_of,render_credential_kinds,)@@ -17,6 +20,7 @@ from products.warehouse_sources.backend.types import ExternalDataSourceTypeALL_SOURCES = SourceRegistry.get_all_sources()SOURCE_TYPES = sorted(ALL_SOURCES.keys(), key=str)+SOURCE_TYPES_WITH_KIND = [s for s in SOURCE_TYPES if ALL_SOURCES[s].credential_kind is not None]@pytest.mark.parametrize(@@ -35,31 +39,49 @@ def test_kebab_case(name: str, expected: str) -> None:@pytest.mark.parametrize(- "source_type,oauth_kind,expected",+ "source_type,expected",+ [+ # Sources that set their kind in their own class.+ (ExternalDataSourceType.POSTGRES, "postgresql"),+ (ExternalDataSourceType.REDSHIFT, "aws-redshift"),+ (ExternalDataSourceType.AMAZONS3, "aws-s3"),+ (ExternalDataSourceType.CUSTOMERIO, "customer-io"),+ (ExternalDataSourceType.MYSQL, "mysql"),+ (ExternalDataSourceType.TIKTOKADS, "tiktok-ads"),+ # Subclasses of an overriding source set their own kind.+ (ExternalDataSourceType.NEON, "neon"),+ (ExternalDataSourceType.SUPABASE, "supabase"),+ (ExternalDataSourceType.PLANETSCALEMYSQL, "planetscale-mysql"),+ (ExternalDataSourceType.AWSDOCUMENTDB, "aws-documentdb"),+ (ExternalDataSourceType.CUSTOM, None),+ # Sources that use the default.+ (ExternalDataSourceType.GITHUB, "github"),+ (ExternalDataSourceType.STRIPE, "stripe"),+ (ExternalDataSourceType.GOOGLECLOUDSTORAGE, "google-cloud-storage"),+ (ExternalDataSourceType.AWSCLOUDTRAIL, "aws-cloud-trail"),+ (ExternalDataSourceType.SNAPCHATADS, "snapchat"),+ (ExternalDataSourceType.GOOGLEADS, "google-ads"),+ ],+)+def test_credential_kind(source_type: ExternalDataSourceType, expected: str | None) -> None:+ assert ALL_SOURCES[source_type].credential_kind == expected+++ "source_type,expected",[- (ExternalDataSourceType.POSTGRES, None, "postgresql"),- (ExternalDataSourceType.REDSHIFT, None, "aws-redshift"),- (ExternalDataSourceType.AMAZONS3, None, "aws-s3"),- (ExternalDataSourceType.BIGQUERY, None, "google-cloud-service-account"),- (ExternalDataSourceType.GITLAB, None, "gitlab"),- (ExternalDataSourceType.APPLESEARCHADS, None, "apple-ads"),- (ExternalDataSourceType.CUSTOMERIO, None, "customer-io"),- (ExternalDataSourceType.MYSQL, None, "mysql"),- (ExternalDataSourceType.MONGODB, None, "mongodb"),- (ExternalDataSourceType.CLICKHOUSE, None, "clickhouse"),- (ExternalDataSourceType.SHAREPOINT, None, "sharepoint"),- (ExternalDataSourceType.TIKTOKADS, None, "tiktok-ads"),- (ExternalDataSourceType.GITHUB, None, "github"),- (ExternalDataSourceType.STRIPE, None, "stripe"),- (ExternalDataSourceType.GOOGLECLOUDSTORAGE, None, "google-cloud-storage"),- (ExternalDataSourceType.AWSCLOUDTRAIL, None, "aws-cloud-trail"),# The kind of the OAuth field wins, because those rows exist already.- (ExternalDataSourceType.SNAPCHATADS, "snapchat", "snapchat"),- (ExternalDataSourceType.POSTGRES, "custom-kind", "custom-kind"),+ (ExternalDataSourceType.SNAPCHATADS, "snapchat"),+ (ExternalDataSourceType.BIGQUERY, "google-cloud-service-account"),+ (ExternalDataSourceType.YOUTUBEANALYTICS, "youtube-analytics"),+ # Without an OAuth field, the kebab-case source type. These sources override it.+ (ExternalDataSourceType.POSTGRES, "postgres"),+ (ExternalDataSourceType.MYSQL, "my-sql"),+ (ExternalDataSourceType.STRIPE, "stripe"),],)-def test_derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind: str | None, expected: str) -> None:- assert derive_credential_kind(source_type, oauth_kind) == expected+def test_default_credential_kind(source_type: ExternalDataSourceType, expected: str) -> None:+ assert default_credential_kind(source_type, ALL_SOURCES[source_type].get_source_config) == expected@pytest.mark.parametrize(@@ -74,15 +96,23 @@ def test_derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind:],)def test_oauth_kind_of(source_type: ExternalDataSourceType, expected: str | None) -> None:- assert oauth_kind_of(source_type) == expected+ assert oauth_kind_of(ALL_SOURCES[source_type].get_source_config) == expected+++def test_compute_reads_the_kind_from_the_source_class() -> None:+ stripe = type(ALL_SOURCES[ExternalDataSourceType.STRIPE])+ with patch.object(stripe, "credential_kind", new_callable=PropertyMock, return_value="stripe-test"):+ kinds = compute_credential_kinds()++ assert kinds[ExternalDataSourceType.STRIPE.value] == "stripe-test"+ assert ExternalDataSourceType.CUSTOM.value not in kindsdef test_custom_source_has_no_credential_kind() -> None:- assert ExternalDataSourceType.CUSTOM in CREDENTIAL_KIND_EXCLUDEDassert ExternalDataSourceType.CUSTOM.value not in CREDENTIAL_KINDS[email protected]("source_type", [s for s in SOURCE_TYPES if s not in CREDENTIAL_KIND_EXCLUDED], ids=str)[email protected]("source_type", SOURCE_TYPES_WITH_KIND, ids=str)def test_every_source_has_a_valid_credential_kind(source_type: ExternalDataSourceType) -> None:kind = CREDENTIAL_KINDS[source_type.value]@@ -92,7 +122,7 @@ def test_every_source_has_a_valid_credential_kind(source_type: ExternalDataSourcassert not kind.startswith("-") and not kind.endswith("-")-def test_credential_kinds_are_unique_apart_from_shared_oauth_providers() -> None:+def test_credential_kinds_are_unique() -> None:sources_by_kind: dict[str, list[str]] = {}for source_type, kind in CREDENTIAL_KINDS.items():sources_by_kind.setdefault(kind, []).append(source_type)@@ -101,8 +131,38 @@ def test_credential_kinds_are_unique_apart_from_shared_oauth_providers() -> Noneassert not shared, f"Source types that share one credential kind: {shared}"+ "_name,computed,allow_changed,expected",+ [+ ("unchanged", {"a": "x"}, False, {"a": "x"}),+ ("new_key_is_added", {"a": "x", "b": "y"}, False, {"a": "x", "b": "y"}),+ ("removed_key_stays", {}, False, {"a": "x"}),+ ("changed_value_raises", {"a": "z"}, False, ChangedFrozenValueError),+ ("changed_value_with_the_flag", {"a": "z"}, True, {"a": "z"}),+ ],+)+def test_merge_append_only(+ _name: str, computed: dict[str, str], allow_changed: bool, expected: dict[str, str] | type[Exception]+) -> None:+ if isinstance(expected, dict):+ assert merge_append_only({"a": "x"}, computed, name="kinds", allow_changed=allow_changed) == expected+ else:+ with pytest.raises(expected, match="--allow-changed-defaults"):+ merge_append_only({"a": "x"}, computed, name="kinds", allow_changed=allow_changed)+++def test_a_changed_kind_fails_the_generation() -> None:+ stripe = type(ALL_SOURCES[ExternalDataSourceType.STRIPE])+ with patch.object(stripe, "credential_kind", new_callable=PropertyMock, return_value="stripe-test"):+ with pytest.raises(ChangedFrozenValueError, match="Stripe: 'stripe' -> 'stripe-test'"):+ render_credential_kinds()+ allowed = render_credential_kinds(allow_changed=True)++ assert ' "Stripe": "stripe-test",' in allowed++def test_credential_kinds_file_is_up_to_date() -> None:- assert dict(CREDENTIAL_KINDS) == compute_credential_kinds()+ assert dict(CREDENTIAL_KINDS) == frozen_credential_kinds()assert CREDENTIAL_KINDS_PATH.read_text() == render_credential_kinds(), ("The frozen credential kinds are out of date. ""Run `python manage.py generate_source_field_classification` and commit the result."diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.pyindex 9ca32cc8332f..3f93204e379b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py@@ -58,6 +58,10 @@ class CosmosDBSource(def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.COSMOSDB+ @property+ def credential_kind(self) -> str | None:+ return "cosmosdb"+@propertydef connection_host_fields(self) -> list[str]:# The stored key signs every request to this endpoint, so retargeting it must re-require the key.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.pyindex 00f83a91846a..813fde6189e9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py@@ -877,6 +877,11 @@ class CustomSource(SimpleSource[CustomSourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.CUSTOM+ @property+ def credential_kind(self) -> str | None:+ # Custom keeps its own `job_inputs`, and its OAuth rows are not `Integration` rows.+ return None+# Custom REST sources are capped per team to keep abuse of the arbitrary-URL fetcher bounded.max_instances_per_team = MAX_CUSTOM_SOURCES_PER_TEAMdiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.pyindex 198078975559..53f95c7e47a7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py@@ -172,6 +172,10 @@ class CustomerIOSource(def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.CUSTOMERIO+ @property+ def credential_kind(self) -> str | None:+ return "customer-io"+@propertydef webhook_template(self) -> Optional["HogFunctionTemplateDC"]:from products.warehouse_sources.backend.temporal.data_imports.sources.customer_io.webhook_template import (diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.pyindex aa7ef44b3185..92d0be69a4be 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py@@ -55,6 +55,10 @@ class DynamoDBSource(ResumableSource[DynamoDBSourceConfig, DynamoDBResumeConfig]def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.DYNAMODB+ @property+ def credential_kind(self) -> str | None:+ return "dynamodb"+@propertydef connection_host_fields(self) -> list[str]:# The region picks the endpoint the signed request — and so the customer's key — is sent to.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.pyindex c567b345a8cd..be95a7b17cf2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py@@ -51,6 +51,10 @@ class E2BSource(ResumableSource[E2BSourceConfig, E2BResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.E2B+ @property+ def credential_kind(self) -> str | None:+ return "e2b"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.pyindex 99da4da5b9d6..04554fae8b64 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py@@ -54,6 +54,10 @@ class GitLabSource(ResumableSource[GitLabSourceConfig, GitLabResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.GITLAB+ @property+ def credential_kind(self) -> str | None:+ return "gitlab"+@propertydef connection_host_fields(self) -> list[str]:# `gitlab_host` is where the stored token is sent; retargeting it must re-require the token.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.pyindex db1b8299891f..bd02258fca9b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py@@ -59,6 +59,10 @@ class GNewsSource(ResumableSource[GNewsSourceConfig, GNewsResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.GNEWS+ @property+ def credential_kind(self) -> str | None:+ return "gnews"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.pyindex 1dd2cc9319a2..1f3ae6fa1a77 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py@@ -266,6 +266,10 @@ class MongoDBSource(ResumableSource[MongoDBSourceConfig, MongoResumeConfig], Valdef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.MONGODB+ @property+ def credential_kind(self) -> str | None:+ return "mongodb"+def get_non_retryable_errors(self) -> dict[str, str | None]:auth_failed_msg = _MONGO_AUTHENTICATION_FAILED_MESSAGEreturn {diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.pyindex 35c9368c3e21..3cf5d485f219 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py@@ -223,6 +223,10 @@ class MySQLSource(def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.MYSQL+ @property+ def credential_kind(self) -> str | None:+ return "mysql"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.pyindex d037f174fd69..123352d2e90a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py@@ -63,6 +63,11 @@ class NeonSource(PostgresSource):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.NEON+ @property+ def credential_kind(self) -> str | None:+ # Without this, the source inherits the kind of `PostgresSource`.+ return "neon"+@staticmethoddef _adjust_field(field: FieldType) -> FieldType:if isinstance(field, SourceFieldInputConfig):diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.pyindex 920d202ed67f..325b2fda8834 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py@@ -121,6 +121,11 @@ class PlanetScaleMySQLSource(MySQLSource):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.PLANETSCALEMYSQL+ @property+ def credential_kind(self) -> str | None:+ # Without this, the source inherits the kind of `MySQLSource`.+ return "planetscale-mysql"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.pyindex 570f79819e71..ff3da3ffda51 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py@@ -89,6 +89,11 @@ class PlanetScalePostgresSource(PostgresSource):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.PLANETSCALEPOSTGRES+ @property+ def credential_kind(self) -> str | None:+ # Without this, the source inherits the kind of `PostgresSource`.+ return "planetscale-postgres"+@staticmethoddef _adjust_field(field: FieldType) -> FieldType:if not isinstance(field, SourceFieldInputConfig):diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.pyindex 32c2c5f370db..824bdbd752ca 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py@@ -741,6 +741,10 @@ class PostgresSource(def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.POSTGRES+ @property+ def credential_kind(self) -> str | None:+ return "postgresql"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.pyindex 9a1b8080f017..2f1de9ace308 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py@@ -41,6 +41,10 @@ class PyPISource(SimpleSource[PyPISourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.PYPI+ @property+ def credential_kind(self) -> str | None:+ return "pypi"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.pyindex aa0d9ee8a2e9..d67ffff50d91 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py@@ -13,6 +13,10 @@ class RB2BSource(SimpleSource[RB2BSourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.RB2B+ @property+ def credential_kind(self) -> str | None:+ return "rb2b"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.pyindex 501b0970baab..1d9aac5af906 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py@@ -125,6 +125,10 @@ class RedshiftSource(SQLSource[RedshiftSourceConfig], SSHTunnelMixin, ValidateDadef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.REDSHIFT+ @property+ def credential_kind(self) -> str | None:+ return "aws-redshift"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.pyindex 4e603e3fd242..f022c2611d8e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py@@ -82,6 +82,10 @@ class SharePointSource(ResumableSource[SharePointSourceConfig, SharePointResumeCdef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.SHAREPOINT+ @property+ def credential_kind(self) -> str | None:+ return "sharepoint"+def get_non_retryable_errors(self) -> dict[str, str | None]:return {INVALID_SITE_URL_ERROR: None,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.pyindex d62f01b69094..9996a898fd22 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py@@ -13,6 +13,10 @@ class SQLiteSource(SimpleSource[SQLiteSourceConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.SQLITE+ @property+ def credential_kind(self) -> str | None:+ return "sqlite"+@propertydef get_source_config(self) -> SourceConfig:return SourceConfig(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.pyindex ce020ce23628..4bcf4d6de126 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py@@ -235,6 +235,11 @@ class SupabaseSource(PostgresSource):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.SUPABASE+ @property+ def credential_kind(self) -> str | None:+ # Without this, the source inherits the kind of `PostgresSource`.+ return "supabase"+def get_non_retryable_errors(self) -> dict[str, str | None]:return {_SUPABASE_REALTIME_PARTITION_ERROR: _SUPABASE_REALTIME_PARTITION_MESSAGE,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.pyindex 41eac94152fd..e420180e913b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py@@ -86,6 +86,10 @@ class TikTokAdsSource(ResumableSource[TikTokAdsSourceConfig, TikTokAdsResumeConfdef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.TIKTOKADS+ @property+ def credential_kind(self) -> str | None:+ return "tiktok-ads"+def get_canonical_descriptions(self) -> CanonicalDescriptions:from products.warehouse_sources.backend.temporal.data_imports.sources.tiktok_ads.canonical_descriptions import (CANONICAL_DESCRIPTIONS,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.pyindex ab657314a993..9aa5f7a3838b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py@@ -46,6 +46,10 @@ class TMDbSource(ResumableSource[TMDbSourceConfig, TMDbResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.TMDB+ @property+ def credential_kind(self) -> str | None:+ return "tmdb"+def resume_covers_run(self,*,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.pyindex 6879eaabd96c..47a35459cfaf 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py@@ -65,6 +65,10 @@ class YouTubeAnalyticsSource(ResumableSource[YouTubeAnalyticsSourceConfig, YouTudef source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.YOUTUBEANALYTICS+ @property+ def credential_kind(self) -> str | None:+ return "youtube-analytics"+def get_non_retryable_errors(self) -> dict[str, str | None]:return {"401 Client Error": (diff --git a/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts b/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.tsindex efe5901e40b3..e79acef8067f 100644--- a/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts+++ b/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts@@ -191,6 +191,7 @@ export class FakeEmailDomainBackend {const sender: IntegrationConfigApi = {id: this.nextSenderId++,kind: 'email',+ auth_type: 'credentials',config: {provider: 'ses',name: 'Acme',diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.tsindex 1c379085fc87..57bcd4a245c8 100644--- a/services/mcp/src/api/generated.ts+++ b/services/mcp/src/api/generated.ts@@ -65896,12 +65896,32 @@ export namespace Schemas {reconnect_required: boolean;}+ /**+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials+ */+ export type IntegrationAuthTypeEnum = typeof IntegrationAuthTypeEnum[keyof typeof IntegrationAuthTypeEnum];+++ export const IntegrationAuthTypeEnum = {+ Oauth: 'oauth',+ Token: 'token',+ Credentials: 'credentials',+ } as const;+/*** Standard Integration serializer.*/export interface IntegrationConfig {readonly id: number;kind: IntegrationKindEnum;+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.+ *+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials */+ readonly auth_type: IntegrationAuthTypeEnum;config?: unknown;readonly created_at: string;readonly created_by: UserBasic;@@ -88937,6 +88957,12 @@ export namespace Schemas {export interface PatchedIntegrationConfig {readonly id?: number;kind?: IntegrationKindEnum;+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.+ *+ * * `oauth` - OAuth+ * * `token` - Token+ * * `credentials` - Credentials */+ readonly auth_type?: IntegrationAuthTypeEnum;config?: unknown;readonly created_at?: string;readonly created_by?: UserBasic;diff --git a/tools/hogli-commands/hogli_commands/projections.py b/tools/hogli-commands/hogli_commands/projections.pyindex 010a9eca8f76..5535ebb2e7af 100644--- a/tools/hogli-commands/hogli_commands/projections.py+++ b/tools/hogli-commands/hogli_commands/projections.py@@ -15,8 +15,8 @@ each output, keyed by its repo-relative path. It never writes files and never paarguments; this module does both, so ``--check`` and the write path cannot disagree.A renderer returns valid source and leaves the style to the formatter of the workspace-that owns the output: oxfmt at the repo root, Biome under ``products/desktop`` and-``packages/agent``. The runner formats every output before it compares or writes, so a+that owns the output: ruff for Python, oxfmt at the repo root, Biome under ``products/desktop``+and ``packages/agent``. The runner formats every output before it compares or writes, so agenerated file looks like the code around it and no formatter needs an exemption for it."""@@ -25,8 +25,10 @@ from __future__ import annotationsimport sysimport jsonimport runpy+import tomllibimport tempfileimport subprocess+import importlib.metadatafrom collections.abc import Callable, Iterablefrom dataclasses import dataclassfrom pathlib import Path@@ -101,6 +103,16 @@ PROJECTIONS: tuple[Projection, ...] = ("products/commands/COMMANDS.generated.md",),),+ Projection(+ name="integration-kinds",+ renderer="posthog/integrations/kinds_projection.py",+ inputs=(+ "posthog/integrations/catalogue.py",+ "posthog/integrations/contracts.py",+ "posthog/integrations/enums.py",+ ),+ outputs=("posthog/integrations/kinds_generated.py",),+ ),)@@ -140,10 +152,22 @@ AGENT_BIOME = Formatter(lockfile="packages/agent/pnpm-lock.yaml",stdin_args=("format", "--stdin-file-path"),)-FORMATTERS = (OXFMT, BIOME, AGENT_BIOME)+# A Python package, not an npm one: the runner reads its version from uv.lock and runs it from the+# Python environment, which the Python CI job has without node_modules.+RUFF = Formatter(+ package="ruff",+ binary="ruff",+ workspace=".",+ config="pyproject.toml",+ lockfile="uv.lock",+ stdin_args=("format", "--stdin-filename"),+)+FORMATTERS = (OXFMT, BIOME, AGENT_BIOME, RUFF)def formatter_for(path: str) -> Formatter:+ if path.endswith(".py"):+ return RUFFfor formatter in (BIOME, AGENT_BIOME):if path.startswith(f"{formatter.workspace}/"):return formatter@@ -175,6 +199,9 @@ class OutputFormatter:self._npm_prefix = tempfile.TemporaryDirectory()def _locked_version(self, formatter: Formatter) -> str:+ if formatter == RUFF:+ lock = tomllib.loads((self.repo_root / formatter.lockfile).read_text())+ return next(package["version"] for package in lock["package"] if package["name"] == formatter.package)loader = getattr(yaml, "CSafeLoader", yaml.SafeLoader)lock = yaml.load((self.repo_root / formatter.lockfile).read_text(), Loader=loader)return lock["importers"]["."]["devDependencies"][formatter.package]["version"]@@ -182,6 +209,17 @@ class OutputFormatter:def _command(self, formatter: Formatter) -> list[str]:# The installed copy when it matches the lockfile, so a local run stays offline.# CI's Python job has no node_modules, so it runs the locked version through npm.+ if formatter not in self._commands and formatter == RUFF:+ version = self._locked_version(formatter)+ installed_version: str | None+ try:+ installed_version = importlib.metadata.version(formatter.package)+ except importlib.metadata.PackageNotFoundError:+ installed_version = None+ if installed_version == version:+ self._commands[formatter] = [sys.executable, "-m", formatter.binary]+ else:+ self._commands[formatter] = ["uvx", f"{formatter.package}=={version}"]if formatter not in self._commands:version = self._locked_version(formatter)node_modules = self.repo_root / formatter.workspace / "node_modules"diff --git a/tools/hogli-commands/hogli_commands/tests/test_projections.py b/tools/hogli-commands/hogli_commands/tests/test_projections.pyindex defa07e4ed0e..883a10f7c6ea 100644--- a/tools/hogli-commands/hogli_commands/tests/test_projections.py+++ b/tools/hogli-commands/hogli_commands/tests/test_projections.py@@ -1,11 +1,22 @@from __future__ import annotations+import sys+import importlib.metadatafrom pathlib import Pathimport pytestimport click-from hogli_commands.projections import OutputFormatter, Projection, ProjectionRunner+from hogli_commands.projections import (+ BIOME,+ OXFMT,+ RUFF,+ Formatter,+ OutputFormatter,+ Projection,+ ProjectionRunner,+ formatter_for,+)RENDERER = """def render():@@ -88,3 +99,38 @@ class TestProjectionRunner:with pytest.raises(click.ClickException, match="Keep the two in step"):runner.stale(PROJECTION)+++def _uv_lock(tmp_path: Path, version: str) -> None:+ (tmp_path / "uv.lock").write_text(f'version = 1\n\n[[package]]\nname = "ruff"\nversion = "{version}"\n')+++class TestRuffFormatter:+ @pytest.mark.parametrize(+ "path,expected",+ [+ ("posthog/integrations/kinds_generated.py", RUFF),+ ("products/desktop/scripts/tool.py", RUFF),+ ("frontend/src/lib/oauthScopes.generated.ts", OXFMT),+ ("products/desktop/packages/core/src/a.generated.ts", BIOME),+ ],+ )+ def test_formatter_for_picks_ruff_for_python(self, path: str, expected: Formatter) -> None:+ assert formatter_for(path) == expected++ @pytest.mark.parametrize(+ "locked_matches,expected_prefix",+ [(True, [sys.executable, "-m", "ruff"]), (False, ["uvx", "ruff==0.0.1"])],+ ids=["installed-version-matches-the-lock", "falls-back-to-the-locked-version"],+ )+ def test_ruff_runs_from_the_environment_only_at_the_locked_version(+ self, tmp_path: Path, locked_matches: bool, expected_prefix: list[str]+ ) -> None:+ _uv_lock(tmp_path, importlib.metadata.version("ruff") if locked_matches else "0.0.1")++ assert OutputFormatter(tmp_path)._command(RUFF) == expected_prefix++ def test_formats_python_output(self, tmp_path: Path) -> None:+ _uv_lock(tmp_path, importlib.metadata.version("ruff"))++ assert OutputFormatter(tmp_path).format("out/kinds.py", "x={ 'a':1 }") == 'x = {"a": 1}\n'