Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(integrations): add the integrations SDK with three auth types and a model-access ratchet (#124844)

Tom Owers005d709492f9
diff --git a/.agents/skills/implementing-warehouse-sources/SKILL.md b/.agents/skills/implementing-warehouse-sources/SKILL.md
index 2980a4fc9a33..017cef242467 100644
--- a/.agents/skills/implementing-warehouse-sources/SKILL.md
+++ b/.agents/skills/implementing-warehouse-sources/SKILL.md
@@ -401,19 +401,37 @@ Rules:
- When you are not sure, keep the default (connection). A setting stored as a connection field still works. A credential marked as a sync setting stays in plain source config.
- A field in `connection_host_fields` must stay a connection field. The invariant test fails otherwise.
- Never set `sync_setting=True` on a field that is `secret=True`, a password or a file upload. The invariant test fails otherwise.
-- Every new source and every field change needs the audit CSV regenerated (step 6 of the workflow). The command also rewrites `facade/credential_kinds.py`.
+- Every new source and every field change needs the audit CSV regenerated (step 6 of the workflow). The command also rewrites `facade/credential_kinds.py` and `facade/credential_auth_types.py`.
- A source that has an OAuth field and an account id field must mark the account id field `account_selector=True`. A test flags the unmarked ones by name.
#### Credential kind
-Each source has a credential kind: the `Integration` kind that holds its typed-in credentials. It is at most 32 characters. `sources/common/credential_kind.py` derives it in this order:
+Each source has a credential kind: the `Integration` kind that holds its typed-in credentials. It is at most 32 characters. The source gives it with the `credential_kind` property on its own class. The default on `BaseSource` is:
1. The kind of the source's OAuth field (`SourceFieldOauthConfig.kind`), at any depth.
-2. `EXISTING_KIND_OVERRIDES`, for sources that share the shape of an existing kind (`Postgres` is `postgresql`, `AmazonS3` is `aws-s3`).
-3. `BRAND_CASING_OVERRIDES`, for one-word brands that kebab-case splits (`MySQL` is `mysql`, not `my-sql`).
-4. The kebab-case source type (`AwsCloudTrail` is `aws-cloud-trail`).
+2. Else the kebab-case source type (`AwsCloudTrail` is `aws-cloud-trail`).
-The result is frozen in the generated file `facade/credential_kinds.py` (`CREDENTIAL_KINDS`), because a kind is stored data. A test fails when the file is out of date, when a kind is longer than 32 characters, or when two source types share a kind. Add an override when a new source needs one of the cases above. The Custom source has no credential kind.
+Override `credential_kind` in your `source.py` when the default is wrong:
+
+- The source has the credential shape of an existing `Integration` kind (`PostgresSource` returns `postgresql`, `AmazonS3Source` returns `aws-s3`).
+- Kebab-case splits a one-word brand name (`MySQLSource` returns `mysql`, not `my-sql`).
+- The source has no credential kind. Return `None` (`CustomSource` does this).
+
+A subclass inherits the override of its parent, so a subclass of an overriding source sets its own kind (`NeonSource` returns `neon`, not the `postgresql` of `PostgresSource`).
+
+The result is frozen in the generated file `facade/credential_kinds.py` (`CREDENTIAL_KINDS`), because a kind is stored data. Never change the kind of a released source. A test fails when the file is out of date, when a kind is longer than 32 characters, or when two source types share a kind.
+
+Each credential kind also has a default auth type: the `credential_auth_type` property on the source class. The default on `BaseSource` (`derive_credential_auth_type` in `sources/common/credential_auth_type.py`) reads the connection fields:
+
+- `oauth`: the source has an OAuth field and no typed-in connection field.
+- `token`: the source has exactly one typed-in field, and it is a secret.
+- `credentials`: every other source.
+
+Override `credential_auth_type` in your `source.py` only when this rule gives the wrong type for the source.
+
+The result is frozen in `facade/credential_auth_types.py` (`CREDENTIAL_AUTH_TYPES`), because a row stores its auth type only when it is not the default. The product registers these kinds with the integrations SDK at startup. Read and write their rows with `IntegrationManager.source(kind)` from `posthog.integrations`. A kind that is also in the integrations catalogue (`stripe`, `postgresql`) keeps the default of the catalogue.
+
+Both generated files are append-only. A new source adds its entry. A removed source keeps its entry, because rows of its kind can exist. When a change to a source would change a committed kind or default auth type, `generate_source_field_classification` fails and writes nothing. Keep the old value with an override in the source class. If the change is necessary, write a data migration for the stored rows first, then rerun the command with `--allow-changed-defaults`.
#### Reading and writing `job_inputs`
@@ -809,6 +827,9 @@ If new:
2. **Integration kind**.
- Add to the `IntegrationKind` enum in `posthog/models/integration/model.py`.
+ - Add the kind to `KINDS` in `posthog/integrations/catalogue.py`, with `default=_OAUTH`, then run `hogli build:projections --only integration-kinds`.
+ Read and write its rows through the integrations SDK, for example `IntegrationManager.your_source.oauth.get(team_id, integration_id)`.
+ See `posthog/integrations/README.md`.
- Add to `OauthIntegration.supported_kinds` in `posthog/models/integration/oauth.py`.
- Add an `elif kind == "your-source": return OauthConfig(...)` branch in `oauth_config_for_kind()`, also in `oauth.py`.
Raise `NotImplementedError("<Source> app not configured")` when the env vars are empty — that's the
diff --git a/.github/workflows/ci-python.yml b/.github/workflows/ci-python.yml
index 689f3cba1482..38c96772b9d3 100644
--- a/.github/workflows/ci-python.yml
+++ b/.github/workflows/ci-python.yml
@@ -509,6 +509,12 @@ jobs:
[ "$RUN_BIN_SCRIPTS" = "true" ] || { echo "bin/ unchanged — skipping"; exit 0; }
pytest bin/test/test_find_python_dependencies.py -v --junitxml=junit-dependency-detection.xml
+ - name: Run semgrep ratchet script tests
+ shell: bash
+ run: |
+ [ "$RUN_BIN_SCRIPTS" = "true" ] || { echo "bin/ unchanged — skipping"; exit 0; }
+ pytest bin/test/test_semgrep_ratchet.py -v
+
- name: Run granian shared socket tests
shell: bash
# The bin_scripts filter includes uv.lock, so every granian upgrade runs this.
diff --git a/.github/workflows/ci-security.yaml b/.github/workflows/ci-security.yaml
index fa15071100b3..11182df34b52 100644
--- a/.github/workflows/ci-security.yaml
+++ b/.github/workflows/ci-security.yaml
@@ -486,6 +486,30 @@ jobs:
--timeout 300 \
bin/ common/ docs/onboarding/ ee/ frontend/ packages/ posthog/ products/ services/ tools/
+ # Per-file ratchet: the INFO rule integration-model-through-sdk has a backlog, and its
+ # baseline file lists the allowed count per file. More findings than the baseline fails, and
+ # so do fewer, so each migration lowers the baseline. See .semgrep/rules/devex/README.md.
+ # Skip on unrebased branches: the rule may not exist there yet.
+ - name: Integration model ratchet (blocking)
+ run: |
+ if [ ! -f .semgrep/rules/devex/integration-model-through-sdk.yaml ]; then
+ echo "integration-model-through-sdk not present on this branch yet, skipping (rebase to pick it up)"
+ exit 0
+ fi
+ docker run --rm -v "${{ github.workspace }}:/src" -w /src \
+ -e SEMGREP_ENABLE_VERSION_CHECK=false \
+ ${{ env.SEMGREP_IMAGE }} \
+ semgrep \
+ --config ".semgrep/rules/devex/integration-model-through-sdk.yaml" \
+ --json \
+ --output semgrep-integration-model.json \
+ --metrics=off \
+ --timeout 300 \
+ common/ ee/ posthog/ products/
+ python3 bin/semgrep_ratchet.py \
+ --findings semgrep-integration-model.json \
+ --baseline posthog/integrations/model_access_baseline.txt
+
# Pass 3 (PRs only): fail on newly-introduced WARNING findings.
# --baseline-commit diffs against the PR base (first parent of the
# merge commit), so the pass-1 backlog is grandfathered while new
diff --git a/.semgrep/rules/devex/README.md b/.semgrep/rules/devex/README.md
index ea5e1b97a48e..1a9830900138 100644
--- a/.semgrep/rules/devex/README.md
+++ b/.semgrep/rules/devex/README.md
@@ -25,3 +25,18 @@ A WARNING rule typically gets flipped to ERROR once its backlog hits zero. The r
- Include the migration playbook (or a pointer to a skill) in the rule's `message:` field — that's what reviewers and authors see when the rule fires.
- Scope with `paths.include:` rather than relying only on `paths.exclude:` — narrower scope means faster scans and clearer intent.
- For legitimate exceptions, prefer `paths.exclude:` for categorical cases (whole dirs/files) and `# nosemgrep: <rule-id>` for case-by-case exemptions at the call site.
+
+## Per-file baseline ratchets
+
+A WARNING rule grandfathers its backlog through `--baseline-commit`, so the backlog has no count that a migration can lower, and a moved file looks new.
+For a migration that runs over many PRs, use a per-file baseline instead:
+
+1. Give the rule `severity: INFO`. The three passes above select WARNING and ERROR only, so they do not run it.
+2. Check in a baseline file that lists `<path> <count>` for each file with findings. `bin/semgrep_ratchet.py --update` writes it from `semgrep --json` output.
+3. Add a step to the `semgrep-devex` job that runs the rule with `--json` and then `bin/semgrep_ratchet.py --findings <json> --baseline <file>`.
+
+The ratchet fails when a file has more findings than its baseline count, and also when it has fewer. So each migration PR lowers the count or removes the line.
+
+| Rule | Baseline |
+| ------------------------------- | ------------------------------------------------ |
+| `integration-model-through-sdk` | `posthog/integrations/model_access_baseline.txt` |
diff --git a/.semgrep/rules/devex/github-installation-first-team.py b/.semgrep/rules/devex/github-installation-first-team.py
index 7ccaa52e4211..cf52e1434f7c 100644
--- a/.semgrep/rules/devex/github-installation-first-team.py
+++ b/.semgrep/rules/devex/github-installation-first-team.py
@@ -101,3 +101,43 @@ def every_integration(installation_id):
def justified_exception(installation_id):
# nosemgrep: github-installation-first-team -- the installation has one project by construction here
return Integration.objects.filter(kind="github", integration_id=installation_id).first()
+
+
+def first_sdk_row(installation_id):
+ # ruleid: github-installation-first-team
+ return IntegrationManager.github.oauth.list_by_external_id(str(installation_id))[0]
+
+
+def first_sdk_row_through_next(installation_id):
+ # ruleid: github-installation-first-team
+ return next(iter(IntegrationManager.kinds("github").list_by_external_id(str(installation_id))), None)
+
+
+def first_sdk_row_through_variable(installation_id):
+ rows = IntegrationManager.github.oauth.list_by_external_id(str(installation_id))
+ if not rows:
+ return None
+ # ruleid: github-installation-first-team
+ return rows[0]
+
+
+def first_sdk_row_of_some_teams(installation_id, team_ids):
+ # ok: github-installation-first-team
+ return IntegrationManager.github.oauth.list_by_external_id(str(installation_id), team_ids=team_ids)[0]
+
+
+def first_sdk_row_of_some_teams_through_variable(installation_id, team_ids):
+ rows = IntegrationManager.github.oauth.list_by_external_id(str(installation_id), team_ids=team_ids)
+ # ok: github-installation-first-team
+ return next(iter(rows), None)
+
+
+def every_sdk_row(installation_id):
+ # ok: github-installation-first-team
+ for row in IntegrationManager.github.oauth.list_by_external_id(str(installation_id)):
+ notify(row.team_id)
+
+
+def first_slack_sdk_row(slack_team_id):
+ # ok: github-installation-first-team
+ return IntegrationManager.slack.oauth.list_by_external_id(slack_team_id)[0]
diff --git a/.semgrep/rules/devex/github-installation-first-team.yaml b/.semgrep/rules/devex/github-installation-first-team.yaml
index b12ec7b33cd6..db2ab0e1d3da 100644
--- a/.semgrep/rules/devex/github-installation-first-team.yaml
+++ b/.semgrep/rules/devex/github-installation-first-team.yaml
@@ -52,6 +52,26 @@ rules:
- pattern-not: Integration.objects. ... .get(..., team=$TEAM, ...)
- pattern-not: Integration.objects. ... .filter(..., team_id=$TEAM, ...). ... [$INDEX]
- pattern-not: Integration.objects. ... .filter(..., team=$TEAM, ...). ... [$INDEX]
+ # One GitHub row out of an SDK installation lookup that no project filter narrows.
+ - patterns:
+ - pattern-either:
+ - pattern: $ACCESSOR.list_by_external_id(...)[$INDEX]
+ - pattern: next(iter($ACCESSOR.list_by_external_id(...)), ...)
+ - patterns:
+ - pattern-inside: |
+ $ROWS = $ACCESSOR.list_by_external_id(...)
+ ...
+ - pattern-not-inside: |
+ $ROWS = $ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)
+ ...
+ - pattern-either:
+ - pattern: $ROWS[$INDEX]
+ - pattern: next(iter($ROWS), ...)
+ - metavariable-regex:
+ metavariable: $ACCESSOR
+ regex: (?i).*github.*
+ - pattern-not: $ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)[$INDEX]
+ - pattern-not: next(iter($ACCESSOR.list_by_external_id(..., team_ids=$TEAMS, ...)), ...)
paths:
include:
- ee/**/*.py
diff --git a/.semgrep/rules/devex/integration-model-through-sdk.py b/.semgrep/rules/devex/integration-model-through-sdk.py
new file mode 100644
index 000000000000..c814f771ff38
--- /dev/null
+++ b/.semgrep/rules/devex/integration-model-through-sdk.py
@@ -0,0 +1,192 @@
+# Test cases for integration-model-through-sdk.
+# ruff: noqa
+
+# ruleid: integration-model-through-sdk
+from posthog.models import Integration
+
+# ruleid: integration-model-through-sdk
+from posthog.models import Integration, Team
+
+# ruleid: integration-model-through-sdk
+from posthog.models.integration import Integration as TeamIntegration
+
+# ruleid: integration-model-through-sdk
+from posthog.models.integration.model import Integration
+
+# ruleid: integration-model-through-sdk
+import posthog.models.integration.model
+
+# ok: integration-model-through-sdk
+from posthog.models.integration import SlackIntegration
+
+# ok: integration-model-through-sdk
+from posthog.integrations import IntegrationManager, IntegrationRecord
+
+from django.apps import apps
+from django.db import models
+from django.db.models import F, Q
+from django.db.models import ForeignKey
+
+team_id = 1
+
+# ruleid: integration-model-through-sdk
+Integration.objects.filter(team_id=team_id, kind="github").first()
+
+# ruleid: integration-model-through-sdk
+TeamIntegration.objects.get(id=1)
+
+# ruleid: integration-model-through-sdk
+manager = posthog.models.integration.model.Integration.objects
+
+# ruleid: integration-model-through-sdk
+IntegrationRecord.objects.filter(team_id=team_id)
+
+# ruleid: integration-model-through-sdk
+facade_api.IntegrationRecord.objects.get(id=1)
+
+# ruleid: integration-model-through-sdk
+Integration._base_manager.filter(id=1)
+
+# ruleid: integration-model-through-sdk
+IntegrationRecord._default_manager.all()
+
+# ruleid: integration-model-through-sdk
+posthog.models.Integration._default_manager.all()
+
+# ruleid: integration-model-through-sdk
+TeamIntegration._base_manager.get(id=1)
+
+# ok: integration-model-through-sdk
+SlackThread._default_manager.all()
+
+# ruleid: integration-model-through-sdk
+model = apps.get_model("posthog", "Integration")
+
+# ruleid: integration-model-through-sdk
+model = apps.get_model("posthog.Integration")
+
+# ruleid: integration-model-through-sdk
+model = apps.get_model("posthog.integration")
+
+# ok: integration-model-through-sdk
+model = apps.get_model("posthog", "Team")
+
+# ok: integration-model-through-sdk
+model = apps.get_model("posthog", "UserIntegration")
+
+
+class SlackThread(models.Model):
+ # ruleid: integration-model-through-sdk
+ integration = models.ForeignKey(Integration, on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ other = models.ForeignKey("posthog.Integration", on_delete=models.SET_NULL, null=True)
+ # ruleid: integration-model-through-sdk
+ third = models.ForeignKey(to="posthog.Integration", on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ fourth = models.OneToOneField("posthog.Integration", on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ fifth = models.ForeignKey(IntegrationRecord, on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ sixth = models.ForeignKey("posthog.integration", on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ seventh = models.ManyToManyField("posthog.Integration", related_name="+")
+ # ruleid: integration-model-through-sdk
+ eighth = models.ManyToManyField(to=Integration, related_name="+")
+ # ruleid: integration-model-through-sdk
+ ninth = models.OneToOneField(to=IntegrationRecord, on_delete=models.CASCADE)
+ # ruleid: integration-model-through-sdk
+ tenth = ForeignKey("POSTHOG.INTEGRATION", on_delete=models.CASCADE)
+ # ok: integration-model-through-sdk
+ user_link = models.ForeignKey("posthog.UserIntegration", on_delete=models.CASCADE)
+ # ok: integration-model-through-sdk
+ org_links = models.ManyToManyField("posthog.OrganizationIntegration")
+ # ok: integration-model-through-sdk
+ integration_id_only = models.BigIntegerField(null=True)
+ # ok: integration-model-through-sdk
+ team = models.ForeignKey("posthog.Team", on_delete=models.CASCADE)
+
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.select_related("integration").all()
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.select_related("team", "slack_integration").all()
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.prefetch_related("integration__created_by").all()
+
+# ok: integration-model-through-sdk
+SlackThread.objects.select_related("team", "created_by").all()
+
+# ok: integration-model-through-sdk
+SlackThread.objects.select_related("github_user_integration").all()
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.filter(integration__kind="slack")
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.filter(team_id=team_id, slack_integration__config__team_id="T1")
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.exclude(Q(thread__integration__kind="slack"))
+
+# ok: integration-model-through-sdk
+SlackThread.objects.filter(integration_id=3)
+
+# ok: integration-model-through-sdk
+SlackThread.objects.filter(team_id=team_id)
+
+# ok: integration-model-through-sdk
+SlackThread.objects.filter(github_user_integration__isnull=True)
+
+# ok: integration-model-through-sdk
+SlackThread.objects.filter(organization_integration__kind="vercel")
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.values("id", "integration__kind")
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.values_list("slack_integration__config", flat=True)
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.order_by("-integration__created_at")
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.annotate(kind=F("thread__integration__kind"))
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.filter(**{"integration__kind": "slack"})
+
+# ruleid: integration-model-through-sdk
+SlackThread.objects.exclude(Q(**{"integration__config__team_id": "T1"}))
+
+# ok: integration-model-through-sdk
+SlackThread.objects.values("integration_id", "team__name")
+
+# ok: integration-model-through-sdk
+SlackThread.objects.order_by("-user_integration__created_at")
+
+# ok: integration-model-through-sdk
+SlackThread.objects.values_list("organization_integration__kind", flat=True)
+
+# ok: integration-model-through-sdk
+SlackThread.objects.filter(**{"integration_id": 3})
+
+
+def handler(self, user, thread, integration, slack_integration, org_integration):
+ # ruleid: integration-model-through-sdk
+ user.integration_set.all()
+ # ruleid: integration-model-through-sdk
+ integration.save()
+ # ruleid: integration-model-through-sdk
+ thread.integration.save(update_fields=["config"])
+ # ruleid: integration-model-through-sdk
+ slack_integration.save()
+ # ok: integration-model-through-sdk
+ thread.save()
+ # ok: integration-model-through-sdk
+ org_integration.save()
+ # ok: integration-model-through-sdk
+ self.integration.save(update_fields=["config"])
+ # ok: integration-model-through-sdk
+ IntegrationManager.github.oauth.update(team_id, 1, config_patch={"a": 1})
diff --git a/.semgrep/rules/devex/integration-model-through-sdk.yaml b/.semgrep/rules/devex/integration-model-through-sdk.yaml
new file mode 100644
index 000000000000..c63c5705cd90
--- /dev/null
+++ b/.semgrep/rules/devex/integration-model-through-sdk.yaml
@@ -0,0 +1,130 @@
+rules:
+ # Severity INFO keeps this rule out of the three semgrep-devex passes, which select WARNING and
+ # ERROR. The "Integration model ratchet" step of that job runs it on its own and compares the
+ # findings per file with posthog/integrations/model_access_baseline.txt.
+ - id: integration-model-through-sdk
+ message: |
+ Read and write integration rows through the integrations SDK, not through the `Integration` model:
+
+ from posthog.integrations import IntegrationManager, IntegrationRecord
+
+ IntegrationManager.github.oauth.get(team_id, integration_id)
+ IntegrationManager.stripe.any.list(team_id)
+ IntegrationManager.any.get(team_id, integration_id)
+
+ The integration rows move to a separate service. The SDK is the closed set of operations that
+ keeps working after the move, so code outside it must not import the model, use one of its
+ managers (`Integration.objects`, `_default_manager`, `_base_manager`), load it with
+ `apps.get_model`, add a relation field to it, join it with `select_related`, `prefetch_related`
+ or an `integration__` lookup, or call `.save()` on a row. For a DRF field, use
+ `IntegrationReferenceField` from `posthog/integrations/fields.py`.
+ See `posthog/integrations/README.md`.
+
+ After you move a file to the SDK, lower its count in
+ `posthog/integrations/model_access_baseline.txt`, or remove the line at zero.
+ severity: INFO
+ languages: [python]
+ paths:
+ include:
+ - 'common/**/*.py'
+ - 'ee/**/*.py'
+ - 'posthog/**/*.py'
+ - 'products/**/*.py'
+ exclude:
+ - 'products/desktop/**'
+ - '**/test/**'
+ - '**/tests/**'
+ - '**/test_*.py'
+ - '**/*_test.py'
+ - '**/conftest*.py'
+ - '**/migrations/**'
+ # The model, its per-kind adapters and the SDK are the implementation.
+ - 'posthog/models/integration/**'
+ - 'posthog/models/github_integration_base.py'
+ - 'posthog/integrations/**'
+ # Django admin works on the model by design.
+ - 'posthog/admin/**'
+ - 'products/*/backend/admin.py'
+ - 'products/*/backend/admin/**'
+ pattern-either:
+ # The model class, imported by name or reached through its module.
+ - pattern: from posthog.models import Integration
+ - pattern: from posthog.models.integration import Integration
+ - pattern: from posthog.models.integration.model import Integration
+ - pattern: import posthog.models.integration.model
+ # A manager of the model. `IntegrationRecord` is the same class while the SDK returns models.
+ # `$MODULE.Integration` also matches an import alias such as `Integration as TeamIntegration`,
+ # because semgrep resolves the alias to its module path.
+ - pattern: Integration.objects
+ - pattern: $MODULE.Integration.objects
+ - pattern: $MODULE.Integration._base_manager
+ - pattern: $MODULE.Integration._default_manager
+ - patterns:
+ - pattern-either:
+ - pattern: $MODEL.objects
+ - pattern: $MODEL._base_manager
+ - pattern: $MODEL._default_manager
+ - metavariable-regex:
+ metavariable: $MODEL
+ regex: ^(.*\.)?(Integration|IntegrationRecord)$
+ # The model class looked up by its label.
+ - patterns:
+ - pattern-either:
+ - pattern: $APPS.get_model($APP, "$NAME")
+ - pattern: $APPS.get_model("$NAME")
+ - metavariable-regex:
+ metavariable: $NAME
+ regex: ^(?i:(posthog\.)?integration)$
+ # A relation field to the model: the class, or its label in any letter case.
+ - patterns:
+ - pattern-either:
+ - pattern: $FIELD($TARGET, ...)
+ - pattern: $FIELD(..., to=$TARGET, ...)
+ - metavariable-regex:
+ metavariable: $FIELD
+ regex: ^(\w+\.)*(ForeignKey|OneToOneField|ManyToManyField)$
+ - metavariable-regex:
+ metavariable: $TARGET
+ regex: ^((\w+\.)*(Integration|IntegrationRecord)|["'](?i:posthog\.integration)["'])$
+ # A join through an integration relation.
+ - patterns:
+ - pattern-either:
+ - pattern: $QUERYSET.select_related(..., "$RELATION", ...)
+ - pattern: $QUERYSET.prefetch_related(..., "$RELATION", ...)
+ # `user_integration` and `organization_integration` name other models.
+ - metavariable-regex:
+ metavariable: $RELATION
+ regex: ^(?!(\w+_)?(user|org|organization)_integration)(\w+_)?integration(_set)?(__.*)?$
+ # A lookup through an integration relation, such as `filter(integration__kind="slack")`.
+ - patterns:
+ - pattern-either:
+ - pattern: $QUERYSET.filter(...)
+ - pattern: $QUERYSET.exclude(...)
+ - pattern: $QUERYSET.get(...)
+ - pattern: $QUERYSET.update(...)
+ - pattern: Q(...)
+ - pattern-regex: \b(?!\w*(user|org|organization)_integration__)(\w+__)?(\w+_)?integration__\w+\s*=
+ # The same lookup as a string, such as `values("integration__kind")` or `filter(**{"integration__kind": k})`.
+ - patterns:
+ - pattern-either:
+ - pattern: $QUERYSET.filter(...)
+ - pattern: $QUERYSET.exclude(...)
+ - pattern: $QUERYSET.get(...)
+ - pattern: $QUERYSET.values(...)
+ - pattern: $QUERYSET.values_list(...)
+ - pattern: $QUERYSET.order_by(...)
+ - pattern: F(...)
+ - pattern: Q(...)
+ - pattern-regex: '["'']-?(\w+__)*(?!(\w+_)?(user|org|organization)_integration__)(\w+_)?integration__\w+["'']'
+ - pattern: $OBJECT.integration_set
+ # A write to a row outside the SDK. Names only: `self.integration` and `org_integration` are
+ # often a UserIntegration or an OrganizationIntegration.
+ - patterns:
+ - pattern: $ROW.save(...)
+ - metavariable-regex:
+ metavariable: $ROW
+ regex: ^(?!self\.)(?!(.*\.)?(user|org|organization)_integration$)(.*\.)?(\w+_)?integration$
+ metadata:
+ category: best-practice
+ subcategory:
+ - audit
diff --git a/bin/semgrep_ratchet.py b/bin/semgrep_ratchet.py
new file mode 100755
index 000000000000..f939a3226795
--- /dev/null
+++ b/bin/semgrep_ratchet.py
@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+# ruff: noqa: T201 allow print statements
+"""Hold the findings of one semgrep rule to a checked-in, per-file baseline.
+
+The baseline lists each file that still has findings, with its finding count. The check fails
+when a file has more findings than its baseline count (new debt), and also when it has fewer
+(the baseline is stale). So the baseline only goes down, and each migration lowers it.
+
+A file that semgrep reports an error for (a timeout, a parse error) can have missing findings, so
+the check does not call its line stale, and `--update` does not lower its line.
+
+ semgrep --config <rule>.yaml --json --output findings.json <paths>
+ bin/semgrep_ratchet.py --findings findings.json --baseline <baseline>.txt
+ bin/semgrep_ratchet.py --findings findings.json --baseline <baseline>.txt --update
+"""
+
+import sys
+import json
+import argparse
+from collections import Counter
+from dataclasses import dataclass, field
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class RatchetResult:
+ over: dict[str, tuple[int, int]] = field(default_factory=dict)
+ stale: dict[str, tuple[int, int]] = field(default_factory=dict)
+ # Fewer findings than the baseline in a file with a semgrep error. Not a failure.
+ unchecked: dict[str, tuple[int, int]] = field(default_factory=dict)
+
+ @property
+ def ok(self) -> bool:
+ return not self.over and not self.stale
+
+
+def count_findings(semgrep_output: dict) -> Counter[str]:
+ return Counter(result["path"] for result in semgrep_output.get("results", []))
+
+
+def error_paths(semgrep_output: dict) -> frozenset[str]:
+ paths: set[str] = set()
+ for error in semgrep_output.get("errors", []):
+ if error.get("path"):
+ paths.add(error["path"])
+ paths.update(span["file"] for span in error.get("spans") or [] if span.get("file"))
+ return frozenset(paths)
+
+
+def parse_baseline(text: str) -> dict[str, int]:
+ baseline: dict[str, int] = {}
+ for line_number, raw_line in enumerate(text.splitlines(), start=1):
+ line = raw_line.strip()
+ if not line or line.startswith("#"):
+ continue
+ path, _, count = line.rpartition(" ")
+ if not path or not count.isdigit() or int(count) < 1:
+ raise ValueError(f"Baseline line {line_number} is not '<path> <count>': {raw_line!r}")
+ if path in baseline:
+ raise ValueError(f"Baseline line {line_number} repeats {path}")
+ baseline[path] = int(count)
+ return baseline
+
+
+def compare(
+ findings: Counter[str], baseline: dict[str, int], incomplete: frozenset[str] = frozenset()
+) -> RatchetResult:
+ over: dict[str, tuple[int, int]] = {}
+ stale: dict[str, tuple[int, int]] = {}
+ unchecked: dict[str, tuple[int, int]] = {}
+ for path in sorted(set(findings) | set(baseline)):
+ found, allowed = findings.get(path, 0), baseline.get(path, 0)
+ if found > allowed:
+ over[path] = (found, allowed)
+ elif found < allowed:
+ (unchecked if path in incomplete else stale)[path] = (found, allowed)
+ return RatchetResult(over=over, stale=stale, unchecked=unchecked)
+
+
+def updated_findings(findings: Counter[str], baseline: dict[str, int], incomplete: frozenset[str]) -> Counter[str]:
+ result = Counter(findings)
+ for path in incomplete:
+ if baseline.get(path, 0) > result.get(path, 0):
+ result[path] = baseline[path]
+ return result
+
+
+def render_baseline(header: str, findings: Counter[str]) -> str:
+ lines = [line for line in header.splitlines() if line.startswith("#")]
+ lines += [f"{path} {count}" for path, count in sorted(findings.items())]
+ return "\n".join(lines) + "\n"
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--findings", type=Path, required=True, help="semgrep --json output")
+ parser.add_argument("--baseline", type=Path, required=True, help="the checked-in baseline file")
+ parser.add_argument("--update", action="store_true", help="rewrite the baseline from the findings")
+ args = parser.parse_args(argv)
+
+ semgrep_output = json.loads(args.findings.read_text())
+ incomplete = error_paths(semgrep_output)
+ if semgrep_output.get("errors"):
+ print(f"semgrep reported {len(semgrep_output['errors'])} errors. The counts can be incomplete.")
+ findings = count_findings(semgrep_output)
+ baseline_text = args.baseline.read_text() if args.baseline.exists() else ""
+ baseline = parse_baseline(baseline_text)
+
+ if args.update:
+ findings = updated_findings(findings, baseline, incomplete)
+ args.baseline.write_text(render_baseline(baseline_text, findings))
+ print(f"Wrote {sum(findings.values())} findings in {len(findings)} files to {args.baseline}")
+ return 0
+
+ result = compare(findings, baseline, incomplete)
+ for path, (found, allowed) in result.unchecked.items():
+ print(
+ f"Warning: semgrep reported an error for {path}, so its {found} findings (baseline: {allowed}) are not checked."
+ )
+ if result.ok:
+ print(f"{sum(findings.values())} findings in {len(findings)} files. They match {args.baseline}.")
+ return 0
+
+ if result.over:
+ print(f"New findings. Fix them, or add `# nosemgrep: <rule-id>` with a reason. Do not raise {args.baseline}.")
+ new_paths = set(result.over)
+ for item in semgrep_output.get("results", []):
+ if item["path"] in new_paths:
+ print(f" {item['path']}:{item['start']['line']}: {item.get('check_id', '')}")
+ for path, (found, allowed) in result.over.items():
+ print(f" {path}: {found} findings, baseline allows {allowed}")
+ if result.stale:
+ print(f"Fewer findings than {args.baseline} allows. Lower these lines (remove a line at 0):")
+ for path, (found, allowed) in result.stale.items():
+ print(f" {path} {found} (baseline: {allowed})")
+ return 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/bin/test/test_semgrep_ratchet.py b/bin/test/test_semgrep_ratchet.py
new file mode 100644
index 000000000000..0ccca56f3c03
--- /dev/null
+++ b/bin/test/test_semgrep_ratchet.py
@@ -0,0 +1,147 @@
+#!/usr/bin/env python3
+
+import json
+import tempfile
+from collections import Counter
+from pathlib import Path
+
+import unittest
+
+from parameterized import parameterized
+
+from bin.semgrep_ratchet import compare, error_paths, main, parse_baseline, render_baseline
+
+
+def semgrep_output(*paths: str, errors: list[dict] | None = None) -> dict:
+ return {
+ "results": [{"path": path, "check_id": "rule", "start": {"line": 1}} for path in paths],
+ "errors": errors or [],
+ }
+
+
+TIMEOUT = {"type": "Timeout", "path": "a.py"}
+
+
+class TestSemgrepRatchet(unittest.TestCase):
+ @parameterized.expand(
+ [
+ ("matches", Counter({"a.py": 2}), {"a.py": 2}, {}, {}),
+ ("no_findings_no_baseline", Counter(), {}, {}, {}),
+ ("new_file", Counter({"a.py": 1, "b.py": 1}), {"a.py": 1}, {"b.py": (1, 0)}, {}),
+ ("more_in_a_listed_file", Counter({"a.py": 3}), {"a.py": 2}, {"a.py": (3, 2)}, {}),
+ ("fewer_in_a_listed_file", Counter({"a.py": 1}), {"a.py": 2}, {}, {"a.py": (1, 2)}),
+ ("file_fully_migrated", Counter(), {"a.py": 2}, {}, {"a.py": (0, 2)}),
+ (
+ "moved_file",
+ Counter({"new/a.py": 2}),
+ {"old/a.py": 2},
+ {"new/a.py": (2, 0)},
+ {"old/a.py": (0, 2)},
+ ),
+ ]
+ )
+ def test_compare(
+ self,
+ _name: str,
+ findings: Counter[str],
+ baseline: dict[str, int],
+ expected_over: dict[str, tuple[int, int]],
+ expected_stale: dict[str, tuple[int, int]],
+ ) -> None:
+ result = compare(findings, baseline)
+
+ assert (result.over, result.stale, result.unchecked) == (expected_over, expected_stale, {})
+
+ @parameterized.expand(
+ [
+ ("fewer", Counter({"a.py": 1}), {"a.py": 2}, {}, {}, {"a.py": (1, 2)}),
+ ("none", Counter(), {"a.py": 2}, {}, {}, {"a.py": (0, 2)}),
+ ("more_still_fails", Counter({"a.py": 3}), {"a.py": 2}, {"a.py": (3, 2)}, {}, {}),
+ ("other_file_still_stale", Counter({"a.py": 2}), {"a.py": 2, "b.py": 2}, {}, {"b.py": (0, 2)}, {}),
+ ]
+ )
+ def test_compare_with_an_error_in_a_file(
+ self,
+ _name: str,
+ findings: Counter[str],
+ baseline: dict[str, int],
+ expected_over: dict[str, tuple[int, int]],
+ expected_stale: dict[str, tuple[int, int]],
+ expected_unchecked: dict[str, tuple[int, int]],
+ ) -> None:
+ result = compare(findings, baseline, incomplete=frozenset({"a.py"}))
+
+ assert (result.over, result.stale, result.unchecked) == (expected_over, expected_stale, expected_unchecked)
+
+ def test_error_paths_reads_the_path_and_the_spans(self) -> None:
+ output = semgrep_output(
+ errors=[TIMEOUT, {"type": "Syntax error", "spans": [{"file": "b.py"}, {"file": "c.py"}]}, {"type": "x"}]
+ )
+
+ assert error_paths(output) == frozenset({"a.py", "b.py", "c.py"})
+
+ @parameterized.expand(
+ [
+ ("no_count", "a.py\n"),
+ ("zero_count", "a.py 0\n"),
+ ("not_a_number", "a.py two\n"),
+ ("repeated_path", "a.py 1\na.py 2\n"),
+ ]
+ )
+ def test_parse_baseline_rejects_bad_lines(self, _name: str, text: str) -> None:
+ with self.assertRaises(ValueError):
+ parse_baseline(text)
+
+ def test_render_keeps_the_header_and_round_trips(self) -> None:
+ text = render_baseline("# header\n#\nold.py 9\n", Counter({"b.py": 1, "a b/c.py": 2}))
+
+ assert text == "# header\n#\na b/c.py 2\nb.py 1\n"
+ assert parse_baseline(text) == {"a b/c.py": 2, "b.py": 1}
+
+ @parameterized.expand(
+ [
+ ("matches", ("a.py", "a.py"), "a.py 2\n", 0),
+ ("new_finding", ("a.py", "a.py", "b.py"), "a.py 2\n", 1),
+ ("stale_entry", ("a.py",), "a.py 2\n", 1),
+ ("stale_entry_with_an_error", ("a.py",), "a.py 2\n", 0, [TIMEOUT]),
+ ("new_finding_with_an_error", ("a.py", "a.py", "a.py"), "a.py 2\n", 1, [TIMEOUT]),
+ ]
+ )
+ def test_main_exit_code(
+ self, _name: str, paths: tuple[str, ...], baseline: str, expected: int, errors: list[dict] | None = None
+ ) -> None:
+ with tempfile.TemporaryDirectory() as directory:
+ findings_file = Path(directory) / "findings.json"
+ baseline_file = Path(directory) / "baseline.txt"
+ findings_file.write_text(json.dumps(semgrep_output(*paths, errors=errors)))
+ baseline_file.write_text(baseline)
+
+ exit_code = main(["--findings", str(findings_file), "--baseline", str(baseline_file)])
+
+ assert exit_code == expected
+
+ def test_main_update_writes_a_baseline_that_then_passes(self) -> None:
+ with tempfile.TemporaryDirectory() as directory:
+ findings_file = Path(directory) / "findings.json"
+ baseline_file = Path(directory) / "baseline.txt"
+ findings_file.write_text(json.dumps(semgrep_output("a.py", "b.py", "a.py")))
+ baseline_file.write_text("# header\nold.py 3\n")
+ args = ["--findings", str(findings_file), "--baseline", str(baseline_file)]
+
+ update_exit_code = main([*args, "--update"])
+ check_exit_code = main(args)
+ written = baseline_file.read_text()
+
+ assert (update_exit_code, check_exit_code, written) == (0, 0, "# header\na.py 2\nb.py 1\n")
+
+ def test_main_update_keeps_the_count_of_a_file_with_an_error(self) -> None:
+ with tempfile.TemporaryDirectory() as directory:
+ findings_file = Path(directory) / "findings.json"
+ baseline_file = Path(directory) / "baseline.txt"
+ findings_file.write_text(json.dumps(semgrep_output("b.py", errors=[TIMEOUT])))
+ baseline_file.write_text("a.py 3\nb.py 2\n")
+
+ main(["--findings", str(findings_file), "--baseline", str(baseline_file), "--update"])
+ written = baseline_file.read_text()
+
+ assert written == "a.py 3\nb.py 1\n"
diff --git a/hogli.yaml b/hogli.yaml
index 474be0401b02..20ba6062933f 100644
--- a/hogli.yaml
+++ b/hogli.yaml
@@ -1303,3 +1303,7 @@ environment:
bin_script: rasterize-recording-file
description: 'Render a recording JSONL file to video with the rasterizer'
hidden: true
+ lint:semgrep-ratchet:
+ bin_script: semgrep_ratchet.py
+ description: 'Compare semgrep findings per file with a checked-in baseline'
+ hidden: true
diff --git a/posthog/admin/admins/integration_admin.py b/posthog/admin/admins/integration_admin.py
index d2f77dcf4962..13166cdf4b44 100644
--- a/posthog/admin/admins/integration_admin.py
+++ b/posthog/admin/admins/integration_admin.py
@@ -1,7 +1,9 @@
from django.contrib import admin
+from django.db.models import QuerySet
+from django.http import HttpRequest
from django.utils.html import format_html
-from posthog.models.integration import Integration
+from posthog.models.integration import ANY, Integration
from posthog.models.organization_integration import OrganizationIntegration
from posthog.models.user_integration import UserIntegration
@@ -9,7 +11,7 @@ from posthog.models.user_integration import UserIntegration
@admin.register(Integration)
class IntegrationAdmin(admin.ModelAdmin):
list_select_related = ("team", "created_by")
- list_display = ("id", "kind", "integration_id", "team_link", "created_by", "created_at")
+ list_display = ("id", "kind", "auth_type", "integration_id", "team_link", "created_by", "created_at")
list_display_links = ("id",)
list_filter = (
"kind",
@@ -34,6 +36,12 @@ class IntegrationAdmin(admin.ModelAdmin):
),
]
+ def get_queryset(self, request: HttpRequest) -> QuerySet[Integration]:
+ # The default manager holds only rows with the default auth type of their kind.
+ queryset = Integration.objects._with_auth(ANY)
+ ordering = self.get_ordering(request)
+ return queryset.order_by(*ordering) if ordering else queryset
+
@admin.display(description="Team")
def team_link(self, obj: Integration) -> str:
return format_html("{} ({})", obj.team.name, obj.team_id)
diff --git a/posthog/api/integration.py b/posthog/api/integration.py
index eea14de8a700..3017847779c0 100644
--- a/posthog/api/integration.py
+++ b/posthog/api/integration.py
@@ -77,6 +77,8 @@ from posthog.models.integration import (
ANTHROPIC_DEFAULT_INTEGRATION_ID_PREFIX,
ANTHROPIC_MANAGED_AGENT_LIST_PAGE_LIMIT,
ANTHROPIC_WORKSPACE_LABEL_MAX_LENGTH,
+ ANY,
+ DEFAULT,
ERROR_TOKEN_REFRESH_FAILED,
GITHUB_REPOSITORY_REFRESH_COOLDOWN_SECONDS,
POSTHOG_CONNECT_ALLOWED_REGIONS,
@@ -107,7 +109,9 @@ from posthog.models.integration import (
GoogleCloudIntegration,
GoogleCloudServiceAccountIntegration,
Integration,
+ IntegrationAuthType,
IntegrationError,
+ IntegrationQuerySet,
JiraIntegration,
LinearIntegration,
LinkedInAdsIntegration,
@@ -124,6 +128,7 @@ from posthog.models.integration import (
defer_repository_cache_fields,
resolve_aliased_oauth_kind,
)
+from posthog.models.integration.auth_types import rows_without_source_marker_q
from posthog.models.integration.github_audit import GitHubAudit
from posthog.models.integration.stripe import stripe_signing_secret
from posthog.models.integration.twitter_ads import TwitterAdsIntegration
@@ -769,6 +774,15 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial
"""Standard Integration serializer."""
created_by = UserBasicSerializer(read_only=True)
+ auth_type = serializers.ChoiceField(
+ choices=IntegrationAuthType.choices,
+ read_only=True,
+ help_text=(
+ "How this integration authenticates with the provider: `oauth` for a connection the provider manages "
+ "(an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several "
+ "typed-in fields. A kind can have several auth types."
+ ),
+ )
files_write_requestable = serializers.SerializerMethodField(
help_text="Slack only: whether reconnecting can request the files:write scope."
)
@@ -791,6 +805,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial
fields = [
"id",
"kind",
+ "auth_type",
"config",
"created_at",
"created_by",
@@ -802,6 +817,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial
]
read_only_fields = [
"id",
+ "auth_type",
"created_at",
"created_by",
"errors",
@@ -813,11 +829,15 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial
@extend_schema_field(serializers.BooleanField())
def get_files_write_requestable(self, obj: Integration) -> bool:
- return obj.kind == "slack" and "files:write" in POSTHOG_SLACK_SCOPE.split(",")
+ return (
+ obj.kind == "slack"
+ and obj.auth_type == IntegrationAuthType.OAUTH
+ and "files:write" in POSTHOG_SLACK_SCOPE.split(",")
+ )
@extend_schema_field(serializers.BooleanField(allow_null=True))
def get_installation_shared(self, obj: Integration) -> bool | None:
- if obj.kind != "github" or not obj.integration_id:
+ if obj.kind != "github" or obj.auth_type != IntegrationAuthType.OAUTH or not obj.integration_id:
return None
# Mirrors the check in IntegrationViewSet.perform_destroy: only other *team* rows keep the App
# installed on GitHub. Personal rows don't count because destroy deletes them along with it.
@@ -829,7 +849,7 @@ class IntegrationSerializer(serializers.ModelSerializer, UserAccessControlSerial
@extend_schema_field(serializers.ChoiceField(choices=GITHUB_INSTALLATION_STATUS_CHOICES, allow_null=True))
def get_installation_status(self, obj: Integration) -> str | None:
- if obj.kind != "github":
+ if obj.kind != "github" or obj.auth_type != IntegrationAuthType.OAUTH:
return None
return (
"unavailable"
@@ -1557,7 +1577,8 @@ class IntegrationViewSet(
# page boundary. Clients page this list to find one kind, so a dropped row reads as
# "not configured". Order oldest-first: several clients take the first row of a kind as their
# default connection.
- queryset = defer_repository_cache_fields(Integration.objects.all()).order_by("created_at", "id")
+ # Lists every auth type, so typed-in credentials show next to installations and OAuth grants.
+ queryset = defer_repository_cache_fields(Integration.objects._with_auth(ANY)).order_by("created_at", "id")
serializer_class = IntegrationSerializer
filter_backends = [DjangoFilterBackend]
filterset_fields = ["kind"]
@@ -1568,7 +1589,14 @@ class IntegrationViewSet(
# creator. Unlike team-shared integrations, don't expose other members' connections in
# list/retrieve — scope `posthog` rows to the requesting user. Other kinds stay team-visible.
user_id = getattr(self.request.user, "id", None)
- return queryset.exclude(Q(kind=POSTHOG_CONNECT_KIND) & ~Q(created_by_id=user_id))
+ queryset = queryset.exclude(Q(kind=POSTHOG_CONNECT_KIND) & ~Q(created_by_id=user_id))
+ # The other actions (destroy, channels, github_repos, ...) call provider code that reads the
+ # row in the default shape of its kind, so they keep to rows with the default auth type, and
+ # skip warehouse source rows, the same as the implicit filter of `Integration.objects`.
+ if self.action not in ("list", "retrieve"):
+ assert isinstance(queryset, IntegrationQuerySet)
+ queryset = queryset._with_auth(DEFAULT).filter(rows_without_source_marker_q())
+ return queryset
def handle_exception(self, exc: Exception) -> Response:
# GitHub rate limits surface from any GitHub-backed action (teams, repos, branches, refresh);
@@ -2974,17 +3002,18 @@ class IntegrationViewSet(
queryset = self.filter_queryset(self.get_queryset())
page = self.paginate_queryset(queryset)
instances = page if page is not None else list(queryset)
- for instance in instances:
- if instance.kind == "github":
- GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()
+ # Only App installations have an installation id to heal from. A typed-in token row does not.
+ github_apps = [
+ instance
+ for instance in instances
+ if instance.kind == "github" and instance.auth_type == IntegrationAuthType.OAUTH
+ ]
+ for instance in github_apps:
+ GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()
# One grouped lookup for the whole page instead of an `exists()` per row, so
# `installation_shared` stays a fixed one query however many GitHub rows there are.
reference_counts = GitHubIntegration.installation_reference_counts(
- {
- instance.integration_id
- for instance in instances
- if instance.kind == "github" and instance.integration_id
- },
+ {instance.integration_id for instance in github_apps if instance.integration_id},
include_personal=False,
)
serializer = self.get_serializer(
@@ -3000,6 +3029,6 @@ class IntegrationViewSet(
# Inlined from RetrieveModelMixin too: deferring to it would re-run `get_object`, paying a
# second query and permission check for the row already in hand.
instance = self.get_object()
- if instance.kind == "github":
+ if instance.kind == "github" and instance.auth_type == IntegrationAuthType.OAUTH:
GitHubIntegration(instance, source="integrations_api", priority=Priority.NORMAL).ensure_account_name()
return Response(self.get_serializer(instance).data)
diff --git a/posthog/api/test/test_integration.py b/posthog/api/test/test_integration.py
index fb9b916f7fc2..93d291f70c29 100644
--- a/posthog/api/test/test_integration.py
+++ b/posthog/api/test/test_integration.py
@@ -3272,6 +3272,39 @@ class TestIntegrationAPIKeyAccess:
assert "github" in kinds
assert "twilio" in kinds
+ def test_list_and_retrieve_show_every_auth_type(self, client: HttpClient):
+ client.force_login(self.user)
+ token_row = Integration.objects._with_auth(Integration.AuthType.TOKEN).create(
+ team=self.team, kind="github", integration_id="typed-in", sensitive_config={"token": "secret"}
+ )
+ source_row = Integration.objects._with_auth(Integration.AuthType.CREDENTIALS).create(
+ team=self.team, kind="postgresql", integration_id="source", config={"warehouse_source_type": "Postgres"}
+ )
+ base_url = f"/api/environments/{self.team.pk}/integrations"
+
+ with patch.object(GitHubIntegration, "ensure_account_name") as heal:
+ listed = client.get(f"{base_url}/").json()["results"]
+ retrieved = client.get(f"{base_url}/{token_row.id}/")
+ repos = client.get(f"{base_url}/{token_row.id}/github_repos/")
+
+ assert {(row["id"], row["auth_type"]) for row in listed} == {
+ (self.github_integration.id, "oauth"),
+ (self.twilio_integration.id, "token"),
+ (token_row.id, "token"),
+ (source_row.id, "credentials"),
+ }
+ # Code that does not state its auth types does not see the warehouse source row.
+ assert not Integration.objects.filter(team=self.team, kind="postgresql").exists()
+ assert retrieved.status_code == status.HTTP_200_OK
+ assert (retrieved.json()["auth_type"], retrieved.json()["installation_status"]) == ("token", None)
+ assert repos.status_code == status.HTTP_404_NOT_FOUND
+ # Only the App row of the list heals its account name. The token row has no installation.
+ assert heal.call_count == 1
+ # Actions other than list and retrieve do not reach the warehouse source row.
+ deleted = client.delete(f"{base_url}/{source_row.id}/")
+ assert deleted.status_code == status.HTTP_404_NOT_FOUND
+ assert Integration._base_manager.filter(id=source_row.id).exists()
+
def test_list_integrations_filtered_by_kind(self, client: HttpClient):
client.force_login(self.user)
diff --git a/posthog/integrations/AGENTS.md b/posthog/integrations/AGENTS.md
new file mode 100644
index 000000000000..4c1cb0443230
--- /dev/null
+++ b/posthog/integrations/AGENTS.md
@@ -0,0 +1,13 @@
+# Integrations SDK: rules for callers
+
+Read `README.md` before you change the SDK or add an operation.
+
+- Read and write `Integration` rows only through `IntegrationManager` from `posthog.integrations`. Never import the `Integration` model, its managers or its relations. The semgrep ratchet `integration-model-through-sdk` fails CI on new uses.
+- Pick the accessor by kind and auth type: `IntegrationManager.<kind>.<oauth|token|credentials>` for code that reads one config shape. Use `.any` or `IntegrationManager.kinds(...)` only for code that handles every auth type the same way, such as a picker or a delete.
+- Rows are read-only `IntegrationRecord`s. Do not call `.save()`, `.delete()` or `.refresh_from_db()` on a row, and do not write to `row.config`. Write with `update` or `upsert`, and delete with `delete`.
+- Do not read related rows (`row.team`, `row.created_by`). Use `row.team_id` and `row.created_by_id`. Set a foreign key with its id column (`integration_id=row.id`).
+- Warehouse source rows (`config["warehouse_source_type"]`) are hidden by default. Add `.with_source_rows()` only in code that handles every row the same way, such as the integrations list and delete API. Use `IntegrationManager.source(kind)` for the source credentials themselves.
+- Refresh an OAuth token with `IntegrationManager.<kind>.oauth.refresh_access_token(team_id, id, stale_access_token=...)`. Do not lock a row yourself.
+- When an operation is missing, add a small named operation to the SDK. Do not widen `find(where=...)`, which compares named config keys for equality only, and never return a queryset.
+- The model moves behind an HTTP API in the credentials service. Every operation must work over that transport: no callbacks into caller code inside a transaction, no querysets, no model instances that callers write.
+- When you move a file to the SDK, lower its count in `model_access_baseline.txt`, or remove the line at zero.
diff --git a/posthog/integrations/CLAUDE.md b/posthog/integrations/CLAUDE.md
new file mode 120000
index 000000000000..47dc3e3d863c
--- /dev/null
+++ b/posthog/integrations/CLAUDE.md
@@ -0,0 +1 @@
+AGENTS.md
\ No newline at end of file
diff --git a/posthog/integrations/README.md b/posthog/integrations/README.md
new file mode 100644
index 000000000000..6a2c9c3c0215
--- /dev/null
+++ b/posthog/integrations/README.md
@@ -0,0 +1,171 @@
+# Integrations SDK
+
+The integrations SDK is the only way to read and write `Integration` rows (`posthog/models/integration/model.py`).
+The rows move to a separate credentials service soon.
+The SDK is a small, closed set of operations, so only its transport changes after the move.
+No operation returns a queryset, and no operation runs caller code inside its transaction.
+
+```python
+from posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationRecord
+from posthog.integrations.fields import IntegrationReferenceField
+```
+
+The rules for callers are in `AGENTS.md`.
+
+| Module | Holds |
+| -------------------- | -------------------------------------------------------------------------- |
+| `__init__.py` | The public names |
+| `accessors.py` | The operations |
+| `catalogue.py` | The kinds and their auth types (hand-written) |
+| `kinds_generated.py` | One accessor class per kind, generated from the catalogue |
+| `contracts.py` | `IntegrationRecord`, `IntegrationPatch`, `IntegrationUsage` and the errors |
+| `source_kinds.py` | The registry of warehouse source kinds |
+| `usages.py` | The registry of usage providers |
+| `fields.py` | `IntegrationReferenceField` for DRF serializers |
+
+The model module imports `catalogue.py`, `contracts.py` and `enums.py`. These modules must not import the model.
+`__init__.py` loads `IntegrationManager` on first use for the same reason.
+
+## Read rows
+
+An accessor fixes the kind, the auth type, or both. Each operation takes `team_id` first and returns rows in the scope of the accessor only.
+
+| Accessor | Rows | Operations |
+| ---------------------------------------- | ------------------------------------------------------ | ---------------- |
+| `IntegrationManager.github.oauth` | One kind with one auth type | Reads and writes |
+| `IntegrationManager.stripe.any` | One kind with every auth type | Reads and delete |
+| `IntegrationManager.kinds("slack", ...)` | Several kinds, with the default auth type of each kind | Reads and delete |
+| `IntegrationManager.any` | Every kind with every auth type | Reads and delete |
+| `IntegrationManager.source(kind).token` | Warehouse source rows of one kind with one auth type | Reads and writes |
+| `IntegrationManager.source(kind).any` | Warehouse source rows of one kind with every auth type | Reads and delete |
+
+Only `IntegrationManager.source(kind)` reads warehouse source rows (rows with `config["warehouse_source_type"]`).
+The other accessors skip them, because their config has the shape of the warehouse source, not of the kind.
+Code that handles every row the same way, such as the integrations list and delete API, adds them with `with_source_rows()`:
+
+```python
+IntegrationManager.any.with_source_rows().delete(team_id, integration_id)
+IntegrationManager.slack.any.with_source_rows().list(team_id)
+IntegrationManager.kinds("github", "stripe", auth_types="any").with_source_rows().get_many(team_id, ids)
+```
+
+Reads:
+
+- `get(team_id, id)` returns the row, or raises `IntegrationNotFound` when the row is not in the scope.
+- `get_many(team_id, ids)` returns `{id: row}` for the ids in the scope.
+- `find(team_id, external_id=..., where={...})` returns the oldest matching row or `None`.
+- `list(team_id, external_id=..., where={...}, newest_first=False)` and `exists(...)`.
+- `find` and `exists` also take `team_ids=[...]` instead of `team_id`, for example the teams of an organization. Pass exactly one of the two. An empty `team_ids` matches no row.
+- `usages(team_id, id)` returns the PostHog objects that use the row.
+
+These reads go across teams, for example to the teams that share one GitHub installation:
+
+- `list_by_external_id(external_id, team_ids=None)` returns the rows with this external id.
+- `get_by_external_id(external_id, id)` returns the row with this id and this external id, or raises `IntegrationNotFound`.
+- `count_by_external_id(external_id)` and `exists_by_external_id(external_id)`.
+
+They need an accessor with a kind: on `IntegrationManager.any` they raise `ValueError`, because external ids are unique only per provider.
+Do not take the first row of `list_by_external_id` without `team_ids`: the rule `github-installation-first-team` flags it for GitHub.
+
+`IntegrationManager.kinds(...)` takes `auth_types`: `"default"` (the default auth type of each kind), `"any"` for every auth type, or a list of auth types.
+
+Reads do not load the GitHub repository cache columns (`repository_cache`, `repository_cache_updated_at`). Django loads them with one more query when code reads them.
+
+`where` compares named config keys for equality, for example `where={"region": "eu"}`.
+Equality also compares the JSON type, so `where={"installation_id": "123"}` misses a row that stores the number `123`.
+For a GitHub installation, use the external id, which always holds the installation id as a string: `IntegrationManager.github.oauth.find(team_id, external_id=str(installation_id))`.
+It does not accept lookups (`a__b`), nested values or the keys that the SDK writes (`auth_type`, `warehouse_source_type`), and raises `UnsupportedIntegrationFilter`.
+When you need more, add a named method to the SDK.
+
+Each operation has an async form with an `a` prefix: `aget`, `alist`, `aupdate`, and so on.
+
+## Write rows
+
+Only an accessor with one auth type can write, because a new row has exactly one auth type.
+
+- `create(team_id, external_id=..., config=..., sensitive_config=..., created_by_id=...)`. It raises `IntegrationConflict` when the team already has a row of the kind with this external id, of any auth type.
+- `upsert(team_id, external_id, config=..., ...)` returns `(row, created)`. It replaces the config of an existing row. It raises `IntegrationConflict` when the existing row is not in the scope, for example a row with another auth type.
+- `update(team_id, id, config_patch=..., config_remove=..., sensitive_config_patch=..., errors=...)` merges a patch. `errors=None` keeps the errors.
+- `update_by_external_id(external_id, patch)` patches the rows of every team that are in the scope. It writes only the rows that the patch changes, in one `bulk_update`, and returns their count. A `bulk_update` sends no `post_save` signal.
+
+Delete rows with `delete(team_id, id)`. It raises `IntegrationInUse` with the usages when other objects use the row. Pass `ignore_usages=True` to delete it anyway.
+`delete_by_external_id(external_id)` deletes the rows of every team in one transaction. It deletes each row on its own, so the delete signals fire. Like `list_by_external_id`, it needs an accessor with a kind.
+
+An OAuth accessor (`IntegrationManager.<kind>.oauth`) also refreshes access tokens:
+
+- `refresh_access_token(team_id, id, stale_access_token=None)` locks the row, refreshes the token when it expires soon, saves the new tokens and returns the row.
+ Pass the token that the provider refused as `stale_access_token`, to refresh also when it is still the stored token.
+ When another worker already stored a newer token, the row comes back without a refresh.
+ The lock stops two workers from spending the same refresh token, which matters for providers that rotate refresh tokens (Resend, Jira).
+ A refresh that fails does not raise: the row comes back with `errors` set.
+ It raises `ValueError` for a kind that `OauthIntegration` cannot refresh.
+
+## Rows are read-only
+
+The SDK returns rows typed as `IntegrationRecord`, a read-only `Protocol`:
+`id`, `team_id`, `kind`, `integration_id`, `config`, `sensitive_config`, `errors`, `created_at`, `created_by_id`, `auth_type`, `access_token`, `refresh_token` and `can_be_managed_by_creator(user_id)`.
+`config` and `sensitive_config` are `Mapping`s, so mypy rejects `row.config["x"] = 1`.
+Mypy also rejects `.save()`, `.delete()`, `.refresh_from_db()`, managers and related rows such as `row.team`.
+Use `row.team_id` and `row.created_by_id`, and load the team or the user from their own models.
+Write with `update` or `upsert`. Read the row again with `get` after a change.
+To set a foreign key to a row, set the id column: `task.github_integration_id = row.id`.
+
+At runtime the rows are still model instances. A later change returns frozen records with the same reads, and callers do not change.
+The per-kind adapters next to the model (`SlackIntegration`, `GitHubIntegration`, `OauthIntegration`, ...) take an `IntegrationRecord` and narrow it to the model with `integration_model`, because they still write the row.
+
+## Pick an auth type
+
+Each kind supports some of three auth types, and has one default:
+
+| Auth type | Use it for |
+| ------------- | ----------------------------------------------------------------------------------------------- |
+| `oauth` | A connection that the provider manages: an OAuth grant or an app installation |
+| `token` | One static secret: an API key, a personal access token, a bot token or a webhook secret |
+| `credentials` | Several typed-in fields: host, user and password, an access key and a secret, a service account |
+
+Use the auth type of the rows that your code can handle.
+A kind class has an attribute only for the auth types that the kind supports, so mypy rejects `IntegrationManager.github.credentials`.
+Use `.any` only when your code handles every auth type of the kind the same way, for example a picker or a delete.
+
+A row stores its auth type in `config["auth_type"]` only when it is not the default of its kind. The SDK writes that key. Do not write it yourself.
+
+## Add a kind
+
+1. Add the kind to `Integration.IntegrationKind` in `posthog/models/integration/model.py`.
+2. Add the kind to `KINDS` in `catalogue.py`, with its default auth type and its other auth types.
+3. Run `hogli build:projections --only integration-kinds` and commit `kinds_generated.py`.
+
+Use `IntegrationKind` from `posthog.integrations` for kind values in code, not `Integration.IntegrationKind`.
+
+The default of a kind is stored data. Do not change the default of a kind that has rows.
+
+Warehouse source kinds are not in the catalogue. The warehouse product registers them at startup with `register_source_kinds`, and callers reach their rows through `IntegrationManager.source(kind)`.
+A warehouse source row has `config["warehouse_source_type"]`. Every other accessor skips these rows unless you call `with_source_rows()`, and `IntegrationManager.source(kind)` reads only these rows.
+
+## Report usages
+
+A product that uses integration rows reports them from `products/<product>/backend/integration_usages.py`:
+
+```python
+def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]: ...
+```
+
+The SDK finds the module on the first call to `usages` or `delete`. Code outside a product calls `register_usage_provider` instead.
+
+## Reference a row from a serializer
+
+Use `IntegrationReferenceField` from `posthog.integrations.fields` instead of a `PrimaryKeyRelatedField` on `Integration`.
+It reads the team from the serializer context, so a row of another team does not exist for the field.
+Pass `kinds=[...]` to accept only rows of these kinds.
+With `kinds`, the field accepts only the default auth type of each kind, because code that reads a kind expects its usual config shape.
+Pass `auth_types=[AuthType.TOKEN]` to accept other auth types, or `auth_types="any"` to accept every auth type.
+The field never accepts a warehouse source row.
+
+## Why direct model use fails CI
+
+The semgrep rule `.semgrep/rules/devex/integration-model-through-sdk.yaml` finds uses of the model outside the SDK: imports of `Integration`, its managers (`objects`, `_default_manager`, `_base_manager`, also through `IntegrationRecord`), `apps.get_model` lookups of it, relation fields to it, `select_related` and `prefetch_related` on an integration relation, `integration__` lookups (as keyword arguments or as strings), `integration_set`, and `.save()` on a row.
+The "Integration model ratchet" step in `.github/workflows/ci-security.yaml` compares the findings per file with `model_access_baseline.txt`.
+It fails when a file has more findings than its baseline count, and also when it has fewer.
+A file that semgrep reports an error for (a timeout or a parse error) is not checked for fewer findings, and `--update` keeps its count.
+When you move a file to the SDK, lower its count in the baseline, or remove the line at zero.
+Tests, migrations, Django admin, the model and its per-kind adapters (`posthog/models/integration/`) are outside the rule.
diff --git a/posthog/integrations/__init__.py b/posthog/integrations/__init__.py
new file mode 100644
index 000000000000..a2d7fc1c878f
--- /dev/null
+++ b/posthog/integrations/__init__.py
@@ -0,0 +1,92 @@
+"""The integrations SDK: the only way to read and write `Integration` rows. See README.md and AGENTS.md.
+
+from posthog.integrations import IntegrationManager, IntegrationNotFound, IntegrationRecord
+from posthog.integrations.fields import IntegrationReferenceField
+"""
+
+import importlib
+from typing import TYPE_CHECKING, Any
+
+from posthog.integrations.contracts import (
+ SOURCE_MARKER_CONFIG_KEY,
+ ConfigValue,
+ IntegrationConflict,
+ IntegrationInUse,
+ IntegrationNotFound,
+ IntegrationPatch,
+ IntegrationRecord,
+ IntegrationUsage,
+ KindSpec,
+ UnknownIntegrationKind,
+ UnsupportedAuthType,
+ UnsupportedIntegrationFilter,
+)
+from posthog.integrations.enums import AuthType, IntegrationAuthType
+
+if TYPE_CHECKING:
+ from posthog.integrations.kinds_generated import IntegrationKind
+ from posthog.integrations.manager import (
+ AuthAccessor,
+ IntegrationManager,
+ IntegrationReader,
+ OAuthAccessor,
+ SourceKindAccessor,
+ UsageProvider,
+ register_source_kinds,
+ register_usage_provider,
+ )
+
+# Loaded on first use (PEP 562). The model module imports the catalogue from this package, and
+# importing a submodule runs this file first, so an eager import of the accessors (which import
+# the model) would be a circular import.
+_LAZY_NAMES = {
+ "IntegrationKind": "posthog.integrations.kinds_generated",
+ **dict.fromkeys(
+ (
+ "AuthAccessor",
+ "IntegrationManager",
+ "IntegrationReader",
+ "OAuthAccessor",
+ "SourceKindAccessor",
+ "UsageProvider",
+ "register_source_kinds",
+ "register_usage_provider",
+ ),
+ "posthog.integrations.manager",
+ ),
+}
+
+
+def __getattr__(name: str) -> Any:
+ if name not in _LAZY_NAMES:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ value = getattr(importlib.import_module(_LAZY_NAMES[name]), name)
+ globals()[name] = value
+ return value
+
+
+__all__ = [
+ "SOURCE_MARKER_CONFIG_KEY",
+ "AuthAccessor",
+ "AuthType",
+ "ConfigValue",
+ "IntegrationAuthType",
+ "IntegrationConflict",
+ "IntegrationInUse",
+ "IntegrationKind",
+ "IntegrationManager",
+ "IntegrationNotFound",
+ "IntegrationPatch",
+ "IntegrationReader",
+ "IntegrationRecord",
+ "IntegrationUsage",
+ "KindSpec",
+ "OAuthAccessor",
+ "SourceKindAccessor",
+ "UnknownIntegrationKind",
+ "UnsupportedAuthType",
+ "UnsupportedIntegrationFilter",
+ "UsageProvider",
+ "register_source_kinds",
+ "register_usage_provider",
+]
diff --git a/posthog/integrations/accessors.py b/posthog/integrations/accessors.py
new file mode 100644
index 000000000000..e113cef98834
--- /dev/null
+++ b/posthog/integrations/accessors.py
@@ -0,0 +1,676 @@
+"""The operations of the integrations SDK.
+
+An accessor holds a scope: the kinds, the auth type, and whether it sees warehouse source rows. Every
+operation reads or writes only rows in that scope. No accessor sees warehouse source rows unless the
+caller asks for them with `with_source_rows()` or uses `IntegrationManager.source(kind)`.
+`IntegrationReader` has the reads and the deletes. `AuthAccessor` adds the writes, and exists only for
+one kind with one auth type, because a new row has exactly one auth type. `OAuthAccessor` adds the
+token refresh.
+
+No operation returns a queryset, and no operation runs caller code inside its transaction. The
+integration rows move to a separate service later, so the operations here are the whole contract, and
+only their transport changes. Rows come back typed as the read-only `IntegrationRecord`.
+"""
+
+import re
+import json
+import builtins
+from collections.abc import Iterable, Mapping
+from enum import Enum
+from typing import ClassVar, Literal
+
+from django.db import IntegrityError, transaction
+from django.db.models import Q
+
+from posthog.integrations.catalogue import KINDS
+from posthog.integrations.contracts import (
+ SOURCE_MARKER_CONFIG_KEY,
+ ConfigValue,
+ IntegrationConflict,
+ IntegrationInUse,
+ IntegrationNotFound,
+ IntegrationPatch,
+ IntegrationRecord,
+ IntegrationUsage,
+ UnknownIntegrationKind,
+ UnsupportedAuthType,
+ UnsupportedIntegrationFilter,
+)
+from posthog.integrations.enums import AuthType
+from posthog.integrations.source_kinds import is_source_kind
+from posthog.integrations.usages import integration_usages
+from posthog.models.integration.auth_types import ANY, AUTH_TYPE_CONFIG_KEY, default_auth_type, kind_auth_types
+from posthog.models.integration.model import Integration, IntegrationQuerySet, defer_repository_cache_fields
+from posthog.models.integration.oauth import OauthIntegration
+from posthog.sync import database_sync_to_async
+
+# The SDK writes these keys itself. A caller cannot set, remove or filter on them.
+RESERVED_CONFIG_KEYS = frozenset({AUTH_TYPE_CONFIG_KEY, SOURCE_MARKER_CONFIG_KEY})
+
+_CONFIG_KEY = re.compile(r"^[A-Za-z][A-Za-z0-9_]*$")
+# The unique key of a row: (team, kind, external id).
+_UNIQUE_KEY_CONSTRAINT = "posthog_integration_kind_id_unique"
+_SCALAR_TYPES = (str, int, float, bool, type(None))
+
+
+class SourceRows(Enum):
+ # Named accessors skip warehouse source rows, so code that reads a kind in its usual shape never
+ # gets a row with typed-in source credentials.
+ EXCLUDE = "exclude"
+ ONLY = "only"
+ INCLUDE = "include"
+
+
+def _rows() -> IntegrationQuerySet:
+ # The GitHub repository cache can be large. Django loads a deferred column when code reads it.
+ return defer_repository_cache_fields(Integration.objects._with_auth(ANY))
+
+
+# The auth types an accessor reads: one auth type, several, or the default auth type of each kind.
+AuthTypes = AuthType | frozenset[AuthType] | Literal["default"]
+
+
+def _auth_type_q(kind: str, auth_type: AuthType) -> Q:
+ stored = Q(config__auth_type=auth_type.value)
+ if auth_type == default_auth_type(kind):
+ # A JSON null counts as unset, the same as in `auth_types.rows_with_default_auth_type_q`. A
+ # stored default is possible too: `QuerySet.update()` skips the `normalized_config` of `save()`.
+ return Q(config__auth_type__isnull=True) | Q(config__auth_type=None) | stored
+ return stored
+
+
+def _where_q(where: Mapping[str, ConfigValue] | None) -> Q:
+ q = Q()
+ for key, value in (where or {}).items():
+ if not isinstance(key, str) or not _CONFIG_KEY.match(key) or "__" in key or key in RESERVED_CONFIG_KEYS:
+ raise UnsupportedIntegrationFilter(f"Cannot filter integrations on the config key {key!r}")
+ if not isinstance(value, _SCALAR_TYPES):
+ raise UnsupportedIntegrationFilter(f"Cannot compare the config key {key!r} with a {type(value).__name__}")
+ # `__exact`, so a key named like a lookup (`contains`, `regex`) still reads as a key.
+ # nosemgrep: orm-field-injection - the key is one checked JSON key segment, and `__exact` fixes the lookup
+ q &= Q(**{f"config__{key}__exact": value})
+ return q
+
+
+def _is_unique_key_violation(error: IntegrityError) -> bool:
+ constraint = getattr(getattr(error.__cause__, "diag", None), "constraint_name", None)
+ return constraint == _UNIQUE_KEY_CONSTRAINT or _UNIQUE_KEY_CONSTRAINT in str(error)
+
+
+def _checked_config_keys(keys: Iterable[str]) -> None:
+ reserved = sorted(RESERVED_CONFIG_KEYS.intersection(keys))
+ if reserved:
+ raise ValueError(f"The integrations SDK writes {reserved} itself")
+
+
+def _patched(
+ current: Mapping[str, object] | None, patch: Mapping[str, object], remove: Iterable[str]
+) -> dict[str, object]:
+ result = dict(current or {})
+ result.update(patch)
+ for key in remove:
+ result.pop(key, None)
+ return result
+
+
+class IntegrationReader:
+ """Reads and deletes of the integration rows in one scope."""
+
+ def __init__(
+ self,
+ *,
+ kinds: frozenset[str] | None,
+ auth_type: AuthTypes | None = None,
+ source_rows: SourceRows = SourceRows.EXCLUDE,
+ ) -> None:
+ """`auth_type=None` reads every auth type. `"default"` reads the default auth type of each kind."""
+ if auth_type is not None and not kinds:
+ raise ValueError("An accessor with an auth type needs its kinds")
+ self._kinds = kinds
+ self._auth_type = auth_type
+ self._source_rows = source_rows
+
+ def _auth_types_of(self, kind: str) -> builtins.list[AuthType]:
+ if self._auth_type == "default":
+ return [default_auth_type(kind)]
+ if isinstance(self._auth_type, AuthType):
+ return [self._auth_type]
+ if isinstance(self._auth_type, frozenset):
+ return sorted(self._auth_type)
+ return []
+
+ def __repr__(self) -> str:
+ kinds = sorted(self._kinds) if self._kinds is not None else "every kind"
+ return f"{type(self).__name__}({kinds}, {self._auth_type or 'any auth type'}, {self._source_rows.value})"
+
+ def _scope_q(self) -> Q:
+ q = Q()
+ if self._kinds is not None:
+ if self._auth_type is None:
+ q &= Q(kind__in=sorted(self._kinds))
+ else:
+ # An empty Q() in an OR matches every row, so start from a clause that matches none.
+ by_kind = Q(pk__in=[])
+ for kind in sorted(self._kinds):
+ for auth_type in self._auth_types_of(kind):
+ by_kind |= Q(kind=kind) & _auth_type_q(kind, auth_type)
+ q &= by_kind
+ if self._source_rows == SourceRows.EXCLUDE:
+ q &= ~Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)
+ elif self._source_rows == SourceRows.ONLY:
+ q &= Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)
+ return q
+
+ def _cross_team_rows(self, operation: str) -> IntegrationQuerySet:
+ # A cross-team operation without a kind would reach every row with the external id, of any
+ # provider, in every team. External ids are only unique per provider.
+ if self._kinds is None:
+ raise ValueError(f"{operation} needs an accessor with a kind, such as IntegrationManager.github.oauth")
+ return _rows().filter(self._scope_q())
+
+ def _team_rows(self, team_id: int) -> IntegrationQuerySet:
+ return _rows().filter(self._scope_q(), team_id=team_id)
+
+ def _rows_of_teams(self, team_id: int | None, team_ids: Iterable[int] | None) -> IntegrationQuerySet:
+ if (team_id is None) == (team_ids is None):
+ raise ValueError("Pass exactly one of team_id and team_ids")
+ if team_id is not None:
+ return self._team_rows(team_id)
+ # An empty list matches no row.
+ return _rows().filter(self._scope_q(), team_id__in=builtins.list(team_ids or ()))
+
+ def _matching(
+ self,
+ team_id: int | None,
+ external_id: str | None,
+ where: Mapping[str, ConfigValue] | None,
+ team_ids: Iterable[int] | None = None,
+ ) -> IntegrationQuerySet:
+ rows = self._rows_of_teams(team_id, team_ids).filter(_where_q(where))
+ return rows if external_id is None else rows.filter(integration_id=external_id)
+
+ def with_source_rows(self) -> "IntegrationReader":
+ """A reader of the same scope that also sees warehouse source rows.
+
+ Use it only for code that handles every row the same way, such as the integrations list and
+ delete API. Code that reads the config of a kind must not see source rows: their config has the
+ shape of the warehouse source, not of the kind.
+ """
+ if self._source_rows == SourceRows.ONLY:
+ raise ValueError("This accessor reads only warehouse source rows")
+ return IntegrationReader(kinds=self._kinds, auth_type=self._auth_type, source_rows=SourceRows.INCLUDE)
+
+ def _row(self, team_id: int, id: int, *, lock: bool = False) -> Integration:
+ rows = self._team_rows(team_id)
+ try:
+ return (rows.select_for_update() if lock else rows).get(id=id)
+ except (Integration.DoesNotExist, ValueError, TypeError) as error:
+ raise IntegrationNotFound(f"No integration {id!r} in {self!r} for team {team_id}") from error
+
+ def get(self, team_id: int, id: int) -> IntegrationRecord:
+ """The row with this id. Raises `IntegrationNotFound` when the row is not in the scope."""
+ return self._row(team_id, id)
+
+ def get_many(self, team_id: int, ids: Iterable[int]) -> dict[int, IntegrationRecord]:
+ """The rows with these ids, keyed by id. Ids that are not in the scope are missing from the result."""
+ return {row.id: row for row in self._team_rows(team_id).filter(id__in=list(ids))}
+
+ def find(
+ self,
+ team_id: int | None = None,
+ *,
+ team_ids: Iterable[int] | None = None,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ ) -> IntegrationRecord | None:
+ """The oldest row with this external id and these config values, or None.
+
+ Pass `team_id` or `team_ids`, not both. An empty `team_ids` finds nothing.
+ """
+ return self._matching(team_id, external_id, where, team_ids).order_by("created_at", "id").first()
+
+ def list(
+ self,
+ team_id: int,
+ *,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ newest_first: bool = False,
+ ) -> builtins.list[IntegrationRecord]:
+ order = ("-created_at", "-id") if newest_first else ("created_at", "id")
+ return list(self._matching(team_id, external_id, where).order_by(*order))
+
+ def exists(
+ self,
+ team_id: int | None = None,
+ *,
+ team_ids: Iterable[int] | None = None,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ ) -> bool:
+ """Pass `team_id` or `team_ids`, not both. An empty `team_ids` is False."""
+ return self._matching(team_id, external_id, where, team_ids).exists()
+
+ def list_by_external_id(
+ self, external_id: str, *, team_ids: Iterable[int] | None = None
+ ) -> builtins.list[IntegrationRecord]:
+ """Rows of every team with this external id, for example the teams that share one GitHub installation.
+
+ Raises `ValueError` on `IntegrationManager.any`, because the result would mix providers.
+ """
+ rows = self._cross_team_rows("list_by_external_id").filter(integration_id=external_id)
+ if team_ids is not None:
+ rows = rows.filter(team_id__in=list(team_ids))
+ return list(rows.order_by("created_at", "id"))
+
+ def get_by_external_id(self, external_id: str, id: int) -> IntegrationRecord:
+ """The row with this id and this external id, in any team.
+
+ For a callback that names both, such as a GitHub installation and one of its rows. Raises
+ `IntegrationNotFound` when the row has another external id or is not in the scope.
+ """
+ rows = self._cross_team_rows("get_by_external_id")
+ try:
+ return rows.get(integration_id=external_id, id=id)
+ except (Integration.DoesNotExist, ValueError, TypeError) as error:
+ raise IntegrationNotFound(f"No integration {id!r} with external id {external_id!r} in {self!r}") from error
+
+ def count_by_external_id(self, external_id: str) -> int:
+ """The number of rows of every team with this external id."""
+ return self._cross_team_rows("count_by_external_id").filter(integration_id=external_id).count()
+
+ def exists_by_external_id(self, external_id: str) -> bool:
+ """True when a row of any team has this external id."""
+ return self._cross_team_rows("exists_by_external_id").filter(integration_id=external_id).exists()
+
+ def usages(self, team_id: int, id: int) -> builtins.list[IntegrationUsage]:
+ """The PostHog objects of the team that use the row. See `usages.py`."""
+ row = self.get(team_id, id)
+ return integration_usages(team_id, row.id)
+
+ def delete(self, team_id: int, id: int, *, ignore_usages: bool = False) -> None:
+ """Delete the row. Raises `IntegrationInUse` when other objects use it, unless `ignore_usages`."""
+ row = self._row(team_id, id)
+ if not ignore_usages:
+ usages = integration_usages(team_id, row.id)
+ if usages:
+ raise IntegrationInUse(usages)
+ row.delete()
+
+ def delete_by_external_id(self, external_id: str) -> int:
+ """Delete the rows of every team with this external id, for example after an app is uninstalled.
+
+ Raises `ValueError` on `IntegrationManager.any`, because it would delete rows of other providers.
+ All the deletes commit together or not at all. Each row is deleted on its own, so the delete
+ signals fire.
+ """
+ deleted = 0
+ with transaction.atomic():
+ for row in self._cross_team_rows("delete_by_external_id").filter(integration_id=external_id):
+ row.delete()
+ deleted += 1
+ return deleted
+
+ async def aget(self, team_id: int, id: int) -> IntegrationRecord:
+ return await database_sync_to_async(self.get)(team_id, id)
+
+ async def aget_many(self, team_id: int, ids: Iterable[int]) -> dict[int, IntegrationRecord]:
+ return await database_sync_to_async(self.get_many)(team_id, list(ids))
+
+ async def afind(
+ self,
+ team_id: int | None = None,
+ *,
+ team_ids: Iterable[int] | None = None,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ ) -> IntegrationRecord | None:
+ return await database_sync_to_async(self.find)(
+ team_id, team_ids=None if team_ids is None else list(team_ids), external_id=external_id, where=where
+ )
+
+ async def alist(
+ self,
+ team_id: int,
+ *,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ newest_first: bool = False,
+ ) -> builtins.list[IntegrationRecord]:
+ return await database_sync_to_async(self.list)(
+ team_id, external_id=external_id, where=where, newest_first=newest_first
+ )
+
+ async def aexists(
+ self,
+ team_id: int | None = None,
+ *,
+ team_ids: Iterable[int] | None = None,
+ external_id: str | None = None,
+ where: Mapping[str, ConfigValue] | None = None,
+ ) -> bool:
+ return await database_sync_to_async(self.exists)(
+ team_id, team_ids=None if team_ids is None else list(team_ids), external_id=external_id, where=where
+ )
+
+ async def ausages(self, team_id: int, id: int) -> builtins.list[IntegrationUsage]:
+ return await database_sync_to_async(self.usages)(team_id, id)
+
+ async def adelete(self, team_id: int, id: int, *, ignore_usages: bool = False) -> None:
+ await database_sync_to_async(self.delete)(team_id, id, ignore_usages=ignore_usages)
+
+
+class AuthAccessor(IntegrationReader):
+ """Reads, deletes and writes of the rows of one kind with one auth type."""
+
+ def __init__(self, *, kind: str, auth_type: AuthType, source_rows: SourceRows = SourceRows.EXCLUDE) -> None:
+ if source_rows == SourceRows.INCLUDE:
+ raise ValueError("A write accessor writes either named rows or warehouse source rows")
+ super().__init__(kinds=frozenset({kind}), auth_type=auth_type, source_rows=source_rows)
+ self._kind = kind
+ self._write_auth_type = auth_type
+
+ def _config_for_write(self, config: Mapping[str, object]) -> dict[str, object]:
+ if self._write_auth_type not in kind_auth_types(self._kind).supported:
+ raise UnsupportedAuthType(f"Integration kind {self._kind!r} does not support {self._write_auth_type}")
+ if AUTH_TYPE_CONFIG_KEY in config:
+ raise ValueError("The integrations SDK writes the auth type itself")
+ has_marker = SOURCE_MARKER_CONFIG_KEY in config
+ if self._source_rows == SourceRows.ONLY and not has_marker:
+ raise ValueError(f"A warehouse source row needs config[{SOURCE_MARKER_CONFIG_KEY!r}]")
+ if self._source_rows == SourceRows.EXCLUDE and has_marker:
+ raise ValueError("Write warehouse source rows through IntegrationManager.source(kind)")
+ result = dict(config)
+ if self._write_auth_type != default_auth_type(self._kind):
+ result[AUTH_TYPE_CONFIG_KEY] = self._write_auth_type.value
+ return result
+
+ def _in_scope(self, row: Integration) -> bool:
+ return self._team_rows(row.team_id).filter(id=row.id).exists()
+
+ def create(
+ self,
+ team_id: int,
+ *,
+ external_id: str | None,
+ config: Mapping[str, object],
+ sensitive_config: Mapping[str, object] | None = None,
+ created_by_id: int | None = None,
+ ) -> IntegrationRecord:
+ """Create a row. The SDK writes `config["auth_type"]` when the auth type is not the kind's default.
+
+ Raises `IntegrationConflict` when the team already has a row of the kind with this external id.
+ """
+ new_config = self._config_for_write(config)
+ try:
+ with transaction.atomic():
+ return _rows().create(
+ team_id=team_id,
+ kind=self._kind,
+ integration_id=external_id,
+ config=new_config,
+ sensitive_config=dict(sensitive_config or {}),
+ created_by_id=created_by_id,
+ )
+ except IntegrityError as error:
+ if not _is_unique_key_violation(error):
+ raise
+ raise IntegrationConflict(
+ f"Team {team_id} already has a {self._kind!r} integration with external id {external_id!r}"
+ ) from error
+
+ def upsert(
+ self,
+ team_id: int,
+ external_id: str,
+ *,
+ config: Mapping[str, object],
+ sensitive_config: Mapping[str, object] | None = None,
+ created_by_id: int | None = None,
+ ) -> tuple[IntegrationRecord, bool]:
+ """Create the row with this external id, or replace the config of the existing one.
+
+ Returns the row and True when it is new. Raises `IntegrationConflict` when a row with this
+ external id exists outside the scope, for example with another auth type.
+ """
+ new_config = self._config_for_write(config)
+ new_sensitive_config = dict(sensitive_config or {})
+ key = {"team_id": team_id, "kind": self._kind, "integration_id": external_id}
+ with transaction.atomic():
+ row = _rows().select_for_update().filter(**key).first()
+ if row is None:
+ try:
+ with transaction.atomic():
+ created = _rows().create(
+ **key, config=new_config, sensitive_config=new_sensitive_config, created_by_id=created_by_id
+ )
+ return created, True
+ except IntegrityError as error:
+ if not _is_unique_key_violation(error):
+ raise
+ row = _rows().select_for_update().get(**key)
+ if not self._in_scope(row):
+ raise IntegrationConflict(f"Integration {row.id} with external id {external_id!r} is not in {self!r}")
+ row.config = new_config
+ row.sensitive_config = new_sensitive_config
+ update_fields = ["config", "sensitive_config"]
+ if created_by_id is not None:
+ row.created_by_id = created_by_id
+ update_fields.append("created_by")
+ row.save(update_fields=update_fields)
+ return row, False
+
+ def update(
+ self,
+ team_id: int,
+ id: int,
+ *,
+ config_patch: Mapping[str, object] | None = None,
+ config_remove: Iterable[str] = (),
+ sensitive_config_patch: Mapping[str, object] | None = None,
+ sensitive_config_remove: Iterable[str] = (),
+ errors: str | None = None,
+ ) -> IntegrationRecord:
+ """Merge `config_patch` into the config and drop the keys in `config_remove`. `errors=None` keeps the errors."""
+ patch = IntegrationPatch(
+ config_patch=dict(config_patch or {}),
+ config_remove=tuple(config_remove),
+ sensitive_config_patch=dict(sensitive_config_patch or {}),
+ sensitive_config_remove=tuple(sensitive_config_remove),
+ errors=errors,
+ )
+ _checked_config_keys([*patch.config_patch, *patch.config_remove])
+ with transaction.atomic():
+ row = self._row(team_id, id, lock=True)
+ update_fields = _patch_row(row, patch)
+ if update_fields:
+ row.save(update_fields=update_fields)
+ return row
+
+ def update_by_external_id(self, external_id: str, patch: IntegrationPatch) -> int:
+ """Apply `patch` to the rows of every team with this external id. Returns the number of rows that changed.
+
+ One `bulk_update` writes the changed rows, so no `post_save` signal fires. Rows that the patch
+ does not change are not written.
+ """
+ _checked_config_keys([*patch.config_patch, *patch.config_remove])
+ with transaction.atomic():
+ rows = (
+ self._cross_team_rows("update_by_external_id")
+ .select_for_update()
+ .filter(integration_id=external_id)
+ .order_by("id")
+ )
+ changed: builtins.list[Integration] = []
+ fields: set[str] = set()
+ for row in rows:
+ row_fields = _patch_row(row, patch)
+ if row_fields:
+ changed.append(row)
+ fields.update(row_fields)
+ if changed:
+ _rows().bulk_update(changed, sorted(fields))
+ return len(changed)
+
+ async def acreate(
+ self,
+ team_id: int,
+ *,
+ external_id: str | None,
+ config: Mapping[str, object],
+ sensitive_config: Mapping[str, object] | None = None,
+ created_by_id: int | None = None,
+ ) -> IntegrationRecord:
+ return await database_sync_to_async(self.create)(
+ team_id,
+ external_id=external_id,
+ config=config,
+ sensitive_config=sensitive_config,
+ created_by_id=created_by_id,
+ )
+
+ async def aupsert(
+ self,
+ team_id: int,
+ external_id: str,
+ *,
+ config: Mapping[str, object],
+ sensitive_config: Mapping[str, object] | None = None,
+ created_by_id: int | None = None,
+ ) -> tuple[IntegrationRecord, bool]:
+ return await database_sync_to_async(self.upsert)(
+ team_id, external_id, config=config, sensitive_config=sensitive_config, created_by_id=created_by_id
+ )
+
+ async def aupdate(
+ self,
+ team_id: int,
+ id: int,
+ *,
+ config_patch: Mapping[str, object] | None = None,
+ config_remove: Iterable[str] = (),
+ sensitive_config_patch: Mapping[str, object] | None = None,
+ sensitive_config_remove: Iterable[str] = (),
+ errors: str | None = None,
+ ) -> IntegrationRecord:
+ return await database_sync_to_async(self.update)(
+ team_id,
+ id,
+ config_patch=config_patch,
+ config_remove=tuple(config_remove),
+ sensitive_config_patch=sensitive_config_patch,
+ sensitive_config_remove=tuple(sensitive_config_remove),
+ errors=errors,
+ )
+
+
+def _same_json(a: object, b: object) -> bool:
+ # Python equality treats `1`, `1.0` and `True` as equal, but JSON stores them as different values.
+ return json.dumps(a, sort_keys=True, default=str) == json.dumps(b, sort_keys=True, default=str)
+
+
+def _patch_row(row: Integration, patch: IntegrationPatch) -> list[str]:
+ """Apply `patch` to the row in memory. Returns the fields whose value changed."""
+ changed: list[str] = []
+ config = _patched(row.config, patch.config_patch, patch.config_remove)
+ if not _same_json(config, row.config or {}):
+ row.config = config
+ changed.append("config")
+ sensitive_config = _patched(row.sensitive_config, patch.sensitive_config_patch, patch.sensitive_config_remove)
+ if not _same_json(sensitive_config, row.sensitive_config or {}):
+ row.sensitive_config = sensitive_config
+ changed.append("sensitive_config")
+ if patch.errors is not None and patch.errors != row.errors:
+ row.errors = patch.errors
+ changed.append("errors")
+ return changed
+
+
+class OAuthAccessor(AuthAccessor):
+ """Reads and writes of the OAuth rows of one kind, and the refresh of their access tokens."""
+
+ def __init__(self, *, kind: str, source_rows: SourceRows = SourceRows.EXCLUDE) -> None:
+ super().__init__(kind=kind, auth_type=AuthType.OAUTH, source_rows=source_rows)
+
+ def refresh_access_token(
+ self, team_id: int, id: int, *, stale_access_token: str | None = None
+ ) -> IntegrationRecord:
+ """Refresh the access token of the row when it expires soon, and return the row.
+
+ The refresh holds a lock on the row. Some providers rotate the refresh token on each refresh
+ and revoke the grant when a token is spent twice, so two workers must not refresh one row at
+ the same time. The second worker waits for the lock, and then gets the token that the first
+ worker stored.
+
+ Pass `stale_access_token` to refresh also when that token is still the stored access token,
+ for example after the provider refused it. When the row has another token, the row comes back
+ without a refresh. A refresh that fails does not raise: the row comes back with `errors` set.
+ Raises `ValueError` when the kind has no OAuth token refresh.
+ """
+ if self._kind not in OauthIntegration.supported_kinds:
+ raise ValueError(f"Integration kind {self._kind!r} has no OAuth token refresh")
+ with transaction.atomic():
+ row = self._row(team_id, id, lock=True)
+ oauth = OauthIntegration(row)
+ stale = stale_access_token is not None and row.access_token == stale_access_token
+ if stale or oauth.access_token_expired():
+ oauth.refresh_access_token()
+ return row
+
+ async def arefresh_access_token(
+ self, team_id: int, id: int, *, stale_access_token: str | None = None
+ ) -> IntegrationRecord:
+ # Not thread-sensitive: the provider call can be slow, and it must not hold the shared thread
+ # that every other sync database call of this event loop waits for. The row lock still applies.
+ return await database_sync_to_async(self.refresh_access_token, thread_sensitive=False)(
+ team_id, id, stale_access_token=stale_access_token
+ )
+
+
+class KindAccessor:
+ """The base of the generated accessor of one kind. Each subclass has one attribute per supported auth type."""
+
+ kind: ClassVar[str]
+ any: IntegrationReader
+
+
+class SourceKindAccessor:
+ """The rows of one warehouse source kind. Only rows with `config["warehouse_source_type"]` are in scope."""
+
+ def __init__(self, kind: str) -> None:
+ self.kind = kind
+ self.oauth = AuthAccessor(kind=kind, auth_type=AuthType.OAUTH, source_rows=SourceRows.ONLY)
+ self.token = AuthAccessor(kind=kind, auth_type=AuthType.TOKEN, source_rows=SourceRows.ONLY)
+ self.credentials = AuthAccessor(kind=kind, auth_type=AuthType.CREDENTIALS, source_rows=SourceRows.ONLY)
+ self.any = IntegrationReader(kinds=frozenset({kind}), source_rows=SourceRows.ONLY)
+
+
+class IntegrationManagerBase:
+ """The hand-written part of `IntegrationManager`. The generated subclass adds one attribute per kind."""
+
+ # Every row of every kind and every auth type, without warehouse source rows. For reads by id with a
+ # kind that the caller does not know. `IntegrationManager.any.with_source_rows()` adds the source rows.
+ any = IntegrationReader(kinds=None)
+
+ def source(self, kind: str) -> SourceKindAccessor:
+ """The warehouse source rows of a registered source kind. Raises `UnknownIntegrationKind`."""
+ if not is_source_kind(kind):
+ raise UnknownIntegrationKind(f"No warehouse source registered the integration kind {kind!r}")
+ return SourceKindAccessor(kind)
+
+ def kinds(
+ self, *kinds: str, auth_types: Iterable[AuthType] | Literal["any", "default"] = "default"
+ ) -> IntegrationReader:
+ """Reads of the named rows of several catalogue kinds.
+
+ By default it reads the default auth type of each kind, because code that names kinds expects
+ their usual config shape. `auth_types="any"` reads every auth type. A list reads those auth types.
+ """
+ unknown = sorted(set(kinds) - set(KINDS))
+ if not kinds or unknown:
+ raise UnknownIntegrationKind(f"Unknown integration kinds: {unknown or 'none given'}")
+ if auth_types == "any":
+ return IntegrationReader(kinds=frozenset(kinds))
+ if auth_types == "default":
+ return IntegrationReader(kinds=frozenset(kinds), auth_type="default")
+ return IntegrationReader(kinds=frozenset(kinds), auth_type=frozenset(AuthType(t) for t in auth_types))
diff --git a/posthog/integrations/catalogue.py b/posthog/integrations/catalogue.py
new file mode 100644
index 000000000000..c2252bc0777d
--- /dev/null
+++ b/posthog/integrations/catalogue.py
@@ -0,0 +1,74 @@
+"""The integration kinds and the auth types that each kind supports.
+
+This is the hand-written source of `kinds_generated.py`. After a change here, run
+`hogli build:projections --only integration-kinds` and commit the result. A kind also needs a value
+in `Integration.IntegrationKind` (`posthog/models/integration/model.py`); a test compares the two.
+
+A row stores its auth type in `config["auth_type"]` only when it is not the default of its kind, so
+the default of a kind is stored data. Do not change the default of a kind that has rows.
+
+Warehouse source kinds are not in this file. The warehouse product registers them at startup with
+`register_source_kinds`.
+"""
+
+from collections.abc import Mapping
+
+from posthog.integrations.contracts import KindSpec
+from posthog.integrations.enums import AuthType
+
+_OAUTH = AuthType.OAUTH
+_TOKEN = AuthType.TOKEN
+_CREDENTIALS = AuthType.CREDENTIALS
+
+KINDS: Mapping[str, KindSpec] = {
+ "anthropic": KindSpec(label="Anthropic", default=_TOKEN),
+ "apns": KindSpec(label="Apple Push Notification service", default=_CREDENTIALS),
+ "apple-ads": KindSpec(label="Apple Ads", default=_OAUTH, others=frozenset({_CREDENTIALS})),
+ "aws-redshift": KindSpec(label="Amazon Redshift", default=_CREDENTIALS),
+ "aws-s3": KindSpec(label="Amazon S3", default=_CREDENTIALS),
+ "azure-blob": KindSpec(label="Azure Blob Storage", default=_CREDENTIALS),
+ "bing-ads": KindSpec(label="Bing Ads", default=_OAUTH),
+ "clickup": KindSpec(label="ClickUp", default=_OAUTH, others=frozenset({_TOKEN})),
+ "customerio-app": KindSpec(label="Customer.io App API", default=_TOKEN),
+ "customerio-track": KindSpec(label="Customer.io Track API", default=_TOKEN),
+ "customerio-webhook": KindSpec(label="Customer.io webhook", default=_TOKEN),
+ "databricks": KindSpec(label="Databricks", default=_CREDENTIALS, others=frozenset({_TOKEN})),
+ "email": KindSpec(label="Email", default=_CREDENTIALS),
+ "firebase": KindSpec(label="Firebase", default=_CREDENTIALS),
+ "github": KindSpec(label="GitHub", default=_OAUTH, others=frozenset({_TOKEN})),
+ "gitlab": KindSpec(label="GitLab", default=_TOKEN),
+ "google-ads": KindSpec(label="Google Ads", default=_OAUTH),
+ "google-analytics": KindSpec(label="Google Analytics", default=_OAUTH),
+ "google-calendar": KindSpec(label="Google Calendar", default=_OAUTH),
+ "google-cloud-service-account": KindSpec(label="Google Cloud service account", default=_CREDENTIALS),
+ "google-cloud-storage": KindSpec(label="Google Cloud Storage", default=_CREDENTIALS),
+ "google-pubsub": KindSpec(label="Google Pub/Sub", default=_CREDENTIALS),
+ "google-search-console": KindSpec(label="Google Search Console", default=_OAUTH),
+ "google-sheets": KindSpec(label="Google Sheets", default=_OAUTH, others=frozenset({_CREDENTIALS})),
+ "helpscout": KindSpec(label="Help Scout", default=_OAUTH),
+ "hubspot": KindSpec(label="HubSpot", default=_OAUTH),
+ "instagram": KindSpec(label="Instagram", default=_OAUTH),
+ "intercom": KindSpec(label="Intercom", default=_OAUTH),
+ "jira": KindSpec(label="Jira", default=_OAUTH, others=frozenset({_TOKEN})),
+ "linear": KindSpec(label="Linear", default=_OAUTH),
+ "linkedin-ads": KindSpec(label="LinkedIn Ads", default=_OAUTH),
+ "meta-ads": KindSpec(label="Meta Ads", default=_OAUTH),
+ "pardot": KindSpec(label="Pardot", default=_OAUTH),
+ "pinterest-ads": KindSpec(label="Pinterest Ads", default=_OAUTH),
+ "postgresql": KindSpec(label="PostgreSQL", default=_CREDENTIALS),
+ "posthog": KindSpec(label="PostHog", default=_OAUTH),
+ "reddit-ads": KindSpec(label="Reddit Ads", default=_OAUTH),
+ "resend": KindSpec(label="Resend", default=_OAUTH, others=frozenset({_TOKEN})),
+ "s3-compatible": KindSpec(label="S3-compatible storage", default=_CREDENTIALS),
+ "salesforce": KindSpec(label="Salesforce", default=_OAUTH),
+ "slack": KindSpec(label="Slack", default=_OAUTH, others=frozenset({_TOKEN})),
+ "slack-posthog-code": KindSpec(label="Slack (PostHog Code)", default=_OAUTH),
+ "snapchat": KindSpec(label="Snapchat", default=_OAUTH),
+ "snowflake": KindSpec(label="Snowflake", default=_CREDENTIALS),
+ "stripe": KindSpec(label="Stripe", default=_OAUTH, others=frozenset({_TOKEN})),
+ "tiktok-ads": KindSpec(label="TikTok Ads", default=_OAUTH),
+ "twilio": KindSpec(label="Twilio", default=_TOKEN),
+ "twitter-ads": KindSpec(label="X Ads", default=_OAUTH),
+ "vercel": KindSpec(label="Vercel", default=_OAUTH, others=frozenset({_TOKEN})),
+ "youtube-analytics": KindSpec(label="YouTube Analytics", default=_OAUTH),
+}
diff --git a/posthog/integrations/contracts.py b/posthog/integrations/contracts.py
new file mode 100644
index 000000000000..21cce799df3d
--- /dev/null
+++ b/posthog/integrations/contracts.py
@@ -0,0 +1,142 @@
+"""Data shapes and errors of the integrations SDK."""
+
+import dataclasses
+from collections.abc import Mapping
+from datetime import datetime
+from typing import Any, Protocol
+
+from posthog.dataclasses import frozen
+from posthog.integrations.enums import AuthType
+
+# A value that `find`, `list` and `exists` can compare a config key with. Equality only.
+ConfigValue = str | int | float | bool | None
+
+# A row that holds the typed-in credentials of a warehouse source stores the source type under this
+# config key. Only `IntegrationManager.source(kind)` and a reader from `with_source_rows()` read these rows.
+SOURCE_MARKER_CONFIG_KEY = "warehouse_source_type"
+
+
+class IntegrationRecord(Protocol):
+ """One integration row, as the SDK returns it. Read-only.
+
+ At runtime the SDK returns the Django model, but callers see only these reads. So mypy rejects
+ `.save()`, `.delete()`, managers and writes to `config`. Write through the SDK, for example with
+ `update` or `upsert`. Later the SDK returns a frozen record with the same reads, so callers do not
+ change. Add a read here only when callers need it, and never a related row: use `team_id` and
+ `created_by_id`, and load the team or the user from their own models.
+ """
+
+ @property
+ def id(self) -> int: ...
+
+ @property
+ def team_id(self) -> int: ...
+
+ @property
+ def kind(self) -> str: ...
+
+ # The id of the connection in the external system, for example a GitHub installation id.
+ @property
+ def integration_id(self) -> str | None: ...
+
+ @property
+ def config(self) -> Mapping[str, Any]: ...
+
+ @property
+ def sensitive_config(self) -> Mapping[str, Any]: ...
+
+ @property
+ def errors(self) -> str: ...
+
+ @property
+ def created_at(self) -> datetime: ...
+
+ @property
+ def created_by_id(self) -> int | None: ...
+
+ # `config["auth_type"]`, or the default auth type of the kind when the row does not store one.
+ @property
+ def auth_type(self) -> str: ...
+
+ # The decrypted `sensitive_config["access_token"]`. Raises `UndecryptedIntegrationSecretError`
+ # when the stored value cannot be decrypted.
+ @property
+ def access_token(self) -> str | None: ...
+
+ @property
+ def refresh_token(self) -> str | None: ...
+
+ def can_be_managed_by_creator(self, user_id: int | None) -> bool: ...
+
+
+@frozen
+class KindSpec:
+ """The auth types of one integration kind. A row without `config["auth_type"]` has the default."""
+
+ label: str
+ default: AuthType
+ others: frozenset[AuthType] = frozenset()
+
+ @property
+ def supported(self) -> frozenset[AuthType]:
+ return self.others | {self.default}
+
+
+@frozen
+class IntegrationPatch:
+ """A change to one row. `update` and `update_by_external_id` apply it.
+
+ `errors=None` keeps the stored errors. Pass `""` to clear them.
+ """
+
+ config_patch: Mapping[str, object] = dataclasses.field(default_factory=dict)
+ config_remove: tuple[str, ...] = ()
+ sensitive_config_patch: Mapping[str, object] = dataclasses.field(default_factory=dict)
+ sensitive_config_remove: tuple[str, ...] = ()
+ errors: str | None = None
+
+
+@frozen
+class IntegrationUsage:
+ """One PostHog object that uses an integration row, such as a warehouse source or a workflow."""
+
+ # The kind of object, for example "warehouse_source". The product that reports it chooses the value.
+ type: str
+ id: str
+ name: str
+
+
+class IntegrationNotFound(LookupError):
+ """The row does not exist, belongs to another team, or is not in the scope of the accessor."""
+
+
+class UnknownIntegrationKind(ValueError):
+ """The kind is not in the catalogue, and no product registered it."""
+
+
+class UnsupportedIntegrationFilter(ValueError):
+ """`where` holds a key or a value that the SDK cannot compare.
+
+ The SDK compares named config keys for equality. Anything more needs its own SDK method.
+ """
+
+
+class UnsupportedAuthType(ValueError):
+ """The kind does not support the auth type of the write."""
+
+
+class IntegrationConflict(ValueError):
+ """A write hit the unique key (team, kind, external id) of a row.
+
+ `create` raises it for any row with the same key. `upsert` raises it for a row with the same key
+ that the accessor cannot write, for example a row with another auth type, or a warehouse source
+ row found through a named accessor.
+ """
+
+
+class IntegrationInUse(Exception):
+ """`delete` found PostHog objects that use the row."""
+
+ def __init__(self, usages: list[IntegrationUsage]) -> None:
+ super().__init__(f"The integration has {len(usages)} usages")
+ self.usages = usages
diff --git a/posthog/integrations/enums.py b/posthog/integrations/enums.py
new file mode 100644
index 000000000000..b24488121cbc
--- /dev/null
+++ b/posthog/integrations/enums.py
@@ -0,0 +1,21 @@
+"""Enums of the integrations SDK."""
+
+from posthog.enums import LabeledStrEnum
+
+
+class IntegrationAuthType(LabeledStrEnum):
+ """How an integration row authenticates with its provider.
+
+ A kind names the provider. One kind can support several auth types, and has one default.
+ """
+
+ # A connection that the provider manages: an OAuth grant or an app installation. It has scopes or permissions.
+ OAUTH = "oauth", "OAuth"
+ # One static secret string: an API key, a personal access token, a bot token or a webhook secret.
+ TOKEN = "token", "Token"
+ # Several typed-in fields: a host, a user and a password, an access key and a secret, or a service account.
+ CREDENTIALS = "credentials", "Credentials"
+
+
+# The short name for callers. The class keeps the long name, because the OpenAPI enum name comes from it.
+AuthType = IntegrationAuthType
diff --git a/posthog/integrations/fields.py b/posthog/integrations/fields.py
new file mode 100644
index 000000000000..2e20c741bdfc
--- /dev/null
+++ b/posthog/integrations/fields.py
@@ -0,0 +1,98 @@
+"""DRF fields that reference an integration row. Not in `posthog.integrations`, so the SDK does not load DRF."""
+
+from collections.abc import Iterable
+from typing import Any, Literal
+
+from drf_spectacular.types import OpenApiTypes
+from drf_spectacular.utils import extend_schema_field
+from rest_framework import serializers
+from rest_framework.fields import empty, get_attribute
+from rest_framework.relations import PKOnlyObject
+
+from posthog.integrations.contracts import IntegrationNotFound, IntegrationRecord
+from posthog.integrations.enums import AuthType
+from posthog.integrations.manager import IntegrationManager
+
+
+@extend_schema_field(OpenApiTypes.INT)
+class IntegrationReferenceField(serializers.Field):
+ """An integration id of the request's team. Use it instead of a `PrimaryKeyRelatedField` on `Integration`.
+
+ It reads the team from the serializer context (`team_id`, or `get_team`), so a row of another team
+ is "does not exist". Pass `kinds` to accept only rows of these catalogue kinds. The field refuses
+ warehouse source rows, the same as `IntegrationManager.any` and `IntegrationManager.kinds(...)`.
+
+ With `kinds`, `auth_types` selects the auth types the field accepts. The default (None) accepts only
+ the default auth type of each kind, because code that reads a kind expects its usual config shape.
+ Pass a list of auth types, or `"any"` for every auth type. Without `kinds`, the field accepts every row.
+ """
+
+ default_error_messages = {
+ "does_not_exist": 'Invalid pk "{pk_value}" - object does not exist.',
+ "incorrect_type": "Incorrect type. Expected pk value, received {data_type}.",
+ "no_team": "The serializer context has no team.",
+ }
+
+ def __init__(
+ self,
+ *,
+ kinds: Iterable[str] = (),
+ auth_types: Iterable[AuthType] | Literal["any"] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ self.kinds = frozenset(kinds)
+ if not self.kinds:
+ if auth_types is not None:
+ raise ValueError("IntegrationReferenceField needs kinds to filter by auth type")
+ self._rows = IntegrationManager.any
+ else:
+ self._rows = IntegrationManager.kinds(
+ *sorted(self.kinds), auth_types="default" if auth_types is None else auth_types
+ )
+ super().__init__(**kwargs)
+
+ def _team_id(self) -> int:
+ team_id = self.context.get("team_id")
+ if team_id is None and "get_team" in self.context:
+ team_id = self.context["get_team"]().id
+ if team_id is None:
+ self.fail("no_team")
+ return int(team_id)
+
+ def run_validation(self, data: object = empty) -> object:
+ # Same as DRF's `RelatedField`: an empty string means no row, so `allow_null` and `required` decide.
+ if data == "":
+ data = None
+ return super().run_validation(data)
+
+ def to_internal_value(self, data: object) -> IntegrationRecord:
+ if isinstance(data, bool) or not isinstance(data, int | str):
+ self.fail("incorrect_type", data_type=type(data).__name__)
+ try:
+ integration_id = int(data)
+ except ValueError:
+ self.fail("incorrect_type", data_type=type(data).__name__)
+ try:
+ return self._rows.get(self._team_id(), integration_id)
+ except IntegrationNotFound:
+ self.fail("does_not_exist", pk_value=data)
+
+ def get_attribute(self, instance: object) -> object:
+ # Read the foreign key column instead of the related row, so a list does not query once per row.
+ if self.source_attrs:
+ try:
+ parent = get_attribute(instance, self.source_attrs[:-1])
+ value = parent.serializable_value(self.source_attrs[-1])
+ except AttributeError:
+ pass
+ else:
+ # `serializable_value` returns the attribute itself when it is not a model field.
+ return PKOnlyObject(pk=getattr(value, "pk", value))
+ return super().get_attribute(instance)
+
+ def to_representation(self, value: IntegrationRecord | PKOnlyObject | int) -> int:
+ if isinstance(value, int):
+ return value
+ if isinstance(value, PKOnlyObject):
+ return value.pk
+ return value.id
diff --git a/posthog/integrations/kinds_generated.py b/posthog/integrations/kinds_generated.py
new file mode 100644
index 000000000000..7253484d034c
--- /dev/null
+++ b/posthog/integrations/kinds_generated.py
@@ -0,0 +1,432 @@
+"""Generated from `posthog/integrations/catalogue.py`. Do not edit by hand.
+
+Run `hogli build:projections --only integration-kinds` after a change to the catalogue. Import
+`IntegrationManager` from `posthog.integrations`, not the classes here.
+"""
+
+from enum import StrEnum
+
+from posthog.integrations.accessors import (
+ AuthAccessor,
+ IntegrationManagerBase,
+ IntegrationReader,
+ KindAccessor,
+ OAuthAccessor,
+)
+from posthog.integrations.enums import AuthType
+
+
+class IntegrationKind(StrEnum):
+ ANTHROPIC = "anthropic"
+ APNS = "apns"
+ APPLE_ADS = "apple-ads"
+ AWS_REDSHIFT = "aws-redshift"
+ AWS_S3 = "aws-s3"
+ AZURE_BLOB = "azure-blob"
+ BING_ADS = "bing-ads"
+ CLICKUP = "clickup"
+ CUSTOMERIO_APP = "customerio-app"
+ CUSTOMERIO_TRACK = "customerio-track"
+ CUSTOMERIO_WEBHOOK = "customerio-webhook"
+ DATABRICKS = "databricks"
+ EMAIL = "email"
+ FIREBASE = "firebase"
+ GITHUB = "github"
+ GITLAB = "gitlab"
+ GOOGLE_ADS = "google-ads"
+ GOOGLE_ANALYTICS = "google-analytics"
+ GOOGLE_CALENDAR = "google-calendar"
+ GOOGLE_CLOUD_SERVICE_ACCOUNT = "google-cloud-service-account"
+ GOOGLE_CLOUD_STORAGE = "google-cloud-storage"
+ GOOGLE_PUBSUB = "google-pubsub"
+ GOOGLE_SEARCH_CONSOLE = "google-search-console"
+ GOOGLE_SHEETS = "google-sheets"
+ HELPSCOUT = "helpscout"
+ HUBSPOT = "hubspot"
+ INSTAGRAM = "instagram"
+ INTERCOM = "intercom"
+ JIRA = "jira"
+ LINEAR = "linear"
+ LINKEDIN_ADS = "linkedin-ads"
+ META_ADS = "meta-ads"
+ PARDOT = "pardot"
+ PINTEREST_ADS = "pinterest-ads"
+ POSTGRESQL = "postgresql"
+ POSTHOG = "posthog"
+ REDDIT_ADS = "reddit-ads"
+ RESEND = "resend"
+ S3_COMPATIBLE = "s3-compatible"
+ SALESFORCE = "salesforce"
+ SLACK = "slack"
+ SLACK_POSTHOG_CODE = "slack-posthog-code"
+ SNAPCHAT = "snapchat"
+ SNOWFLAKE = "snowflake"
+ STRIPE = "stripe"
+ TIKTOK_ADS = "tiktok-ads"
+ TWILIO = "twilio"
+ TWITTER_ADS = "twitter-ads"
+ VERCEL = "vercel"
+ YOUTUBE_ANALYTICS = "youtube-analytics"
+
+
+class AnthropicKind(KindAccessor):
+ kind = "anthropic"
+ token = AuthAccessor(kind="anthropic", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"anthropic"}))
+
+
+class ApnsKind(KindAccessor):
+ kind = "apns"
+ credentials = AuthAccessor(kind="apns", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"apns"}))
+
+
+class AppleAdsKind(KindAccessor):
+ kind = "apple-ads"
+ oauth = OAuthAccessor(kind="apple-ads") # the default
+ credentials = AuthAccessor(kind="apple-ads", auth_type=AuthType.CREDENTIALS)
+ any = IntegrationReader(kinds=frozenset({"apple-ads"}))
+
+
+class AwsRedshiftKind(KindAccessor):
+ kind = "aws-redshift"
+ credentials = AuthAccessor(kind="aws-redshift", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"aws-redshift"}))
+
+
+class AwsS3Kind(KindAccessor):
+ kind = "aws-s3"
+ credentials = AuthAccessor(kind="aws-s3", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"aws-s3"}))
+
+
+class AzureBlobKind(KindAccessor):
+ kind = "azure-blob"
+ credentials = AuthAccessor(kind="azure-blob", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"azure-blob"}))
+
+
+class BingAdsKind(KindAccessor):
+ kind = "bing-ads"
+ oauth = OAuthAccessor(kind="bing-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"bing-ads"}))
+
+
+class ClickupKind(KindAccessor):
+ kind = "clickup"
+ oauth = OAuthAccessor(kind="clickup") # the default
+ token = AuthAccessor(kind="clickup", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"clickup"}))
+
+
+class CustomerioAppKind(KindAccessor):
+ kind = "customerio-app"
+ token = AuthAccessor(kind="customerio-app", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"customerio-app"}))
+
+
+class CustomerioTrackKind(KindAccessor):
+ kind = "customerio-track"
+ token = AuthAccessor(kind="customerio-track", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"customerio-track"}))
+
+
+class CustomerioWebhookKind(KindAccessor):
+ kind = "customerio-webhook"
+ token = AuthAccessor(kind="customerio-webhook", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"customerio-webhook"}))
+
+
+class DatabricksKind(KindAccessor):
+ kind = "databricks"
+ token = AuthAccessor(kind="databricks", auth_type=AuthType.TOKEN)
+ credentials = AuthAccessor(kind="databricks", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"databricks"}))
+
+
+class EmailKind(KindAccessor):
+ kind = "email"
+ credentials = AuthAccessor(kind="email", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"email"}))
+
+
+class FirebaseKind(KindAccessor):
+ kind = "firebase"
+ credentials = AuthAccessor(kind="firebase", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"firebase"}))
+
+
+class GithubKind(KindAccessor):
+ kind = "github"
+ oauth = OAuthAccessor(kind="github") # the default
+ token = AuthAccessor(kind="github", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"github"}))
+
+
+class GitlabKind(KindAccessor):
+ kind = "gitlab"
+ token = AuthAccessor(kind="gitlab", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"gitlab"}))
+
+
+class GoogleAdsKind(KindAccessor):
+ kind = "google-ads"
+ oauth = OAuthAccessor(kind="google-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"google-ads"}))
+
+
+class GoogleAnalyticsKind(KindAccessor):
+ kind = "google-analytics"
+ oauth = OAuthAccessor(kind="google-analytics") # the default
+ any = IntegrationReader(kinds=frozenset({"google-analytics"}))
+
+
+class GoogleCalendarKind(KindAccessor):
+ kind = "google-calendar"
+ oauth = OAuthAccessor(kind="google-calendar") # the default
+ any = IntegrationReader(kinds=frozenset({"google-calendar"}))
+
+
+class GoogleCloudServiceAccountKind(KindAccessor):
+ kind = "google-cloud-service-account"
+ credentials = AuthAccessor(kind="google-cloud-service-account", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"google-cloud-service-account"}))
+
+
+class GoogleCloudStorageKind(KindAccessor):
+ kind = "google-cloud-storage"
+ credentials = AuthAccessor(kind="google-cloud-storage", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"google-cloud-storage"}))
+
+
+class GooglePubsubKind(KindAccessor):
+ kind = "google-pubsub"
+ credentials = AuthAccessor(kind="google-pubsub", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"google-pubsub"}))
+
+
+class GoogleSearchConsoleKind(KindAccessor):
+ kind = "google-search-console"
+ oauth = OAuthAccessor(kind="google-search-console") # the default
+ any = IntegrationReader(kinds=frozenset({"google-search-console"}))
+
+
+class GoogleSheetsKind(KindAccessor):
+ kind = "google-sheets"
+ oauth = OAuthAccessor(kind="google-sheets") # the default
+ credentials = AuthAccessor(kind="google-sheets", auth_type=AuthType.CREDENTIALS)
+ any = IntegrationReader(kinds=frozenset({"google-sheets"}))
+
+
+class HelpscoutKind(KindAccessor):
+ kind = "helpscout"
+ oauth = OAuthAccessor(kind="helpscout") # the default
+ any = IntegrationReader(kinds=frozenset({"helpscout"}))
+
+
+class HubspotKind(KindAccessor):
+ kind = "hubspot"
+ oauth = OAuthAccessor(kind="hubspot") # the default
+ any = IntegrationReader(kinds=frozenset({"hubspot"}))
+
+
+class InstagramKind(KindAccessor):
+ kind = "instagram"
+ oauth = OAuthAccessor(kind="instagram") # the default
+ any = IntegrationReader(kinds=frozenset({"instagram"}))
+
+
+class IntercomKind(KindAccessor):
+ kind = "intercom"
+ oauth = OAuthAccessor(kind="intercom") # the default
+ any = IntegrationReader(kinds=frozenset({"intercom"}))
+
+
+class JiraKind(KindAccessor):
+ kind = "jira"
+ oauth = OAuthAccessor(kind="jira") # the default
+ token = AuthAccessor(kind="jira", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"jira"}))
+
+
+class LinearKind(KindAccessor):
+ kind = "linear"
+ oauth = OAuthAccessor(kind="linear") # the default
+ any = IntegrationReader(kinds=frozenset({"linear"}))
+
+
+class LinkedinAdsKind(KindAccessor):
+ kind = "linkedin-ads"
+ oauth = OAuthAccessor(kind="linkedin-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"linkedin-ads"}))
+
+
+class MetaAdsKind(KindAccessor):
+ kind = "meta-ads"
+ oauth = OAuthAccessor(kind="meta-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"meta-ads"}))
+
+
+class PardotKind(KindAccessor):
+ kind = "pardot"
+ oauth = OAuthAccessor(kind="pardot") # the default
+ any = IntegrationReader(kinds=frozenset({"pardot"}))
+
+
+class PinterestAdsKind(KindAccessor):
+ kind = "pinterest-ads"
+ oauth = OAuthAccessor(kind="pinterest-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"pinterest-ads"}))
+
+
+class PostgresqlKind(KindAccessor):
+ kind = "postgresql"
+ credentials = AuthAccessor(kind="postgresql", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"postgresql"}))
+
+
+class PosthogKind(KindAccessor):
+ kind = "posthog"
+ oauth = OAuthAccessor(kind="posthog") # the default
+ any = IntegrationReader(kinds=frozenset({"posthog"}))
+
+
+class RedditAdsKind(KindAccessor):
+ kind = "reddit-ads"
+ oauth = OAuthAccessor(kind="reddit-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"reddit-ads"}))
+
+
+class ResendKind(KindAccessor):
+ kind = "resend"
+ oauth = OAuthAccessor(kind="resend") # the default
+ token = AuthAccessor(kind="resend", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"resend"}))
+
+
+class S3CompatibleKind(KindAccessor):
+ kind = "s3-compatible"
+ credentials = AuthAccessor(kind="s3-compatible", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"s3-compatible"}))
+
+
+class SalesforceKind(KindAccessor):
+ kind = "salesforce"
+ oauth = OAuthAccessor(kind="salesforce") # the default
+ any = IntegrationReader(kinds=frozenset({"salesforce"}))
+
+
+class SlackKind(KindAccessor):
+ kind = "slack"
+ oauth = OAuthAccessor(kind="slack") # the default
+ token = AuthAccessor(kind="slack", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"slack"}))
+
+
+class SlackPosthogCodeKind(KindAccessor):
+ kind = "slack-posthog-code"
+ oauth = OAuthAccessor(kind="slack-posthog-code") # the default
+ any = IntegrationReader(kinds=frozenset({"slack-posthog-code"}))
+
+
+class SnapchatKind(KindAccessor):
+ kind = "snapchat"
+ oauth = OAuthAccessor(kind="snapchat") # the default
+ any = IntegrationReader(kinds=frozenset({"snapchat"}))
+
+
+class SnowflakeKind(KindAccessor):
+ kind = "snowflake"
+ credentials = AuthAccessor(kind="snowflake", auth_type=AuthType.CREDENTIALS) # the default
+ any = IntegrationReader(kinds=frozenset({"snowflake"}))
+
+
+class StripeKind(KindAccessor):
+ kind = "stripe"
+ oauth = OAuthAccessor(kind="stripe") # the default
+ token = AuthAccessor(kind="stripe", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"stripe"}))
+
+
+class TiktokAdsKind(KindAccessor):
+ kind = "tiktok-ads"
+ oauth = OAuthAccessor(kind="tiktok-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"tiktok-ads"}))
+
+
+class TwilioKind(KindAccessor):
+ kind = "twilio"
+ token = AuthAccessor(kind="twilio", auth_type=AuthType.TOKEN) # the default
+ any = IntegrationReader(kinds=frozenset({"twilio"}))
+
+
+class TwitterAdsKind(KindAccessor):
+ kind = "twitter-ads"
+ oauth = OAuthAccessor(kind="twitter-ads") # the default
+ any = IntegrationReader(kinds=frozenset({"twitter-ads"}))
+
+
+class VercelKind(KindAccessor):
+ kind = "vercel"
+ oauth = OAuthAccessor(kind="vercel") # the default
+ token = AuthAccessor(kind="vercel", auth_type=AuthType.TOKEN)
+ any = IntegrationReader(kinds=frozenset({"vercel"}))
+
+
+class YoutubeAnalyticsKind(KindAccessor):
+ kind = "youtube-analytics"
+ oauth = OAuthAccessor(kind="youtube-analytics") # the default
+ any = IntegrationReader(kinds=frozenset({"youtube-analytics"}))
+
+
+class GeneratedIntegrationManager(IntegrationManagerBase):
+ anthropic = AnthropicKind()
+ apns = ApnsKind()
+ apple_ads = AppleAdsKind()
+ aws_redshift = AwsRedshiftKind()
+ aws_s3 = AwsS3Kind()
+ azure_blob = AzureBlobKind()
+ bing_ads = BingAdsKind()
+ clickup = ClickupKind()
+ customerio_app = CustomerioAppKind()
+ customerio_track = CustomerioTrackKind()
+ customerio_webhook = CustomerioWebhookKind()
+ databricks = DatabricksKind()
+ email = EmailKind()
+ firebase = FirebaseKind()
+ github = GithubKind()
+ gitlab = GitlabKind()
+ google_ads = GoogleAdsKind()
+ google_analytics = GoogleAnalyticsKind()
+ google_calendar = GoogleCalendarKind()
+ google_cloud_service_account = GoogleCloudServiceAccountKind()
+ google_cloud_storage = GoogleCloudStorageKind()
+ google_pubsub = GooglePubsubKind()
+ google_search_console = GoogleSearchConsoleKind()
+ google_sheets = GoogleSheetsKind()
+ helpscout = HelpscoutKind()
+ hubspot = HubspotKind()
+ instagram = InstagramKind()
+ intercom = IntercomKind()
+ jira = JiraKind()
+ linear = LinearKind()
+ linkedin_ads = LinkedinAdsKind()
+ meta_ads = MetaAdsKind()
+ pardot = PardotKind()
+ pinterest_ads = PinterestAdsKind()
+ postgresql = PostgresqlKind()
+ posthog = PosthogKind()
+ reddit_ads = RedditAdsKind()
+ resend = ResendKind()
+ s3_compatible = S3CompatibleKind()
+ salesforce = SalesforceKind()
+ slack = SlackKind()
+ slack_posthog_code = SlackPosthogCodeKind()
+ snapchat = SnapchatKind()
+ snowflake = SnowflakeKind()
+ stripe = StripeKind()
+ tiktok_ads = TiktokAdsKind()
+ twilio = TwilioKind()
+ twitter_ads = TwitterAdsKind()
+ vercel = VercelKind()
+ youtube_analytics = YoutubeAnalyticsKind()
diff --git a/posthog/integrations/kinds_projection.py b/posthog/integrations/kinds_projection.py
new file mode 100644
index 000000000000..9be86b95bf50
--- /dev/null
+++ b/posthog/integrations/kinds_projection.py
@@ -0,0 +1,73 @@
+"""Render `kinds_generated.py`, the typed accessors of the integrations SDK, from `catalogue.py`.
+
+A renderer in the projection registry (`tools/hogli-commands/hogli_commands/projections.py`). Run
+`hogli build:projections --only integration-kinds` after a change to the catalogue.
+
+Each kind gets a class with one attribute per auth type that it supports, plus `any`. So mypy rejects
+an auth type that a kind does not support, for example `IntegrationManager.github.credentials`.
+"""
+
+from posthog.integrations.catalogue import KINDS
+from posthog.integrations.contracts import KindSpec
+from posthog.integrations.enums import AuthType
+
+OUTPUT = "posthog/integrations/kinds_generated.py"
+
+HEADER = '''"""Generated from `posthog/integrations/catalogue.py`. Do not edit by hand.
+
+Run `hogli build:projections --only integration-kinds` after a change to the catalogue. Import
+`IntegrationManager` from `posthog.integrations`, not the classes here.
+"""
+
+from enum import StrEnum
+
+from posthog.integrations.accessors import (
+ AuthAccessor,
+ IntegrationManagerBase,
+ IntegrationReader,
+ KindAccessor,
+ OAuthAccessor,
+)
+from posthog.integrations.enums import AuthType
+'''
+
+
+def _attribute(kind: str) -> str:
+ return kind.replace("-", "_")
+
+
+def _class_name(kind: str) -> str:
+ return "".join(part.capitalize() for part in kind.split("-")) + "Kind"
+
+
+def _kind_class(kind: str, spec: KindSpec) -> list[str]:
+ lines = ["", "", f"class {_class_name(kind)}(KindAccessor):", f' kind = "{kind}"']
+ for auth_type in AuthType:
+ if auth_type not in spec.supported:
+ continue
+ note = " # the default" if auth_type == spec.default else ""
+ if auth_type == AuthType.OAUTH:
+ lines.append(f' oauth = OAuthAccessor(kind="{kind}"){note}')
+ else:
+ lines.append(
+ f' {auth_type.value} = AuthAccessor(kind="{kind}", auth_type=AuthType.{auth_type.name}){note}'
+ )
+ lines.append(f' any = IntegrationReader(kinds=frozenset({{"{kind}"}}))')
+ return lines
+
+
+def render_kinds() -> str:
+ kinds = sorted(KINDS.items())
+ # A plain StrEnum: the OpenAPI enum names come from the model's `Integration.IntegrationKind`, and a
+ # second labeled class with the same name would make both names ambiguous.
+ lines = [HEADER, "", "class IntegrationKind(StrEnum):"]
+ lines += [f' {_attribute(kind).upper()} = "{kind}"' for kind, _ in kinds]
+ for kind, spec in kinds:
+ lines += _kind_class(kind, spec)
+ lines += ["", "", "class GeneratedIntegrationManager(IntegrationManagerBase):"]
+ lines += [f" {_attribute(kind)} = {_class_name(kind)}()" for kind, _ in kinds]
+ return "\n".join(lines) + "\n"
+
+
+def render() -> dict[str, str]:
+ return {OUTPUT: render_kinds()}
diff --git a/posthog/integrations/manager.py b/posthog/integrations/manager.py
new file mode 100644
index 000000000000..45839fa1c0c7
--- /dev/null
+++ b/posthog/integrations/manager.py
@@ -0,0 +1,38 @@
+"""The entry point of the integrations SDK. Import the names from `posthog.integrations`, not from here.
+
+ from posthog.integrations import IntegrationManager
+
+ IntegrationManager.github.oauth.get(team_id, integration_id) # one kind, one auth type: reads and writes
+ IntegrationManager.stripe.any.list(team_id) # one kind, every auth type: reads and deletes
+ IntegrationManager.any.get(team_id, integration_id) # every kind: reads and deletes
+ IntegrationManager.any.with_source_rows().delete(team_id, id) # every kind, warehouse source rows too
+ IntegrationManager.source("ab-tasty").token.create(...) # warehouse source rows of a registered kind
+
+A kind class has an attribute only for the auth types that the kind supports, so mypy rejects
+`IntegrationManager.github.credentials`. `.any` has no writes, because a new row has exactly one auth
+type. See `posthog/integrations/README.md`.
+"""
+
+from collections.abc import Mapping
+
+from posthog.integrations import accessors, source_kinds, usages
+from posthog.integrations.enums import AuthType
+from posthog.integrations.kinds_generated import GeneratedIntegrationManager
+
+IntegrationReader = accessors.IntegrationReader
+AuthAccessor = accessors.AuthAccessor
+OAuthAccessor = accessors.OAuthAccessor
+SourceKindAccessor = accessors.SourceKindAccessor
+UsageProvider = usages.UsageProvider
+
+IntegrationManager = GeneratedIntegrationManager()
+
+
+def register_source_kinds(kinds: Mapping[str, AuthType]) -> None:
+ """Register warehouse source kinds and the default auth type of each. Call it from `AppConfig.ready`."""
+ source_kinds.register_source_kinds(kinds)
+
+
+def register_usage_provider(provider: UsageProvider) -> None:
+ """Add a provider for `usages`. A product adds `backend/integration_usages.py` instead."""
+ usages.register_usage_provider(provider)
diff --git a/posthog/integrations/model_access_baseline.txt b/posthog/integrations/model_access_baseline.txt
new file mode 100644
index 000000000000..ea3da4b65320
--- /dev/null
+++ b/posthog/integrations/model_access_baseline.txt
@@ -0,0 +1,241 @@
+# Files that use the `Integration` model outside the integrations SDK, and how many findings each
+# file has. Rule: .semgrep/rules/devex/integration-model-through-sdk.yaml. The count only goes down:
+# after you move a file to the SDK, lower its count or remove its line.
+ee/api/integration.py 2
+ee/api/vercel/tasks.py 3
+ee/api/vercel/vercel_connect.py 5
+ee/api/vercel/vercel_sso.py 2
+ee/management/commands/backfill_vercel_secrets.py 2
+ee/tasks/slack.py 2
+ee/vercel/integration.py 6
+posthog/api/comments.py 2
+posthog/api/github_callback/installation_events.py 5
+posthog/api/github_callback/personal_finish.py 3
+posthog/api/github_callback/team_services.py 11
+posthog/api/integration.py 7
+posthog/api/integration_push_tokens.py 1
+posthog/api/organization_integration.py 5
+posthog/api/posthog_connection.py 2
+posthog/api/team/team_config.py 2
+posthog/api/user_integration.py 5
+posthog/domain_connect.py 3
+posthog/github/attribution.py 1
+posthog/github/installations.py 2
+posthog/management/commands/flag_legacy_oauth_integrations.py 3
+posthog/management/commands/migrate_ses_tenants.py 2
+posthog/management/commands/normalize_apns_integrations.py 3
+posthog/models/comment/slack_thread.py 1
+posthog/models/integration_repository_cache.py 2
+posthog/models/user.py 1
+posthog/models/user_integration.py 1
+posthog/ownership/github_files.py 2
+posthog/session_recordings/models/session_recording_external_reference.py 2
+posthog/session_recordings/session_recording_api.py 1
+posthog/session_recordings/session_recording_external_reference_api.py 2
+posthog/slack/channels.py 1
+posthog/tasks/comment_slack_sync.py 3
+posthog/tasks/integrations.py 11
+posthog/temporal/ai_observability/eval_reports/delivery.py 2
+products/ai_observability/backend/api/evaluation_reports.py 2
+products/alerts/backend/presentation/views/alert.py 2
+products/alerts_platform/backend/delivery/slack.py 2
+products/batch_exports/backend/debug/debugger.py 1
+products/batch_exports/backend/debug/start_session.py 1
+products/batch_exports/backend/facade/api.py 2
+products/batch_exports/backend/logic/exports.py 3
+products/batch_exports/backend/models/batch_export.py 1
+products/batch_exports/backend/presentation/views/batch_export/destinations.py 3
+products/batch_exports/backend/presentation/views/batch_export/exports.py 1
+products/batch_exports/backend/temporal/destinations/azure_blob_batch_export.py 2
+products/batch_exports/backend/temporal/destinations/bigquery_batch_export.py 2
+products/batch_exports/backend/temporal/destinations/databricks_batch_export.py 2
+products/batch_exports/backend/temporal/destinations/postgres_batch_export.py 2
+products/batch_exports/backend/temporal/destinations/redshift_batch_export.py 3
+products/batch_exports/backend/temporal/destinations/s3_batch_export.py 2
+products/batch_exports/backend/temporal/destinations/snowflake_batch_export.py 2
+products/billing_alerts/backend/facade/api.py 2
+products/business_knowledge/backend/github_repos.py 2
+products/canvas/backend/welcome.py 2
+products/cdp/backend/validation.py 3
+products/commands/backend/channels/slack/mentions.py 3
+products/commands/backend/logic/github.py 2
+products/commands/backend/logic/subjects.py 2
+products/conversations/backend/api/github_setup.py 4
+products/conversations/backend/facade/api.py 2
+products/conversations/backend/services/gmail_sync.py 7
+products/conversations/backend/tasks/github.py 2
+products/customer_analytics/backend/facade/api.py 5
+products/customer_analytics/backend/logic/calendar_sync.py 5
+products/customer_analytics/backend/logic/event_stream_destination.py 2
+products/customer_analytics/backend/logic/feature_request_github.py 2
+products/customer_analytics/backend/logic/feature_requests.py 2
+products/customer_analytics/backend/models/event_stream.py 1
+products/customer_analytics/backend/models/feature_request.py 1
+products/customer_analytics/backend/temporal/calendar_sync.py 2
+products/data_modeling/backend/models/github_sync_config.py 1
+products/engineering_analytics/backend/logic/job_log_insights.py 2
+products/engineering_analytics/backend/logic/job_logs/activity.py 2
+products/engineering_analytics/backend/logic/ownership_files.py 2
+products/engineering_analytics/backend/logic/quarantine.py 2
+products/error_tracking/backend/logic/__init__.py 1
+products/error_tracking/backend/logic/alerts.py 2
+products/error_tracking/backend/logic/external_references.py 4
+products/error_tracking/backend/logic/github_external_references.py 2
+products/error_tracking/backend/models.py 3
+products/error_tracking/backend/presentation/views/git_provider_file_link_resolver.py 3
+products/error_tracking/backend/temporal/alerts/delivery.py 2
+products/growth/backend/product_push/surfaces.py 2
+products/messaging/backend/models/optout_sync_config.py 3
+products/messaging/backend/models/provider_import_run.py 1
+products/messaging/backend/presentation/views/customerio_webhook.py 2
+products/messaging/backend/presentation/views/message_templates.py 2
+products/messaging/backend/remote_config.py 2
+products/messaging/backend/services/customerio_config.py 13
+products/messaging/backend/services/customerio_sync_service.py 1
+products/messaging/backend/services/customerio_webhook.py 1
+products/messaging/backend/services/transactional_email.py 2
+products/review_hog/backend/activity_logging.py 2
+products/review_hog/backend/automatic_reviews.py 3
+products/review_hog/backend/label_reviews.py 2
+products/review_hog/backend/ownership.py 2
+products/review_hog/backend/repository_settings.py 2
+products/review_hog/backend/reviewer/status_comment.py 2
+products/review_hog/backend/temporal/activities.py 2
+products/review_hog/backend/temporal/resolution.py 2
+products/signals/backend/github_optional.py 2
+products/signals/backend/inbox_summary.py 1
+products/signals/backend/models.py 4
+products/signals/backend/onboarding_completion.py 2
+products/signals/backend/report_generation/resolve_reviewers.py 3
+products/signals/backend/scout_harness/profile/builders.py 2
+products/signals/backend/scout_harness/serializers.py 2
+products/signals/backend/scout_harness/slack_delivery.py 2
+products/signals/backend/serializers.py 2
+products/signals/backend/slack_inbox_notifications.py 4
+products/signals/backend/slack_notification_targets.py 2
+products/signals/backend/temporal/agentic/select_repository.py 2
+products/signals/backend/tracker_issues.py 4
+products/signals/backend/views.py 2
+products/skills/backend/api/community_publish_services.py 1
+products/slack_app/backend/analytics.py 1
+products/slack_app/backend/api.py 28
+products/slack_app/backend/discussion_replies.py 1
+products/slack_app/backend/facade/api.py 3
+products/slack_app/backend/feature_flags.py 1
+products/slack_app/backend/inbox_channel.py 1
+products/slack_app/backend/logic/feature_tips/catalog.py 3
+products/slack_app/backend/logic/feature_tips/interactivity.py 2
+products/slack_app/backend/logic/feature_tips/offer.py 1
+products/slack_app/backend/logic/interview_recruitment.py 2
+products/slack_app/backend/logic/link_unfurl_charts.py 2
+products/slack_app/backend/logic/thread_join.py 1
+products/slack_app/backend/models.py 3
+products/slack_app/backend/onboarding.py 2
+products/slack_app/backend/services/commands.py 3
+products/slack_app/backend/services/followup_invite.py 1
+products/slack_app/backend/services/inbox_interactivity.py 2
+products/slack_app/backend/services/integration_resolver.py 6
+products/slack_app/backend/services/project_picker.py 1
+products/slack_app/backend/services/slack_app_home.py 6
+products/slack_app/backend/services/slack_app_home_stats.py 1
+products/slack_app/backend/services/slack_auth.py 1
+products/slack_app/backend/services/slack_messages.py 2
+products/slack_app/backend/services/slack_scopes.py 1
+products/slack_app/backend/services/slack_settings.py 1
+products/slack_app/backend/services/slack_user_info.py 3
+products/slack_app/backend/services/slack_welcome_messages.py 1
+products/slack_app/backend/services/turn_feedback.py 2
+products/slack_app/backend/signals.py 2
+products/slack_app/backend/slack_link_unfurl.py 1
+products/slack_app/backend/slack_thread.py 2
+products/slack_app/backend/slack_workflow_events.py 2
+products/slack_app/backend/tasks.py 2
+products/slack_app/backend/temporal/activities/billing.py 2
+products/slack_app/backend/temporal/activities/classifiers.py 6
+products/slack_app/backend/temporal/activities/messaging.py 9
+products/slack_app/backend/temporal/activities/model_router.py 2
+products/slack_app/backend/temporal/activities/onboarding.py 2
+products/slack_app/backend/temporal/activities/repo_selection.py 4
+products/slack_app/backend/temporal/activities/rules.py 2
+products/slack_app/backend/temporal/activities/task_creation.py 4
+products/slack_app/backend/temporal/activities/thread.py 2
+products/slack_app/backend/temporal/activities/untagged_question.py 2
+products/slack_app/backend/temporal/activities/user_resolution.py 2
+products/slack_app/backend/temporal/attachments.py 1
+products/slack_app/backend/views/slack_user_link.py 2
+products/slack_app/evals/eval_project_classifier.py 1
+products/stamphog/backend/logic/channel_resolution.py 2
+products/stamphog/backend/logic/slack_digest.py 3
+products/subscriptions/backend/logic/slack_subscriptions.py 2
+products/subscriptions/backend/logic/subscription_writes.py 2
+products/subscriptions/backend/models/subscription.py 1
+products/subscriptions/backend/temporal/activities.py 2
+products/subscriptions/backend/temporal/ai_subscription/delivery.py 1
+products/subscriptions/backend/temporal/delivery_common.py 1
+products/tasks/backend/facade/api.py 15
+products/tasks/backend/github_repository_access.py 2
+products/tasks/backend/logic/pull_request_titles.py 2
+products/tasks/backend/logic/repo_selection/agent.py 3
+products/tasks/backend/logic/services/comment_slack_dm.py 2
+products/tasks/backend/logic/services/dev_sandbox_context.py 2
+products/tasks/backend/logic/services/pr_reconciliation.py 1
+products/tasks/backend/logic/services/workflow_tasks.py 3
+products/tasks/backend/logic/task_review.py 3
+products/tasks/backend/max_tools.py 4
+products/tasks/backend/models.py 5
+products/tasks/backend/presentation/serializers.py 6
+products/tasks/backend/repository_readiness.py 2
+products/tasks/backend/repository_resolver.py 4
+products/tasks/backend/temporal/build_image/activities.py 2
+products/tasks/backend/temporal/create_snapshot/activities/create_sandbox.py 2
+products/tasks/backend/temporal/create_snapshot/activities/get_snapshot_context.py 2
+products/tasks/backend/temporal/create_snapshot/utils.py 2
+products/tasks/backend/temporal/process_task/activities/create_sandbox_from_snapshot.py 1
+products/tasks/backend/temporal/process_task/activities/feature_flags.py 2
+products/tasks/backend/temporal/process_task/activities/get_pr_context.py 2
+products/tasks/backend/temporal/process_task/activities/get_task_processing_context.py 1
+products/tasks/backend/temporal/process_task/activities/provision_sandbox.py 1
+products/tasks/backend/temporal/process_task/activities/refresh_sandbox_credentials.py 1
+products/tasks/backend/temporal/process_task/activities/send_followup_to_sandbox.py 1
+products/tasks/backend/temporal/process_task/activities/start_agent_server.py 2
+products/tasks/backend/temporal/process_task/sandbox_credentials.py 1
+products/tasks/backend/temporal/process_task/utils.py 3
+products/tasks/scripts/run_agent_in_docker.py 2
+products/visual_review/backend/logic/github_api.py 2
+products/visual_review/backend/logic/team_channels.py 2
+products/warehouse_sources/backend/models/external_data_destination.py 1
+products/warehouse_sources/backend/presentation/destination_connection_check.py 1
+products/warehouse_sources/backend/presentation/destination_connection_testing.py 1
+products/warehouse_sources/backend/presentation/views/external_data_destination.py 2
+products/warehouse_sources/backend/presentation/views/external_data_source/oauth_accounts.py 2
+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/bigquery.py 2
+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/databricks.py 2
+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/postgres.py 2
+products/warehouse_sources/backend/temporal/data_imports/pipelines/pipeline_v3/destinations_load/writers/redshift.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/oauth.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/bigquery.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/common/mixins.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/google_ads.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/google_ads/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/google_analytics.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/google_analytics/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/google_search_console.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/google_search_console/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/helpscout/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/linkedin_ads/linkedin_ads.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/linkedin_ads/source.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/meta_ads.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/meta_ads/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/resend/oauth.py 2
+products/warehouse_sources/backend/temporal/data_imports/sources/slack/source.py 1
+products/warehouse_sources/backend/temporal/data_imports/sources/twitter_ads/twitter_ads.py 1
+products/wizard/backend/logic/workers/repository_publisher.py 3
+products/workflows/backend/management/commands/delete_stale_ses_identity.py 2
+products/workflows/backend/management/commands/disable_ses_feedback_forwarding.py 2
+products/workflows/backend/services/customerio_import/content.py 3
+products/workflows/backend/services/data_suggestions/team_context.py 5
+products/workflows/backend/services/email_health.py 3
+products/workflows/backend/services/setup_status.py 2
+products/workflows/backend/services/transactional_email_workflows.py 2
+products/workflows/backend/tasks/ses_tenant_state.py 2
diff --git a/posthog/integrations/source_kinds.py b/posthog/integrations/source_kinds.py
new file mode 100644
index 000000000000..668805882548
--- /dev/null
+++ b/posthog/integrations/source_kinds.py
@@ -0,0 +1,46 @@
+"""Kinds that hold the typed-in credentials of warehouse sources.
+
+There are about a thousand of these kinds, so they get no generated accessor. The warehouse product
+registers them at startup, and callers reach their rows through `IntegrationManager.source(kind)`.
+A row of a source kind carries the source type in `config["warehouse_source_type"]`.
+"""
+
+import threading
+from collections.abc import Mapping
+
+from posthog.integrations.catalogue import KINDS
+from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY, KindSpec
+from posthog.integrations.enums import AuthType
+from posthog.models.integration.auth_types import register_kind_auth_types
+
+__all__ = ["SOURCE_MARKER_CONFIG_KEY", "is_source_kind", "register_source_kinds"]
+
+_lock = threading.Lock()
+_source_kinds: set[str] = set()
+
+
+def register_source_kinds(kinds: Mapping[str, AuthType]) -> None:
+ """Register warehouse source kinds with the default auth type of each.
+
+ A kind that is also in the catalogue, such as `stripe` or `postgresql`, keeps its catalogue default.
+ The auth type given here only joins the auth types the kind supports, so its source rows can be written.
+ """
+ with _lock:
+ for kind, auth_type in kinds.items():
+ catalogued = KINDS.get(kind)
+ if catalogued is None:
+ register_kind_auth_types(kind, KindSpec(label=kind, default=AuthType(auth_type)))
+ elif AuthType(auth_type) not in catalogued.supported:
+ register_kind_auth_types(
+ kind,
+ KindSpec(
+ label=catalogued.label,
+ default=catalogued.default,
+ others=catalogued.others | {AuthType(auth_type)},
+ ),
+ )
+ _source_kinds.add(kind)
+
+
+def is_source_kind(kind: str) -> bool:
+ return kind in _source_kinds
diff --git a/posthog/integrations/test/__init__.py b/posthog/integrations/test/__init__.py
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/posthog/integrations/test/test_accessors.py b/posthog/integrations/test/test_accessors.py
new file mode 100644
index 000000000000..4569fc96fdef
--- /dev/null
+++ b/posthog/integrations/test/test_accessors.py
@@ -0,0 +1,872 @@
+import time
+import threading
+from collections.abc import Callable, Iterable, Iterator
+from contextlib import contextmanager
+from types import SimpleNamespace
+from typing import Any
+
+from posthog.test.base import BaseTest
+from unittest.mock import patch
+
+from django.db import connection
+from django.test.utils import CaptureQueriesContext
+
+from asgiref.sync import async_to_sync
+from parameterized import parameterized
+
+from posthog.integrations import IntegrationManager, register_source_kinds, register_usage_provider, usages
+from posthog.integrations.accessors import AuthAccessor, IntegrationReader, OAuthAccessor, SourceRows
+from posthog.integrations.contracts import (
+ IntegrationConflict,
+ IntegrationInUse,
+ IntegrationNotFound,
+ IntegrationPatch,
+ IntegrationRecord,
+ IntegrationUsage,
+ UnknownIntegrationKind,
+ UnsupportedAuthType,
+ UnsupportedIntegrationFilter,
+)
+from posthog.integrations.enums import AuthType
+from posthog.models import Team
+from posthog.models.integration import ANY, Integration, auth_types
+from posthog.models.integration.auth_types import default_auth_type, kind_auth_types
+from posthog.models.integration.model import IntegrationQuerySet
+from posthog.models.integration.oauth import OauthIntegration
+
+MARKER = "warehouse_source_type"
+
+# name -> (kind, config). The name is also the external id.
+ROWS: dict[str, tuple[str, dict[str, Any]]] = {
+ "github_oauth": ("github", {"account": {"name": "a"}}),
+ "github_token": ("github", {"auth_type": "token", "account": {"name": "a"}}),
+ "slack_oauth": ("slack", {}),
+ "stripe_oauth": ("stripe", {}),
+ "stripe_token": ("stripe", {"auth_type": "token"}),
+ "stripe_source": ("stripe", {"auth_type": "token", MARKER: "Stripe"}),
+ "postgres": ("postgresql", {"host": "db.example.com"}),
+ "postgres_source": ("postgresql", {"host": "db.example.com", MARKER: "Postgres"}),
+ "ably_source": ("ably", {MARKER: "Ably"}),
+}
+SOURCE_ROWS = {name for name, (_kind, config) in ROWS.items() if MARKER in config}
+
+
+@contextmanager
+def _usage_providers(*providers: usages.UsageProvider) -> Iterator[None]:
+ with patch.object(usages, "_registered", []), patch.object(usages, "_product_providers", list(providers)):
+ yield
+
+
+class TestIntegrationAccessors(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.rows = {
+ name: Integration.objects._with_auth(ANY).create(
+ team=self.team, kind=kind, integration_id=name, config=config
+ )
+ for name, (kind, config) in ROWS.items()
+ }
+ self.other_team = Team.objects.create(organization=self.organization, name="Other")
+ self.other_row = Integration.objects._with_auth(ANY).create(
+ team=self.other_team, kind="github", integration_id="github_oauth", config={}
+ )
+
+ def _names(self, rows: Iterable[IntegrationRecord]) -> set[str]:
+ return {row.integration_id for row in rows if row.team_id == self.team.id and row.integration_id}
+
+ @parameterized.expand(
+ [
+ ("single_auth_default", lambda: IntegrationManager.github.oauth, {"github_oauth"}),
+ ("single_auth_non_default", lambda: IntegrationManager.github.token, {"github_token"}),
+ ("kind_any", lambda: IntegrationManager.github.any, {"github_oauth", "github_token"}),
+ ("skips_source_rows", lambda: IntegrationManager.stripe.token, {"stripe_token"}),
+ ("kind_any_skips_source_rows", lambda: IntegrationManager.stripe.any, {"stripe_oauth", "stripe_token"}),
+ ("credentials_default", lambda: IntegrationManager.postgresql.credentials, {"postgres"}),
+ ("everything_but_source_rows", lambda: IntegrationManager.any, set(ROWS) - SOURCE_ROWS),
+ ("everything_with_source_rows", lambda: IntegrationManager.any.with_source_rows(), set(ROWS)),
+ (
+ "kind_any_with_source_rows",
+ lambda: IntegrationManager.stripe.any.with_source_rows(),
+ {"stripe_oauth", "stripe_token", "stripe_source"},
+ ),
+ ("source_of_a_catalogue_kind", lambda: IntegrationManager.source("stripe").token, {"stripe_source"}),
+ ("source_any", lambda: IntegrationManager.source("stripe").any, {"stripe_source"}),
+ ("source_of_a_registered_kind", lambda: IntegrationManager.source("ably").token, {"ably_source"}),
+ ("source_other_auth_type", lambda: IntegrationManager.source("ably").credentials, set()),
+ (
+ "several_kinds_default_auth_type_by_default",
+ lambda: IntegrationManager.kinds("github", "slack"),
+ {"github_oauth", "slack_oauth"},
+ ),
+ (
+ "several_kinds_any_auth_type",
+ lambda: IntegrationManager.kinds("github", "slack", auth_types="any"),
+ {"github_oauth", "github_token", "slack_oauth"},
+ ),
+ # The source rows join the scope, and the auth type filter still applies to them.
+ (
+ "several_kinds_default_auth_type_with_source_rows",
+ lambda: IntegrationManager.kinds("stripe", "postgresql").with_source_rows(),
+ {"stripe_oauth", "postgres", "postgres_source"},
+ ),
+ (
+ "several_kinds_any_auth_type_with_source_rows",
+ lambda: IntegrationManager.kinds("stripe", auth_types="any").with_source_rows(),
+ {"stripe_oauth", "stripe_token", "stripe_source"},
+ ),
+ (
+ "several_kinds_default_auth_type",
+ lambda: IntegrationManager.kinds("github", "stripe", "postgresql", auth_types="default"),
+ {"github_oauth", "stripe_oauth", "postgres"},
+ ),
+ (
+ "several_kinds_listed_auth_types",
+ lambda: IntegrationManager.kinds("github", "stripe", "postgresql", auth_types=[AuthType.TOKEN]),
+ {"github_token", "stripe_token"},
+ ),
+ ]
+ )
+ def test_list_returns_only_rows_in_scope(
+ self, _name: str, accessor: Callable[[], IntegrationReader], expected: set[str]
+ ) -> None:
+ assert self._names(accessor().list(self.team.id)) == expected
+
+ @parameterized.expand(
+ [
+ ("other_auth_type", lambda: IntegrationManager.github.oauth, "github_token"),
+ ("source_row_through_named_accessor", lambda: IntegrationManager.stripe.any, "stripe_source"),
+ ("named_row_through_source_accessor", lambda: IntegrationManager.source("stripe").token, "stripe_token"),
+ ("other_kind", lambda: IntegrationManager.slack.oauth, "github_oauth"),
+ ("source_row_through_any", lambda: IntegrationManager.any, "stripe_source"),
+ ("other_auth_type_through_kinds", lambda: IntegrationManager.kinds("github"), "github_token"),
+ ]
+ )
+ def test_get_raises_for_a_row_out_of_scope(
+ self, _name: str, accessor: Callable[[], IntegrationReader], row_name: str
+ ) -> None:
+ with self.assertRaises(IntegrationNotFound):
+ accessor().get(self.team.id, self.rows[row_name].id)
+
+ @parameterized.expand([("missing_id", 10**9), ("not_a_number", "abc"), ("none", None)])
+ def test_get_raises_for_a_bad_id(self, _name: str, row_id: Any) -> None:
+ with self.assertRaises(IntegrationNotFound):
+ IntegrationManager.any.get(self.team.id, row_id)
+
+ def test_a_stored_default_auth_type_reads_as_the_default(self) -> None:
+ row = self.rows["github_oauth"]
+ # `QuerySet.update()` skips `save()`, so it can store the default auth type of the kind.
+ Integration._base_manager.filter(id=row.id).update(config={"auth_type": "oauth"})
+
+ assert IntegrationManager.github.oauth.get(self.team.id, row.id).id == row.id
+ assert self._names(IntegrationManager.kinds("github", auth_types="default").list(self.team.id)) == {
+ "github_oauth"
+ }
+ with self.assertRaises(IntegrationNotFound):
+ IntegrationManager.github.token.get(self.team.id, row.id)
+
+ def test_get_does_not_cross_teams(self) -> None:
+ with self.assertRaises(IntegrationNotFound):
+ IntegrationManager.github.oauth.get(self.team.id, self.other_row.id)
+
+ assert IntegrationManager.github.oauth.get(self.other_team.id, self.other_row.id) == self.other_row
+
+ def test_get_many_drops_ids_out_of_scope(self) -> None:
+ ids = [self.rows["github_oauth"].id, self.rows["github_token"].id, self.other_row.id, 10**9]
+
+ assert IntegrationManager.github.oauth.get_many(self.team.id, ids) == {
+ self.rows["github_oauth"].id: self.rows["github_oauth"]
+ }
+
+ @parameterized.expand(
+ [
+ ("external_id", {"external_id": "github_token"}, "github_token"),
+ ("nested_config_value", {"where": {"account": {"name": "a"}}}, None),
+ ("scalar_config_value", {"where": {"auth_type_hint": None}}, None),
+ ("oldest_without_filters", {}, "github_oauth"),
+ ("no_match", {"external_id": "missing"}, None),
+ ]
+ )
+ def test_find(self, _name: str, kwargs: dict[str, Any], expected: str | None) -> None:
+ if isinstance(next(iter(kwargs.get("where", {}).values()), None), dict):
+ with self.assertRaises(UnsupportedIntegrationFilter):
+ IntegrationManager.github.any.find(self.team.id, **kwargs)
+ return
+ row = IntegrationManager.github.any.find(self.team.id, **kwargs)
+
+ assert (row.integration_id if row else None) == expected
+
+ @parameterized.expand(
+ [
+ ("string", {"host": "db.example.com"}, True),
+ ("string_mismatch", {"host": "other.example.com"}, False),
+ ("missing_key_matches_none", {"port": None}, False),
+ ("key_named_like_a_lookup", {"contains": "x"}, False),
+ ]
+ )
+ def test_exists_compares_config_keys_for_equality(self, _name: str, where: dict[str, Any], expected: bool) -> None:
+ assert IntegrationManager.postgresql.credentials.exists(self.team.id, where=where) is expected
+
+ def test_find_returns_the_oldest_row_and_list_can_reverse(self) -> None:
+ newer = IntegrationManager.github.oauth.create(self.team.id, external_id="newer", config={})
+
+ assert IntegrationManager.github.oauth.find(self.team.id) == self.rows["github_oauth"]
+ assert IntegrationManager.github.oauth.list(self.team.id, newest_first=True)[0] == newer
+
+ @parameterized.expand(
+ [
+ ("every_team", None, {"github_oauth": 2}),
+ ("one_team", "other", {"github_oauth": 1}),
+ ]
+ )
+ def test_list_by_external_id(self, _name: str, team_ids: str | None, expected: dict[str, int]) -> None:
+ rows = IntegrationManager.github.oauth.list_by_external_id(
+ "github_oauth", team_ids=[self.other_team.id] if team_ids else None
+ )
+
+ assert {"github_oauth": len(rows)} == expected
+ assert all(row.integration_id == "github_oauth" for row in rows)
+
+ @parameterized.expand(
+ [
+ ("both_teams", ["own", "other"], "github_oauth"),
+ ("other_team_only", ["other"], "other"),
+ ("empty", [], None),
+ ("unknown_team", [10**9], None),
+ ]
+ )
+ def test_find_and_exists_with_team_ids(self, _name: str, teams: list[Any], expected: str | None) -> None:
+ by_name = {"own": self.team.id, "other": self.other_team.id}
+ team_ids = [by_name.get(team, team) for team in teams]
+ expected_row = {"github_oauth": self.rows["github_oauth"], "other": self.other_row}.get(expected or "")
+
+ row = IntegrationManager.github.oauth.find(team_ids=team_ids, external_id="github_oauth")
+
+ assert row == expected_row
+ assert IntegrationManager.github.oauth.exists(team_ids=iter(team_ids)) is (expected_row is not None)
+ assert async_to_sync(IntegrationManager.github.oauth.afind)(team_ids=team_ids) == expected_row
+ assert async_to_sync(IntegrationManager.github.oauth.aexists)(team_ids=team_ids) is (expected_row is not None)
+
+ @parameterized.expand(
+ [
+ ("neither", {}),
+ ("both", {"team_id": 1, "team_ids": [1]}),
+ ]
+ )
+ def test_find_and_exists_need_exactly_one_team_argument(self, _name: str, kwargs: dict[str, Any]) -> None:
+ for read in (IntegrationManager.github.oauth.find, IntegrationManager.github.oauth.exists):
+ with self.assertRaises(ValueError):
+ read(**kwargs)
+
+ @parameterized.expand(
+ [
+ ("own_row", "github_oauth", lambda self: self.rows["github_oauth"], True),
+ ("row_of_another_team", "github_oauth", lambda self: self.other_row, True),
+ ("row_with_another_external_id", "github_oauth", lambda self: self.rows["github_token"], False),
+ ("another_workspace_id_for_the_external_id", "github_token", lambda self: self.other_row, False),
+ ("missing_id", "github_oauth", lambda self: SimpleNamespace(id=10**9), False),
+ ("not_a_number", "github_oauth", lambda self: SimpleNamespace(id="abc"), False),
+ ]
+ )
+ def test_get_by_external_id(self, _name: str, external_id: str, row: Callable[[Any], Any], found: bool) -> None:
+ expected = row(self)
+
+ if found:
+ assert IntegrationManager.github.oauth.get_by_external_id(external_id, expected.id) == expected
+ else:
+ with self.assertRaises(IntegrationNotFound):
+ IntegrationManager.github.oauth.get_by_external_id(external_id, expected.id)
+
+ @parameterized.expand(
+ [
+ ("every_team", lambda: IntegrationManager.github.oauth, "github_oauth", 2),
+ ("other_auth_type_is_out_of_scope", lambda: IntegrationManager.github.oauth, "github_token", 0),
+ ("kind_any", lambda: IntegrationManager.github.any, "github_token", 1),
+ ("source_row_is_out_of_scope", lambda: IntegrationManager.stripe.any, "stripe_source", 0),
+ ("missing", lambda: IntegrationManager.github.oauth, "missing", 0),
+ ]
+ )
+ def test_count_and_exists_by_external_id(
+ self, _name: str, accessor: Callable[[], IntegrationReader], external_id: str, expected: int
+ ) -> None:
+ assert accessor().count_by_external_id(external_id) == expected
+ assert accessor().exists_by_external_id(external_id) is (expected > 0)
+
+ @parameterized.expand(
+ [
+ ("get", lambda self, a: [a.get(self.team.id, self.rows["github_oauth"].id)]),
+ ("get_many", lambda self, a: list(a.get_many(self.team.id, [self.rows["github_oauth"].id]).values())),
+ ("find", lambda self, a: [a.find(self.team.id)]),
+ ("list", lambda self, a: a.list(self.team.id)),
+ ("list_by_external_id", lambda self, a: a.list_by_external_id("github_oauth")),
+ ("get_by_external_id", lambda self, a: [a.get_by_external_id("github_oauth", self.other_row.id)]),
+ ("update", lambda self, a: [a.update(self.team.id, self.rows["github_oauth"].id, errors="")]),
+ ("upsert_of_an_existing_row", lambda self, a: [a.upsert(self.team.id, "github_oauth", config={})[0]]),
+ ]
+ )
+ def test_reads_defer_the_repository_cache(
+ self, _name: str, read: Callable[[Any, AuthAccessor], list[Integration]]
+ ) -> None:
+ Integration._base_manager.filter(integration_id="github_oauth").update(repository_cache=[{"name": "repo"}])
+
+ rows = read(self, IntegrationManager.github.oauth)
+
+ assert rows
+ for row in rows:
+ assert {"repository_cache", "repository_cache_updated_at"} <= row.get_deferred_fields()
+ assert row.repository_cache == [{"name": "repo"}]
+
+ def test_with_source_rows_refuses_a_source_only_accessor(self) -> None:
+ with self.assertRaises(ValueError):
+ IntegrationManager.source("stripe").any.with_source_rows()
+
+ @parameterized.expand(
+ [
+ ("list_by_external_id", lambda: IntegrationManager.any.list_by_external_id("github_oauth")),
+ ("delete_by_external_id", lambda: IntegrationManager.any.delete_by_external_id("github_oauth")),
+ ("get_by_external_id", lambda: IntegrationManager.any.get_by_external_id("github_oauth", 1)),
+ ("count_by_external_id", lambda: IntegrationManager.any.count_by_external_id("github_oauth")),
+ ("exists_by_external_id", lambda: IntegrationManager.any.exists_by_external_id("github_oauth")),
+ ]
+ )
+ def test_cross_team_operations_need_a_kind(self, _name: str, call: Callable[[], Any]) -> None:
+ with self.assertRaises(ValueError):
+ call()
+
+ assert Integration._base_manager.filter(integration_id="github_oauth").count() == 2
+ assert len(IntegrationManager.kinds("github").list_by_external_id("github_oauth")) == 2
+
+ @parameterized.expand(
+ [
+ ("default_auth_type_stores_nothing", lambda: IntegrationManager.github.oauth, {}),
+ ("non_default_auth_type_is_stored", lambda: IntegrationManager.github.token, {"auth_type": "token"}),
+ (
+ "token_on_a_credentials_kind",
+ lambda: IntegrationManager.databricks.token,
+ {"auth_type": "token"},
+ ),
+ ("credentials_default", lambda: IntegrationManager.databricks.credentials, {}),
+ ("source_kind_default", lambda: IntegrationManager.source("ably").token, {MARKER: "Ably"}),
+ (
+ "source_row_of_a_catalogue_kind",
+ lambda: IntegrationManager.source("stripe").token,
+ {MARKER: "Stripe", "auth_type": "token"},
+ ),
+ ]
+ )
+ def test_create_stores_the_auth_type_only_when_it_is_not_the_default(
+ self, _name: str, accessor: Callable[[], Any], expected_config: dict[str, Any]
+ ) -> None:
+ config = {MARKER: expected_config[MARKER]} if MARKER in expected_config else {}
+
+ row = accessor().create(
+ self.team.id, external_id="new", config=config, sensitive_config={"secret": "s"}, created_by_id=self.user.id
+ )
+
+ stored = Integration._base_manager.get(id=row.id)
+ assert (stored.config, stored.sensitive_config, stored.created_by_id) == (
+ expected_config,
+ {"secret": "s"},
+ self.user.id,
+ )
+ assert accessor().get(self.team.id, row.id) == row
+
+ @parameterized.expand(
+ [
+ ("auth_type_in_config", lambda: IntegrationManager.github.token, {"auth_type": "token"}, ValueError),
+ ("marker_through_named_accessor", lambda: IntegrationManager.stripe.token, {MARKER: "Stripe"}, ValueError),
+ ("source_row_without_marker", lambda: IntegrationManager.source("ably").token, {}, ValueError),
+ (
+ "auth_type_the_kind_does_not_support",
+ lambda: IntegrationManager.source("postgresql").token,
+ {MARKER: "Postgres"},
+ UnsupportedAuthType,
+ ),
+ ]
+ )
+ def test_create_rejects(
+ self, _name: str, accessor: Callable[[], Any], config: dict[str, Any], error: type[Exception]
+ ) -> None:
+ with self.assertRaises(error):
+ accessor().create(self.team.id, external_id="new", config=config)
+
+ @parameterized.expand(
+ [
+ ("same_scope", lambda: IntegrationManager.github.oauth, "github_oauth", {}),
+ ("other_auth_type", lambda: IntegrationManager.github.token, "github_oauth", {}),
+ ("source_row_through_named_accessor", lambda: IntegrationManager.stripe.oauth, "stripe_source", {}),
+ (
+ "named_row_through_source_accessor",
+ lambda: IntegrationManager.source("stripe").token,
+ "stripe_token",
+ {MARKER: "Stripe"},
+ ),
+ ]
+ )
+ def test_create_raises_conflict_on_the_unique_key(
+ self, _name: str, accessor: Callable[[], Any], external_id: str, config: dict[str, Any]
+ ) -> None:
+ with self.assertRaises(IntegrationConflict):
+ accessor().create(self.team.id, external_id=external_id, config=config)
+
+ # The failed insert rolls back to a savepoint, so the transaction stays usable.
+ assert IntegrationManager.any.with_source_rows().list(self.team.id, external_id=external_id) == [
+ self.rows[external_id]
+ ]
+
+ def test_upsert_creates_then_replaces_the_config(self) -> None:
+ row, created = IntegrationManager.github.token.upsert(self.team.id, "new", config={"a": 1})
+ again, created_again = IntegrationManager.github.token.upsert(
+ self.team.id, "new", config={"b": 2}, sensitive_config={"token": "t"}, created_by_id=self.user.id
+ )
+
+ again = Integration._base_manager.get(id=again.id)
+ assert (created, created_again, again.id) == (True, False, row.id)
+ assert (again.config, again.sensitive_config, again.created_by_id) == (
+ {"b": 2, "auth_type": "token"},
+ {"token": "t"},
+ self.user.id,
+ )
+
+ @parameterized.expand(
+ [
+ ("other_auth_type", lambda: IntegrationManager.github.token, "github_oauth", {}),
+ ("source_row", lambda: IntegrationManager.stripe.token, "stripe_source", {}),
+ (
+ "named_row_through_source",
+ lambda: IntegrationManager.source("stripe").token,
+ "stripe_token",
+ {MARKER: "Stripe"},
+ ),
+ ]
+ )
+ def test_upsert_raises_on_a_row_out_of_scope(
+ self, _name: str, accessor: Callable[[], Any], external_id: str, config: dict[str, Any]
+ ) -> None:
+ before = Integration._base_manager.get(id=self.rows[external_id].id).config
+
+ with self.assertRaises(IntegrationConflict):
+ accessor().upsert(self.team.id, external_id, config=config)
+
+ assert Integration._base_manager.get(id=self.rows[external_id].id).config == before
+
+ def test_update_merges_and_removes_keys(self) -> None:
+ row = self.rows["github_token"]
+
+ IntegrationManager.github.token.update(
+ self.team.id,
+ row.id,
+ config_patch={"login": "octocat"},
+ config_remove=["account", "missing"],
+ sensitive_config_patch={"token": "t"},
+ errors="TOKEN_REFRESH_FAILED",
+ )
+
+ stored = Integration._base_manager.get(id=row.id)
+ assert (stored.config, stored.sensitive_config, stored.errors) == (
+ {"auth_type": "token", "login": "octocat"},
+ {"token": "t"},
+ "TOKEN_REFRESH_FAILED",
+ )
+
+ @parameterized.expand(
+ [
+ ("patch_auth_type", {"config_patch": {"auth_type": "oauth"}}, ValueError),
+ ("remove_auth_type", {"config_remove": ["auth_type"]}, ValueError),
+ ("patch_marker", {"config_patch": {MARKER: "Stripe"}}, ValueError),
+ ]
+ )
+ def test_update_rejects_reserved_keys(self, _name: str, kwargs: dict[str, Any], error: type[Exception]) -> None:
+ with self.assertRaises(error):
+ IntegrationManager.github.token.update(self.team.id, self.rows["github_token"].id, **kwargs)
+
+ def test_upsert_without_created_by_keeps_the_creator(self) -> None:
+ row = IntegrationManager.github.token.create(
+ self.team.id, external_id="new", config={}, created_by_id=self.user.id
+ )
+
+ again, created = IntegrationManager.github.token.upsert(self.team.id, "new", config={"b": 2})
+
+ assert (created, again.id) == (False, row.id)
+ assert Integration._base_manager.get(id=row.id).created_by_id == self.user.id
+
+ def test_upsert_updates_the_row_that_a_concurrent_insert_won(self) -> None:
+ original_first = IntegrationQuerySet.first
+ reads: list[int] = []
+
+ def first_after_a_competing_insert(queryset: Any) -> Integration | None:
+ result = original_first(queryset)
+ reads.append(1)
+ if len(reads) == 1:
+ assert result is None
+ Integration.objects._with_auth(ANY).create(
+ team=self.team, kind="github", integration_id="raced", config={"auth_type": "token", "old": True}
+ )
+ return result
+
+ with patch.object(IntegrationQuerySet, "first", autospec=True, side_effect=first_after_a_competing_insert):
+ row, created = IntegrationManager.github.token.upsert(
+ self.team.id, "raced", config={"new": True}, created_by_id=self.user.id
+ )
+
+ stored = Integration._base_manager.filter(team=self.team, kind="github", integration_id="raced")
+ saved = stored.get()
+ assert (created, stored.count(), saved.id) == (False, 1, row.id)
+ assert (saved.config, saved.created_by_id) == ({"new": True, "auth_type": "token"}, self.user.id)
+
+ @parameterized.expand(
+ [
+ ("nothing", {}),
+ ("empty_collections", {"config_patch": {}, "config_remove": (), "sensitive_config_patch": {}}),
+ ]
+ )
+ def test_an_empty_patch_writes_nothing(self, _name: str, kwargs: dict[str, Any]) -> None:
+ row = self.rows["github_oauth"]
+ before = Integration._base_manager.get(id=row.id)
+
+ with patch.object(Integration, "save") as save:
+ updated = IntegrationManager.github.oauth.update(self.team.id, row.id, **kwargs)
+
+ save.assert_not_called()
+ assert (updated.config, updated.sensitive_config, updated.errors) == (
+ before.config,
+ before.sensitive_config,
+ before.errors,
+ )
+
+ @parameterized.expand(
+ [
+ ("int_to_bool", 1, True),
+ ("bool_to_int", True, 1),
+ ("int_to_float", 1, 1.0),
+ ("zero_to_false", 0, False),
+ ("nested_int_to_bool", {"a": 1}, {"a": True}),
+ ]
+ )
+ def test_update_writes_a_value_of_another_json_type(self, _name: str, stored: object, new: object) -> None:
+ row = self.rows["github_oauth"]
+ IntegrationManager.github.oauth.update(self.team.id, row.id, config_patch={"flag": stored})
+
+ IntegrationManager.github.oauth.update(self.team.id, row.id, config_patch={"flag": new})
+
+ saved = Integration._base_manager.get(id=row.id).config["flag"]
+ assert (saved, type(saved)) == (new, type(new))
+
+ @parameterized.expand(
+ [
+ ("auth_type_without_kinds", {"kinds": None, "auth_type": AuthType.TOKEN}),
+ ("auth_type_with_empty_kinds", {"kinds": frozenset(), "auth_type": AuthType.TOKEN}),
+ ]
+ )
+ def test_a_reader_with_an_auth_type_needs_kinds(self, _name: str, kwargs: dict[str, Any]) -> None:
+ with self.assertRaises(ValueError):
+ IntegrationReader(**kwargs)
+
+ def test_a_write_accessor_refuses_to_include_source_rows(self) -> None:
+ with self.assertRaises(ValueError):
+ AuthAccessor(kind="github", auth_type=AuthType.OAUTH, source_rows=SourceRows.INCLUDE)
+
+ def test_update_raises_for_a_row_out_of_scope(self) -> None:
+ with self.assertRaises(IntegrationNotFound):
+ IntegrationManager.github.oauth.update(self.team.id, self.rows["github_token"].id, errors="")
+
+ def _resend_row(self, *, refreshed_ago: int) -> Integration:
+ return Integration.objects._with_auth(ANY).create(
+ team=self.team,
+ kind="resend",
+ integration_id="resend-account",
+ config={"expires_in": 900, "refreshed_at": int(time.time()) - refreshed_ago},
+ sensitive_config={"access_token": "old", "refresh_token": "r1"},
+ )
+
+ @parameterized.expand(
+ [
+ ("expired", 800, None, True),
+ ("fresh", 10, None, False),
+ ("fresh_but_the_caller_holds_the_stored_token", 10, "old", True),
+ ("fresh_and_another_worker_rotated_the_token", 10, "older", False),
+ ("expired_and_another_worker_rotated_the_token", 800, "older", True),
+ ]
+ )
+ def test_refresh_access_token_refreshes_only_when_needed(
+ self, _name: str, refreshed_ago: int, stale_access_token: str | None, refreshes: bool
+ ) -> None:
+ row = self._resend_row(refreshed_ago=refreshed_ago)
+
+ def rotate(oauth: OauthIntegration) -> None:
+ oauth.integration.sensitive_config = {"access_token": "new", "refresh_token": "r2"}
+ oauth.integration.save()
+
+ with (
+ patch.object(OauthIntegration, "refresh_access_token", autospec=True, side_effect=rotate) as refresh,
+ CaptureQueriesContext(connection) as queries,
+ ):
+ result = IntegrationManager.resend.oauth.refresh_access_token(
+ self.team.id, row.id, stale_access_token=stale_access_token
+ )
+
+ expected_token = "new" if refreshes else "old"
+ assert refresh.call_count == (1 if refreshes else 0)
+ assert (result.id, result.sensitive_config["access_token"]) == (row.id, expected_token)
+ assert Integration._base_manager.get(id=row.id).sensitive_config["access_token"] == expected_token
+ assert any("FOR UPDATE" in query["sql"] for query in queries.captured_queries)
+
+ @parameterized.expand(
+ [
+ ("kind_without_a_refresh", lambda: IntegrationManager.github.oauth, "github_oauth", ValueError),
+ ("row_of_another_kind", lambda: IntegrationManager.stripe.oauth, "github_oauth", IntegrationNotFound),
+ (
+ "row_with_another_auth_type",
+ lambda: IntegrationManager.stripe.oauth,
+ "stripe_token",
+ IntegrationNotFound,
+ ),
+ ]
+ )
+ def test_refresh_access_token_raises(
+ self, _name: str, accessor: Callable[[], Any], row_name: str, error: type[Exception]
+ ) -> None:
+ with patch.object(OauthIntegration, "refresh_access_token") as refresh:
+ with self.assertRaises(error):
+ accessor().refresh_access_token(self.team.id, self.rows[row_name].id)
+
+ refresh.assert_not_called()
+
+ @parameterized.expand(
+ [
+ # Queries: the savepoint, the locked read, the one bulk update, the savepoint release.
+ ("changes_every_team", [], IntegrationPatch(config_patch={"suspended": True}), 2, 4),
+ ("skips_a_row_that_already_matches", ["other"], IntegrationPatch(config_patch={"suspended": True}), 1, 4),
+ (
+ "writes_nothing_when_nothing_changes",
+ ["own", "other"],
+ IntegrationPatch(config_patch={"suspended": True}),
+ 0,
+ 3,
+ ),
+ ("removes_a_key", ["own"], IntegrationPatch(config_remove=("suspended",)), 1, 4),
+ ("errors_only", [], IntegrationPatch(errors="SUSPENDED"), 2, 4),
+ ("empty_patch", [], IntegrationPatch(), 0, 3),
+ ]
+ )
+ def test_update_by_external_id_reaches_every_team(
+ self, _name: str, already_suspended: list[str], patch: IntegrationPatch, changed: int, queries: int
+ ) -> None:
+ by_name = {"own": self.rows["github_oauth"], "other": self.other_row}
+ for name in already_suspended:
+ Integration._base_manager.filter(id=by_name[name].id).update(config={"suspended": True})
+
+ with self.assertNumQueries(queries):
+ count = IntegrationManager.github.oauth.update_by_external_id("github_oauth", patch)
+
+ stored = Integration._base_manager.filter(integration_id="github_oauth")
+ assert count == changed
+ if patch.config_patch:
+ assert {row.config.get("suspended") for row in stored} == {True}
+ if patch.config_remove:
+ assert {row.config.get("suspended") for row in stored} == {None}
+ if patch.errors:
+ assert {row.errors for row in stored} == {"SUSPENDED"}
+ assert "suspended" not in Integration._base_manager.get(id=self.rows["github_token"].id).config
+
+ def test_update_by_external_id_rejects_reserved_keys(self) -> None:
+ with self.assertRaises(ValueError):
+ IntegrationManager.github.oauth.update_by_external_id(
+ "github_oauth", IntegrationPatch(config_patch={MARKER: "Github"})
+ )
+
+ def test_delete_checks_usages(self) -> None:
+ row = self.rows["github_oauth"]
+
+ def provider(team_id: int, integration_id: int) -> list[IntegrationUsage]:
+ return [IntegrationUsage(type="workflow", id="1", name="Flow")] if integration_id == row.id else []
+
+ with _usage_providers(provider):
+ with self.assertRaises(IntegrationInUse) as raised:
+ IntegrationManager.github.oauth.delete(self.team.id, row.id)
+ assert raised.exception.usages == [IntegrationUsage(type="workflow", id="1", name="Flow")]
+
+ IntegrationManager.github.oauth.delete(self.team.id, row.id, ignore_usages=True)
+ IntegrationManager.github.token.delete(self.team.id, self.rows["github_token"].id)
+
+ assert not Integration._base_manager.filter(id__in=[row.id, self.rows["github_token"].id]).exists()
+
+ def test_delete_raises_for_a_row_out_of_scope(self) -> None:
+ with _usage_providers(), self.assertRaises(IntegrationNotFound):
+ IntegrationManager.github.oauth.delete(self.team.id, self.rows["github_token"].id)
+
+ def test_delete_by_external_id_reaches_every_team(self) -> None:
+ assert IntegrationManager.github.oauth.delete_by_external_id("github_oauth") == 2
+ assert not Integration._base_manager.filter(integration_id="github_oauth").exists()
+ assert Integration._base_manager.filter(id=self.rows["github_token"].id).exists()
+
+ def test_delete_by_external_id_rolls_back_every_row_when_one_delete_fails(self) -> None:
+ original_delete = Integration.delete
+ calls: list[int] = []
+
+ def delete_then_fail_on_the_second_row(row: Integration, *args: Any, **kwargs: Any) -> Any:
+ calls.append(row.id)
+ if len(calls) == 2:
+ raise RuntimeError("delete failed")
+ return original_delete(row, *args, **kwargs)
+
+ with patch.object(Integration, "delete", autospec=True, side_effect=delete_then_fail_on_the_second_row):
+ with self.assertRaises(RuntimeError):
+ IntegrationManager.github.oauth.delete_by_external_id("github_oauth")
+
+ assert len(calls) == 2
+ assert Integration._base_manager.filter(integration_id="github_oauth").count() == 2
+
+ def test_usages_reads_registered_and_product_providers_in_order(self) -> None:
+ row = self.rows["postgres"]
+ product_module = SimpleNamespace(
+ integration_usages=lambda team_id, integration_id: [
+ IntegrationUsage(type="warehouse_source", id="2", name="b"),
+ IntegrationUsage(type="warehouse_source", id="1", name="a"),
+ ]
+ )
+
+ with (
+ patch.object(usages, "_registered", []),
+ patch.object(usages, "_product_providers", None),
+ patch.object(usages, "load_product_modules", return_value=iter([product_module])) as load,
+ ):
+ register_usage_provider(
+ lambda team_id, integration_id: [IntegrationUsage(type="batch_export", id="9", name="z")]
+ )
+ first = IntegrationManager.postgresql.credentials.usages(self.team.id, row.id)
+ IntegrationManager.postgresql.credentials.usages(self.team.id, row.id)
+
+ assert [(usage.type, usage.name) for usage in first] == [
+ ("batch_export", "z"),
+ ("warehouse_source", "a"),
+ ("warehouse_source", "b"),
+ ]
+ load.assert_called_once_with("integration_usages")
+
+ def test_a_provider_module_that_registers_at_import_does_not_deadlock(self) -> None:
+ def import_a_module_that_registers(_name: str) -> Iterable[SimpleNamespace]:
+ register_usage_provider(lambda team_id, integration_id: [])
+ return [SimpleNamespace(integration_usages=lambda team_id, integration_id: [])]
+
+ with (
+ patch.object(usages, "_registered", []),
+ patch.object(usages, "_product_providers", None),
+ patch.object(usages, "load_product_modules", side_effect=import_a_module_that_registers),
+ ):
+ # A daemon thread, so a deadlock fails the test instead of hanging the run.
+ loader = threading.Thread(target=usages.integration_usages, args=(self.team.id, 1), daemon=True)
+ loader.start()
+ loader.join(timeout=10)
+
+ assert not loader.is_alive()
+ assert len(usages._registered) == 1
+
+ @parameterized.expand(
+ [
+ ("aget", lambda t, r: ("get", (t, r), {})),
+ ("aget_many", lambda t, r: ("get_many", (t, [r]), {})),
+ ("afind", lambda t, r: ("find", (t,), {"external_id": "github_oauth"})),
+ ("alist", lambda t, r: ("list", (t,), {})),
+ ("aexists", lambda t, r: ("exists", (t,), {"external_id": "github_oauth"})),
+ ("ausages", lambda t, r: ("usages", (t, r), {})),
+ ]
+ )
+ def test_async_reads_match_the_sync_reads(
+ self, name: str, args: Callable[[int, int], tuple[str, tuple[Any, ...], dict[str, Any]]]
+ ) -> None:
+ accessor = IntegrationManager.github.oauth
+ sync_name, call_args, call_kwargs = args(self.team.id, self.rows["github_oauth"].id)
+
+ with _usage_providers(
+ lambda team_id, integration_id: [IntegrationUsage(type="warehouse_source", id="1", name="a")]
+ ):
+ expected = getattr(accessor, sync_name)(*call_args, **call_kwargs)
+ actual = async_to_sync(getattr(accessor, name))(*call_args, **call_kwargs)
+
+ assert actual == expected
+ assert actual not in (None, [], {}, False)
+
+ def test_async_delete_deletes_and_honors_usages(self) -> None:
+ accessor = IntegrationManager.github.oauth
+ row = self.rows["github_oauth"]
+
+ with _usage_providers(
+ lambda team_id, integration_id: [IntegrationUsage(type="warehouse_source", id="1", name="a")]
+ ):
+ with self.assertRaises(IntegrationInUse):
+ async_to_sync(accessor.adelete)(self.team.id, row.id)
+ assert Integration._base_manager.filter(id=row.id).exists()
+ async_to_sync(accessor.adelete)(self.team.id, row.id, ignore_usages=True)
+
+ assert not Integration._base_manager.filter(id=row.id).exists()
+
+ def test_async_writes_match_the_sync_writes(self) -> None:
+ accessor = IntegrationManager.github.token
+ row = self.rows["github_token"]
+
+ created = async_to_sync(accessor.acreate)(
+ self.team.id, external_id="made", config={"a": 1}, sensitive_config={"s": 1}, created_by_id=self.user.id
+ )
+ upserted, was_created = async_to_sync(accessor.aupsert)(self.team.id, "made", config={"b": 2})
+ updated = async_to_sync(accessor.aupdate)(self.team.id, row.id, config_patch={"x": 1}, errors="boom")
+
+ assert (created.config, created.sensitive_config, created.created_by_id) == (
+ {"a": 1, "auth_type": "token"},
+ {"s": 1},
+ self.user.id,
+ )
+ assert (upserted.id, was_created, upserted.config) == (created.id, False, {"b": 2, "auth_type": "token"})
+ assert (updated.config["x"], updated.errors) == (1, "boom")
+ assert Integration._base_manager.get(id=row.id).config["x"] == 1
+
+ def test_async_refresh_delegates_to_the_sync_refresh(self) -> None:
+ # The async form runs in its own thread with its own connection, which cannot see the rows
+ # of this test's transaction, so the sync refresh is replaced here.
+ accessor = IntegrationManager.resend.oauth
+ row = self._resend_row(refreshed_ago=800)
+
+ with patch.object(OAuthAccessor, "refresh_access_token", autospec=True, return_value=row) as refresh:
+ refreshed = async_to_sync(accessor.arefresh_access_token)(self.team.id, row.id, stale_access_token="old")
+
+ assert refreshed is row
+ refresh.assert_called_once_with(accessor, self.team.id, row.id, stale_access_token="old")
+
+
+class TestRegistration(BaseTest):
+ @parameterized.expand(
+ [
+ ("unknown_source_kind", lambda: IntegrationManager.source("acme-crm-unregistered")),
+ ("unknown_kind", lambda: IntegrationManager.kinds("github", "acme-crm")),
+ ("no_kinds", lambda: IntegrationManager.kinds()),
+ ]
+ )
+ def test_unknown_kinds_raise(self, _name: str, build: Callable[[], Any]) -> None:
+ with self.assertRaises(UnknownIntegrationKind):
+ build()
+
+ def test_register_source_kinds_sets_the_default_of_a_new_kind_only(self) -> None:
+ with patch.dict(auth_types._KIND_AUTH_TYPES):
+ register_source_kinds({"acme-sdk-test": AuthType.TOKEN, "stripe": AuthType.CREDENTIALS})
+
+ assert default_auth_type("acme-sdk-test") == AuthType.TOKEN
+ assert default_auth_type("stripe") == AuthType.OAUTH
+ assert AuthType.CREDENTIALS in kind_auth_types("stripe").supported
+ assert IntegrationManager.source("acme-sdk-test").token.list(self.team.id) == []
+
+ def test_source_rows_of_a_catalogue_kind_use_the_warehouse_auth_type(self) -> None:
+ row = IntegrationManager.source("clickup").credentials.create(
+ self.team.id, external_id="new", config={MARKER: "ClickUp"}
+ )
+
+ assert default_auth_type("clickup") == AuthType.OAUTH
+ assert row.config == {MARKER: "ClickUp", "auth_type": "credentials"}
+
+ def test_register_source_kinds_rejects_a_changed_default(self) -> None:
+ register_source_kinds({"acme-sdk-test-2": AuthType.TOKEN})
+
+ with self.assertRaises(ValueError):
+ register_source_kinds({"acme-sdk-test-2": AuthType.CREDENTIALS})
+
+ def test_warehouse_sources_register_their_kinds_at_startup(self) -> None:
+ assert IntegrationManager.source("ably").token is not None
+ assert default_auth_type("ably") == AuthType.TOKEN
diff --git a/posthog/integrations/test/test_catalogue.py b/posthog/integrations/test/test_catalogue.py
new file mode 100644
index 000000000000..50211284d466
--- /dev/null
+++ b/posthog/integrations/test/test_catalogue.py
@@ -0,0 +1,222 @@
+import shutil
+import textwrap
+from pathlib import Path
+from typing import Any
+
+import pytest
+
+from django.test import SimpleTestCase
+
+import mypy.api
+from hogli_commands.projections import PROJECTIONS, OutputFormatter, ProjectionRunner
+from parameterized import parameterized
+
+from posthog.integrations import IntegrationManager
+from posthog.integrations.accessors import AuthAccessor, IntegrationReader, OAuthAccessor
+from posthog.integrations.catalogue import KINDS
+from posthog.integrations.contracts import UnsupportedIntegrationFilter
+from posthog.integrations.enums import AuthType
+from posthog.integrations.kinds_generated import IntegrationKind
+from posthog.models.integration import Integration
+
+REPO_ROOT = Path(__file__).resolve().parents[3]
+SDK_DIR = REPO_ROOT / "posthog/integrations"
+
+
+# The default auth type of a kind is stored data: a row without `config["auth_type"]` has it.
+# A change here needs a data migration for the rows of the kind, and a deliberate edit of this mapping.
+FROZEN_KIND_DEFAULTS: dict[str, str] = {
+ "anthropic": "token",
+ "apns": "credentials",
+ "apple-ads": "oauth",
+ "aws-redshift": "credentials",
+ "aws-s3": "credentials",
+ "azure-blob": "credentials",
+ "bing-ads": "oauth",
+ "clickup": "oauth",
+ "customerio-app": "token",
+ "customerio-track": "token",
+ "customerio-webhook": "token",
+ "databricks": "credentials",
+ "email": "credentials",
+ "firebase": "credentials",
+ "github": "oauth",
+ "gitlab": "token",
+ "google-ads": "oauth",
+ "google-analytics": "oauth",
+ "google-calendar": "oauth",
+ "google-cloud-service-account": "credentials",
+ "google-cloud-storage": "credentials",
+ "google-pubsub": "credentials",
+ "google-search-console": "oauth",
+ "google-sheets": "oauth",
+ "helpscout": "oauth",
+ "hubspot": "oauth",
+ "instagram": "oauth",
+ "intercom": "oauth",
+ "jira": "oauth",
+ "linear": "oauth",
+ "linkedin-ads": "oauth",
+ "meta-ads": "oauth",
+ "pardot": "oauth",
+ "pinterest-ads": "oauth",
+ "postgresql": "credentials",
+ "posthog": "oauth",
+ "reddit-ads": "oauth",
+ "resend": "oauth",
+ "s3-compatible": "credentials",
+ "salesforce": "oauth",
+ "slack": "oauth",
+ "slack-posthog-code": "oauth",
+ "snapchat": "oauth",
+ "snowflake": "credentials",
+ "stripe": "oauth",
+ "tiktok-ads": "oauth",
+ "twilio": "token",
+ "twitter-ads": "oauth",
+ "vercel": "oauth",
+ "youtube-analytics": "oauth",
+}
+
+
+class TestCatalogue(SimpleTestCase):
+ def test_kinds_match_the_model_choices(self) -> None:
+ assert set(KINDS) == set(Integration.IntegrationKind.values) == {kind.value for kind in IntegrationKind}
+
+ def test_kind_defaults_do_not_change(self) -> None:
+ assert {kind: spec.default.value for kind, spec in KINDS.items()} == FROZEN_KIND_DEFAULTS
+
+ def test_generated_file_is_in_sync_with_the_catalogue(self) -> None:
+ [projection] = [projection for projection in PROJECTIONS if projection.name == "integration-kinds"]
+ runner = ProjectionRunner(REPO_ROOT, [projection], OutputFormatter(REPO_ROOT))
+
+ assert runner.stale(projection) == [], "Run `hogli build:projections --only integration-kinds`."
+
+ @parameterized.expand([(kind,) for kind in sorted(KINDS)])
+ def test_each_kind_has_exactly_its_supported_auth_types(self, kind: str) -> None:
+ accessor = getattr(IntegrationManager, kind.replace("-", "_"))
+ present = {auth_type for auth_type in AuthType if hasattr(accessor, auth_type.value)}
+
+ assert present == KINDS[kind].supported
+ expected = {AuthType.OAUTH: OAuthAccessor, AuthType.TOKEN: AuthAccessor, AuthType.CREDENTIALS: AuthAccessor}
+ assert all(type(getattr(accessor, auth_type.value)) is expected[auth_type] for auth_type in present)
+ assert type(accessor.any) is IntegrationReader
+
+ @parameterized.expand(
+ [
+ ("create",),
+ ("upsert",),
+ ("update",),
+ ("refresh_access_token",),
+ ("update_by_external_id",),
+ ("acreate",),
+ ("aupdate",),
+ ]
+ )
+ def test_any_readers_have_no_writes(self, method: str) -> None:
+ assert not hasattr(IntegrationManager.any, method)
+ assert not hasattr(IntegrationManager.stripe.any, method)
+ assert hasattr(IntegrationManager.stripe.oauth, method)
+
+ @parameterized.expand(
+ [
+ ("lookup_in_key", {"account__name": "a"}),
+ ("auth_type_key", {"auth_type": "token"}),
+ ("marker_key", {"warehouse_source_type": "Stripe"}),
+ ("empty_key", {"": "a"}),
+ ("key_with_a_dash", {"a-b": "a"}),
+ ("key_starts_with_a_digit", {"1a": "a"}),
+ ("dict_value", {"account": {"name": "a"}}),
+ ("list_value", {"ids": [1, 2]}),
+ ("tuple_value", {"ids": (1,)}),
+ ]
+ )
+ def test_find_rejects_unsupported_filters(self, _name: str, where: dict[str, Any]) -> None:
+ with self.assertRaises(UnsupportedIntegrationFilter):
+ IntegrationManager.github.oauth.find(1, where=where)
+
+
+# The SDK modules are copied into an empty directory, so mypy reads them without the Django plugin
+# and without following imports into the rest of the repo. Imports it cannot find become `Any`.
+SDK_FILES = (
+ "__init__.py",
+ "accessors.py",
+ "catalogue.py",
+ "contracts.py",
+ "enums.py",
+ "kinds_generated.py",
+ "manager.py",
+ "source_kinds.py",
+ "usages.py",
+)
+
+SNIPPET = """
+from posthog.integrations import IntegrationManager
+
+IntegrationManager.github.oauth.create(1, external_id="x", config={})
+IntegrationManager.github.token.upsert(1, "x", config={})
+IntegrationManager.source("ably").token.create(1, external_id="x", config={})
+IntegrationManager.stripe.any.get(1, 2)
+IntegrationManager.any.list(1)
+IntegrationManager.stripe.any.create(1, external_id="x", config={})
+IntegrationManager.github.credentials
+IntegrationManager.any.update(1, 2, errors="")
+IntegrationManager.postgresql.oauth
+IntegrationManager.stripe.any.upsert(1, "x", config={})
+row = IntegrationManager.github.oauth.get(1, 2)
+row.save()
+row.delete()
+row.config["x"] = 1
+row.objects
+IntegrationManager.github.token.refresh_access_token(1, 2)
+row.config.get("x")
+IntegrationManager.resend.oauth.refresh_access_token(1, 2, stale_access_token="t").sensitive_config
+"""
+
+
+def test_mypy_rejects_unsupported_auth_types_writes_through_any_and_row_writes(
+ tmp_path: Path, monkeypatch: pytest.MonkeyPatch
+) -> None:
+ # mypy also searches the working directory, which would pull in the whole repo.
+ monkeypatch.chdir(tmp_path)
+ root = tmp_path / "src"
+ for relative in SDK_FILES:
+ target = root / "posthog/integrations" / relative
+ target.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy(SDK_DIR / relative, target)
+ (root / "posthog/__init__.py").touch()
+ (root / "snippet.py").write_text(textwrap.dedent(SNIPPET))
+ (tmp_path / "mypy.ini").write_text("[mypy]\n")
+
+ stdout, stderr, _ = mypy.api.run(
+ [
+ "--config-file",
+ str(tmp_path / "mypy.ini"),
+ "--cache-dir",
+ str(tmp_path / "cache"),
+ "--no-site-packages",
+ "--ignore-missing-imports",
+ "--no-error-summary",
+ "--hide-error-context",
+ str(root / "snippet.py"),
+ ]
+ )
+
+ errors = sorted(
+ (int(line.split(":")[1]), line.split(": error: ")[1].split(" [")[0])
+ for line in stdout.splitlines()
+ if "snippet.py" in line and ": error: " in line
+ )
+ assert errors == [
+ (9, '"IntegrationReader" has no attribute "create"'),
+ (10, '"GithubKind" has no attribute "credentials"'),
+ (11, '"IntegrationReader" has no attribute "update"'),
+ (12, '"PostgresqlKind" has no attribute "oauth"'),
+ (13, '"IntegrationReader" has no attribute "upsert"'),
+ (15, '"IntegrationRecord" has no attribute "save"'),
+ (16, '"IntegrationRecord" has no attribute "delete"'),
+ (17, 'Unsupported target for indexed assignment ("Mapping[str, Any]")'),
+ (18, '"IntegrationRecord" has no attribute "objects"'),
+ (19, '"AuthAccessor" has no attribute "refresh_access_token"'),
+ ], stdout + stderr
diff --git a/posthog/integrations/test/test_fields.py b/posthog/integrations/test/test_fields.py
new file mode 100644
index 000000000000..8abaaa7fbe6c
--- /dev/null
+++ b/posthog/integrations/test/test_fields.py
@@ -0,0 +1,183 @@
+from typing import Any
+
+from posthog.test.base import BaseTest
+
+from parameterized import parameterized
+from rest_framework import serializers
+
+from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY
+from posthog.integrations.enums import AuthType
+from posthog.integrations.fields import IntegrationReferenceField
+from posthog.models import Team
+from posthog.models.comment.slack_thread import CommentSlackThread
+from posthog.models.integration import ANY, Integration
+
+
+class _Serializer(serializers.Serializer):
+ integration = IntegrationReferenceField()
+ github = IntegrationReferenceField(kinds=["github"], required=False, allow_null=True)
+ github_any = IntegrationReferenceField(kinds=["github"], auth_types="any", required=False)
+ github_token = IntegrationReferenceField(kinds=["github"], auth_types=[AuthType.TOKEN], required=False)
+
+
+class TestIntegrationReferenceField(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.slack = Integration.objects._with_auth(ANY).create(team=self.team, kind="slack", integration_id="s")
+ self.github_oauth = Integration.objects._with_auth(ANY).create(
+ team=self.team, kind="github", integration_id="o"
+ )
+ self.github = Integration.objects._with_auth(ANY).create(
+ team=self.team, kind="github", integration_id="g", config={"auth_type": "token"}
+ )
+ self.github_source_row = Integration.objects._with_auth(ANY).create(
+ team=self.team,
+ kind="github",
+ integration_id="gs",
+ config={"auth_type": "token", SOURCE_MARKER_CONFIG_KEY: "Github"},
+ )
+ other_team = Team.objects.create(organization=self.organization, name="Other")
+ self.other = Integration.objects._with_auth(ANY).create(team=other_team, kind="slack", integration_id="s")
+
+ def _validate(self, data: dict[str, Any], context: dict[str, Any] | None = None) -> _Serializer:
+ serializer = _Serializer(data=data, context={"team_id": self.team.id} if context is None else context)
+ serializer.is_valid()
+ return serializer
+
+ @parameterized.expand(
+ [
+ ("int", lambda self: self.slack.id),
+ ("string", lambda self: str(self.slack.id)),
+ ]
+ )
+ def test_accepts_an_id_of_the_team(self, _name: str, value: Any) -> None:
+ serializer = self._validate({"integration": value(self), "github": self.github_oauth.id})
+
+ assert serializer.errors == {}
+ assert serializer.validated_data == {"integration": self.slack, "github": self.github_oauth}
+
+ @parameterized.expand(
+ [
+ ("no_kinds_accepts_every_auth_type", "integration", "github", None),
+ ("default_accepts_the_kind_default", "github", "github_oauth", None),
+ ("default_refuses_another_auth_type", "github", "github", "does_not_exist"),
+ ("any_accepts_the_kind_default", "github_any", "github_oauth", None),
+ ("any_accepts_another_auth_type", "github_any", "github", None),
+ ("any_refuses_a_source_row", "github_any", "github_source_row", "does_not_exist"),
+ ("listed_accepts_its_auth_type", "github_token", "github", None),
+ ("listed_refuses_the_kind_default", "github_token", "github_oauth", "does_not_exist"),
+ ]
+ )
+ def test_auth_types(self, _name: str, field: str, row: str, error_code: str | None) -> None:
+ serializer = self._validate({"integration": self.slack.id, field: getattr(self, row).id})
+
+ if error_code is None:
+ assert serializer.validated_data[field] == getattr(self, row)
+ else:
+ assert [error.code for error in serializer.errors[field]] == [error_code]
+
+ @parameterized.expand(
+ [
+ ("allow_null", "github", None),
+ ("not_allow_null", "github_any", "null"),
+ ("not_allow_null_without_kinds", "integration", "null"),
+ ]
+ )
+ def test_an_empty_string_means_null(self, _name: str, field: str, error_code: str | None) -> None:
+ serializer = self._validate({"integration": self.slack.id, field: ""})
+
+ if error_code is None:
+ assert serializer.errors == {}
+ assert serializer.validated_data[field] is None
+ else:
+ assert [error.code for error in serializer.errors[field]] == [error_code]
+
+ def test_auth_types_without_kinds_raises(self) -> None:
+ with self.assertRaises(ValueError):
+ IntegrationReferenceField(auth_types="any")
+
+ def test_reads_the_team_from_get_team(self) -> None:
+ serializer = self._validate({"integration": self.slack.id}, context={"get_team": lambda: self.team})
+
+ assert serializer.validated_data == {"integration": self.slack}
+
+ @parameterized.expand(
+ [
+ ("other_team", lambda self: {"integration": self.other.id}, "integration", "does_not_exist"),
+ ("missing", lambda self: {"integration": 10**9}, "integration", "does_not_exist"),
+ (
+ "other_kind",
+ lambda self: {"integration": self.slack.id, "github": self.slack.id},
+ "github",
+ "does_not_exist",
+ ),
+ # The field reads only named rows, so a warehouse source row does not exist for it.
+ (
+ "source_row_without_kinds",
+ lambda self: {"integration": self.github_source_row.id},
+ "integration",
+ "does_not_exist",
+ ),
+ (
+ "source_row_of_the_kind",
+ lambda self: {"integration": self.slack.id, "github": self.github_source_row.id},
+ "github",
+ "does_not_exist",
+ ),
+ ("not_a_number", lambda self: {"integration": "abc"}, "integration", "incorrect_type"),
+ ("bool", lambda self: {"integration": True}, "integration", "incorrect_type"),
+ ("list", lambda self: {"integration": [1]}, "integration", "incorrect_type"),
+ ]
+ )
+ def test_rejects(self, _name: str, data: Any, field: str, code: str) -> None:
+ serializer = self._validate(data(self))
+
+ assert [error.code for error in serializer.errors[field]] == [code]
+
+ def test_needs_a_team_in_the_context(self) -> None:
+ serializer = self._validate({"integration": self.slack.id}, context={})
+
+ assert [error.code for error in serializer.errors["integration"]] == ["no_team"]
+
+ def test_represents_the_id(self) -> None:
+ field = IntegrationReferenceField()
+
+ assert (field.to_representation(self.slack), field.to_representation(self.slack.id)) == (
+ self.slack.id,
+ self.slack.id,
+ )
+
+
+class _ReadSerializer(serializers.Serializer):
+ integration = IntegrationReferenceField(allow_null=True)
+
+
+class _Holder:
+ def __init__(self, integration: Integration | None) -> None:
+ self.integration = integration
+
+
+class TestIntegrationReferenceFieldRead(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.slack = Integration.objects._with_auth(ANY).create(team=self.team, kind="slack", integration_id="s")
+
+ def test_reads_the_foreign_key_column_without_a_query(self) -> None:
+ thread = CommentSlackThread(team=self.team, integration_id=self.slack.id)
+
+ with self.assertNumQueries(0):
+ data = _ReadSerializer(thread).data
+
+ assert data == {"integration": self.slack.id}
+
+ @parameterized.expand(
+ [
+ ("model_field_null", lambda self: CommentSlackThread(team=self.team, integration_id=None), None),
+ ("plain_attribute", lambda self: _Holder(self.slack), "slack"),
+ ("plain_attribute_null", lambda self: _Holder(None), None),
+ ]
+ )
+ def test_represents(self, _name: str, build: Any, expected: str | None) -> None:
+ data = _ReadSerializer(build(self)).data
+
+ assert data == {"integration": None if expected is None else getattr(self, expected).id}
diff --git a/posthog/integrations/usages.py b/posthog/integrations/usages.py
new file mode 100644
index 000000000000..3b30ea843b9a
--- /dev/null
+++ b/posthog/integrations/usages.py
@@ -0,0 +1,42 @@
+"""The registry of the products that report which PostHog objects use an integration row.
+
+A product reports its usages from `products/<product>/backend/integration_usages.py`, with a function
+`integration_usages(team_id: int, integration_id: int) -> Iterable[IntegrationUsage]`. The SDK finds
+these modules on the first call, so no core module has to import a product. Code outside a product
+calls `register_usage_provider` instead.
+"""
+
+import threading
+from collections.abc import Callable, Iterable
+
+from posthog.integrations.contracts import IntegrationUsage
+from posthog.products import load_product_modules
+
+UsageProvider = Callable[[int, int], Iterable[IntegrationUsage]]
+
+PRODUCT_MODULE = "integration_usages"
+PRODUCT_FUNCTION = "integration_usages"
+
+# Reentrant: a provider module can call `register_usage_provider` while `_providers` imports it.
+_lock = threading.RLock()
+_registered: list[UsageProvider] = []
+_product_providers: list[UsageProvider] | None = None
+
+
+def register_usage_provider(provider: UsageProvider) -> None:
+ with _lock:
+ if provider not in _registered:
+ _registered.append(provider)
+
+
+def _providers() -> list[UsageProvider]:
+ global _product_providers
+ with _lock:
+ if _product_providers is None:
+ _product_providers = [getattr(module, PRODUCT_FUNCTION) for module in load_product_modules(PRODUCT_MODULE)]
+ return [*_product_providers, *_registered]
+
+
+def integration_usages(team_id: int, integration_id: int) -> list[IntegrationUsage]:
+ usages = [usage for provider in _providers() for usage in provider(team_id, integration_id)]
+ return sorted(usages, key=lambda usage: (usage.type, usage.name, usage.id))
diff --git a/posthog/models/integration/__init__.py b/posthog/models/integration/__init__.py
index eb0922152a84..efa63c38c926 100644
--- a/posthog/models/integration/__init__.py
+++ b/posthog/models/integration/__init__.py
@@ -16,6 +16,15 @@ from .anthropic import (
AnthropicIntegrationError,
)
from .assignees import Assignee, AssigneeLookupFailed, ReconnectRequired
+from .auth_types import (
+ ANY,
+ DEFAULT,
+ AuthRequirement,
+ AuthTypeSelector,
+ IntegrationAuthType,
+ KindAuthTypes,
+ register_kind_auth_types,
+)
from .aws import (
AWSCredentialsIntegration,
AWSRedshiftIntegration,
@@ -73,7 +82,7 @@ from .linkedin_ads import LinkedInAdsIntegration
from .meta import InstagramIntegration, MetaAdsIntegration, MetaGraphIntegration
from .model import (
Integration,
- IntegrationManager,
+ IntegrationModelManager,
IntegrationQuerySet,
UndecryptedIntegrationSecretError,
aget_integration_by_id,
@@ -150,10 +159,17 @@ from .stripe import STRIPE_POSTHOG_SECRET_NAMES, StripeIntegration
from .twilio import TwilioIntegration
__all__ = [
+ "ANY",
+ "DEFAULT",
+ "AuthRequirement",
+ "AuthTypeSelector",
+ "IntegrationAuthType",
+ "KindAuthTypes",
+ "register_kind_auth_types",
"UndecryptedIntegrationSecretError",
"integration_secret_decrypt_counter",
"IntegrationQuerySet",
- "IntegrationManager",
+ "IntegrationModelManager",
"Integration",
"defer_repository_cache_fields",
"aget_integration_by_id",
diff --git a/posthog/models/integration/anthropic.py b/posthog/models/integration/anthropic.py
index c9da24e6b05a..9229bef3c920 100644
--- a/posthog/models/integration/anthropic.py
+++ b/posthog/models/integration/anthropic.py
@@ -10,6 +10,7 @@ from django.db import transaction
import structlog
from rest_framework.exceptions import ValidationError
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.models.utils import IntegrityError
@@ -48,13 +49,13 @@ class AnthropicIntegration:
integration: model.Integration
_client: "Anthropic"
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != model.Integration.IntegrationKind.ANTHROPIC.value:
# Internal misuse, not a user-input issue.
raise AnthropicIntegrationError(
f"AnthropicIntegration init called with Integration kind={integration.kind!r}"
)
- self.integration = integration
+ self.integration = model.integration_model(integration)
api_key = integration.sensitive_config.get("api_key")
if not api_key:
raise AnthropicIntegrationError(f"Anthropic integration {integration.id} is missing 'api_key'")
diff --git a/posthog/models/integration/auth_types.py b/posthog/models/integration/auth_types.py
new file mode 100644
index 000000000000..3ad606cede29
--- /dev/null
+++ b/posthog/models/integration/auth_types.py
@@ -0,0 +1,229 @@
+"""Auth types of `Integration` rows, and the catalogue of the auth types that each kind supports.
+
+A kind names the provider (`github`, `stripe`, `postgresql`). An auth type names how the row
+authenticates with that provider: `oauth` (a connection the provider manages), `token` (one static
+secret) or `credentials` (several typed-in fields). A kind can have several auth types. Each kind
+has one default auth type: the meaning the kind had before auth types existed. A row stores its auth
+type in `config["auth_type"]` only when it is not the default. A row without that key has the
+default auth type of its kind, so rows written before auth types existed need no migration.
+
+Code outside the integrations SDK (`posthog/integrations/`) does not query the model. The SDK
+states the auth types of each query with the internal `IntegrationQuerySet._with_auth`.
+
+This module must not import `model.py`: `model.py` imports it to build the queryset filters. It
+imports only the light SDK modules (catalogue, contracts, enums), which do not import the model.
+"""
+
+from collections.abc import Iterable, Mapping
+from enum import Enum
+from typing import Any, Self
+
+from django.db.models import Q
+
+from posthog.dataclasses import frozen
+from posthog.integrations.catalogue import KINDS
+from posthog.integrations.contracts import SOURCE_MARKER_CONFIG_KEY, KindSpec
+from posthog.integrations.enums import IntegrationAuthType
+
+AUTH_TYPE_CONFIG_KEY = "auth_type"
+
+
+# The auth types and the kind catalogue belong to the integrations SDK. The model reads them here for
+# the storage rule and the implicit default filter of the model manager.
+KindAuthTypes = KindSpec
+
+
+class AuthTypeSelector(Enum):
+ # Every auth type. For code that handles each row the same way, such as the integrations list.
+ ANY = "any"
+ # The default auth type of each row's kind. For code that handles several kinds by their usual meaning.
+ DEFAULT = "default"
+
+
+ANY = AuthTypeSelector.ANY
+DEFAULT = AuthTypeSelector.DEFAULT
+
+# Kinds that are not in the catalogue get this entry until they register their own with
+# `register_kind_auth_types`.
+UNKNOWN_KIND_AUTH_TYPES = KindSpec(label="Unknown", default=IntegrationAuthType.CREDENTIALS)
+
+_KIND_AUTH_TYPES: dict[str, KindSpec] = dict(KINDS)
+
+
+def register_kind_auth_types(kind: str, auth_types: KindSpec) -> None:
+ """Add a kind to the catalogue, before the first query of the kind runs. `register_source_kinds` calls it."""
+ existing = _KIND_AUTH_TYPES.get(kind)
+ # Adding supported auth types is safe. Changing the default is not: rows store only a non-default.
+ widens = (
+ existing is not None and existing.default == auth_types.default and existing.supported <= auth_types.supported
+ )
+ if existing is not None and existing != auth_types and not widens:
+ raise ValueError(f"Integration kind {kind!r} already has auth types {existing!r}")
+ _KIND_AUTH_TYPES[kind] = auth_types
+
+
+def kind_auth_types(kind: str) -> KindSpec:
+ return _KIND_AUTH_TYPES.get(kind, UNKNOWN_KIND_AUTH_TYPES)
+
+
+def catalogued_kinds() -> Mapping[str, KindSpec]:
+ return dict(_KIND_AUTH_TYPES)
+
+
+def default_auth_type(kind: str) -> IntegrationAuthType:
+ return kind_auth_types(kind).default
+
+
+def stored_auth_type(config: Any) -> str | None:
+ if not isinstance(config, dict):
+ return None
+ return config.get(AUTH_TYPE_CONFIG_KEY)
+
+
+def effective_auth_type(kind: str, config: Any) -> str:
+ stored = stored_auth_type(config)
+ return stored if stored is not None else default_auth_type(kind).value
+
+
+def _unset_q() -> Q:
+ # `isnull=True` on a JSON key matches a missing key only. A JSON null counts as unset too,
+ # the same as `config.get("auth_type")` in Python.
+ return Q(config__auth_type__isnull=True) | Q(config__auth_type=None)
+
+
+def _kinds_with_default_q(auth_type: IntegrationAuthType) -> Q | None:
+ kinds = [kind for kind, entry in _KIND_AUTH_TYPES.items() if entry.default == auth_type]
+ clauses: list[Q] = []
+ if kinds:
+ clauses.append(Q(kind__in=kinds))
+ if UNKNOWN_KIND_AUTH_TYPES.default == auth_type:
+ clauses.append(~Q(kind__in=list(_KIND_AUTH_TYPES)))
+ return _any_of(clauses)
+
+
+def _any_of(clauses: Iterable[Q]) -> Q | None:
+ result: Q | None = None
+ for clause in clauses:
+ result = clause if result is None else result | clause
+ return result
+
+
+def rows_with_auth_type_q(auth_type: IntegrationAuthType) -> Q:
+ explicit = Q(config__auth_type=auth_type.value)
+ kinds = _kinds_with_default_q(auth_type)
+ return explicit if kinds is None else explicit | (_unset_q() & kinds)
+
+
+def rows_with_default_auth_type_q() -> Q:
+ # Correct because `normalized_config` never lets a row store the default auth type of its kind.
+ return _unset_q()
+
+
+def rows_without_source_marker_q() -> Q:
+ return ~Q(config__has_key=SOURCE_MARKER_CONFIG_KEY)
+
+
+def normalized_config(kind: Any, config: Any) -> Any:
+ """Drop a stored auth type that is null or the default of the kind.
+
+ So a row has `config["auth_type"]` only when its auth type is not the default, and the filter for
+ default rows stays one short clause. `Integration.save` and `bulk_create` call it.
+ """
+ if not isinstance(config, dict) or AUTH_TYPE_CONFIG_KEY not in config or not isinstance(kind, str):
+ return config
+ stored = config[AUTH_TYPE_CONFIG_KEY]
+ if stored is not None and stored != default_auth_type(kind):
+ return config
+ return {key: value for key, value in config.items() if key != AUTH_TYPE_CONFIG_KEY}
+
+
+@frozen
+class AuthRequirement:
+ """The auth types that a caller can handle.
+
+ It is the input of `_with_auth`. Later it can also carry the scopes that a caller needs, so that
+ one check compares them with the scopes that a row grants.
+ """
+
+ any: bool = False
+ kind_default: bool = False
+ auth_types: frozenset[IntegrationAuthType] = frozenset()
+
+ @classmethod
+ def of(cls, *selectors: "IntegrationAuthType | AuthTypeSelector | str") -> Self:
+ if not selectors:
+ raise ValueError("_with_auth() needs at least one auth type, DEFAULT or ANY")
+ if ANY in selectors:
+ if len(selectors) > 1:
+ raise ValueError("ANY already includes every auth type, so pass it alone")
+ return cls(any=True)
+ auth_types: set[IntegrationAuthType] = set()
+ for selector in selectors:
+ if selector is DEFAULT:
+ continue
+ if isinstance(selector, AuthTypeSelector) or selector not in IntegrationAuthType.values:
+ raise ValueError(f"Unknown integration auth type: {selector!r}")
+ auth_types.add(IntegrationAuthType(selector))
+ return cls(kind_default=DEFAULT in selectors, auth_types=frozenset(auth_types))
+
+ @classmethod
+ def kind_default_only(cls) -> Self:
+ return cls(kind_default=True)
+
+ def q(self) -> Q:
+ if self.any:
+ return Q()
+ clauses = [rows_with_auth_type_q(auth_type) for auth_type in sorted(self.auth_types)]
+ if self.kind_default:
+ clauses.append(rows_with_default_auth_type_q())
+ result = _any_of(clauses)
+ assert result is not None
+ return result
+
+ def accepts(self, kind: str | None, auth_type: str) -> bool:
+ if self.any or auth_type in self.auth_types:
+ return True
+ return self.kind_default and kind is not None and auth_type == default_auth_type(kind)
+
+ @property
+ def single_auth_type(self) -> IntegrationAuthType | None:
+ if self.any or self.kind_default or len(self.auth_types) != 1:
+ return None
+ return next(iter(self.auth_types))
+
+
+def config_for_write(kind: Any, config: Any, requirement: AuthRequirement) -> Any:
+ """Return the config to write for a new or updated row under `requirement`.
+
+ A requirement with one auth type writes it into the config when it is not the kind's default.
+ A requirement that does not cover the row's auth type raises, so a write never makes a row that
+ the same query cannot read back. `kind` is None when the caller does not pass it.
+ """
+ if config is None:
+ config = {}
+ if not isinstance(config, dict) or not (kind is None or isinstance(kind, str)):
+ return config
+
+ stored = stored_auth_type(config)
+ if stored is not None:
+ if kind is not None and stored not in kind_auth_types(kind).supported:
+ raise ValueError(f"Integration kind {kind!r} does not support auth type {stored!r}")
+ if not requirement.accepts(kind, stored):
+ raise ValueError(f"Auth type {stored!r} is not in the _with_auth() of this query")
+ return config
+
+ target = requirement.single_auth_type
+ if target is not None:
+ if kind is None:
+ if any(entry.default == target for entry in _KIND_AUTH_TYPES.values()):
+ raise ValueError(f"Pass the kind, so the write can tell if {target.value!r} is its default")
+ return {**config, AUTH_TYPE_CONFIG_KEY: target.value}
+ if target not in kind_auth_types(kind).supported:
+ raise ValueError(f"Integration kind {kind!r} does not support auth type {target.value!r}")
+ if target == default_auth_type(kind):
+ return config
+ return {**config, AUTH_TYPE_CONFIG_KEY: target.value}
+
+ if kind is not None and not requirement.accepts(kind, default_auth_type(kind).value):
+ raise ValueError("A write through _with_auth() needs exactly one auth type, or one that is the kind's default")
+ return config
diff --git a/posthog/models/integration/aws.py b/posthog/models/integration/aws.py
index 92092e958016..baf0a3f89d14 100644
--- a/posthog/models/integration/aws.py
+++ b/posthog/models/integration/aws.py
@@ -14,6 +14,7 @@ from rest_framework.exceptions import ValidationError
from posthog.credentials import AWSKeyPair
from posthog.credentials_service.callers import CredentialsCaller
from posthog.credentials_service.settings_fallback import get_or_settings
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.models.utils import IntegrityError
from posthog.security.url_validation import is_url_allowed
@@ -258,13 +259,13 @@ class AWSRoleBasedIntegration:
integration_kind: ClassVar[_AWSKindType]
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != self.integration_kind:
raise common.IntegrationError(
"Integration provided is not the expected AWS integration "
f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')"
)
- self.integration = integration
+ self.integration = model.integration_model(integration)
try:
self.aws_role_arn = integration.config["aws_role_arn"]
except KeyError:
@@ -413,14 +414,14 @@ class AWSCredentialsIntegration:
integration_kind: ClassVar[_AWSKindType]
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != self.integration_kind:
raise common.IntegrationError(
"Integration provided is not the expected AWS integration "
f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')"
)
- self.integration = integration
- self.credentials = _read_aws_credentials(integration)
+ self.integration = model.integration_model(integration)
+ self.credentials = _read_aws_credentials(self.integration)
@property
def aws_account_id(self) -> str | None:
@@ -540,13 +541,13 @@ class S3CompatibleIntegration:
credentials: AWSKeyPair
endpoint_url: str
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != model.Integration.IntegrationKind.S3_COMPATIBLE:
raise common.IntegrationError(
f"Integration provided is not an S3-compatible integration (got kind='{integration.kind}')"
)
- self.integration = integration
- self.credentials = _read_aws_credentials(integration)
+ self.integration = model.integration_model(integration)
+ self.credentials = _read_aws_credentials(self.integration)
try:
self.endpoint_url = integration.config["endpoint_url"]
except KeyError:
diff --git a/posthog/models/integration/azure_blob.py b/posthog/models/integration/azure_blob.py
index 4b1fc1dc6a01..0aebb9bb65a1 100644
--- a/posthog/models/integration/azure_blob.py
+++ b/posthog/models/integration/azure_blob.py
@@ -6,6 +6,7 @@ from itertools import groupby
from operator import itemgetter
from urllib.parse import urlsplit
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.security.url_validation import (
ResolvedIPs,
@@ -33,12 +34,12 @@ class AzureBlobIntegration:
integration: model.Integration
connection_string: str
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != model.Integration.IntegrationKind.AZURE_BLOB.value:
raise AzureBlobIntegrationError(
f"Integration provided is not an Azure Blob integration (got kind='{integration.kind}')"
)
- self.integration = integration
+ self.integration = model.integration_model(integration)
try:
self.connection_string = self.integration.sensitive_config["connection_string"]
diff --git a/posthog/models/integration/clickup.py b/posthog/models/integration/clickup.py
index d7133ec3cb2d..121827c942e7 100644
--- a/posthog/models/integration/clickup.py
+++ b/posthog/models/integration/clickup.py
@@ -5,6 +5,7 @@ import structlog
from rest_framework.exceptions import ValidationError
from posthog.exceptions_capture import capture_exception
+from posthog.integrations.contracts import IntegrationRecord
from . import common, model
@@ -14,11 +15,11 @@ logger = structlog.get_logger(__name__)
class ClickUpIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "clickup":
raise Exception("ClickUpIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
def _check_auth_error(self, response: requests.Response, context: str) -> None:
if response.status_code == 401:
diff --git a/posthog/models/integration/databricks.py b/posthog/models/integration/databricks.py
index b49a0363279a..71fe5b740882 100644
--- a/posthog/models/integration/databricks.py
+++ b/posthog/models/integration/databricks.py
@@ -2,6 +2,7 @@
import re
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.security.url_validation import is_url_allowed
@@ -61,10 +62,10 @@ class DatabricksIntegration:
client_id: str
client_secret: str
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != model.Integration.IntegrationKind.DATABRICKS.value:
raise DatabricksIntegrationError("Integration provided is not a Databricks integration")
- self.integration = integration
+ self.integration = model.integration_model(integration)
try:
self.server_hostname = self.integration.config["server_hostname"].lower()
diff --git a/posthog/models/integration/email.py b/posthog/models/integration/email.py
index 132309a4899d..9a25cf0b274f 100644
--- a/posthog/models/integration/email.py
+++ b/posthog/models/integration/email.py
@@ -18,6 +18,7 @@ from rest_framework.exceptions import ValidationError
from posthog.dataclasses import frozen
from posthog.exceptions_capture import capture_exception
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.team.team import Team
from posthog.models.user import User
from posthog.plugins.plugin_server_api import reload_integrations_on_workers
@@ -42,10 +43,10 @@ class EmailDomainStatusReport:
class EmailIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "email":
raise Exception("EmailIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@classmethod
def create_native_integration(
diff --git a/posthog/models/integration/github.py b/posthog/models/integration/github.py
index 11cc061ca850..b07afc04ec09 100644
--- a/posthog/models/integration/github.py
+++ b/posthog/models/integration/github.py
@@ -16,6 +16,7 @@ import structlog
from posthog.egress.github.transport import GitHubRateLimitError, github_request
from posthog.egress.limiter.policies import Priority
from posthog.egress.transport.transport import EgressBudgetExhausted
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.github_integration_base import (
GitHubIntegrationBase,
GitHubIntegrationError,
@@ -368,10 +369,10 @@ class GitHubIntegration(GitHubIntegrationBase):
raise exhausted
return None
- def __init__(self, integration: model.Integration, *, source: str, priority: Priority) -> None:
+ def __init__(self, integration: IntegrationRecord, *, source: str, priority: Priority) -> None:
if integration.kind != "github":
raise Exception("GitHubIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
self.source = source
self.priority = priority
diff --git a/posthog/models/integration/gitlab.py b/posthog/models/integration/gitlab.py
index e4fef7d0233a..c72f894c7392 100644
--- a/posthog/models/integration/gitlab.py
+++ b/posthog/models/integration/gitlab.py
@@ -6,6 +6,7 @@ from urllib.parse import urlencode, urlparse
import requests
+from posthog.integrations.contracts import IntegrationRecord
from posthog.security.url_validation import is_url_allowed
from . import common, model
@@ -77,10 +78,10 @@ class GitLabIntegration:
return integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "gitlab":
raise Exception("GitLabIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@property
def project_path(self) -> str:
diff --git a/posthog/models/integration/google_ads.py b/posthog/models/integration/google_ads.py
index 44fbe16eb90c..403da540156d 100644
--- a/posthog/models/integration/google_ads.py
+++ b/posthog/models/integration/google_ads.py
@@ -11,6 +11,7 @@ from posthog.credentials_service.callers import CredentialsCaller
from posthog.credentials_service.settings_fallback import get_or_settings
from posthog.egress.slack.client import SlackWebClient as WebClient
from posthog.exceptions_capture import capture_exception
+from posthog.integrations.contracts import IntegrationRecord
from . import common, model
@@ -51,11 +52,11 @@ def _google_ads_request(method: str, url: str, **kwargs) -> requests.Response:
class GoogleAdsIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "google-ads":
raise Exception("GoogleAdsIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@property
def client(self) -> WebClient:
diff --git a/posthog/models/integration/google_cloud.py b/posthog/models/integration/google_cloud.py
index 0cbb015b78b5..4b8a9d94dc79 100644
--- a/posthog/models/integration/google_cloud.py
+++ b/posthog/models/integration/google_cloud.py
@@ -8,6 +8,7 @@ from google.auth.transport.requests import Request as GoogleRequest
from google.oauth2 import service_account
from rest_framework.exceptions import ValidationError
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.plugins.plugin_server_api import reload_integrations_on_workers
@@ -71,8 +72,8 @@ def is_unique_service_account_by_organization_id(service_account_email: str, org
class GoogleCloudServiceAccountIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
- self.integration = integration
+ def __init__(self, integration: IntegrationRecord) -> None:
+ self.integration = model.integration_model(integration)
@classmethod
def integration_from_service_account(
@@ -158,8 +159,8 @@ class GoogleCloudIntegration:
supported_kinds = ["google-pubsub", "google-cloud-storage"]
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
- self.integration = integration
+ def __init__(self, integration: IntegrationRecord) -> None:
+ self.integration = model.integration_model(integration)
@classmethod
def integration_from_key(
diff --git a/posthog/models/integration/jira.py b/posthog/models/integration/jira.py
index 83b89d4a2fd6..08bff68e9889 100644
--- a/posthog/models/integration/jira.py
+++ b/posthog/models/integration/jira.py
@@ -10,6 +10,7 @@ import structlog
from rest_framework.exceptions import ValidationError
from posthog.exceptions_capture import capture_exception
+from posthog.integrations.contracts import IntegrationRecord
from . import common, model, oauth
from .assignees import MAX_ASSIGNEES, Assignee, AssigneeLookupFailed, ReconnectRequired
@@ -71,11 +72,11 @@ def description_to_adf(description: str) -> dict[str, Any]:
class JiraIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "jira":
raise Exception("JiraIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
def cloud_id(self) -> str | None:
"""Get the Atlassian cloud ID from the integration config"""
diff --git a/posthog/models/integration/linear.py b/posthog/models/integration/linear.py
index e2c38a60549d..73c3c5840a9b 100644
--- a/posthog/models/integration/linear.py
+++ b/posthog/models/integration/linear.py
@@ -6,6 +6,8 @@ import requests
import structlog
from rest_framework.exceptions import ValidationError
+from posthog.integrations.contracts import IntegrationRecord
+
from . import common, model
from .assignees import MAX_ASSIGNEES, Assignee, AssigneeLookupFailed
@@ -15,11 +17,11 @@ logger = structlog.get_logger(__name__)
class LinearIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "linear":
raise Exception("LinearIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
def url_key(self) -> str:
return common.dot_get(self.integration.config, "data.viewer.organization.urlKey")
diff --git a/posthog/models/integration/linkedin_ads.py b/posthog/models/integration/linkedin_ads.py
index 3615cb6490dc..213fac4529ad 100644
--- a/posthog/models/integration/linkedin_ads.py
+++ b/posthog/models/integration/linkedin_ads.py
@@ -5,6 +5,7 @@ import structlog
from rest_framework.exceptions import ValidationError
from posthog.egress.slack.client import SlackWebClient as WebClient
+from posthog.integrations.contracts import IntegrationRecord
from . import common, model
@@ -14,11 +15,11 @@ logger = structlog.get_logger(__name__)
class LinkedInAdsIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "linkedin-ads":
raise Exception("LinkedInAdsIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@property
def client(self) -> WebClient:
diff --git a/posthog/models/integration/meta.py b/posthog/models/integration/meta.py
index b49bacff4aff..d5242a7a0ac6 100644
--- a/posthog/models/integration/meta.py
+++ b/posthog/models/integration/meta.py
@@ -5,6 +5,8 @@ import time
import requests
import structlog
+from posthog.integrations.contracts import IntegrationRecord
+
from . import common, model, oauth, refresh_tracking
logger = structlog.get_logger(__name__)
@@ -22,10 +24,10 @@ class MetaGraphIntegration:
kind: str = ""
api_version: str = common.META_GRAPH_API_VERSION
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != self.kind:
raise Exception(f"{type(self).__name__} init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
def refresh_access_token(self):
oauth_config = oauth.OauthIntegration.oauth_config_for_kind(self.integration.kind)
diff --git a/posthog/models/integration/model.py b/posthog/models/integration/model.py
index 2e5e0b2cb7ec..46568df96364 100644
--- a/posthog/models/integration/model.py
+++ b/posthog/models/integration/model.py
@@ -1,20 +1,34 @@
"""The Integration Django model, its manager, and generic secret-decryption helpers."""
+from collections.abc import Iterable, Mapping
from typing import Any, Self, cast
from django.db import models
from django.db.models import Q
+from django.db.models.manager import ManagerDescriptor
from django.utils.functional import Promise
import structlog
from prometheus_client import Counter
from posthog.helpers.encrypted_fields import FERNET_TOKEN_PREFIX, EncryptedJSONField
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.rbac.decorators import field_access_control
from posthog.sync import database_sync_to_async
from . import common
+from .auth_types import (
+ AuthRequirement,
+ AuthTypeSelector,
+ IntegrationAuthType,
+ config_for_write,
+ default_auth_type,
+ effective_auth_type,
+ normalized_config,
+ rows_with_default_auth_type_q,
+ rows_without_source_marker_q,
+)
logger = structlog.get_logger(__name__)
@@ -87,24 +101,120 @@ def _decrypted_sensitive_value(integration: "Integration", field_name: str) -> s
class IntegrationQuerySet(models.QuerySet["Integration"]):
+ # Set by `_with_auth`. None means the query did not state its auth types.
+ _auth_requirement: AuthRequirement | None = None
+ # True on the queryset that `Integration.objects` returns without `_with_auth`. It holds only
+ # rows with the default auth type of their kind and without the warehouse source marker, so code
+ # that does not state its auth types never gets a row it cannot handle. A source row can have
+ # the default auth type of its kind and still hold a different config shape.
+ _implicit_default_auth: bool = False
+
+ def _clone(self) -> Self:
+ clone = super()._clone() # type: ignore[misc]
+ clone._auth_requirement = self._auth_requirement
+ clone._implicit_default_auth = self._implicit_default_auth
+ return clone
+
+ def _with_auth(self, *auth_types: IntegrationAuthType | AuthTypeSelector | str) -> Self:
+ """Keep only the rows whose auth type is in `auth_types`. Internal to the integrations SDK.
+
+ Pass `DEFAULT` for the default auth type of each row's kind, or `ANY` for every row. A row
+ without `config["auth_type"]` has the default auth type of its kind. A create through a
+ `_with_auth` with one non-default auth type writes it into `config["auth_type"]`.
+ """
+ if self._implicit_default_auth:
+ raise ValueError(
+ "Call _with_auth() directly on Integration.objects. The queryset without it holds only rows "
+ "with the default auth type of their kind."
+ )
+ if self._auth_requirement is not None and not self._auth_requirement.any:
+ # A second selector would filter by both but write by the last, so a create could make a
+ # row that this queryset cannot read back. Narrow only from ANY.
+ raise ValueError("This queryset already has _with_auth(). Narrow only from _with_auth(ANY).")
+ requirement = AuthRequirement.of(*auth_types)
+ queryset = self._chain() if requirement.any else self.filter(requirement.q()) # type: ignore[attr-defined]
+ queryset._auth_requirement = requirement
+ return queryset
+
+ def _with_implicit_default_auth(self) -> Self:
+ queryset = self.filter(rows_with_default_auth_type_q(), rows_without_source_marker_q())
+ queryset._implicit_default_auth = True
+ return queryset
+
+ def _write_requirement(self) -> AuthRequirement | None:
+ if self._auth_requirement is not None:
+ return self._auth_requirement
+ return AuthRequirement.kind_default_only() if self._implicit_default_auth else None
+
+ def _with_config_for_write(self, values: dict[str, Any], kind: Any) -> dict[str, Any]:
+ requirement = self._write_requirement()
+ if requirement is None:
+ return values
+ return {**values, "config": config_for_write(kind, values.get("config"), requirement)}
+
+ # The django-stubs plugin rebinds this subclass to its own TypeVar, so a concrete signature cannot match.
+ def create(self, **kwargs: Any) -> "Integration": # type: ignore[override]
+ return super().create(**self._with_config_for_write(kwargs, kwargs.get("kind")))
+
+ def bulk_create(self, objs: Iterable["Integration"], *args: Any, **kwargs: Any) -> list["Integration"]: # type: ignore[override]
+ objs = list(objs)
+ requirement = self._write_requirement()
+ for obj in objs:
+ if requirement is not None:
+ obj.config = config_for_write(obj.kind, obj.config, requirement)
+ # bulk_create does not call save(), so normalize here too.
+ obj.config = normalized_config(obj.kind, obj.config)
+ return cast(list[Integration], super().bulk_create(cast(Iterable[Any], objs), *args, **kwargs))
+
+ def update_or_create( # type: ignore[override]
+ self,
+ defaults: Mapping[str, Any] | None = None,
+ create_defaults: Mapping[str, Any] | None = None,
+ **kwargs: Any,
+ ) -> tuple["Integration", bool]:
+ # The update path sets `config` from `defaults` on the row it found. Without this, a new
+ # config without the key would turn a typed-in row back into a default row.
+ kind = kwargs.get("kind") or (defaults or {}).get("kind") or (create_defaults or {}).get("kind")
+ if defaults is not None and "config" in defaults:
+ defaults = self._with_config_for_write(dict(defaults), kind)
+ return super().update_or_create(defaults=defaults, create_defaults=create_defaults, **kwargs)
+
def for_github_installation_id(self, installation_id: str | int) -> Self:
return self.filter(
Q(config__installation_id=str(installation_id)) | Q(config__installation_id=int(installation_id))
)
-class IntegrationManager(models.Manager["Integration"]):
+class IntegrationModelManager(models.Manager["Integration"]):
_queryset_class = IntegrationQuerySet
+ # False on `_AllRowsManager`, the default manager that Django internals use.
+ _filters_to_default_auth: bool = True
def get_queryset(self) -> IntegrationQuerySet:
- return IntegrationQuerySet(self.model, using=self._db)
+ queryset = IntegrationQuerySet(self.model, using=self._db, hints=self._hints) # type: ignore[attr-defined]
+ if self._is_related_manager() or not self._filters_to_default_auth:
+ return queryset
+ return queryset._with_implicit_default_auth()
+
+ def _with_auth(self, *auth_types: IntegrationAuthType | AuthTypeSelector | str) -> IntegrationQuerySet:
+ if self._is_related_manager():
+ return self.get_queryset()._with_auth(*auth_types)
+ queryset = IntegrationQuerySet(self.model, using=self._db, hints=self._hints) # type: ignore[attr-defined]
+ return queryset._with_auth(*auth_types)
def filter(self, *args: Any, **kwargs: Any) -> IntegrationQuerySet:
return self.get_queryset().filter(*args, **kwargs)
+ def _is_related_manager(self) -> bool:
+ # Django builds related managers such as `team.integration_set` as subclasses of this class,
+ # and only they have `core_filters`. They stay unfiltered, the same as the base manager that
+ # cascades, `save()` and `refresh_from_db()` use.
+ return hasattr(self, "core_filters")
+
def first_github_for_team_installation(self, team_id: int, installation_id: str) -> "Integration | None":
return (
- self.filter(team_id=team_id, kind=Integration.IntegrationKind.GITHUB)
+ self._with_auth(IntegrationAuthType.OAUTH)
+ .filter(team_id=team_id, kind=Integration.IntegrationKind.GITHUB)
.for_github_installation_id(installation_id)
.first()
)
@@ -112,18 +222,25 @@ class IntegrationManager(models.Manager["Integration"]):
def first_github_for_user_installation(self, user: User, installation_id: str) -> "Integration | None":
user_team_ids = user.teams.values_list("id", flat=True)
return (
- self.filter(team_id__in=user_team_ids, kind=Integration.IntegrationKind.GITHUB)
+ self._with_auth(IntegrationAuthType.OAUTH)
+ .filter(team_id__in=user_team_ids, kind=Integration.IntegrationKind.GITHUB)
.for_github_installation_id(installation_id)
.first()
)
+class _AllRowsManager(IntegrationModelManager):
+ _filters_to_default_auth = False
+
+
def integration_kind_choices() -> list[tuple[str, str | Promise]]:
# Callable so growing the enum doesn't generate a no-op migration.
return list(Integration.IntegrationKind.choices)
class Integration(models.Model):
+ AuthType = IntegrationAuthType
+
class IntegrationKind(models.TextChoices):
ANTHROPIC = "anthropic"
APPLE_ADS = "apple-ads"
@@ -178,7 +295,11 @@ class Integration(models.Model):
VERCEL = "vercel"
YOUTUBE_ANALYTICS = "youtube-analytics"
+ # Declared for type checkers that do not read Django fields: ty refuses the model as an
+ # `IntegrationRecord` without them.
+ id: int
team = models.ForeignKey("Team", on_delete=models.CASCADE)
+ team_id: int
# The integration type identifier
kind = field_access_control(models.CharField(max_length=32, choices=integration_kind_choices), "project", "admin")
@@ -202,8 +323,14 @@ class Integration(models.Model):
# Meta
created_at = models.DateTimeField(auto_now_add=True, blank=True)
created_by = models.ForeignKey("User", on_delete=models.SET_NULL, null=True, blank=True)
+ created_by_id: int | None
- objects: IntegrationManager = IntegrationManager()
+ # Django's default manager, because it is declared first. Admin widgets, DRF relation fields,
+ # `validate_constraints` and `get_object_or_404` read through it, so it must see every row. After
+ # the class, `Integration.objects` points to `_implicit_default_auth_objects` instead. The Django
+ # name stays `objects`, so the migration state does not change.
+ objects: IntegrationModelManager = _AllRowsManager()
+ _implicit_default_auth_objects: IntegrationModelManager = IntegrationModelManager()
class Meta:
constraints = [
@@ -213,8 +340,20 @@ class Integration(models.Model):
]
indexes = [models.Index(fields=["kind", "integration_id"], name="posthog_integration_kind_ext")]
+ def save(self, *args: Any, **kwargs: Any) -> None:
+ self.config = normalized_config(self.kind, self.config)
+ super().save(*args, **kwargs)
+
+ @property
+ def auth_type(self) -> str:
+ """The stored `config["auth_type"]`, or the default auth type of the kind when the key is missing."""
+ return effective_auth_type(self.kind, self.config)
+
@property
def display_name(self) -> str:
+ if self.auth_type != default_auth_type(self.kind):
+ # The branches below read the config shape of the default auth type.
+ return self.config.get("display_name") or f"ID: {self.integration_id}"
if self.kind == "twitter-ads":
return self.config.get("screen_name") or self.integration_id
if self.kind == "pinterest-ads":
@@ -311,7 +450,20 @@ class Integration(models.Model):
return _decrypted_sensitive_value(self, "refresh_token")
-def defer_repository_cache_fields(queryset: models.QuerySet[Integration]) -> models.QuerySet[Integration]:
+# `setattr`, because the type of `objects` is the manager, not its descriptor.
+setattr(Integration, "objects", ManagerDescriptor(Integration._implicit_default_auth_objects)) # noqa: B010
+
+
+def integration_model(record: IntegrationRecord) -> Integration:
+ """The model behind a row of the integrations SDK.
+
+ The SDK returns its rows as the read-only `IntegrationRecord`, and they are model instances at
+ runtime. Only the per-kind adapters next to the model use it, because they still write the row.
+ """
+ return cast(Integration, record)
+
+
+def defer_repository_cache_fields[QuerySetT: models.QuerySet[Integration]](queryset: QuerySetT) -> QuerySetT:
return queryset.defer("repository_cache", "repository_cache_updated_at")
diff --git a/posthog/models/integration/oauth.py b/posthog/models/integration/oauth.py
index 144016df4a3f..117714c49d4e 100644
--- a/posthog/models/integration/oauth.py
+++ b/posthog/models/integration/oauth.py
@@ -23,6 +23,7 @@ from posthog.credentials_service.instance_settings import get_many_or_instance_s
from posthog.credentials_service.settings_fallback import get_many_or_settings
from posthog.dataclasses import frozen
from posthog.egress.slack.observability import record_slack_api_response
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.plugins.plugin_server_api import reload_integrations_on_workers
from posthog.schema_enums import SlackIntegrationScope
@@ -555,8 +556,8 @@ class OauthIntegration:
def __str__(self) -> str:
return f"OauthIntegration(integration={self.integration.id}, kind={self.integration.kind}, team={self.integration.team_id})"
- def __init__(self, integration: model.Integration) -> None:
- self.integration = integration
+ def __init__(self, integration: IntegrationRecord) -> None:
+ self.integration = model.integration_model(integration)
@classmethod
@cache_for(timedelta(minutes=5))
diff --git a/posthog/models/integration/postgres.py b/posthog/models/integration/postgres.py
index b8a45edfcb43..90ba9835278c 100644
--- a/posthog/models/integration/postgres.py
+++ b/posthog/models/integration/postgres.py
@@ -2,6 +2,7 @@
from typing import ClassVar, Literal, NamedTuple
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.psycopg_helpers import has_ipv6_route
from posthog.security.postgres_hosts import ipv6_only_host_message
@@ -68,14 +69,14 @@ class PostgreSQLServerIntegration:
integration: model.Integration
integration_kind: ClassVar[_PostgreSQLServerKindType]
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != self.integration_kind:
raise common.IntegrationError(
"Integration provided is not the expected PostgreSQL server integration "
f"(got kind='{integration.kind}', expected kind='{self.integration_kind}')"
)
- self.integration = integration
+ self.integration = model.integration_model(integration)
@classmethod
def integration_from_config(
diff --git a/posthog/models/integration/push.py b/posthog/models/integration/push.py
index 0085b4afb213..cb0aaec7c0b7 100644
--- a/posthog/models/integration/push.py
+++ b/posthog/models/integration/push.py
@@ -12,6 +12,7 @@ from google.auth.transport.requests import Request as GoogleRequest
from google.oauth2 import service_account
from rest_framework.exceptions import ValidationError
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.plugins.plugin_server_api import reload_integrations_on_workers
@@ -124,10 +125,10 @@ def preserved_push_config(
class FirebaseIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "firebase":
raise Exception("FirebaseIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@classmethod
def integration_from_key(
@@ -278,10 +279,10 @@ class ApplePushIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "apns":
raise Exception("ApplePushIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
@classmethod
def integration_from_key(
diff --git a/posthog/models/integration/slack.py b/posthog/models/integration/slack.py
index 3273a167dcf2..3bc3aae0d1c2 100644
--- a/posthog/models/integration/slack.py
+++ b/posthog/models/integration/slack.py
@@ -28,6 +28,7 @@ from posthog.credentials_service.settings_fallback import get_many_or_settings
from posthog.egress.slack.client import SlackWebClient as WebClient
from posthog.ingress.slack.provider import build_slack_signature_scheme
from posthog.ingress.verify.schemes import VerificationOutcome, hmac_sha256_signature
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.instance_setting import get_instance_settings
from . import model
@@ -91,11 +92,11 @@ SLACK_MEMBERS_MAX_REQUESTS = 10
class SlackIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration, *, source: str = "integration") -> None:
+ def __init__(self, integration: IntegrationRecord, *, source: str = "integration") -> None:
if integration.kind not in SLACK_INTEGRATION_KINDS:
raise Exception("SlackIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
# The egress `source` label. A flow that posts on its own names itself here, so its Slack
# calls and 429s can be told apart from every other caller of the same integration.
self.source = source
diff --git a/posthog/models/integration/snowflake.py b/posthog/models/integration/snowflake.py
index 927f2b467ade..3220f2b8ac23 100644
--- a/posthog/models/integration/snowflake.py
+++ b/posthog/models/integration/snowflake.py
@@ -4,6 +4,7 @@ import re
from django.db import transaction
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.user import User
from posthog.models.utils import IntegrityError
@@ -46,12 +47,12 @@ class SnowflakeIntegration:
user: str
authentication_type: str
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != model.Integration.IntegrationKind.SNOWFLAKE:
raise SnowflakeIntegrationError(
f"Integration provided is not a Snowflake integration (got kind='{integration.kind}')"
)
- self.integration = integration
+ self.integration = model.integration_model(integration)
try:
self.account = integration.config["account"]
self.user = integration.config["user"]
diff --git a/posthog/models/integration/stripe.py b/posthog/models/integration/stripe.py
index 5160a5c7e141..a6794f032b73 100644
--- a/posthog/models/integration/stripe.py
+++ b/posthog/models/integration/stripe.py
@@ -17,6 +17,7 @@ from posthog.credentials_service.errors import CredentialsServiceFailure
from posthog.credentials_service.settings_fallback import get_or_settings
from posthog.dataclasses import frozen
from posthog.exceptions_capture import capture_exception
+from posthog.integrations.contracts import IntegrationRecord
from posthog.models.oauth import OAuthAccessToken, OAuthApplication, OAuthRefreshToken, lock_oauth_connection
from posthog.models.user import User
from posthog.models.utils import generate_random_oauth_access_token, generate_random_oauth_refresh_token
@@ -129,10 +130,10 @@ class StripeIntegration:
]
)
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "stripe":
raise ValueError(f"Expected stripe integration, got {integration.kind}")
- self.integration = integration
+ self.integration = model.integration_model(integration)
def _stripe_client(self) -> "StripeClient | None":
# Returns None when the required env vars are missing so callers can skip Stripe
diff --git a/posthog/models/integration/twilio.py b/posthog/models/integration/twilio.py
index 2be240e2b9e4..9c11de19a2d0 100644
--- a/posthog/models/integration/twilio.py
+++ b/posthog/models/integration/twilio.py
@@ -4,6 +4,8 @@ from typing import TYPE_CHECKING
from rest_framework.exceptions import ValidationError
+from posthog.integrations.contracts import IntegrationRecord
+
from . import model
if TYPE_CHECKING:
@@ -13,10 +15,10 @@ if TYPE_CHECKING:
class TwilioIntegration:
integration: model.Integration
- def __init__(self, integration: model.Integration) -> None:
+ def __init__(self, integration: IntegrationRecord) -> None:
if integration.kind != "twilio":
raise Exception("TwilioIntegration init called with Integration with wrong 'kind'")
- self.integration = integration
+ self.integration = model.integration_model(integration)
self._account_sid = self.integration.config["account_sid"]
self._auth_token = self.integration.sensitive_config["auth_token"]
diff --git a/posthog/models/test/test_integration_auth_types.py b/posthog/models/test/test_integration_auth_types.py
new file mode 100644
index 000000000000..527bdd189fd6
--- /dev/null
+++ b/posthog/models/test/test_integration_auth_types.py
@@ -0,0 +1,299 @@
+from collections.abc import Callable
+from typing import Any
+
+from posthog.test.base import BaseTest
+
+from django.db import models
+from django.db.migrations.state import ModelState
+from django.test import SimpleTestCase
+
+from parameterized import parameterized
+
+from posthog.integrations.catalogue import KINDS
+from posthog.models import Team, User
+from posthog.models.integration import ANY, DEFAULT, AuthRequirement, Integration, IntegrationAuthType
+from posthog.models.integration.auth_types import config_for_write
+
+AuthType = IntegrationAuthType
+
+
+class TestIntegrationAuthTypeCatalogue(SimpleTestCase):
+ def test_every_integration_kind_has_a_catalogue_entry(self) -> None:
+ assert set(KINDS) == set(Integration.IntegrationKind.values)
+
+ @parameterized.expand(
+ [
+ ("nothing", ()),
+ ("any_with_a_type", (ANY, AuthType.OAUTH)),
+ ("any_with_default", (ANY, DEFAULT)),
+ ("unknown_string", ("bogus",)),
+ ]
+ )
+ def test_requirement_rejects_invalid_selectors(self, _name: str, selectors: tuple[Any, ...]) -> None:
+ with self.assertRaises(ValueError):
+ AuthRequirement.of(*selectors)
+
+ @parameterized.expand(
+ [
+ ("non_default_type_is_written", "github", {}, (AuthType.TOKEN,), {"auth_type": "token"}),
+ ("plain_string_type_is_written", "github", {"a": 1}, ("token",), {"a": 1, "auth_type": "token"}),
+ ("missing_config_gets_the_type", "slack", None, (AuthType.TOKEN,), {"auth_type": "token"}),
+ ("token_on_a_credentials_kind_is_written", "databricks", {}, (AuthType.TOKEN,), {"auth_type": "token"}),
+ ("default_type_is_not_written", "github", {}, (AuthType.OAUTH,), {}),
+ ("stored_type_in_requirement_is_kept", "github", {"auth_type": "token"}, (AuthType.TOKEN,), None),
+ ("several_types_with_the_default", "github", {}, (AuthType.OAUTH, AuthType.TOKEN), {}),
+ ("default_selector", "github", {}, (DEFAULT,), {}),
+ ("any_keeps_config", "github", {"a": 1}, (ANY,), {"a": 1}),
+ ("any_keeps_stored_type", "github", {"auth_type": "token"}, (ANY,), None),
+ ("unknown_kind_default", "acme-crm", {}, (AuthType.CREDENTIALS,), {}),
+ ]
+ )
+ def test_config_for_write(
+ self,
+ _name: str,
+ kind: str | None,
+ config: dict | None,
+ selectors: tuple[Any, ...],
+ expected: dict | None,
+ ) -> None:
+ result = config_for_write(kind, config, AuthRequirement.of(*selectors))
+
+ assert result == (config if expected is None else expected)
+
+ @parameterized.expand(
+ [
+ ("type_the_kind_does_not_support", "postgresql", {}, (AuthType.TOKEN,)),
+ ("stored_type_the_kind_does_not_support", "postgresql", {"auth_type": "token"}, (ANY,)),
+ ("stored_type_outside_requirement", "github", {"auth_type": "token"}, (AuthType.OAUTH,)),
+ ("several_types_without_the_default", "github", {}, (AuthType.TOKEN, AuthType.CREDENTIALS)),
+ ("unknown_kind_with_a_specific_type", "acme-crm", {}, (AuthType.TOKEN,)),
+ # Every auth type is the default of some kind, so a write without the kind cannot tell.
+ ("no_kind_with_oauth", None, {}, (AuthType.OAUTH,)),
+ ("no_kind_with_token", None, {}, (AuthType.TOKEN,)),
+ ]
+ )
+ def test_config_for_write_rejects(
+ self, _name: str, kind: str | None, config: dict, selectors: tuple[Any, ...]
+ ) -> None:
+ with self.assertRaises(ValueError):
+ config_for_write(kind, config, AuthRequirement.of(*selectors))
+
+ @parameterized.expand(
+ [
+ ("missing_key", {}, None),
+ ("stored_default", {"auth_type": "oauth"}, None),
+ ("stored_non_default", {"auth_type": "token"}, ValueError),
+ ]
+ )
+ def test_config_for_write_without_with_auth(
+ self, _name: str, config: dict, expected_error: type[Exception] | None
+ ) -> None:
+ requirement = AuthRequirement.kind_default_only()
+
+ if expected_error is None:
+ assert config_for_write("github", config, requirement) == config
+ else:
+ with self.assertRaises(expected_error):
+ config_for_write("github", config, requirement)
+
+
+ROWS: dict[str, tuple[str, dict]] = {
+ "github_oauth": ("github", {}),
+ "github_oauth_explicit": ("github", {"auth_type": "oauth"}),
+ "github_null": ("github", {"auth_type": None}),
+ "github_token": ("github", {"auth_type": "token"}),
+ "slack_oauth": ("slack", {}),
+ "slack_token": ("slack", {"auth_type": "token"}),
+ "postgres": ("postgresql", {}),
+ "postgres_source": ("postgresql", {"warehouse_source_type": "Postgres"}),
+ "unknown_kind": ("acme-crm", {}),
+}
+DEFAULT_ROWS = {
+ "github_oauth",
+ "github_oauth_explicit",
+ "github_null",
+ "slack_oauth",
+ "postgres",
+ "postgres_source",
+ "unknown_kind",
+}
+# An unqualified query also skips warehouse source rows, whose config has another shape.
+IMPLICIT_ROWS = DEFAULT_ROWS - {"postgres_source"}
+GITHUB_OAUTH_ROWS = {"github_oauth", "github_oauth_explicit", "github_null"}
+
+
+class TestIntegrationWithAuth(BaseTest):
+ def setUp(self) -> None:
+ super().setUp()
+ self.rows = {
+ name: Integration.objects._with_auth(ANY).create(
+ team=self.team, kind=kind, integration_id=name, config=config
+ )
+ for name, (kind, config) in ROWS.items()
+ }
+
+ def _names(self, queryset: models.QuerySet[Integration]) -> set[str]:
+ return {row.integration_id for row in queryset.filter(team=self.team) if row.integration_id}
+
+ @parameterized.expand(
+ [
+ ("oauth", lambda: Integration.objects._with_auth(AuthType.OAUTH), GITHUB_OAUTH_ROWS | {"slack_oauth"}),
+ (
+ "token_across_kinds",
+ lambda: Integration.objects._with_auth(AuthType.TOKEN),
+ {"github_token", "slack_token"},
+ ),
+ (
+ "token_with_kind_in",
+ lambda: Integration.objects._with_auth(AuthType.TOKEN).filter(kind__in=["slack", "postgresql"]),
+ {"slack_token"},
+ ),
+ (
+ "credentials_with_the_unknown_kind_default",
+ lambda: Integration.objects._with_auth(AuthType.CREDENTIALS),
+ {"postgres", "postgres_source", "unknown_kind"},
+ ),
+ (
+ "oauth_with_kind",
+ lambda: Integration.objects._with_auth(AuthType.OAUTH).filter(kind="github"),
+ GITHUB_OAUTH_ROWS,
+ ),
+ ("default_selector", lambda: Integration.objects._with_auth(DEFAULT), DEFAULT_ROWS),
+ ("default_and_token", lambda: Integration.objects._with_auth(DEFAULT, AuthType.TOKEN), set(ROWS)),
+ ("any", lambda: Integration.objects._with_auth(ANY), set(ROWS)),
+ ("unqualified_query", lambda: Integration.objects.all(), IMPLICIT_ROWS),
+ ("unqualified_filter", lambda: Integration.objects.filter(kind="github"), GITHUB_OAUTH_ROWS),
+ (
+ "unqualified_filter_skips_source_rows",
+ lambda: Integration.objects.filter(kind="postgresql"),
+ {"postgres"},
+ ),
+ ("default_manager", lambda: Integration._default_manager.all(), set(ROWS)),
+ (
+ "narrowed_twice",
+ lambda: Integration.objects._with_auth(ANY)._with_auth(AuthType.TOKEN),
+ {"github_token", "slack_token"},
+ ),
+ ]
+ )
+ def test_with_auth_filters_rows(
+ self, _name: str, queryset: Callable[[], models.QuerySet[Integration]], expected: set[str]
+ ) -> None:
+ assert self._names(queryset()) == expected
+
+ def test_auth_type_property_reads_config_or_kind_default(self) -> None:
+ assert {
+ row.integration_id: row.auth_type for row in Integration.objects._with_auth(ANY).filter(team=self.team)
+ } == {
+ "github_oauth": "oauth",
+ "github_oauth_explicit": "oauth",
+ "github_null": "oauth",
+ "github_token": "token",
+ "slack_oauth": "oauth",
+ "slack_token": "token",
+ "postgres": "credentials",
+ "postgres_source": "credentials",
+ "unknown_kind": "credentials",
+ }
+
+ def test_save_drops_a_stored_auth_type_that_is_null_or_the_kind_default(self) -> None:
+ token_row = self.rows["github_token"]
+ token_row.config = {"auth_type": "oauth", "a": 1}
+ token_row.save()
+
+ stored = {
+ row.integration_id: row.config
+ for row in Integration._base_manager.filter(
+ team=self.team,
+ integration_id__in=["github_oauth_explicit", "github_null", "github_token", "slack_token"],
+ )
+ }
+ assert stored == {
+ "github_oauth_explicit": {},
+ "github_null": {},
+ "github_token": {"a": 1},
+ "slack_token": {"auth_type": "token"},
+ }
+
+ def test_unqualified_queries_do_not_touch_non_default_rows(self) -> None:
+ updated = Integration.objects.filter(team=self.team, kind="github").update(errors="x")
+ deleted, _ = Integration.objects.filter(team=self.team, kind="slack").delete()
+
+ assert (updated, deleted) == (3, 1)
+ assert Integration.objects._with_auth(AuthType.TOKEN).filter(team=self.team).count() == 2
+
+ @parameterized.expand(
+ [
+ ("create", lambda qs, **kw: qs.create(**kw)),
+ ("get_or_create", lambda qs, **kw: qs.get_or_create(**kw)[0]),
+ ("update_or_create", lambda qs, **kw: qs.update_or_create(**kw)[0]),
+ ("bulk_create", lambda qs, **kw: qs.bulk_create([Integration(**kw)])[0]),
+ ]
+ )
+ def test_writes_through_one_non_default_type_store_it(self, _name: str, write: Callable[..., Integration]) -> None:
+ row = write(Integration.objects._with_auth(AuthType.TOKEN), team=self.team, kind="github", integration_id="new")
+
+ assert Integration._base_manager.get(team=self.team, integration_id="new").config == {"auth_type": "token"}
+ assert row.auth_type == "token"
+ assert not Integration.objects.filter(team=self.team, integration_id="new").exists()
+
+ @parameterized.expand([("create",), ("get_or_create",), ("update_or_create",)])
+ def test_writes_through_the_default_type_store_nothing(self, method: str) -> None:
+ getattr(Integration.objects._with_auth(AuthType.OAUTH), method)(
+ team=self.team, kind="github", integration_id="new"
+ )
+
+ row = Integration.objects.get(team=self.team, integration_id="new")
+ assert (row.config, row.auth_type) == ({}, "oauth")
+
+ def test_update_or_create_keeps_the_auth_type_of_an_existing_row(self) -> None:
+ row, created = Integration.objects._with_auth(AuthType.TOKEN).update_or_create(
+ team=self.team, kind="github", integration_id="github_token", defaults={"config": {"account": "a"}}
+ )
+
+ row.refresh_from_db()
+ assert (created, row.config) == (False, {"account": "a", "auth_type": "token"})
+
+ def test_unqualified_create_rejects_a_non_default_type(self) -> None:
+ with self.assertRaises(ValueError):
+ Integration.objects.create(
+ team=self.team, kind="github", integration_id="new", config={"auth_type": "token"}
+ )
+
+ @parameterized.expand(
+ [
+ ("all", lambda: Integration.objects.all()._with_auth(ANY)), # type: ignore[attr-defined]
+ ("filter", lambda: Integration.objects.filter(kind="github")._with_auth(AuthType.TOKEN)),
+ (
+ "chained_auth_types",
+ lambda: Integration.objects._with_auth(AuthType.OAUTH)._with_auth(AuthType.TOKEN),
+ ),
+ ]
+ )
+ def test_with_auth_after_an_unqualified_query_raises(self, _name: str, build: Callable[[], Any]) -> None:
+ with self.assertRaises(ValueError):
+ build()
+
+ def test_related_and_base_managers_are_not_filtered(self) -> None:
+ token_row = self.rows["github_token"]
+ prefetched_team = Team.objects.prefetch_related("integration_set").get(pk=self.team.pk)
+
+ token_row.refresh_from_db()
+ assert {row.integration_id for row in self.team.integration_set.all()} == set(ROWS)
+ assert {row.integration_id for row in prefetched_team.integration_set.all()} == set(ROWS)
+ assert self._names(Integration._base_manager.all()) == set(ROWS)
+ assert self._names(self.team.integration_set._with_auth(AuthType.TOKEN)) == {"github_token", "slack_token"}
+
+ def test_the_unfiltered_default_manager_does_not_change_the_migration_state(self) -> None:
+ # A default manager that is not named `objects` would need a migration.
+ assert (Integration._default_manager.name, ModelState.from_model(Integration).managers) == ("objects", [])
+ assert Integration.objects is not Integration._default_manager
+ assert self._names(Integration._default_manager._with_auth(AuthType.TOKEN)) == {"github_token", "slack_token"}
+
+ def test_deleting_the_creator_reaches_non_default_rows(self) -> None:
+ creator = User.objects.create_and_join(self.organization, "[email protected]", None)
+ Integration._base_manager.filter(team=self.team).update(created_by=creator)
+
+ creator.delete()
+
+ assert not Integration._base_manager.filter(team=self.team, created_by__isnull=False).exists()
diff --git a/posthog/owners.yaml b/posthog/owners.yaml
index 2de43abd6d05..8217231dd6ba 100644
--- a/posthog/owners.yaml
+++ b/posthog/owners.yaml
@@ -19,6 +19,8 @@ rules:
owners: team-web-analytics
- match: '/hogql/'
owners: team-data-tools
+ - match: '/integrations/'
+ owners: team-warehouse-sources
- match: '/management/migration_analysis/'
owners: team-devex
# Each line acknowledges a risky migration, so one added line lets that migration skip the check.
diff --git a/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr b/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr
index 256d1841491a..262e99026415 100644
--- a/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr
+++ b/posthog/tasks/test/__snapshots__/test_process_scheduled_changes.ambr
@@ -38,7 +38,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -290,7 +293,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -1196,7 +1202,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
diff --git a/products/desktop/packages/api-client/src/generated.ts b/products/desktop/packages/api-client/src/generated.ts
index bc567b1f6bff..fb60fcdbd537 100644
--- a/products/desktop/packages/api-client/src/generated.ts
+++ b/products/desktop/packages/api-client/src/generated.ts
@@ -16391,6 +16391,12 @@ export namespace Schemas {
* * `unavailable` - unavailable
*/
export type InstallationStatusEnum = "connected" | "unavailable";
+ /**
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials
+ */
+ export type IntegrationAuthTypeEnum = "oauth" | "token" | "credentials";
/**
* * `anthropic` - Anthropic
* * `apple-ads` - Apple Ads
@@ -16500,6 +16506,14 @@ export namespace Schemas {
export type IntegrationConfig = {
id: number;
kind: IntegrationKindEnum;
+ /**
+ * How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.
+ *
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials
+ */
+ auth_type: IntegrationAuthTypeEnum & unknown;
/**
* Standard Integration serializer.
*/
diff --git a/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr b/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr
index d6286d5b8ad7..78aef96b3c42 100644
--- a/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr
+++ b/products/early_access_features/backend/test/__snapshots__/test_early_access_feature.ambr
@@ -38,7 +38,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -287,7 +290,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -452,7 +458,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -878,7 +887,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
@@ -1108,7 +1120,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
diff --git a/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr b/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr
index a6d93d281e93..e95941f512d8 100644
--- a/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr
+++ b/products/feature_flags/backend/api/test/__snapshots__/test_organization_feature_flag.ambr
@@ -380,7 +380,10 @@
"posthog_integration"."created_at",
"posthog_integration"."created_by_id"
FROM "posthog_integration"
- WHERE ("posthog_integration"."kind" = 'vercel'
+ WHERE ((("posthog_integration"."config" -> 'auth_type') IS NULL
+ OR ("posthog_integration"."config" -> 'auth_type') = 'null'::jsonb)
+ AND NOT ("posthog_integration"."config" ? 'warehouse_source_type')
+ AND "posthog_integration"."kind" = 'vercel'
AND "posthog_integration"."team_id" = 99999)
LIMIT 21
'''
diff --git a/products/integrations/frontend/generated/api.schemas.ts b/products/integrations/frontend/generated/api.schemas.ts
index e07e9a291b51..713480dfb364 100644
--- a/products/integrations/frontend/generated/api.schemas.ts
+++ b/products/integrations/frontend/generated/api.schemas.ts
@@ -257,6 +257,19 @@ export const IntegrationKindEnumApi = {
YoutubeAnalytics: 'youtube-analytics',
} as const
+/**
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials
+ */
+export type IntegrationAuthTypeEnumApi = (typeof IntegrationAuthTypeEnumApi)[keyof typeof IntegrationAuthTypeEnumApi]
+
+export const IntegrationAuthTypeEnumApi = {
+ Oauth: 'oauth',
+ Token: 'token',
+ Credentials: 'credentials',
+} as const
+
/**
* * `connected` - connected
* * `unavailable` - unavailable
@@ -274,6 +287,12 @@ export const InstallationStatusEnumApi = {
export interface IntegrationConfigApi {
readonly id: number
kind: IntegrationKindEnumApi
+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.
+ *
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials */
+ readonly auth_type: IntegrationAuthTypeEnumApi
config?: unknown
readonly created_at: string
readonly created_by: UserBasicApi
@@ -332,6 +351,12 @@ export interface SlackChannelsResponseApi {
export interface PatchedIntegrationConfigApi {
readonly id?: number
kind?: IntegrationKindEnumApi
+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.
+ *
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials */
+ readonly auth_type?: IntegrationAuthTypeEnumApi
config?: unknown
readonly created_at?: string
readonly created_by?: UserBasicApi
diff --git a/products/warehouse_sources/backend/apps.py b/products/warehouse_sources/backend/apps.py
index 7f1ae3c23c44..2854587ea072 100644
--- a/products/warehouse_sources/backend/apps.py
+++ b/products/warehouse_sources/backend/apps.py
@@ -20,3 +20,13 @@ class WarehouseSourcesConfig(AppConfig):
from products.warehouse_sources.backend.sync_alerts.registration import register # noqa: PLC0415
register()
+
+ # Source kinds have no generated accessor. Callers reach their rows through `IntegrationManager.source(kind)`.
+ # The SDK imports the Integration model, which needs the app registry.
+ from posthog.integrations import AuthType, register_source_kinds # noqa: PLC0415
+
+ from products.warehouse_sources.backend.facade.credential_auth_types import ( # noqa: PLC0415
+ CREDENTIAL_AUTH_TYPES,
+ )
+
+ register_source_kinds({kind: AuthType(auth_type) for kind, auth_type in CREDENTIAL_AUTH_TYPES.items()})
diff --git a/products/warehouse_sources/backend/facade/credential_auth_types.py b/products/warehouse_sources/backend/facade/credential_auth_types.py
new file mode 100644
index 000000000000..c66cf7a1ab73
--- /dev/null
+++ b/products/warehouse_sources/backend/facade/credential_auth_types.py
@@ -0,0 +1,1425 @@
+"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.
+
+The default auth type of each warehouse source credential kind, keyed by kind. A row stores its auth
+type only when it is not the default, so a change to an existing value needs a data migration. See
+`sources/common/credential_auth_type.py` for the derivation. A kind in the integrations catalogue, such
+as `stripe`, keeps the default of the catalogue. The file is append-only: a removed kind keeps its
+entry.
+"""
+
+from collections.abc import Mapping
+
+CREDENTIAL_AUTH_TYPES: Mapping[str, str] = {
+ "ab-tasty": "credentials",
+ "ably": "token",
+ "abnormal-security": "credentials",
+ "acast": "token",
+ "acculynx": "token",
+ "actionstep": "credentials",
+ "active-campaign": "credentials",
+ "acuity-scheduling": "credentials",
+ "ad-roll": "credentials",
+ "adapty": "token",
+ "adjust": "token",
+ "adobe-analytics": "credentials",
+ "adobe-commerce": "credentials",
+ "adp-workforce-now": "credentials",
+ "adyen": "credentials",
+ "aftership": "token",
+ "agile-crm": "credentials",
+ "aha": "credentials",
+ "aha-ideas": "credentials",
+ "ahrefs": "credentials",
+ "aikido-security": "credentials",
+ "air-ops": "token",
+ "airbrake": "token",
+ "airbridge": "credentials",
+ "airbyte": "credentials",
+ "aircall": "credentials",
+ "airship": "credentials",
+ "airtable": "token",
+ "airwallex": "credentials",
+ "aiven": "token",
+ "akamai-reporting": "credentials",
+ "akeneo": "credentials",
+ "alation": "credentials",
+ "alegra": "credentials",
+ "algolia": "credentials",
+ "alguna": "token",
+ "allegro": "credentials",
+ "alpaca-broker-api": "credentials",
+ "alpha-vantage": "token",
+ "amazon-ads": "credentials",
+ "amazon-cloud-watch": "credentials",
+ "amazon-event-bridge": "credentials",
+ "amazon-kinesis": "credentials",
+ "amazon-selling-partner": "credentials",
+ "amazon-sns": "credentials",
+ "amazon-sqs": "credentials",
+ "amplemarket": "token",
+ "amplitude": "credentials",
+ "anodot-cost": "credentials",
+ "anomalo": "credentials",
+ "anthropic": "token",
+ "anvil": "token",
+ "apaleo": "credentials",
+ "apify-dataset": "credentials",
+ "apitally": "token",
+ "apollo": "token",
+ "app-lovin": "credentials",
+ "app-store-connect": "credentials",
+ "appcues": "credentials",
+ "appdirect": "credentials",
+ "appdynamics": "credentials",
+ "appfigures": "token",
+ "appfolio": "credentials",
+ "appfollow": "token",
+ "apple-ads": "credentials",
+ "apps-flyer": "credentials",
+ "appsignal": "credentials",
+ "appstack": "token",
+ "apptivo": "credentials",
+ "appwrite": "credentials",
+ "arcade": "credentials",
+ "argocd": "credentials",
+ "arxiv": "credentials",
+ "asaas": "credentials",
+ "asana": "token",
+ "ashby": "token",
+ "asknicely": "credentials",
+ "assembly-ai": "credentials",
+ "astronomer": "credentials",
+ "athenahealth": "credentials",
+ "atlan": "credentials",
+ "attentive": "token",
+ "attio": "token",
+ "audio-go": "credentials",
+ "auth0": "credentials",
+ "autodesk-construction-cloud": "credentials",
+ "automox": "credentials",
+ "autumn": "token",
+ "avalara": "credentials",
+ "aviationstack": "credentials",
+ "aviator": "token",
+ "awin": "credentials",
+ "aws-athena": "credentials",
+ "aws-batch": "credentials",
+ "aws-budgets": "credentials",
+ "aws-cloud-trail": "credentials",
+ "aws-cloudformation": "credentials",
+ "aws-compute-optimizer": "credentials",
+ "aws-config": "credentials",
+ "aws-connect": "credentials",
+ "aws-cost-and-usage-report": "credentials",
+ "aws-cost-anomaly-detection": "credentials",
+ "aws-cost-explorer": "credentials",
+ "aws-documentdb": "credentials",
+ "aws-glue-data-catalog": "credentials",
+ "aws-guardduty": "credentials",
+ "aws-health": "credentials",
+ "aws-iam-access-analyzer": "credentials",
+ "aws-inspector": "credentials",
+ "aws-macie": "credentials",
+ "aws-organizations": "credentials",
+ "aws-rds-performance-insights": "credentials",
+ "aws-redshift": "credentials",
+ "aws-s3": "credentials",
+ "aws-s3-tables": "credentials",
+ "aws-sagemaker": "credentials",
+ "aws-savings-plans": "credentials",
+ "aws-security-hub": "credentials",
+ "aws-ses": "credentials",
+ "aws-step-functions": "credentials",
+ "aws-support": "credentials",
+ "aws-systems-manager": "credentials",
+ "aws-trusted-advisor": "credentials",
+ "aws-waf": "credentials",
+ "aws-xray": "credentials",
+ "axiom": "credentials",
+ "azure-activity-log": "credentials",
+ "azure-advisor": "credentials",
+ "azure-api-management": "credentials",
+ "azure-application-insights": "credentials",
+ "azure-blob": "credentials",
+ "azure-cost-management": "credentials",
+ "azure-data-explorer": "credentials",
+ "azure-data-factory": "credentials",
+ "azure-dev-ops": "credentials",
+ "azure-log-analytics": "credentials",
+ "azure-monitor-alerts": "credentials",
+ "azure-monitor-metrics": "credentials",
+ "azure-openai-usage": "credentials",
+ "azure-policy-insights": "credentials",
+ "azure-reservations": "credentials",
+ "azure-resource-graph": "credentials",
+ "azure-resource-health": "credentials",
+ "azure-service-health": "credentials",
+ "azure-synapse": "credentials",
+ "azure-table-storage": "credentials",
+ "babelforce": "credentials",
+ "back-market": "token",
+ "backblaze": "credentials",
+ "bamboo-hr": "credentials",
+ "basecamp": "credentials",
+ "baserow": "credentials",
+ "baseten": "token",
+ "bcms": "credentials",
+ "beacons": "credentials",
+ "beamer": "token",
+ "beehiiv": "credentials",
+ "better-stack": "token",
+ "bettermode": "credentials",
+ "bexio": "credentials",
+ "big-commerce": "credentials",
+ "big-mailer": "token",
+ "bigeye": "credentials",
+ "bill-com": "credentials",
+ "billit": "credentials",
+ "billomat": "credentials",
+ "bing-ads": "oauth",
+ "bing-webmaster-tools": "credentials",
+ "bitbucket": "credentials",
+ "bitly": "credentials",
+ "bitrise": "token",
+ "bitwarden": "credentials",
+ "blackbaud-raisers-edge-nxt": "credentials",
+ "blackboard-learn": "credentials",
+ "bland-ai": "token",
+ "bling": "credentials",
+ "blogger": "credentials",
+ "bloomerang": "token",
+ "blueshift": "credentials",
+ "bluesky": "credentials",
+ "bluetally": "credentials",
+ "bol-retailer": "credentials",
+ "bold-sign": "credentials",
+ "boulevard": "credentials",
+ "box": "credentials",
+ "braintree": "credentials",
+ "braintrust": "credentials",
+ "branch": "credentials",
+ "braze": "credentials",
+ "breezometer": "token",
+ "breezy-hr": "credentials",
+ "brevo": "token",
+ "brex": "token",
+ "browse-ai": "token",
+ "browser-use": "token",
+ "browserbase": "token",
+ "buffer": "credentials",
+ "bugherd": "token",
+ "bugsnag": "token",
+ "build-better": "token",
+ "buildium": "credentials",
+ "buildkite": "credentials",
+ "bunny": "token",
+ "buttondown": "token",
+ "buy-me-a-coffee": "token",
+ "buzzsprout": "credentials",
+ "cal-com": "credentials",
+ "calendarific": "token",
+ "calendly": "token",
+ "calibre": "credentials",
+ "call-rail": "credentials",
+ "campaign-manager360": "credentials",
+ "campaign-monitor": "credentials",
+ "campayn": "credentials",
+ "campfire": "token",
+ "canny": "token",
+ "canvas-lms": "credentials",
+ "capsule-crm": "token",
+ "captain-data": "credentials",
+ "capterra": "credentials",
+ "captivate": "credentials",
+ "care-quality-commission": "credentials",
+ "cart-com": "credentials",
+ "cashfree": "credentials",
+ "cast-ai": "token",
+ "castor-edc": "credentials",
+ "catchpoint": "token",
+ "cdc-open-data": "token",
+ "census": "credentials",
+ "chameleon": "token",
+ "channel-io": "credentials",
+ "chargeback-stop": "credentials",
+ "chargebee": "credentials",
+ "chargedesk": "token",
+ "chargeflow": "credentials",
+ "chargify": "credentials",
+ "chart-hop": "credentials",
+ "chart-mogul": "token",
+ "chatwoot": "credentials",
+ "check-hq": "credentials",
+ "checkly": "credentials",
+ "checkmarx": "credentials",
+ "checkout-com": "credentials",
+ "chess-com": "credentials",
+ "chift": "credentials",
+ "chorus": "credentials",
+ "churnkey": "credentials",
+ "cimis": "token",
+ "cin7": "credentials",
+ "circle-ci": "credentials",
+ "circle-so": "credentials",
+ "circleci-insights": "credentials",
+ "cisco-duo": "credentials",
+ "cisco-meraki": "credentials",
+ "clari": "credentials",
+ "clarifai": "credentials",
+ "clarify": "credentials",
+ "classy": "credentials",
+ "clay": "token",
+ "clazar": "credentials",
+ "cleartax": "credentials",
+ "clerk": "token",
+ "clever": "token",
+ "clevertap": "credentials",
+ "clickhouse": "credentials",
+ "clickhouse-cloud": "credentials",
+ "clickup": "credentials",
+ "cliniko": "token",
+ "clio": "credentials",
+ "clip": "credentials",
+ "clockify": "token",
+ "clockodo": "credentials",
+ "close": "token",
+ "cloudability": "credentials",
+ "cloudbeds": "credentials",
+ "cloudflare": "token",
+ "cloudflare-d1": "credentials",
+ "cloudflare-r2": "credentials",
+ "cloudinary": "credentials",
+ "cloudsmith": "credentials",
+ "cloudzero": "token",
+ "clover": "credentials",
+ "coassemble": "credentials",
+ "cockroachdb": "credentials",
+ "coda": "token",
+ "codacy": "credentials",
+ "codecov": "credentials",
+ "codefresh": "token",
+ "codemagic": "token",
+ "codescene": "credentials",
+ "cody": "credentials",
+ "cohere": "token",
+ "coin-api": "token",
+ "coin-gecko": "credentials",
+ "coin-market-cap": "token",
+ "collibra": "credentials",
+ "commercetools": "credentials",
+ "commission-junction": "credentials",
+ "commslayer": "credentials",
+ "companycam": "token",
+ "concord": "credentials",
+ "conecto-crm": "credentials",
+ "conekta": "token",
+ "config-cat": "credentials",
+ "confluence": "credentials",
+ "confluent-cloud": "credentials",
+ "constant-contact": "credentials",
+ "conta-azul": "credentials",
+ "contentsquare": "credentials",
+ "convert-kit": "token",
+ "convex": "credentials",
+ "convonite": "credentials",
+ "coolify": "credentials",
+ "copper": "credentials",
+ "coralogix": "credentials",
+ "cortex": "token",
+ "cosmosdb": "credentials",
+ "couchbase": "credentials",
+ "coupa": "credentials",
+ "courier": "token",
+ "coveralls": "credentials",
+ "crates-io": "credentials",
+ "creem": "credentials",
+ "crisp": "credentials",
+ "criteo": "credentials",
+ "cronitor": "token",
+ "crossref": "credentials",
+ "crowdstrike-falcon": "credentials",
+ "crunchbase": "token",
+ "cube-cloud": "credentials",
+ "culture-amp": "credentials",
+ "cursor": "token",
+ "curve": "credentials",
+ "customer-io": "credentials",
+ "customerly": "token",
+ "cybersource": "credentials",
+ "d2l-brightspace": "credentials",
+ "dagster-cloud": "credentials",
+ "data-for-seo": "credentials",
+ "databricks": "credentials",
+ "datadog": "credentials",
+ "datahub": "credentials",
+ "datascope": "credentials",
+ "dato-cms": "credentials",
+ "datorama": "credentials",
+ "dayforce": "credentials",
+ "db2": "credentials",
+ "dbt": "credentials",
+ "debugbear": "token",
+ "decagon": "token",
+ "deel": "token",
+ "deel-flows": "credentials",
+ "deepgram": "token",
+ "deepsource": "credentials",
+ "demodesk": "token",
+ "deno-deploy": "token",
+ "depot": "token",
+ "deputy": "credentials",
+ "descope": "credentials",
+ "develocity": "credentials",
+ "devin-ai": "credentials",
+ "dialpad": "credentials",
+ "digital-ocean": "token",
+ "ding-connect": "token",
+ "directus": "credentials",
+ "discord": "credentials",
+ "discourse": "credentials",
+ "display-video360": "credentials",
+ "dixa": "token",
+ "do-it": "token",
+ "dockerhub": "credentials",
+ "docuseal": "credentials",
+ "docusign": "credentials",
+ "dodo-payments": "credentials",
+ "dokploy": "credentials",
+ "dolibarr": "credentials",
+ "donorbox": "credentials",
+ "doorloop": "credentials",
+ "doppler": "token",
+ "dovetail": "token",
+ "dragonboat": "credentials",
+ "drata": "credentials",
+ "drchrono": "credentials",
+ "dreamdata": "credentials",
+ "dremio": "credentials",
+ "drip": "credentials",
+ "dripify": "credentials",
+ "dropbox": "credentials",
+ "dropbox-sign": "token",
+ "dub": "token",
+ "dubsado": "credentials",
+ "duck-lake": "credentials",
+ "dwolla": "credentials",
+ "dynamics365": "credentials",
+ "dynamics365-business-central": "credentials",
+ "dynamodb": "credentials",
+ "dynatrace": "credentials",
+ "e-conomic": "credentials",
+ "e2b": "credentials",
+ "easybill": "token",
+ "easypost": "token",
+ "easypromos": "token",
+ "ebay": "credentials",
+ "ecb-data-portal": "credentials",
+ "elasticemail": "token",
+ "elasticsearch": "credentials",
+ "electricity-maps": "token",
+ "eleven-labs": "token",
+ "eloqua": "credentials",
+ "email-octopus": "token",
+ "emarsys": "credentials",
+ "embrace": "credentials",
+ "employment-hero": "credentials",
+ "encharge": "credentials",
+ "entsoe": "credentials",
+ "env0": "credentials",
+ "eppo": "token",
+ "etsy": "credentials",
+ "eurostat": "credentials",
+ "eventbrite": "token",
+ "eventee": "token",
+ "eventzilla": "token",
+ "everflow": "credentials",
+ "everhour": "token",
+ "exact-online": "credentials",
+ "exchange-rates-api": "token",
+ "expensify": "credentials",
+ "expo": "credentials",
+ "ez-office-inventory": "credentials",
+ "facebook-pages": "credentials",
+ "factorial": "token",
+ "faire": "token",
+ "faros-ai": "credentials",
+ "fastbill": "credentials",
+ "fastly": "token",
+ "fauna": "credentials",
+ "featurebase": "token",
+ "feishu": "credentials",
+ "fieldpulse": "token",
+ "fieldwire": "credentials",
+ "file-upload": "credentials",
+ "filevine": "credentials",
+ "fillout": "credentials",
+ "finage": "token",
+ "financial-modelling": "token",
+ "finnhub": "token",
+ "finnworlds": "token",
+ "finout": "credentials",
+ "fintoc": "credentials",
+ "fire-hydrant": "credentials",
+ "firebase": "credentials",
+ "firebolt": "credentials",
+ "firecrawl": "token",
+ "fireworks-ai": "credentials",
+ "first-promoter": "credentials",
+ "five9": "credentials",
+ "flagsmith": "credentials",
+ "fleetio": "credentials",
+ "flexera-cloud-cost": "credentials",
+ "flexmail": "credentials",
+ "flexport": "credentials",
+ "float-app": "token",
+ "flowlu": "credentials",
+ "flutterwave": "credentials",
+ "fly-io": "credentials",
+ "folk": "token",
+ "formbricks": "credentials",
+ "fortnox": "credentials",
+ "fourthwall": "credentials",
+ "framer": "credentials",
+ "fred": "token",
+ "free-agent": "credentials",
+ "freightview": "credentials",
+ "fresh-books": "credentials",
+ "freshcaller": "credentials",
+ "freshchat": "credentials",
+ "freshdesk": "credentials",
+ "freshsales": "credentials",
+ "freshservice": "credentials",
+ "frill": "token",
+ "front": "token",
+ "frontegg": "credentials",
+ "fulcrum": "token",
+ "full-story": "token",
+ "fusion-auth": "credentials",
+ "g2": "credentials",
+ "gainsight-cs": "credentials",
+ "gainsight-px": "credentials",
+ "gcore": "token",
+ "gcp-apigee": "credentials",
+ "gcp-artifact-registry": "credentials",
+ "gcp-bigtable": "credentials",
+ "gcp-chronicle": "credentials",
+ "gcp-cloud-asset-inventory": "credentials",
+ "gcp-cloud-billing": "credentials",
+ "gcp-cloud-build": "credentials",
+ "gcp-cloud-deploy": "credentials",
+ "gcp-cloud-dns": "credentials",
+ "gcp-cloud-functions": "credentials",
+ "gcp-cloud-logging": "credentials",
+ "gcp-cloud-monitoring": "credentials",
+ "gcp-cloud-run": "credentials",
+ "gcp-cloud-spanner": "credentials",
+ "gcp-cloud-sql": "credentials",
+ "gcp-cloud-trace": "credentials",
+ "gcp-cloud-workflows": "credentials",
+ "gcp-compute-engine": "credentials",
+ "gcp-container-analysis": "credentials",
+ "gcp-dataflow": "credentials",
+ "gcp-dataplex": "credentials",
+ "gcp-dataproc": "credentials",
+ "gcp-error-reporting": "credentials",
+ "gcp-gke": "credentials",
+ "gcp-pubsub": "credentials",
+ "gcp-recaptcha-enterprise": "credentials",
+ "gcp-recommender": "credentials",
+ "gcp-security-command-center": "credentials",
+ "gdelt": "credentials",
+ "gem": "credentials",
+ "genesys-cloud": "credentials",
+ "gerrit": "credentials",
+ "get-stream": "credentials",
+ "getdx": "token",
+ "ghost": "credentials",
+ "giphy": "token",
+ "git-book": "token",
+ "gitea": "credentials",
+ "gitguardian": "credentials",
+ "github": "token",
+ "gitlab": "credentials",
+ "givebutter": "token",
+ "gladly": "credentials",
+ "glassfrog": "token",
+ "gleif": "credentials",
+ "gmail": "credentials",
+ "gnews": "token",
+ "go-cardless": "credentials",
+ "go-login": "token",
+ "gojiberry": "credentials",
+ "goldcast": "token",
+ "gong": "credentials",
+ "google-ad-manager": "credentials",
+ "google-ad-mob": "credentials",
+ "google-ad-sense": "credentials",
+ "google-ads": "oauth",
+ "google-analytics": "oauth",
+ "google-business-profile": "credentials",
+ "google-calendar": "credentials",
+ "google-chat": "credentials",
+ "google-classroom": "credentials",
+ "google-cloud-service-account": "credentials",
+ "google-cloud-storage": "credentials",
+ "google-directory": "credentials",
+ "google-drive": "credentials",
+ "google-forms": "credentials",
+ "google-merchant-center": "credentials",
+ "google-page-speed-insights": "token",
+ "google-play-console": "credentials",
+ "google-postmaster-tools": "credentials",
+ "google-search-console": "oauth",
+ "google-sheets": "credentials",
+ "google-tasks": "credentials",
+ "google-webfonts": "token",
+ "google-workspace-admin-reports": "credentials",
+ "gorgias": "credentials",
+ "grafana": "credentials",
+ "grafana-irm": "credentials",
+ "granola": "token",
+ "greenhouse": "credentials",
+ "greyt-hr": "credentials",
+ "gridly": "credentials",
+ "groq": "token",
+ "growi": "credentials",
+ "growth-book": "credentials",
+ "guardian": "token",
+ "guesty": "credentials",
+ "gumloop": "credentials",
+ "gumroad": "token",
+ "guru": "credentials",
+ "gusto": "credentials",
+ "harness": "credentials",
+ "harness-ccm": "credentials",
+ "harness-sei": "credentials",
+ "harvest": "credentials",
+ "harvey": "credentials",
+ "hatchet": "credentials",
+ "healthchecks": "credentials",
+ "healthie": "credentials",
+ "heap": "credentials",
+ "height": "token",
+ "helicone": "credentials",
+ "hellobaton": "credentials",
+ "helpscout": "oauth",
+ "heroku": "token",
+ "hetzner": "token",
+ "hex": "credentials",
+ "hey-gen": "token",
+ "hey-reach": "token",
+ "hi-bob": "credentials",
+ "high-level": "credentials",
+ "hightouch": "token",
+ "hitpay": "credentials",
+ "hivebrite": "credentials",
+ "holded": "credentials",
+ "honeybadger": "token",
+ "honeycomb": "credentials",
+ "hookdeck": "token",
+ "hooray-hr": "token",
+ "hootsuite": "credentials",
+ "hostaway": "credentials",
+ "housecall-pro": "token",
+ "hubplanner": "token",
+ "hubspot": "oauth",
+ "hugging-face": "credentials",
+ "humanitec": "credentials",
+ "humanitix": "token",
+ "huntr": "token",
+ "hyperspell": "credentials",
+ "hyros": "token",
+ "ikas": "credentials",
+ "illumina-basespace": "credentials",
+ "imagga": "credentials",
+ "imf-data": "credentials",
+ "impact": "credentials",
+ "impact-partner": "credentials",
+ "imperva": "credentials",
+ "incident-io": "token",
+ "infisical": "credentials",
+ "inflowinventory": "credentials",
+ "influxdb-cloud": "credentials",
+ "infor-nexus": "credentials",
+ "inngest": "credentials",
+ "insightful": "credentials",
+ "insightly": "credentials",
+ "instagram": "oauth",
+ "instana": "credentials",
+ "instantly": "token",
+ "instatus": "token",
+ "intercom": "oauth",
+ "interzoid": "credentials",
+ "inth": "credentials",
+ "intruder": "token",
+ "invoiced": "token",
+ "invoiceninja": "credentials",
+ "ip2-whois": "token",
+ "iron-source-ads": "credentials",
+ "iterable": "credentials",
+ "iyzico": "credentials",
+ "jamf-pro": "credentials",
+ "jellyfish": "token",
+ "jenkins": "credentials",
+ "jfrog-artifactory": "credentials",
+ "jira": "credentials",
+ "job-nimbus": "token",
+ "jobber": "credentials",
+ "jobtread": "credentials",
+ "jotform": "credentials",
+ "judge-me-reviews": "credentials",
+ "jumpcloud": "credentials",
+ "just-call": "credentials",
+ "just-sift": "token",
+ "k6-cloud": "credentials",
+ "kafka": "credentials",
+ "kajabi": "credentials",
+ "kalshi": "credentials",
+ "kameleoon": "credentials",
+ "kandji": "credentials",
+ "kapa-ai": "credentials",
+ "katana": "token",
+ "kaufland-marketplace": "credentials",
+ "keka": "credentials",
+ "kernel": "token",
+ "kestra": "credentials",
+ "kick": "credentials",
+ "kickscale": "credentials",
+ "kickstarter": "credentials",
+ "kinde": "credentials",
+ "kion": "credentials",
+ "kisi": "token",
+ "kissmetrics": "credentials",
+ "klar": "credentials",
+ "klarna": "credentials",
+ "klaus": "credentials",
+ "klaviyo": "token",
+ "knock": "token",
+ "knowbe4": "credentials",
+ "kommo": "credentials",
+ "komodor": "credentials",
+ "kong-konnect": "credentials",
+ "koyeb": "token",
+ "kubecost": "credentials",
+ "kustomer": "credentials",
+ "kyve": "credentials",
+ "labelbox": "credentials",
+ "lacework": "credentials",
+ "lago": "credentials",
+ "lambda-labs": "token",
+ "lang-smith": "credentials",
+ "langfuse": "credentials",
+ "laravel-cloud": "credentials",
+ "latitude": "credentials",
+ "lattice": "credentials",
+ "launch-darkly": "token",
+ "lawmatics": "credentials",
+ "leadfeeder": "token",
+ "leantime": "credentials",
+ "learnworlds": "credentials",
+ "ledyer": "credentials",
+ "leexi": "credentials",
+ "lemlist": "token",
+ "lemon-squeezy": "token",
+ "less-annoying-crm": "token",
+ "lettr-labs": "credentials",
+ "lever": "token",
+ "lexware-office": "token",
+ "liana": "credentials",
+ "lightdash": "credentials",
+ "lightfield": "token",
+ "lightspeed-retail": "credentials",
+ "linear": "oauth",
+ "linearb": "token",
+ "lingo-dev": "token",
+ "linkedin-ads": "oauth",
+ "linkedin-pages": "credentials",
+ "linkrunner": "token",
+ "linnworks": "credentials",
+ "linode": "token",
+ "listmonk": "credentials",
+ "lite-llm": "credentials",
+ "liveblocks": "credentials",
+ "livestorm": "token",
+ "llama-cloud": "credentials",
+ "lob": "token",
+ "lodgify": "token",
+ "logicmonitor": "credentials",
+ "logrocket": "credentials",
+ "logz-io": "credentials",
+ "lokalise": "token",
+ "looker": "credentials",
+ "loom": "credentials",
+ "loop-returns": "token",
+ "loops": "token",
+ "lovable": "token",
+ "luma": "token",
+ "m3ter": "credentials",
+ "mailchimp": "token",
+ "mailer-lite": "token",
+ "mailer-send": "token",
+ "mailgun": "credentials",
+ "mailjet": "credentials",
+ "mailosaur": "token",
+ "mailtrap": "token",
+ "mantle": "credentials",
+ "manychat": "credentials",
+ "marketo": "credentials",
+ "marketstack": "token",
+ "mastodon": "credentials",
+ "matomo": "credentials",
+ "maxio": "credentials",
+ "medusa": "credentials",
+ "meetup": "credentials",
+ "megaphone": "credentials",
+ "meltwater": "credentials",
+ "mem0": "credentials",
+ "memberful": "credentials",
+ "membrain": "credentials",
+ "mendeley": "credentials",
+ "mention": "token",
+ "mercado-ads": "credentials",
+ "mercado-pago": "token",
+ "mercury": "token",
+ "merge": "credentials",
+ "meta-ads": "oauth",
+ "metabase": "credentials",
+ "metaplane": "token",
+ "meteostat": "token",
+ "metorial": "token",
+ "metricool": "credentials",
+ "metriport": "credentials",
+ "metronome": "token",
+ "mews": "credentials",
+ "mezmo": "credentials",
+ "microsoft-advertising": "credentials",
+ "microsoft-clarity": "token",
+ "microsoft-dataverse": "credentials",
+ "microsoft-defender-cloud-apps": "credentials",
+ "microsoft-defender-endpoint": "credentials",
+ "microsoft-defender-for-cloud": "credentials",
+ "microsoft-entra-id": "credentials",
+ "microsoft-excel": "credentials",
+ "microsoft-intune": "credentials",
+ "microsoft-lists": "credentials",
+ "microsoft-purview": "credentials",
+ "microsoft-purview-audit": "credentials",
+ "microsoft-sentinel": "credentials",
+ "microsoft-teams": "credentials",
+ "microsoft-teams-call-records": "credentials",
+ "microsoft365-usage-reports": "credentials",
+ "midtrans": "credentials",
+ "mighty-networks": "credentials",
+ "mindbody": "credentials",
+ "mintlify": "credentials",
+ "mirakl": "credentials",
+ "miro": "credentials",
+ "missive": "credentials",
+ "mistral-ai": "token",
+ "mix-max": "token",
+ "mixpanel": "credentials",
+ "mo-engage": "credentials",
+ "modal": "credentials",
+ "mode": "credentials",
+ "moesif": "credentials",
+ "mollie": "token",
+ "monaco": "credentials",
+ "monday": "token",
+ "moneybird": "credentials",
+ "mongodb": "credentials",
+ "mono": "token",
+ "monte-carlo": "credentials",
+ "moodle": "credentials",
+ "motherduck": "credentials",
+ "motion": "token",
+ "moxie": "credentials",
+ "msg91": "credentials",
+ "mssql": "credentials",
+ "mux": "credentials",
+ "my-hours": "token",
+ "mycase": "credentials",
+ "mysql": "credentials",
+ "n8n": "credentials",
+ "nager-date": "credentials",
+ "nango": "credentials",
+ "nasa": "credentials",
+ "nation-builder": "credentials",
+ "navan": "credentials",
+ "naver-ads": "credentials",
+ "nebius-ai": "token",
+ "neo4j": "credentials",
+ "neon": "credentials",
+ "neon-crm": "credentials",
+ "net-suite": "credentials",
+ "netlify": "token",
+ "new-relic": "credentials",
+ "new-york-times": "token",
+ "news-api": "token",
+ "news-data": "token",
+ "nexhealth": "credentials",
+ "nexiopay": "credentials",
+ "nextdoor-ads": "credentials",
+ "ninja-one-rmm": "credentials",
+ "no-crm": "credentials",
+ "noaa-cdo": "token",
+ "nobl9": "credentials",
+ "nolt": "credentials",
+ "nops": "credentials",
+ "northbeam": "credentials",
+ "northflank": "token",
+ "northpass-lms": "token",
+ "notion": "credentials",
+ "npm-registry": "credentials",
+ "nuget": "credentials",
+ "nuntly": "token",
+ "nutshell": "credentials",
+ "nylas": "credentials",
+ "octolens": "token",
+ "octopus-deploy": "credentials",
+ "odoo": "credentials",
+ "oecd": "credentials",
+ "okendo": "credentials",
+ "okta": "credentials",
+ "omni": "credentials",
+ "omnisend": "token",
+ "oncehub": "token",
+ "one-drive": "credentials",
+ "one-hundred-ms": "credentials",
+ "one-password": "credentials",
+ "one-signal": "credentials",
+ "oneleet": "credentials",
+ "onelogin": "credentials",
+ "onepagecrm": "credentials",
+ "onfleet": "token",
+ "open-ai": "credentials",
+ "open-ai-ads": "token",
+ "open-aq": "token",
+ "open-data-dc": "credentials",
+ "open-dental": "credentials",
+ "open-exchange-rates": "token",
+ "open-fda": "token",
+ "open-meteo": "token",
+ "open-router": "token",
+ "open-weather": "token",
+ "openalex": "token",
+ "opencorporates": "token",
+ "openfec": "credentials",
+ "opinion-stage": "token",
+ "opn-payments": "token",
+ "opsgenie": "credentials",
+ "opslevel": "credentials",
+ "optimizely": "token",
+ "opus-watch": "token",
+ "oracle": "credentials",
+ "oracle-ebs": "credentials",
+ "oracle-fusion": "credentials",
+ "orb": "token",
+ "orbit": "credentials",
+ "orca-security": "credentials",
+ "ortto": "credentials",
+ "otto-market": "credentials",
+ "oura": "token",
+ "outbrain": "credentials",
+ "outlook": "credentials",
+ "outreach": "credentials",
+ "oveit": "credentials",
+ "ownerrez": "credentials",
+ "pabbly-subscriptions-billing": "credentials",
+ "packagist": "credentials",
+ "paddle": "token",
+ "pagbank": "credentials",
+ "pager-duty": "token",
+ "panda-doc": "token",
+ "paperform": "token",
+ "papersign": "token",
+ "pardot": "credentials",
+ "particle": "credentials",
+ "partner-stack": "credentials",
+ "partnerize": "credentials",
+ "patreon": "credentials",
+ "pax8": "credentials",
+ "pay-fit": "token",
+ "pay-pal": "credentials",
+ "paychex": "credentials",
+ "paylocity": "credentials",
+ "paymob": "credentials",
+ "paymongo": "token",
+ "paystack": "token",
+ "peec-ai": "credentials",
+ "pendo": "credentials",
+ "pennylane": "credentials",
+ "perigon": "token",
+ "perk": "credentials",
+ "persist-iq": "token",
+ "persona": "token",
+ "personio": "credentials",
+ "pexels": "token",
+ "pg-analyze": "credentials",
+ "phonepe": "credentials",
+ "phyllo": "credentials",
+ "picqer": "credentials",
+ "pike13": "credentials",
+ "pinecone": "token",
+ "pingdom": "token",
+ "pingone": "credentials",
+ "pinterest-ads": "oauth",
+ "pinterest-organic": "credentials",
+ "pipedrive": "credentials",
+ "pipeliner": "credentials",
+ "pivotal-tracker": "credentials",
+ "piwik": "credentials",
+ "plaid": "credentials",
+ "plain": "token",
+ "plane": "credentials",
+ "planetscale-mysql": "credentials",
+ "planetscale-postgres": "credentials",
+ "planhat": "token",
+ "planning-center": "credentials",
+ "platform-sh": "credentials",
+ "plausible": "credentials",
+ "plivo": "credentials",
+ "plunk": "credentials",
+ "pluralsight-flow": "credentials",
+ "pocket": "credentials",
+ "podbean": "credentials",
+ "podium": "credentials",
+ "polar": "token",
+ "polygon": "credentials",
+ "polymarket": "credentials",
+ "poplar": "token",
+ "post-nord": "credentials",
+ "postgresql": "credentials",
+ "postmark": "token",
+ "postscript": "token",
+ "power-bi-admin": "credentials",
+ "practicepanther": "credentials",
+ "prefect-cloud": "credentials",
+ "preset": "credentials",
+ "presta-shop": "credentials",
+ "pretix": "credentials",
+ "primetric": "credentials",
+ "printavo": "credentials",
+ "printify": "token",
+ "procore": "credentials",
+ "productboard": "token",
+ "productiv": "credentials",
+ "productive": "credentials",
+ "profound": "token",
+ "prompt-watch": "credentials",
+ "prompting-company": "credentials",
+ "proofpoint-tap": "credentials",
+ "propertyware": "credentials",
+ "pubnub": "credentials",
+ "pulumi-cloud": "credentials",
+ "pylon": "token",
+ "pypi": "credentials",
+ "qdrant": "credentials",
+ "qonto": "credentials",
+ "qualaroo": "credentials",
+ "qualtrics": "credentials",
+ "qualys-vmdr": "credentials",
+ "quay": "credentials",
+ "quick-books": "credentials",
+ "quo": "token",
+ "railway": "token",
+ "railz": "credentials",
+ "rainforest-pay": "credentials",
+ "raisely": "credentials",
+ "raken": "credentials",
+ "rakuten-advertising": "credentials",
+ "ramp": "credentials",
+ "rapid7-insightvm": "credentials",
+ "raygun": "token",
+ "razorpay": "credentials",
+ "rb2b": "credentials",
+ "rd-station-marketing": "credentials",
+ "recall-ai": "credentials",
+ "recharge": "token",
+ "recreation": "token",
+ "recruitee": "credentials",
+ "recurly": "credentials",
+ "reddit": "credentials",
+ "reddit-ads": "oauth",
+ "redis": "credentials",
+ "redpanda-cloud": "credentials",
+ "referral-hero": "token",
+ "render": "credentials",
+ "rent-cast": "token",
+ "rent-manager": "credentials",
+ "repairshopr": "credentials",
+ "replicate": "token",
+ "reply-io": "token",
+ "resend": "token",
+ "retail-express": "credentials",
+ "retell-ai": "token",
+ "retently": "token",
+ "retreaver": "credentials",
+ "revenue-cat": "credentials",
+ "reverb": "token",
+ "revolut-merchant": "credentials",
+ "rewardful": "token",
+ "ring-central": "credentials",
+ "rippling": "token",
+ "rki-covid": "credentials",
+ "roark": "token",
+ "rocket-chat": "credentials",
+ "rocket-matter": "credentials",
+ "rocketlane": "token",
+ "rokt-ads": "credentials",
+ "rollbar": "token",
+ "rootly": "token",
+ "rss": "token",
+ "rubygems": "credentials",
+ "rudder-stack": "credentials",
+ "ruddr": "token",
+ "run-pod": "token",
+ "safety-culture": "token",
+ "sage-hr": "credentials",
+ "sage-intacct": "credentials",
+ "sailthru": "credentials",
+ "sales-loft": "token",
+ "salesflare": "token",
+ "salesforce": "oauth",
+ "salesforce-marketing-cloud": "credentials",
+ "salestrics": "credentials",
+ "sam-cart": "credentials",
+ "sanity": "credentials",
+ "sap-concur": "credentials",
+ "sap-erp": "credentials",
+ "sap-fieldglass": "credentials",
+ "sap-hana": "credentials",
+ "sap-success-factors": "credentials",
+ "savvy-cal": "token",
+ "scale-ai": "token",
+ "scaleway": "credentials",
+ "scalr": "credentials",
+ "scarf": "credentials",
+ "schematic": "token",
+ "scrunch": "token",
+ "search-ads360": "credentials",
+ "sec-edgar": "credentials",
+ "secoda": "token",
+ "secureframe": "credentials",
+ "segment": "credentials",
+ "select-star": "token",
+ "semantic-scholar": "token",
+ "semaphore": "credentials",
+ "semgrep": "token",
+ "semrush": "credentials",
+ "send-grid": "token",
+ "send-pulse": "credentials",
+ "sendowl": "credentials",
+ "senseforce": "credentials",
+ "sentinelone": "credentials",
+ "sentry": "credentials",
+ "sequenzy": "credentials",
+ "serpstat": "credentials",
+ "service-fusion": "credentials",
+ "service-now": "credentials",
+ "servicem8": "token",
+ "servicetitan": "credentials",
+ "servicetrade": "credentials",
+ "sevalla": "credentials",
+ "sevdesk": "token",
+ "seven-shifts": "credentials",
+ "sftp": "credentials",
+ "sharepoint": "credentials",
+ "sharetribe": "credentials",
+ "ship-station": "credentials",
+ "shipmail": "token",
+ "shippo": "token",
+ "shop-wired": "credentials",
+ "shopify": "credentials",
+ "shopware": "credentials",
+ "shortcut": "token",
+ "shortio": "token",
+ "shutterstock": "credentials",
+ "side-shift": "credentials",
+ "sig-noz": "credentials",
+ "sigma-computing": "credentials",
+ "sign-now": "credentials",
+ "sim": "credentials",
+ "sim-fin": "token",
+ "similarweb": "token",
+ "simon-data": "credentials",
+ "simple-cast": "token",
+ "simplesat": "token",
+ "simpro": "credentials",
+ "sinch": "credentials",
+ "singlestore": "token",
+ "singular": "token",
+ "site24x7": "credentials",
+ "skio": "token",
+ "skyvern": "credentials",
+ "slack": "token",
+ "slash": "credentials",
+ "sleekplan": "token",
+ "sleuth": "credentials",
+ "smaily": "credentials",
+ "smart-engage": "token",
+ "smartlead": "token",
+ "smartlook": "credentials",
+ "smartreach": "credentials",
+ "smartrecruiters": "credentials",
+ "smartsheet": "token",
+ "smartwaiver": "token",
+ "smokeball": "credentials",
+ "snapchat": "oauth",
+ "snovio": "credentials",
+ "snowflake": "credentials",
+ "snowplow": "credentials",
+ "snyk": "credentials",
+ "social-pilot": "credentials",
+ "soda-cloud": "credentials",
+ "solarwinds-service-desk": "credentials",
+ "sonar-cloud": "credentials",
+ "sonarqube": "credentials",
+ "sonatype-nexus": "credentials",
+ "soniox": "credentials",
+ "sourcegraph": "credentials",
+ "spacelift": "credentials",
+ "spark-post": "credentials",
+ "speedcurve": "token",
+ "split-io": "token",
+ "splunk-observability-cloud": "credentials",
+ "spot-io": "credentials",
+ "spotify-ads": "credentials",
+ "spotler-crm": "token",
+ "sprig": "token",
+ "sprinklr": "credentials",
+ "sprinto": "credentials",
+ "sprout-social": "credentials",
+ "sqlite": "credentials",
+ "squadcast": "credentials",
+ "square": "credentials",
+ "squarespace": "token",
+ "stack-adapt": "credentials",
+ "stack-overflow-for-teams": "credentials",
+ "starburst": "credentials",
+ "statsig": "token",
+ "statuscake": "token",
+ "statuspage": "token",
+ "steam": "token",
+ "stigg": "token",
+ "stock-data": "token",
+ "stockx": "credentials",
+ "strato": "token",
+ "strava": "credentials",
+ "stream-elements": "token",
+ "streamlabs": "credentials",
+ "stripe": "token",
+ "stytch": "credentials",
+ "substack": "credentials",
+ "sumo-logic": "credentials",
+ "sumsub": "credentials",
+ "supabase": "credentials",
+ "supermetrics": "credentials",
+ "superwall": "credentials",
+ "survey-monkey": "credentials",
+ "survey-sparrow": "credentials",
+ "survicate": "token",
+ "svix": "token",
+ "swan": "credentials",
+ "swarmia": "token",
+ "swonkie": "credentials",
+ "synthesia": "token",
+ "systeme": "token",
+ "taboola": "credentials",
+ "tackle-io": "credentials",
+ "tailscale": "credentials",
+ "talkdesk": "credentials",
+ "talkwalker": "credentials",
+ "tally": "token",
+ "tana": "credentials",
+ "tavus": "token",
+ "tawk-to": "credentials",
+ "teachable": "token",
+ "teamcity": "credentials",
+ "teamtailor": "token",
+ "teamup-fitness": "credentials",
+ "teamwork": "credentials",
+ "tebra": "credentials",
+ "telli": "token",
+ "telnyx": "token",
+ "tempo": "token",
+ "temporal-io": "credentials",
+ "tenable-vulnerability-management": "credentials",
+ "tenjin": "token",
+ "tera-box": "credentials",
+ "ternary": "credentials",
+ "terra-api": "credentials",
+ "terraform-cloud": "credentials",
+ "tessitura": "credentials",
+ "test-dino": "credentials",
+ "testomatio": "credentials",
+ "testrail": "credentials",
+ "thinkific": "credentials",
+ "thinkific-courses": "credentials",
+ "thoughtspot": "credentials",
+ "thousandeyes": "credentials",
+ "threads": "credentials",
+ "thrive-learning": "credentials",
+ "tick-tick": "credentials",
+ "ticket-tailor": "token",
+ "ticketmaster": "token",
+ "tiktok-ads": "oauth",
+ "tiktok-shop": "credentials",
+ "tile38": "credentials",
+ "timely": "credentials",
+ "tiny-erp": "credentials",
+ "tinybird": "credentials",
+ "tinyemail": "token",
+ "tipalti": "credentials",
+ "tmdb": "token",
+ "toast": "credentials",
+ "todoist": "token",
+ "together-ai": "token",
+ "toggl": "credentials",
+ "torii": "token",
+ "track-pms": "credentials",
+ "tracklution": "credentials",
+ "tradable-bits": "credentials",
+ "traffic-stars": "token",
+ "transistor": "token",
+ "travis-ci": "token",
+ "trello": "credentials",
+ "tremendous": "credentials",
+ "trigger-dev": "credentials",
+ "trino": "credentials",
+ "triple-whale": "credentials",
+ "trunk-io": "credentials",
+ "trust-pilot": "credentials",
+ "trustradius": "token",
+ "turso": "credentials",
+ "tv-maze": "credentials",
+ "twelve-data": "token",
+ "twelve-labs": "token",
+ "twenty": "credentials",
+ "twilio": "credentials",
+ "twitch": "credentials",
+ "twitter": "credentials",
+ "twitter-ads": "oauth",
+ "two-c2p": "credentials",
+ "tyntec-sms": "token",
+ "typeform": "credentials",
+ "typesense": "credentials",
+ "ubidots": "credentials",
+ "uk-companies-house": "credentials",
+ "uk-ons": "credentials",
+ "umami": "credentials",
+ "un-comtrade": "credentials",
+ "unity-ads": "credentials",
+ "unleash": "credentials",
+ "unstructured": "credentials",
+ "up-promote": "token",
+ "uploadcare": "credentials",
+ "upstash": "credentials",
+ "uptick": "credentials",
+ "uptimerobot": "token",
+ "us-bea": "token",
+ "us-bls": "credentials",
+ "us-census": "token",
+ "us-eia": "token",
+ "us-treasury-fiscal-data": "credentials",
+ "user-com": "credentials",
+ "userback": "credentials",
+ "usersnap": "credentials",
+ "uservoice": "credentials",
+ "vanta": "credentials",
+ "vantage": "token",
+ "vapi": "token",
+ "veeqo": "token",
+ "vellum": "token",
+ "vendr": "token",
+ "veracode": "credentials",
+ "vercel": "credentials",
+ "vespa": "credentials",
+ "vimeo": "token",
+ "virtuous": "credentials",
+ "visma-economic": "credentials",
+ "vitally": "credentials",
+ "vonage": "credentials",
+ "vturb": "credentials",
+ "vultr": "token",
+ "vwo": "credentials",
+ "waiteraid": "credentials",
+ "walmart-marketplace": "credentials",
+ "wasabi": "token",
+ "watchmode": "token",
+ "waydev": "token",
+ "wayfair": "credentials",
+ "webflow": "credentials",
+ "weights-and-biases": "credentials",
+ "whatsapp-business-management": "credentials",
+ "when-i-work": "credentials",
+ "whmcs": "credentials",
+ "who-gho": "credentials",
+ "whop": "credentials",
+ "wikipedia-pageviews": "credentials",
+ "windmill": "credentials",
+ "windsor-ai": "credentials",
+ "wise": "credentials",
+ "wispr-flow": "credentials",
+ "wistia": "credentials",
+ "wix": "credentials",
+ "wiz": "credentials",
+ "wompi": "credentials",
+ "woo-commerce": "credentials",
+ "woo-payments": "credentials",
+ "wordpress": "credentials",
+ "work-os": "token",
+ "workable": "credentials",
+ "workato": "credentials",
+ "workday": "credentials",
+ "workflowmax": "credentials",
+ "workiz": "token",
+ "workramp": "credentials",
+ "world-bank": "credentials",
+ "wps-office": "credentials",
+ "wrike": "credentials",
+ "writesonic": "credentials",
+ "wufoo": "credentials",
+ "xendit": "credentials",
+ "xero": "credentials",
+ "xmatters": "credentials",
+ "xsolla": "credentials",
+ "yahoo-finance": "credentials",
+ "yandex-metrica": "credentials",
+ "ynab": "token",
+ "yoco": "token",
+ "yotpo": "credentials",
+ "you-sign": "credentials",
+ "younium": "credentials",
+ "youtube-analytics": "oauth",
+ "youtube-data": "credentials",
+ "zalando-zdirect": "credentials",
+ "zap-sign": "credentials",
+ "zapier-supported-storage": "token",
+ "zellify": "credentials",
+ "zenchef": "credentials",
+ "zendesk": "credentials",
+ "zendesk-sell": "token",
+ "zendesk-sunshine": "credentials",
+ "zenduty": "token",
+ "zenefits": "credentials",
+ "zenloop": "token",
+ "zep": "token",
+ "zero": "token",
+ "zitadel": "credentials",
+ "zluri": "credentials",
+ "zoho-analytics": "credentials",
+ "zoho-bigin": "credentials",
+ "zoho-billing": "credentials",
+ "zoho-books": "credentials",
+ "zoho-campaign": "credentials",
+ "zoho-crm": "credentials",
+ "zoho-desk": "credentials",
+ "zoho-expense": "credentials",
+ "zoho-inventory": "credentials",
+ "zoho-invoice": "credentials",
+ "zonka-feedback": "credentials",
+ "zoom": "credentials",
+ "zoom-info": "credentials",
+ "zuora": "credentials",
+ "zylo": "credentials",
+}
diff --git a/products/warehouse_sources/backend/facade/credential_kinds.py b/products/warehouse_sources/backend/facade/credential_kinds.py
index a14adde5c75f..f01e59362cbf 100644
--- a/products/warehouse_sources/backend/facade/credential_kinds.py
+++ b/products/warehouse_sources/backend/facade/credential_kinds.py
@@ -2,7 +2,7 @@
The `Integration` kind that holds the typed-in credentials of each warehouse source type, keyed by
source type value. A kind is stored data, so a change to an existing value needs a data migration.
-See `sources/common/credential_kind.py` for the derivation.
+The file is append-only: a removed source keeps its entry. See `sources/common/credential_kind.py`.
"""
from collections.abc import Mapping
diff --git a/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py
index 32a01901d0d5..7a55231d6f8b 100644
--- a/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py
+++ b/products/warehouse_sources/backend/management/commands/generate_source_field_classification.py
@@ -1,9 +1,14 @@
from typing import Any
-from django.core.management.base import BaseCommand
+from django.core.management.base import BaseCommand, CommandError, CommandParser
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import (
+ CREDENTIAL_AUTH_TYPES_PATH,
+ render_credential_auth_types,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (
CREDENTIAL_KINDS_PATH,
+ ChangedFrozenValueError,
render_credential_kinds,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
@@ -16,11 +21,30 @@ class Command(BaseCommand):
help = (
"Write the audit table of which job_inputs keys of each warehouse source are connection fields, "
"references, selectors, sync settings or runtime state, and write the frozen credential kind of "
- "each source. A test fails when either committed file is out of date."
+ "each source and the default auth type of each kind. A test fails when a committed file is out of date."
)
+ def add_arguments(self, parser: CommandParser) -> None:
+ parser.add_argument(
+ "--allow-changed-defaults",
+ action="store_true",
+ help=(
+ "Write a changed credential kind or default auth type for an existing entry. Integration rows "
+ "store these values, so the change needs a data migration."
+ ),
+ )
+
def handle(self, *args: Any, **options: Any) -> None:
- CREDENTIAL_KINDS_PATH.write_text(render_credential_kinds())
+ allow_changed = options["allow_changed_defaults"]
+ try:
+ # Render both before writing either, so a refused change writes nothing.
+ kinds = render_credential_kinds(allow_changed=allow_changed)
+ auth_types = render_credential_auth_types(allow_changed=allow_changed)
+ except ChangedFrozenValueError as error:
+ raise CommandError(str(error)) from error
+ CREDENTIAL_KINDS_PATH.write_text(kinds)
self.stdout.write(self.style.SUCCESS(f"Wrote {CREDENTIAL_KINDS_PATH}"))
+ CREDENTIAL_AUTH_TYPES_PATH.write_text(auth_types)
+ self.stdout.write(self.style.SUCCESS(f"Wrote {CREDENTIAL_AUTH_TYPES_PATH}"))
CLASSIFICATION_AUDIT_PATH.write_text(render_classification_audit())
self.stdout.write(self.style.SUCCESS(f"Wrote {CLASSIFICATION_AUDIT_PATH}"))
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py
index 80aff19b1aaa..f8bacacd1d26 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/amazon_s3/source.py
@@ -63,6 +63,10 @@ class AmazonS3Source(ObjectStorageSource[AmazonS3SourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.AMAZONS3
+ @property
+ def credential_kind(self) -> str | None:
+ return "aws-s3"
+
@property
def connection_host_fields(self) -> list[str]:
# The URL picks the bucket and region the stored key is sent to, so changing it must re-require the key.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
index 9c46e424a4a5..1a53bcf88c61 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/apple_search_ads/source.py
@@ -90,6 +90,10 @@ class AppleSearchAdsSource(
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.APPLESEARCHADS
+ @property
+ def credential_kind(self) -> str | None:
+ return "apple-ads"
+
@property
def connection_host_fields(self) -> list[str]:
# The stored private key is sent against whichever ad account (Platform API) or
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py
index 1d78cb197a39..4834058fb512 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_documentdb/source.py
@@ -39,6 +39,11 @@ class AwsDocumentDBSource(MongoDBSource):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.AWSDOCUMENTDB
+ @property
+ def credential_kind(self) -> str | None:
+ # Without this, the source inherits the kind of `MongoDBSource`.
+ return "aws-documentdb"
+
@staticmethod
def _adjust_field(field: FieldType) -> FieldType:
if isinstance(field, SourceFieldInputConfig) and field.name == "connection_string":
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
index 4325d596e00d..b4d1ea2e88b3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/bigquery/source.py
@@ -251,6 +251,10 @@ class BigQuerySource(SQLSource[BigQuerySourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.BIGQUERY
+ @property
+ def credential_kind(self) -> str | None:
+ return "google-cloud-service-account"
+
def get_retryable_errors(self) -> set[str]:
# The client-side limits on a query job and on the wait for one page of rows. BigQuery was
# reachable and then gave no result in time, so a later attempt can succeed.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py
index 5f05bca1e1c9..4e440a52251a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clickhouse/source.py
@@ -240,6 +240,10 @@ class ClickHouseSource(SimpleSource[ClickHouseSourceConfig], SSHTunnelMixin, Val
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.CLICKHOUSE
+ @property
+ def credential_kind(self) -> str | None:
+ return "clickhouse"
+
def _bypass_env_proxy(self, config: ClickHouseSourceConfig, team_id: int) -> BypassEnvProxy:
"""Why this connection may skip the egress proxy, or None to stay proxied.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py
index 41b9ee0dee19..837fcac97747 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/clickup/source.py
@@ -49,6 +49,10 @@ class ClickUpSource(ResumableSource[ClickUpSourceConfig, ClickUpResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.CLICKUP
+ @property
+ def credential_kind(self) -> str | None:
+ return "clickup"
+
@property
def connection_host_fields(self) -> list[str]:
# `workspace_id` selects which ClickUp workspace the stored API token is used against.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py
index 2ad35e70a75f..4b42d552a54c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cockroachdb/source.py
@@ -15,6 +15,10 @@ class CockroachDBSource(SimpleSource[CockroachDBSourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.COCKROACHDB
+ @property
+ def credential_kind(self) -> str | None:
+ return "cockroachdb"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
index 8c9a9b92f2a0..7af300b187c6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/base.py
@@ -33,6 +33,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can
CanonicalDescriptions,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.config import Config
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (
+ default_credential_kind,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import SourceSchema
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import (
@@ -518,6 +521,29 @@ class _BaseSource(ABC, Generic[ConfigType]):
a differently named field (e.g. Okta's ``okta_domain``) should list it here."""
return []
+ @property
+ def credential_kind(self) -> str | None:
+ """The ``Integration`` kind that holds this source's typed-in credentials, or None when the source has none.
+
+ The default is the kind of the source's OAuth field, else the kebab-case source type. Override it when the
+ source has the credential shape of an existing kind (Postgres is ``postgresql``), or when kebab-case splits
+ a brand name (MySQL is ``mysql``). The kind is stored data and `facade/credential_kinds.py` freezes it, so
+ never change the kind of a released source. A subclass inherits an override, so it must set its own kind."""
+ return default_credential_kind(self.source_type, self.get_source_config)
+
+ @property
+ def credential_auth_type(self) -> str:
+ """The default auth type (``oauth``, ``token`` or ``credentials``) of rows of ``credential_kind``.
+
+ The default comes from the connection fields. It is stored data and `facade/credential_auth_types.py`
+ freezes it, so never change it for a released source."""
+ # The derivation module imports this module.
+ from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import ( # noqa: PLC0415
+ derive_credential_auth_type,
+ )
+
+ return derive_credential_auth_type(self.get_source_config)
+
def is_unreachable_validation_error(self, error: str) -> bool:
"""Whether a ``validate_credentials`` message means only that the host could not be reached
over the network. The update serializer may then save the change with a warning for a team
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.py
new file mode 100644
index 000000000000..5a5cb5ff7aea
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_auth_type.py
@@ -0,0 +1,119 @@
+"""The default auth type of each warehouse source credential kind.
+
+A row of a credential kind stores its auth type in `config["auth_type"]` only when it is not the default
+of the kind, so the default is stored data. The result is frozen in the generated file
+`facade/credential_auth_types.py`, and the warehouse product registers it with the integrations SDK at
+startup. `generate_source_field_classification` writes that file, and a test fails when it is out of date.
+
+Each source gives the auth type of its kind with `credential_auth_type` on its class. The default,
+`derive_credential_auth_type`, comes from the source's connection fields, the same split as
+`job_input_classification`:
+
+- `oauth`: the source has an OAuth field and no typed-in connection field.
+- `token`: the source has exactly one typed-in field, and it is a secret.
+- `credentials`: every other source.
+"""
+
+from collections.abc import Iterator
+from pathlib import Path
+
+from products.warehouse_sources.backend.facade.credential_auth_types import CREDENTIAL_AUTH_TYPES
+from products.warehouse_sources.backend.facade.source_config import (
+ SourceConfig,
+ SourceFieldInputConfig,
+ SourceFieldSelectConfig,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import merge_append_only
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.job_input_classification import (
+ JobInputKind,
+ field_kind,
+ reference_option_values,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+
+CREDENTIAL_AUTH_TYPES_PATH = Path(__file__).resolve().parents[4] / "facade" / "credential_auth_types.py"
+
+OAUTH = "oauth"
+TOKEN = "token"
+CREDENTIALS = "credentials"
+
+
+class _Unclear(Exception):
+ pass
+
+
+def _typed_in_fields(field: FieldType) -> Iterator[FieldType]:
+ """The fields a user types in for one connection field. Raises `_Unclear` when the shape has a choice."""
+ if isinstance(field, SourceFieldSelectConfig):
+ typed_in_options = [
+ option for option in field.options if option.fields and option.value not in reference_option_values(field)
+ ]
+ if not any(option.fields for option in field.options):
+ # A plain choice, such as a region. It is a typed-in field itself.
+ yield field
+ elif len(typed_in_options) == 1:
+ for nested in typed_in_options[0].fields or []:
+ yield from _typed_in_fields(nested)
+ elif typed_in_options:
+ # Several typed-in shapes, such as an API key or a key pair.
+ raise _Unclear
+ return
+ yield field
+
+
+def derive_credential_auth_type(source_config: SourceConfig) -> str:
+ fields = list(source_config.fields)
+ has_oauth = any(field_kind(field) == JobInputKind.REFERENCE or reference_option_values(field) for field in fields)
+ try:
+ typed_in = [
+ leaf for field in fields if field_kind(field) == JobInputKind.CONNECTION for leaf in _typed_in_fields(field)
+ ]
+ except _Unclear:
+ return CREDENTIALS
+ if not typed_in:
+ return OAUTH if has_oauth else CREDENTIALS
+ if len(typed_in) == 1 and isinstance(typed_in[0], SourceFieldInputConfig) and typed_in[0].secret:
+ return TOKEN
+ return CREDENTIALS
+
+
+def compute_credential_auth_types() -> dict[str, str]:
+ """The default auth type of each credential kind, keyed by kind."""
+ auth_types: dict[str, str] = {}
+ for source in SourceRegistry.get_all_sources().values():
+ kind = source.credential_kind
+ if kind is not None:
+ auth_types[kind] = source.credential_auth_type
+ return dict(sorted(auth_types.items()))
+
+
+def frozen_credential_auth_types(*, allow_changed: bool = False) -> dict[str, str]:
+ """The committed default auth types plus the ones of new kinds. See `merge_append_only`."""
+ return merge_append_only(
+ CREDENTIAL_AUTH_TYPES,
+ compute_credential_auth_types(),
+ name="default auth types",
+ allow_changed=allow_changed,
+ )
+
+
+def render_credential_auth_types(*, allow_changed: bool = False) -> str:
+ lines = [
+ '"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.',
+ "",
+ "The default auth type of each warehouse source credential kind, keyed by kind. A row stores its auth",
+ "type only when it is not the default, so a change to an existing value needs a data migration. See",
+ "`sources/common/credential_auth_type.py` for the derivation. A kind in the integrations catalogue, such",
+ "as `stripe`, keeps the default of the catalogue. The file is append-only: a removed kind keeps its",
+ "entry.",
+ '"""',
+ "",
+ "from collections.abc import Mapping",
+ "",
+ "CREDENTIAL_AUTH_TYPES: Mapping[str, str] = {",
+ ]
+ auth_types = frozen_credential_auth_types(allow_changed=allow_changed)
+ lines += [f' "{kind}": "{auth_type}",' for kind, auth_type in auth_types.items()]
+ lines += ["}", ""]
+ return "\n".join(lines)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py
index 000fed4e47e7..2c5a35bde14c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/credential_kind.py
@@ -1,16 +1,21 @@
"""The `Integration` kind that holds a warehouse source's typed-in credentials.
-The kind comes from `derive_credential_kind`. The result is frozen in the generated file
-`facade/credential_kinds.py`, because a kind is stored data: renaming a kind after the first row
-exists needs a data migration. `generate_source_field_classification` writes that file, and a test
-fails when it is out of date.
+Each source gives its kind with `credential_kind` on its class. The default comes from
+`default_credential_kind`. The result is frozen in the generated file `facade/credential_kinds.py`,
+because a kind is stored data: renaming a kind after the first row exists needs a data migration.
+`generate_source_field_classification` writes that file, and a test fails when it is out of date.
+
+The generated files are append-only: a new source adds its entry, a removed source keeps its entry,
+and a changed value fails the generation unless `--allow-changed-defaults` is passed.
"""
import re
from collections.abc import Iterator, Mapping
from pathlib import Path
+from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS
from products.warehouse_sources.backend.facade.source_config import (
+ SourceConfig,
SourceFieldConfig,
SourceFieldOauthConfig,
SourceFieldSelectConfig,
@@ -24,43 +29,6 @@ CREDENTIAL_KINDS_PATH = Path(__file__).resolve().parents[4] / "facade" / "creden
# `Integration.kind` is `CharField(max_length=32)`.
CREDENTIAL_KIND_MAX_LENGTH = 32
-# Sources that have no credential kind. Custom keeps its own `job_inputs` and its OAuth rows are not `Integration` rows.
-CREDENTIAL_KIND_EXCLUDED: frozenset[ExternalDataSourceType] = frozenset({ExternalDataSourceType.CUSTOM})
-
-# A source whose credentials have the same shape as an existing `Integration` kind uses that kind.
-EXISTING_KIND_OVERRIDES: Mapping[ExternalDataSourceType, str] = {
- ExternalDataSourceType.POSTGRES: "postgresql",
- ExternalDataSourceType.REDSHIFT: "aws-redshift",
- ExternalDataSourceType.AMAZONS3: "aws-s3",
- ExternalDataSourceType.BIGQUERY: "google-cloud-service-account",
- ExternalDataSourceType.GITLAB: "gitlab",
- ExternalDataSourceType.APPLESEARCHADS: "apple-ads",
- ExternalDataSourceType.CUSTOMERIO: "customer-io",
- ExternalDataSourceType.CLICKUP: "clickup",
-}
-
-# The kebab-case rule splits a one-word brand name at its inner capitals. These names keep the brand spelling.
-BRAND_CASING_OVERRIDES: Mapping[ExternalDataSourceType, str] = {
- ExternalDataSourceType.MYSQL: "mysql",
- ExternalDataSourceType.MONGODB: "mongodb",
- ExternalDataSourceType.CLICKHOUSE: "clickhouse",
- ExternalDataSourceType.SHAREPOINT: "sharepoint",
- ExternalDataSourceType.TIKTOKADS: "tiktok-ads",
- ExternalDataSourceType.YOUTUBEANALYTICS: "youtube-analytics",
- ExternalDataSourceType.DYNAMODB: "dynamodb",
- ExternalDataSourceType.COCKROACHDB: "cockroachdb",
- ExternalDataSourceType.COSMOSDB: "cosmosdb",
- ExternalDataSourceType.PLANETSCALEMYSQL: "planetscale-mysql",
- ExternalDataSourceType.PLANETSCALEPOSTGRES: "planetscale-postgres",
- ExternalDataSourceType.AWSDOCUMENTDB: "aws-documentdb",
- ExternalDataSourceType.SQLITE: "sqlite",
- ExternalDataSourceType.GNEWS: "gnews",
- ExternalDataSourceType.PYPI: "pypi",
- ExternalDataSourceType.TMDB: "tmdb",
- ExternalDataSourceType.RB2B: "rb2b",
- ExternalDataSourceType.E2B: "e2b",
-}
-
_WORD_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])|(?<=[A-Z])(?=[A-Z][a-z])")
@@ -68,17 +36,6 @@ def kebab_case(name: str) -> str:
return _WORD_BOUNDARY.sub("-", name).lower()
-def derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind: str | None = None) -> str:
- """The kind for a source. A source with an OAuth field uses the kind that field names, because its rows exist already."""
- if oauth_kind:
- return oauth_kind
- if source_type in EXISTING_KIND_OVERRIDES:
- return EXISTING_KIND_OVERRIDES[source_type]
- if source_type in BRAND_CASING_OVERRIDES:
- return BRAND_CASING_OVERRIDES[source_type]
- return kebab_case(source_type.value)
-
-
def _walk(fields: list[SourceFieldConfig]) -> Iterator[SourceFieldConfig]:
for field in fields:
yield field
@@ -89,36 +46,73 @@ def _walk(fields: list[SourceFieldConfig]) -> Iterator[SourceFieldConfig]:
yield from _walk(option.fields or [])
-def oauth_kind_of(source_type: ExternalDataSourceType) -> str | None:
+def oauth_kind_of(source_config: SourceConfig) -> str | None:
"""The `Integration` kind of the source's OAuth field, at any depth. None when the source has no such field."""
- for field in _walk(list(SourceRegistry.get_source(source_type).get_source_config.fields)):
+ for field in _walk(list(source_config.fields)):
if isinstance(field, SourceFieldOauthConfig):
return field.kind
return None
+def default_credential_kind(source_type: ExternalDataSourceType, source_config: SourceConfig) -> str:
+ """A source with an OAuth field uses the kind that field names, because its rows exist already."""
+ return oauth_kind_of(source_config) or kebab_case(source_type.value)
+
+
+class ChangedFrozenValueError(ValueError):
+ """The generation would change a value that rows already store."""
+
+
+def merge_append_only(
+ committed: Mapping[str, str], computed: Mapping[str, str], *, name: str, allow_changed: bool = False
+) -> dict[str, str]:
+ """`committed` with the new keys of `computed`. A key that is gone from `computed` stays.
+
+ Raises `ChangedFrozenValueError` when `computed` changes a committed value, unless `allow_changed`.
+ """
+ changed = {
+ key: (committed[key], value) for key, value in computed.items() if key in committed and committed[key] != value
+ }
+ if changed and not allow_changed:
+ lines = "\n".join(f" {key}: {old!r} -> {new!r}" for key, (old, new) in sorted(changed.items()))
+ raise ChangedFrozenValueError(
+ f"The generation changes committed {name}, which integration rows store:\n{lines}\n"
+ "Keep the old value in the source class, or write a data migration for the stored rows and "
+ "rerun with --allow-changed-defaults."
+ )
+ return dict(sorted({**committed, **computed}.items()))
+
+
def compute_credential_kinds() -> dict[str, str]:
sources = SourceRegistry.get_all_sources()
- return {
- source_type.value: derive_credential_kind(source_type, oauth_kind_of(source_type))
+ kinds = {
+ source_type.value: sources[source_type].credential_kind
for source_type in sorted(sources, key=lambda source_type: source_type.value)
- if source_type not in CREDENTIAL_KIND_EXCLUDED
}
+ return {source_type: kind for source_type, kind in kinds.items() if kind is not None}
+
+
+def frozen_credential_kinds(*, allow_changed: bool = False) -> dict[str, str]:
+ """The committed kinds plus the kinds of new sources. See `merge_append_only`."""
+ return merge_append_only(
+ CREDENTIAL_KINDS, compute_credential_kinds(), name="credential kinds", allow_changed=allow_changed
+ )
-def render_credential_kinds() -> str:
+def render_credential_kinds(*, allow_changed: bool = False) -> str:
lines = [
'"""Generated by `python manage.py generate_source_field_classification`. Do not edit by hand.',
"",
"The `Integration` kind that holds the typed-in credentials of each warehouse source type, keyed by",
"source type value. A kind is stored data, so a change to an existing value needs a data migration.",
- "See `sources/common/credential_kind.py` for the derivation.",
+ "The file is append-only: a removed source keeps its entry. See `sources/common/credential_kind.py`.",
'"""',
"",
"from collections.abc import Mapping",
"",
"CREDENTIAL_KINDS: Mapping[str, str] = {",
]
- lines += [f' "{source_type}": "{kind}",' for source_type, kind in compute_credential_kinds().items()]
+ kinds = frozen_credential_kinds(allow_changed=allow_changed)
+ lines += [f' "{source_type}": "{kind}",' for source_type, kind in kinds.items()]
lines += ["}", ""]
return "\n".join(lines)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.py
new file mode 100644
index 000000000000..f863a30b74e3
--- /dev/null
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_auth_type.py
@@ -0,0 +1,92 @@
+from pathlib import Path
+
+import pytest
+from unittest.mock import PropertyMock, patch
+
+from django.core.management import CommandError, call_command
+
+import products.warehouse_sources.backend.temporal.data_imports.sources._load_all # noqa: F401
+from products.warehouse_sources.backend.facade.credential_auth_types import CREDENTIAL_AUTH_TYPES
+from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS
+from products.warehouse_sources.backend.management.commands import (
+ generate_source_field_classification as command_module,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_auth_type import (
+ CREDENTIAL_AUTH_TYPES_PATH,
+ compute_credential_auth_types,
+ derive_credential_auth_type,
+ frozen_credential_auth_types,
+ render_credential_auth_types,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (
+ ChangedFrozenValueError,
+)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.registry import SourceRegistry
+from products.warehouse_sources.backend.types import ExternalDataSourceType
+
+
+ "source_type,expected",
+ [
+ # One secret input.
+ (ExternalDataSourceType.ABLY, "token"),
+ (ExternalDataSourceType.KLAVIYO, "token"),
+ # An auth method select with an OAuth option and one typed-in option that holds one secret.
+ (ExternalDataSourceType.STRIPE, "token"),
+ (ExternalDataSourceType.GITHUB, "token"),
+ # Only an OAuth field.
+ (ExternalDataSourceType.HUBSPOT, "oauth"),
+ (ExternalDataSourceType.GOOGLEADS, "oauth"),
+ # Several typed-in fields.
+ (ExternalDataSourceType.POSTGRES, "credentials"),
+ (ExternalDataSourceType.MYSQL, "credentials"),
+ (ExternalDataSourceType.ZENDESK, "credentials"),
+ ],
+)
+def test_credential_auth_type(source_type: ExternalDataSourceType, expected: str) -> None:
+ source = SourceRegistry.get_source(source_type)
+
+ assert source.credential_auth_type == expected
+ assert derive_credential_auth_type(source.get_source_config) == expected
+
+
+def test_compute_reads_the_auth_type_from_the_source_class() -> None:
+ zendesk = type(SourceRegistry.get_source(ExternalDataSourceType.ZENDESK))
+ with patch.object(zendesk, "credential_auth_type", new_callable=PropertyMock, return_value="token"):
+ auth_types = compute_credential_auth_types()
+
+ assert auth_types[CREDENTIAL_KINDS[ExternalDataSourceType.ZENDESK.value]] == "token"
+
+
+def test_every_credential_kind_has_an_auth_type() -> None:
+ # A kind that a source stopped using keeps its entry, because rows of the kind can exist.
+ assert set(CREDENTIAL_KINDS.values()) <= set(CREDENTIAL_AUTH_TYPES)
+ assert set(CREDENTIAL_AUTH_TYPES.values()) <= {"oauth", "token", "credentials"}
+
+
+def test_a_changed_default_auth_type_fails_the_generation(tmp_path: Path) -> None:
+ zendesk = type(SourceRegistry.get_source(ExternalDataSourceType.ZENDESK))
+ kind = CREDENTIAL_KINDS[ExternalDataSourceType.ZENDESK.value]
+ with (
+ patch.object(zendesk, "credential_auth_type", new_callable=PropertyMock, return_value="token"),
+ patch.object(command_module, "CREDENTIAL_KINDS_PATH", tmp_path / "kinds.py"),
+ patch.object(command_module, "CREDENTIAL_AUTH_TYPES_PATH", tmp_path / "auth_types.py"),
+ patch.object(command_module, "CLASSIFICATION_AUDIT_PATH", tmp_path / "audit.csv"),
+ ):
+ with pytest.raises(ChangedFrozenValueError, match=f"{kind}: 'credentials' -> 'token'"):
+ render_credential_auth_types()
+ with pytest.raises(CommandError):
+ call_command("generate_source_field_classification")
+ written_after_refusal = sorted(path.name for path in tmp_path.iterdir())
+ call_command("generate_source_field_classification", "--allow-changed-defaults")
+
+ assert written_after_refusal == []
+ assert f' "{kind}": "token",' in (tmp_path / "auth_types.py").read_text()
+
+
+def test_credential_auth_types_file_is_up_to_date() -> None:
+ assert dict(CREDENTIAL_AUTH_TYPES) == frozen_credential_auth_types()
+ assert CREDENTIAL_AUTH_TYPES_PATH.read_text() == render_credential_auth_types(), (
+ "The credential auth types are out of date. "
+ "Run `python manage.py generate_source_field_classification` and commit the result."
+ )
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py
index 1d16551de311..0b99c0b9954d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/common/test/test_credential_kind.py
@@ -1,14 +1,17 @@
import pytest
+from unittest.mock import PropertyMock, patch
import products.warehouse_sources.backend.temporal.data_imports.sources._load_all # noqa: F401
from products.warehouse_sources.backend.facade.credential_kinds import CREDENTIAL_KINDS
from products.warehouse_sources.backend.temporal.data_imports.sources.common.credential_kind import (
- CREDENTIAL_KIND_EXCLUDED,
CREDENTIAL_KIND_MAX_LENGTH,
CREDENTIAL_KINDS_PATH,
+ ChangedFrozenValueError,
compute_credential_kinds,
- derive_credential_kind,
+ default_credential_kind,
+ frozen_credential_kinds,
kebab_case,
+ merge_append_only,
oauth_kind_of,
render_credential_kinds,
)
@@ -17,6 +20,7 @@ from products.warehouse_sources.backend.types import ExternalDataSourceType
ALL_SOURCES = SourceRegistry.get_all_sources()
SOURCE_TYPES = sorted(ALL_SOURCES.keys(), key=str)
+SOURCE_TYPES_WITH_KIND = [s for s in SOURCE_TYPES if ALL_SOURCES[s].credential_kind is not None]
@pytest.mark.parametrize(
@@ -35,31 +39,49 @@ def test_kebab_case(name: str, expected: str) -> None:
@pytest.mark.parametrize(
- "source_type,oauth_kind,expected",
+ "source_type,expected",
+ [
+ # Sources that set their kind in their own class.
+ (ExternalDataSourceType.POSTGRES, "postgresql"),
+ (ExternalDataSourceType.REDSHIFT, "aws-redshift"),
+ (ExternalDataSourceType.AMAZONS3, "aws-s3"),
+ (ExternalDataSourceType.CUSTOMERIO, "customer-io"),
+ (ExternalDataSourceType.MYSQL, "mysql"),
+ (ExternalDataSourceType.TIKTOKADS, "tiktok-ads"),
+ # Subclasses of an overriding source set their own kind.
+ (ExternalDataSourceType.NEON, "neon"),
+ (ExternalDataSourceType.SUPABASE, "supabase"),
+ (ExternalDataSourceType.PLANETSCALEMYSQL, "planetscale-mysql"),
+ (ExternalDataSourceType.AWSDOCUMENTDB, "aws-documentdb"),
+ (ExternalDataSourceType.CUSTOM, None),
+ # Sources that use the default.
+ (ExternalDataSourceType.GITHUB, "github"),
+ (ExternalDataSourceType.STRIPE, "stripe"),
+ (ExternalDataSourceType.GOOGLECLOUDSTORAGE, "google-cloud-storage"),
+ (ExternalDataSourceType.AWSCLOUDTRAIL, "aws-cloud-trail"),
+ (ExternalDataSourceType.SNAPCHATADS, "snapchat"),
+ (ExternalDataSourceType.GOOGLEADS, "google-ads"),
+ ],
+)
+def test_credential_kind(source_type: ExternalDataSourceType, expected: str | None) -> None:
+ assert ALL_SOURCES[source_type].credential_kind == expected
+
+
+ "source_type,expected",
[
- (ExternalDataSourceType.POSTGRES, None, "postgresql"),
- (ExternalDataSourceType.REDSHIFT, None, "aws-redshift"),
- (ExternalDataSourceType.AMAZONS3, None, "aws-s3"),
- (ExternalDataSourceType.BIGQUERY, None, "google-cloud-service-account"),
- (ExternalDataSourceType.GITLAB, None, "gitlab"),
- (ExternalDataSourceType.APPLESEARCHADS, None, "apple-ads"),
- (ExternalDataSourceType.CUSTOMERIO, None, "customer-io"),
- (ExternalDataSourceType.MYSQL, None, "mysql"),
- (ExternalDataSourceType.MONGODB, None, "mongodb"),
- (ExternalDataSourceType.CLICKHOUSE, None, "clickhouse"),
- (ExternalDataSourceType.SHAREPOINT, None, "sharepoint"),
- (ExternalDataSourceType.TIKTOKADS, None, "tiktok-ads"),
- (ExternalDataSourceType.GITHUB, None, "github"),
- (ExternalDataSourceType.STRIPE, None, "stripe"),
- (ExternalDataSourceType.GOOGLECLOUDSTORAGE, None, "google-cloud-storage"),
- (ExternalDataSourceType.AWSCLOUDTRAIL, None, "aws-cloud-trail"),
# The kind of the OAuth field wins, because those rows exist already.
- (ExternalDataSourceType.SNAPCHATADS, "snapchat", "snapchat"),
- (ExternalDataSourceType.POSTGRES, "custom-kind", "custom-kind"),
+ (ExternalDataSourceType.SNAPCHATADS, "snapchat"),
+ (ExternalDataSourceType.BIGQUERY, "google-cloud-service-account"),
+ (ExternalDataSourceType.YOUTUBEANALYTICS, "youtube-analytics"),
+ # Without an OAuth field, the kebab-case source type. These sources override it.
+ (ExternalDataSourceType.POSTGRES, "postgres"),
+ (ExternalDataSourceType.MYSQL, "my-sql"),
+ (ExternalDataSourceType.STRIPE, "stripe"),
],
)
-def test_derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind: str | None, expected: str) -> None:
- assert derive_credential_kind(source_type, oauth_kind) == expected
+def test_default_credential_kind(source_type: ExternalDataSourceType, expected: str) -> None:
+ assert default_credential_kind(source_type, ALL_SOURCES[source_type].get_source_config) == expected
@pytest.mark.parametrize(
@@ -74,15 +96,23 @@ def test_derive_credential_kind(source_type: ExternalDataSourceType, oauth_kind:
],
)
def test_oauth_kind_of(source_type: ExternalDataSourceType, expected: str | None) -> None:
- assert oauth_kind_of(source_type) == expected
+ assert oauth_kind_of(ALL_SOURCES[source_type].get_source_config) == expected
+
+
+def test_compute_reads_the_kind_from_the_source_class() -> None:
+ stripe = type(ALL_SOURCES[ExternalDataSourceType.STRIPE])
+ with patch.object(stripe, "credential_kind", new_callable=PropertyMock, return_value="stripe-test"):
+ kinds = compute_credential_kinds()
+
+ assert kinds[ExternalDataSourceType.STRIPE.value] == "stripe-test"
+ assert ExternalDataSourceType.CUSTOM.value not in kinds
def test_custom_source_has_no_credential_kind() -> None:
- assert ExternalDataSourceType.CUSTOM in CREDENTIAL_KIND_EXCLUDED
assert ExternalDataSourceType.CUSTOM.value not in CREDENTIAL_KINDS
[email protected]("source_type", [s for s in SOURCE_TYPES if s not in CREDENTIAL_KIND_EXCLUDED], ids=str)
[email protected]("source_type", SOURCE_TYPES_WITH_KIND, ids=str)
def test_every_source_has_a_valid_credential_kind(source_type: ExternalDataSourceType) -> None:
kind = CREDENTIAL_KINDS[source_type.value]
@@ -92,7 +122,7 @@ def test_every_source_has_a_valid_credential_kind(source_type: ExternalDataSourc
assert not kind.startswith("-") and not kind.endswith("-")
-def test_credential_kinds_are_unique_apart_from_shared_oauth_providers() -> None:
+def test_credential_kinds_are_unique() -> None:
sources_by_kind: dict[str, list[str]] = {}
for source_type, kind in CREDENTIAL_KINDS.items():
sources_by_kind.setdefault(kind, []).append(source_type)
@@ -101,8 +131,38 @@ def test_credential_kinds_are_unique_apart_from_shared_oauth_providers() -> None
assert not shared, f"Source types that share one credential kind: {shared}"
+ "_name,computed,allow_changed,expected",
+ [
+ ("unchanged", {"a": "x"}, False, {"a": "x"}),
+ ("new_key_is_added", {"a": "x", "b": "y"}, False, {"a": "x", "b": "y"}),
+ ("removed_key_stays", {}, False, {"a": "x"}),
+ ("changed_value_raises", {"a": "z"}, False, ChangedFrozenValueError),
+ ("changed_value_with_the_flag", {"a": "z"}, True, {"a": "z"}),
+ ],
+)
+def test_merge_append_only(
+ _name: str, computed: dict[str, str], allow_changed: bool, expected: dict[str, str] | type[Exception]
+) -> None:
+ if isinstance(expected, dict):
+ assert merge_append_only({"a": "x"}, computed, name="kinds", allow_changed=allow_changed) == expected
+ else:
+ with pytest.raises(expected, match="--allow-changed-defaults"):
+ merge_append_only({"a": "x"}, computed, name="kinds", allow_changed=allow_changed)
+
+
+def test_a_changed_kind_fails_the_generation() -> None:
+ stripe = type(ALL_SOURCES[ExternalDataSourceType.STRIPE])
+ with patch.object(stripe, "credential_kind", new_callable=PropertyMock, return_value="stripe-test"):
+ with pytest.raises(ChangedFrozenValueError, match="Stripe: 'stripe' -> 'stripe-test'"):
+ render_credential_kinds()
+ allowed = render_credential_kinds(allow_changed=True)
+
+ assert ' "Stripe": "stripe-test",' in allowed
+
+
def test_credential_kinds_file_is_up_to_date() -> None:
- assert dict(CREDENTIAL_KINDS) == compute_credential_kinds()
+ assert dict(CREDENTIAL_KINDS) == frozen_credential_kinds()
assert CREDENTIAL_KINDS_PATH.read_text() == render_credential_kinds(), (
"The frozen credential kinds are out of date. "
"Run `python manage.py generate_source_field_classification` and commit the result."
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py
index 9ca32cc8332f..3f93204e379b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cosmosdb/source.py
@@ -58,6 +58,10 @@ class CosmosDBSource(
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.COSMOSDB
+ @property
+ def credential_kind(self) -> str | None:
+ return "cosmosdb"
+
@property
def connection_host_fields(self) -> list[str]:
# The stored key signs every request to this endpoint, so retargeting it must re-require the key.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py
index 00f83a91846a..813fde6189e9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/custom/source.py
@@ -877,6 +877,11 @@ class CustomSource(SimpleSource[CustomSourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.CUSTOM
+ @property
+ def credential_kind(self) -> str | None:
+ # Custom keeps its own `job_inputs`, and its OAuth rows are not `Integration` rows.
+ return None
+
# Custom REST sources are capped per team to keep abuse of the arbitrary-URL fetcher bounded.
max_instances_per_team = MAX_CUSTOM_SOURCES_PER_TEAM
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py
index 198078975559..53f95c7e47a7 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/customer_io/source.py
@@ -172,6 +172,10 @@ class CustomerIOSource(
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.CUSTOMERIO
+ @property
+ def credential_kind(self) -> str | None:
+ return "customer-io"
+
@property
def webhook_template(self) -> Optional["HogFunctionTemplateDC"]:
from products.warehouse_sources.backend.temporal.data_imports.sources.customer_io.webhook_template import (
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py
index aa7ef44b3185..92d0be69a4be 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/dynamodb/source.py
@@ -55,6 +55,10 @@ class DynamoDBSource(ResumableSource[DynamoDBSourceConfig, DynamoDBResumeConfig]
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.DYNAMODB
+ @property
+ def credential_kind(self) -> str | None:
+ return "dynamodb"
+
@property
def connection_host_fields(self) -> list[str]:
# The region picks the endpoint the signed request — and so the customer's key — is sent to.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py
index c567b345a8cd..be95a7b17cf2 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/e2b/source.py
@@ -51,6 +51,10 @@ class E2BSource(ResumableSource[E2BSourceConfig, E2BResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.E2B
+ @property
+ def credential_kind(self) -> str | None:
+ return "e2b"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py
index 99da4da5b9d6..04554fae8b64 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gitlab/source.py
@@ -54,6 +54,10 @@ class GitLabSource(ResumableSource[GitLabSourceConfig, GitLabResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.GITLAB
+ @property
+ def credential_kind(self) -> str | None:
+ return "gitlab"
+
@property
def connection_host_fields(self) -> list[str]:
# `gitlab_host` is where the stored token is sent; retargeting it must re-require the token.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
index db1b8299891f..bd02258fca9b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/gnews/source.py
@@ -59,6 +59,10 @@ class GNewsSource(ResumableSource[GNewsSourceConfig, GNewsResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.GNEWS
+ @property
+ def credential_kind(self) -> str | None:
+ return "gnews"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py
index 1dd2cc9319a2..1f3ae6fa1a77 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mongodb/source.py
@@ -266,6 +266,10 @@ class MongoDBSource(ResumableSource[MongoDBSourceConfig, MongoResumeConfig], Val
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.MONGODB
+ @property
+ def credential_kind(self) -> str | None:
+ return "mongodb"
+
def get_non_retryable_errors(self) -> dict[str, str | None]:
auth_failed_msg = _MONGO_AUTHENTICATION_FAILED_MESSAGE
return {
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
index 35c9368c3e21..3cf5d485f219 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mysql/source.py
@@ -223,6 +223,10 @@ class MySQLSource(
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.MYSQL
+ @property
+ def credential_kind(self) -> str | None:
+ return "mysql"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py
index d037f174fd69..123352d2e90a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/neon/source.py
@@ -63,6 +63,11 @@ class NeonSource(PostgresSource):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.NEON
+ @property
+ def credential_kind(self) -> str | None:
+ # Without this, the source inherits the kind of `PostgresSource`.
+ return "neon"
+
@staticmethod
def _adjust_field(field: FieldType) -> FieldType:
if isinstance(field, SourceFieldInputConfig):
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
index 920d202ed67f..325b2fda8834 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_mysql/source.py
@@ -121,6 +121,11 @@ class PlanetScaleMySQLSource(MySQLSource):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.PLANETSCALEMYSQL
+ @property
+ def credential_kind(self) -> str | None:
+ # Without this, the source inherits the kind of `MySQLSource`.
+ return "planetscale-mysql"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py
index 570f79819e71..ff3da3ffda51 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/planetscale_postgres/source.py
@@ -89,6 +89,11 @@ class PlanetScalePostgresSource(PostgresSource):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.PLANETSCALEPOSTGRES
+ @property
+ def credential_kind(self) -> str | None:
+ # Without this, the source inherits the kind of `PostgresSource`.
+ return "planetscale-postgres"
+
@staticmethod
def _adjust_field(field: FieldType) -> FieldType:
if not isinstance(field, SourceFieldInputConfig):
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
index 32c2c5f370db..824bdbd752ca 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/postgres/source.py
@@ -741,6 +741,10 @@ class PostgresSource(
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.POSTGRES
+ @property
+ def credential_kind(self) -> str | None:
+ return "postgresql"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
index 9a1b8080f017..2f1de9ace308 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pypi/source.py
@@ -41,6 +41,10 @@ class PyPISource(SimpleSource[PyPISourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.PYPI
+ @property
+ def credential_kind(self) -> str | None:
+ return "pypi"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py
index aa0d9ee8a2e9..d67ffff50d91 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/rb2b/source.py
@@ -13,6 +13,10 @@ class RB2BSource(SimpleSource[RB2BSourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.RB2B
+ @property
+ def credential_kind(self) -> str | None:
+ return "rb2b"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
index 501b0970baab..1d9aac5af906 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/redshift/source.py
@@ -125,6 +125,10 @@ class RedshiftSource(SQLSource[RedshiftSourceConfig], SSHTunnelMixin, ValidateDa
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.REDSHIFT
+ @property
+ def credential_kind(self) -> str | None:
+ return "aws-redshift"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
index 4e603e3fd242..f022c2611d8e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sharepoint/source.py
@@ -82,6 +82,10 @@ class SharePointSource(ResumableSource[SharePointSourceConfig, SharePointResumeC
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.SHAREPOINT
+ @property
+ def credential_kind(self) -> str | None:
+ return "sharepoint"
+
def get_non_retryable_errors(self) -> dict[str, str | None]:
return {
INVALID_SITE_URL_ERROR: None,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py
index d62f01b69094..9996a898fd22 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sqlite/source.py
@@ -13,6 +13,10 @@ class SQLiteSource(SimpleSource[SQLiteSourceConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.SQLITE
+ @property
+ def credential_kind(self) -> str | None:
+ return "sqlite"
+
@property
def get_source_config(self) -> SourceConfig:
return SourceConfig(
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py
index ce020ce23628..4bcf4d6de126 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/supabase/source.py
@@ -235,6 +235,11 @@ class SupabaseSource(PostgresSource):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.SUPABASE
+ @property
+ def credential_kind(self) -> str | None:
+ # Without this, the source inherits the kind of `PostgresSource`.
+ return "supabase"
+
def get_non_retryable_errors(self) -> dict[str, str | None]:
return {
_SUPABASE_REALTIME_PARTITION_ERROR: _SUPABASE_REALTIME_PARTITION_MESSAGE,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py
index 41eac94152fd..e420180e913b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tiktok_ads/source.py
@@ -86,6 +86,10 @@ class TikTokAdsSource(ResumableSource[TikTokAdsSourceConfig, TikTokAdsResumeConf
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.TIKTOKADS
+ @property
+ def credential_kind(self) -> str | None:
+ return "tiktok-ads"
+
def get_canonical_descriptions(self) -> CanonicalDescriptions:
from products.warehouse_sources.backend.temporal.data_imports.sources.tiktok_ads.canonical_descriptions import (
CANONICAL_DESCRIPTIONS,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py
index ab657314a993..9aa5f7a3838b 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/tmdb/source.py
@@ -46,6 +46,10 @@ class TMDbSource(ResumableSource[TMDbSourceConfig, TMDbResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.TMDB
+ @property
+ def credential_kind(self) -> str | None:
+ return "tmdb"
+
def resume_covers_run(
self,
*,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
index 6879eaabd96c..47a35459cfaf 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/youtube_analytics/source.py
@@ -65,6 +65,10 @@ class YouTubeAnalyticsSource(ResumableSource[YouTubeAnalyticsSourceConfig, YouTu
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.YOUTUBEANALYTICS
+ @property
+ def credential_kind(self) -> str | None:
+ return "youtube-analytics"
+
def get_non_retryable_errors(self) -> dict[str, str | None]:
return {
"401 Client Error": (
diff --git a/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts b/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts
index efe5901e40b3..e79acef8067f 100644
--- a/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts
+++ b/products/workflows/frontend/Channels/EmailDomain/__mocks__/fakeEmailDomainBackend.ts
@@ -191,6 +191,7 @@ export class FakeEmailDomainBackend {
const sender: IntegrationConfigApi = {
id: this.nextSenderId++,
kind: 'email',
+ auth_type: 'credentials',
config: {
provider: 'ses',
name: 'Acme',
diff --git a/services/mcp/src/api/generated.ts b/services/mcp/src/api/generated.ts
index 1c379085fc87..57bcd4a245c8 100644
--- a/services/mcp/src/api/generated.ts
+++ b/services/mcp/src/api/generated.ts
@@ -65896,12 +65896,32 @@ export namespace Schemas {
reconnect_required: boolean;
}
+ /**
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials
+ */
+ export type IntegrationAuthTypeEnum = typeof IntegrationAuthTypeEnum[keyof typeof IntegrationAuthTypeEnum];
+
+
+ export const IntegrationAuthTypeEnum = {
+ Oauth: 'oauth',
+ Token: 'token',
+ Credentials: 'credentials',
+ } as const;
+
/**
* Standard Integration serializer.
*/
export interface IntegrationConfig {
readonly id: number;
kind: IntegrationKindEnum;
+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.
+ *
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials */
+ readonly auth_type: IntegrationAuthTypeEnum;
config?: unknown;
readonly created_at: string;
readonly created_by: UserBasic;
@@ -88937,6 +88957,12 @@ export namespace Schemas {
export interface PatchedIntegrationConfig {
readonly id?: number;
kind?: IntegrationKindEnum;
+ /** How this integration authenticates with the provider: `oauth` for a connection the provider manages (an OAuth grant or an app installation), `token` for one static secret, or `credentials` for several typed-in fields. A kind can have several auth types.
+ *
+ * * `oauth` - OAuth
+ * * `token` - Token
+ * * `credentials` - Credentials */
+ readonly auth_type?: IntegrationAuthTypeEnum;
config?: unknown;
readonly created_at?: string;
readonly created_by?: UserBasic;
diff --git a/tools/hogli-commands/hogli_commands/projections.py b/tools/hogli-commands/hogli_commands/projections.py
index 010a9eca8f76..5535ebb2e7af 100644
--- a/tools/hogli-commands/hogli_commands/projections.py
+++ b/tools/hogli-commands/hogli_commands/projections.py
@@ -15,8 +15,8 @@ each output, keyed by its repo-relative path. It never writes files and never pa
arguments; this module does both, so ``--check`` and the write path cannot disagree.
A renderer returns valid source and leaves the style to the formatter of the workspace
-that owns the output: oxfmt at the repo root, Biome under ``products/desktop`` and
-``packages/agent``. The runner formats every output before it compares or writes, so a
+that owns the output: ruff for Python, oxfmt at the repo root, Biome under ``products/desktop``
+and ``packages/agent``. The runner formats every output before it compares or writes, so a
generated file looks like the code around it and no formatter needs an exemption for it.
"""
@@ -25,8 +25,10 @@ from __future__ import annotations
import sys
import json
import runpy
+import tomllib
import tempfile
import subprocess
+import importlib.metadata
from collections.abc import Callable, Iterable
from dataclasses import dataclass
from pathlib import Path
@@ -101,6 +103,16 @@ PROJECTIONS: tuple[Projection, ...] = (
"products/commands/COMMANDS.generated.md",
),
),
+ Projection(
+ name="integration-kinds",
+ renderer="posthog/integrations/kinds_projection.py",
+ inputs=(
+ "posthog/integrations/catalogue.py",
+ "posthog/integrations/contracts.py",
+ "posthog/integrations/enums.py",
+ ),
+ outputs=("posthog/integrations/kinds_generated.py",),
+ ),
)
@@ -140,10 +152,22 @@ AGENT_BIOME = Formatter(
lockfile="packages/agent/pnpm-lock.yaml",
stdin_args=("format", "--stdin-file-path"),
)
-FORMATTERS = (OXFMT, BIOME, AGENT_BIOME)
+# A Python package, not an npm one: the runner reads its version from uv.lock and runs it from the
+# Python environment, which the Python CI job has without node_modules.
+RUFF = Formatter(
+ package="ruff",
+ binary="ruff",
+ workspace=".",
+ config="pyproject.toml",
+ lockfile="uv.lock",
+ stdin_args=("format", "--stdin-filename"),
+)
+FORMATTERS = (OXFMT, BIOME, AGENT_BIOME, RUFF)
def formatter_for(path: str) -> Formatter:
+ if path.endswith(".py"):
+ return RUFF
for formatter in (BIOME, AGENT_BIOME):
if path.startswith(f"{formatter.workspace}/"):
return formatter
@@ -175,6 +199,9 @@ class OutputFormatter:
self._npm_prefix = tempfile.TemporaryDirectory()
def _locked_version(self, formatter: Formatter) -> str:
+ if formatter == RUFF:
+ lock = tomllib.loads((self.repo_root / formatter.lockfile).read_text())
+ return next(package["version"] for package in lock["package"] if package["name"] == formatter.package)
loader = getattr(yaml, "CSafeLoader", yaml.SafeLoader)
lock = yaml.load((self.repo_root / formatter.lockfile).read_text(), Loader=loader)
return lock["importers"]["."]["devDependencies"][formatter.package]["version"]
@@ -182,6 +209,17 @@ class OutputFormatter:
def _command(self, formatter: Formatter) -> list[str]:
# The installed copy when it matches the lockfile, so a local run stays offline.
# CI's Python job has no node_modules, so it runs the locked version through npm.
+ if formatter not in self._commands and formatter == RUFF:
+ version = self._locked_version(formatter)
+ installed_version: str | None
+ try:
+ installed_version = importlib.metadata.version(formatter.package)
+ except importlib.metadata.PackageNotFoundError:
+ installed_version = None
+ if installed_version == version:
+ self._commands[formatter] = [sys.executable, "-m", formatter.binary]
+ else:
+ self._commands[formatter] = ["uvx", f"{formatter.package}=={version}"]
if formatter not in self._commands:
version = self._locked_version(formatter)
node_modules = self.repo_root / formatter.workspace / "node_modules"
diff --git a/tools/hogli-commands/hogli_commands/tests/test_projections.py b/tools/hogli-commands/hogli_commands/tests/test_projections.py
index defa07e4ed0e..883a10f7c6ea 100644
--- a/tools/hogli-commands/hogli_commands/tests/test_projections.py
+++ b/tools/hogli-commands/hogli_commands/tests/test_projections.py
@@ -1,11 +1,22 @@
from __future__ import annotations
+import sys
+import importlib.metadata
from pathlib import Path
import pytest
import click
-from hogli_commands.projections import OutputFormatter, Projection, ProjectionRunner
+from hogli_commands.projections import (
+ BIOME,
+ OXFMT,
+ RUFF,
+ Formatter,
+ OutputFormatter,
+ Projection,
+ ProjectionRunner,
+ formatter_for,
+)
RENDERER = """
def render():
@@ -88,3 +99,38 @@ class TestProjectionRunner:
with pytest.raises(click.ClickException, match="Keep the two in step"):
runner.stale(PROJECTION)
+
+
+def _uv_lock(tmp_path: Path, version: str) -> None:
+ (tmp_path / "uv.lock").write_text(f'version = 1\n\n[[package]]\nname = "ruff"\nversion = "{version}"\n')
+
+
+class TestRuffFormatter:
+ @pytest.mark.parametrize(
+ "path,expected",
+ [
+ ("posthog/integrations/kinds_generated.py", RUFF),
+ ("products/desktop/scripts/tool.py", RUFF),
+ ("frontend/src/lib/oauthScopes.generated.ts", OXFMT),
+ ("products/desktop/packages/core/src/a.generated.ts", BIOME),
+ ],
+ )
+ def test_formatter_for_picks_ruff_for_python(self, path: str, expected: Formatter) -> None:
+ assert formatter_for(path) == expected
+
+ @pytest.mark.parametrize(
+ "locked_matches,expected_prefix",
+ [(True, [sys.executable, "-m", "ruff"]), (False, ["uvx", "ruff==0.0.1"])],
+ ids=["installed-version-matches-the-lock", "falls-back-to-the-locked-version"],
+ )
+ def test_ruff_runs_from_the_environment_only_at_the_locked_version(
+ self, tmp_path: Path, locked_matches: bool, expected_prefix: list[str]
+ ) -> None:
+ _uv_lock(tmp_path, importlib.metadata.version("ruff") if locked_matches else "0.0.1")
+
+ assert OutputFormatter(tmp_path)._command(RUFF) == expected_prefix
+
+ def test_formats_python_output(self, tmp_path: Path) -> None:
+ _uv_lock(tmp_path, importlib.metadata.version("ruff"))
+
+ assert OutputFormatter(tmp_path).format("out/kinds.py", "x={ 'a':1 }") == 'x = {"a": 1}\n'