LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
UN-4481 [FEAT] Show each organization's home region in the organization list (#2370)
Athulf5daa5d95dac
diff --git a/backend/account_v2/ReadMe.md b/backend/account_v2/ReadMe.mdindex 70e5b1ff8..415a56e2f 100644--- a/backend/account_v2/ReadMe.md+++ b/backend/account_v2/ReadMe.md@@ -53,8 +53,8 @@| Field | Present when | Value ||---|---|---|| `role` | The caller has a membership in the organization on this deployment. | The stored role: `admin` or `user` in OSS; `unstract_admin`, `unstract_user`, `unstract_supervisor` or `unstract_reviewer` in cloud. |-| `member_count` | The organization exists on this deployment. | The people the members page lists: the organization's member records on this deployment. Platform API key accounts and pending invitations are not counted. |-| `region` | The organization exists on this deployment and the deployment sets `REGION`. | The region code (`US`, `EU`; `STAGING`, `DEV` or `INTEGRATION` outside production). The client maps it to a label. OSS and on-prem never send it: their settings don't read `REGION`, whatever the environment sets. It is the region of the deployment that answered: an organization from another region that the user once opened here has a local copy, so it shows this region and this deployment's member count. |+| `member_count` | The organization exists on this deployment and, where the deployment sets `REGION`, its metadata doesn't name a different region. An organization created before the stamp and not back-filled carries no region, so a local copy of one from another region still sends the copy's count. | The people the members page lists: the organization's member records on this deployment. Platform API key accounts and pending invitations are not counted. |+| `region` | The deployment sets `REGION`, and either the organization's metadata names a region or the organization exists on this deployment. | The region code (`US`, `EU`; `STAGING`, `DEV` or `INTEGRATION` outside production), from the metadata trimmed and uppercased, else this deployment's `REGION`. The client maps it to a label. OSS and on-prem never send it: their settings don't read `REGION`, whatever the environment sets. Cloud stamps `region` on an organization's metadata where it is created. An organization from another region that the user once opened here has a local copy; its row shows the home region and no `member_count`, since this deployment counts only the people who opened the copy. Organizations created before the stamp and not back-filled show the region of the deployment that answered. |If working out these three fields fails, the list is returned without them and the failure is logged.diff --git a/backend/account_v2/organization_details.py b/backend/account_v2/organization_details.pyindex bd6deb38b..572a91da0 100644--- a/backend/account_v2/organization_details.py+++ b/backend/account_v2/organization_details.py@@ -6,12 +6,23 @@ field is added only where this deployment can answer it, and is otherwise leftout of the row rather than sent as null:- ``role``: the caller has a membership row in that organization here.-- ``member_count``: the organization exists here. Counts the people the- members page lists: the organization's member records here, without- platform API key accounts. Pending invitations have no record yet.-- ``region``: the organization exists here and ``settings.REGION`` is set.+- ``member_count``: the organization exists here and, where+ ``settings.REGION`` is set, its metadata names no other region. Counts the+ people the members page lists: the organization's member records here,+ without platform API key accounts. Pending invitations have no record yet.+- ``region``: only where ``settings.REGION`` is set: the region named in the+ organization's metadata, else ``settings.REGION`` when the organization+ exists here.++Cloud stamps ``region`` on an organization's metadata where it is created.+Opening it from another region makes a local copy there, so where the stamp+names another region the row shows that region and leaves out this+deployment's member count, which covers only the people who opened the copy.+An organization without the stamp (created before it and not back-filled) is+treated as this region's: a copy of one shows this region and the copy's count."""+import loggingfrom collections.abc import Sequencefrom typing import Any, Protocol@@ -21,6 +32,8 @@ from tenant_account_v2.models import OrganizationMemberfrom account_v2.models import Organization, User+logger = logging.getLogger(__name__)+class ListedOrganization(Protocol):"""What the auth services return for each organization.@@ -34,6 +47,21 @@ class ListedOrganization(Protocol):name: str+def _home_region(metadata: object) -> str | None:+ """The ``region`` in an organization's metadata (from the identity provider).++ Returned trimmed and uppercased; None when there is no usable stamp.+ """+ stamp = metadata.get("region") if isinstance(metadata, dict) else None+ if stamp is None:+ return None+ code = stamp.strip().upper() if isinstance(stamp, str) else ""+ if not code:+ logger.warning("Ignoring malformed region stamp %r", stamp)+ return None+ return code++def with_member_details(organizations: Sequence[ListedOrganization], user: User) -> list[dict[str, Any]]:@@ -67,21 +95,26 @@ def with_member_details(user=user, organization__organization_id__in=ids).values_list("organization__organization_id", "role"))- region = getattr(settings, "REGION", None)+ region = (getattr(settings, "REGION", None) or "").strip().upper() or Nonerows = []for org in organizations:+ metadata = getattr(org, "metadata", None)+ home = _home_region(metadata) if region else Nonerow: dict[str, Any] = {"id": org.id,"display_name": org.display_name,"name": org.name,- "metadata": getattr(org, "metadata", None),+ "metadata": metadata,}if org.id in roles:row["role"] = roles[org.id]- if org.id in member_counts:+ exists_here = org.id in member_counts+ is_copy = home is not None and home != region+ if exists_here and not is_copy:row["member_count"] = member_counts[org.id]- if region:- row["region"] = region+ org_region = home or (region if exists_here else None)+ if org_region:+ row["region"] = org_regionrows.append(row)return rowsdiff --git a/backend/account_v2/serializer.py b/backend/account_v2/serializer.pyindex 8fb47a3b2..6c3acfb98 100644--- a/backend/account_v2/serializer.py+++ b/backend/account_v2/serializer.py@@ -49,13 +49,18 @@ class GetOrganizationsResponseSerializer(serializers.Serializer):required=False,help_text="The people the members page lists, without platform API key ""accounts. Absent if the organization has no data on this deployment, "- "or if the details could not be loaded.",+ "if the deployment has regions and the organization's metadata names "+ "another one, or if the details could not be loaded. Organizations "+ "without a region in their metadata count as this deployment's: see "+ "account_v2/ReadMe.md.",)region = serializers.CharField(required=False,- help_text="The deployment's region code, e.g. US or EU. Absent where "- "the deployment has no regions, the organization has no data here, or "- "the details could not be loaded.",+ help_text="The region code, e.g. US or EU, named in the organization's "+ "metadata, else this deployment's region if the organization exists "+ "here. Absent where the deployment has no regions, the region is not "+ "known here, or the details could not be loaded. See account_v2/ReadMe.md "+ "for organizations without a region in their metadata.",)def to_representation(self, instance): # type: ignorediff --git a/backend/account_v2/test_organization_details.py b/backend/account_v2/test_organization_details.pyindex 439462b76..f6ac43287 100644--- a/backend/account_v2/test_organization_details.py+++ b/backend/account_v2/test_organization_details.py@@ -164,3 +164,108 @@ class WithMemberDetailsTests(TestCase):def test_no_organizations(self) -> None:self.assertEqual(with_member_details([], self.caller), [])+++# Cloud stamps the region an organization was created in on its identity+# provider metadata, which the auth service returns with each organization.+@override_settings(REGION="US")+class HomeRegionTests(TestCase):+ def setUp(self) -> None:+ self.caller = _user("[email protected]")+ self.org = _org("org-a", "Org A")+ OrganizationMember.objects.create(+ organization=self.org, user=self.caller, role="admin"+ )++ def _row(self, metadata: object, org_id: str = "org-a") -> dict:+ org = SimpleNamespace(+ id=org_id, display_name="Org A", name=org_id, metadata=metadata+ )+ return with_member_details([org], self.caller)[0]++ def test_home_organisation_keeps_this_regions_details(self) -> None:+ row = self._row({"region": "US"})+ self.assertEqual((row["region"], row["member_count"]), ("US", 1))++ def test_local_copy_shows_its_home_region(self) -> None:+ row = self._row({"region": "EU"})+ self.assertEqual(row["region"], "EU")+ # This deployment's count covers only the people who opened the copy here.+ self.assertNotIn("member_count", row)+ self.assertEqual(row["role"], "admin")++ def test_organisation_with_no_data_here_still_shows_its_region(self) -> None:+ row = self._row({"region": "EU"}, org_id="org-x")+ self.assertEqual(row["region"], "EU")+ self.assertNotIn("member_count", row)+ self.assertNotIn("role", row)++ def test_unstamped_organisation_falls_back_to_this_region(self) -> None:+ # Organizations created before the stamp, until they are back-filled.+ row = self._row({"environment": "cloud-production"})+ self.assertEqual((row["region"], row["member_count"]), ("US", 1))++ @override_settings(REGION=None)+ def test_deployment_without_regions_ignores_the_stamp(self) -> None:+ # OSS and on-prem: a region key in the provider's metadata changes nothing.+ row = self._row({"region": "EU"})+ self.assertNotIn("region", row)+ self.assertEqual(row["member_count"], 1)++ def test_stamp_that_is_not_a_non_empty_string_is_ignored(self) -> None:+ for stamp in ("", " ", ["EU"], 7):+ with self.subTest(stamp=repr(stamp)):+ with self.assertLogs("account_v2.organization_details", "WARNING"):+ row = self._row({"region": stamp})+ self.assertEqual((row["region"], row["member_count"]), ("US", 1))++ def test_stamp_is_compared_without_case_or_padding(self) -> None:+ for stamp, region, count in (+ ("us", "US", 1),+ (" US ", "US", 1),+ ("eu", "EU", None),+ ):+ with self.subTest(stamp=stamp):+ row = self._row({"region": stamp})+ self.assertEqual(+ (row["region"], row.get("member_count")), (region, count)+ )++ @override_settings(REGION="us")+ def test_this_deployments_region_is_compared_without_case(self) -> None:+ row = self._row({"region": "US"})+ self.assertEqual((row["region"], row["member_count"]), ("US", 1))++ def test_metadata_that_is_not_a_dict_counts_as_unstamped(self) -> None:+ for metadata in (None, "EU", ["EU"]):+ with self.subTest(metadata=repr(metadata)):+ row = self._row(metadata)+ self.assertEqual((row["region"], row["member_count"]), ("US", 1))+ self.assertEqual(row["metadata"], metadata)++ def test_this_regions_organisation_with_no_data_here(self) -> None:+ row = self._row({"region": "US"}, org_id="org-x")+ self.assertEqual(row["region"], "US")+ self.assertNotIn("member_count", row)+ self.assertNotIn("role", row)++ def test_mixed_list_keeps_order_and_queries(self) -> None:+ _org("org-b", "Org B")+ orgs = [+ CloudOrg(id="org-a", display_name="A", name="a", metadata={"region": "EU"}),+ CloudOrg(id="org-b", display_name="B", name="b", metadata=None),+ CloudOrg(id="org-x", display_name="X", name="x", metadata={"region": "US"}),+ ]+ with self.assertNumQueries(2):+ rows = with_member_details(orgs, self.caller)+ self.assertEqual(+ [+ (r["id"], r.get("region"), r.get("member_count"), r.get("role"))+ for r in rows+ ],+ [+ ("org-a", "EU", None, "admin"),+ ("org-b", "US", 0, None),+ ("org-x", "US", None, None),+ ],+ )