Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Keep usage-analytics lookups long enough to be reused in a large workspace (CHOO-3667)

Abel Dantas90ec8947887a
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8b9791612..de9220ae0 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -159,6 +159,14 @@ version of their own to them without also giving them a release of their own.
compares it only when someone signs in.
#### Changed
+- **Usage analytics read the database far less per message in a large
+ workspace.** With usage reporting on, each message looked up its room's
+ platform and member counts, its sender and the agent's runtime, each kept
+ for five minutes in caches of at most 10,000 entries. Quiet rooms missed
+ every time, and with thousands of agents the caches evicted what they were
+ about to be asked again. A room's facts are now kept for fifteen minutes,
+ a sender and an agent's runtime, which do not change, for an hour, and up
+ to 50,000 of each.
- **An agent's stream reads in one database session per pass, and pooled
connections are no longer pinged before use.** A pass that delivered a
message opened a session for each of its reads, six pool checkouts and six
diff --git a/core/switch_core/telemetry/messages.py b/core/switch_core/telemetry/messages.py
index 82077aebd..31ee6e176 100644
--- a/core/switch_core/telemetry/messages.py
+++ b/core/switch_core/telemetry/messages.py
@@ -8,10 +8,12 @@ thing being reported on. Three rules keep it off that path:
record and return. One worker task looks up what the event needs (the
room's platform, type and membership, who the sender is) and emits.
- **Lookups are cached.** A busy room says a lot and changes membership
- rarely, so a room is read at most once per `_CACHE_TTL_SECONDS` rather than
- once per message. The member counts can therefore be that far behind. A
- lookup that finds nothing is cached too, as `unknown`: a room deleted while
- its last messages were queued must not cost a query per message.
+ rarely, so a room is read at most once per `_ROOM_TTL_SECONDS` rather than
+ once per message. The member counts can therefore be that far behind. Who
+ a sender is and an agent's runtime do not change, and are kept for
+ `_IDENTITY_TTL_SECONDS`. A lookup that finds nothing is cached too, as
+ `unknown`: a room deleted while its last messages were queued must not cost
+ a query per message.
- **A failing lookup is left alone.** A room, sender or agent whose lookup
failed is not looked up again for `_LOOKUP_BACKOFF_SECONDS`, and events
report it as `unknown` meanwhile. Only that key: one tenant or room that
@@ -53,8 +55,15 @@ from switch_core.transport.observer import ParticipantMessage
logger = logging.getLogger(__name__)
_QUEUE_SIZE = 10_000
-_CACHE_TTL_SECONDS = 300.0
-_CACHE_MAX_ENTRIES = 10_000
+# A room's facts are re-read this often, so its member counts are at most
+# this far behind. Most rooms are quiet: kept for minutes rather than an hour,
+# a room's lookup is a query for most of its messages.
+_ROOM_TTL_SECONDS = 900.0
+# Who a sender is, and an agent's runtime: neither changes.
+_IDENTITY_TTL_SECONDS = 3600.0
+# Above the agents and rooms of a large deployment, so a cache at its limit
+# does not evict what it is about to be asked again.
+_CACHE_MAX_ENTRIES = 50_000
_WARNING_INTERVAL_SECONDS = 60.0
_LOOKUP_BACKOFF_SECONDS = 5.0
# Failures in a row, from any keys, that read as the database failing rather
@@ -164,13 +173,13 @@ class MessageTelemetry:
self._worker: asyncio.Task[None] | None = None
self._closed = False
self._rooms: _TtlCache[tuple[str, str], _RoomFacts] = _TtlCache(
- _CACHE_TTL_SECONDS, _CACHE_MAX_ENTRIES
+ _ROOM_TTL_SECONDS, _CACHE_MAX_ENTRIES
)
self._sender_kinds: _TtlCache[tuple[str, str], str] = _TtlCache(
- _CACHE_TTL_SECONDS, _CACHE_MAX_ENTRIES
+ _IDENTITY_TTL_SECONDS, _CACHE_MAX_ENTRIES
)
self._agent_runtimes: _TtlCache[tuple[str, str], str] = _TtlCache(
- _CACHE_TTL_SECONDS, _CACHE_MAX_ENTRIES
+ _IDENTITY_TTL_SECONDS, _CACHE_MAX_ENTRIES
)
# Keys whose lookup failed, as `(lookup, tenant_id, id)`, until they
# may be tried again.
diff --git a/core/tests/switch_core/telemetry/test_message_events.py b/core/tests/switch_core/telemetry/test_message_events.py
index ad124d320..23fee7851 100644
--- a/core/tests/switch_core/telemetry/test_message_events.py
+++ b/core/tests/switch_core/telemetry/test_message_events.py
@@ -12,6 +12,7 @@ import asyncio
import time
import uuid
from types import SimpleNamespace
+from unittest.mock import MagicMock
import pytest
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
@@ -670,6 +671,33 @@ class TestCaching:
] == ["unknown", "unknown"]
+class TestHowLongLookupsAreKept:
+ def test_a_room_for_a_quarter_hour_and_who_someone_is_for_an_hour(
+ self, monkeypatch: pytest.MonkeyPatch
+ ) -> None:
+ clock = SimpleNamespace(now=100.0)
+ monkeypatch.setattr(
+ messages_module, "time", SimpleNamespace(monotonic=lambda: clock.now)
+ )
+ telemetry = MessageTelemetry(telemetry=MagicMock(), session_factory=MagicMock())
+ room = _RoomFacts(
+ bridge_platform="slack", channel_type="channel", user_count=2, agent_count=1
+ )
+ telemetry._rooms.put(("t", "room"), room)
+ telemetry._sender_kinds.put(("t", "@alice:test"), "user")
+ telemetry._agent_runtimes.put(("t", "client"), "claude_code")
+
+ clock.now += 14 * 60
+ assert telemetry._rooms.get(("t", "room")) == room
+ clock.now += 2 * 60
+ assert telemetry._rooms.get(("t", "room")) is None
+ assert telemetry._sender_kinds.get(("t", "@alice:test")) == "user"
+ assert telemetry._agent_runtimes.get(("t", "client")) == "claude_code"
+ clock.now += 45 * 60
+ assert telemetry._sender_kinds.get(("t", "@alice:test")) is None
+ assert telemetry._agent_runtimes.get(("t", "client")) is None
+
+
class TestTtlCache:
def test_an_entry_expires_after_its_age(
self, monkeypatch: pytest.MonkeyPatch
diff --git a/docs/old/telemetry-events.md b/docs/old/telemetry-events.md
index 0c0bce8f7..92fd1d9a3 100644
--- a/docs/old/telemetry-events.md
+++ b/docs/old/telemetry-events.md
@@ -262,8 +262,8 @@ reasons. Each is now answered rather than ignored:
costs events, never memory.
2. **The sender must not pay for it.** The transport and the router hand a
small record to a queue and return. One worker looks up the room and
- the sender — cached, so a busy room is read once every five minutes rather
- than once per message — and emits. That queue is bounded too, and drops with
+ the sender — cached, so a busy room is read once every fifteen minutes
+ rather than once per message — and emits. That queue is bounded too, and drops with
a warning when it falls behind.
3. **Content risk.** The events carry the same kind of property as every
other: closed sets, booleans and counts. There is no property a message body,
@@ -667,8 +667,9 @@ rather than something the product wants to know.
All three come from `telemetry/messages.py`, reported off the sender's path as
described [above](#message-events-and-what-they-cannot-tell-you). A post with
-several files is one message. Room facts are cached for five minutes, so a
-member count can be that far behind. A room, sender or agent whose lookup
+several files is one message. Room facts are cached for fifteen minutes, so a
+member count can be that far behind; who a sender is and an agent's runtime,
+which do not change, for an hour. A room, sender or agent whose lookup
fails is not looked up again for five seconds, and its events report it as
`unknown` meanwhile; every other room carries on. Three failures in a row,
with no lookup succeeding between them, pause every lookup for five seconds.