Skip to content

Open-source AI teammates in Slack, Teams, Discord, Telegram and Mattermost. Self-host it, with any agent harness and any model.

Optimize garbage collection, so full passes stop stalling the main loop (#712)

Abel Dantas7dfa3e7f251f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2199b6e26..97052e097 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -136,6 +136,13 @@ version of their own to them without also giving them a release of their own.
Existing duplicates are left as they are and can be reached by id.
#### Fixed
+- **Full garbage collections no longer stall the server for hundreds of
+ milliseconds.** Once boot has built what lasts for the life of the process
+ (modules, the ORM's mappers, the app's routes), the server freezes it out of
+ garbage collection, so a full pass walks only what was made since. The
+ clients and bridges started after that stay collectable, since replicas hand
+ them to one another. The dashboard's pause chart now shows the average pause
+ next to p99, as Datadog keeps the average for longer.
- **Agents no longer read offline while their replica shuts down.** With several
replicas, a stopping replica released its lease before its 5 s drain, so for
up to about 6 s the other replicas counted its connected agents as gone: `@role`
diff --git a/core/switch_core/main.py b/core/switch_core/main.py
index 6eff94617..9d17aa22b 100644
--- a/core/switch_core/main.py
+++ b/core/switch_core/main.py
@@ -1,6 +1,7 @@
from __future__ import annotations
import asyncio
+import gc
import hashlib
import logging
import os
@@ -1457,6 +1458,11 @@ async def run(config: SwitchConfig, process_id: str) -> None:
agent_bridge_app.router.lifespan_context = lifespan # type: ignore[assignment]
+ # Before the runtime starts: what follows is the clients, bridges and work
+ # this process owns, which ownership hands to another replica as replicas
+ # come and go, so it must stay collectable.
+ _freeze_boot_objects()
+
# ── Start runtime ────────────────────────────────────────────────────────
await client_lifecycle.start_all()
# Before the bridges start, so none on a shared app runs without the
@@ -1594,6 +1600,32 @@ async def run(config: SwitchConfig, process_id: str) -> None:
await _begin_shutdown.finish()
+def _freeze_boot_objects() -> None:
+ """Take everything boot created out of the garbage collector's way.
+
+ A full collection walks every tracked object while holding the interpreter
+ lock, and most of the heap is what boot built and keeps for the life of the
+ process: modules, the ORM's mappers, the app's routes. Walking that once a
+ minute is what stalls the event loop for hundreds of milliseconds.
+ `gc.freeze` moves it to a permanent generation the collector skips, so
+ full collections only walk what was allocated since.
+
+ Called before the runtime starts, not once it is serving. A frozen object
+ is still freed when its last reference goes, but one caught in a reference
+ cycle never is, and the clients and bridges a process runs are full of
+ cycles (tasks, callbacks, sessions). With several replicas, ownership moves
+ that work between processes whenever one joins or leaves, so freezing it
+ would keep every client and bridge this process started at boot alive
+ after it hands them over.
+ """
+ gc.collect()
+ gc.freeze()
+ logger.info(
+ "Froze %d objects created during boot out of garbage collection.",
+ gc.get_freeze_count(),
+ )
+
+
async def _seed_admin_and_bootstrap_key(
session_factory: async_sessionmaker[AsyncSession],
user_store: UserStore,
diff --git a/deploy/observability/dashboard.json b/deploy/observability/dashboard.json
index 41eeb3dfb..def5e4e20 100644
--- a/deploy/observability/dashboard.json
+++ b/deploy/observability/dashboard.json
@@ -562,7 +562,7 @@
{
"definition": {
"type": "timeseries",
- "title": "Garbage-collection pause p99 / max (ms) by generation. Needs distribution mapping, see README",
+ "title": "Garbage-collection pause p99 / avg (ms) by generation. Needs distribution mapping, see README",
"show_legend": true,
"requests": [
{
@@ -570,7 +570,7 @@
"display_type": "line"
},
{
- "q": "max:switch.runtime.gc_pause{$env,$service} by {generation}",
+ "q": "avg:switch.runtime.gc_pause{$env,$service} by {generation}",
"display_type": "bars"
}
]