Skip to content

Self-improving AI Employees for everyone

chore(temporal): pool-tuning hygiene + document benign UpdateTaskQueue log noise

trohitgc5a245bb9f11
diff --git a/docs-internal/TEMPORAL_ARCHITECTURE.md b/docs-internal/TEMPORAL_ARCHITECTURE.md
index 67c436a0..7ed2eaf7 100644
--- a/docs-internal/TEMPORAL_ARCHITECTURE.md
+++ b/docs-internal/TEMPORAL_ARCHITECTURE.md
@@ -460,7 +460,7 @@ The Temporal binary + persistence layer are managed in-process by the plugin-fol
| Setting | Env var | Default | Purpose |
|---|---|---|---|
-| `temporal_backend` | `TEMPORAL_BACKEND` | `sqlite` | Backend selector — `sqlite` (dev) / `postgres` (prod). |
+| `temporal_backend` | `TEMPORAL_BACKEND` | `postgres` | Backend selector — `postgres` (default, durable) / `sqlite` (fast-iteration dev). |
| `temporal_frontend_grpc_port` | `TEMPORAL_FRONTEND_GRPC_PORT` | `7233` | gRPC port the SDK client connects to. Drives the readiness probe + cluster `rpcAddress`. |
| `temporal_matching_grpc_port` | `TEMPORAL_MATCHING_GRPC_PORT` | `7235` | Internal matching service. |
| `temporal_history_grpc_port` | `TEMPORAL_HISTORY_GRPC_PORT` | `7234` | Internal history service. |
@@ -469,7 +469,7 @@ The Temporal binary + persistence layer are managed in-process by the plugin-fol
| `temporal_num_history_shards` | `TEMPORAL_NUM_HISTORY_SHARDS` | `4` | History-shard count. Bump for higher write throughput; not online-resizable. |
| `temporal_default_max_conns` | `TEMPORAL_DEFAULT_MAX_CONNS` | `20` | Postgres pool size for the `temporal` (history) datastore. |
| `temporal_visibility_max_conns` | `TEMPORAL_VISIBILITY_MAX_CONNS` | `4` | Postgres pool size for the `temporal_visibility` datastore. |
-| `temporal_max_conn_lifetime` | `TEMPORAL_MAX_CONN_LIFETIME` | `1h` | Postgres connection rotation interval. Accepts Go-duration strings (`30m`, `2h`, etc.). |
+| `temporal_max_conn_lifetime` | `TEMPORAL_MAX_CONN_LIFETIME` | `5m` | Postgres connection rotation interval. Accepts Go-duration strings (`30m`, `2h`, etc.). Short window forces periodic refresh — Temporal community recommendation. |
| `temporal_binary_version` | `TEMPORAL_BINARY_VERSION` | `1.31.0` | Pinned `temporalio/temporal` release tag. Bumping requires a matching entry in `_install.py:_CHECKSUMS_BY_VERSION`. |
| `temporal_graceful_shutdown_seconds` | `TEMPORAL_GRACEFUL_SHUTDOWN_SECONDS` | `30` | SIGTERM → SIGKILL grace window for the Temporal binary supervisor. Shared with the embedded worker shutdown. |
| `temporal_postgres_dsn` | `TEMPORAL_POSTGRES_DSN` | (unset) | Reserved for multi-host deployments — when set, bypasses pgserver and points Temporal at an external Postgres. Out of scope for current sprint. |
@@ -481,6 +481,17 @@ Two supervisor-build-time env vars (read inside `_temporal_specs.py`, not in `Se
| `TEMPORAL_PG_READY_TIMEOUT_SECONDS` | `60` | How long the supervisor waits for pgserver to be ready before marking the spec failed. |
| `TEMPORAL_SERVER_READY_TIMEOUT_SECONDS` | `120` | How long the supervisor waits for Temporal's gRPC port to come up. Includes the first-run binary download (~90 MB). |
+### Known benign log noise
+
+Temporal's matching engine periodically writes task-queue metadata for the **internal `temporal-system` namespace** (e.g. `/_sys/default-worker-tq/1`). When two writes contend, the older one's request context gets canceled — Temporal then auto-retries, and workflow execution is unaffected. The cancellation surfaces in the server log as:
+
+```
+ERROR msg="Operation failed with internal error." error="UpdateTaskQueue failed. Failed to start transaction. Error: context canceled" operation=UpdateTaskQueue
+ERROR msg="Persistent store operation failure" component=matching-engine wf-task-queue-name=/_sys/default-worker-tq/1 wf-namespace=temporal-system
+```
+
+These are **not caused by the Postgres backend** — they appeared identically with the previous SQLite path and are reported as benign across the Temporal community ([forum thread](https://community.temporal.io/t/persistent-store-operation-failure-on-updatetaskqueue-operations/11173)). The connection-pool tunables above (`temporal_default_max_conns`, `temporal_max_conn_lifetime`) are sized per Temporal's recommended baseline to minimise the frequency. No further action is required unless you see them in a **user namespace** (e.g. `default` instead of `temporal-system`), which would indicate a real persistence problem.
+
## Debugging
```bash
diff --git a/server/core/config.py b/server/core/config.py
index b1f1b07f..b92f8ea3 100644
--- a/server/core/config.py
+++ b/server/core/config.py
@@ -125,8 +125,14 @@ class Settings(BaseSettings):
temporal_visibility_max_conns: int = Field(
default=4, env="TEMPORAL_VISIBILITY_MAX_CONNS", ge=1, le=500,
)
+ # Postgres connection rotation interval. Temporal community
+ # recommendation is ~5 minutes (much shorter than the 1h default)
+ # because Postgres' default `tcp_keepalives_idle` and pooled-driver
+ # state can lead to "context canceled" errors on idle connections
+ # being reused — periodic refresh sidesteps the issue. Accepts
+ # Go-duration strings (`30s`, `5m`, `1h`).
temporal_max_conn_lifetime: str = Field(
- default="1h", env="TEMPORAL_MAX_CONN_LIFETIME",
+ default="5m", env="TEMPORAL_MAX_CONN_LIFETIME",
)
temporal_binary_version: str = Field(
default="1.31.0", env="TEMPORAL_BINARY_VERSION",
diff --git a/server/services/temporal/_config.py b/server/services/temporal/_config.py
index a9f64cf8..3a067b1b 100644
--- a/server/services/temporal/_config.py
+++ b/server/services/temporal/_config.py
@@ -57,6 +57,12 @@ def _sql_datastore(
pg: dict[str, Any], db_name: str, *,
max_conns: int, max_conn_lifetime: str,
) -> dict[str, Any]:
+ # ``maxIdleConns`` matches ``maxConns`` so the pool stays fully
+ # warm — Temporal's matching-engine periodic UpdateTaskQueue
+ # writes are bursty, and a too-small idle pool causes "context
+ # canceled" errors when the burst arrives faster than the pool
+ # can create new connections. Mirrors the Temporal community
+ # recommendation; see docs-internal/TEMPORAL_ARCHITECTURE.md.
return {
"sql": {
"pluginName": "postgres12",
@@ -66,7 +72,7 @@ def _sql_datastore(
"user": pg["user"],
"password": pg["password"],
"maxConns": max_conns,
- "maxIdleConns": max(2, max_conns // 4),
+ "maxIdleConns": max_conns,
"maxConnLifetime": max_conn_lifetime,
},
}