The open-source agent harness - the runtime layer that turns an LLM into a working agent.
feat: add Redis Sentinel and TLS support in TrueForge (#765)
Sujai Kumar Guptacf55de9aed64
diff --git a/.changeset/redis-sentinel.md b/.changeset/redis-sentinel.mdnew file mode 100644index 000000000..72e00471b--- /dev/null+++ b/.changeset/redis-sentinel.md@@ -0,0 +1,6 @@+---+'@truefoundry/trueforge': minor+'@truefoundry/trueforge-core': minor+---++Add Redis Sentinel + TLS with exclusive transport fail-fast (`REDIS_CONNECTION` DU). Redis is optional at config load for controller/migrate; server still requires it at connect. Sentinel shared-client errors no longer stop the peering heartbeat.diff --git a/charts/trueforge/Chart.yaml b/charts/trueforge/Chart.yamlindex d1abde1bf..7150ae069 100644--- a/charts/trueforge/Chart.yaml+++ b/charts/trueforge/Chart.yaml@@ -4,7 +4,7 @@ description: TrueForge server (API + UI) served from a single container image.type: application# version / appVersion are maintained on main (bot chart-release PR or human).# Publishing is gated by git tag charts/trueforge@<version> (must match version).-version: "0.2.2"+version: "0.2.3"appVersion: "0.2.1"kubeVersion: ">=1.25.0-0"home: https://truefoundry.comdiff --git a/charts/trueforge/README.md b/charts/trueforge/README.mdindex eaeb13c22..d80dc2068 100644--- a/charts/trueforge/README.md+++ b/charts/trueforge/README.md@@ -38,7 +38,7 @@ docs.Postgres and Redis ship as **bundled** dependencies (the Bitnami `postgresql`and `redis` charts, pulled from the public Bitnami OCI archive and pinned by`Chart.lock`). They are enabled by default, so a basic install needs **no-required values**. The chart wires the server's `POSTGRES_*` and `REDIS_URL`+required values**. The chart wires the server's `POSTGRES_*` and `REDIS_*`env to the bundled services automatically.The Bitnami **charts** are still public, but the **container images** they@@ -213,26 +213,33 @@ externalPostgres:The server always runs peered (`STANDALONE=false`), so Redis is always required.Bundled by default (`redis.enabled=true`, **auth disabled** — fine only whenRedis stays unreachable outside the cluster trust boundary). To use an-**external** Redis, set `redis.enabled=false` and provide `externalRedis.url`-as a string or `valueFrom.secretKeyRef`:+**external** Redis, set `redis.enabled=false`, `externalRedis.enabled=true`, and exactly one+of `externalRedis.url`, `externalRedis.host`, or Sentinel. Fields accept a string or+`valueFrom.secretKeyRef`. url, host, and sentinel are mutually exclusive:```yamlredis:enabled: falseexternalRedis:- url:- valueFrom:- secretKeyRef:- name: my-redis-secret- key: redis-url+ enabled: true+ url: redis://:[email protected]:6379+ # or host + auth (not with url):+ # host: redis-master.databases.svc+ # port: 6379+ # auth:+ # password:+ # valueFrom:+ # secretKeyRef:+ # name: my-redis-secret+ # key: redis-password````redis.nameOverride` defaults to `trueforge-redis` so bundled Redis objects donot share names with other Redis chart dependencies when this chart is adependency of some other chart.-For passworded Redis, prefer an external instance and load `REDIS_URL` via-`valueFrom`.+For passworded Redis, prefer an external instance and set `externalRedis.url` or+`externalRedis.auth` (and TLS/Sentinel as needed) via string or `valueFrom`.## OIDC@@ -308,8 +315,10 @@ chart does **not** create Secrets for chart-owned fields — supplyFields that accept string | `valueFrom.secretKeyRef`:`externalPostgres.host`, `externalPostgres.port`, `externalPostgres.database`,-`externalPostgres.user`, `externalPostgres.password`, `externalRedis.url`,-`configs.oidc.clientSecret`.+`externalPostgres.user`, `externalPostgres.password`, `externalRedis.url` / `host` / `auth`,+`externalRedis.tls` (`caCert`, `cert`, `key`, `keyPassphrase`, `serverName`),+`externalRedis.sentinel.auth`, `configs.oidc.clientSecret`.+`externalRedis.tls.rejectUnauthorized` is a plain boolean (default true).`configs.oidc.issuerUrl` and `clientId` are plain strings only.**Bundled Postgres password** still uses Bitnami's API (`postgresql.auth.existingSecret`,@@ -395,7 +404,7 @@ also sets the `/tmp` `emptyDir.sizeLimit`.- **Enable `configs.oidc`** — leaving it off grants shared admin to anyone who can reach the server.- **Replace the `apiKey` placeholder** — create a Secret for `TRUEFORGE_API_KEY` and set `apiKey.valueFrom.secretKeyRef` (do not leave `placeholder-value-please-generate-your-own`).- **Replace the bundled Postgres password** (`trueforge`) or set `postgresql.auth.existingSecret`.-- Treat bundled Redis (`redis.auth.enabled: false`) as cluster-internal only, or switch to external passworded Redis via `externalRedis.url`.+- Treat bundled Redis (`redis.auth.enabled: false`) as cluster-internal only, or switch to external passworded Redis via `externalRedis`.- Set `server.publicBaseUrl` to the real public application URL before using MCP OAuth or OIDC (include a pathname when the UI is served under a stripped prefix).- Prefer `valueFrom.secretKeyRef` for Postgres password, Redis URL, and OIDC client secret; do not commit secrets in values files.- Prefer external managed Postgres/Redis over the bundled subcharts for production HA.diff --git a/charts/trueforge/templates/NOTES.txt b/charts/trueforge/templates/NOTES.txtindex 13e1114df..038303d86 100644--- a/charts/trueforge/templates/NOTES.txt+++ b/charts/trueforge/templates/NOTES.txt@@ -39,7 +39,7 @@ before any shared / public deploy.{{- if and .Values.redis.enabled (not .Values.redis.auth.enabled) }}WARNING: Bundled Redis has auth disabled. Keep it cluster-internal (NetworkPolicy)-or switch to external passworded Redis via externalRedis.url.+or switch to external passworded Redis via externalRedis (url/host/auth).{{- end }}{{- if and .Values.configs.oidc.enabled (not .Values.server.publicBaseUrl) }}diff --git a/charts/trueforge/templates/_helpers.tpl b/charts/trueforge/templates/_helpers.tplindex b6257e67e..060026b7c 100644--- a/charts/trueforge/templates/_helpers.tpl+++ b/charts/trueforge/templates/_helpers.tpl@@ -209,7 +209,7 @@ postgresql subchart (existingSecret override or <release>-postgresql).{{- end }}{{/*-Bitnami redis fullname (mirrors common.names.fullname) so REDIS_URL tracks+Bitnami redis fullname (mirrors common.names.fullname) so REDIS_HOST tracksredis.nameOverride / redis.fullnameOverride.*/}}{{- define "trueforge.redis.fullname" -}}@@ -433,20 +433,88 @@ fields, wires bundled Postgres/Redis, optional OIDC, then server.extraEnv.{{- $env = append $env (dict "name" "STANDALONE" "value" "false") -}}{{- $env = append $env (dict "name" "GRACEFUL_TIMEOUT_SECONDS" "value" (.Values.server.gracefulTimeoutSeconds | toString)) -}}+{{- $externalRedis := .Values.externalRedis | default dict -}}+{{- $sentinel := $externalRedis.sentinel | default dict -}}+{{- $auth := $externalRedis.auth | default dict -}}+{{- $tls := $externalRedis.tls | default dict -}}+{{- $sentinelEnabled := eq (toString ($sentinel.enabled | default false)) "true" -}}+{{- $tlsEnabled := eq (toString ($tls.enabled | default false)) "true" -}}+{{- if and .Values.redis.enabled $externalRedis.enabled -}}+{{- fail "redis.enabled and externalRedis.enabled are mutually exclusive" -}}+{{- end -}}{{- if .Values.redis.enabled -}}{{- $env = append $env (dict "name" "REDIS_URL" "value" (include "trueforge.redis.bundledUrl" .)) -}}-{{- else -}}-{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_URL" "field" "externalRedis.url" "value" .Values.externalRedis.url) | fromJson) -}}-{{- $sentinel := .Values.externalRedis.sentinel | default dict -}}-{{- if $sentinel.enabled -}}-{{- $_ := required "externalRedis.sentinel.hosts is required when externalRedis.sentinel.enabled is true" $sentinel.hosts -}}+{{- else if $externalRedis.enabled -}}+{{- if and $sentinelEnabled (or $externalRedis.url $externalRedis.host) -}}+{{- fail "externalRedis.sentinel cannot be combined with externalRedis.url or externalRedis.host" -}}+{{- end -}}+{{- if and $externalRedis.url $externalRedis.host -}}+{{- fail "externalRedis.url and externalRedis.host are mutually exclusive" -}}+{{- end -}}+{{- if and (not $sentinelEnabled) (not $externalRedis.url) (not $externalRedis.host) -}}+{{- fail "externalRedis.url or externalRedis.host is required when using external Redis without sentinel" -}}+{{- end -}}+{{- if and (not $sentinelEnabled) $externalRedis.url (or $auth.username $auth.password (ne (($externalRedis.db | default 0) | toString) "0")) -}}+{{- fail "externalRedis.auth and a non-zero externalRedis.db are ignored when externalRedis.url is set; encode them in the url or use host mode" -}}+{{- end -}}+{{- if $sentinelEnabled -}}+{{- $_ := required "externalRedis.sentinel.nodes is required when externalRedis.sentinel.enabled is true" (join "," $sentinel.nodes) -}}{{- $_ := required "externalRedis.sentinel.masterName is required when externalRedis.sentinel.enabled is true" $sentinel.masterName -}}-{{- $env = append $env (dict "name" "REDIS_SENTINEL_HOSTS" "value" $sentinel.hosts) -}}+{{- $env = append $env (dict "name" "REDIS_SENTINEL_ENABLED" "value" "true") -}}+{{- $env = append $env (dict "name" "REDIS_SENTINEL_NODES" "value" (join "," $sentinel.nodes)) -}}{{- $env = append $env (dict "name" "REDIS_SENTINEL_MASTER_NAME" "value" $sentinel.masterName) -}}-{{- if $sentinel.password -}}-{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_SENTINEL_PASSWORD" "field" "externalRedis.sentinel.password" "value" $sentinel.password) | fromJson) -}}+{{- $env = append $env (dict "name" "REDIS_DB" "value" (($externalRedis.db | default 0) | toString)) -}}+{{- if $auth.username -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_USERNAME" "field" "externalRedis.auth.username" "value" $auth.username) | fromJson) -}}+{{- end -}}+{{- if $auth.password -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_PASSWORD" "field" "externalRedis.auth.password" "value" $auth.password) | fromJson) -}}+{{- end -}}+{{- $sentinelAuth := $sentinel.auth | default dict -}}+{{- if $sentinelAuth.username -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_SENTINEL_USERNAME" "field" "externalRedis.sentinel.auth.username" "value" $sentinelAuth.username) | fromJson) -}}+{{- end -}}+{{- if $sentinelAuth.password -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_SENTINEL_PASSWORD" "field" "externalRedis.sentinel.auth.password" "value" $sentinelAuth.password) | fromJson) -}}+{{- end -}}+{{- else if $externalRedis.url -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_URL" "field" "externalRedis.url" "value" $externalRedis.url) | fromJson) -}}+{{- else -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_HOST" "field" "externalRedis.host" "value" $externalRedis.host) | fromJson) -}}+{{- $env = append $env (dict "name" "REDIS_PORT" "value" (($externalRedis.port | default 6379) | toString)) -}}+{{- $env = append $env (dict "name" "REDIS_DB" "value" (($externalRedis.db | default 0) | toString)) -}}+{{- if $auth.username -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_USERNAME" "field" "externalRedis.auth.username" "value" $auth.username) | fromJson) -}}+{{- end -}}+{{- if $auth.password -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_PASSWORD" "field" "externalRedis.auth.password" "value" $auth.password) | fromJson) -}}+{{- end -}}+{{- end -}}+{{- $env = append $env (dict "name" "REDIS_TLS_ENABLED" "value" (ternary "true" "false" $tlsEnabled)) -}}+{{- if $tlsEnabled -}}+{{- $rejectUnauthorized := true -}}+{{- if hasKey $tls "rejectUnauthorized" -}}+{{- $rejectUnauthorized = eq (toString $tls.rejectUnauthorized) "true" -}}+{{- end -}}+{{- $env = append $env (dict "name" "REDIS_TLS_REJECT_UNAUTHORIZED" "value" (ternary "true" "false" $rejectUnauthorized)) -}}+{{- if $tls.caCert -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_TLS_CA_CERT" "field" "externalRedis.tls.caCert" "value" $tls.caCert) | fromJson) -}}{{- end -}}+{{- if $tls.serverName -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_TLS_SERVERNAME" "field" "externalRedis.tls.serverName" "value" $tls.serverName) | fromJson) -}}{{- end -}}+{{- if $tls.cert -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_TLS_CERT" "field" "externalRedis.tls.cert" "value" $tls.cert) | fromJson) -}}+{{- end -}}+{{- if $tls.key -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_TLS_KEY" "field" "externalRedis.tls.key" "value" $tls.key) | fromJson) -}}+{{- end -}}+{{- if $tls.keyPassphrase -}}+{{- $env = append $env (include "trueforge.env.fromStringOrValueFrom" (dict "name" "REDIS_TLS_KEY_PASSPHRASE" "field" "externalRedis.tls.keyPassphrase" "value" $tls.keyPassphrase) | fromJson) -}}+{{- end -}}+{{- end -}}+{{- else -}}+{{- fail "set redis.enabled or externalRedis.enabled" -}}{{- end -}}{{- if .Values.postgresql.enabled -}}diff --git a/charts/trueforge/values.yaml b/charts/trueforge/values.yamlindex ff99c111f..54aa65a58 100644--- a/charts/trueforge/values.yaml+++ b/charts/trueforge/values.yaml@@ -226,9 +226,9 @@ postgresql:size: 8Gi# --- Bundled Redis (Bitnami subchart) ------------------------------------------# Rendered only when redis.enabled is true. Auth is disabled (dev default) so-# REDIS_URL is a plain redis://host:6379. Safe only while Redis stays inside-# the cluster trust boundary. For passworded Redis, disable the subchart and-# use externalRedis with a Secret.+# Auth is disabled (dev default) so REDIS_HOST points at the bundled master+# without credentials. Safe only while Redis stays inside the cluster trust+# boundary. For passworded Redis, disable the subchart and use externalRedis.redis:enabled: truearchitecture: standalone@@ -271,23 +271,40 @@ externalPostgres:sslKeyPath: ""sslCaPath: ""externalRedis:- # Full connection URL, e.g. redis://:[email protected]:6379.- # string, or { valueFrom: { secretKeyRef: { name, key } } }. Prefer valueFrom in prod.+ ## Use an external Redis instead of the bundled Redis (`redis.enabled` must be false).+ ## Requires `enabled: true`, then exactly one transport: url, host, or sentinel.+ enabled: false+ ## Full connection URL, e.g. redis://:[email protected]:6379.+ ## string or valueFrom. Mutually exclusive with host and sentinel.url: ""- # Redis Sentinel connection settings, injected as REDIS_SENTINEL_* env.- # url:- # valueFrom:- # secretKeyRef:- # name: my-redis-secret- # key: redis-url+ ## Standalone Redis host. Mutually exclusive with url and sentinel.+ host: ""+ port: 6379+ db: 0+ auth:+ ## string or valueFrom (host/sentinel modes; put credentials in url when using url)+ username: ""+ password: ""+ tls:+ enabled: false+ ## Path or inline PEM+ caCert: ""+ ## Verify the server certificate. Set false only for self-signed certs without a CA.+ rejectUnauthorized: true+ serverName: ""+ cert: ""+ key: ""+ keyPassphrase: ""sentinel:+ ## Mutually exclusive with url and host. Requires nodes + masterName when enabled.enabled: false- # Comma-separated host:port list -> REDIS_SENTINEL_HOSTS.- hosts: ""- # Monitored master name -> REDIS_SENTINEL_MASTER_NAME.+ ## Array of host:port entries+ nodes: []masterName: ""- # string or valueFrom -> REDIS_SENTINEL_PASSWORD.- password: ""+ auth:+ ## string or valueFrom+ username: ""+ password: ""# --- Bundled dev sandbox server (tfy-sandbox-server subchart) -------------------# Managed by the parent chart; off for standalone installs. When true, the# subchart is deployed and the app gets TRUEFOUNDRY_SANDBOX_* env pointing at it.diff --git a/packages/trueforge-core/src/request-reply/client.ts b/packages/trueforge-core/src/request-reply/client.tsindex 99d9fee5b..b580928bf 100644--- a/packages/trueforge-core/src/request-reply/client.ts+++ b/packages/trueforge-core/src/request-reply/client.ts@@ -1,7 +1,7 @@import { randomUUID } from 'node:crypto';import { performance } from 'node:perf_hooks';-import type { RedisClientType } from 'redis';import { NoResponderError, RequestTimeoutError } from './errors';+import type { RedisClient } from './redisClient';import type { JSONReply, JSONValue, PublishedRequest, RequestEnvelope } from './types';import { jsonReplySchema } from './types';import { heartbeatKey, replyKey, requestChannel, sleep } from './utils';@@ -28,7 +28,7 @@ function parseReplyPayload(raw: string) {return jsonReplySchema.parse(parsed);}-async function getDelReply(redisClient: RedisClientType, rKey: string): Promise<JSONReply | null> {+async function getDelReply(redisClient: RedisClient, rKey: string): Promise<JSONReply | null> {const raw = await redisClient.getDel(rKey);if (raw === null) {return null;@@ -48,7 +48,7 @@ export async function redisRequest<T extends JSONValue>({request,options,}: {- redis: RedisClientType;+ redis: RedisClient;executorId: string;path: string;request: RequestEnvelope<T>;diff --git a/packages/trueforge-core/src/request-reply/executor.ts b/packages/trueforge-core/src/request-reply/executor.tsindex 6b2e774d5..b1936e8e9 100644--- a/packages/trueforge-core/src/request-reply/executor.ts+++ b/packages/trueforge-core/src/request-reply/executor.ts@@ -1,9 +1,9 @@import { randomUUID } from 'node:crypto';-import type { RedisClientType } from 'redis';import type { Logger } from 'winston';import z from 'zod';import { extractErrorLogFields } from '../core/util/errorLogFields';import { ReplyError } from './errors';+import type { RedisClient } from './redisClient';import type { JSONReply, RequestHandler } from './types';import { publishedRequestSchema } from './types';import { heartbeatKey, requestChannel } from './utils';@@ -44,8 +44,8 @@ export class RequestReplyExecutor {readonly executorId: string;/** `tfg:rr:req:<executorId>` — the channel this executor subscribes to. */readonly channel: string;- private readonly redis: RedisClientType;- private readonly subscriberClient: RedisClientType;+ private readonly redis: RedisClient;+ private readonly subscriberClient: RedisClient;private readonly logger: Logger;private readonly heartbeatIntervalMs: number;private readonly heartbeatTtlMs: number;@@ -74,9 +74,9 @@ export class RequestReplyExecutor {}: {executorId: string;/** Connected command client, used only for SET (reply + heartbeat). Caller owns its lifecycle. */- redis: RedisClientType;+ redis: RedisClient;/** Connected client to SUBSCRIBE on (duplicate or Sentinel). Caller owns its lifecycle. */- subscriberClient: RedisClientType;+ subscriberClient: RedisClient;requestHandler: RequestHandler;onError?: RequestReplyErrorHandler | undefined;logger: Logger;@@ -108,7 +108,7 @@ export class RequestReplyExecutor {...extractErrorLogFields(err),});this.onError?.(err, { executorId: this.executorId, channel: this.channel });- this.stopHeartbeat();+ // error is often transient; the client can reconnect and fire ready again. Do not stop the heartbeat.});this.subscriberClient.on('end', () => {this.logger.warn('[RequestReplyExecutor] Subscriber connection ended', { executorId: this.executorId });diff --git a/packages/trueforge-core/src/request-reply/index.ts b/packages/trueforge-core/src/request-reply/index.tsindex f8937ebfc..bf5cfd5cf 100644--- a/packages/trueforge-core/src/request-reply/index.ts+++ b/packages/trueforge-core/src/request-reply/index.ts@@ -10,6 +10,7 @@ export type { SendRequestOptions } from './client';export { NoResponderError, ReplyError, RequestTimeoutError } from './errors';export { RequestReplyExecutor } from './executor';export type { RequestReplyErrorHandler, RunExecutorOptions } from './executor';+export type { RedisClient } from './redisClient';export { RequestReplyRouter } from './router';export type { RouteHandler } from './router';export { jsonReplySchema } from './types';diff --git a/packages/trueforge-core/src/request-reply/redisClient.ts b/packages/trueforge-core/src/request-reply/redisClient.tsnew file mode 100644index 000000000..26fcbbee5--- /dev/null+++ b/packages/trueforge-core/src/request-reply/redisClient.ts@@ -0,0 +1,4 @@+import type { RedisClientType, RedisSentinelType } from 'redis';++/** Standalone or Sentinel client for request-reply / command use. */+export type RedisClient = RedisClientType | RedisSentinelType;diff --git a/packages/trueforge/.env.example b/packages/trueforge/.env.exampleindex daf3e77e6..43161d6ab 100644--- a/packages/trueforge/.env.example+++ b/packages/trueforge/.env.example@@ -58,10 +58,38 @@ PORT=8790## Windows: %LOCALAPPDATA%\trueforge\Data\db\db.sqlite# SQLITE_PATH=-## Used when STANDALONE=false (ignored in standalone). Redis connection URL for-## executor peering. Defaults to redis://localhost:6379 if unset.+## Used when STANDALONE=false for the server (ignored in standalone; optional for+## controller / migrate). Redis transports are mutually exclusive — set exactly+## one of REDIS_URL, REDIS_HOST, or Sentinel.## `docker-compose.yml` overrides this to redis://redis:6379 for the server container.REDIS_URL=redis://localhost:6379+# Or host-based (mutually exclusive with REDIS_URL / Sentinel):+# REDIS_HOST=localhost+# REDIS_PORT=6379+# REDIS_DB=0+# REDIS_USERNAME=+# REDIS_PASSWORD=++## Redis TLS (STANDALONE=false only). Applies to data nodes and Sentinel sockets.+# REDIS_TLS_ENABLED=false+# REDIS_TLS_CA_CERT=+# REDIS_TLS_REJECT_UNAUTHORIZED=true+# REDIS_TLS_SERVERNAME=+# REDIS_TLS_CERT=+# REDIS_TLS_KEY=+# REDIS_TLS_KEY_PASSPHRASE=++## Redis Sentinel (STANDALONE=false only). Mutually exclusive with REDIS_URL / REDIS_HOST.+## When enabled, nodes + master name are required; REDIS_USERNAME/PASSWORD apply to data nodes.+# REDIS_SENTINEL_ENABLED=false+# REDIS_SENTINEL_NODES=sentinel-0:26379,sentinel-1:26379+# REDIS_SENTINEL_MASTER_NAME=mymaster+# REDIS_SENTINEL_USERNAME=+# REDIS_SENTINEL_PASSWORD=++## Redis client timeouts (STANDALONE=false only). Optional.+# REDIS_CONNECT_TIMEOUT_MS=20000+# REDIS_PING_INTERVAL_MS=5000## Redis request/reply peering knobs (STANDALONE=false only). All optional.## Max ms to wait for a peer executor's reply before failing with 424. Default 60000.diff --git a/packages/trueforge/src/apis/sessions.ts b/packages/trueforge/src/apis/sessions.tsindex d238c48e0..cbdf0a944 100644--- a/packages/trueforge/src/apis/sessions.ts+++ b/packages/trueforge/src/apis/sessions.ts@@ -14,11 +14,11 @@ import { extractErrorLogFields } from '@truefoundry/trueforge-core/core';import {redisRequest,RequestTimeoutError,+ type RedisClient,type RouteHandler as RequestReplyRouteHandler,type RequestReplyRouter,} from '@truefoundry/trueforge-core/request-reply';import type { Context } from 'hono';-import type { RedisClientType } from 'redis';import type { Logger } from 'winston';import { z } from 'zod';import type { Authorizer } from '../auth/authorizer';@@ -81,8 +81,8 @@ export interface SessionsRouterDeps {resolveSkillStore: ResolveSkillStore;resolveAgentStore: (c: Context) => IAgentStore;resolveSandboxProviderStore: (c: Context) => ISandboxProviderStore;+ redis?: RedisClient | undefined;resolveWebSearchProviderStore: (c: Context) => IWebSearchProviderStore;- redis?: RedisClientType | undefined;requestReplyRouter: RequestReplyRouter;resolveRequestContext: ResolveRequestContext;logger: Logger;@@ -128,7 +128,7 @@ export interface CancelTurnDeps {activeTurns: ActiveTurnRegistry;session: Pick<SessionHandle, 'session_id' | 'freezeTurn'>;sessionStore: Pick<ISessionStore, 'getTurn'>;- redis?: RedisClientType | undefined;+ redis?: RedisClient | undefined;logger: Pick<Logger, 'warn'>;}diff --git a/packages/trueforge/src/app.ts b/packages/trueforge/src/app.tsindex cbfecdedb..d893e2f46 100644--- a/packages/trueforge/src/app.ts+++ b/packages/trueforge/src/app.ts@@ -3,12 +3,11 @@ import { swaggerUI } from '@hono/swagger-ui';import { OpenAPIHono, z } from '@hono/zod-openapi';import type { ISessionStore, Sessions, TurnStreamingEvent } from '@truefoundry/trueforge-core/agent-session';import { extractErrorLogFields } from '@truefoundry/trueforge-core/core';-import type { RequestReplyRouter } from '@truefoundry/trueforge-core/request-reply';+import type { RedisClient, RequestReplyRouter } from '@truefoundry/trueforge-core/request-reply';import type { Context, ErrorHandler, MiddlewareHandler } from 'hono';import { bodyLimit } from 'hono/body-limit';import { HTTPException } from 'hono/http-exception';import type { Configuration } from 'openid-client';-import type { RedisClientType } from 'redis';import type { Logger } from 'winston';import { createAgentImportRouter } from './apis/agentImport';import { createAgentsRouter } from './apis/agents';@@ -211,7 +210,7 @@ export interface ServerDeps<TTransaction> {sessions: Sessions;activeTurns: ActiveTurnRegistry;/** Primary Redis client (server-owned); undefined in standalone mode. */- redis?: RedisClientType | undefined;+ redis?: RedisClient | undefined;/** Request-reply dispatch table served by this replica's executor. */requestReplyRouter: RequestReplyRouter;/** Hands out each turn's resumable event stream to the create and subscribe handlers. */diff --git a/packages/trueforge/src/config.ts b/packages/trueforge/src/config.tsindex 6426caa2a..6c89a6fea 100644--- a/packages/trueforge/src/config.ts+++ b/packages/trueforge/src/config.ts@@ -8,10 +8,10 @@** `STANDALONE` is a discriminated mode selector:* - `true` (default): SQLite only; no Redis / executor peering.- * - `false`: Postgres + Redis (defaults to local trueforge credentials /- * `redis://localhost:6379`).+ * - `false`: Postgres; Redis required for the server (optional for controller / migrate).*/import { existsSync, readFileSync } from 'node:fs';+import { isIPv6 } from 'node:net';import os from 'node:os';import path from 'node:path';import { fileURLToPath } from 'node:url';@@ -19,6 +19,10 @@ import { fileURLToPath } from 'node:url';import envPaths from 'env-paths';import { z } from 'zod';+import { parseRedisSentinelNodes, type RedisConnection } from './runtime/redis';++export type { RedisConnection };+const DEFAULT_PORT = 8790;/** Loopback default; container images set HOST=0.0.0.0 so probes and Service traffic reach the process. */const DEFAULT_HOST = 'localhost';@@ -40,7 +44,6 @@ const DEFAULT_POSTGRES_HOST = 'localhost';const DEFAULT_POSTGRES_PORT = 5432;/** Default Postgres schema for app tables + Kysely migration bookkeeping. */export const DEFAULT_POSTGRES_SCHEMA = 'trueforge';-const DEFAULT_REDIS_URL = 'redis://localhost:6379';/** Unquoted Postgres identifier: letter/underscore start, then alnum/underscore, ≤63 chars. */const POSTGRES_SCHEMA_NAME_RE = /^[a-z_][a-z0-9_]{0,62}$/;/**@@ -214,6 +217,18 @@ function parsePositiveInt(options: { envKey: string; raw: string | undefined; dereturn value;}+function parseNonNegativeInt(options: { envKey: string; raw: string | undefined; defaultValue: number }): number {+ const { envKey, raw, defaultValue } = options;+ if (raw === undefined || raw.trim() === '') {+ return defaultValue;+ }+ const value = Number(raw);+ if (!Number.isInteger(value) || value < 0) {+ throw new Error(`Environment variable ${envKey} must be a non-negative integer, got "${raw}"`);+ }+ return value;+}+/** Parses a boolean env var; anything but `true`/`false` throws instead of reading as `false`. */function parseBoolean(options: { envKey: string; raw: string | undefined; defaultValue: boolean }): boolean {const { envKey, raw, defaultValue } = options;@@ -349,15 +364,122 @@ function resolveCodeModeSocketParent(): string {return path.join(os.tmpdir(), 'tf_cms');}-/** Redis peering URL for distributed mode. Env: `REDIS_URL`. */-function resolveRedisUrl(): string {- const raw = getEnv('REDIS_URL', { defaultValue: DEFAULT_REDIS_URL }) ?? DEFAULT_REDIS_URL;- if (raw.trim() === '') {- throw new Error('Environment variable REDIS_URL must be non-empty when STANDALONE=false.');+/** Env string as-is, or undefined when unset/empty (no trimming). */+function optionalNonEmptyEnv(envKey: string): string | undefined {+ const raw = getEnv(envKey);+ if (raw === undefined || raw === '') {+ return undefined;}return raw;}+/**+ * Build a standalone `redis://` URL from discrete host fields.+ * TLS stays on socket options (`REDIS_TLS_*`), not the URL scheme — CA/cert/key+ * cannot be expressed in `rediss://`, and mixing scheme + socket TLS is ambiguous.+ * IPv6 hosts are bracketed so `new URL()` accepts them; username/password are+ * percent-encoded by the URL setters.+ */+export function buildRedisStandaloneUrl(parts: {+ host: string;+ port: number;+ database: number;+ username: string | undefined;+ password: string | undefined;+}): string {+ const bareHost = parts.host.startsWith('[') && parts.host.endsWith(']') ? parts.host.slice(1, -1) : parts.host;+ const host = isIPv6(bareHost) ? `[${bareHost}]` : bareHost;+ const url = new URL(`redis://${host}:${String(parts.port)}/${String(parts.database)}`);+ if (parts.username !== undefined) {+ url.username = parts.username;+ }+ if (parts.password !== undefined) {+ url.password = parts.password;+ }+ return url.href;+}++/**+ * Exactly one Redis transport when configured.+ * Transports are mutually exclusive: Sentinel, `REDIS_URL`, or `REDIS_HOST`.+ * Returns undefined when none are set so controller / migrate can boot without Redis;+ * the server fails at connect time if still unset.+ */+function resolveRedisConnection(): RedisConnection | undefined {+ const sentinelEnabled = parseBoolean({+ envKey: 'REDIS_SENTINEL_ENABLED',+ raw: getEnv('REDIS_SENTINEL_ENABLED'),+ defaultValue: false,+ });+ const url = optionalNonEmptyEnv('REDIS_URL');+ const host = optionalNonEmptyEnv('REDIS_HOST');+ const database = parseNonNegativeInt({+ envKey: 'REDIS_DB',+ raw: getEnv('REDIS_DB'),+ defaultValue: 0,+ });+ const username = optionalNonEmptyEnv('REDIS_USERNAME');+ const password = getEnv('REDIS_PASSWORD');+ const passwordOrUndefined = password === undefined || password === '' ? undefined : password;++ if (sentinelEnabled) {+ const nodes = optionalNonEmptyEnv('REDIS_SENTINEL_NODES');+ const masterName = optionalNonEmptyEnv('REDIS_SENTINEL_MASTER_NAME');+ if (!nodes || !masterName || parseRedisSentinelNodes(nodes).length === 0) {+ throw new Error(+ 'REDIS_SENTINEL_ENABLED=true requires non-empty REDIS_SENTINEL_NODES and REDIS_SENTINEL_MASTER_NAME.',+ );+ }+ if (url !== undefined || host !== undefined) {+ throw new Error(+ 'Redis transports are mutually exclusive: unset REDIS_URL and REDIS_HOST when REDIS_SENTINEL_ENABLED=true.',+ );+ }+ const sentinelUsername = optionalNonEmptyEnv('REDIS_SENTINEL_USERNAME');+ const sentinelPasswordRaw = getEnv('REDIS_SENTINEL_PASSWORD');+ const sentinelPassword =+ sentinelPasswordRaw === undefined || sentinelPasswordRaw === '' ? undefined : sentinelPasswordRaw;+ return {+ mode: 'sentinel',+ nodes,+ masterName,+ database,+ ...(username !== undefined ? { username } : {}),+ ...(passwordOrUndefined !== undefined ? { password: passwordOrUndefined } : {}),+ ...(sentinelUsername !== undefined ? { sentinelUsername } : {}),+ ...(sentinelPassword !== undefined ? { sentinelPassword } : {}),+ };+ }++ if (url !== undefined && host !== undefined) {+ throw new Error(+ 'Redis transports are mutually exclusive: set only one of REDIS_URL or REDIS_HOST (or enable Sentinel).',+ );+ }+ if (url !== undefined) {+ return { mode: 'standalone', url };+ }+ if (host !== undefined) {+ const port = parsePositiveInt({+ envKey: 'REDIS_PORT',+ raw: getEnv('REDIS_PORT'),+ defaultValue: 6379,+ });+ return {+ mode: 'standalone',+ url: buildRedisStandaloneUrl({+ host,+ port,+ database,+ username,+ password: passwordOrUndefined,+ }),+ };+ }++ return undefined;+}+/*** Postgres connection string for distributed mode.* Prefers `DATABASE_URL` when set (Railway / managed Postgres); otherwise builds from `POSTGRES_*`.@@ -744,8 +866,30 @@ export type DistributedServerConfiguration = SharedServerConfiguration & {* Env: `POSTGRES_SCHEMA`. Default `trueforge`.*/POSTGRES_SCHEMA: string;- /** Peering URL shared by all replicas. Env: `REDIS_URL`. Default `redis://localhost:6379`. */- REDIS_URL: string;+ /**+ * Resolved Redis transport (exactly one of url / host / sentinel), or undefined.+ * Required for the server process; optional for controller / migrate (no peering).+ * Env: mutually exclusive `REDIS_URL`, `REDIS_HOST`, or `REDIS_SENTINEL_*`.+ */+ REDIS_CONNECTION: RedisConnection | undefined;+ /** Socket connect timeout for Redis clients. Env: `REDIS_CONNECT_TIMEOUT_MS`. Default 20000. */+ REDIS_CONNECT_TIMEOUT_MS: number;+ /** Client ping interval for Redis keepalive. Env: `REDIS_PING_INTERVAL_MS`. Default 5000. */+ REDIS_PING_INTERVAL_MS: number;+ /** Enable TLS for Redis (and Sentinel when used). Env: `REDIS_TLS_ENABLED`. Default false. */+ REDIS_TLS_ENABLED: boolean;+ /** CA cert path or inline PEM. Env: `REDIS_TLS_CA_CERT`. */+ REDIS_TLS_CA_CERT: string | undefined;+ /** Verify server cert. Env: `REDIS_TLS_REJECT_UNAUTHORIZED`. Default true. */+ REDIS_TLS_REJECT_UNAUTHORIZED: boolean;+ /** TLS SNI server name. Env: `REDIS_TLS_SERVERNAME`. */+ REDIS_TLS_SERVERNAME: string | undefined;+ /** Client cert path or inline PEM (mTLS). Env: `REDIS_TLS_CERT`. */+ REDIS_TLS_CERT: string | undefined;+ /** Client key path or inline PEM (mTLS). Env: `REDIS_TLS_KEY`. */+ REDIS_TLS_KEY: string | undefined;+ /** Client key passphrase. Env: `REDIS_TLS_KEY_PASSPHRASE`. */+ REDIS_TLS_KEY_PASSPHRASE: string | undefined;/*** OIDC configuration for server authentication.* Undefined means browser login is disabled.@@ -990,7 +1134,32 @@ const configuration: ServerConfiguration = standalonedefaultValue: 60_000,}),POSTGRES_SCHEMA: parsePostgresSchema(getEnv('POSTGRES_SCHEMA')),- REDIS_URL: resolveRedisUrl(),+ REDIS_CONNECTION: resolveRedisConnection(),+ REDIS_CONNECT_TIMEOUT_MS: parsePositiveInt({+ envKey: 'REDIS_CONNECT_TIMEOUT_MS',+ raw: getEnv('REDIS_CONNECT_TIMEOUT_MS'),+ defaultValue: 20_000,+ }),+ REDIS_PING_INTERVAL_MS: parsePositiveInt({+ envKey: 'REDIS_PING_INTERVAL_MS',+ raw: getEnv('REDIS_PING_INTERVAL_MS'),+ defaultValue: 5_000,+ }),+ REDIS_TLS_ENABLED: parseBoolean({+ envKey: 'REDIS_TLS_ENABLED',+ raw: getEnv('REDIS_TLS_ENABLED'),+ defaultValue: false,+ }),+ REDIS_TLS_CA_CERT: getEnv('REDIS_TLS_CA_CERT'),+ REDIS_TLS_REJECT_UNAUTHORIZED: parseBoolean({+ envKey: 'REDIS_TLS_REJECT_UNAUTHORIZED',+ raw: getEnv('REDIS_TLS_REJECT_UNAUTHORIZED'),+ defaultValue: true,+ }),+ REDIS_TLS_SERVERNAME: getEnv('REDIS_TLS_SERVERNAME'),+ REDIS_TLS_CERT: getEnv('REDIS_TLS_CERT'),+ REDIS_TLS_KEY: getEnv('REDIS_TLS_KEY'),+ REDIS_TLS_KEY_PASSPHRASE: getEnv('REDIS_TLS_KEY_PASSPHRASE'),OIDC: resolveOIDCConfig(),AUTOMATICALLY_MOVE_TRUEFORGE_TABLES_FROM_PUBLIC_TO_TRUEFORGE_SCHEMA: parseBoolean({envKey: 'AUTOMATICALLY_MOVE_TRUEFORGE_TABLES_FROM_PUBLIC_TO_TRUEFORGE_SCHEMA',diff --git a/packages/trueforge/src/main.ts b/packages/trueforge/src/main.tsindex 3308c51e5..50a85f796 100644--- a/packages/trueforge/src/main.ts+++ b/packages/trueforge/src/main.ts@@ -56,9 +56,8 @@ import {type ISessionStore,type TurnStreamingEvent,} from '@truefoundry/trueforge-core/agent-session';-import { RequestReplyExecutor, RequestReplyRouter } from '@truefoundry/trueforge-core/request-reply';+import { RequestReplyExecutor, RequestReplyRouter, type RedisClient } from '@truefoundry/trueforge-core/request-reply';import type { Kysely, Transaction } from 'kysely';-import type { RedisClientType } from 'redis';import type { Logger } from 'winston';import { createServerApp } from './app';@@ -93,6 +92,7 @@ import type { IOAuthTokenStore } from './mcp/auth/types';import { PACKAGE_VERSION } from './packageVersion';import { ActiveTurnRegistry } from './runtime/activeTurns';import { EventSubscriptionRegistry } from './runtime/event-subscription';+import type { ConnectedRedis } from './runtime/redis';import { printStandaloneStartupBanner } from './startupBanner';import { InlineMcpServerStore } from './truefoundry/InlineMcpServerStore';import { parseInlineMcpServers, parseInlineSkills, X_TFG_MCP, X_TFG_SKILLS } from './truefoundry/inlineResources';@@ -136,7 +136,8 @@ interface ServerPersistence<TTransaction> {agentStore: IAgentStore<TTransaction>;turnSkillsResolverStore: Pick<ISkillStore<TTransaction>, 'resolveTurnSkills'>;destroyDb: () => Promise<void>;- redis: RedisClientType | undefined;+ /** Connected Redis (client + mode) for distributed peering; undefined in standalone. */+ redis: ConnectedRedis | undefined;/** One shared client for TrueFoundry store resolvers + auth; undefined when TrueFoundry mode is off. */serviceFoundryClient: TrueFoundryServiceFoundryServerClient | undefined;}@@ -379,7 +380,16 @@ async function createDistributedPersistence(options: {DATABASE_POOL_MAX: databasePoolMax,POSTGRES_STATEMENT_TIMEOUT_MS: statementTimeoutMs,POSTGRES_IDLE_IN_TRANSACTION_SESSION_TIMEOUT_MS: idleInTransactionSessionTimeoutMs,- REDIS_URL: redisUrl,+ REDIS_CONNECTION: redisConnection,+ REDIS_CONNECT_TIMEOUT_MS: redisConnectTimeoutMs,+ REDIS_PING_INTERVAL_MS: redisPingIntervalMs,+ REDIS_TLS_ENABLED: redisTlsEnabled,+ REDIS_TLS_CA_CERT: redisTlsCaCert,+ REDIS_TLS_REJECT_UNAUTHORIZED: redisTlsRejectUnauthorized,+ REDIS_TLS_SERVERNAME: redisTlsServerName,+ REDIS_TLS_CERT: redisTlsCert,+ REDIS_TLS_KEY: redisTlsKey,+ REDIS_TLS_KEY_PASSPHRASE: redisTlsKeyPassphrase,EXECUTOR_ID: executorId,} = configuration;@@ -423,6 +433,12 @@ async function createDistributedPersistence(options: {});await migrateToLatest(db);logger.info(`Executor id: ${executorId}`);+ if (redisConnection === undefined) {+ throw new Error(+ 'Set exactly one Redis transport for the server when STANDALONE=false: REDIS_URL, REDIS_HOST, or ' ++ 'REDIS_SENTINEL_ENABLED with REDIS_SENTINEL_NODES and REDIS_SENTINEL_MASTER_NAME.',+ );+ }const serviceFoundryClient = createServiceFoundryServerClient(logger);const tokenStore = new PostgresOAuthTokenStore(db);const modelProviderStore = new PostgresModelProviderStore(db);@@ -502,7 +518,21 @@ async function createDistributedPersistence(options: {agentStore,turnSkillsResolverStore,destroyDb: () => db.destroy(),- redis: await connectRedis({ url: redisUrl, logger }),+ redis: await connectRedis({+ connection: redisConnection,+ connectTimeoutMs: redisConnectTimeoutMs,+ pingIntervalMs: redisPingIntervalMs,+ logger,+ tls: {+ enabled: redisTlsEnabled,+ caCert: redisTlsCaCert,+ rejectUnauthorized: redisTlsRejectUnauthorized,+ serverName: redisTlsServerName,+ cert: redisTlsCert,+ key: redisTlsKey,+ keyPassphrase: redisTlsKeyPassphrase,+ },+ }),serviceFoundryClient,};}@@ -526,7 +556,7 @@ async function createServerRuntime<TTransaction>(persistence: ServerPersistence<const activeTurns = new ActiveTurnRegistry();const requestReplyRouter = new RequestReplyRouter();- const eventSubscriptions = new EventSubscriptionRegistry<TurnStreamingEvent>(redis);+ const eventSubscriptions = new EventSubscriptionRegistry<TurnStreamingEvent>(redis?.client);const sessions = new Sessions({ sessionStore });const oidc = isOidcConfigured(configuration) ? configuration.OIDC : undefined;@@ -626,7 +656,7 @@ async function createServerRuntime<TTransaction>(persistence: ServerPersistence<sessionMetricsStore,sessions,activeTurns,- redis,+ redis: redis?.client,requestReplyRouter,eventSubscriptions,logger,@@ -693,22 +723,26 @@ try {}// After createServerApp so every request-reply route is registered before- // the executor starts consuming messages. The executor needs a dedicated+ // the executor starts consuming messages. Standalone Redis needs a dedicated// subscriber connection (a subscribed client cannot issue normal commands);- // this process owns its lifecycle. Connect before init() so init() awaits- // the initial subscribe + heartbeat — the replica is reachable for peering- // before the HTTP server starts.- let requestReplySubscriber: RedisClientType | undefined;+ // Sentinel owns pub/sub on the shared client. Connect before init() so init()+ // awaits the initial subscribe + heartbeat — the replica is reachable for+ // peering before the HTTP server starts.+ let requestReplySubscriber: RedisClient | undefined;let requestReplyExecutor: RequestReplyExecutor | undefined;if (redis) {- requestReplySubscriber = redis.duplicate();- requestReplySubscriber.on('error', (error: Error) => {- logger.error('[RedisSubscriber] Client error', extractErrorLogFields(error));- });- await requestReplySubscriber.connect();+ if (redis.mode !== 'sentinel') {+ requestReplySubscriber = redis.client.duplicate();+ requestReplySubscriber.on('error', (error: Error) => {+ logger.error('[RedisSubscriber] Client error', extractErrorLogFields(error));+ });+ await requestReplySubscriber.connect();+ } else {+ requestReplySubscriber = redis.client;+ }requestReplyExecutor = new RequestReplyExecutor({executorId: configuration.EXECUTOR_ID,- redis,+ redis: redis.client,subscriberClient: requestReplySubscriber,requestHandler: requestReplyRouter.createRequestHandler(),logger,@@ -779,14 +813,20 @@ try {await activeTurns.shutdownAndWait(CancellationReason.Abandoned);await closed;// Stop serving peer requests (waits for in-flight replies), then close- // the clients this process owns: the subscriber duplicate and the primary.+ // Redis clients (Set dedupes when Sentinel shares one client for pub/sub).await requestReplyExecutor?.drain();- await requestReplySubscriber?.close().catch((error: unknown) => {- logger.warn('[Redis] Error closing subscriber client during shutdown', extractErrorLogFields(error));- });- await redis?.close().catch((error: unknown) => {- logger.warn('[Redis] Error closing client during shutdown', extractErrorLogFields(error));- });+ const redisClients = new Set<RedisClient>();+ if (requestReplySubscriber !== undefined) {+ redisClients.add(requestReplySubscriber);+ }+ if (redis !== undefined) {+ redisClients.add(redis.client);+ }+ for (const client of redisClients) {+ await client.close().catch((error: unknown) => {+ logger.warn('[Redis] Error closing client during shutdown', extractErrorLogFields(error));+ });+ }if (configuration.STANDALONE) {await removeCodeModeSocketParent(configuration.CODE_MODE_SOCKET_PARENT).catch((error: unknown) => {logger.warn('Error removing Code Mode socket parent during shutdown', extractErrorLogFields(error));diff --git a/packages/trueforge/src/runtime/event-subscription/index.ts b/packages/trueforge/src/runtime/event-subscription/index.tsindex 0485b4730..1fe701545 100644--- a/packages/trueforge/src/runtime/event-subscription/index.ts+++ b/packages/trueforge/src/runtime/event-subscription/index.ts@@ -1,4 +1,4 @@-import type { RedisClientType } from 'redis';+import type { RedisClient } from '@truefoundry/trueforge-core/request-reply';import { InMemoryEventStreamStore, InMemoryEventSubscription } from './inMemory';import { RedisEventSubscription } from './redis';@@ -47,7 +47,7 @@ export class EventSubscriptionRegistry<T extends object> {/** One store for the whole process so producers and subscribers share streams. */private readonly memoryStore = new InMemoryEventStreamStore<T>();- constructor(private readonly redis: RedisClientType | undefined) {}+ constructor(private readonly redis: RedisClient | undefined) {}get(streamId: string): EventSubscription<T> {if (this.redis) {diff --git a/packages/trueforge/src/runtime/event-subscription/redis.ts b/packages/trueforge/src/runtime/event-subscription/redis.tsindex 54b2bb8ff..f0ee5e8fb 100644--- a/packages/trueforge/src/runtime/event-subscription/redis.ts+++ b/packages/trueforge/src/runtime/event-subscription/redis.ts@@ -1,5 +1,5 @@+import type { RedisClient } from '@truefoundry/trueforge-core/request-reply';import { setTimeout as sleep } from 'node:timers/promises';-import type { RedisClientType } from 'redis';import {StreamGoneError,SUBSCRIBE_STREAM_THRESHOLD_MS,@@ -31,7 +31,7 @@ export class RedisEventSubscription<T extends object> implements EventSubscriptiprivate nextSequenceNumber = 1;constructor(- private readonly redis: RedisClientType,+ private readonly redis: RedisClient,private readonly streamId: string,) {}diff --git a/packages/trueforge/src/runtime/redis.ts b/packages/trueforge/src/runtime/redis.tsindex 3710c4db7..9563ba04b 100644--- a/packages/trueforge/src/runtime/redis.ts+++ b/packages/trueforge/src/runtime/redis.ts@@ -1,21 +1,227 @@/*** Primary Redis connection, owned by the server: created and connected at* boot, closed last during shutdown. Injected into the request-reply- * transport (which duplicates it only for its subscriber).+ * transport (which duplicates it only for its subscriber in standalone mode;+ * Sentinel shares the same client for pub/sub).*/+import { existsSync, readFileSync } from 'node:fs';+import { setTimeout as sleep } from 'node:timers/promises';+import { extractErrorLogFields } from '@truefoundry/trueforge-core/core';-import { createClient, type RedisClientType } from 'redis';+import { createClient, createSentinel, type RedisClientType, type RedisSentinelType } from 'redis';import type { Logger } from 'winston';-export async function connectRedis(input: { url: string; logger: Logger }): Promise<RedisClientType> {+const DEFAULT_SENTINEL_PORT = 26379;+const SENTINEL_RETRY_BASE_MS = 200;+const SENTINEL_RETRY_MAX_MS = 3_000;++/** Exactly one Redis transport, resolved at config load. */+export type RedisConnection =+ | { mode: 'standalone'; url: string }+ | {+ mode: 'sentinel';+ nodes: string;+ masterName: string;+ database: number;+ username?: string;+ password?: string;+ sentinelUsername?: string;+ sentinelPassword?: string;+ };++export type ConnectedRedis =+ { mode: 'standalone'; client: RedisClientType } | { mode: 'sentinel'; client: RedisSentinelType };++/** Parse comma-separated `host:port` list into Sentinel root nodes. */+export function parseRedisSentinelNodes(raw: string): { host: string; port: number }[] {+ return raw+ .split(',')+ .map(entry => entry.trim())+ .filter(Boolean)+ .map(entry => {+ const lastColon = entry.lastIndexOf(':');+ if (lastColon === -1) {+ return { host: entry, port: DEFAULT_SENTINEL_PORT };+ }+ const host = entry.slice(0, lastColon);+ const parsedPort = Number.parseInt(entry.slice(lastColon + 1), 10);+ return { host, port: Number.isNaN(parsedPort) ? DEFAULT_SENTINEL_PORT : parsedPort };+ });+}++export interface RedisTlsInput {+ enabled: boolean | undefined;+ caCert: string | undefined;+ rejectUnauthorized: boolean | undefined;+ serverName: string | undefined;+ cert: string | undefined;+ key: string | undefined;+ keyPassphrase: string | undefined;+}++/** node-redis socket TLS fields shared by data-node and Sentinel clients. */+export interface RedisTlsSocketOptions {+ tls: true;+ rejectUnauthorized: boolean;+ ca?: string;+ cert?: string;+ key?: string;+ passphrase?: string;+ servername?: string;+}++const PEM_MARKER = '-----BEGIN';++function resolvePemMaterial({ value, label }: { value: string; label: string }): string {+ if (value.includes(PEM_MARKER)) {+ return value;+ }+ if (existsSync(value)) {+ try {+ return readFileSync(value, 'utf8');+ } catch (error) {+ throw new Error(+ `[Redis] Failed to read ${label} from path "${value}": ${error instanceof Error ? error.message : String(error)}`,+ { cause: error },+ );+ }+ }+ throw new Error(`[Redis] ${label} is neither a readable file path nor inline PEM (no "${PEM_MARKER}" marker found).`);+}++/** Shared TLS socket options for data nodes and Sentinel clients. */+function buildTlsSocketOptions(tls: RedisTlsInput | undefined): RedisTlsSocketOptions | undefined {+ if (!tls?.enabled) {+ return undefined;+ }+ if ((tls.cert && !tls.key) || (tls.key && !tls.cert)) {+ throw new Error(+ '[Redis] TLS misconfigured: REDIS_TLS_CERT and REDIS_TLS_KEY must be set together ' ++ '(provide both for mutual TLS, or neither).',+ );+ }++ return {+ tls: true,+ rejectUnauthorized: tls.rejectUnauthorized ?? true,+ ...(tls.caCert ? { ca: resolvePemMaterial({ value: tls.caCert, label: 'REDIS_TLS_CA_CERT' }) } : {}),+ ...(tls.cert ? { cert: resolvePemMaterial({ value: tls.cert, label: 'REDIS_TLS_CERT' }) } : {}),+ ...(tls.key ? { key: resolvePemMaterial({ value: tls.key, label: 'REDIS_TLS_KEY' }) } : {}),+ ...(tls.keyPassphrase ? { passphrase: tls.keyPassphrase } : {}),+ ...(tls.serverName ? { servername: tls.serverName } : {}),+ };+}++/**+ * Connect via Redis Sentinel with rebuild-and-retry.+ *+ * A standalone `createClient()` can recover from later socket drops via the+ * library's automatic reconnect. A Sentinel client cannot: if the first+ * `connect()` rejects, the instance is wedged (retrying `connect()` throws+ * "already attempting to open"). Discard and rebuild on each failed attempt;+ * only auth failures (`WRONGPASS` / `NOAUTH`) abort permanently.+ */+async function connectSentinelWithRetry(input: {+ connection: Extract<RedisConnection, { mode: 'sentinel' }>;+ clientDefaults: { disableOfflineQueue: true; pingInterval: number };+ connectTimeoutMs: number;+ socketTls: RedisTlsSocketOptions | undefined;+ logger: Logger;+}): Promise<ReturnType<typeof createSentinel>> {+ const { connection } = input;+ const auth = {+ database: connection.database,+ ...(connection.username ? { username: connection.username } : {}),+ ...(connection.password ? { password: connection.password } : {}),+ };+ let attempt = 0;+ for (;;) {+ const client = createSentinel({+ name: connection.masterName,+ sentinelRootNodes: parseRedisSentinelNodes(connection.nodes),+ nodeClientOptions: {+ ...input.clientDefaults,+ ...auth,+ socket: { connectTimeout: input.connectTimeoutMs, ...(input.socketTls ?? {}) },+ },+ sentinelClientOptions: {+ socket: { connectTimeout: input.connectTimeoutMs, ...(input.socketTls ?? {}) },+ ...(connection.sentinelUsername ? { username: connection.sentinelUsername } : {}),+ ...(connection.sentinelPassword ? { password: connection.sentinelPassword } : {}),+ },+ });+ // Without an 'error' listener node-redis crashes the process on emit.+ client.on('error', (error: Error) => {+ input.logger.error('[Redis] Client error', extractErrorLogFields(error));+ });+ try {+ await client.connect();+ return client;+ } catch (error) {+ attempt += 1;+ try {+ await client.close();+ } catch {+ // ignore — may never have opened+ }+ const message = error instanceof Error ? error.message : String(error);+ if (message.includes('WRONGPASS') || message.includes('NOAUTH')) {+ throw new Error(`[Redis] Sentinel connect failed: ${message}`, { cause: error });+ }+ input.logger.error('[Redis] Sentinel connect failed, rebuilding and retrying', {+ attempt,+ ...extractErrorLogFields(error),+ });+ await sleep(Math.min(SENTINEL_RETRY_BASE_MS * 2 ** attempt, SENTINEL_RETRY_MAX_MS));+ }+ }+}++export interface ConnectRedisInput {+ connection: RedisConnection;+ tls: RedisTlsInput | undefined;+ logger: Logger;+ connectTimeoutMs: number;+ pingIntervalMs: number;+}++export async function connectRedis(input: ConnectRedisInput): Promise<ConnectedRedis> {input.logger.info('Connecting to Redis');- const client: RedisClientType = createClient({ url: input.url });- // Without an 'error' listener node-redis crashes the process on emit;- // reconnects are automatic, so log and keep running.- client.on('error', (error: Error) => {- input.logger.error('[Redis] Client error', extractErrorLogFields(error));- });- await client.connect();- input.logger.info('Connected to Redis');- return client;++ const socketTls = buildTlsSocketOptions(input.tls);+ const clientDefaults = {+ disableOfflineQueue: true,+ pingInterval: input.pingIntervalMs,+ } as const;++ switch (input.connection.mode) {+ case 'sentinel': {+ // @ts-expect-error TS2375 createSentinel return not assignable under EOPT+ const client: RedisSentinelType = await connectSentinelWithRetry({+ connection: input.connection,+ clientDefaults,+ connectTimeoutMs: input.connectTimeoutMs,+ socketTls,+ logger: input.logger,+ });+ input.logger.info('Connected to Redis');+ return { mode: 'sentinel', client };+ }+ case 'standalone': {+ const created = createClient({+ url: input.connection.url,+ ...clientDefaults,+ socket: { connectTimeout: input.connectTimeoutMs, ...(socketTls ?? {}) },+ });+ // Without an 'error' listener node-redis crashes the process on emit.+ created.on('error', (error: Error) => {+ input.logger.error('[Redis] Client error', extractErrorLogFields(error));+ });+ await created.connect();+ input.logger.info('Connected to Redis');+ // @ts-expect-error TS2375 createClient return not assignable under EOPT+ const client: RedisClientType = created;+ return { mode: 'standalone', client };+ }+ }}diff --git a/packages/trueforge/tests/unit/config/config.test.ts b/packages/trueforge/tests/unit/config/config.test.tsindex 9b0dbe016..deb701b62 100644--- a/packages/trueforge/tests/unit/config/config.test.ts+++ b/packages/trueforge/tests/unit/config/config.test.ts@@ -1,4 +1,5 @@-import { getPublicUiBasePath, type ServerConfiguration } from '../../../src/config';+import type { ServerConfiguration } from '../../../src/config';+import { buildRedisStandaloneUrl, getPublicUiBasePath } from '../../../src/config';import { resolveTrueFoundrySandboxProviderConfig } from '../../../src/truefoundry/resolveTrueFoundrySandboxProviderConfig';/** Minimal distributed config slice for resolve tests (unused fields are irrelevant). */@@ -129,3 +130,32 @@ describe('getPublicUiBasePath', () => {).toBe('/custom/proxy/path/');});});++describe('buildRedisStandaloneUrl', () => {+ const base = {+ port: 6379,+ database: 0,+ username: undefined,+ password: undefined,+ };++ it('builds a hostname URL', () => {+ expect(buildRedisStandaloneUrl({ ...base, host: 'redis.internal' })).toBe('redis://redis.internal:6379/0');+ });++ it('brackets bare IPv6 hosts so the URL is parseable', () => {+ const url = buildRedisStandaloneUrl({ ...base, host: '::1' });+ expect(url).toBe('redis://[::1]:6379/0');+ expect(() => new URL(url)).not.toThrow();+ });++ it('keeps already-bracketed IPv6 hosts', () => {+ expect(buildRedisStandaloneUrl({ ...base, host: '[2001:db8::1]', port: 6380, database: 2 })).toBe(+ 'redis://[2001:db8::1]:6380/2',+ );+ });++ it('leaves IPv4 hosts unbracketed', () => {+ expect(buildRedisStandaloneUrl({ ...base, host: '127.0.0.1' })).toBe('redis://127.0.0.1:6379/0');+ });+});