Skip to content

Open-source platform for 3D AI agents. Turn text or a photo into a rigged, animated GLB avatar, give it an LLM brain, memory and a wallet, and embed it anywhere with one web component. Remote MCP server, x402 payments, Three.js, browser-native.

fix(triage): classify the cryptocurrency.cv news stream's 300 s cutoff as expected

nirholas0275a9ae2993
diff --git a/docs/ops/production-log-triage.md b/docs/ops/production-log-triage.md
index 34ef9d3fe..a94c18148 100644
--- a/docs/ops/production-log-triage.md
+++ b/docs/ops/production-log-triage.md
@@ -1114,6 +1114,29 @@ ERROR 502 POST /api/pump/launch-agent
---
+## 🟢 `Truncated response body. Usually implies that the request timed out...` (cryptocurrency-cv ONLY): `ccv-sse-stream-timeout`
+
+- **Source:** Cloud Run itself (`run.googleapis.com/varlog/system`), on the
+ `cryptocurrency-cv` service. The service's source lives in its own
+ repository, `nirholas/cryptocurrency.cv`.
+- **What it means:** `/api/news/stream` is a Server-Sent Events feed. A client
+ holds it open until Cloud Run's 300 s request timeout closes the response,
+ and Cloud Run logs every response it closes that way. Measured 2026-10-08:
+ 12 of 12 sampled lines joined by `trace` to a `GET /api/news/stream` request
+ that ran 300.99 to 301.00 s, about one every two minutes. Browsers'
+ `EventSource` reconnects on its own, so the reader sees an unbroken feed.
+- **Resolve:** 🟢 nothing required. To confirm a line, join it to its request:
+ `gcloud logging read 'trace="<trace from the warning>" httpRequest:*' --freshness=24h --format='value(httpRequest.requestUrl,httpRequest.latency)'`.
+ Investigate only if a trace lands on a route that is not a stream.
+- **Scope note:** classified **only** for `cryptocurrency-cv`. The same line
+ from any other service stays `investigate`, because there it can mean a
+ handler that never finished its response.
+- **Monitor signature:** `ccv-sse-stream-timeout` in
+ [scripts/gcp-triage.mjs](../../scripts/gcp-triage.mjs), `self-healing`
+ (service-scoped via its `services` field).
+
+---
+
## The owner runbook — every fix as an exact command
Everything red/yellow above, condensed to the actions only the owner can take,
diff --git a/scripts/gcp-triage.mjs b/scripts/gcp-triage.mjs
index 296aba2a5..7e49a2e39 100644
--- a/scripts/gcp-triage.mjs
+++ b/scripts/gcp-triage.mjs
@@ -124,6 +124,15 @@ const KNOWN_SIGNATURES = [
class: 'owner',
action: `cryptocurrency.cv walked its whole AI chain and every provider refused: Groq's free tier spent its daily token budget, Gemini and Vertex answer "Lightning dunning decision is deny" (a billing hold on project aerial-vehicle-466722-p5, not an IAM or key problem), and the OpenRouter account behind OPENROUTER_FALLBACK_KEYS is out of credit. /api/sentiment degrades to a keyword reading and /api/narratives to its last good analysis, so this is noise until the owner restores one provider. ${RUNBOOK} §ccv-ai-providers-unavailable.`,
},
+ {
+ // Service-scoped: Cloud Run writes this line for ANY response it cut
+ // short, so on every other service it stays `investigate`.
+ id: 'ccv-sse-stream-timeout',
+ match: /Truncated response body\. Usually implies that the request timed out/i,
+ services: ['cryptocurrency-cv'],
+ class: 'self-healing',
+ action: `cryptocurrency.cv's /api/news/stream is a Server-Sent Events feed that stays open until Cloud Run's 300 s request timeout closes it; Cloud Run then logs the cut body. Every sampled line on 2026-10-08 (12 of 12) traced to that route at 300.99-301.00 s, and EventSource clients reconnect on their own. Investigate only if a trace lands on a non-streaming route: gcloud logging read 'trace="<trace>" httpRequest:*' --freshness=24h. ${RUNBOOK} §ccv-sse-stream-timeout.`,
+ },
{
id: 'helius-backoff',
match: /helius quota|rate.?limited|refresh deferred \(transient upstream\)/i,
diff --git a/tests/gcp-triage.test.js b/tests/gcp-triage.test.js
index c6e41e3dd..75a651587 100644
--- a/tests/gcp-triage.test.js
+++ b/tests/gcp-triage.test.js
@@ -41,6 +41,15 @@ describe('gcp-triage classify', () => {
expect(classify(line, ['three-ws-api'])?.id).not.toBe('ccv-ai-providers-unavailable');
});
+ it('classifies a truncated body as the SSE stream timeout only on cryptocurrency-cv', () => {
+ const line = 'Truncated response body. Usually implies that the request timed out or the application exited before the response was finished.';
+ const onCcv = classify(line, ['cryptocurrency-cv']);
+ expect(onCcv?.id).toBe('ccv-sse-stream-timeout');
+ expect(onCcv?.class).toBe('self-healing');
+ expect(classify(line, ['three-ws-api'])).toBeNull();
+ expect(classify(line, [])).toBeNull();
+ });
+
it('classifies a bare SIGKILL as the trellis OOM only on model-trellis', () => {
const line = 'Container terminated on signal 9.';
const onTrellis = classify(line, ['model-trellis']);