Skip to content

armgit / cua

Public Imported from GitHub
0

Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.

feat(cyclops): flag PostgreSQL namespace reads with traced fallback (#7699)

r33drichards746cb2df35c6
diff --git a/libs/fleet/backend/handlers/NAMESPACE_READS.md b/libs/fleet/backend/handlers/NAMESPACE_READS.md
new file mode 100644
index 0000000000..51a336fb18
--- /dev/null
+++ b/libs/fleet/backend/handlers/NAMESPACE_READS.md
@@ -0,0 +1,110 @@
+# Namespace read rollout
+
+`GET /api/namespaces` retains its existing JSON response and authorization.
+Terraform seeds `/feature-flags/cyclops-cs/ff-list-ns-read-pg` with `true`,
+so production starts with PostgreSQL reads. Runtime flag changes remain
+operator-controlled. Missing flags and evaluation errors still select Kubernetes.
+
+- Disabled, missing, or failed flag evaluation: read Kubernetes only.
+- Enabled: query namespace objects from `k8s_api.current_resources` using the
+ existing tenant-role executor and PostgreSQL row-level security.
+- PostgreSQL success: return the result, including an empty `[]`.
+- PostgreSQL failure (including unavailable executor, deadline, or invalid row):
+ discard all partial results, log an error with the trace ID, then perform the
+ existing tenant-selected, impersonated Kubernetes read.
+- Apply the existing GitHub namespace allowlist to either backend's result.
+- If Kubernetes also fails, preserve its error/status behavior.
+
+The PostgreSQL attempt has a five-second context deadline. Database cleanup
+has separately bounded contexts (up to three seconds each for rollback and
+connection close). Kubernetes fallback uses the original request context, not
+PostgreSQL's expired context. Flag evaluation has a three-second deadline.
+
+## Enablement
+
+In development, with the development OpenFeature provider:
+
+```sh
+CYCLOPS_CS_FF_LIST_NS_READ_PG=true
+```
+
+In production, new SSM parameters start at `true`. Set the flag to `false`
+to force Kubernetes reads, or back to `true` to resume PostgreSQL reads.
+Terraform retains `ignore_changes = [value]`, so subsequent applies do not
+overwrite manual toggles. If the parameter already exists with `false`, changing
+the Terraform starting value does not flip it; set that existing flag explicitly
+when ready. PR previews also seed the flag on as described below.
+
+The state reader requires `STATE_QUERY_DATABASE_DSN` and
+`STATE_QUERY_TENANT_PASSWORD`. Its initialization is independent of the
+application database configured by `DATABASE_URL`.
+Verify projector health, namespace rows, and tenant-role registration before
+turning the flag on. A stale or empty projection that queries successfully does
+**not** trigger fallback. The query uses the tenant-visible namespace rows in
+this deployment's state store; it does not add a cluster filter.
+
+## Trace shape
+
+```text
+namespaces.list [read.backend, read.fallback, namespaces.count]
+ feature_flag.evaluate [feature_flag.key, feature_flag.result.value]
+ identity.resolve_tenant
+ namespaces.read.postgres (when enabled)
+ db.execute [db.system=postgresql]
+ db.connect
+ db.transaction.begin [db.transaction.read_only=true]
+ db.query
+ db.rows.read_decode [db.response.returned_rows]
+ db.transaction.rollback
+ db.connection.close
+ namespaces.read.kubernetes (disabled or PostgreSQL failed)
+ kubernetes.request [method, status, connection reuse]
+ namespaces.decode
+ namespaces.filter [input, returned, excluded counts]
+ response.encode_write
+```
+
+The Kubernetes request span includes timestamped connection acquisition, DNS,
+TCP, TLS, and first-response-byte events when those phases occur. Reused
+connections naturally omit DNS/TCP/TLS setup events. `db.query` measures the
+initial query call; result fetching, row decoding, and result-writer work are
+measured in `db.rows.read_decode`.
+
+A recovered PostgreSQL failure marks the PostgreSQL child as failed, not the
+successful request. The parent records `read.fallback=true`,
+`read.fallback.reason=postgres_error`, and a `namespaces.read.fallback` event.
+Query text, credentials, HTTP headers, and returned objects are not attached to
+these new spans. Error details remain in the fallback log; spans record safe
+stage status and error types.
+
+These changes do not instrument authentication middleware or change namespace
+creation/deletion. PostgreSQL stage spans also cover `QUERY /api/state/query`,
+since both endpoints use the same executor.
+
+## PR previews
+
+Previews seed `ff-list-ns-read-pg=true` into the process-local feature flag
+management store. The flag appears in the preview admin feature flags UI and
+can be toggled off/on there. Changes affect only that preview pod, never
+production SSM, and reset to the seed when the pod restarts.
+
+`cyclops-preview-state-query` imports only `state_query_dsn` and
+`state_query_tenant_password` from the existing state database secret. These
+credentials connect as the authenticated tenant's read-only RLS role, not as
+an application, writer, migration, or role-admin user. Namespace data comes
+from the existing production projection, matching the production Kubernetes
+cluster that previews already query. The production projector and tenant-role
+controller populate that store; previews do not run duplicate controllers or
+write production state. The preview's `DATABASE_URL` remains its isolated
+`cyclops-preview-database` application connection.
+
+The query credential references are optional so a missing ExternalSecret does
+not prevent the backend from starting. Missing credentials leave the executor
+unavailable and trigger the normal Kubernetes fallback. Secret values are
+injected as environment variables, so restart the preview pod after credentials
+first appear or rotate. Once query credentials are present, namespace requests should report `read.backend=postgres`;
+force the flag off to compare against `read.backend=kubernetes`.
+
+Only trusted same-repository PR previews receive these credentials, following
+the existing preview secret isolation/fork guard. This change does not modify
+production flag values, credential values, or production database grants.
diff --git a/libs/fleet/backend/handlers/namespaces.go b/libs/fleet/backend/handlers/namespaces.go
index 4686d89ab9..05a0060ed4 100644
--- a/libs/fleet/backend/handlers/namespaces.go
+++ b/libs/fleet/backend/handlers/namespaces.go
@@ -222,75 +222,57 @@ func (h Handlers) ListNamespaces(w http.ResponseWriter, r *http.Request) {
attribute.String("user.id", user.ID),
))
defer span.End()
-
- // Scope the list to the caller's own Tenant. Every namespace owned by a
- // Tenant carries capsule.clastix.io/tenant=<tenant> (set on create — see
- // CreateNamespace — and enforced by Capsule). Selecting on it makes this
- // fail-closed: even if Capsule Proxy isn't filtering (e.g. while
- // CapsuleConfiguration.userGroups is being reconciled, or right after a
- // capsule-proxy restart with a cold cache), the apiserver only returns the
- // caller's namespaces. The Tenant name follows the standalone controller
- // convention "user-<sub>" and is derived from the authenticated subject,
- // so a caller can never widen it to another tenant.
- selector := "capsule.clastix.io/tenant=" + identity.PersonalGroup(ctx, user.ID)
- path := "/api/v1/namespaces?labelSelector=" + url.QueryEscape(selector)
- resp, err := h.k8sImpersonate(ctx, "GET", path, nil, user.ID)
- if err != nil {
- span.RecordError(err)
- span.SetStatus(codes.Error, "namespace list request failed")
- slog.Warn("namespace list: k8s request failed", "err", err)
- writeErr(w, http.StatusBadGateway, "kubectl-proxy unavailable")
- return
- }
- defer resp.Body.Close()
-
- if resp.StatusCode != http.StatusOK {
- body, _ := io.ReadAll(resp.Body)
- span.SetStatus(codes.Error, "namespace list returned non-200")
- span.SetAttributes(attribute.Int("http.status_code", resp.StatusCode))
- slog.Warn("namespace list: k8s error", "status", resp.StatusCode, "body", string(body))
- writeErr(w, resp.StatusCode, "k8s: "+string(body))
- return
+ enabled := namespacePostgresEnabled(ctx, user.ID)
+ tenant := namespaceTenant(ctx, user.ID)
+ span.SetAttributes(attribute.Bool("feature_flag.ff-list-ns-read-pg", enabled), attribute.Bool("read.fallback", false))
+ var items []namespaceObject
+ var err error
+ backend := "kubernetes"
+ if enabled {
+ items, err = h.readNamespacesPostgres(ctx, tenant)
+ if err == nil {
+ backend = "postgres"
+ } else {
+ span.SetAttributes(attribute.Bool("read.fallback", true), attribute.String("read.fallback.reason", "postgres_error"))
+ span.AddEvent("namespaces.read.fallback", trace.WithAttributes(attribute.String("read.fallback.reason", "postgres_error")))
+ slog.ErrorContext(ctx, "namespace PostgreSQL read failed; falling back to Kubernetes", "err", err, "trace_id", span.SpanContext().TraceID().String())
+ }
}
-
- // Parse K8s NamespaceList.
- var nsList struct {
- Items []struct {
- Metadata struct {
- Name string `json:"name"`
- Labels map[string]string `json:"labels"`
- CreationTimestamp string `json:"creationTimestamp"`
- } `json:"metadata"`
- Status struct {
- Phase string `json:"phase"`
- } `json:"status"`
- } `json:"items"`
- }
- if err := json.NewDecoder(resp.Body).Decode(&nsList); err != nil {
- span.RecordError(err)
- span.SetStatus(codes.Error, "namespace list decode failed")
- slog.Warn("namespace list: decode error", "err", err)
- writeErr(w, http.StatusBadGateway, "bad response from k8s")
- return
+ span.SetAttributes(attribute.String("read.backend", backend))
+ if backend == "kubernetes" {
+ var status int
+ items, status, err = h.readNamespacesK8s(ctx, tenant, user.ID)
+ if err != nil {
+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))
+ span.SetStatus(codes.Error, "namespace list failed")
+ span.SetAttributes(attribute.Int("http.status_code", status))
+ writeErr(w, status, err.Error())
+ return
+ }
}
-
- out := make([]NamespaceResponse, 0, len(nsList.Items))
- for _, ns := range nsList.Items {
+ _, filterSpan := handlerTracer().Start(ctx, "namespaces.filter")
+ out := make([]NamespaceResponse, 0, len(items))
+ for _, ns := range items {
if isGitHubPrincipal(user) && !namespaceAllowed(user, ns.Metadata.Name) {
continue
}
out = append(out, NamespaceResponse{
- Name: ns.Metadata.Name,
- Status: ns.Status.Phase,
- CreatedAt: ns.Metadata.CreationTimestamp,
- Labels: ns.Metadata.Labels,
+ Name: ns.Metadata.Name, Status: ns.Status.Phase,
+ CreatedAt: ns.Metadata.CreationTimestamp, Labels: ns.Metadata.Labels,
})
}
- span.SetAttributes(
- attribute.Int("http.status_code", http.StatusOK),
- attribute.Int("namespaces.count", len(out)),
- )
- writeJSON(w, http.StatusOK, out)
+ filterSpan.SetAttributes(attribute.Int("namespaces.input_count", len(items)), attribute.Int("namespaces.count", len(out)), attribute.Int("namespaces.excluded_count", len(items)-len(out)))
+ filterSpan.End()
+ span.SetAttributes(attribute.Int("http.status_code", http.StatusOK), attribute.Int("namespaces.count", len(out)))
+ _, responseSpan := handlerTracer().Start(ctx, "response.encode_write")
+ w.Header().Set("Content-Type", "application/json")
+ w.WriteHeader(http.StatusOK)
+ err = json.NewEncoder(w).Encode(out)
+ finishNamespaceSpan(responseSpan, err)
+ if err != nil {
+ span.RecordError(err)
+ span.SetStatus(codes.Error, "namespace response write failed")
+ }
}
// GetNamespace godoc
diff --git a/libs/fleet/backend/handlers/namespaces_pg_test.go b/libs/fleet/backend/handlers/namespaces_pg_test.go
new file mode 100644
index 0000000000..b22458b771
--- /dev/null
+++ b/libs/fleet/backend/handlers/namespaces_pg_test.go
@@ -0,0 +1,255 @@
+package handlers
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "net/http"
+ "net/http/httptest"
+ "testing"
+ "time"
+
+ "cyclops-cs-backend/statequery"
+ "github.com/open-feature/go-sdk/openfeature"
+
+ "cyclops-cs-backend/auth"
+ "github.com/trycua/cloud/pkg/featureflags"
+ "go.opentelemetry.io/otel"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ sdktrace "go.opentelemetry.io/otel/sdk/trace"
+ "go.opentelemetry.io/otel/sdk/trace/tracetest"
+)
+
+func TestListNamespacesPostgresFlag(t *testing.T) {
+ for _, test := range []struct {
+ name, flag string
+ executor *fakeStateQueryExecutor
+ want string
+ k8sCalls int
+ }{
+ {"disabled", "false", &fakeStateQueryExecutor{}, "k8s-ns", 1},
+ {"missing flag", "", &fakeStateQueryExecutor{}, "k8s-ns", 1},
+ {"postgres", "true", &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"pg-ns","labels":{"test":"value"}},"status":{"phase":"Active"}}`}}}, "pg-ns", 0},
+ {"empty success", "true", &fakeStateQueryExecutor{}, "", 0},
+ {"unavailable", "true", nil, "k8s-ns", 1},
+ {"partial failure", "true", &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"partial"}}`}}, err: errors.New("query failed")}, "k8s-ns", 1},
+ {"invalid JSON", "true", &fakeStateQueryExecutor{rows: [][]any{{`broken`}}}, "k8s-ns", 1},
+ {"invalid column", "true", &fakeStateQueryExecutor{rows: [][]any{{123}}}, "k8s-ns", 1},
+ {"invalid column count", "true", &fakeStateQueryExecutor{rows: [][]any{{}}}, "k8s-ns", 1},
+ {"missing namespace name", "true", &fakeStateQueryExecutor{rows: [][]any{{`{}`}}}, "k8s-ns", 1},
+ } {
+ t.Run(test.name, func(t *testing.T) {
+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", test.flag)
+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {
+ t.Fatal(err)
+ }
+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))
+ defer proxy.server.Close()
+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")
+ handler := Handlers{}
+ if test.executor != nil {
+ handler.Features = FeaturesWith(test.executor, nil)
+ }
+ response := httptest.NewRecorder()
+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))
+ if response.Code != 200 {
+ t.Fatalf("status %d: %s", response.Code, response.Body.String())
+ }
+ var namespaces []NamespaceResponse
+ if err := json.Unmarshal(response.Body.Bytes(), &namespaces); err != nil {
+ t.Fatal(err)
+ }
+ if test.want == "" {
+ if namespaces == nil || len(namespaces) != 0 {
+ t.Fatalf("want [], got %s", response.Body.String())
+ }
+ } else if len(namespaces) != 1 || namespaces[0].Name != test.want {
+ t.Fatalf("namespaces = %+v", namespaces)
+ }
+ if len(proxy.requests) != test.k8sCalls {
+ t.Fatalf("Kubernetes calls = %d, want %d", len(proxy.requests), test.k8sCalls)
+ }
+ if test.executor != nil {
+ if test.flag == "true" && test.executor.tenant != "user-alice" {
+ t.Fatalf("tenant = %q", test.executor.tenant)
+ }
+ if test.flag != "true" && test.executor.query != "" {
+ t.Fatal("queried PostgreSQL with flag off")
+ }
+ }
+ })
+ }
+}
+
+func TestListNamespacesPostgresFilteringAndTracing(t *testing.T) {
+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")
+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {
+ t.Fatal(err)
+ }
+ recorder := tracetest.NewSpanRecorder()
+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
+ previous := otel.GetTracerProvider()
+ otel.SetTracerProvider(provider)
+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })
+ executor := &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"allowed"}}`}, {`{"metadata":{"name":"denied"}}`}}}
+ handler := Handlers{Features: FeaturesWith(executor, nil)}
+ response := httptest.NewRecorder()
+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice", PrincipalType: auth.PrincipalTypeGitHubOIDC, AllowedNamespaces: []string{"allowed"}}))
+ var namespaces []NamespaceResponse
+ if err := json.Unmarshal(response.Body.Bytes(), &namespaces); err != nil {
+ t.Fatal(err)
+ }
+ if len(namespaces) != 1 || namespaces[0].Name != "allowed" {
+ t.Fatalf("namespaces = %+v", namespaces)
+ }
+ for _, name := range []string{"namespaces.list", "feature_flag.evaluate", "identity.resolve_tenant", "namespaces.read.postgres", "namespaces.filter", "response.encode_write"} {
+ found := false
+ for _, span := range recorder.Ended() {
+ if span.Name() == name {
+ found = true
+ if name != "namespaces.list" && !span.Parent().IsValid() {
+ t.Fatalf("%s missing parent", name)
+ }
+ }
+ }
+ if !found {
+ t.Errorf("missing span %s", name)
+ }
+ }
+}
+
+func TestListNamespacesFallbackTracing(t *testing.T) {
+ for _, status := range []int{http.StatusOK, http.StatusForbidden} {
+ t.Run(http.StatusText(status), func(t *testing.T) {
+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")
+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {
+ t.Fatal(err)
+ }
+ proxy := newFakeK8s(status, nsListResponse("k8s-ns"))
+ defer proxy.server.Close()
+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")
+ recorder := tracetest.NewSpanRecorder()
+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
+ previous := otel.GetTracerProvider()
+ otel.SetTracerProvider(provider)
+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })
+ handler := Handlers{Features: FeaturesWith(&fakeStateQueryExecutor{err: errors.New("pg failed")}, nil)}
+ response := httptest.NewRecorder()
+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))
+ if response.Code != status {
+ t.Fatalf("status = %d, want %d", response.Code, status)
+ }
+ spans := map[string]sdktrace.ReadOnlySpan{}
+ for _, span := range recorder.Ended() {
+ spans[span.Name()] = span
+ }
+ for _, name := range []string{"namespaces.list", "namespaces.read.postgres", "namespaces.read.kubernetes", "kubernetes.request"} {
+ if spans[name] == nil {
+ t.Fatalf("missing span %s", name)
+ }
+ }
+ parent := spans["namespaces.list"]
+ if spans["namespaces.read.postgres"].Status().Code != codes.Error {
+ t.Error("PG failure not marked")
+ }
+ if status == http.StatusOK && parent.Status().Code == codes.Error {
+ t.Error("successful fallback marked as error")
+ }
+ if status != http.StatusOK && parent.Status().Code != codes.Error {
+ t.Error("failed fallback not marked as error")
+ }
+ attrs := map[string]attribute.Value{}
+ for _, attr := range parent.Attributes() {
+ attrs[string(attr.Key)] = attr.Value
+ }
+ if !attrs["read.fallback"].AsBool() || attrs["read.backend"].AsString() != "kubernetes" {
+ t.Fatalf("fallback attributes = %v", attrs)
+ }
+ if spans["namespaces.read.postgres"].Parent().SpanID() != parent.SpanContext().SpanID() || spans["namespaces.read.kubernetes"].Parent().SpanID() != parent.SpanContext().SpanID() {
+ t.Error("read attempts must be siblings")
+ }
+ foundFirstByte := false
+ for _, event := range spans["kubernetes.request"].Events() {
+ if event.Name == "http.response.first_byte" {
+ foundFirstByte = true
+ }
+ }
+ if !foundFirstByte {
+ t.Error("missing network timing event")
+ }
+ })
+ }
+}
+
+type namespaceFlagErrorProvider struct{ featureflags.SimpleEnvProvider }
+
+func (namespaceFlagErrorProvider) BooleanEvaluation(context.Context, string, bool, openfeature.FlattenedContext) openfeature.BoolResolutionDetail {
+ return openfeature.BoolResolutionDetail{Value: true, ProviderResolutionDetail: openfeature.ProviderResolutionDetail{ResolutionError: openfeature.NewGeneralResolutionError("flag provider unavailable")}}
+}
+
+func TestListNamespacesFlagErrorUsesKubernetes(t *testing.T) {
+ if err := openfeature.SetProviderAndWait(&namespaceFlagErrorProvider{}); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = openfeature.SetProviderAndWait(&featureflags.SimpleEnvProvider{}) })
+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))
+ defer proxy.server.Close()
+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")
+ executor := &fakeStateQueryExecutor{}
+ handler := Handlers{Features: FeaturesWith(executor, nil)}
+ response := httptest.NewRecorder()
+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))
+ if response.Code != http.StatusOK || len(proxy.requests) != 1 || executor.query != "" {
+ t.Fatalf("flag error did not use Kubernetes: status=%d requests=%d query=%q", response.Code, len(proxy.requests), executor.query)
+ }
+}
+
+type namespaceDeadlineExecutor struct{ testing *testing.T }
+
+func (executor namespaceDeadlineExecutor) Execute(ctx context.Context, _, _ string, _ statequery.ResultWriter) error {
+ deadline, ok := ctx.Deadline()
+ if !ok || time.Until(deadline) > namespacePostgresTimeout {
+ executor.testing.Fatal("PostgreSQL attempt is not bounded")
+ }
+ <-ctx.Done()
+ return ctx.Err()
+}
+
+func TestListNamespacesDeadlineFallsBackWithLiveContext(t *testing.T) {
+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")
+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {
+ t.Fatal(err)
+ }
+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))
+ defer proxy.server.Close()
+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")
+ handler := Handlers{Features: FeaturesWith(namespaceDeadlineExecutor{testing: t}, nil)}
+ response := httptest.NewRecorder()
+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))
+ if response.Code != http.StatusOK || len(proxy.requests) != 1 {
+ t.Fatalf("deadline fallback failed: status=%d requests=%d", response.Code, len(proxy.requests))
+ }
+}
+
+func TestNamespaceReadErrorsPreserveCauses(t *testing.T) {
+ writer := &namespaceResultWriter{}
+ var syntaxErr *json.SyntaxError
+ if err := writer.WriteRow([]any{"invalid"}); !errors.As(err, &syntaxErr) || err.Error() != "namespace query returned invalid object" {
+ t.Fatalf("namespace row decode must preserve a redacted cause: %v", err)
+ }
+ proxy := newFakeK8s(http.StatusOK, "invalid")
+ defer proxy.server.Close()
+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")
+ handler := Handlers{}
+ _, status, err := handler.readNamespacesK8s(context.Background(), "user-alice", "alice")
+ if status != http.StatusBadGateway || !errors.As(err, &syntaxErr) || err.Error() != "bad response from k8s" {
+ t.Fatalf("Kubernetes decode must preserve a redacted cause: %d %v", status, err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+ _, status, err = handler.readNamespacesK8s(ctx, "user-alice", "alice")
+ if status != http.StatusBadGateway || !errors.Is(err, context.Canceled) || err.Error() != "kubectl-proxy unavailable" {
+ t.Fatalf("Kubernetes transport must preserve a redacted cause: %d %v", status, err)
+ }
+}
diff --git a/libs/fleet/backend/handlers/namespaces_read.go b/libs/fleet/backend/handlers/namespaces_read.go
new file mode 100644
index 0000000000..24de4f8963
--- /dev/null
+++ b/libs/fleet/backend/handlers/namespaces_read.go
@@ -0,0 +1,159 @@
+package handlers
+
+import (
+ "context"
+ "crypto/tls"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "log/slog"
+ "net/http"
+ "net/http/httptrace"
+ "net/url"
+ "time"
+
+ "cyclops-cs-backend/identity"
+ "cyclops-cs-backend/internal/redactederror"
+ "github.com/jackc/pgx/v5/pgconn"
+ "github.com/open-feature/go-sdk/openfeature"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ "go.opentelemetry.io/otel/trace"
+)
+
+const namespacePostgresFlag = "/feature-flags/cyclops-cs/ff-list-ns-read-pg"
+const namespacePostgresTimeout = 5 * time.Second
+const namespaceListSQL = `SELECT object::text FROM k8s_api.current_resources
+WHERE api_group = '' AND resource = 'namespaces' AND namespace = '' ORDER BY name`
+
+type namespaceObject struct {
+ Metadata struct {
+ Name string `json:"name"`
+ Labels map[string]string `json:"labels"`
+ CreationTimestamp string `json:"creationTimestamp"`
+ } `json:"metadata"`
+ Status struct {
+ Phase string `json:"phase"`
+ } `json:"status"`
+}
+
+type namespaceResultWriter struct{ items []namespaceObject }
+
+func (writer *namespaceResultWriter) WriteFieldDescriptions(_ []pgconn.FieldDescription) error {
+ return nil
+}
+func (writer *namespaceResultWriter) WriteRow(values []any) error {
+ if len(values) != 1 {
+ return errors.New("namespace query returned unexpected columns")
+ }
+ object, ok := values[0].(string)
+ if !ok {
+ return errors.New("namespace query returned non-text object")
+ }
+ var item namespaceObject
+ if err := json.Unmarshal([]byte(object), &item); err != nil {
+ return redactederror.New("namespace query returned invalid object", err)
+ }
+ if item.Metadata.Name == "" {
+ return errors.New("namespace query returned object without name")
+ }
+ writer.items = append(writer.items, item)
+ return nil
+}
+
+func finishNamespaceSpan(span trace.Span, err error) {
+ if err != nil {
+ span.RecordError(errors.New("namespace operation failed"))
+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))
+ span.SetStatus(codes.Error, "namespace operation failed")
+ }
+ span.End()
+}
+
+func namespacePostgresEnabled(ctx context.Context, subject string) bool {
+ ctx, span := handlerTracer().Start(ctx, "feature_flag.evaluate", trace.WithAttributes(attribute.String("feature_flag.key", namespacePostgresFlag)))
+ callCtx, cancel := context.WithTimeout(ctx, 3*time.Second)
+ defer cancel()
+ enabled, err := openfeature.NewClient("cyclops-cs-namespaces").BooleanValue(callCtx, namespacePostgresFlag, false, openfeature.NewEvaluationContext(subject, nil))
+ if err != nil {
+ enabled = false
+ slog.WarnContext(ctx, "namespace read flag evaluation failed; using Kubernetes", "err", err)
+ }
+ span.SetAttributes(attribute.Bool("feature_flag.result.value", enabled))
+ finishNamespaceSpan(span, err)
+ return enabled
+}
+
+func namespaceTenant(ctx context.Context, subject string) string {
+ ctx, span := handlerTracer().Start(ctx, "identity.resolve_tenant")
+ defer span.End()
+ return identity.PersonalGroup(ctx, subject)
+}
+
+func (h Handlers) readNamespacesPostgres(ctx context.Context, tenant string) (items []namespaceObject, err error) {
+ ctx, span := handlerTracer().Start(ctx, "namespaces.read.postgres", trace.WithAttributes(attribute.String("read.backend", "postgres")))
+ defer func() { finishNamespaceSpan(span, err) }()
+ executor := h.Features.StateQuery()
+ if executor == nil {
+ return nil, errors.New("namespace PostgreSQL reader unavailable")
+ }
+ callCtx, cancel := context.WithTimeout(ctx, namespacePostgresTimeout)
+ defer cancel()
+ writer := &namespaceResultWriter{}
+ if err = executor.Execute(callCtx, tenant, namespaceListSQL, writer); err != nil {
+ return nil, err
+ }
+ span.SetAttributes(attribute.Int("namespaces.count", len(writer.items)))
+ return writer.items, nil
+}
+
+func (h Handlers) readNamespacesK8s(ctx context.Context, tenant, subject string) (items []namespaceObject, status int, err error) {
+ ctx, span := handlerTracer().Start(ctx, "namespaces.read.kubernetes", trace.WithAttributes(attribute.String("read.backend", "kubernetes")))
+ defer func() { span.SetAttributes(attribute.Int("http.status_code", status)); finishNamespaceSpan(span, err) }()
+ path := "/api/v1/namespaces?labelSelector=" + url.QueryEscape("capsule.clastix.io/tenant="+tenant)
+ requestCtx, requestSpan := handlerTracer().Start(ctx, "kubernetes.request", trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(attribute.String("http.request.method", "GET"), attribute.String("k8s.resource.name", "namespaces")))
+ requestCtx = httptrace.WithClientTrace(requestCtx, &httptrace.ClientTrace{
+ GetConn: func(_ string) { requestSpan.AddEvent("http.connection.acquire") },
+ GotConn: func(info httptrace.GotConnInfo) {
+ requestSpan.SetAttributes(attribute.Bool("http.connection.reused", info.Reused))
+ requestSpan.AddEvent("http.connection.acquired")
+ },
+ DNSStart: func(_ httptrace.DNSStartInfo) { requestSpan.AddEvent("dns.start") },
+ DNSDone: func(_ httptrace.DNSDoneInfo) { requestSpan.AddEvent("dns.done") },
+ ConnectStart: func(_, _ string) { requestSpan.AddEvent("tcp.connect.start") },
+ ConnectDone: func(_, _ string, _ error) { requestSpan.AddEvent("tcp.connect.done") },
+ TLSHandshakeStart: func() { requestSpan.AddEvent("tls.handshake.start") },
+ TLSHandshakeDone: func(_ tls.ConnectionState, _ error) { requestSpan.AddEvent("tls.handshake.done") },
+ GotFirstResponseByte: func() { requestSpan.AddEvent("http.response.first_byte") },
+ })
+ resp, requestErr := h.k8sImpersonate(requestCtx, "GET", path, nil, subject)
+ if requestErr != nil {
+ finishNamespaceSpan(requestSpan, requestErr)
+ slog.WarnContext(ctx, "namespace list: k8s request failed", "err", requestErr)
+ return nil, http.StatusBadGateway, redactederror.New("kubectl-proxy unavailable", requestErr)
+ }
+ defer resp.Body.Close()
+ requestSpan.SetAttributes(attribute.Int("http.response.status_code", resp.StatusCode))
+ if resp.StatusCode != http.StatusOK {
+ body, readErr := io.ReadAll(resp.Body)
+ statusErr := fmt.Errorf("k8s: %s", body)
+ if readErr != nil {
+ statusErr = redactederror.New("k8s response read failed", readErr)
+ }
+ finishNamespaceSpan(requestSpan, statusErr)
+ return nil, resp.StatusCode, statusErr
+ }
+ finishNamespaceSpan(requestSpan, nil)
+ _, decodeSpan := handlerTracer().Start(ctx, "namespaces.decode")
+ var list struct {
+ Items []namespaceObject `json:"items"`
+ }
+ decodeErr := json.NewDecoder(resp.Body).Decode(&list)
+ finishNamespaceSpan(decodeSpan, decodeErr)
+ if decodeErr != nil {
+ return nil, http.StatusBadGateway, redactederror.New("bad response from k8s", decodeErr)
+ }
+ span.SetAttributes(attribute.Int("namespaces.count", len(list.Items)))
+ return list.Items, http.StatusOK, nil
+}
diff --git a/libs/fleet/backend/statequery/executor.go b/libs/fleet/backend/statequery/executor.go
index 8a2f466f26..0b317e54e1 100644
--- a/libs/fleet/backend/statequery/executor.go
+++ b/libs/fleet/backend/statequery/executor.go
@@ -6,6 +6,12 @@ import (
"encoding/hex"
"errors"
"fmt"
+ "time"
+
+ "go.opentelemetry.io/otel"
+ "go.opentelemetry.io/otel/attribute"
+ "go.opentelemetry.io/otel/codes"
+ "go.opentelemetry.io/otel/trace"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
@@ -46,30 +52,67 @@ func (e *Executor) connectionConfig(tenant string) *pgx.ConnConfig {
return config
}
-func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer ResultWriter) error {
+func finishDatabaseSpan(span trace.Span, err error) {
+ if err != nil {
+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))
+ var postgresErr *pgconn.PgError
+ if errors.As(err, &postgresErr) {
+ span.SetAttributes(attribute.String("db.postgresql.sqlstate", postgresErr.Code))
+ }
+ span.SetStatus(codes.Error, "database operation failed")
+ }
+ span.End()
+}
+
+func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer ResultWriter) (resultErr error) {
+ tracer := otel.Tracer("cyclops-cs-backend/statequery")
+ ctx, span := tracer.Start(ctx, "db.execute", trace.WithAttributes(attribute.String("db.system", "postgresql")))
+ defer func() { finishDatabaseSpan(span, resultErr) }()
if tenant == "" {
return fmt.Errorf("%w: tenant is required", ErrExecution)
}
- conn, err := pgx.ConnectConfig(ctx, e.connectionConfig(tenant))
+ connectCtx, connectSpan := tracer.Start(ctx, "db.connect", trace.WithSpanKind(trace.SpanKindClient))
+ conn, err := pgx.ConnectConfig(connectCtx, e.connectionConfig(tenant))
+ finishDatabaseSpan(connectSpan, err)
if err != nil {
return errors.Join(fmt.Errorf("%w: connect tenant role: %v", ErrExecution, err), err)
}
- defer conn.Close(ctx)
+ defer func() {
+ closeCtx, closeSpan := tracer.Start(context.WithoutCancel(ctx), "db.connection.close")
+ closeCtx, cancel := context.WithTimeout(closeCtx, 3*time.Second)
+ defer cancel()
+ finishDatabaseSpan(closeSpan, conn.Close(closeCtx))
+ }()
- tx, err := conn.BeginTx(ctx, pgx.TxOptions{AccessMode: pgx.ReadOnly})
+ beginCtx, beginSpan := tracer.Start(ctx, "db.transaction.begin", trace.WithAttributes(attribute.Bool("db.transaction.read_only", true)))
+ tx, err := conn.BeginTx(beginCtx, pgx.TxOptions{AccessMode: pgx.ReadOnly})
+ finishDatabaseSpan(beginSpan, err)
if err != nil {
return errors.Join(fmt.Errorf("%w: begin transaction", ErrExecution), err)
}
- defer tx.Rollback(ctx)
+ defer func() {
+ rollbackCtx, rollbackSpan := tracer.Start(context.WithoutCancel(ctx), "db.transaction.rollback")
+ rollbackCtx, cancel := context.WithTimeout(rollbackCtx, 3*time.Second)
+ defer cancel()
+ finishDatabaseSpan(rollbackSpan, tx.Rollback(rollbackCtx))
+ }()
- rows, err := tx.Query(ctx, sql)
+ queryCtx, querySpan := tracer.Start(ctx, "db.query", trace.WithSpanKind(trace.SpanKindClient))
+ rows, err := tx.Query(queryCtx, sql)
+ finishDatabaseSpan(querySpan, err)
if err != nil {
return errors.Join(fmt.Errorf("%w: execute query", ErrExecution), err)
}
- defer rows.Close()
+ _, rowsSpan := tracer.Start(ctx, "db.rows.read_decode")
+ rowCount := 0
+ defer func() {
+ rows.Close()
+ rowsSpan.SetAttributes(attribute.Int("db.response.returned_rows", rowCount))
+ finishDatabaseSpan(rowsSpan, resultErr)
+ }()
if err := writer.WriteFieldDescriptions(rows.FieldDescriptions()); err != nil {
return err
}
@@ -82,7 +125,7 @@ func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer Resul
if err := writer.WriteRow(values); err != nil {
return err
}
+ rowCount++
}
- rows.Close()
return rows.Err()
}
diff --git a/libs/fleet/backend/statequery/tracing_test.go b/libs/fleet/backend/statequery/tracing_test.go
new file mode 100644
index 0000000000..472c7fd06d
--- /dev/null
+++ b/libs/fleet/backend/statequery/tracing_test.go
@@ -0,0 +1,59 @@
+package statequery
+
+import (
+ "context"
+ "net"
+ "testing"
+
+ "go.opentelemetry.io/otel"
+ "go.opentelemetry.io/otel/codes"
+ sdktrace "go.opentelemetry.io/otel/sdk/trace"
+ "go.opentelemetry.io/otel/sdk/trace/tracetest"
+)
+
+func TestExecutorTracesConnectionFailure(t *testing.T) {
+ listener, err := net.Listen("tcp", "127.0.0.1:0")
+ if err != nil {
+ t.Fatal(err)
+ }
+ address := listener.Addr().String()
+ _ = listener.Close()
+ recorder := tracetest.NewSpanRecorder()
+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))
+ previous := otel.GetTracerProvider()
+ otel.SetTracerProvider(provider)
+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })
+ executor, err := NewExecutor("postgres://"+address+"/cyclops?sslmode=disable&connect_timeout=1", "secret")
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, parent := provider.Tracer("test").Start(context.Background(), "request")
+ err = executor.Execute(ctx, "user-alice", "select 'sensitive-query'", &collectingResultWriter{})
+ parent.End()
+ if err == nil {
+ t.Fatal("expected connection failure")
+ }
+ for _, name := range []string{"db.execute", "db.connect"} {
+ found := false
+ for _, span := range recorder.Ended() {
+ if span.Name() != name {
+ continue
+ }
+ found = true
+ if span.Status().Code != codes.Error {
+ t.Errorf("%s did not report error", name)
+ }
+ if !span.Parent().IsValid() {
+ t.Errorf("%s missing parent", name)
+ }
+ for _, attr := range span.Attributes() {
+ if string(attr.Key) == "db.statement" || string(attr.Key) == "db.query.text" {
+ t.Error("query text exposed")
+ }
+ }
+ }
+ if !found {
+ t.Errorf("missing span %s", name)
+ }
+ }
+}