Scale computer-use 2.0 with open-source drivers, cross-OS sandboxes, and benchmarks for training, evaluation, and data generation.
Rust agentai-agentapplecomputer-usecomputer-use-agentcontainerizationcuadesktop-automationhacktoberfestlumemacosmanusoperatorswiftvirtualizationvirtualization-frameworkwindowswindows-sandbox
feat(cyclops): flag PostgreSQL namespace reads with traced fallback (#7699)
r33drichards746cb2df35c6
diff --git a/libs/fleet/backend/handlers/NAMESPACE_READS.md b/libs/fleet/backend/handlers/NAMESPACE_READS.mdnew file mode 100644index 0000000000..51a336fb18--- /dev/null+++ b/libs/fleet/backend/handlers/NAMESPACE_READS.md@@ -0,0 +1,110 @@+# Namespace read rollout++`GET /api/namespaces` retains its existing JSON response and authorization.+Terraform seeds `/feature-flags/cyclops-cs/ff-list-ns-read-pg` with `true`,+so production starts with PostgreSQL reads. Runtime flag changes remain+operator-controlled. Missing flags and evaluation errors still select Kubernetes.++- Disabled, missing, or failed flag evaluation: read Kubernetes only.+- Enabled: query namespace objects from `k8s_api.current_resources` using the+ existing tenant-role executor and PostgreSQL row-level security.+- PostgreSQL success: return the result, including an empty `[]`.+- PostgreSQL failure (including unavailable executor, deadline, or invalid row):+ discard all partial results, log an error with the trace ID, then perform the+ existing tenant-selected, impersonated Kubernetes read.+- Apply the existing GitHub namespace allowlist to either backend's result.+- If Kubernetes also fails, preserve its error/status behavior.++The PostgreSQL attempt has a five-second context deadline. Database cleanup+has separately bounded contexts (up to three seconds each for rollback and+connection close). Kubernetes fallback uses the original request context, not+PostgreSQL's expired context. Flag evaluation has a three-second deadline.++## Enablement++In development, with the development OpenFeature provider:++```sh+CYCLOPS_CS_FF_LIST_NS_READ_PG=true+```++In production, new SSM parameters start at `true`. Set the flag to `false`+to force Kubernetes reads, or back to `true` to resume PostgreSQL reads.+Terraform retains `ignore_changes = [value]`, so subsequent applies do not+overwrite manual toggles. If the parameter already exists with `false`, changing+the Terraform starting value does not flip it; set that existing flag explicitly+when ready. PR previews also seed the flag on as described below.++The state reader requires `STATE_QUERY_DATABASE_DSN` and+`STATE_QUERY_TENANT_PASSWORD`. Its initialization is independent of the+application database configured by `DATABASE_URL`.+Verify projector health, namespace rows, and tenant-role registration before+turning the flag on. A stale or empty projection that queries successfully does+**not** trigger fallback. The query uses the tenant-visible namespace rows in+this deployment's state store; it does not add a cluster filter.++## Trace shape++```text+namespaces.list [read.backend, read.fallback, namespaces.count]+ feature_flag.evaluate [feature_flag.key, feature_flag.result.value]+ identity.resolve_tenant+ namespaces.read.postgres (when enabled)+ db.execute [db.system=postgresql]+ db.connect+ db.transaction.begin [db.transaction.read_only=true]+ db.query+ db.rows.read_decode [db.response.returned_rows]+ db.transaction.rollback+ db.connection.close+ namespaces.read.kubernetes (disabled or PostgreSQL failed)+ kubernetes.request [method, status, connection reuse]+ namespaces.decode+ namespaces.filter [input, returned, excluded counts]+ response.encode_write+```++The Kubernetes request span includes timestamped connection acquisition, DNS,+TCP, TLS, and first-response-byte events when those phases occur. Reused+connections naturally omit DNS/TCP/TLS setup events. `db.query` measures the+initial query call; result fetching, row decoding, and result-writer work are+measured in `db.rows.read_decode`.++A recovered PostgreSQL failure marks the PostgreSQL child as failed, not the+successful request. The parent records `read.fallback=true`,+`read.fallback.reason=postgres_error`, and a `namespaces.read.fallback` event.+Query text, credentials, HTTP headers, and returned objects are not attached to+these new spans. Error details remain in the fallback log; spans record safe+stage status and error types.++These changes do not instrument authentication middleware or change namespace+creation/deletion. PostgreSQL stage spans also cover `QUERY /api/state/query`,+since both endpoints use the same executor.++## PR previews++Previews seed `ff-list-ns-read-pg=true` into the process-local feature flag+management store. The flag appears in the preview admin feature flags UI and+can be toggled off/on there. Changes affect only that preview pod, never+production SSM, and reset to the seed when the pod restarts.++`cyclops-preview-state-query` imports only `state_query_dsn` and+`state_query_tenant_password` from the existing state database secret. These+credentials connect as the authenticated tenant's read-only RLS role, not as+an application, writer, migration, or role-admin user. Namespace data comes+from the existing production projection, matching the production Kubernetes+cluster that previews already query. The production projector and tenant-role+controller populate that store; previews do not run duplicate controllers or+write production state. The preview's `DATABASE_URL` remains its isolated+`cyclops-preview-database` application connection.++The query credential references are optional so a missing ExternalSecret does+not prevent the backend from starting. Missing credentials leave the executor+unavailable and trigger the normal Kubernetes fallback. Secret values are+injected as environment variables, so restart the preview pod after credentials+first appear or rotate. Once query credentials are present, namespace requests should report `read.backend=postgres`;+force the flag off to compare against `read.backend=kubernetes`.++Only trusted same-repository PR previews receive these credentials, following+the existing preview secret isolation/fork guard. This change does not modify+production flag values, credential values, or production database grants.diff --git a/libs/fleet/backend/handlers/namespaces.go b/libs/fleet/backend/handlers/namespaces.goindex 4686d89ab9..05a0060ed4 100644--- a/libs/fleet/backend/handlers/namespaces.go+++ b/libs/fleet/backend/handlers/namespaces.go@@ -222,75 +222,57 @@ func (h Handlers) ListNamespaces(w http.ResponseWriter, r *http.Request) {attribute.String("user.id", user.ID),))defer span.End()-- // Scope the list to the caller's own Tenant. Every namespace owned by a- // Tenant carries capsule.clastix.io/tenant=<tenant> (set on create — see- // CreateNamespace — and enforced by Capsule). Selecting on it makes this- // fail-closed: even if Capsule Proxy isn't filtering (e.g. while- // CapsuleConfiguration.userGroups is being reconciled, or right after a- // capsule-proxy restart with a cold cache), the apiserver only returns the- // caller's namespaces. The Tenant name follows the standalone controller- // convention "user-<sub>" and is derived from the authenticated subject,- // so a caller can never widen it to another tenant.- selector := "capsule.clastix.io/tenant=" + identity.PersonalGroup(ctx, user.ID)- path := "/api/v1/namespaces?labelSelector=" + url.QueryEscape(selector)- resp, err := h.k8sImpersonate(ctx, "GET", path, nil, user.ID)- if err != nil {- span.RecordError(err)- span.SetStatus(codes.Error, "namespace list request failed")- slog.Warn("namespace list: k8s request failed", "err", err)- writeErr(w, http.StatusBadGateway, "kubectl-proxy unavailable")- return- }- defer resp.Body.Close()-- if resp.StatusCode != http.StatusOK {- body, _ := io.ReadAll(resp.Body)- span.SetStatus(codes.Error, "namespace list returned non-200")- span.SetAttributes(attribute.Int("http.status_code", resp.StatusCode))- slog.Warn("namespace list: k8s error", "status", resp.StatusCode, "body", string(body))- writeErr(w, resp.StatusCode, "k8s: "+string(body))- return+ enabled := namespacePostgresEnabled(ctx, user.ID)+ tenant := namespaceTenant(ctx, user.ID)+ span.SetAttributes(attribute.Bool("feature_flag.ff-list-ns-read-pg", enabled), attribute.Bool("read.fallback", false))+ var items []namespaceObject+ var err error+ backend := "kubernetes"+ if enabled {+ items, err = h.readNamespacesPostgres(ctx, tenant)+ if err == nil {+ backend = "postgres"+ } else {+ span.SetAttributes(attribute.Bool("read.fallback", true), attribute.String("read.fallback.reason", "postgres_error"))+ span.AddEvent("namespaces.read.fallback", trace.WithAttributes(attribute.String("read.fallback.reason", "postgres_error")))+ slog.ErrorContext(ctx, "namespace PostgreSQL read failed; falling back to Kubernetes", "err", err, "trace_id", span.SpanContext().TraceID().String())+ }}-- // Parse K8s NamespaceList.- var nsList struct {- Items []struct {- Metadata struct {- Name string `json:"name"`- Labels map[string]string `json:"labels"`- CreationTimestamp string `json:"creationTimestamp"`- } `json:"metadata"`- Status struct {- Phase string `json:"phase"`- } `json:"status"`- } `json:"items"`- }- if err := json.NewDecoder(resp.Body).Decode(&nsList); err != nil {- span.RecordError(err)- span.SetStatus(codes.Error, "namespace list decode failed")- slog.Warn("namespace list: decode error", "err", err)- writeErr(w, http.StatusBadGateway, "bad response from k8s")- return+ span.SetAttributes(attribute.String("read.backend", backend))+ if backend == "kubernetes" {+ var status int+ items, status, err = h.readNamespacesK8s(ctx, tenant, user.ID)+ if err != nil {+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))+ span.SetStatus(codes.Error, "namespace list failed")+ span.SetAttributes(attribute.Int("http.status_code", status))+ writeErr(w, status, err.Error())+ return+ }}-- out := make([]NamespaceResponse, 0, len(nsList.Items))- for _, ns := range nsList.Items {+ _, filterSpan := handlerTracer().Start(ctx, "namespaces.filter")+ out := make([]NamespaceResponse, 0, len(items))+ for _, ns := range items {if isGitHubPrincipal(user) && !namespaceAllowed(user, ns.Metadata.Name) {continue}out = append(out, NamespaceResponse{- Name: ns.Metadata.Name,- Status: ns.Status.Phase,- CreatedAt: ns.Metadata.CreationTimestamp,- Labels: ns.Metadata.Labels,+ Name: ns.Metadata.Name, Status: ns.Status.Phase,+ CreatedAt: ns.Metadata.CreationTimestamp, Labels: ns.Metadata.Labels,})}- span.SetAttributes(- attribute.Int("http.status_code", http.StatusOK),- attribute.Int("namespaces.count", len(out)),- )- writeJSON(w, http.StatusOK, out)+ filterSpan.SetAttributes(attribute.Int("namespaces.input_count", len(items)), attribute.Int("namespaces.count", len(out)), attribute.Int("namespaces.excluded_count", len(items)-len(out)))+ filterSpan.End()+ span.SetAttributes(attribute.Int("http.status_code", http.StatusOK), attribute.Int("namespaces.count", len(out)))+ _, responseSpan := handlerTracer().Start(ctx, "response.encode_write")+ w.Header().Set("Content-Type", "application/json")+ w.WriteHeader(http.StatusOK)+ err = json.NewEncoder(w).Encode(out)+ finishNamespaceSpan(responseSpan, err)+ if err != nil {+ span.RecordError(err)+ span.SetStatus(codes.Error, "namespace response write failed")+ }}// GetNamespace godocdiff --git a/libs/fleet/backend/handlers/namespaces_pg_test.go b/libs/fleet/backend/handlers/namespaces_pg_test.gonew file mode 100644index 0000000000..b22458b771--- /dev/null+++ b/libs/fleet/backend/handlers/namespaces_pg_test.go@@ -0,0 +1,255 @@+package handlers++import (+ "context"+ "encoding/json"+ "errors"+ "net/http"+ "net/http/httptest"+ "testing"+ "time"++ "cyclops-cs-backend/statequery"+ "github.com/open-feature/go-sdk/openfeature"++ "cyclops-cs-backend/auth"+ "github.com/trycua/cloud/pkg/featureflags"+ "go.opentelemetry.io/otel"+ "go.opentelemetry.io/otel/attribute"+ "go.opentelemetry.io/otel/codes"+ sdktrace "go.opentelemetry.io/otel/sdk/trace"+ "go.opentelemetry.io/otel/sdk/trace/tracetest"+)++func TestListNamespacesPostgresFlag(t *testing.T) {+ for _, test := range []struct {+ name, flag string+ executor *fakeStateQueryExecutor+ want string+ k8sCalls int+ }{+ {"disabled", "false", &fakeStateQueryExecutor{}, "k8s-ns", 1},+ {"missing flag", "", &fakeStateQueryExecutor{}, "k8s-ns", 1},+ {"postgres", "true", &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"pg-ns","labels":{"test":"value"}},"status":{"phase":"Active"}}`}}}, "pg-ns", 0},+ {"empty success", "true", &fakeStateQueryExecutor{}, "", 0},+ {"unavailable", "true", nil, "k8s-ns", 1},+ {"partial failure", "true", &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"partial"}}`}}, err: errors.New("query failed")}, "k8s-ns", 1},+ {"invalid JSON", "true", &fakeStateQueryExecutor{rows: [][]any{{`broken`}}}, "k8s-ns", 1},+ {"invalid column", "true", &fakeStateQueryExecutor{rows: [][]any{{123}}}, "k8s-ns", 1},+ {"invalid column count", "true", &fakeStateQueryExecutor{rows: [][]any{{}}}, "k8s-ns", 1},+ {"missing namespace name", "true", &fakeStateQueryExecutor{rows: [][]any{{`{}`}}}, "k8s-ns", 1},+ } {+ t.Run(test.name, func(t *testing.T) {+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", test.flag)+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {+ t.Fatal(err)+ }+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))+ defer proxy.server.Close()+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")+ handler := Handlers{}+ if test.executor != nil {+ handler.Features = FeaturesWith(test.executor, nil)+ }+ response := httptest.NewRecorder()+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))+ if response.Code != 200 {+ t.Fatalf("status %d: %s", response.Code, response.Body.String())+ }+ var namespaces []NamespaceResponse+ if err := json.Unmarshal(response.Body.Bytes(), &namespaces); err != nil {+ t.Fatal(err)+ }+ if test.want == "" {+ if namespaces == nil || len(namespaces) != 0 {+ t.Fatalf("want [], got %s", response.Body.String())+ }+ } else if len(namespaces) != 1 || namespaces[0].Name != test.want {+ t.Fatalf("namespaces = %+v", namespaces)+ }+ if len(proxy.requests) != test.k8sCalls {+ t.Fatalf("Kubernetes calls = %d, want %d", len(proxy.requests), test.k8sCalls)+ }+ if test.executor != nil {+ if test.flag == "true" && test.executor.tenant != "user-alice" {+ t.Fatalf("tenant = %q", test.executor.tenant)+ }+ if test.flag != "true" && test.executor.query != "" {+ t.Fatal("queried PostgreSQL with flag off")+ }+ }+ })+ }+}++func TestListNamespacesPostgresFilteringAndTracing(t *testing.T) {+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {+ t.Fatal(err)+ }+ recorder := tracetest.NewSpanRecorder()+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))+ previous := otel.GetTracerProvider()+ otel.SetTracerProvider(provider)+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })+ executor := &fakeStateQueryExecutor{rows: [][]any{{`{"metadata":{"name":"allowed"}}`}, {`{"metadata":{"name":"denied"}}`}}}+ handler := Handlers{Features: FeaturesWith(executor, nil)}+ response := httptest.NewRecorder()+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice", PrincipalType: auth.PrincipalTypeGitHubOIDC, AllowedNamespaces: []string{"allowed"}}))+ var namespaces []NamespaceResponse+ if err := json.Unmarshal(response.Body.Bytes(), &namespaces); err != nil {+ t.Fatal(err)+ }+ if len(namespaces) != 1 || namespaces[0].Name != "allowed" {+ t.Fatalf("namespaces = %+v", namespaces)+ }+ for _, name := range []string{"namespaces.list", "feature_flag.evaluate", "identity.resolve_tenant", "namespaces.read.postgres", "namespaces.filter", "response.encode_write"} {+ found := false+ for _, span := range recorder.Ended() {+ if span.Name() == name {+ found = true+ if name != "namespaces.list" && !span.Parent().IsValid() {+ t.Fatalf("%s missing parent", name)+ }+ }+ }+ if !found {+ t.Errorf("missing span %s", name)+ }+ }+}++func TestListNamespacesFallbackTracing(t *testing.T) {+ for _, status := range []int{http.StatusOK, http.StatusForbidden} {+ t.Run(http.StatusText(status), func(t *testing.T) {+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {+ t.Fatal(err)+ }+ proxy := newFakeK8s(status, nsListResponse("k8s-ns"))+ defer proxy.server.Close()+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")+ recorder := tracetest.NewSpanRecorder()+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))+ previous := otel.GetTracerProvider()+ otel.SetTracerProvider(provider)+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })+ handler := Handlers{Features: FeaturesWith(&fakeStateQueryExecutor{err: errors.New("pg failed")}, nil)}+ response := httptest.NewRecorder()+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))+ if response.Code != status {+ t.Fatalf("status = %d, want %d", response.Code, status)+ }+ spans := map[string]sdktrace.ReadOnlySpan{}+ for _, span := range recorder.Ended() {+ spans[span.Name()] = span+ }+ for _, name := range []string{"namespaces.list", "namespaces.read.postgres", "namespaces.read.kubernetes", "kubernetes.request"} {+ if spans[name] == nil {+ t.Fatalf("missing span %s", name)+ }+ }+ parent := spans["namespaces.list"]+ if spans["namespaces.read.postgres"].Status().Code != codes.Error {+ t.Error("PG failure not marked")+ }+ if status == http.StatusOK && parent.Status().Code == codes.Error {+ t.Error("successful fallback marked as error")+ }+ if status != http.StatusOK && parent.Status().Code != codes.Error {+ t.Error("failed fallback not marked as error")+ }+ attrs := map[string]attribute.Value{}+ for _, attr := range parent.Attributes() {+ attrs[string(attr.Key)] = attr.Value+ }+ if !attrs["read.fallback"].AsBool() || attrs["read.backend"].AsString() != "kubernetes" {+ t.Fatalf("fallback attributes = %v", attrs)+ }+ if spans["namespaces.read.postgres"].Parent().SpanID() != parent.SpanContext().SpanID() || spans["namespaces.read.kubernetes"].Parent().SpanID() != parent.SpanContext().SpanID() {+ t.Error("read attempts must be siblings")+ }+ foundFirstByte := false+ for _, event := range spans["kubernetes.request"].Events() {+ if event.Name == "http.response.first_byte" {+ foundFirstByte = true+ }+ }+ if !foundFirstByte {+ t.Error("missing network timing event")+ }+ })+ }+}++type namespaceFlagErrorProvider struct{ featureflags.SimpleEnvProvider }++func (namespaceFlagErrorProvider) BooleanEvaluation(context.Context, string, bool, openfeature.FlattenedContext) openfeature.BoolResolutionDetail {+ return openfeature.BoolResolutionDetail{Value: true, ProviderResolutionDetail: openfeature.ProviderResolutionDetail{ResolutionError: openfeature.NewGeneralResolutionError("flag provider unavailable")}}+}++func TestListNamespacesFlagErrorUsesKubernetes(t *testing.T) {+ if err := openfeature.SetProviderAndWait(&namespaceFlagErrorProvider{}); err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { _ = openfeature.SetProviderAndWait(&featureflags.SimpleEnvProvider{}) })+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))+ defer proxy.server.Close()+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")+ executor := &fakeStateQueryExecutor{}+ handler := Handlers{Features: FeaturesWith(executor, nil)}+ response := httptest.NewRecorder()+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))+ if response.Code != http.StatusOK || len(proxy.requests) != 1 || executor.query != "" {+ t.Fatalf("flag error did not use Kubernetes: status=%d requests=%d query=%q", response.Code, len(proxy.requests), executor.query)+ }+}++type namespaceDeadlineExecutor struct{ testing *testing.T }++func (executor namespaceDeadlineExecutor) Execute(ctx context.Context, _, _ string, _ statequery.ResultWriter) error {+ deadline, ok := ctx.Deadline()+ if !ok || time.Until(deadline) > namespacePostgresTimeout {+ executor.testing.Fatal("PostgreSQL attempt is not bounded")+ }+ <-ctx.Done()+ return ctx.Err()+}++func TestListNamespacesDeadlineFallsBackWithLiveContext(t *testing.T) {+ t.Setenv("CYCLOPS_CS_FF_LIST_NS_READ_PG", "true")+ if err := featureflags.SetupProvider(context.Background(), "development", featureflags.AWSCredentials{}); err != nil {+ t.Fatal(err)+ }+ proxy := newFakeK8s(http.StatusOK, nsListResponse("k8s-ns"))+ defer proxy.server.Close()+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")+ handler := Handlers{Features: FeaturesWith(namespaceDeadlineExecutor{testing: t}, nil)}+ response := httptest.NewRecorder()+ handler.ListNamespaces(response, withUser(httptest.NewRequest("GET", "/api/namespaces", nil), &auth.User{ID: "alice"}))+ if response.Code != http.StatusOK || len(proxy.requests) != 1 {+ t.Fatalf("deadline fallback failed: status=%d requests=%d", response.Code, len(proxy.requests))+ }+}++func TestNamespaceReadErrorsPreserveCauses(t *testing.T) {+ writer := &namespaceResultWriter{}+ var syntaxErr *json.SyntaxError+ if err := writer.WriteRow([]any{"invalid"}); !errors.As(err, &syntaxErr) || err.Error() != "namespace query returned invalid object" {+ t.Fatalf("namespace row decode must preserve a redacted cause: %v", err)+ }+ proxy := newFakeK8s(http.StatusOK, "invalid")+ defer proxy.server.Close()+ overrideK8sClient(proxy.server.Client(), proxy.server.URL, "token")+ handler := Handlers{}+ _, status, err := handler.readNamespacesK8s(context.Background(), "user-alice", "alice")+ if status != http.StatusBadGateway || !errors.As(err, &syntaxErr) || err.Error() != "bad response from k8s" {+ t.Fatalf("Kubernetes decode must preserve a redacted cause: %d %v", status, err)+ }+ ctx, cancel := context.WithCancel(context.Background())+ cancel()+ _, status, err = handler.readNamespacesK8s(ctx, "user-alice", "alice")+ if status != http.StatusBadGateway || !errors.Is(err, context.Canceled) || err.Error() != "kubectl-proxy unavailable" {+ t.Fatalf("Kubernetes transport must preserve a redacted cause: %d %v", status, err)+ }+}diff --git a/libs/fleet/backend/handlers/namespaces_read.go b/libs/fleet/backend/handlers/namespaces_read.gonew file mode 100644index 0000000000..24de4f8963--- /dev/null+++ b/libs/fleet/backend/handlers/namespaces_read.go@@ -0,0 +1,159 @@+package handlers++import (+ "context"+ "crypto/tls"+ "encoding/json"+ "errors"+ "fmt"+ "io"+ "log/slog"+ "net/http"+ "net/http/httptrace"+ "net/url"+ "time"++ "cyclops-cs-backend/identity"+ "cyclops-cs-backend/internal/redactederror"+ "github.com/jackc/pgx/v5/pgconn"+ "github.com/open-feature/go-sdk/openfeature"+ "go.opentelemetry.io/otel/attribute"+ "go.opentelemetry.io/otel/codes"+ "go.opentelemetry.io/otel/trace"+)++const namespacePostgresFlag = "/feature-flags/cyclops-cs/ff-list-ns-read-pg"+const namespacePostgresTimeout = 5 * time.Second+const namespaceListSQL = `SELECT object::text FROM k8s_api.current_resources+WHERE api_group = '' AND resource = 'namespaces' AND namespace = '' ORDER BY name`++type namespaceObject struct {+ Metadata struct {+ Name string `json:"name"`+ Labels map[string]string `json:"labels"`+ CreationTimestamp string `json:"creationTimestamp"`+ } `json:"metadata"`+ Status struct {+ Phase string `json:"phase"`+ } `json:"status"`+}++type namespaceResultWriter struct{ items []namespaceObject }++func (writer *namespaceResultWriter) WriteFieldDescriptions(_ []pgconn.FieldDescription) error {+ return nil+}+func (writer *namespaceResultWriter) WriteRow(values []any) error {+ if len(values) != 1 {+ return errors.New("namespace query returned unexpected columns")+ }+ object, ok := values[0].(string)+ if !ok {+ return errors.New("namespace query returned non-text object")+ }+ var item namespaceObject+ if err := json.Unmarshal([]byte(object), &item); err != nil {+ return redactederror.New("namespace query returned invalid object", err)+ }+ if item.Metadata.Name == "" {+ return errors.New("namespace query returned object without name")+ }+ writer.items = append(writer.items, item)+ return nil+}++func finishNamespaceSpan(span trace.Span, err error) {+ if err != nil {+ span.RecordError(errors.New("namespace operation failed"))+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))+ span.SetStatus(codes.Error, "namespace operation failed")+ }+ span.End()+}++func namespacePostgresEnabled(ctx context.Context, subject string) bool {+ ctx, span := handlerTracer().Start(ctx, "feature_flag.evaluate", trace.WithAttributes(attribute.String("feature_flag.key", namespacePostgresFlag)))+ callCtx, cancel := context.WithTimeout(ctx, 3*time.Second)+ defer cancel()+ enabled, err := openfeature.NewClient("cyclops-cs-namespaces").BooleanValue(callCtx, namespacePostgresFlag, false, openfeature.NewEvaluationContext(subject, nil))+ if err != nil {+ enabled = false+ slog.WarnContext(ctx, "namespace read flag evaluation failed; using Kubernetes", "err", err)+ }+ span.SetAttributes(attribute.Bool("feature_flag.result.value", enabled))+ finishNamespaceSpan(span, err)+ return enabled+}++func namespaceTenant(ctx context.Context, subject string) string {+ ctx, span := handlerTracer().Start(ctx, "identity.resolve_tenant")+ defer span.End()+ return identity.PersonalGroup(ctx, subject)+}++func (h Handlers) readNamespacesPostgres(ctx context.Context, tenant string) (items []namespaceObject, err error) {+ ctx, span := handlerTracer().Start(ctx, "namespaces.read.postgres", trace.WithAttributes(attribute.String("read.backend", "postgres")))+ defer func() { finishNamespaceSpan(span, err) }()+ executor := h.Features.StateQuery()+ if executor == nil {+ return nil, errors.New("namespace PostgreSQL reader unavailable")+ }+ callCtx, cancel := context.WithTimeout(ctx, namespacePostgresTimeout)+ defer cancel()+ writer := &namespaceResultWriter{}+ if err = executor.Execute(callCtx, tenant, namespaceListSQL, writer); err != nil {+ return nil, err+ }+ span.SetAttributes(attribute.Int("namespaces.count", len(writer.items)))+ return writer.items, nil+}++func (h Handlers) readNamespacesK8s(ctx context.Context, tenant, subject string) (items []namespaceObject, status int, err error) {+ ctx, span := handlerTracer().Start(ctx, "namespaces.read.kubernetes", trace.WithAttributes(attribute.String("read.backend", "kubernetes")))+ defer func() { span.SetAttributes(attribute.Int("http.status_code", status)); finishNamespaceSpan(span, err) }()+ path := "/api/v1/namespaces?labelSelector=" + url.QueryEscape("capsule.clastix.io/tenant="+tenant)+ requestCtx, requestSpan := handlerTracer().Start(ctx, "kubernetes.request", trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(attribute.String("http.request.method", "GET"), attribute.String("k8s.resource.name", "namespaces")))+ requestCtx = httptrace.WithClientTrace(requestCtx, &httptrace.ClientTrace{+ GetConn: func(_ string) { requestSpan.AddEvent("http.connection.acquire") },+ GotConn: func(info httptrace.GotConnInfo) {+ requestSpan.SetAttributes(attribute.Bool("http.connection.reused", info.Reused))+ requestSpan.AddEvent("http.connection.acquired")+ },+ DNSStart: func(_ httptrace.DNSStartInfo) { requestSpan.AddEvent("dns.start") },+ DNSDone: func(_ httptrace.DNSDoneInfo) { requestSpan.AddEvent("dns.done") },+ ConnectStart: func(_, _ string) { requestSpan.AddEvent("tcp.connect.start") },+ ConnectDone: func(_, _ string, _ error) { requestSpan.AddEvent("tcp.connect.done") },+ TLSHandshakeStart: func() { requestSpan.AddEvent("tls.handshake.start") },+ TLSHandshakeDone: func(_ tls.ConnectionState, _ error) { requestSpan.AddEvent("tls.handshake.done") },+ GotFirstResponseByte: func() { requestSpan.AddEvent("http.response.first_byte") },+ })+ resp, requestErr := h.k8sImpersonate(requestCtx, "GET", path, nil, subject)+ if requestErr != nil {+ finishNamespaceSpan(requestSpan, requestErr)+ slog.WarnContext(ctx, "namespace list: k8s request failed", "err", requestErr)+ return nil, http.StatusBadGateway, redactederror.New("kubectl-proxy unavailable", requestErr)+ }+ defer resp.Body.Close()+ requestSpan.SetAttributes(attribute.Int("http.response.status_code", resp.StatusCode))+ if resp.StatusCode != http.StatusOK {+ body, readErr := io.ReadAll(resp.Body)+ statusErr := fmt.Errorf("k8s: %s", body)+ if readErr != nil {+ statusErr = redactederror.New("k8s response read failed", readErr)+ }+ finishNamespaceSpan(requestSpan, statusErr)+ return nil, resp.StatusCode, statusErr+ }+ finishNamespaceSpan(requestSpan, nil)+ _, decodeSpan := handlerTracer().Start(ctx, "namespaces.decode")+ var list struct {+ Items []namespaceObject `json:"items"`+ }+ decodeErr := json.NewDecoder(resp.Body).Decode(&list)+ finishNamespaceSpan(decodeSpan, decodeErr)+ if decodeErr != nil {+ return nil, http.StatusBadGateway, redactederror.New("bad response from k8s", decodeErr)+ }+ span.SetAttributes(attribute.Int("namespaces.count", len(list.Items)))+ return list.Items, http.StatusOK, nil+}diff --git a/libs/fleet/backend/statequery/executor.go b/libs/fleet/backend/statequery/executor.goindex 8a2f466f26..0b317e54e1 100644--- a/libs/fleet/backend/statequery/executor.go+++ b/libs/fleet/backend/statequery/executor.go@@ -6,6 +6,12 @@ import ("encoding/hex""errors""fmt"+ "time"++ "go.opentelemetry.io/otel"+ "go.opentelemetry.io/otel/attribute"+ "go.opentelemetry.io/otel/codes"+ "go.opentelemetry.io/otel/trace""github.com/jackc/pgx/v5""github.com/jackc/pgx/v5/pgconn"@@ -46,30 +52,67 @@ func (e *Executor) connectionConfig(tenant string) *pgx.ConnConfig {return config}-func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer ResultWriter) error {+func finishDatabaseSpan(span trace.Span, err error) {+ if err != nil {+ span.SetAttributes(attribute.String("error.type", fmt.Sprintf("%T", err)))+ var postgresErr *pgconn.PgError+ if errors.As(err, &postgresErr) {+ span.SetAttributes(attribute.String("db.postgresql.sqlstate", postgresErr.Code))+ }+ span.SetStatus(codes.Error, "database operation failed")+ }+ span.End()+}++func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer ResultWriter) (resultErr error) {+ tracer := otel.Tracer("cyclops-cs-backend/statequery")+ ctx, span := tracer.Start(ctx, "db.execute", trace.WithAttributes(attribute.String("db.system", "postgresql")))+ defer func() { finishDatabaseSpan(span, resultErr) }()if tenant == "" {return fmt.Errorf("%w: tenant is required", ErrExecution)}- conn, err := pgx.ConnectConfig(ctx, e.connectionConfig(tenant))+ connectCtx, connectSpan := tracer.Start(ctx, "db.connect", trace.WithSpanKind(trace.SpanKindClient))+ conn, err := pgx.ConnectConfig(connectCtx, e.connectionConfig(tenant))+ finishDatabaseSpan(connectSpan, err)if err != nil {return errors.Join(fmt.Errorf("%w: connect tenant role: %v", ErrExecution, err), err)}- defer conn.Close(ctx)+ defer func() {+ closeCtx, closeSpan := tracer.Start(context.WithoutCancel(ctx), "db.connection.close")+ closeCtx, cancel := context.WithTimeout(closeCtx, 3*time.Second)+ defer cancel()+ finishDatabaseSpan(closeSpan, conn.Close(closeCtx))+ }()- tx, err := conn.BeginTx(ctx, pgx.TxOptions{AccessMode: pgx.ReadOnly})+ beginCtx, beginSpan := tracer.Start(ctx, "db.transaction.begin", trace.WithAttributes(attribute.Bool("db.transaction.read_only", true)))+ tx, err := conn.BeginTx(beginCtx, pgx.TxOptions{AccessMode: pgx.ReadOnly})+ finishDatabaseSpan(beginSpan, err)if err != nil {return errors.Join(fmt.Errorf("%w: begin transaction", ErrExecution), err)}- defer tx.Rollback(ctx)+ defer func() {+ rollbackCtx, rollbackSpan := tracer.Start(context.WithoutCancel(ctx), "db.transaction.rollback")+ rollbackCtx, cancel := context.WithTimeout(rollbackCtx, 3*time.Second)+ defer cancel()+ finishDatabaseSpan(rollbackSpan, tx.Rollback(rollbackCtx))+ }()- rows, err := tx.Query(ctx, sql)+ queryCtx, querySpan := tracer.Start(ctx, "db.query", trace.WithSpanKind(trace.SpanKindClient))+ rows, err := tx.Query(queryCtx, sql)+ finishDatabaseSpan(querySpan, err)if err != nil {return errors.Join(fmt.Errorf("%w: execute query", ErrExecution), err)}- defer rows.Close()+ _, rowsSpan := tracer.Start(ctx, "db.rows.read_decode")+ rowCount := 0+ defer func() {+ rows.Close()+ rowsSpan.SetAttributes(attribute.Int("db.response.returned_rows", rowCount))+ finishDatabaseSpan(rowsSpan, resultErr)+ }()if err := writer.WriteFieldDescriptions(rows.FieldDescriptions()); err != nil {return err}@@ -82,7 +125,7 @@ func (e *Executor) Execute(ctx context.Context, tenant, sql string, writer Resulif err := writer.WriteRow(values); err != nil {return err}+ rowCount++}- rows.Close()return rows.Err()}diff --git a/libs/fleet/backend/statequery/tracing_test.go b/libs/fleet/backend/statequery/tracing_test.gonew file mode 100644index 0000000000..472c7fd06d--- /dev/null+++ b/libs/fleet/backend/statequery/tracing_test.go@@ -0,0 +1,59 @@+package statequery++import (+ "context"+ "net"+ "testing"++ "go.opentelemetry.io/otel"+ "go.opentelemetry.io/otel/codes"+ sdktrace "go.opentelemetry.io/otel/sdk/trace"+ "go.opentelemetry.io/otel/sdk/trace/tracetest"+)++func TestExecutorTracesConnectionFailure(t *testing.T) {+ listener, err := net.Listen("tcp", "127.0.0.1:0")+ if err != nil {+ t.Fatal(err)+ }+ address := listener.Addr().String()+ _ = listener.Close()+ recorder := tracetest.NewSpanRecorder()+ provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(recorder))+ previous := otel.GetTracerProvider()+ otel.SetTracerProvider(provider)+ t.Cleanup(func() { otel.SetTracerProvider(previous); _ = provider.Shutdown(context.Background()) })+ executor, err := NewExecutor("postgres://"+address+"/cyclops?sslmode=disable&connect_timeout=1", "secret")+ if err != nil {+ t.Fatal(err)+ }+ ctx, parent := provider.Tracer("test").Start(context.Background(), "request")+ err = executor.Execute(ctx, "user-alice", "select 'sensitive-query'", &collectingResultWriter{})+ parent.End()+ if err == nil {+ t.Fatal("expected connection failure")+ }+ for _, name := range []string{"db.execute", "db.connect"} {+ found := false+ for _, span := range recorder.Ended() {+ if span.Name() != name {+ continue+ }+ found = true+ if span.Status().Code != codes.Error {+ t.Errorf("%s did not report error", name)+ }+ if !span.Parent().IsValid() {+ t.Errorf("%s missing parent", name)+ }+ for _, attr := range span.Attributes() {+ if string(attr.Key) == "db.statement" || string(attr.Key) == "db.query.text" {+ t.Error("query text exposed")+ }+ }+ }+ if !found {+ t.Errorf("missing span %s", name)+ }+ }+}