The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
feat(skills): add kubernetes-patterns skill (#2178)
Sahil Agharae116d69c65a9
diff --git a/skills/kubernetes-patterns/SKILL.md b/skills/kubernetes-patterns/SKILL.mdnew file mode 100644index 000000000..1d5a23e89--- /dev/null+++ b/skills/kubernetes-patterns/SKILL.md@@ -0,0 +1,755 @@+---+name: kubernetes-patterns+description: Kubernetes workload patterns, resource management, RBAC, probes, autoscaling, ConfigMap/Secret handling, and kubectl debugging for production-grade deployments.+origin: ECC+---++# Kubernetes Patterns++Production-grade Kubernetes patterns for deploying, managing, and debugging workloads reliably.++## When to Activate++- Writing Kubernetes manifests (Deployments, Services, Ingress, Jobs)+- Configuring resource requests/limits, liveness/readiness probes+- Setting up RBAC, namespaces, or ServiceAccounts+- Managing configuration and secrets in K8s+- Debugging CrashLoopBackOff, OOMKilled, pending pods, or image pull errors+- Configuring HPA (Horizontal Pod Autoscaler) or PodDisruptionBudgets+- Reviewing K8s YAML for security or correctness++## When to Use++> Same as **When to Activate** above. This alias satisfies repo skill-format conventions. Use this skill any time you are writing, reviewing, or debugging Kubernetes YAML and workloads.++## How It Works++This skill provides **copy-pasteable, production-grade YAML patterns** and **kubectl debugging commands** organized by task:++1. **Deployment template** — A fully configured production `Deployment` with security context, rolling update strategy, all three probe types, resource limits, and environment injection from ConfigMap/Secret.+2. **Probes** — Decision table for startup vs liveness vs readiness, with correct `failureThreshold × periodSeconds` math.+3. **Services & Ingress** — ClusterIP, LoadBalancer, and TLS Ingress patterns with cert-manager annotations.+4. **ConfigMaps & Secrets** — `envFrom`, file-mount, and external secrets guidance.+5. **Resource management** — Requests vs limits rules of thumb by workload type (web API, JVM, worker, sidecar).+6. **RBAC** — Least-privilege ServiceAccount → Role → RoleBinding chain.+7. **HPA & PDB** — Autoscaling and node-drain safety configurations.+8. **Jobs & CronJobs** — One-off and scheduled workload patterns with correct `restartPolicy`.+9. **kubectl cheatsheet** — Logs, exec, rollback, port-forward, dry-run, and common error diagnosis commands.+10. **Anti-patterns & checklist** — What NOT to do, and a security/reliability/observability checklist.++## Examples++See the sections below for complete, runnable examples. Quick references:++| Task | Jump to |+|------|---------|+| Full production Deployment YAML | [Core Workload Patterns](#core-workload-patterns) |+| Probe configuration | [Probes](#probes--liveness-readiness-startup) |+| RBAC least-privilege setup | [RBAC](#rbac--roles-and-serviceaccounts) |+| Debug a CrashLoopBackOff | [kubectl Debugging Cheatsheet](#kubectl-debugging-cheatsheet) |+| Autoscaling | [HPA](#horizontal-pod-autoscaler-hpa) |++---++## Core Workload Patterns++### Deployment — Production Template++```yaml+apiVersion: apps/v1+kind: Deployment+metadata:+ name: my-app+ namespace: my-namespace+ labels:+ app: my-app+ version: "1.0.0"+spec:+ replicas: 3+ selector:+ matchLabels:+ app: my-app+ strategy:+ type: RollingUpdate+ rollingUpdate:+ maxSurge: 1 # Allow 1 extra pod during update+ maxUnavailable: 0 # Never reduce below desired count+ template:+ metadata:+ labels:+ app: my-app+ version: "1.0.0"+ spec:+ # Security context at pod level+ securityContext:+ runAsNonRoot: true+ runAsUser: 1001+ fsGroup: 1001++ # Graceful shutdown+ terminationGracePeriodSeconds: 30++ containers:+ - name: my-app+ image: ghcr.io/org/my-app:1.0.0 # Never use :latest+ imagePullPolicy: IfNotPresent++ ports:+ - containerPort: 8080+ protocol: TCP++ # Resource requests AND limits are both required+ resources:+ requests:+ cpu: "100m"+ memory: "128Mi"+ limits:+ cpu: "500m"+ memory: "256Mi"++ # Container security context+ securityContext:+ allowPrivilegeEscalation: false+ readOnlyRootFilesystem: true+ capabilities:+ drop:+ - ALL++ # Probes (see Probes section below)+ startupProbe:+ httpGet:+ path: /health+ port: 8080+ failureThreshold: 30+ periodSeconds: 5+ livenessProbe:+ httpGet:+ path: /health+ port: 8080+ initialDelaySeconds: 0+ periodSeconds: 30+ failureThreshold: 3+ readinessProbe:+ httpGet:+ path: /ready+ port: 8080+ initialDelaySeconds: 5+ periodSeconds: 10+ failureThreshold: 2++ # Environment from ConfigMap and Secret+ envFrom:+ - configMapRef:+ name: my-app-config+ env:+ - name: DB_PASSWORD+ valueFrom:+ secretKeyRef:+ name: my-app-secrets+ key: db-password++ # Writable tmp directory when readOnlyRootFilesystem: true+ volumeMounts:+ - name: tmp+ mountPath: /tmp++ volumes:+ - name: tmp+ emptyDir: {}+```++---++## Probes — Liveness, Readiness, Startup++Understanding when to use each probe is critical:++| Probe | Failure Action | Use For |+|-------|---------------|---------|+| `startupProbe` | Kills container if slow to start | Slow-starting apps (JVM, Python) |+| `livenessProbe` | Restarts container | Deadlock / hung process detection |+| `readinessProbe` | Removes from Service endpoints | Temporary unavailability (DB reconnect) |++```yaml+# Correct pattern: startupProbe covers slow startup,+# then liveness/readiness take over+startupProbe:+ httpGet:+ path: /health+ port: 8080+ failureThreshold: 30 # 30 * 5s = 150s max startup time+ periodSeconds: 5++livenessProbe:+ httpGet:+ path: /health+ port: 8080+ periodSeconds: 30+ failureThreshold: 3 # 3 * 30s = 90s before restart++readinessProbe:+ httpGet:+ path: /ready # Separate endpoint: checks DB, cache, etc.+ port: 8080+ periodSeconds: 10+ failureThreshold: 2+```++```yaml+# WRONG: initialDelaySeconds without startupProbe+# If the app takes 60s to start, set a startupProbe instead+livenessProbe:+ httpGet:+ path: /health+ port: 8080+ initialDelaySeconds: 60 # BAD: Arbitrary wait, race condition+```++---++## Services and Ingress++### Service Types++```yaml+# ClusterIP (default) — internal-only+apiVersion: v1+kind: Service+metadata:+ name: my-app+ namespace: my-namespace+spec:+ selector:+ app: my-app+ ports:+ - port: 80+ targetPort: 8080+ protocol: TCP+ type: ClusterIP+```++```yaml+# LoadBalancer — external traffic (cloud providers)+spec:+ type: LoadBalancer+ ports:+ - port: 443+ targetPort: 8080+```++### Ingress with TLS++```yaml+apiVersion: networking.k8s.io/v1+kind: Ingress+metadata:+ name: my-app+ namespace: my-namespace+ annotations:+ nginx.ingress.kubernetes.io/ssl-redirect: "true"+ cert-manager.io/cluster-issuer: "letsencrypt-prod"+spec:+ ingressClassName: nginx+ tls:+ - hosts:+ - myapp.example.com+ secretName: my-app-tls+ rules:+ - host: myapp.example.com+ http:+ paths:+ - path: /+ pathType: Prefix+ backend:+ service:+ name: my-app+ port:+ number: 80+```++---++## ConfigMaps and Secrets++### ConfigMap — Non-sensitive configuration++```yaml+apiVersion: v1+kind: ConfigMap+metadata:+ name: my-app-config+ namespace: my-namespace+data:+ LOG_LEVEL: "info"+ APP_ENV: "production"+ MAX_CONNECTIONS: "100"+ # Mount as a file for complex config+ app.yaml: |+ server:+ port: 8080+ timeout: 30s+```++```yaml+# Mount ConfigMap as a file+volumes:+ - name: config+ configMap:+ name: my-app-config+ items:+ - key: app.yaml+ path: app.yaml+volumeMounts:+ - name: config+ mountPath: /etc/app+ readOnly: true+```++### Secrets — Sensitive data++```bash+# Create secret from literal (CLI, then store in Vault/SOPS)+kubectl create secret generic my-app-secrets \+ --from-literal=db-password='s3cr3t' \+ --namespace=my-namespace \+ --dry-run=client -o yaml | kubectl apply -f -+```++```yaml+apiVersion: v1+kind: Secret+metadata:+ name: my-app-secrets+ namespace: my-namespace+type: Opaque+# Values are base64-encoded (NOT encrypted — use Sealed Secrets or ESO for real encryption)+data:+ db-password: czNjcjN0 # base64 of 's3cr3t'+```++> **Important:** Raw Kubernetes Secrets are only base64-encoded, not encrypted at rest unless your cluster has encryption configured. Use [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) or [External Secrets Operator](https://external-secrets.io) for production.++---++## Resource Requests and Limits++```yaml+resources:+ requests: # Scheduler uses this to place the pod+ cpu: "100m" # 100 millicores = 0.1 CPU+ memory: "128Mi"+ limits: # Container is killed/throttled above this+ cpu: "500m"+ memory: "256Mi"+```++**Rules of thumb:**++| Workload Type | CPU Request | Memory Request | Notes |+|---------------|-------------|----------------|-------|+| Web API | 100–250m | 128–256Mi | Set limits 2-4x requests |+| Worker/consumer | 250–500m | 256–512Mi | Memory limit = request for predictability |+| JVM app | 500m–1 | 512Mi–2Gi | Allow headroom above `-Xmx` for JVM overhead |+| Sidecar | 10–50m | 32–64Mi | Keep minimal |++```yaml+# WRONG: No requests or limits — unpredictable scheduling, OOM evictions+containers:+ - name: app+ image: myapp:latest+ # Missing resources: {} — this is dangerous in production++# WRONG: Limits without requests — requests default to limits, over-reserves capacity+resources:+ limits:+ cpu: "2"+ memory: "1Gi"+ # requests missing — will default to limits values+```++---++## RBAC — Roles and ServiceAccounts++### Principle of Least Privilege++**Two patterns depending on whether the app calls the Kubernetes API:**++#### Pattern A — App does NOT need the Kubernetes API (most apps)++Disable token automounting on the ServiceAccount. The Role/RoleBinding are not needed.++```yaml+# ServiceAccount with token disabled — safest default+apiVersion: v1+kind: ServiceAccount+metadata:+ name: my-app-sa+ namespace: my-namespace+automountServiceAccountToken: false # No K8s API token injected into pods+```++```yaml+# Reference in Deployment — no token, no API access+spec:+ template:+ spec:+ serviceAccountName: my-app-sa+ automountServiceAccountToken: false # Belt-and-suspenders: also set at pod level+```++#### Pattern B — App DOES need the Kubernetes API (operators, controllers, config watchers)++Enable the token and grant only the permissions actually required.++```yaml+# 1. ServiceAccount — enable token for this SA+apiVersion: v1+kind: ServiceAccount+metadata:+ name: my-app-sa+ namespace: my-namespace+automountServiceAccountToken: true # Token required: app calls K8s API+```++```yaml+# 2. Role — grant only what the app needs (namespace-scoped)+apiVersion: rbac.authorization.k8s.io/v1+kind: Role+metadata:+ name: my-app-role+ namespace: my-namespace+rules:+ - apiGroups: [""]+ resources: ["configmaps"]+ verbs: ["get", "list", "watch"] # Read-only, specific resource+ - apiGroups: [""]+ resources: ["secrets"]+ resourceNames: ["my-app-secrets"] # Restrict to specific secret by name+ verbs: ["get"]+```++```yaml+# 3. Bind Role to ServiceAccount+apiVersion: rbac.authorization.k8s.io/v1+kind: RoleBinding+metadata:+ name: my-app-rolebinding+ namespace: my-namespace+subjects:+ - kind: ServiceAccount+ name: my-app-sa+ namespace: my-namespace+roleRef:+ kind: Role+ apiGroup: rbac.authorization.k8s.io+ name: my-app-role+```++```yaml+# 4. Reference SA in Deployment+spec:+ template:+ spec:+ serviceAccountName: my-app-sa+ # automountServiceAccountToken defaults to true from SA — token is injected+```++---++## Horizontal Pod Autoscaler (HPA)++```yaml+apiVersion: autoscaling/v2+kind: HorizontalPodAutoscaler+metadata:+ name: my-app-hpa+ namespace: my-namespace+spec:+ scaleTargetRef:+ apiVersion: apps/v1+ kind: Deployment+ name: my-app+ minReplicas: 2 # Always at least 2 for HA+ maxReplicas: 10+ metrics:+ - type: Resource+ resource:+ name: cpu+ target:+ type: Utilization+ averageUtilization: 70 # Scale up when avg CPU > 70%+ - type: Resource+ resource:+ name: memory+ target:+ type: Utilization+ averageUtilization: 80+```++> HPA requires `resources.requests` to be set on all containers — it calculates utilization as `current / request`.++---++## PodDisruptionBudget (PDB)++Prevent too many pods going down during node drains or rolling updates:++```yaml+apiVersion: policy/v1+kind: PodDisruptionBudget+metadata:+ name: my-app-pdb+ namespace: my-namespace+spec:+ minAvailable: 2 # OR use maxUnavailable: 1+ selector:+ matchLabels:+ app: my-app+```++---++## Namespaces and Multi-Tenancy++```bash+# Create namespace with resource quotas+kubectl create namespace my-namespace++# Apply ResourceQuota to limit namespace consumption+kubectl apply -f - <<EOF+apiVersion: v1+kind: ResourceQuota+metadata:+ name: my-namespace-quota+ namespace: my-namespace+spec:+ hard:+ requests.cpu: "4"+ requests.memory: 4Gi+ limits.cpu: "8"+ limits.memory: 8Gi+ pods: "20"+EOF+```++---++## Jobs and CronJobs++```yaml+# One-off Job (DB migration, data processing)+apiVersion: batch/v1+kind: Job+metadata:+ name: db-migrate+ namespace: my-namespace+spec:+ backoffLimit: 3 # Retry up to 3 times on failure+ ttlSecondsAfterFinished: 3600 # Auto-delete after 1h+ template:+ spec:+ restartPolicy: OnFailure # Never for Jobs (not Always)+ containers:+ - name: migrate+ image: ghcr.io/org/my-app:1.0.0+ command: ["python", "manage.py", "migrate"]+ resources:+ requests:+ cpu: "100m"+ memory: "256Mi"+```++```yaml+# CronJob+apiVersion: batch/v1+kind: CronJob+metadata:+ name: cleanup-job+ namespace: my-namespace+spec:+ schedule: "0 2 * * *" # 2am daily+ concurrencyPolicy: Forbid # Don't run if previous still running+ successfulJobsHistoryLimit: 3+ failedJobsHistoryLimit: 1+ jobTemplate:+ spec:+ template:+ spec:+ restartPolicy: OnFailure+ containers:+ - name: cleanup+ image: ghcr.io/org/cleanup:1.0.0+ resources:+ requests:+ cpu: "50m"+ memory: "64Mi"+```++---++## kubectl Debugging Cheatsheet++```bash+# --- Pod status and logs ---+kubectl get pods -n my-namespace+kubectl get pods -n my-namespace -o wide # Show node assignment+kubectl describe pod <pod-name> -n my-namespace # Events and state details+kubectl logs <pod-name> -n my-namespace # Current logs+kubectl logs <pod-name> -n my-namespace --previous # Logs from crashed container+kubectl logs <pod-name> -n my-namespace -c <container> # Multi-container pod++# --- Execute into a running container ---+kubectl exec -it <pod-name> -n my-namespace -- sh+kubectl exec -it <pod-name> -n my-namespace -- bash++# --- Check resource usage ---+kubectl top pods -n my-namespace+kubectl top nodes++# --- Deployment operations ---+kubectl rollout status deployment/my-app -n my-namespace+kubectl rollout history deployment/my-app -n my-namespace+kubectl rollout undo deployment/my-app -n my-namespace # Rollback+kubectl rollout undo deployment/my-app --to-revision=2 -n my-namespace++# --- Scale manually ---+kubectl scale deployment my-app --replicas=5 -n my-namespace++# --- Inspect events (cluster-wide issues) ---+kubectl get events -n my-namespace --sort-by='.lastTimestamp'++# --- Port-forward for local debugging ---+kubectl port-forward pod/<pod-name> 8080:8080 -n my-namespace+kubectl port-forward svc/my-app 8080:80 -n my-namespace++# --- Dry-run to validate YAML ---+kubectl apply -f deployment.yaml --dry-run=client+kubectl apply -f deployment.yaml --dry-run=server # Validates against live cluster+```++### Diagnosing Common Errors++```bash+# CrashLoopBackOff: container keeps crashing+kubectl logs <pod-name> --previous -n my-namespace # Check crash logs+kubectl describe pod <pod-name> -n my-namespace # Check exit code & OOMKilled++# ImagePullBackOff: can't pull image+kubectl describe pod <pod-name> -n my-namespace # Check Events section+# Causes: wrong image tag, missing imagePullSecret, private registry++# Pending pod: not scheduled+kubectl describe pod <pod-name> -n my-namespace+# Causes: insufficient resources, no matching node selector, taint/toleration mismatch++# OOMKilled: out of memory+# Increase memory limits, check for memory leaks+kubectl describe pod <pod-name> -n my-namespace | grep -A5 "Last State"+```++---++## Anti-Patterns++```yaml+# BAD: Using :latest tag — non-deterministic deployments+image: myapp:latest++# GOOD: Pin to a specific immutable tag (SHA or semver)+image: ghcr.io/org/myapp:1.4.2+# or+image: ghcr.io/org/myapp@sha256:abc123...++# ---++# BAD: Running as root+securityContext: {} # Defaults to root++# GOOD: Non-root with explicit UID+securityContext:+ runAsNonRoot: true+ runAsUser: 1001++# ---++# BAD: No resource limits — one pod can starve the entire node+containers:+ - name: app+ image: myapp:1.0.0+ # No resources defined++# GOOD: Always set requests and limits+resources:+ requests:+ cpu: "100m"+ memory: "128Mi"+ limits:+ cpu: "500m"+ memory: "256Mi"++# ---++# BAD: Storing plaintext secrets in ConfigMaps+apiVersion: v1+kind: ConfigMap+data:+ DB_PASSWORD: "mysecretpassword" # NEVER — use Secret or external secrets manager++# ---++# BAD: ClusterAdmin for application service accounts+apiVersion: rbac.authorization.k8s.io/v1+kind: ClusterRoleBinding+roleRef:+ kind: ClusterRole+ name: cluster-admin # Grants god-mode to your app++# ---++# BAD: minAvailable: 0 in PDB — defeats the purpose+spec:+ minAvailable: 0++# ---++# BAD: restartPolicy: Always in a Job (causes infinite restart loop)+spec:+ restartPolicy: Always # Use OnFailure or Never for Jobs+```++---++## Best Practices Checklist++### Security+- [ ] Container runs as non-root (`runAsNonRoot: true`, `runAsUser` set)+- [ ] `readOnlyRootFilesystem: true` with `emptyDir` for writable paths+- [ ] `allowPrivilegeEscalation: false`+- [ ] All capabilities dropped (`capabilities.drop: [ALL]`)+- [ ] Dedicated ServiceAccount per app, not `default`+- [ ] `automountServiceAccountToken: false` unless needed+- [ ] RBAC follows least privilege (use `Role`, not `ClusterRole` unless needed)+- [ ] Secrets managed via Sealed Secrets or External Secrets Operator++### Reliability+- [ ] All 3 probe types configured (startup + liveness + readiness)+- [ ] Resource requests AND limits set on every container+- [ ] `minReplicas: 2+` for any production workload+- [ ] PodDisruptionBudget defined for stateful or critical services+- [ ] `RollingUpdate` strategy with `maxUnavailable: 0`+- [ ] HPA configured for variable-load services++### Observability+- [ ] App exposes `/health` (liveness) and `/ready` (readiness) endpoints+- [ ] Structured JSON logging (no PII in logs)+- [ ] Resource labels: `app`, `version`, `environment`++---++## Related Skills++- `docker-patterns` — Multi-stage Dockerfiles and image security+- `deployment-patterns` — CI/CD pipelines, rollback strategy, health check endpoints+- `security-review` — Broader security hardening context+- `git-workflow` — GitOps integration with K8s (ArgoCD / Flux patterns)