Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
Implemented extensive improvements to the Firecrawl Helm chart: (#2938)
Jakob Stadlhuber2ebcec128b7b
diff --git a/examples/kubernetes/firecrawl-helm/Chart.yaml b/examples/kubernetes/firecrawl-helm/Chart.yamlindex dc97ecad8..98ec33428 100644--- a/examples/kubernetes/firecrawl-helm/Chart.yaml+++ b/examples/kubernetes/firecrawl-helm/Chart.yaml@@ -2,4 +2,4 @@ apiVersion: v2name: firecrawldescription: A Helm chart for deploying the Firecrawl applicationtype: application-version: 0.1.0+version: 0.2.0diff --git a/examples/kubernetes/firecrawl-helm/README.md b/examples/kubernetes/firecrawl-helm/README.mdindex f8744818d..5cdf1f34b 100644--- a/examples/kubernetes/firecrawl-helm/README.md+++ b/examples/kubernetes/firecrawl-helm/README.md@@ -1,51 +1,120 @@# Firecrawl Helm Chart-This Helm chart deploys Firecrawl components (API, Worker, Playwright service, Redis, etc.) on a Kubernetes cluster using environment-specific overlays.-## Pre-deployment+This chart deploys Firecrawl on Kubernetes with:+- `api`+- `worker` (queue-worker)+- `extract-worker`+- `nuq-worker`+- `nuq-prefetch-worker`+- `playwright-service`+- `redis`+- `nuq-postgres`+- `rabbitmq`-1. **Configure Secrets and ConfigMaps:**- - Update `values.yaml` with the necessary environment-specific values in the `overlays/<dev|prod>/values.yaml` file.- - **Note:** If using `REDIS_PASSWORD`, adjust the `REDIS_URL` and `REDIS_RATE_LIMIT_URL` in the ConfigMap to include the password.+## Image Strategy-2. **Build and Push Docker Images:**- - API/Worker:- ```bash- docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl:latest ../../../apps/api- docker push ghcr.io/winkk-dev/firecrawl:latest- ```- - Playwright Service:- ```bash- docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl-playwright:latest ../../../apps/playwright-service- docker push ghcr.io/winkk-dev/firecrawl-playwright:latest- ```+- **x86-only cluster**: use official Firecrawl images from GHCR (`ghcr.io/firecrawl/...`).+- **ARM or mixed ARM+x86 cluster**: use your multi-arch `winkkgmbh` images.-## Deployment+Official Firecrawl images are fine for x86. Use winkk images only when ARM support is needed.++## Configure Values++Use `values.yaml` plus one overlay.++Important fields:+- `secret.*` for API keys and sensitive values.+- `config.extra` / `secret.extra` for custom env vars.+- `image.dockerSecretEnabled` and `imagePullSecrets` for private registries.+- `resources.enabled` enables/disables all container resource requests/limits.+ Default: `false`.+- `rabbitmq.enabled`, `extractWorker.enabled`, `nuqPrefetchWorker.enabled` to toggle components.++## Deploy++Render:-Render the manifests for review:```bash-helm template winkk-ai . -f values.yaml -f overlays/dev/values.yaml -n winkk-ai+HELM_NO_PLUGINS=1 helm template firecrawl . \+ -f values.yaml \+ -f overlays/prod/values.yaml \+ -n firecrawl```-Deploy or upgrade the release:+Install/upgrade:+```bash-helm upgrade firecrawl . -f values.yaml -f overlays/dev/values.yaml -n firecrawl --install --create-namespace+HELM_NO_PLUGINS=1 helm upgrade firecrawl . \+ -f values.yaml \+ -f overlays/prod/values.yaml \+ -n firecrawl \+ --install \+ --create-namespace```-## Testing+### Use Official Firecrawl Images (x86-only)++If your cluster is x86-only and you want official images, override repositories:-Forward the API service port to your local machine:```bash-kubectl port-forward svc/firecrawl-api 3002:3002 -n firecrawl+HELM_NO_PLUGINS=1 helm upgrade firecrawl . \+ -f values.yaml \+ -f overlays/prod/values.yaml \+ --set image.repository=ghcr.io/firecrawl/firecrawl \+ --set playwright.repository=ghcr.io/firecrawl/playwright-service \+ --set nuqPostgres.image.repository=ghcr.io/firecrawl/nuq-postgres \+ -n firecrawl \+ --install \+ --create-namespace```-## Cleanup+## Build and Push Multi-Arch Containers (ARM+x86)++Run from `examples/kubernetes/firecrawl-helm`:-To uninstall the deployment:```bash-helm uninstall firecrawl -n firecrawl+docker buildx create --name multiarch --use --bootstrap+```++```bash+docker buildx build --platform linux/amd64,linux/arm64 --push \+ -t docker.io/winkkgmbh/firecrawl:latest \+ ../../../apps/api++docker buildx build --platform linux/amd64,linux/arm64 --push \+ -t docker.io/winkkgmbh/firecrawl-playwright:latest \+ ../../../apps/playwright-service-ts++docker buildx build --platform linux/amd64,linux/arm64 --push \+ -t docker.io/winkkgmbh/nuq-postgres:latest \+ ../../../apps/nuq-postgres```++## Package and Push Helm Chart (OCI)++```bash+HELM_NO_PLUGINS=1 helm package . --destination /tmp/helm-packages+HELM_NO_PLUGINS=1 helm push /tmp/helm-packages/firecrawl-0.2.0.tgz oci://registry-1.docker.io/winkkgmbh+```++Install from OCI:++```bash+HELM_NO_PLUGINS=1 helm upgrade --install firecrawl oci://registry-1.docker.io/winkkgmbh/firecrawl \+ --version 0.2.0 \+ -n firecrawl --create-namespace \+ -f values.yaml \+ -f overlays/prod/values.yaml```----+## Test-This README provides a quick guide to configuring, deploying, testing, and cleaning up your Firecrawl installation using Helm.\ No newline at end of file+```bash+kubectl port-forward svc/firecrawl-firecrawl-api 3002:3002 -n firecrawl+```++## Cleanup++```bash+helm uninstall firecrawl -n firecrawl+```diff --git a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/configmap.yamlindex 7d69a8691..e36e1fc61 100644--- a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml@@ -1,14 +1,50 @@+{{- $fullname := include "firecrawl.fullname" . -}}+{{- $redisDefault := printf "redis://%s-redis:%v" $fullname .Values.service.redis.port -}}+{{- $playwrightDefault := printf "http://%s-playwright:%v/scrape" $fullname .Values.service.playwright.port -}}+{{- $pgUser := default "postgres" .Values.nuqPostgres.auth.username -}}+{{- $pgPassword := default "password" .Values.nuqPostgres.auth.password -}}+{{- $pgDb := default "postgres" .Values.nuqPostgres.auth.database -}}+{{- $nuqDbDefault := printf "postgresql://%s:%s@%s-nuq-postgres:5432/%s" $pgUser $pgPassword $fullname $pgDb -}}+{{- $nuqDbUrl := default $nuqDbDefault .Values.config.NUQ_DATABASE_URL -}}+{{- $rabbitMqDefault := printf "amqp://%s-rabbitmq:%v" $fullname .Values.service.rabbitmq.port -}}+{{- $apiHostDefault := printf "%s-api" $fullname -}}+{{- $apiPortDefault := printf "%v" .Values.service.api.port -}}apiVersion: v1kind: ConfigMapmetadata:name: {{ include "firecrawl.fullname" . }}-configdata:NUM_WORKERS_PER_QUEUE: {{ .Values.config.NUM_WORKERS_PER_QUEUE | quote }}- PORT: {{ .Values.config.PORT | quote }}+ PORT: {{ default (printf "%v" .Values.service.api.port) .Values.config.PORT | quote }}HOST: {{ .Values.config.HOST | quote }}- REDIS_URL: {{ .Values.config.REDIS_URL | quote }}- REDIS_RATE_LIMIT_URL: {{ .Values.config.REDIS_RATE_LIMIT_URL | quote }}- PLAYWRIGHT_MICROSERVICE_URL: {{ .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}+ WORKER_PORT: {{ default (printf "%v" .Values.worker.port) .Values.config.WORKER_PORT | quote }}+ EXTRACT_WORKER_PORT: {{ default (printf "%v" .Values.extractWorker.port) .Values.config.EXTRACT_WORKER_PORT | quote }}+ NUQ_WORKER_PORT: {{ default (printf "%v" .Values.nuqWorker.port) .Values.config.NUQ_WORKER_PORT | quote }}+ NUQ_PREFETCH_WORKER_PORT: {{ default (printf "%v" .Values.nuqPrefetchWorker.port) .Values.config.NUQ_PREFETCH_WORKER_PORT | quote }}+ NUQ_WORKER_COUNT: {{ default (printf "%v" .Values.nuqWorker.replicaCount) .Values.config.NUQ_WORKER_COUNT | quote }}+ REDIS_URL: {{ default $redisDefault .Values.config.REDIS_URL | quote }}+ REDIS_RATE_LIMIT_URL: {{ default $redisDefault .Values.config.REDIS_RATE_LIMIT_URL | quote }}+ PLAYWRIGHT_MICROSERVICE_URL: {{ default $playwrightDefault .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}+ NUQ_DATABASE_URL: {{ $nuqDbUrl | quote }}+ NUQ_DATABASE_URL_LISTEN: {{ default $nuqDbUrl .Values.config.NUQ_DATABASE_URL_LISTEN | quote }}+ NUQ_RABBITMQ_URL: {{ default $rabbitMqDefault .Values.config.NUQ_RABBITMQ_URL | quote }}USE_DB_AUTHENTICATION: {{ .Values.config.USE_DB_AUTHENTICATION | quote }}- HDX_NODE_BETA_MODE: {{ .Values.config.HDX_NODE_BETA_MODE | quote }}- NUQ_DATABASE_URL: {{ .Values.config.NUQ_DATABASE_URL | quote }}+ IS_KUBERNETES: {{ .Values.config.IS_KUBERNETES | quote }}+ ENV: {{ .Values.config.ENV | quote }}+ LOGGING_LEVEL: {{ .Values.config.LOGGING_LEVEL | quote }}+ FIRECRAWL_APP_SCHEME: {{ .Values.config.FIRECRAWL_APP_SCHEME | quote }}+ FIRECRAWL_APP_HOST: {{ default $apiHostDefault .Values.config.FIRECRAWL_APP_HOST | quote }}+ FIRECRAWL_APP_PORT: {{ default $apiPortDefault .Values.config.FIRECRAWL_APP_PORT | quote }}+ OPENAI_BASE_URL: {{ .Values.config.OPENAI_BASE_URL | quote }}+ MODEL_NAME: {{ .Values.config.MODEL_NAME | quote }}+ MODEL_EMBEDDING_NAME: {{ .Values.config.MODEL_EMBEDDING_NAME | quote }}+ OLLAMA_BASE_URL: {{ .Values.config.OLLAMA_BASE_URL | quote }}+ PROXY_SERVER: {{ .Values.config.PROXY_SERVER | quote }}+ PROXY_USERNAME: {{ .Values.config.PROXY_USERNAME | quote }}+ SEARXNG_ENDPOINT: {{ .Values.config.SEARXNG_ENDPOINT | quote }}+ SEARXNG_ENGINES: {{ .Values.config.SEARXNG_ENGINES | quote }}+ SEARXNG_CATEGORIES: {{ .Values.config.SEARXNG_CATEGORIES | quote }}+ SELF_HOSTED_WEBHOOK_URL: {{ .Values.config.SELF_HOSTED_WEBHOOK_URL | quote }}+ {{- range $k, $v := .Values.config.extra }}+ {{ $k }}: {{ $v | quote }}+ {{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/deployment.yamlindex ba0be85fe..602fdf057 100644--- a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml@@ -5,7 +5,7 @@ metadata:labels:app: {{ include "firecrawl.name" . }}-apispec:- replicas: {{ .Values.replicaCount }}+ replicas: {{ .Values.api.replicaCount | default .Values.replicaCount }}selector:matchLabels:app: {{ include "firecrawl.name" . }}-api@@ -18,21 +18,31 @@ spec:imagePullSecrets:{{- toYaml .Values.imagePullSecrets | nindent 8 }}{{- end }}+ terminationGracePeriodSeconds: 180containers:- name: apiimage: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"imagePullPolicy: {{ .Values.image.pullPolicy }}- args: [ "pnpm", "run", "start" ]+ command: [ "node" ]+ args: [ "--max-old-space-size=6144", "dist/src/index.js" ]ports:- containerPort: {{ .Values.service.api.port }}env:- name: FLY_PROCESS_GROUPvalue: "app"+ - name: NUQ_POD_NAME+ valueFrom:+ fieldRef:+ fieldPath: metadata.nameenvFrom:- configMapRef:name: {{ include "firecrawl.fullname" . }}-config- secretRef:name: {{ include "firecrawl.fullname" . }}-secret+ {{- if and .Values.resources.enabled .Values.api.resources }}+ resources:+ {{- toYaml .Values.api.resources | nindent 12 }}+ {{- end }}livenessProbe:httpGet:path: /v0/health/livenessdiff --git a/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yamlnew file mode 100644index 000000000..8457a0a0d--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yaml@@ -0,0 +1,67 @@+{{- if .Values.extractWorker.enabled }}+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-extract-worker+ labels:+ app: {{ include "firecrawl.name" . }}-extract-worker+spec:+ replicas: {{ .Values.extractWorker.replicaCount | default 1 }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-extract-worker+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-extract-worker+ spec:+ {{- if .Values.image.dockerSecretEnabled }}+ imagePullSecrets:+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}+ {{- end }}+ terminationGracePeriodSeconds: 60+ containers:+ - name: extract-worker+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"+ imagePullPolicy: {{ .Values.image.pullPolicy }}+ command: [ "node" ]+ args: [ "--max-old-space-size=3072", "dist/src/services/extract-worker.js" ]+ ports:+ - containerPort: {{ .Values.extractWorker.port | int }}+ env:+ - name: FLY_PROCESS_GROUP+ value: "extract-worker"+ - name: EXTRACT_WORKER_PORT+ value: {{ .Values.extractWorker.port | quote }}+ - name: NUQ_POD_NAME+ valueFrom:+ fieldRef:+ fieldPath: metadata.name+ envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-config+ - secretRef:+ name: {{ include "firecrawl.fullname" . }}-secret+ {{- if and .Values.resources.enabled .Values.extractWorker.resources }}+ resources:+ {{- toYaml .Values.extractWorker.resources | nindent 12 }}+ {{- end }}+ livenessProbe:+ httpGet:+ path: /liveness+ port: {{ .Values.extractWorker.port | int }}+ initialDelaySeconds: 10+ periodSeconds: 10+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+ readinessProbe:+ httpGet:+ path: /health+ port: {{ .Values.extractWorker.port | int }}+ initialDelaySeconds: 10+ periodSeconds: 10+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+{{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yamlindex 4a5a01dcc..df41f8ca8 100644--- a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml@@ -26,7 +26,7 @@ spec:- name: POSTGRES_USERvalue: "{{ .Values.nuqPostgres.auth.username | default "postgres" }}"- name: POSTGRES_PASSWORD- value: "password"+ value: "{{ .Values.nuqPostgres.auth.password | default "password" }}"- name: POSTGRES_DBvalue: "{{ .Values.nuqPostgres.auth.database | default "postgres" }}"ports:@@ -34,17 +34,19 @@ spec:volumeMounts:- name: postgres-storagemountPath: /var/lib/postgresql/data- {{- if .Values.nuqPostgres.resources }}+ {{- if .Values.resources.enabled }}+ {{- if .Values.nuqPostgres.resources }}resources:- {{- toYaml .Values.nuqPostgres.resources | nindent 12 }}- {{- else }}+ {{- toYaml .Values.nuqPostgres.resources | nindent 12 }}+ {{- else }}resources:- requests:- memory: "512Mi"- cpu: "250m"- limits:- memory: "1Gi"- cpu: "500m"+ requests:+ memory: "512Mi"+ cpu: "250m"+ limits:+ memory: "1Gi"+ cpu: "500m"+ {{- end }}{{- end }}volumes:- name: postgres-storagediff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yamlnew file mode 100644index 000000000..59c49341d--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yaml@@ -0,0 +1,17 @@+{{- if .Values.nuqPostgres.persistence.enabled }}+apiVersion: v1+kind: PersistentVolumeClaim+metadata:+ name: {{ include "firecrawl.fullname" . }}-nuq-postgres-pvc+ labels:+ app: {{ include "firecrawl.name" . }}-nuq-postgres+spec:+ accessModes:+ - ReadWriteOnce+ resources:+ requests:+ storage: {{ .Values.nuqPostgres.persistence.size | quote }}+ {{- if .Values.nuqPostgres.persistence.storageClass }}+ storageClassName: {{ .Values.nuqPostgres.persistence.storageClass | quote }}+ {{- end }}+{{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yamlnew file mode 100644index 000000000..2247e346a--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yaml@@ -0,0 +1,69 @@+{{- if .Values.nuqPrefetchWorker.enabled }}+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-nuq-prefetch-worker+ labels:+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker+spec:+ replicas: {{ .Values.nuqPrefetchWorker.replicaCount | default 1 }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker+ spec:+ {{- if .Values.image.dockerSecretEnabled }}+ imagePullSecrets:+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}+ {{- end }}+ terminationGracePeriodSeconds: 60+ containers:+ - name: nuq-prefetch-worker+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"+ imagePullPolicy: {{ .Values.image.pullPolicy }}+ command: [ "node" ]+ args: [ "--max-old-space-size=2048", "dist/src/services/worker/nuq-prefetch-worker.js" ]+ ports:+ - containerPort: {{ .Values.nuqPrefetchWorker.port | int }}+ env:+ - name: FLY_PROCESS_GROUP+ value: "nuq-prefetch-worker"+ - name: NUQ_PREFETCH_WORKER_PORT+ value: {{ .Values.nuqPrefetchWorker.port | quote }}+ - name: NUQ_PREFETCH_REPLICAS+ value: {{ .Values.nuqPrefetchWorker.replicaCount | quote }}+ - name: NUQ_POD_NAME+ valueFrom:+ fieldRef:+ fieldPath: metadata.name+ envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-config+ - secretRef:+ name: {{ include "firecrawl.fullname" . }}-secret+ {{- if and .Values.resources.enabled .Values.nuqPrefetchWorker.resources }}+ resources:+ {{- toYaml .Values.nuqPrefetchWorker.resources | nindent 12 }}+ {{- end }}+ livenessProbe:+ httpGet:+ path: /health+ port: {{ .Values.nuqPrefetchWorker.port | int }}+ initialDelaySeconds: 10+ periodSeconds: 10+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+ readinessProbe:+ httpGet:+ path: /health+ port: {{ .Values.nuqPrefetchWorker.port | int }}+ initialDelaySeconds: 10+ periodSeconds: 10+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+{{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yamlindex d36d83403..d01967714 100644--- a/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml@@ -25,32 +25,40 @@ spec:imagePullPolicy: {{ .Values.image.pullPolicy }}command: [ "node" ]args: [ "--max-old-space-size=3072", "dist/src/services/worker/nuq-worker.js" ]+ ports:+ - containerPort: {{ .Values.nuqWorker.port | int }}env:- name: FLY_PROCESS_GROUPvalue: "nuq-worker"- name: NUQ_WORKER_PORT- value: "3006"+ value: {{ .Values.nuqWorker.port | quote }}+ - name: NUQ_POD_NAME+ valueFrom:+ fieldRef:+ fieldPath: metadata.nameenvFrom:- configMapRef:name: {{ include "firecrawl.fullname" . }}-config- secretRef:name: {{ include "firecrawl.fullname" . }}-secret- {{- if .Values.nuqWorker.resources }}+ {{- if .Values.resources.enabled }}+ {{- if .Values.nuqWorker.resources }}resources:- {{- toYaml .Values.nuqWorker.resources | nindent 12 }}- {{- else }}+ {{- toYaml .Values.nuqWorker.resources | nindent 12 }}+ {{- else }}resources:- requests:- memory: "3G"- cpu: "1000m"- limits:- memory: "4G"- cpu: "1000m"+ requests:+ memory: "3G"+ cpu: "1000m"+ limits:+ memory: "4G"+ cpu: "1000m"+ {{- end }}{{- end }}livenessProbe:httpGet:path: /health- port: 3006+ port: {{ .Values.nuqWorker.port | int }}initialDelaySeconds: 5periodSeconds: 5timeoutSeconds: 5@@ -59,7 +67,7 @@ spec:readinessProbe:httpGet:path: /health- port: 3006+ port: {{ .Values.nuqWorker.port | int }}initialDelaySeconds: 5periodSeconds: 5timeoutSeconds: 5diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yamlindex bff1866c9..f3be19085 100644--- a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml@@ -3,4 +3,6 @@ kind: ConfigMapmetadata:name: {{ include "firecrawl.fullname" . }}-playwright-configdata:- PORT: {{ .Values.playwrightConfig.PORT | quote }}+ PORT: {{ default (printf "%v" .Values.service.playwright.port) .Values.playwrightConfig.PORT | quote }}+ BLOCK_MEDIA: {{ .Values.playwrightConfig.BLOCK_MEDIA | quote }}+ MAX_CONCURRENT_PAGES: {{ .Values.playwrightConfig.MAX_CONCURRENT_PAGES | quote }}diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yamlindex 8f7c1243a..bfb1d8335 100644--- a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml@@ -5,7 +5,7 @@ metadata:labels:app: {{ include "firecrawl.name" . }}-playwrightspec:- replicas: {{ .Values.replicaCount }}+ replicas: {{ .Values.playwright.replicaCount | default .Values.replicaCount }}selector:matchLabels:app: {{ include "firecrawl.name" . }}-playwright@@ -25,11 +25,15 @@ spec:ports:- containerPort: {{ .Values.service.playwright.port }}envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-config+ - secretRef:+ name: {{ include "firecrawl.fullname" . }}-secret- configMapRef:name: {{ include "firecrawl.fullname" . }}-playwright-configlivenessProbe:httpGet:- path: /health/liveness+ path: /healthport: {{ .Values.service.playwright.port }}initialDelaySeconds: 30periodSeconds: 30@@ -38,7 +42,7 @@ spec:failureThreshold: 3readinessProbe:httpGet:- path: /health/readiness+ path: /healthport: {{ .Values.service.playwright.port }}initialDelaySeconds: 30periodSeconds: 30diff --git a/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yamlnew file mode 100644index 000000000..ddd1d9b65--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yaml@@ -0,0 +1,60 @@+{{- if .Values.rabbitmq.enabled }}+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-rabbitmq+ labels:+ app: {{ include "firecrawl.name" . }}-rabbitmq+spec:+ replicas: {{ .Values.rabbitmq.replicaCount | default 1 }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-rabbitmq+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-rabbitmq+ spec:+ containers:+ - name: rabbitmq+ image: {{ .Values.rabbitmq.image }}+ ports:+ - containerPort: {{ .Values.service.rabbitmq.port }}+ - containerPort: {{ .Values.service.rabbitmq.managementPort }}+ {{- if and .Values.resources.enabled .Values.rabbitmq.resources }}+ resources:+ {{- toYaml .Values.rabbitmq.resources | nindent 12 }}+ {{- end }}+ livenessProbe:+ tcpSocket:+ port: {{ .Values.service.rabbitmq.port }}+ initialDelaySeconds: 20+ periodSeconds: 10+ timeoutSeconds: 5+ failureThreshold: 6+ readinessProbe:+ tcpSocket:+ port: {{ .Values.service.rabbitmq.port }}+ initialDelaySeconds: 10+ periodSeconds: 5+ timeoutSeconds: 3+ failureThreshold: 6+---+apiVersion: v1+kind: Service+metadata:+ name: {{ include "firecrawl.fullname" . }}-rabbitmq+spec:+ type: {{ .Values.service.rabbitmq.type }}+ selector:+ app: {{ include "firecrawl.name" . }}-rabbitmq+ ports:+ - name: amqp+ protocol: TCP+ port: {{ .Values.service.rabbitmq.port }}+ targetPort: {{ .Values.service.rabbitmq.port }}+ - name: management+ protocol: TCP+ port: {{ .Values.service.rabbitmq.managementPort }}+ targetPort: {{ .Values.service.rabbitmq.managementPort }}+{{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/secret.yaml b/examples/kubernetes/firecrawl-helm/templates/secret.yamlindex 3840b7c4e..585e3f512 100644--- a/examples/kubernetes/firecrawl-helm/templates/secret.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/secret.yaml@@ -9,8 +9,13 @@ data:LLAMAPARSE_API_KEY: {{ .Values.secret.LLAMAPARSE_API_KEY | b64enc | quote }}BULL_AUTH_KEY: {{ .Values.secret.BULL_AUTH_KEY | b64enc | quote }}TEST_API_KEY: {{ .Values.secret.TEST_API_KEY | b64enc | quote }}- SCRAPING_BEE_API_KEY: {{ .Values.secret.SCRAPING_BEE_API_KEY | b64enc | quote }}- STRIPE_PRICE_ID_STANDARD: {{ .Values.secret.STRIPE_PRICE_ID_STANDARD | b64enc | quote }}- STRIPE_PRICE_ID_SCALE: {{ .Values.secret.STRIPE_PRICE_ID_SCALE | b64enc | quote }}FIRE_ENGINE_BETA_URL: {{ .Values.secret.FIRE_ENGINE_BETA_URL | b64enc | quote }}+ SUPABASE_ANON_TOKEN: {{ .Values.secret.SUPABASE_ANON_TOKEN | b64enc | quote }}+ SUPABASE_URL: {{ .Values.secret.SUPABASE_URL | b64enc | quote }}+ SUPABASE_SERVICE_TOKEN: {{ .Values.secret.SUPABASE_SERVICE_TOKEN | b64enc | quote }}+ PROXY_PASSWORD: {{ .Values.secret.PROXY_PASSWORD | b64enc | quote }}+ SELF_HOSTED_WEBHOOK_HMAC_SECRET: {{ .Values.secret.SELF_HOSTED_WEBHOOK_HMAC_SECRET | b64enc | quote }}REDIS_PASSWORD: {{ .Values.secret.REDIS_PASSWORD | b64enc | quote }}+ {{- range $k, $v := .Values.secret.extra }}+ {{ $k }}: {{ $v | b64enc | quote }}+ {{- end }}diff --git a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yamlindex 787dcfe4a..64aaa5722 100644--- a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml+++ b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml@@ -5,7 +5,7 @@ metadata:labels:app: {{ include "firecrawl.name" . }}-workerspec:- replicas: {{ .Values.replicaCount }}+ replicas: {{ .Values.worker.replicaCount | default .Values.replicaCount }}selector:matchLabels:app: {{ include "firecrawl.name" . }}-worker@@ -18,16 +18,39 @@ spec:imagePullSecrets:{{- toYaml .Values.imagePullSecrets | nindent 8 }}{{- end }}+ terminationGracePeriodSeconds: 60containers:- name: workerimage: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"imagePullPolicy: {{ .Values.image.pullPolicy }}- args: [ "pnpm", "run", "workers" ]+ command: [ "node" ]+ args: [ "--max-old-space-size=3072", "dist/src/services/queue-worker.js" ]+ ports:+ - containerPort: {{ .Values.worker.port | int }}env:- name: FLY_PROCESS_GROUPvalue: "worker"+ - name: WORKER_PORT+ value: {{ .Values.worker.port | quote }}+ - name: NUQ_POD_NAME+ valueFrom:+ fieldRef:+ fieldPath: metadata.nameenvFrom:- configMapRef:name: {{ include "firecrawl.fullname" . }}-config- secretRef:name: {{ include "firecrawl.fullname" . }}-secret+ {{- if and .Values.resources.enabled .Values.worker.resources }}+ resources:+ {{- toYaml .Values.worker.resources | nindent 12 }}+ {{- end }}+ livenessProbe:+ httpGet:+ path: /liveness+ port: {{ .Values.worker.port | int }}+ initialDelaySeconds: 10+ periodSeconds: 10+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3diff --git a/examples/kubernetes/firecrawl-helm/values.yaml b/examples/kubernetes/firecrawl-helm/values.yamlindex 853462b3d..afa9868fb 100644--- a/examples/kubernetes/firecrawl-helm/values.yaml+++ b/examples/kubernetes/firecrawl-helm/values.yaml@@ -1,7 +1,44 @@replicaCount: 1+resources:+ enabled: false++api:+ replicaCount: 1+ resources:+ requests:+ memory: "4G"+ cpu: "2000m"+ limits:+ memory: "6G"+ cpu: "2000m"++worker:+ replicaCount: 1+ port: 3005+ resources:+ requests:+ memory: "3G"+ cpu: "1000m"+ limits:+ memory: "4G"+ cpu: "1000m"++extractWorker:+ enabled: true+ replicaCount: 1+ port: 3004+ resources:+ requests:+ memory: "2G"+ cpu: "1000m"+ limits:+ memory: "3G"+ cpu: "1000m"+nuqWorker:replicaCount: 5+ port: 3006resources:requests:memory: "3G"@@ -10,14 +47,27 @@ nuqWorker:memory: "4G"cpu: "1000m"+nuqPrefetchWorker:+ enabled: true+ replicaCount: 1+ port: 3011+ resources:+ requests:+ memory: "1G"+ cpu: "500m"+ limits:+ memory: "2G"+ cpu: "1000m"+nuqPostgres:replicaCount: 1image:- repository: ghcr.io/firecrawl/nuq-postgres+ repository: docker.io/winkkgmbh/nuq-postgrestag: latestpullPolicy: Alwaysauth:username: postgres+ password: passworddatabase: postgresresources:requests:@@ -28,17 +78,32 @@ nuqPostgres:cpu: "500m"persistence:enabled: false+ size: 10Gi+ storageClass: ""++rabbitmq:+ enabled: true+ image: rabbitmq:3-management+ replicaCount: 1+ resources:+ requests:+ memory: "512Mi"+ cpu: "250m"+ limits:+ memory: "1Gi"+ cpu: "500m"image:- repository: ghcr.io/firecrawl/firecrawl+ repository: docker.io/winkkgmbh/firecrawltag: "latest"pullPolicy: Always- dockerSecretEnabled: true+ dockerSecretEnabled: falseplaywright:- repository: ghcr.io/firecrawl/playwright-service+ repository: docker.io/winkkgmbh/firecrawl-playwrighttag: "latest"pullPolicy: Always+ replicaCount: 1redis:image: redis:alpine@@ -54,20 +119,49 @@ service:redis:type: ClusterIPport: 6379+ rabbitmq:+ type: ClusterIP+ port: 5672+ managementPort: 15672config:NUM_WORKERS_PER_QUEUE: "8"PORT: "3002"HOST: "0.0.0.0"- REDIS_URL: "redis://redis:6379"- REDIS_RATE_LIMIT_URL: "redis://redis:6379"- PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000"+ WORKER_PORT: "3005"+ EXTRACT_WORKER_PORT: "3004"+ NUQ_WORKER_PORT: "3006"+ NUQ_PREFETCH_WORKER_PORT: "3011"+ NUQ_WORKER_COUNT: "5"USE_DB_AUTHENTICATION: "false"- HDX_NODE_BETA_MODE: "1"- NUQ_DATABASE_URL: "postgresql://postgres:password@nuq-postgres:5432/postgres"+ IS_KUBERNETES: "true"+ ENV: "production"+ LOGGING_LEVEL: "INFO"+ FIRECRAWL_APP_SCHEME: "http"+ FIRECRAWL_APP_HOST: ""+ FIRECRAWL_APP_PORT: ""+ REDIS_URL: ""+ REDIS_RATE_LIMIT_URL: ""+ PLAYWRIGHT_MICROSERVICE_URL: ""+ NUQ_DATABASE_URL: ""+ NUQ_DATABASE_URL_LISTEN: ""+ NUQ_RABBITMQ_URL: ""+ OPENAI_BASE_URL: ""+ MODEL_NAME: ""+ MODEL_EMBEDDING_NAME: ""+ OLLAMA_BASE_URL: ""+ PROXY_SERVER: ""+ PROXY_USERNAME: ""+ SEARXNG_ENDPOINT: ""+ SEARXNG_ENGINES: ""+ SEARXNG_CATEGORIES: ""+ SELF_HOSTED_WEBHOOK_URL: ""+ extra: {}playwrightConfig:PORT: "3000"+ BLOCK_MEDIA: ""+ MAX_CONCURRENT_PAGES: "10"secret:OPENAI_API_KEY: ""@@ -75,11 +169,14 @@ secret:LLAMAPARSE_API_KEY: ""BULL_AUTH_KEY: ""TEST_API_KEY: ""- SCRAPING_BEE_API_KEY: ""- STRIPE_PRICE_ID_STANDARD: ""- STRIPE_PRICE_ID_SCALE: ""FIRE_ENGINE_BETA_URL: ""+ SUPABASE_ANON_TOKEN: ""+ SUPABASE_URL: ""+ SUPABASE_SERVICE_TOKEN: ""+ PROXY_PASSWORD: ""+ SELF_HOSTED_WEBHOOK_HMAC_SECRET: ""REDIS_PASSWORD: ""+ extra: {}imagePullSecrets:- name: docker-registry-secret