Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

Implemented extensive improvements to the Firecrawl Helm chart: (#2938)

Jakob Stadlhuber2ebcec128b7b
diff --git a/examples/kubernetes/firecrawl-helm/Chart.yaml b/examples/kubernetes/firecrawl-helm/Chart.yaml
index dc97ecad8..98ec33428 100644
--- a/examples/kubernetes/firecrawl-helm/Chart.yaml
+++ b/examples/kubernetes/firecrawl-helm/Chart.yaml
@@ -2,4 +2,4 @@ apiVersion: v2
name: firecrawl
description: A Helm chart for deploying the Firecrawl application
type: application
-version: 0.1.0
+version: 0.2.0
diff --git a/examples/kubernetes/firecrawl-helm/README.md b/examples/kubernetes/firecrawl-helm/README.md
index f8744818d..5cdf1f34b 100644
--- a/examples/kubernetes/firecrawl-helm/README.md
+++ b/examples/kubernetes/firecrawl-helm/README.md
@@ -1,51 +1,120 @@
# Firecrawl Helm Chart
-This Helm chart deploys Firecrawl components (API, Worker, Playwright service, Redis, etc.) on a Kubernetes cluster using environment-specific overlays.
-## Pre-deployment
+This chart deploys Firecrawl on Kubernetes with:
+- `api`
+- `worker` (queue-worker)
+- `extract-worker`
+- `nuq-worker`
+- `nuq-prefetch-worker`
+- `playwright-service`
+- `redis`
+- `nuq-postgres`
+- `rabbitmq`
-1. **Configure Secrets and ConfigMaps:**
- - Update `values.yaml` with the necessary environment-specific values in the `overlays/<dev|prod>/values.yaml` file.
- - **Note:** If using `REDIS_PASSWORD`, adjust the `REDIS_URL` and `REDIS_RATE_LIMIT_URL` in the ConfigMap to include the password.
+## Image Strategy
-2. **Build and Push Docker Images:**
- - API/Worker:
- ```bash
- docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl:latest ../../../apps/api
- docker push ghcr.io/winkk-dev/firecrawl:latest
- ```
- - Playwright Service:
- ```bash
- docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl-playwright:latest ../../../apps/playwright-service
- docker push ghcr.io/winkk-dev/firecrawl-playwright:latest
- ```
+- **x86-only cluster**: use official Firecrawl images from GHCR (`ghcr.io/firecrawl/...`).
+- **ARM or mixed ARM+x86 cluster**: use your multi-arch `winkkgmbh` images.
-## Deployment
+Official Firecrawl images are fine for x86. Use winkk images only when ARM support is needed.
+
+## Configure Values
+
+Use `values.yaml` plus one overlay.
+
+Important fields:
+- `secret.*` for API keys and sensitive values.
+- `config.extra` / `secret.extra` for custom env vars.
+- `image.dockerSecretEnabled` and `imagePullSecrets` for private registries.
+- `resources.enabled` enables/disables all container resource requests/limits.
+ Default: `false`.
+- `rabbitmq.enabled`, `extractWorker.enabled`, `nuqPrefetchWorker.enabled` to toggle components.
+
+## Deploy
+
+Render:
-Render the manifests for review:
```bash
-helm template winkk-ai . -f values.yaml -f overlays/dev/values.yaml -n winkk-ai
+HELM_NO_PLUGINS=1 helm template firecrawl . \
+ -f values.yaml \
+ -f overlays/prod/values.yaml \
+ -n firecrawl
```
-Deploy or upgrade the release:
+Install/upgrade:
+
```bash
-helm upgrade firecrawl . -f values.yaml -f overlays/dev/values.yaml -n firecrawl --install --create-namespace
+HELM_NO_PLUGINS=1 helm upgrade firecrawl . \
+ -f values.yaml \
+ -f overlays/prod/values.yaml \
+ -n firecrawl \
+ --install \
+ --create-namespace
```
-## Testing
+### Use Official Firecrawl Images (x86-only)
+
+If your cluster is x86-only and you want official images, override repositories:
-Forward the API service port to your local machine:
```bash
-kubectl port-forward svc/firecrawl-api 3002:3002 -n firecrawl
+HELM_NO_PLUGINS=1 helm upgrade firecrawl . \
+ -f values.yaml \
+ -f overlays/prod/values.yaml \
+ --set image.repository=ghcr.io/firecrawl/firecrawl \
+ --set playwright.repository=ghcr.io/firecrawl/playwright-service \
+ --set nuqPostgres.image.repository=ghcr.io/firecrawl/nuq-postgres \
+ -n firecrawl \
+ --install \
+ --create-namespace
```
-## Cleanup
+## Build and Push Multi-Arch Containers (ARM+x86)
+
+Run from `examples/kubernetes/firecrawl-helm`:
-To uninstall the deployment:
```bash
-helm uninstall firecrawl -n firecrawl
+docker buildx create --name multiarch --use --bootstrap
+```
+
+```bash
+docker buildx build --platform linux/amd64,linux/arm64 --push \
+ -t docker.io/winkkgmbh/firecrawl:latest \
+ ../../../apps/api
+
+docker buildx build --platform linux/amd64,linux/arm64 --push \
+ -t docker.io/winkkgmbh/firecrawl-playwright:latest \
+ ../../../apps/playwright-service-ts
+
+docker buildx build --platform linux/amd64,linux/arm64 --push \
+ -t docker.io/winkkgmbh/nuq-postgres:latest \
+ ../../../apps/nuq-postgres
```
+
+## Package and Push Helm Chart (OCI)
+
+```bash
+HELM_NO_PLUGINS=1 helm package . --destination /tmp/helm-packages
+HELM_NO_PLUGINS=1 helm push /tmp/helm-packages/firecrawl-0.2.0.tgz oci://registry-1.docker.io/winkkgmbh
+```
+
+Install from OCI:
+
+```bash
+HELM_NO_PLUGINS=1 helm upgrade --install firecrawl oci://registry-1.docker.io/winkkgmbh/firecrawl \
+ --version 0.2.0 \
+ -n firecrawl --create-namespace \
+ -f values.yaml \
+ -f overlays/prod/values.yaml
```
----
+## Test
-This README provides a quick guide to configuring, deploying, testing, and cleaning up your Firecrawl installation using Helm.
\ No newline at end of file
+```bash
+kubectl port-forward svc/firecrawl-firecrawl-api 3002:3002 -n firecrawl
+```
+
+## Cleanup
+
+```bash
+helm uninstall firecrawl -n firecrawl
+```
diff --git a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml
index 7d69a8691..e36e1fc61 100644
--- a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml
@@ -1,14 +1,50 @@
+{{- $fullname := include "firecrawl.fullname" . -}}
+{{- $redisDefault := printf "redis://%s-redis:%v" $fullname .Values.service.redis.port -}}
+{{- $playwrightDefault := printf "http://%s-playwright:%v/scrape" $fullname .Values.service.playwright.port -}}
+{{- $pgUser := default "postgres" .Values.nuqPostgres.auth.username -}}
+{{- $pgPassword := default "password" .Values.nuqPostgres.auth.password -}}
+{{- $pgDb := default "postgres" .Values.nuqPostgres.auth.database -}}
+{{- $nuqDbDefault := printf "postgresql://%s:%s@%s-nuq-postgres:5432/%s" $pgUser $pgPassword $fullname $pgDb -}}
+{{- $nuqDbUrl := default $nuqDbDefault .Values.config.NUQ_DATABASE_URL -}}
+{{- $rabbitMqDefault := printf "amqp://%s-rabbitmq:%v" $fullname .Values.service.rabbitmq.port -}}
+{{- $apiHostDefault := printf "%s-api" $fullname -}}
+{{- $apiPortDefault := printf "%v" .Values.service.api.port -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "firecrawl.fullname" . }}-config
data:
NUM_WORKERS_PER_QUEUE: {{ .Values.config.NUM_WORKERS_PER_QUEUE | quote }}
- PORT: {{ .Values.config.PORT | quote }}
+ PORT: {{ default (printf "%v" .Values.service.api.port) .Values.config.PORT | quote }}
HOST: {{ .Values.config.HOST | quote }}
- REDIS_URL: {{ .Values.config.REDIS_URL | quote }}
- REDIS_RATE_LIMIT_URL: {{ .Values.config.REDIS_RATE_LIMIT_URL | quote }}
- PLAYWRIGHT_MICROSERVICE_URL: {{ .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}
+ WORKER_PORT: {{ default (printf "%v" .Values.worker.port) .Values.config.WORKER_PORT | quote }}
+ EXTRACT_WORKER_PORT: {{ default (printf "%v" .Values.extractWorker.port) .Values.config.EXTRACT_WORKER_PORT | quote }}
+ NUQ_WORKER_PORT: {{ default (printf "%v" .Values.nuqWorker.port) .Values.config.NUQ_WORKER_PORT | quote }}
+ NUQ_PREFETCH_WORKER_PORT: {{ default (printf "%v" .Values.nuqPrefetchWorker.port) .Values.config.NUQ_PREFETCH_WORKER_PORT | quote }}
+ NUQ_WORKER_COUNT: {{ default (printf "%v" .Values.nuqWorker.replicaCount) .Values.config.NUQ_WORKER_COUNT | quote }}
+ REDIS_URL: {{ default $redisDefault .Values.config.REDIS_URL | quote }}
+ REDIS_RATE_LIMIT_URL: {{ default $redisDefault .Values.config.REDIS_RATE_LIMIT_URL | quote }}
+ PLAYWRIGHT_MICROSERVICE_URL: {{ default $playwrightDefault .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}
+ NUQ_DATABASE_URL: {{ $nuqDbUrl | quote }}
+ NUQ_DATABASE_URL_LISTEN: {{ default $nuqDbUrl .Values.config.NUQ_DATABASE_URL_LISTEN | quote }}
+ NUQ_RABBITMQ_URL: {{ default $rabbitMqDefault .Values.config.NUQ_RABBITMQ_URL | quote }}
USE_DB_AUTHENTICATION: {{ .Values.config.USE_DB_AUTHENTICATION | quote }}
- HDX_NODE_BETA_MODE: {{ .Values.config.HDX_NODE_BETA_MODE | quote }}
- NUQ_DATABASE_URL: {{ .Values.config.NUQ_DATABASE_URL | quote }}
+ IS_KUBERNETES: {{ .Values.config.IS_KUBERNETES | quote }}
+ ENV: {{ .Values.config.ENV | quote }}
+ LOGGING_LEVEL: {{ .Values.config.LOGGING_LEVEL | quote }}
+ FIRECRAWL_APP_SCHEME: {{ .Values.config.FIRECRAWL_APP_SCHEME | quote }}
+ FIRECRAWL_APP_HOST: {{ default $apiHostDefault .Values.config.FIRECRAWL_APP_HOST | quote }}
+ FIRECRAWL_APP_PORT: {{ default $apiPortDefault .Values.config.FIRECRAWL_APP_PORT | quote }}
+ OPENAI_BASE_URL: {{ .Values.config.OPENAI_BASE_URL | quote }}
+ MODEL_NAME: {{ .Values.config.MODEL_NAME | quote }}
+ MODEL_EMBEDDING_NAME: {{ .Values.config.MODEL_EMBEDDING_NAME | quote }}
+ OLLAMA_BASE_URL: {{ .Values.config.OLLAMA_BASE_URL | quote }}
+ PROXY_SERVER: {{ .Values.config.PROXY_SERVER | quote }}
+ PROXY_USERNAME: {{ .Values.config.PROXY_USERNAME | quote }}
+ SEARXNG_ENDPOINT: {{ .Values.config.SEARXNG_ENDPOINT | quote }}
+ SEARXNG_ENGINES: {{ .Values.config.SEARXNG_ENGINES | quote }}
+ SEARXNG_CATEGORIES: {{ .Values.config.SEARXNG_CATEGORIES | quote }}
+ SELF_HOSTED_WEBHOOK_URL: {{ .Values.config.SELF_HOSTED_WEBHOOK_URL | quote }}
+ {{- range $k, $v := .Values.config.extra }}
+ {{ $k }}: {{ $v | quote }}
+ {{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml
index ba0be85fe..602fdf057 100644
--- a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml
@@ -5,7 +5,7 @@ metadata:
labels:
app: {{ include "firecrawl.name" . }}-api
spec:
- replicas: {{ .Values.replicaCount }}
+ replicas: {{ .Values.api.replicaCount | default .Values.replicaCount }}
selector:
matchLabels:
app: {{ include "firecrawl.name" . }}-api
@@ -18,21 +18,31 @@ spec:
imagePullSecrets:
{{- toYaml .Values.imagePullSecrets | nindent 8 }}
{{- end }}
+ terminationGracePeriodSeconds: 180
containers:
- name: api
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
- args: [ "pnpm", "run", "start" ]
+ command: [ "node" ]
+ args: [ "--max-old-space-size=6144", "dist/src/index.js" ]
ports:
- containerPort: {{ .Values.service.api.port }}
env:
- name: FLY_PROCESS_GROUP
value: "app"
+ - name: NUQ_POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-config
- secretRef:
name: {{ include "firecrawl.fullname" . }}-secret
+ {{- if and .Values.resources.enabled .Values.api.resources }}
+ resources:
+ {{- toYaml .Values.api.resources | nindent 12 }}
+ {{- end }}
livenessProbe:
httpGet:
path: /v0/health/liveness
diff --git a/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yaml
new file mode 100644
index 000000000..8457a0a0d
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/extract-worker-deployment.yaml
@@ -0,0 +1,67 @@
+{{- if .Values.extractWorker.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-extract-worker
+ labels:
+ app: {{ include "firecrawl.name" . }}-extract-worker
+spec:
+ replicas: {{ .Values.extractWorker.replicaCount | default 1 }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-extract-worker
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-extract-worker
+ spec:
+ {{- if .Values.image.dockerSecretEnabled }}
+ imagePullSecrets:
+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}
+ {{- end }}
+ terminationGracePeriodSeconds: 60
+ containers:
+ - name: extract-worker
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ command: [ "node" ]
+ args: [ "--max-old-space-size=3072", "dist/src/services/extract-worker.js" ]
+ ports:
+ - containerPort: {{ .Values.extractWorker.port | int }}
+ env:
+ - name: FLY_PROCESS_GROUP
+ value: "extract-worker"
+ - name: EXTRACT_WORKER_PORT
+ value: {{ .Values.extractWorker.port | quote }}
+ - name: NUQ_POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-config
+ - secretRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
+ {{- if and .Values.resources.enabled .Values.extractWorker.resources }}
+ resources:
+ {{- toYaml .Values.extractWorker.resources | nindent 12 }}
+ {{- end }}
+ livenessProbe:
+ httpGet:
+ path: /liveness
+ port: {{ .Values.extractWorker.port | int }}
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: {{ .Values.extractWorker.port | int }}
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+{{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml
index 4a5a01dcc..df41f8ca8 100644
--- a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-deployment.yaml
@@ -26,7 +26,7 @@ spec:
- name: POSTGRES_USER
value: "{{ .Values.nuqPostgres.auth.username | default "postgres" }}"
- name: POSTGRES_PASSWORD
- value: "password"
+ value: "{{ .Values.nuqPostgres.auth.password | default "password" }}"
- name: POSTGRES_DB
value: "{{ .Values.nuqPostgres.auth.database | default "postgres" }}"
ports:
@@ -34,17 +34,19 @@ spec:
volumeMounts:
- name: postgres-storage
mountPath: /var/lib/postgresql/data
- {{- if .Values.nuqPostgres.resources }}
+ {{- if .Values.resources.enabled }}
+ {{- if .Values.nuqPostgres.resources }}
resources:
- {{- toYaml .Values.nuqPostgres.resources | nindent 12 }}
- {{- else }}
+ {{- toYaml .Values.nuqPostgres.resources | nindent 12 }}
+ {{- else }}
resources:
- requests:
- memory: "512Mi"
- cpu: "250m"
- limits:
- memory: "1Gi"
- cpu: "500m"
+ requests:
+ memory: "512Mi"
+ cpu: "250m"
+ limits:
+ memory: "1Gi"
+ cpu: "500m"
+ {{- end }}
{{- end }}
volumes:
- name: postgres-storage
diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yaml
new file mode 100644
index 000000000..59c49341d
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-postgres-pvc.yaml
@@ -0,0 +1,17 @@
+{{- if .Values.nuqPostgres.persistence.enabled }}
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-nuq-postgres-pvc
+ labels:
+ app: {{ include "firecrawl.name" . }}-nuq-postgres
+spec:
+ accessModes:
+ - ReadWriteOnce
+ resources:
+ requests:
+ storage: {{ .Values.nuqPostgres.persistence.size | quote }}
+ {{- if .Values.nuqPostgres.persistence.storageClass }}
+ storageClassName: {{ .Values.nuqPostgres.persistence.storageClass | quote }}
+ {{- end }}
+{{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yaml
new file mode 100644
index 000000000..2247e346a
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-prefetch-worker-deployment.yaml
@@ -0,0 +1,69 @@
+{{- if .Values.nuqPrefetchWorker.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-nuq-prefetch-worker
+ labels:
+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker
+spec:
+ replicas: {{ .Values.nuqPrefetchWorker.replicaCount | default 1 }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-nuq-prefetch-worker
+ spec:
+ {{- if .Values.image.dockerSecretEnabled }}
+ imagePullSecrets:
+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}
+ {{- end }}
+ terminationGracePeriodSeconds: 60
+ containers:
+ - name: nuq-prefetch-worker
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ command: [ "node" ]
+ args: [ "--max-old-space-size=2048", "dist/src/services/worker/nuq-prefetch-worker.js" ]
+ ports:
+ - containerPort: {{ .Values.nuqPrefetchWorker.port | int }}
+ env:
+ - name: FLY_PROCESS_GROUP
+ value: "nuq-prefetch-worker"
+ - name: NUQ_PREFETCH_WORKER_PORT
+ value: {{ .Values.nuqPrefetchWorker.port | quote }}
+ - name: NUQ_PREFETCH_REPLICAS
+ value: {{ .Values.nuqPrefetchWorker.replicaCount | quote }}
+ - name: NUQ_POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
+ envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-config
+ - secretRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
+ {{- if and .Values.resources.enabled .Values.nuqPrefetchWorker.resources }}
+ resources:
+ {{- toYaml .Values.nuqPrefetchWorker.resources | nindent 12 }}
+ {{- end }}
+ livenessProbe:
+ httpGet:
+ path: /health
+ port: {{ .Values.nuqPrefetchWorker.port | int }}
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: {{ .Values.nuqPrefetchWorker.port | int }}
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+{{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml
index d36d83403..d01967714 100644
--- a/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/nuq-worker-deployment.yaml
@@ -25,32 +25,40 @@ spec:
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: [ "node" ]
args: [ "--max-old-space-size=3072", "dist/src/services/worker/nuq-worker.js" ]
+ ports:
+ - containerPort: {{ .Values.nuqWorker.port | int }}
env:
- name: FLY_PROCESS_GROUP
value: "nuq-worker"
- name: NUQ_WORKER_PORT
- value: "3006"
+ value: {{ .Values.nuqWorker.port | quote }}
+ - name: NUQ_POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-config
- secretRef:
name: {{ include "firecrawl.fullname" . }}-secret
- {{- if .Values.nuqWorker.resources }}
+ {{- if .Values.resources.enabled }}
+ {{- if .Values.nuqWorker.resources }}
resources:
- {{- toYaml .Values.nuqWorker.resources | nindent 12 }}
- {{- else }}
+ {{- toYaml .Values.nuqWorker.resources | nindent 12 }}
+ {{- else }}
resources:
- requests:
- memory: "3G"
- cpu: "1000m"
- limits:
- memory: "4G"
- cpu: "1000m"
+ requests:
+ memory: "3G"
+ cpu: "1000m"
+ limits:
+ memory: "4G"
+ cpu: "1000m"
+ {{- end }}
{{- end }}
livenessProbe:
httpGet:
path: /health
- port: 3006
+ port: {{ .Values.nuqWorker.port | int }}
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
@@ -59,7 +67,7 @@ spec:
readinessProbe:
httpGet:
path: /health
- port: 3006
+ port: {{ .Values.nuqWorker.port | int }}
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 5
diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml
index bff1866c9..f3be19085 100644
--- a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml
@@ -3,4 +3,6 @@ kind: ConfigMap
metadata:
name: {{ include "firecrawl.fullname" . }}-playwright-config
data:
- PORT: {{ .Values.playwrightConfig.PORT | quote }}
+ PORT: {{ default (printf "%v" .Values.service.playwright.port) .Values.playwrightConfig.PORT | quote }}
+ BLOCK_MEDIA: {{ .Values.playwrightConfig.BLOCK_MEDIA | quote }}
+ MAX_CONCURRENT_PAGES: {{ .Values.playwrightConfig.MAX_CONCURRENT_PAGES | quote }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml
index 8f7c1243a..bfb1d8335 100644
--- a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml
@@ -5,7 +5,7 @@ metadata:
labels:
app: {{ include "firecrawl.name" . }}-playwright
spec:
- replicas: {{ .Values.replicaCount }}
+ replicas: {{ .Values.playwright.replicaCount | default .Values.replicaCount }}
selector:
matchLabels:
app: {{ include "firecrawl.name" . }}-playwright
@@ -25,11 +25,15 @@ spec:
ports:
- containerPort: {{ .Values.service.playwright.port }}
envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-config
+ - secretRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-playwright-config
livenessProbe:
httpGet:
- path: /health/liveness
+ path: /health
port: {{ .Values.service.playwright.port }}
initialDelaySeconds: 30
periodSeconds: 30
@@ -38,7 +42,7 @@ spec:
failureThreshold: 3
readinessProbe:
httpGet:
- path: /health/readiness
+ path: /health
port: {{ .Values.service.playwright.port }}
initialDelaySeconds: 30
periodSeconds: 30
diff --git a/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yaml
new file mode 100644
index 000000000..ddd1d9b65
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/rabbitmq-deployment.yaml
@@ -0,0 +1,60 @@
+{{- if .Values.rabbitmq.enabled }}
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-rabbitmq
+ labels:
+ app: {{ include "firecrawl.name" . }}-rabbitmq
+spec:
+ replicas: {{ .Values.rabbitmq.replicaCount | default 1 }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-rabbitmq
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-rabbitmq
+ spec:
+ containers:
+ - name: rabbitmq
+ image: {{ .Values.rabbitmq.image }}
+ ports:
+ - containerPort: {{ .Values.service.rabbitmq.port }}
+ - containerPort: {{ .Values.service.rabbitmq.managementPort }}
+ {{- if and .Values.resources.enabled .Values.rabbitmq.resources }}
+ resources:
+ {{- toYaml .Values.rabbitmq.resources | nindent 12 }}
+ {{- end }}
+ livenessProbe:
+ tcpSocket:
+ port: {{ .Values.service.rabbitmq.port }}
+ initialDelaySeconds: 20
+ periodSeconds: 10
+ timeoutSeconds: 5
+ failureThreshold: 6
+ readinessProbe:
+ tcpSocket:
+ port: {{ .Values.service.rabbitmq.port }}
+ initialDelaySeconds: 10
+ periodSeconds: 5
+ timeoutSeconds: 3
+ failureThreshold: 6
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-rabbitmq
+spec:
+ type: {{ .Values.service.rabbitmq.type }}
+ selector:
+ app: {{ include "firecrawl.name" . }}-rabbitmq
+ ports:
+ - name: amqp
+ protocol: TCP
+ port: {{ .Values.service.rabbitmq.port }}
+ targetPort: {{ .Values.service.rabbitmq.port }}
+ - name: management
+ protocol: TCP
+ port: {{ .Values.service.rabbitmq.managementPort }}
+ targetPort: {{ .Values.service.rabbitmq.managementPort }}
+{{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/secret.yaml b/examples/kubernetes/firecrawl-helm/templates/secret.yaml
index 3840b7c4e..585e3f512 100644
--- a/examples/kubernetes/firecrawl-helm/templates/secret.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/secret.yaml
@@ -9,8 +9,13 @@ data:
LLAMAPARSE_API_KEY: {{ .Values.secret.LLAMAPARSE_API_KEY | b64enc | quote }}
BULL_AUTH_KEY: {{ .Values.secret.BULL_AUTH_KEY | b64enc | quote }}
TEST_API_KEY: {{ .Values.secret.TEST_API_KEY | b64enc | quote }}
- SCRAPING_BEE_API_KEY: {{ .Values.secret.SCRAPING_BEE_API_KEY | b64enc | quote }}
- STRIPE_PRICE_ID_STANDARD: {{ .Values.secret.STRIPE_PRICE_ID_STANDARD | b64enc | quote }}
- STRIPE_PRICE_ID_SCALE: {{ .Values.secret.STRIPE_PRICE_ID_SCALE | b64enc | quote }}
FIRE_ENGINE_BETA_URL: {{ .Values.secret.FIRE_ENGINE_BETA_URL | b64enc | quote }}
+ SUPABASE_ANON_TOKEN: {{ .Values.secret.SUPABASE_ANON_TOKEN | b64enc | quote }}
+ SUPABASE_URL: {{ .Values.secret.SUPABASE_URL | b64enc | quote }}
+ SUPABASE_SERVICE_TOKEN: {{ .Values.secret.SUPABASE_SERVICE_TOKEN | b64enc | quote }}
+ PROXY_PASSWORD: {{ .Values.secret.PROXY_PASSWORD | b64enc | quote }}
+ SELF_HOSTED_WEBHOOK_HMAC_SECRET: {{ .Values.secret.SELF_HOSTED_WEBHOOK_HMAC_SECRET | b64enc | quote }}
REDIS_PASSWORD: {{ .Values.secret.REDIS_PASSWORD | b64enc | quote }}
+ {{- range $k, $v := .Values.secret.extra }}
+ {{ $k }}: {{ $v | b64enc | quote }}
+ {{- end }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml
index 787dcfe4a..64aaa5722 100644
--- a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml
+++ b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml
@@ -5,7 +5,7 @@ metadata:
labels:
app: {{ include "firecrawl.name" . }}-worker
spec:
- replicas: {{ .Values.replicaCount }}
+ replicas: {{ .Values.worker.replicaCount | default .Values.replicaCount }}
selector:
matchLabels:
app: {{ include "firecrawl.name" . }}-worker
@@ -18,16 +18,39 @@ spec:
imagePullSecrets:
{{- toYaml .Values.imagePullSecrets | nindent 8 }}
{{- end }}
+ terminationGracePeriodSeconds: 60
containers:
- name: worker
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
- args: [ "pnpm", "run", "workers" ]
+ command: [ "node" ]
+ args: [ "--max-old-space-size=3072", "dist/src/services/queue-worker.js" ]
+ ports:
+ - containerPort: {{ .Values.worker.port | int }}
env:
- name: FLY_PROCESS_GROUP
value: "worker"
+ - name: WORKER_PORT
+ value: {{ .Values.worker.port | quote }}
+ - name: NUQ_POD_NAME
+ valueFrom:
+ fieldRef:
+ fieldPath: metadata.name
envFrom:
- configMapRef:
name: {{ include "firecrawl.fullname" . }}-config
- secretRef:
name: {{ include "firecrawl.fullname" . }}-secret
+ {{- if and .Values.resources.enabled .Values.worker.resources }}
+ resources:
+ {{- toYaml .Values.worker.resources | nindent 12 }}
+ {{- end }}
+ livenessProbe:
+ httpGet:
+ path: /liveness
+ port: {{ .Values.worker.port | int }}
+ initialDelaySeconds: 10
+ periodSeconds: 10
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
diff --git a/examples/kubernetes/firecrawl-helm/values.yaml b/examples/kubernetes/firecrawl-helm/values.yaml
index 853462b3d..afa9868fb 100644
--- a/examples/kubernetes/firecrawl-helm/values.yaml
+++ b/examples/kubernetes/firecrawl-helm/values.yaml
@@ -1,7 +1,44 @@
replicaCount: 1
+resources:
+ enabled: false
+
+api:
+ replicaCount: 1
+ resources:
+ requests:
+ memory: "4G"
+ cpu: "2000m"
+ limits:
+ memory: "6G"
+ cpu: "2000m"
+
+worker:
+ replicaCount: 1
+ port: 3005
+ resources:
+ requests:
+ memory: "3G"
+ cpu: "1000m"
+ limits:
+ memory: "4G"
+ cpu: "1000m"
+
+extractWorker:
+ enabled: true
+ replicaCount: 1
+ port: 3004
+ resources:
+ requests:
+ memory: "2G"
+ cpu: "1000m"
+ limits:
+ memory: "3G"
+ cpu: "1000m"
+
nuqWorker:
replicaCount: 5
+ port: 3006
resources:
requests:
memory: "3G"
@@ -10,14 +47,27 @@ nuqWorker:
memory: "4G"
cpu: "1000m"
+nuqPrefetchWorker:
+ enabled: true
+ replicaCount: 1
+ port: 3011
+ resources:
+ requests:
+ memory: "1G"
+ cpu: "500m"
+ limits:
+ memory: "2G"
+ cpu: "1000m"
+
nuqPostgres:
replicaCount: 1
image:
- repository: ghcr.io/firecrawl/nuq-postgres
+ repository: docker.io/winkkgmbh/nuq-postgres
tag: latest
pullPolicy: Always
auth:
username: postgres
+ password: password
database: postgres
resources:
requests:
@@ -28,17 +78,32 @@ nuqPostgres:
cpu: "500m"
persistence:
enabled: false
+ size: 10Gi
+ storageClass: ""
+
+rabbitmq:
+ enabled: true
+ image: rabbitmq:3-management
+ replicaCount: 1
+ resources:
+ requests:
+ memory: "512Mi"
+ cpu: "250m"
+ limits:
+ memory: "1Gi"
+ cpu: "500m"
image:
- repository: ghcr.io/firecrawl/firecrawl
+ repository: docker.io/winkkgmbh/firecrawl
tag: "latest"
pullPolicy: Always
- dockerSecretEnabled: true
+ dockerSecretEnabled: false
playwright:
- repository: ghcr.io/firecrawl/playwright-service
+ repository: docker.io/winkkgmbh/firecrawl-playwright
tag: "latest"
pullPolicy: Always
+ replicaCount: 1
redis:
image: redis:alpine
@@ -54,20 +119,49 @@ service:
redis:
type: ClusterIP
port: 6379
+ rabbitmq:
+ type: ClusterIP
+ port: 5672
+ managementPort: 15672
config:
NUM_WORKERS_PER_QUEUE: "8"
PORT: "3002"
HOST: "0.0.0.0"
- REDIS_URL: "redis://redis:6379"
- REDIS_RATE_LIMIT_URL: "redis://redis:6379"
- PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000"
+ WORKER_PORT: "3005"
+ EXTRACT_WORKER_PORT: "3004"
+ NUQ_WORKER_PORT: "3006"
+ NUQ_PREFETCH_WORKER_PORT: "3011"
+ NUQ_WORKER_COUNT: "5"
USE_DB_AUTHENTICATION: "false"
- HDX_NODE_BETA_MODE: "1"
- NUQ_DATABASE_URL: "postgresql://postgres:password@nuq-postgres:5432/postgres"
+ IS_KUBERNETES: "true"
+ ENV: "production"
+ LOGGING_LEVEL: "INFO"
+ FIRECRAWL_APP_SCHEME: "http"
+ FIRECRAWL_APP_HOST: ""
+ FIRECRAWL_APP_PORT: ""
+ REDIS_URL: ""
+ REDIS_RATE_LIMIT_URL: ""
+ PLAYWRIGHT_MICROSERVICE_URL: ""
+ NUQ_DATABASE_URL: ""
+ NUQ_DATABASE_URL_LISTEN: ""
+ NUQ_RABBITMQ_URL: ""
+ OPENAI_BASE_URL: ""
+ MODEL_NAME: ""
+ MODEL_EMBEDDING_NAME: ""
+ OLLAMA_BASE_URL: ""
+ PROXY_SERVER: ""
+ PROXY_USERNAME: ""
+ SEARXNG_ENDPOINT: ""
+ SEARXNG_ENGINES: ""
+ SEARXNG_CATEGORIES: ""
+ SELF_HOSTED_WEBHOOK_URL: ""
+ extra: {}
playwrightConfig:
PORT: "3000"
+ BLOCK_MEDIA: ""
+ MAX_CONCURRENT_PAGES: "10"
secret:
OPENAI_API_KEY: ""
@@ -75,11 +169,14 @@ secret:
LLAMAPARSE_API_KEY: ""
BULL_AUTH_KEY: ""
TEST_API_KEY: ""
- SCRAPING_BEE_API_KEY: ""
- STRIPE_PRICE_ID_STANDARD: ""
- STRIPE_PRICE_ID_SCALE: ""
FIRE_ENGINE_BETA_URL: ""
+ SUPABASE_ANON_TOKEN: ""
+ SUPABASE_URL: ""
+ SUPABASE_SERVICE_TOKEN: ""
+ PROXY_PASSWORD: ""
+ SELF_HOSTED_WEBHOOK_HMAC_SECRET: ""
REDIS_PASSWORD: ""
+ extra: {}
imagePullSecrets:
- name: docker-registry-secret