Skip to content

Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥

feat: add initial Helm chart for Firecrawl deployment (#1262)

Jakob Stadlhuberd98277864bf1
diff --git a/SELF_HOST.md b/SELF_HOST.md
index 4074eb220..08f80a43b 100644
--- a/SELF_HOST.md
+++ b/SELF_HOST.md
@@ -209,3 +209,7 @@ By addressing these common issues, you can ensure a smoother setup and operation
## Install Firecrawl on a Kubernetes Cluster (Simple Version)
Read the [examples/kubernetes/cluster-install/README.md](https://github.com/firecrawl/firecrawl/blob/main/examples/kubernetes/cluster-install/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster.
+
+## Install Firecrawl on a Kubernetes Cluster with Helm
+
+Read the [examples/kubernetes/firecrawl-helm/README.md](https://github.com/firecrawl/firecrawl/blob/main/examples/kubernetes/firecrawl-helm/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster with Helm.
diff --git a/examples/kubernetes/firecrawl-helm/.helmignore b/examples/kubernetes/firecrawl-helm/.helmignore
new file mode 100644
index 000000000..0e8a0eb36
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/.helmignore
@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*.orig
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+.vscode/
diff --git a/examples/kubernetes/firecrawl-helm/Chart.yaml b/examples/kubernetes/firecrawl-helm/Chart.yaml
new file mode 100644
index 000000000..dc97ecad8
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/Chart.yaml
@@ -0,0 +1,5 @@
+apiVersion: v2
+name: firecrawl
+description: A Helm chart for deploying the Firecrawl application
+type: application
+version: 0.1.0
diff --git a/examples/kubernetes/firecrawl-helm/README.md b/examples/kubernetes/firecrawl-helm/README.md
new file mode 100644
index 000000000..f8744818d
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/README.md
@@ -0,0 +1,51 @@
+# Firecrawl Helm Chart
+This Helm chart deploys Firecrawl components (API, Worker, Playwright service, Redis, etc.) on a Kubernetes cluster using environment-specific overlays.
+
+## Pre-deployment
+
+1. **Configure Secrets and ConfigMaps:**
+ - Update `values.yaml` with the necessary environment-specific values in the `overlays/<dev|prod>/values.yaml` file.
+ - **Note:** If using `REDIS_PASSWORD`, adjust the `REDIS_URL` and `REDIS_RATE_LIMIT_URL` in the ConfigMap to include the password.
+
+2. **Build and Push Docker Images:**
+ - API/Worker:
+ ```bash
+ docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl:latest ../../../apps/api
+ docker push ghcr.io/winkk-dev/firecrawl:latest
+ ```
+ - Playwright Service:
+ ```bash
+ docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl-playwright:latest ../../../apps/playwright-service
+ docker push ghcr.io/winkk-dev/firecrawl-playwright:latest
+ ```
+
+## Deployment
+
+Render the manifests for review:
+```bash
+helm template winkk-ai . -f values.yaml -f overlays/dev/values.yaml -n winkk-ai
+```
+
+Deploy or upgrade the release:
+```bash
+helm upgrade firecrawl . -f values.yaml -f overlays/dev/values.yaml -n firecrawl --install --create-namespace
+```
+
+## Testing
+
+Forward the API service port to your local machine:
+```bash
+kubectl port-forward svc/firecrawl-api 3002:3002 -n firecrawl
+```
+
+## Cleanup
+
+To uninstall the deployment:
+```bash
+helm uninstall firecrawl -n firecrawl
+```
+```
+
+---
+
+This README provides a quick guide to configuring, deploying, testing, and cleaning up your Firecrawl installation using Helm.
\ No newline at end of file
diff --git a/examples/kubernetes/firecrawl-helm/overlays/dev/values.yaml b/examples/kubernetes/firecrawl-helm/overlays/dev/values.yaml
new file mode 100644
index 000000000..f45223749
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/overlays/dev/values.yaml
@@ -0,0 +1 @@
+# Override the default values for the dev environment
diff --git a/examples/kubernetes/firecrawl-helm/overlays/prod/values.yaml b/examples/kubernetes/firecrawl-helm/overlays/prod/values.yaml
new file mode 100644
index 000000000..2b6554909
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/overlays/prod/values.yaml
@@ -0,0 +1 @@
+# Override the default values for the prod environment
diff --git a/examples/kubernetes/firecrawl-helm/templates/_helpers.tpl b/examples/kubernetes/firecrawl-helm/templates/_helpers.tpl
new file mode 100644
index 000000000..41646a924
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/_helpers.tpl
@@ -0,0 +1,18 @@
+{{/*
+Return the name of the chart.
+*/}}
+{{- define "firecrawl.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Return the fully qualified name of the chart.
+*/}}
+{{- define "firecrawl.fullname" -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+{{- end -}}
diff --git a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml
new file mode 100644
index 000000000..ade1e1049
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml
@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-config
+data:
+ NUM_WORKERS_PER_QUEUE: {{ .Values.config.NUM_WORKERS_PER_QUEUE | quote }}
+ PORT: {{ .Values.config.PORT | quote }}
+ HOST: {{ .Values.config.HOST | quote }}
+ REDIS_URL: {{ .Values.config.REDIS_URL | quote }}
+ REDIS_RATE_LIMIT_URL: {{ .Values.config.REDIS_RATE_LIMIT_URL | quote }}
+ PLAYWRIGHT_MICROSERVICE_URL: {{ .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}
+ USE_DB_AUTHENTICATION: {{ .Values.config.USE_DB_AUTHENTICATION | quote }}
+ HDX_NODE_BETA_MODE: {{ .Values.config.HDX_NODE_BETA_MODE | quote }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml
new file mode 100644
index 000000000..20d772fb1
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml
@@ -0,0 +1,53 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-api
+ labels:
+ app: {{ include "firecrawl.name" . }}-api
+spec:
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-api
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-api
+ spec:
+ {{- if .Values.image.dockerSecretEnabled }}
+ imagePullSecrets:
+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}
+ {{- end }}
+ containers:
+ - name: api
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ args: [ "pnpm", "run", "start:production" ]
+ ports:
+ - containerPort: {{ .Values.service.api.port }}
+ env:
+ - name: FLY_PROCESS_GROUP
+ value: "app"
+ envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-config
+ - secretRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
+ livenessProbe:
+ httpGet:
+ path: /v0/health/liveness
+ port: {{ .Values.service.api.port }}
+ initialDelaySeconds: 30
+ periodSeconds: 30
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+ readinessProbe:
+ httpGet:
+ path: /v0/health/readiness
+ port: {{ .Values.service.api.port }}
+ initialDelaySeconds: 30
+ periodSeconds: 30
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml
new file mode 100644
index 000000000..bff1866c9
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml
@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-playwright-config
+data:
+ PORT: {{ .Values.playwrightConfig.PORT | quote }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml
new file mode 100644
index 000000000..8f7c1243a
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml
@@ -0,0 +1,47 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-playwright
+ labels:
+ app: {{ include "firecrawl.name" . }}-playwright
+spec:
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-playwright
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-playwright
+ spec:
+ {{- if .Values.image.dockerSecretEnabled }}
+ imagePullSecrets:
+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}
+ {{- end }}
+ containers:
+ - name: playwright-service
+ image: "{{ .Values.playwright.repository }}:{{ .Values.playwright.tag }}"
+ imagePullPolicy: {{ .Values.playwright.pullPolicy }}
+ ports:
+ - containerPort: {{ .Values.service.playwright.port }}
+ envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-playwright-config
+ livenessProbe:
+ httpGet:
+ path: /health/liveness
+ port: {{ .Values.service.playwright.port }}
+ initialDelaySeconds: 30
+ periodSeconds: 30
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
+ readinessProbe:
+ httpGet:
+ path: /health/readiness
+ port: {{ .Values.service.playwright.port }}
+ initialDelaySeconds: 30
+ periodSeconds: 30
+ timeoutSeconds: 5
+ successThreshold: 1
+ failureThreshold: 3
diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-service.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-service.yaml
new file mode 100644
index 000000000..ca5322d85
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-service.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-playwright
+spec:
+ type: {{ .Values.service.playwright.type }}
+ selector:
+ app: {{ include "firecrawl.name" . }}-playwright
+ ports:
+ - protocol: TCP
+ port: {{ .Values.service.playwright.port }}
+ targetPort: {{ .Values.service.playwright.port }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yaml
new file mode 100644
index 000000000..f6bfa7371
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yaml
@@ -0,0 +1,35 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-redis
+ labels:
+ app: {{ include "firecrawl.name" . }}-redis
+spec:
+ replicas: {{ .Values.redis.replicaCount }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-redis
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-redis
+ spec:
+ containers:
+ - name: redis
+ image: {{ .Values.redis.image }}
+ command: [ "/bin/sh", "-c" ]
+ args:
+ - |
+ if [ -n "$REDIS_PASSWORD" ]; then
+ echo "Starting Redis with authentication"
+ exec redis-server --bind 0.0.0.0 --requirepass "$REDIS_PASSWORD"
+ else
+ echo "Starting Redis without authentication"
+ exec redis-server --bind 0.0.0.0
+ fi
+ env:
+ - name: REDIS_PASSWORD
+ valueFrom:
+ secretKeyRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
+ key: REDIS_PASSWORD
diff --git a/examples/kubernetes/firecrawl-helm/templates/redis-service.yaml b/examples/kubernetes/firecrawl-helm/templates/redis-service.yaml
new file mode 100644
index 000000000..cc2d69296
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/redis-service.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-redis
+spec:
+ type: {{ .Values.service.redis.type }}
+ selector:
+ app: {{ include "firecrawl.name" . }}-redis
+ ports:
+ - protocol: TCP
+ port: {{ .Values.service.redis.port }}
+ targetPort: {{ .Values.service.redis.port }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/secret.yaml b/examples/kubernetes/firecrawl-helm/templates/secret.yaml
new file mode 100644
index 000000000..94add392f
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/secret.yaml
@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Secret
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-secret
+type: Opaque
+data:
+ OPENAI_API_KEY: {{ .Values.secret.OPENAI_API_KEY | b64enc | quote }}
+ SLACK_WEBHOOK_URL: {{ .Values.secret.SLACK_WEBHOOK_URL | b64enc | quote }}
+ LLAMAPARSE_API_KEY: {{ .Values.secret.LLAMAPARSE_API_KEY | b64enc | quote }}
+ BULL_AUTH_KEY: {{ .Values.secret.BULL_AUTH_KEY | b64enc | quote }}
+ TEST_API_KEY: {{ .Values.secret.TEST_API_KEY | b64enc | quote }}
+ POSTHOG_API_KEY: {{ .Values.secret.POSTHOG_API_KEY | b64enc | quote }}
+ POSTHOG_HOST: {{ .Values.secret.POSTHOG_HOST | b64enc | quote }}
+ SCRAPING_BEE_API_KEY: {{ .Values.secret.SCRAPING_BEE_API_KEY | b64enc | quote }}
+ STRIPE_PRICE_ID_STANDARD: {{ .Values.secret.STRIPE_PRICE_ID_STANDARD | b64enc | quote }}
+ STRIPE_PRICE_ID_SCALE: {{ .Values.secret.STRIPE_PRICE_ID_SCALE | b64enc | quote }}
+ FIRE_ENGINE_BETA_URL: {{ .Values.secret.FIRE_ENGINE_BETA_URL | b64enc | quote }}
+ REDIS_PASSWORD: {{ .Values.secret.REDIS_PASSWORD | b64enc | quote }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/service.yaml b/examples/kubernetes/firecrawl-helm/templates/service.yaml
new file mode 100644
index 000000000..fb7139ee0
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/service.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-api
+spec:
+ type: {{ .Values.service.api.type }}
+ selector:
+ app: {{ include "firecrawl.name" . }}-api
+ ports:
+ - protocol: TCP
+ port: {{ .Values.service.api.port }}
+ targetPort: {{ .Values.service.api.port }}
diff --git a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml
new file mode 100644
index 000000000..787dcfe4a
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml
@@ -0,0 +1,33 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: {{ include "firecrawl.fullname" . }}-worker
+ labels:
+ app: {{ include "firecrawl.name" . }}-worker
+spec:
+ replicas: {{ .Values.replicaCount }}
+ selector:
+ matchLabels:
+ app: {{ include "firecrawl.name" . }}-worker
+ template:
+ metadata:
+ labels:
+ app: {{ include "firecrawl.name" . }}-worker
+ spec:
+ {{- if .Values.image.dockerSecretEnabled }}
+ imagePullSecrets:
+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}
+ {{- end }}
+ containers:
+ - name: worker
+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+ imagePullPolicy: {{ .Values.image.pullPolicy }}
+ args: [ "pnpm", "run", "workers" ]
+ env:
+ - name: FLY_PROCESS_GROUP
+ value: "worker"
+ envFrom:
+ - configMapRef:
+ name: {{ include "firecrawl.fullname" . }}-config
+ - secretRef:
+ name: {{ include "firecrawl.fullname" . }}-secret
diff --git a/examples/kubernetes/firecrawl-helm/values.yaml b/examples/kubernetes/firecrawl-helm/values.yaml
new file mode 100644
index 000000000..fb6ca2ba4
--- /dev/null
+++ b/examples/kubernetes/firecrawl-helm/values.yaml
@@ -0,0 +1,57 @@
+replicaCount: 1
+
+image:
+ repository: ghcr.io/firecrawl/firecrawl
+ tag: "latest"
+ pullPolicy: Always
+ dockerSecretEnabled: true
+
+playwright:
+ repository: ghcr.io/firecrawl/playwright-service
+ tag: "latest"
+ pullPolicy: Always
+
+redis:
+ image: redis:alpine
+ replicaCount: 1
+
+service:
+ api:
+ type: ClusterIP
+ port: 3002
+ playwright:
+ type: ClusterIP
+ port: 3000
+ redis:
+ type: ClusterIP
+ port: 6379
+
+config:
+ NUM_WORKERS_PER_QUEUE: "8"
+ PORT: "3002"
+ HOST: "0.0.0.0"
+ REDIS_URL: "redis://redis:6379"
+ REDIS_RATE_LIMIT_URL: "redis://redis:6379"
+ PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000"
+ USE_DB_AUTHENTICATION: "false"
+ HDX_NODE_BETA_MODE: "1"
+
+playwrightConfig:
+ PORT: "3000"
+
+secret:
+ OPENAI_API_KEY: ""
+ SLACK_WEBHOOK_URL: ""
+ LLAMAPARSE_API_KEY: ""
+ BULL_AUTH_KEY: ""
+ TEST_API_KEY: ""
+ POSTHOG_API_KEY: ""
+ POSTHOG_HOST: ""
+ SCRAPING_BEE_API_KEY: ""
+ STRIPE_PRICE_ID_STANDARD: ""
+ STRIPE_PRICE_ID_SCALE: ""
+ FIRE_ENGINE_BETA_URL: ""
+ REDIS_PASSWORD: ""
+
+imagePullSecrets:
+ - name: docker-registry-secret