Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat: add initial Helm chart for Firecrawl deployment (#1262)
Jakob Stadlhuberd98277864bf1
diff --git a/SELF_HOST.md b/SELF_HOST.mdindex 4074eb220..08f80a43b 100644--- a/SELF_HOST.md+++ b/SELF_HOST.md@@ -209,3 +209,7 @@ By addressing these common issues, you can ensure a smoother setup and operation## Install Firecrawl on a Kubernetes Cluster (Simple Version)Read the [examples/kubernetes/cluster-install/README.md](https://github.com/firecrawl/firecrawl/blob/main/examples/kubernetes/cluster-install/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster.++## Install Firecrawl on a Kubernetes Cluster with Helm++Read the [examples/kubernetes/firecrawl-helm/README.md](https://github.com/firecrawl/firecrawl/blob/main/examples/kubernetes/firecrawl-helm/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster with Helm.diff --git a/examples/kubernetes/firecrawl-helm/.helmignore b/examples/kubernetes/firecrawl-helm/.helmignorenew file mode 100644index 000000000..0e8a0eb36--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/.helmignore@@ -0,0 +1,23 @@+# Patterns to ignore when building packages.+# This supports shell glob matching, relative path matching, and+# negation (prefixed with !). Only one pattern per line.+.DS_Store+# Common VCS dirs+.git/+.gitignore+.bzr/+.bzrignore+.hg/+.hgignore+.svn/+# Common backup files+*.swp+*.bak+*.tmp+*.orig+*~+# Various IDEs+.project+.idea/+*.tmproj+.vscode/diff --git a/examples/kubernetes/firecrawl-helm/Chart.yaml b/examples/kubernetes/firecrawl-helm/Chart.yamlnew file mode 100644index 000000000..dc97ecad8--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/Chart.yaml@@ -0,0 +1,5 @@+apiVersion: v2+name: firecrawl+description: A Helm chart for deploying the Firecrawl application+type: application+version: 0.1.0diff --git a/examples/kubernetes/firecrawl-helm/README.md b/examples/kubernetes/firecrawl-helm/README.mdnew file mode 100644index 000000000..f8744818d--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/README.md@@ -0,0 +1,51 @@+# Firecrawl Helm Chart+This Helm chart deploys Firecrawl components (API, Worker, Playwright service, Redis, etc.) on a Kubernetes cluster using environment-specific overlays.++## Pre-deployment++1. **Configure Secrets and ConfigMaps:**+ - Update `values.yaml` with the necessary environment-specific values in the `overlays/<dev|prod>/values.yaml` file.+ - **Note:** If using `REDIS_PASSWORD`, adjust the `REDIS_URL` and `REDIS_RATE_LIMIT_URL` in the ConfigMap to include the password.++2. **Build and Push Docker Images:**+ - API/Worker:+ ```bash+ docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl:latest ../../../apps/api+ docker push ghcr.io/winkk-dev/firecrawl:latest+ ```+ - Playwright Service:+ ```bash+ docker build --no-cache --platform linux/amd64 -t ghcr.io/winkk-dev/firecrawl-playwright:latest ../../../apps/playwright-service+ docker push ghcr.io/winkk-dev/firecrawl-playwright:latest+ ```++## Deployment++Render the manifests for review:+```bash+helm template winkk-ai . -f values.yaml -f overlays/dev/values.yaml -n winkk-ai+```++Deploy or upgrade the release:+```bash+helm upgrade firecrawl . -f values.yaml -f overlays/dev/values.yaml -n firecrawl --install --create-namespace+```++## Testing++Forward the API service port to your local machine:+```bash+kubectl port-forward svc/firecrawl-api 3002:3002 -n firecrawl+```++## Cleanup++To uninstall the deployment:+```bash+helm uninstall firecrawl -n firecrawl+```+```++---++This README provides a quick guide to configuring, deploying, testing, and cleaning up your Firecrawl installation using Helm.\ No newline at end of filediff --git a/examples/kubernetes/firecrawl-helm/overlays/dev/values.yaml b/examples/kubernetes/firecrawl-helm/overlays/dev/values.yamlnew file mode 100644index 000000000..f45223749--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/overlays/dev/values.yaml@@ -0,0 +1 @@+# Override the default values for the dev environmentdiff --git a/examples/kubernetes/firecrawl-helm/overlays/prod/values.yaml b/examples/kubernetes/firecrawl-helm/overlays/prod/values.yamlnew file mode 100644index 000000000..2b6554909--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/overlays/prod/values.yaml@@ -0,0 +1 @@+# Override the default values for the prod environmentdiff --git a/examples/kubernetes/firecrawl-helm/templates/_helpers.tpl b/examples/kubernetes/firecrawl-helm/templates/_helpers.tplnew file mode 100644index 000000000..41646a924--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/_helpers.tpl@@ -0,0 +1,18 @@+{{/*+Return the name of the chart.+*/}}+{{- define "firecrawl.name" -}}+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}+{{- end -}}++{{/*+Return the fully qualified name of the chart.+*/}}+{{- define "firecrawl.fullname" -}}+{{- $name := default .Chart.Name .Values.nameOverride -}}+{{- if .Values.fullnameOverride -}}+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}+{{- else -}}+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}+{{- end -}}+{{- end -}}diff --git a/examples/kubernetes/firecrawl-helm/templates/configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/configmap.yamlnew file mode 100644index 000000000..ade1e1049--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/configmap.yaml@@ -0,0 +1,13 @@+apiVersion: v1+kind: ConfigMap+metadata:+ name: {{ include "firecrawl.fullname" . }}-config+data:+ NUM_WORKERS_PER_QUEUE: {{ .Values.config.NUM_WORKERS_PER_QUEUE | quote }}+ PORT: {{ .Values.config.PORT | quote }}+ HOST: {{ .Values.config.HOST | quote }}+ REDIS_URL: {{ .Values.config.REDIS_URL | quote }}+ REDIS_RATE_LIMIT_URL: {{ .Values.config.REDIS_RATE_LIMIT_URL | quote }}+ PLAYWRIGHT_MICROSERVICE_URL: {{ .Values.config.PLAYWRIGHT_MICROSERVICE_URL | quote }}+ USE_DB_AUTHENTICATION: {{ .Values.config.USE_DB_AUTHENTICATION | quote }}+ HDX_NODE_BETA_MODE: {{ .Values.config.HDX_NODE_BETA_MODE | quote }}diff --git a/examples/kubernetes/firecrawl-helm/templates/deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/deployment.yamlnew file mode 100644index 000000000..20d772fb1--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/deployment.yaml@@ -0,0 +1,53 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-api+ labels:+ app: {{ include "firecrawl.name" . }}-api+spec:+ replicas: {{ .Values.replicaCount }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-api+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-api+ spec:+ {{- if .Values.image.dockerSecretEnabled }}+ imagePullSecrets:+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}+ {{- end }}+ containers:+ - name: api+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"+ imagePullPolicy: {{ .Values.image.pullPolicy }}+ args: [ "pnpm", "run", "start:production" ]+ ports:+ - containerPort: {{ .Values.service.api.port }}+ env:+ - name: FLY_PROCESS_GROUP+ value: "app"+ envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-config+ - secretRef:+ name: {{ include "firecrawl.fullname" . }}-secret+ livenessProbe:+ httpGet:+ path: /v0/health/liveness+ port: {{ .Values.service.api.port }}+ initialDelaySeconds: 30+ periodSeconds: 30+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+ readinessProbe:+ httpGet:+ path: /v0/health/readiness+ port: {{ .Values.service.api.port }}+ initialDelaySeconds: 30+ periodSeconds: 30+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yamlnew file mode 100644index 000000000..bff1866c9--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-configmap.yaml@@ -0,0 +1,6 @@+apiVersion: v1+kind: ConfigMap+metadata:+ name: {{ include "firecrawl.fullname" . }}-playwright-config+data:+ PORT: {{ .Values.playwrightConfig.PORT | quote }}diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yamlnew file mode 100644index 000000000..8f7c1243a--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-deployment.yaml@@ -0,0 +1,47 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-playwright+ labels:+ app: {{ include "firecrawl.name" . }}-playwright+spec:+ replicas: {{ .Values.replicaCount }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-playwright+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-playwright+ spec:+ {{- if .Values.image.dockerSecretEnabled }}+ imagePullSecrets:+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}+ {{- end }}+ containers:+ - name: playwright-service+ image: "{{ .Values.playwright.repository }}:{{ .Values.playwright.tag }}"+ imagePullPolicy: {{ .Values.playwright.pullPolicy }}+ ports:+ - containerPort: {{ .Values.service.playwright.port }}+ envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-playwright-config+ livenessProbe:+ httpGet:+ path: /health/liveness+ port: {{ .Values.service.playwright.port }}+ initialDelaySeconds: 30+ periodSeconds: 30+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3+ readinessProbe:+ httpGet:+ path: /health/readiness+ port: {{ .Values.service.playwright.port }}+ initialDelaySeconds: 30+ periodSeconds: 30+ timeoutSeconds: 5+ successThreshold: 1+ failureThreshold: 3diff --git a/examples/kubernetes/firecrawl-helm/templates/playwright-service.yaml b/examples/kubernetes/firecrawl-helm/templates/playwright-service.yamlnew file mode 100644index 000000000..ca5322d85--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/playwright-service.yaml@@ -0,0 +1,12 @@+apiVersion: v1+kind: Service+metadata:+ name: {{ include "firecrawl.fullname" . }}-playwright+spec:+ type: {{ .Values.service.playwright.type }}+ selector:+ app: {{ include "firecrawl.name" . }}-playwright+ ports:+ - protocol: TCP+ port: {{ .Values.service.playwright.port }}+ targetPort: {{ .Values.service.playwright.port }}diff --git a/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yamlnew file mode 100644index 000000000..f6bfa7371--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/redis-deployment.yaml@@ -0,0 +1,35 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-redis+ labels:+ app: {{ include "firecrawl.name" . }}-redis+spec:+ replicas: {{ .Values.redis.replicaCount }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-redis+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-redis+ spec:+ containers:+ - name: redis+ image: {{ .Values.redis.image }}+ command: [ "/bin/sh", "-c" ]+ args:+ - |+ if [ -n "$REDIS_PASSWORD" ]; then+ echo "Starting Redis with authentication"+ exec redis-server --bind 0.0.0.0 --requirepass "$REDIS_PASSWORD"+ else+ echo "Starting Redis without authentication"+ exec redis-server --bind 0.0.0.0+ fi+ env:+ - name: REDIS_PASSWORD+ valueFrom:+ secretKeyRef:+ name: {{ include "firecrawl.fullname" . }}-secret+ key: REDIS_PASSWORDdiff --git a/examples/kubernetes/firecrawl-helm/templates/redis-service.yaml b/examples/kubernetes/firecrawl-helm/templates/redis-service.yamlnew file mode 100644index 000000000..cc2d69296--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/redis-service.yaml@@ -0,0 +1,12 @@+apiVersion: v1+kind: Service+metadata:+ name: {{ include "firecrawl.fullname" . }}-redis+spec:+ type: {{ .Values.service.redis.type }}+ selector:+ app: {{ include "firecrawl.name" . }}-redis+ ports:+ - protocol: TCP+ port: {{ .Values.service.redis.port }}+ targetPort: {{ .Values.service.redis.port }}diff --git a/examples/kubernetes/firecrawl-helm/templates/secret.yaml b/examples/kubernetes/firecrawl-helm/templates/secret.yamlnew file mode 100644index 000000000..94add392f--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/secret.yaml@@ -0,0 +1,18 @@+apiVersion: v1+kind: Secret+metadata:+ name: {{ include "firecrawl.fullname" . }}-secret+type: Opaque+data:+ OPENAI_API_KEY: {{ .Values.secret.OPENAI_API_KEY | b64enc | quote }}+ SLACK_WEBHOOK_URL: {{ .Values.secret.SLACK_WEBHOOK_URL | b64enc | quote }}+ LLAMAPARSE_API_KEY: {{ .Values.secret.LLAMAPARSE_API_KEY | b64enc | quote }}+ BULL_AUTH_KEY: {{ .Values.secret.BULL_AUTH_KEY | b64enc | quote }}+ TEST_API_KEY: {{ .Values.secret.TEST_API_KEY | b64enc | quote }}+ POSTHOG_API_KEY: {{ .Values.secret.POSTHOG_API_KEY | b64enc | quote }}+ POSTHOG_HOST: {{ .Values.secret.POSTHOG_HOST | b64enc | quote }}+ SCRAPING_BEE_API_KEY: {{ .Values.secret.SCRAPING_BEE_API_KEY | b64enc | quote }}+ STRIPE_PRICE_ID_STANDARD: {{ .Values.secret.STRIPE_PRICE_ID_STANDARD | b64enc | quote }}+ STRIPE_PRICE_ID_SCALE: {{ .Values.secret.STRIPE_PRICE_ID_SCALE | b64enc | quote }}+ FIRE_ENGINE_BETA_URL: {{ .Values.secret.FIRE_ENGINE_BETA_URL | b64enc | quote }}+ REDIS_PASSWORD: {{ .Values.secret.REDIS_PASSWORD | b64enc | quote }}diff --git a/examples/kubernetes/firecrawl-helm/templates/service.yaml b/examples/kubernetes/firecrawl-helm/templates/service.yamlnew file mode 100644index 000000000..fb7139ee0--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/service.yaml@@ -0,0 +1,12 @@+apiVersion: v1+kind: Service+metadata:+ name: {{ include "firecrawl.fullname" . }}-api+spec:+ type: {{ .Values.service.api.type }}+ selector:+ app: {{ include "firecrawl.name" . }}-api+ ports:+ - protocol: TCP+ port: {{ .Values.service.api.port }}+ targetPort: {{ .Values.service.api.port }}diff --git a/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yamlnew file mode 100644index 000000000..787dcfe4a--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/templates/worker-deployment.yaml@@ -0,0 +1,33 @@+apiVersion: apps/v1+kind: Deployment+metadata:+ name: {{ include "firecrawl.fullname" . }}-worker+ labels:+ app: {{ include "firecrawl.name" . }}-worker+spec:+ replicas: {{ .Values.replicaCount }}+ selector:+ matchLabels:+ app: {{ include "firecrawl.name" . }}-worker+ template:+ metadata:+ labels:+ app: {{ include "firecrawl.name" . }}-worker+ spec:+ {{- if .Values.image.dockerSecretEnabled }}+ imagePullSecrets:+ {{- toYaml .Values.imagePullSecrets | nindent 8 }}+ {{- end }}+ containers:+ - name: worker+ image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"+ imagePullPolicy: {{ .Values.image.pullPolicy }}+ args: [ "pnpm", "run", "workers" ]+ env:+ - name: FLY_PROCESS_GROUP+ value: "worker"+ envFrom:+ - configMapRef:+ name: {{ include "firecrawl.fullname" . }}-config+ - secretRef:+ name: {{ include "firecrawl.fullname" . }}-secretdiff --git a/examples/kubernetes/firecrawl-helm/values.yaml b/examples/kubernetes/firecrawl-helm/values.yamlnew file mode 100644index 000000000..fb6ca2ba4--- /dev/null+++ b/examples/kubernetes/firecrawl-helm/values.yaml@@ -0,0 +1,57 @@+replicaCount: 1++image:+ repository: ghcr.io/firecrawl/firecrawl+ tag: "latest"+ pullPolicy: Always+ dockerSecretEnabled: true++playwright:+ repository: ghcr.io/firecrawl/playwright-service+ tag: "latest"+ pullPolicy: Always++redis:+ image: redis:alpine+ replicaCount: 1++service:+ api:+ type: ClusterIP+ port: 3002+ playwright:+ type: ClusterIP+ port: 3000+ redis:+ type: ClusterIP+ port: 6379++config:+ NUM_WORKERS_PER_QUEUE: "8"+ PORT: "3002"+ HOST: "0.0.0.0"+ REDIS_URL: "redis://redis:6379"+ REDIS_RATE_LIMIT_URL: "redis://redis:6379"+ PLAYWRIGHT_MICROSERVICE_URL: "http://playwright-service:3000"+ USE_DB_AUTHENTICATION: "false"+ HDX_NODE_BETA_MODE: "1"++playwrightConfig:+ PORT: "3000"++secret:+ OPENAI_API_KEY: ""+ SLACK_WEBHOOK_URL: ""+ LLAMAPARSE_API_KEY: ""+ BULL_AUTH_KEY: ""+ TEST_API_KEY: ""+ POSTHOG_API_KEY: ""+ POSTHOG_HOST: ""+ SCRAPING_BEE_API_KEY: ""+ STRIPE_PRICE_ID_STANDARD: ""+ STRIPE_PRICE_ID_SCALE: ""+ FIRE_ENGINE_BETA_URL: ""+ REDIS_PASSWORD: ""++imagePullSecrets:+ - name: docker-registry-secret