Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
mendableai -> firecrawl
Gergő Móricz2f3bc4e7a7b1
diff --git a/.github/workflows/deploy-image-staging.yml b/.github/workflows/deploy-image-staging.ymlindex 5856d3a32..b4a4cba38 100644--- a/.github/workflows/deploy-image-staging.yml+++ b/.github/workflows/deploy-image-staging.yml@@ -25,5 +25,5 @@ jobs:- name: 'Build Inventory Image'run: |- docker build . --tag ghcr.io/mendableai/firecrawl-staging:latest- docker push ghcr.io/mendableai/firecrawl-staging:latest\ No newline at end of file+ docker build . --tag ghcr.io/firecrawl/firecrawl-staging:latest+ docker push ghcr.io/firecrawl/firecrawl-staging:latest\ No newline at end of filediff --git a/.github/workflows/deploy-image.yml b/.github/workflows/deploy-image.ymlindex 01d291245..884ba9bbe 100644--- a/.github/workflows/deploy-image.yml+++ b/.github/workflows/deploy-image.yml@@ -36,6 +36,6 @@ jobs:with:context: ./apps/apipush: true- tags: ghcr.io/mendableai/firecrawl:latest- cache-from: type=registry,ref=ghcr.io/mendableai/firecrawl:buildcache- cache-to: type=registry,ref=ghcr.io/mendableai/firecrawl:buildcache,mode=max\ No newline at end of file+ tags: ghcr.io/firecrawl/firecrawl:latest+ cache-from: type=registry,ref=ghcr.io/firecrawl/firecrawl:buildcache+ cache-to: type=registry,ref=ghcr.io/firecrawl/firecrawl:buildcache,mode=max\ No newline at end of filediff --git a/.github/workflows/deploy-playwright.yml b/.github/workflows/deploy-playwright.ymlindex 0c7bf4656..cac1b67a0 100644--- a/.github/workflows/deploy-playwright.yml+++ b/.github/workflows/deploy-playwright.yml@@ -30,5 +30,5 @@ jobs:- name: 'Build Inventory Image'run: |- docker build . --tag ghcr.io/mendableai/playwright-service:latest- docker push ghcr.io/mendableai/playwright-service:latest\ No newline at end of file+ docker build . --tag ghcr.io/firecrawl/playwright-service:latest+ docker push ghcr.io/firecrawl/playwright-service:latest\ No newline at end of filediff --git a/.github/workflows/deploy-redis.yml b/.github/workflows/deploy-redis.ymlindex e6c37897a..b1ddc29ac 100644--- a/.github/workflows/deploy-redis.yml+++ b/.github/workflows/deploy-redis.yml@@ -30,5 +30,5 @@ jobs:- name: 'Build Inventory Image'run: |- docker build . --tag ghcr.io/mendableai/firecrawl-redis:latest- docker push ghcr.io/mendableai/firecrawl-redis:latest\ No newline at end of file+ docker build . --tag ghcr.io/firecrawl/firecrawl-redis:latest+ docker push ghcr.io/firecrawl/firecrawl-redis:latest\ No newline at end of filediff --git a/.github/workflows/ghcr-clean.yml b/.github/workflows/ghcr-clean.ymlindex c80cb3b66..2e93ff864 100644--- a/.github/workflows/ghcr-clean.yml+++ b/.github/workflows/ghcr-clean.yml@@ -10,7 +10,7 @@ jobs:steps:- uses: bots-house/[email protected]with:- owner: mendableai+ owner: firecrawlname: firecrawl# NOTE: using Personal Access Tokentoken: ${{secrets.GITHUB_TOKEN}}diff --git a/.gitmodules b/.gitmodulesindex b42c5d23c..5e51b274d 100644--- a/.gitmodules+++ b/.gitmodules@@ -1,6 +1,6 @@[submodule "apps/go-sdk/firecrawl-go"]path = apps/go-sdk/firecrawl-go- url = https://github.com/mendableai/firecrawl-go+ url = https://github.com/firecrawl/firecrawl-go[submodule "apps/go-sdk/firecrawl-go-examples"]path = apps/go-sdk/firecrawl-go-examples- url = https://github.com/mendableai/firecrawl-go-examples+ url = https://github.com/firecrawl/firecrawl-go-examplesdiff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex 83f814114..290c31142 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -76,7 +76,7 @@ Now, navigate to the apps/api/ directory and run:```bashpnpm run workers-# if you are going to use the [llm-extract feature](https://github.com/mendableai/firecrawl/pull/586/), you should also export OPENAI_API_KEY=sk-______+# if you are going to use the [llm-extract feature](https://github.com/firecrawl/firecrawl/pull/586/), you should also export OPENAI_API_KEY=sk-______```This will start the workers who are responsible for processing crawl jobs.diff --git a/README.md b/README.mdindex 8fd53adc7..c3941d013 100644--- a/README.md+++ b/README.md@@ -1,19 +1,19 @@<h3 align="center"><a name="readme-top"></a><img- src="https://raw.githubusercontent.com/mendableai/firecrawl/main/img/firecrawl_logo.png"+ src="https://raw.githubusercontent.com/firecrawl/firecrawl/main/img/firecrawl_logo.png"height="200"></h3><div align="center">- <a href="https://github.com/mendableai/firecrawl/blob/main/LICENSE">- <img src="https://img.shields.io/github/license/mendableai/firecrawl" alt="License">+ <a href="https://github.com/firecrawl/firecrawl/blob/main/LICENSE">+ <img src="https://img.shields.io/github/license/firecrawl/firecrawl" alt="License"></a><a href="https://pepy.tech/project/firecrawl-py"><img src="https://static.pepy.tech/badge/firecrawl-py" alt="Downloads"></a>-<a href="https://GitHub.com/mendableai/firecrawl/graphs/contributors">- <img src="https://img.shields.io/github/contributors/mendableai/firecrawl.svg" alt="GitHub Contributors">+<a href="https://GitHub.com/firecrawl/firecrawl/graphs/contributors">+ <img src="https://img.shields.io/github/contributors/firecrawl/firecrawl.svg" alt="GitHub Contributors"></a><a href="https://firecrawl.dev"><img src="https://img.shields.io/badge/Visit-firecrawl.dev-orange" alt="Visit firecrawl.dev">@@ -45,8 +45,8 @@ _This repository is in development, and we’re still integrating custom modules_Pst. hey, you, join our stargazers :)_-<a href="https://github.com/mendableai/firecrawl">- <img src="https://img.shields.io/github/stars/mendableai/firecrawl.svg?style=social&label=Star&maxAge=2592000" alt="GitHub stars">+<a href="https://github.com/firecrawl/firecrawl">+ <img src="https://img.shields.io/github/stars/firecrawl/firecrawl.svg?style=social&label=Star&maxAge=2592000" alt="GitHub stars"></a>## How to use it?@@ -62,7 +62,7 @@ Check out the following resources to get started:- [x] **Others**: [Zapier](https://zapier.com/apps/firecrawl/integrations), [Pabbly Connect](https://www.pabbly.com/connect/integrations/firecrawl/)- [ ] Want an SDK or Integration? Let us know by opening an issue.-To run locally, refer to guide [here](https://github.com/mendableai/firecrawl/blob/main/CONTRIBUTING.md).+To run locally, refer to guide [here](https://github.com/firecrawl/firecrawl/blob/main/CONTRIBUTING.md).### API Key@@ -620,7 +620,7 @@ To deliver the best possible product, we offer a hosted version of Firecrawl aloFirecrawl Cloud is available at [firecrawl.dev](https://firecrawl.dev) and offers a range of features that are not available in the open source version:-+## Contributing@@ -631,8 +631,8 @@ _It is the sole responsibility of the end users to respect websites' policies wh## Contributors-<a href="https://github.com/mendableai/firecrawl/graphs/contributors">- <img alt="contributors" src="https://contrib.rocks/image?repo=mendableai/firecrawl"/>+<a href="https://github.com/firecrawl/firecrawl/graphs/contributors">+ <img alt="contributors" src="https://contrib.rocks/image?repo=firecrawl/firecrawl"/></a>## License Disclaimerdiff --git a/SELF_HOST.md b/SELF_HOST.mdindex 143f9b138..4074eb220 100644--- a/SELF_HOST.md+++ b/SELF_HOST.md@@ -6,7 +6,7 @@ Welcome to [Firecrawl](https://firecrawl.dev) 🔥! Here are some instructions oIf you're contributing, note that the process is similar to other open-source repos, i.e., fork Firecrawl, make changes, run tests, PR.-If you have any questions or would like help getting on board, join our Discord community [here](https://discord.gg/gSmWdAkdwd) for more information or submit an issue on Github [here](https://github.com/mendableai/firecrawl/issues/new/choose)!+If you have any questions or would like help getting on board, join our Discord community [here](https://discord.gg/gSmWdAkdwd) for more information or submit an issue on Github [here](https://github.com/firecrawl/firecrawl/issues/new/choose)!## Why?@@ -208,4 +208,4 @@ By addressing these common issues, you can ensure a smoother setup and operation## Install Firecrawl on a Kubernetes Cluster (Simple Version)-Read the [examples/kubernetes/cluster-install/README.md](https://github.com/mendableai/firecrawl/blob/main/examples/kubernetes/cluster-install/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster.+Read the [examples/kubernetes/cluster-install/README.md](https://github.com/firecrawl/firecrawl/blob/main/examples/kubernetes/cluster-install/README.md) for instructions on how to install Firecrawl on a Kubernetes Cluster.diff --git a/apps/api/sharedLibs/html-transformer/Cargo.lock b/apps/api/sharedLibs/html-transformer/Cargo.lockindex 65be5cdde..4cfb944ce 100644--- a/apps/api/sharedLibs/html-transformer/Cargo.lock+++ b/apps/api/sharedLibs/html-transformer/Cargo.lock@@ -580,7 +580,7 @@ checksum = "650eef8c711430f1a879fdd01d4745a7deea475becfb90269c06775983bbf086"[[package]]name = "nodesig"version = "1.0.0"-source = "git+https://github.com/mendableai/nodesig#b8ebc4a710810dbc36e189d9a0df87b877702b3c"+source = "git+https://github.com/firecrawl/nodesig#b8ebc4a710810dbc36e189d9a0df87b877702b3c"dependencies = ["base64","hex",diff --git a/apps/api/sharedLibs/html-transformer/Cargo.toml b/apps/api/sharedLibs/html-transformer/Cargo.tomlindex f12de603c..34d06a614 100644--- a/apps/api/sharedLibs/html-transformer/Cargo.toml+++ b/apps/api/sharedLibs/html-transformer/Cargo.toml@@ -10,7 +10,7 @@ kuchikiki = "0.8.2"serde = { version = "1.0", features = ["derive"] }serde_json = "1.0"url = "2.5.4"-nodesig = { git = "https://github.com/mendableai/nodesig" }+nodesig = { git = "https://github.com/firecrawl/nodesig" }[lib]crate-type = ["cdylib"]diff --git a/apps/js-sdk/firecrawl/package.json b/apps/js-sdk/firecrawl/package.jsonindex 7127d37f4..e6440d353 100644--- a/apps/js-sdk/firecrawl/package.json+++ b/apps/js-sdk/firecrawl/package.json@@ -21,7 +21,7 @@},"repository": {"type": "git",- "url": "git+https://github.com/mendableai/firecrawl.git"+ "url": "git+https://github.com/firecrawl/firecrawl.git"},"author": "Mendable.ai","license": "MIT",@@ -32,9 +32,9 @@"zod-to-json-schema": "^3.23.0"},"bugs": {- "url": "https://github.com/mendableai/firecrawl/issues"+ "url": "https://github.com/firecrawl/firecrawl/issues"},- "homepage": "https://github.com/mendableai/firecrawl#readme",+ "homepage": "https://github.com/firecrawl/firecrawl#readme","devDependencies": {"@jest/globals": "^30.0.5","@types/dotenv": "^8.2.0",diff --git a/apps/python-sdk/pyproject.toml b/apps/python-sdk/pyproject.tomlindex cf5f4f0ba..c9a3ac15f 100644--- a/apps/python-sdk/pyproject.toml+++ b/apps/python-sdk/pyproject.toml@@ -47,8 +47,8 @@ keywords = ["SDK", "API", "firecrawl"][project.urls]"Documentation" = "https://docs.firecrawl.dev"-"Source" = "https://github.com/mendableai/firecrawl"-"Tracker" = "https://github.com/mendableai/firecrawl/issues"+"Source" = "https://github.com/firecrawl/firecrawl"+"Tracker" = "https://github.com/firecrawl/firecrawl/issues"[tool.setuptools.packages.find]where = ["."]diff --git a/apps/python-sdk/setup.py b/apps/python-sdk/setup.pyindex 1fb31664e..4fd7d8f42 100644--- a/apps/python-sdk/setup.py+++ b/apps/python-sdk/setup.py@@ -19,7 +19,7 @@ def get_version():setup(name="firecrawl-py",version=get_version(),- url="https://github.com/mendableai/firecrawl",+ url="https://github.com/firecrawl/firecrawl",author="Mendable.ai",author_email="[email protected]",description="Python SDK for Firecrawl API",@@ -61,8 +61,8 @@ setup(keywords="SDK API firecrawl",project_urls={"Documentation": "https://docs.firecrawl.dev",- "Source": "https://github.com/mendableai/firecrawl",- "Tracker": "https://github.com/mendableai/firecrawl/issues",+ "Source": "https://github.com/firecrawl/firecrawl",+ "Tracker": "https://github.com/firecrawl/firecrawl/issues",},license="GNU General Public License v3 (GPLv3)",)diff --git a/apps/rust-sdk/Cargo.toml b/apps/rust-sdk/Cargo.tomlindex a117deb67..11637503b 100644--- a/apps/rust-sdk/Cargo.toml+++ b/apps/rust-sdk/Cargo.toml@@ -5,7 +5,7 @@ version = "1.2.1"edition = "2021"license = "MIT"homepage = "https://www.firecrawl.dev/"-repository ="https://github.com/mendableai/firecrawl"+repository ="https://github.com/firecrawl/firecrawl"description = "Rust SDK for Firecrawl API."authors = ["Gergő Móricz <[email protected]>", "sanix-darker <[email protected]>", "kkharji <[email protected]>"]diff --git a/docker-compose.yaml b/docker-compose.yamlindex e0111757e..c6714dd5a 100644--- a/docker-compose.yaml+++ b/docker-compose.yaml@@ -3,7 +3,7 @@ name: firecrawlx-common-service: &common-service# NOTE: If you don't want to build the service locally,# uncomment the build: statement and comment out the image: statement- # image: ghcr.io/mendableai/firecrawl+ # image: ghcr.io/firecrawl/firecrawlbuild: apps/apiulimits:@@ -48,7 +48,7 @@ services:playwright-service:# NOTE: If you don't want to build the service locally,# uncomment the build: statement and comment out the image: statement- # image: ghcr.io/mendableai/playwright-service:latest+ # image: ghcr.io/firecrawl/playwright-service:latestbuild: apps/playwright-service-tsenvironment:PORT: 3000diff --git a/examples/blog-articles/amazon-price-tracking/notebook.md b/examples/blog-articles/amazon-price-tracking/notebook.mdindex 59ca47a58..4e598e6a0 100644--- a/examples/blog-articles/amazon-price-tracking/notebook.md+++ b/examples/blog-articles/amazon-price-tracking/notebook.md@@ -1215,7 +1215,7 @@ Congratulations for making it to the end of this extremely long tutorial! We'veHowever, the project is far from perfect. Since we took a top-down approach to building this app, our project code is scattered across multiple files and often doesn't follow programming best practices. For this reason, I've recreated the same project in a much more sophisticated manner with production-level features. [This new version on GitHub](https://github.com/BexTuychiev/automated-price-tracking) implements proper database session management, faster operations and overall smoother user experience. Also, this version includes buttons for removing products from the database and visiting them through the app.-If you decide to stick with the basic version, you can find the full project code and notebook in [the official Firecrawl GitHub repository's example projects](https://github.com/mendableai/firecrawl/tree/main/examples/automated_price_tracking). I also recommend that you [deploy your Streamlit app to Streamlit Cloud](https://share.streamlit.io) so that you have a functional app accessible everywhere you go.+If you decide to stick with the basic version, you can find the full project code and notebook in [the official Firecrawl GitHub repository's example projects](https://github.com/firecrawl/firecrawl/tree/main/examples/automated_price_tracking). I also recommend that you [deploy your Streamlit app to Streamlit Cloud](https://share.streamlit.io) so that you have a functional app accessible everywhere you go.Here are some further improvements you might consider for the app:diff --git a/examples/blog-articles/scheduling_scrapers/notebook.ipynb b/examples/blog-articles/scheduling_scrapers/notebook.ipynbindex 188aced8c..7c7d4ca4a 100644--- a/examples/blog-articles/scheduling_scrapers/notebook.ipynb+++ b/examples/blog-articles/scheduling_scrapers/notebook.ipynb@@ -463,7 +463,7 @@" save_firecrawl_news_data()\n","```\n","\n",- "The complete scraper script is available in [our GitHub repository](https://github.com/mendableai/firecrawl/blob/main/examples/hacker_news_scraper/firecrawl_scraper.py). For reference, we also provide [a BeautifulSoup implementation of the same scraper](https://github.com/mendableai/firecrawl/blob/main/examples/hacker_news_scraper/bs4_scraper.py)."+ "The complete scraper script is available in [our GitHub repository](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/firecrawl_scraper.py). For reference, we also provide [a BeautifulSoup implementation of the same scraper](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/bs4_scraper.py)."]},{diff --git a/examples/blog-articles/scheduling_scrapers/notebook.md b/examples/blog-articles/scheduling_scrapers/notebook.mdindex 0ee1d4bbc..19cf902e0 100644--- a/examples/blog-articles/scheduling_scrapers/notebook.md+++ b/examples/blog-articles/scheduling_scrapers/notebook.md@@ -254,7 +254,7 @@ if __name__ == "__main__":save_firecrawl_news_data()```-The complete scraper script is available in [our GitHub repository](https://github.com/mendableai/firecrawl/blob/main/examples/hacker_news_scraper/firecrawl_scraper.py). For reference, we also provide [a BeautifulSoup implementation of the same scraper](https://github.com/mendableai/firecrawl/blob/main/examples/hacker_news_scraper/bs4_scraper.py).+The complete scraper script is available in [our GitHub repository](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/firecrawl_scraper.py). For reference, we also provide [a BeautifulSoup implementation of the same scraper](https://github.com/firecrawl/firecrawl/blob/main/examples/hacker_news_scraper/bs4_scraper.py).## Local Web Scraping Automation Methodsdiff --git a/examples/full_example_apps/README.md b/examples/full_example_apps/README.mdindex f631a0e20..1c1bce50b 100644--- a/examples/full_example_apps/README.md+++ b/examples/full_example_apps/README.md@@ -1 +1 @@-Full examples apps built with Firecrawl can be found at this repo: https://github.com/mendableai/firecrawl-app-examples+Full examples apps built with Firecrawl can be found at this repo: https://github.com/firecrawl/firecrawl-app-examplesdiff --git a/examples/kubernetes/cluster-install/api.yaml b/examples/kubernetes/cluster-install/api.yamlindex 1c9c897cf..91f120507 100644--- a/examples/kubernetes/cluster-install/api.yaml+++ b/examples/kubernetes/cluster-install/api.yaml@@ -17,7 +17,7 @@ spec:- name: docker-registry-secretcontainers:- name: api- image: ghcr.io/mendableai/firecrawl:latest+ image: ghcr.io/firecrawl/firecrawl:latestimagePullPolicy: Alwayscommand: [ "node" ]args: [ "--max-old-space-size=6144", "dist/src/index.js" ]diff --git a/examples/kubernetes/cluster-install/playwright-service.yaml b/examples/kubernetes/cluster-install/playwright-service.yamlindex 626f6d2c8..b40cfa835 100644--- a/examples/kubernetes/cluster-install/playwright-service.yaml+++ b/examples/kubernetes/cluster-install/playwright-service.yaml@@ -23,7 +23,7 @@ spec:- name: docker-registry-secretcontainers:- name: playwright-service- image: ghcr.io/mendableai/playwright-service:latest+ image: ghcr.io/firecrawl/playwright-service:latestimagePullPolicy: Alwaysports:- containerPort: 3000diff --git a/examples/kubernetes/cluster-install/worker.yaml b/examples/kubernetes/cluster-install/worker.yamlindex d99afc6c4..69ef79c01 100644--- a/examples/kubernetes/cluster-install/worker.yaml+++ b/examples/kubernetes/cluster-install/worker.yaml@@ -17,7 +17,7 @@ spec:terminationGracePeriodSeconds: 60containers:- name: worker- image: ghcr.io/mendableai/firecrawl:latest+ image: ghcr.io/firecrawl/firecrawl:latestimagePullPolicy: Alwayscommand: [ "node" ]args: [ "--max-old-space-size=3072", "dist/src/services/queue-worker.js" ]