Skip to content

armgit / txtai

Public Imported from GitHub
0

💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows

Update docs

davidmezzettia5d097197eda
diff --git a/docs/pipeline/data/urlretrieve.md b/docs/pipeline/data/urlretrieve.md
new file mode 100644
index 00000000..27b6db25
--- /dev/null
+++ b/docs/pipeline/data/urlretrieve.md
@@ -0,0 +1,62 @@
+# URL Retrieve
+
+![pipeline](../../images/pipeline.png#only-light)
+![pipeline](../../images/pipeline-dark.png#only-dark)
+
+The URL Retrieve pipeline retrieves content from a HTTP(s) URL.
+
+## Example
+
+The following shows a simple example using this pipeline.
+
+```python
+from txtai.pipeline import URLRetrieve
+
+# Create and run pipeline
+urlretrieve = URLRetrieve()
+urlretrieve("https://github.com/neuml/txtai")
+```
+
+## Configuration-driven example
+
+Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).
+
+### config.yml
+```yaml
+# Create pipeline using lower case class name
+urlretrieve:
+
+# Run pipeline with workflow
+workflow:
+ retrieve:
+ tasks:
+ - action: urlretrieve
+```
+
+### Run with Workflows
+
+```python
+from txtai import Application
+
+# Create and run pipeline with workflow
+app = Application("config.yml")
+list(app.workflow("urlretrieve", ["https://github.com/neuml/txtai"]))
+```
+
+### Run with API
+
+```bash
+CONFIG=config.yml uvicorn "txtai.api:app" &
+
+curl \
+ -X POST "http://localhost:8000/workflow" \
+ -H "Content-Type: application/json" \
+ -d '{"name":"retrieve", "elements":["http://github.com/neuml/txtai"]}'
+```
+
+## Methods
+
+Python documentation for the pipeline.
+
+### ::: txtai.pipeline.URLRetrieve.__init__
+### ::: txtai.pipeline.URLRetrieve.__call__
diff --git a/docs/pipeline/index.md b/docs/pipeline/index.md
index 6e146b45..779effa2 100644
--- a/docs/pipeline/index.md
+++ b/docs/pipeline/index.md
@@ -27,6 +27,7 @@ The `LLM` and `RAG` pipelines also have integrations for [llama.cpp](https://git
- [Tabular](data/tabular)
- [Text extraction](data/textractor)
- [Tokenizer](data/tokenizer)
+ - [URLRetrieve](data/urlretrieve)
- Image
- [Caption](image/caption)
- [Image Hash](image/imagehash)
diff --git a/mkdocs.yml b/mkdocs.yml
index 4a0cf52b..6af5c533 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -92,6 +92,7 @@ nav:
- Tabular: pipeline/data/tabular.md
- Textractor: pipeline/data/textractor.md
- Tokenizer: pipeline/data/tokenizer.md
+ - URL Retrieve: pipeline/data/urlretrieve.md
- Image:
- Caption: pipeline/image/caption.md
- Image Hash: pipeline/image/imagehash.md
diff --git a/src/python/txtai/app/base.py b/src/python/txtai/app/base.py
index 5f824038..8254bd59 100644
--- a/src/python/txtai/app/base.py
+++ b/src/python/txtai/app/base.py
@@ -142,7 +142,7 @@ class Application:
config["embeddings"] = None
config["similarity"] = self.pipelines["similarity"]
- elif pipeline == "textractor":
+ elif pipeline in ("textractor", "urlretrieve"):
# Default to safeopen enabled
config["safeopen"] = config.get("safeopen", True)