💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Update docs
davidmezzettia5d097197eda
diff --git a/docs/pipeline/data/urlretrieve.md b/docs/pipeline/data/urlretrieve.mdnew file mode 100644index 00000000..27b6db25--- /dev/null+++ b/docs/pipeline/data/urlretrieve.md@@ -0,0 +1,62 @@+# URL Retrieve+++++The URL Retrieve pipeline retrieves content from a HTTP(s) URL.++## Example++The following shows a simple example using this pipeline.++```python+from txtai.pipeline import URLRetrieve++# Create and run pipeline+urlretrieve = URLRetrieve()+urlretrieve("https://github.com/neuml/txtai")+```++## Configuration-driven example++Pipelines are run with Python or configuration. Pipelines can be instantiated in [configuration](../../../api/configuration/#pipeline) using the lower case name of the pipeline. Configuration-driven pipelines are run with [workflows](../../../workflow/#configuration-driven-example) or the [API](../../../api#local-instance).++### config.yml+```yaml+# Create pipeline using lower case class name+urlretrieve:++# Run pipeline with workflow+workflow:+ retrieve:+ tasks:+ - action: urlretrieve+```++### Run with Workflows++```python+from txtai import Application++# Create and run pipeline with workflow+app = Application("config.yml")+list(app.workflow("urlretrieve", ["https://github.com/neuml/txtai"]))+```++### Run with API++```bash+CONFIG=config.yml uvicorn "txtai.api:app" &++curl \+ -X POST "http://localhost:8000/workflow" \+ -H "Content-Type: application/json" \+ -d '{"name":"retrieve", "elements":["http://github.com/neuml/txtai"]}'+```++## Methods++Python documentation for the pipeline.++### ::: txtai.pipeline.URLRetrieve.__init__+### ::: txtai.pipeline.URLRetrieve.__call__diff --git a/docs/pipeline/index.md b/docs/pipeline/index.mdindex 6e146b45..779effa2 100644--- a/docs/pipeline/index.md+++ b/docs/pipeline/index.md@@ -27,6 +27,7 @@ The `LLM` and `RAG` pipelines also have integrations for [llama.cpp](https://git- [Tabular](data/tabular)- [Text extraction](data/textractor)- [Tokenizer](data/tokenizer)+ - [URLRetrieve](data/urlretrieve)- Image- [Caption](image/caption)- [Image Hash](image/imagehash)diff --git a/mkdocs.yml b/mkdocs.ymlindex 4a0cf52b..6af5c533 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -92,6 +92,7 @@ nav:- Tabular: pipeline/data/tabular.md- Textractor: pipeline/data/textractor.md- Tokenizer: pipeline/data/tokenizer.md+ - URL Retrieve: pipeline/data/urlretrieve.md- Image:- Caption: pipeline/image/caption.md- Image Hash: pipeline/image/imagehash.mddiff --git a/src/python/txtai/app/base.py b/src/python/txtai/app/base.pyindex 5f824038..8254bd59 100644--- a/src/python/txtai/app/base.py+++ b/src/python/txtai/app/base.py@@ -142,7 +142,7 @@ class Application:config["embeddings"] = Noneconfig["similarity"] = self.pipelines["similarity"]- elif pipeline == "textractor":+ elif pipeline in ("textractor", "urlretrieve"):# Default to safeopen enabledconfig["safeopen"] = config.get("safeopen", True)