💡 All-in-one AI framework for semantic search, LLM orchestration and language model workflows
Python agentsaiai-agentsembeddingsinformation-retrievallanguage-modellarge-language-modelsllmnlppythonragretrieval-augmented-generationsearchsearch-enginesemantic-searchsentence-embeddingstransformerstxtaivector-databasevector-search
Storage Task

The Storage Task expands a local directory or cloud storage bucket into a list of URLs to process.
Example
The following shows a simple example using this task as part of a workflow.
from txtai.workflow import StorageTask, Workflow
workflow = Workflow([StorageTask()])
workflow(["s3://path/to/bucket", "local://local/directory"])
Configuration-driven example
This task can also be created with workflow configuration.
workflow:
name:
tasks:
- task: storage
Methods
Python documentation for the task.