Skip to content

LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

docs: Updated tool registry documentation (#64)

Chandrasekharan M40e6148b0283
diff --git a/README.md b/README.md
index 19cf3e86e..271aa3694 100644
--- a/README.md
+++ b/README.md
@@ -205,13 +205,14 @@ We just need to override default Traefik proxy routing to allow this, that's all
Run the services.
-#### Generate Encryption key to be used in backend and Platform service
+#### Generate Encryption key to be used in `backend` and `platform-service`
- Generate Fernet Key Refer https://pypi.org/project/cryptography/
-
- `ENCRYPTION_KEY=$(python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())")`
+An encryption key is used to securely encrypt and store data, for example credentials of connectors or adapters.
+We make use of [cryptography's](https://pypi.org/project/cryptography/) Fernet to perform this encryption. Use this snippet to generate a key that can be set in your respective `backend` and `platform-service` `.env` files.
- use the above generated encryption, key in ENV's of platform and backend
+```bash
+ENCRYPTION_KEY=$(python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())")
+```
#### Conflicting Host Names
diff --git a/backend/README.md b/backend/README.md
index aee7d5613..1b2675342 100644
--- a/backend/README.md
+++ b/backend/README.md
@@ -100,6 +100,10 @@ GOOGLE_OAUTH2_KEY="<client-id>"
GOOGLE_OAUTH2_SECRET="<client-secret>"
```
+## Tool Registry
+
+Information regarding how tools are added and maintained can be found [here](/unstract/tool-registry/README.md).
+
# Archived - (EXPERIMENTAL)
diff --git a/backend/sample.env b/backend/sample.env
index 363212649..2e620a84b 100644
--- a/backend/sample.env
+++ b/backend/sample.env
@@ -102,5 +102,5 @@ X2TEXT_HOST=http://unstract-x2text-service
X2TEXT_PORT=3004
# Encryption Key
-# key must be 32 url-safe base64-encoded bytes.
+# Key must be 32 url-safe base64-encoded bytes. Check the README.md for details
ENCRYPTION_KEY="Sample-Key"
diff --git a/unstract/tool-registry/README.md b/unstract/tool-registry/README.md
index 39de42f47..72195c570 100644
--- a/unstract/tool-registry/README.md
+++ b/unstract/tool-registry/README.md
@@ -1,44 +1,54 @@
# Tool Registry
## Overview
+
The Tool Registry is a crucial component of our platform that enables the registration and management of various tools.
This document explains the structure of the Tool Registry and provides instructions on how to add new tools to it.
-
## Registry Configuration
-The Tool Registry relies on the `registry.yaml` file to maintain a comprehensive list of registered tools. These tools may encompass private tools, distinct from the publicly available **tools** that are listed in `public-tools.json`. Private tools cater to specific use cases or are intended for privileged access.
+The Tool Registry relies on a `registry.yaml` file to maintain a comprehensive list of registered tools. Tools can be made public or private
+
+### Registry
-## Registry
-- Registry.yaml is used to register new private tools apart from the publicly available unstract tools
+- `registry.yaml` helps register tools by specifying a tool image and version from a container registry.
-## Public Tools
-- Public tools are openly accessible to all organizations and are readily available in the platform's hub. These public tools are cataloged in `public_tools.json`.
+### Public Tools
-## Private Tools
-- In contrast, private tools are documented in the `private_tools.json` file. These tools are designated for restricted or specialized use and are not accessible to the general public via the cloud platform
+- Public tools are openly accessible to all organizations and are readily available in the platform's `Workflows` page. These public tools are cataloged in `public_tools.json`. They are pulled from Docker hub and hence require no authentication.
+
+### Private Tools
+
+- In contrast, private tools are documented in the `private_tools.json` file. This file is not tracked by git. These tools are meant for restricted or specialized use and are visible only to the running Unstract instance. They are generally pulled from a supported private container registry and hence require authentication.
## Registering New Tools (Private Tools)
-1. Build the docker image for the tool that needs to be registered. This can be done with the Dockerfile residing in the tool's folder.
+
+1. Build the docker image for the tool that needs to be registered. This can be done with the Dockerfile residing in the [tool's folder](/tools/).
Optionally this can be pushed to a registry of your choice.
-```
-docker build --pull --rm -f "tools/<tool_folder>/Dockerfile" -t <image_name>:<image_tag>
-```
+
+ ```bash
+ docker build --pull --rm -f "tools/<tool_folder>/Dockerfile" -t <image_name>:<image_tag>
+ ```
+
1. Refer the [sample_registry.yaml](/unstract/tool-registry/src/unstract/tool_registry/config/sample_registry.yaml) and create a new `registry.yaml` with your desired tools.
1. You can specify the tool's location using one of the following methods:
- Local Path: If the tool is stored locally on your system, you can define it like this. The default tag is `latest`
+
```yml
tools:
- local:<tool_image_name>:<tool_image_tag>
```
+
- Docker: If the tool is available as a Docker image, define it as follows. The default tag is `latest`
+
```yml
tools:
- docker:unstract/<tool_image_name>:<tool_image_tag>
```
-1. Run the [load_tools_to_json.py](/unstract/tool-registry/scripts/load_tools_to_json.py) script. This script is responsible for processing the registry.yaml file and updating the private_tools.json file.
-1. Once the script has executed, the Tool Registry will be updated with the newly registered private tools.
-1. Users of the platform will be able to access and utilize all tools listed in both the private_tools.json and public_tools.json files. This ensures that all available tools are accessible to the platform's users, whether they are public or private.
+1. Run the [load_tools_to_json.py](/unstract/tool-registry/scripts/load_tools_to_json.py) script. This script uses the `registry.yaml file` and updates the `private_tools.json` file.
+1. Once the script has executed, the **Tool Registry** will be updated with the newly registered private tools.
+1. Optionally in order to make the tool public, move the generated JSON to `public_tools.json` and commit the changes.
+1. Users of the platform will be able to access and utilize all tools listed in both the user's `private_tools.json` and `public_tools.json` files. This ensures that all available tools are accessible to the platform's users, whether they are public or private.
By following these steps, you can effectively add new private tools to the Tool Registry and make them available within the platform, enhancing the capabilities and offerings of the system.