LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows
Python ai-agentsdata-engineeringdocument-aigenerative-aiidpjson-extractionllmmcp-serverocrpdf-extractionprompt-engineeringstructured-output
docs: Updated README.md on LLMWhisperer setup (#80)
Chandrasekharan M3041fa41a9df
diff --git a/README.md b/README.mdindex 271aa3694..c1389f072 100644--- a/README.md+++ b/README.md@@ -47,11 +47,27 @@ Now visit [http://frontend.unstract.localhost](http://frontend.unstract.localhosThat's all. Enjoy!+## Configuring a Text Extractor++Unstract predominantly works with PDF documents and it requires a `Text Extractor` to be configured in the application which helps retrieve text from the documents. Currently supported text extractors include++- [LLMWhisperer](https://unstract-api-resource.developer.azure-api.net/) (works best)+- Unstructured Community+- Unstructured Enterprise++### Steps to use LLMWhisperer Service++[LLMWhisperer](https://unstract-api-resource.developer.azure-api.net/) is our text extraction service which provides best results with Unstract.+- Create an account in the [developer portal](https://unstract-api-resource.developer.azure-api.net/signup)+- Create a `Subscription` under [your profile](https://unstract-api-resource.developer.azure-api.net/profile) and copy the `Primary Key`+- Try the APIs from the portal by passing the copied key in the request header `unstract-key`+- This key needs to be passed in our application while creating an `LLM Whisperer Text Extractor`+## Running with docker composeSee [Docker README.md](docker/README.md).-## Running locally+## Setup Unstract for local development### Installation@@ -83,7 +99,6 @@ eval "$(pdm venv activate in-project)"pdm venv remove in-project```-### Install dependencies with PDM[PDM](https://github.com/pdm-project/pdm) is used for dependency management.@@ -205,15 +220,6 @@ We just need to override default Traefik proxy routing to allow this, that's allRun the services.-#### Generate Encryption key to be used in `backend` and `platform-service`--An encryption key is used to securely encrypt and store data, for example credentials of connectors or adapters.-We make use of [cryptography's](https://pypi.org/project/cryptography/) Fernet to perform this encryption. Use this snippet to generate a key that can be set in your respective `backend` and `platform-service` `.env` files.--```bash-ENCRYPTION_KEY=$(python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())")-```-#### Conflicting Host NamesWhen same host name environment variables are used by both the service running locally and a service@@ -232,3 +238,12 @@ locally and the Tools are in containers, we could set the value to```<host_local_ip> prompt-service```++## Generate Encryption key to be used in `backend` and `platform-service`++An encryption key is used to securely encrypt and store data, for example credentials of connectors or adapters.+We make use of [cryptography's](https://pypi.org/project/cryptography/) Fernet to perform this encryption. Use this snippet to generate a key that can be set in your respective `backend` and `platform-service` `.env` files.++```bash+ENCRYPTION_KEY=$(python -c "import secrets, base64; print(base64.urlsafe_b64encode(secrets.token_bytes(32)).decode())")+```