Skip to content

armgit / dspy

Public Imported from GitHub
0

DSPy: The framework for programming—not prompting—language models

Python

Add local language model folder

Herumb Shandilyae155a45eaf7c
diff --git a/docs/api/local_language_model_clients/HFModel.md b/docs/api/local_language_model_clients/HFModel.md
new file mode 100644
index 000000000..162238ae5
--- /dev/null
+++ b/docs/api/local_language_model_clients/HFModel.md
@@ -0,0 +1,7 @@
+# dspy.HFModel
+
+Initialize `HFModel` within your program with the desired model to load in. Here's an example call:
+
+```python
+llama = dspy.HFModel(model = 'meta-llama/Llama-2-7b-hf')
+```
\ No newline at end of file
diff --git a/docs/api/local_language_model_clients/MLC.md b/docs/api/local_language_model_clients/MLC.md
new file mode 100644
index 000000000..6a36f374b
--- /dev/null
+++ b/docs/api/local_language_model_clients/MLC.md
@@ -0,0 +1,41 @@
+# dspy.ChatModuleClient
+
+## Prerequisites
+
+1. Install the required packages using the following commands:
+
+ ```shell
+ pip install --no-deps --pre --force-reinstall mlc-ai-nightly-cu118 mlc-chat-nightly-cu118 -f https://mlc.ai/wheels
+ pip install transformers
+ git lfs install
+ ```
+
+ Adjust the pip wheels according to your OS/platform by referring to the provided commands in [MLC packages](https://mlc.ai/package/).
+
+## Running MLC Llama-2 models
+
+1. Create a directory for prebuilt models:
+
+ ```shell
+ mkdir -p dist/prebuilt
+ ```
+
+2. Clone the necessary libraries from the repository:
+
+ ```shell
+ git clone https://github.com/mlc-ai/binary-mlc-llm-libs.git dist/prebuilt/lib
+ cd dist/prebuilt
+ ```
+
+3. Choose a Llama-2 model from [MLC LLMs](https://huggingface.co/mlc-ai) and clone the model repository:
+
+ ```shell
+ git clone https://huggingface.co/mlc-ai/mlc-chat-Llama-2-7b-chat-hf-q4f16_1
+ ```
+
+4. Initialize the `ChatModuleClient` within your program with the desired parameters. Here's an example call:
+
+ ```python
+ llama = dspy.ChatModuleClient(model='dist/prebuilt/mlc-chat-Llama-2-7b-chat-hf-q4f16_1', model_path='dist/prebuilt/lib/Llama-2-7b-chat-hf-q4f16_1-cuda.so')
+ ```
+Please refer to the [official MLC repository](https://github.com/mlc-ai/mlc-llm) for more detailed information and [documentation](https://mlc.ai/mlc-llm/docs/get_started/try_out.html).
diff --git a/docs/api/local_language_model_clients/Ollama.md b/docs/api/local_language_model_clients/Ollama.md
new file mode 100644
index 000000000..9a063e661
--- /dev/null
+++ b/docs/api/local_language_model_clients/Ollama.md
@@ -0,0 +1,45 @@
+# dspy.OllamaLocal
+
+:::note
+Adapted from documentation provided by https://github.com/insop
+:::
+
+Ollama is a good software tool that allows you to run LLMs locally, such as Mistral, Llama2, and Phi.
+The following are the instructions to install and run Ollama.
+
+### Prerequisites
+
+Install Ollama by following the instructions from this page:
+
+- https://ollama.ai
+
+Download model: `ollama pull`
+
+Download a model by running the `ollama pull` command. You can download Mistral, Llama2, and Phi.
+
+```bash
+# download mistral
+ollama pull mistral
+```
+
+Here is the list of other models you can download:
+- https://ollama.ai/library
+
+### Running Ollama model
+
+Run model: `ollama run`
+
+You can test a model by running the model with the `ollama run` command.
+
+```bash
+# run mistral
+ollama run mistral
+```
+
+### Sending requests to the server
+
+Here is the code to load a model through Ollama:
+
+```python
+lm = dspy.OllamaLocal(model='mistral')
+```
diff --git a/docs/api/local_language_model_clients/TGI.md b/docs/api/local_language_model_clients/TGI.md
new file mode 100644
index 000000000..0b8311cc8
--- /dev/null
+++ b/docs/api/local_language_model_clients/TGI.md
@@ -0,0 +1,61 @@
+# dspy.HFClientTGI
+
+## Prerequisites
+
+- Docker must be installed on your system. If you don't have Docker installed, you can get it from [here](https://docs.docker.com/get-docker/).
+
+## Setting up the Text-Generation-Inference Server
+
+1. Clone the Text-Generation-Inference repository from GitHub by executing the following command:
+
+ ```
+ git clone https://github.com/huggingface/text-generation-inference.git
+ ```
+
+2. Change into the cloned repository directory:
+
+ ```
+ cd text-generation-inference
+ ```
+
+3. Execute the Docker command under the "Get Started" section to run the server:
+
+
+ ```
+ model=meta-llama/Llama-2-7b-hf # set to the specific Hugging Face model ID you wish to use.
+ num_shard=2 # set to the number of shards you wish to use.
+ volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run
+
+ docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:0.9 --model-id $model --num-shard $num_shard
+ ```
+
+ This command will start the server and make it accessible at `http://localhost:8080`.
+
+If you want to connect to [Meta Llama 2 models](https://huggingface.co/meta-llama), make sure to use version 9.3 (or higher) of the docker image (ghcr.io/huggingface/text-generation-inference:0.9.3) and pass in your huggingface token as an environment variable.
+
+```
+ docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data -e HUGGING_FACE_HUB_TOKEN={your_token} ghcr.io/huggingface/text-generation-inference:0.9.3 --model-id $model --num-shard $num_shard
+```
+
+## Sending requests to the server
+
+After setting up the text-generation-inference server and ensuring that it displays "Connected" when it's running, you can interact with it using the `HFClientTGI`.
+
+Initialize the `HFClientTGI` within your program with the desired parameters. Here is an example call:
+
+ ```python
+ lm = dspy.HFClientTGI(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")
+ ```
+
+ Customize the `model`, `port`, and `url` according to your requirements. The `model` parameter should be set to the specific Hugging Face model ID you wish to use.
+
+
+### FAQs
+
+1. If your model doesn't require any shards, you still need to set a value for `num_shard`, but you don't need to include the parameter `--num-shard` on the command line.
+
+2. If your model runs into any "token exceeded" issues, you can set the following parameters on the command line to adjust the input length and token limit:
+ - `--max-input-length`: Set the maximum allowed input length for the text.
+ - `--max-total-tokens`: Set the maximum total tokens allowed for text generation.
+
+Please refer to the [official Text-Generation-Inference repository](https://github.com/huggingface/text-generation-inference) for more detailed information and documentation.
diff --git a/docs/api/local_language_model_clients/_category_.json b/docs/api/local_language_model_clients/_category_.json
new file mode 100644
index 000000000..8965dcf41
--- /dev/null
+++ b/docs/api/local_language_model_clients/_category_.json
@@ -0,0 +1,8 @@
+{
+ "label": "Local Language Model Clients",
+ "position": 6,
+ "link": {
+ "type": "generated-index",
+ "description": "DSPy supports various methods including `built-in wrappers`, `server integration`, and `external package integration` for model loading. This documentation provides a concise introduction on how to load in models within DSPy extending these capabilities for your specific needs."
+ }
+}
\ No newline at end of file
diff --git a/docs/api/local_language_model_clients/vLLM.md b/docs/api/local_language_model_clients/vLLM.md
new file mode 100644
index 000000000..6658addd2
--- /dev/null
+++ b/docs/api/local_language_model_clients/vLLM.md
@@ -0,0 +1,31 @@
+# dspy.HFClientVLLM
+
+### Setting up the vLLM Server
+
+Follow these steps to set up the vLLM Server:
+
+1. Build the server from source by following the instructions provided in the [Build from Source guide](https://vllm.readthedocs.io/en/latest/getting_started/installation.html#build-from-source).
+
+2. Start the server by running the following command, and specify your desired model, host, and port using the appropriate arguments. The default server address is http://localhost:8000.
+
+Example command:
+
+```bash
+ python -m vllm.entrypoints.openai.api_server --model mosaicml/mpt-7b --port 8000
+```
+
+This will launch the vLLM server.
+
+### Sending requests to the server
+
+After setting up the vLLM server and ensuring that it displays "Connected" when it's running, you can interact with it using the `HFClientVLLM`.
+
+Initialize the `HFClientVLLM` within your program with the desired parameters. Here is an example call:
+
+```python
+ lm = dspy.HFClientVLLM(model="mosaicml/mpt-7b", port=8000, url="http://localhost")
+```
+
+Customize the `model`, `port`, `url`, and `max_tokens` according to your requirements. The `model` parameter should be set to the specific Hugging Face model ID you wish to use.
+
+Please refer to the [official vLLM repository](https://github.com/vllm-project/vllm) for more detailed information and documentation.