DSPy: The framework for programming—not prompting—language models
Python
Add local models back
Herumb Shandilya9f726ff588f5
diff --git a/docs/docs/deep-dive/language_model_clients/local_models/HFClientTGI.mdx b/docs/docs/deep-dive/language_model_clients/local_models/HFClientTGI.mdxnew file mode 100644index 000000000..2550ed3c5--- /dev/null+++ b/docs/docs/deep-dive/language_model_clients/local_models/HFClientTGI.mdx@@ -0,0 +1,90 @@+import AuthorDetails from '@site/src/components/AuthorDetails';++## [HFClient TGI](https://github.com/huggingface/text-generation-inference)++### Prerequisites - Launching TGI Server locally++Refer to the [Text Generation-Inference Server API](/api/local_language_model_clients/TGI) for setting up the TGI server locally.++```bash+#Example TGI Server Launch++model=meta-llama/Llama-2-7b-hf # set to the specific Hugging Face model ID you wish to use.+num_shard=1 # set to the number of shards you wish to use.+volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run++docker run --gpus all --shm-size 1g -p 8080:80 -v $volume:/data -e HUGGING_FACE_HUB_TOKEN={your_token} ghcr.io/huggingface/text-generation-inference:latest --model-id $model --num-shard $num_shard+```++This command will start the server and make it accessible at `http://localhost:8080`.+++### Setting up the TGI Client++The constructor initializes the `HFModel` base class to support the handling of prompting HuggingFace models. It configures the client for communicating with the hosted TGI server to generate requests. This requires the following parameters:++- `model` (_str_): ID of Hugging Face model connected to the TGI server.+- `port` (_int_ or _list_): Port for communicating to the TGI server. This can be a single port number (`8080`) or a list of TGI ports (`[8080, 8081, 8082]`) to route the requests to.+- `url` (_str_): Base URL of hosted TGI server. This will often be `"http://localhost"`.+- `http_request_kwargs` (_dict_): Dictionary of additional keyword agruments to pass to the HTTP request function to the TGI server. This is `None` by default.+- `**kwargs`: Additional keyword arguments to configure the TGI client.++Example of the TGI constructor:++```python+class HFClientTGI(HFModel):+ def __init__(self, model, port, url="http://future-hgx-1", http_request_kwargs=None, **kwargs):+```++### Under the Hood++#### `_generate(self, prompt, **kwargs) -> dict`++**Parameters:**+- `prompt` (_str_): Prompt to send to model hosted on TGI server.+- `**kwargs`: Additional keyword arguments for completion request.++**Returns:**+- `dict`: dictionary with `prompt` and list of response `choices`.++Internally, the method handles the specifics of preparing the request prompt and corresponding payload to obtain the response.++After generation, the method parses the JSON response received from the server and retrieves the output through `json_response["generated_text"]`. This is then stored in the `completions` list.++If the JSON response includes the additional `details` argument and correspondingly, the `best_of_sequences` within `details`, this indicates multiple sequences were generated. This is also usually the case when `best_of > 1` in the initialized kwargs. Each of these sequences is accessed through `x["generated_text"]` and added to the `completions` list.++Lastly, the method constructs the response dictionary with two keys: the original request `prompt` and `choices`, a list of dictionaries representing generated completions with the key `text` holding the response's generated text.+++### Using the TGI Client++```python+tgi_llama2 = dspy.HFClientTGI(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")+```++### Sending Requests via TGI Client++1) _**Recommended**_ Configure default LM using `dspy.configure`.++This allows you to define programs in DSPy and simply call modules on your input fields, having DSPy internally call the prompt on the configured LM.++```python+dspy.configure(lm=tgi_llama2)++#Example DSPy CoT QA program+qa = dspy.ChainOfThought('question -> answer')++response = qa(question="What is the capital of Paris?") #Prompted to tgi_llama2+print(response.answer)+```++2) Generate responses using the client directly.++```python+response = tgi_llama2._generate(prompt='What is the capital of Paris?')+print(response)+```++***++<AuthorDetails name="Arnav Singhvi"/>\ No newline at end of filediff --git a/docs/docs/deep-dive/language_model_clients/local_models/HFClientVLLM.mdx b/docs/docs/deep-dive/language_model_clients/local_models/HFClientVLLM.mdxnew file mode 100644index 000000000..059c2a9be--- /dev/null+++ b/docs/docs/deep-dive/language_model_clients/local_models/HFClientVLLM.mdx@@ -0,0 +1,82 @@+import AuthorDetails from '@site/src/components/AuthorDetails';++## [HFClient vLLM](https://github.com/vllm-project/vllm)++### Prerequisites - Launching vLLM Server locally++Refer to the [vLLM Server API](/api/local_language_model_clients/vLLM) for setting up the vLLM server locally.++```bash+#Example vLLM Server Launch++ python -m vllm.entrypoints.api_server --model meta-llama/Llama-2-7b-hf --port 8080+```++This command will start the server and make it accessible at `http://localhost:8080`.+++### Setting up the vLLM Client++The constructor initializes the `HFModel` base class to support the handling of prompting models, configuring the client for communicating with the hosted vLLM server to generate requests. This requires the following parameters:++- `model` (_str_): ID of model connected to the vLLM server.+- `port` (_int_): Port for communicating to the vLLM server.+- `url` (_str_): Base URL of hosted vLLM server. This will often be `"http://localhost"`.+- `**kwargs`: Additional keyword arguments to configure the vLLM client.++Example of the vLLM constructor:++```python+class HFClientVLLM(HFModel):+ def __init__(self, model, port, url="http://localhost", **kwargs):+```++### Under the Hood++#### `_generate(self, prompt, **kwargs) -> dict`++**Parameters:**+- `prompt` (_str_): Prompt to send to model hosted on vLLM server.+- `**kwargs`: Additional keyword arguments for completion request.++**Returns:**+- `dict`: dictionary with `prompt` and list of response `choices`.++Internally, the method handles the specifics of preparing the request prompt and corresponding payload to obtain the response.++After generation, the method parses the JSON response received from the server and retrieves the output through `json_response["choices"]` and stored as the `completions` list.++Lastly, the method constructs the response dictionary with two keys: the original request `prompt` and `choices`, a list of dictionaries representing generated completions with the key `text` holding the response's generated text.++### Using the vLLM Client++```python+vllm_llama2 = dspy.HFClientVLLM(model="meta-llama/Llama-2-7b-hf", port=8080, url="http://localhost")+```++### Sending Requests via vLLM Client++1) _**Recommended**_ Configure default LM using `dspy.configure`.++This allows you to define programs in DSPy and simply call modules on your input fields, having DSPy internally call the prompt on the configured LM.++```python+dspy.configure(lm=vllm_llama2)++#Example DSPy CoT QA program+qa = dspy.ChainOfThought('question -> answer')++response = qa(question="What is the capital of Paris?") #Prompted to vllm_llama2+print(response.answer)+```++2) Generate responses using the client directly.++```python+response = vllm_llama2._generate(prompt='What is the capital of Paris?')+print(response)+```++***++<AuthorDetails name="Arnav Singhvi"/>\ No newline at end of file