github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-02 11:39:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Harrison/self hosted runhouse (#1154)

Browse Source 
Co-authored-by: Donny Greenberg <dongreenberg2@gmail.com>
Co-authored-by: John Dagdelen <jdagdelen@users.noreply.github.com>
Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
Co-authored-by: Andrew White <white.d.andrew@gmail.com>
Co-authored-by: Peng Qu <82029664+pengqu123@users.noreply.github.com>
Co-authored-by: Matt Robinson <mthw.wm.robinson@gmail.com>
Co-authored-by: jeff <tangj1122@gmail.com>
Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MacBook-Pro.local>
Co-authored-by: zanderchase <zander@unfold.ag>
Co-authored-by: Charles Frye <cfrye59@gmail.com>
Co-authored-by: zanderchase <zanderchase@gmail.com>
Co-authored-by: Shahriar Tajbakhsh <sh.tajbakhsh@gmail.com>
Co-authored-by: Stefan Keselj <skeselj@princeton.edu>
Co-authored-by: Francisco Ingham <fpingham@gmail.com>
Co-authored-by: Dhruv Anand <105786647+dhruv-anand-aintech@users.noreply.github.com>
Co-authored-by: cragwolfe <cragcw@gmail.com>
Co-authored-by: Anton Troynikov <atroyn@users.noreply.github.com>
Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
Co-authored-by: Oliver Klingefjord <oliver@klingefjord.com>
Co-authored-by: blob42 <contact@blob42.xyz>
Co-authored-by: blob42 <spike@w530>
Co-authored-by: Enrico Shippole <henryshippole@gmail.com>
Co-authored-by: Ibis Prevedello <ibiscp@gmail.com>
Co-authored-by: jped <jonathanped@gmail.com>
Co-authored-by: Justin Torre <justintorre75@gmail.com>
Co-authored-by: Ivan Vendrov <ivan@anthropic.com>
Co-authored-by: Sasmitha Manathunga <70096033+mmz-001@users.noreply.github.com>
Co-authored-by: Ankush Gola <9536492+agola11@users.noreply.github.com>
Co-authored-by: Matt Robinson <mrobinson@unstructuredai.io>
Co-authored-by: Jeff Huber <jeffchuber@gmail.com>
Co-authored-by: Akshay <64036106+akshayvkt@users.noreply.github.com>
Co-authored-by: Andrew Huang <jhuang16888@gmail.com>
Co-authored-by: rogerserper <124558887+rogerserper@users.noreply.github.com>
Co-authored-by: seanaedmiston <seane999@gmail.com>
Co-authored-by: Hasegawa Yuya <52068175+Hase-U@users.noreply.github.com>
Co-authored-by: Ivan Vendrov <ivendrov@gmail.com>
Co-authored-by: Chen Wu (吴尘) <henrychenwu@cmu.edu>
Co-authored-by: Dennis Antela Martinez <dennis.antela@gmail.com>
Co-authored-by: Maxime Vidal <max.vidal@hotmail.fr>
Co-authored-by: Rishabh Raizada <110235735+rishabh-ti@users.noreply.github.com>
This commit is contained in:
Harrison Chase
2023-02-19 09:53:45 -08:00
committed by GitHub
parent af8f5c1a49
commit 9d6d8f85da
12 changed files with 1378 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

31
docs/ecosystem/runhouse.md Normal file
View File

@@ -0,0 +1,31 @@
# Runhouse
This page covers how to use the [Runhouse](https://github.com/run-house/runhouse) ecosystem within LangChain.
It is broken into three parts: installation and setup, LLMs, and Embeddings.
## Installation and Setup
- Install the Python SDK with `pip install runhouse`
- If you'd like to use on-demand cluster, check your cloud credentials with `sky check`
## Self-hosted LLMs
For a basic self-hosted LLM, you can use the `SelfHostedHuggingFaceLLM` class. For more
custom LLMs, you can use the `SelfHostedPipeline` parent class.
```python
from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
```
For a more detailed walkthrough of the Self-hosted LLMs, see [this notebook](../modules/llms/integrations/self_hosted_examples.ipynb)
## Self-hosted Embeddings
There are several ways to use self-hosted embeddings with LangChain via Runhouse.
For a basic self-hosted embedding from a Hugging Face Transformers model, you can use
the `SelfHostedEmbedding` class.
```python
from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM
```
For a more detailed walkthrough of the Self-hosted Embeddings, see [this notebook](../modules/utils/combine_docs_examples/embeddings.ipynb)
##

2
docs/modules/llms/integrations.rst
View File

@@ -27,6 +27,8 @@ The examples here are all "how-to" guides for how to integrate with various LLM
`Anthropic <./integrations/anthropic_example.html>`_: Covers how to use Anthropic models with Langchain.
`Self-Hosted Models (via Runhouse) <./integrations/self_hosted_examples.html>`_: Covers how to run models on existing or on-demand remote compute with Langchain.
.. toctree::
:maxdepth: 1

296
docs/modules/llms/integrations/self_hosted_examples.ipynb Normal file
View File

@@ -0,0 +1,296 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "9597802c",
"metadata": {},
"source": [
"# Self-Hosted Models via Runhouse\n",
"This example goes over how to use LangChain and [Runhouse](https://github.com/run-house/runhouse) to interact with models hosted on your own GPU, or on-demand GPUs on AWS, GCP, AWS, or Lambda.\n",
"\n",
"For more information, see [Runhouse](https://github.com/run-house/runhouse) or the [Runhouse docs](https://runhouse-docs.readthedocs-hosted.com/en/latest/)."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6fb585dd",
"metadata": {},
"outputs": [],
"source": [
"from langchain.llms import SelfHostedPipeline, SelfHostedHuggingFaceLLM\n",
"from langchain import PromptTemplate, LLMChain\n",
"import runhouse as rh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "06d6866e",
"metadata": {},
"outputs": [],
"source": [
"# For an on-demand A100 with GCP, Azure, or Lambda\n",
"gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\", use_spot=False)\n",
"\n",
"# For an on-demand A10G with AWS (no single A100s on AWS)\n",
"# gpu = rh.cluster(name='rh-a10x', instance_type='g5.2xlarge', provider='aws')\n",
"\n",
"# For an existing cluster\n",
"# gpu = rh.cluster(ips=['<ip of the cluster>'], \n",
"# ssh_creds={'ssh_user': '...', 'ssh_private_key':'<path_to_key>'},\n",
"# name='rh-a10x')"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "035dea0f",
"metadata": {},
"outputs": [],
"source": [
"template = \"\"\"Question: {question}\n",
"\n",
"Answer: Let's think step by step.\"\"\"\n",
"\n",
"prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3f3458d9",
"metadata": {},
"outputs": [],
"source": [
"llm = SelfHostedHuggingFaceLLM(model_id=\"gpt2\", hardware=gpu, model_reqs=[\"pip:./\", \"transformers\", \"torch\"])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a641dbd9",
"metadata": {},
"outputs": [],
"source": [
"llm_chain = LLMChain(prompt=prompt, llm=llm)"
]
},
{
"cell_type": "code",
"execution_count": 31,
"id": "6fb6fdb2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO | 2023-02-17 05:42:23,537 | Running _generate_text via gRPC\n",
"INFO | 2023-02-17 05:42:24,016 | Time to send message: 0.48 seconds\n"
]
},
{
"data": {
"text/plain": [
"\"\\n\\nLet's say we're talking sports teams who won the Super Bowl in the year Justin Beiber\""
]
},
"execution_count": 31,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
"\n",
"llm_chain.run(question)"
]
},
{
"cell_type": "markdown",
"id": "c88709cd",
"metadata": {},
"source": [
"You can also load more custom models through the SelfHostedHuggingFaceLLM interface:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "22820c5a",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"llm = SelfHostedHuggingFaceLLM(\n",
" model_id=\"google/flan-t5-small\",\n",
" task=\"text2text-generation\",\n",
" hardware=gpu,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 39,
"id": "1528e70f",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO | 2023-02-17 05:54:21,681 | Running _generate_text via gRPC\n",
"INFO | 2023-02-17 05:54:21,937 | Time to send message: 0.25 seconds\n"
]
},
{
"data": {
"text/plain": [
"'berlin'"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"llm(\"What is the capital of Germany?\")"
]
},
{
"cell_type": "markdown",
"id": "7a0c3746",
"metadata": {},
"source": [
"Using a custom load function, we can load a custom pipeline directly on the remote hardware:"
]
},
{
"cell_type": "code",
"execution_count": 34,
"id": "893eb1d3",
"metadata": {},
"outputs": [],
"source": [
"def load_pipeline():\n",
" from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline # Need to be inside the fn in notebooks\n",
" model_id = \"gpt2\"\n",
" tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
" model = AutoModelForCausalLM.from_pretrained(model_id)\n",
" pipe = pipeline(\n",
" \"text-generation\", model=model, tokenizer=tokenizer, max_new_tokens=10\n",
" )\n",
" return pipe\n",
"\n",
"def inference_fn(pipeline, prompt, stop = None):\n",
" return pipeline(prompt)[0][\"generated_text\"][len(prompt):]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "087d50dc",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"llm = SelfHostedHuggingFaceLLM(model_load_fn=load_pipeline, hardware=gpu, inference_fn=inference_fn)"
]
},
{
"cell_type": "code",
"execution_count": 36,
"id": "feb8da8e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"INFO | 2023-02-17 05:42:59,219 | Running _generate_text via gRPC\n",
"INFO | 2023-02-17 05:42:59,522 | Time to send message: 0.3 seconds\n"
]
},
{
"data": {
"text/plain": [
"'john w. bush'"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"llm(\"Who is the current US president?\")"
]
},
{
"cell_type": "markdown",
"id": "af08575f",
"metadata": {},
"source": [
"You can send your pipeline directly over the wire to your model, but this will only work for small models (<2 Gb), and will be pretty slow:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "d23023b9",
"metadata": {},
"outputs": [],
"source": [
"pipeline = load_pipeline()\n",
"llm = SelfHostedPipeline.from_pipeline(\n",
" pipeline=pipeline, hardware=gpu, model_reqs=model_reqs\n",
")"
]
},
{
"cell_type": "markdown",
"id": "fcb447a1",
"metadata": {},
"source": [
"Instead, we can also send it to the hardware's filesystem, which will be much faster."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7206b7d6",
"metadata": {},
"outputs": [],
"source": [
"rh.blob(pickle.dumps(pipeline), path=\"models/pipeline.pkl\").save().to(gpu, path=\"models\")\n",
"\n",
"llm = SelfHostedPipeline.from_pipeline(pipeline=\"models/pipeline.pkl\", hardware=gpu)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.15"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

149
docs/modules/utils/combine_docs_examples/embeddings.ipynb
View File

@@ -313,13 +313,156 @@
"query_result = embeddings.embed_query(text)"
]
},
{
"cell_type": "markdown",
"id": "eec4efda",
"metadata": {},
"source": [
"## Self Hosted Embeddings\n",
"Let's load the SelfHostedEmbeddings, SelfHostedHuggingFaceEmbeddings, and SelfHostedHuggingFaceInstructEmbeddings classes."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a961cdb5",
"id": "d338722a",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"from langchain.embeddings import (\n",
" SelfHostedEmbeddings, \n",
" SelfHostedHuggingFaceEmbeddings, \n",
" SelfHostedHuggingFaceInstructEmbeddings\n",
")\n",
"import runhouse as rh"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "146559e8",
"metadata": {},
"outputs": [],
"source": []
"source": [
"# For an on-demand A100 with GCP, Azure, or Lambda\n",
"gpu = rh.cluster(name=\"rh-a10x\", instance_type=\"A100:1\", use_spot=False)\n",
"\n",
"# For an on-demand A10G with AWS (no single A100s on AWS)\n",
"# gpu = rh.cluster(name='rh-a10x', instance_type='g5.2xlarge', provider='aws')\n",
"\n",
"# For an existing cluster\n",
"# gpu = rh.cluster(ips=['<ip of the cluster>'], \n",
"# ssh_creds={'ssh_user': '...', 'ssh_private_key':'<path_to_key>'},\n",
"# name='my-cluster')"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1230f7df",
"metadata": {},
"outputs": [],
"source": [
"embeddings = SelfHostedHuggingFaceEmbeddings(hardware=gpu)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "2684e928",
"metadata": {},
"outputs": [],
"source": [
"text = \"This is a test document.\""
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1dc5e606",
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"query_result = embeddings.embed_query(text)"
]
},
{
"cell_type": "markdown",
"id": "cef9cc54",
"metadata": {},
"source": [
"And similarly for SelfHostedHuggingFaceInstructEmbeddings:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "81a17ca3",
"metadata": {},
"outputs": [],
"source": [
"embeddings = SelfHostedHuggingFaceInstructEmbeddings(hardware=gpu)"
]
},
{
"cell_type": "markdown",
"id": "5a33d1c8",
"metadata": {},
"source": [
"Now let's load an embedding model with a custom load function:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "c4af5679",
"metadata": {},
"outputs": [],
"source": [
"def get_pipeline():\n",
" from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline # Must be inside the function in notebooks\n",
" model_id = \"facebook/bart-base\"\n",
" tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
" model = AutoModelForCausalLM.from_pretrained(model_id)\n",
" return pipeline(\"feature-extraction\", model=model, tokenizer=tokenizer)\n",
"\n",
"def inference_fn(pipeline, prompt):\n",
" # Return last hidden state of the model\n",
" if isinstance(prompt, list):\n",
" return [emb[0][-1] for emb in pipeline(prompt)] \n",
" return pipeline(prompt)[0][-1]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "8654334b",
"metadata": {},
"outputs": [],
"source": [
"embeddings = SelfHostedEmbeddings(\n",
" model_load_fn=get_pipeline, \n",
" hardware=gpu,\n",
" model_reqs=[\"./\", \"torch\", \"transformers\"],\n",
" inference_fn=inference_fn\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "fc1bfd0f",
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"query_result = embeddings.embed_query(text)"
]
}
],
"metadata": {
@@ -338,7 +481,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
"version": "3.10.9"
},
"vscode": {
"interpreter": {