mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-03 12:07:36 +00:00
Update Modal.com integration docs (#8014)
Hey, I'm a Modal Labs engineer and I'm making this docs update after getting a user question in [our beta Slack space](https://join.slack.com/t/modalbetatesters/shared_invite/zt-1xl9gbob8-1QDgUY7_PRPg6dQ49hqEeQ) about the Langchain integration docs. 🔗 [Modal beta-testers link to docs discussion thread](https://modalbetatesters.slack.com/archives/C031Z7DBQFL/p1689777700594819?thread_ts=1689775859.855849&cid=C031Z7DBQFL)
This commit is contained in:
@@ -6,12 +6,12 @@
|
||||
"source": [
|
||||
"# Modal\n",
|
||||
"\n",
|
||||
"The [Modal Python Library](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
|
||||
"The `Modal` itself does not provide any LLMs but only the infrastructure.\n",
|
||||
"The [Modal cloud platform](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
|
||||
"Use `modal` to run your own custom LLM models instead of depending on LLM APIs.\n",
|
||||
"\n",
|
||||
"This example goes over how to use LangChain to interact with `Modal`.\n",
|
||||
"This example goes over how to use LangChain to interact with a `modal` HTTPS [web endpoint](https://modal.com/docs/guide/webhooks).\n",
|
||||
"\n",
|
||||
"[Here](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example how to use LangChain to interact with `Modal`."
|
||||
"[_Question-answering with LangChain_](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example of how to use LangChain alonside `Modal`. In that example, Modal runs the LangChain application end-to-end and uses OpenAI as its LLM API."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -22,7 +22,7 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install modal-client"
|
||||
"!pip install modal"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -36,20 +36,14 @@
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\u001b[?25lLaunching login page in your browser window\u001b[33m...\u001b[0m\n",
|
||||
"\u001b[2KIf this is not showing up, please copy this URL into your web browser manually:\n",
|
||||
"\u001b[2Km⠙\u001b[0m Waiting for authentication in the web browser...\n",
|
||||
"\u001b]8;id=417802;https://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b\\\u001b[4;94mhttps://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b[0m\u001b]8;;\u001b\\\n",
|
||||
"\n",
|
||||
"\u001b[2K\u001b[32m⠙\u001b[0m Waiting for authentication in the web browser...\n",
|
||||
"\u001b[1A\u001b[2K^C\n",
|
||||
"\n",
|
||||
"\u001b[31mAborted.\u001b[0m\n"
|
||||
"Launching login page in your browser window...\n",
|
||||
"If this is not showing up, please copy this URL into your web browser manually:\n",
|
||||
"https://modal.com/token-flow/tf-Dzm3Y01234mqmm1234Vcu3\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# register and get a new token\n",
|
||||
"# Register an account with Modal and get a new token.\n",
|
||||
"\n",
|
||||
"!modal token new"
|
||||
]
|
||||
@@ -58,7 +52,53 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Follow [these instructions](https://modal.com/docs/guide/secrets) to deal with secrets."
|
||||
"The [`langchain.llms.modal.Modal`](https://github.com/hwchase17/langchain/blame/master/langchain/llms/modal.py) integration class requires that you deploy a Modal application with a web endpoint that complies with the following JSON interface:\n",
|
||||
"\n",
|
||||
"1. The LLM prompt is accepted as a `str` value under the key `\"prompt\"`\n",
|
||||
"2. The LLM response returned as a `str` value under the key `\"prompt\"`\n",
|
||||
"\n",
|
||||
"**Example request JSON:**\n",
|
||||
"\n",
|
||||
"```json\n",
|
||||
"{\n",
|
||||
" \"prompt\": \"Identify yourself, bot!\",\n",
|
||||
" \"extra\": \"args are allowed\",\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"**Example response JSON:**\n",
|
||||
"\n",
|
||||
"```json\n",
|
||||
"{\n",
|
||||
" \"prompt\": \"This is the LLM speaking\",\n",
|
||||
"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"An example 'dummy' Modal web endpoint function fulfilling this interface would be\n",
|
||||
"\n",
|
||||
"```python\n",
|
||||
"...\n",
|
||||
"...\n",
|
||||
"\n",
|
||||
"class Request(BaseModel):\n",
|
||||
" prompt: str\n",
|
||||
"\n",
|
||||
"@stub.function()\n",
|
||||
"@modal.web_endpoint(method=\"POST\")\n",
|
||||
"def web(request: Request):\n",
|
||||
" _ = request # ignore input\n",
|
||||
" return {\"prompt\": \"hello world\"}\n",
|
||||
"```\n",
|
||||
"\n",
|
||||
"* See Modal's [web endpoints](https://modal.com/docs/guide/webhooks#passing-arguments-to-web-endpoints) guide for the basics of setting up an endpoint that fulfils this interface.\n",
|
||||
"* See Modal's ['Run Falcon-40B with AutoGPTQ'](https://modal.com/docs/guide/ex/falcon_gptq) open-source LLM example as a starting point for your custom LLM!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Once you have a deployed Modal web endpoint, you can pass its URL into the `langchain.llms.modal.Modal` LLM class. This class can then function as a building block in your chain."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -90,7 +130,8 @@
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm = Modal(endpoint_url=\"YOUR_ENDPOINT_URL\")"
|
||||
"endpoint_url = \"https://ecorp--custom-llm-endpoint.modal.run\" # REPLACE ME with your deployed Modal web endpoint's URL\n",
|
||||
"llm = Modal(endpoint_url=endpoint_url)"
|
||||
]
|
||||
},
|
||||
{
|
||||
|
Reference in New Issue
Block a user