github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-07-01 02:43:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update llama.cpp integration (#11864)

Browse Source 
<!-- 
- **Description:** removed redondant link, replaced it with Meta's LLaMA
repo, add resources for models' hardware requirements,
  - **Issue:** None,
  - **Dependencies:** None,
  - **Tag maintainer:** None,
  - **Twitter handle:** @ElliotAlladaye
 -->
This commit is contained in:
ElliotKetchup 2023-11-02 00:32:02 +01:00 committed by GitHub
parent a228f340f1
commit 58b90f30b0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 8 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
docs/docs/integrations/llms/llamacpp.ipynb
View File

@ -6,9 +6,9 @@
"source": [ "source": [
"# Llama.cpp\n", "# Llama.cpp\n",
"\n", "\n",
"[llama-cpp-python](https://github.com/abetlen/llama-cpp-python) is a Python binding for [llama.cpp](https://github.com/ggerganov/llama.cpp). \n", "[llama-cpp-python](https://github.com/abetlen/llama-cpp-python) is a Python binding for [llama.cpp](https://github.com/ggerganov/llama.cpp).\n",
"\n", "\n",
"It supports inference for [many LLMs](https://github.com/ggerganov/llama.cpp), which can be accessed on [HuggingFace](https://huggingface.co/TheBloke).\n", "It supports inference for [many LLMs](https://github.com/ggerganov/llama.cpp#description) models, which can be accessed on [HuggingFace](https://huggingface.co/TheBloke).\n",
"\n", "\n",
"This notebook goes over how to run `llama-cpp-python` within LangChain.\n", "This notebook goes over how to run `llama-cpp-python` within LangChain.\n",
"\n", "\n",
@ -54,7 +54,7 @@
"source": [ "source": [
"### Installation with OpenBLAS / cuBLAS / CLBlast\n", "### Installation with OpenBLAS / cuBLAS / CLBlast\n",
"\n", "\n",
"`lama.cpp` supports multiple BLAS backends for faster processing. Use the `FORCE_CMAKE=1` environment variable to force the use of cmake and install the pip package for the desired BLAS backend ([source](https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast)).\n", "`llama.cpp` supports multiple BLAS backends for faster processing. Use the `FORCE_CMAKE=1` environment variable to force the use of cmake and install the pip package for the desired BLAS backend ([source](https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast)).\n",
"\n", "\n",
"Example installation with cuBLAS backend:" "Example installation with cuBLAS backend:"
] ]
@ -177,7 +177,11 @@
"\n", "\n",
"You don't need an `API_TOKEN` as you will run the LLM locally.\n", "You don't need an `API_TOKEN` as you will run the LLM locally.\n",
"\n", "\n",
"It is worth understanding which models are suitable to be used on the desired machine." "It is worth understanding which models are suitable to be used on the desired machine.\n",
"\n",
"[TheBloke's](https://huggingface.co/TheBloke) Hugging Face models have a `Provided files` section that exposes the RAM required to run models of different quantisation sizes and methods (eg: [Llama2-7B-Chat-GGUF](https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF#provided-files)).\n",
"\n",
"This [github issue](https://github.com/facebookresearch/llama/issues/425) is also relevant to find the right model for your machine."
] ]
}, },
{ {