github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-05 13:06:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add prompt hub for various use-cases (#9879)

Browse Source 
Use prompt hub in our use-case docs and guides.
This commit is contained in:
Lance Martin
2023-09-03 15:32:22 -07:00
committed by GitHub
parent 00a7c31ffd
commit 16a27ab244
5 changed files with 335 additions and 478 deletions
Download Patch File Download Diff File Expand all files Collapse all files

262
docs/extras/guides/local_llms.ipynb
View File

@@ -264,88 +264,19 @@
"metadata": {},
"outputs": [],
"source": [
"pip install llama-cpp-python"
"CMAKE_ARGS=\"-DLLAMA_METAL=on\" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dirclear"
]
},
{
"cell_type": "code",
"execution_count": 43,
"id": "9d5f94b5",
"execution_count": null,
"id": "a88bf0c8-e989-4bcd-bcb7-4d7757e684f2",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"objc[10142]: Class GGMLMetalClass is implemented in both /Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/libreplit-mainline-metal.dylib (0x2a0c4c208) and /Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/libllama.dylib (0x2c28bc208). One of the two will be used. Which one is undefined.\n",
"llama.cpp: loading model from /Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\n",
"llama_model_load_internal: format = ggjt v3 (latest)\n",
"llama_model_load_internal: n_vocab = 32000\n",
"llama_model_load_internal: n_ctx = 2048\n",
"llama_model_load_internal: n_embd = 5120\n",
"llama_model_load_internal: n_mult = 256\n",
"llama_model_load_internal: n_head = 40\n",
"llama_model_load_internal: n_layer = 40\n",
"llama_model_load_internal: n_rot = 128\n",
"llama_model_load_internal: freq_base = 10000.0\n",
"llama_model_load_internal: freq_scale = 1\n",
"llama_model_load_internal: ftype = 2 (mostly Q4_0)\n",
"llama_model_load_internal: n_ff = 13824\n",
"llama_model_load_internal: model size = 13B\n",
"llama_model_load_internal: ggml ctx size = 0.09 MB\n",
"llama_model_load_internal: mem required = 8953.71 MB (+ 1608.00 MB per state)\n",
"llama_new_context_with_model: kv self size = 1600.00 MB\n",
"ggml_metal_init: allocating\n",
"ggml_metal_init: using MPS\n",
"ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'\n",
"ggml_metal_init: loaded kernel_add 0x47774af60\n",
"ggml_metal_init: loaded kernel_mul 0x47774bc00\n",
"ggml_metal_init: loaded kernel_mul_row 0x47774c230\n",
"ggml_metal_init: loaded kernel_scale 0x47774c890\n",
"ggml_metal_init: loaded kernel_silu 0x47774cef0\n",
"ggml_metal_init: loaded kernel_relu 0x10e33e500\n",
"ggml_metal_init: loaded kernel_gelu 0x47774b2f0\n",
"ggml_metal_init: loaded kernel_soft_max 0x47771a580\n",
"ggml_metal_init: loaded kernel_diag_mask_inf 0x47774dab0\n",
"ggml_metal_init: loaded kernel_get_rows_f16 0x47774e110\n",
"ggml_metal_init: loaded kernel_get_rows_q4_0 0x47774e7d0\n",
"ggml_metal_init: loaded kernel_get_rows_q4_1 0x13efd7170\n",
"ggml_metal_init: loaded kernel_get_rows_q2_K 0x13efd73d0\n",
"ggml_metal_init: loaded kernel_get_rows_q3_K 0x13efd7630\n",
"ggml_metal_init: loaded kernel_get_rows_q4_K 0x13efd7890\n",
"ggml_metal_init: loaded kernel_get_rows_q5_K 0x4744c9740\n",
"ggml_metal_init: loaded kernel_get_rows_q6_K 0x4744ca6b0\n",
"ggml_metal_init: loaded kernel_rms_norm 0x4744cb250\n",
"ggml_metal_init: loaded kernel_norm 0x4744cb970\n",
"ggml_metal_init: loaded kernel_mul_mat_f16_f32 0x10e33f700\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 0x10e33fcd0\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 0x4744cc2d0\n",
"ggml_metal_init: loaded kernel_mul_mat_q2_K_f32 0x4744cc6f0\n",
"ggml_metal_init: loaded kernel_mul_mat_q3_K_f32 0x4744cd6b0\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_K_f32 0x4744cde20\n",
"ggml_metal_init: loaded kernel_mul_mat_q5_K_f32 0x10e33ff30\n",
"ggml_metal_init: loaded kernel_mul_mat_q6_K_f32 0x10e340190\n",
"ggml_metal_init: loaded kernel_rope 0x10e3403f0\n",
"ggml_metal_init: loaded kernel_alibi_f32 0x10e340de0\n",
"ggml_metal_init: loaded kernel_cpy_f32_f16 0x10e3416d0\n",
"ggml_metal_init: loaded kernel_cpy_f32_f32 0x10e342080\n",
"ggml_metal_init: loaded kernel_cpy_f16_f16 0x10e342ca0\n",
"ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
"ggml_metal_init: hasUnifiedMemory = true\n",
"ggml_metal_init: maxTransferRate = built-in GPU\n",
"ggml_metal_add_buffer: allocated 'data ' buffer, size = 6984.06 MB, ( 6986.19 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1032.00 MB, ( 8018.19 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'kv ' buffer, size = 1602.00 MB, ( 9620.19 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr0 ' buffer, size = 426.00 MB, (10046.19 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr1 ' buffer, size = 512.00 MB, (10558.19 / 21845.34)\n",
"AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | \n"
]
}
],
"outputs": [],
"source": [
"from langchain.llms import LlamaCpp\n",
"llm = LlamaCpp(\n",
" model_path=\"/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\",\n",
" model_path=\"/Users/rlm/Desktop/Code/llama.cpp/models/openorca-platypus2-13b.gguf.q4_0.bin\",\n",
" n_gpu_layers=1,\n",
" n_batch=512,\n",
" n_ctx=2048,\n",
@@ -448,87 +379,10 @@
},
{
"cell_type": "code",
"execution_count": 46,
"id": "b55a2147",
"execution_count": null,
"id": "915ecd4c-8f6b-4de3-a787-b64cb7c682b4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found model file at /Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\n",
"llama_new_context_with_model: max tensor size = 87.89 MB\n",
"llama_new_context_with_model: max tensor size = 87.89 MB\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"llama.cpp: using Metal\n",
"llama.cpp: loading model from /Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\n",
"llama_model_load_internal: format = ggjt v3 (latest)\n",
"llama_model_load_internal: n_vocab = 32001\n",
"llama_model_load_internal: n_ctx = 2048\n",
"llama_model_load_internal: n_embd = 5120\n",
"llama_model_load_internal: n_mult = 256\n",
"llama_model_load_internal: n_head = 40\n",
"llama_model_load_internal: n_layer = 40\n",
"llama_model_load_internal: n_rot = 128\n",
"llama_model_load_internal: ftype = 2 (mostly Q4_0)\n",
"llama_model_load_internal: n_ff = 13824\n",
"llama_model_load_internal: n_parts = 1\n",
"llama_model_load_internal: model size = 13B\n",
"llama_model_load_internal: ggml ctx size = 0.09 MB\n",
"llama_model_load_internal: mem required = 9031.71 MB (+ 1608.00 MB per state)\n",
"llama_new_context_with_model: kv self size = 1600.00 MB\n",
"ggml_metal_init: allocating\n",
"ggml_metal_init: using MPS\n",
"ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/gpt4all/llmodel_DO_NOT_MODIFY/build/ggml-metal.metal'\n",
"ggml_metal_init: loaded kernel_add 0x37944d850\n",
"ggml_metal_init: loaded kernel_mul 0x37944f350\n",
"ggml_metal_init: loaded kernel_mul_row 0x37944fdd0\n",
"ggml_metal_init: loaded kernel_scale 0x3794505a0\n",
"ggml_metal_init: loaded kernel_silu 0x379450800\n",
"ggml_metal_init: loaded kernel_relu 0x379450a60\n",
"ggml_metal_init: loaded kernel_gelu 0x379450cc0\n",
"ggml_metal_init: loaded kernel_soft_max 0x379450ff0\n",
"ggml_metal_init: loaded kernel_diag_mask_inf 0x379451250\n",
"ggml_metal_init: loaded kernel_get_rows_f16 0x3794514b0\n",
"ggml_metal_init: loaded kernel_get_rows_q4_0 0x379451710\n",
"ggml_metal_init: loaded kernel_get_rows_q4_1 0x379451970\n",
"ggml_metal_init: loaded kernel_get_rows_q2_k 0x379451bd0\n",
"ggml_metal_init: loaded kernel_get_rows_q3_k 0x379451e30\n",
"ggml_metal_init: loaded kernel_get_rows_q4_k 0x379452090\n",
"ggml_metal_init: loaded kernel_get_rows_q5_k 0x3794522f0\n",
"ggml_metal_init: loaded kernel_get_rows_q6_k 0x379452550\n",
"ggml_metal_init: loaded kernel_rms_norm 0x3794527b0\n",
"ggml_metal_init: loaded kernel_norm 0x379452a10\n",
"ggml_metal_init: loaded kernel_mul_mat_f16_f32 0x379452c70\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 0x379452ed0\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 0x379453130\n",
"ggml_metal_init: loaded kernel_mul_mat_q2_k_f32 0x379453390\n",
"ggml_metal_init: loaded kernel_mul_mat_q3_k_f32 0x3794535f0\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_k_f32 0x379453850\n",
"ggml_metal_init: loaded kernel_mul_mat_q5_k_f32 0x379453ab0\n",
"ggml_metal_init: loaded kernel_mul_mat_q6_k_f32 0x379453d10\n",
"ggml_metal_init: loaded kernel_rope 0x379453f70\n",
"ggml_metal_init: loaded kernel_alibi_f32 0x3794541d0\n",
"ggml_metal_init: loaded kernel_cpy_f32_f16 0x379454430\n",
"ggml_metal_init: loaded kernel_cpy_f32_f32 0x379454690\n",
"ggml_metal_init: loaded kernel_cpy_f16_f16 0x3794548f0\n",
"ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
"ggml_metal_init: hasUnifiedMemory = true\n",
"ggml_metal_init: maxTransferRate = built-in GPU\n",
"ggml_metal_add_buffer: allocated 'data ' buffer, size = 6984.06 MB, (17542.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1024.00 MB, (18566.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'kv ' buffer, size = 1602.00 MB, (20168.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr0 ' buffer, size = 512.00 MB, (20680.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr1 ' buffer, size = 512.00 MB, (21192.94 / 21845.34)\n",
"ggml_metal_free: deallocating\n"
]
}
],
"outputs": [],
"source": [
"from langchain.llms import GPT4All\n",
"llm = GPT4All(model=\"/Users/rlm/Desktop/Code/gpt4all/models/nous-hermes-13b.ggmlv3.q4_0.bin\")"
@@ -564,89 +418,21 @@
"\n",
"Some LLMs will benefit from specific prompts.\n",
"\n",
"For example, llama2 can use [special tokens](https://twitter.com/RLanceMartin/status/1681879318493003776?s=20).\n",
"For example, LLaMA will use [special tokens](https://twitter.com/RLanceMartin/status/1681879318493003776?s=20).\n",
"\n",
"We can use `ConditionalPromptSelector` to set prompt based on the model type."
]
},
{
"cell_type": "code",
"execution_count": 57,
"id": "d082b10a",
"execution_count": null,
"id": "16759b7c-7903-4269-b7b4-f83b313d8091",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"llama.cpp: loading model from /Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\n",
"llama_model_load_internal: format = ggjt v3 (latest)\n",
"llama_model_load_internal: n_vocab = 32000\n",
"llama_model_load_internal: n_ctx = 2048\n",
"llama_model_load_internal: n_embd = 5120\n",
"llama_model_load_internal: n_mult = 256\n",
"llama_model_load_internal: n_head = 40\n",
"llama_model_load_internal: n_layer = 40\n",
"llama_model_load_internal: n_rot = 128\n",
"llama_model_load_internal: freq_base = 10000.0\n",
"llama_model_load_internal: freq_scale = 1\n",
"llama_model_load_internal: ftype = 2 (mostly Q4_0)\n",
"llama_model_load_internal: n_ff = 13824\n",
"llama_model_load_internal: model size = 13B\n",
"llama_model_load_internal: ggml ctx size = 0.09 MB\n",
"llama_model_load_internal: mem required = 8953.71 MB (+ 1608.00 MB per state)\n",
"llama_new_context_with_model: kv self size = 1600.00 MB\n",
"ggml_metal_init: allocating\n",
"ggml_metal_init: using MPS\n",
"ggml_metal_init: loading '/Users/rlm/miniforge3/envs/llama/lib/python3.9/site-packages/llama_cpp/ggml-metal.metal'\n",
"ggml_metal_init: loaded kernel_add 0x4744d09d0\n",
"ggml_metal_init: loaded kernel_mul 0x3781cb3d0\n",
"ggml_metal_init: loaded kernel_mul_row 0x37813bb60\n",
"ggml_metal_init: loaded kernel_scale 0x474481080\n",
"ggml_metal_init: loaded kernel_silu 0x4744d29f0\n",
"ggml_metal_init: loaded kernel_relu 0x3781254c0\n",
"ggml_metal_init: loaded kernel_gelu 0x47447f280\n",
"ggml_metal_init: loaded kernel_soft_max 0x4744cf470\n",
"ggml_metal_init: loaded kernel_diag_mask_inf 0x4744cf6d0\n",
"ggml_metal_init: loaded kernel_get_rows_f16 0x4744cf930\n",
"ggml_metal_init: loaded kernel_get_rows_q4_0 0x4744cfb90\n",
"ggml_metal_init: loaded kernel_get_rows_q4_1 0x4744cfdf0\n",
"ggml_metal_init: loaded kernel_get_rows_q2_K 0x4744d0050\n",
"ggml_metal_init: loaded kernel_get_rows_q3_K 0x4744ce980\n",
"ggml_metal_init: loaded kernel_get_rows_q4_K 0x4744cebe0\n",
"ggml_metal_init: loaded kernel_get_rows_q5_K 0x4744cee40\n",
"ggml_metal_init: loaded kernel_get_rows_q6_K 0x4744cf0a0\n",
"ggml_metal_init: loaded kernel_rms_norm 0x474482450\n",
"ggml_metal_init: loaded kernel_norm 0x4744826b0\n",
"ggml_metal_init: loaded kernel_mul_mat_f16_f32 0x474482910\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_0_f32 0x474482b70\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_1_f32 0x474482dd0\n",
"ggml_metal_init: loaded kernel_mul_mat_q2_K_f32 0x474483030\n",
"ggml_metal_init: loaded kernel_mul_mat_q3_K_f32 0x474483290\n",
"ggml_metal_init: loaded kernel_mul_mat_q4_K_f32 0x4744834f0\n",
"ggml_metal_init: loaded kernel_mul_mat_q5_K_f32 0x474483750\n",
"ggml_metal_init: loaded kernel_mul_mat_q6_K_f32 0x4744839b0\n",
"ggml_metal_init: loaded kernel_rope 0x474483c10\n",
"ggml_metal_init: loaded kernel_alibi_f32 0x474483e70\n",
"ggml_metal_init: loaded kernel_cpy_f32_f16 0x4744840d0\n",
"ggml_metal_init: loaded kernel_cpy_f32_f32 0x474484330\n",
"ggml_metal_init: loaded kernel_cpy_f16_f16 0x474484590\n",
"ggml_metal_init: recommendedMaxWorkingSetSize = 21845.34 MB\n",
"ggml_metal_init: hasUnifiedMemory = true\n",
"ggml_metal_init: maxTransferRate = built-in GPU\n",
"ggml_metal_add_buffer: allocated 'data ' buffer, size = 6984.06 MB, ( 6986.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'eval ' buffer, size = 1032.00 MB, ( 8018.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'kv ' buffer, size = 1602.00 MB, ( 9620.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr0 ' buffer, size = 426.00 MB, (10046.94 / 21845.34)\n",
"ggml_metal_add_buffer: allocated 'scr1 ' buffer, size = 512.00 MB, (10558.94 / 21845.34)\n",
"AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | VSX = 0 | \n"
]
}
],
"outputs": [],
"source": [
"# Set our LLM\n",
"llm = LlamaCpp(\n",
" model_path=\"/Users/rlm/Desktop/Code/llama.cpp/llama-2-13b-chat.ggmlv3.q4_0.bin\",\n",
" model_path=\"/Users/rlm/Desktop/Code/llama.cpp/models/openorca-platypus2-13b.gguf.q4_0.bin\",\n",
" n_gpu_layers=1,\n",
" n_batch=512,\n",
" n_ctx=2048,\n",
@@ -661,7 +447,7 @@
"id": "66656084",
"metadata": {},
"source": [
"Set the associated prompt."
"Set the associated prompt based upon the model version."
]
},
{
@@ -759,6 +545,18 @@
"llm_chain.run({\"question\":question})"
]
},
{
"cell_type": "markdown",
"id": "6e0d37e7-f1d9-4848-bf2c-c22392ee141f",
"metadata": {},
"source": [
"We also can use the LangChain Prompt Hub to fetch and / or store prompts that are model specific.\n",
"\n",
"This will work with your [LangSmith API key](https://docs.smith.langchain.com/).\n",
"\n",
"For example, [here](https://smith.langchain.com/hub/rlm/rag-prompt-llama) is a prompt for RAG with LLaMA-specific tokens."
]
},
{
"cell_type": "markdown",
"id": "6ba66260",
@@ -770,16 +568,12 @@
"\n",
"For example, here is a guide to [RAG](docs/use_cases/question_answering/how_to/local_retrieval_qa) with local LLMs.\n",
"\n",
"In general, use cases for local model can be driven by at least two factors:\n",
"In general, use cases for local LLMs can be driven by at least two factors:\n",
"\n",
"* `Privacy`: private data (e.g., journals, etc) that a user does not want to share \n",
"* `Cost`: text preprocessing (extraction/tagging), summarization, and agent simulations are token-use-intensive tasks\n",
"\n",
"There are a few approach to support specific use-cases: \n",
"\n",
"* Fine-tuning (e.g., [gpt-llm-trainer](https://github.com/mshumer/gpt-llm-trainer), [Anyscale](https://www.anyscale.com/blog/fine-tuning-llama-2-a-comprehensive-case-study-for-tailoring-models-to-unique-applications)) \n",
"* [Function-calling](https://github.com/MeetKai/functionary/tree/main) for use-cases like extraction or tagging\n",
"\n"
"In addition, [here](https://blog.langchain.dev/using-langsmith-to-support-fine-tuning-of-open-source-llms/) is an overview on fine-tuning, which can utilize open source LLMs."
]
}
],