github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-04-29 12:25:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
c2a3021bb0
langchain/docs/docs/how_to/code_splitter.ipynb
Erick Friis c2a3021bb0
multiple: pydantic 2 compatibility, v0.3 () 
Signed-off-by: ChengZi <chen.zhang@zilliz.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
Co-authored-by: Dan O'Donovan <dan.odonovan@gmail.com>
Co-authored-by: Tom Daniel Grande <tomdgrande@gmail.com>
Co-authored-by: Grande <Tom.Daniel.Grande@statsbygg.no>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: ccurme <chester.curme@gmail.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com>
Co-authored-by: ZhangShenao <15201440436@163.com>
Co-authored-by: Friso H. Kingma <fhkingma@gmail.com>
Co-authored-by: ChengZi <chen.zhang@zilliz.com>
Co-authored-by: Nuno Campos <nuno@langchain.dev>
Co-authored-by: Morgante Pell <morgantep@google.com>
2024-09-13 14:38:45 -07:00

784 lines
22 KiB
Plaintext
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"id": "44b9976d",
"metadata": {},
"source": [
"# How to split code\n",
"\n",
"[RecursiveCharacterTextSplitter](https://python.langchain.com/api_reference/text_splitters/character/langchain_text_splitters.character.RecursiveCharacterTextSplitter.html) includes pre-built lists of separators that are useful for splitting text in a specific programming language.\n",
"\n",
"Supported languages are stored in the `langchain_text_splitters.Language` enum. They include:\n",
"\n",
"```\n",
"\"cpp\",\n",
"\"go\",\n",
"\"java\",\n",
"\"kotlin\",\n",
"\"js\",\n",
"\"ts\",\n",
"\"php\",\n",
"\"proto\",\n",
"\"python\",\n",
"\"rst\",\n",
"\"ruby\",\n",
"\"rust\",\n",
"\"scala\",\n",
"\"swift\",\n",
"\"markdown\",\n",
"\"latex\",\n",
"\"html\",\n",
"\"sol\",\n",
"\"csharp\",\n",
"\"cobol\",\n",
"\"c\",\n",
"\"lua\",\n",
"\"perl\",\n",
"\"haskell\"\n",
"```\n",
"\n",
"To view the list of separators for a given language, pass a value from this enum into\n",
"```python\n",
"RecursiveCharacterTextSplitter.get_separators_for_language`\n",
"```\n",
"\n",
"To instantiate a splitter that is tailored for a specific language, pass a value from the enum into\n",
"```python\n",
"RecursiveCharacterTextSplitter.from_language\n",
"```\n",
"\n",
"Below we demonstrate examples for the various languages."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2bb9c73f-9d00-4a19-a81f-cab2f0fd921a",
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU langchain-text-splitters"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a9e37aa1",
"metadata": {},
"outputs": [],
"source": [
"from langchain_text_splitters import (\n",
" Language,\n",
" RecursiveCharacterTextSplitter,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "082807cb-dfba-4495-af12-0441f63f30e1",
"metadata": {},
"source": [
"To view the full list of supported languages:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e21a2434",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['cpp',\n",
" 'go',\n",
" 'java',\n",
" 'kotlin',\n",
" 'js',\n",
" 'ts',\n",
" 'php',\n",
" 'proto',\n",
" 'python',\n",
" 'rst',\n",
" 'ruby',\n",
" 'rust',\n",
" 'scala',\n",
" 'swift',\n",
" 'markdown',\n",
" 'latex',\n",
" 'html',\n",
" 'sol',\n",
" 'csharp',\n",
" 'cobol',\n",
" 'c',\n",
" 'lua',\n",
" 'perl',\n",
" 'haskell']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"[e.value for e in Language]"
]
},
{
"cell_type": "markdown",
"id": "56669f16-266a-4820-a7e7-d90ade9e642f",
"metadata": {},
"source": [
"You can also see the separators used for a given language:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c92fb913",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
]
},
{
"cell_type": "markdown",
"id": "dcb8931b",
"metadata": {},
"source": [
"## Python\n",
"\n",
"Here's an example using the PythonTextSplitter:\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a58512b9",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='def hello_world():\\n print(\"Hello, World!\")'),\n",
" Document(page_content='# Call the function\\nhello_world()')]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PYTHON_CODE = \"\"\"\n",
"def hello_world():\n",
" print(\"Hello, World!\")\n",
"\n",
"# Call the function\n",
"hello_world()\n",
"\"\"\"\n",
"python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
")\n",
"python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
"python_docs"
]
},
{
"cell_type": "markdown",
"id": "354f60a5",
"metadata": {},
"source": [
"## JS\n",
"Here's an example using the JS text splitter:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "7db0d486",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='function helloWorld() {\\n console.log(\"Hello, World!\");\\n}'),\n",
" Document(page_content='// Call the function\\nhelloWorld();')]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"JS_CODE = \"\"\"\n",
"function helloWorld() {\n",
" console.log(\"Hello, World!\");\n",
"}\n",
"\n",
"// Call the function\n",
"helloWorld();\n",
"\"\"\"\n",
"\n",
"js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.JS, chunk_size=60, chunk_overlap=0\n",
")\n",
"js_docs = js_splitter.create_documents([JS_CODE])\n",
"js_docs"
]
},
{
"cell_type": "markdown",
"id": "a739f545",
"metadata": {},
"source": [
"## TS\n",
"Here's an example using the TS text splitter:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "aee738a4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='function helloWorld(): void {'),\n",
" Document(page_content='console.log(\"Hello, World!\");\\n}'),\n",
" Document(page_content='// Call the function\\nhelloWorld();')]"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"TS_CODE = \"\"\"\n",
"function helloWorld(): void {\n",
" console.log(\"Hello, World!\");\n",
"}\n",
"\n",
"// Call the function\n",
"helloWorld();\n",
"\"\"\"\n",
"\n",
"ts_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.TS, chunk_size=60, chunk_overlap=0\n",
")\n",
"ts_docs = ts_splitter.create_documents([TS_CODE])\n",
"ts_docs"
]
},
{
"cell_type": "markdown",
"id": "ee2361f8",
"metadata": {},
"source": [
"## Markdown\n",
"\n",
"Here's an example using the Markdown text splitter:\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "ac9295d3",
"metadata": {},
"outputs": [],
"source": [
"markdown_text = \"\"\"\n",
"# 🦜️🔗 LangChain\n",
"\n",
"⚡ Building applications with LLMs through composability ⚡\n",
"\n",
"## Quick Install\n",
"\n",
"# Hopefully this code block isn't split\n",
"pip install langchain\n",
"\n",
"As an open-source project in a rapidly developing field, we are extremely open to contributions.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "3a0cb17a",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='# 🦜️🔗 LangChain'),\n",
" Document(page_content='⚡ Building applications with LLMs through composability ⚡'),\n",
" Document(page_content='## Quick Install'),\n",
" Document(page_content=\"# Hopefully this code block isn't split\"),\n",
" Document(page_content='pip install langchain'),\n",
" Document(page_content='As an open-source project in a rapidly developing field, we'),\n",
" Document(page_content='are extremely open to contributions.')]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"md_docs = md_splitter.create_documents([markdown_text])\n",
"md_docs"
]
},
{
"cell_type": "markdown",
"id": "7aa306f6",
"metadata": {},
"source": [
"## Latex\n",
"\n",
"Here's an example on Latex text:\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "77d1049d",
"metadata": {},
"outputs": [],
"source": [
"latex_text = \"\"\"\n",
"\\documentclass{article}\n",
"\n",
"\\begin{document}\n",
"\n",
"\\maketitle\n",
"\n",
"\\section{Introduction}\n",
"Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
"\n",
"\\subsection{History of LLMs}\n",
"The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
"\n",
"\\subsection{Applications of LLMs}\n",
"LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
"\n",
"\\end{document}\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "4dbc47e1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='\\\\documentclass{article}\\n\\n\\x08egin{document}\\n\\n\\\\maketitle'),\n",
" Document(page_content='\\\\section{Introduction}'),\n",
" Document(page_content='Large language models (LLMs) are a type of machine learning'),\n",
" Document(page_content='model that can be trained on vast amounts of text data to'),\n",
" Document(page_content='generate human-like language. In recent years, LLMs have'),\n",
" Document(page_content='made significant advances in a variety of natural language'),\n",
" Document(page_content='processing tasks, including language translation, text'),\n",
" Document(page_content='generation, and sentiment analysis.'),\n",
" Document(page_content='\\\\subsection{History of LLMs}'),\n",
" Document(page_content='The earliest LLMs were developed in the 1980s and 1990s,'),\n",
" Document(page_content='but they were limited by the amount of data that could be'),\n",
" Document(page_content='processed and the computational power available at the'),\n",
" Document(page_content='time. In the past decade, however, advances in hardware and'),\n",
" Document(page_content='software have made it possible to train LLMs on massive'),\n",
" Document(page_content='datasets, leading to significant improvements in'),\n",
" Document(page_content='performance.'),\n",
" Document(page_content='\\\\subsection{Applications of LLMs}'),\n",
" Document(page_content='LLMs have many applications in industry, including'),\n",
" Document(page_content='chatbots, content creation, and virtual assistants. They'),\n",
" Document(page_content='can also be used in academia for research in linguistics,'),\n",
" Document(page_content='psychology, and computational linguistics.'),\n",
" Document(page_content='\\\\end{document}')]"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"latex_docs = latex_splitter.create_documents([latex_text])\n",
"latex_docs"
]
},
{
"cell_type": "markdown",
"id": "c29adadf",
"metadata": {},
"source": [
"## HTML\n",
"\n",
"Here's an example using an HTML text splitter:\n"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "0fc78794",
"metadata": {},
"outputs": [],
"source": [
"html_text = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <head>\n",
" <title>🦜️🔗 LangChain</title>\n",
" <style>\n",
" body {\n",
" font-family: Arial, sans-serif;\n",
" }\n",
" h1 {\n",
" color: darkblue;\n",
" }\n",
" </style>\n",
" </head>\n",
" <body>\n",
" <div>\n",
" <h1>🦜️🔗 LangChain</h1>\n",
" <p>⚡ Building applications with LLMs through composability ⚡</p>\n",
" </div>\n",
" <div>\n",
" As an open-source project in a rapidly developing field, we are extremely open to contributions.\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "e3e3fca1",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='<!DOCTYPE html>\\n<html>'),\n",
" Document(page_content='<head>\\n <title>🦜️🔗 LangChain</title>'),\n",
" Document(page_content='<style>\\n body {\\n font-family: Aria'),\n",
" Document(page_content='l, sans-serif;\\n }\\n h1 {'),\n",
" Document(page_content='color: darkblue;\\n }\\n </style>\\n </head'),\n",
" Document(page_content='>'),\n",
" Document(page_content='<body>'),\n",
" Document(page_content='<div>\\n <h1>🦜️🔗 LangChain</h1>'),\n",
" Document(page_content='<p>⚡ Building applications with LLMs through composability ⚡'),\n",
" Document(page_content='</p>\\n </div>'),\n",
" Document(page_content='<div>\\n As an open-source project in a rapidly dev'),\n",
" Document(page_content='eloping field, we are extremely open to contributions.'),\n",
" Document(page_content='</div>\\n </body>\\n</html>')]"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"html_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.HTML, chunk_size=60, chunk_overlap=0\n",
")\n",
"html_docs = html_splitter.create_documents([html_text])\n",
"html_docs"
]
},
{
"cell_type": "markdown",
"id": "fcaf7abf",
"metadata": {},
"source": [
"## Solidity\n",
"Here's an example using the Solidity text splitter:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "49a1df11",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='pragma solidity ^0.8.20;'),\n",
" Document(page_content='contract HelloWorld {\\n function add(uint a, uint b) pure public returns(uint) {\\n return a + b;\\n }\\n}')]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"SOL_CODE = \"\"\"\n",
"pragma solidity ^0.8.20;\n",
"contract HelloWorld {\n",
" function add(uint a, uint b) pure public returns(uint) {\n",
" return a + b;\n",
" }\n",
"}\n",
"\"\"\"\n",
"\n",
"sol_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.SOL, chunk_size=128, chunk_overlap=0\n",
")\n",
"sol_docs = sol_splitter.create_documents([SOL_CODE])\n",
"sol_docs"
]
},
{
"cell_type": "markdown",
"id": "edd0052c",
"metadata": {},
"source": [
"## C#\n",
"Here's an example using the C# text splitter:\n"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "1524ae0f",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='using System;'),\n",
" Document(page_content='class Program\\n{\\n static void Main()\\n {\\n int age = 30; // Change the age value as needed'),\n",
" Document(page_content='// Categorize the age without any console output\\n if (age < 18)\\n {\\n // Age is under 18'),\n",
" Document(page_content='}\\n else if (age >= 18 && age < 65)\\n {\\n // Age is an adult\\n }\\n else\\n {'),\n",
" Document(page_content='// Age is a senior citizen\\n }\\n }\\n}')]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"C_CODE = \"\"\"\n",
"using System;\n",
"class Program\n",
"{\n",
" static void Main()\n",
" {\n",
" int age = 30; // Change the age value as needed\n",
"\n",
" // Categorize the age without any console output\n",
" if (age < 18)\n",
" {\n",
" // Age is under 18\n",
" }\n",
" else if (age >= 18 && age < 65)\n",
" {\n",
" // Age is an adult\n",
" }\n",
" else\n",
" {\n",
" // Age is a senior citizen\n",
" }\n",
" }\n",
"}\n",
"\"\"\"\n",
"c_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.CSHARP, chunk_size=128, chunk_overlap=0\n",
")\n",
"c_docs = c_splitter.create_documents([C_CODE])\n",
"c_docs"
]
},
{
"cell_type": "markdown",
"id": "af9de667-230e-4c2a-8c5f-122a28515d97",
"metadata": {},
"source": [
"## Haskell\n",
"Here's an example using the Haskell text splitter:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "688185b5",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='main :: IO ()'),\n",
" Document(page_content='main = do\\n putStrLn \"Hello, World!\"\\n-- Some'),\n",
" Document(page_content='sample functions\\nadd :: Int -> Int -> Int\\nadd x y'),\n",
" Document(page_content='= x + y')]"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"HASKELL_CODE = \"\"\"\n",
"main :: IO ()\n",
"main = do\n",
" putStrLn \"Hello, World!\"\n",
"-- Some sample functions\n",
"add :: Int -> Int -> Int\n",
"add x y = x + y\n",
"\"\"\"\n",
"haskell_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.HASKELL, chunk_size=50, chunk_overlap=0\n",
")\n",
"haskell_docs = haskell_splitter.create_documents([HASKELL_CODE])\n",
"haskell_docs"
]
},
{
"cell_type": "markdown",
"id": "4a11f7cd-cd85-430c-b307-5b5b5f07f8db",
"metadata": {},
"source": [
"## PHP\n",
"Here's an example using the PHP text splitter:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "90c66e7e-87a5-4a81-bece-7949aabf2369",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='<?php\\nnamespace foo;'),\n",
" Document(page_content='class Hello {'),\n",
" Document(page_content='public function __construct() { }\\n}'),\n",
" Document(page_content='function hello() {\\n echo \"Hello World!\";\\n}'),\n",
" Document(page_content='interface Human {\\n public function breath();\\n}'),\n",
" Document(page_content='trait Foo { }\\nenum Color\\n{\\n case Red;'),\n",
" Document(page_content='case Blue;\\n}')]"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PHP_CODE = \"\"\"<?php\n",
"namespace foo;\n",
"class Hello {\n",
" public function __construct() { }\n",
"}\n",
"function hello() {\n",
" echo \"Hello World!\";\n",
"}\n",
"interface Human {\n",
" public function breath();\n",
"}\n",
"trait Foo { }\n",
"enum Color\n",
"{\n",
" case Red;\n",
" case Blue;\n",
"}\"\"\"\n",
"php_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.PHP, chunk_size=50, chunk_overlap=0\n",
")\n",
"php_docs = php_splitter.create_documents([PHP_CODE])\n",
"php_docs"
]
},
{
"cell_type": "markdown",
"id": "e9fa62c1",
"metadata": {},
"source": [
"## PowerShell\n",
"Here's an example using the PowerShell text splitter:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7e6893ad",
"metadata": {},
"outputs": [],
"source": [
"POWERSHELL_CODE = \"\"\"\n",
"$directoryPath = Get-Location\n",
"\n",
"$items = Get-ChildItem -Path $directoryPath\n",
"\n",
"$files = $items | Where-Object { -not $_.PSIsContainer }\n",
"\n",
"$sortedFiles = $files | Sort-Object LastWriteTime\n",
"\n",
"foreach ($file in $sortedFiles) {\n",
" Write-Output (\"Name: \" + $file.Name + \" | Last Write Time: \" + $file.LastWriteTime)\n",
"}\n",
"\"\"\"\n",
"powershell_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.POWERSHELL, chunk_size=100, chunk_overlap=0\n",
")\n",
"powershell_docs = powershell_splitter.create_documents([POWERSHELL_CODE])\n",
"powershell_docs"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.4"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink