text-splitters[minor]: Add lua code splitting (#20421)

- **Description:** Complete the support for Lua code in
langchain.text_splitter module.
- **Dependencies:** No
- **Twitter handle:** @saberuster

If no one reviews your PR within a few days, please @-mention one of
baskaryan, efriis, eyurtsev, hwchase17.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
saberuster
2024-04-14 06:42:51 +08:00
committed by GitHub
parent 4b6b0a87b6
commit 160bcaeb93
2 changed files with 64 additions and 0 deletions

View File

@@ -571,6 +571,23 @@ class RecursiveCharacterTextSplitter(TextSplitter):
" ",
"",
]
elif language == Language.LUA:
return [
# Split along variable and table definitions
"\nlocal ",
# Split along function definitions
"\nfunction ",
# Split along control flow statements
"\nif ",
"\nfor ",
"\nwhile ",
"\nrepeat ",
# Split by the normal type of lines
"\n\n",
"\n",
" ",
"",
]
elif language == Language.HASKELL:
return [
# Split along function definitions