Community[minor]: Add language parser for Elixir (#22742)

Hi 👋 

First off, thanks a ton for your work on this 💚 Really appreciate what
you're providing here for the community.

## Description

This PR adds a basic language parser for the
[Elixir](https://elixir-lang.org/) programming language. The parser code
is based upon the approach outlined in
https://github.com/langchain-ai/langchain/pull/13318: it's using
`tree-sitter` under the hood and aligns with all the other `tree-sitter`
based parses added that PR.

The `CHUNK_QUERY` I'm using here is probably not the most sophisticated
one, but it worked for my application. It's a starting point to provide
"core" parsing support for Elixir in LangChain. It enables people to use
the language parser out in real world applications which may then lead
to further tweaking of the queries. I consider this PR just the ground
work.

- **Dependencies:** requires `tree-sitter` and `tree-sitter-languages`
from the extended dependencies
- **Twitter handle:**`@bitcrowd`

## Checklist

- [x] **PR title**: "package: description"
- [x] **Add tests and docs**
- [x] **Lint and test**: Run `make format`, `make lint` and `make test`
from the root of the package(s) you've modified.

<!-- If no one reviews your PR within a few days, please @-mention one
of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. -->
This commit is contained in:
Max Mulatz
2024-06-10 17:56:57 +02:00
committed by GitHub
parent 28e956735c
commit 058a64c563
6 changed files with 124 additions and 0 deletions

View File

@@ -343,6 +343,30 @@ class RecursiveCharacterTextSplitter(TextSplitter):
" ",
"",
]
elif language == Language.ELIXIR:
return [
# Split along method function and module definiton
"\ndef ",
"\ndefp ",
"\ndefmodule ",
"\ndefprotocol ",
"\ndefmacro ",
"\ndefmacrop ",
# Split along control flow statements
"\nif ",
"\nunless ",
"\nwhile ",
"\ncase ",
"\ncond ",
"\nwith ",
"\nfor ",
"\ndo ",
# Split by the normal type of lines
"\n\n",
"\n",
" ",
"",
]
elif language == Language.RUST:
return [
# Split along function definitions