chore: add support for TypeScript code splitting (#11160)

- **Description:** Adds typescript language to `TextSplitter`

---------

Co-authored-by: Jacob Lee <jacoblee93@gmail.com>
This commit is contained in:
Fynn Flügge
2023-09-29 01:41:51 +02:00
committed by GitHub
parent 17fcbed92c
commit b738ccd91e
3 changed files with 85 additions and 0 deletions

View File

@@ -18,6 +18,7 @@ from langchain.text_splitter import (
'go',
'java',
'js',
'ts',
'php',
'proto',
'python',
@@ -107,6 +108,36 @@ js_docs
</CodeOutputBlock>
## TS
Here's an example using the TS text splitter:
```python
TS_CODE = """
function helloWorld(): void {
console.log("Hello, World!");
}
// Call the function
helloWorld();
"""
ts_splitter = RecursiveCharacterTextSplitter.from_language(
language=Language.TS, chunk_size=60, chunk_overlap=0
)
ts_docs = ts_splitter.create_documents([TS_CODE])
ts_docs
```
<CodeOutputBlock lang="python">
```
[Document(page_content='function helloWorld(): void {\n console.log("Hello, World!");\n}', metadata={}),
Document(page_content='// Call the function\nhelloWorld();', metadata={})]
```
</CodeOutputBlock>
## Markdown
Here's an example using the Markdown text splitter: