mirror of
https://github.com/csunny/DB-GPT.git
synced 2025-09-03 10:05:13 +00:00
doc:update knowledge api document
This commit is contained in:
@@ -105,7 +105,12 @@ Document type can be .txt, .pdf, .md, .doc, .ppt.
|
||||
|
||||
Note that the default vector model used is text2vec-large-chinese (which is a large model, so if your personal computer configuration is not enough, it is recommended to use text2vec-base-chinese). Therefore, ensure that you download the model and place it in the models directory.
|
||||
|
||||
- `pdf_embedding <./knowledge/pdf_embedding.html>`_: supported pdf embedding.
|
||||
- `pdf_embedding <./knowledge/pdf/pdf_embedding.html>`_: supported pdf embedding.
|
||||
- `markdown_embedding <./knowledge/markdown/markdown_embedding.html>`_: supported markdown embedding.
|
||||
- `word_embedding <./knowledge/word/word_embedding.html>`_: supported word embedding.
|
||||
- `url_embedding <./knowledge/url/url_embedding.html>`_: supported url embedding.
|
||||
- `ppt_embedding <./knowledge/ppt/ppt_embedding.html>`_: supported ppt embedding.
|
||||
- `string_embedding <./knowledge/string/string_embedding.html>`_: supported raw text embedding.
|
||||
|
||||
|
||||
.. toctree::
|
||||
@@ -118,4 +123,5 @@ Note that the default vector model used is text2vec-large-chinese (which is a la
|
||||
./knowledge/markdown/markdown_embedding.md
|
||||
./knowledge/word/word_embedding.md
|
||||
./knowledge/url/url_embedding.md
|
||||
./knowledge/ppt/ppt_embedding.md
|
||||
./knowledge/ppt/ppt_embedding.md
|
||||
./knowledge/string/string_embedding.md
|
@@ -1,4 +1,4 @@
|
||||
MarkdownEmbedding
|
||||
Markdown
|
||||
==================================
|
||||
markdown embedding can import md text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
|
@@ -1,4 +1,4 @@
|
||||
PDFEmbedding
|
||||
PDF
|
||||
==================================
|
||||
pdfembedding can import PDF text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
|
@@ -1,4 +1,4 @@
|
||||
PPTEmbedding
|
||||
PPT
|
||||
==================================
|
||||
ppt embedding can import ppt text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
|
41
docs/modules/knowledge/string/string_embedding.md
Normal file
41
docs/modules/knowledge/string/string_embedding.md
Normal file
@@ -0,0 +1,41 @@
|
||||
String
|
||||
==================================
|
||||
string embedding can import a long raw text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
inheriting the SourceEmbedding
|
||||
```
|
||||
class StringEmbedding(SourceEmbedding):
|
||||
"""string embedding for read string document."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
file_path,
|
||||
vector_store_config,
|
||||
text_splitter: Optional[TextSplitter] = None,
|
||||
):
|
||||
"""Initialize raw text word path."""
|
||||
super().__init__(file_path=file_path, vector_store_config=vector_store_config)
|
||||
self.file_path = file_path
|
||||
self.vector_store_config = vector_store_config
|
||||
self.text_splitter = text_splitter or None
|
||||
```
|
||||
|
||||
implement read() and data_process()
|
||||
read() method allows you to read data and split data into chunk
|
||||
```
|
||||
@register
|
||||
def read(self):
|
||||
"""Load from String path."""
|
||||
metadata = {"source": "raw text"}
|
||||
return [Document(page_content=self.file_path, metadata=metadata)]
|
||||
```
|
||||
data_process() method allows you to pre processing your ways
|
||||
```
|
||||
@register
|
||||
def data_process(self, documents: List[Document]):
|
||||
i = 0
|
||||
for d in documents:
|
||||
documents[i].page_content = d.page_content.replace("\n", "")
|
||||
i += 1
|
||||
return documents
|
||||
```
|
@@ -1,4 +1,4 @@
|
||||
URL Embedding
|
||||
URL
|
||||
==================================
|
||||
url embedding can import PDF text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
|
@@ -1,4 +1,4 @@
|
||||
WordEmbedding
|
||||
Word
|
||||
==================================
|
||||
word embedding can import word doc/docx text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
|
||||
|
||||
|
Reference in New Issue
Block a user