doc:update knowledge api document

This commit is contained in:
aries_ckt
2023-07-12 16:33:34 +08:00
parent 16d6ce8c89
commit 30adbaf4fd
12 changed files with 90 additions and 12 deletions

View File

@@ -105,7 +105,12 @@ Document type can be .txt, .pdf, .md, .doc, .ppt.
Note that the default vector model used is text2vec-large-chinese (which is a large model, so if your personal computer configuration is not enough, it is recommended to use text2vec-base-chinese). Therefore, ensure that you download the model and place it in the models directory.
- `pdf_embedding <./knowledge/pdf_embedding.html>`_: supported pdf embedding.
- `pdf_embedding <./knowledge/pdf/pdf_embedding.html>`_: supported pdf embedding.
- `markdown_embedding <./knowledge/markdown/markdown_embedding.html>`_: supported markdown embedding.
- `word_embedding <./knowledge/word/word_embedding.html>`_: supported word embedding.
- `url_embedding <./knowledge/url/url_embedding.html>`_: supported url embedding.
- `ppt_embedding <./knowledge/ppt/ppt_embedding.html>`_: supported ppt embedding.
- `string_embedding <./knowledge/string/string_embedding.html>`_: supported raw text embedding.
.. toctree::
@@ -118,4 +123,5 @@ Note that the default vector model used is text2vec-large-chinese (which is a la
./knowledge/markdown/markdown_embedding.md
./knowledge/word/word_embedding.md
./knowledge/url/url_embedding.md
./knowledge/ppt/ppt_embedding.md
./knowledge/ppt/ppt_embedding.md
./knowledge/string/string_embedding.md

View File

@@ -1,4 +1,4 @@
MarkdownEmbedding
Markdown
==================================
markdown embedding can import md text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.

View File

@@ -1,4 +1,4 @@
PDFEmbedding
PDF
==================================
pdfembedding can import PDF text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.

View File

@@ -1,4 +1,4 @@
PPTEmbedding
PPT
==================================
ppt embedding can import ppt text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.

View File

@@ -0,0 +1,41 @@
String
==================================
string embedding can import a long raw text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.
inheriting the SourceEmbedding
```
class StringEmbedding(SourceEmbedding):
"""string embedding for read string document."""
def __init__(
self,
file_path,
vector_store_config,
text_splitter: Optional[TextSplitter] = None,
):
"""Initialize raw text word path."""
super().__init__(file_path=file_path, vector_store_config=vector_store_config)
self.file_path = file_path
self.vector_store_config = vector_store_config
self.text_splitter = text_splitter or None
```
implement read() and data_process()
read() method allows you to read data and split data into chunk
```
@register
def read(self):
"""Load from String path."""
metadata = {"source": "raw text"}
return [Document(page_content=self.file_path, metadata=metadata)]
```
data_process() method allows you to pre processing your ways
```
@register
def data_process(self, documents: List[Document]):
i = 0
for d in documents:
documents[i].page_content = d.page_content.replace("\n", "")
i += 1
return documents
```

View File

@@ -1,4 +1,4 @@
URL Embedding
URL
==================================
url embedding can import PDF text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.

View File

@@ -1,4 +1,4 @@
WordEmbedding
Word
==================================
word embedding can import word doc/docx text into a vector knowledge base. The entire embedding process includes the read (loading data), data_process (data processing), and index_to_store (embedding to the vector database) methods.