Switch to md5 for deduplication in neo4j integrations (#18846)

Deduplicate documents using MD5 of the page_content. Also allows for
custom deduplication with graph ingestion method by providing metadata
id attribute

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
This commit is contained in:
Tomaz Bratanic
2024-03-09 22:28:55 +01:00
committed by GitHub
parent 246724faab
commit a28be31a96
3 changed files with 15 additions and 7 deletions

View File

@@ -1,7 +1,6 @@
from langchain_community.agent_toolkits import SQLDatabaseToolkit
from langchain_community.agent_toolkits import SQLDatabaseToolkit, create_sql_agent
from langchain_community.utilities.sql_database import SQLDatabase
from langchain.agents import create_sql_agent
from tests.unit_tests.llms.fake_llm import FakeLLM