mirror of
https://github.com/hwchase17/langchain.git
synced 2025-06-27 17:08:47 +00:00
⚡ Building applications with LLMs through composability ⚡
f23c3e2444
Please see PR #27678 for context ## Overview This pull request presents a refactor of the `HTMLHeaderTextSplitter` class aimed at improving its maintainability and readability. The primary enhancements include simplifying the internal structure by consolidating multiple private helper functions into a single private method, thereby reducing complexity and making the codebase easier to understand and extend. Importantly, all existing functionalities and public interfaces remain unchanged. ## PR Goals 1. **Simplify Internal Logic**: - **Consolidation of Private Methods**: The original implementation utilized multiple private helper functions (`_header_level`, `_dom_depth`, `_get_elements`) to manage different aspects of HTML parsing and document generation. This fragmentation increased cognitive load and potential maintenance overhead. - **Streamlined Processing**: By merging these functionalities into a single private method (`_generate_documents`), the class now offers a more straightforward flow, making it easier for developers to trace and understand the processing steps. (Thanks to @eyurtsev) 2. **Enhance Readability**: - **Clearer Method Responsibilities**: With fewer private methods, each method now has a more focused responsibility. The primary logic resides within `_generate_documents`, which handles both HTML traversal and document creation in a cohesive manner. - **Reduced Redundancy**: Eliminating redundant checks and consolidating logic reduces the code's verbosity, making it more concise without sacrificing clarity. 3. **Improve Maintainability**: - **Easier Debugging and Extension**: A simplified internal structure allows for quicker identification of issues and easier implementation of future enhancements or feature additions. - **Consistent Header Management**: The new implementation ensures that headers are managed consistently within a single context, reducing the likelihood of bugs related to header scope and hierarchy. 4. **Maintain Backward Compatibility**: - **Unchanged Public Interface**: All public methods (`split_text`, `split_text_from_url`, `split_text_from_file`) and their signatures remain unchanged, ensuring that existing integrations and usage patterns are unaffected. - **Preserved Docstrings**: Comprehensive docstrings are retained, providing clear documentation for users and developers alike. ## Detailed Changes 1. **Removed Redundant Private Methods**: - **Eliminated `_header_level`, `_dom_depth`, and `_get_elements`**: These methods were merged into the `_generate_documents` method, centralizing the logic for HTML parsing and document generation. 2. **Consolidated Document Generation Logic**: - **Single Private Method `_generate_documents`**: This method now handles the entire process of parsing HTML, tracking active headers, managing document chunks, and yielding `Document` instances. This consolidation reduces the number of moving parts and simplifies the overall processing flow. 3. **Simplified Header Management**: - **Immediate Header Scope Handling**: Headers are now managed within the traversal loop of `_generate_documents`, ensuring that headers are added or removed from the active headers dictionary in real-time based on their DOM depth and hierarchy. - **Removed `chunk_dom_depth` Attribute**: The need to track chunk DOM depth separately has been eliminated, as header scopes are now directly managed within the traversal logic. 4. **Streamlined Chunk Finalization**: - **Enhanced `finalize_chunk` Function**: The chunk finalization process has been simplified to directly yield a single `Document` when needed, without maintaining an intermediate list. This change reduces unnecessary list operations and makes the logic more straightforward. 5. **Improved Variable Naming and Flow**: - **Descriptive Variable Names**: Variables such as `current_chunk` and `node_text` provide clear insights into their roles within the processing logic. - **Direct Header Removal Logic**: Headers that are out of scope are removed immediately during traversal, ensuring that the active headers dictionary remains accurate and up-to-date. 6. **Preserved Comprehensive Docstrings**: - **Unchanged Documentation**: All existing docstrings, including class-level and method-level documentation, remain intact. This ensures that users and developers continue to have access to detailed usage instructions and method explanations. ## Testing All existing test cases from `test_html_header_text_splitter.py` have been executed against the refactored code. The results confirm that: - **Functionality Remains Intact**: The splitter continues to accurately parse HTML content, respect header hierarchies, and produce the expected `Document` objects with correct metadata. - **Backward Compatibility is Maintained**: No changes were required in the test cases, and all tests pass without modifications, demonstrating that the refactor does not introduce any regressions or alter existing behaviors. This example remains fully operational and behaves as before, returning a list of `Document` objects with the expected metadata and content splits. ## Conclusion This refactor achieves a more maintainable and readable codebase by simplifying the internal structure of the `HTMLHeaderTextSplitter` class. By consolidating multiple private methods into a single, cohesive private method, the class becomes easier to understand, debug, and extend. All existing functionalities are preserved, and comprehensive tests confirm that the refactor maintains the expected behavior. These changes align with LangChain’s standards for clean, maintainable, and efficient code. --- --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com> |
||
|---|---|---|
| .devcontainer | ||
| .github | ||
| cookbook | ||
| docs | ||
| libs | ||
| scripts | ||
| .gitattributes | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .readthedocs.yaml | ||
| CITATION.cff | ||
| LICENSE | ||
| Makefile | ||
| MIGRATE.md | ||
| poetry.toml | ||
| pyproject.toml | ||
| README.md | ||
| SECURITY.md | ||
| uv.lock | ||
| yarn.lock | ||
Note
Looking for the JS/TS library? Check out LangChain.js.
LangChain is a framework for building LLM-powered applications. It helps you chain together interoperable components and third-party integrations to simplify AI application development — all while future-proofing decisions as the underlying technology evolves.
pip install -U langchain
To learn more about LangChain, check out the docs. If you’re looking for more advanced customization or agent orchestration, check out LangGraph, our framework for building controllable agent workflows.
Why use LangChain?
LangChain helps developers build applications powered by LLMs through a standard interface for models, embeddings, vector stores, and more.
Use LangChain for:
- Real-time data augmentation. Easily connect LLMs to diverse data sources and external / internal systems, drawing from LangChain’s vast library of integrations with model providers, tools, vector stores, retrievers, and more.
- Model interoperability. Swap models in and out as your engineering team experiments to find the best choice for your application’s needs. As the industry frontier evolves, adapt quickly — LangChain’s abstractions keep you moving without losing momentum.
LangChain’s ecosystem
While the LangChain framework can be used standalone, it also integrates seamlessly with any LangChain product, giving developers a full suite of tools when building LLM applications.
To improve your LLM application development, pair LangChain with:
- LangSmith - Helpful for agent evals and observability. Debug poor-performing LLM app runs, evaluate agent trajectories, gain visibility in production, and improve performance over time.
- LangGraph - Build agents that can reliably handle complex tasks with LangGraph, our low-level agent orchestration framework. LangGraph offers customizable architecture, long-term memory, and human-in-the-loop workflows — and is trusted in production by companies like LinkedIn, Uber, Klarna, and GitLab.
- LangGraph Platform - Deploy and scale agents effortlessly with a purpose-built deployment platform for long running, stateful workflows. Discover, reuse, configure, and share agents across teams — and iterate quickly with visual prototyping in LangGraph Studio.
Additional resources
- Tutorials: Simple walkthroughs with guided examples on getting started with LangChain.
- How-to Guides: Quick, actionable code snippets for topics such as tool calling, RAG use cases, and more.
- Conceptual Guides: Explanations of key concepts behind the LangChain framework.
- API Reference: Detailed reference on navigating base packages and integrations for LangChain.