github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-07 05:52:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Generalize Comment on Streaming Support for LLM Implementations and add examples (#8115)

Browse Source 
The example provided demonstrates the usage of the
HuggingFaceTextGenInference implementation with streaming enabled.
This commit is contained in:
Adarsh Shirawalmath
2023-07-24 11:29:59 +05:30
committed by GitHub
parent 80b3ec5869
commit 8ea840432f
2 changed files with 31 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/snippets/modules/model_io/models/llms/how_to/streaming_llm.mdx
View File

@@ -1,5 +1,4 @@
Currently, we support streaming for the `OpenAI`, `ChatOpenAI`, and `ChatAnthropic` implementations. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using `StreamingStdOutCallbackHandler`.
Currently, we support streaming for a broad range of LLM implementations, including but not limited to `OpenAI`, `ChatOpenAI`, `ChatAnthropic`, `Hugging Face Text Generation Inference`, and `Replicate`. This feature has been expanded to accommodate most of the models. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using `StreamingStdOutCallbackHandler`.
```python
from langchain.llms import OpenAI
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler