Taqi Jaffri b7290f01d8 Batching for hf_pipeline (#10795) ... The huggingface pipeline in langchain (used for locally hosted models) does not support batching. If you send in a batch of prompts, it just processes them serially using the base implementation of _generate: https://github.com/docugami/langchain/blob/master/libs/langchain/langchain/llms/base.py#L1004C2-L1004C29 This PR adds support for batching in this pipeline, so that GPUs can be fully saturated. I updated the accompanying notebook to show GPU batch inference. --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>		2023-09-25 18:23:11 +01:00
..
experimental	bump 301 (#11018)	2023-09-25 08:50:47 -07:00
langchain	Batching for hf_pipeline (#10795)	2023-09-25 18:23:11 +01:00