yuehuayingxueluo bc1da87366 [Fix/Inference] Fix format of input prompts and input model in inference engine (#5395) ...

* Fix bugs in inference_engine

* fix bugs in engine.py

* rm  CUDA_VISIBLE_DEVICES

* add request_ids in generate

* fix bug in engine.py

* add logger.debug for BatchBucket

2024-02-23 10:51:35 +08:00

..

__init__.py

[doc] updated inference readme (#5343)

2024-02-02 14:31:10 +08:00

engine.py

[Fix/Inference] Fix format of input prompts and input model in inference engine (#5395)

2024-02-23 10:51:35 +08:00

request_handler.py

[Inference] Optimize and Refactor Inference Batching/Scheduling (#5367)

2024-02-19 17:18:20 +08:00