Zhongkai Zhao 70885d707d [hotfix] Suport extra_kwargs in ShardConfig (#5031) ...

* [refactor]: replace inference args with extra_kwargs in ShardConfig

* modify shardconfig

* polish code

* fix policy bug in llama

* fix bug in auto policy

* remove setattr in ShardConfig

2023-11-10 10:49:50 +08:00

..

__init__.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

get_tokenizer.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

infer_batch.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

io_struct.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

ray_dist_init.py

[hotfix] Suport extra_kwargs in ShardConfig (#5031)

2023-11-10 10:49:50 +08:00

ray_init_config.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

req_queue.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

sampling_params.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00

stats.py

[Inference] Dynamic Batching Inference, online and offline (#4953)

2023-10-30 10:52:19 +08:00