github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-04-26 17:53:08 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,878 Commits 62 Branches 51 Tags
bc09b95f504ffd9155d5d57eb370dd82cc641a0b
Commit Graph

2 Commits

Author SHA1 Message Date
Zhongkai Zhao
70885d707d [hotfix] Suport extra_kwargs in ShardConfig (#5031) 
* [refactor]: replace inference args with extra_kwargs in ShardConfig

* modify shardconfig

* polish code

* fix policy bug in llama

* fix bug in auto policy

* remove setattr in ShardConfig
 2023-11-10 10:49:50 +08:00
Yuanheng Zhao
573f270537 [Infer] Serving example w/ ray-serve (multiple GPU case) (#4841) 
* fix imports

* add ray-serve with Colossal-Infer tp

* trivial: send requests script

* add README

* fix worker port

* fix readme

* use app builder and autoscaling

* trivial: input args

* clean code; revise readme

* testci (skip example test)

* use auto model/tokenizer

* revert imports fix (fixed in other PRs)
 2023-10-02 17:48:38 +08:00