github/ColossalAI
1
0
Fork 0
mirror of https://github.com/hpcaitech/ColossalAI.git synced 2025-09-01 01:06:00 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Hotfix] Fix bugs in testing continuous batching (#5270)

Browse Source 
* fix bug

* fix bugs

* fix bugs

* fix bugs and add padding

* add funcs and fix bugs

* fix typos

* fix bugs

* add func
This commit is contained in:
Jianghai
2024-01-18 16:31:14 +08:00
committed by GitHub
parent 5ae9099f92
commit 9e2342bde2
6 changed files with 86 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
examples/inference/benchmark_llama.py
View File

@@ -95,11 +95,10 @@ def benchmark_inference(args):
if args.dtype == "fp16":
model = model.half()
elif args.dtype == "bf16":
elif args.dtype == "fp16":
model = model.to(torch.bfloat16)
# mbsz = args.mbsz
mbsz = args.batch_size
mbsz = args.mbsz
if args.mode == "caiinference":
inference_config = InferenceConfig(
dtype=args.dtype,