mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-07-01 01:33:10 +00:00
fd6482ad8c
* [inference] support only TP (#4998) * support only tp * enable tp * add support for bloom (#5008) * [refactor] refactor gptq and smoothquant llama (#5012) * refactor gptq and smoothquant llama * fix import error * fix linear import torch-int * fix smoothquant llama import error * fix import accelerate error * fix bug * fix import smooth cuda * fix smoothcuda * [Inference Refactor] Merge chatglm2 with pp and tp (#5023) merge chatglm with pp and tp * [Refactor] remove useless inference code (#5022) * remove useless code * fix quant model * fix test import bug * mv original inference legacy * fix chatglm2 * [Refactor] refactor policy search and quant type controlling in inference (#5035) * [Refactor] refactor policy search and quant type controling in inference * [inference] update readme (#5051) * update readme * update readme * fix architecture * fix table * fix table * [inference] udpate example (#5053) * udpate example * fix run.sh * fix rebase bug * fix some errors * update readme * add some features * update interface * update readme * update benchmark * add requirements-infer --------- Co-authored-by: Bin Jia <45593998+FoolPlayer@users.noreply.github.com> Co-authored-by: Zhongkai Zhao <kanezz620@gmail.com>
28 lines
764 B
Python
28 lines
764 B
Python
import ray
|
|
import requests
|
|
|
|
|
|
@ray.remote
|
|
def send_query(text):
|
|
resp = requests.get("http://localhost:8000/?text={}".format(text))
|
|
return resp.text
|
|
|
|
|
|
test_sentences = [
|
|
"Introduce some landmarks in Beijing",
|
|
"What is the weather today",
|
|
"Coding requires practice and patience",
|
|
"Rainy days inspire cozy reading",
|
|
"Laughter is contagious and heartwarming",
|
|
"Hiking mountains builds strength and resilience",
|
|
"Family bonds grow stronger with time",
|
|
"Science unlocks mysteries of the universe",
|
|
"Music soothes the soul and ignites passion",
|
|
"Artistic expression knows no boundaries",
|
|
]
|
|
|
|
results = ray.get([send_query.remote(text) for text in test_sentences])
|
|
print("Result returned:")
|
|
for res in results:
|
|
print(res)
|