mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-07-13 12:05:48 +00:00

Files

Jianghai 0e616462a7 [Inference] add logit processor and request handler (#5166 )

* add logit processor and request handler

* add

* add

* add

* fix

* add search tokens and update func

* finish request handler

* add running list test

* fix test

* fix some bug

* add

* add

* fix bugs

* fix some bugs

* fix bug

* fix

* fix

* add copy fun

* del useless attn

* fix request status

---------

Co-authored-by: CjhHa1 <cjh18671720497outlook.com>

2024-01-11 13:39:56 +00:00

core

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

kv_cache

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

modeling/policy

[Inference] Add the logic of the inference engine (#5173 )

2024-01-11 13:39:56 +00:00

__init__.py

[Inference] First PR for rebuild colossal-infer (#5143 )

2024-01-11 13:39:29 +00:00

config.py

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

logit_processors.py

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

readme.md

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

sampler.py

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

struct.py

[Inference] add logit processor and request handler (#5166 )

2024-01-11 13:39:56 +00:00

readme.md

Colossal-Infer

Introduction

Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.

Structures

Overview

The main design will be released later on.

Roadmap

[] design of structures
[] Core components
- [] engine
- [] request handler
- [] kv cache manager
- [] modeling
- [] custom layers
- [] online server
[] supported models
- [] llama2