mirror of https://github.com/hpcaitech/ColossalAI.git synced 2026-04-25 01:03:35 +00:00

Files

Jianghai 93aeacca34 [Inference]Update inference config and fix test (#5178 )

* unify the config setting

* fix test

* fix import

* fix test

* fix

* fix

* add logger

* revise log info

---------

Co-authored-by: CjhHa1 <cjh18671720497outlook.com>

2024-01-11 13:39:29 +00:00

core

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

kv_cache

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

__init__.py

[Inference] First PR for rebuild colossal-infer (#5143 )

2024-01-11 13:39:29 +00:00

config.py

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

readme.md

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

struct.py

[Inference]Update inference config and fix test (#5178 )

2024-01-11 13:39:29 +00:00

readme.md

Colossal-Infer

Introduction

Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.

Structures

Overview

The main design will be released later on.

Roadmap

[] design of structures
[] Core components
- [] engine
- [] request handler
- [] kv cache manager
- [] modeling
- [] custom layers
- [] online server
[] supported models
- [] llama2