ColossalAI/colossalai/inference

Colossal-Infer

Introduction

Colossal-Infer is a library for inference of LLMs and MLMs. It is built on top of Colossal AI.

Structures

Overview

The main design will be released later on.

Roadmap

• [] design of structures
• [] Core components
  • [] engine
  • [] request handler
  • [] kv cache manager
  • [] modeling
  • [] custom layers
  • [] online server
• [] supported models
  • [] llama2