A low-latency & high-throughput serving engine for LLMs

FlashInfer: Kernel Library for LLM Serving

Machine Learning Engineering Open Book

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (OSDI'24)

Inference code for LLaMA models

This is a Chinese translation of the CUDA programming guide

LLM inference in C/C++

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Fast and memory-efficient exact attention

Curated collection of papers in MoE model inference

SGLang is a fast serving framework for large language models and vision language models.

Code for the paper Fine-Tuning Language Models from Human Preferences

Fast Multimodal LLM on Mobile Devices

Material for gpu-mode lectures

Efficient and easy multi-instance LLM serving

The code based on vLLM for the paper “ Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention”.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

Transformer related optimization, including BERT, GPT

High-speed Large Language Model Serving for Local Deployment

A self-learning tutorail for CUDA High Performance Programing.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficie…

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.