CUDA Python: Performance meets Productivity

Distributed Triton for Parallel Systems

Expert Parallelism Load Balancer

A Datacenter Scale Distributed Inference Serving Framework

DeeperGEMM: crazy optimized version

RAFT contains fundamental widely-used algorithms and primitives for machine learning and information retrieval. The algorithms are CUDA-accelerated and form building blocks for more easily writing …

Fastest kernels written from scratch

A fast communication-overlapping library for tensor/expert parallelism on GPUs.

CUDA Templates for Linear Algebra Subroutines

A bidirectional pipeline parallelism algorithm for computation-communication overlap in V3/R1 training.

DeepGEMM: clean and efficient FP8 GEMM kernels with fine-grained scaling

DeepEP: an efficient expert-parallel communication library

FlashMLA: Efficient MLA decoding kernels

MoBA: Mixture of Block Attention for Long-Context LLMs

CUDA Core Compute Libraries

Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3.

verl: Volcano Engine Reinforcement Learning for LLMs

CUTLASS and CuTe Examples

Universal LLM Deployment Engine with ML Compilation

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT)

[ICLR 2025] DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

CUDA 算子手撕与面试指南

The official implementation of the paper <MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression>

Tile primitives for speedy kernels

使用 CUDA C++ 实现的 llama 模型推理框架

[NeurIPS'24 Spotlight, ICLR'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an …