An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT)

The open-source materials for paper "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity".

A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 🍓 and reasoning techniques.

Code for paper "Achieving Sparse Activation in Small Language Models"

Awesome list for LLM pruning.

[NeurIPS 2024] The official code of "U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers"

Sparseout: Controlling Sparsity in Deep Networks

主要记录大语言大模型（LLMs） 算法（应用）工程师相关的知识及面试题

List of papers related to neural network quantization in recent AI conferences and journals.

🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), ga…

[NeurIPS 2024] SimPO: Simple Preference Optimization with a Reference-Free Reward

LLM training in simple, raw C/CUDA

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)

Learning Sparse Neural Networks through L0 regularization

Model Quantization Benchmark

[ICML'24] The official implementation of “Rethinking Optimization and Architecture for Tiny Language Models”

EE-LLM is a framework for large-scale training and inference of early-exit (EE) large language models (LLMs).

📚A curated list of Awesome LLM/VLM Inference Papers with codes: WINT8/4, FlashAttention, PagedAttention, MLA, Parallelism etc.

ChineseNER based on BERT, with BiLSTM+CRF layer

Event-based Vision Resources. Community effort to collect knowledge on event-based vision technology (papers, workshops, datasets, code, videos, etc)

Code for the ICML 2023 paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".

A simple and effective LLM pruning approach.

Rethinking the Value of Network Pruning (Pytorch) (ICLR 2019)

[NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support Llama-3/3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan, TinyLlama, etc.

Data augmentation for NLP, presented at EMNLP 2019

🍀 Pytorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution, which is helpful to further understand papers.⭐⭐⭐

Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.

A curated list of neural network pruning resources.