Code for the ICML 2023 paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".

source code for paper "Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models"

Official code for our paper, "LoRA-Pro: Are Low-Rank Adapters Properly Optimized? "

[NeurIPS 2024] BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Models

A library for parameter-efficient and composable transfer learning for NLP with sparse fine-tunings.

Official implementation of the paper Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers

Parameter Efficient Transfer Learning with Diff Pruning

Code for the paper: "SuS-X: Training-Free Name-Only Transfer of Vision-Language Models" [ICCV'23]

Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).

[ICLR'24] "DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training" by Aochuan Chen*, Yimeng Zhang*, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zha…

LOMO: LOw-Memory Optimization

This repository contains code for the MicroAdam paper.

[NeurIPS 2023] MeZO: Fine-Tuning Language Models with Just Forward Passes. https://arxiv.org/abs/2305.17333

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining (NeurIPS 2024)

An Efficient LLM Fine-Tuning Factory Optimized for MoE PEFT

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

A family of compressed models obtained via pruning and knowledge distillation

[NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support Llama-3/3.1, Llama-2, LLaMA, BLOOM, Vicuna, Baichuan, TinyLlama, etc.

Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint?

torch-optimizer -- collection of optimizers for Pytorch

Code for NOLA, an implementation of "nola: Compressing LoRA using Linear Combination of Random Basis"

CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for task-aware parameter-efficient fine-tuning(NeurIPS 2024)