Open-source ChatGPT / Comprehensive / Lightweight / Easy-to-use
The code is open-source, feel free to use it, contributions are welcome! Note: The license of the model depends on the provider of the model.
- 💥Latest News
- 💫OpenLLaMA2
- 💫Features
- 📄Running Example
- 📄Result Display
- ⛏️Pull Request
- 💐References & Acknowledgements
- 🌟Sponsor Us
- 🌈Starchart
- 🏆Contributors
- 2023/9/13: Upload DPO/PPO checkpoint
- 2023/9/9: Add DPO (direct-preference-optimization)
- 2023/9/7: Add pre-commit
- 2023/9/6: Support FlashAttention2 (--flash_attn)
- 2023/8/26: Support wandb logs (--wandb)
- 2023/8/22: Support Ceval
- 2023/8/20: Add some PPO vs SFT examples
- 2023/8/18: support LLaMA2 7B PPO training on Single A100, add pre-trained SFT/RM checkpoints
- 2023/8/13: LLaMA2 7B + SFT+ RM + PPO + DeepSpeed training features
- 2023/07/30: OpenLLaMA2 project officially launched
OpenLLaMA2 aims to develop a high-performance LLaMA2 RLHF training framework (for > 30B models).
The sister project of this project is chinese-llama2 ↗, which aims to fine-tune a Chinese LLaMA2 based on OpenLLaMA2.
- [✔️] A fast LLaMA2 SFT/PPO Training Framework based on DeepSpeed.
- [✔️] Multi-nodes training scripts for Slurm.
- [✔️] Wandb log.
- [✔️] Support conda env.
- [✔️] FlashAttention2.
- [✔️] Support DPO (direct-preference-optimization).
- [WIP] Distributed RLHF based on HTTP server (for > 30B models).
- [WIP] Distributed RLHF based on Ray (for > 30B models).
- [WIP] Rejection Sampling.
- [WIP] Multiple Reward models.
- [WIP] Support QLora.
- [WIP] Better docs and examples
- [WIP] Develop the RLHF datasets ↗ for Multiple reward models.
- [WIP] Train a chinese-llama2 ↗ RLHF model.
- [TODO] Support Context Distillation.
- [TODO] Large-scale model (> 70B) support with ZeRO++ and TRT-LLM inference.
- Verified envs
You can build openllama2 from nvidia-docker(recomended) or from conda envs.
Clone the repository:
git clone https://github.com/openllmai/OpenLLaMA2.git
# Download the pre-trained SFT/RM checkpoints (Optional)
git lfs install
git clone https://huggingface.co/chuyi777/openllama2_checkpoint- Single-node training with nvidia-docker
cd examples/scripts
# install nvidia-docker (Optional)
./nvidia_docker_install.sh
# launch nvidia container
./docker_run.sh
# cd in container
cd /openllama2/examples/scripts
# build OpenLLaMA2 (i.e, pip install)
./build_openllama2.sh
# huggingface login
~/.local/bin/huggingface-cli login
# train SFT model
./train_sft_llama.sh
# train RM model
./train_rm_llama.sh
# train PPO model
./train_ppo_llama.sh- Multi-nodes training on Slurm
cd examples/scripts
# huggingface login on Slurm
pip install transformers
huggingface-cli login
# Moidfy the Slurm Account/Nodes ... in `train_llama_slurm.sh`
# For SFT, RM, and PPO training stage:
# Modify the variable `training_script` in `train_llama_slurm.sh` to
readonly training_script="train_sft_llama.sh"
readonly training_script="train_rm_llama.sh"
readonly training_script="train_ppo_llama.sh"
# set `GPUS_PER_NODE` in `train_llama_slurm.sh`
readonly GPUS_PER_NODE=8
# run multi-nodes training script
# train_llama_slurm.sh will load the training args from `training_script`
sbatch ./train_llama_slurm.sh- build openllama2 from conda envs
If you really don't want to use nvidia-docker, we also provide tutorials for building openllama2 from a conda environment. (We prefer nvidia-docker to avoid errors caused by the environment.)
# we need conda
conda create -n llama2 python=3.10
# so, we need install some package manualy: when installing torch, you may need to match the corresponding cuda version.
pip install packaging ninja
pip install torch --index-url https://download.pytorch.org/whl/cu118
# check ninjia
ninja --version
echo $?
# install flash-attn: may take some time
pip install flash-attn=2.1.1 --no-build-isolation
./build_openllama2.sh
# enjoy it!After completing the training, you can evaluate your model by using the inference script:
./inference_llama.sh { model_path }If you want to contribute code please format the code using the following command,
pip install pre-commit
pre-commit install
git add .
git commit -m "xxx"
We would like to express our gratitude to the following projects and organizations for their contributions to the field of AI and NLP:
How to Join?
- Email us at [email protected](official email) or [email protected]/[email protected](PIC). Please include the following details:
- Your name
- Your GitHub username
- Your areas of interest
- Your skills and experience related to NLP and/or AI
- You can also join us through the official GitHub OpenLLaMA2 ↗ project page. Just create an issue about your interest to contribute and we will get back to you.
What can you do?
- Join the team and participate in the development of the OpenLLaMA2 project.
- Contribute to the project by submitting pull requests.
- Help improve documentation, fix bugs, or create new features.
- Share the project and help us grow the community.
Your sponsorship can help us maintain and improve OpenLLaMA2. If you find this project useful, please consider sponsoring us. You can sponsor us on Open Collective ↗.
A big thank you to all our contributors! If you want to contribute, feel free to make a pull request or create an issue.
@misc{openllmai23,
author = {OpenLLMAI},
title = {OpenLLaMA2},
year={2023},
howpublished = {\url{https://github.com/OpenLLMAI/OpenLLaMA2}}
}
OpenLLaMA2 © 2023 OpenLLMAI. All Rights Reserved.