📘 Documentation | 🤔 Reporting Issues | ⚙️ Model Zoo

English | 简体中文

📣 News

• [2023.08.xx] We release xTuner, with multiple fine-tuned adapters.

📖 Introduction

xTuner is a toolkit for efficiently fine-tuning LLM, developed by the MMRazor and MMDeploy teams.

• Efficiency: Support LLM fine-tuning on consumer-grade GPUs. The minimum GPU memory required for 7B LLM fine-tuning is only 15GB, indicating that users can leverage the free resource, e.g., Colab, to fine-tune their custom LLM models.
• Versatile: Support various LLMs (InternLM, Llama2, Qwen, Baichuan), datasets (MOSS_003_SFT, Arxiv GenTitle, OpenOrca, Alpaca, oasst1, Chinese Medical Dialogue) and algorithms (QLoRA, LoRA), allowing users to choose the most suitable solution for their requirements.
• Compatibility: Compatible with DeepSpeed and the HuggingFace training pipeline, enabling effortless integration and utilization.

🌟 Demos

• QLoRA fine-tune for InternLM-7B
• Chat with Llama2-7B-Plugins
• Integrate xTuner into HuggingFace's pipeline

🔥 Supports

Models	SFT Datasets	Parallel Strategies	Algorithms
InternLM InternLM-Chat Llama Llama2 Llama2-Chat Qwen Qwen-Chat Baichuan-7B Baichuan-13B-Base Baichuan-13B-Chat ...	MOSS-003-SFT Arxiv GenTitle OpenOrca Alpaca en / zh oasst1 Chinese Medical Dialogue ...	(Distributed) Data Parallel DeepSpeed 🚀	QLoRA LoRA Full parameter fine-tune ...

🛠️ Quick Start

Installation

Install xTuner with pip

pip install xtuner

or from source

git clone https://github.com/InternLM/xtuner.git
cd xtuner
pip install -e .

Chat

Examples of Plugins-based Chat 🔥🔥🔥

xTuner provides the tools to chat with pretrained / fine-tuned LLMs.

• For example, we can start the chat with Llama2-7B-Plugins by

  xtuner chat hf meta-llama/Llama-2-7b-hf --adapter xtuner/Llama-2-7b-qlora-moss-003-sft --bot-name Llama2 --prompt-template moss_sft --with-plugins calculate solve search --command-stop-word "<eoc>" --answer-stop-word "<eom>" --no-streamer

For more usages, please see chat.md.

Fine-tune

xTuner supports the efficient fine-tune (e.g., QLoRA) for LLMs.

• Step 0, prepare the config. xTuner provides many ready-to-use configs and we can view all configs by

  xtuner list-cfg

  Or, if the provided configs cannot meet the requirements, we can copy the provided config to the specified directory and make modifications by

  xtuner copy-cfg ${CONFIG_NAME} ${SAVE_DIR}

• Step 1, start fine-tuning. For example, we can start the QLoRA fine-tuning of InternLM-7B with oasst1 dataset by

  # On a single GPU
  xtuner train internlm_7b_qlora_oasst1
  # On multiple GPUs
  xtuner dist_train internlm_7b_qlora_oasst1 ${GPU_NUM}

  For more usages, please see finetune.md.

Deployment

• Step 0, convert the pth adapter to HuggingFace adapter, by

  xtuner convert adapter_pth_2_hf \
      ${CONFIG} \
      ${PATH_TO_PTH_ADAPTER} \
      ${SAVE_PATH_TO_HF_ADAPTER}

  or, directly merge pth adapter to pretrained LLM, by

  xtuner convert merge_adapter \
      ${CONFIG} \
      ${PATH_TO_PTH_ADAPTER} \
      ${SAVE_PATH_TO_MERGED_LLM} \
      --max-shard-size 2GB

• Step 1, deploy fine-tuned LLM with any other framework, such as LMDeploy 🚀.

  pip install lmdeploy
  python -m lmdeploy.pytorch.chat ${NAME_OR_PATH_TO_LLM} \
      --max_new_tokens 256 \
      --temperture 0.8 \
      --top_p 0.95 \
      --seed 0

  🎯 We are woking closely with LMDeploy, to implement the deployment of plugins-based chat!

Evaluation

• We recommend using OpenCompass, a comprehensive and systematic LLM evaluation library, which currently supports 50+ datasets with about 300,000 questions.

🔜 Roadmap

🤝 Contributing

We appreciate all contributions to xTuner. Please refer to CONTRIBUTING.md for the contributing guideline.

🎖️ Acknowledgement

• Llama 2
• QLoRA
• LMDeploy

License

This project is released under the Apache License 2.0. Please also adhere to the Licenses of models and datasets being used.