merge upstream readme (PaddlePaddle#766)

Co-authored-by: lyuwenyu <[email protected]>
war3et · Oct 21, 2024 · 8b3f410 · 8b3f410
1 parent a50ab8a
commit 8b3f410
Show file tree

Hide file tree

Showing 9 changed files with 442 additions and 147 deletions.
diff --git a/README.md b/README.md
diff --git a/README_EN.md b/README_EN.md
diff --git a/applications/README.md b/applications/README.md
@@ -15,8 +15,11 @@
   <a href=#快速开始> 快速开始 </a>
 </h4>
 
+
+
 **PaddleMIX**应用示例基于paddlemix、ppdiffusers和paddlenlp开发，**简单易用**且**功能强大**。聚合业界**优质预训练模型**并提供**开箱即用**的开发体验，覆盖跨模态和多场景的模型库搭配，可满足开发者**灵活定制**的需求。
 
+<img src="https://github.com/user-attachments/assets/4c695140-bf4c-46db-bbb5-5dd8197be947" align="center" />
 
 ## 快速开始
 
@@ -70,6 +73,7 @@ Appflow提供丰富的开箱即用工具集，覆盖跨模态多场景应用，
 
 
 
+
 ### 跨模态多场景应用
 | 应用名称                           | 调用模型                         | 静态图推理    |
 | :--------------------------------- | -------------------------------- | ----------|

diff --git a/applications/README_en.md b/applications/README_en.md
@@ -17,6 +17,8 @@
 
 **PaddleMIX** application example is developed based on paddlemix, ppdiffusers, and Paddlenlp，which is **simple** and **easy** to use  and **powerful**. Aggregating industry high-quality pre trained models and providing out of the box development experience, covering cross modal and multi scenario model library matching, can meet the needs of developers flexible customization .
 
+<img src="https://github.com/user-attachments/assets/4c695140-bf4c-46db-bbb5-5dd8197be947" align="center" />
+
 
 ## Quick Start
 Please confirm if it has been installed first [PaddleMIX](../README_EN.md/#installation) and [ppdiffusers](../README_EN.md/#installation)

diff --git a/build_env.sh b/build_env.sh
@@ -14,6 +14,8 @@ echo "安装 ppdiffusers..."
 cd ppdiffusers
 pip install -e .
 cd ..
+#注：ppdiffusers部分模型需要依赖 CUDA 11.2 及以上版本，如果本地机器不符合要求，建议前往 [AI Studio](https://aistudio.baidu.com/index) 进行模型训练、推理任务。
+#如果希望使用**bf16**训练推理，请使用支持**bf16**的GPU，如A100。
 
 # 安装依赖包
 echo "安装依赖包..."

diff --git a/docs/FAQ.md b/docs/FAQ.md
diff --git a/paddlemix/examples/README.md b/paddlemix/examples/README.md
@@ -6,21 +6,34 @@ paddlemix `examples` 目录下提供模型的一站式体验，包括模型推
 
 | Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
 | --- | --- | --- | --- | --- | --- | --- | 
-| [qwen_vl](./qwen_vl/) | ✅  | ❌  | ✅  | ✅  | ✅  | ❌ |
+| [YOLO-World](./YOLO-World/) | ✅  | ❌  | ❌  | ❌  | ❌ | ❌ |
+| [audioldm2](./audioldm2/) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
 | [blip2](./blip2/) | ✅  | ✅ | ✅  | ✅ |  ❌ | ❌ |
 | [clip](./clip) | ✅  | ✅ | ❌ | ❌ | ❌ | ❌ |
 | [coca](./coca/) |  ✅  | ✅ | ❌ | ❌ | ❌ | ❌ |
+| [CogVLM && CogAgent](./cogvlm/) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
 | [eva02](./eva02/)|    ✅  |  ✅  |  ✅  |  ❌   | ❌   | ❌ |
 | [evaclip](./evaclip/) | ✅ | ✅ | ❌ | ❌ |  ❌ | ❌ |
 | [groundingdino](./groundingdino/) | ✅ | ❌  | 🚧   | ❌  | ✅  | ❌ |
 | [imagebind](./imagebind/) |   ✅  |  ❌   |  ❌  | ❌ | ❌ | ❌ |
+| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
+| [Internvl2](./internvl2/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
+| [llava](./llava/) | ✅  | ✅  | ✅  | ✅  | 🚧  | ✅ |
+| [llava-next](./llava_next_interleave/) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
 | [minigpt4](./minigpt4) | ✅ | ✅ | ✅   |  ❌  | ✅  | ❌ |
+| [minimonkey](./minimonkey/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
+| [qwen2_vl](./qwen2_vl/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
+| [qwen_vl](./qwen_vl/) | ✅  | ❌  | ✅  | ✅  | ✅  | ❌ |
 | [sam](./sam/) | ✅ | ❌ | ❌ | ❌ | ✅  | ❌ |
 | [visualglm](./visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
-| [llava](./llava/) | ✅  | ✅  | ✅  | ✅  | 🚧  | ✅ |
-| [YOLO-World](./YOLO-World/) | ✅  | ❌  | ❌  | ❌  | ❌ | ❌ |
-| [CogVLM && CogAgent](./cogvlm/) | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
-| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
+
+
+
+
+
+
+
+
 
 
 * ✅: Supported

diff --git a/paddlemix/tools/README.md b/paddlemix/tools/README.md
@@ -3,13 +3,30 @@ PaddleMIX工具箱秉承了飞桨套件一站式体验、性能极致、生态
 
 [[English](README_en.md)]
 
-##  🛠️ 支持模型列表 🛠️
-| Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
-| --- | --- | --- | --- | --- | --- | --- |
-| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ | ❌ |
-| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  | ❌ |
-| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
-| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  | ✅ |
+##  🛠️ Unified Fine-tuning Tool for Multimodal Understanding 🛠️
+
+| Model |  SFT | LoRA | Deploy | NPU training |
+| --- |  --- | --- | --- | --- | 
+| [YOLO-World](./YOLO-World/) | ❌  | ❌  | ❌ | ❌ |
+| [audioldm2](./audioldm2/) | ❌ | ❌ | ❌ | ❌ |
+| [blip2](./blip2/) | ✅  | ✅ |  ❌ | ❌ |
+| [clip](./clip) |❌ | ❌ | ❌ | ❌ |
+| [coca](./coca/) |  ❌ | ❌ | ❌ | ❌ |
+| [CogVLM && CogAgent](./cogvlm/) |❌ | ❌ | ❌ | ❌ |
+| [eva02](./eva02/)|   ✅  |  ❌   | ❌   | ❌ |
+| [evaclip](./evaclip/) | ❌ | ❌ |  ❌ | ❌ |
+| [groundingdino](./groundingdino/) |  🚧   | ❌  | ✅  | ❌ |
+| [imagebind](./imagebind/) |  ❌  | ❌ | ❌ | ❌ |
+| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ❌ | ❌ |
+| [Internvl2](./internvl2/)| ✅ | ❌ | ❌ | ❌ |
+| [llava](./llava/)  | ✅  | ✅  | 🚧  | ✅ |
+| [llava-next](./llava_next_interleave/) | ❌ | ❌ | ❌ | ❌ |
+| [minigpt4](./minigpt4) | ✅   |  ❌  | ✅  | ❌ |
+| [minimonkey](./minimonkey/) | ✅ | ❌ | ❌ | ❌ |
+| [qwen2_vl](./qwen2_vl/)| ✅ | ❌ | ❌ | ❌ |
+| [qwen_vl](./qwen_vl/)  | ✅  | ✅  | ✅  | ❌ |
+| [sam](./sam/) | ❌ | ❌ | ✅  | ❌ |
+| [visualglm](./visualglm/) | ✅ | ✅ | ❌ | ❌ |
 
 * ✅: Supported
 * 🚧: In Progress

diff --git a/paddlemix/tools/README_en.md b/paddlemix/tools/README_en.md
@@ -4,12 +4,28 @@ The PaddleMIX toolkit embodies the design philosophy of one-stop experience, ult
 [[中文文档](README.md)]
 
 ##  🛠️ Supported Model List 🛠️
-| Model | Inference |Pretrain | SFT | LoRA | Deploy | NPU training |
-| --- | --- | --- | --- | --- | --- | --- |
-| [qwen_vl](../examples/qwen_vl/) | ✅  | ❌  | ✅  | ✅  |  ✅ | ❌ |
-| [blip2](../examples/blip2/) | ✅  | ✅ | ✅  | ✅ | ✅  | ❌ |
-| [visualglm](../examples/visualglm/) | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
-| [llava](../examples/llava/) | ✅  | ✅   | ✅  | ✅  | 🚧  | ✅ |
+| Model |  SFT | LoRA | Deploy | NPU training |
+| --- |  --- | --- | --- | --- | 
+| [YOLO-World](./YOLO-World/) | ❌  | ❌  | ❌ | ❌ |
+| [audioldm2](./audioldm2/) | ❌ | ❌ | ❌ | ❌ |
+| [blip2](./blip2/) | ✅  | ✅ |  ❌ | ❌ |
+| [clip](./clip) |❌ | ❌ | ❌ | ❌ |
+| [coca](./coca/) |  ❌ | ❌ | ❌ | ❌ |
+| [CogVLM && CogAgent](./cogvlm/) |❌ | ❌ | ❌ | ❌ |
+| [eva02](./eva02/)|   ✅  |  ❌   | ❌   | ❌ |
+| [evaclip](./evaclip/) | ❌ | ❌ |  ❌ | ❌ |
+| [groundingdino](./groundingdino/) |  🚧   | ❌  | ✅  | ❌ |
+| [imagebind](./imagebind/) |  ❌  | ❌ | ❌ | ❌ |
+| [InternLM-XComposer2](./internlm_xcomposer2/) | ✅ | ❌ | ❌ | ❌ |
+| [Internvl2](./internvl2/)| ✅ | ❌ | ❌ | ❌ |
+| [llava](./llava/)  | ✅  | ✅  | 🚧  | ✅ |
+| [llava-next](./llava_next_interleave/) | ❌ | ❌ | ❌ | ❌ |
+| [minigpt4](./minigpt4) | ✅   |  ❌  | ✅  | ❌ |
+| [minimonkey](./minimonkey/) | ✅ | ❌ | ❌ | ❌ |
+| [qwen2_vl](./qwen2_vl/)| ✅ | ❌ | ❌ | ❌ |
+| [qwen_vl](./qwen_vl/)  | ✅  | ✅  | ✅  | ❌ |
+| [sam](./sam/) | ❌ | ❌ | ✅  | ❌ |
+| [visualglm](./visualglm/) | ✅ | ✅ | ❌ | ❌ |
 
 * ✅: Supported
 * 🚧: In Progress