Skip to content

Commit

Permalink
updates on fingpt3
Browse files Browse the repository at this point in the history
  • Loading branch information
@oliverwang15
oliverwang15 committed Aug 24, 2023
1 parent adec8a4 commit a3c70bb
Showing 1 changed file with 22 additions and 22 deletions.
44 changes: 22 additions & 22 deletions fingpt/FinGPT-v3/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,28 +49,28 @@ for sentiment in out_text:

## Ⅱ. Benchmark Results

| **Weighted F1** | BloombergGPT | [ChatGLM2](https://github.com/THUDM/ChatGLM2-6B) | [v3](https://huggingface.co/oliverwang15/FinGPT_ChatGLM2_Sentiment_Instruction_LoRA_FT) | [v3.1](https://huggingface.co/oliverwang15/FinGPT_v31_ChatGLM2_Sentiment_Instruction_LoRA_FT) | v3.1.1 (8bit) | v3.1.2 (QLoRA) | [Llama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | [v3.2](https://huggingface.co/oliverwang15/FinGPT_v32_Llama2_Sentiment_Instruction_LoRA_FT) |
| --------------- | :----------: | :----------------------------------------------: | :----------------------------------------------------------: | :--------------: | :----------------: | :------------------------: | :------------------------: | :------------------------: |
| FPB [1] | 0.511 | 0.381 | 0.701 | <ins>**0.855**</ins> | <ins>**0.855**</ins> | 0.777 | 0.390 | *0.850* |
| FiQA-SA [2] | 0.751 | 0.790 | 0.760 | <ins>0.850</ins> | *0.847* | 0.752 | 0.800 | **0.860** |
| TFNS [3] | - | 0.189 | 0.740 | *0.875* | <ins>0.879</ins> | 0.828 | 0.296 | **0.894** |
| NWGI [4] | - | 0.449 | 0.578 | **0.642** | <ins>0.636</ins> | 0.583 | 0.503 | *0.632* |
| Mean | | 0.452 | 0.730 | <ins>0.805</ins> | *0.804* | 0.735 | 0.497 | **0.809** |
| Std | | 0.217 | **0.091** | *0.095* | 0.098 | <ins>0.092</ins> | 0.189 | 0.103 |
| **ACC/F1 Micro** | | | | | | | | |
| FPB [1] | - | 0.464 | 0.775 | <ins>**0.856**</ins> | <ins>**0.856**</ins> | 0.797 | 0.462 | *0.851* |
| FiQA-SA [2] | - | 0.822 | 0.731 | <ins>*0.836*</ins> | <ins>*0.836*</ins> | 0.713 | 0.822 | **0.844** |
| TFNS [3] | - | 0.331 | 0.738 | *0.876* | <ins>0.879</ins> | 0.837 | 0.386 | **0.894** |
| NWGI [4] | - | 0.560 | 0.588 | **0.642** | <ins>0.638</ins> | 0.587 | 0.583 | *0.636* |
| Mean | | 0.544 | 0.735 | <ins>0.803</ins> | *0.802* | 0.733 | 0.563 | **0.806** |
| Std | | 0.180 | **0.090** | <ins>0.094</ins> | *0.096* | *0.096* | 0.165 | 0.100 |
| **Macro F1** | | | | | | | | |
| FPB [1] | - | 0.487 | 0.745 | **0.841** | *0.837* | 0.752 | 0.517 | <ins>0.840</ins> |
| FiQA-SA [2] | - | 0.560 | 0.641 | <ins>0.746</ins> | *0.743* | 0.641 | 0.610 | **0.752** |
| TFNS [3] | - | 0.340 | 0.681 | *0.841* | <ins>0.845</ins> | 0.774 | 0.401 | **0.866** |
| NWGI [4] | - | 0.489 | 0.579 | **0.650** | <ins>0.645</ins> | 0.592 | 0.539 | *0.644* |
| Mean | | 0.469 | 0.675 | <ins>0.769</ins> | *0.767* | 0.690 | 0.517 | **0.776** |
| Std | | 0.080 | **0.069** | 0.079 | 0.081 | *0.076* | <ins>0.075</ins> | 0.087 |
| **Weighted F1** | BloombergGPT | ChatGPT | GPT-4 | [ChatGLM2](https://github.com/THUDM/ChatGLM2-6B) | [v3](https://huggingface.co/oliverwang15/FinGPT_ChatGLM2_Sentiment_Instruction_LoRA_FT) | [v3.1](https://huggingface.co/oliverwang15/FinGPT_v31_ChatGLM2_Sentiment_Instruction_LoRA_FT) | v3.1.1 (8bit) | v3.1.2 (QLoRA) | [Llama2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | [v3.2](https://huggingface.co/oliverwang15/FinGPT_v32_Llama2_Sentiment_Instruction_LoRA_FT) |
| --------------- | :----------: | :----------------------------------------------: | :----------------------------------------------------------: | :--------------: | :----------------: | :------------------------: | :------------------------: | :------------------------: | :------------------------: | :------------------------: |
| FPB [1] | 0.511 | 0.781 | 0.833 | 0.381 | 0.701 | <ins>**0.855**</ins> | <ins>**0.855**</ins> | 0.777 | 0.390 | *0.850* |
| FiQA-SA [2] | 0.751 | 0.730 | 0.630 | 0.790 | 0.760 | <ins>0.850</ins> | *0.847* | 0.752 | 0.800 | **0.860** |
| TFNS [3] | - | 0.736 | 0.808 | 0.189 | 0.740 | *0.875* | <ins>0.879</ins> | 0.828 | 0.296 | **0.894** |
| NWGI [4] | - | - | - | 0.449 | 0.578 | **0.642** | <ins>0.636</ins> | 0.583 | 0.503 | *0.632* |
| Mean | | | | 0.452 | 0.730 | <ins>0.805</ins> | *0.804* | 0.735 | 0.497 | **0.809** |
| Std | | | | 0.217 | **0.091** | *0.095* | 0.098 | <ins>0.092</ins> | 0.189 | 0.103 |
| **ACC/F1 Micro** | | | | | | | | | | |
| FPB [1] | - | 0.781 | 0.834 | 0.464 | 0.775 | <ins>**0.856**</ins> | <ins>**0.856**</ins> | 0.797 | 0.462 | *0.851* |
| FiQA-SA [2] | - | 0.662 | 0.545 | 0.822 | 0.731 | <ins>*0.836*</ins> | <ins>*0.836*</ins> | 0.713 | 0.822 | **0.844** |
| TFNS [3] | - | 0.731 | 0.813 | 0.331 | 0.738 | *0.876* | <ins>0.879</ins> | 0.837 | 0.386 | **0.894** |
| NWGI [4] | - | - | | 0.560 | 0.588 | **0.642** | <ins>0.638</ins> | 0.587 | 0.583 | *0.636* |
| Mean | | | | 0.544 | 0.735 | <ins>0.803</ins> | *0.802* | 0.733 | 0.563 | **0.806** |
| Std | | | | 0.180 | **0.090** | <ins>0.094</ins> | *0.096* | *0.096* | 0.165 | 0.100 |
| **Macro F1** | | | | | | | | | | |
| FPB [1] | - | 0.770 | 0.827 | 0.487 | 0.745 | **0.841** | *0.837* | 0.752 | 0.517 | <ins>0.840</ins> |
| FiQA-SA [2] | - | 0.611 | 0.539 | 0.560 | 0.641 | <ins>0.746</ins> | *0.743* | 0.641 | 0.610 | **0.752** |
| TFNS [3] | - | 0.693 | 0.758 | 0.340 | 0.681 | *0.841* | <ins>0.845</ins> | 0.774 | 0.401 | **0.866** |
| NWGI [4] | - | | | 0.489 | 0.579 | **0.650** | <ins>0.645</ins> | 0.592 | 0.539 | *0.644* |
| Mean | | | | 0.469 | 0.675 | <ins>0.769</ins> | *0.767* | 0.690 | 0.517 | **0.776** |
| Std | | | | 0.080 | **0.069** | 0.079 | 0.081 | *0.076* | <ins>0.075</ins> | 0.087 |

​ **X**: Best score, <ins>X</ins>: Second best score, *X*: Third best score

Expand Down

0 comments on commit a3c70bb

Please sign in to comment.