Add source code for RAG based FinGPT for Financial Sentiment Analysis

fingpt/FinGPT_sentiment/README.md

@@ -1,35 +1,94 @@
-## FinGPT Sentiment Analysis
+# Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models: Open Source Code
 
-## Motivations:
-* Classify financial statements to help traders aggregate and digest financial news
-## Methods:
-* FinGPT fine-tuning
-* Retrieval-Augmented Generation
+## Overview
 
+This repository provides the implementation of a retrieval-augmented Large Language Models (LLMs) framework tailored for financial sentiment analysis. Directly applying LLMs for financial sentiment analysis has challenges, especially when faced with the succinct nature of financial news. Our method addresses these challenges by introducing a two-fold approach: an instruction-tuned LLMs module and a retrieval-augmentation module.
+![Framwork](assets/framework.jpg)
+## Abstract
 
-## Setup
+Financial sentiment analysis is critical for valuation and investment decision-making. Traditional NLP models, limited by their parameter size and the scope of their training datasets, often fall short in this area. Large Language Models (LLMs) have demonstrated superior performance across various NLP tasks, but directly applying them to financial sentiment analysis presents its own set of challenges. Our proposed framework, benchmarked against traditional models and other LLMs, achieves a performance gain of 15% to 48% in accuracy and F1 score.
 
-* Visit environment_news_scraping.yml for the environment setup
-* Set up your .env file, can refer to /FinGPT_sentiment/.env.example
+## Key Contributions
 
-``` python
+1. Introduction of a novel retrieval-augmented large language model framework tailored for financial sentiment analysis, ensuring more nuanced and informed predictions.
+2. The unique method of instruction tuning provides a more accurate response to user-intended financial sentiment analysis tasks.
+3. Through extensive evaluations, we show that our approach significantly outperforms both traditional sentiment analysis models and renowned general-purpose LLMs.
 
-python news_scraper.py
+## Repository Contents
 
+- `multisource_retrieval`: Main code directory for the retrieval augmentation module which enhance the concise query with context.
+- `instruction-FinGPT`: Directory containing the code and data related to the instruction-tuned LLMs module.
+
+
+## Performance
+Zero-shot evaluation between BloombergGPT, general-purpose LLMs like ChatGPT, and our model on the dataset of financial phaseBank (FPB).
+| Metrics | Accuracy | F1 |
+|---------|----------|----|
+| BloombergGPT [^1^] | - | 0.51 |
+| ChatGPT 4.0 [^2^] | 0.64 | 0.51 |
+| ChatGLM2-6B [^3^] | 0.47 | 0.40 |
+| Llama-7B [^4^] | 0.60 | 0.40 |
+| Ours | **0.76** | **0.74** |
+
+[^1^]: [https://arxiv.org/abs/2303.17564]
+[^2^]: [https://arxiv.org/abs/2203.02155]
+[^3^]: [https://arxiv.org/abs/2210.02414]
+[^4^]: [https://arxiv.org/abs/2302.13971]
+
+Experimental results on the Twitter Val dataset.
+| Metrics          | Accuracy | F1    |
+|------------------|----------|-------|
+| FinBert [^1^]    | 0.725    | 0.668 |
+| ChatGLM2-6B [^2^]| 0.482    | 0.381 |
+| LLaMA-7B [^3^]   | 0.54     | 0.36  |
+| Ours w/o Context | 0.86     | 0.811 |
+| Ours w/ RAG Context | **0.88**  | **0.842** |
+
+[^1^]: [https://arxiv.org/abs/1908.10063]
+[^2^]: [https://arxiv.org/abs/2210.02414]
+[^3^]: [https://arxiv.org/abs/2302.13971]
+
+### An showcase of RAG-instruction-tuned LLM
+![Case](assets/showcase.png)
+
+## Installation and Usage
+
+1. Clone the repository:
+```bash
+git clone https://github.com/AI4Finance-Foundation/FinGPT.git
+```
+
+2. Navigate to the project directory and install the required dependencies:
+```bash
+cd fingpt/FinGPT_sentiment
+pip install -r requirements.txt
 ```
 
-## I. Data Preparation
-Task 1: GPT-based News Classification
+3. Instructions on how to run the model, fine-tune, or evaluate can be found in the respective directories (`instruction-FinGPT`, `multisource_retrieval`). Note the generated .csv file from multisource_retrieval is the input for instruction tuned LLM during evaluation.
+
+## Citation
+
+If you find our work useful or use it in your projects, please consider citing our original paper. The bibtex reference will be updated once available.
+
+```
+@misc{zhang2023instructfingpt,
+      title={Instruct-FinGPT: Financial Sentiment Analysis by Instruction Tuning of General-Purpose Large Language Models}, 
+      author={Boyu Zhang and Hongyang Yang and Xiao-Yang Liu},
+      year={2023},
+      eprint={2306.12659},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+## Feedback and Contributions
+
+We welcome feedback, bug reports, and any contributions to improve the code or extend the model capabilities. Please open an issue or create a pull request, and we'll attend to it promptly.
+
+## License
 
-1. On UI, select csv file to load
-2. Creates "classification" column for each financial statement
-3. Using "default_classification_prompt" to ask GPT to classify the news
-4. Saves .csv
+This project is licensed under the MIT License. See the `LICENSE` file for details.
 
-Task 2: Context Retrieval
+---
 
-1. On UI, select csv file to load
-2. Creates "contextualized_sentence" for each financial statement
-3. Using Google and various news sources to retrieve the context
-   4. Add relevant news paragraphs to form "contextualized_sentence" for each financial statement
-5. Saves .csv
+This README provides a structured overview for potential users or contributors to easily understand, use, and potentially contribute to the open-source project.

fingpt/FinGPT_sentiment/instruct-FinGPT/README.md

@@ -0,0 +1,53 @@
+
+A fast, affordable, scalable and open system framework for enabling end-to-end Instruction Tuning experience to generate high-quality Instruct-FinGPT models at all scales.
+
+## Training
+Use the following command to instruction finetune the llama7b model on the financial sentiment analysis datasets.
+```
+python train.py --actor-model decapoda-research/sent-llama-7b-hf --deployment-type single_node --output-dir checkpoints
+```
+
+Choose the expected deployment-type，(ranging from single_gpu, single_node to multi_node)。These deployment type corresponds to different training scripts in the "training" folder. Modify the parameters of these scripts according to needs. Specially, if you want to finetune with LoRA, you can modify the script in training/supervised_finetuning/single_node/run_sent-llama-7b.sh as:
+```
+OUTPUT=$1
+ZERO_STAGE=$2
+if [ "$OUTPUT" == "" ]; then
+    OUTPUT=./output
+fi
+if [ "$ZERO_STAGE" == "" ]; then
+    ZERO_STAGE=3
+fi
+mkdir -p $OUTPUT
+
+python3 main.py \
+   --data_path zeroshot/twitter-financial-news-sentiment chiapudding/kaggle-financial-sentiment \
+   --data_split 2,4,4 \
+   --model_name_or_path decapoda-research/llama-7b-hf \
+   --per_device_train_batch_size 4 \
+   --per_device_eval_batch_size 4 \
+   --max_seq_len 512 \
+   --learning_rate 1e-5 \
+   --weight_decay 0.1 \
+   --num_train_epochs 2  \
+   --gradient_accumulation_steps 1 \
+   --lr_scheduler_type cosine \
+   --num_warmup_steps 0 \
+   --seed 1234 \
+   --gradient_checkpointing \
+   --zero_stage $ZERO_STAGE \
+   --deepspeed \
+   --output_dir $OUTPUT \
+   --lora_dim 128 \
+   --lora_module_name model.layers. \
+   &> $OUTPUT/training.log
+   ```
+
+## Testing
+Use the following command to implement inference.
+```
+python ./inference/batchbot_torch.py --path checkpoints/actor-models/sent-llama-7b --max_new_tokens 16
+```
+
+
+## Acknowledgement
+This code is developed based on [DeepSpeed-Chat](https://github.com/microsoft/DeepSpeed/tree/master/blogs/deepspeed-chat).

fingpt/FinGPT_sentiment/instruct-FinGPT/chat.py

@@ -0,0 +1,31 @@
+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0
+
+# DeepSpeed Team
+
+import argparse
+import subprocess
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--path",
+                        type=str,
+                        help="Directory containing trained actor model")
+    parser.add_argument("--phase",
+                        type=str,
+                        choices=('chat', 'infer'),
+                        help="whether to run chat or inference")
+    parser.add_argument(
+        "--max_new_tokens",
+        type=int,
+        default=128,
+        help="Maximum new tokens to generate per response",
+    )
+    args = parser.parse_args()
+
+    if args.phase == 'chat':
+        cmd = f"python3 ./inference/chatbot.py --path {args.path} --max_new_tokens {args.max_new_tokens}"
+    else:
+        cmd = f"python3 ./inference/batchbot.py --path {args.path} --max_new_tokens {args.max_new_tokens}"
+    p = subprocess.Popen(cmd, shell=True)
+    p.wait()