Skip to content

Commit

Permalink
add multi-node script
Browse files Browse the repository at this point in the history
  • Loading branch information
boyuZh committed Aug 21, 2023
1 parent d4aa63f commit f04561a
Show file tree
Hide file tree
Showing 5 changed files with 78 additions and 2 deletions.
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -130,3 +130,8 @@ dmypy.json
.DS_Store
.idea/FinGPT.iml
*.xml

# Job scripts
fingpt/FinGPT_sentiment/instruct-FinGPT/run.sh
fingpt/FinGPT_sentiment/instruct-FinGPT/checkpoints
fingpt/FinGPT_sentiment/instruct-FinGPT/ds_results_all_10_v2_1.*
2 changes: 1 addition & 1 deletion fingpt/FinGPT_sentiment/instruct-FinGPT/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A fast, affordable, scalable and open system framework for enabling end-to-end I
## Training
Use the following command to instruction finetune the llama7b model on the financial sentiment analysis datasets.
```
python train.py --actor-model decapoda-research/sent-llama-7b-hf --deployment-type single_node --output-dir checkpoints
python train.py --actor-model /path/to/llama7b --deployment-type single_node --output-dir checkpoints
```

Choose the expected deployment-type,(ranging from single_gpu, single_node to multi_node)。These deployment type corresponds to different training scripts in the "training" folder. Modify the parameters of these scripts according to needs. Specially, if you want to finetune with LoRA, you can modify the script in training/supervised_finetuning/single_node/run_sent-llama-7b.sh as:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0

# DeepSpeed Team
OUTPUT=$1
ZERO_STAGE=$2
if [ "$OUTPUT" == "" ]; then
OUTPUT=./output
fi
if [ "$ZERO_STAGE" == "" ]; then
ZERO_STAGE=2
fi
mkdir -p $OUTPUT

deepspeed main.py \
--data_path zeroshot/twitter-financial-news-sentiment chiapudding/kaggle-financial-sentiment \
--data_split 10,0,0 \
--model_name_or_path facebook/opt-1.3b \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--max_seq_len 512 \
--learning_rate 9.65e-6 \
--weight_decay 0.1 \
--num_train_epochs 2 \
--gradient_accumulation_steps 1 \
--lr_scheduler_type cosine \
--num_warmup_steps 0 \
--seed 1234 \
--zero_stage $ZERO_STAGE \
--deepspeed \
--output_dir $OUTPUT \
&> $OUTPUT/training.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0

# DeepSpeed Team
OUTPUT=$1
ZERO_STAGE=$2
if [ "$OUTPUT" == "" ]; then
OUTPUT=./output
fi
if [ "$ZERO_STAGE" == "" ]; then
ZERO_STAGE=3
fi
mkdir -p $OUTPUT

deepspeed main.py \
--data_path zeroshot/twitter-financial-news-sentiment chiapudding/kaggle-financial-sentiment \
--data_split 10,0,0 \
--model_name_or_path decapoda-research/llama-7b-hf \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--max_seq_len 512 \
--learning_rate 1e-5 \
--weight_decay 0.1 \
--num_train_epochs 2 \
--gradient_accumulation_steps 1 \
--lr_scheduler_type cosine \
--num_warmup_steps 0 \
--seed 1234 \
--gradient_checkpointing \
--zero_stage $ZERO_STAGE \
--deepspeed \
--output_dir $OUTPUT \
&> $OUTPUT/training.log


# --lora_dim 128 \
# --lora_module_name model.layers. \
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ mkdir -p $OUTPUT

deepspeed main.py \
--data_path zeroshot/twitter-financial-news-sentiment chiapudding/kaggle-financial-sentiment \
--data_split 2,4,4 \
--data_split 10,0,0 \
--model_name_or_path decapoda-research/llama-7b-hf \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
Expand Down

0 comments on commit f04561a

Please sign in to comment.