🤔 SemCoder: Training Code Language Models with Comprehensive Semantics

🤔 Overview | 🤖 Models | 📚 Dataset | 🛠️ Get Started | 🕹️ Demo | 📝 Citation | 🙏 Acknowledgements

📰 News

• []

🤔 Overview

🤖 Models

Model	Checkpoint	Size	HEval (+)	MBPP (+)	CRUXEval-I	CRUXEval-O	License
SemCoder	🤗 HF Link	6.7B	--.- (--.-)	--.- (--.-)	--.- (--.-)	--.- (--.-)	DeepSeek
SemCoder-S	🤗 HF Link	6.7B	--.- (--.-)	--.- (--.-)	--.- (--.-)	--.- (--.-)	DeepSeek

📚 Dataset

• PyX: ???
• PyX-Monologue: ???.
• PyX-R: ???.

🛠️ Get Started

Install Environment

git clone https://github.com/ARiSE-Lab/SemCoder.git;
cd SemCoder;
conda env create --name semcoder --file=environment.yml;
conda activate semcoder;
export PYTHONPATH=$(pwd);

🕹️ Demo

We follow Magicoder script to lanuch a gradio server for the local demo. You can launch your local gradio demo as following:

CUDA_VISIBLE_DEVICES=0 python semcoder_demo.py \
   --base_model "semcoder/semcoder_s" \
   --device "cuda:0" \
   --port 8080

Evaluation

• To evaluate SemCoder on EvalPlus, run

bash scripts/eval/eval_evalplus.sh

• To evaluate SemCoder on CRUXEval, you need to firstly clone their official release:

git clone https://github.com/facebookresearch/cruxeval.git

Update the $CRUXEVAL_HOME to be the absolute path of the cloned repository in this script and run:

bash scripts/eval/eval_cruxeval.sh

• To finetune SemCoder for debugging and self-refinement, please refer to this script

• To evaluate SemCoder for iterative self-refinement on EvalPlus, please run

bash scripts/eval/eval_finetune_refine.sh

• ✨ [New] ✨ To evaluate SemCoder on LiveCodeBench for code generation, please follow these steps:

# Clone our adapted LiveCodeBench
git clone https://github.com/Robin-Y-Ding/LiveCodeBench.git;

# Set up environment
cd LiveCodeBench;
conda create -n livecodebench Python=3.10;
conda activate livecodebench;
pip install poetry;
poetry install --with with-gpu;

# Run evaluation
export CUDA_VISIBLE_DEVICES=0;
python -m lcb_runner.runner.main \
  --model semcoder/semcoder_s \
  --scenario codegeneration \
  --evaluate

• ✨ [New] ✨ To evaluate SemCoder on LiveCodeBench for code execution, you can run:

export CUDA_VISIBLE_DEVICES=0;
cd LiveCodeBench;
python -m lcb_runner.runner.main \
    --model semcoder/semcoder_s \
    --scenario codeexecution \
    --cot_code_execution \
    --n 1 \
    --evaluate

📝 Citation

@article{ding2024semcoder,
  title={SemCoder: Training Code Language Models with Comprehensive Semantics},
  author={Yangruibo Ding and Jinjun Peng and Marcus J. Min and Gail Kaiser and Junfeng Yang and Baishakhi Ray},
  journal={arXiv preprint arXiv:2406.01006},
  year={2024}
}

🙏 Acknowledgements

We thank the following amazing projects that inspired our design choices:

• MagicCoder: Synthetic Code Generation.
• EvalPlus: Test-case Generation & Augmentation.
• DeepSeek-Coder: Base model for SemCoder.