Skip to content

Commit

Permalink
Merge branch 'master' into add/experiment-configuration
Browse files Browse the repository at this point in the history
  • Loading branch information
vadim0x60 authored Jan 15, 2024
2 parents b9de2e3 + 4d028c2 commit 602b921
Showing 1 changed file with 6 additions and 2 deletions.
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
A framework for AI-assisted program synthesis.
Given a problem description and some input-output examples, the framework generates a program that solves the problem.

## Paper

You can find an in-depth discussion of this tool, the philosophy it implements and its usage in our paper, [Fully Autonomous Programming with Large Language Models](https://dl.acm.org/doi/abs/10.1145/3583131.3590481). Consider citing it if you use SEIDR in your research.

## Usage

```
Expand All @@ -12,7 +16,7 @@ help(dev)

## Reproducing the experiments from our paper

The experiments reported in [the blog post](https://vadim.me/posts/unreasonable) and in the upcoming paper are contained in `benchmark.py` file. When you run this file, the AI-generated programs are commited to a dedicated github repository, while the metrics (i.e. how many tests every program passes) will be logged in your [Weights and Biases](https://wandb.ai)
The experiments are contained in `benchmark.py` and `benchmark_humaneval.py` files. When you run this file, the AI-generated programs are commited to a dedicated github repository, while the metrics (i.e. how many tests every program passes) will be logged in your [Weights and Biases](https://wandb.ai)

### Prerequisites
#### Set up Weights and Biases
Expand Down Expand Up @@ -108,4 +112,4 @@ python3 benchmark.py \
--ollama_url "http://URL:PORT"
```

Example Slurm scripts are stored in `example_scripts/` and tables with hyperparameters in `/config`
Example Slurm scripts are stored in `example_scripts/` and tables with hyperparameters in `/config`

0 comments on commit 602b921

Please sign in to comment.