Agents on DSpy

Building an agent using DSPy that can interact with the webarena environment.

We are looking to achieve SOTA performance on webarena benchmark, by implementing various ideas like:

• SteP: Stacked LLM Policies for Web Actions
• Tree Search For Language Model Agents

Sample Agent

The sample agent is for solving the task of finding the walking distance between 2 locations on OpenStreetMap.

Installation

1. Install webarena depedencies

cd webarena
uv venv -p 3.11 --seed
source .venv/bin/activate
pip install -r requirements.txt
playwright install
pip install -e .

2. Configure the environment

# export MAP="http://ec2-3-131-244-37.us-east-2.compute.amazonaws.com:3000"
export MAP="https://www.openstreetmap.org"
export SHOPPING="<your_shopping_site_domain>:7770"
export SHOPPING_ADMIN="<your_e_commerce_cms_domain>:7780/admin"
export REDDIT="<your_reddit_domain>:9999"
export GITLAB="<your_gitlab_domain>:8023"

3. Obtain the auto-login cookies for all websites

mkdir -p ./.auth
python browser_env/auto_login.py

4. Copy the map configs to config_data folder

python scripts/generate_test_data.py
For Fedora:
  grep -ol "\"map\"" config_files/*.json | xargs cp -t ../config_data/
For Mac:
  grep -ol "\"map\"" config_files/*.json | xargs -I {} cp {} ../config_data/
rm ../config_data/test*.json

Running

1. Setup the environment from the root directory

source webarena/.venv/bin/activate
python -m scripts.evaluate.debug_webarena