Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add benchmarks API-Bank, APIBench, Nexus #1136

Merged
merged 48 commits into from
Dec 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
713a2d2
add benchmark gorilla, nexus
HHHHHejia Oct 30, 2024
9398ffb
add apibank
HHHHHejia Nov 1, 2024
f8ba0d7
Merge branch 'master' into benchmark_hejia
harryeqs Nov 19, 2024
5352cea
Merge branch 'master' into benchmark_hejia
harryeqs Nov 24, 2024
f6d3436
Merge branch 'master' into benchmark_hejia
harryeqs Nov 25, 2024
638dc57
Convert gorilla and nexusraven to regular directories
HHHHHejia Nov 26, 2024
679225e
Merge branch 'master' into benchmark_hejia
harryeqs Dec 2, 2024
1c113d1
refactor: Constructed NexusBenchmark following BaseBenchmark from GAI…
harryeqs Dec 4, 2024
ca1e490
Merge branch 'master' into benchmark_hejia
harryeqs Dec 4, 2024
b5a7d0b
Merge branch 'master' into benchmark_hejia
harryeqs Dec 5, 2024
9083048
refactor: Refactored the integration of APIBench (Gorilla) benchmark
harryeqs Dec 6, 2024
1b151e0
Merge branch 'master' into benchmark_hejia
harryeqs Dec 6, 2024
14ee80a
refactor: Modified code structure
harryeqs Dec 7, 2024
42b7407
refactor: Integrated APIBank
harryeqs Dec 8, 2024
f908681
refactor: Change directory name for smoother merge
harryeqs Dec 8, 2024
9a77c78
Merge branch 'master' into benchmark_hejia
harryeqs Dec 8, 2024
0d71313
docs: Update docs and put into benchmarks directory
harryeqs Dec 9, 2024
7745c37
docs: Included examples
harryeqs Dec 9, 2024
8ad922a
update poetry lock
harryeqs Dec 9, 2024
acd676c
fix: Fix tree_sitter_import issue
harryeqs Dec 9, 2024
1511711
Merge branch 'master' into benchmark_hejia
harryeqs Dec 9, 2024
5a26acf
update poetry lock
harryeqs Dec 9, 2024
68c4e30
update poetry lock
harryeqs Dec 9, 2024
7c427ce
clean code for upcoming refactoring
harryeqs Dec 10, 2024
4730c16
Merge branch 'master' into benchmark_hejia
harryeqs Dec 10, 2024
bc057ab
fix tests
harryeqs Dec 11, 2024
0cce601
Merge branch 'benchmark_hejia' of https://github.com/HHHHHejia/camel …
harryeqs Dec 11, 2024
b757d8d
updated download method for APIBank benchmark
harryeqs Dec 11, 2024
03221e1
updated APIBench download and eval_ast
harryeqs Dec 11, 2024
2f52186
Merge branch 'master' into benchmark_hejia
harryeqs Dec 11, 2024
0e942ef
updated docstrings
harryeqs Dec 12, 2024
35f2afa
Merge branch 'master' into benchmark_hejia
harryeqs Dec 12, 2024
ad777cf
updated docstrings
harryeqs Dec 12, 2024
ded55bf
Merge branch 'master' into benchmark_hejia
harryeqs Dec 14, 2024
83805f0
update type annotations
harryeqs Dec 14, 2024
f534efe
clean code and update docstrings
harryeqs Dec 15, 2024
1eab6e3
Merge branch 'master' into benchmark_hejia
harryeqs Dec 15, 2024
34e33cf
Merge branch 'master' into benchmark_hejia
harryeqs Dec 16, 2024
1d280a9
Merge branch 'master' into benchmark_hejia
harryeqs Dec 17, 2024
41fe3ed
Merge branch 'master' into benchmark_hejia
harryeqs Dec 18, 2024
9a0dba3
update pyproject.toml and poetry.lock
harryeqs Dec 18, 2024
a367501
Merge branch 'master' into benchmark_hejia
harryeqs Dec 19, 2024
a5127f9
improve structure and add unit tests
harryeqs Dec 23, 2024
511fbe0
Merge branch 'master' into benchmark_hejia
harryeqs Dec 23, 2024
f3da00c
resolve conflicts
harryeqs Dec 23, 2024
8907b94
update poetry.lock
harryeqs Dec 23, 2024
c38f619
Merge branch 'master' into benchmark_hejia
harryeqs Dec 27, 2024
d2b7e9d
Merge branch 'master' into benchmark_hejia
Wendong-Fan Dec 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add apibank
  • Loading branch information
HHHHHejia committed Nov 1, 2024
commit 9398ffb15d5ac4ce6ff2fdc19edb470b11d088b0
71 changes: 0 additions & 71 deletions camel/benchmark/datasets/apibank/README.md

This file was deleted.

2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/add_agenda.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/add_alarm.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/add_meeting.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/add_reminder.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/add_scene.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import random

class AddScene(API):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime
import random

Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/book_hotel.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/calculator.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .api import API
from camel.benchmark.datasets.apibank.apis.api import API

class Calculator(API):
description = 'This API provides basic arithmetic operations: addition, subtraction, multiplication, and division.'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class CancelRegistration(API):
description = 'This API cancels the registration of a patient given appointment ID.'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class CancelTimedSwitch(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/check_token.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json

class CheckToken(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_account.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json

class DeleteAccount(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_agenda.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_alarm.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_meeting.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_reminder.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/delete_scene.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class DeleteScene(API):

Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/dictionary.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
# from api import API
import requests

Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/document_qa.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class DocumentQA(API):
description = 'This API answers the question from a given document url.'
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class EmergencyKnowledge(API):

Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/forgot_password.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import random

Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/get_today.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from .api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class GetToday(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/get_user_token.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
class GetUserToken(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/image_caption.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class ImageCaption(API):
description = 'This API generates a caption for a given image.'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_agenda.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_alarm.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_meeting.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_password.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json

class ModifyPassword(API):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class ModifyRegistration(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_reminder.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/modify_scene.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis import API
from camel.benchmark.datasets.apibank.apis import API

class ModifyScene(API):
description = 'This API modifies a scene of smart home system, given the scene name and a list of smart devices'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/open_bank_account.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/play_music.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class PlayMusic(API):
description = 'This API triggers a music player to play music.'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_agenda.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_alarm.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_balance.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class QueryBalance(API):
description = 'This API queries the balance of a given user.'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_health_data.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class QueryHealthData(API):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# from api import API
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class QueryHistoryToday(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_meeting.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class QueryRegistration(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_reminder.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import json
import os
import datetime
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_scene.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class QueryScene(API):
description = 'This API queries a scene of smart home system, given the scene name'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/query_stock.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class QueryStock(API):
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import datetime

class RecordHealthData(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/register_user.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import string
import random
import json
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/search_engine.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
from rank_bm25 import BM25Okapi
import numpy as np
import nltk
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/send_email.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
# from api import API
import re

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class SpeechRecognition(API):
description = 'This API recognizes the speech from a given audio url.'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/symptom_search.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class SymptomSearch(API):
description = 'This API searches for a given symptom.'
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/timed_switch.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis import API
from camel.benchmark.datasets.apibank.apis import API
import datetime

class TimedSwitch(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/tool_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
logging.getLogger('sentence_transformers').setLevel(logging.WARNING)

import json
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
import os

class ToolSearcher(API):
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/translate.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API
# from api import API
import googletrans
from googletrans import Translator
Expand Down
2 changes: 1 addition & 1 deletion camel/benchmark/datasets/apibank/apis/wiki.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from apis.api import API
from camel.benchmark.datasets.apibank.apis.api import API

class Wiki(API):
description = 'This API for searching a keyword in Wikipedia.'
Expand Down
Loading
Loading