feat: Add benchmarks API-Bank, APIBench, Nexus

[HHHHHejia]

I've added the APIbank APIbench and Nexus benchmark, main method see benchmark test and utils folder (benchmark_base.py)

There're some problem to be solved for the APIBank, APIbench(gorilla) and Nexus benchmark. listed as below.

For Nexus:
run python nexus_test.py. You'll get error
1.OpenAI limits the size of the function passed into the function call api (function name, function description length, number of functions, etc.). You need to add judgment logic in Camel. If OpenAI does not allow function call, use structure output instead.

2.Critical: while true bug in camel.chatagent.step. When the incoming api is not executed correctly, while true will not terminate.The while true logic should be eliminated. You cannot assume that the function passed by the user will always be executed correctly.

For APIbench
There're three datasets 'torchhub', 'tensorhub', 'huggingface’ . "torchhub"works well. BUT
3.'tensorhub', 'huggingface’ could not be correctly evaluted by the ast matching program. This is a problem within the original repo. I have already proposed an issue. [(https://github.com/ShishirPatil/gorilla/issues/729)]

It could be version problem of tree_sitter, but if you don't use tree_sitter==0.20.4, you'll get an another bug.

For APIbank
There're three datasets 'level1', 'level2', 'level3’ . BUT
4.NO ONE knows how to eveluate 'level3'. See the issue in original repo:
[https://github.com/AlibabaResearch/DAMO-ConvAI/issues/167]
[https://github.com/AlibabaResearch/DAMO-ConvAI/issues/102]
[https://github.com/AlibabaResearch/DAMO-ConvAI/issues/114]

5.APIbank involves multiple "User-Assistant-System" messages as History Records. Camel ChatAgent does not support adding multiple rounds of system messages yet. Temporary solution: Use record_message and make_assistant_message instead of system messages.

6.The version conflict between openai in camel, Https, and Google translate in original repo, see
[https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/api-bank#demo]. Camel, Https and Google translate lib doesn't work together.
For now two way works:
-use original repo without camel, Google translate and Https works well.
-use camel, remove Google translate, it works but without Google translate tool.
See:
[https://github.com/microsoft/TaskWeaver/issues/172]

7.Some datasets need to be hosted on GitHub/HuggingFace. The original author did not do this, but we do not want to include these data in Camel's GitHub.

[harryeqs]

@Wendong-Fan Hi Wendong, the three functional calling benchmarks have been integrated following the pattern of the GAIA benchmark integration. Sorry that the retriever has not yet been integrated into the APIBench benchmark due to time constraints and that can be done by Wednesday at the earliest, but the other parts are ready for review.

I also have a few questions regarding the integration:

1. The code of all three benchmarks is under Apache 2.0 license, are there any patterns to follow when referencing the code I copied/adapted?
2. What should be included in the unit test for these three benchmarks? I feel like the GAIA benchmark test is hard to reference for these benchmarks.
3. The level 3 dataset of APIBank still can't run since there haven't been any updates from the authors on how to use the evaluator.
4. How do you run the pre-commit test when committing to this fork?

Thanks!

[Wendong-Fan]

Thanks @HHHHHejia @harryeqs ! Left some comments for APIBank

[Wendong-Fan]

I think the current run method in BaseBenchmark should be refactored. cc @liuxukun2000

[harryeqs]

Hi Wendong should this be done in this PR, or shall we set up a new issue and PR for the refactoring?

[Wendong-Fan]

yeah we can do it in another pr, issue created here:#1338

[Wendong-Fan]

some errors with running pre-commit run --all-files need to be fixed

camel/benchmarks/utils/ast_eval.py:28: error: Cannot find implementation or library stub for module named "tree_sitter_python"  [import-not-found]
camel/benchmarks/utils/ast_eval.py:28: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
camel/benchmarks/utils/ast_eval.py:29: error: Library stubs not installed for "tree_sitter"  [import-untyped]

Hi Wendong sorry I tried but could not reproduce this error. Could you please check if the tree-sitter-python package has been successfully installed in your venv after running poetry install --with dev,docs -E all. This package was added to the poetry lock and its stub check should have been disabled so there shouldn't be any errors?

Sorry @harryeqs , my bad, I didn't switch my environment properly

[harryeqs]

Thanks @HHHHHejia @harryeqs ! Left some comments for APIBank

Thanks @Wendong-Fan for the comprehensive review! I have made some changes and please have a look when possible.
Thanks!

[Wendong-Fan]

thanks @harryeqs and sorry for the late review, left some comments below

[Wendong-Fan]

where did these dependencies used?

[harryeqs]

Hi Wendong, the googletrans-py is used in the APIs defined for the API-Bank, while the tree-sitter and tree-sitter-python are used for evaluation of APIBench.

[Wendong-Fan]

hey @harryeqs , I noticed some review comments were marked as resolved, but the updates don’t appear to have been pushed. Did you forget to push the code? Please ensure all comments are fully addressed before marking them as resolved.

[harryeqs]

hey @harryeqs , I noticed some review comments were marked as resolved, but the updates don’t appear to have been pushed. Did you forget to push the code? Please ensure all comments are fully addressed before marking them as resolved.

Hi Wendong I am very sorry that the update was delayed due to the hackathon. I've addressed some comments locally but have not pushed since I am still adding the tests. I will finish the tests asap this afternoon and push the code. Sorry for the wait!

[harryeqs]

thanks @harryeqs and sorry for the late review, left some comments below

Thank you @Wendong-Fan very much for the review! Sorry for the delay in updating the benchmarks. The tests have been added but they are based on a number of mocks as downloading the actual datasets is time-consuming and requires extra storage.

[Wendong-Fan]

from Guohao: in the example we should add tools to the ChatAgent