b106a61352
* ci: Mitigate security issues in autogpt-ci.yml - Remove unnecessary pull_request_target paths and related variables and config - Set permissions for contents to read only * ci: Simplify steps in autogpt-ci.yml workflow using GitHub CLI - Simplify step in 'autogpt-ci.yml' by using GitHub CLI instead of API for adding label and comment functionality - Replace curl command with 'gh issue edit' to add "behaviour change" label to the pull request - Replace gh api command with 'gh issue comment' to leave a comment about the changed behavior of AutoGPT in the pull request * ci: Fix issues in workflows - Move environment variable definition to top level in benchmark-ci.yml (because the other job also needs it) - Removed invalid 'branches: [hackathon]' restriction in hackathon.yml workflow - Removed redundant 'ref' and 'repository' fields in the 'checkout' step of both workflows. * ci: Delete legacy benchmarks.yml workflow * ci: Add triggers for CI workflows - Add triggers to run CI workflows when they are edited. - Update the paths for the CI workflows in the trigger configuration. * fix: Fix benchmark lint error - Removed unnecessary blank lines in report_types.py - Fixed string quotes in challenge.py to maintain consistency * fix: Update task description in password generator data.json - Update task description in `data.json` file for the password generator challenge to clarify the input requirements and error handling. - This change is made in an attempt to make the Benchmark CI pass. * fix: Fix PasswordGenerator challenge in CI - Fix the behavior of the reference password_generator.py to align with the task description - Use default password length 8 instead of a random length in the generate_password function - Retrieve the password length from the command line arguments if "--length" is provided, else set it to 8
As a user
pip install auto-gpt-benchmarks- Add boilerplate code to run and kill agent
agbenchmark--category challenge_categoryto run tests in a specific category--mockto only run mock tests if they exists for each test--noregto skip any tests that have passed in the past. When you run without this flag and a previous challenge that passed fails, it will now not be regression tests
- We call boilerplate code for your agent
- Show pass rate of tests, logs, and any other metrics
Contributing
Diagrams: https://whimsical.com/agbenchmark-5n4hXBq1ZGzBwRsK4TVY7x
To run the existing mocks
- clone the repo
auto-gpt-benchmarks pip install poetrypoetry shellpoetry installcp .env_example .envgit submodule update --init --remote --recursiveuvicorn server:app --reloadagbenchmark --mockKeep config the same and watch the logs :)
To run with mini-agi
- Navigate to
auto-gpt-benchmarks/agent/mini-agi pip install -r requirements.txtcp .env_example .env, setPROMPT_USER=falseand add yourOPENAI_API_KEY=. SsetMODEL="gpt-3.5-turbo"if you don't have access togpt-4yet. Also make sure you have Python 3.10^ installed- set
AGENT_NAME=mini-agiin.envfile and where you want yourREPORT_LOCATIONto be - Make sure to follow the commands above, and remove mock flag
agbenchmark
- To add requirements
poetry add requirement.
Feel free to create prs to merge with main at will (but also feel free to ask for review) - if you can't send msg in R&D chat for access.
If you push at any point and break things - it'll happen to everyone - fix it asap. Step 1 is to revert master to last working commit
Let people know what beautiful code you write does, document everything well
Share your progress :)
Dataset
Manually created, existing challenges within Auto-Gpt, https://osu-nlp-group.github.io/Mind2Web/
How do I add new agents to agbenchmark ?
Example with smol developer.
1- Create a github branch with your agent following the same pattern as this example:
https://github.com/smol-ai/developer/pull/114/files
2- Create the submodule and the github workflow by following the same pattern as this example:
https://github.com/Significant-Gravitas/Auto-GPT-Benchmarks/pull/48/files
How do I run agent in different environments?
To just use as the benchmark for your agent. pip install the package and run agbenchmark
For internal Auto-GPT ci runs, specify the AGENT_NAME you want you use and set the HOME_ENV.
Ex. AGENT_NAME=mini-agi
To develop agent alongside benchmark, you can specify the AGENT_NAME you want you use and add as a submodule to the repo