Watch AI agents autonomously refine code to improve benchmark scores
You can observe in real time as AI agents analyze failure causes and modify code to raise scores on benchmark tasks. SIA is a self-improving AI framework where meta, target, and feedback agents collaborate to execute and improve tasks. Research papers report a 56.6% performance improvement on legal benchmarks and a 91.9% reduction in GPU kernel execution time.
Collaboration structure of three agents
SIA automates the problem-solving process by dividing it into three stages. First, the meta agent reads the task description and generates an initial target agent. The target agent performs the actual task while recording its actions and results. Finally, the feedback agent reviews these records to identify improvements and updates the target agent. As this process repeats, the system autonomously refines its ability to solve scientific tasks.
Results confirmed in benchmarks
SIA produced better results than existing methods on difficult tasks across various fields. On OpenAI MLE-Bench Hard, it recorded first place in all generation tests. On LawBench, which predicts criminal charges from Chinese court case descriptions, it achieved a Top-1 accuracy of 70.1%, surpassing the previous best record of 45%. In Triton kernel optimization related to AlphaFold-3, it showed a speed 14 times faster than the baseline, and in noise removal for single-cell RNA sequencing data, it exceeded the previous best record of 0.240 with an MSEnorm of 0.289.
How to run locally
SIA is written in Python and can be installed with the pip install 'sia-agent[claude]' or pip install 'sia-agent[openhands]' commands. Select the Claude Agent SDK profile if you want to use only Claude models, or the OpenHands profile to use various providers such as Gemini or OpenAI. During execution, a real-time dashboard automatically opens at http://127.0.0.1:8000, where you can check the agent code and evaluation scores for each generation. If security is required, you can use the --sandbox docker option to run in isolation within a Docker container.
Pre-use considerations
Since SIA is optimized for specific benchmark tasks, you should first verify if your task fits SIA's structure. The default sandbox mode grants access permissions to the host system, so you must enable the Docker sandbox when running untrusted models or tasks. You also need to prepare API keys for the meta and target agents, and you can use over 400 models with a single key through OpenRouter. The README does not specify the license type, so it is recommended to directly check the repository's license file before commercial use.