Score legal agents on real-world M&A data room tasks
You can see how accurately AI agents follow instructions in realistic legal environments, such as M&A data rooms. This is a benchmark specialized for legal work, used to evaluate and improve agent capabilities. It scores the entire process of an agent reading documents and producing deliverables, rather than focusing on complex theory.
Validation via M&A data room scenarios
This tool evaluates the process of completing realistic M&A data room tasks, not just simple question-and-answer interactions. It tracks the entire workflow where an agent reads instructions, analyzes relevant documents, and produces the required deliverables. Each task includes the instruction given to the agent, the reference documents, and a rubric for grading. This allows you to determine how reliable the agent's results are in a real legal environment.
Execution harness and grading system structure
The tool consists of two parts: a task dataset containing agent instructions, documents, and grading criteria, and an execution harness that runs the agent and evaluates the results. The harness provides the environment for the agent to perform tasks and grades the generated output against predefined criteria. Specifically, when grading .docx format outputs, you must install pandoc CLI 3.5 or higher on your system. This setup separates agent execution from evaluation to ensure reproducible results.
Using it as a library
The execution harness and grader are provided as a Python package called lab-core. Installing this package allows you to run and evaluate agents outside the repository. During installation, you need to set the LAB_ROOT environment variable to the path of the task dataset. You can also use various features via the python -m lab_core.… command. This approach makes it easy for researchers or developers to connect their own agents to this benchmark.
Things to check before use
This project is in a continuously evolving stage, with task sets and the execution harness being added and improved regularly. Therefore, it is important to use the latest version. If you want to grade .docx files, you must verify that pandoc is installed. Additionally, because this tool is specialized for legal work, it may not be suitable for evaluating agents in other fields. If you use it for research purposes, it is recommended to follow the citation format provided in the README.
