Verify whether agent skills improve real-world task performance
If you want to confirm that an agent skill actually improves task outcomes before deployment, this tool is necessary. Developed by NVIDIA, SkillEvaluator checks skills in three stages, from static quality to live agent execution. This allows for an objective assessment of whether a skill is safe, non-redundant, and positively impacts agent performance.
Three-stage evaluation structure
SkillEvaluator structures its evaluation into three independent tiers. Tier 1 is a validation stage that checks if the skill is safe, well-formatted, and compatible with the agent harness. Tier 2 is a deduplication stage that detects if the content overlaps or duplicates other skills. Tier 3 is a live evaluation stage that runs actual agents to compare results with and without the skill. Each tier can be run independently, and you can selectively apply only the necessary stages.
Basic validation without API keys
The most basic Tier 1 validation can be run without external API keys, GPUs, or Docker. If uv and Git are installed, you can receive a static quality score for a skill using the skillevaluator quality-check command. This process saves deterministic quality scores and identified issues to a report. The default passing threshold is 70 points, and the example skill passes with a score of 86.2. However, since this stage is limited to static analysis, security scanners and live agent evaluations must be configured separately.
Model-based evaluation and live agent execution
For more in-depth evaluation, you can use an NVIDIA Build API key. Setting this key enables model-based rubric scores and feedback, as well as Tier 2 embedding-based duplicate detection. Tier 3 live evaluation requires a supported agent and a sandbox environment. This stage measures how the skill affects the agent's actual behavior. Costs may be incurred depending on model usage and managed sandbox usage, so you should verify the cost structure before execution.
Essential checks before deployment
This tool is currently classified as experimental, and NVIDIA does not provide enterprise support or SLAs. Therefore, it is appropriate to use it as a quality control tool in the early stages of skill development rather than applying it directly to production environments. Additionally, this project downloads and installs third-party open-source software such as SkillSpector, Gitleaks, and Semgrep, so you must review the license terms of that software before use. If a security vulnerability is suspected, it should be reported through a private process rather than a public issue.