EvalSutra AI

What it grades

From the EvalSutra README. npm published 0.3.0 on 2026-11-07. The git package.json the same day said 0.3.1. The GitHub Pages site returned 404.

• It grades recorded observations: final output, tool calls, and latency. It does not run the agent.
• Missing evidence is inconclusive, never a pass. A model opinion cannot prove success.
• The README's bad-demo case fails because a forbidden tool, read_private_key, was called and a canary leaked. The latency case is inconclusive because no latency was recorded.
• You get report.html, report.md, report.json, and junit.xml.
• Needs Node.js 20 or newer. The README says 0 runtime dependencies, no API key, no account, and no network.

Assertions, statuses, and the gate

• 11 assertions: equals, normalized_equals, contains, not_contains, regex, json_has_keys, required_tools, forbidden_tools, tool_order, max_steps, max_latency_ms.
• Statuses: pass, fail, inconclusive, error, skipped, not_supported.
• model_judge and unverified_claim sources cannot prove a pass.
• Gates: NO_GO, REVIEW, GO_CANDIDATE. GO_CANDIDATE means the recorded evidence met your thresholds. It is not "safe to ship".
• compare returns PASS, FAIL, or INCONCLUSIVE.
• Exit codes: 0 ok, 1 error, 2 NO_GO or regression or audit fail, 3 REVIEW or inconclusive, 4 invalid evidence.

The 18 skills

The README says install writes only inside the project, never overwrites, and refuses symlink escapes. Only the file copy is tested.

• prime plans and holds a conservative gate. bench builds suites. truth checks trusted references.
• trace checks tool calls, order, and permissions. task checks the end state, not a polished reply.
• guard builds synthetic injection and secret-handling tests. robust covers timeouts, retries, and recovery.
• drift compares a baseline with a candidate. port checks IDE and OS compatibility.
• judge holds rubrics and must not pass alone. cost tracks latency, tokens, and units.
• data covers provenance, licence, and leakage. arena is a paired comparison. explain gives a root cause with evidence.
• audit checks grader validity. verify is an independent re-run. report writes the reports. pluginwatch is a static plugin review.

Not built, and not verified

• Not implemented: live agent runner, sandbox, server, dashboard, SARIF, OpenTelemetry, Promptfoo, Inspect AI, Harbor, MCP Inspector adapters, a calibrated LLM judge, audit-plugin, and JSON Schema files.
• Unverified: skill discovery inside live Cursor, Codex, Gemini CLI, and Copilot, and Linux or macOS results beyond the public CI history.
• The README says the grader had 0 false positives and 0 false negatives on 36 known-answer cases, the audit is a self-check, and the data is synthetic.
• No regulatory compliance is claimed. Trademark clearance for the name has not been performed.