Claude Code release notes: 2.1.269 adds plugin evals

Claude Code 2.1.269 adds claude plugin eval, which tests a plugin against repeatable cases and compares the results with runs that do not load the plugin. Released on September 11, the feature produces scored results with JSON and HTML reports.

A plugin is an extension that packages capabilities for Claude Code. These tests check how the extension affects Claude's work, rather than just whether its files have valid syntax.

Sources: Claude Code 2.1.269 release notes. Anthropic's plugin eval guide.

How Claude Code plugin evals work

Each test case contains a prompt and graders that return a pass or fail. A grader can check the reply's text, whether Claude called a particular tool, or whether the response meets criteria judged by another model.

By default, each case runs three times with the plugin and three times without it. The report shows both scores and their difference. A high score in both groups does not show that the plugin improved that task.

Source: Anthropic's explanation of eval runs and the no-plugin baseline.

Create and run an eval suite

Human step: Use Claude Code 2.1.269 or later and open a terminal at the root of a working plugin. Run:

claude plugin eval init

Claude reads the plugin, asks what a good result should contain, proposes cases and graders, pilots them, and writes the files under evals/. Then leave the interactive session with /exit or Ctrl+D.

Back at your terminal, run:

claude plugin eval . --no-publish

The flag keeps the report local instead of publishing it. Open the HTML file at the path after Report: and inspect which graders failed.

Source: Anthropic's steps for creating and running the suite.

Cost and current version

Eval runs, model-based grading, and the interactive setup use your account's model access and count against your plan's limits or API bill. Tools beyond the default read-only set require an explicit grant.

Version 2.1.270 was the latest on npm when checked on September 14. It fixes a regression introduced in 2.1.269 where read-only Git commands could unexpectedly ask for permission after a session had been running for a while.

Sources: Anthropic's requirements and tool permissions. Claude Code 2.1.270 release notes. Claude Code in the npm registry.