Behavioral Evaluations & EDK Guide

This guide introduces the Eval Development Kit (EDK) and details how to write, validate, run, and report on behavioral evaluations in the Gemini CLI codebase.

---

Overview

Behavioral evaluations are automated tests designed to assert on the behavior of the Gemini CLI agent (e.g., verifying which tools are called, checking call ordering, or avoiding destructive commands) rather than checking the final prose output.

Evaluating agent behavior is critical because:

1. Model responses are non-deterministic, making exact prose matching highly fragile.
2. We must ensure the model utilizes the most efficient tools (e.g., batching files via read_many_files instead of sequential read_file calls).
3. We must enforce safety boundaries (e.g., preventing execution of raw shell commands when safe alternatives exist).

All behavioral evaluations are stored under the evals/ directory.

---

EDK Developer Commands

The EDK provides CLI tools under scripts/ to help contributors audit, check, and monitor evals.

1. npm run eval:inventory

Scans all eval files under evals/, statically parses them, and provides a structured overview of what exists in the repository.

• Usage:

  npm run eval:inventory

• JSON Output: For CI integration or inventory indexing, generate a machine-readable JSON report:

  npm run eval:inventory -- --json

• Custom Root: Run against another directory or repository:

  npm run eval:inventory -- --root /path/to/other/repo

---

2. npm run eval:validate

A lint-like checker that validates eval source files against standard structural guidelines and best practices.

• Usage:

  npm run eval:validate

• Custom Scopes: Validate a specific file:

  npm run eval:validate -- evals/my-test.eval.ts

Validation Rules & Severities

Rule ID	Severity	Description
file-naming	Error	File must match *.eval.ts or *.eval.tsx naming conventions.
valid-policy	Error	Policy must be one of ALWAYS_PASSES, USUALLY_PASSES, or USUALLY_FAILS.
suite-metadata	Error	Both suiteName and suiteType must be present as static string literals.
prompt-presence	Error	Every eval case must have a non-empty prompt string.
case-name-static	Error	The case name must be a static string literal, not computed dynamically.
invalid-tool-refs	Error	All tools referenced in assertions must match known built-in or legacy tools.
positive-assertion	Error	Evaluation cases must assert on at least one tool call (e.g., check waitForToolCall has been invoked).
workspace-setup	Error	Workspace behaviors (like file-system edits/reads) must set up a files object.
new-evals-policy	Warning	New evals must not use ALWAYS_PASSES policy initially (they should be promoted after nightly data proves stability).

Warnings (new-evals-policy) will be logged with ⚠ and will not cause the CLI process to exit with status 1. Errors (✗) will block CI builds and return exit status 1.

---

3. npm run eval:report

Aggregates local vitest report.json artifacts, maps them against inventory policies, and summarizes the pass rates per model.

• Usage:

  npm run eval:report

  By default, it scans evals/logs/ recursively for report.json files.
• Specifying Directory:

  npm run eval:report -- /path/to/logs

• JSON Output:

  npm run eval:report -- --json

---

Contributor Workflow

When writing a new behavioral evaluation, adhere to this workflow to ensure high-quality, non-flaky test runs.

Step-by-Step Guide

1. Identify the Target Behavior: Determine which tool calls need verification (e.g., web_fetch must be called).
2. Author the Eval File: Create your file under evals/<name>.eval.ts naming it properly.
3. Configure Workspace Files: If the eval reads or edits files, define them inside the files metadata field.
4. Assert Behavior, Not Prose: Ensure the assert block checks tool interactions using rig.waitForToolCall or similar. Do not check final prose.
5. Run Locally:

   RUN_EVALS=true npx vitest run evals/my-test.eval.ts

6. Deflake: Run your eval at least 3 times locally to verify it does not fail due to model variance.
7. Run Validation: Run npm run eval:validate to ensure no linting errors are present.

Acceptance Criteria Checklist

• Naming: File ends with .eval.ts or .eval.tsx.
• Policy: New evals start as USUALLY_PASSES.
• Metadata: Static suiteName and suiteType (e.g. 'behavioral') are specified.
• Assertions: Uses rig.waitForToolCall or asserts tool arguments explicitly.
• Clean workspace: Does not write to files outside rig.testDir.

Common Anti-Patterns to Avoid

• Restricting core tools: Never override settings.tools.core to limit tools. Evals must run against the default toolset.
• Checking model prose: Avoid expect(result).toContain('something') since model wording is non-deterministic.
• Integration-only testing: Evals that only write files without checking realistic model prompts are integration tests and belong under integration-tests/.

---

CI & Dashboard Integration

You can easily automate behavioral evaluations or compile dashboard data using EDK’s JSON reporters.

CI Validation Block

Add a step in your PR checks or GitHub workflows to automatically lint new evals and block pull requests containing validation errors:

- name: Run Eval Validator
  run: npm run eval:validate

Publishing to a Dashboard

To record nightly performance metrics across multiple models:

1. Configure your workflow to run evaluations with the JSON reporter:

   cross-env GEMINI_MODEL=gemini-2.5-pro npx vitest run --config evals/vitest.config.ts --reporter=json --outputFile="evals/logs/eval-logs-gemini-2.5-pro/report.json"

2. Aggregate all test runs using the reporting tool:

   npm run eval:report -- evals/logs --json > aggregated_report.json

3. Upload aggregated_report.json to your dashboard storage backend to visualize pass rates over time.