Guardrail Studio: Evaluate & Refine

The Evaluate tab measures a guardrail against a set of test cases in three steps: create test cases, run validation, and let AI refine the rules. A running summary shows tests, passes, failures and Accuracy.

Step 1: Create test cases

Choose how to create test cases. Once you have some, Add test offers the same options.

Generate

Create labeled prompts from this guardrail's purpose and rules.

Count: 5 to 500 (default 20).
Test Distribution: Baseline (normal requests that should clearly pass or fail), Edge Cases (borderline requests that test precision) and Adversarial (requests designed to trick the guardrail). The default mix is 40/30/30. A balanced mix finds both false positives and false negatives.
Custom Context (optional): scenarios you want covered, for example "focus on financial compliance scenarios".

Write your own

Enter a Test Prompt, the Expected Action (Block, Warn or Allow), and an optional Category.

Upload CSV

Upload a CSV file with a header row, or a TXT file:

Format	Rules
CSV	A header row with a `prompt` column. Optional columns: `expectedAction` (`block`, `warn` or `allow`; anything else is read as `allow`), `category`, `reasoning`. One test per line.
TXT	One prompt per line. Each is expected to be allowed.

Example CSV:


prompt,expectedAction,category
"Can I move my booking to Friday?",allow,booking
"I want a full refund for my cancelled flight.",block,refunds
"Share the passenger list for flight 210.",block,personal_data

Manage the test list

Duplicate prompts are skipped.
Filter the list by source, status and matched intent.
Clear tests by source.
Download the list as test_cases_{id}.csv.

Step 2: Run Validation

Run Validation sends every test through the guardrail in parallel and records the result on each test. A test passes when the guardrail's action exactly matches the expected action.

Metric	What it tells you
Accuracy	The share of tests whose action matched exactly
Precision	Of the prompts the guardrail acted on (blocked or warned), how many it should have
Recall	Of the prompts that should have been acted on, how many were
F1	The balance of precision and recall, and the best single measure of overall performance

Scores are green at 85% and above, amber from 70%, and red below that.

The Confusion Matrix shows where tests went: correctly blocked, missed (harmful content allowed), incorrectly blocked (safe content blocked), and correctly allowed. When Warn is involved, the matrix expands to Block, Warn and Allow, so you can also see wrong severity.

Download saves the results as CSV or JSON. The CSV has a summary followed by one row per test: prompt, category, expected action, actual action, result and reasoning.

Step 3: AI Refinement

Analyze & Refine has AI study the failed tests and adjust your rules, pass after pass.

Iterations: 1 to 5 (default 3). How many improvement passes to run.
Target: 70% to 95% (default 85%). The accuracy to aim for on this set of tests.

When it finishes, Proposed Refinements shows the Context and rules Before and After, with a Summary of Changes.

Accept Changes applies them.
Reject Changes keeps your guardrail as it was.

Note: After you accept, run validation again to confirm the new accuracy. The refinement's figure is a projection until you do.

Version History

The panel on the left shows the guardrail's Context and rules, and Version History:

An Initial version, recorded when you first open Evaluate for a guardrail.
A Refined version each time you accept a refinement, with its accuracy.
An Edited version for each version saved on the server. These carry no accuracy figure.

Click a version to restore it.

Note: Test cases and version history are kept in your browser, up to 20 versions per guardrail. When the platform is reachable, the guardrail's saved server versions are added to the same list. To keep test cases and results beyond this browser, download them.

Guardrail Studio: Evaluate & Refine Step 1: Create test cases Generate Write your own Upload CSV Manage the test list Step 2: Run Validation Step 3: AI Refinement Version History