AI checks

Use an AI check when meaning or appearance needs judgement, such as whether an error explains how to continue. You choose a model, called a judge, and give it evidence and a requirement. Keep exact facts, including a saved title, in ordinary expect checks.

Configure a judge #Link to Configure a judge

Retest's Vercel AI SDK adapter supports Anthropic, OpenAI and Azure OpenAI. Choose a provider below, install its optional packages and add the judge declaration to retest.config.ts. Replace the model or deployment name with one that accepts the evidence you plan to send.

$ npm install --save-dev ai@^7.0.127 @ai-sdk/anthropic@^4.0.71

import { env } from '@rehearsal-labs/retest'const judge = {  adapter: '@rehearsal-labs/retest/evaluation/ai-sdk',  credentials: { apiKey: env('RETEST_EVALUATION_ANTHROPIC_KEY') },  options: { provider: 'anthropic', model: 'your-model-id' },  accepts: ['text', 'images', 'frames'] as const,}

$ npm install --save-dev ai@^7.0.127 @ai-sdk/openai@^4.0.83

import { env } from '@rehearsal-labs/retest'const judge = {  adapter: '@rehearsal-labs/retest/evaluation/ai-sdk',  credentials: { apiKey: env('RETEST_EVALUATION_OPENAI_KEY') },  options: { provider: 'openai', model: 'your-model-id' },  accepts: ['text', 'images', 'frames'] as const,}

$ npm install --save-dev ai@^7.0.127 @ai-sdk/azure@^4.0.90

import { env } from '@rehearsal-labs/retest'const judge = {  adapter: '@rehearsal-labs/retest/evaluation/ai-sdk',  credentials: { apiKey: env('RETEST_EVALUATION_AZURE_KEY') },  options: {    provider: 'azure',    resourceName: 'my-resource',    model: 'my-deployment',  },  accepts: ['text', 'images', 'frames'] as const,}

Put the key in the environment variable named by env(). These integrations use API keys. Retest's own tests have called a live Azure deployment; the Anthropic and OpenAI requests were checked against local copies of those APIs. For Azure, model names your deployment, and you must give exactly one of resourceName or baseURL. The adapter reads the credential you declare rather than the SDK's own key variables.

Below the judge declaration, add the app and evaluation settings. Registering the config also lets TypeScript check judge names and evidence kinds in your tests.

import { chrome, defineConfig } from '@rehearsal-labs/retest'const config = defineConfig({  apps: { web: chrome({ baseUrl: 'http://127.0.0.1:4173' }) },  evaluation: {    judges: { visual: judge },    defaultJudge: 'visual',    timeoutMs: 30_000,    limits: { callsPerTest: 5, callsPerRun: 100 },  },})export default configdeclare module '@rehearsal-labs/retest' {  interface Register {    config: typeof config  }}

The adapter loads when a check first needs it. Ordinary tests need neither these packages nor a model key. The time limit includes judge setup, and a check cannot take longer than its test has left.

Judge the current page #Link to Judge the current page

Bring the page to a known state before asking the judge. Here, the exact title check waits for the save response; the AI check then asks about the page's presentation.

import { expect, test } from '@rehearsal-labs/retest'test('makes the saved title readable', async ({ page }) => {  await page.goto('/')  await page.getByTestId('task-title').fill('Release checklist')  await page.getByTestId('save-task').click()  await expect(page.getByTestId('saved-task')).toHaveText('Release checklist')  await test.evaluate({    requirement: 'The saved title "Release checklist" is readable below Save.',    evidence: { capture: 'screenshot' },  })})

Retest saves the screenshot, calls the judge and validates its answer. The judge receives evidence and criteria without browser tools, so it cannot act on the app. Its answer describes what the screenshot shows; use a separate check if you need to prove the server saved the task.

Choose the evidence #Link to Choose the evidence

A screenshot is useful for visual requirements. For wording alone, you can supply text directly. This example judges the task app's failure message against a requirement for helpful errors.

await test.evaluate({  requirement: 'The message explains why saving failed and what to try next.',  evidence: {    text: 'Could not save',    label: 'Save response',  },  mode: 'advisory',})

This judges the supplied words rather than reading the page. advisory records a warning when the check does not pass, so you can assess a new requirement before making it mandatory. Keep an ordinary assertion in the test because advisory checks do not count as assertions.

You can pass several evidence items in an array and name an app for captures in a test with several apps. To include captured logs, select { diagnostics: ['console', 'network'] }. The judge must accept text, and no diagnostics reach it unless you select them.

Check a recorded step #Link to Check a recorded step

For something that happens over time, enable recording and wrap the work in a named step. A frame check selects the latest started step with that name. If the step is still running, its interval ends at the moment of the check.

await test.step('Save the task', async () => {  await page.getByTestId('save-task').click()  await expect(page.getByTestId('saved-task')).toHaveText('Release checklist')})await test.evaluate({  requirement: {    saved: {      kind: 'state',      requirement: 'The last frame shows "Release checklist" below Save.',    },  },  evidence: { recording: { step: 'Save the task' } },})

The judge must accept frames. Choose the criterion kind for what you want to learn from the recording:

• State. state judges the last frame held for the interval. Use it to check the screen at the end of a step.
• Seen. seen asks whether something appeared in a captured frame. Without a witnessed appearance, the result is inconclusive.
• Never. never forbids an appearance. A cited frame showing the forbidden event can fail the check even when capture is partial.

Partial frames cannot support a pass. Even complete capture consists of samples, so it cannot prove that a brief event was absent between frames. Use an ordinary assertion or a recorded event when the outcome needs that proof.

Read the verdict #Link to Read the verdict

Checks are required unless you choose advisory. Every criterion must pass for a required check to pass, and catching its thrown error does not remove its recorded verdict.

A failed capture or a judge that cannot decide leaves a check inconclusive. A missing key, missing package, expired time limit or invalid answer is an error.

The failure card and report show the criterion, the judge's explanation and the evidence it cited. Each evaluation record keeps the provider and model, evidence references, latency and token counts when returned. retest inspect --test places the check in the timeline.

Support and limits #Link to Support and limits

A check that exceeds a call or input limit ends with evaluation_error. Read Configuration options before adding many checks or large evidence sets.

A frame check on an app that was not recorded is an error. If recording was requested but failed, the check is inconclusive. Evidence forbidden by the pixel policy also produces an error.

Screenshot evidence works across browser engines and in Electron and native apps. Live native diagnostics are unavailable to AI checks. Screenshots sent to a judge cannot be cropped or masked; the pixel policy still applies.

Next steps #Link to Next steps

• ChecksUse exact assertions for facts your test can measure.
• Recordings and evidenceRecord a run and understand capture gaps.
• Configuration optionsLook up judge settings and evaluation limits.