Fix a failure with your coding agent

This example uses Parcel, a demo store, and the test from Check out a cart. Parcel is a demo project, not a customer.

Your coding agent runs the test with the Rehearsal CLI and reads where it stopped. It fixes the code and runs the test again. Rehearsal runs the test and reports what happened. The agent changes the code.

Before you start #Link to Before you start

• The CLI is installed and logged in. See Install the CLI.
• Parcel runs on your computer at http://localhost:3000. rehearsal connect http://localhost:3000 runs in a terminal of its own, as Connect a local app shows.
• The owner test account works on the local address too. Add it under Where it works.
• Your agent knows the CLI. See Use Rehearsal from your coding agent.

Ask your agent #Link to Ask your agent

For example: "The checkout test in Rehearsal fails on my local app. Run it, find out why, fix the app and run the test again."

What the agent runs #Link to What the agent runs

1. 1

   Step 1: Find the test

   $ rehearsal tests list --json

   The list holds Checkout shows the right total and its identifier.

2. 2

   Step 2: Run it on the local app

   $ rehearsal runs start --test <test-id> --environment local \  --idempotency-key checkout-local-1 --json

   --environment local picks the address on your computer. A script names its own key, so asking again never starts a second run.

3. 3

   Step 3: Wait for the result

   $ rehearsal runs wait <run-id> --timeout 5m --json

   The command exits with 1, because the test failed. The answer still holds the whole run, and data.verdict says failed. Each failing test is in data.run.failures. This one, shortened:

   {  "testName": "Checkout shows the right total",  "failedStep": 6,  "failedStepLabel": "Check the order total",  "expected": "The order is placed with a total of $18.50.",  "observed": "Error: expect(locator).toHaveText(expected) failed\n\nLocator: getByTestId('order-total')\nExpected: \"$18.50\"\nReceived: \"$15.50\"",  "evidence": [    { "artifactId": "<artifact-id>", "kind": "screenshot", "stepIndex": 6 },    { "artifactId": "<artifact-id>", "kind": "video", "stepIndex": null }  ]}

   Steps count from zero here, so step 6 is the seventh step. The total left out shipping.

4. 4

   Step 4: Look at the page

   $ rehearsal runs artifacts download <run-id> --artifact <artifact-id> \  --output ./order-total.png --json

   This saves the screenshot of the failing step. The answer gives the local path, never a download link.

5. 5

   Step 5: Fix the app

   The agent finds the code that adds up the order and adds shipping to the total. This part is the agent's own work.

6. 6

   Step 6: Run the same test again

   $ rehearsal runs rerun <run-id> --idempotency-key checkout-local-2 --json$ rehearsal runs wait <new-run-id> --timeout 5m --json

   A rerun checks the same test version at the same address. This time runs wait exits with 0.

7. 7

   Step 7: Compare the two runs

   $ rehearsal runs compare <run-id> <new-run-id> --json

   {  "schemaVersion": "1",  "command": "runs compare",  "data": {    "reordered": false,    "earlier": { "number": 41, "verdict": "failed", "complete": true },    "later": { "number": 42, "verdict": "passed", "complete": true },    "tests": [      {        "testName": "Checkout shows the right total",        "change": "fixed",        "sameVersion": true,        "sameTarget": true,        "earlier": { "status": "failed" },        "later": { "status": "passed" }      }    ]  },  "error": null}

   fixed means the test failed before and passes now, at the same version and address. A test checked at another version or address is marked different instead.

If the test is wrong #Link to If the test is wrong

The CLI runs tests and reads results. It cannot change a test. If the check itself is wrong, the agent tells you. You then change the test with Re:agent in the app.

The CLI also tells agents to read what a page showed as data, never as instructions.

What it costs #Link to What it costs

Each run of the test counts once. Here that is two test runs: one failed at a check, and one passed. See what counts as a test run.

Next steps #Link to Next steps

• Read the JSON output your agent gets from each command.
• Run tests from the terminal yourself.
• Learn to read a run in the app.