Read a run

Find out why a test passed or failed from its steps, recording, screenshot, trace and console, and what each outcome means.

Every run keeps a report of what happened. It says how the run ended, and for each test it shows the steps, a recording of the browser, the page where something went wrong and what the page logged. This page walks through that report, then explains every outcome, so you can tell a problem in your app from a test that needs changing.

Find a run Link to Find a run

Runs lists the project's runs, newest first, each with its number, such as Run 42, its tests and when it started. The pills at the top count them by where they stand, such as Failed or Running, and Filter narrows the list by test, suite, environment, how a run started and when. The runs Rehearsal makes to check a new test are left out until you pick Checked by Rehearsal under How it started.

A test's own page shows its last ten runs as a row of marks, one colour for each outcome. Press a mark to read that run on the test's page. Recent runs under the recording lists the newest five, and View all opens Runs for that test alone.

The top of the report Link to The top of the report

The heading names what ran: the test, the suite with its number of tests, or a count such as 3 tests, with the run's number beside it. Under it, one sentence says how the run ended, when it was started, the step where a single test stopped, how long the tests took and how the run started:

Output
Failed 2 hours ago at step 4, took 41s, started manually

When a run ended Blocked or Could not run, an amber notice under the sentence says why. A line of facts under the player names the browser, the environment and the address the run used, and the commit for a run GitHub started. A repeated run links to the run it repeated, and a run that checked a bug names the bug.

The buttons at the top are Run again, or Stop while the run is going, and Open test. The menu beside them holds Edit in chat, Download the trace and Console log, which all belong to the test you are reading.

A run of several tests Link to A run of several tests

When a run holds more than one test, they are listed beside the player, the failures first, then the tests that need an update or were blocked. The report opens on the first failure, or on the first test when nothing failed, and you press another test to read its recording and steps. A test that was never checked stays on the list with Not ready and the reason, or with No result recorded when the run ended before its turn.

The steps Link to The steps

The steps stand beside the recording, numbered across the whole test and, when it visited more than one page, grouped under the title of each page. A step that checks something carries a small checklist mark and keeps its words bright, so the checks stand out from the clicks and typing around them. Press any step and the recording jumps to it.

A step shows how long it took only when it was a second or more slower than the steps around it. Point at any other step to see its time.

When a step fails Link to When a step fails

The step that failed is open in the list. When the test compared what it expected with what was on the page, one sentence says both, and under it is the line the test runner itself reported, word for word. Show the screenshot puts the page as it looked at that step in place of the recording, and Show the recording switches back.

The recording Link to The recording

The player shows the browser as the test used it, with play, replay and a choice of speed. The bar under the picture has one stretch for each step, as wide as the step took and coloured by how it ended, so a failure is easy to spot. Point at the bar to see which step is where, and click it to jump there.

While a test is still running, the page reads the run again every few seconds, and the recording appears once that test has finished. Moments when sign-in details were being typed are left out of the video and long pauses are shortened, and a line under the Console and Network cards says when either happened.

Console and network Link to Console and network

Under the recording, Console lists the warnings and errors the page logged while the test ran, and Network lists the requests that failed or answered with a status of 400 or more. Each card shows the newest twenty and says how many there were in all. Press the step beside an entry to move the recording there.

Download the file on each card saves everything the run kept, including the messages and requests the card does not show. When a file could not be kept, a sentence under the cards says so, such as "The recording could not be saved."

The trace Link to The trace

Download the trace in the run's menu saves one web page for the test you are reading. It lists every step with its kind, how it ended, how long it took and the page and address it ended on. Open it in any browser, or attach it to an issue for someone without access to Rehearsal.

What each outcome means Link to What each outcome means

A run has one outcome, and each test in it has its own. The run's outcome comes from its tests:

RunWhat it means
PassedEvery test the run was asked to check passed.
FailedAt least one test failed or needs an update.
BlockedNo test failed, but Rehearsal could not check at least one, because it was not ready when the run started or something it needs was missing.
CancelledThe run ended before every test had a result, usually because someone pressed Stop.
Could not runA problem on our side stopped the run. It counts towards nothing.
Queued and RunningThe run is waiting to start, or its tests are under way.

Each test in the run ends in one of these:

TestWhat it means
PassedThe test took every step and every check held.
FailedA step or a check did not work. The step that failed is open in the list.
Needs updateThe test could not find something on the page that it acts on, such as a button or a field. The run counts it as a failure.
BlockedRehearsal could not check the test, and the reason is under its steps.
Not readyThe test was not ready when the run started, so the run did not check it. Its line names why, such as a test Rehearsal had not verified yet or one that was archived.
No result recordedThe run ended before this test's turn.

Good to know

A run passes only when every test it was asked to check passed. A test that was not ready, or never got its turn, keeps the whole run from passing, so a run marked Passed always means everything you asked for was checked.

A test that needs an update Link to A test that needs an update

Needs update asks a question only you can answer, so Rehearsal never decides it for you. The test looked for something it uses, such as a button, and the page no longer had it where the test expected. If your app changed on purpose, open the run's menu, press Edit in chat and tell Re:agent what changed, and what it saves becomes the test's next version. If the change was a mistake, fix your app and press Run again.

The CLI calls this outcome "a test needs review", and run wait exits with its own code for it.

A blocked test Link to A blocked test

A blocked test is one Rehearsal could not try, so it says nothing about your app. The sentence under its steps names the reason, such as a test account or an upload file that is no longer there, a connection to your computer that closed during the run, or a browser other than Chromium. Fix what it names and run the test again.

Could not run Link to Could not run

Could not run is our failure, never yours. Rehearsal tried the run again before it gave up and kept any results it had, and the run counts towards nothing, so run it again when you are ready. Runs we could not finish has the details.

A GitHub check uses the same words for something else. On a check's list of tests, Could not run marks a test that was blocked, the one this report calls Blocked. Only a check titled "Could not run these tests. You were not charged." means the run itself ended Could not run, as Checks on commits shows.

Emails about runs Link to Emails about runs

When a run fails, is blocked or could not run, Rehearsal emails every active member of the workspace. When a later run at the same address passes after one that did not, they all get an email saying so. Rehearsal's own checks of a new test send nothing, and scheduled runs send alerts of their own, as Scheduled runs explains.

No screen turns these emails off. The Email chat and schedule alerts switch in Settings, Notifications, Email covers chats and schedules, so it does not stop the emails about runs.

Next steps Link to Next steps