Run tests

Start a run of one test, several tests or a suite from the app, and choose the screens and the address it checks.

A run checks a set of your saved tests against one address of your app and keeps a report of what each test did. This page shows how to start one from the app, for one test, a few, a whole suite or the tests an exploration has made. To run tests from a script or a CI job, see Block a release that breaks, and to run them on every pull request, see Rules for pull requests and deployments.

Every way of starting a run opens the same dialog. It tells you what will run, lets you pick the screens and, when an app has more than one address, asks where to run it. Run then starts the run and opens its page, where the results arrive as each test finishes. Starting a run needs the Manage projects permission.

Where a run starts Link to Where a run starts

  • One test. In Tests, point at the test's row and press Run test at its end, or open the test and press Run test at the top of its page.
  • Several tests. Tick them in Tests and press Run selected. They go into one run, in the order the list shows them.
  • A suite. Pick it under Suites in the Tests sidebar and press Run suite. Suites shows how to make one.
  • New tests. Once the tests you created from an exploration are ready, the Test cases card in the panel beside the chat offers Run once now, which runs every ready test once. A test made in the chat has Run now on its card.
  • A run you already have. Press Run again on its page, or tick several runs in Runs and press Run again.

You can also ask Re:agent in the chat, for example to run the release suite, and it starts the run with your own permissions.

Start a run Link to Start a run

Here Parcel's web app has two addresses, its production site at https://parcel.example and http://localhost:3000 on your computer, and you run two of its tests.

  1. Step 1: Choose the tests

    Open Tests, tick Check out a cart and Save a card, and press Run selected. The dialog opens as Run 2 tests, and its first line counts the work once Rehearsal has checked the selection:

    Output
    This run checks 2 tests on 1 screen: 2 test runs.

    A test run is one test checked once, on one screen in one browser, and it is what your plan counts. The dialog names a price only when the run would go past what your plan includes, and Plans and runs explains the rest.

  2. Step 2: Choose the screens

    Desktop is picked when the dialog opens. Add Tablet or Phone to check the same tests at those sizes as well. Rehearsal starts one run for each screen, so two screens give you two runs with their own reports.

    ScreenSize in pixels
    Desktop1440 × 900
    Tablet834 × 1194
    Phone390 × 844

    Tablet and phone are Chromium at that size with touch turned on, not real devices. Tests run only in Chromium, so if the dialog also offers Browsers, keep Chromium. A test run in Firefox or WebKit ends Blocked.

  3. Step 3: Choose the address

    An app with one address is tested there without a question. For an app with several, the dialog shows the app's name over a list of its addresses, such as Production · https://parcel.example, and you pick one. A run tests each app at one address.

    An address on your computer stays greyed out until your computer is connected, because Rehearsal's browsers reach it through the CLI. Run rehearsal connect web http://localhost:3000 and keep it open until the run ends, as Test an app on your computer shows.

  4. Step 4: Press Run

    The button says Starting while Rehearsal records the run, then Run 43 started, and the dialog closes onto the run's page. When you picked more than one screen, it opens the first of the runs, and the others wait beside it in Runs.

The run's page reads the run again every few seconds until it ends, so you can leave it open and watch the results arrive. Read a run explains what you see there.

Which tests a run checks Link to Which tests a run checks

A run checks only tests that are ready. A new test becomes ready once Rehearsal has checked it, as Create tests explains, and until then the Tests list marks it Verifying or Could not verify. A test you change stays ready, because runs keep using the newest version that passed its checks until the new version passes them too.

When a run starts, it writes down each test in the newest version Rehearsal has verified, together with the address it will use. A change you make to a test or a suite after that belongs to the next run. The tests then run one at a time, in order, and each one has three minutes to finish.

Good to know

A test that is not ready still goes into the run when you choose it, and the dialog warns you, for example "1 test is not ready and uses no test runs." The run lists that test as Not ready without checking it, and the run ends Blocked, or Failed if another of its tests failed, because a run never passes on fewer tests than you asked for. If none of the tests is ready, nothing starts.

Run again Link to Run again

Run again repeats a run exactly, with the same test versions at the same addresses on the same screen. That makes it the way to check a fix to your app against the tests that failed. To try a newer version of a test or another address, start a new run instead.

Ticking several runs in Runs and pressing Run again repeats each of them, and the dialog says how many started. The runs Rehearsal makes to check a new test are its own, so they have no Run again.

Stop a run Link to Stop a run

While a run is queued or running, its page shows Stop in place of Run again. Once you confirm, the run ends Cancelled straight away. The test that was running records no result, so it says No result recorded, like each test the run never reached.

When a run cannot finish Link to When a run cannot finish

If a problem on our side stops a run, Rehearsal picks it up again, keeps the results it already has and runs the rest. A run that still cannot finish ends Could not run. That failure is ours, so the run counts towards nothing, as Runs we could not finish explains.

Runs without the app Link to Runs without the app

The same runs start from your terminal, which is how a release script or a CI job waits for the result before it ships:

Terminal
rehearsal run --suite <suite-id> --environment production --request-key release-1842rehearsal run wait <run-id> --timeout 30m

A script names its own --request-key, such as a build number, so that sending the command again returns the run it already started instead of starting a second one. run wait stops after 60 seconds with exit code 7 unless --timeout sets a longer wait, so allow as long as your tests take. Block a release that breaks shows the whole gate, with signing in from CI, choosing the tests and the address, and the exit codes. A GitHub rule runs your tests on each pull request and posts the result as a check, as Rules for pull requests and deployments shows, and a schedule runs a suite at the times you choose, as Scheduled runs shows.

Next steps Link to Next steps