Suites

Group the tests you run together, such as the ones a release depends on, and run them with one press.

A suite is a named group of tests that you run together, such as the tests Parcel's checkout and saved cards depend on before a release. It belongs to a project and can hold tests of any of the project's apps, up to 200 of them. A GitHub rule, a schedule and the CLI can each run a suite, so it is the usual way to name the tests that matter.

Anyone who can view the project sees its suites. Creating, changing and running one needs the Manage projects permission.

Create a suite Link to Create a suite

  1. Step 1: Press New suite

    Open Tests. The sidebar lists the project's suites under Suites, with New suite at the end of the list.

  2. Step 2: Name it

    Give the suite a name, here release, and if you like a description of what it covers. A name can be used once in a project, whatever its capitals, so a second Release is turned away. Press New suite, and the suite's page opens.

  3. Step 3: Add tests

    Press Add tests. The list holds every test in the project, with a search box over it. Pick Check out a cart, Save a card and the others the release needs, then press Save.

You can also start from Tests. Tick the tests and press Add to suite, or use Add to suite in the menu of a test's row or page, then pick the suite or New suite. A suite you make that way already holds the tests you chose.

A test written to reproduce a bug is marked Reproduction and joins a suite only once the bug's fix is verified and it becomes a Regression test. An archived test can never be added.

Run a suite Link to Run a suite

Open the suite and press Run suite. The dialog has no choice of screens, because each test in a suite runs on the screens the suite keeps for it, and a test you add starts on desktop. If an app in the suite has more than one address, the dialog asks which one to use, as in Run tests.

The run writes down the suite's tests when it starts. While it is still going, the suite page says so, and a change you make to the suite in the meantime applies to the next run.

Tests that are not ready Link to Tests that are not ready

The line at the top of the suite page counts its tests and how many are ready, such as "14 tests, all ready." When some are not, an amber notice takes its place and says that a run of the suite ends Blocked until those tests are fixed or removed. Each test that is not ready shows its state, such as Verifying or Could not verify, in the list below.

Good to know

A suite never passes on fewer tests than it holds. A test that is still being verified keeps the whole run from passing, so before you rely on a suite in a pull request or a release script, check that the line at the top of its page says every test is ready.

The suite page Link to The suite page

  • Tests in this suite lists each test with its state and its latest results. The cross at the end of a row takes the test out of the suite and keeps it in the project.
  • Recent runs shows the suite's newest five runs, and View all opens Runs for this suite alone, so last night's scheduled run is one press away.
  • Schedule holds the times the suite runs by itself, as Scheduled runs explains.
  • Rename sits at the top beside Run suite, and Remove suite at the foot of the page.

In the sidebar, each suite shows how many tests it holds, and Scheduled or Paused when it has a schedule. Not in a suite lists the tests that no suite runs, which is a quick way to find a new test you have not added anywhere yet.

Run a suite from elsewhere Link to Run a suite from elsewhere

A GitHub rule can run a suite on every pull request, as Rules for pull requests and deployments shows. From your terminal or a CI job, rehearsal suites list gives each suite's identifier and how many of its tests are ready, run runs it, and run wait follows the run until it ends:

Terminal
rehearsal suites listrehearsal run --suite <suite-id> --environment production --request-key release-1842rehearsal run wait <run-id> --timeout 30m

A script has to name its own --request-key, such as a build number, and run wait stops after 60 seconds unless --timeout sets a longer wait. A suite that runs on more than one screen starts one run for each, so a script waits for every one of them. Block a release that breaks shows the whole gate. The CLI reads suites but does not change them, so you make and edit them in the app.

Remove a suite Link to Remove a suite

Remove suite asks you to confirm, then removes the suite and stops its schedules. The tests stay in the project, and the runs the suite started keep their results.

Next steps Link to Next steps