Run your tests faster with parallel cucumber….

Here in the JUST EAT web team, we have quite a few UI tests to run on each build we produce. We know that in order for these tests to remain useful to the team, it’s important to keep the speed of execution to a minimum. There are a few ways that we can improve speed of execution for our tests. One approach is to not have many UI tests, and to write fast running, in memory, integration tests instead (this is something we’re currently working on – I’ll write a new blog post about it in a few months)! Another approach is to parallelise the UI tests. Recently, we’ve been experimenting with the parallel_tests gem in Ruby to do just that.
First of all you need to install the parallel_test gem in your directory. You can find more info here: GitHub. Once you’ve successfully installed the parallel_tests gem, then you’re ready to roll.

Run features in parallel

The following basic command will run all the features within the ‘features’ directory:

$ bundle exec parallel_cucumber features/

But if you want to run parallel_tests using a specific profile or tags, here’s an example of how your can run your tests:

$ bundle exec parallel_cucumber features/ -n 6 -o '-t  @smoke -t ~@manual'

You can specify the number of processes to speed up tests executions by using option ‘-n’, as in the above example. By default, it uses 4 processes.

Outcome of the parallel tests

The following reports indicate the summary of the test execution before and after implementing parallel_tests in our project. We were able to reduce the test execution time from 45% to 62% as a percentage for different test builds.
The test execution time varies with each test builds and also depends on few factors  such as…

  • number of feature files
  • number of tests in each feature file
  • number of headless and browser tests in each build

Overall, we’ve reduced the test execution time by 56% on average.
Before
old_final
After
new_final

Generating HTML Reports

We can generate HTML report for the each process. We chose to create a profile called ‘parallel’ in our cucumber.yml file as follows:

parallel: --format html --out report<%= ENV['TEST_ENV_NUMBER']%>.html

This will create HTML report with name as report.html, report2.html and so on. The variable ENV[‘TEST_ENV_NUMBER’] gives details about all the processes that cucumber scenarios are executing.

Re-running Failed Scenarios

This section describes how you can re-run only the failed scenarios with parallel cucumber.

Set up

Update your existing parallel test profile to log failing scenarios in cucumber.yml as follows:

parallel: --format html --out report<%= ENV['TEST_ENV_NUMBER']%>.html --format ParallelTests::Cucumber::FailuresLogger --out cucumber_failures.log

This will create cucumber_failures.log with all the failed scenarios which we can use to rerun.
Now, run your features with that ‘parallel’ profile:

$ bundle exec parallel_cucumber features/ -n 6 -o '-t %tags% -t ~@manual ENVIRONMENT=%environment% -p parallel'

This will run your entire features in six different processes and will log the failed scenarios in the ‘cucumber_failures.log’ file.

Re-run failed scenarios

Since we have ‘cucumber_failures.log’ file with all the failed scenarios, we can use that file to re-run failed cucumber scenarios.

$ bundle exec cucumber @cucumber_failures.log --format html --out test_results_rerun.html ENVIRONMENT=%environment%

test_results_rerun.html gives the report of re-running failed scenarios. Also, you need to make sure you delete cucumber_failures.log before each test execution.
I’ve created the following powershell task (as the cleanup task) in the TeamCity to delete the log file before the execution of parallel test command.

$filePath = "$scriptDirectory\cucumber_failures.log"
if (Test-Path $filePath) {
 Remove-Item $filePath
}

Here’s an example of how the steps have been executed for smoke tests in the TeamCity…
TC_configuration
But you can create a separate rake tasks to run parallel tests, rerun failing tests and cleanup failure logs depend on your work environment.

Summary

There’s a couple of things to bear in mind with this tool. The first – the best way to speed up your UI tests is to not have many UI tests! However, if you are going down this route, then you need to keep in mind that one feature will run on one process. This tool will not break down your scenarios which are written in one feature file, into many processes, so you don’t get much control over how the feature files are split into processes unfortunately.
 
Thanks for reading!
Deepthi

  1. Did you manage to get Cucumber test failures outputting to Team City’s Tests tab?
    Thanks,
    Sam

    1. Yes. As I mentioned above when you run the tests with ‘parallel’ profile (including ‘–format ParallelTests::Cucumber::FailuresLogger –out cucumber_failures.log’), it will create the cucumber_failures.log in the cucumber directory. Then make sure to select the cucumber directory as the Artifact paths: (under General Settings ) in the Team City. It will create the cucumber_failures.log as a output (Artifacts) in the test tab.

    2. Sam, I also didn’t see it at first since this webpage is limited in width. If you hover over the example code an arrow icon appears which when you click opens the code fully in a separate window. The entire command you are looking for is:
      1 parallel: –format html –out report.html –format ParallelTests::Cucumber::FailuresLogger –out cucumber_failures.log

Comments are closed.