Why fewer End-to-End Tests?

At Just Eat, we undertake a lot of end to end (e2e) tests that continuously run every time we change something on our website. It goes without saying that tests give you more confidence about the product’s quality since it covers more use cases from the user’s perspective. However time is crucial in the Software Development Life Cycle – the more you wait to get feedback, the slower the development process will be.

The issues we face having more e2e website tests are…
  • They require lot of knowledge, skills and time to write quality e2e tests.
  • They can take a long time to execute, which means feedback is slow.
  • They can be very brittle, due to relying on UI which changes often.
  • Testing negative paths can be complex if integrating with APIs or databases.
  • They can be fragile since they normally rely on external dependencies and environment.
  • And when e2e test fails… it can be like finding a needle in a haystack.

 
So how do we overcome these issues? Well we need to make sure we are writing the right amount of tests at the right level… and by level I mean unit, acceptance and end to end.
If we consider unit tests, they test a small unit/component in isolation and are fast, often very reliable and normally make easy to find the bug when they fail. The main disadvantage of a unit test is that even if the unit/component works well in isolation, we do not know if it works well with the rest of the system. In other words we don’t test the communications between each component. For that we need to have an integration test and these tests should focus on the contracts between each component and how they act when this contract is met and not met. But despite the above issues with e2e tests we know that they are the only tests which simulate real user scenarios, ie when I place an order or when I leave a review, etc – so it’s important to have a right balance of all these test types.
The best visual indication of this is the Agile Testing Pyramid (see below).
 
pyramid_thumb
According to the pyramid, the best combination will be 70% unit tests, 20% integration/acceptance tests and only 10% end to end tests.

In-memory acceptance tests

We allow each developer to run all of the integration/acceptance level test locally. To achieve this we’ve developed a framework to minimise the issues normally encountered. The framework supports real tests via browser by hosting the website in-memory and mocks out all of the endpoint calls so it’s not dependent on the actual QA environment.
inmemory test framework diagram
FiddlerCore – Is the base component used by Fiddler and allows you to capture and modify HTTP and HTTPS traffic and we use it injecting a proxy.
Coypu/Selenium – This is a wrapper for browser automation tools on .Net with Selenium WebDriver (same as Capybara in the ruby framework).
IIS Express – Visual Studio .NET built in IIS server which we use here to host the Website.
When you trigger an in-memory test the Selenium driver; with the Coypu wrapper, communicates with the browser whilst Fiddlercore will inject the proxy to the request via a header.
The browser accesses the Website; hosted in IIS Express, exercising the server side code which attempts to communicate with various endpoints (APIs). Through FiddlerCore, we can listen to the API call being made, inject a proxy and mock the API response, so we can test the presentation layer in isolation. You can mock scenarios where the API can fail or return unexpected data and handle how this will affect the user journey – in most cases, you can just show a minimal set to a user instead of an error page.
In some cases eg Authentication, you can inject an in-memory implementation of the identity server that can hijack authentication requests and issue tokens that your application trusts. Ideally, a developer should be able to run most of the tests, including all unit tests and a huge subset of the acceptance tests suite without being connected to a network.

The benefits of this framework are…
  • Faster and reliable than e2e tests.
  • Can also be used to write tests to simulate real user scenarios.
  • Can also be used to write integration/api tests.
  • They provide the ability to write scenarios where you can gracefully degrade in terms of functionality.
  • No dependency on the environment or QA.
  • Earlier feedback in the development cycle.
  • They can be run locally.
  • Works well with continuous integration.
  • Motivates the developer to write their own e2e and Integration tests. Also it helps developers to think about and simplify the architecture with simpler dependencies.

Summary :

Having more e2e tests don’t automatically reflect on faster delivery but it is important to have a right balance of tests and a good test automation strategy. So it goes; without saying, less e2e tests will save you time plus you can now also spend more time on exploratory testing. Also “right” set of tests allows you to evolve your architecture, refactor your code in order to continuously improve.
As a footnote I would like to mention ‘Rajpal Wilkhu’ who architectured and helped us to develop this amazing framework.
 
Thanks for reading …
Deepthi Lansakara

Comments are closed.