Performance Testing
Automated performance testing was the next logical step from automated development and continuous deployment. As you can imagine with the architecture to support Just Eat, which is not only a high-transaction e-commerce solution, for our customers, but also involves a distributed messaging system, for the restaurants. There are a lot of inter-connected pieces, so when we make an API change we need to ensure that this does not have an adverse affect, on either the front end site, the mobile devices, the communications with the restaurants, or the back end administrative tools.
We have a VPC setup for each team to performance test in and with a simple change of the parameters passed into the CloudFormations, we are able to build an as live version of any or all of our features. As you can imagine, having this as live setup, for all applications, can get quite expensive. So rather than having this environment up all day we have implemented the following design which enables us to run automated nightly performance tests.
Automated performance testing in AWS
On a set schedule all stacks required for the desire application to function are created in our VPC of a production size and configuration. Once these stacks are online, a performance stack is created. This stack consists of a Test Runner to co-ordinate the test, and an auto scaling group of Test Agents to execute distributed load across the application being tested (see bees with machine guns or wapt) . After the test has run the results from the test runner are uploaded to S3.
These are the collated along with all the relevant metrics from the monitoring server. This data is then analysed and the build will pass or fail depending on how these performance metrics compared to previous runs. At this stage artifacts are published with the relevant graphs onto the build server, this allows developers to easily see why a build has failed and compare the metrics to previous runs.
As this performance environment is built from scratch every night we can be confident that what is set up and deployed on this environment, is identical to what will be deployed on production.
I hope this has given you a brief insight into how we at Just Eat are using AWS to help us develop and test in a reliable, repeatable and scale-able fashion.