Just-Eat spectrum-bottom spectrum-top facebook google-plus instagram linkedIn pinterest reddit rss twitter_like twitter_reply twitter_share twitter_veriviedtwitter vimeo whatsapp youtube error_filled error file info-filled info loading star tick arrow arrowLeft arrowRight close download minus-filled minus move play plus-filled plus searchIcon settings

Tag : Testing


Reliably Testing HTTP Integrations in a .NET Application


Testing HTTP dependencies in modern web applications is a common problem, but it’s also something that can create difficulty for authoring reliable tests.

Today, we’re open-sourcing a library to help reduce the friction many developers have with this common requirement: JustEat.HttpClientInterception.

You can find the repository in our GitHub organisation and can find the package available to download at JustEat.HttpClientInterception on NuGet.org.

The Problem

Many modern software applications integrate with external Application Programming Interfaces (APIs) to provide solutions for problems within their domain of responsibility, whether it be delivering your night in, booking a flight, trading financial instruments, or monitoring transport infrastructure in real-time.

These APIs are very often HTTP-based, with RESTful APIs that consume and produce JSON often being the implementation of choice. Of course integrations might not be with full-blown APIs, but just external resources that are available over HTTP, such as a website exposing HTML, or a file download portal.

In .NET applications, whether these are console-based, rich GUIs, background services, or ASP.NET web apps, a common go-to way of consuming such HTTP-based services from code is using the HttpClient class.

HttpClient provides a simple API surface for GET-ing and POST-ing resources over HTTP(S) to external services in many different data formats, as well as functionality for reading and writing HTTP headers and more advanced extensibility capabilities.

It is also commonly exposed as a dependency that can be injected into other services, such as third-party dependencies for tasks such as implementing OAuth-based authentication.

Overall this makes HttpClient a common and appropriate choice for writing HTTP-based integrations for .NET applications.

An important part of software development is not just implementing a solution to a given problem, but also writing tests for your applications. A good test suite helps ensure that delivered software is of a high quality, is functionally correct, is resilient in the face of failure, and provides a safety net against regression for future work.

When your application depends on external resources though, then testing becomes a bit more involved. You don’t want to have the code under test making network calls to these external services for a myriad of reasons. They make your tests brittle and hard to maintain, require a network connection to be able to run successfully, might cost you money, and slow down your test suite, to name but a few examples.

These issues lead to approaches using things like mocks and stubs. HttpClient, and its more low-level counterpart HttpMessageHandler, are not simple to mock however. While not difficult to do, their lack of interface and design lead to a requirement to implement classes that derive from HttpMessageHandler in order to override protected members to drive test scenarios, and to build non-primitive types by hand, such as HttpResponseMessage.

Another approach that can be used to simplify the ability to use mocks is to create your own custom IHttpClient interface and wrap your usage of HttpClient within an implementation of this interface. This creates its own problems in non-trivial integrations though, with the interface often swelling to the point of being a one-to-one representation of HttpClient itself to expose enough functionality for your use-cases.

While this mocking and wrapping is feasible, once your application does more than one or two simple interactions with an HTTP-based service, the amount of test code required to drive your test scenarios can balloon quite quickly and become a burden to maintain.

It is also an approach that only works in a typical unit test approach. As usage of HttpClient is typically fairly low down in your application’s stack, this does not make it a viable solution for other test types, such as functional and integration tests.

A Solution

Today we’re publishing a way of solving some of these problems by releasing our JustEat.HttpClientInterception .NET library as open-source to our organisation in GitHub.com under the Apache 2.0 licence.

A compiled version of the .NET assembly is also available from JustEat.HttpClientInterception on NuGet.org that supports .NET Standard 1.3 (and later) and .NET Framework 4.6.1 (and later).

JustEat.HttpClientInterception provides a number of types that allow HTTP requests and their corresponding responses to be declared using the builder pattern to register interceptions for HTTP requests in your code to bypass the network and return responses that drive your test scenarios.

Below is a simple example that shows registering an interception for an HTTP GET request to the Just Eat Public API:

The library provides a strongly-typed API that supports easily setting up interceptions for arbitrary HTTP requests to any URL and for any HTTP verb, returning responses that consist of either raw bytes, strings or objects that are serialized into a response as-and-when they are required.

Fault injection is also supported by allowing arbitrary HTTP codes to be set for intercepted responses, as well as latency injection via an ability to specify a custom asynchronous call-back that is invoked before the intercepted response is made available to the code under test.

With ASP.NET Core adding Dependency Injection as a first-class feature and being easy to self-host for use within test projects, a small number of changes to your production code allows HttpClientInterceptorOptions to be injected into your application’s dependencies for use with integration tests without your application needing to take a dependency on JustEat.HttpClientInterception for itself.

With the library injected into the application, HTTP requests using HttpClient and/or HttpMessageHandler that are resolved by your IoC container of choice can be inspected and intercepted as-required before any network connections are made. You can also opt-in to behaviour that throws an exception for any un-intercepted requests, allowing you to flush out all HTTP requests made by your application from your tests.

Further examples of using the library can be found at these links:

The Benefits

We’ve used this library successfully with two internal applications we’re developing with ASP.NET Core (one an API, the other an MVC website) to really simplify our tests, and provide good code coverage, by using a test approach that is primarily a black-box approach.

The applications’ test suites self-host the application using Kestrel, with the service registration set-up to create a chain of DelegatingHandler implementations when resolving instances of HttpClient and HttpMessageHandler. With HttpClientInterceptorOptions registered to provide instances of DelegatingHandler by the test start-up code when the application is self-hosted, this allows all HTTP calls within the self-hosted application in the tests to be intercepted to drive the tests.

The tests themselves then either initiate HTTP calls to the public surface of the self-hosted server with a vanilla HttpClient in the case of the API, or use Selenium to test the rendered pages using browser automation in the case of the website.

This approach provides many benefits, such as:

  • Simple setup for testing positive and negative code paths for HTTP responses, such as for error handling.
  • Exercises serialization and deserialization code for HTTP request and response bodies.
  • Testing behaviour in degraded scenarios, such as network latency, for handling of timeouts.
  • Removes dependencies on external services for the tests to pass and the need to have access to an active network connection for services that may only be resolvable on a internal/private network.
  • No administrative permissions required to set-up port bindings.
  • Speeds up test execution by removing IO-bound network operations.
  • Allows you to skip set-up steps to create test data for CRUD operations, such as having to create resources to test their deletion.
  • Can be integrated in a way that other delegating handlers your application may use are still exercised and tested implicitly.
  • Allows us to intercept calls to IdentityServer for our user authentication and issue valid self-signed JSON Web Tokens (JWTs) in the tests to authenticate browser calls in Selenium tests.

In the case of the ASP.NET Core API using this test approach, at the time of writing, we’ve been able to achieve over 90% statement coverage of a several thousand line application with just over 200 unit, integration and end-to-end tests. Using our TeamCity server, the build installs the .NET Core runtime, restores its dependencies from NuGet, compiles all the code and runs all the tests in just over three-and-a-half minutes.

Some Caveats

Of course such a solution is not a silver bullet. Intercepting all of your HTTP dependencies isolates you from interface changes in your dependencies.

If an external service changes its interfaces, such as by adding a new API version or deprecating the one you use, adds new fields to the responses, or changes to require all traffic to support HTTPS instead of HTTP, your integration tests will not find such changes. It also does not validate that your application integrates correctly with APIs that require authentication or apply rate-limits.

Similarly, the black-box approach is relatively heavyweight compared to a simple unit test, so may not be suited to testing all of the edge cases in your code and low-level assertions on your responses.

Finally, your intercepted responses will only cater for the behaviour you’ve seen and catered-for within your tests. A real external dependency may change its behaviour over time in ways that your static simulated behaviours will not necessarily emulate.

A good mixture of unit, interception-based integration tests, and end-to-end tests against your real dependencies are needed to give you a good robust test suite that runs quickly and also gives you confidence in your changes as you develop your application over time. Shipping little and often is a key tenet of Continuous Delivery.

In Conclusion

We hope that you’ve found this blog post interesting and that you find JustEat.HttpClientInterception useful in your own test suites for simplifying things and making your applications even more awesome.

You can find the project in our organisation on GitHub and you can download the library to use in your .NET projects from the JustEat.HttpClientInterception package page on NuGet.org.

Contributions to the library are welcome – check out the contributing guide if you’d like to get involved!

If you like solving problems at scale and don’t think testing is just for QAs, why not check out open positions in Technology at Just Eat?

About the Author

This blog post was written by Martin Costello, a Senior Engineer at Just Eat, who works on consumer-facing websites and the back-end services that power them.


Unit testing front-end JavaScript with AVA and jsdom

Writing tests for JavaScript code that interacts with the DOM can be tricky. Luckily, using a combination of AVA and jsdom, writing those tests becomes a lot easier.

This article will walk you through how to set everything up so you can get started writing your tests today.

What is AVA?

AVA is described as a “Futuristic JavaScript test runner“. Sounds fancy, huh?! So, what is it exactly that makes it “futuristic“?!

Tests run quickly

AVA runs test files in parallel, each in its own separate process, with the tests inside those files running concurrently. This offers better performance than other test runners that run tests serially, such as Mocha. This also means that each test file is run in an isolated environment — great for writing atomic tests.

Simple API

AVA’s API is very small because, in AVA’s own words, it is “highly opinionated“. You won’t find any assertion aliases here! This reduces the cognitive load required when writing tests.

Write tests in ES2015

You don’t need to do anything to be able to write tests in ES2015, AVA supports this out of the box! Under the covers it’s using Babel to transpile with the es2015 and stage-2 presets.

No implicit globals

AVA has no implicit globals, simply import it into your test file and you have everything you need.

Other benefits

There are a whole host of other benefits which AVA offers such as:

  • Promise support
  • Generator function support
  • Async function support
  • Observable support
  • Enhanced assertion messages
  • Clean stack traces

All of this combined sounds very “futuristic” to me!

Getting off the launchpad with AVA

Now that we know more about AVA, let’s create a new project and start writing some tests.

Start by running npm init inside a new project folder. This will create a package.json file, which will contain various pieces of information about the project such as its name, authors, and dependencies, among others. Hitting enter for each question will fill in a default value.

Installing AVA

Add AVA to the project by typing npm install ava --save-dev, then update the scripts section in package.json:

The --verbose flag enables the verbose reporter, which means more information is displayed when the tests are run.

When using npm scripts, the path to AVA in the node_modules folder will be resolved for us, so all we need to do is type npm test on the command line. Doing so at the moment this will give us an exception:

Let’s fix that by adding a test.

Writing a test

Create a test directory, with a file named demo.test.js inside, then add a test:

First, AVA is imported into the module, then the test function is called, passing a string as the first parameter which describes what the test is doing. The second parameter is the test implementation function which contains the body of the test, this provides us with an object, t, from which we can call the assertion functions.

The is assertion is used here, which takes two values and checks that they are both equal (using === so there is no type conversion).

Note: You can choose any name you like for the t parameter, such as assert. However, using the t convention in AVA will wrap the assertions with power-assert which provides more descriptive messages.

Run npm test and the test result will be printed out

Success! Our test passed as expected. To see an example of what a failing test would look like change the test assertion to t.is(1 + 1, 1). Run the test now and you’ll see an error

As you can see, there is a lot of useful information provided in order to help us track down the issue.

Testing modules

To demonstrate how to test a module, create a new folder called src in the root of the project with a file inside called demo-module.js with the contents:

Update demo.test.js by first importing the module, then adding a new test:

Running npm test now will give you the following exception

Uh oh, what happened?

AVA will transpile ES2015 code in your tests; however, it won’t transpile code in modules imported from outside those tests. This is so that AVA has zero impact on your production environment.

If our source modules are written in ES2015, how do we tell AVA that we’d like them to be transpiled too?

Transpiling source files

To transpile source files, the quick and dirty option is to tell AVA to load babel-register which will automatically transpile the source files on the fly. This is ok if you have a small number of test files, but there is a performance cost which comes with loading babel-register in every forked process.

The other option is to transpile your sources before running the tests in order to improve performance.

The next two sections look at how each technique can be achieved.

Transpile with babel-register

Add babel-register by running npm install babel-register --save-dev, then add a "babel" config to package.json

Next, add "babel-register" to the AVA "require" section

Run npm test and the tests will once again pass, great!

The recommendation from the AVA team is to use babel-registeruntil the performance penalty becomes too great“. As your test base grows you’ll need to look into setting up a precompilation step.

Setting up a precompilation step

A precompilation step will transpile your source modules before the tests are run in order to improve performance. Let’s look at one way to set this up.

Note: If you were following along with the last section you’ll need to remove the references to babel-register. First run npm uninstall babel-register --save-dev, then remove "babel-register" from the AVA "require" section in package.json.

Start by adding the babel-cli and babel-preset-es2015 packages to the project: npm install babel-cli babel-preset-es2015 --save-dev.

Next, add a "babel" config to package.json

In order to run the tests, we need to update the npm scripts. Add a new npm script called precompile

The precompile npm script will tell Babel to take the files in the src directory, transpile them, then output the results to the dist directory.

Next, the test npm script needs to be updated so that it runs the precompile step before running the tests

The double ampersand (&&) tells npm to first run the precompile script and then the AVA tests.

The final task is to update the reference to demo-module inside demo.test.js to point at the compiled code, we do this by replacing ../src with ../dist:

Run npm test and we’re presented with all green tests!

Testing the DOM using Node

So far we have the ability to test JavaScript code, but what if we’d like to test a function which makes use of the DOM? Node doesn’t have a DOM tree, so how do we get around this?

One option is to use a combination of a test runner and a browser — a popular combination is Karma and PhantomJS. These offer a lot of benefits like being able to test against real browsers, run UI tests, take screenshots, and the ability to be run as part of a CI process.

However, they typically come with a fairly large overhead, so running lots of small tests can take minutes at a time. Wouldn’t it be great if there was a JavaScript implementation of the DOM?

Welcome to the stage; jsdom!


jsdom is described as “A JavaScript implementation of the WHATWG DOM and HTML standards, for use with Node.js“.

It supports the DOM, HTML, canvas, and many other web platform APIs, making it ideal for our requirements.

Because it’s purely JavaScript, jsdom has very little overhead when creating a new document instance which means that tests run quickly.

There is a downside to using a JavaScript implementation over an actual browser – you are putting your trust in the standards being implemented and tested correctly, and any inconsistencies between browsers will not be detected. This is a deal breaker for some, but for the purposes of unit testing I think it is a reasonable risk to take; jsdom has been around since early 2010, is actively maintained, and thoroughly tested. If you are looking to write UI tests then a combination of something like Karma and PhantomJS may be a better fit for you.

Integrating jsdom

Setting up jsdom can be a daunting task, the documentation is great, but very lengthy and goes into a lot of detail (you should still read it!). Luckily a package called browser-env can help us out.

Add browser-env to the project npm install browser-env --save-dev.

Create a helpers directory (which is ignored by convention when using AVA) inside test, then add setup-browser-env.js with the contents

We need to tell AVA to require this module before any of the tests are run so that browser-env can create the full browser environment before any DOM references are encountered. Inside your package.json add

Note: You may have noticed that this file is written in ES5. This is because AVA will transpile ES2015 code in the tests, yet it won’t transpile any modules imported or, in this case, required from outside the tests — see the transpiling source files section.

Testing the DOM

Let’s write a test which makes use of the document global which has been provided thanks to jsdom. Add a new test to the end of demo.test.js:

First, we add a paragraph element with some text to the document body, then query for that element using document.querySelector, and finally, we verify that the selected paragraph tag has an innerHTML value equal to 'Hello, world'.

Run the tests with npm test

Congratulations, you’ve just unit-tested the (virtual) DOM!

Test coverage with nyc

As a bonus let’s quickly set up some test coverage. Because AVA runs each test file in a separate Node.js process, we need a code coverage tool which supports this. nyc ticks the box — it’s basically istanbul with support for subprocesses.

Add it to the project with npm install nyc --save-dev, then update the test npm script by adding nyc before the call to ava:

You’ll also need to update the Babel config to tell it to include source maps when developing so that the reporter can output the correct lines for the transpiled code:

Run the tests and witness the awesome code coverage table!

What next?

If you’re interested in what else you can do with AVA, have a look through the AVA readme, check out the AVA recipe docs, read about common pitfalls, and listen to this JavaScript Air podcast episode. I’d also recommend looking into setting up linting for your code.

You can browse the source code for this blog post on GitHub.

So, now you have no excuse for not testing your front-end JavaScript!

Damian Mullins is a UI Engineer at Just Eat. Progressive enhancement advocate, web standards supporter, JavaScript enthusiast.


Calabash Page Objects

Faster development of Calabash tests

While creating the page object classes in our Calabash mobile test suites at JUST EAT, we found ourselves repeating a lot of actions when waiting for, scrolling to and interacting with elements on the screen. We abstracted these actions into a library to avoid this unnecessary duplication of code and made these actions agnostic to screen size. This library has now been published as a ruby gem called calabash-page-objects.

Why use this?

Dealing with small screens

Sometimes you have to scroll to elements on small screens but not on larger screens. We initially used if-statements dealing with an environment variable for ‘small screen’ inside your test code – not good!

We wrote a method to scroll to an element if it wasn’t immediately there. This method was then included in many of the methods available to our elements; touching, inputting text, asserting presence etc.

Multiple scrollable views

When attempting to use Calabash’s default scroll method, we noticed that sometimes it didn’t appear to scroll the view we wanted if there were multiple scrollable views on the screen.

After looking into the Calabash methods, we noticed that you could perform scroll actions on locators. We wrapped up this method in the gem too so that we could pass both the element we’re searching for and the view it belongs in into all the helper methods. This became the ‘parent’ parameter that the gem methods can optionally take.

How to use?

The calabash-page-objects gem exposes two element classes, one for iOS the other for Android. They are implemented in the same way regardless of the platform under test. These element classes have methods for, waiting for them to be visible, waiting for them to disappear, touching them, etc. These methods all take variables in a consistent format.

More information

See the project on Github for more information, and a more detailed description of the methods and parameters. Feel free to fork it and contribute too.


Acceptance Testing in Appium

Why use acceptance testing?

A well-tested application measures the quality of the application and helps achieve good compatibility between the users and devices (iOS and Android devices in our case). To get the most out of our testing efforts and test coverage, a robust and cross-platform tool is required to automate the application. Appium is a pretty good choice which fits the bill.

Acceptance testing gives us confidence in two main areas: that new features have been built to the correct specification; and existing features continue to function after new integrations.
It’s notoriously difficult to prove that developers have built the feature they were asked for. One reason for this is that the use of natural language to describe feature specifications can result in ambiguities and misunderstandings between developers and stakeholders.


One approach being undertaken by a number of teams in International Engineering at JUST EAT is to produce executable specifications.

This has numerous benefits in terms of understanding the requirements of a project through user stories but, in particular, it give us specific steps that can be tested to verify a feature has been implemented correctly. These step sets are called scenarios, and capture the flow of a specific task that a user wants to perform.

The following are some examples from our own specifications…

Scenario Outline: Make a successful order – Login to JUST EAT through Facebook
 Given I am on the home screen

 And I search for a restaurant

 And I add items to basket

 And I login through <social> account <email> and <password>

 When I checkout with cash payment

 Then the order should be placed successfully

 |social        |email                    |password |

 |facebook   |test@test.com |test |

Scenario: View order details on Order History screen
  Given I am on the home screen

  And I have previously ordered from this device

  And I navigate to order history

  When I select an order

    |Test Restaurant |

  Then I am shown the details of that order

These specifications are written in Gherkin syntax, which is supported by the Cucumber suite of test tools. The specific tools we chose to use are outlined in the next section.

Summary of technology


The iOS Webkit Debug Proxy allows access to webviews and their content. It provides an interface to the various components of a web page within a webview using the same contract as that used for the native application. For the JUST EAT app we use it to facilitate automation of the checkout flow so that we can enter delivery details, make an order and read the details of the order confirmation page.

As part of the initialisation of the Appium process, it launches the iOS Simulator. Appium communicates with the simulator using Apple’s UI Automation layer. This layer provides Javascript APIs to access the UI of the application at runtime. Appium then wraps this interface in another API that conforms to the JSON Wire Protocol. The benefit of using this abstraction is that it standardises UI access for different platforms meaning the same tests can be run for a variety of platforms.

While the Appium server provides the interface to access the UI of the running application, Cucumber JS is used to execute our scenarios defined in Gherkin syntax. The code that backs these steps contains the procedures to send commands to Appium.

NodeJS underlies most of the technologies listed above. NodeJS implements a non-blocking I/O model and an event loop that uses a single thread and performs I/O asynchronously. Mocha, Chai and Chai-as-Promised were among other modules used to provide additional testing support.

Page object model


Since we have the apps on iOS and Android platforms we created a single test suite which can run the same set of tests on both platforms, to avoid duplication of test code and save time, and Page Object Model helped us achieve this.

Page object models each represent a screen or group of controls in the user interface. For example, the home screen of the iOS app can be represented by a page object that provides a text field and a search button as its interactive elements. These elements can be obtained by ID, Xpath, Class name or by Name, and stored as properties on the model to be used during the test.

This adds a level of abstraction and the tests use an instance of the page object to interact with the UI elements rather than worrying about the element locators in the test itself. The abstraction is useful because the model is tightly coupled to the user’s perception of the user interface, and so is more appropriate for acceptance testing.

Another benefit is that we can add elements to a given model over time, as needed by the tests, instead of including all the elements on the screen that the tests do not interact with.

Below is a diagram of the test project structure, indicating where the page objects are located.



We direct the tests to use the page objects from the right folder, based on the platform we are running the tests on.

A potential issue with a cross-platform test suite is that you may have non-uniform UI designs across platforms. Luckily, our UX designers and developers make sure the flow is uniform across iOS and Android, meaning that we don’t have to add lots of “if(platform)” statements in the test code.

Continuous integration

A major use of automated acceptance test is verifying the validity of new and existing features during automated build processes. To this end, we created a new script that runs the tests on TeamCity.

The script itself takes a number of arguments to allow configuration of the test environment for different build jobs:


  • Platform Name specifies which platform it runs on, i.e. iOS or Android.
  • Device Name specifies the type of device to run on, e.g. iPhone 6 Plus, Samsung Galaxy S4.
  • Platform Version allow a particular SDK to be targeted, e.g. iOS 7.1, Android 4.0.4.
  • App Path specifies a path to the app executable under test.
  • Environment was a custom option introduced to allow the selection of a particular QA environment, e.g. QA19, QA99, Staging.
  • Country Code lets the tests know which flavour of the app is under test.
  • Cucumber Flags allows some additional configuration to be passed to the grunt task runner.

To integrate it with TeamCity we took the following steps…

  1. We created a new build template and added a number of configurable parameters for the script arguments.
  2. We added build features to allow consumption of the JUnit report and archiving of the HTML report.

  3. We added an artifact dependency to ensure that the most recent valid build would always be under test.



Reporting test results is an important step in continuous integration as it allows tracking of test results over time, as well as providing a quick insight for interested parties without requiring them to read test metadata files such as XML and JSON.

We use a selection of Javascript tools to produce and convert the output of Cucumber JS. Grunt provides the framework in which to execute Cucumber JS and consume the test output, through a simple task runner with various configurable settings.

The JSON output produced is simple and readable but not necessarily compatible with continuous integration reporting tools. To this end we use protractor-cucumber-junit which converts the JSON into two formats:

  • HTML provides a simple and readable page that any non-technical user can access for a quick overview of test results.
  • JUnit XML is almost universally consumable by CI tools allowing native presentation of results in your CI front-end of choice, as well as enabling CI tools to track trends in testing over time, code coverage, and so on.


Simulation across platforms

iOS simulators are used, which are included as part of Xcode and hence can only be run from a mac. Xcode can run only one instance of instruments at a time and therefore, we cannot parallelise the iOS test run.

Genymotion is being used for Android emulators as it’s more promising than the Android emulator that comes with the Android sdk. Read an awesome blog on how to use Android emulators on CI, by Andrew Barnett, here [//tech.just-eat.com/2015/02/26/using-android-emulators-on-ci/]

Problems and challenges

We’re using simulators and emulators to run our acceptance tests – as they give us more coverage in terms of devices and also an easy way to test any small changes. Nevertheless, not all types of user interactions can be tested on emulators and the iOS simulators in particular have problems with certain Appium gestures.

Further, during the test execution there are no real events like battery consumption and interrupts and hence, we do not get accurate results regarding the performance of the app. Since we are in initial stages of automating the apps, we find simulators/emulators as a good stop-gap while experimenting with continuous integration.

In the future it would be desirable to use a cloud-hosted service such as SauceLabs. This would allow tests to be run across many devices and OS versions in parallel.


How to take screenshots for failing tests with KIF

How to take screenshots for failing tests with KIF

In the Consumer iOS app team we use KIF to test our application. KIF (Keep It Functional) is an open source tool for iOS automation from square. More details here: //github.com/kif-framework/KIF

Taking screenshots for failing tests is a good way to understand why the test is failing.

Most test automation tools come with some way of taking screenshots. To enable taking screenshots for failing tests in KIF, we need to set an environment variable “KIF_SCREENSHOTS= ‘location’”. Location is the path which tells KIF where to save the screenshots.

To make sure that screenshots are saved for all failing tests, irrespective of the machine they run on, ‘location’ should be relative. I used the location ‘$PROJECT_DIR/Failed_Screenshots’.

$PROJECT_DIR is an Xcode build setting which stores the value of the Xcode project directory. This will already be set to where the project is on your system. To see the value of $PROJECT_DIR, go to the terminal, navigate to where the Xcode project is, and type ‘xcodebuild -showBuildSettings | grep PROJECT_DIR’.

KIF creates the Failed_Screenshots folder automatically, whenever a test fails and saves screenshots for all the failing tests.

Below are the steps to enable screenshots from Xcode:

How to enable screenshots using Xcode

  • Select the test scheme in Xcode


  • Select Edit Scheme


  • Select Run and set an environment variable KIF_SCREENSHOTS= ‘location_to_Save_screenshots’

variable settings

  • Select Close


  • Write a failing test

test file

test file2

This is a sample test which I created to demonstrate how KIF can take screenshots. This test is looking for an accessibility label which does not exist and always fails.

  • Run the test using Xcode or xcodebuild on command line


  • Check the screenshot in the failed screenshots folder, which you set as the value of the KIF_SCREENSHOTS variable. The screenshot is saved with the same name as the test class and also includes the line number of the failing step. For example, ‘JETESTScreenshots.m, line 17.png’, in this case ‘JETESTScreenshots.m’ is the test class and line 17 is the line number of the failing step.


  • Give the path to the ‘Failed_Screenshots’ folder in the artifact paths field of the project in your CI system, to save the screenshots as artifact for every failing build job.


Thanks for reading.