Protecting our inputs in a microservices world

At JUST EAT, we’re increasingly moving towards an interconnected web of internal APIs that each own their own data. We’re doing this so we can slice up our centralised database along domain boundaries, and scale parts of our system independently in terms of performance and rate of change. Small things are easier.
In this sort of world, it’s more important than ever to have consistent behaviour for non-functional requirements that span across different components. It’s a waste of everyone’s time to keep reinventing the wheel, but slightly differently, each time. These cross-cutting concerns are the usual suspects — security, validation, logging, metrics. This post deals with validation and consistency in how to respond when errors happen (when rules are violated).
It’s really important to validate one’s inputs — Garbage In, Garbage Out. Further, it’s valuable to reject invalid input as early as possible, so that the internals of the platform don’t have to worry about invalid data so much (not that they should trust internal clients more than external ones, but at least one can go tap on the shoulder of an engineer that looks after an internal thing).

Rejecting invalid input & requests politely

By validation, broadly speaking, I’m talking about two classes of error that I think are distinct:

  • Errors because the rules about the request were invalid – for example, the “Email” field was blank, or not an actual email address – rules that don’t require the API to ask anything external
  • Errors because the request was valid, but could not be processed because some sort of business-rule was not obeyed – for example, the credentials did not match, or the order could not be accepted because the restaurant ran out of chicken for the evening while the customer was ordering – rules that do require input from other components

Personally, I don’t really want clients of my APIs to have an inconsistent experience. So, if I give them back a 400 Bad Request, I want them to be able to say ‘Oh, I got a bad request, here’s a list of things I need to fix to make it work’, rather than ‘Oh, I got a bad request, I’ll go and ask the API team how to fix it — because it’s different from that other operation from that other API’. This means my APIs should give back an error response body that is standard, across operations and across APIs.

Open Source!

I’m really pleased to announce that today, we’re open-sourcing a library that we’ve built to do exactly this – JE.ApiValidation. It contains

  • A standard error response contract DTO
  • The assumption that you’ll be using the excellent FluentValidation library to implement your validation rules
  • Request-validation support for WebApi and OpenRasta, for requests that have a validator defined for them registered in your container:
    • WebApi: plug in to modelstate stage via a global action-executing filter
    • OpenRasta: plug in as an OperationInterceptor
  • Error-during-processing support for WebApi and OpenRasta, for requests that fail a business-rule check
    • WebApi: an exception-filter catches any FluentValidation.ValidationException and transforms that to the standard error response
    • OpenRasta: an OperationInterceptor catches any FluentValidation.ValidationException and transforms that to the standard error response
  • virtual methods for you to override, should you want to log warnings via your logging framework of choice, or publish metrics, or anything else

Error contract

The standard error response looks like:

{
  "Code": 40000,
  "Message": "some text about what class of error this might be",
  "Errors": {
    "FieldName": [
      {
        "Code": 0,
        "Message": "Error message for the rule being violated",
        "AttemptedValue": "What value was rejected",
        "CustomState": {
          "Something": "Custom"
        }
      }
    ]
  }
}

There are a few points about this design of response:

  • so that clients can use arithmetic to decide what class of error this is, each error response is categorised, and each category has a (hopefully unique) code number. So far, 40000 is “invalid request”, 45000 is “error during processing”. The codes are modelled after http and smtp, which suggest classes of error be denoted by ranges of values, so it’s easier for machines to interpret.
  • it’s very similar to what FluentValidation gives you out of the box.
  • it fulfils the requirement about telling the client as much as possible about what went wrong, rather than forcing them to solve each problem they may encounter one request & code-change at a time
  • it could be improved by adding some data about whether the request should be retried, or is a final-state (for example, if there was a network transient, probably retry it. If the payment failed because the customer’s card expired, there’s no point retrying)

Source code & how to install

The source code for the library can be found at http://github.com/justeat/JE.ApiValidation. There are a few different nugets here, all installable from nuget.org and the package manager or console:

  • JE.ApiValidation.DTOs – the contract. Take this if you want the notion of the standard contract, but the implementation for it doesn’t suit you (PRs welcome!)
  • JE.ApiValidation.OpenRasta – the OpenRasta OperationInterceptors for request validation and error-processing
  • JE.ApiValidation.WebApi – the WebApi request validation attribute – no dependency on FluentValidation, in case you happen to use DataAnnotations to do your validation already (and don’t want to change that)
  • JE.ApiValidation.WebApi.FluentValidation – the WebApi error-processing exception-filter (that depends on FluentValidation)

Getting started

There are examples for how to use it:

  • WebApi request validation
    using JE.ApiValidation.WebApi;
    public class MyWebApiConfiguration : HttpConfiguration {
      public MyWebApiConfiguration() {
        // ...
        ConfigureRequestValidation();
      }
      private void ConfigureRequestValidation() {
        FluentValidationModelValidatorProvider.Configure(this, provider => provider.ValidatorFactory = new ValidatorFactoryForYourContainer());
        Filters.Add(new FilterForInvalidRequestsAttribute());
      }
    }
    // and, assuming a class called "Widget" with an string:Email property
    public class WidgetValidator : AbstractValidator {
      public WidgetValidator() {
        RuleFor(x => x.Email)
          .NotNull()
          .NotEmpty()
          .EmailAddress();
      }
    }
    
  • WebApi errors during response-processing
    using JE.ApiValidation.WebApi;
    using JE.ApiValidation.WebApi.FluentValidation;
    public class MyWebApiConfiguration : HttpConfiguration {
      public MyWebApiConfiguration() {
        // ...
        ConfigureResponseProcessingErrorHandling();
      }
      private void ConfigureResponseProcessingErrorHandling() {
        Filters.Add(new ResponseProcessingErrorAttribute());
      }
    // and then, assuming a class verbosely named InternalRepresentationOfAWidget
    public class WidgetBusinessRules : AbstractValidator {
      RuleFor(x => x.Something)
        .Must(x => x.Something == "something")
        .WithMessage("Some arcane rule was not met.");
    }
    // and finally, within a controller or service class (where _validator is injected via your container):
    void DoSomething(InternalRepresentationOfAWidget w) {
      _validator.ValidateAndThrow(w);
      // ... do things knowing that the business rules passed
    }
    
  • OpenRasta request validation (link to example)
  • WebApi errors during response-processing (link to example)

For the time being, continuous integration and publishing the nugets will be internal; that will change in due course.