Just-Eat spectrum-bottom spectrum-top facebook google-plus instagram linkedIn pinterest reddit rss twitter_like twitter_reply twitter_share twitter_veriviedtwitter vimeo whatsapp youtube error_filled error file info-filled info loading star tick arrow arrowLeft arrowRight close download minus-filled minus move play plus-filled plus searchIcon settings

Tag : DevOps


Deployment Automation using Powershell DSC

Immutable Servers using Desired State Configuration

At JUST EAT, when we upgrade our platform services within our AWS hosted environment, we don’t just update the package on the instances, we replace the old servers with new ones built from scratch. This is an extension of the phoenix server approach. The longer a server has been provisioned and running, the more likely it is in an unknown state, creating a problem known as Snowflake servers.

We continuously deploy our components many times a day replacing almost every part of our platform infrastructure. Our continuous deployment strategy encompasses repeatable environments and extremely fast deployment times through pre-built static images (Amazon images – AMIs) and service orchestration times through decentralised orchestration tools (AWS Cloudformation). It’s difficult to guarantee repeatable deployments of the same server due to dependencies on packages, network availability and changes in the environment descriptions hence we achieve predictability using machine images that have been pre-built and tested within a Continuous Integration pipeline. Using version controlled recipes to define server configurations is an important part of continuous delivery.

Desired State Configuration (DSC) is a new management model for Powershell that allows us to deploy and manage configuration data for software services and environments, in turn allowing us to configure how to bring new and existing machines into compliance. DSC provides a set of Windows Powershell language extensions and resources that can be used to specify how you want your server to be configured allowing us to create Pheonix servers that can be recreated reliably from scratch.

DSC is the layer that accepts a configuration in the form of a static vendor neutral Management Object Format (MOF) file, a plain-text file in a format developed by the Distributed Management Task Force (DMTF), and implements it. The extensions in Powershell merely provide a Domain Specific Language (DSL) to define a configuration and generate the final MOF which is then executed by DSC. You could potentially write your own administrative interface to DSC. Moreover, since the MOF is based on an open standard, you could implement your own agent that provides the local and native resources to manage an OS. Hence, DSC is an open, cross platform solution to configuration management. The DSC agent on Windows is called the Local Configuration Manager (LCM) and is easily configured via a DSC configuration. You can use DSC to build a LINUX machine using the Open Management Infrastructure. This post aims to introduce DSC and is an introduction to a series of posts that will dive into advanced features of DSC including the different modes in which DSC operates (push and pull), the LCM and building LINUX AMIs. In order to generate machine images (AMIs) in AWS, we will be pushing a configuration to a remote instance and asking DSC to evaluate it.

Generating an AMI for EC2 in AWS

An Amazon Machine Image (AMI) defines the programs and settings that will be applied when you launch an EC2 instance. We dynamically install and configure our services on EC2 at instance launch time, pulling packages from S3, deploying files and ensuring services are started using CloudFormation. The AMI has the necessary OS features and tools pre-installed in order to speed up stack creation.

In order to create an AMI, we are going to require AWS Tools for Powershell installed. The following steps allow you to create an EC2 instance to create an image from, install features using DSC and finally create an image from it using just Powershell.

1) Generate a key-pair and a security group for managing access to the instance whilst you are building it. You will need to open open up ports for WinRM.

2) Find the standard Windows 2012 R2 image from the Amazon store and create an EC2 instance from it and set its attached EBS volumes in the exact way you want them created in the final image. Set the key-pair and security group to the ones created in step 1. Set the user data block to the following:

3) Extract the DNS name from the API and use the key-pair to extract the administrator password as shown below. You can now configure the machine using DSC.

Executing a DSC configuration against an EC2 instance

The following is an example configuration we use at JUST EAT to install the web server features on server core:

The keyword Configuration is an extension to the Powershell language. The keyword Node specifies the machine where the configuration will execute. WindowsFeatrue is a DSC Resource provided out-of-the-box with Windows Management Framework 4.0.

Within this config we are asking DSC to ensure the features Web-AppInit, Web-Asp-Net45, Web-Http-Tracing, Web-Mgmt-Service, Web-Net-Ext, Web-Server, Web-WebSockets and Web-Mgmt-Compat are installed on the machine with name $computerName. We set the property Ensure to Present in order to ask DSC to make sure the feature is present. We can set the value to Absent to remove a feature. You can use the Powershell commandlet Get-WindowsFeature to find the feature names.

You can see the declarative syntax of DSC at work here; we are specifying the desired state of the machine, we don’t really care how it happens. DSC will check if the feature already exists and install it if not found, which means that you can keep running the configuration above to ensure compliance.

This configuration can be extracted into a composite DSC resource, which you can use in multiple configurations. This is quite useful as you can develop a suite of behaviour tests for your own resources which can then be combined to create a robust set of configurations for your images. Creating your own resources is an important part of authoring DSC configurations and we’ll cover this in another post.

Once you have defined a configuration, generate the DSC MOF using:

$outpath will contain the DSC MOF file which will be pushed to the EC2 instance.

We can push the MOF to the remote machine and ask DSC to force the remote instance to evaluate the configuration using the following script:

And that’s it!

-Wait will wait until the configuration is applied and -Verbose will output the verbose logging in the resources to your local console. Once done, stop the EC2 instance and generate the AMI using the following powershell script:

Setting up a task in CI to generate an AMI

DSC is already pre-installed on Windows Server 2012 R2 and if you use just DSC to configure your AMIs, then everything is just a set of text files that live in a version control system like Git. Just clone the repo and execute the scripts on a Windows Server 2012 R2 build agent. The only requirement is that you have the AWS SDK for Windows installed.

The future of DSC

DSC was first mentioned in the Monad Manifesto in 2002 by Jeffery Snover, a distinguished engineer at Microsoft and lead architect for Windows Server and System Center. He describes the delivery of DSC as completing an eleven year journey to deploy automation to the data center in the form of Powershell.

The Windows OS, unlike LINUX is built around an enormous set of disparate APIs, meaning that getting something like Chef, Puppet or even DSC working on Windows is not only harder but can be a stressful task. Microsoft is pouring time and money into creating DSC resources that can, through a very simple and consistent interface, configure most of the OS. The next version of the Windows Management Framework, WMF 5.0, also shipped with Windows 10 will allow installing DSC resources directly from the Powershell Gallery, hence providing a mechanism to share DSC resources, guaranteeing that the coverage of the OS and applications will grow exponentially. Microsoft and other companies out there will be doing a lot of work that you don’t have to!

Microsoft recently announced Nanoserver, a purpose-built OS designed to run cloud applications and containers. Powershell DSC will be the preferred way to remotely manage the OS in the future.

Microsoft is working with Chef to create a set of Cookbooks based on Powershell DSC in order to provide enterprises with a rock-solid, native automation experience for the provision of compute and storage instances on Azure (and AWS). If you are not already using DSC to configure your Windows instances, then you are going to miss out on the work that Microsoft is doing with the community to save them and their users the work to configure the OS.

Useful links

The following book by Jez Humble and David Farley is highly recommended if you want to dig deeper into Deployment Pipelines: Continuous Delivery


Advanced Powershell DSC series: //channel9.msdn.com/Series/Advanced-PowerShell-Desired-State-Configuration-DSC-and-Custom-Resources


Presentation: AWS Summit 2014 London

Nearly 1,000 takeaways ordered a minute from hungry consumers, with near real time confirmation from restaurants and delivery of their food just 45 minutes later is a hard technical challenge.

At the AWS Summit 2014 in London I presented on how AWS allows the many small engineering teams at JUST EAT to take responsibility to meet that challenge, as we build and operate a platform that empowers consumers to love their takeaway experience.

Here are the slides from the session, but please do get in touch if you’d like to know more:


Presentation: Embracing DevOps to operate in the cloud

Yesterday, I presented at the April meetup for London DevOps, a usergroup focussing on DevOps. A whole set of us from JUST EAT Tech came along and supplied beer and pizza to the 50 or so people that turned up to hear me talk about how we get our developers to operate our platform.

As promised, here are the slides!