Just-Eat

December 9, 2015

1383 views

In Memory DynamoDb

Problem

Here at JUST EAT we use DynamoDb in a lot of our components. The DynamoDb API can be awkward and slow to work with at times and this has sometimes lead to a decision between having complicated tests or sacrificing coverage.

Usually our integration tests are run against a QA AWS account that mirrors production. This gives us confidence that our changes are good to push live, but can be slow to iterate on when designing new features. To speed up our cycle time, we wanted a way to decouple our integration tests from AWS without losing confidence in our code.

Solution

Amazon provide a java app that can be used to deploy a local version of DynamoDb. We used this to develop a new feature, and it was the fastest we’ve ever worked with DynamoDb. We decided to wrap it up in a nuget package for use in other projects and when that succeeded just as well we decided to open source it.

The nuget contains a wrapper for this jar file to start and stop it, and provides a DynamoDb client that is configured to use the local instance of dynamo.

This eliminated the need to mock out the complicated DynamoDb api in our tests without sacrificing speed or coverage. It also removed a lot of lines from our tests.

Local Dynamo Test Example

Caveats

There are a couple of potential pitfalls with our approach.

AWS have licensed the DynamoDb jar file in such a way that we couldn’t include it in the nuget. This means that as part of setup, you need to download the file and its dependencies from Amazon and include them in your solution. This has the downside that the version you are testing against has the potential to get out of sync with the version that is running in AWS.

Our QA environments have a mirror of our production security groups, therefore any tests passing in QA give us confidence that our feature will work as intended in production. The local version of DynamoDb provided by Amazon has no concept of security groups, meaning that this aspect of our testing would no longer be covered. To ensure security groups continued to be tested adequately, we implemented a dependency check endpoint in our API that executes a read and write against dynamo and returns an http response giving success or failure information. This check is run post-deploy and has to pass before a deployment is considered successful.

The project is available on nuget: nuget.org/packages/LocalDynamoDb and on github at: github.com/justeat/LocalDynamoDb

Alan Nichols & Steve Brazier

AWS Testing

March 19, 2014

1304 views

Using DynamoDB Global Secondary Indexes

DynamoDB

DynamoDB is a schema-less NOSQL storage hosted by AWS, it has auto-scaling, read/write capacity management and alerting controlled using CloudWatch.

When creating a Dynamo table you assign a Key consisting of either a Hash or a Hash and Range, and the read/write capacity (amount of consistent reads and writes that can be performed per second). When Dynamo was first released there were three different ways of accessing an item, by Key (if a Key has both a Hash and Range then both must be set), Query and Scan. Accessing an item using the Key requires you to know the exact value of the Hash or the Hash and Range. Query allows you to perform comparisons against the Range part of the Key, but you must still specify which Hash you want to search. Scan performs a search across all columns without having to specify the Hash/Range, but in doing so it reads every item in your table, so can be an expensive operation if your table has 100s/1000s of items.

Global Secondary Indexes to the rescue

Recently Dynamo released Global Secondary Indexes, which are additional indexes (max 5 at present) that can be set against any column in your table, the only limitation being they must to be created up front. Each index always returns the Key from the main table and an optional list of projected attributes (equivalent to a column in SQL).

When designing your index there are two considerations to make, cost versus performance. Each index is a copy of the data from the main table and will contain at least an index Key and the table’s Key, plus any projected attributes. Including all attributes in an index will double the storage cost as there will be duplicate copies of the data, but will not require any additional reads to the table. An index that only has a Key will be slower as you will need to perform additional reads to the table using the table’s Key from the Query result. You can of course project a subset of the attributes, but as an index has to be created when the table is created any schema changes i.e. additional attributes, can not be added to an index at a later date.

To Query an index you set the IndexName and define a list of KeyConditions, these are a list of operations that you want to perform against the Key in the specified index, but be warned, you can only perform an equality (EQ) query against a Hash Key, whereas you can perform any operation against a Range Key.

var query = new QueryRequest
{
	TableName = "Ratings",
	IndexName = "RestId-Index",
	KeyConditions = new Dictionary&lt;String, Condition&gt;
	{
		{
		"RestaurantId", new Condition
			{
				ComparisonOperator = "EQ",
				AttributeValueList = {new AttributeValue {N = "1"}}
			}
		}
	},
	Limit = 10,
	ScanIndexForward = true
};

var query = new QueryRequest

{

TableName = "Ratings",

IndexName = "RestId-Index",

KeyConditions = new Dictionary<String, Condition>

{

"RestaurantId", new Condition

{

ComparisonOperator = "EQ",

AttributeValueList = {new AttributeValue {N = "1"}}

}

Limit = 10,

ScanIndexForward = true

};

When you Query an index the data is returned as a List of Dictionary<string, AttributeValue>, the string is the name of the attribute and the AttributeValue the value. Attributes can be three different base types: string (S), number (N), binary (B) or it can be a set of types (SS, NS and BS). The AttributeValue has a property for each type which makes mapping the data to an entity more difficult and not very DRY.

Query Result Mapping

Here are the steps to create your own generic entity mapper.

1) Get list of public properties from T.

var type = typeof(T);
var ret = new T();
var entityProperties = type.GetProperties(BindingFlags.Instance | BindingFlags.Public);

var type = typeof(T);

var ret = new T();

var entityProperties = type.GetProperties(BindingFlags.Instance | BindingFlags.Public);

2) Loop each property in T.

3) Get the attribute name.

private static string GetAttributeName(PropertyInfo property)
{
	var attributeName = property.Name;
	var dynamoDbProperty = property.GetCustomAttributes(typeof (DynamoDBPropertyAttribute), true).SingleOrDefault();
	if (dynamoDbProperty != null)
	{
		var dynamoDbPropertyValue = (DynamoDBPropertyAttribute) dynamoDbProperty;
		if (!string.IsNullOrWhiteSpace(dynamoDbPropertyValue.AttributeName))
		{
			attributeName = dynamoDbPropertyValue.AttributeName;
		}
	}
	return attributeName;
}

private static string GetAttributeName(PropertyInfo property)

{

var attributeName = property.Name;

var dynamoDbProperty = property.GetCustomAttributes(typeof (DynamoDBPropertyAttribute), true).SingleOrDefault();

if (dynamoDbProperty != null)

{

var dynamoDbPropertyValue = (DynamoDBPropertyAttribute) dynamoDbProperty;

if (!string.IsNullOrWhiteSpace(dynamoDbPropertyValue.AttributeName))

{

attributeName = dynamoDbPropertyValue.AttributeName;

}

return attributeName;

}

4) Find the attribute in the Dictionary.

private static AttributeValue GetAttribute(IDictionary&lt;string, AttributeValue&gt; item, string attributeName)
{
	if (!item.ContainsKey(attributeName))
	{
		return null;
	}
	AttributeValue value;
	return !item.TryGetValue(attributeName, out value) ? null : value;
}

private static AttributeValue GetAttribute(IDictionary<string, AttributeValue> item, string attributeName)

{

if (!item.ContainsKey(attributeName))

{

return null;

}

AttributeValue value;

return !item.TryGetValue(attributeName, out value) ? null : value;

}

5) If the attribute is not found, no mapping required, so the property’s default will be returned.

6) If the attribute is found cast it to the property’s type.

var propertyType = property.PropertyType;
switch (Type.GetTypeCode(propertyType))
{
	case TypeCode.String:
		property.SetValue(ret, attribute.S, null);
		break;
	case TypeCode.Int32:
		var intValue = ParseInt(attribute, attributeName);
		property.SetValue(ret, intValue, null);
		break;
	case TypeCode.Boolean:
		var booleanValue = ParseBoolean(attribute);
		property.SetValue(ret, booleanValue, null);
		break;
	case TypeCode.DateTime:
		var dateTimeValue = ParseDateTime(attribute, attributeName);
		property.SetValue(ret, dateTimeValue, null);
		break;
	default:
		var attributeValue = GetNonPrimitiveTypeValue(attribute);
		if (attributeValue != null)
		{
			var serializedItem = attributeValue is string
				? attributeValue.ToString()
				: _serializer.Serialize(attributeValue);
			var deserializedItem = _serializer.Deserialize(serializedItem, propertyType);
			if (deserializedItem != null)
			{
				property.SetValue(ret, deserializedItem, null);
			}
		}
		break;
}

var propertyType = property.PropertyType;

switch (Type.GetTypeCode(propertyType))

{

case TypeCode.String:

property.SetValue(ret, attribute.S, null);

break;

case TypeCode.Int32:

var intValue = ParseInt(attribute, attributeName);

property.SetValue(ret, intValue, null);

break;

case TypeCode.Boolean:

var booleanValue = ParseBoolean(attribute);

property.SetValue(ret, booleanValue, null);

break;

case TypeCode.DateTime:

var dateTimeValue = ParseDateTime(attribute, attributeName);

property.SetValue(ret, dateTimeValue, null);

break;

default:

var attributeValue = GetNonPrimitiveTypeValue(attribute);

if (attributeValue != null)

{

var serializedItem = attributeValue is string

? attributeValue.ToString()

: _serializer.Serialize(attributeValue);

var deserializedItem = _serializer.Deserialize(serializedItem, propertyType);

if (deserializedItem != null)

{

property.SetValue(ret, deserializedItem, null);

}

break;

}

To get a non-primitive type value you must check each type in the AttributeValue.

private static object GetNonPrimitiveTypeValue(AttributeValue value)
{
	if (!string.IsNullOrWhiteSpace(value.S))
	{
		return value.S;
	}
	if (!string.IsNullOrWhiteSpace(value.N))
	{
		return value.N;
	}
	if (value.SS != null &amp;&amp; value.SS.Any())
	{
		return value.SS;
	}
	if (value.NS != null &amp;&amp; value.NS.Any())
	{
		return value.NS;
	}
	return null;
}

private static object GetNonPrimitiveTypeValue(AttributeValue value)

{

if (!string.IsNullOrWhiteSpace(value.S))

{

return value.S;

}

if (!string.IsNullOrWhiteSpace(value.N))

{

return value.N;

}

if (value.SS != null && value.SS.Any())

{

return value.SS;

}

if (value.NS != null && value.NS.Any())

{

return value.NS;

}

return null;

}

IJsonSerializer interface is a wrapper around the Serialization logic.

public interface IJsonSerializer
{
	string Serialize(T item);
	object Deserialize(string value, Type type);
}

public interface IJsonSerializer

{

string Serialize(T item);

object Deserialize(string value, Type type);

}

This has been tested against String, Integer, Float, DateTime, Boolean, Collection and Non-Primitive Types so it should work against most custom entity classes.

Full class implementation

Conclusion

Dynamo is schema-less so the attributes can be of any type, when you Query an index you will not always get back all of the attributes of an item, only those defined in the projected fields. In my opinion, it is best to create a different entity/view to map against each index, this will help to avoid attempting to cast properties that do not exist in the index.

Pros:

Different read capacity per index so can be tailored depending on how often each index will be accessed
Return only a subset of attributes to help performance

Cons:

Cannot create new indexes or edit existing ones after table creation
Limited to only 5 indexes
Can only perform equality on Hash Key
Custom mapping required, although the code above fixes this limitation

Dynamo has come a long way since it was first launched and Global Secondary Indexes are a nice addition that adds some much needed functionality.

AWS

Tag : DynamoDB

In Memory DynamoDb

Problem

Solution

Caveats

Using DynamoDB Global Secondary Indexes

DynamoDB

Global Secondary Indexes to the rescue

Query Result Mapping

Conclusion

Recent Posts

Follow Us

Tag : DynamoDB

In Memory DynamoDb

Problem

Solution

Caveats

Using DynamoDB Global Secondary Indexes

DynamoDB

Global Secondary Indexes to the rescue

Query Result Mapping

Conclusion

Recent Posts

Tags

Follow Us