SimpleDB Batching in LightSpeed 3.11

One of the enhancements made to LightSpeed 3.11 was to improve the efficiency of inserts when working with SimpleDB. If you’ve not heard of SimpleDB, it’s a cloud based database offering from Amazon. It’s not a traditional relational database, but rather a key/value store which is pretty neat if you’re into that sort of thing. We first released support for mapping over SimpleDB last year and got a great response from people wanting to explore SimpleDB. Nicely, LightSpeed is still the only .NET ORM that supports SimpleDB :-)

Storm clouds

One of the potential performance problems with cloud based databases is that queries take the form of a web request. As you’ll be well aware, these are an order of magnitude slower that calling a database running on your machine, or even on your same network segment. Unfortunately this means that if you wanted to push a heap of inserts into your database the SimpleDB provider would chug away making a call… getting a response… making the next call… getting a response… and so on. This latency made performance really quite slow for large data loading exercises.

Batched put attributes

Jeremy started investigating what could be done and found Amazon had enhanced the API for SimpleDB (and surfaced it through their .NET wrapper) to allow for “batched puts” which is SimpleDB lingo for batching of inserts. With a little bit of work on the core engine, LightSpeed can be configured to batch up inserts into collections of 25 – which is the limit that Amazon imposes on the batch size. This yields ~25x performance for inserts.

Turning on batching for SimpleDB

As this capability was not being released as part of a major version change we’ve left it to developers to enable this functionality. It’s pretty easy to do, just add the following to your connection string for SimpleDB providers:

Enable Batch Insert=true

More improvements planned

Currently batching is only enabled for the insert operations. Support for update operations is planned and then we will likely investigate some additional performance enhancements we can make to ensure that LightSpeed continues to earn it’s name. Also, a special mention must be made to our LightSpeed user “MiddleTommy” who has provided great feedback on working with LightSpeed which lead to this enhancement being made – thanks! :-)

Hope that helps all our SimpleDB users out there! Also, if you’re tinkering with SimpleDB, be sure to check out our SimpleDB Management Tools which gives a SQL Management Studio feel to working with SimpleDB directly from within Visual Studio.

Tagged as LightSpeed

Leave a Reply

Archives

Join our mailer

You should join our newsletter! Sent monthly:

Back to Top