SimpleDB support

The SimpleDB support is great. I'm prototyping a project I wanted to use SimpleDB on and the Mindscape tools are coming in very handy. You're going to eventually translate .Net data types into something queryable/sortable on the SimpleDB end, right? e.g. DateTime stored in a format like "2009-01-07T16:31:47.284-06:00", numbers with leading zeros, etc. How are you going to handle the item limit on queries? Will Lightspeed transparently load all the records if the query returns more than 250? Something nice would be a feature that got around the 1k attribute value limit. e.g. the 3KB string in MyEntity.Text gets stored in Text__1, Text__2, Text__3

Hello Luke,

Thanks for the feedback.  We're currently considering how to handle storing and querying typed data in SimpleDB.  The plan is indeed to store DateTime in universal sortable format.  Numbers are a bit trickier because we don't know how many leading zeroes to store them with -- I'm currently thinking this may have to be an attribute-based approach.  Or perhaps we could try to infer it from a range validation?  Negative numbers are also a major problem here.  We would love to hear what you feel are the key "must haves" in this area.

In the current build we do not do anything about the item limit: we simply return the results of the initial query.  So if SimpleDB returns a "more results" token, right now it's tough luck.  This is just a limitation of the current state of development, and the 2.2 release will load result sets larger than 250 items transparently.  For simplicity, though, we will be perfoming follow-up queries synchronously (and without limit), so there will be a significant performance impact from large queries (not to mention possible usage charges).  The Amazon API with its stateful "nextToken" cookie is clearly designed to be friendly to paging UIs, but LightSpeed queries are stateless and it will be tricky for us to surface this feature.  Again feedback is welcome.

We currently have no plans to do anything clever in the way of splitting blobs larger than 1K.  It's a cool idea though and we'll keep it on the radar.  Thanks!

Maybe allow folks to customize it, but the defaults should be "all types save and load without the developer worrying about it". I think that follows "convention over configuration". An int32 with a value of 123 might have seven leading zeros but I don't really care while developing. Let me tweak it later if the AWS bill is too high.

Is the Lucene search stuff going to work on SimpleDB? Where are indexes going to be stored? Another feature idea, automatic S3-backing for larger field types like BLOBs and nvarchar(max).

S3 backing for large field types is something we're considering (and would probably replace the attribute splitting idea we discussed earlier in the thread), but won't be in for the 2.2 release.

Hi Luke,

Lucene indexing is available with the SimpleDB connectivity because the full text indexing capabilities of LightSpeed are not provider specific but hook into the entity life cycle. The indexes are stored in a directory  that is configurable.

Appreciate the feature ideas - we often add features based on feedback and are currently interested in seeing how people use our SimpleDB capabilities (both with LightSpeed & SimpleDB Management Tools).

I hope that helps,

John-Daniel

 

Hi Luke,

Just to let you know that we have now implemented the following features:

* Numeric values are stored in SimpleDB with leading zeroes, and dates in sortable format, so that ordering comparisons work as expected on numeric types.  (If you have existing data without leading zeroes, we will still be able to load it, but comparisons will not work as expected.)  Numeric types are always formatted to the maximum width of the numeric type -- thus an int will always take up 10 digits, a long 20 digits and a double 300+ digits.  We recommend not using the double type in entities that back onto SimpleDB!

* If a query returns more than 100 results, we now issue follow-up queries until we have the entire result set.  As previously mentioned, this happens synchronously, so large results sets may take a long time to download.

These features will be included in nightly builds dated 15 Jan 2009 and above, available from  http://www.mindscape.co.nz/products/lightspeed/nightlybuilds.aspx (Express edition) or the store (retail edition).  Please let us know if you run into any bugs or have any feedback.

Why are the nightly builds going backwards? Every time I check that page, it's an earlier version.

Hi Luke,

Thanks for the heads up - we'll have a look into this and post an update when it's fixed.