Enhancements to LINQ provider for LightSpeed 3.0
Tagged as In LightSpeed 2.0 we introduced support for LINQ queries through our LightSpeed LINQ query provider (Mindscape.LightSpeed.Linq) which has been universally appreciated by our customers; however we have heard from customers who wanted to write some fairly sophisticated queries and found that our LINQ provider has some restrictions and omissions – notably support for explicit joins, groupings and certain type of set operators.
The LINQ provider in LightSpeed 2.0 was all about translating LINQ expressions back against our existing query API. In LightSpeed 3.0 we are extending the query API to support these additional scenarios and to that end we have added in support for join’s, groupings, multi-part traversals and better support for translation of functions that you would generally expect to work.
Can you just make it so any query I write will just work?
The job of our LINQ provider is to translate an expression which you have expressed in code into a query against the underlying data source in a sensible way to return you the correct result – which some times may be an entity, a collection of entities, or even a projection to a totally new type. You are also allowed to chain queries, such that you could select a collection of entities and project them to a new set, then transform that set, then select a subset of that query, filter it and then project it to a new set (and so on) – so while its perfectly valid for you write that query, translating that to a sensible query for LightSpeed may tricky or even may not be possible at all.
When an expression is defined, we first need to translate that into its client and server components – where a server component is something we can execute against the database, and the client side part is something we need to do in memory using LINQ to Objects. Why? Because it might not be possible for us to have the server actually do all the work which you expect. A simple example is where part of your query contains a property on the .NET class rather than being against a database column – to handle these we either need to translate this property into something which can be extrapolated from the underlying database row data or we have to evaluate the property once the object has been bought back into memory.
Currently we throw exceptions where we encounter a situation we cant directly handle, rather than fetching the server component and doing the rest in memory – you can always get around this by forcing the results to be evaluated by calling .ToList() or the like, but the reason we do this is to avoid situations where you might end up writing a query which selects a Cartesian product and then has to do the rest in memory – in general we think we should be continuing with this approach – what do you think?
Calling all LINQ queries!
Do you have a query you want to see serviced by LightSpeed 3.0? If so, email us a small sample of your model and the query (which should currently fail!) or post it up in our forums and we will add it to our test suite. While we cant guarantee to support everything for the reasons I mentioned above, we will be working hard to make sure everything that is sensible is supported and we would love to hear your thoughts on this matter :)
Posted by Jeremy Boyd on 26 March 2009 
18 Responses to “Enhancements to LINQ provider for LightSpeed 3.0”
Leave a Reply
Categories
BrainDump (1)
Community Code (4)
Events (15)
F# (11)
General (50)
Lab Samples (2)
LightSpeed (249)
MegaPack (7)
News (68)
NHibernate Designer (18)
Nightly news (40)
Phone Elements (22)
Products (87)
Projects (5)
Screencast (6)
SharePoint (3)
Silverlight (14)
Silverlight Elements (59)
SimpleDB Management Tools (20)
Visual Studio (9)
VS File Explorer (7)
Web Workbench (20)
WPF (43)
WPF Diagrams (53)
WPF Elements (91)
WPF Property Grid (32)
Hi Jeremy,
I know it is _very_ hard to write a proper LINQ provider for already a cooked and working O/RM, it has been discussed all around the web couple of times.
I haven’t dived completely into this topic, but aren’t you hitting a dead end somehow with “translating LINQ queries to an LS API”?
I think the same “temporary” approach was done by NHibernate folks which (for the sake of having it quickly) translated their LINQ queries to an existing Criteria Query Lng but they logically hit an end how far they can go with it.
NHibernate (by surprise of many) didn’t have any AST parser for their queries (for HQL it worked with an object model) so now they are working on of having one included in future releases.
It will translate HQL/LINQ queries, for which they are using already finished Java project (so they are “just” porting it):
http://www.antlr.org/
I know NHibernate stands on the opposite side of a competition for you (being the most complex to work with, supporting legacy DBs and complex mapping) so its object model and capabilities are much broader, so having a proper LINQ provider supporting numerous NHibernate features probably required it.
Just my 0.02$, as the whole idea of LINQ is to hit DB engine (with proper optimized SQL queries for each specific DB) instead of doing filtering in memory.
Sorry if that is an advert on NHibernate, I used it as an comparison of LINQ providers (NH still doesn’t have any LINQ finished yet) as any developer which used the 2 O/RMs can’t really compare them.
Both are used for different projects, and whenever we can we would choose LS without a question.
Great work guys anyway (JOIN support sounds sweeeeet!), how many Microsoft LINQ test have you passed btw?
Hey guys I would love to see support for translation of LINQ extention methods.
from t in PartDefinitions
where t.Parts.Contains(p=>p.CutOn==null)
select t;
You’re making me a little nervous by explaining why you can’t get all LINQ queries to work and calling for examples from others to test :)
I think most customers would expect you to be on par with LINQ To SQL and/or the Entity Framework, and if not, provide examples as to where you fall short.
Sounds like you guys are making progress, however, so kudos to that!
Thanks for the comments guys.
@DavidHayden – “All queries” would mean you need to defer work to L2O a lot of the time (because of the client/server components in most sophisticated queries. Our baseline is L2S and Microsoft 101, we could do more, but they throw on a lot of cases where it would be sensible for us to throw as well when handling certain client side scenarios. We are worried about you being able to write a query where the only executable server part would be SELECT * FROM TABLE and the rest having to be done client side because of a .NET method in the next section of the expression tree. Should we assume you know best, or should we throw similar to most other LINQ providers when certain ops are not sensible to translate?
@nefajciar – Ideally we want to convert most if not all queries to the L2S API for execution because of provider specific nuances (ie.. we dont want to make DB provider specific assumptions in our LINQ provider), currently we are passing most of the 101 tests.
@MiddleTommy – thanks, we will set up a test case for that one :)
Jeremy,
I trust you guys to make the right decision as I have not tried to create a LINQ provider and hence don’t know all the ins and outs.
Again, I just think you guys need to be on par with LINQ To SQL and the Entity Framework, and if you are not, let us know why.
I think with LightSpeed now being at version 3.0 and the potential licensing change there shouldn’t be any surprises or deficiencies in the LINQ implementation when comparing to the Microsoft implementations we get out-of-the-box.
Regards,
Dave
If you want sophisticated linq queries to test, check my weblog on the articles about writing linq to llblgen pro :)
About mixing linq to objects with linq to database, it’s easy to find those, just use a funcletizer (in one of my articles I mention it) and compile them up-front, then also compile the projection lambda after you’ve replaced db element referencing expressions with referals to an object you pass as input to the compiled expression (i.e. the row coming from the data-reader).
The complex issues with linq providers are:
- selectmany
- groupby
- multiple-hop referals in mixed joins (nested froms + defaultifempty + regular joins
- nested queries in projection.
And a truckload of other details, you’ll likely run into along the way.
Hi Frans,
Thanks for the comment. We enjoyed your in depth coverage of writing a LINQ provider – it’s always great to see vendors sharing their experiences.
Effectively our initial LINQ provider included everything except the complex items you listed. LightSpeed 3.0 is not complete yet but we have all those types of queries now working in our tests and hence why we’re not looking for the crazy edge cases.
It raises an interesting issue – what constitutes a complete LINQ provider. We initially started by thinking passing the LINQ 101 cases was sufficient (and what the folks @ Telerik believe is enough) however it didn’t take long to realise that LINQ 101 cases are overly simplistic and don’t reflect the real world all too well. I wonder if there is a case to be made for an open source query set which could be used by various vendors to validate their capabilities?
By the way, Jeremy would have responded but he’s off on holiday for the week.
Thanks for the comment :-)
LightSpeedLsgenHelper class that could be use together with other code generation system
- columns, private/public
- relations, private/public
- attributes for the current column/relation
- everything in “LightSpeed” naming conventions
@csa – We’re looking at ways of exposing our meta model in LightSpeed 3.0 which should further help folks who are wanting to build tooling support on top of LightSpeed.
:-)
var g = from t in work.MSizes
group t by new {t.type, t.size} into grp
select grp;
Doesnt work with the latest build
Hi MiddleTommy,
Support for the join and group by operators is being added only in the 3.0 release and isn’t included in the 2.x nightly builds. Thanks for the test query though!
Thanks a million
Any update on the release of 3.0? I’ve downloaded the product, showed my team at work, but without being able to do LINQ joins we can’t justify purchasing the product.
Hi Tim,
Thanks for your feedback – we’re absolutely racing to get the first beta’s of LightSpeed 3.0 out the door but our original time frames are currently likely to stand.
We should be in a position to start talking about concrete dates in several weeks.
Thanks for your feedback, I hope this helps.
Tim,
Perhaps this will help you for the time being: http://www.mindscape.co.nz/blog/index.php/2009/06/08/join-queries-in-lightspeed-2/
I hope that helps.
The Lightspeed 3.0 LINQ Projections have me perplexed… if I project just three columns/properties from a single table using LINQ, the actual T-SQL that is executed selects all columns for the entity.
This isn’t optimal, because it means that large columns that are stored out-of-table are always retrieved, whether they’re needed or not. This, in turn, means that the server and the network are unnecessarily taxed.
Is there a deliberate reason for this, or should we hold our breath for future improvements in the LINQ provider?
LINQ to LightSpeed normally perform projections server-side, and selects only the columns that are required for the projection. However, if it can’t translate the projection into a set of columns for a SELECT, it may decide to retrieve the full entities and run the projection client-side. If you could post the query that was resulting in a full entity load, we’d be happy to identify why it wasn’t doing a limited select (and either fix it or suggest ways of changing the query so that the projection did happen server-side). Thanks!