Lightspeed and Cursors / Large data sets

Hi guys,

another question, this time regarding lightspeed and whether or not the standard query object uses cursors or not.  I did see the docs regarding Pages - however I don't think that they are really what I was looking for.

Say we have the following :

var query = from p in unitOfWork.Names select p;
foreach (var name in query)
{
  // do something
}

Where Names has a hundred million rows.  What happens - will lightspeed just bring the required objects into memory as we iterate, or will it try and load them all up front?

thanks,

Ben

It will load them all up front.  A LINQ query that returns entities resolves down to a call to IUnitOfWork.Find<T>, which returns an IList<T> rather than a lazy sequence.

hey Ivan,

thanks for the speedy response. 

ok, so I take it that using Page is indeed the way to go? - or is there something (obvious) that I'm missing?

Well, you can use whatever partitioning strategy you want, but yes, paging queries (or the LINQ equivalents, Skip and Take) are probably the simplest and most predictable if you want to iterate over a data set that is too large to load in one go.  Don't forget entities once loaded remain in the unit of work so you will need to do your paged queries in different units of work; and if other apps are writing to the database at the same time then be aware this could throw off your pages.

 nice one, and thanks for the tip about the unitOfWork holding onto older pages (and yes, I was concerned about other apps writing data - which cursors solve (at least with the correct type of db isolation) - but I'll do some sort of higher level sync in my project

cheers,

Ben

Does calling SaveChanges(true) flush entities out from the unit of work so that they are no longer in memory?

It removes references from the UOW's identity map, so the UOW will not be retaining them in memory.  There may of course be other reachable references, which via associations could make the entire entity graph reachable.  If you're loading large numbers of entities, and it's important that they become eligible for collection promptly, then needless to say you'll want to stress test.

As long as the unit of work isn't hoarding references to them after SaveChanges(true) then all is good in the hood.