This thread looks to be a little on the old side and therefore may no longer be relevant. Please see if there is a newer thread on the subject and ensure you're using the most recent build of any software if your question regards a particular product.
This thread has been locked and is no longer accepting new posts, if you have a question regarding this topic please email us at support@mindscape.co.nz
|
Hi guys,
Lately we've been having a bit of a performance issue with our system, which has been running for a while. We made an (unrelated) change to the configuration of the system (ie. changed some records in the database) and now we see this issue. It might have existed before, but only now has it presented itself.
Anyway, after some analysis I found that the problem is a piece of code where we insert a large(-ish) number of records. The system gets a couple of thousand records from various external sources and then goes through these records, creates Lightspeed entities from them and adds these entities to the entity map and at the end calls a SaveChanges(). When actually inserting the data into the database the system starts using resources (ie. memory... For a typical number of rows (20000 to 50000, it can take up to 5Gb (!!!!) of memory) and time.
I've been playing with calling the SaveChanges more often (which I don't really like) and this seems to help a little, but not as much as I would like to.
As I said, I'm adding the entities to the entity map, but I don't really need them on the entity map. I've been looking at the update method to directly create the records in the database, but I don't completely see how this works.
I've checked out the database statements going to the database (SQL Server 2008), but I don't see anything out of the ordinary, except that the system only handles about 5-10 records a second. I would expect this number to be much higher. The table that the data is inserted into does not have any weird indexes or anything.
Do you guys have any pointers towards how I might resolve this? Note that I'm using a data layer based on your RepositoryBase base class, with a strongly typed UnitOfWork. The system is a Windows service, using PerThreadUnitOfWorkScope. I've tried to tweak the updatebatchsize and identityblocksize and this helped a little, but still not enough.
Thanks in advance,
Robert
|
|
|
As an extra piece of information: Storing of the data is done in a thread, separate from the system's main thread, but even when this thread ends, I still don't see the memory usage go down, which seems to indicate the entity map is not cleared at the end of the thread (which, with a PerThreadUnitOfWorkScope, I would expect)...
Hope this helps,
Robert
|
|
|
Yes, 5-10 records per second sounds extremely slow -- I haven't done any benchmarking recently but I think the last time I had to do bulk insert testing I was able to insert thousands or tens of thousands of records per second (admittedly tiny records to a local database, but still). However, without more detailed info it is hard to know what might be causing the problem. I guess the first thing I would do is see if the inserts are taking a long time to execute on the database. A quick and dirty way of doing this is to turn on LightSpeed logging, as this will print the time taken for the database round-trip after each logged batch. SQL Server Profiler will of course provide more additional info from the server side. Basically I think you want to be trying to track down where the time is taken between: * Passing through the LightSpeed code (i.e. how long the SaveChanges takes, minus the time taken on the database round-trip) * Time on the network (i.e. time reported by LightSpeed for the round-trip, minus time taken in the SQL engine) * Time taken by the SQL engine (from SQL Profiler) (You will probably want to test this with smaller data sets -- if the insert rate is 5-10 per second then you should still be able to get meaningful figures with data sets of 10 or so, which will be easier to analyse because they'll be a single batch. If the problem only shows with very large data sets then that's another useful data point and, combined with the memory usage and apparent leakage, suggests something awry with the building and disposal of the database objects within LightSpeed.) If a lot of the time is spent within LightSpeed (and given the memory usage it's possible this is happening, if only due to thrashing), then it would be great if you could put together a repro case so we can profile it here. You could also try profiling it at your end, though unless you have a source code licence this may not be very informative. One thing to check out: You note that the problem only manifested after you made a change to the system configuration. Are you able to roll that change back on a test instance, and see if that restores the previous (better) performance? Finally, concerning the update method: this is unlikely to help you because it only updates existing records using their IDs. It can't be used for inserting new records. |
|
|
Hi Ivan,
Thanks for your (fast, as usual) reply.
I will shortly look into the actual SQL stuff, but I've first attached a Memory Profiler and found one issue which could (at least partially) explain memory usage/leakage.
The issue is that because we have a Windows Service, there is a Main Thread. On this main thread, we have some stuff running (not too much. almost all work is handed off to separate threads). one of these things is a timer which elapses every minute to check if there is some job to re-run. In this handler, a number of entities is retrieved. At the end of the handler these entities seem to remain on the entity map. I would expect that a timer elapsed event handler would run on a seperate thread (and thus, due to PerThreadUnitOfWorkScope with its own Unit Of Work), but this does not seem to be the case.
Since the main thread only exits when the service stops, I would have to remove these entities from the entity map manually. I could (in my service layer) call SaveChanges(true), but this would also save any other pending changes. Probably this is not a big deal, but is there another way to remove/detach entities that are no longer needed from the entity map? can I dereference them? (something along the line of entities.ForEach(e => e = null);)
Next I will check the actual sql going over the wire to see if there is anything obvious there.
Kind regards,
Robert
|
|
|
Spoke too soon... The retry function is spun off in the timer elapsed event handler using a ThreadPool.QueueUserWorkItem call. I've asked a question regarding ThreadPools and PerThreadUnitOfWorkScope before on this forum, so I'll convert this call to a 'real' thread, to see if that improves the situation.
FYI
Robert.
|
|
|
Ok, I've made the modification to have this method run in a real thread and I still see the memory creep up slowly... The Memory Profiler tool has a filter to show objects on the finalizer queue that are not disposed and the one that I see there is my UnitOfWork (I'm not running any jobs at the moment, so all activity is from this timer event handler). If my UnitOfWork is still around, then I guess that the items on the entity map of that UoW also still exist or not?
I'm not 100% sure if this is actually it, as I'm still figuring out how to fully understand the memory profile.
Hope this helps.
Regards,
Robert
|
|
|
Yes, a UnitOfWork retains references to all the entities in its identity map. But if the UOW is on the finalizer queue then it means it is eligible for reclamation, and therefore the entities should also be eligible for reclamation. Does the UOW eventually get removed from the finalizer queue? (E.g. if you call GC.Collect a few times.) If so, does that stabilise the memory use (once it plateaus)? Disposing the UOW rather than waiting for the finalizer is probably a good thing to do anyway. Undisposed UOWs retain entity references and hold connections, which could be starving the database. I have to admit I am leery of the PerThreadUnitOfWorkScope in multithreaded contexts -- I think its disposal logic is incorrect for the multithreaded scenario and suspect that it should really be called the SingleThreadedPerProgramUnitOfWorkScope -- so if you are getting undisposed UOWs then I would suggest that you maybe look at replacing this with a custom scope class or just explicit management of the worker threads' unit of work objects. Let us know if you want support or assistance on this. |
|
|
Hi Ivan,
I'm afraid I won't have time to make a full-fledged PerThreadUnitOfWorkScope, so I'm looking for a way to properly dispose the UoW at the end of a thread. As I said, I'm using the RepositoryBase pattern, so if you have any pointers on how to do this (\Is the RepositoryBase aware of when the thread ends or does it just go out of scope? Or would I dispose of the UoW when the RepositoryBase is disposed (Does it implement IDisposable or should I implement this interface myself?
So many questions, so little time ;-)
But as always your help is greatly appreciated. Your prompt support is one of the main reasons we like your product (besides the great product itself ofcourse).
Regards,
Robert
|
|
|
RepositoryBase is not thread-aware. RepositoryBase is not IDisposable, so you would need to implement the interface yourself (and call dispose when the thread exits). The subtlety of the implementation is that if you have a shared (across threads) repository instance with a unit of work per thread (e.g. via a [ThreadStatic] member), you can't just dispose it when your thread finishes because another thread may be using the same repository -- but with a different UOW. And when it is finally safe to dispose it, it needs to dispose all the UOWs on all threads. (This is pretty much the problem with sharing a PerThreadUnitOfWorkScope across threads.) So your life will be a lot easier if you can use a different repository instance on each thread. In fact, in this case I think you can probably get away with using PerThreadUnitOfWorkScope, provided you use a different scope object per repository (i.e. per thread). Your Repository.Dispose method can then dispose the scope which in turn disposes the UOW. This may be marginally nicer than managing the UOW directly because the scope handles creation, but you may feel that managing the UOW directly gives you better visibility of the UOW lifecycle -- which is clearly important in a diagnostic situation like this. Is "repository per thread + dispose on thread exit" a viable pattern for you? If not, let us know and we can see if we can think of alternatives... |
|
|
Hi Ivan,
I see where you're going, but this is quite a significant change in my app, since we spin off a lot of threads during running of jobs, so I will have to think about it. Since it's my birthday and the end of my working day (yes, I work on my birthday), I'll have to shift this to a later day (hopefully tomorrow, but I also have a deadline on another app, so it might be later).
I might go to a CleanupUoW method on the Repository. Not so nice, but quickest to implement, I guess. It's a bit in line with a solution we discussed earlier on a different thread (no pun intended) on PerThreadUnitOfWorkScope, where we have a (capsulation-breaking) method on the repository to be able to get to the SaveChanges(true) method on the UoW...
If you DO come up with a nicer solution, please let me know. Any of this (multi-threaded UoW scoping) in the pipeline for v3.0?
Kind regards,
Robert
|
|
|
Hi guys,
I've been playing with this a bit more and found out something weird... I implemented a method on my repository DisposeUoW, which disposes the Unit of work and this works fine on my development machine.
However, when I move it to our staging server, which is an x64 machine, the memory used does not seem to get released and thus the amount of memory in use by the service is steadily increasing (I stopped the service when it was up to using 7 gigs).
I then built the service specific for x86 (as opposed to the default Any CPU setting) and now it DOES stabilize and release memory as expected...
When playing around with different options (x64, release vs. debug), I found that only the x86 option works (release or debug makes no difference).
Any idea what could cause this? Just FYI, I created the following method:
public void DisposeUoW()
{
UnitOfWorkScope.Current.Dispose();
UnitOfWorkScope.Dispose();
}
Using only the UnitOfWorkScope.Current.Dispose(); does not yield the expected result, not even on my dev machine.
When calling this method on the repository, at the end of a thread, I (now) also specifically add GC.Collect and GC.WaitForPendingFinalizers. Not exactly as I would like, but (in x86) seems to do the trick...
Kind Regards,
Robert
|
|
|
We've no idea what would cause this behaviour, and we're very interested to reproduce this. Would you be able to provide us with a simple repro project that demonstrates the leakage when built for x64 / Any CPU (and run on x64) but correct behaviour when built for x86? We appreciate that you have a workaround, and that this may therefore no longer be a priority for you, but if you do have time to put something together then we'd definitely like to investigate this further. Thanks! |
|
|
Hi Ivan,
I'll try to get a sample together, although this particular case is stretching the possibilities of Lightspeed (although it works great), so I'll need some time. It IS however the only place in our code where we use the Stored Procedure functionality. Could this somehow have anything to do with it?
I'll try to get back to you with a sample (if I can reproduce it) as soon as I can.
Kind regards,
Robert
|
|
|
Hi Guys,
Just to follow up, I've NOT been able to create a project that reproduces the problem, eventhough I do consistently see it on our actual application (which is obviously a lot more elaborate).
I'll try to find more time to investigate, but as always there are more pressing deadlines on my to-do list...
Regards,
Robert
|
|