Memory Leak when Full text search using Lucene

Hello, I'm encountering the below error when doing a search. It was working earlier, but after updated to recent version (a month ago), it have this problem now.

[OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.]
   System.LocalDataStore.PopulateElement(LocalDataStoreSlot slot) +214
   System.LocalDataStore.SetData(LocalDataStoreSlot slot, Object data) +71
   System.Threading.Thread.SetData(LocalDataStoreSlot slot, Object data) +49
   Lucene.Net.Index.TermInfosReader.GetEnum() +68
   Lucene.Net.Index.TermInfosReader.Get(Term term) +40
   Lucene.Net.Index.SegmentReader.DocFreq(Term t) +12
   Lucene.Net.Search.IndexSearcher.DocFreq(Term term) +15
   Lucene.Net.Search.Similarity.Idf(Term term, Searcher searcher) +20
   Lucene.Net.Search.TermWeight..ctor(TermQuery enclosingInstance, Searcher searcher) +58
   Lucene.Net.Search.TermQuery.CreateWeight(Searcher searcher) +29
   Lucene.Net.Search.Query.Weight(Searcher searcher) +34
   Lucene.Net.Search.Hits..ctor(Searcher s, Query q, Filter f) +67
   Lucene.Net.Search.Searcher.Search(Query query, Filter filter) +33
   Mindscape.LightSpeed.Search.LuceneSearchEngine.Search(String query, String[] scopes) +401
   Mindscape.LightSpeed.Search.SearchEngineBroker.Search(String query, Type[] types) +102
   ..ƒ(Query , Type[] ) +156
   Mindscape.LightSpeed.UnitOfWork.Search(Query query, Type[] types) +79
   Mindscape.LightSpeed.UnitOfWorkBase.Search(String searchQuery, Type[] types) +77
   Project.Controllers.HomeController.Search(String s) in ...\Controllers\HomeController.cs:318
   lambda_method(Closure , ControllerBase , Object[] ) +108

Are you able to provide some more details about your situation? As the error is occuring within Lucene rather than LightSpeed this is not something we will be able to specifically debug but we might be able to help track down what is going on here.

• How big is your Lucene index? And roughly how many entities are you indexing?
• Do any of your entities contain large data properties - e.g. blobs?
• What was the query you were performing at the time?
• Does this issue occur for all queries or just specific queries?
• Is the box physically running out of memory?

If there is a repro example you can provide then we can certainly have a look into this locally and try and gain a bit more insight into this, although I would suspect this is highly likely to be dependent on your Lucene index.

Jeremy

How big is your Lucene index? And roughly how many entities are you indexing?

• It is currently 639 KB

Do any of your entities contain large data properties - e.g. blobs?

• No. Only varchar(max)

What was the query you were performing at the time?

• No query, just searching using Lucene.

Does this issue occur for all queries or just specific queries?

• Only during search, but it happened before on all queries (was due to memory leak in webimage)

Is the box physically running out of memory?

• No, but restarting the web app (not IIS) solves the problem.

Sorry, I wasn't particularly clear in the previous post - by queries I am meaning the search calls rather than querying for entities.

Does this occur for all search calls or only when searching for specific terms etc? Also what was the search call you were performing at the time when you get the crash (if this is any term I guess that makes it easy to identify).

Given this is a Web App, does the issue present itself immediately following an app pool restart or only after a certain amount of time?

Can you reproduce the issue if you create a console application and issue a single search call? If so would you be able to send this through and I can have a look into this locally.

Jeremy

It happened on all search terms, I was just searching for a single word, e.g. collection.

The issue only occur after a certain amount of time, restarting the app pool solves the problem. Also, I noticed that after awhile, the web page becomes slower, restarting the app pool return to its snappy response. Probably its related to memory limit / leak per app pool?

I don't it can be reproducible on a console app.

Ok, that does sound like there is a memory leak of some description going on there.

Probably the best way to understand a bit more about this given its probably only reproducible on your local deployment would be to track the memory growth of the w3wp process over time in relation to the requests to see if there is a correlation there. Presumably it will be leaking for all requests or only for certain requests because of something not being disposed properly in those situations.

It would also be useful to see if you disable the use of the search broker entirely if the memory use profile changes as a result - this would help pin down if its specifically leaking out of the use of the search broker or if the leak may be occuring elsewhere but its just due to a large allocation occuring within Lucene that you are triggering the OutOfMemoryException there first.

How are you currently managing your UOW disposal per request for the web application? Also are you caching entities any longer than the per request scope?

Jeremy

Does disabling the search broker break my app?

For now I'm using UOW on per request basis. There are some entities being cached within a minute or so.

Will the cached entities keep the UOW from disposing?

It would certainly break any functionality which directly calls to the SearchBroker so you will need to take that into consideration and work around it, I am assuming you have a dev environment where you are replicating this so disabling this while you run through some test load should only have the impact of generating some 500 responses for requests where you are performing a search.

Just to check, are you using explicitly disposing your UnitOfWork at the end of each request, or are you using an MVC based application and using the LightSpeedControllerBase as the base for your controllers?

Jeremy

I'm currently using the below:

BaseController : AsyncController
{
    protected LightSpeedContext<DBUnitOfWork> db = MvcApplication.LightSpeedContext;

}

HomeController : BaseController
{
   using (var uow = db.CreateUnitOfWork())
   {
   ...
   }
}

If you are scoping all of your UnitOfWork instances in a short lived fashion with a using block as above then that would certainly not lead to any incorrect disposal issues so I guess we can rule that out.

I am at a bit of a loss however as to what might be occurring other than to think there is something which is leaking memory somewhere in your web app and eventually there is a large allocation requested by Lucene in responding to a search query which triggers the OutOfMemoryException.

I think your best course of action would be to try and profile the system as suggested earlier so you can try and understand a bit more about what might be causing the memory leak (e.g. is a specific page, is it every page, if its every page then you could look at whats common etc) and then try and isolate it further.

Jeremy