Memory leak in built application

[Mark Yedinak]

I am running into an issue with a memory leak. Due to NDA I cannot post code. The issue is have is that the deployed exe has a memory leak yet I do not see it when running in the IDE. The code is fairly basic and consists of three tasks. The first task is a basic UI event task and it does updates two strip charts during execution. The second task is a data collection task that retrieves data from a device using HTTP. The third task is a data logging task that writes the data to a database. Messages are passed between the tasks using dynamic events. Each task is basically a while loop with an event structure. Tasks two and three are subVIs on the top level block diagram.

I have scoured the code for any unitialized shift registers or run away arrays. Tasks two and three both open their connections (HTTP and DB) once and use the same connection throughout the execution. Error processing will open a new connection is there is an error but it will cleanup the old connection. Also, I have not recorded any errors so they application should be using the original connections it opened. The DB stuff (using the database toolkit) is executing stored procedures. It does free the resources for each transaction.

I have run the execution trace tool (not on the built exe), profiler and VI analyzer and nothing is jumping out. Any ideas on how to isolate what is causing the memory leak? As I stated the leak only exists in the built exe. Running in the IDE shows memory usage to be very stable. I am running out of places to look. Any thoughts or ideas would be appreciated.

(Sorry about the lack of code, but I'm not at liberity to post it.)

[ShaunR]

My first instinct is that it is probably the "dynamic registration".

You seem to have fairly well partitioned application (into 3 parts). I would suggest making 3 separate exes with each of the parts and see which one (if any) the leak occurs in. That may narrow it down a bit.

I'm afraid however, without any code. for us It's like trying to swat a fly blindfolded.

[Mark Yedinak]

My first instinct is that it is probably the "dynamic registration".

You seem to have fairly well partitioned application (into 3 parts). I would suggest making 3 separate exes with each of the parts and see which one (if any) the leak occurs in. That may narrow it down a bit.

I'm afraid however, without any code. for us It's like trying to swat a fly blindfolded.

Yes, I do realize it is very difficult to give advice without the code. Anything in particular to look at with the dynamic registration? I have two sets of user events that the tasks register. One is what I call application events that are general events like start, stop, exit, etc. The other is application specific events which as the name appies is events specific to this application. I use an AE for each set of the events and separate VIs to actually post the specific events. The events only get registered once in each of the tasks.

Your idea abouth separating the tasks is interesting but I would need to have some different method of passing the messages between them if they were separate applications. the events would not pass application boundaries. Without the messaging only the data collection task would actually do anything. The other two are acting on messages out of the collection task.

[ShaunR]

Yes, I do realize it is very difficult to give advice without the code. Anything in particular to look at with the dynamic registration? I have two sets of user events that the tasks register. One is what I call application events that are general events like start, stop, exit, etc. The other is application specific events which as the name appies is events specific to this application. I use an AE for each set of the events and separate VIs to actually post the specific events. The events only get registered once in each of the tasks.

Nothing specifically. It's not a technique I have much of a use for so my "instinct" is probably biased to that end. But realistically, it could be anywhere.

Your idea abouth separating the tasks is interesting but I would need to have some different method of passing the messages between them if they were separate applications. the events would not pass application boundaries. Without the messaging only the data collection task would actually do anything. The other two are acting on messages out of the collection task.

Not sure how easy it is with your code, but what I quite often do is just put indicators on each of the message input queue to collect them in arrays. Then when you have a few, change it to a control, save as default, and just play them back over and over again with a while loop in the module.

[crelf]

You seem to have fairly well partitioned application (into 3 parts). I would suggest making 3 separate exes with each of the parts and see which one (if any) the leak occurs in. That may narrow it down a bit.

I agree - even if you can't partition them that well, just start removing functionality until it levels out - a laborious task, I know, but without being able to see the code, it could be anything and everything.

[Mark Yedinak]

Not sure how easy it is with your code, but what I quite often do is just put indicators on each of the message input queue to collect them in arrays. Then when you have a few, change it to a control, save as default, and just play them back over and over again with a while loop in the module.

This would probably work. I think I would try this with the HTTP and DB tasks. I suspect it is one of those causing the issue and not the UI task. My first inclination is the DB task but there isn't much there on my end and the tollkit VIs are password protected so it is hard to say what they are doing under the hood.

[ShaunR]

This would probably work. I think I would try this with the HTTP and DB tasks. I suspect it is one of those causing the issue and not the UI task. My first inclination is the DB task but there isn't much there on my end and the tollkit VIs are password protected so it is hard to say what they are doing under the hood.

Well. If it doesn't show up in those two then, by elimination, you know it must be the third one.

Which toolkit is it? The NI one isn't password protected.

[Aristos Queue]

Do you make any DLL calls in your code? The execution trace tool cannot generally track allocations made in DLLs.

[Anders Björk]

I am running into an issue with a memory leak. Due to NDA I cannot post code. The issue is have is that the deployed exe has a memory leak yet I do not see it when running in the IDE. The code is fairly basic and consists of three tasks. The first task is a basic UI event task and it does updates two strip charts during execution. The second task is a data collection task that retrieves data from a device using HTTP. The third task is a data logging task that writes the data to a database. Messages are passed between the tasks using dynamic events. Each task is basically a while loop with an event structure. Tasks two and three are subVIs on the top level block diagram.

I have scoured the code for any unitialized shift registers or run away arrays. Tasks two and three both open their connections (HTTP and DB) once and use the same connection throughout the execution. Error processing will open a new connection is there is an error but it will cleanup the old connection. Also, I have not recorded any errors so they application should be using the original connections it opened. The DB stuff (using the database toolkit) is executing stored procedures. It does free the resources for each transaction.

I have run the execution trace tool (not on the built exe), profiler and VI analyzer and nothing is jumping out. Any ideas on how to isolate what is causing the memory leak? As I stated the leak only exists in the built exe. Running in the IDE shows memory usage to be very stable. I am running out of places to look. Any thoughts or ideas would be appreciated.

(Sorry about the lack of code, but I'm not at liberity to post it.)

Which labview version are you using for this? I have a weak memory that the db-toolkit had some problem memory leaks.

[Rolf Kalbermatter]

Which labview version are you using for this? I have a weak memory that the db-toolkit had some problem memory leaks.

My first hunch before hearing about the password protected DB VIs was also the DB task. I know of some problems with result sets in the NI DB Toolkit. But as Shaun already said, I can't imagine it to be the NI DB Toolkit since that one is NOT password protected. And as far as I remember, the result set memory leak exists in the IDE too.

[Daklu]

Can you correlate the memory leak with any particular application activity? Is the growth rate the same when the app is doing nothing vs when it is engaged in an activity?

One thing that might be useful is Dr. Damien's WinDbg logging vi. He told me about it at the summit and I have found it extremely useful for grabbing quick debug logs from executables. I believe it is in his nuggets thread but I don't recall exactly.

(Every so often someone posts about an unusual problem they're having with an app using a user-event based messaging system. I won't claim anyone is wrong for using it, but I honestly don't understand its appeal. Compared to a queue based messaging system it is less flexible, less capable, and less transparent. Even if this leak isn't related to user events at the very least a queue based system would let you eliminate that as a source of errors. Is it that user events support built-in dynamic registration/unregistration and queues do not?)

Not sure how easy it is with your code, but what I quite often do is just put indicators on each of the message input queue to collect them in arrays. Then when you have a few, change it to a control, save as default, and just play them back over and over again with a while loop in the module.

Smart. I hadn't thought of that.

[jzoller]

In a similar fashion to grabbing the actual queue, you might consider creating a fourth component that just sends out messages to the other three. Then, "spike" the messages with a heavy bias towards suspect states, and see which cause the largest memory increase.

You could also mock or fake existing components to isolate the one of the others...

2 cents

Joe Z.

[Mark Yedinak]

(Every so often someone posts about an unusual problem they're having with an app using a user-event based messaging system. I won't claim anyone is wrong for using it, but I honestly don't understand its appeal. Compared to a queue based messaging system it is less flexible, less capable, and less transparent. Even if this leak isn't related to user events at the very least a queue based system would let you eliminate that as a source of errors. Is it that user events support built-in dynamic registration/unregistration and queues do not?)

The beauty of user events is that you effectively get a one to many queue. Notifiers are one to many but can only contain the lastest data. User events will queue up th eevents like a queue but the sender only needs to generate a single event to broadcast it to many parts of the application. Queues on the other hand are many to one. To get a broadcast message either the lower level code has to know everyone who will get the message or you implement some middle man task to receive the message and post it to the queues for everything (publish and subscribe) interested in getting it.

[Mark Yedinak]

Do you make any DLL calls in your code? The execution trace tool cannot generally track allocations made in DLLs.

And no, no DLL's that I know of. If the password protected stuuf is calling something though I would have no way of knowing.

[asbo]

Can you build a debug executable, connect to it, and then use the built-in profiler to see if any VIs crop up with large memory usage?

[JamesMc86]

Another couple of ideas to add:

1. You can run the desktop execution trace toolkit on an exe (with VI server enabled), though I suspect the sort of issues this would see would be consistent with the IDE or EXE.
   
2. I would use perfmon to monitor the memory and handles. File, network connection etc. reference leaks will tend to show an increase in handle count.
   

Bar that I think breaking it down is the good first step. Knowing the rate of the leak and your main data paths can sometimes point to specific sections of code as well, but normally only when you have a few big data sets that you are moving around.

[Mark Yedinak]

OK, I haven't broken the application down into the component parts yets but did rebuild it on my local machine (this application actually runs at a client site) and monitored it overnight. The memory is slowing increasing and the stats from perfmon shows that the handle count is extremely high. The handle count was well over 7000000 after about 6.5 hours of running. The memory size was 5 times greater than when the application started. It is colecting the data from a remote site using HTTP once per second. It then logs that same data into a DB. Both the HTTP connection and DB connection are left open. Does the extreme handle count provide any clues where to look. I will break it into three separate applications for test purposes but thought the handle count may point to a specific area.

[Daklu]

The memory is slowing increasing and the stats from perfmon shows that the handle count is extremely high.

You don't happen to have a screenshot do you? I don't recall perfmon having a "memory handle count." Maybe I'm using a different version...

Anyhoo, 7 M "memory handles" over 6.5 hours is roughly 300 memory handles per second. I assume you don't have any loops going at 300 Hz or some fundamental frequency of 300 Hz?

[JamesMc86]

It is just listed as handles. You can also view them in task manager.

Each handle is a OS object and you would not expect this to be anywhere out of the 1000s. I know files would represent an object, network sockets beyond that I am not sure but I do know it is likely to be a reference leak if it is a handle leak. There is quite a detailed document at http://blogs.technet.com/b/markrussinovich/archive/2009/09/29/3283844.aspx but I can't say I've tried tried these myself before. I think Daklu's question is a good place to start as this is a pretty large rate of handle loss.

[Daklu]

There is quite a detailed document at...

Russinovich's articles are always good. I wish I had 6 months to just sit down and read his stuff. Of interest was this comment,

"Any process that has more than a ten thousand handles open at any given point in time is likely either poorly designed or has a handle leak..."

So I checked...

[drjdpowell]

I was investigating the resources used in loading large numbers of subVI clones a month ago. There were 5 handles opened per clone (for a clone with no meaningful code in it other than a short wait).

Edit later: actually it’s 3 handles.

[Aristos Queue]

"Any process that has more than a ten thousand handles open at any given point in time is likely either poorly designed or has a handle leak..."

Or the process is a dataflow execution environment that uses the heap instead of the stack for much of its program execution, thus many function calls are handles that would be stack entries in other programming languages.

Or the execution environment is being tricky. I am not a C# expert, but the bit I know about their execution under the hood, they have hundreds of "handles", but they ask for a single block of memory from the operating system, which shows up as 1 handle, and then manage that very large block themselves through the garbage collector.

I would suggest it is a decent rule of thumb for applications written in C/C++, but probably doesn't apply to other execution environments/compilation patterns.

[Rolf Kalbermatter]

Or the execution environment is being tricky. I am not a C# expert, but the bit I know about their execution under the hood, they have hundreds of "handles", but they ask for a single block of memory from the operating system, which shows up as 1 handle, and then manage that very large block themselves through the garbage collector.

I believe LabVIEW used to do that too, when it was including the Great Cycle Smart Memory Heap Manager. Or were there two separate memory manager backends that LabVIEW could use? I'm not sure anymore and the details are fading, but fact is that LabVIEW had some intermediate memory manager layer on top of the OS memory manager that allowed it to not only debug memory usage on a more detailed level, but also consolidate many individual memory request by LabVIEW into much fewer and bigger ones for the OS.

Great Circle was absorbed at some time and is of little significance nowadays other than for special, bad performing applications that get patched up by using Great Circle . Smartheap is apparently still on sale at http://www.microquill.com/.

So it seems LabVIEW had at some point the option to replace the standard C Runtime memory manager functions with different backends, and was also shipped with them at some point. This supposedly helped to debug memory issues, and may still be used internally but could be an option to reduce the handle hunger from LabVIEW. So that LabVIEW is even by Mark Russinovich's standards a well behaving application .

[Mark Yedinak]

No, The data collection loop is running at a whopping 1 sample per second. The display and logging tasks are driven by the collection tasks. That is what is so baffling about this leak. This application is running at a very slow pace and to see the increase in handles and memory over a short period of time is quite surprising. I will be digging into this more later. I don't think I'll get a chance to work on it today though.

Thanks to all for the suggestions.

[0_o]

I would try one last thing:

copy the vi that causes the trouble to another new vi and remove any advanced GUI controls (silverlight, customized...).

If this doesn't help copy it part by part and check when the leaks start.