Jump to content

Memory grab after several hours of running


Recommended Posts

Hi all,

I need a LAVA category of "Strange behaviour that I can't explain!"

Here's the problem: I dynamically call a vi that, in essence, reads in data via TCP (64kS/s), buffers that data if needed, performs an FFT on it, and displays the results on a waveform graph. This program will run just fine for hours, when all of the sudden it starts chewing up memory, at the rate of about 200k per second (as reported in the "Processes" tab of Task Manager), until the machine runs out of memory and crashes. If I close the vi in the midst of the memory grab and reopen it, it's fine again (until the next time it happens). This bug doesn't always occur (I've run it for days with no problem) but it happens often enough that it's more than just an annoyance.

This has happened in both 7.1.1 and 8.5.1, on a variety of computers, all running WinXP. The code has been built into an executable. I've never had the source code show this behaviour, but it hasn't been used as much as the exe.

Any thoughts?

Cat

Link to comment

QUOTE (Cat @ Mar 3 2009, 08:08 AM)

Hi all,

I need a LAVA category of "Strange behaviour that I can't explain!"

Here's the problem: I dynamically call a vi that, in essence, reads in data via TCP (64kS/s), buffers that data if needed, performs an FFT on it, and displays the results on a waveform graph. This program will run just fine for hours, when all of the sudden it starts chewing up memory, at the rate of about 200k per second (as reported in the "Processes" tab of Task Manager), until the machine runs out of memory and crashes. If I close the vi in the midst of the memory grab and reopen it, it's fine again (until the next time it happens). This bug doesn't always occur (I've run it for days with no problem) but it happens often enough that it's more than just an annoyance.

This has happened in both 7.1.1 and 8.5.1, on a variety of computers, all running WinXP. The code has been built into an executable. I've never had the source code show this behaviour, but it hasn't been used as much as the exe.

Any thoughts?

Cat

One way to get more info about the conditions when the program crashes would be to run a network protocol analyzer in the background (Wireshark is a good choice) to log the TCP/IP traffic. Then, when the program goes haywire, you have a record of what was happening to the network at least. If this log doesn't show any weirdness, then you can start looking at other parts of the program. This should be relatively easy to do (download Wireshark http://www.wireshark.org/ ) and install it and set up message logging on the stream in question.

Mark

Link to comment

QUOTE (Cat @ Mar 3 2009, 04:08 PM)

Hi all,

I need a LAVA category of "Strange behaviour that I can't explain!"

Here's the problem: I dynamically call a vi that, in essence, reads in data via TCP (64kS/s), buffers that data if needed, performs an FFT on it, and displays the results on a waveform graph. This program will run just fine for hours, when all of the sudden it starts chewing up memory, at the rate of about 200k per second (as reported in the "Processes" tab of Task Manager), until the machine runs out of memory and crashes. If I close the vi in the midst of the memory grab and reopen it, it's fine again (until the next time it happens). This bug doesn't always occur (I've run it for days with no problem) but it happens often enough that it's more than just an annoyance.

This has happened in both 7.1.1 and 8.5.1, on a variety of computers, all running WinXP. The code has been built into an executable. I've never had the source code show this behaviour, but it hasn't been used as much as the exe.

Any thoughts?

Cat

Hi,

Well.. How big is you code ? I mean how many VIs more or less.. Have you identified which VI in your code is chewing up your memory ?

It's hard to have an intelligent guess without seeing the code.. can you post it ?

Link to comment

QUOTE (Antoine Châlons @ Mar 3 2009, 10:40 AM)

Well.. How big is you code ? I mean how many VIs more or less.. Have you identified which VI in your code is chewing up your memory ?

It's hard to have an intelligent guess without seeing the code.. can you post it ?

The whole executable has around 2000 vis in it, give or take a few. The section of the code where the problem occurs is around 100 vis. I can tell you that the Out of Memory error occurs when buffering the data, however, I can't tell if that's actually what's *causing* the problem. As I said I can run this for days with no problem, so it's hard to troubleshoot. I've never known it to fail in less than 2 or 3 hours, tho, if it was going to fail (which makes it very time consuming to troubleshoot). If I run the VI metrics tool that comes with LV, nothing stands out as a problem. But then again, maybe it just wasn't going to fail those times I tested it, or maybe it only occurs in the executable.

I know this is a tough one, especially since I have so few details to give. I guess I'm hoping someone out there has seen something like this happen before and can point me to something to fix, or at least commiserate. :-)

Cat

Link to comment

Can you save the data and process offline to see if there is something in the data causing your processing to leak memory?

Can you isolate just the TCP stuff and run that to see if its the communication thats causing the problem?

Are there any other processes running on your machine that could cause issues?

Can you run the processing routines in a loop (without the TCP) to see if there is something in there causing a problem?

Are you using a statemachine architecture? I usually have a "Next State.vi" to flip states. I can log every state transition that occurs. This helps isolate the states where trouble is brewing.

post-2680-1236104585.jpg?width=400

There is also a version that can be used to log states into a string, but this can grow very large so is to be used with caution for limited runs:

post-2680-1236104696.jpg?width=400

Troubleshooting is never easy, and the basic principles are the same: isolate modules and figure out which one is causing the problem, then fix it (easier said than done) :headbang:

Neville.

Link to comment

QUOTE (neBulus @ Mar 3 2009, 10:46 AM)

The http://sine.ni.com/nips/cds/view/p/lang/en/nid/206790' target="_blank">execution trace toolkit should help to nail that issue.

It works with exe's and should "lift the hood" to let you peek in at what is happening while the memory is being goobled.

Coolio!! (as my 13 yr old daughter would say)

I got about 10 words into the description of it to my normally tightfisted team leader, and he said "Buy it!!"

I'm getting a new toy! I mean, I'm getting an important tool to help make my software more efficient and bug-free...

Thanks for the suggestion!

Cat (I think I just used up my exclamation point quota for the day! :D )

Link to comment

Are you sure you are using a Waveform Graph and not a Waveform Chart? If it is a Waveform Chart, I remember having a similar problem in LV 8.5. I was updating the chart by wiring an array to the chart terminal. On Task Manager I saw memory being chewed up. I tried to use the Performance and Profile Window to find any memory leaks but couldn't find any (are you sure you used VI Metrics and not the Performance and Profile Window?).

Finally I found out the problem was the "Chart History Length" of the chart. Whenever the size of the array was larger than the chart history length, the extra data was apparently being stored in memory and that was what was chewing up the memory.

If this turns out to be the problem, then you could try increasing the chart history length, OR make sure the array you feed to the chart terminal only has as many elements as the chart history length. I'd be curious to hear if that was the problem.

I tried recreating this in LabVIEW 8.6 and couldn't.

QUOTE (Cat @ Mar 3 2009, 09:08 AM)

Hi all,

I need a LAVA category of "Strange behaviour that I can't explain!"

Here's the problem: I dynamically call a vi that, in essence, reads in data via TCP (64kS/s), buffers that data if needed, performs an FFT on it, and displays the results on a waveform graph. This program will run just fine for hours, when all of the sudden it starts chewing up memory, at the rate of about 200k per second (as reported in the "Processes" tab of Task Manager), until the machine runs out of memory and crashes. If I close the vi in the midst of the memory grab and reopen it, it's fine again (until the next time it happens). This bug doesn't always occur (I've run it for days with no problem) but it happens often enough that it's more than just an annoyance.

This has happened in both 7.1.1 and 8.5.1, on a variety of computers, all running WinXP. The code has been built into an executable. I've never had the source code show this behaviour, but it hasn't been used as much as the exe.

Any thoughts?

Cat

Link to comment

QUOTE (jasonw @ Mar 3 2009, 03:57 PM)

It took that long to get to the 'crelf on LAVAG said' part of the description. :laugh:

Well, yes, then it took another 10 minutes attempting to describe what a "crelf" is. :P

Link to comment

QUOTE (crelf @ Mar 3 2009, 04:29 PM)

...

Although, neBulus beat me to it :)

In all fairness I made that recomendation based on something that crelf said in a meeting about a month ago. :ninja:

So like most stories here on LAVA they end with a carrige return and Line-feed with a little "e' thown in to allow access to the imaginary*.

Ben

* If you think that was bad, I can out do that! Here is a LAVA palindrome.

"Avalon did no LAVA"

Link to comment

QUOTE (Cat @ Mar 4 2009, 06:54 AM)

Ha! I actually LOLed :D My wife has long since given up trying to explain what I am ;)

QUOTE (neBulus @ Mar 4 2009, 08:57 AM)

So like most stories here on LAVA they end with a carrige return and Line-feed with a little "e' thown in to allow access to the imaginary.

*sniff* that's, that's beautiful Ben! *sniff*

Link to comment
  • 3 weeks later...

QUOTE (neBulus @ Mar 3 2009, 11:46 AM)

The http://sine.ni.com/nips/cds/view/p/lang/en/nid/206790' target="_blank">execution trace toolkit should help to nail that issue.

It works with exe's and should "lift the hood" to let you peek in at what is happening while the memory is being goobled.

So I have my new toy (Desktop Execution Trace Toolkit), and I'm having all sorts of problems running it. The example project they send along is trivial, to say the least. My project has a lot of vis doing a lot of things, and the trace runs out of space (999999 event limit) very quickly. I know there's filtering, but it seems to work only intermittantly, or not at all. You maybe able to run it with executables, but you don't know what vi is generating the event in that mode, so it's not really very helpful. And as far as I can tell in executable mode you can only start it up once (at least when running source code, if your event queue fills up you can stop and start it again). Oh, and I manage to crash it once or twice an hour.

I've searched around the NI site and can't find anything more useful than what little online help the software comes with. I've spent 2 days trying to get it configured for what I need to do and keep running into road blocks. I'm kinda frustrated, because this is the first time my team leader has actually cracked open the wallet to pay for something like this, and if I can't figure out how to use it, it may be the last time. I must admit, I'm expecting a lot from something we just paid $999 for. Maybe I shouldn't be expecting so much...

I know this is a relatively new product, but does anyone out there have experience with it and wouldn't mind answering a few questions?

(As a side note, is there some way I can change this topic to "Desktop Execution Trace Toolkit" or something else more pertinent?)

Link to comment

QUOTE (Cat @ Mar 23 2009, 11:33 AM)

So I have my new toy (Desktop Execution Trace Toolkit), and I'm having all sorts of problems running it. The example project they send along is trivial, to say the least. My project has a lot of vis doing a lot of things, and the trace runs out of space (999999 event limit) very quickly. I know there's filtering, but it seems to work only intermittantly, or not at all. You maybe able to run it with executables, but you don't know what vi is generating the event in that mode, so it's not really very helpful. And as far as I can tell in executable mode you can only start it up once (at least when running source code, if your event queue fills up you can stop and start it again). Oh, and I manage to crash it once or twice an hour.

I've searched around the NI site and can't find anything more useful than what little online help the software comes with. I've spent 2 days trying to get it configured for what I need to do and keep running into road blocks. I'm kinda frustrated, because this is the first time my team leader has actually cracked open the wallet to pay for something like this, and if I can't figure out how to use it, it may be the last time. I must admit, I'm expecting a lot from something we just paid $999 for. Maybe I shouldn't be expecting so much...

I know this is a relatively new product, but does anyone out there have experience with it and wouldn't mind answering a few questions?

(As a side note, is there some way I can change this topic to "Desktop Execution Trace Toolkit" or something else more pertinent?)

Probably doesn't make you feel any better, but I was at the local NI Developer Day last week and they demoed the execution trace toolkit. The guy running the demo (who's a pretty sharp LabVIEW guy from the NI home office) managed to crash it and had a hell of a time to get it to do what it was supposed to. He admitted he wasn't very familiar with the tool, but I did get the distinct impression it may not be ready for prime time. When it did work you could see how useful it can be when it does work. NI generally makes good stuff, but they may have jumped the gun on release of this product.

Sorry I can't really help.

Mark

Link to comment

QUOTE (Cat @ Mar 23 2009, 12:33 PM)

So I have my new toy (Desktop Execution Trace Toolkit), and I'm having all sorts of problems running it. The example project they send along is trivial, to say the least. My project has a lot of vis doing a lot of things, and the trace runs out of space (999999 event limit) very quickly. I know there's filtering, but it seems to work only intermittantly, or not at all. You maybe able to run it with executables, but you don't know what vi is generating the event in that mode, so it's not really very helpful. And as far as I can tell in executable mode you can only start it up once (at least when running source code, if your event queue fills up you can stop and start it again). Oh, and I manage to crash it once or twice an hour.

I've searched around the NI site and can't find anything more useful than what little online help the software comes with. I've spent 2 days trying to get it configured for what I need to do and keep running into road blocks. I'm kinda frustrated, because this is the first time my team leader has actually cracked open the wallet to pay for something like this, and if I can't figure out how to use it, it may be the last time. I must admit, I'm expecting a lot from something we just paid $999 for. Maybe I shouldn't be expecting so much...

I know this is a relatively new product, but does anyone out there have experience with it and wouldn't mind answering a few questions?

(As a side note, is there some way I can change this topic to "Desktop Execution Trace Toolkit" or something else more pertinent?)

Hi Cat,

Please tell me you have been talking with NI Support to address those issues (please).

That is one of the new widgets NI is pushing so you should be able to get some decent support.

Ben

PS After you have your issues solved and you become our expert on this tool, you may want to take a look http://forums.ni.com/ni/board/message?board.id=BreakPoint&view=by_date_ascending&message.id=409#M409' target="_blank">at this Sea Story I posted over on the Dark-Side.

Link to comment

QUOTE (mesmith @ Mar 23 2009, 02:24 PM)

Actually, it *does* make me feel better, in a small small way. :) At least I'm not a complete idiot; it may be the software and not me. Maybe...

QUOTE (neBulus @ Mar 23 2009, 02:25 PM)

Please tell me you have been talking with NI Support to address those issues (please).

That is one of the new widgets NI is pushing so you should be able to get some decent support.

Well...... I have never had a lot of luck with NI support. Generally I've thrashed a problem up one side and down the other before I call them, and by that time I've tried everything they can suggest. Then it becomes a bug report...

My favorite hardware experience was with the GPIB/ENet converter box. I think my serial number for that thing was less than 10. I was all excited to put it in my system. After I did, my program that had been taking a few hundred milliseconds to respond to commands started taking many seconds, sometimes minutes. After persistently bugging NI support for several days, they finally came back with the infamous line: "It's just going to be slow." My cow-orkers actually put that on a t-shirt for me when I left that lab. :)

But, I guess I'll probably have to give NI support another try.

QUOTE (neBulus @ Mar 23 2009, 02:25 PM)

over on the Dark-Side.

Great story! Now I understand the bump on my head! :P

Link to comment

Cat,

History and jokes aside, I was re-reading all the earlier posts (actually related to your issue..) and it seems to me there are a lot of good ideas there. Forget about the toolkit and the $ your boss spent to get it for you. Use those tried and true methods and I'm sure you can isolate where the memory leak is occuring.

Track the states, log them to file (like I pointed out), use WireShark to track TCP transactions to isolate if the communication is the cause (like someone else pointed out).

Build a "test harness" set of VI's that don't have parts of the code and run that (for example no TCP just reading from file).

Put case structures in suspect VI's run with certain parts of the code de-activated and see if it occurs.

Run test cases on multiple PC's to speed debugging up.

Good old-fashioned debugging is drudgery and fancy tools are all well and good, but basically it is "99% perspiration and 1% inspiration"..

PS I tried using the Real-time trace toolkit a few weeks ago, and I couldn't make head nor tail of the information, running a vision application on a quad-core RT target with multiple timed and regular while loops doing processing and TCP-IP communication via Gig-E. I gave that up and ran a series of tests with and without the timed loops to evaluate.

PS2 Post back with the results of your debugging for more help from the community.

Neville.

Link to comment

QUOTE (Cat @ Mar 24 2009, 06:09 AM)

...But, I guess I'll probably have to give NI support another try.

Great story! Now I understand the bump on my head! :P

1) If you don't make any good progress (if you do not have premier support that could be an issue) use the phrase "I understand that it should be possible to ESCALATE this call, could you do that please?" Mnay AE's (particularly the rookies) will fight that thinking that they have failed if they do so.

2) I believe it is Elijah Kerry form NI who is pushing those new tools. Since he is in the marketing side you can just ask for him when you call NI. Tell him that "Ben NI" said that "he (Elijah) can make recomendations on who to talk to to get the tool working the way it has to." Have your service request handy when you call for him since he can use that to escalate the call.

3) Talk to your local NI person. They picked-up a little extra in their pay-check when you bought that software, let them help out.

I aggre with Neville that traditional approach may hel pnarrow down the issue to a point where can invoke the toolkit to get that extra info you require.

Still trying to help,

Ben

Link to comment

QUOTE (Neville D @ Mar 24 2009, 12:18 PM)

History and jokes aside, I was re-reading all the earlier posts (actually related to your issue..) and it seems to me there are a lot of good ideas there. Forget about the toolkit and the $ your boss spent to get it for you. Use those tried and true methods and I'm sure you can isolate where the memory leak is occuring.

Track the states, log them to file (like I pointed out), use WireShark to track TCP transactions to isolate if the communication is the cause (like someone else pointed out).

Build a "test harness" set of VI's that don't have parts of the code and run that (for example no TCP just reading from file).

Put case structures in suspect VI's run with certain parts of the code de-activated and see if it occurs.

Run test cases on multiple PC's to speed debugging up.

Good old-fashioned debugging is drudgery and fancy tools are all well and good, but basically it is "99% perspiration and 1% inspiration"..

PS I tried using the Real-time trace toolkit a few weeks ago, and I couldn't make head nor tail of the information, running a vision application on a quad-core RT target with multiple timed and regular while loops doing processing and TCP-IP communication via Gig-E. I gave that up and ran a series of tests with and without the timed loops to evaluate.

PS2 Post back with the results of your debugging for more help from the community.

Yes, sir! :)

Seriously now folks... I understand completely that debugging is drudgery. I've already started trying to divide and conquer. Unfortunately, there are at least 148 vis running when this problem has occurred. Plus the fact that if it's going to happen at all, it takes hours before the memory grab starts happening. And I have to closely monitor it, because if it starts happening, it only takes a minute or two before it chews thru all the memory on the computer. yadda yadda blah blah blah. I.e. it's a pain, but I'm going to have to do it.

I've got a test state where the data just comes from a file. I'll run that overnight tonight and see if it crashes. I'm going to need to automate some user actions to simulate actual use... anyway, I've already started the perspiration part of it. I was just hoping the Execution Trace Toolkit would do what I thought it said it would do and narrow things down a little for me. Oh well.

And if I can ever narrow it down myself to something less than 100 or so vis, I'll post here for more help.

Thanks,

Cat

QUOTE (neBulus @ Mar 24 2009, 12:34 PM)

1) If you don't make any good progress (if you do not have premier support that could be an issue) use the phrase "I understand that it should be possible to ESCALATE this call, could you do that please?" Mnay AE's (particularly the rookies) will fight that thinking that they have failed if they do so.

2) I believe it is Elijah Kerry form NI who is pushing those new tools. Since he is in the marketing side you can just ask for him when you call NI. Tell him that "Ben NI" said that "he (Elijah) can make recomendations on who to talk to to get the tool working the way it has to." Have your service request handy when you call for him since he can use that to escalate the call.

3) Talk to your local NI person. They picked-up a little extra in their pay-check when you bought that software, let them help out.

I aggre with Neville that traditional approach may hel pnarrow down the issue to a point where can invoke the toolkit to get that extra info you require.

Still trying to help,

Ben

1) I do have the SSP for LV, but that probably doesn't count. I'll try escalating, if necessary.

2) Thanks for the name. "It's not what you know, it's who you know." ;)

3) Hmm, yes I could try that. I don't have a sense of his technical expertise, but he might have another name to contact.

Yes, maybe if I don't push the toolkit too hard, it can help at some point.

Frankly, right now it's more a matter of not having time to beta test NI's new software. If I had lots of time to play with it, it would be a different matter. But I've been tasked with 3 projects I estimated to take about 3 weeks each, and oh, BTW, I only have 1 month to get them all done. I was hoping the toolkit would help, not add to the load.

Thanks for your help, Ben (and everyone else)! Even if all these responses don't absolutely answer the problem, they get me thinking about different ways to attack it that maybe I haven't tried already.

Link to comment
  • 1 month later...

After many many days of testing with multiple different setups, the problem is solved. Or rather, I don't know *exactly* what is causing the problem, but I do know the circustances under which it will occur.

Turns out the problem wasn't with all the TCP traffic, and it wasn't with continually moving large amounts of data around in memory (tho I will have to admit I was using a global to do that -- now it's all properly Q'd up :) ).

I am using old functions from lvsound.dll to output the time series data to sound. For some (as yet) unknown reason, on certain computers, after approx 4.25 hours of continually writing 2 channels of I16 data to the sound card, something in the system starts grabbing memory at a huge rate.

These "certain computers" are from different manufacturers and have all been purchased in the past couple years, so I'm assuming there is something in newer computers that doesn't play well with the old LabVIEW sound functions. This is even true within a single brand. My old IBM/Lenovo T-43 works fine, but my newer T-61 doesn't. Unfortunately, because I need to use the old machines in order to develop code on a least common denominator, I've been using the T-43 and of course could never replicate this problem there. That made the whole debugging process even more interesting.

I've done a quick fix to the problem by reinitializing the sound output every half hour. This action clears out whatever buffer is building up and the code will run fine for at least 3 days (my longest test). The longer fix is to incorporate the newer LV sound functions (lvsound2.dll), which from my preliminary tests work fine in both the old and the new systems. I'm just not sure if everyone is going to have DirectX8.0 on their computers. Yes, it's been around forever, but so have some of our laptops.

I'm not really sure whether or not to report this as a bug, since you have to look really hard to even find lvsound.dll anymore -- all the LV sound functions point to the newer version. Plus, I haven't identified exactly what's causing the problem; whether it's a sound card hardware or software issue. Any thoughts on this as a reportable bug?? Or maybe someone has already reported it and I just wasted 2 weeks. Luckily I have enough other laptops I didn't come to a complete halt on my other projects.

Thanks again to everyone who weighed in on this with suggestions and encouragment!

Cat

Link to comment

QUOTE (Cat @ Apr 28 2009, 08:04 AM)

....

Thanks again to everyone who weighed in on this with suggestions and encouragment!

Cat

What was the method you used to narrow it down to the lvsound.dll?

Not that I question that as the reason but I am curious what signs other than the memory jumping that got you looking at that dll?

Thanks for the update!

Ben

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.