rRIO runs out of RAM and gets stuck

[KoBe]

Hi people,

I'm using:

1x cRIO-9073 with 1x NI-9203 and 1x NI-9205

1x WSN-9791 Gateway with 3x WSN-3202

1x MBus Modbus converter connected to RS232 of the cRIO

Labview 8.6.1 + FPGA + Real Time

NI-RIO-3.4.0 without Scan Engine

My system runs as a datalogger. Ac mains are measured with 10KHz and the mean of 2 seconds of current and voltage are stored. All other devices are sampled with 2 seconds. A further loop stores that data in an array and after 1 minute it makes the mean for each 10 seconds (5 samples) and stores the result in a TDMS file.

The problem is, that the application starts with memory occupation of about 78,6% (View in System Manager) and with each second it increseas slightly by 0,001% or even less. BUT that's enough that after one or two hours the cRIO runs out of memory and gets stuck. The problem is, that this device is in a remote location and should run for 2 years and I can not go their every 2 hours to reset it manually because it does not respond in remote.

In the screenshot you see the main VI of the cRIO. First of all I have some init stuff, a 1 element cluster Queue which is use like a notifier (single writer=>lossy enqueue and multiple readers=>preview) and a 10 element string Queue which drives the overall init and control finite state machine. This state machine, after init toggles between Idle and Remote with a periode of 2 seconds to read the values of functional global variables which I'm using to control the system in remote. (not really needed much, because it's a "stupid" datalogger application which does not need any user interaction appart from freeing disk space by downloading periodically the tdms data).

The second big loop in the upper part includes 2 timed loops and one while loop to make acquisition and data storage.

1) Timed loop with 200ms periode reads FPGA DMA and elaborates data. If a package of data (2048) values are in memory, a 50Hz single tone frequency will be search to measure AC mains frequency, RMS voltage/ current. This elaborated data is passed into a functional global variable (FGV).

2) Timed loop with variable periode (in this case 2 seconds) reads WSN data, AC data from FGV, Modbus, analog inputs of the cRIO and inserts the data array into a RT-FIFO

3) Reads RT-FIFO, waits 500ms to free CPU if no further element is available. After a variable time (60s) all collected data will be processed and is then stored in a TDMS file. The TDMS reference is opened once and so long the system doesn't loose supply it is stored as FGV. Only after restart the reference will be reopened again.

**************

And somewhere in this code the cRIO is eating piece by piece my free memory :-(. Each SubVI contains allready "Deallocate Memory" but it didn't improve the system. I'm really desperate. Can someone see from the screenshot if I'm using a programming structure or method where some issues with memory are known? I have no idea.... It happens also with an other project, where I just read 5 analog inputs with a sampling periode of 2s. There after 20 days the cRIO is stuck again, means memory is also full. Could the problem also come from VxWorks RTOS behind Labview?

**************

Don't know where the problem comes from and tested allready so many things.... the problem is that is takes hours or day to reproduce the error and that really a problem, because I can never be sure that the system is not running out of RAM memory after some weeks?

Thanks for all suggestions you can give me, I'm really looking forward to any single reply, because otherwise my whole pipeline full of projects won't be successful.

Ciao

Kobe

[KoBe]

Hi people,

it's me again.

Could it be possible that "Build Array" leaves some overhead in the RAM behind, even the memory should be deallocated by the RTOS? How can I force deallocation in a VI which I never leave (main VI). Can it help to make SubVI's around each "Build Array" with a "Deallocate Memory" inside?

I know that it is better to preallocate all the memory before and then insert the new values in the existing array. That works fine for fixed size data like the single array I'm using. But what would happen if I a would preallocate an array of 6 error clusters? The cluster can't be of fixed size because of the string? Or it's better to do that also for the error clusters to be sure no new memory is allocated at least in the case that their is no error an all error strings remain "" which mains of constant size most of the time??

.....

.....

.....

I tried to use only preallocated arrays and it seems to work... I have to run again some hours, days, weeks, months of test, then I will let you know :-)

Bye

Kobe

[KoBe]

It's getting weird!!!!

VI in debug mode: used memory remains constant

VI in rtexe mode: used memory constantly increases till crash

I think some NI Application Engineer is needed who knows what could be the difference in memory allocation or usage depending on debug mode or rtexe mode...

[asbo]

It would be much easier if you uploaded the code itself to look at; that's a fairly involved BD ...

Pre-allocated arrays are a good step, but are you familiar with circular buffers? Used properly, they practically guarantee constant memory consumption.

[KoBe]

It would be much easier if you uploaded the code itself to look at; that's a fairly involved BD ...

The whole library behind the project is huge... would prefer to not upload it.

Pre-allocated arrays are a good step, but are you familiar with circular buffers? Used properly, they practically guarantee constant memory consumption.

Now I use pre-allocated memory and I replace elements in the existing array instead of building a new array. That work also fine in debug mode as described before, but it does not work if i build a rtexe

[KoBe]

Maybe I did run also into that known issue: http://zone.ni.com/devzone/cda/tut/p/id/11014#86-reademptyfifo

Reading Empty Target to Host DMA FIFO with Timeout Set to Zero Gradually Starves CPU in built LabVIEW RT executables on cRIO targets

In built LabVIEW RT applications on cRIO, if a Target to Host DMA FIFO read executes with a timeout of zero and the FIFO is empty a processor leak occurs that increases the CPU usage on the controller.

Workaround— Read zero elements to find elements remaining, instead of using a zero timeout

Date Added—01/15/2009

I have to check that now....

[KoBe]

Maybe I did run also into that known issue: http://zone.ni.com/d...6-reademptyfifo

True true, now CPU load is reduce, but memory issue is still not solved...

[Tim Erickson]

You keep getting a reference to Meetbox FIFO but never close it.

[KoBe]

You keep getting a reference to Meetbox FIFO but never close it.

Thanks Tim! You're right, I'll insert Close FIFO.

If the program would work without problems, I would reach that point after 2 years of running the loop with Meetbox FIFO which means ending the data logging period. And in that case it doesn't mather if the program crashes. But for correctness i'll insert everything.

Nevertheless I don't know why my code would run infinitely in debug mode and on the other hand causes RAM overflow running with a build rtexe.....

[GoGators]

Have you tried the Performance and memory profiler. You can profile the memory and determine which VI sucking up the memory.

[KoBe]

Have you tried the Performance and memory profiler. You can profile the memory and determine which VI sucking up the memory.

At the moment I'm "offline" / have no possibility to access the cRIO to try your suggestions. Nevertheless I thank you very much!

In the meantime I found exactly what I was searching for: an official NI statement about memory leakage with Queues. When I'm online again I'll try to fix that leakage like described below and the I'll let you know if it works.

http://zone.ni.com/reference/en-XX/help/371361E-01/glang/create_queue/

Obtain Queue Details

Use named queues to pass data between two sections of a block diagram or between two VIs in the same application instance. If you do not wire name, the function creates a new, unnamed queue reference. If you wire name, the function searches for an existing queue with the same name and returns a new reference to the existing queue. If a queue with the same name does not already exist and create if not found? is TRUE, the function creates a new, named queue reference.

If you use the Obtain Queue function to return a reference to a named queue inside a loop, LabVIEW creates a new reference to the named queue each time the loop iterates. If you use Obtain Queue in a tight loop, LabVIEW slowly increases how much memory it uses because each new reference uses an additional four bytes. These bytes are released automatically when the VI stops running. However, in a long-running application it may appear as if LabVIEW is leaking memory since the memory usage keeps increasing. To prevent this unintended memory allocation, use the Release Queue function in the loop to release the queue reference for each iteration.

This function might return error codes 1, 2, 1094, 1100, 1491, or 1548.

[ShaunR]

At the moment I'm "offline" / have no possibility to access the cRIO to try your suggestions. Nevertheless I thank you very much!

In the meantime I found exactly what I was searching for: an official NI statement about memory leakage with Queues. When I'm online again I'll try to fix that leakage like described below and the I'll let you know if it works.

http://zone.ni.com/r...g/create_queue/

Obtain Queue Details

Use named queues to pass data between two sections of a block diagram or between two VIs in the same application instance. If you do not wire name, the function creates a new, unnamed queue reference. If you wire name, the function searches for an existing queue with the same name and returns a new reference to the existing queue. If a queue with the same name does not already exist and create if not found? is TRUE, the function creates a new, named queue reference.

If you use the Obtain Queue function to return a reference to a named queue inside a loop, LabVIEW creates a new reference to the named queue each time the loop iterates. If you use Obtain Queue in a tight loop, LabVIEW slowly increases how much memory it uses because each new reference uses an additional four bytes. These bytes are released automatically when the VI stops running. However, in a long-running application it may appear as if LabVIEW is leaking memory since the memory usage keeps increasing. To prevent this unintended memory allocation, use the Release Queue function in the loop to release the queue reference for each iteration.

This function might return error codes 1, 2, 1094, 1100, 1491, or 1548.

I've been bitten by this before too.Now I use this vi. It ensures there is only ever 1 reference to a queue, you don't need thousands of wires all over your diagram and you can easily access queues from other VIs.

[KoBe]

I've been bitten by this before too.Now I use this vi. It ensures there is only ever 1 reference to a queue, you don't need thousands of wires all over your diagram and you can easily access queues from other VIs.

Thanks a lot for your VI. I allready programmed something else:

I'm opening the Queue in the loop, preview an element and then close it again with force destroy=false.

Didn't have yet the possibility to test the code. What happens if I close a Queue reference with force destry=false? Doesn't it mean that at least one reference remains open somewhere? A final destruction is done with force destry=true ??

[ShaunR]

Thanks a lot for your VI. I allready programmed something else:

I'm opening the Queue in the loop, preview an element and then close it again with force destroy=false.

Didn't have yet the possibility to test the code. What happens if I close a Queue reference with force destry=false? Doesn't it mean that at least one reference remains open somewhere? A final destruction is done with force destry=true ??

If it is false only the reference you wire to it will be closed. If it is the last reference, the queue is also destroyed regardless if you set it to true or false.What can happen is that you get "intermittent" failures if you don't get the close exactly right. Since one part of your code may close the last ref just before another part tries to dequeue/enqueue etc.. Then a loop that obtains a reference comes around again an re-creates it but you cannot track down where the error is coming from.

Edited October 15, 2010 by ShaunR