Jump to content
beefly

Memory deallocation bug

Recommended Posts

Hello!
It was found that the function Request Deallocation does not free memory after using an array of strings (or any other multi-dimensional array).
Description of the experiment:
There is a VI from which subVI is called by command. In the subVI only one-dimensional array (numbers or strings) generated. If the array element type is number (I32), then allocated memory is released (using the function Request Deallocation). If the array element type is string (or any other multi-dimensional array), the memory will not be released. Anyone faced this problem? If yes, what should we do?
Excuse for bad english.

main.vi

memoryusage.vi

Share this post


Link to post
Share on other sites

Why are you using that function?  The reason I ask is because no program that I have ever written has needed it.  Every time I see it used it didn't need to be.  I'm not saying it doesn't serve a purpose I am just trying to say you don't need to use it. 

 

That function as stated will release memory not being used.  How do you know which memory is not being used?  The compiler will decide what memory it needs and doesn't and will best optimize itself.  So you running that function may do nothing and that would be expected.  Or it may do something but what the something is, will likely be unpredictable.

 

My advice is to just trust the compiler and not use that function to release memory, unless someone at NI has said to for some fix to an issue.

 

EDIT: Okay after re-reading the function description I think it could be used and have some benefit in some programs that perform some work seldomly, and you want to discard the data before exiting the application.  One reason your code may not work is there is a race condition, where you are requesting a deallocation in parallel with creating your array.  Perform one then the other using some data flow or sequence structure. 

Share this post


Link to post
Share on other sites

I do recall a colleague having quite a bit of memory problems with large 2D string arrays. The memory would never de-allocate even after the VI had finished running. This problem was unique to strings, it did not occur when the data type was changed to something else.

 

This was LV2012, so perhaps it has been fixed.

Share this post


Link to post
Share on other sites

I also have described similar situations (I think on the NI forums).  The main takeaway was that the function is in fact named Request Deallocation, so I guess "Request denied!" is a valid response.  Actually I looked at it in some detail at the time and I seem to recall it worked like this:  string arrays are stored as an array of handles to the individual strings.  RD dutifully returns the memory used by the handle array immediately but the memory used by the individual strings is not freed.

 

I could troll AQ and discuss the LV garbage collector, but there is not one so that memory is not going to be freed in the normal sense.  I seem to recall that the memory is reusable by LV so it is not completely lost, but LV can be the RAM roach motel, bytes check in but they don't check out.

 

The quick and dirty solution I have used the few times this has been an issue is to put the offending code into an executable which I can then run dynamically in its own context.   Either as a one-shot process or as a daemon.  Reclaiming memory was as simple as shutting down the process and restarting.

  • Like 2

Share this post


Link to post
Share on other sites
...but LV can be the RAM roach motel, bytes check in but they don't check out.

:lol:

 

Request deallocation can also be a huge time hit.  I've tried it several times on 2D arrays of strings in different situations, and it's... well, not ideal.  I ended up writing my string array to a file, and storing an array of mapped integer file offsets.

 

The deallocation problem first showed up with some changes in LV6.1... apparently, it's either not soluble or not a priority.

  • Like 1

Share this post


Link to post
Share on other sites
Hi everyone, thanks for the replies.

 

It is worth to note that the problem is not incorrectly using the Request Deallocation, and generally is not in this function. Even the subVI is run through the reference (see attachment), and then stoped and the reference is being closed in the main program, the memory is not released for any, any multidimensional array (it may be a 1d array of strings or 2d array of numbers)! It should be said that LabVIEW reuses this memory, but the  resources are not returned to the system. By the way in LabVIEW 2013 this problem remained. If this "feature" are known from the earliest versions LabVIEW, then why it is not in the official list of bugs? I understand, to leave this problem out, it is necessary either to avoide using any complex arrays (above 1st order) or work with multidimensional arrays in a separate executable.

main_reference.vi

memoryusage_reference.vi

Edited by beefly

Share this post


Link to post
Share on other sites

I don't want to insult you, but you are generally a text based programmer aren't you?  I say that because at least from my perspective I don't care what LabVIEW does with memory, but I've seen other text based developers get hung up on trying to understand the low level of what is going on.  The compiler is better equipped to do as it pleases, releasing and reallocating memory as it needs.  I trust the compiler until I see a reason not to (crashes, or run away memory).  I suggest you do the same, trust the compiler.  

 

You stated that memory is released properly when you close LabVIEW, and the memory is reused within the application when it needs to be so what is the problem?  Where is the bug?  If it doesn't release the memory then it must need it for a reason that I could not predict.  I do not see is as a bug, and I deal with very large amounts of data in the form of larger sized arrays of clusters of arrays all the time.

Share this post


Link to post
Share on other sites
I don't want to insult you, but you are generally a text based programmer aren't you?  I say that because at least from my perspective I don't care what LabVIEW does with memory, but I've seen other text based developers get hung up on trying to understand the low level of what is going on.  The compiler is better equipped to do as it pleases, releasing and reallocating memory as it needs.  I trust the compiler until I see a reason not to (crashes, or run away memory).  I suggest you do the same, trust the compiler.  

 

You stated that memory is released properly when you close LabVIEW, and the memory is reused within the application when it needs to be so what is the problem?  Where is the bug?  If it doesn't release the memory then it must need it for a reason that I could not predict.  I do not see is as a bug, and I deal with very large amounts of data in the form of larger sized arrays of clusters of arrays all the time.

 

I cannot agree: The VI posted here (memoryusage_reference.VI) is a reentrant VI that you can easily use to fill the memory in use by LabVIEW - even placing a Request Deallocation in its BD does not help if this VI produces an array of strings.  With a 32-bit LabVIEW, you quite quickly get the famous "Not enough memory to complete this operation" message. Sole reason:  Request Deallocation rejected to properly clean up.  Although the documentation promises to do so.

To my opinon, this is a bug in LabVIEW and as I take from this thread, it is still living in LabVIEW 2013.

I feel encouraged to move code like this to a C/C++ DLL ...

Share this post


Link to post
Share on other sites

Coming from someone who has had to tackle memory barriers a few times in LabVIEW before, I agree the memory manager can be a little aggravating at times, but I don't see a bug here.

 

As beefly presented the VIs when executing the main I can do a single string create operation and watch the memory allocation soar. When I follow up with a release operation memory decreases by a size that seems to correlate to the number of elements (approximately N*4 bytes), but leaves a larger portion of memory allocated to LabVIEW which appears to be the actual string data, consistent with Darin's observations. This is a good example, kudos.

 

However as Hooovahh pointed out, I don't believe the memory is leaked. It is not lost track of. I can indefinitely do create/release pairs with a stable peak memory allocation. The peak will only increase if I hang onto multiple references at a time which is required to support the multiple independent data spaces.

 

This is definitely not ideal. It would be nice if LabVIEW released the memory to the host operating system so, for example a LabVIEW process which has a tendency to balloon from time to time can restore itself to a small memory footprint when idle. While I don't like the behavior, I think it's still managed in a defined way so don't see a bug here.

Share this post


Link to post
Share on other sites

Glad that smb (guentermueller) agreed with me.

But there is a note: it was verified that reentrant VIs use the same memory. Can run SubVI five times, then close the references of subVIs, then run another 5 times and you will see that more memory will not allocate...
The problem is uncontrol of the process to free memory. In case of multi-tasking operating system the deallocation of unused memory is very important in any case. It's not just about how to live with it and how to get around this, but why there is such a difference in behavior in the case of one-dimensional and multi-dimensional arrays. I remind that memory will be never released. I understand that the lack of memory is easier to solve with it increases. But deallocation occurs in different ways, which should not be, I think. Deallocation should follow the same scenario.
Edited by beefly

Share this post


Link to post
Share on other sites

Is it really a 1D vs 2D problem? I didn't try with a 2D numeric array. I was under the impression it was the indirection in the array of strings that was causing issue- that is how the array is really just an array of handles to other data.

Share this post


Link to post
Share on other sites
While I understand and appreciate hoovah's sentiment that you should just trust the compiler, I think that blindly trusting it is not always the best idea.  

 

No blind trust.  I trust it until I find an issue which to be honest isn't all that often.

 

I trust the compiler until I see a reason not to (crashes, or run away memory).

Share this post


Link to post
Share on other sites
But there is a note: it was verified that reentrant VIs use the same memory. Can run SubVI five times, then close the references of subVIs, then run another 5 times and you will see that more memory will not allocate...

 

Running the VI 5 times (I am always referring to the VI in the mode when it produces an array of strings) just uses a good amount of memory that you will not get back - but just can reuse by further calls to these reentrant instances.  So you cannot run another 5 instances of this VI in LabVIEW 32-bit - or do any other very memory consuming task that would have fit into memory before executing these reentrant VIs. Oops!

I am glad to hear that Mr Mike has announced a check of this behaviour and provided a CAR.

Share this post


Link to post
Share on other sites

It's good news that NI paid attention to this problem. Waiting for the result.

 

Running the VI 5 times (I am always referring to the VI in the mode when it produces an array of strings) just uses a good amount of memory that you will not get back - but just can reuse by further calls to these reentrant instances.  So you cannot run another 5 instances of this VI in LabVIEW 32-bit - or do any other very memory consuming task that would have fit into memory before executing these reentrant VIs. Oops!

 

I'm completely agree with you and it's obvious, that if you open hundred different copies of reentrant VI, the allocated memory will be one hundred times greater than for a single call, and physical memory can easily finish. However, in a previous post I just wanted to say that one copy of reentrant VI also reuses its memory after closing. Sorry for my english...

Share this post


Link to post
Share on other sites

Hey everyone,

 

the last response here was almost a year ago, so I wonder if there was a solution for the problem? I am sill working with LV2012 und currently running into problems with memory, that I can't release with the "Request Deallocation" function.

 

Thank you very much.

Share this post


Link to post
Share on other sites

As a work around, what if you open with Option set to x100?  it seems to release memory then.  I don't know if its bad form or opens another can of worms to open with "Prepare to call and collect"  but never collect. The documentation seems to indicate so.  

 

Quote:

If you use this option flag (x100), you must include one Wait On Asynchronous Call node for every call that you begin with a Start Asynchronous Call node to ensure that LabVIEW does not retain any started calls in memory indefinitely.

Share this post


Link to post
Share on other sites

Unfortunately, it's still not fixed in LV 2018 SP1. Didn't check it specifically in 2020, but I guess it's the same.
I wonder what happened to this CAR?

Share this post


Link to post
Share on other sites
1 hour ago, EvgenKo423 said:

Didn't check it specifically in 2020, but I guess it's the same.

Yeah! LabVIEW 2020 64-bit - RD doesn't work for strings as it should do. Even when the VI is unloaded, LabVIEW still holds some memory allocated and never releases.

1 hour ago, EvgenKo423 said:

I wonder what happened to this CAR?

No tracks of it on NI forums. Anyone with internal access to the list?

Share this post


Link to post
Share on other sites

I'm no longer with NI (as of 2014!), but I just got notification on this thread.  I'll forward it to someone who's still there and ask them to respond.

  • Like 2

Share this post


Link to post
Share on other sites
7 hours ago, rob.dye said:

The labels I've placed in the picture attempt to explain it all, but I'll summarize by saying that all the "lost" memory (around 422MB) can be explained by the 10M master pointers managed by LV, plus 10M _freed_ small blocks sitting in pages managed by SmartHeap. 

Wasn't aware that LabVIEW still uses SmartHeap. I knew it did in LabVIEW 2.5 up to around 6.0 or so but thought you guys had dropped that when the memory managers of the underlaying OS got smart enough themselves to not make a complete mess of things (that specifically applied to the Win3.1 memory management and to a lesser extend to the MacOS Classic one).

Edited by Rolf Kalbermatter

Share this post


Link to post
Share on other sites

Thanks for such a detailed explanation, Rob!

 

I just played with string array a bit more and wanna share my findings with the others.

15 hours ago, rob.dye said:

These _freed_ blocks are not "lost". SmartHeap knows where they are and will let us use them again, although because they are small SH keeps them in special low-overhead pools that are used _only_ for small allocations, and who knows when LV will need 10M small blocks again. 

These _freed_ blocks were formerly the 10M strings with one "space" character in them, each of which actually take up probably 40 or 48 bytes. Each string block has a LV-managed 32 byte header, plus a 4 byte length, plus 1 byte for the "space" char. LV asks Smartheap for this 37 byte block, and they probably give us a 40 (rounded up to multiple of 8?) or 48 (rounded up to multiple of 16?) byte smallblock to contain our request. Small blocks in SH are low overhead because they only require a single bit to represent their inUse state and require no header of their own.

According to SmartHeap specs, small should be considered blocks under 256 bytes, but LabVIEW starts to release memory as expected only for blocks which are 1048356 + 36 = 1048392 (1 MiB - 184) bytes in size and above. Is it still an expected behavior for LabVIEW?
Anyway, even 256 bytes is probably quite big length for a typical string to get properly garbage-collected.

TestStringArray.vi

Edited by EvgenKo423

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Similar Content

    • By WILSTi
      Hello,

      I'm developing an application to play "Battleship" with another opponent through serial communication. I currently have the following difficulty:
       I have not been able to fill the Array with my opponent's ships, I can modify a position, but by changing the position, the result of the previous position returns to its original value.
      VI operation:

      For my turn, using the radio buttons, I select the coordinate and press the SEND button. My opponent answers me with a 1 if I hit or with a 0 if I failed, this data is what I couldn't keep in the array.
      Best regards,
      3Q.vi
    • By Ram Prakash
      Can anyone please tell what a DVR [ Data value reference ] is ? I want to know at what situation it will be used and what are the advantages we get by using DVR. I am really confused in this topic . If someone has any code in which they have worked with DVRs. kindly share it to me.
       
      Thank you.
    • By Chen Li
      The program obtains the number of messages in the hardware buffer through ZCAN_GetReceiveNum function, and then calls ZCAN Receive to complete the received data. When the amount of data reaches 1000 frames / s, the program will crash.
      UINT ZCAN_Receive(CHANNEL_HANDLE channel_handle,  ZCAN_Receive_Data* pReceive,  UINT len,  INT wait_time = -1);

      I think there is something wrong with creating the parameter array ZCAN Receive Data* pReceive. Has anyone ever encountered a similar problem?
       
      Receive.zip
    • By hooovahh
      View File Hooovahh Array VIMs
      Here is the Hooovahh Array VIMs.  This initial release contains 14 VIMs for manipulating array data, which are intended to replace OpenG functionality, but with the added benefit of data type propagation, and increased performance using newer array manipulation techniques.  In later versions other Array manipulation functions were added moving all the OpenG stuff to their own palette.
      Most of the OpenG functions are unchanged, but a few use the newer conditional and concatenating tunnels.  And a few functions have added performance based on other inputs.  For instance the Delete Array Elements can operate in a more efficient way if the input indexes are already sorted.  The Filter 1D array can also be more efficient if the input is known to not contain any duplicates.
      Because these packages contain VIMs, they require LabVIEW 2017 or newer.  Having these functions be VIMs mean all functions work with various array data types.  Included functions are:
      Conditional Auto-Indexing Tunnel Delete Elements from (1D or 2D) Array Filter 1D Array Index (1D or 2D) Array Remove Duplicates from 1D Array Reorder (1D or 2D) Array Reverse 1D Array Slice 1D Array Sort (1D or 2D) Array Convert 1D to 2D Convert 2D to 1D Find Subarray Force Array Min/Max Size Foreign Key Sort Submitter hooovahh Submitted 10/11/2017 Category *Uncertified* LabVIEW Version  
    • By Tim_S
      Thought I'd pass this along and see if anyone can reproduce with different versions of LabVIEW. Appreciate it if anyone has seen this and has a fix.
      I'm using shared variables to communicate between applications (1:N). I'd been seeing some memory creep that was inconsistent and somewhat bizarre. Eventually managed to track it down to that I'm programmatically opening a connection to a shared variable in one loop, then reading the value in a different loop (the different loops have to do with reconnecting on connection loss and startup). There is a functional global used to pass the variable to the second loop. The Read Variable primitive deallocates all but 4 bytes of memory for the previous loop handle and then allocates memory for a new handle on each iteration of the while loop, hence creating a leak. This behavior does not occur if there is only one loop where there is an open, while loop with a read, and a close.
      Main.vi demonstrates the issue. Main 2.vi is more like the NI example.
      I've got service request #7728859 with NI going, but I think I got the guy's first day.
      LabVIEW 2015 SP1 32-bit on Win7 64-bit. Shared Variables memory leak.zip
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.