Jump to content

Memory deallocation bug


beefly

Recommended Posts

Hello!
It was found that the function Request Deallocation does not free memory after using an array of strings (or any other multi-dimensional array).
Description of the experiment:
There is a VI from which subVI is called by command. In the subVI only one-dimensional array (numbers or strings) generated. If the array element type is number (I32), then allocated memory is released (using the function Request Deallocation). If the array element type is string (or any other multi-dimensional array), the memory will not be released. Anyone faced this problem? If yes, what should we do?
Excuse for bad english.

main.vi

memoryusage.vi

Link to comment

Why are you using that function?  The reason I ask is because no program that I have ever written has needed it.  Every time I see it used it didn't need to be.  I'm not saying it doesn't serve a purpose I am just trying to say you don't need to use it. 

 

That function as stated will release memory not being used.  How do you know which memory is not being used?  The compiler will decide what memory it needs and doesn't and will best optimize itself.  So you running that function may do nothing and that would be expected.  Or it may do something but what the something is, will likely be unpredictable.

 

My advice is to just trust the compiler and not use that function to release memory, unless someone at NI has said to for some fix to an issue.

 

EDIT: Okay after re-reading the function description I think it could be used and have some benefit in some programs that perform some work seldomly, and you want to discard the data before exiting the application.  One reason your code may not work is there is a race condition, where you are requesting a deallocation in parallel with creating your array.  Perform one then the other using some data flow or sequence structure. 

Link to comment

I do recall a colleague having quite a bit of memory problems with large 2D string arrays. The memory would never de-allocate even after the VI had finished running. This problem was unique to strings, it did not occur when the data type was changed to something else.

 

This was LV2012, so perhaps it has been fixed.

Link to comment

I also have described similar situations (I think on the NI forums).  The main takeaway was that the function is in fact named Request Deallocation, so I guess "Request denied!" is a valid response.  Actually I looked at it in some detail at the time and I seem to recall it worked like this:  string arrays are stored as an array of handles to the individual strings.  RD dutifully returns the memory used by the handle array immediately but the memory used by the individual strings is not freed.

 

I could troll AQ and discuss the LV garbage collector, but there is not one so that memory is not going to be freed in the normal sense.  I seem to recall that the memory is reusable by LV so it is not completely lost, but LV can be the RAM roach motel, bytes check in but they don't check out.

 

The quick and dirty solution I have used the few times this has been an issue is to put the offending code into an executable which I can then run dynamically in its own context.   Either as a one-shot process or as a daemon.  Reclaiming memory was as simple as shutting down the process and restarting.

  • Like 2
Link to comment
...but LV can be the RAM roach motel, bytes check in but they don't check out.

:lol:

 

Request deallocation can also be a huge time hit.  I've tried it several times on 2D arrays of strings in different situations, and it's... well, not ideal.  I ended up writing my string array to a file, and storing an array of mapped integer file offsets.

 

The deallocation problem first showed up with some changes in LV6.1... apparently, it's either not soluble or not a priority.

  • Like 1
Link to comment
Hi everyone, thanks for the replies.

 

It is worth to note that the problem is not incorrectly using the Request Deallocation, and generally is not in this function. Even the subVI is run through the reference (see attachment), and then stoped and the reference is being closed in the main program, the memory is not released for any, any multidimensional array (it may be a 1d array of strings or 2d array of numbers)! It should be said that LabVIEW reuses this memory, but the  resources are not returned to the system. By the way in LabVIEW 2013 this problem remained. If this "feature" are known from the earliest versions LabVIEW, then why it is not in the official list of bugs? I understand, to leave this problem out, it is necessary either to avoide using any complex arrays (above 1st order) or work with multidimensional arrays in a separate executable.

main_reference.vi

memoryusage_reference.vi

Edited by beefly
Link to comment

I don't want to insult you, but you are generally a text based programmer aren't you?  I say that because at least from my perspective I don't care what LabVIEW does with memory, but I've seen other text based developers get hung up on trying to understand the low level of what is going on.  The compiler is better equipped to do as it pleases, releasing and reallocating memory as it needs.  I trust the compiler until I see a reason not to (crashes, or run away memory).  I suggest you do the same, trust the compiler.  

 

You stated that memory is released properly when you close LabVIEW, and the memory is reused within the application when it needs to be so what is the problem?  Where is the bug?  If it doesn't release the memory then it must need it for a reason that I could not predict.  I do not see is as a bug, and I deal with very large amounts of data in the form of larger sized arrays of clusters of arrays all the time.

Link to comment
I don't want to insult you, but you are generally a text based programmer aren't you?  I say that because at least from my perspective I don't care what LabVIEW does with memory, but I've seen other text based developers get hung up on trying to understand the low level of what is going on.  The compiler is better equipped to do as it pleases, releasing and reallocating memory as it needs.  I trust the compiler until I see a reason not to (crashes, or run away memory).  I suggest you do the same, trust the compiler.  

 

You stated that memory is released properly when you close LabVIEW, and the memory is reused within the application when it needs to be so what is the problem?  Where is the bug?  If it doesn't release the memory then it must need it for a reason that I could not predict.  I do not see is as a bug, and I deal with very large amounts of data in the form of larger sized arrays of clusters of arrays all the time.

 

I cannot agree: The VI posted here (memoryusage_reference.VI) is a reentrant VI that you can easily use to fill the memory in use by LabVIEW - even placing a Request Deallocation in its BD does not help if this VI produces an array of strings.  With a 32-bit LabVIEW, you quite quickly get the famous "Not enough memory to complete this operation" message. Sole reason:  Request Deallocation rejected to properly clean up.  Although the documentation promises to do so.

To my opinon, this is a bug in LabVIEW and as I take from this thread, it is still living in LabVIEW 2013.

I feel encouraged to move code like this to a C/C++ DLL ...

Link to comment

Coming from someone who has had to tackle memory barriers a few times in LabVIEW before, I agree the memory manager can be a little aggravating at times, but I don't see a bug here.

 

As beefly presented the VIs when executing the main I can do a single string create operation and watch the memory allocation soar. When I follow up with a release operation memory decreases by a size that seems to correlate to the number of elements (approximately N*4 bytes), but leaves a larger portion of memory allocated to LabVIEW which appears to be the actual string data, consistent with Darin's observations. This is a good example, kudos.

 

However as Hooovahh pointed out, I don't believe the memory is leaked. It is not lost track of. I can indefinitely do create/release pairs with a stable peak memory allocation. The peak will only increase if I hang onto multiple references at a time which is required to support the multiple independent data spaces.

 

This is definitely not ideal. It would be nice if LabVIEW released the memory to the host operating system so, for example a LabVIEW process which has a tendency to balloon from time to time can restore itself to a small memory footprint when idle. While I don't like the behavior, I think it's still managed in a defined way so don't see a bug here.

Link to comment

Glad that smb (guentermueller) agreed with me.

But there is a note: it was verified that reentrant VIs use the same memory. Can run SubVI five times, then close the references of subVIs, then run another 5 times and you will see that more memory will not allocate...
The problem is uncontrol of the process to free memory. In case of multi-tasking operating system the deallocation of unused memory is very important in any case. It's not just about how to live with it and how to get around this, but why there is such a difference in behavior in the case of one-dimensional and multi-dimensional arrays. I remind that memory will be never released. I understand that the lack of memory is easier to solve with it increases. But deallocation occurs in different ways, which should not be, I think. Deallocation should follow the same scenario.
Edited by beefly
Link to comment

Is it really a 1D vs 2D problem? I didn't try with a 2D numeric array. I was under the impression it was the indirection in the array of strings that was causing issue- that is how the array is really just an array of handles to other data.

Link to comment
While I understand and appreciate hoovah's sentiment that you should just trust the compiler, I think that blindly trusting it is not always the best idea.  

 

No blind trust.  I trust it until I find an issue which to be honest isn't all that often.

 

I trust the compiler until I see a reason not to (crashes, or run away memory).
Link to comment
But there is a note: it was verified that reentrant VIs use the same memory. Can run SubVI five times, then close the references of subVIs, then run another 5 times and you will see that more memory will not allocate...

 

Running the VI 5 times (I am always referring to the VI in the mode when it produces an array of strings) just uses a good amount of memory that you will not get back - but just can reuse by further calls to these reentrant instances.  So you cannot run another 5 instances of this VI in LabVIEW 32-bit - or do any other very memory consuming task that would have fit into memory before executing these reentrant VIs. Oops!

I am glad to hear that Mr Mike has announced a check of this behaviour and provided a CAR.

Link to comment

It's good news that NI paid attention to this problem. Waiting for the result.

 

Running the VI 5 times (I am always referring to the VI in the mode when it produces an array of strings) just uses a good amount of memory that you will not get back - but just can reuse by further calls to these reentrant instances.  So you cannot run another 5 instances of this VI in LabVIEW 32-bit - or do any other very memory consuming task that would have fit into memory before executing these reentrant VIs. Oops!

 

I'm completely agree with you and it's obvious, that if you open hundred different copies of reentrant VI, the allocated memory will be one hundred times greater than for a single call, and physical memory can easily finish. However, in a previous post I just wanted to say that one copy of reentrant VI also reuses its memory after closing. Sorry for my english...
Link to comment
  • 11 months later...

Hey everyone,

 

the last response here was almost a year ago, so I wonder if there was a solution for the problem? I am sill working with LV2012 und currently running into problems with memory, that I can't release with the "Request Deallocation" function.

 

Thank you very much.

Link to comment

As a work around, what if you open with Option set to x100?  it seems to release memory then.  I don't know if its bad form or opens another can of worms to open with "Prepare to call and collect"  but never collect. The documentation seems to indicate so.  

 

Quote:

If you use this option flag (x100), you must include one Wait On Asynchronous Call node for every call that you begin with a Start Asynchronous Call node to ensure that LabVIEW does not retain any started calls in memory indefinitely.

Link to comment
  • 5 years later...
1 hour ago, EvgenKo423 said:

Didn't check it specifically in 2020, but I guess it's the same.

Yeah! LabVIEW 2020 64-bit - RD doesn't work for strings as it should do. Even when the VI is unloaded, LabVIEW still holds some memory allocated and never releases.

1 hour ago, EvgenKo423 said:

I wonder what happened to this CAR?

No tracks of it on NI forums. Anyone with internal access to the list?

Link to comment
7 hours ago, rob.dye said:

The labels I've placed in the picture attempt to explain it all, but I'll summarize by saying that all the "lost" memory (around 422MB) can be explained by the 10M master pointers managed by LV, plus 10M _freed_ small blocks sitting in pages managed by SmartHeap. 

Wasn't aware that LabVIEW still uses SmartHeap. I knew it did in LabVIEW 2.5 up to around 6.0 or so but thought you guys had dropped that when the memory managers of the underlaying OS got smart enough themselves to not make a complete mess of things (that specifically applied to the Win3.1 memory management and to a lesser extend to the MacOS Classic one).

Edited by Rolf Kalbermatter
Link to comment

Thanks for such a detailed explanation, Rob!

 

I just played with string array a bit more and wanna share my findings with the others.

15 hours ago, rob.dye said:

These _freed_ blocks are not "lost". SmartHeap knows where they are and will let us use them again, although because they are small SH keeps them in special low-overhead pools that are used _only_ for small allocations, and who knows when LV will need 10M small blocks again. 

These _freed_ blocks were formerly the 10M strings with one "space" character in them, each of which actually take up probably 40 or 48 bytes. Each string block has a LV-managed 32 byte header, plus a 4 byte length, plus 1 byte for the "space" char. LV asks Smartheap for this 37 byte block, and they probably give us a 40 (rounded up to multiple of 8?) or 48 (rounded up to multiple of 16?) byte smallblock to contain our request. Small blocks in SH are low overhead because they only require a single bit to represent their inUse state and require no header of their own.

According to SmartHeap specs, small should be considered blocks under 256 bytes, but LabVIEW starts to release memory as expected only for blocks which are 1048356 + 36 = 1048392 (1 MiB - 184) bytes in size and above. Is it still an expected behavior for LabVIEW?
Anyway, even 256 bytes is probably quite big length for a typical string to get properly garbage-collected.

TestStringArray.vi

Edited by EvgenKo423
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.