Responsibility of deallocation with string arrays and DSSetHandleSize and related

[Matt W]

I'm currently adding a scripting support via LuaJIT to a program I'm working on. The script is working with potentially large arrays (1 million+ elements) of doubles and strings. I pass both types via Adapt to Type Handles by value and I am able to read both of them without issue. I can also modify values within the double array.

Now my problem is I want to output arrays of doubles and strings from my script, but their size is unknown until I run the script (I do know a maximum size, but that will be far larger than needed in the majority of cases). I can think of a few approaches but since I can't find a good description of the semantics of the LabVIEW memory manager and especially it's interaction with DS* functions I'm unsure if what I do will produce leaks or deallocate possibly in use data. I'm just going to list some of the approaches I have come up with with string arrays since double's should be a much easier problem.

The safe method:

Run the script have it store the data separately then read the string array one element at a time into the LabVIEW array. While this is certainly safe it's also horrendously slow.

The I think might be safe method:

Have LabVIEW pass a handle to a string array that's been preallocated with empty strings, the length of the array is the maximum amount of strings I could possibly need. Now use DSSetHandleSize on all the string handles I'm writing to to make space and write the string data and string size. Next use DSDisposeHandle on all of the unused strings (I assume I'm responsible for deallocating them since I'm throwing them out them), then DSSetHandleSize on the array handle and set the array size. With this method I'm assuming LabVIEW is still responsible for cleaning up all the resized handles. The problem with this that the array is probably several orders of magnitude too large, so I'm putting extra pressure on the memory manager.

The method I want to use:

Have LabVIEW pass a handle to an empty string array. Use DSSetHandleSize on the array handle to make space for the strings I'll need and set the array size. Then use DSNewHandle to create new strings to fill up and set their sizes. Now typically the thing responsible for allocating is also responsible for deallocating, is this still true when I'm putting the string handles in the array that LabVIEW is already handling (in my previous approach I deallocted handles that LabVIEW made, would LabVIEW do the same when it's done with the ones I allocated). If I need to deallocate the strings myself do I need to be extra careful that LabVIEW isn't currently using that data already (IE use the always copy node on anything that comes out of the string array, before I pass the string array back in to the script to close it).

There's a decent change I might end up using arrays of single element clusters of arrays of strings. If there's some reason the concepts from the above wouldn't apply to that.

Matt W

[asbo]

I would hesitantly guess that because you're using LV calls to manage memory, it will still "know" about your allocations. Having never used these particular functions, I can't speak in specificity, but it might be work writing out some test code that just allocates a crap load of memory and then returns ... then see what LV does when the VI unloads from memory.

[Matt W]

Using DETT and a test VI LabVIEW did deallocate strings that I allocated and put into a LabVIEW allocated array. The deallocation happened before the vi terminated, since all DS* allocated data would have been cleaned up by then anyway. I'm forcing deallocation by reshaping the array.

So it looks like LabVIEW will deallocate handles placed properly into aggregate data types. I'm not sure how uncontained handles are dealt with (my guess is that those would need to be removed by hand), but that isn't relevant for my issue.

I don't suppose anyone knows where this stuff is documented (I assume the people who used to write CIN's would have experience with this). Testing LabVIEW this way would be good for double checking the documentation does what I think it says, but seems error prone to figuring out what's exactly going without documentation.

I'm going to assume my third approach will work, unless someone knows of some issues with it.

[Rolf Kalbermatter]

The rules are simple C programming rules. If you pass an array of handles to C code and resize the array size, you are fully responsible to deallocate any elements that are occupied by the previous array beyond its new size and to allocate any elements thathaven't existed before in the array. Additionally you have to make sure to resize any existing handle to the required size before you modify its contents.

And your last remark about that who allocated it also needs to deallocate it, that is ambiguous at least. In C the caller is usually responsible for both since there is no standrad way of passing ownership of memory between caller and callee, but APIs can decide to change that, by allocating and returning memory, but that has to be specifically documented by the API and such an API better provides a function to allow the caller to deallocate that memory later on, since the malloc/free from the API may not be compatible and not even refer to the same heap than the malloc/free of the caller.

In LabVIEW the situation is different. LabVIEW uses a standardized memory manager throughout, so this limitation does not exist. Whoever holds onto a handle is responsible to manage and eventually release it and all of its contents. For a C function being called by LabVIEW with native handles, this means you may get in a handle, you may modify it and allocate/deallocate any handles therein, provided you also keep the related information such as the array size consistent. If you then pass the array back (which you basically always do if it is in a value parameter, and usually also if it is a reference parameter, you pass on ownership of that handle back to LabVIEW and it will need to manage it from thereon.

An additional tidbit you should know is that LabVIEW uses for performance reasons often a null handle when an empty handle is required, but treats an empty handle also correctly. So your C code needs to be prepared to handle an array of 1000 empty strings to be really an array of 1000 null handles, meaning when you want to write something into these strings you can not just do a DSSetHandleSize() as that will crash on a null handle. Instead use NumericArrayResize() with element type uB, as this function gets the handle passed by reference and will correctly allocate a new handle if it was null.

[Matt W]

Thanks, that helps clarify some things.

I haven't seen any nulls during my basic testing, so I'll add some code to check for that. Is there a way to encourage LabVIEW to use nulls, I would like to double check that the null handling is correct.

Which part of the handle can null? Only the handle, only the pointer or either?

If I pass a Handle by value into a CLN can I assume that the passed in handle will not be null (otherwise I can't resize that handle, and I would have no way to return a new handle).

[Matt W]

I think I found the answer to my second question

http://zone.ni.com/reference/en-XX/help/371361H-01/lvconcepts/how_labview_stores_data_in_memory/

Sounds like with strings it is either the handle or pointer that can be null, but with arrays it is just the handle.

[Matt W]

I found some more info

http://zone.ni.com/reference/en-XX/help/371361H-01/lvexcodeconcepts/array_and_string_options/

In the section

Determining When to Resize Array and String Handles in Shared Libraries

Note Although you must use DSSetHandleSize to resize a string or an array of arbitrary data type, you can use the NumericArrayResize manager routine to resize arrays of numeric data. NumericArrayResize automatically accounts for platform-dependent alignment requirements.

It sounds like you're not supposed to use NumericArrayResize for string data. I'm guessing the problem is alignment since it looks like in LV64 on windows the string size would be padded with NumericArrayResize, when it shouldn't be padded. On LV32 on windows there is no padding so I guess it wouldn't make a difference in that case (or in any case where alignment is less than or equal to 4 bytes).

[Rolf Kalbermatter]

It sounds like you're not supposed to use NumericArrayResize for string data. I'm guessing the problem is alignment since it looks like in LV64 on windows the string size would be padded with NumericArrayResize, when it shouldn't be padded. On LV32 on windows there is no padding so I guess it wouldn't make a difference in that case (or in any case where alignment is less than or equal to 4 bytes).

Yes you are not supposed to do that and I can't guarantee that it works for 64 Bit OSes. I treated it so far simply as a NumericArray of 32 bit integers for 32 bit OS and 64 bit integer for 64 bit OS, but never really tested any code on a 64 bit system so far.

As to handles passed by value: Yes you can assume that they are not null, as you rightly have reasoned that you could not resize them in your code then. It still means that the pointer inside the handle can be null but DSSetHandleSize() should be able to deal with that correctly.