Filling a cluster with strings and arrays dynamically

[jbone]

Hi,

Here is my new problem:

I am passing a cluster to a DLL function, in this DLL function I want to fill the data of the cluster. So, it works perfectly when I do it with simple elements, but when the cluster has strings or arrays inside, I fill it with the structure explained in How LabVIEW Stores Data in Memory.

_declspec (dllexport) long fillCluster(char* struct_ptr) {	int j;	char* auxS;	char sh[]= "Hello, it works!";	// The cluster stores a handle, that is a pointer to a pointer to the string 	auxS = *(*(char***)struct_ptr);	// Write the size	*auxS = strlen(sh);	auxS += sizeof(long);				for(j=0;j<strlen(sh);j++){		auxS[j]=sh[j];	}	return 0;}

And I can see in my VI that the string gets filled correctly. But then, when I try to save the VI, I get this error:

Fatal Internal Error: "fsane.cpp", line 442

LabVIEW version 10.0f2

It seems than some object is making the sanity check fail... so probably it is because I am not filling the cluster object correctly and it gets corrupted. Do I have to allocate the memory for the cluster? for the string inside the clusert? How?

Can anyone show me how to fill a cluster that is passed to a DLL function as a parameter (pointer to the cluster).

Any sample code or example will be very appreciated.

Thank you very much.

Edited February 15, 2011 by jbone

[jbone]

Ok, If I try to allocate the memory for the string inside the cluster, as shown here:

_declspec (dllexport) long fillCluster(char* struct_ptr) {        int j;        char* auxS;        char sh[]= "Hello, it works!";      	long* size;	// The cluster stores a handle, that is a pointer to a pointer to the string 	*(*(char***)struct_ptr) = (char*) malloc ( sizeof(long) + (sizeof(char)*strlen(sh)) );    	auxS = *(*(char***)ptr);   	size = (long *)auxS;   	        // Write the size	*size = strlen(sh);	size += 1;		auxS = (char*) size;             			        for(j=0;j<strlen(sh);j++){                auxS[j]=sh[j];        }        return 0;}

Then, It runs correctly, I do not get any error while saving or running... but, after closing the VI and the project, when I try to close LabVIEW it gets blocked and will never close (I have to kill it).

Thanks again.

Edited February 15, 2011 by jbone

[ned]

If you're going to allocate memory in your DLL and then hand it back to LabVIEW, you should either use the LabVIEW memory manager functions, or allocate the memory in LabVIEW and then pass pointers to it into your DLL. The LabVIEW memory manager functions are documented in the help, under Code Interface Node functions. Alternately, you can initialize the memory in LabVIEW before calling the DLL by using Initialize Array; for a string, initialize an array of U8 and convert it to a string.

[jbone]

If you're going to allocate memory in your DLL and then hand it back to LabVIEW, you should either use the LabVIEW memory manager functions, or allocate the memory in LabVIEW and then pass pointers to it into your DLL. The LabVIEW memory manager functions are documented in the help, under Code Interface Node functions. Alternately, you can initialize the memory in LabVIEW before calling the DLL by using Initialize Array; for a string, initialize an array of U8 and convert it to a string.

Thanks a lot! It completely works now... What I do not get is why, if it is necessary to use LabVIEW memory manager functions, I found the examples in Calling C/C++ DLLs Containing Simple and Complex Datatypes from LabVIEW using malloc, realloc, etc.

Could anyone please clarify?

[ned]

Thanks a lot! It completely works now... What I do not get is why, if it is necessary to use LabVIEW memory manager functions, I found the examples in Calling C/C++ DLLs Containing Simple and Complex Datatypes from LabVIEW using malloc, realloc, etc.

Could anyone please clarify?

In those examples, the data returned is always a pointer-sized integer, which is then dereferenced by a mysterious XNode call GetValueByPointer. The example provides what I assume is equivalent LabVIEW code for dereferencing a pointer using MoveBlock. Dereferencing the pointer this way copies the data from space allocated by the DLL into space allocated by LabVIEW. I'm guessing that in your case you are trying to return the structure/array directly by reference, allowing the call library node to handle the dereferencing. As a result, you're handing LabVIEW a block of memory that it didn't allocate, so it doesn't know how to manage it nor free it on exit. I'm not sure what happens to the memory allocated by the DLL in the examples; I'd guess this would be a memory leak if called repeatedly, and that the operating system would clean it up on exit, but that's mostly a guess.

[Rolf Kalbermatter]

Thanks a lot! It completely works now... What I do not get is why, if it is necessary to use LabVIEW memory manager functions, I found the examples in Calling C/C++ DLLs Containing Simple and Complex Datatypes from LabVIEW using malloc, realloc, etc.

Could anyone please clarify?

As ned has already explained quite clear you have to use LabVIEW memory manager functions if you want to hand data back to LabVIEW directly. The GetValueByPointer XNode is just a way to deal with native pointers, when you have no way or don't want to modify the DLL itself to deal with LabVIEW handles directly. What it does is copying the data in the pointer into a LabVIEW handle. And if you don't clean up that pointer yourself afterwards it will be leaked. Nothing LabVIEW can do about, since the DLL interface is simply a function interface with no defined conventions about how to allocate and manage memory across that boundery. DLL x could use a standard malloc function for that while DLL Y comes with its own memory manager implementation and LabVIEW has no way of knowing about this at all.

Why LabVIEW uses handles? Because traditionally that was required on older platforms to have efficient variable sized memory structures (strings and arrays) and changing that now is no option at all. Even if it would use pointers the fact that the caller never can know what the callee used to create a particular memory block and vice versa, would still prevent fiddling with pointers created in the other component.

[jbone]

Wooow!!! Very long and complete explanations. Thank you very much Ned and Rolf.

It is very clear now.

[ShaunR]

The GetValueByPointer XNode is just a way to deal with native pointers, when you have no way or don't want to modify the DLL itself to deal with LabVIEW handles directly.

I find it far too slow It's a shame it's an xnode.. It's password protected and I wanted to find out how it determines the length of a string before it dereferences (iterates 1 char at a time and checks for null?) They don't mention how to do that in the move block documentation (dereference a variable length string).

[Rolf Kalbermatter]

I find it far too slow It's a shame it's an xnode.. It's password protected and I wanted to find out how it determines the length of a string before it dereferences (iterates 1 char at a time and checks for null?)

Well if you find a different way to determine the length of a null terminated C string you are definitely a lot smarter than all the C cracks who have worked on the various standard C runtime libraries when implementing strlen().

They don't mention how to do that in the move block documentation (dereference a variable length string).

MoveBlock() operates on a byte array only. There is no way it can determine the length of the source buffer at all. While it could be a typical C string it does not know about that. It's simply a char[] buffer, but char in C does not mean null terminated string at all, it's just the name for a byte sized integer.

If you know that you have a null terminated string buffer you could do something like this instead:

LStrHandle target = NULL;

int len = StrLen(buffer);

MgErr err = NumericArrayResize(uB, 1, (UHandle)&target, len);

MoveBlock(buffer, LStrBuf(*target), len);

LStrLen(*target) = len;

This is as fast as it gets and I'm sure the XNode adds some complications to it, since it works for different parameter types too.

Another LabVIEW manager call that does this more or less in one call would be:

LStrHandle target = DSNewHdlClr(4);

LStrPrintf(target, "%s", buffer);

Since this incures an extra call to the memory manager function DSSetHandleSize() inside LStrPrintf() it is likely slower than the previous solution.

[ShaunR]

Fantastic. Knew it had to be simple.

I think the speed of the xnode stems from (apart from being an xnode) that its polymorphic. I wanted to use i in the SQLIte API but it was far to slow. But now I can add some more features.

Many thanks.

Edited February 17, 2011 by ShaunR

[ned]

I think the speed of the xnode stems from (apart from being an xnode) that its polymorphic. I wanted to use i in the SQLIte API but it was far to slow. But now I can add some more features.

I've never written an xnode, so someone correct me if I'm wrong, but I can't see how it being an xnode would affect execution speed. It's configured to the right type when it is wired. That's an edit-time operation. At run-time, it's no longer polymorphic so there should be no speed penalty.

[Rolf Kalbermatter]

I've never written an xnode, so someone correct me if I'm wrong, but I can't see how it being an xnode would affect execution speed. It's configured to the right type when it is wired. That's an edit-time operation. At run-time, it's no longer polymorphic so there should be no speed penalty.

Not being able to look into the XNode I can't really say much about it. But it most likely doesn't just call MoveBlock() only but probably does some other things too, that are not strictly necessary in all situations. Not knowing the exact circumstances it is called in it may do so just to error on the safe side. It's definitely more complex than just calling two LabVIEW manager functions, and possibly XNodes also are limited in the way they are executed. Wouldn't surprise me if XNodes always executed in the single threaded UI Execution system, and that could be a major performance killer.

[ShaunR]

Not being able to look into the XNode I can't really say much about it. But it most likely doesn't just call MoveBlock() only but probably does some other things too, that are not strictly necessary in all situations. Not knowing the exact circumstances it is called in it may do so just to error on the safe side. It's definitely more complex than just calling two LabVIEW manager functions, and possibly XNodes also are limited in the way they are executed. Wouldn't surprise me if XNodes always executed in the single threaded UI Execution system, and that could be a major performance killer.

Well. I thought I'd get my hands dirty and take the xnode apart. You are right. It doesn't just call moveblock. In fact, it doesn't call it at all!. It calls "GetValueByPointer" in "lvimptsl.dll" (that might call moveblock). But, more importantly (as you surmised), it does this in the UI thread (presumably because the dll isn't thread-safe). So. You're on a roll , That's the reason it's so slow (and totally unusable for 99% of my apps).

But your suggestion works a treat

Edited February 26, 2011 by ShaunR

[Rolf Kalbermatter]

Well. I thought I'd get my hands dirty and take the xnode apart. You are right. It doesn't just call moveblock. In fact, it doesn't call it at all!. It calls "GetValueByPointer" in "lvimptsl.dll" (that might call moveblock). But, more importantly (as you surmised), it does this in the UI thread (presumably because the dll isn't thread-safe). So. You're on a roll , That's the reason it's so slow (and totally unusable for 99% of my apps).

But your suggestion works a treat

Well that the DLL call is done in the UI thread may be just a byproduct. What I was meaning to say is that the execution of the entire XNode may be forced into the UI thread. I have never bothered to look into XNodes and how they work, but it wouldn't surprise me if they are more of a quick and dirty hack added to LabVIEW than a properly designed feature, which might be one of the reasons that it never made it into a public feature. And taking the shortcut of executing XNodes in the UI thread would make creating such a feature much easier. Of course it's not ideal to do it that way but hey I have no idea what the intended use cases were so it may have made perfect sense.

[ShaunR]

Well that the DLL call is done in the UI thread may be just a byproduct. What I was meaning to say is that the execution of the entire XNode may be forced into the UI thread. I have never bothered to look into XNodes and how they work, but it wouldn't surprise me if they are more of a quick and dirty hack added to LabVIEW than a properly designed feature, which might be one of the reasons that it never made it into a public feature. And taking the shortcut of executing XNodes in the UI thread would make creating such a feature much easier. Of course it's not ideal to do it that way but hey I have no idea what the intended use cases were so it may have made perfect sense.

Xnodes don't quite work like that (you would think it was similar to an Xcontrol...But it isn't) They are basically pre-packaged script nodes that are programmed to generate code when executed. So the xnode in question has lots of script to create a CLN and all the function parameters to interface to the DLL. The result is that at some point (not sure exactly when, maybe after compilation, or when you press the run button) the xnode runs, then creates the code and it is this generated code which runs in the place of the xnode - In this case a CLN in the UI thread.

It's a lot of [script] code to generate a relatively small amount of real code that could have been created as you described and wrapped in a polymorphic VI for the adapt-to-type (no need for the intermediary DLL then and it wouldn't have to execute in the UI thread).

[ShaunR]

Well that the DLL call is done in the UI thread may be just a byproduct. What I was meaning to say is that the execution of the entire XNode may be forced into the UI thread. I have never bothered to look into XNodes and how they work, but it wouldn't surprise me if they are more of a quick and dirty hack added to LabVIEW than a properly designed feature, which might be one of the reasons that it never made it into a public feature. And taking the shortcut of executing XNodes in the UI thread would make creating such a feature much easier. Of course it's not ideal to do it that way but hey I have no idea what the intended use cases were so it may have made perfect sense.

Xnodes don't quite work like that (you would think it was similar to an Xcontrol...But it isn't) They are basically pre-packaged script nodes that are programmed to generate code when executed. So the xnode in question has lots of script to create a CLN and all the function parameters to interface to the DLL. The result is that at some point (not sure exactly when, maybe after compilation, or when you press the run button) the xnode runs, then creates the code and it is this generated code which runs in the place of the xnode - In this case a CLN in the UI thread.

It's a lot of [script] code to generate a relatively small amount of real code that could have been created as you described and wrapped in a polymorphic VI for the adapt-to-type (no need for the intermediary DLL then and it wouldn't have to execute in the UI thread).

[Bab]

If you know that you have a null terminated string buffer you could do something like this instead:

LStrHandle target = NULL;

int len = StrLen(buffer);

MgErr err = NumericArrayResize(uB, 1, (UHandle)&target, len);

MoveBlock(buffer, LStrBuf(*target), len);

LStrLen(*target) = len;

This is as fast as it gets and I'm sure the XNode adds some complications to it, since it works for different parameter types too.

Dear Rolf,

thanks for sharing your knowledge!

I am actually using the GetValueByPointer xnode in order to get the string returned by a WinApi call.

Strange enought, the string coming out of the xnode IS Null terminated. That is I am getting the \00 character too. I did expect to get all the characters BEFORE the Null.

I am interesting in using the code you shared as an alternative, but how do I use it? It looks like ShaunR knows too how to use it.

Do I have to create a new C dll? Does it have to do with CIN?

Or maybe you could point me to a topic where is is explained.

I also have trouble understanding how to use the "LabVIEW Manager Functions" described in LV help (http://zone.ni.com/reference/en-XX/help/371361H-01/lvexcodeconcepts/labview_manager_routines/).

What do I need in order to use these? (MS Visual Studio?, NI CVI?) What are you using?

Thanks for any assistance.

BaB.

[Rolf Kalbermatter]

Dear Rolf,

thanks for sharing your knowledge!

I am actually using the GetValueByPointer xnode in order to get the string returned by a WinApi call.

Strange enought, the string coming out of the xnode IS Null terminated. That is I am getting the \00 character too. I did expect to get all the characters BEFORE the Null.

I am interesting in using the code you shared as an alternative, but how do I use it? It looks like ShaunR knows too how to use it.

Do I have to create a new C dll? Does it have to do with CIN?

Or maybe you could point me to a topic where is is explained.

I also have trouble understanding how to use the "LabVIEW Manager Functions" described in LV help (http://zone.ni.com/reference/en-XX/help/371361H-01/lvexcodeconcepts/labview_manager_routines/).

What do I need in order to use these? (MS Visual Studio?, NI CVI?) What are you using?

Thanks for any assistance.

BaB.

The most straight forward way to use them is to write C code, and for that you need a C compiler such as MS Visual Studio or LabWindows CVI. In a time long, long ago, this was used to create so called CINs but they are part of a long gone era, and the way nowadays is to create DLLs (shared libraries on non Windows platforms) and incorporate those shared libraries/DLLs into LabVIEW using the Call Library Node. In order for the C compiler to be able to find the declarations of those functions you have to #include "extcode.h" and possibly other include files from the <LabVIEW>/cintools directory. You also link the resulting code with labviewv.lib from the same directory so that the resulting DLL can be linked and later knows how to link to those manager functions in LabVIEW at runtime.

Now if you need to call only very few of those manager functions, there is a little trick in that you define in the Call Library Node as library Name "LabVIEW" (without quotes and case does matter!!) and then you can configure the Call Library Node to match the definition of the manager function and LabVIEW will call into itself and execute that function. This is however an advanced feature. You do need to know how to configure the Call Library Node correctly, you need to be familiar with how LabVIEW stores its native data in memory, and should also be very savy about pointers and such in general. Also it is not very maintenance friendly since you have to fiddle in the LabVIEW diagram with C intrinsicaties and everytime you need to change something you have to go in there again and try to find out what you did last time. Editing a C file and creating a new DLL/shared library is in the long term so much easier, and once you end up calling more than a few C functions through the Call Library Node you really want to place the complicated C details in a C module and load that into LabVIEW through the Call Library Node, instead of making complicated push ups and more complicated fitness exercises on the LabVIEW diagram.

What your problem is, I'm not sure, but if it is that you do not get the entire string since there seems to be NULL bytes already at the beginning, then you probably got Unicode16 strings. LabVIEW does NOT use Unicode strings at all internally but always uses the MBCS coding, and all LabVIEW manager string functions consequently operate on MBCS strings, so they can't deal with Unicode16 strings at all. In that case you better use Windows API functions that can work with WCHAR strings or alternatively have to convert the Unicode string into MBCS early on using the Windows API function WideCharToMultiByte().

[Bab]

Thanks for the reply.

my problem was that I was getting the proper string out of the WinAPI call, but with a NULL character at the end and I expected that GetValueByPointer would return just the string, without the Null character.

I am not sure if GetValueByPointer always react like this. Is it the expected behavior, or a bug?

As an alternative I now use 2 Call Library Nodes: StrLen to get the string length, and MoveBlock to copy the CString into a Labview wire.

I attached the resulting vi for the function GetUniversalName (discussed in another thread on LAVA...)

It converts a path to a mapped network drive (ex: S:\folder2\file01.ext) into its UNC name ( ex: \\server1\share1\folder1\folder2\file01.ext, if S:\ is mapped to \\server1\share1\folder1).

The MoveBlock function properly truncates the CString an does not return the ending Null character

The GetValueByPointer returns the complete CString, including the ending Null character - which is not what I want here...

I hope someone finds this useful.

PathGetUniversalName_no_Sub.vi