Pointers to Variants

[Aristos Queue]

+1. Plenty for me to play with here. Ta very muchly (did you mean 4 or 8 bytes for the pointer size rather than 8 or 16?)

Yes. Doh!

 

 

If the variant claims right of destruction, what happens to the copies in the buffer? Is this the reason why it crashes on deallocation?

That's why you do a top-swap instead of a move block. There's one allocated inside the variant in the first place and you swap the [pointers/top-level data structure which may our may not contain pointers depending upon the data type under discussion] so that the buffer still contains one.

[ShaunR]

Yes. Doh!

 

 

That's why you do a top-swap instead of a move block. There's one allocated inside the variant in the first place and you swap the [pointers/top-level data structure which may our may not contain pointers depending upon the data type under discussion] so that the buffer still contains one.

Hmmmm. Not sure what this "top swap" actually is. Is that just swapping handles/pointers so basically the pointer in the buffer then points to the empty variant for a read? That would be fine for a write, but for a read the variant needs to stay in the buffer.

Can you demonstrate with my original example?

Edited July 10, 2013 by ShaunR

[Rolf Kalbermatter]

LvVariant is a C++ class. There is a reference count in the variant object, but I wouldn't expect it to come into play here... can't say for certain. When a To Variant primitive executes, LV does "newVariant = new Variant()". Then newVariant.mTD is assigned the data type object -- this is an object with its own fairly complex structure itself. Then the data on the wire is top-swapped into the variant's "newVariant.mData" field (To Variant's input is a stomper so LV has already guaranteed that the data is independent of any other parallel branch and is thus free to be stomped on). There's also an mAttrs field that the attribute table, which is left as null unless/until an attribute is added to the variant.

 

LvVariant does not have any virtual methods so there is no vTable on the C++ object. But the actual data size of the LvVariant class is (1 + sz + sz + 4 + 4 + sz) bytes, where sz is 8 or 16 depending on how many bytes a pointer has on your system. The data is in the first of those pointers. The reference count is the first of the 4 bytes. There's also a transaction count -- the second 4 byte number -- which is unused within Labview.exe and is exported apparently for use with flex data -- you can basically ignore that.

 

Interesting info. The pointer size was already mentioned by Shaun, but I have another question or two. Why the initial 1 byte in the class size? Wouldn't that either cause extra alignment in the structure or otherwise unaligned pointer access, which as far as I know is rather inefficient on non x86 systems?

 

Also would the data and typedescriptor by any change be the ILVDataInterface and ILVDatatypeInterface objects documented in the LabVIEW cintools directory?

 

And does the absence of these header files in preLV2009 mean that the actual handling of data was done differently before, or was that already implemented in earlier LabVIEW versions but just not documented in any way?

Hmmmm. Not sure what this "top swap" actually is. Is that just swapping handles/pointers so basically the pointer in the buffer then points to the empty variant for a read? That would be fine for a write, but for a read the variant needs to stay in the buffer.

Can you demonstrate with my original example?

I suppose he means that the handle have to be swapped between your copy of the data and an empty variant with the same type descriptor. Otherwise LabVIEW still sees the data on the return of the CLN and since that wire is not going anywhere, it concludes that the variant refcount can be decremented resulting in a deallocation of the data if their refcount reaches zero so that your copy of the data pointer is then invalid. I'm still not sure this buys us anything in terms of your MoveBlock() (or SwapBlock()) attempt since the variant obviously needs to be constructed somewhere. I suppose that if you store the variant pointer anywhere you would at least have to increment it's refcount too, and if your store the internal data pointer, you would have to increments its own refcount. That would probably mean calling the AddRef() method on the ILVDataInterface pointer inside the variant, if it is an ILFDataInterface pointer and not just the native data only.

 

And the SwapBlock() has the same parameter types as MoveBlock(). And it is not officially documented. What I know is from accidentally stumbling across some assembly statements during debugging.

[ShaunR]

I'm still not sure this buys us anything in terms of your MoveBlock() (or SwapBlock()) attempt since the variant obviously needs to be constructed somewhere. 

Indeed. However a variant is constructed (constant on the diagram), but from what AQ was saying about descriptors, a simple copy is not adequate since the "void" variant has the wrong ones. I'm now wondering about the LVVariantGetContents and LVVariantSetContents which may be the correct way to modify a variants data and cause the descriptor to be updated appropriately.

[Rolf Kalbermatter]

Indeed. However a variant is constructed (constant on the diagram), but from what AQ was saying about descriptors, a simple copy is not adequate since the "void" variant has the wrong ones. I'm now wondering about the LVVariantGetContents and LVVariantSetContents which may be the correct way to modify a variants data and cause the descriptor to be updated appropriately.

 

It probably is, or some of the other methods that start with LvVariant..

But without some header file that documents at least the parameter list this is a painful process to figure out

[ShaunR]

But without some header file that documents at least the parameter list this is a painful process to figure out

 

I have faith in you ! 

[Rolf Kalbermatter]

I have faith in you ! 

That's very nice, but don't hold your breath for it! I'm currently overloaded with more important things.

[ShaunR]

That's very nice, but don't hold your breath for it! I'm currently overloaded with more important things.

 

[lordexod]

Can be change to C + +, provided that it is supported by the "NI".

LabVIEW CPP Generator.vi

[Rolf Kalbermatter]

Can be change to C + +, provided that it is supported by the "NI".

LabVIEW CPP Generator.vi

 

That may be an interesting approach, but there is no guarantee that the generated C code is binary compatible with what LabVIEW uses itself internally. They are entirely different environments and the CPP generated code is in fact free to use different and simpler datatypes for managing LabVIEW data, than what LabVIEW does use. The CPP generated code only has to provide enough functionality for proper runtime execution while the LabVIEW environment has more requirements for editing operations.

[lordexod]

Compatible with "DFIR" and Labview.

Dynamic libraries already exist in "resource": tdcore_12_0.dll, mgcore_12_0.dll, mgcore_AT_12_0.dll, mgcore_SH_12_0.dll, mgcore_AT_SH_12_0.dll, dfir_12_0.dll .

[Aristos Queue]

Hmmmm. Not sure what this "top swap" actually is. Is that just swapping handles/pointers so basically the pointer in the buffer then points to the empty variant for a read? That would be fine for a write, but for a read the variant needs to stay in the buffer.

My original comment on this topic:

 

You would need to either top swap (swap, not move) the data into the variant or copy the data in, depending upon whether the source data is stompable.

So on the write, you can use the swap because the source is stompable. On the read, you would need to make a copy.

Interesting info. The pointer size was already mentioned by Shaun, but I have another question or two. Why the initial 1 byte in the class size?

It's a bool. It controls something. I don't know what... didn't dig into it. It's private data of a class, and anything that doesn't go through a class' API will just break when someone moves the private data of the class around or changes the data consistency rules for that data. I figured I'd list off the fields I thought you could hack for the moment, but going into all the details of the class is pointless -- the operations that you can manipulate are given in G primitives already.

[Rolf Kalbermatter]

It's a bool. It controls something. I don't know what... didn't dig into it. It's private data of a class, and anything that doesn't go through a class' API will just break when someone moves the private data of the class around or changes the data consistency rules for that data. I figured I'd list off the fields I thought you could hack for the moment, but going into all the details of the class is pointless -- the operations that you can manipulate are given in G primitives already.

 

Well if you say its function is not interesting, I'm more than happy to believe you.

 

But!!! You didn't comment on the fact if the LvVariant is byte packed, in which case access to the pointers in there would incur quite a performance penalty on at least all non-x86 platforms, or if the structure uses natural alignement, in which case your calculation formula about the size would be in fact misleading.

 

Note: Seems the structure uses at least on Windows x86 the standard LabVIEW byte alignment, that is byte packed. All other platforms including Windows x64 likely will have natural/default alignment.

 

But your documentation is definitely not completely correct. The LvVariant looks more like

 

#pragma pack(1)

typedef struct

{

void *something; // maybe there is still a lpvtbl somehow

char flag;   // bool

void *mData;

void *mAttr;

int32 refCount;

int32 transaction;

void *mTD;

} LvVariant;

#pragma pack()

 

This is for Windows x86. For others I assume the pragma pack() would have to be left away.

 

Also I checked in LabVIEW 8.6 and 2011 and they both seem to have the same layout so I think there is some good hope that the layout stays consistent across versions, which still makes this a tricky business to rely upon.

[Aristos Queue]

I don't know much about byte packing. I'll take your word for it. There's no pragma around the code. It is a class, though I don't think that means anything compile was vs struct.

[lordexod]

New better version.

[Rolf Kalbermatter]

New better version.

v2vp.png

 

That is not going to help much. That pointer is invalid at the moment the Call Library Node returns since the variant got deallocated (or at least marked for deallocation at whatever time LabVIEW feels like).

 

And the third parameter is always an int32 but you need to pass 4 for 32 Bit LabVIEW and 8 for 64 byte LabVIEW to it.

[lordexod]

MoveBlock it memmove function: http://www.cplusplus.com/reference/cstring/memmove/

[Rolf Kalbermatter]

MoveBlock it memmove function: http://www.cplusplus.com/reference/cstring/memmove/

 

That is true, but what do you want to say with that?

 

This is in general what MoveBlock() and memove() is about. Nothing magical at all!

void MoveBlock(void *src, void *dst, int32 len)
{
    int32 i;
    if (src & 0x3 || dst & 0x3 || len & 0x3)
    {
        for (i = 0; i < len; i++)
            *(char*)dst++ = *(char*)len++;
    }
    else
    {
        for (i = 0; i < len / 4; i++)
            *(int32*)dst++ = *(int32*)len++;
    }
}

This is not the real MoveBlock() implementation. This implementation would cause "undefined" behaviour if the two memory blocks overlap, while the MoveBlock() documentation states explicitedly that overlapping memory blocks are allowed. Basically the real implementation would have to compare the pointers and either start from the begin or end with copying depending on the comparison result.

 

That does not change anything about the fact that a pointer is just a meaningless collections of numbers if it does point to something that is not allocated in memory anymore. Once your CLN returns in your previous example, there is absolutely nothing that would prevent LabVIEW from deallocating the variant, except lazy deallocation, because it determines that the same variant can be reused the next time the VI is executed. But there is no guarantee that LabVIEW will do lazy deallocation and in real world scenarios it is very likely that LabVIEW will deallocate that variant sooner or later to reuse the memory for something else.

[ShaunR]

MoveBlock it memmove function: http://www.cplusplus.com/reference/cstring/memmove/

 

I couldn't look at your snipit because Lavag seems to be stripping the meta data again and I can't import it as code. But based on your quoted comment........

 

The varpointers.vi I attached in the first post uses MoveBlock and it works once but dies when deallocating the pointer the second time around because LabVIEW kills the variant at some point or, more probably, I've killed it when I shouldn't have.

Edited July 12, 2013 by ShaunR

[lordexod]

That is true, but what do you want to say with that?

 

This is in general what MoveBlock() and memove() is about. Nothing magical at all!

 

void MoveBlock(void *src, void *dst, int32 len){	  int32 i;	  if (src & 0x3 || dst & 0x3 || len & 0x3)	  {		   for (i = 0; i < len; i++)				 *(char*)dst++ = *(char*)len++;	  }	  else	  {		   for (i = 0; i < len / 4; i++)				 *(int32*)dst++ = *(int32*)len++;	  }}

This is not the real MoveBlock() implementation. This implementation would cause "undefined" behaviour if the two memory blocks overlap, while the MoveBlock() documentation states explicitedly that overlapping memory blocks are allowed. Basically the real implementation would have to compare the pointers and either start from the begin or end with copying depending on the comparison result.

 

That does not change anything about the fact that a pointer is just a meaningless collections of numbers if it does point to something that is not allocated in memory anymore. Once your CLN returns in your previous example, there is absolutely nothing that would prevent LabVIEW from deallocating the variant, except lazy deallocation, because it determines that the same variant can be reused the next time the VI is executed. But there is no guarantee that LabVIEW will do lazy deallocation and in real world scenarios it is very likely that LabVIEW will deallocate that variant sooner or later to reuse the memory for something else.

Error you have in your code.

 

Orginalny kod on memmove function:memmove.txt

[Rolf Kalbermatter]

Error you have in your code.

 

Orginalny kod on memmove function:memmove.txt

 

Sigh! I mentioned it is not correct for overlapping buffers. And the Microsoft C Runtime source code is copyright protected so you should not post it anywhere!

[Rolf Kalbermatter]

I believe that the correct definition is as follows:

void *TDR;

#include "lv_prolog.h"
typedef struct
{
	void *lpVTable;		// object C++ virtual table
	LVBoolean flag;		// bool
	void *mData;		// the actual non-flat LabVIEW data
	void *mAttr;		// Looks like a C++ object
	int32 refCount;
	int32 transaction;
	TDR mTD     		// Looks like a C++ object too
} LvVariant, *LvVariantPtr;
#include "lv_epilog.h

But accessing this is highly tricky, LabVIEW version and platform dependent and should therefore be avoided at all costs even though that layout seems to hold true since at least 7.1 and until at least 2018. Also the lpVTable is very long and not a formalized COM interface (I was for a moment hoping it might be the ILVDataType from the ILVDataInterface.h header file in cintools, but alas). I'm still stumped about the presence of ILVDataInterface.h and ILVTypeInterface.h in the cintools directory as it seems to connect to absolutely nothing else in there so it is a bit hard to see how that would be even remotely useful.

But the referencing of the interesting elements can be done by using these function prototypes (gleamed from an ni_extcode.h file contained in the LabVIEW for CUDA SDK download)

#include "extcode.h"

/* from extcode.h in cintools directory
#ifdef __cplusplus
	class LvVariant;
#else
	typedef struct LVVARIANT LvVariant;
#endif
typedef LvVariant* LvVariantPtr; */

/* from ni_extcode.h from CUDA SDK package */
#ifdef __cplusplus
	class TDR;
#else
	typedef struct _TDR TDR;
#endif // __cplusplus

TH_REENTRANT EXTERNC const TDR * _FUNCC LvVariantGetType(LvVariantPtr lvVar);

TH_REENTRANT EXTERNC const void *_FUNCC LvVariantGetDataPtr(LvVariantPtr lvVar);

You just need to be aware that when you pass a LabVIEW Variant through a Call Library Node with the parameter set to Adapt to Type, that LabVIEW actually passes the variant as:

retval FuncName(...., LvVariantPtr *lvVar, ....);

The nasty thing is that the returned TDR data pointer seems indeed to be another C++ object with no declared interface anywhere, and no it does also not match the ILVTypeInterface in the above mentioned header file.

 

And just after hitting the Save button on this post and having spend many hours researching this, the final heureka moment happened. 💡

Those two header files in cintools are of course related to this selection:

Edited May 24, 2022 by Rolf Kalbermatter

[Aristos Queue]

22 hours ago, Rolf Kalbermatter said:

The nasty thing is that the returned TDR data pointer seems indeed to be another C++ object with no declared interface anywhere, and no it does also not match the ILVTypeInterface in the above mentioned header file.

Correct. There's nothing exported about TDRs. It's an opaque hierarchy of types from outside of LabVIEW's core code. 

[Rolf Kalbermatter]

3 hours ago, Aristos Queue said:

Correct. There's nothing exported about TDRs. It's an opaque hierarchy of types from outside of LabVIEW's core code. 

I know. I traced it to tdcore_xx_x.dll but that is as far as I'm willing to invest into this. Trying to find out the signature of C++ object dispatch tables is simply to tedious especially if they are raw dispatch tables and not based on IUnknown COM architecture.