Pointers to Variants

[ShaunR]

I'm probably missing something fundamental about how variants are stored in memory  .

 

I want to create an array of pointers to variants. I seem to be able to write quite happily, but when read; it causes a problem.

The following VI demonstrates what I am trying to do. It runs through once quite happily, but on the second execution it fails on the DSDisposePointer (even though the variant has been written and read correctly). If you disable the read, then it doesn't crash so it must be something to do with the way I'm retrieving the data.

 

Any help appreciated.

 

varpointers.vi

 

 

[Rolf Kalbermatter]

I'm probably missing something fundamental about how variants are stored in memory  .

 

I want to create an array of pointers to variants. I seem to be able to write quite happily, but when read; it causes a problem.

The following VI demonstrates what I am trying to do. It runs through once quite happily, but on the second execution it fails on the DSDisposePointer (even though the variant has been written and read correctly). If you disable the read, then it doesn't crash so it must be something to do with the way I'm retrieving the data.

 

Any help appreciated.

 

varpointers.vi

 

And why do you think a variant is 8 bytes long? I don't know how long it is, but it is either a pointer or a more complex structure whose size you cannot easily determine. In the pointer case, which I would tend to believe it is, it would be either 4 bytes or 8 bytes long depending on if you run this on 32 bit or 64 bit. The pointer theory is further enforced by the lonely definition of the LvVariant typedef in the extcode.h file. It is certainly an object so will hold a virtual dispatch table (a pointer, yes) and the actual data itself in whatever format LabVIEW would like. Most likely they chose a similar approach to the Windows VARIANT layout with a variable telling the type of the data and the actual data either as pointer for non-scalar data and directly embedded for scalars.

 

If you run your VI on LabVIEW for 32 bit, the second MoveBlock will overwrite data in memory beyond the Variantpointer and therefore destroy something in memory.

 

Please note also that LabVIEW knows in fact two variants. One is the native LvVariant and the other is the Windows VARIANT. They look the same on the diagram and LabVIEW will happily coerce from one to the other but in memory they are different. And while you can configure a CLN parameter to be a Windows VARIANT, this is only supported on Windows obviously.

 

Still wish they would document the LvVariant API that is exported by LabVIEW.exe.

[ShaunR]

And why do you think a variant is 8 bytes long? I don't know how long it is, but it is either a pointer or a more complex structure whose size you cannot easily determine. In the pointer case, which I would tend to believe it is, it would be either 4 bytes or 8 bytes long depending on if you run this on 32 bit or 64 bit. The pointer theory is further enforced by the lonely definition of the LvVariant typedef in the extcode.h file.

 

If you run your VI on LabVIEW for 32 bit, the second MoveBlock will overwrite data in memory beyond the Variantpointer and therefore destroy something in memory.

 

Please note also that LabVIEW knows in fact two variants. One is the native LvVariant and the other is the Windows VARIANT. They look the same on the diagram and LabVIEW will happily coerce from one to the other but in memory they are different. And while you can configure a CLN parameter to be a Windows VARIANT, this is only supported on Windows obviously.

 

Still wish they would document the LvVariant API that is exported by LabVIEW.exe.

 

I know. I'm running x64 at the moment so that's why it s 8. Maybe I should have put a conditional disable for people on 32 bit, but it was late at night and I had already pulled most of my hair out.

 

One doc I found states:

"LabVIEW stores variants as handles to a LabVIEW internal data structure. Variant data is made up of 4 bytes"

 

And a handle is a **myhandle. I don't really care what the data structure is only that I can point to the handle so it "should" be just pointer copying. Works fine for the write and once for the read, but second time around the deallocation kills LV.

 

So. Suggestions?

[jcarmody]

[...] It runs through once quite happily, but on the second execution it fails on the DSDisposePointer [...]

 

I don't know if it's helpful, but it fails on the third execution for me (32 bit Win XP SP3).

[ShaunR]

I don't know if it's helpful, but it fails on the third execution for me (32 bit Win XP SP3).

Yes. As Rolf pointed out. For x32 you need to change the 8 to a 4.

But it is the third one that causes the problem sooner or later.

[chris754]

Having trouble posting code/pics. To solve get rid of the variant constand and instead wire the variant over from the "To varaiant" output to where the constant was wired to.

[Rolf Kalbermatter]

Having trouble posting code/pics. To solve get rid of the variant constand and instead wire the variant over from the "To varaiant" output to where the constant was wired to.

 

That is most likely not a real fix but just an avoidance of a symptome. It looks like LabVIEW is actually maintining some form of reference count on Variants. Copying the pointer alone is likely not enough to keep the variant alive, but somehow the reference count has to be incremented too (and properly decremented afterwards again, to avoid lingering Variants.

[chris754]

Yes, I agree with you. I posted as a poissble soultion to his particular problem. You can see that my patch will break if you put variant constants on the BD wired to nothing.

[ShaunR]

That is most likely not a real fix but just an avoidance of a symptome. It looks like LabVIEW is actually maintining some form of reference count on Variants. Copying the pointer alone is likely not enough to keep the variant alive, but somehow the reference count has to be incremented too (and properly decremented afterwards again, to avoid lingering Variants.

 

Maybe that's the difference between working with pointers and working with handles? (DSHandToHand etc). Could be that variants are relocatable.

Edited July 5, 2013 by ShaunR

[Rolf Kalbermatter]

Maybe that's the difference between working with pointers and working with handles? (DSHandToHand etc). Could be that variants are relocatable.

 

I don't think so. Handles do not really maintain a ref count in itself. And I doubt a LvVariant is a real handle although it seems to be a pointer to a pointer (or more precisely a pointer to a C++ class).

 

I would suspect the LvVariant to implement some COM like refcounting but would not know how to get the refcount properly increased and decreased.

[ShaunR]

I don't think so. Handles do not really maintain a ref count in itself. And I doubt a LvVariant is a real handle although it seems to be a pointer to a pointer (or more precisely a pointer to a C++ class).

 

I would suspect the LvVariant to implement some COM like refcounting but would not know how to get the refcount properly increased and decreased.

 

I don't really see why variants need a ref-count since you can disassemble a variant as bytes and reconstitute which would mess up that sort of thing.

 

I read (somewhere) that for handles you have to query the "Master" handle table to get hold of a handle to a particular variable so it is a two step process. So I was thinking perhaps the issue is that I'm actually operating on a master table rather than the real handle.  I was kind-a hoping it would just be just a miss-setting in one or more of the CLFNs. Since you haven't proffered a solution or things to try with pointers, I guess it's not trivial and I'll just have to jump down that rabbit hole to see what Alice is up to..

[Rolf Kalbermatter]

I don't really see why variants need a ref-count since you can disassemble a variant as bytes and reconstitute which would mess up that sort of thing.

 

I read (somewhere) that for handles you have to query the "Master" handle table to get hold of a handle to a particular variable so it is a two step process. So I was thinking perhaps the issue is that I'm actually operating on a master table rather than the real handle.  I was kind-a hoping it would just be just a miss-setting in one or more of the CLFNs. Since you haven't proffered a solution or things to try with pointers, I guess it's not trivial and I'll just have to jump down that rabbit hole to see what Alice is up to..

 

Well, refcounting is for ease of resource management. You simply follow some well defined rules about when to call AddRef() and Release() and then the last Release() automatically deallocates the object. It controls the lifetime of the object, not of the data in the object. Flattening a Variant basically AddRefs() the object, retrieves the data and stores it in a flattened stream and then calls Release() on the object. If nobody else still has AddRefed() the object, it will now be deallocated.

 

Unflattening does a new() for a Variant object which creates the object with one refcount, then AddRef() and then unflatten and store the unflattened data in the object, then Release() again. Works perfect for COM and why not for LabVIEW Variants?

 

The master pointer is an old concept taken from the Mac OS 7 memory manager. Basically the memory manager maintained an array of master pointers from which handles got allocated. The entry in the master pointer array was the first pointer in the handle, which pointed to the actual memory area. This allowed the memory manager to maintain a preallocated pool of master pointers and swap allocated memory areas between an "allocated" master pointer pool and an "previously allocated" pool  when a handle was freed, in order to safe potentially some memory allocations, but also meant that that pool could be exhausted and then you had to call a special function MoreMasterPointers() or similar to extend the pool. The current handle implementation in LabVIEW still inherits some characteristics from this, but as far as I know a handle is not anymore allocated from a master pointer array but instead is just a pointer to a pointer to a memory area. The memory area does actually have a hidden header in front of the handle that stores information about the size of the handle, some flags and at least in some builds of LabVIEW a back pointer to the pointer that points to the memory area.    But most of this information is not anymore actively used (except potentially in special internal debug builds to more easily track down memory corruptions).

[ned]

This comment from Aristos Queue about variants might be enlightening: "VARIANTS AREN'T FLAT. The data is picked up off the wire whole, with all the hair hanging off of it (arrays of clusters of arrays of clusters of arrays of... etc) and put in the variant along with the type of the wire."

 

Based on that, it would seem that the variant you're passing is just a pointer (or handle) to a new type description plus a pointer to the data. Since the data itself isn't copied, there should be some reference counting, the same way LabVIEW would do normally with multiple copies of the same data. Not sure this helps you pull that data apart, though.

[Rolf Kalbermatter]

This comment from Aristos Queue about variants might be enlightening: "VARIANTS AREN'T FLAT. The data is picked up off the wire whole, with all the hair hanging off of it (arrays of clusters of arrays of clusters of arrays of... etc) and put in the variant along with the type of the wire."

 

Based on that, it would seem that the variant you're passing is just a pointer (or handle) to a new type description plus a pointer to the data. Since the data itself isn't copied, there should be some reference counting, the same way LabVIEW would do normally with multiple copies of the same data. Not sure this helps you pull that data apart, though.

 

Well, that made me think! It might be not the LvVariant that is refcounted but the data inside. Take a look at ILVDataInterface.h. I think all the diagram data is probably wrapped in that and refcounted and then the LvVariant is a thin wrapper around that. To manage polymorphic data. The ILVData interfaces in there inherit from IUnknown, most likely the same as the COM IUnknown interface, eventhough NI seems to have gone the path of creating a nidl tool that replaces the midl compiler from MS in order to make it work for non MS-platforms.

 

Quite an undertaking but I would certainly start with the source code from Wine for that, who made a widl tool! While Wine is LGPL, as long as it is only for inhouse use, and I doubt NI ever planned to release that toolchain, this would be basically still be fine.

[ShaunR]

Well, that made me think! It might be not the LvVariant that is refcounted but the data inside. Take a look at ILVDataInterface.h. I think all the diagram data is probably wrapped in that and refcounted and then the LvVariant is a thin wrapper around that. To manage polymorphic data. The ILVData interfaces in there inherit from IUnknown, most likely the same as the COM IUnknown interface, eventhough NI seems to have gone the path of creating a nidl tool that replaces the midl compiler from MS in order to make it work for non MS-platforms.

 

Quite an undertaking but I would certainly start with the source code from Wine for that, who made a widl tool! While Wine is LGPL, as long as it is only for inhouse use, and I doubt NI ever planned to release that toolchain, this would be basically still be fine.

 

Now you are showing off

 

If it's really that hard (and undocumented), I'll do something else.

[ShaunR]

Interestingly.

If you try using the GetValueByPointer Xnode, it accepts a variant but produces broken code. 

 

 

Another damn fine plan hits the dust...lol

Is the lvimptsl dynamic library documented anywhere?

Edited July 7, 2013 by ShaunR

[Rolf Kalbermatter]

Another damn fine plan hits the dust...lol

Is the lvimptsl dynamic library documented anywhere?

 

I'm almost 100% sure it is not documented. And from the looks of it it won't really help you here. The function linked to by the xnode is the only useful exported function in there. If the xnode doesn't know how to deal with variants to make the xnode interface work with that library function, the library function itself most likely doesn't know either.

 

But can you explain what you try to do? Do you really need the runtime variant type feature of a Variant, or do you just want adaptable code that is fine to get the right datatype at edit time? If the second is true, some polymorphic VIs and possibly the ILVDataInterface in combination with Adapt To Type as Call Library Node parameter type might be enough.

 

Just be aware that ILVDataInterface is only available since LabVIEW 2009.

[ShaunR]

I'm almost 100% sure it is not documented. And from the looks of it it won't really help you here. The function linked to by the xnode is the only useful exported function in there. If the xnode doesn't know how to deal with variants to make the xnode interface work with that library function, the library function itself most likely doesn't know either.

 

But can you explain what you try to do? Do you really need the runtime variant type feature of a Variant, or do you just want adaptable code that is fine to get the right datatype at edit time? If the second is true, some polymorphic VIs and possibly the ILVDataInterface in combination with Adapt To Type as Call Library Node parameter type might be enough.

 

Just be aware that ILVDataInterface is only available since LabVIEW 2009.

 

It was a long shot and I didn't really have any expectation it would help. But it was worth asking the question.

 

As to what I want to do. You will need to download the code in the Queues vs Circular buffer thread. It's fixed in the demo using doubles for the buffer. I need to be able to allow wiring of any data type so I thought, naively, just adapt to type to a variant. It seems to work great as long as you don't de-allocate the pointer . Maybe I don't need to deallocate,it but it "feels" wrong and probably leaks memory like a sieve as well as a high chance that a gremlin is sitting there just waiting to raise his GPF flag..

Edited July 7, 2013 by ShaunR

[lordexod]

Maybe something like this.

example.vi

[ShaunR]

Maybe something like this.

example.vi

 

It's just too slow to serialise (~10x slower).

[lordexod]

InitStr.vi - It sees only printable ASCII character.

[lordexod]

Simpler version:

example2.vi

Edited July 9, 2013 by lordexod

[ShaunR]

Now you have switched to byte arrays; you don't need a terminating string. But we still need to get rid of the flatten and unflatten which are too slow.

[Aristos Queue]

LvVariant is a C++ class. There is a reference count in the variant object, but I wouldn't expect it to come into play here... can't say for certain. When a To Variant primitive executes, LV does "newVariant = new Variant()". Then newVariant.mTD is assigned the data type object -- this is an object with its own fairly complex structure itself. Then the data on the wire is top-swapped into the variant's "newVariant.mData" field (To Variant's input is a stomper so LV has already guaranteed that the data is independent of any other parallel branch and is thus free to be stomped on). There's also an mAttrs field that the attribute table, which is left as null unless/until an attribute is added to the variant.

 

LvVariant does not have any virtual methods so there is no vTable on the C++ object. But the actual data size of the LvVariant class is (1 + sz + sz + 4 + 4 + sz) bytes, where sz is 8 or 16 depending on how many bytes a pointer has on your system. The data is in the first of those pointers. The reference count is the first of the 4 bytes. There's also a transaction count -- the second 4 byte number -- which is unused within Labview.exe and is exported apparently for use with flex data -- you can basically ignore that.

 

You'll never be able to change out the type descriptor -- that's the final sz in the size. Those are refcounted C++ objects that you have to go through the constructors to instantiate. So you could in theory change out the data within a variant, provided you started with a variant that had the type descriptor you needed already in it. You would need to either top swap (swap, not move) the data into the variant or copy the data in, depending upon whether the source data is stompable. A variant claims right of destruction over the data pointer it contains.

 

Don't know if any of that helps you.

[ShaunR]

LvVariant is a C++ class. There is a reference count in the variant object, but I wouldn't expect it to come into play here... can't say for certain. When a To Variant primitive executes, LV does "newVariant = new Variant()". Then newVariant.mTD is assigned the data type object -- this is an object with its own fairly complex structure itself. Then the data on the wire is top-swapped into the variant's "newVariant.mData" field (To Variant's input is a stomper so LV has already guaranteed that the data is independent of any other parallel branch and is thus free to be stomped on). There's also an mAttrs field that the attribute table, which is left as null unless/until an attribute is added to the variant.

 

LvVariant does not have any virtual methods so there is no vTable on the C++ object. But the actual data size of the LvVariant class is (1 + sz + sz + 4 + 4 + sz) bytes, where sz is 8 or 16 depending on how many bytes a pointer has on your system. The data is in the first of those pointers. The reference count is the first of the 4 bytes. There's also a transaction count -- the second 4 byte number -- which is unused within Labview.exe and is exported apparently for use with flex data -- you can basically ignore that.

 

You'll never be able to change out the type descriptor -- that's the final sz in the size. Those are refcounted C++ objects that you have to go through the constructors to instantiate. So you could in theory change out the data within a variant, provided you started with a variant that had the type descriptor you needed already in it. You would need to either top swap (swap, not move) the data into the variant or copy the data in, depending upon whether the source data is stompable. A variant claims right of destruction over the data pointer it contains.

 

Don't know if any of that helps you.

+1. Plenty for me to play with here. Ta very muchly (did you mean 4 or 8 bytes for the pointer size rather than 8 or 16?)

If the variant claims right of destruction, what happens to the copies in the buffer? Is this the reason why it crashes on deallocation?

Edited July 10, 2013 by ShaunR