Direct read/write of Transfer Buffer possible?..

[dadreamer]

Long time didn't see those hackish threads about LV internals on LAVA, so here's something. As always it's strictly experimental and "just for fun", so don't run the things below on your work systems, otherwise you'll easily get LV crashed, your files deleted and your disk reformatted. You're warned.

As some of you maybe know, starting from LV 2017 there exists hidden ArrayMemInfo node, available w/ the scripting. It provides some info on the input array including its data pointer.

ArrayMemInfo.vi

My first discovery was that this pointer is stable and doesn't change until VI is stopped. So its common use case could be the memory copy operations for some third-party DLLs. It's obvious and it's kinda boring usage. Playing with it after some time I discovered that I'm able to not only read but also write to that pointer, able to get its handle, resize the array etc. - read on to see the examples. I could conclude that this way it's possible to manipulate Transfer Buffer or DCO's Transfer Data or whatever it's called. From now on I started to question myself, where does ArrayMemInfo node take that pointer from and could we do the same without this instrument. And here was another big surprise for me - yes, we can! LV familiar CLFNs easily do that, when you configure the input parameter as:

Type = Adapt to Type

Constant = Yes (checked)

Data format = Pointers to Handles

Actually you could set Type to Array and set Data type and Array format to the actual data type of your array, it's not that important. What is important, Constant checkbox should be checked. In this case LV not only passes the native data types, but also does not create the input data copy, providing the pointer/handle to the Transfer Buffer's data instead! I was very surprised, because it was there from LV 2009, but I didn't pay attention to it. I also did not find it to be described somewhere in the official documents.

Knowing that I have replaced ArrayMemInfo with CLFNs in my basic samples.

Here's how we could do read/write of a common numeric indicator:

numeric_test.vi

And this is the same for a common numeric array:

array_test.vi

As you can see, there's a major disadvantage of the whole technique - one should pass the wire to the indicator in a loop to get the indicator updated in time. Nevertheless there exists a pair of VIs, that do the similar and do not require any wires to be connected to a control/indicator. I'm saying about Get Control Values by Index and Set Control Values by Index, first introduced in LV 2013. So I dug these VIs deeper and came to these two internal functions, which are used by the foresaid VIs and ArrayMemInfo behind the scenes.

ReadDCOTransferData:

ReadDCOTransferData.vi

WriteDCOTransferData:

WriteDCOTransferData.vi

As their top-level wrappers, these functions don't require any wires and care about low-level operations, i.e. no need to resize the array w/ NumericArrayResize, manually set its dimensions and actual data w/ MoveBlock etc. Although you don't get absolute control over a control or indicator (e.g., its pointer or handle), you definitely don't need it as all the work is done by ReadDCOTransferData/WriteDCOTransferData. Finally we came to the conclusion that the best has already been implemented. But... if you really want... there's a way to get Operate Data pointer for a generic control/indicator.

numeric_test_ncg.vi

array_test_ncg.vi

However I'd like to notice, that this technique contains a lot of pitfalls besides shown low-level manipulations. Front Panel object doesn't get updated at all, for example, so it's required to redraw it manually w/ Invalidate method (or click on the object or hide-show the window). Local variables of the object don't get updated too, even when the object is invalidated. I'm sure, there are other things not done, which usually LV does on normal read/write. So, these diagrams should be perceived as the concept only.

In the end I'd say that I hardly imagine, where I could really use any of these non-standard r/w methods. Surely in LV I will continue to use native Get Control Values by Index and Set Control Values by Index VIs, because they do everything, what I want. There might be one use-case though, when communicating with some external code. It would be easier to write debug (or any other) data from a custom DLL into LV's indicator directly, not using PostLVUserEvent or DbgPrintf. Extcode.h doesn't expose many APIs to the user, so calling ReadDCOTransferData/WriteDCOTransferData might be preferred inside the library. But I'm not going to check/test it now, because I haven't written DLLs for LabVIEW for a while.

Edited March 5, 2020 by dadreamer
fixed slight inaccuracy in ReadDCOTransferData/WriteDCOTransferData CLFNs

[Rolf Kalbermatter]

On 2/21/2020 at 10:49 AM, dadreamer said:

Long time didn't see those hackish threads about LV internals on LAVA, so here's something. As always it's strictly experimental and "just for fun", so don't run the things below on your work systems, otherwise you'll easily get LV crashed, your files deleted and your disk reformatted. You're warned.

As some of you maybe know, starting from LV 2017 there exists hidden ArrayMemInfo node, available w/ the scripting. It provides some info on the input array including its data pointer.

That's all nice and pretty until you place an Array Resize node in the array wire to increase the array size. Et voila, the internal pointer most likely (not necessarily always) will change as LabVIEW will have to reallocate a new memory area and copy the old content into the new area and deallocate the original memory. So while this is no real news to people familiar with the LabVIEW memory manager C function interface, it is at best a brittle approach to rely on.

When you write a C DLL that receives handles from LabVIEW, the handle is only guaranteed to exist for the duration of the function call and after you return control to the LabVIEW diagram, LabVIEW reserves the right to resize, delete, or reuse that array handle for other things as it sees fit.

Your array_test.vi is a very Rube Goldberg solution for something that can be solved with a simple Resize Array node. What you basically do is to format the handle value (which is a pointer to the actual memory pointer) into text, then convert that text back into a pointer (handle) and then resize it with NumericArrayResize() and finally copy in the data into that resized handle. It's equivalent to doing a Resize Array on the original array and then a copy into that resized array, although in LabVIEW you wouldn't resize the handle for this but simply create a new one with that data, most easily by branching the wire but if you really want to, you could also use an autoindexing loop to make it a little bit more Rube Goldberg.

I also kind of question the datatype of the third parameter for your ReadDCOTransferData and WriteDCOTransferData. As you set it, you treat it as a pointer (most likely passed by reference) but it should probably be the datatype of the control (passed by reference) (in this specific case the easiest would be then to simply configure it as Adapt to Type).

Edited March 3, 2020 by Rolf Kalbermatter

[dadreamer]

Quote

That's all nice and pretty until you place an Array Resize node in the array wire to increase the array size. Et voila, the internal pointer most likely (not necessarily always) will change as LabVIEW will have to reallocate a new memory area and copy the old content into the new area and deallocate the original memory. So while this is no real news to people familiar with the LabVIEW memory manager C function interface, it is at best a brittle approach to rely on.

When you write a C DLL that receives handles from LabVIEW, the handle is only guaranteed to exist for the duration of the function call and after you return control to the LabVIEW diagram, LabVIEW reserves the right to resize, delete, or reuse that array handle for other things as it sees fit.

Yeah, I'm aware of it and seen such behaviour during my experiments. In this case ArrayMemInfo node should be called again on a "new" wire to retrieve fresh data pointer. Or ArrayMemInfo might be replaced by those SPrintf nodes to obtain the pointer to array's handle instead. That pointer is also stable while VI is executing and the wire exists, and it's updated by LV with new array's handle, when the array is resized or whatever. Nevertheless I strongly doubt that someone would use all those codes for something except "funny" time wasting in LV, even for simple data display, that's why I did not describe absolutely all the nuances. There are many articles on the subject, e.g. "Using External Code in LabVIEW", so I assume, the experimenter understands the deal.

Quote

Your array_test.vi is a very Rube Goldberg solution for something that can be solved with a simple Resize Array node. What you basically do is to format the handle value (which is a pointer to the actual memory pointer) into text, then convert that text back into a pointer (handle) and then resize it with NumericArrayResize() and finally copy in the data into that resized handle. It's equivalent to doing a Resize Array on the original array and then a copy into that resized array, although in LabVIEW you wouldn't resize the handle for this but simply create a new one with that data, most easily by branching the wire but if you really want to, you could also use an autoindexing loop to make it a little bit more Rube Goldberg.

Agree, it looks a little odd, but I didn't want to introduce additional conditional checks, but in the same time I'd like to show how simple operations like read, write and resize of the array work together. They are combined into one Event frame and do their job there. For that I especially studied the source code of NumericArrayResize and made sure, that when the input array's dimSize and the requested new size totalNewSize do match, then NumericArrayResize does nothing and returns. So, no need to check that in LV as it's already checked internally.

As for that twin SPrintf formatting, I didn't find a reliable way in pure LV to pull out the pointer except this one. Maths in extcode.h aren't adapted for 64-bit integers, so I cannot use Max, Abs or something in 64-bit IDE. There exists a trick with _byteswap_uint64 function from msvcr100.dll, but I feel it's not okay to tie to this specific MSVCR version and some additional operations are needed to restore the pointer's byte order back to normal.

Anyway, it's all a "proof-of-concept" thing, so may be changed/played around as desired.

Quote

I also kind of question the datatype of the third parameter for your ReadDCOTransferData and WriteDCOTransferData. As you set it, you treat it as a pointer (most likely passed by reference) but it should probably be the datatype of the control (passed by reference) (in this specific case the easiest would be then to simply configure it as Adapt to Type).

Which parameter do you mean exactly?

int32_t ReadDCOTransferData(uintptr_t viDS, int32_t dcoIdx, uintptr_t *data, int32_t dType, uintptr_t unused, uintptr_t unused);

int32_t WriteDCOTransferData(uintptr_t viDS, int32_t dcoIdx, uintptr_t *data, int32_t dType, uintptr_t unused, uintptr_t unused, int32_t unused);

If you mean data parameter, it is passed as Pointer to Value, because it's a buffer, which the read/write operations are performed on. If you mean dType, it's integer, because it sets the DCO type in VI Data Space (possible values are 0, 1 and 2 as per my BD). I checked all the arguments many times, they seem to fit fine. Last 2 or 3 parameters aren't actually used in the functions, but passed anyways. Maybe it's some relict of earlier LV versions, I did not have a chance to test yet.

 

Also I tried to reproduce the similar pointer reception on CINs according to this excerpt:

But LV makes a copy for the input data, no matter how hard I try. Did it work as described at all?

Edited March 5, 2020 by dadreamer
small clarification

[Rolf Kalbermatter]

3 hours ago, dadreamer said:

As for that twin SPrintf formatting, I didn't find a reliable way in pure LV to pull out the pointer except this one. Maths in extcode.h aren't adapted for 64-bit integers, so I cannot use Max, Abs or something in 64-bit IDE. There exists a trick with _byteswap_uint64 function from msvcr100.dll, but I feel it's not okay to tie to this specific MSVCR version and some additional operations are needed to restore the pointer's byte order back to normal.

My point is, that for the retrieval of the pointer that a handle is, the two SPrintf() calls are utterly Rube Goldberg. They simply return the pointer that a handle is. If you passed the array as Array Handle, Pass Handle by value to the first MoveBlock() function you would achieve the same!

Quote

 

int32_t ReadDCOTransferData(uintptr_t viDS, int32_t dcoIdx, uintptr_t *data, int32_t dType, uintptr_t unused, uintptr_t unused);

int32_t WriteDCOTransferData(uintptr_t viDS, int32_t dcoIdx, uintptr_t *data, int32_t dType, uintptr_t unused, uintptr_t unused, int32_t unused);

If you mean data parameter, it is passed as Pointer to Value, because it's a buffer, which the read/write operations are performed on. If you mean dType, it's integer, because it sets the DCO type in VI Data Space (possible values are 0, 1 and 2 as per my BD). I checked all the arguments many times, they seem to fit fine. Last 2 or 3 parameters aren't actually used in the functions, but passed anyways. Maybe it's some relict of earlier LV versions, I did not have a chance to test yet.

Yes I mean the data parameter. In your diagram you pass the U64 value of the control/indicator to it and declare the parameter as pointer sized integer (passed by reference?). But it should be the data type of the control passed by reference (so U64 passed by reference). For more complex values like clusters and arrays it is probably best to configure this parameter simply as Adapt to Type (pass handles by reference).

Quote

Also I tried to reproduce the similar pointer reception on CINs according to this excerpt:

But LV makes a copy for the input data, no matter how hard I try. Did it work as described at all?

CINS always passed data by reference. There was no way to configure this differently. There was also no way to tell LabVIEW that a parameter was const or not other than by the fact if the output (right) terminal was connected.

For the Call Library node you have a Constant checkbox in the parameter declaration. This is a hint to LabVIEW that the DLL function will NOT modify (stomp on) the data and LabVIEW is free to schedule code in such a way to first execute this CLN before other functions that may want to modify data in place. If only one sink to a wire is marked as stomper (wanting to modify the data), LabVIEW can save copying the data to pass to the different nodes by making sure to schedule the non stomping nodes first (and even in parallel if possible) before finally executing the one stomper node. Even if there are multiple stomper sinks on a single wire, it will schedule them such that all non-stomping nodes are executed first if possible from the dataflow, and then create n - 1 data copies to pass to the different stomper nodes (with n being the number of nodes that are indicating that they MIGHT stomp on the data).

Yes this might be overzealous if the node decides to not stomp on the data anyways, but LabVIEW always tries to work by the principle to be better safe than sorry in this respect (and has failed in the past sometimes in some cases but that are bugs the LabVIEW team wants to have squished ASAP when they get aware of them).

Edited March 5, 2020 by Rolf Kalbermatter

[dadreamer]

Quote

My point is, that for the retrieval of the pointer that a handle is, the two SPrintf() calls are utterly Rube Goldberg. They simply return the pointer that a handle is. If you passed the array as Array Handle, Pass Handle by value to the first MoveBlock() function you would achieve the same!

I don't need a handle (i.e., TD1 **TD1Hdl) here. I want a pointer to a handle (i.e., TD1Hdl *HdlPtr). That's why I do that cumbersome SPrintf stuff. When the VI is started for the very first time, the array's handle is NULL. Later I resize the array or do some other things with it, so the handle becomes a real value. It also can change between the runs or on a new resize. Of course, I could do some logic and use a shift register to store the actual handle, but as to me it's better to do some complicated task once, rather than do it again and again in a loop. Moreover, I use the similar method not only on arrays, but also on controls, so I need a pointer to the control as well. MoveBlock dereferences the pointer and I actually get nothing to deal with later.

Quote

Yes I mean the data parameter. In your diagram you pass the U64 value of the control/indicator to it and declare the parameter as pointer sized integer (passed by reference?). But it should be the data type of the control passed by reference (so U64 passed by reference). For more complex values like clusters and arrays it is probably best to configure this parameter simply as Adapt to Type (pass handles by reference).

This is my omission. I didn't even notice until you said. Numerous copy-paste operations do their evil things, when using on complex nodes like CLFNs. Fixed the VIs in my first post. Thanks. 😸

Quote

CINS always passed data by reference.

When I had a very limited test of the above VIs on LabVIEW 8.5, I noticed, that LabVIEW gives away fine (non-copied) pointer for array and the whole stuff works, but for numeric control it is no longer working. With CINs the situation is even worse. Regardless of the way how the parameters are passed, LabVIEW makes their copies and passes them instead of the originals. But under some circumstances no copies are created, e.g. when I pass the wire from Random Number VI into CIN directly instead of passing it from the control's terminal. Maybe I could try to find out what’s the matter, if possible.

[Rolf Kalbermatter]

2 hours ago, dadreamer said:

I don't need a handle (i.e., TD1 **TD1Hdl) here. I want a pointer to a handle (i.e., TD1Hdl *HdlPtr). That's why I do that cumbersome SPrintf stuff. When the VI is started for the very first time, the array's handle is NULL. Later I resize the array or do some other things with it, so the handle becomes a real value. It also can change between the runs or on a new resize. Of course, I could do some logic and use a shift register to store the actual handle, but as to me it's better to do some complicated task once, rather than do it again and again in a loop. Moreover, I use the similar method not only on arrays, but also on controls, so I need a pointer to the control as well. MoveBlock dereferences the pointer and I actually get nothing to deal with later.

That makes no sense. But I"m not going to tell you you can't do that. 😀

Quote

When I had a very limited test of the above VIs on LabVIEW 8.5, I noticed, that LabVIEW gives away fine (non-copied) pointer for array and the whole stuff works, but for numeric control it is no longer working. With CINs the situation is even worse. Regardless of the way how the parameters are passed, LabVIEW makes their copies and passes them instead of the originals. But under some circumstances no copies are created, e.g. when I pass the wire from Random Number VI into CIN directly instead of passing it from the control's terminal. Maybe I could try to find out what’s the matter, if possible.

The control value is completely seperated from the data value in the wire. Assuming that they are the same is utterly useless. There might be circumstances where they appear to be the same but threating them like that would simply cause potential trouble. You can NOT control LabVIEWs memory mnagement on such a level. LabVIEW reserves the right to reschedule code execution and memory reallocations depending on seemingly minimal changes and it also can and will change between versions. The only thing you can say is that in a particular version without any change to the diagram you SHOULD get the same result but anything else is simply not safe to assume.

Trying to use this for a (possibly perceived) performance gain is utterly asking for a lot of pain in the long run. So don't do it!

Edited March 5, 2020 by Rolf Kalbermatter

[dadreamer]

8 hours ago, Rolf Kalbermatter said:

That makes no sense. But I"m not going to tell you you can't do that. 😀

Okay, maybe this will make a sense?.. We want to write something into LV control/indicator (say, numeric array, doesn't matter) directly. So, convenient DS... functions are not enough for us. We need some well-established storage in VI Data Space, that is associated with our control. *HdlPtr behaves like that storage, because it's actually a memory address in VI Data Space, where we could read/write from/to and all the changes are immediately applied to our control. If you'd use the traditional method with MoveBlock and DS..., you get a handle equal to NULL. But you want to write to the control, how would you do that then? You need to tell LabVIEW, that you've created a fresh handle and want to have it applied to your control.

When I've been experimenting w/ all this, my perspective was that the pointer can be passed anywhere you want, e.g. into external code, and you can do anything with it there. LabVIEW reacts to your manipulations and updates the control as you wish. In the examples everything is done on BD, but it all could be easily translated into C/C++ or another language of your choice. I might write a sample DLL to illustrate this better, but why to bother, if it may be done from the diagram?.. It's only LV-related disadvantage, that many CLFNs look bulky, in C the corresponding calls occupy few lines of code.

9 hours ago, Rolf Kalbermatter said:

The control value is completely seperated from the data value in the wire. Assuming that they are the same is utterly useless.

I fact, this is what I always kept in mind, when creating this thread. 🙂 There's even some short but neat NI presentation - Improving the Performance of your LabVIEW Applications.

9 hours ago, Rolf Kalbermatter said:

it also can and will change between versions. The only thing you can say is that in a particular version without any change to the diagram you SHOULD get the same result but anything else is simply not safe to assume.

I assume, that something could change in the memory managements, but not so drastically. DCOs/DDOs concept existed from the very first LV versions (I saw it in 2.5 already), so the basic rules to deal with these objects should stay the same. If the changes would be such significant, then you'd get a partial or even full lack of backward compatibility w/ the previous versions. Remember, modern LV still loads and runs all the versions back to 7.1. And I've personally never seen, that a generic control's pointer changes during the execution, or even that I'm unable to get it with the way described. Never liked to be wrong (who likes?), so has checked my samples on many LV versions back to 2009. Anyway, it's all not for the real projects, so even if something changes, who cares?.. 😀 As to me, I very seldomly use hacks in my work, I could count 3 or maybe 4 private property nodes for the whole term of my LV programming. The hacks are just a matter of interest, no more.

9 hours ago, Rolf Kalbermatter said:

Trying to use this for a (possibly perceived) performance gain is utterly asking for a lot of pain in the long run. So don't do it!

Actually I'm stopping at this point, because I feel like it's a waste of time, if speak about CINs, as no any man on Earth uses CINs these days. As to CLFNs, I told the essence of what I know about the subject, so it's done. From the very LV programming days I had a question, if we could get a classic pointer to a wire or a control. Well, now ten years later, I've got the answer. It's not such a classic and not such a practical though.

[Rolf Kalbermatter]

9 hours ago, dadreamer said:

Okay, maybe this will make a sense?.. We want to write something into LV control/indicator (say, numeric array, doesn't matter) directly. So, convenient DS... functions are not enough for us. We need some well-established storage in VI Data Space, that is associated with our control. *HdlPtr behaves like that storage, because it's actually a memory address in VI Data Space, where we could read/write from/to and all the changes are immediately applied to our control. If you'd use the traditional method with MoveBlock and DS..., you get a handle equal to NULL. But you want to write to the control, how would you do that then? You need to tell LabVIEW, that you've created a fresh handle and want to have it applied to your control.

That's a terrible hack. LabVIEW controls are not pointers. They are complex objects whose data elements can be dynamically allocated, resized and deallocated. It may work for a while but is utterly susceptible to LabVIEW versions, platforms, and what else!

Quote

I assume, that something could change in the memory managements, but not so drastically. DCOs/DDOs concept existed from the very first LV versions (I saw it in 2.5 already), so the basic rules to deal with these objects should stay the same. If the changes would be such significant, then you'd get a partial or even full lack of backward compatibility w/ the previous versions. Remember, modern LV still loads and runs all the versions back to 7.1. And I've personally never seen, that a generic control's pointer changes during the execution, or even that I'm unable to get it with the way described. Never liked to be wrong (who likes?), so has checked my samples on many LV versions back to 2009. Anyway, it's all not for the real projects, so even if something changes, who cares?.. 😀 As to me, I very seldomly use hacks in my work, I could count 3 or maybe 4 private property nodes for the whole term of my LV programming. The hacks are just a matter of interest, no more.

DCO/DDO concept is quite something else than the actual memory layout. It's related but there is no hard rule that says that the DCO or DDO layout in memory can't change and it has changed in the past.

Quote

Actually I'm stopping at this point, because I feel like it's a waste of time, if speak about CINs, as no any man on Earth uses CINs these days. As to CLFNs, I told the essence of what I know about the subject, so it's done. From the very LV programming days I had a question, if we could get a classic pointer to a wire or a control. Well, now ten years later, I've got the answer. It's not such a classic and not such a practical though.

CIN support is utterly legacy. There are no tools to create CINs for any of the new platforms that have been released since around LabVIEW 8.2. That includes all the 64-bit platforms as well as the NI Linux RT platforms (both ARM and x64).

One very bad aspect of CINs is that the code resource is platform specific and needs to be inside the VI and there is only space for one code resource per VI per CIN routine. So if you want to support multiplatform you have to create for each platform a copy of the VI and put the according code resource in there and then somehow place the correct VI on the system when installing your library to a particular platform. Utter maintenance nightmare! 

Edited March 6, 2020 by Rolf Kalbermatter