Undocumented LabVIEW

[CopperD]

I am starting to dig into the undocumented LabVIEW functions. I am sure some of us have little crumbs of information from NI and other have figured things out on our own. How can we round up this information so we don't keep repeating ourselves? Currently I am working on the thread management functions and how the IDE is notified of them so they can be aborted. At some point in time I saw a wiki but that seems to no longer exist.

[crelf]

The LabVIEWWiki has a collection of these, compiled from a lot of different sources. Unfortunately, it's down right now, but you can browse from an archive: http://web.archive.org/web/20150531121321/http://labviewwiki.org/LabVIEW_configuration_file

[Michael Aivaliotis]

I moved this thread to the lounge.

The Wiki content still exists, I think. It has some technical issues to resolve. I will take a cursory look into the effort it will take to resurrect it. Before it went dark, the ability to edit was only enabled for account holders and I got zero requests from people to create an account in over 5 years. But I guess it only takes 1 person with determination to get the ball rolling.

[Antoine Chalons]

16 hours ago, CopperD said:

Currently I am working on the thread management functions and how the IDE is notified of them so they can be aborted. At some point in time I saw a wiki but that seems to no longer exist.

Is this what you're looking for?

[Rolf Kalbermatter]

17 hours ago, CopperD said:

I am starting to dig into the undocumented LabVIEW functions. I am sure some of us have little crumbs of information from NI and other have figured things out on our own. How can we round up this information so we don't keep repeating ourselves? Currently I am working on the thread management functions and how the IDE is notified of them so they can be aborted. At some point in time I saw a wiki but that seems to no longer exist.

From your post in for the DCG in the code repository I take it that you are talking here about the THThreadCreate() and friends C API. I'm afraid that there is not really a way to make the IDE aware of this thread in any way. These functions are only thin wrappers around the platform thread managment functions (CreateThread on Windows, pthread_create on Linux, etc.). As such they are used by the LabVIEW kernel to manage threads in a way that makes the actual kernel almost completely independent of the underlaying platform API, but are on such a low level that the IDE is no aware about them unless it created them themselves.

Basically calling any of these LabVIEW manager functions (memory, file, thread, etc) is more or less equal to directly calling the underlaying system API directly but with the advantage that your C code doesn't have to worry about different APIs when you try to compile it for another LabVIEW  target like Linux or Mac OSX. If you only want to work in Windows, calling CreateThread() directly is actually the more direct and simpler way of doing this.

What is your actual issue about wanting to have the IDE be aware of your created threads?

Edited April 29, 2016 by rolfk

[ShaunR]

There also used to be a "Rusty Nails" section on LavaG.org for stuff that was cool but might blow up LabVIEW. I wonder what happened to that?

[CopperD]

1 hour ago, rolfk said:

I'm afraid that there is not really a way to make the IDE aware of this thread in any way.

I view that as a personal challenge. I already learned from Chris that what I want exists but is not exposed. A lack of an export entry is a minor hurdle to overcome. I can generate signatures once I find what I am looking for so the code will keep working unless a major update is done. I need to dig around and figure out how large parts of the IDE work. This is why I suggested a place so I can put this information so no one else needs to repeat what I have done and can pick up where I leave off.

1 hour ago, rolfk said:

What is your actual issue about wanting to have the IDE be aware of your created threads?

When the user hits the abort button in the IDE I want the threads they created to be killed. If this does not occur it will lead to many headaches. I could setup some sort of heartbeat or use what is already built into the IDE.

 

It sounds like you have some of this inside knowledge of how LabVIEW works.

[drjdpowell]

33 minutes ago, CopperD said:

When the user hits the abort button in the IDE I want the threads they created to be killed.

What are you meaning by a “thread” here?  Users don’t create threads.

[CopperD]

The easiest way (Without using a wrapper dll) right now to handle a function pointer is to use CreateThread or ThThreadCreate. This creates a thread that might not return depending on what their code does. It would be nice if the developer can hit the abort button and all such threads are killed otherwise you need to end the primary thread.This is a non-issue on the compiled code and is just need to improve the experience inside the LabVIEW IDE. 

 

I been thinking of adding code to the abort button to kill these threads.

[ShaunR]

1 hour ago, CopperD said:

When the user hits the abort button in the IDE I want the threads they created to be killed. If this does not occur it will lead to many headaches. I could setup some sort of heartbeat or use what is already built into the IDE.

This has other positive implications for developers that interface to external code.

One issue I (we?) have is that we cannot pass opaque pointers to the "abort", "reserved" and "unreserved callbacks" in CLFNs which are direct hooks to the LabVIEW IDE.

Many 3rd party API libraries have an Init and Deinit procedure which is global. You only want to de-init if

1. An init succeeded and allocated resources(no error, say)
2. No residual inits are floating around from the last run.

However, the callbacks for the CLFN are on each node so you only know that you can cater for 1&2 if the last node is executed and other nodes didn't error. Aborting in the IDE LabVIEW stops it dead before the last node and therefore you leak the callback. Any other strategy results in a crash of LabVIEW because you don't know whether the callback actually exists.

If we had an abort hook (and a reserve and unreserve) that could take arguments then we can easily cater for this type of API. 

As an example.

In order for the the OpenSSL library to be thread safe you must attach a callback to the library. This is a global callback, across all instances and invocations of the library and can only exist once as it is basically a mutex function. So when the library is initialised, a thread callback is created and passed to the API . You are supposed to then free the pointer and "de-init" the callback when you shut down but - and this is a "but" with a capital "CRASH" - only if you created the callback rather than another application.....bummer! 

Regardless of that GOTCHA; when you abort the IDE, however, you cannot execute your shutdown function which is in the close or De Init VI. Similarly, There is no way to track the callback instance from the LabVIEW dev environment since the pointer allocation exists outside of LabVIEW so you don;lt know how many de-inits to call to clean up. So when you press abort, you leak that callback until LabVIEW exits and the OS cleans it up for you. Ideally, we would use the abort, reserved and unreserved callback function of the CFLN to clean up. (Note that this is not a problem in an application as a Close or De-Init can always be called and is no worse than forgetting to close a queue refnum)

Now. NI will argue that you can create these functions in you DLL and call them so it's not a problem. Except that, of course that would only work if you run all nodes in the root loop which defeats the whole point of multi-threaded callback -DOH! (as well as a huge performance hit).Additionally and probably even worse for me is that I use unmolested (or I should say "untampered with" so that hashes can be checked) 3rd party code so that users can update binaries without my intervention and, of course I don't run the risk of introducing nasty, hard to trace bugs into an already extensively tested API I'm not intimate with. 

So your abort could also enable a global cleanup of a 3rd party API when in the IDE. We could detect the run-time engine or IDE and snatch whatever arguments the clean-up requires then, using your abort hook, pass that directly to the APIs clean up function.

Edited April 29, 2016 by ShaunR

[CopperD]

This is turning into a very useful discussing. I will focus my efforts into the abort button. I knew sooner or later my craziness would become helpful. 

[Rolf Kalbermatter]

1 hour ago, CopperD said:

I view that as a personal challenge. I already learned from Chris that what I want exists but is not exposed. A lack of an export entry is a minor hurdle to overcome. I can generate signatures once I find what I am looking for so the code will keep working unless a major update is done. I need to dig around and figure out how large parts of the IDE work. This is why I suggested a place so I can put this information so no one else needs to repeat what I have done and can pick up where I leave off.

When the user hits the abort button in the IDE I want the threads they created to be killed. If this does not occur it will lead to many headaches. I could setup some sort of heartbeat or use what is already built into the IDE.

 

It sounds like you have some of this inside knowledge of how LabVIEW works.

There is one publically available interface that could allow something like this although not very well documented. It is the callback configuration in the Call Library Interface Node. Basically every Call Library Node can register "callbacks" that are called when the diagram on which the Call Library Node is placed is initialized and deinitialized. And all these callbacks have a pointer parameter that can also be configured to be passed to the actual Call Library function too.

It's not straightforward but a lot easier than trying to figure out code signatures to call into internal functions that might vary from LabVIEW version to version.

[ShaunR]

3 minutes ago, rolfk said:

There is one publically available interface that could allow something like this although not very well documented. It is the callback configuration in the Call Library Interface Node. Basically every Call Library Node can register "callbacks" that are called when the diagram on which the Call Library Node is placed is initialized and deinitialized. And all these callbacks have a pointer parameter that can also be configured to be passed to the actual Call Library function too.

It's not straightforward but a lot easier than trying to figure out code signatures to call into internal functions that might vary from LabVIEW version to version.

I'm hoping he'll hijack all those and replace them with ones we can send arguments to  10 years ago I complained about this to NI 

We'll need to talk about how we configure such a hack (per project, per VI, per node, per IDE, any or all of the above?  )

[CopperD]

1 minute ago, ShaunR said:

I'm hoping he'll hijack all those and replace them with ones we can send arguments to

Someone who understands!

For what I want to do I need to do more then just jump to the function pointer. I need to call it with the correct calling convention and pass arguments. When abort is called we need it to call some cleanup code that may have arguments that need sent.

 

Idea scratch pad

If you do not know the function pointer to the function you want to use for clean up you can use LoadLibrary to get the handle to the library then pass that to GetProcAddress with the name of the function you want called. This can be automated and is used for the generation of the import table in my PE Loader.

The code for abort will take a pointer to a cluster that contains the number of cleanup entries and an array of cleanup entries. The clean up entries contain the function pointer and arguments to pass. Call each cleanup then return to the normal abort. 

Cleanup entry Function pointer, calling convention, number of parameters, array of parameters.

To use, place VI that will check if inside IDE then inject code to run clean up. A pointer is given where we will put the pointer to our cleanup cluster. We are free to add and remove entries from this cluster during runtime. The user will remain unaware of these pointers so it remains easy to use.

Simple vi that will add or remove functions to this list and update the arguments the developers wants to pass in. 

[Rolf Kalbermatter]

4 hours ago, ShaunR said:

This has other positive implications for developers that interface to external code.

One issue I (we?) have is that we cannot pass opaque pointers to the "abort", "reserved" and "unreserved callbacks" in CLFNs which are direct hooks to the LabVIEW IDE.

Many 3rd party API libraries have an Init and Deinit procedure which is global. You only want to de-init if

1. An init succeeded and allocated resources(no error, say)
2. No residual inits are floating around from the last run.

However, the callbacks for the CLFN are on each node so you only know that you can cater for 1&2 if the last node is executed and other nodes didn't error. Aborting in the IDE LabVIEW stops it dead before the last node and therefore you leak the callback. Any other strategy results in a crash of LabVIEW because you don't know whether the callback actually exists.

If we had an abort hook (and a reserve and unreserve) that could take arguments then we can easily cater for this type of API. 

 

But the three callbacks all have one argument. It's the InstanceDataPtr. This is a pointer to a pointer sized variable that is stored by LabVIEW for each CLN callsite. LabVIEW provides you that storage, what you store in there is up to you. It could be an integer that allows you to identify the resource associated to this callsite or a ponter to a structure as simple or complicated as you wish. The reserve function is called when the diagram is iniitialized. the unreserve before the diagram is unloaded and the abort when the hierarchy that this CLN callsite is located in is aborted. And yes you can configure the CLN to pass exactly this InstanceDataPtr also to the actiual function. It won't show up as a parameter on the CLN icon but LabVIEW will pass exactly this pointer to the function. So your reserve function allocates some datapointer to identify the callsite, the actual function stores whatever it wants into that pointer and the abort function checks if there is anything in there that needs to be aborted, canceled or whatever. The unreserve function needs to deallocate any resources that were allocated in either the reserve or run function and not yet cleared by the abort function.

 

1 hour ago, CopperD said:

Someone who understands!

For what I want to do I need to do more then just jump to the function pointer. I need to call it with the correct calling convention and pass arguments. When abort is called we need it to call some cleanup code that may have arguments that need sent.

 

Idea scratch pad

If you do not know the function pointer to the function you want to use for clean up you can use LoadLibrary to get the handle to the library then pass that to GetProcAddress with the name of the function you want called. This can be automated and is used for the generation of the import table in my PE Loader.

The code for abort will take a pointer to a cluster that contains the number of cleanup entries and an array of cleanup entries. The clean up entries contain the function pointer and arguments to pass. Call each cleanup then return to the normal abort. 

Cleanup entry Function pointer, calling convention, number of parameters, array of parameters.

To use, place VI that will check if inside IDE then inject code to run clean up. A pointer is given where we will put the pointer to our cleanup cluster. We are free to add and remove entries from this cluster during runtime. The user will remain unaware of these pointers so it remains easy to use.

Simple vi that will add or remove functions to this list and update the arguments the developers wants to pass in. 

Sounds pretty much like what the undocumented RTSetCleanupProc() function does:

enum {        /* cleanup modes (when to call cleanup proc) */
    kCleanRemove,
    kCleanExit,                /* only when LabVIEW exits */
    kCleanOnIdle,            /* whenever active vi hierarchy goes idle */
    kCleanAfterReset,        /* whenever active vi goes idle after a reset */
    };

typedef int32 (_FUNCC *CleanupProcPtr)(uintptr_t resource);
TH_REENTRANT int32 _FUNCC RTSetCleanupProc(CleanupProcPtr proc, uintptr_t resource, int32 mode);
 

Basically this function can be used to register a pointer sized integer with according cleanup function pointer, that will be called with the integer parameter whenever the event that is used as mode happens. This is supposedly the functionality that is used by most LabVIEW refnums. They all use the kCleanOnIdle mode. Don't forget to call this function again with the same function pointer and pointer sized integer and the kCleanRemove mode whenever you are finished with this resource so that the cleanup entry gets removed from the internal list.

And yes these entries seem to be stored in a linear list so it might be a good idea not to flood that list with unneccessary resource entries in order to keep your LabVIEW system responsive. If you find this fact disturbing or worse, please forget that you heard about this whole funcitonality.

Edited April 29, 2016 by rolfk

[ShaunR]

11 hours ago, rolfk said:

But the three callbacks all have one argument. It's the InstanceDataPtr. This is a pointer to a pointer sized variable that is stored by LabVIEW for each CLN callsite. LabVIEW provides you that storage, what you store in there is up to you. It could be an integer that allows you to identify the resource associated to this callsite or a ponter to a structure as simple or complicated as you wish. The reserve function is called when the diagram is iniitialized. the unreserve before the diagram is unloaded and the abort when the hierarchy that this CLN callsite is located in is aborted. And yes you can configure the CLN to pass exactly this InstanceDataPtr also to the actiual function. It won't show up as a parameter on the CLN icon but LabVIEW will pass exactly this pointer to the function. So your reserve function allocates some datapointer to identify the callsite, the actual function stores whatever it wants into that pointer and the abort function checks if there is anything in there that needs to be aborted, canceled or whatever. The unreserve function needs to deallocate any resources that were allocated in either the reserve or run function and not yet cleared by the abort function.

Yes. But........

1. 3rd party APIs never create their functions with that in mind and may need more than one parameter or the pointer is passed back as a result rather than a parameter.
2. You can't guarantee what order the multiple CFLN callbacks will be executed in so the "Close" may be before others (can only destroy it once there is no possibility it will be used).
3. Can only populate it from within the DLL requiring special functions and handling in the DLL rather than LabVIEW.
4. Cannot test to see if the pointer is valid (ok, so thats a hard one regardless but you can do stuff to mitigate it)

Some of those aren't an issue for you since you use a wrapper DLL so you can write special functions to fit squares into round holes. You probably don't need it at all because you can make your DLL arbiter and guardian of the pointers.As you know,I choose not to write wrapper DLLs and the InstanceDataPtr doesn't quite cover it.

Edited April 30, 2016 by ShaunR

[CopperD]

On 4/29/2016 at 3:30 PM, rolfk said:

Sounds pretty much like what the undocumented RTSetCleanupProc() function does:

enum {        /* cleanup modes (when to call cleanup proc) */
    kCleanRemove,
    kCleanExit,                /* only when LabVIEW exits */
    kCleanOnIdle,            /* whenever active vi hierarchy goes idle */
    kCleanAfterReset,        /* whenever active vi goes idle after a reset */
    };

typedef int32 (_FUNCC *CleanupProcPtr)(uintptr_t resource);
TH_REENTRANT int32 _FUNCC RTSetCleanupProc(CleanupProcPtr proc, uintptr_t resource, int32 mode);

Looks like both of you are correct.

If this works on abort then it will save me some trouble today. It can call my code which can handle the the more complex functions and ordering.  

Hmm has anyone figured out how to go from a vi refnum to an entry point for the function? If this can be figured out it will make it much easier for people to modify this.

I have many different angles of attack here so I am very confident something will work out.

 

[Rolf Kalbermatter]

53 minutes ago, CopperD said:

a vi refnum to an entry point for the function? If this can be figured out it will make it much easier for people to modify this.

Can you be more clear? Which function? RTSetCleanupProc() only indirectly has a reference to the current top level VI, this function doesn't take a VI reference in any way, It determines that internally.

[CopperD]

This has nothing to do with RTSetCleanupProc() directly. 

 

When you call a vi by reference how does the caller know where the executable code is? 

 

CallVIFromDll looks like an interesting function name. 

Edited May 1, 2016 by CopperD

[Rolf Kalbermatter]

I really don't know the details. A VI reference is simply a somewhat complex index into a list of VI data structures. This VI data structure is a huge C struct that contains all kinds of data elements including pointers to pointers to pointers to pointers of various data. Some of this is the diagram heap, the front panel heap, the data space heap, and the compiled code heap. And this structure changes with every LabVIEW version significantly since there is no external code ever directly accessing it.

Interfacing anything in this structure directly is a complete nogo as it will be totally incompatible with every new LabVIEW version, even service releases and potentially bug fix releases. And the actual machine code is not just a function pointer segment that you can jump too with the parameters on the stack, instead the code directly refers to it's parameters through VI internal structures from the conpane and all. Setting that up correctly on your own is definitely a way into total crazyness. 

[CopperD]

On 5/1/2016 at 6:18 PM, rolfk said:

definitely a way into total crazyness. 

I have already begun the slide into madness. I tore into DFIR and LLVM Sunday evening and monday morning.

CallVIFromDll is not used by anyone in LabVIEW dir. I checked all the imports and found no one using it. I have the function prototype figured out. (Sorry I let that info at home) Normal applications and Dlls don't use this either. This function seems to be used only by .NET Dlls.

 

I have the crazy idea of compiling a simple VI to dll so I can easily get the executable code then searching the IDE's memory space for same code. Once found I can put some breakpoints in the debugger and work backwards.

 

The easy solution is to use RTSetCleanupProc to call my code which will handle cleanup based on info from a cluster. 

[Rolf Kalbermatter]

On 3-5-2016 at 0:48 AM, CopperD said:

CallVIFromDll is not used by anyone in LabVIEW dir. I checked all the imports and found no one using it. I have the function prototype figured out. (Sorry I let that info at home) Normal applications and Dlls don't use this either. This function seems to be used only by .NET Dlls.

This function most likely was added at some point for TestStand to call LabVIEW test adapters. But I'm pretty sure they use different functionality in newer versions. And using this has a high change of breaking at some point. NI controlling both the caller as well as the callee and not having documented that function in any way, can easily decide to change something about this interface. They won't do it just to pester you, as the work to make these changes is significant, but if there is a technical or business reason to do it, they will and will not even acknowledge your complaints of breaking this functionality.

I've been investigating the possibilities of calling VIs directly from a DLL myself in the past and came across this and some other functionality but simply didn't feel it was worth spending to much time on it if it was not somehow publically acknowledged by NI. It would have been a very handy thing to use in LuaVIEW as it would make callbacks into LabVIEW code so much easier for the user than what it does now. And it would most likely allow multiple turnarounds between Lua and LabvIEW code which LuaVIEW currently disallows to even attempt because of the stack frame handling which gets completely out of control if we would try to allow that. 

Edited May 6, 2016 by rolfk