Inelegant DLL Interface [long rant/question]

[tkuiper]

After scouring Google, the darker side, OpenG, and now LAVA, I've all but come to the conclusion that DLL interaction in LabVIEW is inelegant at best, at least as far as what I'm wanting to do. I understand exactly why this is, (runtime engine, etc.) but that doesn't stop me from hating it.

Here's what I'm doing: I have a LabVIEW application that calls into a DLL. This DLL would really like a few callback functions. I can satisfy that by either of 2 options, but the main sticking point is those Callbacks need to access and operate on LabVIEW information. The options I've seen so far:

1. [vastly inferior and may not work after all] writing my own callbacks in C and embedding them into foo.dll to somehow interact with LabVIEW or
2. [inelegant but might actually work!] build those 'callbacks' as VIs and compile down into a LV-DLL, and load/call into this LV-DLL from the original DLL.

The main issue here is that those callbacks (supplied to the DLL) need to access LabVIEW data/GOOP objects. The LabVIEW ActiveX server is not an option (to support distribution where the development environment is not available) and other more bulky workarounds that have previously been suggested (TCP Client/Server, reverse callbacks, etc.) are infeasible due to the large amount of data/decision making necessary in those callbacks.

So for all my research and thoughtwork, I'm left with the inelegant option #2 where I create a LabVIEW DLL (with all the incredibly carefully constructed callback function-VIs and their prototypes) to handle the DLL-to-LabVIEW interface. (Which is entirely probable by passing a few key "user values" - GOOP references, etc - from LabVIEW to DLL and back to LV-DLL.) And, yes, this method does assume that the DLL and the original Application are sharing the same process space. (so the GOOP references are indeed valid) If not, then I have the unenviable task of porting 3.5Mb (yes, three-and-a-half megabytes) of C code to LabVIEW.

[rant]

That's just nasty. Who in their right mind forces you to split your code into 2 separate modules? Especially when those functions are designed to be tightly integrated and very relevant to eachother?

I understand that callbacks violate dataflow programming and that due to the fact that LabVIEW DLLs are still really just compiled bytecode wrapped in a runtime engine (Java/.NET CLR anyone?) it would really, really, really be nice to extend such external-callback functionality to the callback VI subsystem. The mechanisms are present for ActiveX and .NET, why not DLLs?

Yaknow, I wouldn't even mind if those callback functions just fired events (which would, however, require a 'response event' or some other such synchronization structure) caught by an event structure! (Even though that would be far more inelegant, obfuscated, and kludgy than the nasty multiple-dll mess I'm proposing.)

[/rant]

OK, sorry about that...

Have I missed something?

Have any of you LabVIEW gurus come up with a more elegant solution to this problem?

Thanks for your time,

Tony

[Rolf Kalbermatter]

The options I've seen so far:

- [vastly inferior and may not work after all] writing my own callbacks in C and embedding them into foo.dll to somehow interact with LabVIEW or

Why do you say this is inferior? It is not, because doing that is basically the only proper way to merge a C callback with the LabVIEW dataflow programming. There are several possibilities depending on the requirements you have.

1. Calling the PostLVUserEvent() function you can pass data back to LabVIEW, in fact any data you want but you need to create LabVIEW compatible native data for that (LabVIEW handles for strings and arrays). The event is then processed in the according user event in the event loop, so there is a possible problem with serialization of all the callbacks if processed in the same event structure.

2. You can use Occur() to trigger a LabVIEW occurrence. The advantage is that it is not serialized with other occurrences that you might trigger from other callbacks but the disadvantage is that you can not pass data back with the occurrence trigger.

3. To solve the problem with the occurrence not having any means to pass data with, you can do your own queue code in C that holds elements that the callback functions put in before they trigger the occurrence. When the occurrence is triggered in LabVIEW you can read the queue for new data.

So it is definitly not inferior to do that. The only problem with this is that you do need to write some (good and not very trivial) C code, but that is no reason to call this inferior.

- [inelegant but might actually work!] build those 'callbacks' as VIs and compile down into a LV-DLL, and load/call into this LV-DLL from the original DLL.The main issue here is that those callbacks (supplied to the DLL) need to access LabVIEW data/GOOP objects.

The main issue with this is not so much the fact that they need to access GOOP data or any LabVIEW data, but much more a maintenance nightmare. As soon as those callbacks DLLs are not compiled in exactly the same version as the LabVIEW version you use to call them, those DLLs will execute in the according LabVIEW runtime engine and as such are in fact almost out of process from the calling LabVIEW environment. This will indeed make the sharing of data between the DLL and the calling LabVIEW process impossible but there are also other problems why you rather don't want to do that.

The LabVIEW ActiveX server is not an option (to support distribution where the development environment is not available) and other more bulky workarounds that have previously been suggested (TCP Client/Server, reverse callbacks, etc.) are infeasible due to the large amount of data/decision making necessary in those callbacks.

Actually LabVIEW executables can serve as Activex Automation server just as well as the LabVIEW development system. Check out the build settings of your project for the place where you enable ActiveX server and specify the name under which it will be registered. But the problem with ActiveX for doing things will be probably the same as for TCP/IP since they are both out of process technologies so if TCP/IP won't work, ActiveX likely will be even worse.

I understand that callbacks violate dataflow programming and that due to the fact that LabVIEW DLLs are still really just compiled bytecode wrapped in a runtime engine (Java/.NET CLR anyone?) it would really, really, really be nice to extend such external-callback functionality to the callback VI subsystem. The mechanisms are present for ActiveX and .NET, why not DLLs?

Because ActiveX and .Net have a very strictly typed interface description where LabVIEW can get all the necessary information to create the interfacing code for you. In comparison, direct C code interfaces have no formal description of the data types and calling interfaces at all, and no, a header file does not count as that by a very long stretch since it is missing a lot of information that is sometimes buried somewhere in a text documentation, but more often is simply the result of programmer intuition and trial and error with a good source code level debugger. These last three things are something LabVIEW is still several light years away of being able to do.

Of course there could be something like the LabVIEW Call Library Configuration dialog to allow to configure callback interfaces to VIs that can then be passed as function pointers to Call Library Nodes, but the according dialog would likely be even more complicated than the Call Library Node dialog and, considering how much difficulties most LabVIEW users have with the existing CLN already, as such very hard to use for more than 99% of the LabVIEW users.

So for all my research and thoughtwork, I'm left with the inelegant option #2 where I create a LabVIEW DLL (with all the incredibly carefully constructed callback function-VIs and their prototypes) to handle the DLL-to-LabVIEW interface. (Which is entirely probable by passing a few key "user values" - GOOP references, etc - from LabVIEW to DLL and back to LV-DLL.) And, yes, this method does assume that the DLL and the original Application are sharing the same process space. (so the GOOP references are indeed valid) If not, then I have the unenviable task of porting 3.5Mb (yes, three-and-a-half megabytes) of C code to LabVIEW.

Seems to me you got stuck between a wall and a hard place. Why even porting 3.5 MB C code to LabVIEW at all? From what I see you seem to need to actually decide in the callback whatever and return data from there to the caller of the callback so this means synchronous operation of the callback. In that case you are right that the only possible solution would be to wrap LabVIEW VIs into a DLL, making sure this DLL is created in exactly the same LabVIEW version as your calling LabVIEW application and then GetProcAddress() those DLL function pointers and pass them to the Call Library Node as callback pointers. As I already said this is going to be a rather messy maintenance nightmare and another problem is there too! The C wrapper created for calling the actual VIs in those DLLs has to process and convert all variable sized data from the C function pointer into LabVIEW handles and before returning again back to C pointers. This can be a lengthy and performance hungry process for the amount of data you seem to think would make a TCP/IP interface unfeasible.

Rolf Kalbermatter

[tkuiper]

First of all, thanks for the thorough and thoughtful response. I've learned a few things, but I also see that I didn't explain a few other things very well...

Why do you say this is inferior? It is not, because doing that is basically the only proper way to merge a C callback with the LabVIEW dataflow programming. There are several possibilities depending on the requirements you have.

1. Calling the PostLVUserEvent() function you can pass data back to LabVIEW, in fact any data you want but you need to create LabVIEW compatible native data for that (LabVIEW handles for strings and arrays). The event is then processed in the according user event in the event loop, so there is a possible problem with serialization of all the callbacks if processed in the same event structure.

2. You can use Occur() to trigger a LabVIEW occurrence. The advantage is that it is not serialized with other occurrences that you might trigger from other callbacks but the disadvantage is that you can not pass data back with the occurrence trigger.

3. To solve the problem with the occurrence not having any means to pass data with, you can do your own queue code in C that holds elements that the callback functions put in before they trigger the occurrence. When the occurrence is triggered in LabVIEW you can read the queue for new data.

So it is definitely not inferior to do that. The only problem with this is that you do need to write some (good and not very trivial) C code, but that is no reason to call this inferior.

I say that this method is inferior because - in this instance - the outside-world-to-LabVIEW-dataspace API is limited to either the established ActiveX API or some sort of reverse-ActiveX API (a whole series of LV-DLL functions that expose such functionality) and - well, I thought I said it later in the original post but I'll say it here - I will need to do quite a bit of iterating over the data returned and based upon that data, request more data, etc...

This is a highly OO application - the initial data query will be to retrieve an array of GOOP references. I'll then interrogate each reference (via member function calls) in turn searching for a particular reference (based upon which callback was called and the parameters passed to that callback). Once that particular reference is found, I'll need to interrogate it for yet more data, format a few pieces of data based on the arguments to that callback, use its member functions to distribute a few of these pieces of data while caching others locally in the DLL, etc...

So it's not the fact that I need to push data back into LabVIEW, (regardless of size) it's that I need to dialog with a whole series of objects based upon what's happening in those callbacks. The aforementioned methods would by great if I just had to shove 10Mb of samples into LabVIEW, but I need to dialog iteratively...

Performing that dialog in LabVIEW can be difficult (time consuming) as it is, however translating that simple interface into a whole slew of ActiveX API calls (in C) is - IMHO - far inferior. Maybe I'm just a LabVIEW chauvinist, but I happen to love the way LabVIEW interacts with itself. Jumping into a sea of C code (har har) to do the same thing reminds me of weeding a garden with my bare hands. (Some people find it rewarding.. but I'd rather use a shovel where a shovel would be useful! Broken fingernails, fire ants welcoming you into their home, what fun!)

However, this was the main option I was hoping that some brilliant people might have some keen insights into. I don't have a problem writing C - I have a problem writing code that I know is going to be clunky and inefficient just to solve a problem that shouldn't exist in the first place.

The main issue with this is not so much the fact that they need to access GOOP data or any LabVIEW data, but much more a maintenance nightmare. As soon as those callbacks DLLs are not compiled in exactly the same version as the LabVIEW version you use to call them, those DLLs will execute in the according LabVIEW runtime engine and as such are in fact almost out of process from the calling LabVIEW environment. This will indeed make the sharing of data between the DLL and the calling LabVIEW process impossible but there are also other problems why you rather don't want to do that.

This was one of my first concerns, but it's highly mitigated by the fact that the main application also makes heavy use of dynamic VI calls. This in itself requires an annoying amount of preparation, distribution recompilation, and coordination between the LabVIEW versions of the main application itself and the dynamically loaded VIs. So, shortly, that's already a problem we're taking into account and shouldn't be an issue. (Famous last words, muahaha!)

Actually LabVIEW executables can serve as Activex Automation server just as well as the LabVIEW development system. Check out the build settings of your project for the place where you enable ActiveX server and specify the name under which it will be registered. But the problem with ActiveX for doing things will be probably the same as for TCP/IP since they are both out of process technologies so if TCP/IP won't work, ActiveX likely will be even worse.

I had totally forgotten about this and it may actually be the solution I was searching for. The ActiveX interface will work because it will (it must!) be in-process. Even if not, some really nasty sort of solution could be made (maybe?) using dynamic VI load/control populate/execute/indicator fetch routine(s) using the 'open application instance' type of ActiveX interface. I've done this to allow one VI[EXE] to control another VI[source] across the network.

TCP would (with an extensively designed custom command/response interface) work, but the latencies involved and the time to develop/debug/etc that server make it impractical. Using the LabVIEW ActiveX interface is a a slightly better option, but it's still very inefficient in that pages of code would be required to do what a single subvi-call node accomplishes on a block diagram. This is what I'm leaning towards, but I yearn for that simplicity of dropping a subvi onto a block diagram and simply wiring up its connector pane...

Because ActiveX and .Net have a very strictly typed interface description where LabVIEW can get all the necessary information to create the interfacing code for you. In comparison, direct C code interfaces have no formal description of the data types and calling interfaces at all, and no, a header file does not count as that by a very long stretch since it is missing a lot of information that is sometimes buried somewhere in a text documentation, but more often is simply the result of programmer intuition and trial and error with a good source code level debugger. These last three things are something LabVIEW is still several light years away of being able to do.

Of course there could be something like the LabVIEW Call Library Configuration dialog to allow to configure callback interfaces to VIs that can then be passed as function pointers to Call Library Nodes, but the according dialog would likely be even more complicated than the Call Library Node dialog and, considering how much difficulties most LabVIEW users have with the existing CLN already, as such very hard to use for more than 99% of the LabVIEW users.

Indeed, it's that strictly-typed interface for all data types, classes, interfaces, and whatnot that makes interacting with ActiveX and .NET so simple. (The vast majority of all AX/.NET enums even support right-click-create-constant!) That's sort of why I'm decrying LabVIEW's lack of strict data typing/interface prototyping, even though it exports everything via ActiveX already!

Every single bit of information about a VI (well, almost... ok.. not quite, but enough!) is available via ActiveX calls on a VI reference - especially with internal scripting - however that information really should be exported a tiny step further into callbacks - IMO. (Feature request!)

And, yes, calling into direct C code (as a C DLL) will give you no information whatsoever about the function other than its ordinal, name, and procedure address, a C++ export does give you more information, but an ActiveX control or a .NET assembly gives you everything. Yes, header files are problematic at best. C/C++ libraries exist (free from Microsoft, no less) for interrogating an ActiveX control/server and .NET assembly/etc for all necessary information and calling into it. MIDL, ATL, and the like were invented to give developers easy access to such ActiveX interfaces in C++. Yes, you can call into .NET from C - it's quite nasty but it can be done. And we all know that you can call into ActiveX from C - that's where it originally came from!

So what's really necessary here may simply be either a custom library of functions, or ATL-compliant classes, for interactive with the LabVIEW ActiveX server or something similar from NI. This would allow C code to call back into a VI. After all, TestStand does exactly this!

[OT]

Btw, does anybody have any information on manually instantiating a given version of the LabVIEW runtime engine? (Outside of building a VI/EXE in that exact version of LV that instantiates the engine for you just because it's a NI-compiled binary!) I've seen several instances where it would be quite useful to be just like TestStand and find a VI on disc and do something appropriate with it - from a language/program not written by NI.

[/OT]

It might even look roughly analogous to their Embedded development system. They provide the upper layer interface, you fill in the "callbacks." (So you'd really be writing C/C++ code to interface with a VI on a one-to-one basis.. "Just how exactly would I stuff this callback's controls before it's called?" and "How to I extract the necessary information from the VI after its run, and then what do I do with that data afterward?") Hah, so this would essentially mean I'd write a callback function to call a callback VI, which LabVIEW would then call another callback function (or 2! input and output!) to establish the interface between their VI and my code!

Hmm... maybe not...

Another thing... LV2009 allows you to build .NET assemblies, but, again, this is the long/wrong way around.

Seems to me you got stuck between a wall and a hard place. Why even porting 3.5 MB C code to LabVIEW at all? From what I see you seem to need to actually decide in the callback whatever and return data from there to the caller of the callback so this means synchronous operation of the callback. In that case you are right that the only possible solution would be to wrap LabVIEW VIs into a DLL, making sure this DLL is created in exactly the same LabVIEW version as your calling LabVIEW application and then GetProcAddress() those DLL function pointers and pass them to the Call Library Node as callback pointers. As I already said this is going to be a rather messy maintenance nightmare and another problem is there too! The C wrapper created for calling the actual VIs in those DLLs has to process and convert all variable sized data from the C function pointer into LabVIEW handles and before returning again back to C pointers. This can be a lengthy and performance hungry process for the amount of data you seem to think would make a TCP/IP interface unfeasible.

Porting said 3.5Mb of C would allow me to use the code in the DLL in native form - in pure G. If the code existed in G instead of C, then I wouldn't need to ask any questions, I'd just plop a few SubVI calls and be done... However...

Again, it's not the quantity of data, it's the iterative interactions that is the gating requirement.

The LabVIEW ActiveX server calls would perform the data translation for me. (I have to assume that NI has supremely optimized these routines.) I had planned on minimizing the number of LabVIEW-to-DLL calls, 5 or 6 at the most for this interface. Those entry points should provide most, if not all, of the LabVIEW-to-DLL functionality required. It's the other way around that's bothering me. At first glance, I'd either need a very abstracted programmable interface (think printf on steroids) or a function/VI to handle every single permutation of raw data-to-ASCII conversion. (Just an example.. I wish it were as simple as string formatting!)

Tony

[Rolf Kalbermatter]

Why not turn the whole idea around? You do want callback support and ActiveX (long ago) and .Net (since version 8) interfaces in LabVIEW support it out of the box. So the real solution would be to write a thin ActiveX or .Net wrapper around your DLL code that translates the DLL callback in an according ActiveX or .Net event. Then your DLL invokes the callback funciton which in turn sends the ActiveX/.Net event to LabVIEW and LabVIEW invokes the according callback VI which then returns whatever data is required to the ActiveX/.Net event and that returns control to the callback.

Yes it is not trivial and it would not work for tight low level kernel type callback drivers, but nothing will directly work with such callback drivers from a high level environment like LabVIEW. You could make it sort of work from a simple C program but certainly not a .Net application or such either.

If you need to pass large amounts of data from the callback to LabVIEW or vice versa you would have to opt for an in-process ActiveX or .Net wrapper otherwise you can go with an out of process wrapper too. ActiveX/.Net will take care of marshaling the data for out of process servers between said server and LabVIEW (and .Net may do marshaling no matter what). Marshaling is ok for small amounts of data but if you plan to move large amounts of data it is going to create an additional bottleneck.

Rolf Kalbermatter

[tkuiper]

Why not turn the whole idea around? You do want callback support and ActiveX (long ago) and .Net (since version 8) interfaces in LabVIEW support it out of the box. So the real solution would be to write a thin ActiveX or .Net wrapper around your DLL code that translates the DLL callback in an according ActiveX or .Net event. Then your DLL invokes the callback funciton which in turn sends the ActiveX/.Net event to LabVIEW and LabVIEW invokes the according callback VI which then returns whatever data is required to the ActiveX/.Net event and that returns control to the callback.

What an interesting idea...

It would have to be an ActiveX wrapper, as I'm not incredibly interested in porting that 3.5Mb of C code to C#, entertaining tho that may be.

This solution would nicely provide hooks for LabVIEW code to handle the events/callbacks. The main concern I'd have is with returning the hard-won data back into the DLL - err ActiveX control.

Eww.

I'm picturing some very nasty interactions that would go something like this: (In this description "control" refers to the inproc ActiveX server, for lack of simpler terms, incorrect tho it may be.)

1. Main application decides it needs something from the control
2. LabVIEW calls into the control asking for that data
3. The control decides that it in turn needs more information from LabVIEW
4. The control fires an event asking for that data
5. LabVIEW intercepts that event and executes the prior registered callback VI
6. (warning: fuzziness ensues) The callback VI executes and... somehow... supplies that information to the control
7. The control is happy and finishes doing what it needs to do, and
8. The control then fires off a series of events notifying LabVIEW of the status of the apparatus.

Now, the main things I'm not sure of, and I may just have to sit down for a day or 10 (preferably in binary) and test this out:

1. Are ActiveX Events Synchronous or Asynchronous? (i.e. do I have to manage the synchronization of stimulus[event] and response[callback completion]? Do I have to watch out for/manage deadlocks between LabVIEW, which is already steeped highly in ActiveX under the hood, and its hooks into other ActiveX servers?)
2. I'm going to have to supply some way for the callback VI to provide the requested data to the control - probably in the form of control interface function calls since an event is typically (from what I've seen) uni-directional - which would tend to generate deadlock possibilities, especially if the Event is handled internally as Synchronous... Threads, semaphores, IPC, etc...

I'm pretty sure that there should only be a few (information gathering) events [ideally none or 1, but possibly several] spawned in response to any control commands (from LabVIEW). The control would have to be implemented in such a way that after it fires an event, it's looking for a function (maybe one in particular, maybe all of a set?) be called it completes The Transaction (of which there may be several "queued up" in the ActiveX event buffer - or more likely just blasted out willy-nilly) and allows it to perform the final processing, probably via a series of events which will also need to be handled by LabVIEW.

This is sounding similarly kludgy, but may end up being the shortest path to goal. This all depends on being able to easily/reliably pass information back in response to an Event...

Any thoughts?

Thanks again,

Tony

[Rolf Kalbermatter]

Any thoughts?

Thanks again,

Tony

Not really . I think you managed to come up with a few possible obstacles and problems that if they surface, might kill this idea too. But I have no other good alternative at the moment for you.

Rolf Kalbermatter

[tkuiper]

Well, after much time spent being on hold pending further delay, success is close indeed!

Your original suggestion of the ActiveX server was the ticket. Here's a quick overview:

The EXE exports an ActiveX server (as specified in the build settings) and knows whether or not it's an application or in the LabVIEW IDE.

The main initiation DLL call [not called using the UI thread!!!] passes a string to the ActiveX server name (e.g. IsEXE ? "MyServer" : "LabVIEW") which the DLL uses in the call to pLVApp.CreateInstance() (pLVApp is an instance of _ApplicationPtr, since the .tlb was #imported using no_namespace...)

Here's where things get slightly interesting..

I created a helper function (CallVI) that calls pLVApp->GetVIReference, sets controls specified in an stl::map<char*,_variant_t>, Runs that VI, then retrieves all indicators into that same stl::map, cleaning up after it's finished.

This function in turn is wrapped in a framework of macros managing said stl::map that were dynamically generated by a LabVIEW utility that, via scripting, parses the GOOP objects and generates appropriate C 'wrapper' code for each public VI into the appropriate cpp/h file. (Foo object is built into foo.cpp/foo.h, with functions for Foo_DoThis and Foo_GetBar/Foo_SetBar)

The gotchas:

1. All DLL calls to this library must not be in the UI thread (or it'll lock as soon as it tries to access the ActiveX server)
2. #import the .tlb (whether labview.tlb or myapplication.tlb) using no_namespace - note that this requires you to close the VS solution before re-compiling the LV Application!
3. Use the correct ActiveX server! (If you're in the IDE, create in instance of "LabVIEW.Application", if you're built into MyApplication.EXE, then create an instance of "MyApplication.Application" or whatever you configured the build script to export for the ActiveX server name.)
4. Make all VIs that will be called by ActiveX reentrant! (Otherwise the GetVIReference() or Run() will fail with some anonymous IDispatch error)

That about sums it up.

Heh, just thinking about the name of this particular forum.. "Calling External Code" This is more about "Calling Internal Code From External Code."

Thanks again!