clfn scripting

[smithd]

just curious if anyone has ever made an attempt to improve scripting out of dlls. I know theres the built in import wizard, but I've literally never seen that work (has anyone here?).

 

The reason i'm asking is that I was thinking about it and came across this tool: https://github.com/CastXML/CastXML (binaries from the build link at the bottom here: https://github.com/CastXML/CastXMLSuperbuild)

Essentially it takes a C header (or any C file I guess) and processes it with LLVM to produce an XML tree of all the functions and types. For example it takes the header here: https://github.com/nanomsg/nng/blob/master/include/nng/nng.h

and produces xml which includes stuff like this:

  <Typedef id="_8" name="ptrdiff_t" type="_285" context="_1" location="f1:51" file="f1" line="51"/>
  <Typedef id="_9" name="wchar_t" type="_286" context="_1" location="f1:90" file="f1" line="90"/>
  <Typedef id="_14" name="intptr_t" type="_285" context="_1" location="f4:182" file="f4" line="182"/>
  <Union id="_78" name="nng_sockaddr" context="_1" location="f6:151" file="f6" line="151" members="_328 _329 _330 _331 _332 _333" size="1088" align="64"/>
  <Struct id="_61" name="nng_aio" context="_1" location="f6:96" file="f6" line="96" incomplete="1"/>
  <Struct id="_68" name="nng_sockaddr_in6" context="_1" location="f6:124" file="f6" line="124" members="_315 _316 _317" size="160" align="16"/>
  <Field id="_315" name="sa_family" type="_25" context="_68" access="public" location="f6:125" file="f6" line="125" offset="0"/>
  <Field id="_316" name="sa_port" type="_25" context="_68" access="public" location="f6:126" file="f6" line="126" offset="16"/>
  <Field id="_317" name="sa_addr" type="_385" context="_68" access="public" location="f6:127" file="f6" line="127" offset="32"/>
  <Typedef id="_79" name="nng_sockaddr" type="_334" context="_1" location="f6:158" file="f6" line="158"/>
  <Enumeration id="_80" name="nng_sockaddr_family" context="_1" location="f6:160" file="f6" line="160" size="32" align="32">
    <EnumValue name="NNG_AF_UNSPEC" init="0"/>
    <EnumValue name="NNG_AF_INPROC" init="1"/>
    <EnumValue name="NNG_AF_IPC" init="2"/>
    <EnumValue name="NNG_AF_INET" init="3"/>
    <EnumValue name="NNG_AF_INET6" init="4"/>
    <EnumValue name="NNG_AF_ZT" init="5"/>
  </Enumeration>
  <Function id="_84" name="nng_close" returns="_293" context="_1" location="f6:194" file="f6" line="194" mangled="?nng_close@@9" attributes="dllimport">
    <Argument type="_55" location="f6:194" file="f6" line="194"/>
  </Function>
  <Function id="_87" name="nng_setopt" returns="_293" context="_1" location="f6:205" file="f6" line="205" mangled="?nng_setopt@@9" attributes="dllimport">
    <Argument type="_55" location="f6:205" file="f6" line="205"/>
    <Argument type="_338" location="f6:205" file="f6" line="205"/>
    <Argument type="_339" location="f6:205" file="f6" line="205"/>
    <Argument type="_5" location="f6:205" file="f6" line="205"/>
  </Function>
  <FundamentalType id="_293" name="int" size="32" align="32"/>
  <FundamentalType id="_294" name="unsigned char" size="8" align="8"/>
  <FundamentalType id="_295" name="unsigned int" size="32" align="32"/>
  <PointerType id="_353" type="_352" size="64" align="64"/>
  <PointerType id="_354" type="_62" size="64" align="64"/>
  <PointerType id="_355" type="_47" size="64" align="64"/>

Its pretty easy to parse because everything has an ID -- I've got some truly gross code for converting all the struct and enum definitions into labview which is simple enough.

I guess I'm just testing the waters to see if anyone else thinks this is useful. My usual strategy right now is to minimize the number of C calls I make, but if I could just magically import a header and get the library made for me, it would be pretty cool. But on the other hand, theres so many challenges associated with that which I hadn't thought about going in, like handling structs by value, or what to do with numeric return values (is it an error code? is 0 good or bad?). The particular library I selected as a test case above has two features that are super annoying -- its main 'session' reference is not a pointer like in most C apis, but a struct containing a single int, and it also has a union type..for some reason. So even making this library work, which I thought would be an easy way to try it out, has ballooned in complexity a bit. Thoughts?

[JKSH]

On 2/2/2019 at 9:18 AM, smithd said:

I guess I'm just testing the waters to see if anyone else thinks this is useful. My usual strategy right now is to minimize the number of C calls I make, but if I could just magically import a header and get the library made for me, it would be pretty cool. But on the other hand, theres so many challenges associated with that which I hadn't thought about going in, like handling structs by value, or what to do with numeric return values (is it an error code? is 0 good or bad?). The particular library I selected as a test case above has two features that are super annoying -- its main 'session' reference is not a pointer like in most C apis, but a struct containing a single int, and it also has a union type..for some reason. So even making this library work, which I thought would be an easy way to try it out, has ballooned in complexity a bit. Thoughts?

LLVM/Clang is truly a marvel! If you need to write wrapper VIs for DLLs, then I think a tool like this is the way to go.

However, unless the C API has anything non-trivial like passing a large struct by-value or strings/arrays, you'll end up having to write your own wrapper DLL, right?

(side note: I do wish LabVIEW would let us pass Booleans straight into a CLFN by-value...)

 

On 2/2/2019 at 9:18 AM, smithd said:

The particular library I selected as a test case above has two features that are super annoying -- its main 'session' reference is not a pointer like in most C apis, but a struct containing a single int, and it also has a union type..for some reason. So even making this library work, which I thought would be an easy way to try it out, has ballooned in complexity a bit.

• This struct has the same size as a single int, so you can lie to the CLFN: Say that the parameter type is a plain I32.

• What are the possible types in the union? If they're quite simple and small, you could tell the CLFN to treat it as a fixed-size cluster of U8s (where the number of cluster elements equals the number of bytes of the union), then Type Cast the value to the right type. The type cast will treat your cluster as a byte array).

These would require project-specific scripts that don't fit in a generic automatic tool though.

 

On 2/2/2019 at 9:18 AM, smithd said:

just curious if anyone has ever made an attempt to improve scripting out of dlls. I know theres the built in import wizard, but I've literally never seen that work (has anyone here?).

The import wizard was useful for study purposes, to figure out how I should create my wrapper (using a simple sample DLL). That was about it.

[ensegre]

From time to time, albeit sporadically, I have to build wrapper sets to .dll and .so, and I would love any improvement. Back in the days I was used to have to write my wrapper VIs one by one, and by now, especially for large libraries, the import wizard is a lifesaver to me. However, I end up still spending some amount of time, often significant, domesticating the .h files provided with the libraries so that the wizard sees better through them, writing LV translators between C structures and casted down byte arrays, creating ad hoc enum ring typedefs, hunting untranslated pointers, and similar chores. If that could be automated, I would enjoy it, though I agree that beyond some point that becomes project-specific. And at times (callbacks, message pumps i.e.) still no choice but having to write C containers.

Have btw the import tool parsing capabilities got any better across versions? (I vaguely think yes but I don't have fair data for a comparison)

[viSci]

I use the dll wizard often and it has saved me a lot of time.  It took awhile to understand how to massage the .h files into something digestible but once I got the hang of it, it was worth the effort. 

The error handling is pretty good at pointing you to the offending spot in the .h file.  I think that the wizard has improved over time so if your last exposure was years ago I would give it another try.

Edited February 4, 2019 by viSci

[smithd]

13 hours ago, viSci said:

The error handling is pretty good at pointing you to the offending spot in the .h file.  I think that the wizard has improved over time so if your last exposure was years ago I would give it another try.

Fair enough, its definitely been a while. Last I remember I had to have the whole visual C sdk on my machine because it wanted to find some standard header.

On 2/3/2019 at 10:11 AM, ensegre said:

I end up still spending some amount of time, often significant, domesticating the .h files provided with the libraries

To me this is one of the bigger annoyances -- I dont' want to fix up the h file, that seems iffy to me. I suppose it generally doesn't matter since its just use to help the script along, but...

On 2/3/2019 at 8:30 AM, JKSH said:

However, unless the C API has anything non-trivial like passing a large struct by-value or strings/arrays, you'll end up having to write your own wrapper DLL, right

Usually strings and arrays are OK, but yeah big structs can be iffy due to offsets -- that is once nice thing about this LLVM-based tool, is it tells you all the byte offsets so technically one could pass in an appropriately sized buffer and restructure the data...but thats a pain as well.

[Rolf Kalbermatter]

On 2/5/2019 at 3:12 AM, smithd said:

Fair enough, its definitely been a while. Last I remember I had to have the whole visual C sdk on my machine because it wanted to find some standard header.

To me this is one of the bigger annoyances -- I dont' want to fix up the h file, that seems iffy to me. I suppose it generally doesn't matter since its just use to help the script along, but...

Well to be honest LLVM won't be able to process a header file that references other header files with declarations that are used in your header without access to them. It needs access to those referenced headers somehow too and I doubt it comes preinstalled with all Mac OS X, Windows SDK, etc. etc. headers so you end up installing them somehow to your disk too and pointing LLVM to them.

Quote

Usually strings and arrays are OK, but yeah big structs can be iffy due to offsets -- that is once nice thing about this LLVM-based tool, is it tells you all the byte offsets so technically one could pass in an appropriately sized buffer and restructure the data...but thats a pain as well.

The import library wizard has internally this informations too, but once you start going down the path of creating the wrappers with such advanced functionality through scripting you end up in hell. There is simply no way such a "Wizard" can really know how the different information in such structures may or may not interact with each other and with other things in the API so you end up with a overly complicated interface that is very difficult to understand and allows you to still do lots of illegal things. The purpose of a proper LabVIEW DLL wrapper is to simplify the API as much as possible for the user without saddling him with subtleties like if you want this function to return 1000 samples you have to set this numeric to 8000 AND also Initialize an array with 1000 double values in order for the call to not crash. Or to return a string from the function to require the user to even have to specify the size of the buffer as the function documentation specifies that the function requires a buffer of at least 256 characters to fill in. 

Even such a simple thing as an API where you pass in a buffer that the function should fill in with information and a second parameter that tells the function how big the buffer really is, is already a total nogo for even the most advanced wizard, since there is no way the C syntax can describe the specific dependency between these two parameters. So the wizard will create two controls on the front panel and the uninformed user will wire a 1000 to the buffer size but leave the string control that should now contain 1000 bytes at its default of an empty string and -> boom!

We humans after quite some dealings with these kinds of APIs often can infer the correspondence between these two parameters because of a more or less useful naming correlation between the two but even that can often go wrong (is the number in bytes, characters or numeric elements of the array type?). The only real information comes from the documentation, which surprisingly often fails to document these things accurately too, and then the only thing that remains are hopefully sample source code that shows you how these parameters are supposed to be assigned.

 

Edited April 2, 2019 by Rolf Kalbermatter