Pass complex data structure to C function

[Łukasz]

I need help passing a complex data structure — a map of clusters containing an enum type and a string value — to a C function.

My initial solution was to convert this structure to a string and then to byte buffer, pass the buffer and its lenght, and then decode it on the C side. However, this approach has issues when the key or value contains characters that I used as separators. While I could improve my encoding method, I'm looking for alternative solutions to achieve my goal.

Function prototype:

int32_t lv_amqp_basic_publish(int64_t *conn_int, uint16_t channel, CStr exchange, CStr routingkey, uint8_t *headersBuffer, int32_t headersBufferLen, LStrHandle messagebody, LStrHandle error_description);

headersBuffer is used to populate headers table (array of amqp_table_entry_t_):

typedef struct amqp_table_entry_t_ {
  amqp_bytes_t key; /**< the table entry key. Its a null-terminated UTF-8
                     * string, with a maximum size of 128 bytes */
  amqp_field_value_t value; /**< the table entry values */
} amqp_table_entry_t;

LabVIEW code to encode headers map to string:

• The blue arrow represents the encoding signature for each element
• The green arrow indicates an issues with this method and ugly workaround

I can modify both the LabVIEW and C code.  Any suggestions would be greatly appreciated.

[Rolf Kalbermatter]

Well this is one hell of an API to tackle.

amqp_bytes_t seems to be actually a struct similar to a LabVIEW handle contents, with a size_t element indicating how many bytes the following pointer points at. That in itself is already nasty if you want to support both 32-bit and 64-bit LabVIEW, since size_t is a pointer sized unsigned integer and the pointer after is of course pointer sized too!

Then you have the amqp_field_value_t which in principle is a library specific Variant. Basically you want to have an element that consist of a binary string based key value and a variant, except that the Variant manager API in LabVIEW while present is basically undocumented. Well it's not totally undocumented since the NI developers let slip through a header file in the GPU Toolkit download that actually declares quite some of the actual functions. Of course there is the problem that function declarations are hardly any real documentation. It only gives the function signature but doesn't explain anything about how the functions would need to be used. So there is in fact a lot of trial and error here and the realistically present danger, that the Variant datatype and its related functions are subject to change at the simple whim of any LabVIEW developer since the fact that it is not officially documented makes the API "subject to change" at any time, for any reason including simply the desire to change it. The only reason not to do so is that existing NI libraries such as the OPC UA Toolkit, which makes internally use of that API, would also need to be reviewed and changed in order to not crash with a new LabVIEW version. Since NI has a bit of a habit to release LabVIEW version synchronized Toolkits (albeit sometimes with a year long delay or an entire version even missing) this is however not an impossible limitation as it not only would limit the documented version compatibility but also be a technical limitation to help prevent version incompatible Toolkit installations.

Even if you would use LabVIEW variants as value of the key value pair, you would need to do some binary translation since a LabVIEW variant is not the same as your amqp_field_value_t variant.

Personally I would likely use a cluster with a LabVIEW string as key element and a flattened string of the binary data with extra integer for datatype indication. Then in the C code do a translation of these elements to the amqp_bytes_t and amqp_field_value_t data elements. If you allow for a simple 1 to 1 mapping of the field_value element, things could be fairly straightforward.

Something like this:

struct
{
    LStrHandle key;
    LStrHandle value; // really the flattened binary data
    int32 datatype;   // native LabVIEW datatype, could get rather nasty if you want to support complex
                      // datatypes and not just scalars and a string as you would need to allow for a
                      // hierarchical datatype description such as the i16 typedef array of LabVIEW itself
} KeyValueRec;

If you use a native LabVIEW Variant it would instead look like:

struct
{
    LStrHandle key;
    LvVariantPtr value; // Native LabVIEW variant
} KeyValueRec;

But as mentioned the API to actually access LvVariant from C code is completely undocumented.

Edited May 28 by Rolf Kalbermatter

[Łukasz]

Thank you for your reply, 
passing a struct seemed very intuitive, and I considered a similar approach. However, I couldn't find any examples of how to pass an array of structs, which is the main blocker for this idea. Have you seen this approach used anywhere?

[Rolf Kalbermatter]

5 hours ago, Łukasz said:

Thank you for your reply, 
passing a struct seemed very intuitive, and I considered a similar approach. However, I couldn't find any examples of how to pass an array of structs, which is the main blocker for this idea. Have you seen this approach used anywhere?

Pretty simple except if you need to resize the array in the C code.

You can let LabVIEW create the necessary code for the function prototype and any datatypes. Create a VI with a Call Library Node, create all the parameters you want and configure their types. For parameters where you want to have LabVIEW datatypes passed to the C code, choose Adapt to Type. Then right click on the Call Library Node and select "Create C code". Select where to save the resulting file and voila.

This would then look something like this:

/* Call Library source file */

#include "extcode.h"

#include "lv_prolog.h"
/* Typedefs */
typedef struct {
	LStrHandle key;
	int32_t dataType;
	LStrHandle value;
	} TD2;

typedef struct {
	int32_t dimSize;
	TD2 Cluster elt[1];
	} TD1;
typedef TD1 **TD1Hdl;
#include "lv_epilog.h"

void ReadData(uintptr_t connection, TD1Hdl data);

void ReadData(uintptr_t connection, TD1Hdl data)
{

	/* Insert code here */

}

Personally I do not like the generic datatype names and I always rename them in a way like this:

/* Call Library source file */

#include "extcode.h"

#include "lv_prolog.h"
/* Typedefs */
typedef struct {
	LStrHandle key;
	int32_t dataType;
	LStrHandle value;
} KeyValuePairRec;

typedef struct {
	int32_t dimSize;
	KeyValuePairRec elt[1];
} KeyValuePairArr;
typedef KeyValuePairArr **KeyValuePairArrHdl;
#include "lv_epilog.h"

void ReadData(uintptr_t connection, KeyValuePairArrHdl data);

void ReadData(uintptr_t connection, KeyValuePairArrHdl data)
{
    int32_t i = 0;
    KeyValuePairRec *p = (*data)->elt;
  
    for (; i < (*data)->dimSize; i++, p++)
    {
         p->key;
         p->dataType;
         p->value;
    }
}

 

Edited May 28 by Rolf Kalbermatter

[Neil Pate]

13 hours ago, Rolf Kalbermatter said:

For parameters where you want to have LabVIEW datatypes passed to the C code, choose Adapt to Type. Then right click on the Call Library Node and select "Create C code". Select where to save the resulting file and voila.

 

Nice! I did not know about this.

[Łukasz]

Thank you @Rolf Kalbermatter.

Your suggestion works as expected, and the code is much simpler than in the previous version.

As I understand it, performing the opposite operation would not be easy since we can't resize the array. I wonder if there is any other function to allocate memory on the C side using LabVIEW, or if `NumericArrayResize` from extcode.h is the only option?

[Rolf Kalbermatter]

On 6/5/2024 at 9:09 AM, Łukasz said:

Thank you @Rolf Kalbermatter.

Your suggestion works as expected, and the code is much simpler than in the previous version.

As I understand it, performing the opposite operation would not be easy since we can't resize the array. I wonder if there is any other function to allocate memory on the C side using LabVIEW, or if `NumericArrayResize` from extcode.h is the only option?

We could allocate/resize the array but it is highly complicated. There are two basic possibilities:

1) Using NumericArrayResize() is possible but you need to calculate the byte size yourself. With complex datatypes (clusters) the actual byte size can depend on the bitness of your compilation and contain extra alignment (filler) bytes for non Windows 32-bit compilation. Really gets complicated, but the advantage is that it is at least documented.

2) There is an undocumented SetArraySize() function. It can work for arbitrary array elements including clusters and accounts for the platform specific alignment but is tricky since the datatype description for the array element is a LabVIEW type-descriptor. To get that right is pretty much as complicated as trying to calculate the array element size yourself and as it is undocumented you risk that something might suddenly change.

The declaration for that function is:

TH_REENTRANT MgErr _FUNCC SetArraySize(int16 **tdp, int32 off, int32 dims, UHandle *p, int32 size);

tdp is the 16-bit LabVIEW type descriptor for the array element data type. This is basically the same thing that you get from Flatten Variant but you want to normally make sure that it does not contain any element labels as it does not need them and only makes the parsing slower.

off is usually 0 as it allows to specify an offset into a more complex tdp array.

The other parameters are exactly the same as for NumericArrayResize(). In fact NumericArrayResize() is a thin wrapper around this function that uses predefined tdp's depending on the first parameter of it.

Edited June 7 by Rolf Kalbermatter