[CR] JSON LabVIEW

[Stinus Olsen]

Hi guys

 

I just now stumbled upon a (not so) funny mishab/error that i wanted to highlight for further investigation.

I relates to the JSON VIs use of the "Array of VData to VArray__ogtk.vi" to convert an array of independent (but equally typed) variants into a single array (as a variant)..

 

The reason I use the word 'mishab' is because I have a hard time figuring out if this actually classifies as a 'real' error and not just me using the JSON VIs in a wrong way

 

Take a look at the below image..

 

The text lister shows a snippet of the JSON text I'm trying to parse - an array of type Plugin Subscription containing another array of Subscription Elements.

 

The whole JSON text has been written by the JSON VIs themselves!

 

The "Array of VData to VArray__ogtk.vi" function fails because the first parsed element in the JSON text array 'Subscriptions' are used as the type descriptor for the resulting array.

This first index, however, does not have the necessary type information needed to contain the second element of Subscriptions, and thus fails in the conversion..

 

The "Array of VData to VArray__ogtk.vi" function is called as part of the recursive "JSON to Variant" VI, but at the moment I'm still narrowing down on the specific use case when this happens.

 

Updates will follow..

Edited March 6, 2014 by Stinus Olsen

[drjdpowell]

If you attach a small project showing the problem, I can take a look at it.

[Stinus Olsen]

If you attach a small project showing the problem, I can take a look at it.

 

I haven't had time to create a project, but I managed to whip up a small example showing off the unexpected behavior.

 

JSON Test.zip

 

I spend most of the day debugging this issue - and in the end I solved the problem by changing a value in my config data cluster from a type of U16 to U32, which solved the problem instantaneously..

 

Those were some pretty expensive 16 bits 

 

I managed to track down the reason as to why this happens, but still don't know if I know all the cases in which it can happen..

 

 

Edit: I haven't included any of the JSON VIs - so you're on your own doing all the relinking necessary to run the test..

Edited March 6, 2014 by Stinus Olsen

[drjdpowell]

Could you reattach with the control values saved as default?

[Stinus Olsen]

Bugger...now I remember what i forgot

 

I dont have access to the code at the moment, but you can pretty easy set it up:

 

Each of the 5 arrays should have 3 active elements..

 

Each element is a cluster of:

 - A string value: "StringValue"

 - An array of clusters containing only one element - A numeric value: "NumericValue"

 

In the StringValue field, for the 3 indexes in each of the 5 arrays, just type some bogus value - Its really not important for the test..

 

The NumericValue field in the cluster arrays on the other hand are..!

 

In the following you will only need to enter values in the only index that are visible (0) - the value you enter are not important, only if the array itself are empty or not..

 

Array 1: NumericValues are all empty...

Array 2: Idx 1: Empty - Idx 2: Empty - Idx 3: some value

Array 3: Empty - some value - empty

Array 4: Empty - some value - some value

Array 5: some value - Empty - some value

 

Let me know if you got it, - if not I'll attach a new file tomorrow with the standard values set

[drjdpowell]

I have a fix and will post a VIPM package in a moment.  The problem results from converting 8 and 16-bit types to 32-bit, and relying on the OpenG functions to down convert.  This works, but when we have empty arrays, they are left as the default type, leading to arrays of variants that are mixed in bit length, which the OpenG function can’t handle.  So I added explicate type-casts for the 8 and 16-bit integers.

...

Here it is

lava_lib_json_api-1.1.3.21.vip

 

I have not fully tested, so this may not be the only issue; could you run your tests please?

 

— James

[Ton Plomp]

Here's a trimmed down version of the error (http://i.imgur.com/PY0Masp.png).

 

The fix by james (1.1.3.21) fixes this issue

 

However it's similar for Complex Singles and Complex Doubles. This is fixed in commit d966584f2b63 (not build yet).

 

James: could you add Tag's after building the project?

 

Last issue is with FXP's.

I don't understand them well enough to have a good idea how to represent them in JSON...

 

Ton

 

Apparently if the first instance of an array is empty, something goes wrong. I haven't been able to pinpoint it. However the variant we return from 'From JSON' has the correct number of elements (see Variant out).

 

 

I can understand a little bit how this happens.

Finding a fix might be harder.

 

 

Ton

[Stinus Olsen]

Yes the fix works as it should!

 

I use the JSON VIs as one of multiple backend providers for a general configuration module in my applications.

For that purpose I've had to add a couple of additions to the toolset - and I don't see why those could not be a part of the standard functions, so here are a changeset:

JSON Change Set.zip

 

Maybe one of you guys could take 5 minutes to import them to the bitbucket repo?!  (instructions are included )

 

The added functions are:

 - Setting (possibly overwriting) a hierarchy element, by index of an array of strings

 - Removing (deleting) an element, also by using an array of strings as index

 - Setting a value of path type by internally casting it to a string (my own way of circumventing the path-type-unsupported problem)

 

And thanks a ton for some quick work yesterday on the TD fix

Edited March 7, 2014 by Stinus Olsen

[PBD_ctrl]

I have a fix and will post a VIPM package in a moment.  The problem results from converting 8 and 16-bit types to 32-bit, and relying on the OpenG functions to down convert.  This works, but when we have empty arrays, they are left as the default type, leading to arrays of variants that are mixed in bit length, which the OpenG function can’t handle.  So I added explicate type-casts for the 8 and 16-bit integers.

...

Here it is

lava_lib_json_api-1.1.3.21.vip

 

I have not fully tested, so this may not be the only issue; could you run your tests please?

 

— James

Did my code get added into this or did my commit fail?

[drjdpowell]

Did my code get added into this or did my commit fail?

 

Looks like it:

 

[PBD_ctrl]

Looks like it:

 

JSON commits.png

I figured out what I was doing wrong.

 

Here is a list of what I changed...(there are comments in the code and the commit as well)

 

-Added a case to handle refnums in JSON to Variant and Variant to JSON.  I decided to convert the refnums to either I32 or string depending on thier type and then include the class of the refnum proceeded by a "=>".  Including the class is valuable from my viewpoint in that it makes the JSON more readable.  When converting from a JSON sting back to a variant I use the "Variant (Data Type)" input to define the class and ignote the class name in JSON string.

 

-Added a case to JSON to Variant to handle paths explicitly.  This fixes the empty path bug.

 

-Removed OpenG Format Variant Into String from the default case and replaced it with Flattened String to Variant function that mirrors the default case in String to JSON.

 

 

Take a good look at the refnum handling I am guessing that it will be the thing that you guys may not entirely agree on.

[ShaunR]

 I decided to convert the refnums to either I32 or string depending on thier type and then include the class of the refnum proceeded by a "=>".  Including the class is valuable from my viewpoint in that it makes the JSON more readable.  When converting from a JSON sting back to a variant I use the "Variant (Data Type)" input to define the class and ignote the class name in JSON string.

 

Refnums are numerical types in all languages (even labVIEW). However, Whilst it may be "convenient",  I have reservations about injecting labview specific formatting into a language agnostic serialisation.What if a string contains "=>" ?

Edited March 7, 2014 by ShaunR

[PBD_ctrl]

I see where your coming from and that was the way that I originally coded it up.  Maybe it was too easy to do, but it felt useless without some sort of information about the class.

 

My feeling was that the refnums handled by this API are only going to be valid within a LabVIEW application so it wouldn't hurt anything to throw in a class identifier.  I searched around to see if I could find how other JSON APIs handle this and came up short.  Do you know of any other JSON APIs not for LabVIEW that handle refnums?

 

As for strings containing "=>" I could change it so that we search from the end of the string to get around this problem.

 

I should also mention that I don't care that much.  If it gets a new package built faster with the empty path fix to remove the "=>" and class I support it.

[ShaunR]

Do you know of any other JSON APIs not for LabVIEW that handle refnums?

They don't need to as the equivalent of refnums in other languages are pointers (generally) and therefore integers. It is only because LabVIEW is strictly typed that they get their own type.

[drjdpowell]

Sticking with the JSON spec, it might be better to encode LabVIEW refnums as JSON Objects:

 

{

  "DAQmx Refnum": {ref:”SampleTask”, LVtype:”UserDefinedRefnumTag"},

  "File Refnum": {ref:-16777216, LVtype:”ByteStream"},

}

 

On converting back, we should throw and error on any type mismatch.

 

This is rather verbose, but encoding refnums should be a rare use case, limited to “tokens” that are passed back to the original sending application where the refunms are valid.

[ShaunR]

Sticking with the JSON spec, it might be better to encode LabVIEW refnums as JSON Objects:

 

{

  "DAQmx Refnum": {ref:”SampleTask”, LVtype:”UserDefinedRefnumTag"},

  "File Refnum": {ref:-16777216, LVtype:”ByteStream"},

}

 

On converting back, we should throw and error on any type mismatch.

 

This is rather verbose, but encoding refnums should be a rare use case, limited to “tokens” that are passed back to the original sending application where the refunms are valid.

 

You know that "refnum" is short for "reference number", right?  

A file refnum is just a number. In fact. It is the file reference returned by the OS. Same with, for example, TCPIP. It just strikes me that this is a bit like adding the TypeDef information into the JSON stream  (e.g Control. TypeDef, StrictTypeDef) which is interesting, but not useful.

 

 

Let's assume that having specific LabVIEW reference types defined as objects is desirable. What does the LVType:ByteStream give you that the cluster wired as the "template" doesn't? How will you be able to tell that a DAQ Task ID is a UserDefinedRefnumTag  rather than a physical channel? What will the cluster template for the variant decoding look like?

 

DAQ Task IDs are not refnums. In fact,they are the same for all IO (like VISA) - a typed string like "path". If you plonk a DAQ Property Node onto a diagram you can quite happily connect a string to the "ref" input  (which stands for reference, not refnum). You'll see a conversion dot as it gets converted to the Task ID type-same goes for VISA references.

 

Are these "pre-thoughts" to object serialisation? Here, There Be Monsters. 

[drjdpowell]

The distinction between reference numbers and reference strings is unimportant; they are references.  I’m not sure if it is valuable to include reference type information in the JSON serialization, but if we do, it would be best to do this a JSON Object, rather than special string formatting.

 

 

Let's assume that having specific LabVIEW reference types defined as objects is desirable. What does the LVType:ByteStream give you that the cluster wired as the "template" doesn't?

Nothing in that case, other than type-checking and some human readability of the JSON.  

 

What will the cluster template for the variant decoding look like?

Same as the original encoding one.  Why wouldn’t it be?

[ShaunR]

Same as the original encoding one.  Why wouldn’t it be?

Exactly.

 

If we are going to put type information in the JSON stream purely for readability and type checking, shouldn't we also put type information about EXT, DB, SGL, etc.?

 

This just looks to me like a solution looking for a problem.

[drjdpowell]

This just looks to me like a solution looking for a problem.

Perhaps.  But most types can be meaningfully converted.  A SGL can be a DBL.  The number 5.6 can be the string “5.6”.  Even the string “Hello” can be returned as a valid DBL (NaN).  But returning a Queue refnum as a User Event refnum is invalid.  That may not be that important, since the invalid User Event will just throw an error when used, 

[PBD_ctrl]

I agree that my "=>" implementation doesn't exactly work.  I like the idea of representing it as an object but I worry about opening a can of worms over a rare use case.  My vote would be to keep it stupid simple at this point and go with the numerical(string for DAQmx, VISA and IVI references) implementation.  It lacks the readability that I desired but I can concede for simplicity.

[drjdpowell]

Maybe one of you guys could take 5 minutes to import them to the bitbucket repo?!  (instructions are included )

I have added this to Bitbucket but I have not tested it much.  Do you have a Test VI that we can add to the test suite?

[Stinus Olsen]

Here is a small VI that shows the usage (and tests) the functionality:

 

Test and Bug Fix.zip

 

At the same time there is a small fix for one of the removal VIs that caused a null value to be created if the path to the value object to be deleted was non-existing..

 

[drjdpowell]

Could you save it in LabVIEW 2011 please.

[PBD_ctrl]

I committed variant to JSON and JSON to variant after modifying the refnum cases to a simple I32 and string setup.  I look forward to a new package.

[Stinus Olsen]

Could you save it in LabVIEW 2011 please.

Here you go:

Test and Bug Fix_2011.zip