[CR] JSON LabVIEW

[drjdpowell]

To be clear, Args was just a normal cluster, nothing Varianty about it. Ton

Ah, I though your “variant arguments” were variant attributes and couldn’t understand how you were doing that.

[drjdpowell]

Shaun,

I am looking at the VIPM package that Ton and I have put together. Shall we divide into two packages, one not dependent on OpenG and one that contains all the Variant stuff? So people who can’t use OpenG can still use the core functions or your extended API? I can do that just be replacing one OpenG function, Trim Whitespace, in the core (can we use your “Fast Trim” instead).

Ton,

You currently have the package installing under LAVA; don’t we need a LAVA tools approved package before doing that?

[ShaunR]

Shaun,

I am looking at the VIPM package that Ton and I have put together. Shall we divide into two packages, one not dependent on OpenG and one that contains all the Variant stuff? So people who can’t use OpenG can still use the core functions or your extended API? I can do that just be replacing one OpenG function, Trim Whitespace, in the core (can we use your “Fast Trim” instead).

Ton,

You currently have the package installing under LAVA; don’t we need a LAVA tools approved package before doing that?

If you can wait until next weekend, I'll replace all the openG stuff so it's a self contained lib. Just a bit busy at the moment finalising the websocket demo, but that will be finished this week.

[drjdpowell]

If you can wait until next weekend, I'll replace all the openG stuff so it's a self contained lib. Just a bit busy at the moment finalising the websocket demo, but that will be finished this week.

OK, but to me it seems silly for anyone to avoid OpenG yet use this (far, far less tested through experience) third-party open-source software. I’d rather offer the whole package to OpenG.

Now, do it with higher performance and you’ll get me interested.

[ShaunR]

OK, but to me it seems silly for anyone to avoid OpenG yet use this (far, far less tested through experience) third-party open-source software. I’d rather offer the whole package to OpenG.

I'm not opposed to that either but I thought Ton was talking about it going into the Lavag Tools Network..

Now, do it with higher performance and you’ll get me interested.

Yeah. Well its got LVPOOP in it so that's not an option.

[drjdpowell]

Yeah. Well its got LVPOOP in it so that's not an option.

Theoretically, the not-on-the-pallet VIs in vi.lib\Utility\VariantDataType should blow the OpenG stuff out of the water, as OpenG has to flatten the data to access it (expensive), but my only experience with the VariantDataType Vis is that they are glacially slow.

[ShaunR]

Theoretically, the not-on-the-pallet VIs in vi.lib\Utility\VariantDataType should blow the OpenG stuff out of the water, as OpenG has to flatten the data to access it (expensive), but my only experience with the VariantDataType Vis is that they are glacially slow.

In theory. Practice and theory are the same. In practice, they are not

Historically the openG stuff is generally more performant since a lot of them perform the same functions as are available in LV, just optimised. Of course with the NI grown ones they can have C functions in the exe to do the heavy lifting, but usually they use nodes that run the the UI thread which mitigates any performance gained

Saying that. I use the vi.lib getTypeInfo,GetCluster/numeric etc but I haven't bench-marked them since they are "Hobsons Choice" for me.

[drjdpowell]

Saying that. I use the vi.lib getTypeInfo,GetCluster/numeric etc but I haven't bench-marked them since they are "Hobsons Choice" for me.

I’m pretty sure you’ve got some non-UI thread substitutes lying around. I think you posted one once.

[jerl]

I have a 2D array in a cluster in a 1D array in a cluster. after flattening, in the JSON string, the 2d array looks like 1d. if I try to unflatten this string to my cluster, the variant to data primitive throws me an incompatible datatype error. it seems, 2d arrays are generally unsupported?

[drjdpowell]

I had not considered multi-dimentional arrays. JSON doesn’t have a multi-dimentional array type, but we could just have arrays of arrays. I will look into supporting it. Thanks.

[ShaunR]

I had not considered multi-dimentional arrays. JSON doesn’t have a multi-dimentional array type, but we could just have arrays of arrays. I will look into supporting it. Thanks.

N dim arrays don't have a "type" but they are represented

e.g. Array:[[1,2,3,4,5],[1,2,3,4,5]]

This causes us a problem in the way we convert to type from a variant (recursion works against us due to the fact that it is not a recursive function. it is iterative). If we were just dealing with strings, then it would be fairly straight forward.

For example. Using the "Set From Variant" in a for loop works fine for 2D arrays. But for 3D arrays it will give incorrect results (try with the "Example Creation of JSON string.vi"-bug).

One way forward is to detect the number of dims and have a different "Set JSON Array.vi" for each (max 4 dims?). But this is ugly (POOP to the rescue?).

[drjdpowell]

It looks like it can be done with a little OpenG gymnastics (though not trivial). But this is making more work for you if you want to avoid an OpenG dependancy. How do you want to proceed with that?

[ShaunR]

It looks like it can be done with a little OpenG gymnastics (though not trivial). But this is making more work for you if you want to avoid an OpenG dependancy. How do you want to proceed with that?

Well. Decoding an N dim array is not that hard (a 1d array of values with offsets). But I'm not sure how you envisage representing it internally .

Edited December 22, 2012 by ShaunR

[drjdpowell]

Well. Decoding an N dim array is not that hard (a 1d array of values with offsets). But I'm not sure how you envisage representing it internally .

If you can give me a 1d array, and the dimensions to calculate the offsets, it will be easy to recursively convert to nested JSON arrays inside of JSON arrays (and it will work for any number of dimensions).

— James

BTW> One other issue that occurs to me is “ragged” arrays, arrays of other arrays of varying size. In LabVIEW, I would make a ragged array from an array of clusters containing arrays; in JSON, it would be just an array of arrays of varying length. I would probably try and add support for conversion between these types, too, if we’re going to support multi-D arrays.

[ShaunR]

If you can give me a 1d array, and the dimensions to calculate the offsets, it will be easy to recursively convert to nested JSON arrays inside of JSON arrays (and it will work for any number of dimensions).

Done.

BTW> One other issue that occurs to me is “ragged” arrays, arrays of other arrays of varying size. In LabVIEW, I would make a ragged array from an array of clusters containing arrays; in JSON, it would be just an array of arrays of varying length. I would probably try and add support for conversion between these types, too, if we’re going to support multi-D arrays.

Agreed.. All LV arrays are "square" arrays, however this is inefficient for large data-sets. We could actually handle non-square arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "square" when necessary on extraction).......just a thought!

[Anders Björk]

Done. Agreed.. All LV arrays are "square" arrays, however this is inefficient for large data-sets. We could actually handle non-square arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "square" when necessary on extraction).......just a thought!

Square arrays? Equal dimension size? You must mean rectangular arrays or?

[ShaunR]

Square arrays? Equal dimension size? You must mean rectangular arrays or?

Quite right. Poor terminology on my part.

Done. Agreed.. All LV arrays are "rectangular" arrays, however this is inefficient for large data-sets. We could actually handle non-rectangular arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "rectangular" when necessary on extraction).......just a thought!

[drjdpowell]

I have a 2D array in a cluster in a 1D array in a cluster. after flattening, in the JSON string, the 2d array looks like 1d. if I try to unflatten this string to my cluster, the variant to data primitive throws me an incompatible datatype error. it seems, 2d arrays are generally unsupported?

I am having a problem uploading a new version to the Code Repository, but you can download it from bitbucket.  It is extended to work with multi-D (rectangular) arrays.

[Mark Balla]

Certified 2-18-2013

Placed in General Category

[JamesMc86]

Hey Guys,

 

Had an issue with JSON to Variant where types mismatch because of no enum support so variant to data threw an error.

Though it easiest to keep seperate so there is a patch attached. I see it already encodes the enums as string anway so I used the OpenG Variant tools to use this and set the variant value as a string.

EDIT: Won't let me upload a patch, shall I push it into the bitbucket repo or fork first?

 

Cheers,

James

[drjdpowell]

Had an issue with JSON to Variant where types mismatch because of no enum support so variant to data threw an error.

It’s supposed to work with enums, and my test case seems to work.  "Variant to JSON" delegates enums to OpenG’s “Scan from String” which calls OpenG “Set Enum String Value”, so it should work.  What error did you get and what enum string did it have?

[JamesMc86]

Hi,

Thought I had already posted back, sorry!

 

The root cause is that the OpenG Scan From String puts the enum string through a scan from string with %s which removes anything after a space so this bug specifically effects enum strings with spaces.

 

A customer of mine has also had issues with strings in clusters. In JSON Scalar.lvclass:Variant Decode.vi it assumes all array elements are the same size. In the case of an array of strings or an array of clusters with strings this is not the case. I guess this comes back to your discussions about ragged 2d arrays as well. You can see the issue in the attached VI. He has managed to write a fix for this that we can share as well in whatever format is easiest.

[Ton Plomp]

Hi James,

 

Could you fork the repo at bitbucket, add the fix to your fork, and request a Pull request. It woul be great if it can be supplied in LV2009 format.

 

Regards,

 

Ton

[JamesMc86]

Do you mean LV2011?  This is the version the code was originally in. I will get that done.

 

Cheers,

James

[Ton Plomp]

Sorry my memory is mixing up projects.

 

Ton