Jump to content

[CR] JSON LabVIEW


Recommended Posts

Shaun,

I am looking at the VIPM package that Ton and I have put together. Shall we divide into two packages, one not dependent on OpenG and one that contains all the Variant stuff? So people who can’t use OpenG can still use the core functions or your extended API? I can do that just be replacing one OpenG function, Trim Whitespace, in the core (can we use your “Fast Trim” instead).

Ton,

You currently have the package installing under LAVA; don’t we need a LAVA tools approved package before doing that?

Link to comment

Shaun,

I am looking at the VIPM package that Ton and I have put together. Shall we divide into two packages, one not dependent on OpenG and one that contains all the Variant stuff? So people who can’t use OpenG can still use the core functions or your extended API? I can do that just be replacing one OpenG function, Trim Whitespace, in the core (can we use your “Fast Trim” instead).

Ton,

You currently have the package installing under LAVA; don’t we need a LAVA tools approved package before doing that?

If you can wait until next weekend, I'll replace all the openG stuff so it's a self contained lib. Just a bit busy at the moment finalising the websocket demo, but that will be finished this week.

Link to comment
If you can wait until next weekend, I'll replace all the openG stuff so it's a self contained lib. Just a bit busy at the moment finalising the websocket demo, but that will be finished this week.

OK, but to me it seems silly for anyone to avoid OpenG yet use this (far, far less tested through experience) third-party open-source software. I’d rather offer the whole package to OpenG.

Now, do it with higher performance and you’ll get me interested. :)

Link to comment

OK, but to me it seems silly for anyone to avoid OpenG yet use this (far, far less tested through experience) third-party open-source software. I’d rather offer the whole package to OpenG.

I'm not opposed to that either but I thought Ton was talking about it going into the Lavag Tools Network..

Now, do it with higher performance and you’ll get me interested. :)

Yeah. Well its got LVPOOP in it so that's not an option. :P

Link to comment
Yeah. Well its got LVPOOP in it so that's not an option. :P

Theoretically, the not-on-the-pallet VIs in vi.lib\Utility\VariantDataType should blow the OpenG stuff out of the water, as OpenG has to flatten the data to access it (expensive), but my only experience with the VariantDataType Vis is that they are glacially slow.

Link to comment

Theoretically, the not-on-the-pallet VIs in vi.lib\Utility\VariantDataType should blow the OpenG stuff out of the water, as OpenG has to flatten the data to access it (expensive), but my only experience with the VariantDataType Vis is that they are glacially slow.

In theory. Practice and theory are the same. In practice, they are not :P

Historically the openG stuff is generally more performant since a lot of them perform the same functions as are available in LV, just optimised. Of course with the NI grown ones they can have C functions in the exe to do the heavy lifting, but usually they use nodes that run the the UI thread which mitigates any performance gained

Saying that. I use the vi.lib getTypeInfo,GetCluster/numeric etc but I haven't bench-marked them since they are "Hobsons Choice" for me.

Link to comment
  • 1 month later...

I have a 2D array in a cluster in a 1D array in a cluster. after flattening, in the JSON string, the 2d array looks like 1d. if I try to unflatten this string to my cluster, the variant to data primitive throws me an incompatible datatype error. it seems, 2d arrays are generally unsupported?

Link to comment

I had not considered multi-dimentional arrays. JSON doesn’t have a multi-dimentional array type, but we could just have arrays of arrays. I will look into supporting it. Thanks.

N dim arrays don't have a "type" but they are represented

e.g. Array:[[1,2,3,4,5],[1,2,3,4,5]]

This causes us a problem in the way we convert to type from a variant (recursion works against us due to the fact that it is not a recursive function. it is iterative). If we were just dealing with strings, then it would be fairly straight forward.

For example. Using the "Set From Variant" in a for loop works fine for 2D arrays. But for 3D arrays it will give incorrect results (try with the "Example Creation of JSON string.vi"-bug).

One way forward is to detect the number of dims and have a different "Set JSON Array.vi" for each (max 4 dims?). But this is ugly (POOP to the rescue?).

Link to comment

It looks like it can be done with a little OpenG gymnastics (though not trivial). But this is making more work for you if you want to avoid an OpenG dependancy. How do you want to proceed with that?

Well. Decoding an N dim array is not that hard (a 1d array of values with offsets). But I'm not sure how you envisage representing it internally .

Edited by ShaunR
Link to comment
Well. Decoding an N dim array is not that hard (a 1d array of values with offsets). But I'm not sure how you envisage representing it internally .

If you can give me a 1d array, and the dimensions to calculate the offsets, it will be easy to recursively convert to nested JSON arrays inside of JSON arrays (and it will work for any number of dimensions).

— James

BTW> One other issue that occurs to me is “ragged” arrays, arrays of other arrays of varying size. In LabVIEW, I would make a ragged array from an array of clusters containing arrays; in JSON, it would be just an array of arrays of varying length. I would probably try and add support for conversion between these types, too, if we’re going to support multi-D arrays.

Link to comment

If you can give me a 1d array, and the dimensions to calculate the offsets, it will be easy to recursively convert to nested JSON arrays inside of JSON arrays (and it will work for any number of dimensions).

Done.

BTW> One other issue that occurs to me is “ragged” arrays, arrays of other arrays of varying size. In LabVIEW, I would make a ragged array from an array of clusters containing arrays; in JSON, it would be just an array of arrays of varying length. I would probably try and add support for conversion between these types, too, if we’re going to support multi-D arrays.

Agreed.. All LV arrays are "square" arrays, however this is inefficient for large data-sets. We could actually handle non-square arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "square" when necessary on extraction).......just a thought!

Link to comment
Done. Agreed.. All LV arrays are "square" arrays, however this is inefficient for large data-sets. We could actually handle non-square arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "square" when necessary on extraction).......just a thought!

Square arrays? Equal dimension size? You must mean rectangular arrays or?

  • Like 1
Link to comment

Square arrays? Equal dimension size? You must mean rectangular arrays or?

Quite right. Poor terminology on my part.

Done. Agreed.. All LV arrays are "rectangular" arrays, however this is inefficient for large data-sets. We could actually handle non-rectangular arrays by using arrays of DVRs internally (again, the premise being that we would only convert to "rectangular" when necessary on extraction).......just a thought!

Link to comment
  • 2 weeks later...
I have a 2D array in a cluster in a 1D array in a cluster. after flattening, in the JSON string, the 2d array looks like 1d. if I try to unflatten this string to my cluster, the variant to data primitive throws me an incompatible datatype error. it seems, 2d arrays are generally unsupported?

I am having a problem uploading a new version to the Code Repository, but you can download it from bitbucket.  It is extended to work with multi-D (rectangular) arrays.

Link to comment
  • 1 month later...
  • 2 months later...

Hey Guys,

 

Had an issue with JSON to Variant where types mismatch because of no enum support so variant to data threw an error.


Though it easiest to keep seperate so there is a patch attached. I see it already encodes the enums as string anway so I used the OpenG Variant tools to use this and set the variant value as a string.


EDIT: Won't let me upload a patch, shall I push it into the bitbucket repo or fork first?

 

Cheers,

James

Link to comment
Had an issue with JSON to Variant where types mismatch because of no enum support so variant to data threw an error.

It’s supposed to work with enums, and my test case seems to work.  "Variant to JSON" delegates enums to OpenG’s “Scan from String” which calls OpenG “Set Enum String Value”, so it should work.  What error did you get and what enum string did it have?

Link to comment
  • 2 months later...

Hi,


Thought I had already posted back, sorry!

 

The root cause is that the OpenG Scan From String puts the enum string through a scan from string with %s which removes anything after a space so this bug specifically effects enum strings with spaces.

 

A customer of mine has also had issues with strings in clusters. In JSON Scalar.lvclass:Variant Decode.vi it assumes all array elements are the same size. In the case of an array of strings or an array of clusters with strings this is not the case. I guess this comes back to your discussions about ragged 2d arrays as well. You can see the issue in the attached VI. He has managed to write a fix for this that we can share as well in whatever format is easiest.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.