[CR] JSON LabVIEW

[drjdpowell]

Attached is Version 1.2 with all new features.  I will add this to the CR in a few days if there are no objections. Please run this version with your projects to test for errors.

 

lava_lib_json_api-1.2.0.22.vip

[Stinus Olsen]

According to most of the information I could find on the subject, the shortest possible JSON stream allowed is either an empty Object "{}" or an empty array "[]".

However, when trying to parse a string containing only an empty Object "{}", I'm getting an error 1 in return - is this expected behaviour?

 

I recon the JSON Object:Unflatten function should be able to handle at least an empty root object?

[Stinus Olsen]

Here is a fix to the above mentioned problem.

I have added an enclosing selector case as a guard against empty objects in the JSON Object::Unflatten function:

 

JSON Object - Unflatten.zip

(Backsaved to 2011)

 

As I'm still not a BitBucket user, could one of you possibly add it to the repository?

 

To test the code, create a file containing only an opening and a closing bracket "{}", and try to load it before and after the fix is applied.. 

 

[drjdpowell]

New version posted in the CR.  Includes a fix for empty objects.

[drjdpowell]

I’ve made an addition that I could use comment on.  See “Mixed type array experiment JDP.vi” in the Bitbucket repo.  Basically it is about JSON like this:

 

{“instruments”:[{"type":"SimMultimeter","config":{"maxV":10,"maxI":5}},

{"type":"SimVoltmeter","config": {"Voltage":100}},

{"type":"SimDAQ","config": 12.3}]

}

 

Here “instruments" is an array of objects corresponding to modules.  Each has an identifier, and also a configuration object that varies from module to module.  We can’t make a LabVIEW cluster that matches this structure, so at the moment we have to use the lower-level functions to get at it.  However, if we have a means of defining a string in a cluster as a JSON string, then we can hold arbitrary JSON subitems in this string.  To do this I have added support for a “tag”, <JSON>, at the beginning of the name of any string in a cluster.  With this, we can convert the above into a cluster containing an “instruments" array of clusters of two strings, named “type” and “<JSON>config”.  Thus we can still work with static clusters, while retaining dynamic behavior at lower levels.

 

Have a look at the example and see what you think?

 

— James

[drjdpowell]

Also see “Test JSON subitems in Variant.vi”.   I’ve added the ability to use “JSON Value” class objects in the Variant JSON tools.  I should have thought of this first, before using “tagged” string names, so perhaps the tagged string stuff should be dropped.

 

[Stobber]

Will anybody mind if I "clean" up the API and palettes a little? I want to:

- Give short names to palette items

- Create icons for all the palettes

- Rename some palettes

- Rearrange some palette items to group them better, maybe nest things deeper for more organization

- Expand the VI documentation for all public VIs

- Remove the probe VIs from the palettes

- Remove polymorphic instances from the palettes

- Remove test VIs from the palettes

 

I'd also like to do some things that would break backward compatibility:

- Rename several methods and functions to enforce "verb phrase" naming convention

- Standardize the names of many similar methods

- Remove test VIs from the package; I believe those belong in the source repo, not the distribution

 

Finally, I'd like to put out a "call for examples" and include any that are submitted. This is a big API, and it's hard to pick up without straightforward examples of both parsing and generating JSON. I'm still not even sure whether this is a DOM/XPath parser or a stream parser, or both!

[Stobber]

Bug?

 

"JSON.lvlib:Get Item As String.vi" can't find its library on disk, and neither can I.

I found this function on the JSON Array palette.

[drjdpowell]

Will anybody mind if I "clean" up the API and palettes a little? I want to:

What palettes are you looking at?  My JSON API palette only has two items (the Get and Set polymorphic VIs).  

[Stobber]

What palettes are you looking at?  My JSON API palette only has two items (the Get and Set polymorphic VIs).  

 

I see pretty much everything. Are you using auto-generated palettes?

 

[drjdpowell]

I see pretty much everything. Are you using auto-generated palettes?

The VIPM package includes a palette with only the two polymorphic Get/Set VIs.  Are you creating a different VIPM package?

 

[Stobber]

No, I'm not trying to create a new project with the code. (I did try to get to the Bitbucket repo, but the link from the forum thread is broken. Is it still a public repo?)

 

I opened VIPM and realized that it doesn't think the pacakge is installed, even though there's obviously code in my <addons> folder. I reinstalled v1.2.1.23 and the palettes are still messed up.

 

[update]

Uninstalled the package, deleted the <vi.lib>/addons/JSON API folder, and then reinstalled. Palettes still look like my screenshot.

Edited May 1, 2014 by Stobber

[drjdpowell]

The repo is now under “LAVAG”.  I don’t no why your palettes are different  I’ve just noticed that I have TWO “JSON API” palettes under “Addons”, the one with two items, and full set that you see.  I’ll try and investigate.

[CharlesB]

I'm discovering this library, find it to be very mature, and the best replacement to the aging OpenG's variant config library, and more practical to use than AQ Character lineator, that requires to manually handle each field of your classes or clusters.

 

I'd have a suggestion to handle numbers with units: currently it's not recognized as such, and outputs the raw binary representation. I think it should at least issue an error, or represent the unit in the JSON string. I tried to look at what can be done for the latest case, but ran into two problems:

 

First, in order to serialize the unit you have to get a string representation of the unit (like "m.s^-1" for a speed number), which is quite difficult. You just get a cluster with base units and exponent, using vi.lib/Utility/VariantDataType/GetNumericInfo.vi. And also it's difficult, if not impossible, for the JSON representation to set the correct type back.

 

A solution can be to not output the unit string with number, and convert it to raw number in unit base, using the "Strip Units" VI from OpenG Data library. This is what variant config library does, in the  "SGL PQ".."EXT PQ" case of "Read key (variant).vi".

 

The other point I'd like to share is that I have some classes that I would like to serialize. Can we imagine the following thing, à la AQ Character lineator:

A class wants to be serializable, and thus defines methods "to data" and "from data", which take a variant as their respective input/output. Internally this variant would be a cluster representing private data. JSON API, when meeting a class instance, just needs to call the "to data" method VI and pass the resulting variant to "Variant to JSON", and serialize the real class name next to it.

 

On deserialization, JSON creates an instance from the real class name, and call its "from data" method VI.

 

This would require of course that these classes to inherit from a "serializable" class.

 

What do y'all think about this?

 

[drjdpowell]

A solution can be to not output the unit string with number, and convert it to raw number in unit base, using the "Strip Units" VI from OpenG Data library. This is what variant config library does, in the  "SGL PQ".."EXT PQ" case of "Read key (variant).vi".

That might be best, and easy to do.

 

The other point I'd like to share is that I have some classes that I would like to serialize. Can we imagine the following thing, à la AQ Character lineator:

Also easy to do (just one abstract class with two empty methods).  I would have the “to” and “from” methods deal in JSON Value objects rather than a cluster in variant; the User can always call the Variant to JSON methods if they want, and they have the option of using the direct JSON methods as well.

 

— James

[Shaun Hayward]

Hi Guys,

I'm noticing an issue with this library and I'm not sure the best way to work around it:

 

Problem: For better or worse, the API I'm communicating with needs JSON objects to be in a correct order in addition to correctly named. i.e. {"B":1, "C":2, "A":3} is NOT equal to {"A":3,"B":1,"C":2}. However, when creating the JSON object using "Set from JSON string.vi" the order seems to be getting lost. Any subsequent calls to "Get as JSON string.vi" always seem to end with the elements getting returned in alphabetical order (as opposed to the original structure order).

 

In my example, simply calling "Set from JSON String" followed by "Get as JSON string" is causing my object of {"Alias":"string","Type":"string","Configuration":[{object}]} to be converted to {"Alias":"string","Configuration":[{object}],"Type":"string"}

 

"Configuration" and "Type" are getting swapped and this is causing the 3rd party API I'm working with to discard the object.

 

One option I've thought of (but havent tried yet) is to modify the JSON Object class to maintain an array of the element names that "JSON Object.lvclass:Flatten.vi"  uses to make sure that the variant attributes come back in the correct order. Can any of you think of a reason this would not work, or have a better idea? (I haven't poked too deep into this API yet)

 

Thanks,

Shaun

[Shaun Hayward]

I made the modification I suggested to a fork of this library (https://bitbucket.org/shew82/json-api-labview/) .

 

This change, just changes JSON Object.lvclass, and I didnt touch the VIPB configuration file, etc. I also don't have LabVIEW 2011 available, so the changed code is in LabVIEW 2013. Let me know if you want me to do a pull request, etc.

[drjdpowell]

I wouldn’t be in favor of adding overhead to all uses of JSON Object just to support one badly broken implementation in another language.   You might be able to make a child-class of JSON Object (“Ordered JSON Objectâ€?) that you could optionally use.  

[JKSH]

I agree with drdjpowell.

 

The JSON standard says (emphasis added), "An object is an unordered set of name/value pairs."

 

This library is standards-compliant and does not need to be changed. The problem lies in the API you're communicating with. Send its developers a bug report and get them to fix it. The world will thank you for it

Edited October 2, 2014 by JKSH

[ShaunR]

I agree with drdjpowell.

 

The JSON standard says (emphasis added), "An object is an unordered set of name/value pairs."

 

This library is standards-compliant and does not need to be changed. The problem lies in the API you're communicating with. Send its developers a bug report and get them to fix it. The world will thank you for it

 

I sit firmly on the other side of the fence with things like this. When it's a choice between form over function; function wins - especially if the spec can be interpreted ambiguously via tortured semantics. Adhering to specs by observing a strict negative of a positive statement (it doesn't say they cannot be ordered, rather, I expect it was stated thus so it didn't restrict) is why the native 2013 JSON is inferior (IMHO) to this library - speed aside. This library works when the native one throws errors because the native one adheres even more strictly, word for word, to the spec.

 

inf and NAN are examples of where some libraries have implemented function over form. In that case, it was in spite of the spec which specifically disallowed it. There is precedence here if there is enough benefit.

 

Would it really hurt that much to make the output look exactly like the input which is what we all kind of expect and know to be right? Relying on a specs throw away description about an unordered list seems a bit of a cop out to me and it would probably make testing much easier and simpler as you could do a straight input/output compare. It wouldn't break existing code, either. So I'm not sure what the resistance is apart from the effort required which has already been done.

Edited October 2, 2014 by ShaunR

[drjdpowell]

Would it really hurt that much to make the output look exactly like the input which is what we all kind of expect and know to be right? Relying on a specs throw away description about an unordered list seems a bit of a cop out to me and it would probably make testing much easier and simpler as you could do a straight input/output compare. It wouldn't break existing code, either. So I'm not sure what the resistance is apart from the effort required which has already been done.

I haven’t looked at the change, but I imagined that it involved building a separate array of item names.  If so, that is a significant additional overhead that will reduce performance.  If this ordering can be added either without significant overhead (or can be turned on optionally when needed) then I don’t really object.  

 

Note, though, that “what we all kind of expect and know to be right†is the assumption that caused the other-language JSON implementation to be released in such a flawed and brittle state.  

 

— James

[Shaun Hayward]

Sorry for the late reply: I thought I had thread subscription turned on but no emails came up...

 

James, your right in your assessment - this was a very simple implementation that just build a string array of names when the object tree gets built. This was mostly because I didnt want to just throw a request (ie potential work) at someone without at least offering a possible "solution".

 

Using the "Large Test Case.vi" from the source didn't seem to show much of a performance penalty (see below) but if this is a deal breaker could we look into a property for JSON Value.lvclass for "Maintain Order" that when true would build and use the array, and when false (default) would use existing alphabetical ordering?

 

I didn't add this with my first offering as the performance still seemed more than adequate (given the size and complexity of the JSON objects I'm working with and the test case timing results), and adding and handling the property would make the code noticeably more complex (e.g. making sure the flag is propagated through the object tree, what happens if the flag changes  etc).  Subclassing JSON Object also seemed like it could get pretty complex as there's seconds like "JSON Value:Unflatten.vi" that specify class type as a constant on the block diagram.

 

Timing results (both cases are mean of 1000 iterations):

Released code = about 19.4ms  

Modified code = about 20.7ms

 

I (personally) also like encoders/decoders that have the ability to get back the original data exactly as they tend to be easier to validate etc.

 

So overall, whilst I felt this was an acceptable modification (with IMHO, some benefits), if you guys feel that the performance difference is a killer and don't like the idea of adding a property, I can totally understand.

 

Regards,

Shaun

[drjdpowell]

I keep meaning to make time to work on JSON but here’s an idea:

 

Instead of storing the JSON Object’s values in a Variant look-up table, store them in an array and store the index to that array in the lookup table.  That might have better performance since one may be able to do some operations in-place on the array (Variant attributes always make copies, sadly).   As an added advantage, one can output the JSON Object in the original order by using the array order.  There would be some overhead here, as you’d have to sort the names based on the indexes to match them to the array order, but this part could be easily be optional based on a boolean input (similar to “Pretty Printâ€).

 

Let me think about it some more...

[ShaunR]

I keep meaning to make time to work on JSON but here’s an idea:

 

Instead of storing the JSON Object’s values in a Variant look-up table, store them in an array and store the index to that array in the lookup table.  That might have better performance since one may be able to do some operations in-place on the array (Variant attributes always make copies, sadly).   As an added advantage, one can output the JSON Object in the original order by using the array order.  There would be some overhead here, as you’d have to sort the names based on the indexes to match them to the array order, but this part could be easily be optional based on a boolean input (similar to “Pretty Printâ€).

 

Let me think about it some more...

 

<Dumb question>

The request is for values not to be sorted alphabetically.. Is the solution just not to sort? Do we, indeed, sort them? Is it part of the prettyfier rather than the encoding. Where exactly is this sorting taking place?

Edited October 8, 2014 by ShaunR

[Shaun Hayward]

I keep meaning to make time to work on JSON but here’s an idea:

 

Instead of storing the JSON Object’s values in a Variant look-up table, store them in an array and store the index to that array in the lookup table.  That might have better performance since one may be able to do some operations in-place on the array (Variant attributes always make copies, sadly).   As an added advantage, one can output the JSON Object in the original order by using the array order.  There would be some overhead here, as you’d have to sort the names based on the indexes to match them to the array order, but this part could be easily be optional based on a boolean input (similar to “Pretty Printâ€).

 

Let me think about it some more...

 

That is an interesting idea... I would imagine that being able to work in place could definitely offer better performance for large / nested JSON objects.

 

 

<Dumb question>

The request is for values not to be sorted alphabetically.. Is the solution just not to sort? Do we, indeed, sort them? Is it part of the prettyfier rather than the encoding. Where exactly is this sorting taking place?

 

From the poking around I did, the sorting that takes place is an implicit sort when adding the JSON elements as variant attributes to a containing element. They end up getting stored alphabetically (in the variant attribute tree) so when "Get all attributes" is done in the "flattening to JSON string" VI, the objects come out in alphabetical order. The workaround I threw together kept a list of names so that instead of "get all attributes" a "foreach name, get attribute" was done (which preserved original order).