133 replies to this topic

[drjdpowell]

Name: JSON LabVIEW
Submitter: drjdpowell
Submitted: 04 Oct 2012
Category: General
LabVIEW Version: 2011
License Type: BSD (Most common)

JSON is a data interchange format (sometimes compared to XML, but simpler). There are multiple projects to create a JSON package for LabVIEW. This is yet another one motivated by this hijacked conversation originally about a different project to convert JSON into LabVIEW Variants.

This project uses a set of LVOOP classes to match the recursive structure of JSON, rather than variants. It allows conversation to and from JSON. All functionality is available through two polymorphic VIs: Set and Get. In addition to Get and Set VIs for common data types, one can also convert directly to or from complex clusters via variant-JSON tools.

Copyright 2012-2014 James David Powell, Shaun Rumbell, Ton Plomp and James McNally.

 

NOTE: latest version requires VIPM 2014 to install

Click here to download this file

[Ton Plomp]

I tried to use this tool. And it looks great, however the function 'Variant to JSON' fails with a variant inside an object. I made the following changes:
-Added support for Variant. By using 'Unwrap Variant'

-If a <Null> Variant is found, replace this with a Null-Cluster

-When Flattening the Cluster and number of elements is 0 then output '{}'

(note that without this change an empty cluster would flatten to '}')
This allowed me to have a cluster with optional variant Arguments flatten well:
into:
{"api_key":"xx","args":{"owner":"ton","repo_name":"repository","repo_type":"hg"},"id":915,"method":"create_repo"} (with args)
or
{"api_key":"xx","args":{},"id":378,"method":"get_users"} (no args)

I added a method for 'Object' to list all element names:

This will return the names of all the elements in an Object (Cluster), so you can scan them and retrieve them by name.

Ton

Edited by Ton Plomp, 04 October 2012 - 09:10 PM.

[Ton Plomp]

I extended the code with 'Pretty Print' for flattening (carriage returns, indents, adds spaces). I uploaded the source to bitbucket. Get the latest source at https://bitbucket.or...ew/get/tip.zip.
Repo is at: https://bitbucket.or...son-api-labview

[ShaunR]

Took Toms version (love the "prettyfying" btw ) and added support for escaped quotes in strings (\").
Escaped quotes are maintained in the variant table and only converted when read out via the polymorphic VIs.
(back-saved in 2009)

Edited by ShaunR, 06 October 2012 - 01:50 AM.

[Ton Plomp]

On topic:
The testcase include the followin JSON:
{"T1":456.789 , "T2":"test2", "Nest":{"ZZ":123,"NestArray":[1,2,"Hi"] }}
Is that even valid JSON?
Is it allowed in JSON to have different objects types inside an array? (NestArray consist of two numerics and a string)
JSONLint says its valid, but I feel oppposed to that (maybe just limited by LabVIEW)

Bugfixes:

• Boolean JSON text should be lower case (code)
• Numerics should use '.' as decimal sign (code)

Discussion:

What should we do with NaN, -Inf, and +Inf? JSON does not support them. NaN could be null but the others I don't know.
Official JSON sets those three values to 'null' however I lean to this idea. For numerics we should use 1e5000 though.

Ton

Edited by Ton Plomp, 06 October 2012 - 11:42 AM.

[ShaunR]

On topic:
The testcase include the followin JSON:
{"T1":456.789 , "T2":"test2", "Nest":{"ZZ":123,"NestArray":[1,2,"Hi"] }}
Is that even valid JSON?
Is it allowed in JSON to have different objects types inside an array? (NestArray consist of two numerics and a string)
JSONLint says its valid, but I feel oppposed to that (maybe just limited by LabVIEW)

Yes. In fact, it can be and array of anything including other objects and/or arrays so even this is valid:

"myarray":[[123,223,{value1:MyVal1}], value2:"wally"]

This is why, for LabVIEW, it is much better for us to rely on the programmer to decide how to get the info out since with the polymorphic VIs we limit his options to valid LabVIEW types with a fall-back of the raw string.

On a similar note: You don't have to have quotes around a string value value either (it's only required for labels).

What should we do with NaN, -Inf, and +Inf? JSON does not support them. NaN could be null but the others I don't know.
Official JSON sets those three values to 'null' however I lean to this idea. For numerics we should use 1e5000 though.

Ton

I have no position on this. I took the decision in the SQLite API to encode NaN as a string rather than NULL but that was so that extra detection wasn't needed when converting back to LabVIEW types (Search and Replace NULL for NAN). It was an issue of performance that we may not be so fastidious about here. Other languages may not have a representation for NaN, but LabVIEW doesn't have a concept of NULL either except perhaps "empty string" which converts to zero for numeric types.

Off:Topic:
Way to go to get the last word in (comment then get the mods to remove it all...lol)

Edited by crelf, 10 October 2012 - 05:03 PM.
Licensing discussion split to http://lavag.org/topic/16233-licensing-agreements-nicom-vs-lavagorg-vs-labview/

[ShaunR]

For instance ShaunR added support for escape quotes in the product. Now I have to go to his code, detect the changes (which is hard since LabVIEW is binary and he backsaved to 2009), merge these into my Mercurial repo(Done).

I gave up with automatic merging a long time ago with any system. The fact that LabVIEW (especially 2009) re-compiles at the drop-of-a-hat makes most conventional source control systems impotent (and don't get me started on the LV Merge tool). I now only use them for micro-versioning. I rely on comments to identify and track changes and, although it requires an anal approach to commenting, works pretty well-at least for my workflow. You probably saw (simpler versions of) my comments in the code (added, modified etc). Just search for <SR>.

Edited by ShaunR, 06 October 2012 - 04:03 PM.

[ShaunR]

I think some "JSON" implementations have (perhaps wrongly) allowed these
values, so it is probably a good idea to accept things like "Inf", "Infinity", "NaN" when parsing in JSON.

The Json spec (Section 2.4) specifically forbids it for numbers.

Numeric values that cannot be represented as sequences of digits (such as Infinity and NaN) are not permitted.

However. It doesn't say anything about strings We can already cope with recovery of +inf, -inf and Nan (although we will have to add "Infinity") so it's really a decision for encoding (I would suggest string encoding and lose the type but a switch might be more appropriate as you suggest).

PHP will throw an error if the type is a numeric, but will be quite happy if it is a string and, as PHP is untyped once parsed it really makes no difference.:

json_encode(): double INF does not conform to the JSON spec, encoded as 0 in /var/www/html/search.php on line 167

Some Javascript engines allow it since it is valid Javascript but not valid Json however there is no concensus.

• cmj-JSON4Lua: raw tostring() output (invalid JSON).
• dkjson: 'null' (like in the original JSON-implementation).
• Fleece: NaN is 0.0000, freezes on +/-Inf.
• jf-JSON: NaN is 'null', Inf is 1e+9999 (the encode_pretty function still outputs raw tostring()).
• Lua-Yajl: NaN is -0, Inf is 1e+666.
• mp-CJSON: raises invalid JSON error by default, but runtime configurable ('null' or Nan/Inf).
• nm-luajsonlib: 'null' (like in the original JSON-implementation).
• sb-Json: raw tostring() output (invalid JSON).
• th-LuaJSON: JavaScript? constants: NaN is 'NaN', Inf is 'Infinity' (this is valid JavaScript?, but invalid JSON).

(NB. Twitter allows it,but I have no idea what library they use. Lots of people complaining that they get errors from parsers when recovering feeds and the general mood is that parsers should just cope with it).

Edited by ShaunR, 08 October 2012 - 04:58 PM.

[drjdpowell]

I’ve added a new version to the CR (sorry Ton, I will need to learn how to use Github).

I added a JSON to Variant function. Note that I’m trying to introduce as much loose-typing as possible (very natural when going through a string intermediate), so the below example shows conversion between clusters who have many mismatched types, as well as different orders of elements. It would be nice to think about what type conversion should be allowed without throwing an error.



And I’ve started on a low-level set of Get/Set polymathic VIs for managing the conversion from JSON Scalers/Arrays to LabVIEW types (very similar to Shaun’s set but without the access by name array). I’ve reformatted two of Shaun’s VIs to be based of the new lower-level ones. The idea is to restrict the conversion logic (which at some point will have to deal with escaped characters, special logic for null (==NaN), Inf, UTF-8 conversion, allowed Timestamp formats, etc.) to be in only one clearly defined place. At some point, I will redo the Variant stuff to work off of these functions rather than relying on the OpenG String Palette as they do now).

[ShaunR]

I’ve added a new version to the CR (sorry Ton, I will need to learn how to use Github).

I added a JSON to Variant function. Note that I’m trying to introduce as much loose-typing as possible (very natural when going through a string intermediate), so the below example shows conversion between clusters who have many mismatched types, as well as different orders of elements. It would be nice to think about what type conversion should be allowed without throwing an error.



And I’ve started on a low-level set of Get/Set polymathic VIs for managing the conversion from JSON Scalers/Arrays to LabVIEW types (very similar to Shaun’s set but without the access by name array). I’ve reformatted two of Shaun’s VIs to be based of the new lower-level ones. The idea is to restrict the conversion logic (which at some point will have to deal with escaped characters, special logic for null (==NaN), Inf, UTF-8 conversion, allowed Timestamp formats, etc.) to be in only one clearly defined place. At some point, I will redo the Variant stuff to work off of these functions rather than relying on the OpenG String Palette as they do now).

Do you want to list out the tasks that need to be done so that we can apportion them between us?

[drjdpowell]

First thing we could use are functions to convert between JSON-format strings and regular LabVIEW Text (escaped characters, encoding, add/remove the quotes). I’m weak on regex.

[ShaunR]

First thing we could use are functions to convert between JSON-format strings and regular LabVIEW Text (escaped characters, encoding, add/remove the quotes). I’m weak on regex.

I've already implemented the removal of quotes. The only escaped chars that I know "must" be escaped are unicode strings and I'm not sure what to do about that with LabVIEW not supporting Unicode without using OS dependent code.

[drjdpowell]

Repo is at: https://bitbucket.or...son-api-labview

Hi Ton,
This is my first use of Hg or bitbucket. I’ve got a bitbucket account, installed TortoiseHg and successfully downloaded the latest JSON version and created a small “change set” ready for “pushing” up to the repository, but my authorization fails. Do you need to add me to your “team”?

[drjdpowell]

Just added some improvements to the bitbucket repo. Below is the new “Example Extract… V2” example using the polymorphic Get function. Note that there is no object type casting now, as the function do the casting themselves (throwing an error if the input type is invalid).



I’ve also been working on meaningful error messages. On issue I’d like comment on is getting an item in a JSON object when the object isn’t found. Currently this is not an error, but just makes the “found” output false. I would rather get rid of the “found” output and make it an error instead.

— James

[drjdpowell]

Actually, another possible error choice is to basically never throw an error on “Get”; just return a “null” (or zero, NaN, empty string, etc.) if there is no way to convert the input JSON to a meaningful value of that type (this follows the practice of SQLite, which always provides value regardless of a mismatch between requested and stored data type). Then perhaps all “Get” instances should have a “found” boolean output.

Ton, Shaun, what do you think?

[ShaunR]

Actually, another possible error choice is to basically never throw an error on “Get”; just return a “null” (or zero, NaN, empty string, etc.) if there is no way to convert the input JSON to a meaningful value of that type (this follows the practice of SQLite, which always provides value regardless of a mismatch between requested and stored data type). Then perhaps all “Get” instances should have a “found” boolean output.

Ton, Shaun, what do you think?

Not throwing an error makes it quite hard to find out what the problem is in even moderately sized streams. However. I'm of the opinion that it should at least try a best guess and raise a warning (preferably identifying where in the stream). Errors are a two edged sword since you can end up halting your program just because someone left off a quote (we are heavily reliant on quotes being in the right place).

I'm just waiting for Ton to give me a login the the repo since I've implemented the decoding of escaped chars (not unicode I might add-we need to think about that). It doesn't work for your modified JSON_Double[array] though, since it is utilised in the Get Item By Name rather than Get Text. I have, however, made it a VI so you can put it where you think best.

Edited by ShaunR, 01 November 2012 - 04:33 PM.

[drjdpowell]

I’m talking about errors in the conversion from our “JSON” object into LabVIEW datatypes. There are also errors in the initial interpretation of the JSON string (missing quotes, or whatever); there we will definitely need the throw errors, with meaningful information about where the parsing error occurred in the input JSON string.

For debugging type conversion problems, one can use the custom probes to look at the sub-JSON objects fed into the “Get as…”; this will be a subset of the full initial JSON string.

BTW, heres the previous example where I’ve introduced an error into the Timestamp format (and probed the value just before the “Get”):

[ShaunR]

Timestamp is probably the "odd-one-out" since it can't be interpreted any other way apart from as a timestamp. All the numerics are currently coerced (and output default if coercion fails-NaN for doubles, 0 for others). If the input type is a String Array then the ouput is an array of one literal.

Are you suggesting that if the JSON stream has a double and an Int is requested then it should throw an error? I'm not sure I'd find that very useful (probably more of a pain in the backside) and if I wanted a U8 from an U64 (say to get rid of a coercion dot) then I would have to convert it manually.

[drjdpowell]

Are you suggesting that if the JSON stream has a double and an Int is requested then it should throw an error?

Oh no, definitely not that.

I mean what if the User requests:
— the string “Hello” as a DBL? Is this NaN, zero, or an error? What about as an Int32? A timestamp?
— for that matter, what about a boolean? Should anything other than ‘true’/‘false' be an error? Any non-‘null'/non-‘false' be true (including the JSON strings “null” and false”)? Or any non ‘true' be false (even the JSON string “true”)?
— “1.2Hello” as a DBL? Is this 1.2 or an error?
— or just “1.2”, a JSON string, not a JSON numeric? Should we (as we are doing) allow this to be converted to 1.2?
— a JSON Object as an Array of DBL? A “Not an array” error, or an array of all the Objects items converted to DBL?
— a JSON Scalar as an Array of DBL? Error or a single element array?
— a JSON Object as a DBL? Could return the first item, but Objects are unordered, so “first” is a bit problematic.

And what if the User asks for an item by name from:
— an Object that doesn’t have that named item? Currently this is no error, but we have a “found” boolean output that is false.
— an Array or Scalar? Could be an Error, or just return false for “found”.

Then for the JSON to Variant function there is:
— cluster item name not present in the JSON Object: an error or return the default value

Personally, I think we should give as much “loose-typing” as possible, but I’m not sure where the line should be drawn for returning errors.

[ShaunR]

Well. this is how it works at present.

— the string “Hello” as a DBL? Is this NaN, zero, or an error?
NaN (after all NAN= Not A Number)

What about as an Int32?
0

A timestamp?
Like I said. Odd one out since all the other LabVIEW "from Strings" have a default behavior defined (they don't return errors)

— for that matter, what about a boolean? Should anything other than ‘true’/‘false' be an error? Any non-‘null'/non-‘false' be true (including the JSON strings “null” and false”)? Or any non ‘true' be false (even the JSON string “true”)?
True/False, Yes/No, On/Off 0/>0 are catered for. anything else equates to False.,


— “1.2Hello” as a DBL? Is this 1.2 or an error?
NAN
— or just “1.2”, a JSON string, not a JSON numeric? Should we (as we are doing) allow this to be converted to 1.2?
1.2

These
— a JSON Object as an Array of DBL? A “Not an array” error, or an array of all the Objects items converted to DBL?
— a JSON Scalar as an Array of DBL? Error or a single element array?
— a JSON Object as a DBL? Could return the first item, but Objects are unordered, so “first” is a bit problematic.
These I see as purely internal representations (lazy typing-only define type when recovered)

And what if the User asks for an item by name from:
— an Object that doesn’t have that named item? Currently this is no error, but we have a “found” boolean output that is false.
— an Array or Scalar? Could be an Error, or just return false for “found”.
The default value and a warning?

Then for the JSON to Variant function there is:
— cluster item name not present in the JSON Object: an error or return the default value
The default value and a warning?

Personally, I think we should give as much “loose-typing” as possible, but I’m not sure where the line should be drawn for returning errors.

I don't think any typing should be contained in the objects at all. Only interpreted when recovered to the best of our ability (as it does at present). Why enforce typing internally when there is none in the Json objects. Type is purely a convenience for wiring up to indicators/controls

Edited by ShaunR, 02 November 2012 - 08:09 PM.