[CR] JSONtext

[drjdpowell]

View File

JSONtext

---

Package for working with JSON.  Uses high-speed text parsing, rather than building an intermediate representation as with prior LabVIEW JSON libraries (this is much faster).  Allows easy working with "subitems" in JSON format, or one can convert to/from LabVIEW types.  Uses the new "malleable VIs" of LabVIEW 2017 to convert to any LabVIEW type directly.

JSON text makes use of a form a JSON Path notation, allowing easy and rapid access to the subitems of interest.

Requires LabVIEW 2017 and install by VIPM 2017 or later.

Original conversation about JSONtext.

On the LabVIEW Tools Network.

JDP Science Tools group on NI.com.

Copyright 2017 JDP Science Limited

---

• Submitter
  drjdpowell
• Submitted
  08/02/2017
• Category
  General
• LabVIEW Version
  2017
• License Type
  BSD (Most common)

 

[Mark Balla]

Certified on 09-27-17 and placed in General category 

[drjdpowell]

Is any early adopter using the “From JSON Text.vi” polymorphic vi (or its individual instances)?   I want to completely deprecate those in favor of only using the “From JSON text.vim” malleable VI.   Also, is anyone using any off-palette subVIs?   Because I’m about to reorganize them and make them all private.

[drjdpowell]

I’m also considering removing the “Section Cluster” stuff, to reduce the API to support.

[smithd]

I am using those VIs because I can't update to 2017, so I guess this is the part where I just keep my fork separate.

[elsatobias]

Hi, has anyone tested this package with double datatype?

Doubles get encoded to JSON, but on decoding the decimal part gets cut off.

This seems a problem of the scan From String operation, which uses the country typical decimal separator. In JSON only the (.) is used.

I suggest to add the format string "%.; %f" to the vi's which decode floating point data.

Perhaps there is other country related stuff (timestamps could be a candiate).

Greetings Tobi

Edited November 13, 2017 by elsatobias
Investigated problem, added possible solution

[drjdpowell]

Sorry about that.   I had put %.; in the array functions but forgot the scaler ones.

Edit> please try the 1.2.6 version just uploaded.

Edited November 13, 2017 by drjdpowell

[elsatobias]

12 hours ago, drjdpowell said:

Sorry about that.   I had put %.; in the array functions but forgot the scaler ones.

Edit> please try the 1.2.6 version just uploaded.

Hello,

I have tried the 1.2.6 version. But instead of adding "%.;" (dot), you added "%,;" (comma), what is standard definition on my system. So the problem still exists.

With the new version, nobody could import JSON floating point strings, because even on systems using "." as decimal separator, you forced the converter to use ",".

I'm afraid, you have to change it again.

Greetings Tobi

[drjdpowell]

Opps.   I had made it %,; as a way of testing it on my machine with strings like "123,456", but I forgot to switch back!

Edit> added 1.2.7 version; please test

Edited November 14, 2017 by drjdpowell

[elsatobias]

Thanks a lot,

it now works as expected. Thank you for your fast help.

Greetings Tobi

[silmaril]

Hi, I woul like  to discuss an idea to extend the funcionality of JSONtext.

We would like to use JSON as configuration files to replace INI files without the complexity of XML. This works great from the technical point of view. There is only the drawback that JSON does not support comments in the text, which makes it harder for humans to edit the files. The YAML file format supports comments and it claims to be a superset of JSON: http://www.yaml.org/spec/1.2/spec.html#id2759572

I am not asking to build a complete YAML library (even if this would be a great thing, too), but I was asking myself if it made sense to add support for YAML comments into JSONtext.

This might have to be optional, since it would not comply with the original JSON specification. If active, all code searching for the different markers inside the JSON string would have to check for "#" characters that follow whitespace and ignore everything up to the following line end (with the exception of "#" inside strings).

I am not sure about the implication this would have for code writing to the JSON string (the "Merge ..." VIs).

And of course this would lead to the demand for a function that strips the JSON-string of all comments to convert it to valid JSON that can be used with other APIs.

What do you think?

[drjdpowell]

Do you want to be able to programatically add comments, or just ignore them on reading?   I am wary of adding features to the core functions as I want to protect its speed and simplicity, but I don't mind additional functions.  In fact, I've been considering a function that is intended to take human-edited JSON and "fix it up", correcting common errors like leaving an extra comma when deleting.  This could easily strip comments.

Edited March 7, 2018 by drjdpowell

[JKSH]

16 hours ago, drjdpowell said:

This could easily strip comments.

If you're considering this, then the next question is: What style of comments?

• @silmaril suggested YAML, which uses "# ...".
• JavaScript (which is where JSON came from) uses "/* ... */" and "// ..."

Will you choose one to support? Will you support both (and any other style that might appear in the future)?

Edited March 8, 2018 by JKSH

[drjdpowell]

If it's an additional "fix it up function" then I can support many versions easily.  Might be slower, but one doesn't care about raw speed in a config file use case.   But I'm wary of adding more extended features to the core functions, where I do care about speed and simplicity.  As an example of the simplicity issue, currently one can overwrite whitespace when setting values in a JSON.  I use this to allow quicker setting of items in the JSON, allowing update in place where possible.   If I have 10 char of whitespace and overwrite 4 characters, I now have 6 char or whitespace.  But if I have a 10-char comment, and overwrite 4 char, I now have corrupted JSON.

[silmaril]

On 7.3.2018 at 1:47 PM, drjdpowell said:

Do you want to be able to programatically add comments, or just ignore them on reading?

The primary use case would be read-only, which could be covered perfectly by a strip comments function.

The drawback of this is that it makes any programmatic changes to the config files impossible... ok, not really impossible, but changing one item destroys all the comments in the file

If there is some elegant way to simply ignore comments and keep them in place when changing values without slowing down the core functionality - that would be great! But I do understand that this is very difficult at least.

And I have to admit that I currently don't have any use cases where config file changes have to be written programmatically, so your suggestion sounds like the appropriate solution to this.

 

On 8.3.2018 at 6:29 AM, JKSH said:

What style of comments?

• @silmaril suggested YAML, which uses "# ...".
• JavaScript (which is where JSON came from) uses "/* ... */" and "// ..."

That depends on the way you look at your file. If it's "JSON with comments", the JavaScript style is appropriate. If it's "the JSON part of YAML with comments", the YAML style is the way to go. I am in favour of supporting all three of them in the additional "fix it up function". As James pointed out, performance is no concern in this use case.

 

To get into the technical details: This would mean we have two kinds of comments:

• Line comments: start with a marker symbol sequence; end with the next EOL or EOF.
• Block comments: start with one marker symbol sequence, end with another marker symbol sequence.

I think we should add a convention for line comments: They should only be allowed outside of strings, otherwise we could destroy perfectly valid string values or object names. I would also define that the marker must be preceded by some kind of whitespace (space, tab, CR, LF) or be at the beginning of the file.

 

[SGI]

I have a variant of an Enum that I want to convert from JSON text.

When using JSONtext, not only the Enum datastrings are lost, but also the datatype is changed (from unsigned long to signed long)

I also tested "JSON API" with the same data, and that worked just fine.

Am I doing something wrong?

Edited June 15, 2018 by SGI

[drjdpowell]

What are you trying to do?   Are you trying to load "Page 2" of your tab control?  Because the number 1 will be converted by "Variant to Data" to your "Page 2".

With JSONtext, I placed speed over making the Variants perfect.  Variants are a means to an end.  I did the fastest, lowest-overhead way that will produce a Variant that can be converted to your datatype correctly.   For an enum, an integer converts correctly and is probably an order of magnitude faster than what JSON API does.

Why are you using the Variant version of that VI, anyway, rather than the VIM (which hides the variant stuff away entirely)?

 

 

 

[silmaril]

JSONtext is really a great help in our work - thank you very much, James!

I was missing a function to merge JSON objects together recursively (since "Merge into Object.vi" is only considering one level of the data structure), so I wrote a VI that does this (see attached VI).

I am sure this isn't the most efficient way to do this, so I am open for suggestions to improve the code.

Would you consider adding this functionality to JSONtext?

 

For anyone wondering why one would need this kind of merge function: Our use case is a modular configuration file structure, in which a complex configuration resides in one large JSON object. Each JSON file may contain references to include other JSON files into the current configuration. This enables us to structure the test configuration for about a hundred similar but different items in a reasonable way. Even if wrote "large", the JSON files are still small enough that the performance of the merge function is no issue for us.

JSON_Merge_Recursively.vi

[drjdpowell]

Definitely on my list of JSONtext additions.  Ideally, I would want to implement RFC 7396: JSON Merge Patch, with the ability to generate a "patch" of the differences between two JSON texts, then apply the patch to one text to get the other. 

[silmaril]

I wasn't aware of this RFC - thanks for the link!

One interesting aspect: It uses null-values to delete items, which means we should write our software to behave in the same way for a null-item and for a nonexisting item.

In some cases this might leave us with the problem of how to correctly save a NaN value.

 

[SGI]

I have problems converting the data back from JSON to the correct variant data type, even when using the VIM instead of the VI and using 'Variant to data'.

At the moment, I have no alternative for using variants as my data type.

 

Any other suggestions for making this work?

JSONtext - issues reading variant data type from JSON.vi

[drjdpowell]

My questions remain the same as back on June 15: what are you trying to do?

[SGI]

I'm trying to use JSON to store control and indicator values of which I don't know the datatype.

I use dynamic VI calling, I set controls using VI Reference -> Set Control values, run the VI, and get the indicators using 'Get Control values'; those need an give back variants, which I want to load from JSON and store back to JSON.

 

Edited July 3, 2018 by SGI

[drjdpowell]

I think you may be misunderstanding how "Variant to Data" works and what it is for.  Providing a type inside a Variant to the top input does nothing; instead you need to connect the actual type.  It converts variants into a specific data type.   You don't need "Variant to Data" at all for your Controls, as they stay as Variants.  And names never matter; if you get the data structure right, it will work.  "Variant to Data" and "Set Control Value" both ignore that actual names in the data.  

I do Controls to/from JSON all the time, so I will try and dig up an example.

[drjdpowell]

An example (using and array of control references, but you can make Get/Set Control Values work similarly):

To JSON:

From JSON: