[CR] JSONtext

[Aristos Queue]

@ShaunR "Didn't that do JSON?"
Yes. Initially I rolled my own JSON parser, but it was rewritten to call JSONText as subVIs. The Character Lineator is a better framework (my opinion) for importing and exporting objects into/out of JSON than trying to hook the serializer directly. Raw JSON doesn't provide any schema version management nor any data massaging capacities. It's just fields without context. The JSONText library is important for the raw string handling. You need more infrastructure sitting above it to actually use JSON. CL is one possible choice for that infrastructure. The serializer hook that was added to JSONText in its last revision allows users to roll their own infrastructure.

[drjdpowell]

Alternative (6): all attributes are returned as JSON values.  Rather than get the attribute as the actual type, the User would get the attribute as a string, then use From JSON to convert to type. 

One could make VIMs that combine these two steps, and these VIMs might possibly be made to work with non-JSON attributes as well.

[ShaunR]

  On 7/21/2021 at 8:17 PM, Aristos Queue said:

it was rewritten to call JSONText as subVIs.

Expand  

A paean to the utility of JSONText.

Edited July 22, 2021 by ShaunR

[Aristos Queue]

  On 7/22/2021 at 2:32 PM, ShaunR said:

A paean to the utility of JSONText.

Expand  

Kind of. The JSONText VIs are more efficient overall than my originals that I slapped together in a weekend. But the current JSONText VIs do have some inefficiencies that my originals didn’t have (most notably, see the earlier post about accessing all fields of an object) that crop up when translating schemas. And the single  serializer hook is insufficient to do some operations — support for sets/maps or custom translations of complex data structures, for example. These are things that the CL architecture accounts for that I wish JSONText was refactored to support. Having said that, I’m happy enough for my own projects with the existing toolkit and unwilling to donate the refactoring time. 
 

It is definitely a good toolkit. But it still has a long way to go in my opinion. 

[drjdpowell]

  On 7/23/2021 at 12:14 AM, Aristos Queue said:

support for sets/maps

Expand  

There is support for Sets and Maps in the latest version; I wouldn't want such things require any kind of "hook".

[drjdpowell]

  On 7/23/2021 at 12:14 AM, Aristos Queue said:

or custom translations of complex data structures

Expand  

I hope Users can do whatever they want using the lower-level functions plus subJSON with the <JSON> tag.

 

[drjdpowell]

  On 7/23/2021 at 12:14 AM, Aristos Queue said:

But the current JSONText VIs do have some inefficiencies that my originals didn’t have (most notably, see the earlier post about accessing all fields of an object)

Expand  

The Object thing is not where I would spend limited effort in performance improvements, as I would expect the User to actually use the item Values and thus require a copy at some point.   I would rather improve the JSONpath search algorithm.

[ShaunR]

  On 7/23/2021 at 10:51 AM, drjdpowell said:

I hope Users can do whatever they want using the lower-level functions plus subJSON with the <JSON> tag.

Expand  

Indeed. I don't know what AQ is referring to with CL (Command Line? Common Lisp?) but I have a feeling he has an eye on the library supporting JSON Schema-another IETF brain-fart which made HTML a nightmare until they ditched it and brought us XML. They have a habit of taking nice simple solutions and "formalising" them them into bloated, complicated solutions that nobody uses.

  On 7/23/2021 at 12:14 AM, Aristos Queue said:

But it still has a long way to go in my opinion.

Expand  

I think the library is pretty much feature complete at this point-maybe some data types missing but can't think of any off the top of my head - so this sounds peculiar...unless JSON schema is your target.

Remember. Many of us only use toolkits like these because NI refused to make the native JSON primitives' fit-for-purpose.

[Francois Normandin]

  On 7/23/2021 at 11:20 AM, ShaunR said:

Indeed. I don't know what AQ is referring to with CL

Expand  

AQ's Character Lineator

 

[Aristos Queue]

  On 7/23/2021 at 11:20 AM, ShaunR said:

Remember. Many of us only use toolkits like these because NI refused to make the native JSON primitives' fit-for-purpose.

Expand  

Dont misunderstand me: I think it’s a great toolkit, and I think it should be a G toolkit not a language primitive. I’m just finding it’s edge cases. 🙂

[ShaunR]

  On 7/23/2021 at 11:28 PM, Aristos Queue said:

I think it’s a great toolkit, and I think it should be a G toolkit not a language primitive.

Expand  

Possibly in this case as it has searching. But we wouldn't have to write parsers if the current primitives were useful 

I don't know what the current status of the G toolkit is, since the last person that took over maintenance and release is no longer active on this forum.

  On 7/23/2021 at 11:28 PM, Aristos Queue said:

 I’m just finding it’s edge cases. 🙂

Expand  

That's fair enough and I'm sure appreciated. However JSON Schema - if that's your target - isn't an edge case. Your Lineator would be the place to implement that, so I'm confused by the comment "long way to go" for a library that I think is pretty much feature complete.

[Aristos Queue]

Found two bugs in "Variant to JSON Text.vi".

The fixed VI is attached, saved in LV 2015. Included in this is a whole lot of manual diagram cleanup to help me follow some cases I was trying to understand. I noticed that there are several calls to the To JSON primitive that do not have the "enable LV extensions" boolean wired up. I did NOT change any of those, but they may warrant a review. There are also a few places where the error cluster is dropped that I don't think it should be.

FIRST Here's the test case for the major bug:

Expected JSON:

{"a":null,"b":null}

Actual JSON:

[null, null]

SECOND A simple miswire: the LV extensions bool was wired to the pretty print terminal in the Waveform case.

Variant to JSON Text.viFetching info...

[drjdpowell]

  On 8/4/2021 at 10:32 PM, Aristos Queue said:

FIRST Here's the test case for the major bug:

Expected JSON:

{"a":null,"b":null}

Actual JSON:

[null, null]

 

Expand  

That is arguably not a bug, as Variants involves two names: the name of the Variant, and the name of the data contained in the Variant.  In your case your empty Variants contain unnamed void data, and JSONtext flattens clusters of unnamed elements as JSON Arrays. 

It is not obvious if it is the best choice to use the data name rather than the Variant name.  However, I am sort of tied to the behaviour of the Variant-to-Data node that I use internally to convert Clusters to Arrays of Variants.  See this example:

Note that Typed clusters and Clusters of Variants containing Typed Data both convert identically, while Variant elements of a cluster never retain the names of the Variants themselves, just the name on the Data in the Variant).  The produced JSON matches these choices, made by the programmers of the Variant-to-Data node.

Edited August 5, 2021 by drjdpowell

[drjdpowell]

I've had multiple requests to "support Variants", but there is a mismatch between Variants (Data plus full type description including name) and JSON (Data with only weak type info: string, number, etc.).   To support unflattening JSON to Variant would require a form something like this:

{
  "VarA":{"Variant type":"long","name":"TypedA","Value":123},
  "VarB":{"Variant type":"long","name":"TypedB","Value":456}
}

But this is a rabbit hole I'm not going down.  I'd rather ask "Why are you using Variants here instead of subJSON?"

This gives you the behaviour you expect:

[drjdpowell]

People who wish to use Variants and JSON generally want to do this:

1. Put Static-Typed Data in a Variant in a larger data structure
2. Convert larger structure to JSON
3. Send the JSON somewhere
4. Convert JSON back to larger structure (containing Variant)
5. Convert Variant to Static-Typed Data

But Step (4) is a big problem, as at that point we don't have access to the Type-description information to rebuild the Variant.

But I say do this:

1. Convert Static-Typed Data to JSON
2. Assemble larger JSON from this subJSON
3. Send the JSON somewhere
4. Extract the subJSON
5. Convert subJSON to Static-Typed Data

This avoids the problem, as we only need the Type at the first and last steps,where we statically know the type.

[Aristos Queue]

Our use  case is nullable strings and nullable integers. We want to be able to differentiate between "this is the set value of the parameter (including possibly empty string)" and "this parameter has never been set, so defer to the default". So a cluster of variant correctly should record "null" for an empty variant and not-null for a non-empty variant.

  Quote

That is arguably not a bug

Expand  

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

[Aristos Queue]

The fix is easy -- just get the names of the cluster elements and pass into the subarray instead of relying upon the Variant To Data node to capture the names.

[ShaunR]

  On 8/5/2021 at 10:16 AM, drjdpowell said:

To support unflattening JSON to Variant would require a form something like this:

Expand  

For integers you can just reconstitute as i64 regardless. That covers most of them. A note telling the dev that this is so, means they can do things after for the edge cases. Most of the time the native coercion will cover the edge cases when the dev wires them up.

Same goes for Singles and Doubles as Extended by detecting the decimal point, lack of quotes and the E in the JSON string to differentiate them from integers.

So you end up with only 2 numeric types. I64 Integer or extended depending on the JSON format of the number.

[ShaunR]

  On 8/5/2021 at 2:33 PM, Aristos Queue said:

Recording an array when passed a cluster is definitely a bug

Expand  

+1

[drjdpowell]

  On 8/5/2021 at 2:33 PM, Aristos Queue said:

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

Expand  

LabVIEW Clusters and Arrays do not map directly onto JSON Objects and Arrays.   

• LabVIEW Clusters are fixed-size ordered sets of optionally-named values
• JSON Objects are variable-size, unordered sets of must-be-named values
• LabVIEW Arrays are variable sized ordered sets of values all the same type
• JSON Arrays are variable sized ordered sets of values that can all be different type

So a LabVIEW Cluster with unnamed elements cannot map onto a JSON Object, but can map onto a JSON Array.  Similarly, mixed-type JSON Arrays cannot be converted to a LabVIEW array, but can be converted to a LabVIEW cluster (assuming you know the number of elements and types).  JSONtext thus supports two mappings for LabVIEW Clusters, as explained in Help>>JDP Science>>JSONtext...

 

[Aristos Queue]

  Quote

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

Expand  

Allow me to rephrase then.

Recording an array when passed a NAMED cluster is definitely a bug, regardless of the behavior of the variants.

[drjdpowell]

  On 8/5/2021 at 2:33 PM, Aristos Queue said:

Our use  case is nullable strings and nullable integers. We want to be able to differentiate between "this is the set value of the parameter (including possibly empty string)" and "this parameter has never been set, so defer to the default". So a cluster of variant correctly should record "null" for an empty variant and not-null for a non-empty variant.

Expand  

Consider just not including those parameters.  Rather than {"A":123,"B":null,"C":789} just have {"A":123,"C":789}; then "B" will be default.

Alternately you could put named-Variant values inside your variants (which teh Variant-to-Data node will pass through:

Variants are quite tricky, as they can serve both as a temporary container for a value, and a value itself.

[drjdpowell]

  On 8/5/2021 at 4:19 PM, Aristos Queue said:

Recording an array when passed a NAMED cluster is definitely a bug, regardless of the behavior of the variants.

Expand  

Again, Variants are tricky, and have two names and serve two purposes.  Viewed as a container of named data, your Variants are just placeholders, not actual data themselves (which is why I suggested in the last post to put a named Variant as Data inside the placeholder).

Edited August 5, 2021 by drjdpowell

[Niatross]

I am trying to use this to convert a map with an enum as its key. It seems to convert the map to something which looks sensible but when I try and convert the JSON back to a map I am getting an error saying that the it is trying to create a map with identical keys.

I haven't had time to look into what is causing this properly, I was hoping someone had some thoughts.Enum Map.vi

[LogMAN]

  On 8/10/2021 at 10:08 PM, Niatross said:

I am trying to use this to convert a map with an enum as its key. It seems to convert the map to something which looks sensible but when I try and convert the JSON back to a map I am getting an error saying that the it is trying to create a map with identical keys.

I haven't had time to look into what is causing this properly, I was hoping someone had some thoughts.Enum Map.vi

Expand  

Here is what happens:

1. Scalar JSON text to Variant.vi uses the index output of the Search 1D Array function (I32) for the Enum value (U16). JSON to Map.vi then uses Variant To Flattened String (special).vi to extract the data. Enum U16 has 2 Bytes but I32 has 4 Bytes of data, so the Map data gets offset by 2 Bytes for each Key.
2. Scalar JSON text to Variant.vi uses the output of Get U32.vi for all unsigned integers. JSON to Map.vi then uses Variant To Flattened String (special).vi to extract the data. U16 has 2 Bytes but U32 has 4 Bytes of data, so the Map data gets offset by 2 Bytes for each value.

The solution is to cast all values according to their respective types. Here are the two offending sections for your particular case: