[CR] JSONtext

[Aristos Queue]

@ShaunR "Didn't that do JSON?"
Yes. Initially I rolled my own JSON parser, but it was rewritten to call JSONText as subVIs. The Character Lineator is a better framework (my opinion) for importing and exporting objects into/out of JSON than trying to hook the serializer directly. Raw JSON doesn't provide any schema version management nor any data massaging capacities. It's just fields without context. The JSONText library is important for the raw string handling. You need more infrastructure sitting above it to actually use JSON. CL is one possible choice for that infrastructure. The serializer hook that was added to JSONText in its last revision allows users to roll their own infrastructure.

[drjdpowell]

Alternative (6): all attributes are returned as JSON values.  Rather than get the attribute as the actual type, the User would get the attribute as a string, then use From JSON to convert to type. 

One could make VIMs that combine these two steps, and these VIMs might possibly be made to work with non-JSON attributes as well.

[ShaunR]

18 hours ago, Aristos Queue said:

it was rewritten to call JSONText as subVIs.

A paean to the utility of JSONText.

Edited July 22, 2021 by ShaunR

[Aristos Queue]

9 hours ago, ShaunR said:

A paean to the utility of JSONText.

Kind of. The JSONText VIs are more efficient overall than my originals that I slapped together in a weekend. But the current JSONText VIs do have some inefficiencies that my originals didn’t have (most notably, see the earlier post about accessing all fields of an object) that crop up when translating schemas. And the single  serializer hook is insufficient to do some operations — support for sets/maps or custom translations of complex data structures, for example. These are things that the CL architecture accounts for that I wish JSONText was refactored to support. Having said that, I’m happy enough for my own projects with the existing toolkit and unwilling to donate the refactoring time. 
 

It is definitely a good toolkit. But it still has a long way to go in my opinion. 

[drjdpowell]

10 hours ago, Aristos Queue said:

support for sets/maps

There is support for Sets and Maps in the latest version; I wouldn't want such things require any kind of "hook".

[drjdpowell]

10 hours ago, Aristos Queue said:

or custom translations of complex data structures

I hope Users can do whatever they want using the lower-level functions plus subJSON with the <JSON> tag.

 

[drjdpowell]

10 hours ago, Aristos Queue said:

But the current JSONText VIs do have some inefficiencies that my originals didn’t have (most notably, see the earlier post about accessing all fields of an object)

The Object thing is not where I would spend limited effort in performance improvements, as I would expect the User to actually use the item Values and thus require a copy at some point.   I would rather improve the JSONpath search algorithm.

[ShaunR]

3 minutes ago, drjdpowell said:

I hope Users can do whatever they want using the lower-level functions plus subJSON with the <JSON> tag.

Indeed. I don't know what AQ is referring to with CL (Command Line? Common Lisp?) but I have a feeling he has an eye on the library supporting JSON Schema-another IETF brain-fart which made HTML a nightmare until they ditched it and brought us XML. They have a habit of taking nice simple solutions and "formalising" them them into bloated, complicated solutions that nobody uses.

10 hours ago, Aristos Queue said:

But it still has a long way to go in my opinion.

I think the library is pretty much feature complete at this point-maybe some data types missing but can't think of any off the top of my head - so this sounds peculiar...unless JSON schema is your target.

Remember. Many of us only use toolkits like these because NI refused to make the native JSON primitives' fit-for-purpose.

[Francois Normandin]

38 minutes ago, ShaunR said:

Indeed. I don't know what AQ is referring to with CL

AQ's Character Lineator

 

[Aristos Queue]

12 hours ago, ShaunR said:

Remember. Many of us only use toolkits like these because NI refused to make the native JSON primitives' fit-for-purpose.

Dont misunderstand me: I think it’s a great toolkit, and I think it should be a G toolkit not a language primitive. I’m just finding it’s edge cases. 🙂

[ShaunR]

9 hours ago, Aristos Queue said:

I think it’s a great toolkit, and I think it should be a G toolkit not a language primitive.

Possibly in this case as it has searching. But we wouldn't have to write parsers if the current primitives were useful 

I don't know what the current status of the G toolkit is, since the last person that took over maintenance and release is no longer active on this forum.

9 hours ago, Aristos Queue said:

 I’m just finding it’s edge cases. 🙂

That's fair enough and I'm sure appreciated. However JSON Schema - if that's your target - isn't an edge case. Your Lineator would be the place to implement that, so I'm confused by the comment "long way to go" for a library that I think is pretty much feature complete.

[Aristos Queue]

Found two bugs in "Variant to JSON Text.vi".

The fixed VI is attached, saved in LV 2015. Included in this is a whole lot of manual diagram cleanup to help me follow some cases I was trying to understand. I noticed that there are several calls to the To JSON primitive that do not have the "enable LV extensions" boolean wired up. I did NOT change any of those, but they may warrant a review. There are also a few places where the error cluster is dropped that I don't think it should be.

FIRST Here's the test case for the major bug:

Expected JSON:

{"a":null,"b":null}

Actual JSON:

[null, null]

SECOND A simple miswire: the LV extensions bool was wired to the pretty print terminal in the Waveform case.

Variant to JSON Text.vi

[drjdpowell]

11 hours ago, Aristos Queue said:

FIRST Here's the test case for the major bug:

Expected JSON:

{"a":null,"b":null}

Actual JSON:

[null, null]

 

That is arguably not a bug, as Variants involves two names: the name of the Variant, and the name of the data contained in the Variant.  In your case your empty Variants contain unnamed void data, and JSONtext flattens clusters of unnamed elements as JSON Arrays. 

It is not obvious if it is the best choice to use the data name rather than the Variant name.  However, I am sort of tied to the behaviour of the Variant-to-Data node that I use internally to convert Clusters to Arrays of Variants.  See this example:

Note that Typed clusters and Clusters of Variants containing Typed Data both convert identically, while Variant elements of a cluster never retain the names of the Variants themselves, just the name on the Data in the Variant).  The produced JSON matches these choices, made by the programmers of the Variant-to-Data node.

Edited August 5, 2021 by drjdpowell

[drjdpowell]

I've had multiple requests to "support Variants", but there is a mismatch between Variants (Data plus full type description including name) and JSON (Data with only weak type info: string, number, etc.).   To support unflattening JSON to Variant would require a form something like this:

{
  "VarA":{"Variant type":"long","name":"TypedA","Value":123},
  "VarB":{"Variant type":"long","name":"TypedB","Value":456}
}

But this is a rabbit hole I'm not going down.  I'd rather ask "Why are you using Variants here instead of subJSON?"

This gives you the behaviour you expect:

[drjdpowell]

People who wish to use Variants and JSON generally want to do this:

1. Put Static-Typed Data in a Variant in a larger data structure
2. Convert larger structure to JSON
3. Send the JSON somewhere
4. Convert JSON back to larger structure (containing Variant)
5. Convert Variant to Static-Typed Data

But Step (4) is a big problem, as at that point we don't have access to the Type-description information to rebuild the Variant.

But I say do this:

1. Convert Static-Typed Data to JSON
2. Assemble larger JSON from this subJSON
3. Send the JSON somewhere
4. Extract the subJSON
5. Convert subJSON to Static-Typed Data

This avoids the problem, as we only need the Type at the first and last steps,where we statically know the type.

[Aristos Queue]

Our use  case is nullable strings and nullable integers. We want to be able to differentiate between "this is the set value of the parameter (including possibly empty string)" and "this parameter has never been set, so defer to the default". So a cluster of variant correctly should record "null" for an empty variant and not-null for a non-empty variant.

Quote

That is arguably not a bug

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

[Aristos Queue]

The fix is easy -- just get the names of the cluster elements and pass into the subarray instead of relying upon the Variant To Data node to capture the names.

[ShaunR]

4 hours ago, drjdpowell said:

To support unflattening JSON to Variant would require a form something like this:

For integers you can just reconstitute as i64 regardless. That covers most of them. A note telling the dev that this is so, means they can do things after for the edge cases. Most of the time the native coercion will cover the edge cases when the dev wires them up.

Same goes for Singles and Doubles as Extended by detecting the decimal point, lack of quotes and the E in the JSON string to differentiate them from integers.

So you end up with only 2 numeric types. I64 Integer or extended depending on the JSON format of the number.

[ShaunR]

22 minutes ago, Aristos Queue said:

Recording an array when passed a cluster is definitely a bug

+1

[drjdpowell]

1 hour ago, Aristos Queue said:

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

LabVIEW Clusters and Arrays do not map directly onto JSON Objects and Arrays.   

• LabVIEW Clusters are fixed-size ordered sets of optionally-named values
• JSON Objects are variable-size, unordered sets of must-be-named values
• LabVIEW Arrays are variable sized ordered sets of values all the same type
• JSON Arrays are variable sized ordered sets of values that can all be different type

So a LabVIEW Cluster with unnamed elements cannot map onto a JSON Object, but can map onto a JSON Array.  Similarly, mixed-type JSON Arrays cannot be converted to a LabVIEW array, but can be converted to a LabVIEW cluster (assuming you know the number of elements and types).  JSONtext thus supports two mappings for LabVIEW Clusters, as explained in Help>>JDP Science>>JSONtext...

 

[Aristos Queue]

Quote

Recording an array when passed a cluster is definitely a bug, regardless of the behavior of the variants.

Allow me to rephrase then.

Recording an array when passed a NAMED cluster is definitely a bug, regardless of the behavior of the variants.

[drjdpowell]

1 hour ago, Aristos Queue said:

Our use  case is nullable strings and nullable integers. We want to be able to differentiate between "this is the set value of the parameter (including possibly empty string)" and "this parameter has never been set, so defer to the default". So a cluster of variant correctly should record "null" for an empty variant and not-null for a non-empty variant.

Consider just not including those parameters.  Rather than {"A":123,"B":null,"C":789} just have {"A":123,"C":789}; then "B" will be default.

Alternately you could put named-Variant values inside your variants (which teh Variant-to-Data node will pass through:

Variants are quite tricky, as they can serve both as a temporary container for a value, and a value itself.

[drjdpowell]

15 minutes ago, Aristos Queue said:

Recording an array when passed a NAMED cluster is definitely a bug, regardless of the behavior of the variants.

Again, Variants are tricky, and have two names and serve two purposes.  Viewed as a container of named data, your Variants are just placeholders, not actual data themselves (which is why I suggested in the last post to put a named Variant as Data inside the placeholder).

Edited August 5, 2021 by drjdpowell

[Niatross]

I am trying to use this to convert a map with an enum as its key. It seems to convert the map to something which looks sensible but when I try and convert the JSON back to a map I am getting an error saying that the it is trying to create a map with identical keys.

I haven't had time to look into what is causing this properly, I was hoping someone had some thoughts.Enum Map.vi

[LogMAN]

7 hours ago, Niatross said:

I am trying to use this to convert a map with an enum as its key. It seems to convert the map to something which looks sensible but when I try and convert the JSON back to a map I am getting an error saying that the it is trying to create a map with identical keys.

I haven't had time to look into what is causing this properly, I was hoping someone had some thoughts.Enum Map.vi

Here is what happens:

1. Scalar JSON text to Variant.vi uses the index output of the Search 1D Array function (I32) for the Enum value (U16). JSON to Map.vi then uses Variant To Flattened String (special).vi to extract the data. Enum U16 has 2 Bytes but I32 has 4 Bytes of data, so the Map data gets offset by 2 Bytes for each Key.
2. Scalar JSON text to Variant.vi uses the output of Get U32.vi for all unsigned integers. JSON to Map.vi then uses Variant To Flattened String (special).vi to extract the data. U16 has 2 Bytes but U32 has 4 Bytes of data, so the Map data gets offset by 2 Bytes for each value.

The solution is to cast all values according to their respective types. Here are the two offending sections for your particular case: