Un-flatten from JSON to Array of cluster with contained variant loses data

[AndyS]

Hi!

I have to convert a dynamically generated array into a JSON string and back. Unfortunately I found that the un-flatten method loses the variant data. See the screenshot of FP and BD and the comments inside.

JSON_Text_test.vi

 

 

Is this a bug in JSON Text or is my data-construction not supported as expected? In case of the letter I have modify huge parts of my code. So I hope that it is a bug 😉

 

The 2nd thing I recognized is that the name "Value" of the cluster is not used during flatten. Instead the name of the connected constant / control / line is used. I found the green VI ("Set Data Name__ogtk.vi") at OpenG Toolkit that allows me to programmatically set the variant data name. As you can imagine I would prefer not to need the OpenG VI.

 

Thanks in advance for your kind help 🙂

 

Edited December 8, 2021 by AndyS

[AndyS]

Sorry, if you're wondering about the variant to data ... I did a mistake. In my project code it looks like in the following screenshot. But the result / problem stays the same:

JSON_Text_test.vi

Edited December 8, 2021 by AndyS

[LogMAN]

27 minutes ago, AndyS said:

Is this a bug in JSON Text or is my data-construction not supported as expected?

27 minutes ago, AndyS said:

The 2nd thing I recognized is that the name "Value" of the cluster is not used during flatten. Instead the name of the connected constant / control / line is used.

Here is a similar post from the CR thread. The reason for this behavior are explained in the post after that.

JSONtext essentially transforms the data included in your variant, not the variant itself. So when you transform your array into JSON, the variant contains name and type information. But this information doesn't exist when you go the other way around (you could argue that the name exists, but there is no type information). The variant provided to the From JSON Text function contains essentially a nameless element of type Void. JSONtext has no way to transform the text into anything useful. To my knowledge there is no solution to this.

The only idea I have is to read the values as strings, convert them into their respective types and cast them to variant manually.

Edited December 8, 2021 by LogMAN

[drjdpowell]

I would encourage you to try the "subJSON" described in the conversation LogMAN linked to.  Basically, replace your "Value" variant with a string labelled "<JSON>Value", and convert your values to JSON (at all the places where you now have "Set Data Name__ogtk.vi").  Then, wherever you are converting your variants to strict type, instead convert the subJSON to that type.

The problem with using Variants and JSONtext is that the JSON doesn't contain the specific type information needed to recover the data in the Variant.

 

[AndyS]

On 12/8/2021 at 4:02 PM, LogMAN said:

JSONtext essentially transforms the data included in your variant, not the variant itself. So when you transform your array into JSON, the variant contains name and type information. But this information doesn't exist when you go the other way around (you could argue that the name exists, but there is no type information). The variant provided to the From JSON Text function contains essentially a nameless element of type Void. JSONtext has no way to transform the text into anything useful. To my knowledge there is no solution to this.

Ok, it took a while but yes, you're right. How can LabVIEW know the data type of a variant if the only you offer is the value information. It can't.

But: JKI JSON can ... I tested it shortly and somehow they are able to reconstruct the variant. I didn't had the time to went deeper into the code but I assume that they take a closer look to the data. E.g. everything that comes in " " must be a string. Everything without must be a number, enum, bool, ... every number with a . included must be a floating point. That's my assumption without seeing the code. I also assume that the input variant and the output variant might be not exactly the same if there are unusual data types used.
The problem on JKI JSON is the low speed in comparison to JSON text - so I would highly prefer to go on using JSON text. I'm not sure if there is a way to combine the best of both worlds but it would be really cool.

Edited December 9, 2021 by AndyS

[drjdpowell]

I believe JKI JSON converts to Variants containing the basic JSON-matching types: String, Boolean, Number (DBL, I'm guessing).  Then it also provides, if I remember, a special VI to convert that Variant into your cluster of actual specific data types.

One of the reasons I don't like that is that it ties things to a monolithic LabVIEW type, the Cluster, that is not as flexible as JSON Object/Arrays.  You show the same "LabVIEW tunnel vision" in your design of an array of "Group", "Name", "Value" clusters.  That's a common way to handle the limits of LabVIEW.   

In JSON, the natural structure is this:

{
"Basic": 
   {
     "config file":"c:\\test\\config.ini",
     "Operator:"Muller"
   },
"Extended": 
   {
     "Room Temp":21.4,
     "Test No":0,
     "ping":null
   }
}

That isn't really doable with your LabVIEW Arrays and Clusters, even if I give you some extra Variant support.  But you have tools to do stuff like this in JSONtext.  

[AndyS]

On 12/8/2021 at 5:33 PM, drjdpowell said:

I would encourage you to try the "subJSON" described in the conversation LogMAN linked to.  Basically, replace your "Value" variant with a string labelled "<JSON>Value", and convert your values to JSON (at all the places where you now have "Set Data Name__ogtk.vi").  Then, wherever you are converting your variants to strict type, instead convert the subJSON to that type.

The problem with using Variants and JSONtext is that the JSON doesn't contain the specific type information needed to recover the data in the Variant.

 

Ok, I implemented it like suggested and it's working fine. Nevertheless I'm semi-happy with this solution because I have now a dependency to JSON text in a toolkit where I havn't had it before.

One final question for my understanding (see the following picture). In opposite to the screenshot of my first post, I connect now the original array with all variant data information to "From JSON Text.vim". Nevertheless the value can still not be reconstructed. Why?

[Rolf Kalbermatter]

You still have the problem of the embedded variant in your datatype. JSON is inherently string format with some indications to make a distinction between numbers and strings but nothing more. A variant is an AnyType.

What you try to do is: Convert this string (which from the JSON syntax only can be inferred to be a numeric, object, list or string, into Anything! Anything what? The JSON library could elect to convert strings into a string, numerics into a double and anything else into nothing, but that will break down as soon as you want to use that variant as a specific type as you pushed it into the incoming variant in the begin. The JSON string simply lacks every information about the actually wanted type in the variant and therefore you can not convert it into a variant without loosing significant compatibility and functionality. This is a fundamental problem that can not be solved with throwing more code at it, but only by rethinking the way you want to save your configuration data.

Edited December 15, 2021 by Rolf Kalbermatter

[ShaunR]

46 minutes ago, Rolf Kalbermatter said:

This is a fundamental problem that can not be solved with throwing more code at it, but only by rethinking the way you want to save your configuration data

The way I usually get around this is either by not using variants (preferred) or saving as a string and specifying the type as another entry. e.g.

{

"config_file:"c:\temp\config.ini",

"config_file~type":"str",

"value":1.0E6,

"Value~type":"ext"

}

This can be transparent to the user but not applicable if you have a strict field list for a file. Thank [insert deity] we can switch on strings in a case structure, eh?

[drjdpowell]

On 12/15/2021 at 7:48 AM, AndyS said:

Nevertheless the value can still not be reconstructed. Why?

The Type input just sees "array of clusters".  Normally one just inputs an empty array; there is no code to look at individual elements of the array.  Theoretically, this could be done, but that seems a very low-value use case so chances of me doing that are zero.

Actually, it was a mistake of mine to implement the first part of your code; doing it again I would convert Variants in Clusters to 'null' and throw a "Variants not supported" error.  This would keep my options open to introduce, without breaking changes, a better Variant support that includes encoding teh data type (along the lines of what Shaun describes).  There is a number of possibilities, for example:

{
  "config file:["c:\temp\config.ini",{"LVtype":"Path"}],
  "Room Temp":["21.4",{"LVtype":"DBL"}]
}

I am held back from that, partly because it would now be a breaking change, and partly because I have yet seen an example of someone using Variants and JSON that would not be far better implemented in just JSON.  

[ShaunR]

11 hours ago, drjdpowell said:

{
  "config file:["c:\temp\config.ini",{"LVtype":"Path"}],
  "Room Temp":["21.4",{"LVtype":"DBL"}]
}

That is far more difficult to implement.

consider this-an array in a variant and a normal array:

{
"temps":[21.4, 23.2, 33.6],
"press":[1006];
}

LabVIEW arrays are single types so this is straight forward.

{
"temps":[21.4, 23.2, 33.6],
"temps~lvtype":"dbl",
"press":[1006]
}

You don't need to pars the array string ("temps") any more than you do now.

Including the type in the array, you now have to inspect the elements of the array (at least one, if the format is the first or last element) parse it then delete it to return the array. If there are only two elements in the array, you have to see if there is a lvtype to decide if it is really an array or it's a scaler with a type. 

Edited December 16, 2021 by ShaunR

[drjdpowell]

There are a lot of possible options for encoding Variants, all with a different set of advantages and disadvantages.  Without a clear use case, it is difficult to see what the best choice would be.