[CR] JSON LabVIEW

[ShaunR]

  On 12/4/2015 at 3:00 PM, drjdpowell said:

Sorry, forgot to push the changes.  It’s there now.

 

I was just checking to see if it was still the main repository. Can we put a link on the CR page?

Edited December 4, 2015 by ShaunR

[drjdpowell]

Here is a beta version with changes aimed at performance, particularly for large arrays of data.  I managed to eliminate an O(N^2) performance issue with parsing large JSON strings.  I also improved the Variant tools’ speed with one- and two-dimensional arrays.   See the new example “Example of an Array of XY arrays.viâ€.

 

lava_lib_json_api-1.4.1.33.vipFetching info...

 

Please give this a test and let me know of any issues before I place it in the CR.   I’m also thinking of submitting to the LabVIEW Tools Network.

 

— James

[ensegre]

Coming to this thread with my monstruous configuration which is a blurb of about everything including array of classes with children, which so far I dumped as 50KB+ xml, and starting to have cursive looks at the Lineator too, I'd like to reask the same question of https://lavag.org/topic/16217-cr-json-labview/page-6#entry109240 (2nd part, serialize classes): what exactly do I have to do in order to dump my monster as json and viceversa? Create serializable.lvclas parent of all my other classes, perhaps, with two methods - doing exactly what? Is there an example?

 

Also, suppose the monster still mutates, parameters being added or moved: what recommendations would you give in order to maximise the information which can still be retrieved from an out-of-sync configuration dump?

 

TIA, Enrico

[drjdpowell]

  On 12/20/2015 at 11:15 AM, ensegre said:

what exactly do I have to do in order to dump my monster as json and viceversa? Create serializable.lvclas parent of all my other classes, perhaps, with two methods - doing exactly what? Is there an example?

 

The current JSON-Variant tools will store objects as flattened strings.  Not human-readable, but flattened objects have a mutation history so you can freely rearrange things inside the class private data (but not on any condition re-namespace the class).  I find that mixed-mode quite useful and easy.

 

Alternately, to serialize objects as JSON I usually just have methods in the class to convert to/from JSON and I use them as I build up the JSON explicitly (i.e. without using the shortcut of the JSON-Variant tools on a cluster).  

 

There is a possibility that one could make one’s LVOOP classes children of “JSON Valueâ€, and then override “flatten†and “unflattenâ€.  Then your objects would serialize to JSON even inside a cluster using the JSON-Variant tools (those tools recognize “JSON Value†objects and flatten/unflatten them).  But there is a technical issue that I have to look into to make that work. [Never mind, this won’t work.]

 

  On 12/20/2015 at 11:15 AM, ensegre said:

Also, suppose the monster still mutates, parameters being added or moved: what recommendations would you give in order to maximise the information which can still be retrieved from an out-of-sync configuration dump?

 

Adding or removing is fine, but don’t rename or move things, unless you want to have a “schema version†attribute and mutation code than coverts from older schema.

[ensegre]

  On 12/21/2015 at 9:23 AM, drjdpowell said:

Alternately, to serialize objects as JSON I usually just have methods in the class to convert to/from JSON and I use them as I build up the JSON explicitly (i.e. without using the shortcut of the JSON-Variant tools on a cluster).  

 

There is a possibility that one could make one’s LVOOP classes children of “JSON Valueâ€, and then override “flatten†and “unflattenâ€.  Then your objects would serialize to JSON even inside a cluster using the JSON-Variant tools (those tools recognize “JSON Value†objects and flatten/unflatten them).  But there is a technical issue that I have to look into to make that work.

This is the way I'm currently exploring for flattening, and while coding it, it seems to work:

1. make my top device class a child of JSON Object (not Value)
2. for each descendant class, create an override Flatten.vi like this:

 

The first subVI in the chain reads the relevant configuration parameters of my Heater class, and bundles them in a cluster. I just leverage on previous VIs I coded for the configuration UI of the application.

 

For a more complex class, the extension of the exercise seems ok as well:

 

 

(broken wires here in the snippets are only due to the missing class context)

 

Do you see drawbacks?

[drjdpowell]

  On 12/21/2015 at 9:47 AM, ensegre said:

Do you see drawbacks?

Problems show up in the unflattening part.  Flattening, the part you are trying, is strait forward, because you are going from strongly-typed LabVIEW to weakly-typed JSON.  The other way is harder.

 

You don’t need to make your class a child of JSON Object; just have your class have “To JSON†and “From JSONâ€.  You won’t be able to use these objects in the JSON-Variant tools, but you will be able to handle things with the other methods.  This involves more coding, but is faster.  And it allows one to get away from the monster config cluster and towards a distributed config, where different code modules supply different parts of the config.

[CharlesB]

Here's what I've done for this in my implementation.

 

I had the same need, big configurations with nested objects that needed to be serializable. I created a "JSON serializable" class that has "Data to JSON" and "JSON to data" that take JSON values as arguments. My classes inherit from it and de/serialize their private data by overriding these methods, with the option of outputting class name along with the data. JSON.lvlib is modified to handle these classes on get/set JSON. I used the @notation to output the class name, resulting in this kind of output

"Timings": {
  "@type": "Timings",
  "@value": {
    "Exp time": 0.003,
    "Piezo": {
      "Modulation shape": "Square",
      "Nb steps": 4
    }
  }
}

This is the code for deserializing and serializing:http://imgur.com/a/afaKj

 

The drawback is that I need to modify JSON.lvlib, and I didn't take the time to update my fork to stay inline with the development of JSON library. Also it needs LabVIEW 2013, because of the need of "Get LV Class Default Value By Name".
The advantage is that the class name is serialized, and loaded back at deserialization. It allows to save/load child classes of a serializable class, giving more flexibility to the configuration.

 

If someone is interested I can work on updating the fork with last version of JSON.lvlib and publish it

Edited December 21, 2015 by CharlesB

[drjdpowell]

  On 12/21/2015 at 10:18 AM, CharlesB said:

The drawback is that I need to modify JSON.lvlib, and I didn't take the time to update my fork to stay inline with the development of JSON library. 

I could easily make a “JSON serializable†class with abstract “Data to JSON†and “JSON to Data†methods, and have the JSON-Variant tools use them.  Then you could inherit off this class and not have to modify JSON.lvlib.

[ShaunR]

  On 12/21/2015 at 10:09 AM, drjdpowell said:

Problems show up in the unflattening part.  Flattening, the part you are trying, is strait forward, because you are going from strongly-typed LabVIEW to weakly-typed JSON.  The other way is harder.

 

You don’t need to make your class a child of JSON Object; just have your class have “To JSON†and “From JSONâ€.  You won’t be able to use these objects in the JSON-Variant tools, but you will be able to handle things with the other methods.  This involves more coding, but is faster.  And it allows one to get away from the monster config cluster and towards a distributed config, where different code modules supply different parts of the config.

 

The easiest way by far is to use the [un]Flatten to XML primitives, which can take objects and variants as arguments, and convert the XML to/from JSON. with simple regex/search/replace. Why they didn't make the native JSON ones like the XML is beyond me

Edited December 21, 2015 by ShaunR

[ensegre]

  On 12/21/2015 at 10:09 AM, drjdpowell said:

Problems show up in the unflattening part.  Flattening, the part you are trying, is strait forward, because you are going from strongly-typed LabVIEW to weakly-typed JSON.  The other way is harder.

Right, now that I turn to it, I see it. Harder, but perhaps not insurmountable: if I get it correctly, the showstopper is that there is no way to recast the saved class data back into classes by merit of some autodynamical mechanism; the hard way should be to include enough class-discriminating information in the JSON, and supplement a lot of ad hoc fragile code to place it back where needed. Anyway, tedious enough to wonder if it is really preferrable to my original xml dump.

 

 

  ShaunR said:

Why they didn't make the native JSON ones like the XML is beyond me

and me too.

[Stobber]

  On 12/21/2015 at 3:09 PM, ensegre said:

Right, now that I turn to it, I see it. Harder, but perhaps not insurmountable: if I get it correctly, the showstopper is that there is no way to recast the saved class data back into classes by merit of some autodynamical mechanism; the hard way should be to include enough class-discriminating information in the JSON, and supplement a lot of ad hoc fragile code to place it back where needed. Anyway, tedious enough to wonder if it is really preferrable to my original xml dump.

 

I think you mean "objects" everywhere you say "classes". To that end, I once created an "INI Serializable" class that did what jdpowell suggests. In addition to providing overridable protected methods for "Serialize" and "Deserialize" actions, it had infrastructure to generate a GUID for the object to be serialized and all the objects that composed it. The objects' GUIDs were the names of sections in the INI file, and their private data were written to keys in that section. A key was added for each object composing the one being written, and the composing object's GUID was assigned to that key. In this way, the Deserialize method could build up an object's simple-datatype members, then look up any sub-objects by their GUID, build those up, and insert them into the owning object.

 

i.e. the output of "Child of INI Serializable.lvclass:Serialize.vi" would be:

# Flattened object

[5FF01668-0B7A-4EAF-9CF7-B09754EB4A07]
__class = "Child of INI Serializable.lvclass"
__object1 = "64F7B537-BC57-48BE-B425-8C4197D51925"
__object2 = "79C1503B-4745-41FF-B3D9-3B2DCD3B079D"
number = 1.0
string = "Hello World"

[64F7B537-BC57-48BE-B425-8C4197D51925]
__class = "Behavioral Strategy.lvclass"

[79C1503B-4745-41FF-B3D9-3B2DCD3B079D]
__classs = "IO API.lvclass"
visa name = "ASRL2::INSTR"

I wish I'd kept the code, but it's lost to the ages. You could recreate it in a day and probably add stuff for handling variants in the same way, since they amount to key-value dictionaries anyway.

 

 

Edit: This still required that every serializable class override the serialize and deserialize actions to manually output a VCluster of their private data and later accept that VCluster, parse it, and build up its own private data again.

Edited December 21, 2015 by Stobber

[drjdpowell]

Note: I’m thinking of putting in an abstract parent class with “To JSON†and “From JSON†methods, but I’m not going to specify how exactly objects will be serialized.  So Charles can override with his implementation (without modifying JSON.lvlib) without restricting others from creating classes than can be generated from different JSON schema.   I could imagine use cases where one is generating objects from JSON that is created by a non-LabVIEW program that one cannot modify, for example.   Or one just wants a different format; here’s a list of objects from one of my projects:

[
["SwitchRouteGroup",{"Connect":"COMMS1"}],
["DI Test",{"IO":"IO_ID_COMMS_1_PRESENT","ON=0":false,"Value":true}],
["Set DO",{"IO":"IO_ID_COMMS_1_EN","ON=0":true,"Readable?":true,"Value":false}],
["Wait Step",{"ms":2000}],
["DMM Read Voltage",{"High":2,"Low":-2}],
["Set DO",{"IO":"IO_ID_COMMS_1_EN","ON=0":true,"Readable?":true,"Value":true}],
["Wait Step",{"ms":500}],
["DMM Read Voltage",{"High":13,"Low":11}],
["AI Test",{"High":3600,"IO":"IO_ID_COMMS_1_V_SENSE","Low":3400}],
["SwitchRouteGroup",{"Disconnect":"COMMS1"}]
]

[Stobber]

  On 12/22/2015 at 7:30 AM, drjdpowell said:

 

Note: I’m thinking of putting in an abstract parent class with “To JSON†and “From JSON†methods, but I’m not going to specify how exactly objects will be serialized.  So Charles can override with his implementation (without modifying JSON.lvlib) without restricting others from creating classes than can be generated from different JSON schema.   I could imagine use cases where one is generating objects from JSON that is created by a non-LabVIEW program that one cannot modify, for example.   Or one just wants a different format; here’s a list of objects from one of my projects:

 

Caution: a framework keeps all the assumptions but provides none of the code.

 

If you don't handle a lot of the work for the user (making assumptions about how to do so), I don't see what help your new classes will be. All you'll do is give people the opportunity to make your code package a critical dependency of their project. If anything, I recommend using the Strategy pattern to provide a default Serialization Behavior that people can override with their own.

[drjdpowell]

Oh, I’m not even talking about a framework.  Just a hook that allows someone to connect such a thing to the JSON-Variant tools without modifying JSON.lvlib.  Then CharlesB could provide his object serializer as an addon library.  

[ShaunR]

  On 12/22/2015 at 2:22 PM, drjdpowell said:

Oh, I’m not even talking about a framework.  Just a hook that allows someone to connect such a thing to the JSON-Variant tools without modifying JSON.lvlib.  Then CharlesB could provide his object serializer as an addon library.

 

Is this really a box you want to open? You can't inherit from multiple parents and the Character Lineator is designed for this.

Edited December 22, 2015 by ShaunR

[Stobber]

  On 12/22/2015 at 4:22 PM, ShaunR said:

 

Is this really a box you want to open? You can't inherit from multiple parents and the Character Lineator is designed for this.

 

I haven't looked at the Character Lineator, but my impression from forum chatter is that it's a monster that runs slowly. Is it worth use in production software these days?

 

That said, I still don't know whether jdpowell's suggestion is the right way to implement an app-specific ser/des protocol. I feel like that's a different code module than this one, maybe using this one as a dependency with the existing API.

[smithd]

  On 12/22/2015 at 7:30 AM, drjdpowell said:

 

Note: I’m thinking of putting in an abstract parent class with “To JSON†and “From JSON†methods, but I’m not going to specify how exactly objects will be serialized.  So Charles can override with his implementation (without modifying JSON.lvlib) without restricting others from creating classes than can be generated from different JSON schema.   I could imagine use cases where one is generating objects from JSON that is created by a non-LabVIEW program that one cannot modify, for example.   Or one just wants a different format; here’s a list of objects from one of my projects:

I dont have the code in front of me so forgive me if I'm off a bit, but I think there are two options here and I'd like to be sure which you're proposing:

-To Json which produces a raw string

-To json which produces a json.lvlib object, which the library then flattens in the usual way into a string.

I'd prefer the second option myself...is that what you're going for?

 

 

  On 12/22/2015 at 4:55 PM, Stobber said:

I haven't looked at the Character Lineator, but my impression from forum chatter is that it's a monster that runs slowly. Is it worth use in production software these days?

Yeah. Its probably OK on desktop but its realllly slow on RT and probably not suitable for any code which is run regularly. 

Edited January 5, 2016 by smithd

[drjdpowell]

  On 1/5/2016 at 2:42 AM, smithd said:

-To Json which produces a raw string

-To json which produces a json.lvlib object, which the library then flattens in the usual way into a string.

I'd prefer the second option myself...is that what you're going for?

The second option.

 

PS> I’m a bit stuck on this at the moment because of a problem with “Variant to Data†being too “dumb†when it comes to child classes.   If one has a Variant containing a Parent-class datatype holding a child-class object, and you attempt to use "Variant to Data" to cast this to a Child-class wire, it throws a type mismatch error, even though such a conversion is easily possible.  This is a problem when the library-user want to use clusters containing their own child classes.  There are a couple of work around but both are ugly.

[smithd]

what are your workarounds? the thing that comes to mind is always do var->lvobj first, then lvobj->child class, but thats could certainly get ugly. Where specifically is this coming up in the code?

[drjdpowell]

  On 1/7/2016 at 4:44 AM, smithd said:

what are your workarounds? the thing that comes to mind is always do var->lvobj first, then lvobj->child class, but thats could certainly get ugly. Where specifically is this coming up in the code?

 

It come up in JSON to Variant, where the User of the library provides a target cluster containing their custom child classes.   I need to provide a Variant that can be converted using Variant-to-Data into their cluster.  My workarounds are:

1) require the User to override a “Put in Variant†dynamic dispatch method.  Very simple, just puts the object in a Variant with the right wiretype, but a burden on the User. 

2) get as LVObj; flatten to string; unflatten using the type descriptor from the supplied child object.   This allows one to ‘transfer’ the wiretype into the Variant.  But the flattening/unflattening is heavy overhead.  

[FixedWire]

Not sure if this is the best thread...

 

I'm looking to use the JSON implementation to manipulate STIL files (IEEE 1450).

 

Is an extension of the JSON.lvlib a good use case for this implementation?

 

The format looks like this for SPI (serial peripheral interface bus):

 

Signals {

                //My SPI Signal - Direction is Relative to Slave

                CLK      In;

                MOSI     In;

                MISO     Out;

                CS       In;

} // end

 

DCLevels SPI_Levels {

                CLK        {VIH '3.3V'; VIL '0V'; }

                MOSI      {VIH '3.3V'; VIL '0V'; }

                MISO      {VIH '0V'; VIL '0V'; VOH '2.0V'; VOL '1.8V'; }

                CS          {VIH '3.3V'; VIL '0V';}

}

 

 

Any suggestions on the best way to do this? Not using STIL is not an option.

 

Thanks!

[Stobber]

You want to manipulate STIL data using a JSON API? Sounds like turning a screw with a hammer...

 

Can you get your vector generator (or whatever test equipment you're using) to output STDF instead?

[FixedWire]

Yes Stobber, I hear you. The test equipment needs the STIL format & I need a way to both build and manipulate parameters. e.g. change the MOSI max voltage level. JSON was a thought and I'm open to suggestions. Haven't seen any vector generator source files yet... 

Edited January 19, 2016 by FixedWire

[Stobber]

You should go sniffing around for a STIL API. If there's none native to LV, look for one in another language that can be built into a DLL (C++) or called from the command line (Python).

[CharlesB]

  On 1/5/2016 at 12:24 PM, drjdpowell said:

PS> I’m a bit stuck on this at the moment because of a problem with “Variant to Data†being too “dumb†when it comes to child classes.   If one has a Variant containing a Parent-class datatype holding a child-class object, and you attempt to use "Variant to Data" to cast this to a Child-class wire, it throws a type mismatch error, even though such a conversion is easily possible.  This is a problem when the library-user want to use clusters containing their own child classes.  There are a couple of work around but both are ugly.

 

What I do in my implementation is to serialize the class type, and if present when reading it back I instantiate the real class, using LabVIEW's "Get LV Class Default Value By Name". It works neatly in my case. Serializing class type is an option given to the "Data to JSON" VI, so classes that don't have children don't get their type serialized.

 

Serialized class looks like this:

{
  "Timings": {
    "@type": "ChildTimings",
    "@value": {
      "Camera delay": 0,
      "Exp time": 0.001,
      "Another field": "3.14159"
     },
  "Name": "Charles"
}

Attached my version of the package implementing this (don't pay attention to the version number, it was forked from 1.3.x version). Code is in JSON Object.lvclass:Set Class Instance.vi and JSON Object.lvclass:Get as class instance.vi

 

Hope you won't find it ugly :-)

 

lava_lib_json_api-1.4.0.26.vipFetching info...

Edited January 20, 2016 by CharlesB