[CR] JSONtext

[camow7]

I'm trying to parse this JSON data:
 

{
  "id":"fb66aaec-f713-424c-ba04-386d0986fb12",
  "planId":"4c7cbe9c-ba67-4d8b-b731-32d3de0de700",
  "name":"AJ3",
  "redrillOfHoleId":null,
  "design":{
    "angle":0.0,
    "bearing":4.376555,
    "comment":null,
    "decks":[
      {
        "number":1,
        "horizon":25.0,
        "length":5.0,
        "productId":"f3fb75ac-1065-4d9c-bd6a-fdf01c34837b",
        "weight":0.0,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":2,
        "horizon":20.0,
        "length":5.0,
        "productId":"5800171c-e16e-48e7-9431-ac9b88f4852f",
        "weight":441.786469,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":3,
        "horizon":15.0,
        "length":5.0,
        "productId":"f3fb75ac-1065-4d9c-bd6a-fdf01c34837b",
        "weight":0.0,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":4,
        "horizon":10.0,
        "length":5.0,
        "productId":"0fa2dd6d-6b0c-4bfd-998b-e0b50ee64481",
        "weight":353.429169,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":5,
        "horizon":5.0,
        "length":5.0,
        "productId":"f3fb75ac-1065-4d9c-bd6a-fdf01c34837b",
        "weight":0.0,
        "isVariable":true,
        "isBackfill":false
      },
      {
        "number":6,
        "horizon":0.0,
        "length":5.0,
        "productId":"5800171c-e16e-48e7-9431-ac9b88f4852f",
        "weight":441.786469,
        "isVariable":false,
        "isBackfill":false
      }
    ],
    "diameter":300.0,
    "length":30.0,
    "x":-194.97681394419092,
    "y":67.241148799992772,
    "z":0.0,
    "whenCreated":"2020-01-28T01:51:43.4065706Z",
    "whenUpdated":"2020-01-28T01:51:43.4065706Z"
  },
  "adjustedDesign":{
    "angle":0.0,
    "bearing":4.376555,
    "comment":null,
    "decks":[
      {
        "number":1,
        "horizon":25.0,
        "length":5.0,
        "productId":"0fa2dd6d-6b0c-4bfd-998b-e0b50ee64481",
        "weight":353.429,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":2,
        "horizon":20.0,
        "length":5.0,
        "productId":"f3fb75ac-1065-4d9c-bd6a-fdf01c34837b",
        "weight":0.0,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":3,
        "horizon":15.0,
        "length":5.0,
        "productId":"5800171c-e16e-48e7-9431-ac9b88f4852f",
        "weight":441.786,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":4,
        "horizon":10.0,
        "length":5.0,
        "productId":"f3fb75ac-1065-4d9c-bd6a-fdf01c34837b",
        "weight":0.0,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":5,
        "horizon":5.0,
        "length":5.0,
        "productId":"0fa2dd6d-6b0c-4bfd-998b-e0b50ee64481",
        "weight":353.429,
        "isVariable":false,
        "isBackfill":false
      },
      {
        "number":6,
        "horizon":0.0,
        "length":5.0,
        "productId":"5800171c-e16e-48e7-9431-ac9b88f4852f",
        "weight":441.786,
        "isVariable":false,
        "isBackfill":false
      }
    ],
    "diameter":300.0,
    "length":30.0,
    "x":-194.97681394419092,
    "y":67.241148799992772,
    "z":0.0,
    "whenCreated":"2020-01-28T02:04:07.6143372Z"
  },
  "actual":{
    "angle":null,
    "bearing":null,
    "calculatedDecks":[],
    "comment":null,
    "depthToWater":null,
    "diameter":null,
    "expectedDeckCount":null,
    "length":null,
    "accessories":[],
    "measuredDecks":[],
    "notDrilled":false,
    "userName":null,
    "x":null,
    "y":null,
    "z":null
  },
  "drillAdjustedDesign":null,
  "drillActual":null,
  "holeType":"Hole type 1",
  "materialType":null,
  "isDeleted":false,
  "isNonDesignedHole":false,
  "allowTruncatedDesign":false,
  "disableChangeDesignRules":false,
  "scaleDepthOfBurial":null,
  "strataIntercepts":{},
  "tags":["drilling_designed","loading_designed"],
  "whenCreated":"2020-01-28T01:51:43.4065706Z",
  "whenUpdated":"2020-01-28T01:51:43.4065706Z"
}

But am currently getting this error:
 

Quote

JSONtext.lvlib:JSON text to Variant.vi:4410001<ERR>
State: Parser Error
Near: ':nu'
in '"2020-01-28T02:04:07.6143372Z"},"actual":{"angle":null,"bearing":null,"calculatedDecks":[],"comment":null,"depthToWater":null,"diameter":null,"expectedDeckCount":null,"length":null,"accessories":[],"me'

<b>Complete call chain:</b>
     JSONtext.lvlib:JSON text to Variant.vi:4410001
     JSONtext.lvlib:JSON text to Variant.vi:4410003
     Parse Hole JSON.vi

 

[drjdpowell]

Post a simple vi showing the error, please.

[camow7]

@drjdpowell I have attached the vi with the cluster I'm trying to parse it too. I have put the text in as a constant. Let me know if you have trouble running it.

JSON Parsing Error.vi

[drjdpowell]

The error is coming from where you attempt to convert null (as in "comment":null) to a string.   Intuitively, one would think that a null should be equivalent to an empty string (just as when converted to a number, null becomes NaN and does not throw an error).  Thank you for the report.

[camow7]

No problem. Thanks for following it up so quickly. Until a fix is done I could precheck specific fields that may error and set them to empty strings before passing into your conversion vi.

[camow7]

@drjdpowell Also if you go through and set the comment fields to empty strings, the next parsing error hit is this:
 

Quote

Error 1 occurred at JSONtext.lvlib:JSON text to Variant.vi:4410003

Possible reason(s):

Cannot Get as integer: null

Complete call chain:
     JSONtext.lvlib:JSON text to Variant.vi:4410003
     JSONtext.lvlib:JSON text to Variant.vi:4410002
     JSON Parsing Error.vi

This is replicated using the same VI shared above by first setting the comment keys to actual strings

[drjdpowell]

That one is trickier.   What is null supposed to mean, interpreted as an integer?   Unlike for Floats, there is no Not-an-Integer value, nor is there an obvious "null" value like an empty string.   Should one just use zero as null?   Or should one consider the null to be that same as if the item was not present at all in the JSON, those using your default value in the supplied cluster.  Why is whatever generated this JSON providing null values at all?  Especially in place of strings and integers?

[drjdpowell]

13 hours ago, camow7 said:

Until a fix is done I could precheck specific fields that may error and set them to empty strings before passing into your conversion vi.

A possibly better option is to replace your string/integer values with JSON strings, which is LabVIEW strings with <JSON> at the start of the name.  So "<JSON>Comment" rather than "Comment".   JSONtext will happily parse out the value into your cluster as JSON, then when you actually need your value you can convert it to string/integer, with appropriate handling if it is null or throws an error.

This method, by the way, is also how to handle JSON that has variable structure.

[ShaunR]

2 hours ago, drjdpowell said:

Why is whatever generated this JSON providing null values at all?  Especially in place of strings and integers?

I imagine it is C , C++ or Javascript. NULL is a specific type of invalid pointer in C/C++ languages and "not an object" in Javascript (as opposed to "undefined"). In LabVIEW we don't really have either concepts.

In JSON, it is a valid type so it depends on how you want to translate it back into a LabVIEW type. Historically I have converted in the following manner: string->empty, numeric->0, boolean->false etc. Unless, of course, it is in quotes. In that case it is the string "NULL", in whatever case, as it's implicity typed.

Edited January 31, 2020 by ShaunR

[pjr1121]

Is there a plan to implement conditional filters in the JSON paths?  For instance the following should return a single book title, but I get a parsing error due to the question mark.

Or is this already implemented in a different manner?

--tested on jsonpath.com--

$.store.book[?(@.category=='reference')].title

 

{
    "store":
    {
        "book":
        [
            { "category": "reference",
              "author": "Nigel Rees",
              "title": "Sayings of the Century",
              "price": 8.95
            },
            { "category": "fiction",
              "author": "Evelyn Waugh",
              "title": "Sword of Honour",
              "price": 12.99
            },
            { "category": "fiction",
              "author": "Herman Melville",
              "title": "Moby Dick",
              "isbn": "0-553-21311-3",
              "price": 8.99
            },
            { "category": "fiction",
              "author": "J. R. R. Tolkien",
              "title": "The Lord of the Rings",
              "isbn": "0-395-19395-8",
              "price": 22.99
            }
        ],
        "bicycle":
        {
            "color": "red",
            "price": 19.95
        }
    }
}

 

[camow7]

On 1/31/2020 at 11:25 PM, drjdpowell said:

A possibly better option is to replace your string/integer values with JSON strings, which is LabVIEW strings with <JSON> at the start of the name.  So "<JSON>Comment" rather than "Comment".   JSONtext will happily parse out the value into your cluster as JSON, then when you actually need your value you can convert it to string/integer, with appropriate handling if it is null or throws an error.

This method, by the way, is also how to handle JSON that has variable structure.

Tested this and it worked well. Had no idea you could add the <JSON> tag to parse complex data types like dictionaries. Super handy. This looks likes it's solved my problems. Thanks!

[camow7]

@drjdpowell Any plans to include a "Set Null to Default" feature?

[drjdpowell]

On 2/3/2020 at 5:01 AM, camow7 said:

Any plans to include a "Set Null to Default" feature?

Unfortunately this contrasts with the current behaviour that null-->NaN for a floating-point number, rather than being the default number input.  In standard JSON, the float values NaN, Infinity and -Infinity have to become null, and to convert them back to a default value doesn't make sense.  We could add an option to "ignore null items" which would treat nulls as equivalent to that item not existing. 

[drjdpowell]

On 1/31/2020 at 5:59 PM, pjr1121 said:

Is there a plan to implement conditional filters in the JSON paths?  For instance the following should return a single book title, but I get a parsing error due to the question mark.

Or is this already implemented in a different manner?

--tested on jsonpath.com--

$.store.book[?(@.category=='reference')].title

Plan?  Yes, but not a priority.   Note, though, that JSONtext returns substrings, meaning you can implement filtering in LabVIEW without making data copies, so the following code implements your example:

More effort, of course, but it should be fast, and using LabVIEW is very flexible, and more debug-able.  

[direheart]

Hi, awesome work on the library, is it possible to make it compatible with Labview 2014? Thank you!!

[drjdpowell]

5 hours ago, direheart said:

is it possible to make it compatible with Labview 2014?

No, unfortunately, as it uses VIMs that are only availablee in 2017+

[ThomasGutzler]

Help

Is there a way to parse this JSON string into this cluster in the following way:

- The order of elements inside the "Parameters" cluster does not matter. No error
- I can use enums
- The name of the elements matter. "TWOoo" shouldn't end up populating "two" (even if the order of elements matches the cluster). I want an error here instead of "two" being the default value
- Don't care if JSON text has additional elements that aren't in the cluster. No error

In short, features: Strict name checking, order independence, error if element missing

I've tried so many things but they all fail in one way or the other. See attached snippet

[LogMAN]

@ThomasGutzler here is a version that does most of what you describe.

• The order of elements inside Parameters doesn't matter.
• You can use enums.
• It doesn't matter if the JSON string has additional elements (no error)

However, you don't get an error if an element is missing ("two" in your example). That would require something like a "strict" mode, which I don't think is supported by JSONtext.

Unfortunately code is removed from VI snippets on LAVA, so here is the VI. Extract Parameters.vi

[LogMAN]

4 minutes ago, LogMAN said:

However, you don't get an error if an element is missing ("two" in your example). That would require something like a "strict" mode, which I don't think is supported by JSONtext.

Actually, if you turn your cluster back into JSON you can compare it to the original JSON string to see if there are any differences. The string is empty if every element is defined in the source string.

Try this: Extract Parameters.vi

Notice the "Differences" string, which lists all differences (missing elements) in your source JSON string.

[drjdpowell]

Standard JSON specification for JSON objects is that they are a collection of named elements with order unimportant, so any JSON library should ignore order and match on names.

Missing elements is a design choice, though, and I've gone with "use supplied default" rather than "throw error", for a number of reasons.  But as LogMAN shows, there are ways to do checking yourself.

[ThomasGutzler]

Thanks @LogMAN and @drjdpowell, I was unaware of the "Diff" vi. I'll use it to throw my own error.

☝️

[Phillip Brooks]

On 2/4/2020 at 4:28 AM, drjdpowell said:

Unfortunately this contrasts with the current behaviour that null-->NaN for a floating-point number, rather than being the default number input.  In standard JSON, the float values NaN, Infinity and -Infinity have to become null, and to convert them back to a default value doesn't make sense.  We could add an option to "ignore null items" which would treat nulls as equivalent to that item not existing. 

The native LabVIEW Unflatten From JSON Function might be of use.

I used a Salesforce REST API that returned JSON. There were numerous string and numeric fields that returned as NULL.

If you use the native LV function and change 'default null elements?(F)' to true and set 'enable LabVIEW extensions? (T)' to false, you can get the default values assigned in your input type & defaults cluster.

https://zone.ni.com/reference/en-XX/help/371361R-01/glang/unflatten_from_json/

 

[drjdpowell]

On 7/26/2019 at 3:35 PM, andy_r said:

Just small recommendation to change the library.

Concerning timestamp to string conversion:

The current format string "%Y-%m-%dT%H:%M:%S%3uZ" in "Variant to Json Text.vi" is dependent on localization. On my german windows this leads a to timestamp according to Iso 8601 of "2019-07-26T14:17:20,053Z". Notice the comma for the fractional seconds. According to the Iso both, comma and point, is valid (Apparently, the Iso even recommends the comma https://en.wikipedia.org/wiki/ISO_8601#cite_note-25). However, it seems that the point is used more regularly in practice, for example in .net (see Json.net package). So I would suggest to change the format like this: "%Y-%m-%dT%H:%M:%S%.;%3uZ". This ensures using the point instead of comma., hence the format is independent of your local setting.

Sorry, I had missed this comment at the time.   That is a good idea.

https://bitbucket.org/drjdpowell/jsontext/issues/53/force-use-of-decimal-point-in-timestamps

[Porter]

Bug? JSONText doesn't apply input default values to missing cluster items of array elements.

I was hoping to see "Default", 1 and "2", NaN but instead get "",1 and "2",0

I think that it is due to the behavior of the Get Array Information VI.

The workaround is to convert the array elements individually as clusters:

[drjdpowell]

4 hours ago, Porter said:

JSONText doesn't apply input default values to missing cluster items of array elements.

Thanks.  Issue 62: https://bitbucket.org/drjdpowell/jsontext/issues/62/jsontext-doesnt-apply-input-default-values

Although, I'm not sure this is doable, as your default values are the default values of your on-diagram array constant, not part of the data.  As that array is empty, I don't think I can access those default values inside a subVI.  If you had passed a non-empty array with a default value as first element, then I could read that and use it as default.

Edited July 9, 2020 by drjdpowell