drjdpowell

Some performance numbers: I took the "Message on new transaction" JSON from the blockchain.info link that Shaun gave, created a cluster for it, and compared the latest LAVA-JSON-1.4.1**, inbuilt NI-JSON, and my new JSONtext stuff for converting JSON to a cluster. LAVA-JSON: 7.4 ms NI-JSON: 0.08 ms JSONtext: 0.6 ms Then I added a large array of 10,000 numbers to bulk the array out by 50kB. If I add the array to the Cluster I get these numbers: LAVA-JSON: 220 ms NI-JSON: 5.6 ms JSONtext: 9.0 ms (I pass the large array to NI-JSON internally, which is why I'm closer) If I don't add the array to the cluster (say, I'm only interested in the metadata of a measurement): LAVA-JSON: 135 ms NI-JSON: 5.2 ms JSONtext: 1.1 ms The NI tools appear to vet everything very carefully, even unused elements, while I do the minimal checking needed to parse past the large array (in fact, if I find all cluster elements before reaching the array, I just stop, meaning the time to convert is 0.6 ms, as if the array wasn't there). **Note: earlier LAVA-JSON versions would be notably worse, especially for the large case.

I’d add: - Work on a stream (i.e. allow the JSON Value to be followed by something else, like the regular Flatten functions have a “Rest of String” output). - Give useful error messages that include where in the very long JSON text the parser had an issue. - Work with “sub”JSON. Meaning, “I know there is an “Options” item, but it can come in multiple forms, so just return me that item as JSON so I can do more work on it (or pass it on to an application subcomponent that does know what the form is). The library I’m working on, JSONtext, is trying to be sort of an extension to the inbuilt JSON primitives that adds all these features.

One of the performance advantages of working directly with JSON strings is that, when converting JSON to/from a Variant, one can use the NI primitives to handle large numeric arrays, without ever letting it see (and throw a fit over) the entire string. In fact, I can often pull out selected elements of a large JSON text faster than using the NI primitive’s “path” input (I think because the primitive insists on parsing the entire string for errors, while I don’t).

I’m working on a new JSON library that I hope will be much faster for my use cases. It skips any intermediate representation (like the LVOOP objects in the LAVA JSON API, or the Variants in some other JSON toolkits) and works directly on JSON text strings. I’d like to avoid making a tool just for myself, so I’d like to know what other people are using JSON for. Is anyone using JSON for large(ish) data? Application Config files? Communication with remote non-LabVIEW programs? Databases?

XControls don’t automatically save display properties of their contained controls; one has to explicitly save/load in special methods of the XControl. A big pain, given the large number of properties.

It’s Ton Plomp’s XControl, I’d guess.

After studying documentation, my current plan is to use Postgresql with its libpq client dll. I think I can wrap libpq in a similar way to what I did with SQLite.

Anybody used Postgresql?

What toolkits do people use for accessing databases? I’ve been using SQLite a lot, but now need to talk to a proper database server (MySQL or Postgres). I’ve used the NI Database Connectivity Toolkit in the (far) past, but I know there are other options, such as using ADO.NET directly. What do people use for database connectivity? What would you recommend? — James

Speculative, to establish that the work is legitimate R&D of my company. I want to be able to show a tax inspector a company-branded product on the LabVIEW Tools Network. BTW, I’ve found I can separate the new stuff into a separate, non-conflicting project, so LAVA JSON can be left as is (I’ll bring out a new version with some modest performance improvements).

I’m recording my work on Messenger Library and all other reuse libraries, and hope to claim at least some of them also. JSON just happens to be what I am working on right now.

JSON LabVIEW is BSD licensed. It’s not “Copy Left”, so one doesn’t have to open-source any derivative work. BTW, what I’m working on is a major change, that gets rid of the intermediate representation as LVOOP objects entirely. The only representation is text. So I may be able make it an independent package that can be used in parallel with the LAVA JSON library.

Because I forget to submit to NI. Updating LAVA-CR is quick and instant, and I intend to send to NI later when any bugs are discovered, but then it drops of my radar.

https://www.gov.uk/guidance/corporation-tax-research-and-development-tax-relief-for-small-and-medium-sized-enterprises I was thinking of that as an option. The LAVA community one would be version 1.4.1. I’m afraid I’ve checked in some things starting with a switch to 2013; you should ignore those and branch from the last 2011 checkin. I’ll set up a new repo and a new project name.

I’m doing a lot of work at the moment on a “JSON 2.0” version, particularly intended to improve performance in common use cases that I’ve encountered. For example, I have a new VI that flattens a Variant directly to JSON text (skipping the Objects entirely) that works on large arrays of numerics about 30 times faster. I’d like to claim R&D tax credits on the work I do on this (which is otherwise unpaid), and to do that the result of the work needs to be owned by my limited company, so I’d like to release the new version under “JDP Science Limited”, rather than “LAVA”. I would put a BSD license on the new version, and properly comply with the BSD license of the current LAVA version. I hope none of the contributors of "JSON LabVIEW” object? The R&D credit would work out to about 30% of charged rate (if I do the tax math right, which is far from obvious). 30% is a lot better than zero.

I can also recommend Savitski-Golay, which is basically just fitting to a Nth-order polynomial, and estimates multiple derivatives at once. Apply some “straight-line” criteria such as all derivatives beyond the first being “small”.

You have to outsmart the compiler when performance testing. In your code your constant array is copied at the to-variant step because it is needed at the next iteration. Make a new array for each iteration, like this: tests (Modified to avoid Constant Folding).vi

Yes, I try not to shoot myself in the foot.

Actually, that is an Actor Model rule ("messages should be immutable"). It’s a rule I sometimes break, but not without pause for thought. Where I have broken the rule I have sometimes had race-condition bugs. The value of rules is not in blindly obeying them, but in understanding why one should be reluctant to break them.

Variants don’t “flatten”. Putting something in a variant doesn’t involve altering or copying the data. They have overhead but I don’t think the size of the data matters. Never use Globals for big data. Globals always copy when you read them. So does any “Get”. Avoid a copy by extracting only the required data inside the structure you are using. So, inside the IPE with a DVR, inside the Action Engine, or inside the message sending code.

BTW on Action Engines: are they really that simple to pick up for a novice? “Sending a message” seems to me to be a much easier concept than “non-reentrant subVIs with uninitialized shift registers”.

Make it that simple and perhaps I do use Action Engines. Turn the enum into an “(Re)Initialize” boolean, or have the code initialize itself on first call, and you’ll find such things in my code. But none of them will be usable to communicate between code modules like AEs are often used. Logging is a good example of where multiple things can share something without affecting each other.

I must not be understanding the problem you’re presenting. Perhaps you could explain a AE/DVR solution, and illustrate how it cannot be done by messages. You mention Daklu, who only has LapDog as a reuse library. LapDog is less a messaging framework that your Dispatcher. Dispatcher is an implementation of a “Message Broker” messaging pattern, I believe, so one is making more of a design choice by using it than one would be by using LapDog. My “Messenger Library” is meant as a flexible message passing library, where you don’t need to use the “actor” stuff at all if you’d rather not (most of the examples installed with Messenger Library are simple message passing). Its central messaging pattern is “Request-Reply”. The Actor Framework, on the other hand, is very much intended to be a framework to enforce a certain style of program architecture. One should (I hope) find Messenger Library very useful even if one doesn’t follow similar design principles to me, but it is (deliberately) hard to use the AF in a way not intended by AQ. Perhaps, though, you actually meant “principles” rather than “framework”, as Daklu has written a lot about the “actor-oriented” design principles he follows.

Your kidding, right? Just register on “Total bytes downloaded” and look at the increase (which stops when they finish). Trivial. Edit>> or register a Queue for all the “Throughput” messages and average once every 10 secs. Also easy. And are you really worried about sub-millisecond timing uncertainties on a 10-second average? If so, I think you have those with an AE also, just due to OS jitter. Don’t forget your “Dispatcher".

Well, the simplest way to do that would be to Register Notifiers (well, NotifierMessengers) for the info one wants and then For-loop over the Notifiers to get your snapshot. I’m not that fast at coding but it’s no faster for me to make an AE. And don’t you have to modify all your data sources to add calls to this new AE? I don’t have to modify any of the data-producing actors, my actors aren’t code-coupled via this subVI and can be reused as is, and some of my actors can be on remote systems (try that last one with your action engine!). How long would it take for a novice? I’m not as positive on the ability of novices to understand action engines (they seem to love Local variables, from my experience) but learning messaging patterns is more of a learning curve. But learning is an investment. Learning messaging is well worth it.