drjdpowell

Actually, I missed the "doesn't pause till lower loop is stopped bit"; that can't be a compiler issue (unlike updating the indicator after the breakpoint) so seems to be a bug.

How much of a performance loss would you be prepared to take to fix this bug?

I would guess it is a compiler optimization, where the terminal points to the same memory location as the output tunnel. It is arguable that some breakpoint weirdness is better than the forced memory copies just in case a breakpoint might be added at some point (given the huge number of places a breakpoint could be added). Alternately, it could be something to do with "chunking"; dividing a VI into executable chunks. I wouldn't be surprised if a breakpoint can only pause between chunks. Exiting the loop and writing to the indicator terminal could be one chunk.

Saving your images directly in a binary file would probably be the fastest way to save. Not the best format for long-term storage, but if you only keeping them temporarily so they can be read and compressed later then that doesn't matter. I would first consider your Compression step; can you make that any faster? You only need it 33% faster to get 150 FPS. Is this your own developed compression algorithm? How much CPU does it use when it is running? If only one CPU then there are parallelization options (such as compressing multiple images in parallel). BTW: do you really have 8-byte pixels?

Best to state performance numbers in per operation. I assume these are for 5528 images, so the "Drop Table" is 50ms each time? Try eliminating the Drop Table and instead just delete the row. If that works then your remaining dominant step is the 10ms/image for Compression. I think your initial mistake was to go, "Since we want to speed up the process, <we do extra steps and make things extra complicated and in parallel modules in hopes that will be faster>." Better to have said, "Since we want to speed up the process, we will make a simple-as-possible prototype VI that we can benchmark." That prototype would be a simple loop that gets data, compresses, and saves to db.

I would much rather than IMAQ references behaved the same as other LabVIEW references, like Queues.

How to get a list of image buffers? Thanks! That's great.

As a side question, how do people deal with the non-standard way that IMAQ image references work (alway globally named; don't clean up when owning VI goes idle)? For background, I am currently trying to get a large amount of non-reentrant image analysis code to work reentrantly, and have to deal with preventing one clone of a VI modifying an image inadvertently shared with another. I am attacking the problem by auto-generating image names based on call site (ie, a pre-allocate clone that uses its own clone id in the image name):

Is there any way to list all IMAQ images currently in memory? This is as a diagnostic and code check, rather than to actually use this in code.

I note that you haven't given any performance numbers. If I were looking at slow code with four steps, I would have learned how long each step roughly took within a few minutes**. It is not uncommon in such cases for one step to be the bottleneck, taking significantly longer than all the other cases put together. In that case, it does not help to try and do steps in parallel, and you should instead try and improve the performance of that one step. I will withhold any other suggestions, as I think you are possibly just digging a hole for yourself by layering on "performance improvements" without measuring performance. **Using the "Quick Timer" from "Cyclic Table Probes".

It's the inconsistent state that is the problem, where the by-value bit is inconsistent after the by-ref bit is changed in parallel. That doesn't always happen; sometimes either the value or the reference bit never changes. But one needs to be aware of this issue.

Thanks. I have often considered providing a flattening-based implementation for arbitrarily large clusters, but have always paused because of the big step change in performance.

In the LAVA-CR is a 1.14.5 version that includes a service-discovery implementation using UDP messages. See the example "Example Reconnecting TCP Client with UDP Service Discovery.vi".

Sets and Maps are 2019, so I'm not sure they can be supported till JSONtext is in 2019.

I'm afraid it uses Malleable VIs that were introduced in 2017, meaning it can't, as a whole at least, be back ported.

Ah, you mean this then: http://json-schema.org/understanding-json-schema/, and you want a function that converts a LabVIEW Cluster/Array to a basic JSON Scheme. That does not exist in JSONtext but could at some point.

Not sure I understand the question. Can you rephrase it?

No. Only by custom formatting the numbers into JSON yourself. I often use single numerics, as 7 decimal places is more reasonable looking.

Googling suggests PQ should be threadsafe: https://www.postgresql.org/docs/9.1/libpq-threading.html

My mistake. There is an SQLite on Linux RT conversation also going on right now and I mixed the two up.

SQLite is thread safe unless it is changed from the default "Serialized" mode: https://www.sqlite.org/threadsafe.html

It is difficult for me to give good advice without really understanding the project. Some random thoughts: Where are the performance bottlenecks? The first thing I would do is prototype and benchmark the steps and get a feel for what are time-limiting, high-CPU steps. Be wary of designing a complex custom communication pattern "for performance" before you understand what and where the performance issues are. How long does it take to convert your IMAQ image into a DBL array, for example? You might find that effort on a custom queue system is better spent on some other area. Remember that you IMAQdx task is already an asynchronous actor-like thing, with a buffer of images. I often see people placing importance on the loop "taking data" piping that data to an async analysis loop, when really the IMAQdx or DAQmx task is really the one taking data. Also note the design pattern used by IMAQdx Events, where the "Frame Done" event doesn't contain the Image itself, but tells one that an image can be read from a separate queue (ring buffer). If you find you need a custom queue for data, you could consider using this pattern. I'd also try and experiment to see if I could have more than one loop pull different Frames off the IMAQ ring buffer at the same time, as this gets parallelism working as early as possible. If you are working with reference data, either IMAQ images or DVRs, then 75 messages per second is not high, and I personally would just use the standard Messenger Library Actor Event-Notification Messages for passing references to the data, at least at first. Design for simplicity and add complexity only later if your performance benchmarks show value in it.

Brute force. I suck at scripting. I add 20-30 more cases every so often. Maximum possible with this method is 255. This VI is part of the "JDP Science Common Utilities" vipm package, BTW, so one doesn't need to install JSONtext to use it. Originated in the conversations on this idea: Convert an Array of Variants into a Cluster. I wish NI would just implement that idea so this stopgap would be unnecessary, so please kudo that idea. I used to use OpenG Variant Tools a lot, but replaced them with the newer NI functions once they became faster (about LabVIEW 2012, I recall) and I had this workaround. The only missing functionality from OpenG is the ability to set the name of data in a cluster (the above SubVI create a Cluster with generic element names; I decided to just live without that.

What kind of data and data rate?

My own UIs, when I polish them, tend towards mostly white. I could give NI some suggestions on how to improve that UI, but I wouldn't change the basic whiteness. Although my UIs are graph heavy, which I find look better as all white. In a UI whiteout graphs the reasoning is less strong.