Sending and displaying real-time data between Actors

[Mike Le]

I've created a set of Actors that pick up data from a hardware source and display that data live in a Chart indicator.

 

The organization is pretty simple:

 

1) Controller Actor: Launches the Hardware, Plotter, and Analysis actors. Mediates messages between them.

2) Hardware Actor: responsible for setup and shutdown of communication with the DUT. As it picks up data, passes them up to the Controller.

3) Plotter Actor: picks up hardware data from the Controller and plots the data to the Chart. Also provides controls that let the user change what signals are plotting and what they look like.

4) Analysis Actor: picks up hardware data from the Controller for analysis; results and/or raw data can be saved to file.

 

The DUT produces data at a rate of 1kHz. It's theoretically possible to stream about 200 signals at this rate, though in practice it would be exceptional if someone were to gather more than 50 at once.

 

At first, I had the Hardware Actor pick up each data-point individually and immediately forward each point to the Controller. But the Plotter Actor down-stream had trouble keeping up with the 1kHz rate.

 

My fix was to have the Controller "batch" the messages. The Hardware Actor is still producing at 1kHz, but the Controller builds an array until 30 data-points have been received and then forwards the 2D array to the Plotter.

 

Now the system was fast enough for a few signals, maybe 5-10.

 

But as I add more signals, I find that my software starts to fall behind the hardware data.

 

A colleague suggested using a DVR to store the picked up data and passing the reference around instead of the data array. My concern is that this breaks the "contract" of Actors communicating only through Actor Messages. This thread was helpful, but I thought I'd start a new thread for this particular use-case. I imagine others have used Actors for streaming hardware data and have come up with better, more elegant solutions than I could!

Edited January 10, 2014 by Mike Le

[drjdpowell]

Could you not have the Hardware Actor batch up measurements, rather than the Controller? 

 

Does your Plotter Actor always do slow display updates (Property nodes) for each message?  You can write it to only do display updates a few times per second even if new data messages arrive faster.

[Mike Le]

I could move the batching to the Hardware Actor, but I'm not sure this would clear the bottleneck? It seems like the Plotter would see about the same frequency of messages. I can try it and see if there's a performance improvement.

 

I was originally pushing updates through a property node, but as you point out, this was very slow. I switched to pushing the updates to an internal queue, which is then read in a parallel loop of Plotter Actor Core. This queue dumps out directly into the Chart indicator terminal.

 

In terms of update speed, it's desirable by the client for the graphing to appear smooth. Around 20Hz seems to be acceptable for them. I'll try adding a small delay to the queue-to-terminal loop (switching to a flush queue operation) and see if that improves things.

[drjdpowell]

I could move the batching to the Hardware Actor, but I'm not sure this would clear the bottleneck? 

I don’t know where the bottleneck is, but you’ll have problems pushing 50k to 200k messages per second between actors.  For display operations you should definitely have a flush queue in there somewhere, so updates can run at a slower rate than arriving messages.  

 

If you do go with a shared reference, I wouldn’t use a DVR; instead share a data queue like the one you use in your Plotter Actor Core.  Create that in Plotter and pass it by message to Hardware.  Then you have a direct, low-overhead stream that you flush a few times a second.

 

— James

 

BTW, I don’t know the details of you application, but I would be tempted to make basic plotting of raw data a feature of the Hardware Actor itself.

[Mike Le]

BTW, I don’t know the details of you application, but I would be tempted to make basic plotting of raw data a feature of the Hardware Actor itself.

 

This is an idea that's also been pitched around the office. My main hesitation is that the data needs to be passed not just to the Plotter, but also to at least one other Actor (maybe two down the line). Because of that, it made sense to me to isolate Hardware and Plotting, because then the Controller can decide which Actors need access to Hardware data.

 

If I couple the Hardware and Plotter actors, then suddenly that actor also needs to make the decision to forward the Hardware data to another Actor. At that point, it feels like I'm creating a bloated Controller/Hardware/Plotter actor and tightly coupling all of those functions/responsibilities.

 

 

I don’t know where the bottleneck is, but you’ll have problems pushing 50k to 200k messages per second between actors.  For display operations you should definitely have a flush queue in there somewhere, so updates can run at a slower rate than arriving messages.  

 

I was playing with a flush queue at the Chart updating end and couldn't get a performance improvement. It actually slowed down the front panel updates immensely, but that could've been a problem with how I implemented it. Do you mean a flush queue for the Actor messaging? I've never done anything like that before; is it even possible? It doesn't seem consistent with Actor philosophy...

 

 

If you do go with a shared reference, I wouldn’t use a DVR; instead share a data queue like the one you use in your Plotter Actor Core.  Create that in Plotter and pass it by message to Hardware.  Then you have a direct, low-overhead stream that you flush a few times a second.

 

This might be the most expedient way to go if I can't get batching to work. It still concerns me to pass around a "back-channel" reference, but it might be necessary to bypass the overhead of Actor processing.

[kombela]

According to my tests on AF this behavior is normal. AF is just not the fastest messaging framework that can be implemented in LabVIEW. It has two components that make is relatively slow:

1. Uses priority queues. It makes sending atomic messages(messages with little or no data content) about 2 times slower. More on this: https://decibel.ni.com/content/docs/DOC-24589
2. Uses dynamic dispatch. Once per message a dynamic dispatch VI is called. It has an overhead and it makes atomic messaging 3 - 8 times slower. Read more on this: https://decibel.ni.com/content/docs/DOC-13709#comment-31559
   

 

Please don't take my numbers seriously as accurate measurements, but approximately these are what I measured at least on my PC.

If you read all the links you will understand the reason for AF being slow. I think you can get some speed by leaving out priority queues and use normal queues, but I see no immediate solution for the dynamic dispatch problem.

 

I have a similar system in work, and what I do is to use as much data in a packet as my GUI's refresh rate requires (so for a 1kHz signal to achieve 25 fps you can send arrays of 40 data points). I see that you use this trick for the Controller, but use it for the Hardware actor also.This way the user still experiences "real-time" behavior, but you spared with messages.

 

Hope it helps a bit.

 

[this reply was first posted on AF forums, as I didn't notice the discussion already started here]

[drjdpowell]

Because of that, it made sense to me to isolate Hardware and Plotting, because then the Controller can decide which Actors need access to Hardware data.

Sending things to the controller is fine, but if all the Plotter does is plot raw Hardware data, then what are you gaining by splitting them?  If it’s more than that; if Plotter is more than just a straight UI for Hardware, then it makes sense.

 

Personally, I would always start with a “Hardware” actor running by itself, for initial debugging in isolation.  For that, Hardware needs a UI (preferably simple, low-overhead, and one that has no overhead if Hardwares Front Panel isn’t loaded).  Often this UI can serve as a part of the application UI by being in a subpanel.  

 

Ask yourself this: if you reused Hardware in a new application, would you reuse Plotter as is?  If so, maybe it would be simpler to combine them.

 

I was playing with a flush queue at the Chart updating end and couldn't get a performance improvement. It actually slowed down the front panel updates immensely, but that could've been a problem with how I implemented it.

If thing’s got worse when you reduced the UI update rate then you did it wrong.

 

Do you mean a flush queue for the Actor messaging?

No.  Though you can do the equivalent thing with messages by having them flag the need for a UI update, but not do the update till later (after many messages have arrived.

 

This might be the most expedient way to go if I can't get batching to work. It still concerns me to pass around a "back-channel" reference, but it might be necessary to bypass the overad of Actor processing.

Do you batch signals?  Or send a separate 30+ messages/sec for each of 50 signals?  30 messages per second containing all signals can’t have THAT much overhead.

 

Oh, BTW, make sure your not using “Build Array” anywhere as that can be slow.

[Mike Le]

I was using Build Array; factoring it out led to a substantial performance improvement. Thanks for that!

 

I'm still seeing some issues with GUI slowdown plotting at 30Hz, so I still have to play around with it, but that definitely helped a lot.