Action Engines... are we still using these?

[Neil Pate]

5 minutes ago, drjdpowell said:

Single writer is good.  Why “global” then?   I generally have data flowing around by messages and locally stored by anyone who needs to store it.

The other issue is constantly polling a DVR for changes, versus receiving notification messages only when things change, but for “slow” apps that doesn’t matter much (but it affects your ability to reuse your solution in “fast” apps).  

I used to do things like this, where the data was in a message itself. However I found maintenance of the user events became too much of a pain. Now I just have simple "data changed" events, and then the thing listening polls the data it care about from the global data store.

My actors themselves are all 100% ByVal, I just use the data store as the mechanism for access the data across processes

[ShaunR]

27 minutes ago, Neil Pate said:

I used to do things like this, where the data was in a message itself. However I found maintenance of the user events became too much of a pain. Now I just have simple "data changed" events, and then the thing listening polls the data it care about from the global data store.

My actors themselves are all 100% ByVal, I just use the data store as the mechanism for access the data across processes

Then I would suggest staying away from any of the class based messaging systems that have proliferated.  You have to create a class (and all the overrides as well as inherit from their particular fetish for structure) for every damned message - which is comical. This is why I use simple strings for messaging..

Edited September 9, 2016 by ShaunR

[Neil Pate]

Just now, ShaunR said:

Then I would suggest staying away from any of the class based messaging systems that have proliferated. You have to create a class (and all the overrides etc) for every damned message which is comical. This is why I use simple strings for messaging.

I have a class based messaging system already, but the data payload is just a variant (for the exact reasons you have just mentioned).

In retrospect there is not much beneft my message being a class, I just designed it that way originally in case I wanted different message types (which I have never actually used).

[drjdpowell]

53 minutes ago, ShaunR said:

Because of aggregation.

The goal is to average a number of values all being acquired asynchronously. There needs to be a way to snapshot the multiple values at an instance in time. If you use messaging then by the time you messaged each subsystem and got a response the values are not at the same instance in time. If you use events, then the event structure only operates sequentially so you have a similar problem. The AE makes this trivial. Other solution require all sorts of hoops.

My first thought is “consider doing views of past history for real and use SQLite”, as I’m sure you agree.  My second thought is “that sounds trivial to do with the Register-Notify messaging pattern”.  Why in the world would you message subsystems to get data in response?   This is what I mean by focussing on a few complimentary techniques and learning them very well.  SQLite is very complimentary, but if one mixes limited-capability messaging with some similarly simplistic “tag” implementation then you end up with an overly complex result that is weaker than just picking one technique and really understanding it.

[ShaunR]

43 minutes ago, drjdpowell said:

My first thought is “consider doing views of past history for real and use SQLite”, as I’m sure you agree.  My second thought is “that sounds trivial to do with the Register-Notify messaging pattern”.  Why in the world would you message subsystems to get data in response?   This is what I mean by focussing on a few complimentary techniques and learning them very well.  SQLite is very complimentary, but if one mixes limited-capability messaging with some similarly simplistic “tag” implementation then you end up with an overly complex result that is weaker than just picking one technique and really understanding it.

An action engine to do the averaging over N channels is about 2 minutes to produce one VI (~2k), that can be explained to a novice in 30 seconds regardless of the architecture you use. 

A class with a DVR is about 5 minutes, 3 or 4 Vis (about 30K) and you have to promise there will be only one (but that's probably OK for Neil) and could be explained to a novice in 30 secs IF  he was OK with classes to begin with.

What's your overhead and how long until a novice would know it "very well"?

Edited September 9, 2016 by ShaunR
Niel/Neil. People are funny about their names :)

[drjdpowell]

1 hour ago, Neil Pate said:

I used to do things like this, where the data was in a message itself. However I found maintenance of the user events became too much of a pain. Now I just have simple "data changed" events, and then the thing listening polls the data it care about from the global data store.

My actors themselves are all 100% ByVal, I just use the data store as the mechanism for access the data across processes

I have one User Event per “actor”, that basically carry Text-Variant messages.  No maintenance at all.  The main weakness of polling on “data changed” is that you can miss updates.

[Neil Pate]

James, as I said I have been down that path before, exactly as you described. Now I am trying something different, "just because". :-)

[drjdpowell]

23 minutes ago, ShaunR said:

An action engine to do the averaging over N channels is about 2 minutes to produce one VI (~2k), that can be explained to a novice in 30 seconds regardless of the architecture you use. 

A class with a DVR is about 5 minutes, 3 or 4 Vis (about 30K) and you have to promise there will be only one (but that's probably OK for Niel) and could be explained to a novice in 30 secs IF  he was OK with classes to begin with.

What's your overhead and how long until a novice would know it "very well"?

Well, the simplest way to do that would be to Register Notifiers (well, NotifierMessengers) for the info one wants and then For-loop over the Notifiers to get your snapshot.  I’m not that fast at coding but it’s no faster for me to make an AE.  And don’t you have to modify all your data sources to add calls to this new AE?  I don’t have to modify any of the data-producing actors, my actors aren’t code-coupled via this subVI and can be reused as is, and some of my actors can be on remote systems (try that last one with your action engine!).  

How long would it take for a novice?   I’m not as positive on the ability of novices to understand action engines (they seem to love Local variables, from my experience) but learning messaging patterns is more of a learning curve.  But learning is an investment.  Learning messaging is well worth it.

[ShaunR]

1 hour ago, drjdpowell said:

For-loop over the Notifiers to get your snapshot.

That won't give you the snapshot.

Take an extreme example. You are launching download "actors". Each downloader has a MB/s that is emits via your notifiers and once the download is completed the downloader closes itself. You want to know the rolling 10 second average of throughput on your network card. So you launch 1000 downloads.

Now you have the global / concurrent snapshot problem. At any one time you need to know how many are still open so you can for loop through them and obtain the average. As you for loop through them, on each iteration, they will be updating their notifiers before you get to include them in your calculation and some will probably close leaving their last value in there before you've finished looping - therefore including them when they shouldn't be.

This is solvable with some sort of protected global storage (a DB, an AE, a DVR, a Go Pro) but not by pure messaging alone.

1 hour ago, drjdpowell said:

But learning is an investment.  Learning messaging is well worth it.

But it is not learning "messaging". It is learning someone's particular flavour of framework Your's, AQs, Michaels, Daklus. Everyone and his dog has had a go  I was rather amused when watching the NI Systems Groups video about their DCAF thing. Where was the Actor Framework?.  If I could ban one word from the dictionary it would be "Framework". Every single framework I have seen in LabVIEW so far has been like this

And you are still expected to build the bike 

Edited September 9, 2016 by ShaunR

[drjdpowell]

1 hour ago, ShaunR said:

This is solvable with some sort of protected global storage (a DB, an AE, a DVR, a Go Pro) but not by pure messaging alone.

Your kidding, right?  Just register on “Total bytes downloaded” and look at the increase (which stops when they finish).  Trivial.

Edit>> or register a Queue for all the “Throughput” messages and average once every 10 secs.   Also easy.  And are you really worried about sub-millisecond timing uncertainties on a 10-second average?  If so, I think you have those with an AE also, just due to OS jitter.

1 hour ago, ShaunR said:

But it is not learning "messaging". It is learning someone's particular flavour of framework Your's, AQs, Michaels, Daklus.

Don’t forget your “Dispatcher".

Edited September 9, 2016 by drjdpowell

[ShaunR]

18 hours ago, drjdpowell said:

Your kidding, right?  Just register on “Total bytes downloaded” and look at the increase (which stops when they finish).  Trivial.

Edit>> or register a Queue for all the “Throughput” messages and average once every 10 secs.   Also easy.  And are you really worried about sub-millisecond timing uncertainties on a 10-second average?  If so, I think you have those with an AE also, just due to OS jitter.

Don’t forget your “Dispatcher".

Well. there is no "Total bytes downloaded" emitted by the downloaders (or total size, for that matter) but if there was then throttling or congestion would make you think the download had finished so no. That won't do.

A queue isn't as straight forward as it appears on the surface either, Since the MB/s is usually driven by incoming data (receive n bytes in delta N ms and extrapolate bytes/s) rather than seeing how many bytes arrive in 1 second and then emiiting an event. (That's why we need a rolling average) Faster downloads will skew your average since they will be putting more entries in your queue.

I'm not sure why Dispatcher is thrown in here. That isn't a messaging framework or a memory accessor or really anything close to what we are discussing. Dispatcher is some turbocharged TCPIP primitives for a specific purpose (pub/sub). It has more in common with network streams than a messaging framework. It is meant to be a component in your system not the system itself. I suppose when an actor is just code that does something and messaging is just information flow then even a simple API becomes a "Messaging Framework". I don't subscribe to the philosophy of tortured vocabulary, though.

If you are looking for my "Messaging Framework" then you need to look at the VIM Demo where it is one VI @ ~700KB. The merits of each messaging approach is irrelevant in his discussion though. We should instead concentrate on the OPs question.

Edited September 10, 2016 by ShaunR

[Neil Pate]

Guys you are killing my thread :-)

An Action Engine is not a process (actor, whatever we want to call it). Nobody actually said it was either!

I just wanted to know if people still have a use for Action Engines (enough so to actually make one) these days.

 

[JKSH]

8 hours ago, Neil Pate said:

I just wanted to know if people still have a use for Action Engines (enough so to actually make one) these days.

I still use AEs for very simple event logging. My AE consists of 2 actions:

1. "Init" where I wire in the log directory and an optional refnum for a string indicator (which acts as the "session log" output console)
2. "Log" (the default action) where I wire in the message to log.

Then, all I have to do is plop down the VI and wire in the log message from anywhere in my application. The AE takes care of timestamping, log file rotation, and scolling the string indicator (if specified)

Having more than 2 actions or 3 inputs makes AE not that nice to use -- I find myself having to pause and remember which inputs/outputs are used with which actions. I've seen a scripting tool on LAVA that generates named VIs to wrap the AE, but that means it's no longer a simple AE.

[ShaunR]

12 hours ago, Neil Pate said:

Guys you are killing my thread :-)

Such a drama queen

I think most have weighed in and, as per usual, the applied programmers do still use them and the pure programmers don't.

3 hours ago, JKSH said:

Having more than 2 actions or 3 inputs makes AE not that nice to use -- I find myself having to pause and remember which inputs/outputs are used with which actions.

If you can wrap it in a polymorphic VI. You'll not only simplify the connector, you'll also get the adapt to type for the different usages.You don't have to split out each method into a new vi of the polymorph but you may have to split the typedef into a couple that make sense for the connectors.

For example. If you have a Hash AE that has SHA and HMAC. Then two VIs for the polymorph, one with a typedef of SHA1, SHA256 and SHA512 then another with MD4, MD5, SHA etc

[drjdpowell]

On September 10, 2016 at 8:35 AM, ShaunR said:

Well. there is no "Total bytes downloaded" emitted by the downloaders ...

I must not be understanding the problem you’re presenting.   Perhaps you could explain a AE/DVR solution, and illustrate how it cannot be done by messages.  

On September 10, 2016 at 8:35 AM, ShaunR said:

I'm not sure why Dispatcher is thrown in here. That isn't a messaging framework or a memory accessor or really anything close to what we are discussing. Dispatcher is some turbocharged TCPIP primitives for a specific purpose (pub/sub). It has more in common with network streams than a messaging framework. It is meant to be a component in your system not the system itself. I suppose when an actor is just code that does something and messaging is just information flow then even a simple API becomes a "Messaging Framework". I don't subscribe to the philosophy of tortured vocabulary, though.

If you are looking for my "Messaging Framework" then you need to look at the VIM Demo where it is one VI @ ~700KB. The merits of each messaging approach is irrelevant in his discussion though. We should instead concentrate on the OPs question.

You mention Daklu, who only has LapDog as a reuse library.  LapDog is less a messaging framework that your Dispatcher.   Dispatcher is an implementation of a “Message Broker” messaging pattern, I believe, so one is making more of a design choice by using it than one would be by using LapDog.  My “Messenger Library” is meant as a flexible message passing library, where you don’t need to use the “actor” stuff at all if you’d rather not (most of the examples installed with Messenger Library are simple message passing).  Its central messaging pattern is “Request-Reply”.  The Actor Framework, on the other hand, is very much intended to be a framework to enforce a certain style of program architecture.  One should (I hope) find Messenger Library very useful even if one doesn’t follow similar design principles to me, but it is (deliberately) hard to use the AF in a way not intended by AQ. 

Perhaps, though, you actually meant “principles” rather than “framework”, as Daklu has written a lot about the “actor-oriented” design principles he follows.

Edited September 11, 2016 by drjdpowell

[drjdpowell]

9 hours ago, JKSH said:

I still use AEs for very simple event logging. My AE consists of 2 actions:

1. "Init" where I wire in the log directory and an optional refnum for a string indicator (which acts as the "session log" output console)
2. "Log" (the default action) where I wire in the message to log.

Then, all I have to do is plop down the VI and wire in the log message from anywhere in my application. The AE takes care of timestamping, log file rotation, and scolling the string indicator (if specified)

Having more than 2 actions or 3 inputs makes AE not that nice to use -- I find myself having to pause and remember which inputs/outputs are used with which actions. I've seen a scripting tool on LAVA that generates named VIs to wrap the AE, but that means it's no longer a simple AE.

Make it that simple and perhaps I do use Action Engines.  Turn the enum into an “(Re)Initialize” boolean, or have the code initialize itself on first call, and you’ll find such things in my code.  But none of them will be usable to communicate between code modules like AEs are often used.  Logging is a good example of where multiple things can share something without affecting each other.  

[drjdpowell]

BTW on Action Engines:  are they really that simple to pick up for a novice?  “Sending a message” seems to me to be a much easier concept than “non-reentrant subVIs with uninitialized shift registers”.

[pawhan11]

I think sometimes globals or DVRs will be more suitable  than messages. For example when we have a large buffer data points that some process is acquiring and stores in memory, others will use that data. By using globals/dvrs it is just basically set and get. Using messages involves flattening data to variant/string in order pass that data by message implementation. With large data this delay might be significant.

[smithd]

9 hours ago, pawhan11 said:

I think sometimes globals or DVRs will be more suitable  than messages. For example when we have a large buffer data points that some process is acquiring and stores in memory, others will use that data. By using globals/dvrs it is just basically set and get. Using messages involves flattening data to variant/string in order pass that data by message implementation. With large data this delay might be significant.

It sounds like you still need some sync mechanism to be sure all the consumers get the data in the right order and all that (assuming your large buffer is actually a stream -- if its more of a database/tag then sure, a DVR would probably be the better implementation). The amount of overhead related to allocating a N0,000 byte buffer on a machine with 4-64 GB of ram is pretty minimal and I wouldn't step into DVRs over messages until I know for sure its too much data to keep up. To get the best of both worlds you could implement your own preallocated message buffer using a fixed size queue and some DVRs, but then we're really going off the deep end.

Edit: also the global has the same overhead since it generates a copy every time, and a DVR will have overhead unless you process the data inside of the in place structure.

 

Edited September 12, 2016 by smithd

[ShaunR]

15 hours ago, pawhan11 said:

I think sometimes globals or DVRs will be more suitable  than messages. For example when we have a large buffer data points that some process is acquiring and stores in memory, others will use that data. By using globals/dvrs it is just basically set and get. Using messages involves flattening data to variant/string in order pass that data by message implementation. With large data this delay might be significant.

There is nothing to stop you sending the DVR reference as a message. DVRs are storage and messages are pseudo-events. There are no rules (except self imposed ones) that say the data has to be an integral part of a message. Sending a DVR reference (or queue name etc) is like the postman leaving a note telling you to pick the parcel up at the depot because it won't fit through the letterbox.

[drjdpowell]

17 hours ago, pawhan11 said:

I think sometimes globals or DVRs will be more suitable  than messages. For example when we have a large buffer data points that some process is acquiring and stores in memory, others will use that data. By using globals/dvrs it is just basically set and get. Using messages involves flattening data to variant/string in order pass that data by message implementation. With large data this delay might be significant.

Variants don’t “flatten”.  Putting something in a variant doesn’t involve altering or copying the data.  They have overhead but I don’t think the size of the data matters.

Never use Globals for big data.  Globals always copy when you read them.  So does any “Get”.   Avoid a copy by extracting only the required data inside the structure you are using.  So, inside the IPE with a DVR, inside the Action Engine, or inside the message sending code.  

[drjdpowell]

1 hour ago, ShaunR said:

There are no rules (except self imposed ones) that say the data has to be an integral part of a message. 

Actually, that is an Actor Model rule ("messages should be immutable").  It’s a rule I sometimes break, but not without pause for thought.  Where I have broken the rule I have sometimes had race-condition bugs.  The value of rules is not in blindly obeying them, but in understanding why one should be reluctant to break them.

[ShaunR]

12 minutes ago, drjdpowell said:

The value of rules is not in blindly obeying them, but in understanding why one should be reluctant to break them.

The only rules in software are those imposed by the compiler Everything else is just anti foot-shooting aphorisms and  masochism. .

[drjdpowell]

Yes, I try not to shoot myself in the foot.

[Neil Pate]

3 hours ago, drjdpowell said:

Variants don’t “flatten”.  Putting something in a variant doesn’t involve altering or copying the data.  They have overhead but I don’t think the size of the data matters.

Really? I did not know this, always suspected there was something special about variants. Now I am even more glad I use variants as my message data