Actor-Queue Relationship in the Actor Framework

[Daklu]

The inner one doesn't even necessarily have to route through the outer one for outbound messages...

 

I don't think there's anything wrong with the idea of using a filter actor.  I've used them in my own code with no discernable negative effects.  But sending message from an inner actor to an external actor can create unexpected race conditions if the developer isn't careful.  I just posted an example on another thread (and copied below) illustrating the problem.

 

 

There's value in routing all messages between helper loops, subactors, and external actors through a central point inside the actor. (Usually the message handling loop in my code.) That central point maintains all the important state data the actor needs to operate correctly.

 

Once you allow internal loops to communicate directly with external actors, you open the door to race conditions. The component is announcing an event (an error) the brain (the message handler) doesn't know about. To illustrate the race condition, let's pretend you have two actors: Actor and OtherActor.

 

Actor has a message handling loop and an interal helper loop. We don't know how OtherActor is implemented, but we know that if it receives an error message it will in turn send a Reset message back to the Actor. But, we don't want the Actor to always accept and process Reset messages. We only want to do that if the Actor is in an error state.

 

When the messages are routed through the message handler the message sequence goes like this:

1. Helper loop sends error message to message handler.

2. Message handler notes error, switches to error state, and sends message to OtherActor.

3. OtherActor sends Reset message to Actor.

4. Actor receives message, confirms it is in an error state, and initiates the reset.

 

When the helper loop bypasses the message handler this sequence is possible:

1. Helper loop sends error message to message handler and OtherActor.

2. OtherActor sends Reset message to Actor.

3. Actor receives Reset message, finds it is not in an error state, and discards the message.

4. Actor receives error message and switches to error state.

 

Will that happen? Probably not. But the race condition exists and I'd rather *know* my app will behave as expected rather than *hope* my app will behave as expected. (In my experience by the time race condition is discovered there will be so many race conditions present it is very difficult to remove them.)

 

Sending outbound messages directly to recipients and bypassing the outer actor can be done safely, but it's certainly one of those things I would encourage people to avoid until they are very comfortable with actor-oriented programming.

[Aristos Queue]

I am not sure I'm following your terminology. But if I'm reading this right, your example isn't quite what I was thinking, and tweaked, I don't think the race condition exists. Check my logic, please...

 

Here's the original launch stack...

 

Caller Actor

  |

Filter Actor

  |

Nested Actor

 

Nested Actor is given Caller Actor's direct queue so Nested can send to Caller directly... the advantage here is that Filter then only has to worry about messages coming one way and not about sometimes having to pass them up to Caller. Now, here's the fix that I think stops your race condition: Filter should *never* send messages to Caller. Filter's job is all about shielding Nested. If Filter wants to say something to Caller, that's actually a real job, so have it send a message to Nested that says, "Please tell Caller..."

 

Does that close the hole? All the state knowledge is now in Nested, in one place.

[Daklu]

You're right, we are thinking about slightly different things.

 

 

Does that close the hole? All the state knowledge is now in Nested, in one place.

 

Given your launch stack, it depends on how much Nested depends on Filter for correct operation and whether Filter is constant or changable.

 

In the specific case of DoS attacks it probably doesn't matter since a few extra messages getting through Filter are unlikely to break Nested.  The filtering is geared more towards performance and responsiveness than ensuring state diagram correctness.  In situations where failure to filter a message may allow state diagram violations, the presence of a race condition is almost certainly undesirable.  Then filter and state knowledge need to be in the same place.

[Aristos Queue]

In situations where failure to filter a message may allow state diagram violations, the presence of a race condition is almost certainly undesirable.  Then filter and state knowledge need to be in the same place.

So when that case arises, Filter would send a message to Nested to handle it, perhaps packaging as a single message a whole block of messages that lead to the problem. Doing so means that Nested assumes that it is launched by a Filter for correct operation since it does not include its own filtration system (something that would be redundant under the design of a private Nested actor class).

 

Any other holes?

[mje]

To dredge this up again:

 

 

An actor receiving requests is generally in charge of its health and destiny; an actor sent commands is subject to DoS attacks or other hazards from external sources, whether incidental or malicious.

 

A command makes the sender responsible for knowing whether the receiver can process the message without going into some unexpected state.  In the context of the entire system's operation, yes, the sender needs to know when it should send a particular message to a particular receiver, but it should never have to worry about breaking the receiver because it sent a message when it shouldn't have.  (Like I said, lots of gray between actor and non-actor.)

 

I wasn't sure I was buying into this over the last week or two. I completely get the difference but I'm not convinced the dichotomy is relevant to the Actor Framework because depending on which relationships you're looking at (Actor-Message, Actor-Queue) I don't see an Actor living at either end of the spectrum. Depending on how many layers of the onion you peel back, I can imagine looking either perspective.

 

I've also been thinking about the relationship between Actors and their Messages and today I came up with something which sort of blew my mind. Maybe this is obvious to those of you who have been working with the AF for a while but to me this was a surprise that I could do this.

 

 

This is an override of a Message Do.vi method I made as an example. Note it does not return the same Actor supplied as in input to the output. The Message has effectively switched out which Actor is spinning in the Actor Core.vi loop. To me this completely flies in the face of having an Actor be in charge of its destiny and obliterates the idea of sending requests to an actor. I can chop that Actor's head off whenever I please.

 

All I need to do is send a malicious Message to a spinning Actor and I can completely kill it. Can you even contain this behavior if you start working with a plug-in architecture? Without digging too deep I think this stems from the fact that the main Actor Core.vi doesn't enforce the Dynamic Dispatch to the output terminal and would seem to be a conscious design decision.

 

Sample code attached. Open the Test.vi in the root folder to see a functioning example. Made with the stock AF shipping with LV 2012.

Actor Switch.zip

[drjdpowell]

In any “command pattern”-style process the messages are part of the process, so if one can write or load new messages then one is modifying the process.  

[Daklu]

...but I'm not convinced the dichotomy is relevant to the Actor Framework because depending on which relationships you're looking at (Actor-Message, Actor-Queue) I don't see an Actor living at either end of the spectrum.

 

I don't understand what you're saying.  Can you explain it a little more?  Do you mean the dichotomy between a command and a request?  I also don't get what you mean by Actor-Message and Actor-Queue relationships.

 

 

The Message has effectively switched out which Actor is spinning in the Actor Core.vi loop. To me this completely flies in the face of having an Actor be in charge of its destiny and obliterates the idea of sending requests to an actor. I can chop that Actor's head off whenever I please.

 

Being able to swap out actors can be a good thing.  It allows you to use actors to implement a state diagram, where each actor represents a different state.  I've slightly modified the project and added a UI element to change between one of four different actors.  (See Test2.vi)

 

Personally I don't consider swapping out Actor classes to be equivalent to killing off an actor.  The AF strongly ties Actor classes to the abstract concept of actors, but in my mind the essence of all actors is the message handling loop--Actor Core in the AF.  Actor Core isn't ever stopped, so you're not really chopping off the actor's head.

 

If malicious code is a concern you might be able to override Actor Core and test the messages when they arrive, discarding any messages that you did not put on your white list.  Or just don't ever expose the Enqueuer and wrap all message sends in actor methods.  (i.e. OriginalActor.SendGetPathMsg)  I've pretty much decided that if someone has enough access to the source code or runtime environment where they're able to inject malicious code, I've got far bigger problems that hardening the actor framework of choice won't solve.

Actor Switch.zip

[Daklu]

...you might be able to override Actor Core and test the messages when they arrive, discarding any messages that you did not put on your white list.

 

I spent a couple hours this morning trying to implement this and it looks like AQ effectively locked out our ability to implement message filtering that way.  Apparently there is a document somewhere explaining how to implement message filters in AF, but I didn't find it in my quick search through the AF community.

[mje]

Do you mean the dichotomy between a command and a request?

 

Yes, that's what I was referring to. I won't go into detail how I think an Actor doesn't satisfy a command pattern only because there seems to be a good consensus on that. However I don't really believe an Actor is simply a request based system either. I don't really have time to rebut this point today though (if it needs rebuttal). Similar to how Shaun thinks the framework blurs the line between messages and processes, I don't really see the AF as laying on either end of the command-request spectrum.

 

If malicious code is a concern you might be able to override Actor Core and test the messages when they arrive, discarding any messages that you did not put on your white list.  Or just don't ever expose the Enqueuer and wrap all message sends in actor methods.  (i.e. OriginalActor.SendGetPathMsg)  I've pretty much decided that if someone has enough access to the source code or runtime environment where they're able to inject malicious code, I've got far bigger problems that hardening the actor framework of choice won't solve.

Malicious code-- whether the maliciousness is intentional or not-- would be a concern for applications designed to load external components at run-time (plug-ins etc). Basically at this point I'm not sure I'd want to expose any Actors Enqueuers directly through an interface external components would be able to interact with.

 

Since the Enqueuer is sealed off and an Actor demands that overrides of Actor Core.vi call their parent implementation, if I choose to expose the Enqueuer I have absolutely no way of restricting which messages could be sent to the Actor and which Messages get acted on.

 

Now this isn't an insurmountable problem: encapsulate the Enqueuer in yet another layer which provides an interface external components use to interact with. The implementation simply delegates to the Actor by firing off a set of corresponding privately scoped messages down the Enqueuer.

[Daklu]

I won't go into detail how I think an Actor doesn't satisfy a command pattern only because there seems to be a good consensus on that.  However I don't really believe an Actor is simply a request based system either.

 

I must have missed out on (or forgotten) that conversation.  It seems obvious to me the AF's messaging paradigm *is* the command pattern.

 

Important point of clarification:

When I talk about command messages vs request messages, I am specifically not referring to command pattern messages.  Command pattern messaging is one way of implementing the decision making process for invoking the message handling code.  It is contrasted with a message ID system.  Command pattern messaging uses dynamic dispatching; message ID systems use a case structure that switches out on the ID of the message received.

 

Command messages vs request messages is all about deciding who is responsible for making sure the message receiver doesn't do something it is not supposed to do.  If the receiver depends on the sender to only send the subset of messages that are appropriate for its state, the messages are in effect commands.  (That's what I mean by "sender-side filtering.")  The sender is "commanding" the receiver to execute the message regardless of whether or not the receiver is in a state where it should execute the message.  When the receiver implements its own state checks prior to executing a message it receives ("receiver-side filtering") the messages are in effect requests.  The receiver always has the option to refuse the message.

 

 

I don't really see the AF as laying on either end of the command-request spectrum.

 

Correct, the AF doesn't force the developer towards either end of the spectrum (and I can't think of any way a framework could enforce that if it wanted to.)  My assertion is that well-designed actors, regardless of the framework used, always use requests.  And it doesn't only apply to actors; it applies to any concurrent programming model.

 

It's the the test-and-act problem on a different scale.  With concurrent programming, as long as test value and set value are unique operations separated by a finite amount of time, there's no way to guarantee a concurrent process hasn't changed the value after it was tested and before it is set.*  It is a race condition.  In those situations, instead of testing the value first to see if the action can be taken, programmers typically attempt to do the action and see if it generated an error.

 

The principle is the same for actors.  Assuming the actor has more than one behavioral state (i.e. it responds to the same message differently depending on its internal state) any message filtering *has* to be done in the receiving actor's logical thread.  If message filtering is only done by the sending actor then you have a test-and-act race condition.

 

(*You can't make that guarantee in the general case.  Sometimes you can verify the guarantee via code inspection if your application tightly controls who is able to send messages to the actor.)

 

 

Basically at this point I'm not sure I'd want to expose any Actors Enqueuers directly through an interface external components would be able to interact with.

 

Since the Enqueuer is sealed off and an Actor demands that overrides of Actor Core.vi call their parent implementation, if I choose to expose the Enqueuer I have absolutely no way of restricting which messages could be sent to the Actor and which Messages get acted on.

 

Now this isn't an insurmountable problem: encapsulate the Enqueuer in yet another layer which provides an interface external components use to interact with. The implementation simply delegates to the Actor by firing off a set of corresponding privately scoped messages down the Enqueuer.

 

Building a robust plugin framework that prevents rogue plugins from taking down the app or other plugins doesn't strike me as a particularly trivial task, regardless of the framework/messaging system used.  The system has to be designed so each plugin runs in its own logical thread; otherwise a rogue plugin can enter an infinite loop and starve everything else.  That means spinning up an independent actor for each one.  Each plugin also needs a unique queue to send messages to the app.  That makes it so the rogue plugin cannot, a) kill the queue and prevent other plugins from sending messages to the app, and b) flood the queue so the other plugin's messages are drowned out.

 

If robustness against rogue plugins is a requirement, you're going to need to implement another abstraction layer.  I don't think the AF has any disadvantage here.

[sktthemes]

In any “command pattern”-style process the messages are part of the process, so if one can write or load new messages then one is modifying the process.  

[Aristos Queue]

18 hours ago, sktthemes said:

In any “command pattern”-style process the messages are part of the process, so if one can write or load new messages then one is modifying the process.  

False. That assumes that the messages can do things that the existing messages could not do. Since the only operations available are those that were already available, the only thing a new message class can do is different combos of those operations -- the operations the class already defines as being safe to do (otherwise those wouldn't be exposed as public methods).

[drjdpowell]

I think sktthemes is a bot advertising something about photography promotion, as it just copied what I posted earlier.

[Neil Pate]

James I agree, I did report the post a few days ago.