Error Handling in messaging systems

[John Lokanis]

I am trying to solve a bit of a chicken-egg issue with my message system.  Since I think this would apply to any message system, I would like to hear how others have addressed this issue.

In my message system, each parallel process has a message handler loop that receives and executes messages.  A process can also have code that runs in parallel with the message handler to implement additional functionality, such as a UI loop that handles events.

If an error occurs in this UI loop, it sends the error to its message handler.  The message handler then executes the message with the error information, outputting the error as if it happened within the message loop.  This then propagates to an error handler that executes specific error handling code for that process and then sends the error info to an error logger process.

 

I also have implemented a broadcast message to shutdown the application.  This sends the shutdown message to all processes at the same time.

 

The problem I am running into occurs if an error is generated in the UI loop when it tries to send the shutdown message. That error is not getting handled.  This is because the message handler gets the shutdown message before the UI loop has a chance to send the error data.  So, there is nothing listening to the error message.  And even if it was still listening, the error logger process has also already shutdown so the error would not be logged either.

 

So, any ideas on how to solve this?  How do you deal with this in your messaging systems?

 

thanks for the feedback,

 

-John

[Tim_S]

I've solved this two ways:

 

1. Errors during shutdown get dropped. I try do do this only when the program has received a shutdown from the operating system (for instance, when the UPS is running out of juice).

 

2. Sequence the shutdown so that higher-level components shut down first. This has been the reverse order from starting up the system for me (e.g., startup might be load configuration->start error management->start security->load & start plugins->start UI, and shutdown would be stop UI->stop & unload plugins->stop security->stop error management->write configuration).

[John Lokanis]

...error management...

How does your error management work?  Do your parallel processes handle their own errors or do all errors get dispatched to a central error handler?

At first I planned to use a central error handler, but that would create a bottleneck that would block all processes when many errors occurred at once and could lead to deadlock.  So, instead I have each process handle its own errors and only use a central process for logging.

But, this leads to my current problem of the process not being able to handle its own shutdown errors.

I could just ignore them (that is effectively what happens now) but I am concerned that would mask other issues.

[hooovahh]

I'm not sure if this helps or muddies the waters.  One thing to think about is lets say you have your shutdown message is sent, and your logging part shuts down, but it shuts down, before the DAQ part finishes.  The DAQ part could send a message to the logging part waiting for a reply (not sure what kind of message DAQ would wait for a reply for but this is an example)  Your DAQ code could wait forever for your logging part to reply and it never will because it has already shutdown.  For this reason your messaging system either needs a long timeout in case something goes wrong so you don't wait forever, or have your messaging system be aware of the shutdown messages, and reply to messages that need replies saying to perform the shutdown, without needing the logging part of the code to be running.

 

This brings up another opinion of mine, and that is that separate running parts of code should never shutdown, unless they all shutdown.  So say there was an error in logging and you ran clean up, and this caused the logging code to close and stop.  Now we have the same problem as before where DAQ may ask for something from the logging code and it will never see a reply.  I've seen in the past were an error would kill some code, but you need to remember other parts of the code may need it to reply or receive messages.

 

EDIT: Also in my setup each parallel process handles its own errors if the error was not generated by another caller.  So DAQ asks logger for something that generates an error, this causes logger to send the error back to DAQ where the error is handled.  If logger generated an error on its own it will handle its own error (it has no place to send it).  Having a central error system is a novel idea but I too would be worried about system slow down.  In my setup any part of the code that will run often should be as streamline as possible.  So the messaging system is very minimal because I expect messages to happen all the time.  I've also thought about having a central location to keep track of states transitioned in each parallel process.  This has alot of benefit when it comes to debugging errors but again I haven't done it yet due to the overhead involved. 

Edited February 27, 2013 by hooovahh

[Daklu]

I agree with Tim, it requires a two-part solution. To add a bit to what he said:

 

1. You could create an error logging system that successfully logged every error, even during shutdown. That creates an opportunity for the application never closing. If the error logger encounters an error while it is shutting down, it has to start up enough to log *that* error, followed by another shutdown which throws an error... you get the point. So the first step is figuring out what errors you care about logging.

 

2. Professional applications (which I loosly interpret as anything built into an executable) really should have a controlled shutdown process. In my systems the top level actor issues the shutdown command and it is propogated down the hierarchy. However, no actor actually begins shutting itself down until it all its subactors have finished shutting themself down. So while the command is issued from the top-down, the shutdown process itself operates bottom-up. Occasionally I will customize the shutdown sequence so certain components (like an error logger) will be the last to shutdown, even though it is not the top level actor.

 

In practice I ignore most errors generated by an actor or loop once its shutdown process has started. Note that's not necessarily when it first receives the shutdown command. It's when that actor decides it's time to exit the loop.

[Daklu]

For this reason your messaging system...

 

I understand what you're saying, but I would rephrase it and say your messaging protocol needs to be designed appropriately.  Messaging system implies (to me) a framework for sending messages, like LapDog.Messaging, JAMA, or the messaging aspect of the AF.  The messaging protocol is the series of messages your actors exchange to communicate important information.

 

 

This brings up another opinion of mine, and that is that separate running parts of code should never shutdown, unless they all shutdown.

 

In practice that is what I implement.  If a low level actor has a fatal error and shuts down that message is sent up the chain to the top level actor, who in turn issues a shutdown command to everything else.  It's pretty simple to implement and fatal errors are discovered quickly.  There are, however, other valid ways of dealing with the situation.

 

Some applications benefit from being able to dynamically invoke and destroy actors.  In these cases there is no need to shut everything down just because a temporary actor is no longer needed.  My apps haven't required dynamic actors so I don't bother with it, but it is a valid use case.  Also, fatal errors in one actor don't necessarily require the entire app be shutdown.  The owning actor could detect the unexpected shutdown and spin up another actor to replace it.

 

 

Also in my setup each parallel process handles its own errors if the error was not generated by another caller... Having a central error system is a novel idea but I too would be worried about system slow down.

 

I agree every actor should handle its own errors as best as it can.  I've never found a scenario where I thought a centralized error handler was preferable to having each actor define its own behavior when it has an error.  These scenarios might exist--lots of people seem to use centralized error handlers.  Perhaps I'm just lacking in imagination.

 

 

So DAQ asks logger for something that generates an error, this causes logger to send the error back to DAQ where the error is handled.

 

Error handling (and your entire messaging protocol, really) is simplified if you use an event-based messaging protocol.  In other words, instead of the DAQ asking the logger for information the logger automatically sends information to the DAQ when an event of interest occurs.

 

 

I've also thought about having a central location to keep track of states transitioned in each parallel process.  This has alot of benefit when it comes to debugging errors but again I haven't done it yet due to the overhead involved. 

 

I handle these kinds of conveniences using a generic DebugMessage with a string payload.  Status messages that are not critical to system operation but would be helpful during development are wrapped in a DebugMessage and sent up the chain.  All DebugMessages get routed to an actor where the payload is extracted and added to a string indicator.  Sometime the actor is application's UI, sometimes it is a separate debug window.  In my systems the overhead to implement it is minimal.

[hooovahh]

I understand what you're saying, but I would rephrase it and say your messaging protocol needs to be designed appropriately.  Messaging system implies (to me) a framework for sending messages, like LapDog.Messaging, JAMA, or the messaging aspect of the AF.  The messaging protocol is the series of messages your actors exchange to communicate important information.

I explicitly avoided saying Actor throughout my post because not all of us use the Actor Frame work, and one could design the messaging framework in the way I described...and some of us have.

 

Some applications benefit from being able to dynamically invoke and destroy actors. 

In my experiences I've never needed to completely destroy a loop, just call the cleanup and init.  So I may have my logger close what it is doing and re-read settings but it is always running and listening for other "Actors" to respond to.

 

Error handling (and your entire messaging protocol, really) is simplified if you use an event-based messaging protocol.  In other words, instead of the DAQ asking the logger for information the logger automatically sends information to the DAQ when an event of interest occurs.

My example was a little simplistic.  Lets try this instead, a Sequencing Actor calls a DAQ Actor to read some values, the DAQ Actor reads the values then sends them back to the Sequencing Actor with any error that may have happened in the read.  I was trying to say that if shutdown happens the DAQ may shutdown, but my call from Sequencing to DAQ will wait forever waiting for the DAQ to reply with the data it requested.  Previously I used DAQ and Logging in my example but this makes less sense because I couldn't imagine a time when the DAQ would call Logging and want a reply.

 

I handle these kinds of conveniences using a generic DebugMessage with a string payload.  Status messages that are not critical to system operation but would be helpful during development are wrapped in a DebugMessage and sent up the chain.  All DebugMessages get routed to an actor where the payload is extracted and added to a string indicator.  Sometime the actor is application's UI, sometimes it is a separate debug window.  In my systems the overhead to implement it is minimal.

I like this idea but I would want all states transitioned.  So I can see what cases in my state machine, in each Actor went where, then where then where etc.  This then can give a very detailed view of what states my Actor went to and usually tell me why based on decisions that could have made it do those cases.  This central location of states and Actors would need to be a circular buffer of some kind, or possibly logged to TDMS and overwritten, or even just turned on manually.  This could open up all kinds of debugging tools.  Imagine if the User of my software has a bug that is reproducible   I could say turn on this checkbox then make the bug happen.  Then this one file could show all the states all actors went to, and with that I can very quickly narrow down what is happening.

 

I have done this in the past when I expect an error by using the History Probe.  Here I will probe the states wire (assuming I use strings as my state) and this probe will show all states that were executed by the Actor.  Of course this only works when running in source, but with a State Transition Actor I could use this in an EXE.

Edited February 27, 2013 by hooovahh

[John Lokanis]

Just to steer this back on topic, I still do not see a way for an 'Actor' that implements parallel loops to deal with an error when shutting itself down.  The AF uses these parallel loops to implement user interfaces.  But, if the UI code generates an error, it must send it to the message handler loop to process it.  And on shutdown, that loop has stopped.

 

To be more specific, the user commands the UI to shutdown the system.  The UI loop broadcasts the shutdown message to all processes.  One of the calls to a different process fails because of some error with its message transport.  That error is returned from the 'send to all' call up to the UI loop.  The UI loop then tries to send that error to its message handler but since it received and executed the shutdown message, it is no longer running.

The end result is I am unaware of a bug in my code where one of my processes has a bad message transport (for whatever reason).

 

You all seem to be telling me that I should just ignore this issue.  I would prefer to find a solution but it does not look like there is one.

[Daklu]

Just to steer this back on topic...

 

Quit interrupting.  Me and hoovah are trying to have a discussion.  

 

 

...I still do not see a way for an 'Actor' that implements parallel loops to deal with an error when shutting itself down.  The AF uses these parallel loops to implement user interfaces.  But, if the UI code generates an error, it must send it to the message handler loop to process it.  And on shutdown, that loop has stopped.

 

To be more specific, the user commands the UI to shutdown the system.  The UI loop broadcasts the shutdown message to all processes.  One of the calls to a different process fails because of some error with its message transport.  That error is returned from the 'send to all' call up to the UI loop.  The UI loop then tries to send that error to its message handler but since it received and executed the shutdown message, it is no longer running.

The end result is I am unaware of a bug in my code where one of my processes has a bad message transport (for whatever reason).

 

You all seem to be telling me that I should just ignore this issue.  I would prefer to find a solution but it does not look like there is one.

 

A couple comments that may help... or may make you to decide it's not worth the effort.

 

In my actors a message transport failure (i.e. the queue has been unexpectedly released) is a fatal error and always triggers an automatic shutdown.  That could help you in the scenario you describe, but it's not a general solution. 

 

You don't say what your actor tree looks likes, so I'm guessing it's something like this.

 

 

Broadcasting a shutdown message to all actors simultaneously limits your ability to control the shutdown process, even if an actor waits for all subactors to shutdown before shutting itself down.  In this design the error handler doesn't have any subactors, so it shuts down immediately after receiving the message instead of waiting for all the other actors to shutdown first.  You need to use targeted messaging, not broadcast messaging, and implement shutdown logic.

 

This is how I have implemented controlled shutdowns in similar designs:

1. The top level actor (UI) decides to shut down the app, so it sends "Exit" messages to Loops 1 and 2.
   
2. Loop 1 receives the Exit message and shuts down, sending an "Exited" message to the UI.  The UI notes that Loop 1 has exited.
   
3. Loop 2 receives the Exit message and sends Exit messages to Loops 3 and 4.
   
4. Loop 3 shuts down and sends Loop 2 an Exited message.  Loop 2 records that Loop 3 has exited.
   
5. Loop 4 shuts down and sends Loop 2 an Exited message.  Loop 2 records that Loop 4 has exited.  Loop 2 realizes that all its subactors have exited, so it exits and sends the UI an Exited message.
   
6. The UI records that Loop 2 has exited, and realizing that both Loops 1 and 2 have exited and any error messages they may have created would have been received, sends an Exit message to the Error Handler.
   
7. The error handler exits and sends an Exited message to the UI.
   
8. The UI sees that all subactors have exited and exits itself.
   

[ShaunR]

If you have a sequence engine. Then it becomes fairly straight forward to dictate what should happen when. This is one of the reasons why I use messaging with a queue for commands and events for response.

Most of my apps are present in the real world where you need to do things like set a slide to the home position before then turning the power off. It's an extension of your error handling problem and is one of sequencing.

The standard architecture that I have adopted is that only the sequence engine is allowed to issue commands and all other modules can only be listeners to other modules but not interact or act on them. An example of this would be a UI which can listen to the slide controller and show the position, but when the move button is pressed, the command must be sent to the Sequence Engine which will then tell the slide to move (and probably a load of other things too). This topology does not have inherent broadcast capabilities since each message must be placed on the appropriate modules queue and has to be considered and programmed by the designer. This is, however, trivial and the pro's far outweigh the cons since you dictate the order of commands and precisely target the processes. It is very rare that a broadcast control message can be acted upon without consideration to other processes.

[Daklu]

I explicitly avoided saying Actor throughout my post because not all of us use the Actor Frame work, and one could design the messaging framework in the way I described...and some of us have.

 

Whether or not you use actors is irrelevant.  I could have (and perhaps should have) said,

 

The messaging protocol is the series of messages your loops exchange to communicate important information.

 

 

...and one could design the messaging framework in the way I described...and some of us have.

 

The ability of messaging frameworks to incorporate those features doesn't support your claim that,

 

For this reason your messaging system either needs a long timeout...

 

Messaging systems don't need those features any more than a person without a car needs car insurance.  The reason you need those features is because you are using a message protocol designed around synchronous query-response communication.  If your protocol is designed around asynchronous event-announcements then the need for those features goes away.

 

 

In my experiences I've never needed to completely destroy a loop, just call the cleanup and init.  So I may have my logger close what it is doing and re-read settings but it is always running and listening for other "Actors" to respond to.

 

Yeah, me too.  But I can easily envision scenarios where dynamically launching actors loops would be beneficial.

 

 

I was trying to say that if shutdown happens the DAQ may shutdown, but my call from Sequencing to DAQ will wait forever waiting for the DAQ to reply with the data it requested.

 

The particular components involved in the sequence of messages don't matter.  Synchronous messaging (blocking one thread while waiting for another thread to respond) is rarely a good idea.

 

 

I like this idea but I would want all states transitioned.

 

Okay yeah, I can see why you wouldn't want to take the time to implement it.

[drjdpowell]

For this reason your messaging system either needs a long timeout...

 

Messaging systems don't need those features any more than a person without a car needs car insurance.  The reason you need those features is because you are using a message protocol designed around synchronous query-response communication.  If your protocol is designed around asynchronous event-announcements then the need for those features goes away.

 

Question: Doesn’t a fully asynchronous message system still need the concept of a timeout?  In your shutdown example, the UI loop is at some point waiting to receive the “Exited” messages from Loops 1 and 2.  If one of those messages fails to arrive, won’t it be waiting forever?  

[hooovahh]

Messaging systems don't need those features any more than a person without a car needs car insurance.  

No you are right, I didn't mean you need it as in it is a requirement, I meant it as you will would need it to prevent a dead lock, if nothing else was done.  The other suggestion I had (about knowing when a shutdown has occurred) was my preferred method.

 

Synchronous messaging (blocking one thread while waiting for another thread to respond) is rarely a good idea.

If this is true then I don't belong in this conversation.  I'd say the majority of my messages have a reply that a Actor will wait for.  When I ask my Power Supply Actor for the current state I wait for it to tell me the current state before moving on.  When I ask the DAQ to take a measurement I wait to get the value back before asking for the next value.  When I tell the Sequencer to "Start Test" I don't wait for a reply.

 

Okay yeah, I can see why you wouldn't want to take the time to implement it.

That being said I still think it would be a fun experiment.  Implement this with some OO or a DVR and have a central place with all states.  I already have something similar for the Actor messages.  I have an Actor that will intercept all messages between actors (along with the data send/received) and then I can display it using the handy Variant Probe.  Then during debug I can turn this on and can see at a higher level what is happening.  This is only turned on during debugging because it is not a circular buffer and could grown forever.  I've been meaning to fix that at some point.

[John Lokanis]

Just to clarify a few points:

I am not using Actors. I have created my own messaging system based off the way I did things pre-LVOOP.  All my processes (Actors, if you will) are peers.  There is no top level.  They can all message each other directly.  All messages are async.  The issue really is when one of these processes contains more than one loop running in parallel.  I took my design from the examples for implementing a UI with AF.  The UI code has its own loop, but the code also calls the parent class in parallel where the message handler is implemented.  The issue comes from trying to deal with errors at shutdown between the child implementation containing the UI loop and the parent message handler.

I understand Dave's solution and I agree that sequencing a shutdown makes sense among the various processes (Actors) but I am still stuck with this case where we have overridden the parent message code to add a UI.  It still seems like there is no clean way to deal with this.

Edited February 28, 2013 by John Lokanis

[Daklu]

Question: Doesn’t a fully asynchronous message system still need the concept of a timeout?  In your shutdown example, the UI loop is at some point waiting to receive the “Exited” messages from Loops 1 and 2.  If one of those messages fails to arrive, won’t it be waiting forever?  

 

No.

Yes.*

 

You are correct that if one of the Exited messages fails to arrive then the UI loop will never shut down.  That's exactly what I want to happen.  Putting a timeout in the UI loop permits a resource leak--Loop 1 can continue operating after the rest of the application has been shut down.  By not using a timeout any errors in my shutdown logic are immediately apparent.  Fail early, fail often.  The hardest bugs to fix are those I don't know about.

 

(*"Waiting" in the sense that the UI loop will never shut itself down, not in the sense that it is blocked or prevented from processing other messages it receives.)

 

 

If this is true then I don't belong in this conversation.  I'd say the majority of my messages have a reply that a Actor will wait for.  When I ask my Power Supply Actor for the current state I wait for it to tell me the current state before moving on.  When I ask the DAQ to take a measurement I wait to get the value back before asking for the next value.  When I tell the Sequencer to "Start Test" I don't wait for a reply.

 

I guess my question is why do you ask the Power Supply for its state?  Why don't you just have the power supply announce any state changes?  That way your caller always has the most up to date information and you don't have to block it waiting for a response.

 

Even if you do use a query-response protocol, you can design it to be asynchronous and non-blocking.  It often requires the caller keep track of more state information of the kind "I've sent the power supply a QueryState message and am waiting for a reply," but it's still usually preferable to blocking the calling thread in my opinion.

[John Lokanis]

As for sync/async, I handle this by having a request message that contains the name of the sender.  The receiver then executes the message, gets the requested data and then sends a different message to the requester with the answer inside.  The requester executes this message containing the data, taking whatever action needed the data to perform its work.  This allows the requester to perform other actions (like a shutdown) while waiting for the data it requested.

(dammit, not you got me going off topic...    )

Edited February 28, 2013 by John Lokanis

[Daklu]

The UI code has its own loop, but the code also calls the parent class in parallel where the message handler is implemented.

 

I'm not following so let me talk this out.  You tell me where I go wrong.

 

You have a parent class that implements some sort of message handling code in a method I'll call "Execute."  You have created a subclass which contains UI elements and overridden the Parent.Execute method.  On the UI.Execute block diagram, you use Call Parent Method to invoke the parent's message handling code and in parallel you have a loop for catching front panel events from the user.  The problem you're having is that on the UI.Execute block diagram the parent message handler is exiting before the UI loop, which leaves the UI loop stranded.  Is that correct?

 

Do all your actors inherit from that parent?  What kind of message does the parent handle?  Do you use a case structure or command pattern message handler?  Is the UI loop an independent actor able to communicate directly with actors on other block diagrams, or is it a helper loop that works with the parent's message handler (and perhaps other helper loops) to present create an actor?

[John Lokanis]

Dave: Yes, that is basically it.  When the parent quits, a local user event is fired to tell the UI loop to quit.

Yes, all my "actors" inherit from that parent.  Think of it like the Actor Core in AF.  The parent handles all messages for that 'Actor'.  It also contains all state data.  Inside the parent code, I use dynamic dispatch to execute the various messages received.  The UI loop is a 'helper', not another 'actor'.

I have implemented override-able error handling within the parent message code.  My 'helper' UI loop simply sends errors to the message handler to be dealt with.

 

It occurs to me that my only solution might be to have the 'helper' UI loop call the error handler directly instead of passing it to the message handler.  That might mean I cannot reuse my error handler method as easily as I wanted to.

 

Also still leave me with the need to stagger the shutdown for the error logger so it can catch any shutdown errors from the other processes.

[Daklu]

As for sync/async...

 

Yeah, that's the kind of scheme I had in mind.  It can work, but it's not as simple or clear (imo) as an event-announcement protocol.  If the event-announcement messages from an actor occur so frequently there is risk of bogging down the messaging system I'll see if other load reduction strategies can be used, like:

 

-Remove some events from the list of announced events.

-Make the announcement based on a timer event instead of an OnChanged event.

-Bundle multiple announcements into a single message.

 

If these can't be applied I'll look into more complex messaging ideas like Futures or asynchronous query-response messages.  Synchronous query-response messages are a solution of last resort for me.

[John Lokanis]

If you think of each process (Actor) as it's own state machine, it makes sense to link them together with async messages.  I guess it is more of an interrupt-able state machine, however.

Personally, I like my system because my transport can be overridden with different implementations.  For example, it could be a notifier or even an array of notifiers to allow many-to-one lossy messaging.  Or it could be a VI Server transport to send a message to another application.

[Daklu]

It occurs to me that my only solution might be to have the 'helper' UI loop call the error handler directly instead of passing it to the message handler.

 

I don't think you want to do that.  There's value in routing all messages between helper loops, subactors, and external actors through a central point inside the actor.  (Usually the message handling loop in my code.)  That central point maintains all the important state data the actor needs to operate correctly.

 

Once you allow internal loops to communicate directly with external actors, you open the door to race conditions.  The component is announcing an event (an error) the brain (the message handler) doesn't know about.  To illustrate the race condition, let's pretend you have two actors:  Actor and OtherActor. 

 

Actor has a message handling loop and an interal helper loop.  We don't know how OtherActor is implemented, but we know that if it receives an error message it will in turn send a Reset message back to the Actor.  But, we don't want the Actor to always accept and process Reset messages.  We only want to do that if the Actor is in an error state.

 

When the messages are routed through the message handler the message sequence goes like this:

1. Helper loop sends error message to message handler.

2. Message handler notes error, switches to error state, and sends message to OtherActor.

3. OtherActor sends Reset message to Actor.

4. Actor receives message, confirms it is in an error state, and initiates the reset.

 

When the helper loop bypasses the message handler this sequence is possible:

1. Helper loop sends error message to message handler and OtherActor.

2. OtherActor sends Reset message to Actor.

3. Actor receives Reset message, finds it is not in an error state, and discards the message.

4. Actor receives error message and switches to error state.

 

Will that happen?  Probably not.  But the race condition exists and I'd rather *know* my app will behave as expected rather than *hope* my app will behave as expected.  (In my experience by the time race condition is discovered there will be so many race conditions present it is very difficult to remove them.)

 

Alternatives?  You have overridable error handling.  Why not implement overridable shutdown code?  Customize each one to make the message handler waits for the helper loops to shut down before they are allowed to shut themself down.

If you think of each process (Actor) as it's own state machine, it makes sense to link them together with async messages.

 

Each actor certainly has the potential to become an independent state machine.  Or another way to look at it is each actor *is* its own state machine, it just starts out with only one state.  That's kind of what actor-oriented programming is about.  Do you think about actors differently?

[ShaunR]

Dave: Yes, that is basically it.  When the parent quits, a local user event is fired to tell the UI loop to quit.

Yes, all my "actors" inherit from that parent.  Think of it like the Actor Core in AF.  The parent handles all messages for that 'Actor'.  It also contains all state data.  Inside the parent code, I use dynamic dispatch to execute the various messages received.  The UI loop is a 'helper', not another 'actor'.

I have implemented override-able error handling within the parent message code.  My 'helper' UI loop simply sends errors to the message handler to be dealt with.

 

It occurs to me that my only solution might be to have the 'helper' UI loop call the error handler directly instead of passing it to the message handler.  That might mean I cannot reuse my error handler method as easily as I wanted to.

 

Also still leave me with the need to stagger the shutdown for the error logger so it can catch any shutdown errors from the other processes.

Why not just have a "Shut-down" actor which knows what order to shut things down?

[Tim_S]

Why not just have a "Shut-down" actor which knows what order to shut things down?

I've used some aspects of the actor-framework, but not implemented it wholly so this may be a silly question... Wouldn't the top-level know the order to start up/shut down?

[hooovahh]

I guess my question is why do you ask the Power Supply for its state?  Why don't you just have the power supply announce any state changes?  

Because Power Supply (and all other Actors) are Idle. (with a few exceptions)  They can be doing work every timeout but I'd rather they only perform task when needed, this cuts down on processing.  I can have hundreds of Actors all of which are idle, not polling anything.  Also I don't see the benefit from broadcasting my power supply status constantly when I may only want it at the end of a test (to see that it is still stable).  If I want PSU voltage for a UI element I can tell it to do it's thing periodically and push it to a VIG but rarely do I do this because I don't know how hold this data is.  If I request the data and wait for the reply I know that the data I got was from the operation just performed.  

 

What you are describing sounds like a Publisher/Subscriber paradigm, Is this how the Actor Framework works?  I assumed (from what little I've seen) that it was command/response using user events and queues.

 

I feel like these conversations are beneficial, but difficult to digest some times.  We all talk about what we do and how we do it, but it takes alot more time to put into text what we are trying to say and, concepts we are trying to share.  This tends to makes posts long and it doesn't take long to fall behind in the conversation.  Oh and I do like the idea of a Shutdown Actor but I haven't needed on yet.

[Daklu]

I've used some aspects of the actor-framework, but not implemented it wholly so this may be a silly question... Wouldn't the top-level know the order to start up/shut down?

 

The traditional LV model of a top level vi that all other vis are a dependency of doesn't always translate well to actor oriented programming, especially if using the Actor Framework since it dynamically launches actors.  Once the startup code has launched the actors and made the necessary connections it can exit, leaving each actor as its own top level vi.