Dynamic Sub VI Control: Practical Implementation Thereof

[AlexA]

Hi guys,

I've had to re-do some code I've been working on as it needed to be deployed to a real-time system. I'm taking this opportunity to make some architectural changes because some of the inter-process signalling felt really dirty. Here's a brief outline of what I'm doing.

Host machine VI controls Real-Time VI via network stream.

Real-Time VI take messages sent by network stream and performs some "program control" type logic (telling other mediator loops to launch sub-VIs, launch an FPGA based VI, etc. etc).

I'm currently wrestling with the practical implementation of an abstraction I want to use (because it feels right). That is, I want each dynamic sub-VI to be launched with it's own pair of messaging queues, one to receive commands from it's master, the other to send status reports back to the master.

The problem I'm having is I can't think of a really elegant way for the Sub-VI launcher to monitor "n" status queues.

To give some more context, the Sub-VI launcher loop has it's own pair of messaging queues which connect it to the Real-Time VI's master loop, that is, the loop that handles commands from the host machine via the network. Previously the sub-VI launcher loop has presented it's own command queue (that is, the command queue used by the Real-Time master loop to tell it to launch VI's) to listen for and handle status up-dates from the Sub-VI's. This has run me into trouble when trying to elegantly handle shut-down orders as suddenly the sub-VI launcher loop receives a stack of status updates from a raft of different Sub-VI's and must shut them down in the correct order and then report to the master loop so further action can be taken.

I feel as if somehow abstracting communication between the Sub-VI launcher and it's progeny to an individual messenger pair per child will allow me to perform sequences of shut-down events elegantly.

So, to synthesise, how can a master-loop elegantly monitor all its dynamically launched children and still respond to commands from its master?

Thanks for any insight you guys give,

Alex

Edit: A final note, I would prefer to stay away from an OOP solution for now. I've tried in the past to look into it but I just don't have the time to study it at the minute. So if you could confine answers to standard LV "state-machine" type architecture (a while loop enclosing a case structure), that would be much appreciated.

Edited January 17, 2012 by AlexA

[drjdpowell]

I think going away from “one incoming message queue per message loop” would be a mistake. If your having problems on ordering shutdown, perhaps those can be solved a different way. How do you shutdown your processes? If you use a technique that tells all processes to shutdown at the same time (like a common global, say), than a better technique is to shutdown processes individually by sending individual “shutdown” messages to them. Send the shutdown messages in the order needed, and if necessary, wait for each process to acknowledge shutdown before ordering another to shutdown.

[ShaunR]

Usually, each "module" will have it's own command queue and they will all use a common "status" queue rather than separate queues to feedback to the master.

The thing to bear in mind is the topology. Queues are "Many-to-one". So for the commands, you can have many modules/controllers/whatever placing commands on that one modules queue. In reverse, you also have a "many-to-one", but now it is many modules placing status info on the controllers queue.

Another way is to use a single "Control" notifier (a notifier is a "One-to Many") for the modules and each module filters the command sent. This has the advantage that you can "broadcast" to all modules things like Start, Stop, Pause etc which are global to all modules, but still enable defined commands for single modules which the module responds to. It has the downside that you can only have 1 command in the pipe at any-one time so for sequencing you have to arrange for some other mechanism. But the difference in this case is responsibility. The module is responsible for actioning the messages (shutting down, starting up etc) and the "Controller" just becomes the orchestrator. This is what I term "Autonomous Modules" and greatly simplifies the upper hierarchy at the expense of more complicated modules.

You can also use a combination of both. for example, I may have a command "Queue" but broadcast back information via a notifier (great for UI stuff)

[AlexA]

The thing is, if a single status queue is used for the master to action on statuses from the subVIs, that queue must be the same queue that is driving that master's state machine (else you have the problem of the master loop reading two queues anyway, one for statuses and one for commands).

That feels wrong to me, I'm not sure why, but I feel like the purity of communication should be 2 way only, for sub modules to tie into a masters command queue to get their own statuses across feels dirty.

I've actually come up with a message handler that "locks" like a queue with no time out wired, but manages to poll N different high-priority queues and one low priority queue (the masters command queue). I'll link a png of it when I get back to work tomorrow, but basically it polls N high-priority queues and if there is nothing in them polls a low priority queue, if there's nothing in that queue either, then the it reloops until a message is found on at least one queue.

Considering the use case where the user asks to launch 3-4 modules in sequence, there is a danger (handled by setting sufficiently long time outs on the queue polls in the message handler) that somehow the module launcher control(master) queue will progress right through its requested launch sequence before any of the Sub VI's actually manage to fire up and return their statuses. As long as at least one Sub VI gets its statuses and handshaking requests into one of the N high-priority queue it's likely that all of them will. I can't really think of a more elegant way to handle sequences like that.

So to be clear on how I have been passing sequences of commands that require a confirmation. The UI code will take an event and turn it into the correct sequence of Sub VI launches which are sent to a Sub VI handling loop, at the end of this sequence it loads the specific command "Done" which is a handshaking command, what that does is that when the Sub VI handling loop dequeues "Done", it loads its own "Done" command back onto a return queue which goes to the UI loop, which is waiting at a dequeue.

Do you have any recommendations for handshaking logic if you use only a single status report queue? I couldn't figure out how to make the Sub VI handler wait for a hand shake from the subs after each one it tried to launch if there was already a sequence of launch requests in the master queue (hope I'm being clear). The best I came up with when working with a single return queue for all Subs as well as commands from the higher level stuff, was to allow requests from Sub-VI's to be enqueued at the opposite end, so as each Sub VI fired up it loaded its own requests on to the top of the queue. The problem with this, is that there's still a forseeable chain of events where a Sub-VI's start up routine takes so long that before it can load its request onto the queue, the queue finishes its sequence and returns command to the top level code. In fact, I seemed to see this case frequently with a certain launch sequence. I really don't like the idea of incorporating "Wait x seconds" in a launch sequence as it feels crude and superfluous if I can just come up with the right hand-shaking logic.

So, once again, to crystallise my ponderings, when working in an environment which has been specifically coded to be asynchronous, what is the best way to ensure that a strict sequence of events is followed?

Edit: The more I think about it, the more dirty my new code feels to me, some of the launched sub-modules need to pass data between each-other, there's a strong potential for a consumer to request a queue before a producer has published it, requiring re-request code. Ugggh, if I could just think how to ensure that each Sub-VI gets all the way through its launch sequence before the next one is launched (WITHOUT using crude "wait x seconds" code) I'd be set.

Edit 2 (2 minutes after edit 1): I supposed I could format the commands such that they can be parsed for specific information (like the command string could read "Status:Request Data" from a Sub VI and "VI Control:Launch X" from a master level command. That way perhaps commands could be dequeued in some sort of order then requeued if they don't match whats...required...at the time? Hmmm, have to ponder the possibilities a bit more. I wish I wasn't under PhD time pressure, I'm enjoying Labview more than muscle bath design at the moment!

Edited January 19, 2012 by AlexA

[drjdpowell]

You should look into something known as a “callback”, though I just think of them as a “return address”. You make a method of replying to the message part of the message, rather than sending replies to a dedicated command queue. The process making the request can always attach its own command queue if it wants replies to return in sequence with all other commands; but it also has the option of creating and attaching a temporary response queue and waiting for the reply on that queue (destroying it afterwards). Thus you can use a callback asynchronously OR synchronously as needed. So you could Start module A, wait for it to reply “Done”, then Start module B.

Here’s what a subVI that does an synchronous send-and-wait-for-reply looks like in the messaging system I use:

It creates a new queue, attaches it to the message, sends the message, and waits (with timeout) for a reply, before destroying the queue.

The process at the receiving end of the message calls the “Reply” method on the message in order to send a response back along the temporary queue (unless there is an error, in which case it replies with an error message instead):

I use this when I start up modules; I send a series of initializing messages, with the last one being sent as a Query, where I wait for a repines from the module. For example, here is a startup of a “Camera” module:

Ignore the details, but you can see a series of messages sent, with the last message being sent as a Query. I don’t even care what the response is, other than if it is an error (which is automatically sent out the error terminal).

— James

Edited January 19, 2012 by drjdpowell

[ShaunR]

This is a simple example of the sort of thing I do with initilising/launching sub processes.

The top example is fully synchronous and will launch the vi and wait until it gets the feedback from that VI before proceding to the next.

The bottom one is the sort of thing I do mostly since it launches and initialises all the vis and then just waits until it gets the responses back. You can make it as complicated as you wish (check the responses rather than just waiting for a set number as I have shown or "gathering" them in a while loop instead of a for loop) however I think you can see the pattern.

You will notice I am not talking about a "controller" here because it depends where you are using this. Generally, the UI will talk to a controller (you may have more than one e.g IO controller or sequence engine) and the controller will use this to launch its sub vis.

Edited January 19, 2012 by ShaunR

[AlexA]

Hey guys, thanks heaps for your help! You've really seeded my mind with some new avenues to think along. For anyone interested, this is the poll N queues message handler I came up with.

As I noted in my previous post and in the block diagram, this is not a very good method unless you don't care at all what order the sub VI's start in.

Thanks again for the help, will update when I've solidified a method.

[Steen Schmidt]

@AlexA: You could take a look at my ModuleControl toolset for inspiration - it's mentioned here. Unfortunately the toolset isn't yet publically available, but will be in the near furture on NI LabVIEW Tools Network (three other of my toolsets are arriving there as I write this, under the GPower name).

Cheers,

Steen