A Robust Technique for Controlling a Parallel Processing Loop?

[AlexA]

Hey guys,

Some of you may have looked at the thread I posted recently about trying to refactor my code to not be a squiggly mess. I'm trying to break apart the functionality of the program into sub-vis as well as decoupling the UI from the processing. One of the problems I'm having is that I have a series of loops which are interacting with an FPGA. Most are listening to FIFO's waiting for blocks of data to arrive suitable for processing. One is triggering the initiation of data capture. I'm trying to wrap my head around how I can trigger these loops to start listening once the user is ready for that to happen (after they've performed some calibration and constant initialisation), but also to trigger the stop condition on these loops when the user doesn't want to listen anymore, and then to start it again when they're ready to.

Mostly I'm struggling to figure out how I can pass the stop condition across the multiple parallel loops (is the best way really to use a local variable like I have been)? I have tried to diagram below the structure of the code in abstract as I currently imagine it. Can anyone point me in a better direction for what I'm trying to do?

As a general question, is it ok to nest event structures and while loops like that?

Your wisdom is much appreciated.

Regards

[MarkCG]

I've been looking into using user events to this too. 1 event structure per loop works, but I don't know enough about the mechanics of event structures to say why two per loop causes problems.

[asbo]

Effectively what you have there is a Producer-Consumer architecture and that's perfectly okay (and actually a pretty popular pattern in LV apps).

There are a couple other nodes that are designed specifically for communications, check out Notifiers abd Occurrences. You may be more comfortable implementing one of those for signalling across parallel loops. Because you're only looking to send triggers, user events might be overkill.

[drjdpowell]

It might be better to use a notifier:

This design will start and stop the inner loop with the same notifier (and destroying the notifier in the upper loop will shut down the lower loop when you want the program to stop). Using a User event is more complicated.

-- James

[AlexA]

Yeah, there's a problem in that I also need to pass information to the "consumer" loop along with the "go" type instruction. I need to pass a reference to an FPGA vi. I got really excited and started off using a JKI state machine to handle the user interface and a few calibration routines. Now that I want to listen to the output of the stream from the "consumer" loop at any given time, the JKI architecture doesn't make as much sense. To go into a listening state locks up my user interface so the listening state can't be cancelled.

I think I'll have to move to a 3 loop architecture. I saw it around on these forums but can't remember what it's called, MVC maybe? I'm thinking a UI loop based on an events handler which enques messages for a general loop which handles messages and acts on them and a "stream" loop which starts and stops the data stream from the FPGA. When started the stream loop loads data via lossy enque into a queue and the act of dequeuing and processing is handled in the "general loop".

Should I handle constants which are initialised at program start-up, but subject to future reinitilisation depending on the result of a calibration routine, in the UI loop? Or in the "general" loop?

Arrgghh, started with a bang into this state-machine type coding and now it's all spaghetti but I just can't see the wires . Could someone link me a "getting started with state-machine/producer consumer" type reading, preferably free if such a thing exists.

[drjdpowell]

Yeah, there's a problem in that I also need to pass information to the "consumer" loop along with the "go" type instruction. I need to pass a reference to an FPGA vi.

The notifier can pass the information. My example shows a notifier with just a boolean, but you can instead use a cluster of information needed by the consumer loop. I've done exactly this kind of thing in the past. Use a User Event (again, a cluster) to pass the results back to the JKI loop event structure.

-- James

[crichter]

Thanks DRJDPOWELL for posting the notifier example. How would I correctly stop this VI? I tried adding a "stop" event case and I think only the producer loop stopped. I also tried adding a "release notifier" outside the same while loop, resulting in an error 1.

[Rolf Kalbermatter]

Thanks DRJDPOWELL for posting the notifier example. How would I correctly stop this VI? I tried adding a "stop" event case and I think only the producer loop stopped. I also tried adding a "release notifier" outside the same while loop, resulting in an error 1.

You need to close the Notifier refnum wherever you want the consumer loop to shutdown. That should be enough to make the consumer loop exit on the resulting error. You may have to work a little on the Get Notifier Status inside the case structure. The current example mixes in fact the actual notifier event with a means to detect if the inner loop should terminate. I personally would remove this and the inner loop completely unless you really want that loop inside the consumer to execute as long as the switch is on.

[drjdpowell]

Thanks DRJDPOWELL for posting the notifier example. How would I correctly stop this VI? I tried adding a "stop" event case and I think only the producer loop stopped. I also tried adding a "release notifier" outside the same while loop, resulting in an error 1.

Release the Notifier in your “stop” event case in the top loop; that will cause an error in the “wait on notification" in the bottom loop, causing it to shutdown (since the error is wired to the stop of the loop).

I personally would remove this and the inner loop completely unless you really want that loop inside the consumer to execute as long as the switch is on.

That was the use case of the original post; to start and stop listening on FIFO for incoming data. I use secondary loops like this for things I have to wait on, such as a TCP connection. If instead you want a second loop to execute commands from the primary loop, then you are better off using a queue. But you can still use the destruction of the queue in the primary loop to trigger shutdown of the secondary.

[crichter]

Thanks Guys. I guess I am weaning off the labview "milk" and tasting some meat.

[AlexA]

Hey guys,

 

It's been a long time since I've visited this topic but I thought I'd resurrect this rather than start a new one. I'm still uncertain about best practises.

 

Specifically, what is the best way to control a parallel loop that has some "continuous" type behaviour, plus config, init, reinit, stop, etc?

 

I've cartooned up a couple of the ways I'm aware of doing things (the notifier one I kind of made up, but it should work I think EDIT: the case should just read Default, not "False", Default).

 

At the moment, I'm working with a bunch of FPGAs, and our standard approach is to listen to them for data, continuously in a parallel loop, and write config style messages in a separate message handler loop which has direct access to the FPGA reference, so effectively, we're branching the FPGA ref. This has worked fine so far, but I'm always curious if there's a more robust way?

 

In general, how do we feel about handling the "main" continuous function of a loop in the timeout case of some communication system?

 

Regards all,

Alex

Edited November 2, 2014 by AlexA

[crossrulz]

In general, how do we feel about handling the "main" continuous function of a loop in the timeout case of some communication system?

I do that all the time.  Simple queue to send commands to the parallel process.  Have some reasonable timeout, do whatever is needed in the timeout case.  If a command comes in, do what is needed with that command.  But there is usally some need for state.  So sometimes a state machine works better.  Based on your reading from the FPGA, the simple timeout for read will probably work just fine.

 

And with your FPGA, play around with Dynamic FPGA References.  You can take you main FPGA reference and cast it into a more specific reference.  This way, you can pass into your subVI a very specific interface, making the code possibly reusable.  It at least performs some decoupling.

[ShaunR]

I've been using the following topology for a long, long time now and haven't found anything better that suits my needs.

 

It's just the next level up to your "cartoons" but gives you a complete message based service architecture (that transcends networks ). Mine is a little more complicated but not much - just deeper message decomposition and only replies if there is a question mark after a command ala SCPI. You can replace queues with events if you prefer. It's simple, easy to understand and scales better than any other generic framework in LabVIEW. When you hear me talking about "modules", I'm talking about the bottom half of that diagram. The only thing it's missing is a dynamic launcher for the service. You can also bastardise it with classes instead of the case statement if there aren't enough VIs for you

 

All the comments are just saying that whats on the diagram is normally inside sub VIs but the guts are displayed for understanding.

 

 

PS:

You'll find the Queue VI used here in Dispatcher but it's nothing special - just a self initialising queue wrapper for strings that I use in everything!!.

Edited November 3, 2014 by ShaunR

[AlexA]

Hi Shaun,

 

Yeah that looks very much like what I use, but I base it around Daklu's LapDog queue based system. The question I'm asking is targetted at the scenario where the "module" loop is intended to act as an interface for some piece of hardware which has a continuous operating state (constantly filling some sort of buffer with data), but also other states like "setting control parameters" or "updating setpoints".

Do you handle that sort of thing inside a timeout?

[ShaunR]

Hi Shaun,

 

Yeah that looks very much like what I use, but I base it around Daklu's LapDog queue based system. The question I'm asking is targetted at the scenario where the "module" loop is intended to act as an interface for some piece of hardware which has a continuous operating state (constantly filling some sort of buffer with data), but also other states like "setting control parameters" or "updating setpoints".

Do you handle that sort of thing inside a timeout?

 

It depends.

 

If it is a create/destroy type process then you don't need a queue at all (a service with no control interface). Just launch the service with a dynamic launcher and crowbar it (abort) externally using VI server. No queues, only the event broadcasting data and a while loop that you can stop dead in it;s tracks. If you need a bit more (Configure/Start/Stop/Shutdown) then untitled 1 contains a dynamic launcher itself and launches the acquisition when it receives Start, crowbars it when it receives Stop and complains when it gets configure whilst acquiring. So the acquisition is a sub-process of untitled 1 and untitled 1 is the acquisition "controller" that provides the system interface.

 

If you cannot just crowbar it, then you must go the timeout route (zero on the queue timeout) and effectively untitled 1 is a state machine with message injection rather than a message driven action engine (it's an important distinction),

 

There was a discussion on here a long time ago where we discussed these methods and Dalku and I compared notes with Lapdog. The only real difference was I used "->" as a message delimiter and case statements and Dalku used ":" and his Lapdog classes. The arcitectures were similar, it was the implementation that differed. You could also use the Actor Framework but then you are really getting complicated just to start/stop a VI (in fact, the AF always strikes me as complicated regardless of the size of task).

 

If you think about it, you are asking for the same features as an Multiple Document Interface which is just a sub module launcher with event feedback which is what you have above if you include a launcher.

 

Another example of the launch and crowbar approach can be seen in the Websocket server. When a connection comes in, the sub process is launched and runs asynchronously whilst the server listens for more connections and launches other services. If the server is closed, it just goes through the list of running services and crowbars them (abort then close FP). Each sub process is a service as described in my previous post but doesn't launch anything dynamically. So once launched, the client/user talks directly with the service (launched process) through a defined string API using their web browser. 

 

.

[AlexA]

Hi Shaun,

 

Thanks for all that info!

 

Would it be correct to say the distinction between state machine with message injection and message drive action engine (AE) is how the actions map to commands. I.e., a message driven AE gets you an action with a strictly defined end point, in other words it's gauranteed to complete. Whereas, for state machine with message injection, a single message could just result in a different continuous state?

 

A long time ago I looked at Actor Framework but there were so many layers of abstraction I found it impossible to get started, let alone port my code to it.

 

I'm pretty happy with what I've got at the moment, I've slowly improved it but I'm still wondering if I'm maintaining too much state in my message handlers.

 

For example, I have an FPGA which is driving a number of pieces of hardware (motors, electrodes etc.) The VI in charge of the interface to that FPGA consists of a continuous loop which basically just listens to a Target-To-Host DMA FIFO, as well as a message handler which takes requests like "Update Setpoint Profile" and computes a new profile before uploading it to the FPGA.

 

Should everything be done in a single state machine loop with message injection (I.e. listening is done in the timeout case).

 

My uncertainty stems from something someone said to me a long time ago on LAVA, that it was very strange that I branched the FPGA reference to two different loops, and that they had never needed to do that.

 

Thanks again for sharing your experience!

Edit: I get 403 Forbidden errors when trying to follow your links.

Edited November 3, 2014 by AlexA

[ned]

My uncertainty stems from something someone said to me a long time ago on LAVA, that it was very strange that I branched the FPGA reference to two different loops, and that they had never needed to do that.

Do you have a link to that discussion? There are so many things one can do with an FPGA, that I can imagine that in one application it would seem weird to branch an FPGA reference, whereas in another it's completely logical. One of the great things about the FPGA is that parallel loops are truly parallel and independent, so it can make sense (to me, anyway) to treat those loops almost as separate devices. Even if there's only one FPGA loop, I've written code where the host side forks the reference so that one host loop can handle the ongoing, routine operations (such as reading data from a FIFO) while another handles the occasional updates (changing a setting by writing to a front-panel element). At least in my opinion, if branching the FPGA reference to multiple loops lets you separate functions logically and results in clean code, there's no reason not to do it.

[AlexA]

Hi Ned,

Thanks, you've eased my mind a bit. I think this was the discussion http://lavag.org/topic/16908-q-what-causes-unresponsive-fpga-elements-after-restart/ but after re-reading it, I think I read something into it that wasn't there. Or perhaps I'm misrecalling where or if I saw that piece of information. Either way, from what you say I'm not worried about it anymore.

 

Cheers,

Alex

[ShaunR]

Hi Shaun,

 

Thanks for all that info!

 

Would it be correct to say the distinction between state machine with message injection and message drive action engine (AE) is how the actions map to commands. I.e., a message driven AE gets you an action with a strictly defined end point, in other words it's gauranteed to complete. Whereas, for state machine with message injection, a single message could just result in a different continuous state?

No. It's simpler than that. It is state. With State-machines (as the name implies) they control and own system state State is implicit and encapsulated. With a message driven AE, state is external and the interface is basically an API for encapsulation of discrete functions it's a classical action Engine with a message interface rather than typedefs..

 

I'm sure there are better academic labels for what I am describing because it's nothing new

 

A long time ago I looked at Actor Framework but there were so many layers of abstraction I found it impossible to get started, let alone port my code to it.

 

I'm pretty happy with what I've got at the moment, I've slowly improved it but I'm still wondering if I'm maintaining too much state in my message handlers.

 

For example, I have an FPGA which is driving a number of pieces of hardware (motors, electrodes etc.) The VI in charge of the interface to that FPGA consists of a continuous loop which basically just listens to a Target-To-Host DMA FIFO, as well as a message handler which takes requests like "Update Setpoint Profile" and computes a new profile before uploading it to the FPGA.

 

Should everything be done in a single state machine loop with message injection (I.e. listening is done in the timeout case).

 

It is a designers choice and pretty subjective so I'm not going to say "do this one". There are advantages and disadvantages over both. When it comes to RT, the crowbar method becomes much more attractive than in desktops as it has better performance generally and tends to have superior jitter since execution can easily be made constant,deterministic and predictable. With any messaging in the acquisition, then you may not be able to guarantee that and jitter will definitely no longer be constant or predictable and you may even lose determinism depending on what you do when you respond to a control message. However, you cannot configure the former easily "online" (whilst it is running). It is a Configure-Start-Acquire-Stop-Configure-Start-Acquire-Stop type of arrangement (offline). 

 

My uncertainty stems from something someone said to me a long time ago on LAVA, that it was very strange that I branched the FPGA reference to two different loops, and that they had never needed to do that.

 

Thanks again for sharing your experience!

Edit: I get 403 Forbidden errors when trying to follow your links.

 

403? You're IP address is probably banned....lol. Been visiting without a user agent? (PM me with your IP and I'll unban it).

 

later........

 

Nope. You're not banned. Your Uni must be blocking you.

Edited November 4, 2014 by ShaunR