Factoring Code and Instantiating Parallel Processes in subVIs

[AlexA]

Hey all,

I'm currently constructing a whopping program that after tidying (and commiting the crime of hiding a routine in a stacked sequence structure) is a couple of screens across horizontally. I was wondering if anyone would be keen to upload a screen shot of the block diagram of their current massive project. I'm not really interested in what it does, just the individual approaches people are taking to keep their code tidy and easy to interpret. Also, if anyone knows of a really good resource for tips regarding organising your code elegantly in labview, please let me know.

Regards all,

Alex

[dannyt]

Hi,

You seem to link the size of you program /project to the size of you VI block diagram and this is not the case.

For your own sake and the sake of anybody else who works on or supports your LabVIEW code look at breaking your VI down with the correct usage of SubVI's.

Historically our project contained a number of large block diagram but over time we have broken these all done in single screen one, in most case a lot less than a single screen. The benefit are huge in terms of understanding what each subvi is doing and seeing at a glance the flow and intend of your code. I shudder to think how we used to do it.

The creation of SubVI may at first seem like an unwanted overhead in time and effort when you start, but the pay back over time is huge.

Our project control at testing of our full product line, it contains over 1000 VI's 90%+ of which have block diagrams around half a screen size.

I would be worth looking at these sources

http://www.ni.com/pdf/manuals/321393d.pdf Chaper 6 of the LabVIEW Development Guidelines

http://www.bloomy.com/lvstyle/ - a very good book on LabVIEW style

cheers

Danny

[drjdpowell]

Here's an image of a recent large (for me at least) project:

This is the top level of the program, which is a "Database Viewer" for displaying several different types of measurement records in a database. It is based on a free template available from JKI, a form of something known as a "Queued State Machine" or QSM (though Queued Operation Machine would be a more apt name). In a QSM, a large program is organized into various frames of the outer case structure, with some queuing mechanism for calling the frames in order, and some kind of memory for organizing data available to all frames. Google "JKI state machine" for more info.

Not so obvious, but seen in the image is another technique: clustering closely-related information on one wire and having a set of subVI's that act on that wire. This allows the top-level diagram to be much clearer and simpler. The best way to do this is with LVOOP "Classes", though one can instead use type-def clusters. In the image you can see the use of a "Record" class, and most of the complexity of the program is hidden in that class's subVI's.

-- James

[jcarmody]

I, too, like the JKI State Machine. This isn't the largest program I've written, but it's what I'm releasing now. LVOOP, as drdjpowell mentioned, goes a long way toward making your Block Diagram more manageable.

[P de Boevere]

My best advise is to leave the vi for a year and do all sorts of other projects and learn and apply the basics of architecturing large LabVIEW vi's.

Then look at your current work in progress, throw everything away and start from scratch.

[AlexA]

Hey everyone,

This is exactly the sort of advice I was looking for, some really beautiful little block diagrams. I'm definitely going to work on getting the code broken down into sub-vi's. Thanks for your examples.

@P de Boevere: Hahaha, I wish I had that luxury, unfortunately I'm a lowly PhD student trying to thrash out all the stuff I've been tasked with doing.

For your entertainment pleasure, I've included the block diagram of my main vi:

Obviously I've got a lot to learn...

[asbo]

For your entertainment pleasure, I've included the block diagram of my main vi:

I'd like to know how long a BDCleanup takes on that Something like jcarmody's QSM might just kill it

[AlexA]

Hahaha, there's definitely a bit of thumb twiddling going on while the BDCleanup goes. Today is refactor day so I'll post my results in 7 hours or so of how far I've got rationalising it and tidying it all up. Wish me luck!

[dannyt]

For your entertainment pleasure, I've included the block diagram of my main vi: Obviously I've got a lot to learn...

I was actually expecting a lot worse than that from your initial post :-) I have certainly seen a lot worse that that.

[Cat]

For your entertainment pleasure, I've included the block diagram of my main vi:

You just need to liberally use "Create SubVI" in logical places.

I'm generally pretty anal-retentive about the 1 screen BD rule. However, I have one vi that is such a humongous mass of spaghetti code I'm too embarrased to post it. It's about 6 screens. It already has almost 100 subvis and still needs more. At this point I'm looking at just rewriting the whole thing. If something gets that messy visually, there's a good chance it's messy logically, too.

[drjdpowell]

For your entertainment pleasure, I've included the block diagram of my main vi

Could be worse, look at a project of mine from a few years ago before I learned anything about architecture (the labels you can read are in 106pt font!!!):

-- James

PS: One thing you need to attend to, though, is your wire routing. You have many places were your wires go under other structures in very confusing ways (I think one wire is running backwards under another wire!). Learn to use the "Clean Up Wire" Menu option.

[AlexA]

Hey guys,

I imagine you're getting heartily sick of my flurry of questions by now. This one's been bothering me for a couple of days. Is there a conventional way for triggering a continuous process inside a state, without locking the state machine? The only hint I've seen how to do this is to place the process inside a sub-vi and launch it with "wait until complete" set as false.

I've also read a little bit about Daemon's on the lava wiki, is this what I should be looking at?

What I'm trying to achieve is:

I have an FPGA which performs two functions, generates a stream of images, and performs a set of calculations resulting in a single number out which describes a characteristic of each frame. In my host code, I want the user to be able to interact with these two functions independantly. I currently have a state machine/producer-consumer architecture. The UI loop is the producer and loads commands into a queue which is unloaded by the consumer and acted upon. Currently I have a third loop bolted on to this which when triggered, starts listening to the FPGA FIFO for data and packaging it up into a Queue.

I would like to if possible, move this bolted on loop into a sub-vi which can be triggered and stopped all from inside the state machine. I would also like to create a similar sort of triggered "listening" state which dequeues from the FIFO gathering loop and processes.

Anyone got any good ideas for dealing with the goal of triggering continuous processes (and stopping them arbitrarily) from inside states, without locking the state machine.

Regards all,

Alex

Edited September 11, 2011 by AlexA

[AlexA]

So, I've learned the value of scratch pad VI's . I've figured out how to run a VI in the back-ground, haven't tried listening to its queues yet but that's next. Still a little bit stuck on how to close it properly. When I call close VI reference it works but leaves the VI running, when I use abort, then close, it doesn't actually stop and then close reference returns an error. Anyone got any insight into this behaviour?

[drjdpowell]

Hi Alex,

It would be better if you continued your original topic, rather than starting a new one. Conversations like this serve as a resource for later readers (I've learned lots from reading conversations on LAVA) and splitting up the conversation across many topics makes it confusing and less readable.

While dynamically launching a VI as a parallel process ("daemon") certainly works, its a bit tricky and often over-kill for what you need. I would really recommend you use a simpler solution with separate loops on the same block diagram, with queue/notifier/UserEvents connecting them. Like the Notifier-to/UserEvent-from design I suggested in your other topic, which does everything you want. Note that you can easily convert your simple solution to a dynamically-launched VI at a later date, but this is worth doing mainly only if you want to reuse the component in another program or have the ability to "launch" an arbitrary number of them.

-- James

Edited September 12, 2011 by drjdpowell

[SteveChandler]

Top level VI - no controls or indicators at all.

Function loops - the thing on the bottom left of the above VI with a big green arrow.

The GUI. One of the VIs in the above VI.

[AlexA]

Ahh, apologies for the split. Could I request a mod merge this with this thread: http://lavag.org/topic/14883-examples-of-massive-programs-and-your-block-diagrams/ and a change of title to that thread, something like "A conversation on factoring code" (and a subsequent lock to this thread). I intend to upload an example of my now factored code into that thread shortly once I'm happy with some of the functionality.

As I hope this thread will be merged I'll add the following comment here:

I've managed to figure out how to dynamically load VI's and leave them running in the background. It might be overkill but boy do I like it! It really compresses things and makes the block diagram easier to intrepet and modify in my opinion.

[AlexA]

Ok, now I am stumped, I've been merrily going along triggering VI's from my main VI, all working as intended. Until I got to one that I didn't write, a VI that acts as a serial server allowing the transfer of commands through the code to the camera attached to the FPGA. I've tested this VI in two ways. If I try to run it using the Run VI method it results in a time out error with the camera configuration software. If I run it as a sub-vi (the way it's run in the example code provided with the NI-1483 adaptor) inside the exact same state (which locks up the state machine). Then it works fine, but of course I can't then leave it alone and go do other things. I could of course just write a dynamic user event that transfers the start serial server command to another loop, but I'd much rather understand why Run VI doesn't work like launching it as a sub-vi in this case.

So my question, what would be difference between the two runs (I've taken care to use the Ctrl Val.Set method to make sure all the inputs are the same in each case)?

Edited September 13, 2011 by AlexA

[AlexA]

Ok,

I've attached a zip of my project as it currently stands. I've taken the approach where I separate functionality into dynamically launched sub-VI's. I have a feeling like I'm a babe walking into a dark wood and the sun is setting quicker than I realise. It's quickly getting out of my control and it's all stemming from trying to separate things into small chunks which are supposed to operate in parallel but be started individually and arbitrarily.

It would be great if someone could have a glance over the project, specifically "main.vi" and the subs it calls, and tell me if there's a better way of doing what I'm trying to do. Specifically the idea of running parallel processes which are meant to talk to each other from a single, management type, user events handler. The reason I was trying to get things all being triggered from a consumer type events manager is that to do otherwise (i.e. use dynamic events to trigger the parallel loops in the main host) is that in terms of foot-print it really wouldn't be any more efficient than what I was doing already.

Regards,

Alex

[Grampa_of_Oliva_n_Eden]

Ok,

I've attached a zip of my project as it currently stands. I've taken the approach where I separate functionality into dynamically launched sub-VI's. I have a feeling like I'm a babe walking into a dark wood and the sun is setting quicker than I realise. It's quickly getting out of my control and it's all stemming from trying to separate things into small chunks which are supposed to operate in parallel but be started individually and arbitrarily.

It would be great if someone could have a glance over the project, specifically "main.vi" and the subs it calls, and tell me if there's a better way of doing what I'm trying to do. Specifically the idea of running parallel processes which are meant to talk to each other from a single, management type, user events handler. The reason I was trying to get things all being triggered from a consumer type events manager is that to do otherwise (i.e. use dynamic events to trigger the parallel loops in the main host) is that in terms of foot-print it really wouldn't be any more efficient than what I was doing already.

Regards,

Alex

This image from from my Design Albumn on the Dark-Side

is a SSD detailing the interaction of the various components in an application that required spawning an unkown (at development time) number of data collectors display their data and consolidate all reading in a single file.

If that is helpful you may want to review some of the other albumns in my image gallery here (you can skip the family of course). many of those images have links to the original thread where they were posted.

Have fun,

Ben

w

[drjdpowell]

Alex, your project zip is missing the two dynamically-launched VI's. Can you upload a new zip that includes them? If so, I'll have a look at it.

[SteveChandler]

This image from from my Design Albumn on the Dark-Side

Hey Ben, what do you use to create your sequence diagrams?

[Grampa_of_Oliva_n_Eden]

Hey Ben, what do you use to create your sequence diagrams?

Yikes!

I don't have official software so PowerPoint (blush).

Ben

[AlexA]

@drdjpowell

Hey, the files are in there, in the Sub-VI's folder, but in my code they're using absolute paths, so I guess that's why they wouldn't be launching for you.

On another tack:

Last night as I was tossing and turning I decided to browse the LAVA OOP sub-forum and I found this two part tutorial by Michael Aivaliotis from JKI which does exactly what I was trying to do, but more elegantly. So for people following along with my case or looking at a similar problem here is the link again.

If I have edit priviledges, I'm going to change the title of this thread to more accurately reflect what's here.

Note: From "Examples of massive programs and your block diagrams" to "Factoring Code and Instantiating Parallel Processes in Sub-VI's"

Edit: Ok, I don't have priviledges so I'll send a PM to the moderator.

Edit 2: Ok, colour me stupid but I can't seem to find a list of the moderators responsible for this sub-forum, or even a list of moderators full stop, can anyone let me know who to contact?

Edited September 13, 2011 by AlexA

[asbo]

Edit 2: Ok, colour me stupid but I can't seem to find a list of the moderators responsible for this sub-forum, or even a list of moderators full stop, can anyone let me know who to contact?

Well, the members page offers options to filter by group, but it seems like the way that the only admins/mods that show up are the bot-y ones. I would just report your own post and leave a description. Any moderator or admin will be able to see it then.

[drjdpowell]

I've attached a zip of my project as it currently stands.

Hi Alex,

I think your being way too ambitious, and trying to develop many advanced concepts simultaneously. Personally, I couldn't learn and use dynamic VIs, queues, QSM architecture, etc. on top of learning FPGA and your imaging equipment itself. Introducing one new thing is a good learning experience; introducing several is a terrible experience as you'll never untangle the nest of interacting bugs.

And it's heavily over-engineered. This code is to collect images and display and/or save them, right? Why does it need six separate loops (four dynamically launched) to do this simple sequential thing. For example, your "Listen to Image FIFO Stream" loop loads it's image data into a set of three queues and sends it to your "Display and Save Image Data" loop; you could easily do that in one VI and save the (buggy, BTW) implementation of the queues. You could probably do this program with a single QSM, or at most a QSM with one extra loop controlled by a single notifier (as I suggested in your other thread).

The best course of action I can suggest is:

1) Get a basic prototype working that does the basic image collection and display functions you want. ONE VI, ONE loop, no QSM, no architecture, no queues, no control references. Simple data flow only; it's just a prototype.

2) Use that experience to get a set of basic subVI's such as "Get image", "Save Image", "Trigger", etc. (no loops internally, these just do one action each). At this stage, think about clustering related information (type-def clusters or objects).

3) NOW start fresh on a architecture, using your new "driver" of subVI's. It would be best to use someone else's already debugged, and well thought-out, template (such as the JKI-QSM, which is what I use). I suspect you might only need one loop (with the Trigger->Get->Save steps in the timeout case of the JKI-QSM) but if not, use a separate loop on the same diagram controlled by ONE incoming communication method (not multiple polled control references and global variables).

If you want to continue with your current monster, here's some issues I can see:

1) In the QSM design, every state calls "idle", including "idle", which causes the loop to execute the "idle" state (which does nothing and has no delay) at maximum speed, monopolizing the CPU (what's the "idle" state even for?).

2) Your three queues to send one image design is buggy since the queues can get out of step with each other when errors occur, Also, your queues will fill up and crash the program if the receiving "Display and Save Image Data" VI isn't running. And "Display and Save Image Data" will display and save default data if it doesn't receive an image every 5ms (the timeout you've added to each queue).

3) Your "Stop Stream Refnum" design neither starts, nor stops, your "Listen to Image FIFO Stream" VI. It doesn't actually do anything.

As I said, simultaneous bugs on multiple levels is very difficult to work through. Personally, I only use dynamic VIs via a template (worked out and debugged on simple test cases), and use someone else's template for a QSM (JKI). Combined with an initial prototype to learn the new functionality (often I just play with the example VI's provided), this makes debugging manageable.

-- James