What is your LabVIEW application development process?

[Tomi Maila]

Hi folks,

I've been rather inactive lately here on the forums. The reason for this is I'm in a transition phase where I'm leaving my position at the University of Helsinki and moving forward to founding my own company.

Related to setting up new business, I'd like to hear what is your preferred LabVIEW development process like? What are the best practices in LabVIEW development process and what are the worst pitfalls. Are you using an agile process, some other kind of iterative process or waterfall model? I'm interested in all the phases of the LabVIEW software development process starting from user requirements, trough software requirements management, the actual coding methods, unit testing, system level testing, hardware testing, integration testing, build process, release process and all the way to release deployment, application update processes and customer feedback collection.

Please do not concentrate only on good things. Feel free to raise up also problems in your development process that you think may limit your code quality, consume your resources and slow down your development process. Also tell how you deal with communications trough out your process?

I feel this may end up being a long lasting discussion, at least when all of us start to see the pitfalls of our own development processes.

The worst mistake we have made so far is related to requirements. We didn't identify and clarify well enough the user requirements for a complicated software project. As a result the developers in the project were not working for a clear goal. The ambiguity in the user requirement lead to unclarity also in the application requirements. The unclarity of the goals not only steered the project to wrong direction but also made developers unmotivated as the goal was not clear enough. Needles to say, this project failed in many ways.

Tomi

[ASTDan]

I have my own business, and I couldn't agree with you more regarding requirements. Having good requirements/specifications=good software.

The LabVIEW style Guide, and The LabVIEW training classes are great places to start regarding the format of you specification documents.

My current philosophy on getting good requirements is trying to involve the customer as much as possible. I recently purchased Microsoft Live meeting and I can prototype user interfaces to my customers in real time. This helps explain ideas in a way the customer can see and touch instead of a vague intangible idea. This has helped me immensely in writing better code, and I think my customers like being more involved in the process as well.

I also just read a book that said your time on a software (or any engineering project) should be 1/3 specification, 1/3 programming, and 1/3 testing. What jumped out at me initially was how much time is spent on specification. What I have come to learn is that is time very well spent.

I also think recognizing and embracing that requirements gathering, software specification development, and engineering in general is an iterative process. Try as I might sh*& happens. Having a method in your process to adapt to changing requirements (within reason) is I think very important and just good sense.

My advice on building a business is recognizing it is an iterative process also. It is tempting to try to get the perfect business model on the first shot, but that I think is very difficult to do. The best advise I got in building a business was "Hard woods grow slow" i.e. something that sticks around for a long time and can take the slings and arrows of the market takes a long time to build.

Dan

[crelf]

QUOTE (Tomi Maila @ Dec 5 2008, 12:32 PM)

I'd like to hear what is your preferred LabVIEW development process like?

Formally, we use an internally-defined variation of the the VEE Model, but in practise we usually informally extend it to the Dual Vee model. This process takes a little getting used to (you need to balance the benfits from how deep you want to define things). ...and as for requirements - they are key. Learn and live by the IEEE 1288 standards for requirment verbiage - doing so has several benefits: it protects you from scope changes, it makes sure that the customer knows what they're going to get, and it's a standard that is common in military, aerospace, medical, etc industries, and if you're used to using it then you'll have a competitve advantage. Take a look at this.

[ASTDan]

QUOTE (crelf @ Dec 5 2008, 03:30 PM)

Learn and live by the IEEE 1288 standards for requirment verbiage

Just a note buying these specifications from IEEE can be expensive. What I did is go to my local University Library. They had a subscription to every standard you could think of. I was able to get these standards at no cost!

Libraries are a very good thing!

Dan

[LAVA 1.0 Content]

Another useful is the evolving Large Application Development with LabVIEW resource on ni.com. Specifically I recommend the Software Engineering with LabVIEW page.

[Tomi Maila]

Thanks for all for the suggestions so far.

QUOTE (crelf @ Dec 5 2008, 10:30 PM)

Learn and live by the IEEE 1288 standards for requirment verbiage - doing so has several benefits: it protects you from scope changes, it makes sure that the customer knows what they're going to get, and it's a standard that is common in military, aerospace, medical, etc industries, and if you're used to using it then you'll have a competitve advantage.

Crelf, I was browsing the IEEE standards with IEEExplore and didn't find a standard number 1288. Are you confident about the standard number?

Tomi

[Vinny]

QUOTE (Tomi Maila @ Dec 6 2008, 04:11 AM)

Thanks for all for the suggestions so far.

Crelf, I was browsing the IEEE standards with IEEExplore and didn't find a standard number 1288. Are you confident about the standard number?

Tomi

I am not familiar with IEEE standard 1288 either. The Software Requirements Specifications is IEEE standard 830-1998. Not sure it that was the document Crelf was refering to, but it is a good reference as well.

Vinny

[alukindo]

Hi Tomi:

I have owned a consulting firm for the last 4 years. What I have found regarding software development processes, is that one model does not work for all customers and neither will it work for all projects. Your own working style may also be a huge factor on how you end up fashioning your business process. . . . Now, I know that these comments are not all that helpful, but it all goes to show that the software development business does not have silver bullets on what will work 90% of the time. To shed some more light, I have found that some customers do not exactly know what they want, and when you help try to define what it could be, they may not necessarily be willing to pay to get it done. You end up spending most of your time managing scope and expectations.

Some larger companies who outsource software development on a regular basis will be more understanding of a process model and the software development cost. Smaller companies who have been through the software development experience either in-house or via another consultant 'may' be easier to work with. With time, I think that you will develop a preference of the type of 'applications' that you wish to focus on, and the type of 'customer' that you would like to spend your time with. This combination will impose the key factors that define your development process. As you mature as a business, you will also find that developing and licensing your own software 'products' is more sustainable than chasing the one-off projects.

A company called 37Signals that does web based software development, including a project collaboration site called 'Basecamp', have a lot of what I have found to be very practical advise on the software development challenges. They also have a book that you can read on-line for free. Although, their advise is mostly for web-based application development, some of their tips do cross-over.

I hope that this helps.

[crelf]

QUOTE (Tomi Maila @ Dec 6 2008, 04:11 AM)

...didn't find a standard number 1288. Are you confident about the standard number?

Not at all Sorry all, the number is http://ieeexplore.ieee.org/Xplore/login.jsp?url=/stamp/stamp.jsp?arnumber=741940&isnumber=16016' rel='nofollow' target="_blank">1233-1998.

[Tomi Maila]

QUOTE (crelf @ Dec 7 2008, 09:33 PM)

Not at all Sorry all, the number is http://ieeexplore.ieee.org/Xplore/login.jsp?url=/stamp/stamp.jsp?arnumber=741940&isnumber=16016' rel='nofollow' target="_blank">1233-1998.

This one looks more promising, thanks

Anyone else ready to share their experience on their preferred software development process?

[mike_spacex]

Here's a little nugget and also a complaint (one of very few) that I have with LabVIEW development. Before I ever even knew about LabVIEW I was part of a group of developers that belived in (and strictly enforced) test-driven development, which is related to the test-first programming concepts of Extreme Programming. We were encouraged to not write any code unless we had developed all conceivable test cases for it first. The tests might not even compile at first because not all of the classes and methods they require may exist. Nevertheless, it functions as a kind of executable specification. You then get it to compile with minimal production code, so that you can run it and watch it fail. (Sometimes you expect it to fail, and it passes, which is useful information.) You then produce exactly as much code as will enable that test to pass.

Having this practice engrained in me made it difficult to start working in LabVIEW, where you at least need to develop a con pane (and therefore a crude front panel) before you could even write test code that calls your subVI (unless making dynamic calls using VI Server). When I apply test-first programming (as best I can in LabVIEW) I typically write more robust code and avoid the 'fix 1 bug -- create 5 more' problem. It also helps me nail-down the requirements early in the lifecycle. All the tests get added to a 'regression test suite' which we run weekly to verify that nothing has broken. This approach also leaves less open for interpretation than do written requirements documents when handing off tasks to other developers. That's my 2 cents.

[jgcode]

QUOTE (mike_nrao @ Dec 9 2008, 06:51 AM)

All the tests get added to a 'regression test suite' which we run weekly to verify that nothing has broken.

Hi Mike

Are you able to elaborate on your test suite - did you get this to run automated?

I see NI is beta testing unit tests for VIs, looks really good.

[ElijahKerry]

I'm personally very interested to see where this discussion goes. I know that requirements management is a difficult, and all-too-often overlooked aspect of software development. It seems that in the real-world it's very much an iterative process between working with customers, developing a prototype or proof of concept, and documenting findings towards developing specifications.

Check out the new group on ni.com dedicated to large LabVIEW application development. We're just getting it off the ground, but we're hoping to leverage the expertise of those of you working on large applications and struggling with topics like requirements management and code validation, to share your expertise and your knowledge.

Here's a question for everyone on this thread: imagine that you were in charge of coordinating 5+ developers on a single LabVIEW application. How would you do it and what are some of the biggest challenges you think you would face in this task? How, if at all, different would these be from the challenges you would encounter with any other development environment? Have any of you coordinated or been a part of group development on a single project?

I'm glad to see the launch of the Software Engineering wiki on LAVA. You may find some value in reviewing the content c. relf pointed out at ni.com/labview/power

QUOTE (jgcode @ Dec 8 2008, 05:16 PM)

Hi Mike

Are you able to elaborate on your test suite - did you get this to run automated?

I see NI is beta testing unit tests for VIs, looks really good.

It's true - we're preparing to release a new tool aimed at helping developers perform automated unit testing and even requirements based validation. We plan to make these products available in Q1 of 2009, but I would encourage you all to join the http://ni.com/beta' rel='nofollow' target="_blank">beta program for these tools - be sure to select 'LabVIEW Software Engineering Tools.'

The idea is simple, but the use cases allow for advanced testing. You define test vectors consisting of pairs of inputs and expected outputs and effectively run the VI and compare the results with what you had anticipated. This goes beyond troubleshooting because it's not just about demonstrating to ourselves that it works, but it's about proving to someone else that it works exactly as defined, which again ties in very closely with how you define your requirements and how granular they are in so far as describing expected behavior. This tool is also likely to prove valuable for those of you interested in regression testing.

Sign up for the Beta and download the video tutorials along with the installers. You'll see another tool for dynamic code analysis is also available as a part of this program. This tool will help identify the existence and source of issues such as memory leaks or where un-handled errors are coming from.

Hope this sounds like something that will prove useful for those of you who mentioned test driven development.

[LAVA 1.0 Content]

QUOTE (ElijahRock @ Dec 10 2008, 02:35 AM)

...

Here's a question for everyone on this thread: imagine that you were in charge of coordinating 5+ developers on a single LabVIEW application. How would you do it and what are some of the biggest challenges you think you would face in this task? How, if at all, different would these be from the challenges you would encounter with any other development environment? Have any of you coordinated or been a part of group development on a single project?

...

I have done this a number of times. For the most part these adventures have progressed rather smothly when a clean set of specifications and and design are in place. Source Control of some type is very helpful in tracking down and fixing issues. But the big issues that come to mind are;

1) Keeping the kids in their own sand-box

These are minimized when we are working with a modular design with clear-cut lines where a module's functional requirements begin and end. I will generally pass resoncibilty for a single moduale at a time to each developer and let them be as creative as they want provided they meet all of the specs. While a developer is working on a module they are expected to keep a non-broken version of their module on the server so that others are not impacted by "someone throwing sand at them from another sand-box". Cross checks are performed regularly by "testing the app" before sharing updates. But this brings up the next issue...

2) GUI testing

Unit testing of sub-VI is of only limited use when verifying the app behaves as specified. With robust applications the number of scenarios that need to be tested grow at an astonomical rate. The best we can do is try to test all of the edge cases and try to guess at where the peaks and valleys are. So the adventures involved in making sure the app meets specs really depends on the type development process you are using. The Waterfall approach is an ideal case where all of the requirements are documented before wire 1 is dropped. In this environment the specs drive the test set. If the app is large the time involved in verifying that every req is fullfilled after every change can place a large demand on someone's time. Since the the same amount of time is required to run through all of the tests, the edge cases are checked for minor updates and only when a major update is called for is the full test suite performed and documented. Event then issues can arise that were not anticipated at design time so even if I have "checked off every box" I have to expect bug reports from the end user. Using the Unified process is nice since (insert tongue in cheek) we only have deliver what we are sure we can complete in a limited amount of time. This means I don't have to spend a lot of time up front documenting req's BUT the test suite is very fuzzy.

So...

What would really be helpful is to be able to teach LV how to perform a GUI test. That means punching buttons cranking knobs and evealuating if everything looks right. Rather than leave you with an extreme request that seems impossible to implement let me throw out an idea. What if...

The Vision stuff were used to capture screen states and and compare these with the Gold Standard" that was defined when LV was taught how to test the app. It would have to be able to mask-out region where the image is dynamic (does the current time indicator update regularly?). To analyze thing like charts updating there would have to be a known simulated signal and a way of expressing the expected appearence of the chart as a function of the simulated signals.

Throw into that mix that the CPU durring the app testing should be continually monitored to report spikes in CPU demand as result of introducing a change.

Now back to the topic of this thread.

I use whatever process meets the customers needs and budget. What I strongly urge you to do is to take the time to educate your customer and the challenges you are takling in their behalf. Once a customer is "trained" they are much more receptive to your estimates and approaches. So, try to engage every new customer as if they will be your "boss for life". All follow-up projects become much easier.

Done rambling,

Ben

[Tomi Maila]

QUOTE (neB @ Dec 11 2008, 04:14 PM)

What I strongly urge you to do is to take the time to educate your customer and the challenges you are takling in their behalf. Once a customer is "trained" they are much more receptive to your estimates and approaches.

Can you be more specific what kind of "training" does your customers need? What actually you try to tell them before the first project?

Tomi

[LAVA 1.0 Content]

QUOTE (Tomi Maila @ Dec 11 2008, 09:22 AM)

Can you be more specific what kind of "training" does your customers need? What actually you try to tell them before the first project?

Tomi

In brief:

I let them know that I am not only the developer of there code but also the guardian of their budget. Example related to testing:

I can spend all of their budget repeatedly testing everything every time even the smallest things is changed before delivering. After i delever they have to verify the functionality before I get paid. So the the same tests are run twice with the customer having to pay to enusre they will not find issues. I let them know that I can save them money if I minimize the redundent testing and set their expectations that there WILL be bugs. Similarly, if I have a spec and a test for that spec, but the customer doesn't like the behaviour, then "to hell with the test" its getting fixed!

Be clear and upfront! If they ask for a change or a fix let them know what they are asking for in a manner that they can understand. Offer more expedient approaches for the first pass and keep their version around if they decide they really need it work "the hard way".

Do whatever it takes to make your customer "look good" to their boss.

Ben

[ASTDan]

QUOTE (Tomi Maila @ Dec 11 2008, 09:22 AM)

Can you be more specific what kind of "training" does your customers need? What actually you try to tell them before the first project?

Tomi

Educating customers how YOU do a software project. I don't know about everybody else but for most of my customers they have limited experience specicifing software requiremtents. I think part of my job is to help them help me get a good requirements. That means asking good questions and writing requirements. It also means comuncating with your customer along the way.

Managing expectations I feel is very important. Usually I am developing CUSTOM software. There will be bugs and problems. It is smart to be clear with your customers about this up front. Remember most people's interface with technology is with mature products and their expectations are very high. Managing those expectaions is very important.

Dan

[crelf]

QUOTE (neB @ Dec 11 2008, 09:40 AM)

I can spend all of their budget repeatedly testing everything every time even the smallest things is changed before delivering.

Indeed. That's why is important the you really make sure that the customer knows how important the requirements are - they are signing thier names to what they're going to get. If they want to change those requirements during the project, then that's perfectly fine, but there will be consequences. Changes don't always impact schedules negatively - I've issued several zero dollar (and even some negative dollar) ECOs in the past.

QUOTE (neB @ Dec 11 2008, 09:40 AM)

After i delever they have to verify the functionality before I get paid.

Our processes differ on that point - we* create an ATP (acceptance test plan) that the customer agrees on, and then we do (at least) two test rounds: a FAT and a SAT (and dry runs of each, of course). The customer does not do any formal testing - they are present for the SAT, and the SAT is based on the ATP which is directly correlated with the requirements through a traceability matrix. There is opportunity for the customer to make suggestions on things he/she doesn't like, but that's a requirements change right there (note: I'm only talking about fixed priced projects here).

QUOTE (neB @ Dec 11 2008, 09:40 AM)

(I) set their expectations that there WILL be bugs.

Never. If the SAT results meet the requirements, then there are no bugs. There may be issues that the customer wants addressed (and can do so through the ECO process), but there are never bugs. I know it sounds like I'm arguing semantics here, but I can assure you that I'm not. These are crucial distinctions that can make or break a project (and therefore a company) due to process immaturity.

QUOTE (neB @ Dec 11 2008, 09:40 AM)

Similarly, if I have a spec and a test for that spec, but the customer doesn't like the behaviour, then "to hell with the test" its getting fixed!

So how do you know when you're finished? When the system meets the requirements, or when the customer is happy? If you've got a process-mature customer, then those are the same thing.

QUOTE (neB @ Dec 11 2008, 09:40 AM)

Do whatever it takes to make your customer "look good" to their boss.

Wow.

----

*Disclaimer: our project process can't be summarized in one post, so this is a generalized version of just part of it.

[LAVA 1.0 Content]

Good comments Chris!

It looks like you delver mostly fixed-price projects while I delever "by-the-hour". Most of my customers are LV developers themselves but do not have the luxury of being able to concentrate on LV*.

QUOTE (crelf @ Dec 11 2008, 11:01 AM)

Indeed. That's why is important the you really make sure that the customer knows how important the requirements are - ...

Agreed, for "fixed price" projects.

QUOTE (crelf @ Dec 11 2008, 11:01 AM)

...

Never. If the SAT results meet the requirements, then there are no bugs. There may be issues that the customer wants addressed (and can do so through the ECO process), but there are never bugs. I know it sounds like I'm arguing semantics here, but I can assure you that I'm not. ...

Again, fixed price projects are different than "by-the-hour".

QUOTE (crelf @ Dec 11 2008, 11:01 AM)

...

So how do you know when you're finished?...

For hourly projects the project is done when the customer stops asking for changes (money left-over) or when the money runs out.

The key to managing the by-the-hour projects is

1) delivering the "core-functionality" very early in the development process so the customer has a solution to their need in the event complications arise with features and enhancements.

2) Addressing the most demanding issues first.

3) Communicating with the customer re: "how much gas is left in the tank" and using the most expedient "route to the destination".

4) Being honest. If there isn't enough gas to make the trip, let them know as eary as possible so they can decide if they really need to haul along "the Kitchen sink" or if they can get but with some paper towels and some soapy water.

Still just trying to help,

Ben

* A good example of the types of projects I get is the one I worked yesterday. The project spec read like "Up to two weeks of on-site support as required." How do you write a test plan for that, "Have eighty hours been billed?, If true then done, else keep working."

[crelf]

QUOTE (neB @ Dec 11 2008, 11:30 AM)

It looks like you delver mostly fixed-price projects while I delever "by-the-hour".

Yep - that's why I said:

QUOTE (crelf @ Dec 11 2008, 11:01 AM)

note: I'm only talking about fixed priced projects here

I don't mean to say that we only do fixed-price projects - that's certainly not the case. I assumed that the original question was referring to a fixed-price project model and you assumed a time-and-materials project model. Maybe it'll be a mixture of the two? I figure that you're more into LabVIEW consulting, whereas we prefer to work in the complete turn-key solution domain. Tomi: let us know which model you're interested in so we can target our responses more accurately.

QUOTE (neB @ Dec 11 2008, 11:30 AM)

A good example of the types of projects I get is the one I worked yesterday. The project spec read like "Up to two weeks of on-site support as required." How do you write a test plan for that, "Have eighty hours been billed?, If true then done, else keep working."

I'd call that a term of the contract, not a system requirement.

[mike_spacex]

QUOTE (jgcode @ Dec 8 2008, 05:16 PM)

Hi Mike

Are you able to elaborate on your test suite - did you get this to run automated?

The test suite we developed doesn't do any fancy linking of tests to requirements, it just dynamically runs a batch of test VIs (listed in an ini file) that all conform to a common type specifier. Each test is designed to "exercise" some aspect of our software architecture then output a boolean pass/fail, string message, and error out. Results are stored in a txt file and also automatically emailed to a list of recipients. It's easy to setup Windows Task Scheduler to run a .bat file that opens and runs the regression test VI from the development environment. Weekly, daily, whenever... it can even be configured to run using admin privileges, providing the username/password, if necessary.

zipped attachment contains:

regression test VI

ini file that lists the tests to run

.bat file used to schedule autorun

'bad test.vi' -- use this as a template

CPU Usage VI (used in many of our tests)

http://lavag.org/old_files/post-4721-1229361212.zip'>Download File:post-4721-1229361212.zip

[Tomi Maila]

Thank you all for excellent comments

QUOTE (crelf @ Dec 11 2008, 08:21 PM)

Tomi: let us know which model you're interested in so we can target our responses more accurately.

My company will be unlikely to work on "turn-key" projects directly. We provide consulting services for our customers. However, customers themselves can work on fixed price "turn-key" projects and our job will be to help them deliver the project successfully. I'm more likely to work as part of the customers team rather than develop software of my own. Therefore the customers process is the one that matters, not the process of my company. I want to help customers to choose the correct process for their needs. Therefore the question I asked was rather general, I didn't want to specify the process requirements too strictly as they tend to change from customer to customer. Anyway, I guess most of the projects of my customers will be of type fixed expenses, fixed time and somewhat flexible requirements. The important factor will be development speed and quality and ability to flexibly adapt to changing needs. I also tend to prefer agile and other iterative processes, where all the requirements are not known upfront.

[jdunham]

QUOTE (Tomi Maila @ Dec 11 2008, 06:22 AM)

Can you be more specific what kind of "training" does your customers need? What actually you try to tell them before the first project?

QUOTE (Tomi Maila @ Dec 15 2008, 12:25 PM)

The important factor will be development speed and quality and ability to flexibly adapt to changing needs. I also tend to prefer agile and other iterative processes, where all the requirements are not known upfront.

I think you partly answered your own question. Sometimes you will have to educate your customer that your services are different than those of a plumber, accountant, mechanic, or lawyer. Unlike all of those, you generally doing something which has never been done before and is not 100% predictable. The customer, or the people they answer to, can get confused about that.

[LAVA 1.0 Content]

QUOTE (jdunham @ Dec 15 2008, 04:20 PM)

...you generally doing something which has never been done before ...

"and there is the rub" (Hamlet, W. Shakespear sp?)

Ben