Is TestStand the right tool?

[mrstevenund]

Hello all,

I've been asked to evaluate TestStand for use in my department. So far, I'm not quite seeing how it will work for us. I was hoping everyone here would give me some insight into something I'm missing.

In my group, we continually test multiple DUTs for extended periods of time. The typical test have up to 15 DUTs, each DUT with a CAN bus multiplexed to a test computer with a single CAN port, and DUT inputs are all parralleled and controlled via relays. It is not uncommon for tests to last 50-100 days. We already have a test shell built in LV8.20 that outputs a CSV file, one file per day, continuously updated with the Test Time, DUT info, PS voltage, Test Step, Test results and limits.

So now for the first question... I'm sure I can modify the generic XML files that TestStand outputs to something more meaningful to us just by learning about XML, so that's not too much of a problem. But, so far with my playing, it looks like it's designed to run only a single DUT at a time. Everytime you start the test, it prompts the user for a single DUT S/N. Am I wrong about this? Is there a way to run multiple DUTs? Am I just being stubborn and not wanting to change my ways?

Thanks in advance. I'm sure I'll have more questions as I continue working with it.

[Mark Yedinak]

Yes, TestStand could work for your solution and it is capable of testing multiple UUTs at one time. It can even share resources between the parallel tests and automatically switch the resource to a different UUT. In addition, it could run other tests that are not dependent on the shared resource at the same time. TestStand has different models for its testing and the default model is designed to test a single UUT at a time and repeat this over multiple UUTs sequentially. The three common models are sequential (the default example stuff), parallel and batch.

In the past I have wondered myself sometimes if it is better to do a LabVIEW only system of use TestStand and LabVIEW. We are currently in the process of building a fairly large test system and though we are finding some quirks with using TestStand we are happy we chose to use it. It implements many things that we don't have to worry about so we can concentrate on writing the tests and not the test executive. If I were you I would give it a serious look. You may want to contact your NI rep so that they could help you see TestStand's full potential.

[TobyD]

Go to Configure --> Station Options --> Model Tab and browse to ParallelModel.seq. Now click Execute --> Test UUTs and you'll see how it works with multiple units.

TestStand is very powerful, but the learning curve is steep (especially for parallel testing) and there is the added cost of software + deployment licenses. If you already have a system that is working, I don't think I'd invest the time and money for TestStand, but if you're looking to upgrade your systems anyway TestStand might be the way to go.

[jcarmody]

QUOTE (mrsteve @ Apr 10 2009, 01:49 PM)

Hello all,

I've been asked to evaluate TestStand [...]

I've heard many of the benefits of using TestStand but I still don't like it (the only thing I appreciate about it is that it'll manage User privileges for you). I'm sure that I'll find an application for it that I can't accomplish easily in LabVIEW (like when I need to handle multiple UUTs and share resources among them - that'll happen soon (yeah, right)), but for now I can't get past the feeling that it's just another text-based mess.

I mean, who programs in text anymore? Really...

[Mark Yedinak]

QUOTE (jcarmody @ Apr 10 2009, 02:09 PM)

I've heard many of the benefits of using TestStand but I still don't like it (the only thing I appreciate about it is that it'll manage User privileges for you). I'm sure that I'll find an application for it that I can't accomplish easily in LabVIEW (like when I need to handle multiple UUTs and share resources among them - that'll happen soon (yeah, right)), but for now I can't get past the feeling that it's just another text-based mess.

I mean, who programs in text anymore? Really...

TestStand is not a text based scripting tool. Where are you getting that from?

[TobyD]

QUOTE (jcarmody @ Apr 10 2009, 12:09 PM)

I can't get past the feeling that it's just another text-based mess.

What do you mean? There is nothing text based about creating a test in TestStand.

Where TestStand has proven useful for us is in reducing the time it takes to create tests for new devices. The TestStand process model is customized to our data collection system, our report generation system, our failure tracking system, etc. None of this changes from one device to the next. We maintain a library of common modules (done mostly in LabVIEW) that can be reused for multiple products. When it comes time to create a test system for a new DMM, we write the DMM driver (again in LabVIEW) and then it's just a matter of dropping the test VIs into a new sequence file in the right order and setting the limits. The new driver handles communication with the UUT, and once we get the data out we can run it into our standard test VIs (DC Voltage, AC voltage, Capacitance, etc.). We don't have to worry about logging data for each step, or generating a calibration certificate for each UUT because TestStand does all of that for us (based on the process model that we developed once).

[PaulG.]

I've run some fairly large, mulitple DUT testing programs without TestStand. VI templates under an executive VI worked OK for me, but maybe my idea of "large" is different than yours. And I don't think I've ever seen TestStand in action to really appreciate it. I don't like the cost. It's way to expensive. It's hard enough to convince the boss to keep up to date with the latest version of LV.

[Mark Yedinak]

QUOTE (PaulG. @ Apr 10 2009, 02:43 PM)

I don't like the cost. It's way to expensive. It's hard enough to convince the boss to keep up to date with the latest version of LV.

Yes, cost is one negative of TestStand. Unlike LabVIEW you can't freely distribute your executable. TestStand requires a deployment license. In terms of the cost for the development system that is fairly easily recouped by the productivity increase. As mentioned earlier TestStand does have a fairly steep learning curve but once you understand how it works it is fairly easy to work with.

[crelf]

QUOTE (jcarmody @ Apr 10 2009, 03:09 PM)

...I can't get past the feeling that it's just another text-based mess.

I htink you're confusing TestStand with Labwindows/CVI?

[TobyD]

QUOTE (PaulG. @ Apr 10 2009, 12:43 PM)

I don't like the cost. It's way to expensive. It's hard enough to convince the boss to keep up to date with the latest version of LV.

I couldn't agree more. And it's true that there is nothing that TestStand can do that you can't do in LabVIEW alone, but there is nothing LabVIEW can do that you can't do in C++ given enough time yet we all choose LabVIEW. It all comes down to how much of the wheel you need to reinvent. TestStand is way too expensive/complicated to use as a simple sequencer when a LabVIEW state machine could work just as well. But when you get into database logging, report generation, conditional looping, etc. TestStand starts looking a little better.

I actually argued the case for doing our new internal test architecture in LabVIEW alone, but I was the new guy and lost so we went with LabVIEW + TestStand. I am seeing the benifits now, but given the amount of time it took us to learn TestStand and customize it to our needs, I still think I could have made a pretty robust executive.

[crelf]

QUOTE (PaulG. @ Apr 10 2009, 03:43 PM)

I don't like the cost. It's way to expensive. It's hard enough to convince the boss to keep up to date with the latest version of LV.

Then your boss needs to learn the difference between cost and value

For every system I've made that does anything even mildy interesting wrt sequencing, the time I've saved by using TestStand rather than trying to do it myself has far out weighed the purchase and distribution prices. Use the right tool for the job - I don't know who said it, but I agree that you should be concentrating on the design of tests and the low level stuff - having to design, write, test, deploy and maintain your own custom sequencer isn't a good move in the medium to long term.

[mrstevenund]

QUOTE (PaulG. @ Apr 10 2009, 02:43 PM)

I don't like the cost. It's way to expensive. It's hard enough to convince the boss to keep up to date with the latest version of LV.

Agreed.

Thanks everyone. It looks like some more investigation is in order. The cost is (as always) a major concern. But the higher-ups are pushing us to use it for a few new projects coming up. It is looking like a call to an NI rep is in order.

[Jim Kring]

QUOTE (crelf @ Apr 10 2009, 12:55 PM)

Then your boss needs to learn the difference between cost and value

For every system I've made that does anything even mildy interesting wrt sequencing, the time I've saved by using TestStand rather than trying to do it myself has far out weighed the purchase and distribution prices. Use the right tool for the job - I don't know who said it, but I agree that you should be concentrating on the design of tests and the low level stuff - having to design, write, test, deploy and maintain your own custom sequencer isn't a good move in the medium to long term.

But, it's way cheaper to just write your one scripting/programming language/engine. And, it's more fun. Plus, there's also job security in it. And, it makes you feel important, when people call you and ask you to add a new feature like looping, variables, etc. Plus, since you created the language, you get to name it. And, maybe someday there will be an O'Reilly book written for it, and maybe you'll get to pick out the animal they use for the cover -- would you choose a Koala or a Kangaroo?

[ShaunR]

Ive used Test Stand quite extensively over the years for testing and the biggest problem seems to be that it tries to be all things to all people. If it just stuck to being a sequencing engine then it would be fine.

If your tests are all read some data from a device, record the result and check for a pass/fail and and spit out a pass/fail report at the end and your doing that over and over again regardless of number of UUT's and no user interaction, then its superb.

However, if you are doing test like set a pressure and record the response over time and check that it reaches 70% in ...say 1 second, then you find it has severe limitations. In fact, you will probably end up writing a special piece of code to do that test because it's easier. If you have to write most of your tests in code, then basically you are just using it as a sequencing engine.

You seem to end up exchanging one programming environment (e.g Labview, Labwindows,c++ or whatever) for another (Test Stand). And Test Stand doesn't do it quite right! You also have to write a large majority of the tests in code anyway and make them fit in. I now use the rule of thumb that if I have to write special test code for 40% of the tests (like the example above) , then the Test Stand overhead isn't worth it and will opt for a proprietry solution.

As people have mentioned. It is a steep learning curve, but that is because you require an in depth knowledge of its features and functionality even to do fairly trivial things.

[PaulG.]

QUOTE (ShaunR @ Apr 11 2009, 07:45 AM)

... However, if you are doing test like set a pressure and record the response over time and check that it reaches 70% in ...say 1 second, then you find it has severe limitations ...

I had an application that had similar parameters and that's where I used a main "test manager" VI with a number of tests written as VI templates. I won't take all the credit for it since I inherited the original idea. But it worked very well. Need a new DUT test? Just write a new template. What little I know of TE I still know enough that it could have been a real headache trying to write this in a strict TE-type test sequencer environment. A test had to start at any time, it could be a different type of test on a half dozen different DUT's, etc. etc.

[jcarmody]

QUOTE (crelf)

I htink you're confusing TestStand with Labwindows/CVI?

QUOTE (Mark Yedinak)

TestStand is not a text based scripting tool. Where are you getting that from?

QUOTE

What do you mean? There is nothing text based about creating a test in TestStand.

No wires! :thumbdown:

I was nearly finished writing a very articulate, eloquent response to the responses to my earlier troll post when I fat-fingered my laptop's 'back' button. Anyway, I'm sorry for writing the way I did but I've been working on a project that was started in TestStand and LabVIEW and I'm not enjoying myself. I did another project in TestStand and felt that it was overkill. I know that my opinion has been influenced by my lack of understanding/expertise with the tool, so I have two more things to say. No more foolishness, I promise.

First, the learning curve to TestStand is very steep. I expect to spend a lot of time learning to use it effectively, but I expect it to make me more effective (once I get over the hump).

Second, I love how easy it is to log test results to a database. I was able to assist a colleague by first installing MySQL and logging there and then again, later, by connecting to the factory SQL Server database with very little effort

[speerose]

QUOTE (ShaunR @ Apr 11 2009, 09:45 PM)

Ive used Test Stand quite extensively over the years for testing and the biggest problem seems to be that it tries to be all things to all people. If it just stuck to being a sequencing engine then it would be fine.

I agree with you ShaunR. TestStand seems to be trying to please everyone by trying to do everything. I started to feel it that when it introduced more flow controls, like IF, WHILE, etc.. in TestStand 3.5. I mean I appreciate them but it gives me a feeling that TestStand is trying to be more than just a sequencer.

As it becomes more powerful, it brings up a questions of how much test code should be in TestStand vs Code Modules (Labview, etc..)

[ShaunR]

If you start with the Labview Test Sequence Example (which I think we would all agree is useless) and change the individual tests to a single dynamic load, add a report,pre-test and post test VI (again dynamic load). With a bit of thought about the Data interface (2D array works best ) you can pretty much do 80% of what Test Stand does with a text file that tells the program which VI's to load.

Discuss....lol.

[crossrulz]

Our company uses TestStand quite extensively. We write quick driver VIs and then sequence them in TestStand. Our main motivation is that if we have any little error, we have to freeze the configuration (no hardware changes, no more code runs, etc.). This is a NASA rule. Anyways, if we have any errors at all, TestStand automatically will handle it. That is one less thing we have to worry about.

But my main peev with TestStand is that you cannot create an executable. All of your LabVIEW code (and LabVIEW itself) has to be there on the test station and any fool can come and change code on us. I plan on trying out a homemade test executive next time I get to create a test station for this exact reason.

[djolivet]

QUOTE (crossrulz @ Apr 13 2009, 08:47 AM)

But my main peev with TestStand is that you cannot create an executable. All of your LabVIEW code (and LabVIEW itself) has to be there on the test station and any fool can come and change code on us.

Could you not build DLLs from the VIs and call those DLLs from TestStand?

[crossrulz]

QUOTE (djolivet @ Apr 13 2009, 09:15 AM)

Could you not build DLLs from the VIs and call those DLLs from TestStand?

That's an interesting idea. I'll have to look into that.

[ShaunR]

Or you could just password protect them or save without diagrams.

The advantage of DLL's as proposed by djolivet is that they don't require the full development environment (do you really install a Full Labview License on every target machine????); just the run-time engine which is installed as part of test stand anyway! It really depends on how often you need to make changes. Generally, Vi's are always converted to DLL's as the last phase of the release before deployment especially if it is for a customer and not for internal use.

[PaulG.]

QUOTE (crossrulz @ Apr 13 2009, 08:47 AM)

... All of your LabVIEW code (and LabVIEW itself) has to be there on the test station and any fool can come and change code on us. ...

In a NASA environment? At face value that sounds kind of scary.

[jzoller]

QUOTE (ShaunR @ Apr 13 2009, 08:11 AM)

Or you could just password protect them or save without diagrams.

The advantage of DLL's as proposed by djolivet is that they don't require the full development environment (do you really install a Full Labview License on every target machine????); just the run-time engine which is installed as part of test stand anyway! It really depends on how often you need to make changes. Generally, Vi's are always converted to DLL's as the last phase of the release before deployment especially if it is for a customer and not for internal use.

The LV development environment is very useful in debugging on the fly if you have live code. NI even supports it with a licensing option.

... yes, I know, everything is always tested before it hits the floor, and no one would ever dream of pig testing so that widget can ship by Friday. Right?

Joe Z. (not my current employer, btw)

[ShaunR]

QUOTE (jzoller @ Apr 13 2009, 09:40 PM)

The LV development environment is very useful in debugging on the fly if you have live code. NI even supports it with a licensing option.

... yes, I know, everything is always tested before it hits the floor, and no one would ever dream of pig testing so that widget can ship by Friday. Right?

Joe Z. (not my current employer, btw)

I'm thinking more about the cost of deployment. Exes and dll's are don't require licensing.I'd find it impossible to justify that the customer pays thousands for a full package which he/she never uses just so that they can use my software.