Have you ditched TestStand to roll your own sequencer?

[Norm Kirchner]

I know many people never go the route of using TestStand, choosing to roll their own sequencer and make it work (as I did once upon a time)
However I'm starting to hear from a few folks who adopted TestStand and are going back to try to make their own sequencers (as I hope to never do)
So, I shook the corporate tree some and have a meeting tomorrow for which I'm representing y'all, based on what I was hearing at NIWeek

That being said, if any have gone the route of TS only to turn back, and you'd like your voice shared inside the guts of the mothership, please do share here.
FWIW, If you've got specifics as to why you've never tried, that would be best for another thread. Thanks for understanding

~,~
The Captain was here

Tag anyone you may know have done the same

@Mark Yedinak@Mark Balla

[crossrulz]

My only TestStand roadblock so far (and has not been nearly as big of one as I expected) is the deployment license cost.  Though, put that cost against a VST and it is a small drop in the bucket. (The VST is literally half of the cost of my current project's entire test rack.)

As far as going away from TestStand, I have ran into a couple situations where I was told to use TestStand and I convinced the customer/management that the project really just needed a decent State Machine with a few other loops for logging and instrumentation.

Sorry, no real ammo coming from me.  But I am very curious about other people's reasons.

[infinitenothing]

It's always a project by project decision for me. If I need the flexibility, I'll go TS. Otherwise, it's not worth the deployment licensing hassle.

Another advantage to rolling your own is you get to use LabVIEW's block diagram representation of the sequence. It's a great visual—things in parallel execute in parallel. Data flows on wires. What can I say?

[Tim_S]

I've run in to TestStand twice... The first was a project I got pulled in to at the 11th-and-then-some hour to make some changes. I got a quick intro to implementing LabVIEW code in TestStand and made the changes. I didn't get to do much with it, however the impression I got from people who had been with the project was very negative in how difficult the system was to use to do what was wanted and they intended not to use it again. This was with TestStand version 1 (and I believe LabVIEW 5) from people who (today) would be certified LabVIEW CLD, so take that for what you will.

The second time I ran into it was as part of a major revision and re-write of an existing system including sequencer written in LabVIEW. We (two from my company and a local alliance partner) seriously considered using TestStand, but concluded that the amount of work learning TestStand, porting existing code to work with TestStand, build a system around this that did all the other things that were needed, and make it look like a homogeneous system would be more costly that just writing it in LabVIEW (version 8.6 at the time).

[Chris_Collier]

I think our decision to no longer use TestStand stems from our need to run requirements-driven tests. The sequencer engine we have built takes requirements-level data and translates that to our test system for stimulus application and response measurements. No one at our location was knowledgeable enough to create a similar system so that the only thing we had to change while running a TestStand sequence was to point to the new requirement (no change to the test sequence run through TestStand or the modules TestStand calls).

I have a bit of a deeper understanding of how TestStand works now, but am still not comfortable enough with it to create this type of system.

One of the reasons I wouldn't want to create sequences in TestStand for each unit is that it isn't a very straight-forward system to use. I don't want to have all of the test engineers on my team required to know TestStand intimately, just as I don't want them to have to know LabVIEW intimately. Everyone should know the basics, a few should have a deeper understanding of the system as a whole, and at least one person knows the whole system intimately. If sequences were translated from our requirement documents to TestStand sequences, everyone on my team who creates unit tests (which is everyone on my team) would need to know TestStand. With our current engine, they need only know Excel.

If there were more in-depth classes for how to make higher level architectures work in TestStand, and if TestStand didn't require a menu 3-deep for common functions, I might be more inclined to use it.

[smithd]

I tried to (and succeeded, for a given value of succeeded) use teststand as a semi-headless sequencer with a network front-end, but often think may have been a mistake. The driving factor behind my reconsideration is I've now used it for a few months and the API is a pain to use and its pretty buggy in important areas. We're not going to switch at this point because I did eventually get it to work fairly consistently, but that could change if new requests come in -- every new feature seems to take more and more time to get right.

The buggy part is easy to describe: occasionally my whole labview instance will hang when I start a VI with a ts app manager control on it, occasionally it will fail to shut down, and I see sometimes bizarre behavior between traces, like on first run a sequence will trace into the process model but on subsequent runs it will jump directly to the main sequence (with no setting changes in between).

Specific to what I'm doing, I rely on the API for everything and its a big challenge to wrap my head around edit vs runtime objects, different trace modes and flags, error vs failure behavior, threading concerns (at what points am I allowed to safely access engine data?), etc. Fundamentally what I want to do with teststand is this set of methods: startexecution(), getStatus(). Unfortunately, getStatus in reality balloons out into this crazy nest of callbacks and API calls and trying to reconstruct the state of the system through a bunch of trace messages. Also, specific to my desire to run sequences from the API, there are a ton of modal dialogs which pop up whenever you do something wrong. Most are disable-able (once you figure out all the flags to set), but a few related to closing references are not. Modal dialogs are obviously a challenge to work around if there is nobody standing at the UI ready to press buttons.

Edited May 31, 2018 by smithd

[JCAndersen]

I attended my first TestStand course in 2004 and have been working with TestStand on a regular basis since 2008.

To me it seems that every one who has replied to this thread until now is making the most common miss assumption about TestStand: "TestStand is a test sequencer".
Of cause you are right about that, but it is so much more. With TestStand you get a toolbox for creating your own test management frame work. This means that you as a test developer need to spend some time getting to know you toolbox, just like you had to learn how to program in LabVIEW at some point. An yes, TestStand is a complex toolbox to get your head around. That is the trade off for the flexibility you get.

In the posts here I have not seen one challenge that I could not implement and make work in TestStand, just as easy or easier that in a custom LabVIEW sequencer. And when you are on the track with TestStand, you will realize all benefits like batch executions, reporting and property loading. Of cause you can implement and maintain all of this your self, but they are handed to you with TestStand and NI will maintain the fundamental function for you.

However I do agree with Chris_Collier that it would be nice if NI provided a course on how to make high level architectures with TestStand. 

So I guess my point is this: Do not just buy a TestStand license and start using it out of the box as a simple test sequencer. You need to be clear about your goals for the test system, learn how to use TestStand and then learn how to implement you test system with TestStand.

Edited June 1, 2018 by JCAndersen

[Bryan]

From my experiences, TS seemed to be overkill for a lot of applications with which I have been involved, like "trying to swat a fly with a Buick" (to quote a friend).  Add to that the additional cost of deployment/debug licenses for multiple test stations, the bill goes up quickly.

Learning curve also comes into play as TS has seemed to become its own programming language and is not as easy to learn as LabVIEW for those without the budget or time to take the training classes.

With a "roll your own", after the initial time investment of creating a test sequencer in LabVIEW, you can deploy to as many stations as you want without the cost of additional deployment licenses by just building EXEs.

Where I'm currently working, TestStand is just starting to gain a foothold, but ease of deployment and cost of deployment licensing has been a source of contention. 

Now, my opinions are based solely on my experience with using TS on and off for several years without formal TS training.  Ironically, a couple of my coworkers and myself are going to be taking a TestStand classes within the coming weeks, and I'm interested in whether my personal opinion (and those of my coworkers) will change.

At my previous employer, we still had a system running the precursor to TS: Test Executive. We had kept updating it until we ran into an incompatible version of LabVIEW, then just "froze it in time".  Nobody there was very familiar with TestStand and how much it would take to do a migration as the station tested a large variety of boards.  Because there was only one operating station, the risk of downtime for migration was considered too great, so it was never done.  Test Executive seemed so much simpler and a better match for the needs at the time.

A second station was created to do the same as the one running Test Executive.  Test Executive wasn't compatible with the newer version of Windows, and they still didn't want to go the TS route, so we ended up going with a LabVIEW-based test sequencer from CalBay (now Averna) called "iVVivi".  I became very well versed in iVVivi and it was definitely simpler than TS.  It also had an integrated LabVIEW OOP-based HAL, which was very attractive.  However, in some respects it was TOO simple and required some creative LabVIEW routines to mimic some functionality available with Test Executive.  I don't believe iVVivi is even supported anymore, so they may eventually be forced to go the route of TS.  Or, beg NI for the password to access/update the protected VIs to recompile in later LV versions.

 

EDIT (3/2020): After having had the training and applying what I've learned over the past year, I can say that I understand TestStand much more, including its quirks. I still think that it could be suited for some applications, but still way too overly-complicated for the majority of applications that I deal with on the day-to-day and the additional costs per seat/deployment are still a deterrent.  It still fits the "swatting a fly with a Buick" analogy I used before.

Edited March 4, 2020 by Bryan

[Tim_S]

6 hours ago, JCAndersen said:

This means that you as a test developer need to spend some time getting to know you toolbox, just like you had to learn how to program in LabVIEW at some point.

Which is a big point to why we didn't use TestStand in that there is a cost in learning the system which has to be able to fit in budget and schedule of a project.

[hooovahh]

For me the simplest reason I don't use TestStand for my sequencer, is that it doesn't run on real-time.  I want to tell an RT target to run a sequence and have it run deterministically (mostly) and headless.  No more worrying about Windows restarting on an update, blue screening, how to handle PC upgrades and the down time with it, etc.  I just want an embedded device to run my sequence talking to network devices, serial, DAQ, and other IO.  And have the Windows computer just get the data and do report generation.  When RT is not a hard requirement, or in cases when there is an RT target but the sequence itself can be non-deterministic (steps ran on Windows), TS is a fine option.  Just not one I've used often enough to be comfortable with it on large projects.  When a small project does come I just use our sequencer anyway because eating your own dog food is good for other projects when issues are found.

[Phillip Brooks]

Yes.

4 years ago, I took my current job because the lone engineer supporting the manufacturing of my product suddenly left. I believe there were multiple reasons, but after I got involved I realized that I would have probably done the same if I was in his shoes.

There was 10 test stations running 5 test sequences for three specific products.  The test system was based on an older version of CVI / TestStand with run time deployment that originated at a consulting company. The product under test has some diagnostics, but much is tested by configuring it as a customer would. The product software changed over time (as would be expected) and the tests stop working. Sometimes the changes demanded a change to the CVI code, but the previous engineer had no experience. Solution? Replace the functions that don't work with LabVIEW VIs!

Move forward a couple of years and introduce new test requirements, new web based interfaces and the test system turned into quite a mess of curl, perl, CVI, LabVIEW and TestStand RTE.

It got so bad that the product OS was locked down to a specific version of software and then upgraded at the end to the shipping software. CVI code would crap out when IT would install Windows updates (DLL hell).  Test times per UUT went from an original 30 minutes were now almost two hours. The product ended up being tested twice just so we could install the latest software before shipping. First pass yield was 80% with the primary cause being test execution failure; timeouts exceeded and Selenium/Firefox step failures because of JAVA updates or an operator changing the screen resolution.

This is where I came in. I dubbed the existing system "The Jenga Pile". Touch anything and it all came crashing down. It's easy to say that this is an implementation issue. My manager and I decided that we needed to move to a current version of Windows / LabVIEW / TestStand, get rid of cludges like Selenium and curl and rewrite the tests from scratch based on the latest version of the product software.

I was 6 months and 75% or so into the project when the only early advance of the new product software was given to me. Many of my tests were based on a telnet session to the product. Telnet was now REMOVED from the product! (not disabled, but REMOVED - security issues ). I needed to change to SSH or use the serial console port. LabVIEW and SSH (don't get me started)?!?!   

I was already done with my test coding and intended to spend the last two months tweaking the TestStand environment and creating my first RTE deployment. I used SSH.NET here and there where necessary to recover. After taking 4 weeks to rework the telnet related issues, I found I could not reliably create and install TestStand deployments. SSH.NET would not work. Oops! Did I mention I had never created an RTE deployment before? My previous employer used debug deployment licenses and I never once dealt had to deal with the RTE.  Now I know why.

I told my manager to cancel the plans for upgrading our 10 deployment licenses and spent 4 weeks in overdrive creating my own crude sequencer and report tools. I saved the cost of NI upgrades, met the deadline and made my life easier down the road.
Changes now only require compiling a new EXE and dropping it on the test stations. I have a UI that I can change, deploy in minutes and I haven't had to think about Process Models or creating 100s of MB of deployment packages for simple changes.  It is not actor based or even very modular at this point.

I love the promise of TestStand, but I'm not controlling a super-collider and certainly not doing rocket science. Give me TestStand Lite (google it) as an actor where I can register some LabVIEW UI elements to monitor and control execution.

Give me TestStand Lite, limit the engine to running LabVIEW VIs and include the engine in the Professional edition. The TestStand Lite sequence editor should integrate seamlessly into the LabVIEW project.

Edited June 1, 2018 by Phillip Brooks

[viSci]

I was wondering when someone would mention the long forgotten but hopeful promise of Teststand Lite.  I doubt after all these years NI is going to provide this.  It is likely something that will only happen with community support.

[ShaunR]

On 6/1/2018 at 1:31 PM, Phillip Brooks said:

LabVIEW and SSH (don't get me started)?!?!  

I have no problems with LabVIEW and SSH

[ShaunR]

The main issue with TestStand is it tries to be all things to all people.

It's pitched as a test sequence engine but is too complicated and cumbersome for that. The main UI is far too complicated for production and the "screen" hooks are awkward and difficult to implement. Reports seem like an afterthought and the LabVIEW hooks are prone to crashing. If you thought global variables were bad, well we have have several different varieties with different scopes and figuring out where things are defined or coming from is a very deep rabbit hole.

I greatly simplified my life when using Test Stand by having a single VI TCPIP connector that just emits an API string which you define in the test stand editor and running a service VI that receives the string and invokes the actual tests-basically reducing test stand to a command/response recipe script to order tests, retrieve results and throw up a big PASS/FAIL at the end. At that point it really doesn't matter what generates the API strings - test stand or a custom sequencer.

Edited June 2, 2018 by ShaunR

[Norm Kirchner]

All,

Thanks for chiming in. I actually hadn't heard of the promise of TS Lite and will be sure to bug the powers that be about it.

All in all, most things here resonate with the themes I've heard before and am glad to have ammo in hand to go to the conversation with, as it's gotten delayed; so feel free to keep chiming in.

If I had to sum up the big points

• Cost: LV exe is free cheapest TS deployment gets 500+
• Simplicity: Simple enough needs that can get away without the extra features in TS
• Parallelization: not easy to implement or visualize parallel tasks
• Training: Don't want to learn another tool myself to dev or for other people to use; just not intuitive enough
• API woes: Buggy in important areas; hangs between LV
• Data Access: Getting data around, into and out and between the modules can be challenging
• Deployment: Just doesn't go smooth
• History: Been burned before for one situation, why go back
• TestStand Lite: when please?
• UI Editor: complicated and overloaded to the point of distraction
• UI Tester GUI: Understanding components and creating in LV is an untenable beast

I hope that is a good representation.

~,~ The Captain was here

[Benoit]

For my part, I hate TS for the license cost and for it's advantage that is an inconvenient for me. TS is so capable to integrate anything that it becomes not user friendly. The knowledge ramp is quite hard. Doing things in parallel is not intuitive and if you want a good GUI for your operator that is multiple language... GOOD LUCK. In a manufacture context, time and clarity of instruction counts.

I create my own test framework that integrate database, debug tools, instrument command spy and overpass functionality. Everything has a "secret" backdoor to connect to the tester to see what's going on. (I  VI server) I developed a HAL "Hardware abstraction layer" level that allow the technician to change any instrument in a matter of minutes and this is 100% transparent for any test sequence that runs. Per example I support over 15 different power supply (serial, GPIB and USB). (plug it and play) I also integrated Domain login capability and so much more. Did I said that it support batch mode, single mode and asynchronous mode for unlimited test position as well as debug step by step? 

Finally, developing a test sequence with medium complexity (CAN, SPI, I2C, Ethernet, ARINC,+ instrument power supply switch, DAQ, Oscilloscope, camera (bar-code and vision inspection) and more, about 250 measurements take less than a week including deployment and validation. TS cannot offer that.

 

In other words, if you want all those functionality in TestStand, you need to develop them outside TestStand. Training someone on older code made in TestStand is way too long. So a huge license cost just to execute test step no thanks not for me.

By the way... I'm working for an important electronic manufacturer... High volume!!! No TestStand and happy about it.

[Antoine Chalons]

48 minutes ago, Benoit said:

For my part, I hate TS for the license cost and for it's advantage that is an inconvenient for me. TS is so capable to integrate anything that it becomes not user friendly. The knowledge ramp is quite hard. Doing things in parallel is not intuitive and if you want a good GUI for your operator that is multiple language... GOOD LUCK. In a manufacture context, time and clarity of instruction counts.

I create my own test framework that integrate database, debug tools, instrument command spy and overpass functionality. Everything has a "secret" backdoor to connect to the tester to see what's going on. (I  VI server) I developed a HAL "Hardware abstraction layer" level that allow the technician to change any instrument in a matter of minutes and this is 100% transparent for any test sequence that runs. Per example I support over 15 different power supply (serial, GPIB and USB). (plug it and play) I also integrated Domain login capability and so much more. Did I said that it support batch mode, single mode and asynchronous mode for unlimited test position as well as debug step by step? 

Finally, developing a test sequence with medium complexity (CAN, SPI, I2C, Ethernet, ARINC,+ instrument power supply switch, DAQ, Oscilloscope, camera (bar-code and vision inspection) and more, about 250 measurements take less than a week including deployment and validation. TS cannot offer that.

 

In other words, if you want all those functionality in TestStand, you need to develop them outside TestStand. Training someone on older code made in TestStand is way too long. So a huge license cost just to execute test step no thanks not for me.

By the way... I'm working for an important electronic manufacturer... High volume!!! No TestStand and happy about it.

Maybe you should give it a name and start selling it :-o

[Benoit]

Well... it's my design that I did for a company... Cannot sell it... but my value is high. 

Benoit

[pawhan11]

From my perspective,

I have designed sequencers in LV that work fine. 

Recently I had to switch and use teststand as main sequencer and I can tell this tool is amazing. The ammount of flexibility we can have is almost unlimited. Some apis are not ducomented properly and there is high learning curve in my opinion. This is not a tool that You will plug and play in week. It requires investment that returns over time. I have seen NI training materials and to my opinion they provide no valuable information.

[Phillip Brooks]

On 6/27/2018 at 6:01 AM, pawhan11 said:

Recently I had to switch and use teststand as main sequencer and I can tell this tool is amazing. The amount of flexibility we can have is almost unlimited. Some apis are not documented properly and there is high learning curve in my opinion. This is not a tool that You will plug and play in week. It requires investment that returns over time. I have seen NI training materials and to my opinion they provide no valuable information.

It reminds me of the old quote about solving a programming problem using regular expressions. Punch-line : "Now you've got TWO problems"

[Elena Lombrozo]

Hey all,

I know this is an old topic, but thought I'd pitch in in case it's still relevant to anyone- 

Although this topic is about in-house alternatives to TestStand, but there is a commercial alternative that might interest those of you that quitted TestStand because of its cost or because it is not so-user friendly - Testview's TVI is an affordable test automation framework that places at its center core test cases. The GUI is super intuitive for LabVIEW users, and the company is very attentive to costumers - upgrading and maintaining the software to keep it up to date, and to answer costumer's needs. 

If this sounds like something you could be interested in, you can visit their website: https://testview.co.il/

[Omer Nachon]

20 minutes ago, Elena Lombrozo said:

Hey all,

I know this is an old topic, but thought I'd pitch in in case it's still relevant to anyone- 

Although this topic is about in-house alternatives to TestStand, but there is a commercial alternative that might interest those of you that quitted TestStand because of its cost or because it is not so-user friendly - Testview's TVI is an affordable test automation framework that places at its center core test cases. The GUI is super intuitive for LabVIEW users, and the company is very attentive to costumers - upgrading and maintaining the software to keep it up to date, and to answer costumer's needs. 

If this sounds like something you could be interested in, you can visit their website: https://testview.co.il/

You should also know that this works not only with LabVIEW, but also with .NET and C#. And it really is super easy to learn and to use, the training curve maxes at about 4 days.