Open source alternatives to TestStand?

[pawhan11]

Has anyone here successfully migrated from TestStand to some open source automation frameworks?

In our hardware test we have a lot of things happening in parallel and so far I haven't found reliable alternative for TS. 

[Bryan]

I'm very interested in the responses to this.  We're currently looking into migration of existing TestStand systems either back to pure LabVIEW or to a home-grown or other architecture.  TestStand has just become too much of a pain in the butt for us that we're planning on purging it from our test systems.  

I miss the days of Test Executive (TestStand's predecessor).  It was simple and easily customizable for LabVIEW programmers. Even "iVVivi" (created by CalBay - Now Averna??) was better than TestStand for our needs (lacking in some areas), but iVVivi wasn't open source and it seems to have disappeared from existance.

It would have been nice if NI would have made the old Test Executive code open source, but I'm sure it would have hurt the sales of TestStand, (although I'm sure TestStand is doing a pretty good job of that by itself).

Sorry, I didn't mean to hijack - I'll stop right now as I could easily go on a tirade about my experiences with TestStand.

[Rolf Kalbermatter]

Back in those days I had my own Test Executive Framework modeled around how the NI Test Executive was designed. It worked pretty well but was quite a burden to maintain. That was in LabVIEW 5.1 and 6.x days and VI server was just about invented and very limited, so quite painful to implement a system like a Test Executive that should be easily scriptable. We later moved to a system based on Lua for LabVIEW for some of these applications, where Lua was the scripting environment that the test engineer would use to customize the entire product tests. It worked fairly well but had its own challenges as many things were either integrated in the LabVIEW application and only modifiable by us but not the client, or we had to do the interfacing in Lua too and that would usually make the code so complex that the client couldn't change it either.

It's a very tricky thing to get right. In fact I think there is not a single solution that fits all. TestStand went into a certain direction that works fairly well for semiconductor testing and characterization but not as ideal for some other test setups. And it is a system of its own that is fairly complex and needs to be understood well in addition to whatever extra environment you use to implement the actual test connectors (LabVIEW, LabWindows/CVI, Visual Studio VB,C/C++, or Python). Also it is quite database driven so you need to have some ideas about that too or you end up in a mess with it quickly.

I never quite felt at home with TestStand, despite having it used for several customer projects, mainly in the semiconductor industry. On the other hand, writing your own Test Executive is A LOT of work, both for developing it and even much more so for maintaining it. I would not really want to do that anymore unless there would be a very unique opportunity for a specific project, but chances for that are pretty much non-existent.

And inside Averna we have our own product Averna Launch which is build around NI TestStand. So chances to get to develop something else besides that are pretty much moot. 😀  Averna Launch however takes the database control to an even higher level, so you really have to think about that in order to get the maximum benefit of the product.

Edited April 23, 2021 by Rolf Kalbermatter

[Antoine Chalons]

4 hours ago, pawhan11 said:

Has anyone here successfully migrated from TestStand to some open source automation frameworks?

In our hardware test we have a lot of things happening in parallel and so far I haven't found reliable alternative for TS. 

What is driving you away from TestStand, just the cost or some technical reasons?

I mean, you're saying you want to migrate from it, if it's just to cut costs, finding an alternative.. well even if you find cheap/free one, the cost for doing your migration will be proportinal to the size of your existing projects, so think carefully.

[pawhan11]

Mostly cutting costs and company wants to go into open source solutions. Existing systems will stay as they are now until decommissioned, hardware will be reused where possible , question is about how we approach new projects. NI is too expensive compared to other vendors on the market and we are not having complex systems that require high end hardware they offer. For example we have high end pxi scope cards to measure ac frequency and phase rotation from the grid   And also other our observations of NI software portfolio in general:

- LabVIEW is in decline, from what i hear they dropped support for universities, in EU they closed offices in many countries. We are trying to develop new test systems in .net and see how it goes. 

- No significant new features and roadmaps, unknown future of the language. They killed NXG and announced it on forum post!...  Hope this new NI Conect will change my mind.

 

[ShaunR]

55 minutes ago, pawhan11 said:

We are trying to develop new test systems in .net and see how it goes.

Python has taken over the areas where LabVIEW traditionally dominated in test and measurement  There are a couple of reasons for this not least that it's multi-platform and opens up more cost effective hardware options such as Raspberry PI's. 

Edited April 27, 2021 by ShaunR

[crossrulz]

3 hours ago, pawhan11 said:

- No significant new features and roadmaps, unknown future of the language. They killed NXG and announced it on forum post!...  Hope this new NI Conect will change my mind.

What do you mean "no significant new features"?  2017: Malleable VIs, 2019: Maps and Sets, 2020: Interfaces

I'll give you the rest though.  I have about as much hope in NI Connect as I did during NI's marketing event last year.  And NI's communication lately has been lacking at best.

[hooovahh]

I like TestStand but it is the one size fits all type of software.  NI can't know what you'll prioritize in features in a sequencer and so it either has too few features and only can satisfy the simplest of projects, or it has everything thrown at it, in which case there are parts almost no one will use, and it might have a steep learning curve.

That's why one of the justifications I agree with when it comes to a home grown sequencer, is that you know the types of features you need, and the types you don't.  Now obviously requirements change, and you need to change with them, but high level things you know you won't use, can help trim the feature set in replacing TestStand.

For me the custom sequencer grew out of the need to have it run completely on RT.  The Windows application is just there to for starting the test, status, final report generation, and manual control.

I've used PTP Sequencer in the past, and it was pretty basic and not really a TestStand replacement.  But it was helpful in getting some ideas on where I wanted our sequencer to go.  It wasn't open source and I still don't think it is.  As for cost savings.  All the analysis I've seen has favored TestStand when it comes to cost of developing your own.

[Rolf Kalbermatter]

31 minutes ago, hooovahh said:

As for cost savings.  All the analysis I've seen has favored TestStand when it comes to cost of developing your own.

That's definitely true, unless your time costs nothing 😀, which it very rarely does. Maybe someone working at a university could do that, but they hardly will ever need a real Test Executive. Also if you do it as a hobby, the hours don't really count either, but for that a real Test Executive is WAYYY too big to ever hope to tackle it. Leaves the professional use cases and unless you are planning to sell your own Test Executive it almost always will be more expensive to write your own. Actually the initial writing is not even the biggest cost factor although certainly not something to ignore, but the long term maintenance is the more costly and less sexy work you have to do.

If you have unique requirements such as that it runs on the real-time, developing something yourself may be unavoidable, but otherwise it is going to be simply a "not invented here syndrome".

[Lipko]

We also went for home-grown test controllers and sequence editors as the task for us was too simple and specific for the Test Stand mammoth, and all members of our test team has to create and run their own sequences (no dedicated sequence coder person).
Coding a sequence controller for our certain controllers (NI cDAQ +0-10V AOs and DOs, native Labview support) and for our preferred measuring aplifiers (HBM QuantumX, pretty good Labview support) was only a matter of days with Labview. Coding a simple sequence editor was also a matter of days. Sure, it was a bit tedious to use at the beginning but incrementally the thing became a very effective and user friendly environment. I agree that the "not invented here" syndrome is a bad thing but it has a great advantage: the developer fully understands the requirements down to the tiniest details, as he is also an active user of the environment.

Now it also controls climate chambers, tests can be scheduled, sequences has loops/branches/popups/etc, can run endurance tets reliably for weeks or at lest it's easy to recover (on a Win10 PC under corporate firewall!!!!11), it has automatic evaluation and reporting, editor can export to DiAdem Reports, etc. Actually, it is a base that can be used for making dedicated/specialized test bench control application in days, which still takes weeks currently for the test bench programming team. Sure, only our team uses the system (5 persons at the moment). I can imagine it would horribly shit itself if it was deployed without some programming on a slightly different system... It will never become a commercial product, not even used by other groups within the company (luckily no one was interested enough when it won the internal company innovation contest).

Edited April 28, 2021 by Lipko

[Antoine Chalons]

36 minutes ago, Lipko said:

It will never become a commercial product

The initial question was : is there an open source alternative.
You won't make a commercial product, good.

Now... are you going to publish the source?

[ShaunR]

On 4/27/2021 at 3:15 PM, hooovahh said:

All the analysis I've seen has favored TestStand when it comes to cost of developing your own.

I've written quite a few a test sequencers. They all worked roughly the same way which was a little bit like CSS in being able to include other files. Basically scripting. I have always been lucky though. I got to specify the hardware and using SCPI means you can just script the commands directly from the files with a simple TCPIP or Serial squirter and not bother with drivers. But simple wrappers around drivers can also be used for difficult stuff. I just extended the SCPI syntax to include the hardware route. So you end up with "recipe's" and add some custom stuff for delays/waits, a dialogue invoker and reporting and you were pretty much there. The UI was basically file listings and being able to order, add and delete them. The only other things were start, pause and stop. On one occasion I expanded the concept into a fully featured scripting language. After the first 3 times I could create a Test Sequencer in a day from scratch when moving to  a new company. A couple of hours if i could reuse some stuff. Probably couldn't do that now though. I've forgotten too much and have had to make room for other stuff 

I did something very similar when using Test Stand too. I'd write a TCPIP connector and a LabVIEW VI would run in the background routing commands. All TS sequences just became sending strings to the connector and returning string results. Much simpler. Basically reducing TS to a Script Manager and reporting tool without all the usual complexity. You only have to train production people how add, remove and disable sequences and, if you have a techie production person, the format of the string commands.

Edited April 28, 2021 by ShaunR

[Lipko]

3 hours ago, Antoine Chalons said:

The initial question was : is there an open source alternative.
You won't make a commercial product, good.

Now... are you going to publish the source?

I could, if anyone is really interested.

And I know my comment was OFF, but many comments were similar (story telling with no actual use for the OP), sorry.

[Antoine Chalons]

Well... It's your  -or your employer's- choice really.

I'm sure you know all the potential benefits of open-sourcing your code. Really, appart from improvement suggestions, what's gonna come at you?

In my current activities I don't have a need for a sequencer but I'm sure many people would love to take a look and get inspiration, if not more.

 

Edit : after posting the above I just read again ShaunR's signature : "A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort. (Herm Albright 1876-1944)."

;D

Edited April 28, 2021 by Antoine Chalons

[ShaunR]

2 hours ago, Antoine Chalons said:

Edit : after posting the above I just read again ShaunR's signature : "A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort. (Herm Albright 1876-1944)."

Boss: There's no "i" in team.

ShaunR: There's no "u" either but there is "me".

[Lipko]

I'm sure other's have more general solutions, and I thought open-sourcing is not just tossing the code into the wild. Programming is not my main task, so maintaining an open-source project is out of question.

Once I shared a project for fun (okay, it was only a fun game projecjt) and I got zero feedback.

Maybe I can post in the code-in-development section though. But I can tell you the code for all my projects is very messy. As programming is only a tolerated activity of my job, all my projects are rapidly prototyped stuff.

[Rolf Kalbermatter]

7 hours ago, Lipko said:

Once I shared a project for fun (okay, it was only a fun game projecjt) and I got zero feedback.

The zero feedback is a common problem. Or if you get feedback it is often about "if you could add this and that feature, please" which feels rather "beggish" and often the "please" is even missing. Considering that the whole thing is out in the open as open source so anyone who would care can get his hands dirty and get that feature built in themselves, is quite disheartening. And extra bonus would be if the result would be shared back with the community. 😀

[ShaunR]

1 hour ago, Rolf Kalbermatter said:

The zero feedback is a common problem.

No news is good news. 

I have a couple of "tricks" I use here.

Periodically I will remove the software from the public and wait to see if I get any complaints that it's missing. If there is still no feedback then it goes back in my private toolbox-no immediate interest means people are probably no longer using it and you don't have to support it in 10 years time when someone stumbles across it.

Sometimes I release a "Beta" version. They are not really beta's (I don't believe in them) but fully tested products. These are usually for things that have taken me some time to create but created for my own benefit, need or curiosity. It is a way to gauge whether there is interest outside of my own and how much support would be needed to progress the product outside of my own needs. If no-one downloads it or passes judgement-yup, it goes back in my private toolbox

Edited April 29, 2021 by ShaunR

[seles]

Hi!

I found this discussion while doing research on test automation framework. There is an open source project for test automation that I also found while researching called OpenTAP. It is an open-source test sequencer that is well document and has a pretty decent community. Its extendable and has a .NET core. There is another one called OpenHTF that was created by some Google engineers that is a Python test framework, but it has limited documentation. I don't know if this helps because I have limited software and testing knowledge, but this is what I've found. 

 

[Omer Nachon]

Hello i would like to present TVI - Tester Automation Framework software has been implemented in more than 1000 different tests and it’s applied in the defense industry and leading high-tech companies.

Here are a number of the advantages we offer:

• Save 30% -50% of the funds and time of software development and engineering

• Full LabVIEW, Python, .net support using advanced software architectures

• Flexible reports viewer and generator (Word, pdf, csv, Html)

• Easy to use software and station configuration interface

• Built-in steps – run EXE, display text or picture message…

• Support MS-SQL and MySQL results server

• Set test parameters and test configuration

• Create multiple system profiles

 

Please contact: omer@testview.co.il
 

[Bryan]

Is the TVI - Tester Automation Framework an open source alternative?

I see you also posted the exact same text here.  Is it also a LabVIEW software validation tool?  

[Omer Nachon]

The TVI is a commercial software and we do not supply customers with its source code.
We do make changes and upgrades per customer requests to the TVI - it has to coincide with our vision of the software use cases.
 

For example we recently added support to run Python scripts and wafer lever mapping and testing based on customer requests.
Some changes and upgrades we do all the time - based on our internal R&D or customer inputs.
We make an official release with improvements and fixes 4 times a year.

What kind of changes do you have in mind?
What kind of testers are you developing?

[Rolf Kalbermatter]

On 7/27/2021 at 3:16 PM, Omer Nachon said:

The TVI is a commercial software and we do not supply customers with its source code.
We do make changes and upgrades per customer requests to the TVI - it has to coincide with our vision of the software use cases.
 

For example we recently added support to run Python scripts and wafer lever mapping and testing based on customer requests.
Some changes and upgrades we do all the time - based on our internal R&D or customer inputs.
We make an official release with improvements and fixes 4 times a year.

What kind of changes do you have in mind?
What kind of testers are you developing?

I would suggest moving this into another thread. The title of this thread is about an Open Source test executive, and you clearly state that you have no plans to ever open source it.

[Kurt Friday]

We’re looking at making our Medulla ViPER test executive open source.

I presented the ViPER architecture and test executive to the Sydney LabVIEW User Group. You can view the presentation in the link below

ViPER Test Executive Presentation

I’m hoping to have this released before the end of the year but I’d be keen to have some early adopters.

[Jordan Kuehn]

On 4/29/2021 at 10:45 AM, ShaunR said:

No news is good news. 

I have a couple of "tricks" I use here.

Periodically I will remove the software from the public and wait to see if I get any complaints that it's missing. If there is still no feedback then it goes back in my private toolbox-no immediate interest means people are probably no longer using it and you don't have to support it in 10 years time when someone stumbles across it.

Sometimes I release a "Beta" version. They are not really beta's (I don't believe in them) but fully tested products. These are usually for things that have taken me some time to create but created for my own benefit, need or curiosity. It is a way to gauge whether there is interest outside of my own and how much support would be needed to progress the product outside of my own needs. If no-one downloads it or passes judgement-yup, it goes back in my private toolbox

I was looking for one of your old tools the other day and I couldn't find it. Now I stumble across this several months after you posted the "trick". You are evil! 😆