Open source alternatives to TestStand?

[ShaunR]

4 hours ago, Jordan Kuehn said:

I was looking for one of your old tools the other day and I couldn't find it. Now I stumble across this several months after you posted the "trick". You are evil! 😆

Well. I removed my offerings from Lavag about 4 years ago, so it works  

But the real reason i removed them was that there used to be a size limit and I ran out. I deleted all files so I could post images in the threads going forward. You'll also find a lot of my old posts that no longer have images.

Edited September 15, 2021 by ShaunR

[bbean]

On 4/23/2021 at 6:12 AM, pawhan11 said:

Has anyone here successfully migrated from TestStand to some open source automation frameworks?

In our hardware test we have a lot of things happening in parallel and so far I haven't found reliable alternative for TS. 

I would investigate cosmos.  I haven't migrated a complete project to it yet but I'm implementing data collection, storage and telemetry GUIs to it for a new project working with python and FPGA programmers.  A lot of the python programmers use python and it as a replacement for LabVIEW/TestStand.  I haven't used their Script-Runner (TestStand-ish stuff) but the current version is highly capable for data collection, saving, presentation, limit checking.  They are working on cosmos 5 which will run in a docker container and have web based gui's

[Gribo]

OpenTap: OpenTAP: About

[huijun guo]

11 hours ago, Kurt Friday said:

我们正在考虑将我们的Medulla ViPER 测试执行程序开源。

我向悉尼 LabVIEW 用户组展示了 ViPER 架构和测试执行程序。您可以在下面的链接中查看演示文稿

ViPER 测试执行演示

我希望在今年年底之前发布这个版本，但我很想有一些早期采用者。

This is really good news,

[pawhan11]

@Gribo Have You used OpenTAP in commercial applications? Are You able to compare it to NI Teststand?

[Gribo]

I didn't use it at all, even though most of my lab instruments are from Keysight.

[AutoMeasure]

I find TestStand to be a powerful, flexible platform and a very cleverly designed product, with features really well targeted to manufacturing test. But you have to take the time to master it and digest it in order to make it look and behave like you want, rather than be constrained by it. I tried Test Executive Toolkit in '97 and TestStand in '09 and discarded them right away in frustration. And then I was forced to get "over the hump" and master TestStand 2014 and 2016 while at Sonos, and now I like TestStand a lot.

I don't use TestStand at my current company because I can't justify the cost, and we're not at the mass-production stage yet. I see that the current cost is $5,561 per Development seat, $2,814 per station for the Debug version, and $741 per station for Base Deployment (or $581 per station for qty. 5). Just plain too much, and the Base Deployment is just the engine, no sequence editor.

Dear NI - Please price TestStand at $2,500 per Development seat, and $150 per computer for the Base Deployment. You don't need a distinct Debug version (it's the same as the Development version anyway). Also, please advertise your TestStand product! Tell everyone about the new affordable pricing! NI will benefit from the increased sales volume, and TestStand users will benefit from having a larger community of users to talk to and meet with.   🙂

 

[Jordan Kuehn]

11 minutes ago, AutoMeasure said:

and TestStand users will benefit from having a larger community of users to talk to and meet with.   🙂

 

It took NI decades to release a community version of LV and IMO (and I've heard echoed often) it has really hurt the language and wide adoption. Hopefully this shift in mindset can carry over into other products like TestStand. Or opening FPGA up to non NI hardware as another example.

[Bryan]

TestStand has its place.  Yes, it can be a very powerful tool, but where I work, TestStand was someone's hammer and every test solution was a nail.  We have several instances where TestStand was grossly overkill, where a more lightweight test executive would have been a better fit. 

I could write a short novel on all the hassle we've had with our TestStand based testers and the poor planning, development and implementation of it where I work, but I will refrain. (In fact, I did, but deleted it as it went on a long rant). 

I'll just simply say that I've inherited those @%$#@ machines and they've been nothing but a 4-year headache and I'm slowly replacing each one with a LabVIEW-Only solution.  The ones I've done so far perform much better/faster/stronger, are more robust and the operators gripe much less about them if at all.  In fact a "notoriously gripey" operator said that they "absolutely loved the new tester" after I had completely reworked it to eliminate TestStand.

Edited October 29, 2021 by Bryan

[JB_1592]

On 9/16/2021 at 8:23 AM, pawhan11 said:

@Gribo Have You used OpenTAP in commercial applications? Are You able to compare it to NI Teststand?

I have done a minor project with it. Thoughts so far...

I think what you hope to achieve and what you need from it is a big consideration.

Unlike a typical sequence file, OpenTAP TestPlan objects do not have Setup and Cleanup routines. That makes mandatory cleanup steps that must occur even if test is aborted less intuitive. There are a few ways to achieve the basic idea, but they require a shift in thinking and a decision on how best to achieve the goal. For what I've done so far I've gone with the idea of using steps with names like "With Power On" and ensuring power is turned off in a catch block that should run even if an exception is thrown, but I do worry a bit that if I have multiple instruments/interlocks/whatever that deep nesting of these steps could make editing TestPlans cumbersome.

Outside of that, if your goal of going open source is something available for free then you may be in for some disappointment or have a lot of work left for yourself.

• The engine is free, unless you really want to pay Keysight for support.
• The WPF based Editor a community edition, but it is not free for commercial use.
  You can use the open source text based one (TUI) if it meets your needs, but it isn't quite as user friendly or fully functional. (Although it works basically anywhere there's a terminal, unlike the Windows only WPF based Editor.)
  There is a new "EditorX" available, but I haven't tried it. If I understand correctly, the "community edition" license for this application no longer contains the prohibition on commercial use, but does require that you agree to a bunch of telemetry. This one is built using one web-based UI tech or another and is supposed to be cross platform.
• There are some basic things outside of the engine not included, such as printing a nicely formatted report. Keysight does have a HTML5 ResultListener plugin, which I believe needs to be licensed per workstation. That may not be an issue if you don't need to print paper copies of results. There are free plugins for writing results to CSV files and SQLite or PostgreSQL databases.

If you want or need to pay for any of these things then it catches up to (or exceeds) the basic deployment cost of TestStand pretty fast. Of course, you could roll your own for any of those things, but that can add up to some effort.

Then there is the greater .NET vs LabVIEW portion to consider. LabVIEW is built for test and measurement and has a far greater ecosystem in terms of instrument drivers and such*. .NET certainly has more general purpose libraries and larger community, but if you need data visualization you'll need to decide on a third party library to do something as basic as data plotting. I think some of that may be taken for those used to building UIs in LabVIEW. The devops ecosystem is much stronger for .NET if that matters to you.

Overall, a lot of give and take and I'm not 100% sure whether or not this will be the long term path for larger projects.

*OpenTAP does include base classes for instruments, as well as a generic SCPI instrument implementation that can be derived from, which somewhat facilitates creation of instrument plugins, but I don't think you'll find a lot of vendor provided ones. I think even a lot of the Keysight provided ones require license keys... Come to think of it, that seems like a major weakness and barrier to adoption if they really want to drive adoption of this framework,, so... not really sure how I feel about that business model.

[Jack Parmer]

Flojoy is essentially "Open source TestStand" - please see this demo video and blog post.

You can also download Flojoy to try yourself at https://docs.flojoy.ai/studio/installation/

Flojoy also has an open-source database platform called Flojoy Cloud that seamlessly integrates with its test sequencer and has a modern Python SDK for test data upload and retrieval.  You can try Flojoy Cloud for free at https://cloud.flojoy.ai/ or watch this demo video.

Please let us know if you have any questions, suggestions, ideas, or feedback!