Large Test applications without large State Machines

[jcarmody]

I'm intrigued by a few discussions we've had recently and am here to ask what architecture y'all use when writing a program to implement a long test procedure.  Here's the first discussion, and here's the second.

Here's the scenario:

I have two test procedures for a product I need to test: an MTP (Manufacturing Test Procedure) and ATP (Acceptance Test Procedure).  Each procedure contains a dozen or more individual tests (some are common between the two), each test requires configuring and reading a few IO devices and the application must have a reasonably functional user interface that allows automatic testing as well as permit a Technician to debug the UUT.  My approach has been to write a JKI State Machine with states to control the hardware and macros to sequence the tests; this results in a long list of states.  Several respectable LAVA contributors in the aforementioned discussions eschew this design, so my question is: what architecture can perform all my tests without using the kind of State Machine I've described?

[asbo]

TestStand

[jcarmody]

TestStand

So, instead of having a large State Machine I'll have a large list of subsequences?  That's feels better but I can't explain why.

I think I'll use it, but it'll be for the built-in reporting and data storage and not because it's more readable.  What else will I gain that addresses the issues folks have with my old kind of State Machine?

[asbo]

I suppose it would have been great of me to elaborate a bit more, huh?

I've worked on projects that use both and I agree - a large list of subsequences feels a better than a large state machine. TestStand sequences do tend to feel a bit more ... well ... *sequenced.* It's easily to miss a step in a state machine and it can be a pain to track down - so I would definitely count the readability as significant.

TestStand can be pretty powerful if your testing scenario can utilize features like process models. It can be a little harrowing to wrap your head around a complex TestStand set up at first, but it is a really flexible framework. I'm still not accustomed to it yet and have found myself writing VI code for something that could surely be accomplished in the sequence if I had the experience.

[Mark Yedinak]

I suppose it would have been great of me to elaborate a bit more, huh?

I've worked on projects that use both and I agree - a large list of subsequences feels a better than a large state machine. TestStand sequences do tend to feel a bit more ... well ... *sequenced.* It's easily to miss a step in a state machine and it can be a pain to track down - so I would definitely count the readability as significant.

TestStand can be pretty powerful if your testing scenario can utilize features like process models. It can be a little harrowing to wrap your head around a complex TestStand set up at first, but it is a really flexible framework. I'm still not accustomed to it yet and have found myself writing VI code for something that could surely be accomplished in the sequence if I had the experience.

We are in the process of rolling out a very large TestStand application for our internal engineering tests. Currently our test is approximately 2000 sequences and over 200000 test points. What we have found that works well is to use a database to manage the data parameters and expected results. We also dynamically link the sequences that we will run based on the selected UUT. Given the size of our test suite we also implemented a tree view of the sequences. This has been very useful for navigating through the tests and allowing the user to individually select what will run. In our database we also store test dependencies so that we can skip over tests that don't apply to the UUT even if the test was selected to run.

The other thing that we have done is to keep the sequences themselves fairly basic. We created lots of custom step types that implement the complex tasks. All of our custom step types are implemented in LabVIEW. I definitely think you should seriously consider using TestStand.

[crelf]

Firstly, there's a difference between a state machine and a sequence machine. If all you've got is a sequence machine then TestStand is a great solution (not to mention all the stuff that's built into it like sequence control, IVI layers, direct DAQ, database communiation, etc). It's also much easier to validate when you're looking at dicrete components (like your LabVIEW VIs - they can have unit tests so you're sure you're on the right track during development), as well as OTS tracability (you don't need to proove that your sequencing engine works, just the sequence itself).

[jcarmody]

[...] I definitely think you should seriously consider using TestStand.

I purchased a Debug Deployment license for my project.  I will use it, but my TestStand skills today are worse than my LabVIEW skills were two years ago. 

Firstly, there's a difference between a state machine and a sequence machine.

Please help me better understand the distinction.  I use macros for sequencing tests and operations that make up a test, but the sequences can branch depending upon how the UUT responds.  I call it a State Machine...

[crelf]

I purchased a Debug Deployment license for my project. I will use it, but my TestStand skills today are worse than my LabVIEW skills were two years ago.

TestStand's just like LabVIEW in that you can get some pretty cool stuff going with it almost out of the box. To be a true TS guy? That takes a lot of time

Please help me better understand the distinction. I use macros for sequencing tests and operations that make up a test, but the sequences can branch depending upon how the UUT responds. I call it a State Machine...

Yep, then that's a state machine.

The other thing that we have done is to keep the sequences themselves fairly basic. We created lots of custom step types that implement the complex tasks. All of our custom step types are implemented in LabVIEW.

Smart move - we also put multiple IVI calls with some extra stuff in some of our custom steps to streamline our sequences.

[Mark Smith]

I'm a little late to the party here, but I'll put in a vote for LuaVIEW. I have found this to be a very powerful tool for writing Lua scriptable applications for deployment (and I don't mean LabVIEW scripting). I use an object oriented LabVIEW framework for building code that encapsulates the test equipment (and collections of test equipment) for stimulus and measurement and includes classes for data persistence, evaluation, and reporting. I then use Lua (in the OO style calling convention) to build scripts that are specific to particular tests. The heavy lifting gets done by LabVIEW, and the scripts can be very lightweight.

One huge advantage to this approach to my customers is that the deployed app is now scriptable in a language that feels mostly natural to them (most of my customers are in R&D type environments and are scientists and engineers and know languages like Python, Perl, etc). They don't need LabVIEW to modify tests or experiments - and couldn't use it any way, since they don't have the source code on the testers - and there is no additional licensing fee (there is a license fee for the development toolkit) for the deployments. So they can tinker as much as the project API allows - the API is the set of Lua callable LabVIEW functions I expose plus the Lua callable LabVIEW functions LuaVIEW exposes plus the entire native Lua 5.0 language (which is Turing complete) - without me having to worry that they mucked up my source code.

Heres' an example

----------------------------- test function --------------------------

function testOne()

-- create objects - uses unique names and an action engine to

-- create by-val objects that are retrieved by-ref

powerSupply = PowerSupply.new("powerSupply")

powerMeter = RF_Power_Meter.new("powerMeter")

dataSaver = DataSaver.new("dataSaver")

-- configure objects - I use LabVIEW clusters/classes flattened

-- to XML because they are easily embedded in a text script

powerSupply:configure([[

<Cluster>

<Name>UUT Power</Name>

<NumElts>5</NumElts>

<String>

<Name>Description</Name>

<Val>Configure UUT Input Power</Val>

</String>

<Refnum>

<Name>Power Supply (Agilent E6644) VISA Resource Name</Name> --6038A

<RefKind>VISA</RefKind>

<Val>GPIB0::5::INSTR</Val>

</Refnum>

<DBL>

<Name>Voltage</Name>

<Val>28.00000000000000</Val>

</DBL>

<DBL>

<Name>Current Limit</Name>

<Val>2.00000000000000</Val>

</DBL>

<EW>

<Name>foldback protection (0:off)</Name>

<Choice>off</Choice>

<Choice>constant voltage</Choice>

<Choice>constant current</Choice>

<Val>0</Val>

</EW>

</Cluster>

]])

powerMeter:config([[

<Cluster>

<Name>RF Power Meter</Name>

<NumElts>5</NumElts>

<String>

<Name>Description</Name>

<Val>UUT Power Measurement</Val>

</String>

<Refnum>

<Name>Agilent E4418B VISA Resource Name</Name>

<RefKind>VISA</RefKind>

<Val>GPIB0::13::INSTR</Val>

</Refnum>

<I32>

<Name>Unit A (0: dBm)</Name>

<Val>0</Val>

</I32>

<DBL>

<Name>Lower Limit A (-90.00E+0)</Name>

<Val>90.00000000000000</Val>

</DBL>

<DBL>

<Name>Upper Limit A (90.00E+0)</Name>

<Val>90.00000000000000</Val>

</DBL>

</Cluster>

]])

powerSupply:apply() -- applies power to the UUT

power = powerMeter:measure() -- take a measurement - RF Power in this case

dataFile:writeObjectAsXML(powerMeter) -- save the data taken by the powerMeter

powerSupply:remove() -- remove power from the UUT

return power

------------- end of function -----------------------------

-- the test script calls the functions

--#import c:\scripts\functions

result = testOne()

if resultOne >= someLimit then -- if we pass the first test

repeat

result = someOtherTest() -- this test loops until the limit is met

until (result < someOtherLimit)

end

----------- end of test ------------------------------------

The XML configs can be edited in the text editor (I use SciTE since it's free and Lua syntax aware) or I have an app that parses the script, extracts the LV clusters, and loads them into their respective controls for editing.

So, the specific tests become functions that get called in the main script - the main script (and functions) can have any feature of the Lua language - control structures, math functions, system calls, file IO, etc. It's powerful, flexible, and relatively easy to maintain. This approach lets LabVIEW do what it's best at (instrument control, DAQ, threading) and lets Lua do what it's best at (imperative, easy to read and follow test directions). New functions can be deployed to the API without changing the exe (plug-in style).

Mostly I just wanted to present this as an alternative to TestStand (disclaimer - I have no interest in LuaVIEW other than as a satisfied user and I have very little exposure to TestStand, so maybe I've completely misinterpreted what TestStand does). And don't read this as "TestStand is bad" - I don't know TestStand but I think NI in general puts out quality products. I am a LabVIEW user, after all

Mark