Multiple Test Executives

[NeilA]

I have been working on a Test Executive as my main project at work for a while now. It has however been made clear to me now that I could potentially need multiple test executives working together to complete one test. Although this is a way off down the line (I want to think about it now).

I would be interested in the way people have used LabVIEW to create a similar set up or how you may go about it. Our supplier uses different language but does this by using a layer about what effectively is similar to my system that regiters the systems and kind of asset manages them.

I use a LabVIEW written server that monitors a datasocket for communication from a C# web app I created. The C# converts the SOAP message received in to an xml usable in the the test executive and sends it out on the data socket to the LabVIEW server. This is a seperate VI at the moment that I start when I start the test executive. I was wondering if I raised this VI in the heirarcy to be above the test executive. Enabling the Test executives to register with it and put the handling of the test in the hands of the server which splits the responsilities amongst subscribed test execs.

An alternative approach I thought of is to make the test execs have both master and slave capabilities and the first one active becomes master. This throws up the problem of them knowing where to look for the other execs etc.

Really not sure how best to implement something like this, any suggestions or help would be greatly appreciated.

Neil.

[crelf]

I use a LabVIEW written server that monitors a datasocket for communication from a C# web app I created. The C# converts the SOAP message received in to an xml usable in the the test executive and sends it out on the data socket to the LabVIEW server. This is a seperate VI at the moment that I start when I start the test executive.

Sounds complex. Is there a reason you're spending what looks to be significant time to design, develop and maintain all this yourself, instead of using TestStand?

[Michael Aivaliotis]

One problem that you will have to address is how to synchronize the multiple test execs. Also, how do you effectively manage your hardware (test equipment) allocation among test execs. I would seriously consider Teststand as well. You can reuse most if not all your test code written in LabVIEW.

[NeilA]

Took a while to find this it moved I think.

A little background:

The complete system is used for testing satellites, and controls all of the satellite during build. There are sub systems for power, configuration, monitoring, testing etc. Therefore the architecture of our testing system is complex and wierdly the test executive I am writing is only a small part one of the subsystems.

My executive controls an RF measurement system which contains things like spectrum analysers, network analysers, power meters and switching networks for RF testing. My executives' main purpose is to receive communications from the Master Test System Controller (MTSC) which controls the entire measurement and satellite payload and sub systems. It then decyphers the message, selects the correct test and passes the parameters to the test which is effectively a sub VI. My executive also controls execution of the test, then relays the results as they are captured to the MTSC.

The legacy system had an MTSC which was based on Teststand and controlled an in-house RF test system that was an HTBasic relic that lost support and functionality over the years. Our group came to rely on out sourced RF Test systems with teststand remaining. My executive is being developed to claw some of this capability back in house and to fit in to the new system using an MTSC developed in c# by my colleagues. Test Stand does a lot of things but we only used it simply to run sequences. I haven't been in the company long but I understand that teststand has not been used correctly and therefore not worked well. This has led to a redesign. I turned up after the design was created and I already had LabVIEW experience, so I chose labVIEW. It made sense for the tests and talking to the equipment. But has not been simple to implement the SOAP etc. If I used teststand in my executive I would again only use a subset of the teststands functionality and would still have to use a converter for the SOAP handling. The use of LabVIEW in this context is much more reasonable.

There is a requirement to have to test systems working together to acheive a certain type of test. This is being done by our outside provider by making the webservice part of the system Heirarchical leader of the test and delegating to the RF Measurments systems. I am certain that it is possible to either add the functionality to my SOAP handler function and have that delegate the tasks to the execs similar to our provider or add a master slave decision system in to my exec.

Does it make more sense now? I hope it does, I have asked questions about my executive on here before and have gotten a similar answer of Teststand can do that, try teststand. I believe that for my currently requirements the QSM structure is simple and works fine controlling the messaging and test control perfectly. The ability to control two or more test systems at once has again made me wince with concern that the path I have chosen is the right one specially when the answers are Teststand again.

Teststand is not always the right solution and our suppliers are not using it acheive the same thing I want to do so wanted to discuss really if the way they do it is the best. If you were in the same situation what might you do?

Many Thanks,

Neil.

[crelf]

Test Stand does a lot of things but we only used it simply to run sequences.

That's sad, but certainly not uncommon.

I haven't been in the company long but I understand that teststand has not been used correctly and therefore not worked well. This has led to a redesign. I turned up after the design was created and I already had LabVIEW experience, so I chose labVIEW... The use of LabVIEW in this context is much more reasonable... Teststand is not always the right solution and our suppliers are not using it acheive the same thing I want to do so wanted to discuss really if the way they do it is the best. If you were in the same situation what might you do?

Take it from someone who has been where you are, several times before - learn about TestStand and bring it into your life. Sure, you can do a lot of stuff from scratch with LabVIEW, but history suggests it will probably all fall apart sometime. Maybe not today, maybe not tomorrow, but soon. Use LabVIEW for the encapsulated tasks that you can't do with TestStand/IVI/VISA/DLLs/SQL/, and TestStand for everything else. I know it's the same message that you've heard many times before, but maybe that's because it's right.

As for your suppliers, maybe they aren't using it correctly or to it's full feature set. There are poor TestStand architects, just like there are poor LabVIEW architects. I'm not trying to say that TestStand is the be-all and end-all of test software, but if all you have is LabVIEW, everything looks like a VI

[Tim_S]

Sure, you can do a lot of stuff from scratch with LabVIEW, but history suggests it will probably all fall apart sometime. Maybe not today, maybe not tomorrow, but soon. Use LabVIEW for the encapsulated tasks that you can't do with TestStand/IVI/VISA/DLLs/SQL/, and TestStand for everything else. I know it's the same message that you've heard many times before, but maybe that's because it's right.

Crelf -

Could you explain this, especially the "fall apart sometime"?

Tim

[jcarmody]

[...] Maybe not today, maybe not tomorrow, but soon.

"I'm saying it because it's true. Inside of us, we both know you belong with TestStand. It's part of your workflow, the thing that keeps production going. If that project leaves the ground and you're not with TestStand, you'll regret it. Maybe not today. Maybe not tomorrow, but soon and for the rest of your life." Casablanca

[crelf]

Could you explain this, especially the "fall apart sometime"?

Unless you're some uber-architect-God, you'll design something that works for what you need right now, and maybe even add in a method of passing stuff around that you don't even know about yet. Then, after it's in use for a while, someone (including you) will want to make it do something that it's not quirte designed to do, and you'll have a question: redesign (and all the work required to retrofit the work that's already been done) or adjust (hack) the elements already in the design to perform outside of their original function (you can usually do this pretty quickly, and due to production pressures or otherwise, you decide this is the best route (you justify it to yourself by saying "I'm thinking outside the box" rather than thinking you should redesign the box). You might got through a few iterations of this until you end up with an architecture that is so crippled with edge cases that you finally decide it's time to re-design (hey, you've learned a lot of lessons, right?), and then you start the cycle again. Or do you? The production PCs need the new features, you don't have any downtime ro redesign, so you cut corners. You end up with an architecture that's better than the previous one, but it still doesn't quite have the expandability that you'll need in a month, a year, etc. You've invested an amazing amount of time (and a crazy amount of money) into developing a software system that you could have just bought off the shelf (which is cheaper, has much higher quality, and more features than you'll need in the next couple of years, is maintained by someone else, has a support base of thousands of people around the globe, I could go on...)

In short - spend you time wisely: work on the parts that are important to your product and your business - let someone else take care of all the other stuff. I don't know you or your business, but I'm assuming that your product isn't a test sequencing software platform, right? For example: you use LabVIEW - why don't you write your own version of LabVIEW in assembly? Because it just doesn't make sense to do that. So why are you writing your own, less functional, version of TestStand that you're going to need to continually support? (Unless it's job security )

By the way: TestStand really isn't *that* difficult to learn. It's like LabVIEW - you can get in on the ground floor, create some sequences, get stuff running. Then, as your knowledge matures, you can expand your software domain. Take a course, get a consultant in for a couple of days to walk you through it (you might even be able to get NI to give you someone for some time), get the course books and work through those if you don't have a lot of money (or find it difficult to convince upper management that it's valuable). The people that complain that moving to TestStand is too hard because their boss won't pay for it aren't explaining the true value of COTs Vs custom to their bosses right (or, perhaps, their bosses have their heads fair up their own arses).

All this comes down to COTS vs custom, and it's something we've debated here so many times. Use what's appropriate, and, in my experince, custom sequencing engines are rarely the right way to go. I'm not saying that TestStand is always the answer, but I'd bet a beer that it's the right medium- to long-term answer for you.

"I'm saying it because it's true. Inside of us, we both know you belong with TestStand. It's part of your workflow, the thing that keeps production going. If that project leaves the ground and you're not with TestStand, you'll regret it. Maybe not today. Maybe not tomorrow, but soon and for the rest of your life."

LOL

[Tim_S]

Unless you're some uber-architect-God,

Which, of course, I am.

I understand what you meant now. Thanks for the explanation.

By the way: TestStand really isn't *that* difficult to learn.

I worked at the very tail end of a project where the main developer implemented a system in version 1 of TestStand. I recall there being, shall we say, speaking in tongues and gnashing of teeth before the blood oaths were sworn to never speak of this ill again. As such I'm a little leery of working with it, though I have been trying to get some time to give it a serious look-see.

Tim

[crelf]

I understand what you meant now. Thanks for the explanation.

No worries.

I worked at the very tail end of a project where the main developer implemented a system in version 1 of TestStand. I recall there being, shall we say, speaking in tongues and gnashing of teeth before the blood oaths were sworn to never speak of this ill again. As such I'm a little leery of working with it, though I have been trying to get some time to give it a serious look-see.

Let's see:

• Version 1 of TestStand
• You started learning it in the tail end of a project
• You launched into a project without any real training on TestStand itself - only how it related to that project

Gee, I can't think of why that'd be a difficult situation Take another look at TestStand - it's come a loooooong way from v1 to v4.2.

[NeilA]

As I wrote in my posts above, my code is really a protocol converter that is called Test Executive (as that is what it does). My test Executive is able to interpret an XML message (that is received via a SOAP webservice) and starts/iterates through tests based on that XML message.

I have no need to edit test sequences or anything like that. The sequence for testing is held in the XML message. Using a refactoring of the XML I can determine the number of times the test is called etc.

Using TestStand to do what I am doing:-

• I would have to write a SOAP webservice (maybe a service that linked to a .dll or datasocket) that would call to TestStand in probably a similar way that I have already done for LabVIEW.
• I would have to create something to parse the XML, like I have done for LabVIEW (using the company schema not LabVIEWs).
• Then I could write a sequence using these items to call test VIs.

Currently the only difference to using TestStand that I can see would be using a sequence instead of a state machine. The hard parts of what I have done so far would still have needed to be done for Test Stand.

My state machine is a bit complex but at the heart is still a statemachine, a simple architecture that is proven in exactly the same way as TestStand. I may be too rookie with TestStand to know what I am missing but I work with some NI alliance members and they believe my implementation using LabVIEW is good and that TestStand would be wasted for my architecture.

Which is why I wanted to discuss ways in which you could potentially create what I wanted in LabVIEW which is to have two of my programs running at once that could work together to perform a test. For instance one sets a RF signal and the other measures it. Maybe either a heirarchy of VIs with a logging in system etc, or to implement a master/slave relationship where one nominates itself as master at start up and if a master quits there would be a shuffle to re-select a new master. I wanted to discuss the use of items like the network queue VIs etc with perhaps debate about what could be interesting code in the end. I have ideas of what to do if on one PC but on networked machines it gets a bit more confusing.

However, this turned into a discussion about TestStand which was magnified when the post was moved to a TestStand forum section. How am I going to get interesting LabVIEW responses now? I am annoyed that I am constantly told to use TestStand on a LabVIEW forum, I may not be explaining myself very well or something but I dont get what more I can say! I have had lots of help from the members of this site in the past, I am a bit sad this has been such a frustrating experience.

Neil.

[crelf]

I may be too rookie with TestStand to know what I am missing but I work with some NI alliance members and they believe my implementation using LabVIEW is good and that TestStand would be wasted for my architecture.

Are they TestStand experts?

However, this turned into a discussion about TestStand which was magnified when the post was moved to a TestStand forum section. How am I going to get interesting LabVIEW responses now?

I'm not sure who moved it, but I'll move it back for you. No need to be cranky - all you have to do is ask. All we're trying to do here is help each other.

I am annoyed that I am constantly told to use TestStand on a LabVIEW forum, I may not be explaining myself very well or something but I dont get what more I can say! I have had lots of help from the members of this site in the past, I am a bit sad this has been such a frustrating experience.

I think you're right - the reason that you're being constantly told to use TestStand is because, based on your descriptions, the members think that TestStand would be a better way to go. I don't think anyone here has a particular preference to TestStand over LabVIEW per se (as you menetioned, it's primarily a LabVIEW forum), everyone's just trying to help. That said, due to your fierce intent on using LabVIEW over TestStand, irrespective of your motivations, I won't mention TestStand in this thread ever again.

As an aside: maybe other members would understand your concept better and not make the mistake of suggesting the-platform-I-promised-not-to-mention-again-in-this-thread if you uploaded a concept diagram or even some architectural code.

[NeilA]

Was not being cranky, as I said I am just frustrated as I clearly don't know how to get my questions across correctly.

Are they TestStand experts?

I wouldn't go that far, lol! They are TestStand CLD and LabVIEW CLD I think. But I really think my use is a little light for TestStand (it is OK to say I am not really a psycho lol!) and since I have seen a few other Test Executive type questions on the forum I kind of thought my question would spark discussion.

I will see what I can use to explain how my system works and get back to you.

Cheers,

Neil.

[ShaunR]

Let me get this right in my limited stack. Correct me where I've got the wrong end of the stick.

You have a Master Test System (that is written by a 3rd party..not yourself )that manages a whole raft of tests including yours. This system communicates its desires (Test No./Name, number of times to execute, pass/fail criteria) that your "Executive" should execute via some sort of "translation" interface. Your executive goes away and tests the sub-system and then returns the results back to the Master System. A discrete test that the Master system must wait for. Where the Master System knows all and your sub-test just takes parameters and returns results.

This is how it appears after reading and is fairly straightforward except what you mean by "Executive". Many people use "Executive" and "Sequencer" synonymously. I tend to see the difference as an "Executive" manages "sequences" soin my mind, your Master Test System would be an "Executive" and your sub-test(s) would be a sequence(s) and by that definition, you only ever have one "Executive".

But the above seems a little oversimplified (I only get a sense of this) since anyone going to the lengths of incorporating multiple languages and defining XML interfaces to sub-tests, probably have a more flexible system in mind.

Certainly in similar systems I have worked on, the "sub-test" defined in the Master Test System is an entry point into a number of sub-tests. So in your example the entry point would be "RF Tests" and there would be "sub-sub-tests" like Output Power,Carrier Drift, Power Density etc. The question here is where the partitioning is and how simple you want the configuration of the "Executive/Master Test System" to be. Do you still want all the parameters defined in the Master System (huge amount of configuration information for EVERY test), or a simplified alias system where parameters are handled locally by each test. The latter is the preferred topology in production environments where different tests are maintained/designed by different teams, keeping the "Executive" simple and distributing maintenance workload across teams.

[NeilA]

Sorry, I have not had time to respond recently.

ShaunR I like that you have taken the time to understand my requirement and I think you have done well with my limited description.

I don't know where the boundry lies in what I am allowed to say so I have to be a little bit vague I'm afraid just to cover my own back, I hope you can understand.

The set up is like this:-

Now what needs to be understood from this set up is that the blue dotted RF measurement section can be my software (I call it the 'test exec' this is based on a company legacy software that I will replace) or any of our other suppliers. It is designed this way because they can all be attached at once or in any combination to test different parts of the DUT at once they can also be run locally without the Test system controller to perform debug and development and diagnostic checks.

I have this pretty much figured out, the interfaces were set by the rest of the team before I joined, they are using SOAP messaging, so I did what I could and implemented a layer in C# code which acts as a server that talks to LabVIEW and I built a native LabVIEW client.

I am happy with this part of the solution. My executive works really well and handles tests well. I found more confidence and happiness after reading a couple of other post that explained a similar situation using QSM architecture. However my solution is a little convoluted due to the teams choice of using SOAP which not handled well in LabVIEW.

The overall system is designed this way to remove any knowledge of the testing from the test controller to make it usable in all projects. The only parts that are test specific are the data in the database and the Test VI's (in my case) my executive doesn't know what the test is it just loads the test based on a single parameter in the execute XML received from the Test Controller.

OK, now then, my original question was based around the best way to implement what our supplier is doing for this current project. They are effectively running two testers at the same time that work in collaboration to acheive a more complex tests. There is no direct requirement to acheive this straight away but thought it may be worth looking into now as I could add hooks, loops or what ever you want to call them as I can, to help later.

They have done this by adding a layer above their tester which is a comms layer. This layer accepts the comms form the Test system controller and then uses a different simplified dialog/protocol to talk to the testers once the SOAP has been decyphered.

Sort of like this (excuse the quick and approximate drawing):-

I wanted to see how others would approach this sort of thing. Does this make any more sense? This idea seems the most obvious to me and is fairly simple. It envolves an extra layer of communication which I think I could expand my C# interface and use this to work out to which tester the commands should be sent.

I hope that atleast this gives you all an idea of what I am doing and hopefully explains why I am not using TestStand. I will try and answer any more questions if I can.

Many Thanks in advance for all/any help.

Neil.

[ShaunR]

Sorry, I have not had time to respond recently.

ShaunR I like that you have taken the time to understand my requirement and I think you have done well with my limited description.

I don't know where the boundry lies in what I am allowed to say so I have to be a little bit vague I'm afraid just to cover my own back, I hope you can understand.

The set up is like this:-

Now what needs to be understood from this set up is that the blue dotted RF measurement section can be my software (I call it the 'test exec' this is based on a company legacy software that I will replace) or any of our other suppliers. It is designed this way because they can all be attached at once or in any combination to test different parts of the DUT at once they can also be run locally without the Test system controller to perform debug and development and diagnostic checks.

I have this pretty much figured out, the interfaces were set by the rest of the team before I joined, they are using SOAP messaging, so I did what I could and implemented a layer in C# code which acts as a server that talks to LabVIEW and I built a native LabVIEW client.

I am happy with this part of the solution. My executive works really well and handles tests well. I found more confidence and happiness after reading a couple of other post that explained a similar situation using QSM architecture. However my solution is a little convoluted due to the teams choice of using SOAP which not handled well in LabVIEW.

The overall system is designed this way to remove any knowledge of the testing from the test controller to make it usable in all projects. The only parts that are test specific are the data in the database and the Test VI's (in my case) my executive doesn't know what the test is it just loads the test based on a single parameter in the execute XML received from the Test Controller.

OK, now then, my original question was based around the best way to implement what our supplier is doing for this current project. They are effectively running two testers at the same time that work in collaboration to acheive a more complex tests. There is no direct requirement to acheive this straight away but thought it may be worth looking into now as I could add hooks, loops or what ever you want to call them as I can, to help later.

They have done this by adding a layer above their tester which is a comms layer. This layer accepts the comms form the Test system controller and then uses a different simplified dialog/protocol to talk to the testers once the SOAP has been decyphered.

Sort of like this (excuse the quick and approximate drawing):-

I wanted to see how others would approach this sort of thing. Does this make any more sense? This idea seems the most obvious to me and is fairly simple. It envolves an extra layer of communication which I think I could expand my C# interface and use this to work out to which tester the commands should be sent.

I hope that atleast this gives you all an idea of what I am doing and hopefully explains why I am not using TestStand. I will try and answer any more questions if I can.

Many Thanks in advance for all/any help.

Neil.

Ok.I think I'm getting the gist of it.

You will notice that the supplier has broken the direct tie (RMS WS and TCS WS) between the master test system and the test layers (as opposed to just the incoming via your c# server). This is because it enables them to completely manage their inter-process comms without limitation. They can not only interpret requests from the Master System and re-interpret in a form that the subsystems can understand , but can also use a far greater vocabulary for inter process comms and filter/re-interpret back to the master.

I'm not quite sure what the difference is between the "Executive" and the "test" in terms of your labview program since the test vi will have a user interface and it seems only one test vi is used by each the "Executive" so the purpose behind "Main" isn't clear to me (generalised diagram?). I could understand it if the "Executive" could invoke or choose between multiple "tests" because it would basically be a plug-in architecture. But soldiering on.....

I would have used a similar topology as your supplier with what you describe, but the interface layer would have been Labview . The interface would have basically been a client/server with a few special case statements. On the one side (RMS WS) it would include an dynamic loader which could take the test name from the master and invoke the "Executive" for that test, configure it and tell it things like Stop, exit, pause run etc (if it is something I have written or execute and close it if it is a 3rd party exe). Basically invoke the test and pass on the parameters from the master. On the other side(TCS WS) I would have a mechanism (probably a queue) that receives info (status, results, progress, errors etc) from the "Executive" (can be one or more), filters out local information and repackages or retransmits information destined for the master.

How this would be realised is really dependent on how much control you have over the other parts of the system. If one of the tests is just an executable, you may be able to use DDE or perhaps it has a config file you can modify before executing it, but you are at the mercy of the forethought of the originator. If you have written the code, you can make it really slick.

[Dean Mills]

Neil

I have just finished reading this thread...Lots to read. I have created a few test executives the past, including 1 with true parallelism and a scripting language to control it. The best advice I can give, based on your current information, is don't try to add any hooks in now. Generally the reason for adding hooks is to make it easier to upgrade in the future. At Meikle we have legacy code that is over 9 years old that hasn't been touched in that amount of time. The last major upgrade to the test executive was about 7 years ago when I went with a dynamic parallel version. Current updates are more hacks than actual designed code. I wrote the original with what I thought were hooks in it to handle multiple parallel processes. I was somewhat off the mark. I ended up keeping 1 piece which was the actual VI that called all of my steps. The rest I rewrote from the ground up with some software that allowed me to launch, manage the number of parallel engines and test them. The result was 1 user interface with multiple low level engines. I now fondly think of that first version as a pre-proof of principle.

All of engines and steps are based off of the exact same queued state machine template, which now uses event structures, but dynamic events vs queues is a whole other nasty discussion. I cannot stress how important it is to use the exact same template, cause many years down the road you will look at the code and go "what was I thinking, why are they so different." The current architecture is a launcher that will launch the engines and close all the dynamic references, this also loads all the dynamic reentrant steps. Another VI that manages what engine is running what sequence (this is called dynamically from several other VI's with information about what script to run), one very convoluted VI that retains feedback about how each step of each script ran or is currently running, yet another VI which displays the timing information, one other VI that allows for a running from a user interface and the last creates the script. The actual TE VI includes a UI which allows for breakpoints, single stepping and running for debug. Within the language we have if else, rendezvous, semaphores and looping statements. We use these to control parallel processes as a TE step exists that can launch another TE either inline or parallel.

As for Teststand, I remember looking and playing with version 1. It was really suitable for what we were trying to accomplish. The people who were concerned with the cost of deploying our software thought the teststand license fee was better suited to stay at Meikle then send to NI. Besides it was an extremely fun and satisfying challenge to create a TE. Teststand has come a long way but isn't suited to the Meikle business model and industry we reside it.

Currently we are moving completely away from this TE format. We have created very accomplished Ladder Logic editor and engine that runs our stuff. It is very fast and easy to use. The problems and issues we have with our test executive is slowly going away as we are stopping using it. This is mostly based on the requirement we have of controlling an assembly machine or test machine with cylinders, motors, e-stops, MCR's, PLC communcation etc. As a scan of all IO and a sequencer written in Ladder is much easy to integrate.

From being able to upgrade code in the future I tend to look and design software with the philosophy that any VI which is a UI screen, any dynamic called engine, basically any independant entity should be able to be rewritten and plug back into the code very easily, without breaking the existing code. Based on that philosophy, my advice would be to have 1 user interface, 1 main program, 1 script manager that would sequence and control any test script being executed and 1 engine to run the script. Currently this wouldn't have the parallel capability but in the future you rebuild the script manager and the test script engine from the ground up to handle the parallel testing.

Good luck,

Dean

[crelf]

As for Teststand, I remember looking and playing with version 1. It was really suitable for what we were trying to accomplish. The people who were concerned with the cost of deploying our software thought the teststand license fee was better suited to stay at Meikle then send to NI.

Using TestStand in your product doesn't mean that you can charge for a license fee and take a chunk of that yourself (depending on the price pain point you see in your customers' eyes ). Besides, the costs of designing, building, testing, releasing, maintaining, etc your own executive... that's a lot of dosh. As an aside, the TestStand depolyment license fee has dramatically reduced recently.

Besides it was an extremely fun and satisfying challenge to create a TE.

That is totally true (I've done it a few times), but I figure it's also fun and satisfying to create the wheel Seriously though - my company is about test, and while it's great to create tools and libraries that we use internally, I find it more satisifying to create a system that tests something, not so much the tools I used to create it. I liken it to a power supply: you could buy a DC power supply for a few bucks of the shelf, or you could design, build, test, release, maintain, etc your own from discrete electronic components. Sure, building your own can be satisfying, but I'd rather use one that someone else made to create something bigger.

Teststand has come a long way but isn't suited to the Meikle business model and industry we reside it... From being able to upgrade code in the future... my advice would be to have 1 user interface, 1 main program, 1 script manager that would sequence and control any test script being executed and 1 engine to run the script. Currently this wouldn't have the parallel capability but in the future you rebuild the script manager and the test script engine from the ground up to handle the parallel testing.

I know I said I wouldn't say it, but TestStand can do that

OK - seriously, now I'll leave you all alone. Unless you say something about TestStand's capabilities that I think is wrong, I'll try to ignore this thread the best I can

[NeilA]

OK, thanks for the comments.

First off the diagrams were thrown together quickly this afternoon based on a few seperate diagrams I have that describe the seperate components of the system to give a simple overview to help my explanation.

To address a couple of the statements in your post ShaunR I consider the three components inside of the red dotted area to be the executive and the test VI represents the test called for measurement at that moment in time. There have only been a few Test VIs written at this point but they all follow a Queued State Machine architecture. The Executive and the Test VIs each have their own queues, the executive starts both queues but either the exec or test vi can add to the others queue this allows the test vi to call for information from the Master test controller about DUT or to ask the Master test controller to change the DUT cofiguration via the exec which has control of the SOAP messaging. So I think I have a similar sub-language/protocol you discss that is aparant in the supplier software. The C# sharp layer in the diagram is actually in 2 parts the first part is C# layer that is the best way I could find to create a SOAP webservice server, but, what it does is strip out the SOAP namespaces etc and then sends it through a localhost datasocket. The second part is a LabVIEW coded server that is loosely based on the XML-RPC server in the code repository (Sincere thanks to the dev(spoke to them before)) to monitor the localhost port and package the data into a native labview format from XML and place it on the queue.

What I was originally illuding to in my earlier posts is that I think y LabVIEW server could be promoted to a similar duty as the suppliers interface layer. In the main program/vi of the executive I currently examine the received execute messages from the LabVIEW server to discover which test to load this sort of task could also be promoted to the server and could also check for tester specific information that would help me to select the correct Test VI or queue to talk to which could also work with more than one at a time.

Also I would like to add that my Test VIs are hard coded tests which may make my software not strictly a Test Executive in the sequencer sence but just executes and controls the tests as more of an interface. The tests become set based on the test parameters that are sent to them ie frequencies, powers etc. Allowing them to be generic in performance and I forsee there to be a growing repository of tests available to be picked from.

The only problem is that in my system it may be easier to keep the system as it is but allow it to determine a pair of tests that run together and are both controlled by a single executive. But I am unsure how this could be done. Either system would require some sort of synchronisation. Both the synchronisation and queues would have have to be network dependant (which I have seen some things to help on this forum).

Does that make sense?

Have you had to deal with these sorts of issues? Are there any tips?

Dean,

A lot of what you say makes sense and is very informative it also gives me some confidence that my slightly covoluted but modular approach makes sense and that although not always the easiest route to test executives they can be done in LabVIEW. Also the fact that I have seperated the functionality means that I can change parts of the system without massive reprocussions like you suggested

Again I like to say thanks for your input, I am also relatively green but have learnt a lot from watching the forum and asking questions. I would still appreciate any further suggestions if you feel I am doing anything wrong or there is a better way.

Many Thanks,

Neil.

[ShaunR]

To address a couple of the statements in your post ShaunR I consider the three components inside of the red dotted area to be the executive and the test VI represents the test called for measurement at that moment in time. There have only been a few Test VIs written at this point but they all follow a Queued State Machine architecture. The Executive and the Test VIs each have their own queues, the executive starts both queues but either the exec or test vi can add to the others queue this allows the test vi to call for information from the Master test controller about DUT or to ask the Master test controller to change the DUT cofiguration via the exec which has control of the SOAP messaging. So I think I have a similar sub-language/protocol you discss that is aparant in the supplier software.

The C# sharp layer in the diagram is actually in 2 parts the first part is C# layer that is the best way I could find to create a SOAP webservice server, but, what it does is strip out the SOAP namespaces etc and then sends it through a localhost datasocket. The second part is a LabVIEW coded server that is loosely based on the XML-RPC server in the code repository (Sincere thanks to the dev(spoke to them before)) to monitor the localhost port and package the data into a native labview format from XML and place it on the queue.

What I was originally illuding to in my earlier posts is that I think y LabVIEW server could be promoted to a similar duty as the suppliers interface layer. In the main program/vi of the executive I currently examine the received execute messages from the LabVIEW server to discover which test to load this sort of task could also be promoted to the server and could also check for tester specific information that would help me to select the correct Test VI or queue to talk to which could also work with more than one at a time.

I still must be missing something here. I still don't see the purpose behind the "Executive" other than a translator between the "tests" and SOAP server which itself is a translator to the Master system.

Lets assume the C# bit doesn't exist. No better, is completely transparent so as far as you executive is concerned it receives information about a test (e.g. test name and limits) directly from the Master System. Your executive receives this info about a test to perform and does.....what? Calls a single test? Where is the logic that selects a test based on the info received (going by your drawings).

I may just be getting confused by the main and user interface, but it seems that the "executive" should be able to call more than one test so instead of your drawing showing an executive and user interface for each test (the tests are hard coded you say). It could just invoke the appropriate test and show its front panel (which is the user interface). The test can still be run locally by just dbl clicking on the test vi and your "executive" can invoke 3,4, n tests to run concurrently sucking up the messages from your queues and relaying them back to the Master System. In this scenario, the "Executive" is the same as your suppliers interface and there is no "executive" only the tests.

Also I would like to add that my Test VIs are hard coded tests which may make my software not strictly a Test Executive in the sequencer sence but just executes and controls the tests as more of an interface. The tests become set based on the test parameters that are sent to them ie frequencies, powers etc. Allowing them to be generic in performance and I forsee there to be a growing repository of tests available to be picked from.

Hmmm. What about if I described the little bubble in my noggin this way.

If you combined the Labview Server with the Executive. Removed the interface from the executive (hide its front panel) and instead used the Test Vis front panel as an interface. Then gave the Executive the capability to dynamically load tests (i.e execute one, leave it running, execute another, leave it running and monitor the queues). I think then you would have pretty much the same functionality as your supplier with less hierarchical levels.

The only problem is that in my system it may be easier to keep the system as it is but allow it to determine a pair of tests that run together and are both controlled by a single executive. But I am unsure how this could be done. Either system would require some sort of synchronisation. Both the synchronisation and queues would have have to be network dependant (which I have seen some things to help on this forum).

Does that make sense?

This bit does....lol.

I think you are describing 2 tests that may be standalone tests in their own right, but may also have to pause/wait if other tests are running concurrently. For this to happen you have (as I see it) three choices (others people may be able to see more, that's the beauty of forums )

1. A single all powerful intelligent sequencer (classic) that knows what to do and when, and orchestrates everything.

2. Get the tests to chat amongst themselves and only bug the "Executive/translator" when something important happens.

3. Or a (what I call) dumb sequencer (probably a good fit for your topology) that doesn't know anything about the tests (only which tests are running) but routes requests and messages from the tests (that only know about their test) which wait until they get the nod.

You are probably used to the first one. Intimidated by the second and never heard of the 3rd....lol.

The way the 3rd one works is this.

Day 1.

Test A starts running and says "hey can I run now?" and waits. The Sequencer Says "Sure" cos the sequencer knows that test A is the only test running. Test A says "ta very muchly.....here's the result". They all go home to the missus.

Day2.

Test A starts running and says "hey can I run now?" and waits. The sequencer is silent because it knows Test B is already running and test B must get to its lunch break before Test A can start. About 12 o'clock, Test B says to the sequencer "I'm off to lunch now" and pauses. The sequencer finally says to test A "Sure". A mightily relieved Test A says "ta very muchly...here's the result".Later. Test B comes back from lunch and says "hey can I run now?". The sequencer says "sure" because it knows test A slipped out the back door early and now Test B is alone in the lab. Test B says "ta very muchly...here's the result". And they all go home to Test A missus...lol.

Personification aside. What the third option would enable you to do is allow interaction with the master server either by pausing/stopping/reconfiguring the tests or relaying status information back to the Master since the "Executive/translator" is in the message loop (functional events if you like).

Waffle, waffls...lol

[NeilA]

OK to keep it simple I explained the simplest use for the system (single measurement). What the Executive actually does in its simplest guise is run a simple test, for instance a Gain vs Frequency sweep and translate and mediate all of the comms back and forth. In actuality the master system sends an execute xml that contains all the details for 64 sweeps of the DUT the executive then refactors the xml from one complete XML to a test specific XML.

XML structure:-

General test details

Detail 1

Detail 2

...

Detail n

---> Test 1 specific Details

---> Detail 1

---> Detail 2

---> ....

---> Detail n

---> Test 2 specific Details

---> Detail 1

---> Detail 2

---> ....

---> Detail n

Refactored to:-

General test details

Detail 1

Detail 2

...

Detail n

Test 1 specific Details

Detail 1

Detail 2

....

Detail n

I then send this flattened test specific XML to the Test VI. I then control the execution of these tests and inform the master system of the progress and translate the requests.

Not much more than the simple version but it gives you an idea of the reason behind the executive. We dont generally do this , but, the way it is done you to just change the XML and it could potentially run all of the tests with one call to the executive. This is simple because the XML is created by a database script and the data is populate by another script from an excel speadsheet. The spreadsheet become your sequence editor.

Edited August 14, 2009 by NeilA

[NeilA]

I have tried to reply correctly to this a couple of times I will create a more complete response when I get home from work my work set up seems to not like posting to LAVA.

Thanks,

Neil.