Framework for cross platform Automated Testing?

[Thomas Robertson]

We have a fairly large application (5,000+ vi's, 20 years of development).  Much like the Software Engineering books promise will happen, adding functionality without breaking something is proving to be almost impossible.  We would like to get into automated testing/test driven development and I'm trying to come up with a framework for making that happen.  The framework would need to sync to the latest code changes throughout the day and run thousands of test cases and report any failures.  Futhermore it would need to do overnight testing of long tests (e.g. performance, memory leak) and probably weekly tests.

 

At first I looked into the LabVIEW unit test framework and figured we could make lots of unit tests and have a program which simply scours our sourcecode directories and executes any unit tests and generates a report.  However, the unit test framework only works on Windows x86.  Our software runs on x86, x64, and MacOS x64.  The JKI "VI Tester" may also be a possibility for creating test cases.  That said looking at the roadmap it appears that it doesn't allow for automated running of test cases.  I'm also a bit worried that about whether VI Tester is something that will be around long term or if it's getting deprecated by NI's Unit Test Framework.

 

I know that there's plenty of framework's outside LabVIEW such as Google's gtest and I'm wondering if using such an outside tool with individual test cases interfacing to LabVIEW might make sense?

 

Who's done this before?  Since it's supposedly a cornerstone in agile development I would think there should be an off the shelf answer for this.

[OlivierL]

Hi Thomas, considering where you are at right now, I would recommend implementing an automated test by starting at system level because creating Unit Tests for a significant subset of your 5000 VIs will take a lot of time. After creating the system level tests, then I would look into getting more test cases for each individual part of your application.

 

For the implementation, we have used UTF in the past but we usually create our own VIs for system level tests as the setup can be quite complex and we do not feel that UTF offers us much of a benefit at that level. What we do is add a script ran by the OS (in your case you would have a slightly different script running on three PCs or likely 3VMs <- VMs allow you to share HW resources more easily...) every night that:

 1 - Updates SVN folders that are part of the test

 2 - Calls an VI in an executable format (that you could compile independently on each platform) and which performs the system level test while monitoring RAM & CPU usage in the background.

 3 - Call another script/application to parse all the result files and create a separate report including graphs of the logged data overlayed on top of known good results. We do this for the values that are harder to analyze automatically but for which a human eye can analyze in a glance the following morning.

 

Since our error handling includes a FGV holding all the errors/Warnings generated, we can easily include those in the reports along with all other results and be part of our P/F criteria.

 

One thing to keep in mind is that the task often looks daunting but you have to start somewhere. I find it a lot easier to start with a limited scope which grows a lot over time. With this incremental process, creating your first set of tests is more manageable and you can add more test cases as problems are discovered and you want to make sure to cover those in future releases. When implementing the automated testing after the facts, we usually begin creating unit tests as we find the most critical sections that can break, and as we make modifications to existing code. All new code changes should include implementing proper unit test for the given module.

 

Hope this helps.

[ak_nz]

We can run VI Tester tests in an automated fashion using the API (just like the UTF). You'd need the development environment on your CI machine though. There are plenty of posts around about setting up a CI system with LabVIEW.

 

I'd also like to add that VI Tester will not be deprecated by the UTF. They are tools that target testing in different ways - nether one will supplant the other any time soon.

[Thomas Robertson]

I have started exploring VI Tester and it does indeed seem to do what I need (I notice that the testing GUI doesn't work on Mac OS, but the code for running tests automated seems to work on Mac OS).  @OliverL To your point I am going to do system level testing to start with but I think I'll do it within the VI Tester framework.  It seems I can put whatever code into the test cases I want so there's nothing to stop me from putting code that talks to my running application and has it do large scale testing.  After that unit tests could be seperately added in.