JKI VI Tester - Use of suites and cases to test multiple classes in the same way

[Christian Butcher]

Hi,

I'm writing unit tests for some logging code (OK... so I suppose testing the actual logging class would be integration testing, but...) and have (at present) two disk logging classes and an in-memory test implementation.

My test suite/case arrangement is currently a bit of a mess (understatement) and what I really want to do is run the same set of tests using each logging implementation. Obviously I don't want to code up the same tests repeatedly (and have to maintain a set of multiples).

The solution as I understand it is to store a class object of the type to test in either the Suite or the Case (or both?) and then always read from private data when running a test. Writing the individual tests in this way is fine. My confusion lies in the best way to arrange the Suites and Cases to do this.

In particular, if I have some DiskLoggerTestCase containing a collection of tests (for simplicity, all of my tests) and with the Logger object as private data of the Case, then I can create a Suite with code like below in New.vi (duplicatedCasePerSuite.png). Here, I build an array of each Logging object, then use a Write accessor and an autoindexing For loop to build the Suite. Problem: In the VI Tester UI, I see "DiskLoggerTestSuite:DiskLoggerTestCase, DiskLoggerTestSuite:DiskLoggerTestCase". There appears to be no way to distinguish or add names separately (if this is incorrect, I'd love to know how to change it).

My second attempt (SuitePerImplementation.png) was to create a new Suite for the single TestCase, with each suite having a new name but the same TestCase (and using the write accessor, without a For loop). This works out, but means I now need to create a new suite for each additional implementation. It's not so terrible, but changes have to be reproduced in each suite if I want to modify something. Further, if I have more than one TestCase, it isn't possible to place them in a loop without reorganising them to inherit from a shared class providing the "Write Logging Object" VI as Dynamic Dispatch. Adding cases that don't use this require a new loop to discover their test methods. Duplicate code again rears its head.

My next attempt will be to create one Suite of TestCases, in which the Suite has a Logging object, and it passes the object to the necessary cases, whilst allowing me to then create a set of owning Suites that each contain the first Suite. This is probably more or less the same as the second approach, but should at least compartmentalise some of the changes needed when adding Test Cases and separate adding Test Cases (change to the first Suite) from adding implementations (add a new Suite of the second type).

 

Does anyone know the way this should really be done? I feel l'm missing a good solution and just creating difficult-to-maintain messes.

[Omar Mussa]

I personally think of TestSuites as a way to group a bunch of tests into a common test environment rather than as a way to reuse a TestCase for testing multiple parameters.  But I can definitely see the value of being able to set the displayed name of the test case on the VI Tester GUI and how it would help handle the situation you raise of re-running the same TestCase with multiple parameters.

The simplest way that I see to solve this problem is to just have 3 TestCases that share some common test code so that you can have easy to debug and use the three test cases to define the tests for each logger.

In this scenario, I think you'd have a test folder hierarchy like this:

Tests

Tests\Logger\DiskLogger1

Tests\Logger\DiskLogger2

Tests\Logger\InMemory

Tests\Logger\Common

The simplicity of this is that you can specify the specific tests for each logger, you don't have to worry about it being too complicated and you can add all three TestCases to a TestSuite if desired so that they can share some resources or be grouped for logical organizational reasons (or you can skip the TestSuite if you like).  Most likely, this is the way that I would approach that problem.

The other way that is possible to approach the problem is to use test case inheritance.  VI Tester does support TestCase child classes but you will have to make some tweaks to your code to make this work.

On disk you'll have something like this:

Tests\Logger\DiskLogger

Tests\Logger\DiskLogger.1

Tests\Logger\DiskLogger.2

Tests\Logger\DiskLogger.InMemory

where DiskLogger.1, DiskLogger.2 and DiskLogger.InMemory are classes that inherit from DiskLogger (which is the TestCase with all of your test methods).  You can create the child classes just by creating classes from your project - you don't need to create them using VI Tester.  You'll need to modify the test methods to use dynamic dispatch inputs and outputs (which will also require that the re-entrancy type changes to 'shared' - LabVIEW will warn you about this and its pretty easy to update).  You'll also need to create over-ride VIs for any tests in your child classes that you want to execute (currently by default, VI Tester is conservative and assumes that if you don't specify a test VI to over-ride then you don't want to execute it from your child class).  When you create a "New VI for Override" and specify your test method, by default the implementation is to call parent method so this can be done pretty quickly or via scripting.

After all that work, you can use your original TestSuite (the first image) and wire in a bunch of your child TestCases and each child will show its class name on the VI Tester Tree.

[Christian Butcher]

Thanks for the quick response - sorry I didn't see this straight away.

I've implemented something a little like your first suggestion: I now have

DiskLoggerTestSuite

-> LoggingSharedTestCase

-> PathGenTestCase (this is another shared component used by the logging classes

and then

DiskLoggerTypeATestSuite

-> DiskLoggerTestCase

DiskLoggerTypeBTestSuite

-> DiskLoggerTestCase

with the case shared between multiple suites. I need to create a new suite for a new class to test, but the suite is fairly simple and so quick to create.

At present, I have an ugly setUp for my DiskLoggerTestSuite (the shared code suite) since I'm writing a directory to private data of each TestCase, but adding in inheritance for these cases would solve that problem very easily (and I'll presumably do that next time I need to add something, or when I rework this project a bit more). I'm holding off only because of a blog post I read regarding malleable VIs in 2017 SP1 and the possibility of matching without inheritance (but I'll have to wait and see there).

 

Thanks for your thoughts on this - I'll no doubt reference it again next time I stumble into this issue and hopefully can take your advice to heart a little more fully.