Testing private class vis

[Daklu]

JKI's newly announced VI Tester prompted me to finally post a question that's been on my mind for several months...

I often write test vis for a class' private subvis which obviously requires me to put the test vis in the class. I don't really like distributing test code with the class. Manually removing the test vis from the class for every build is a drag. (...and drop too!) I use OpenG Builder and OpenG Package Builder for my code; I believe it is possible to write a vi that will remove test vis from a class that I could use as a pre-build script, but I haven't had the time to really dig into it.

What are some strategies others use for testing private class vis, with or without the OpenG packages?

[jdunham]

QUOTE (Daklu @ Feb 2 2009, 09:08 AM)

I often write test vis for a class' private subvis which obviously requires me to put the test vis in the class.

OK, I'll ask the obvious... Why do you write test VIs for the private subvis? It seems to me that the nature of classes is that they expose a specific interface (all the public parts) and if you test the interface, then by definition your class works correctly. There shouldn't be any need to test the private methods inside the class. If they are not exercised by calling the public methods, then why do they exist at all?

Having said that, I have to admit my unit testing is sometimes lacking, so I'm willing to learn and be corrected.

[Jim Kring]

Once friend classes are available in LabVIEW (and this is likely to be on the roadmap) then a class under test can declare the testing class as a friend, so that the testing class can call the class under test's private methods.

Regarding whether it is good or necessary to perform tests on private methods...

In an ideal world you would probably never need to test private functionality because testing the public interface of a class should be sufficient (since you'll only ever need to use a class through its public interface). However, we're in a real world where we have to make compromises due to limitations in our tools and in our architectures in order to create working software within a reasonable amount of time.

I think it was on Jeff Atwood's blog that I heard something to the effect of:

In theory, there's no difference between theory and practice; but in practice, there is.

Another strategy (aside from making the test VIs members of the class under test) is to put anything that needs to be tested outside of the class under test. For example, if they are generic support functions, you could put them into another lvlib and make them public.

[Aristos Queue]

Hmm... Jim... I think theory and practice are closer in this case. Try this on:

You *shouldn't* test your private VIs. You should only test your public VIs. Why? Your private VIs are written to assume they are being called from within the class. Your private VIs may not be able to handle boundary conditions because the public API prevents those boundaries from ever being tested. Your private VIs may not have any error handling because their inputs are always sanitized by the public API. There may actually be logical errors in many many parts of your private VIs, but that's ok because there is absolutely no way for those ever to be invoked from the public API. Given all this, if you test your private VIs, you're going to become aware of defects that don't matter, and you'll waste development time fixing them. You may even cause slowdowns for your class by adding error checking and bounds checking code to the private parts of your class, replicating the work done by the public VIs. Or, to put it another way... Surgeon General's Warning: Testing your private parts directly may result in decreased performance, wasted development, and redundant material. Avoid testing your private parts in favor of having someone else test you through your public API.

Now, this is *theory* only. What are the thoughts from the practical experimentalists?

PS: Friend classes, should they ever come into being, won't do what you just said you expect them to do, ie, give free, uncontrolled access to the private parts of the class. An object doesn't mind if friends come inside the house, but following the object into the bathroom is going too far. That's one feature that I think is poorly implemented in C++ and JAVA, and if LV ever gets it on my watch, we're going to do something just a bit different...

[Daklu]

QUOTE (jdunham @ Feb 2 2009, 09:41 AM)

OK, I'll ask the obvious... Why do you write test VIs for the private subvis?

I'll tell you why *I* write test cases for private vis, but I'll leave it to those smarter than me to debate the 'correctness' of doing it.

The short answer is sometimes I make changes to private vis that inadvertently 'breaks' (as in it still executes but doesn't operate correctly) my application and it's easy to run a test case on a private vi to make sure it's behaving the way I expect it to.

The long answer is multifacited:

• Many of my classes are hardware dependent and I don't always have the hardware available to test the class as a whole. Testing sub vis gives me some confidence things will work when I connect to the hardware.
  
• Even if I do have the hardware available, you can't always test the results without adding other instruments to the system. (Managers tend to not like spending money on 'stuff to test the stuff I test our stuff with.') For example, my Weather Toolbox has a Rain class that includes a Rain.Stop public vi. I can't automate testing that vi without adding expensive instrumentation, but I can test the sub vis I use to make sure they work as expected. I use test cases to make sure the private vis I use in Rain.Stop are sending the correct messages to the vendor's Rain API.
  
• My private vis sometimes go several levels deep. I find testing private sub vis helps me locate bugs quickly, especially when I've had to change several sub vis at one time.
  
• I don't know any better...
  

I'm always willing to learn better ways of doing things if they address these issues.

QUOTE (Aristos Queue @ Feb 2 2009, 02:54 PM)

Given all this, if you test your private VIs, you're going to become aware of defects that don't matter, and you'll waste development time fixing them.

Only if you first waste development time broadly implementing tests that don't matter. At some point you have to trust the developer to know what needs to be tested and what doesn't in a given vi.

QUOTE (Jim Kring @ Feb 2 2009, 10:32 AM)

Another strategy (aside from making the test VIs members of the class under test) is to put anything that needs to be tested outside of the class under test. For example, if they are generic support functions, you could put them into another lvlib and make them public.

That doesn't work so well in my case--the sub vis I want to test tend to be very specific to the class or library I'm creating. They aren't meaningful outside of the context of the class.

[Omar Mussa]

QUOTE (Aristos Queue @ Feb 2 2009, 02:54 PM)

Avoid testing your private parts in ... public

Yes, you may find legal problems as well (I know it was badly out of context and edited but I couldn't resist )

QUOTE (Daklu @ Feb 2 2009, 04:16 PM)

I'll tell you why *I* write test cases for private vis, but I'll leave it to those smarter than me to debate the 'correctness' of doing it.

• Many of my classes are hardware dependent and I don't always have the hardware available to test the class as a whole. Testing sub vis gives me some confidence things will work when I connect to the hardware.
  
  

This is actually a design issue with your application that you should address. One of the great aspects of having LVOOP with inheritance is that you can create a parent class 'Rain.Abstract' and then have two children that inherit from it called 'Rain.Hardware' and 'Rain.Simulator'. This means that you can run all of your unit tests on the simulator implementation to flush out your object API and then rerun the tests when hardware exists by using the hardware implementation. This should significantly cut your bugs down -- ideally you'll only find issues where you either didn't simulate properly or you didn't imeplement the hardware (i.e. drivers) correctly.

One thing that writing unit tests does is that it forces you to 'FIX' your applications design issues so that you CAN unit test them (especially if you are doing 'Test Driven Development').

VI Tester can really help you with running tests of this nature -- For now, you can create a test object and in the setUp you can choose whether to create a hardware or simulator instance of your object to test. On the VI Tester roadmap is supporting test cases with inheritance so that you could create all of your unit tests in a 'Rain.Test' class and then have a 'Rain.Simulator.Test' class that creates a Rain.Simulator object in its setUp and would allow you to extend the Rain.Test class... We're still actively working on making this feature work well so that's as much as I can say for now.

[Aristos Queue]

QUOTE (Daklu @ Feb 2 2009, 06:16 PM)

The short answer is sometimes I make changes to private vis that inadvertently 'breaks' (as in it still executes but doesn't operate correctly) my application and it's easy to run a test case on a private vi to make sure it's behaving the way I expect it to.

You can always hit the run button on any VI for short, quick checks of functionality. And if you just want to run a quick test, it should be just as easy to run the public VI that calls your private VI as it would be to run the private VI directly. When we start talking "test suites", I'm thinking nightly builds and automated testing on a regular basis. Those auto tests have no business, IMHO, running the private VIs because, as I said, a parameter sweep across them may result in false error reports or may put your object into an unexpected bad state that can never actually occur in real uses -- and I do mean *can* never occur not *should* never occur. That's the thing about privacy enforced by the compiler -- outsiders *cannot* call that VI with arbitrary values. Since this *cannot* happen, we don't have to worry about testing for it.

[Omar Mussa]

QUOTE (Aristos Queue @ Feb 2 2009, 05:16 PM)

You can always hit the run button on any VI for short, quick checks of functionality.

I can almost never hit the run button for my class private (or public) member VIs. They often depend on data existing in the private class data -- some of that data could be queue references or other things that need to be created first. That is practically the whole point of having a unit test framework -- you can use test harnesses (setup/teardown VIs) to initialize data you want to have for each test and validate that the tests work in isolation from other tests.

QUOTE (Aristos Queue @ Feb 2 2009, 05:16 PM)

And if you just want to run a quick test, it should be just as easy to run the public VI that calls your private VI as it would be to run the private VI directly.

I think the problem is one of practicality. Its hard to know a priori exactly which public VIs will call private VIs (especially for VIs that are utilities set to private) -- but we do have a well defined VI that is the 'private' VI -- so it seems like it is reasonable that if we want to test certain ranges that we know public API VIs will need, we should be able to test those ranges. The problem with adding to the Public API to do this is that I don't want my Public API to get harder to describe/more verbose than it has to be -- for example, I don't want a VI called 'TestPrivateMethodX' that is a public member of my class.

QUOTE (Aristos Queue @ Feb 2 2009, 05:16 PM)

When we start talking "test suites", I'm thinking nightly builds and automated testing on a regular basis. Those auto tests have no business, IMHO, running the private VIs because, as I said, a parameter sweep across them may result in false error reports or may put your object into an unexpected bad state that can never actually occur in real uses -- and I do mean *can* never occur not *should* never occur. That's the thing about privacy enforced by the compiler -- outsiders *cannot* call that VI with arbitrary values. Since this *cannot* happen, we don't have to worry about testing for it.

Parameter sweeps across private VIs are probably not the only use case for testing private VIs. Another reason unit tests are 'useful' is for enforcing that changing your code hasn't created bugs. Since I can't unit test my private VIs properly, I can't be certain that making changes to private VIs did not create bugs in my public API (that I may not currently be testing with 100% certainty of coverage). So, IMO, these are valid requests from a unit test perspective. Otherwise, the answer is -- nothing should ever be private that you can't trust to work 100% of the time through your public API. Since I can't gaurantee I'm testing 100% of my public APIs interface with its private members, I'm basically asking for trouble. I don't like that answer from a practical perspective.

[Jim Kring]

I did some poking around and see that this question has been asked before -- here are some stack overflow threads on the topic.

It seems that the general consensus is that unit testing private functionality makes the test code-base more rigid and should be avoided. That said, I like to be able to gain the benefit of violating encapsulation with tests, where needed, provided that I understand the risks and costs.

I guess we'll all have to spend some more time unit testing our code and come back here with our experiences

[Omar Mussa]

QUOTE (Jim Kring @ Feb 2 2009, 05:58 PM)

It seems that the general consensus is that unit testing private functionality makes the test code-base more rigid and should be avoided.

Sort of, a lot of people are use 'Reflection' (google reflection and unit testing if you don't believe me ) to dynamically scope the private methods as public in order to test them (using other languages like java). I can imagine all kinds of s*!@ hitting the fan when doing that in LabVIEW since any class methods that happened to have the same name as the method you changed to public would be broken (due to not having dynamic dispatching enabled but being a part of the same class hierarchy).

Also, so far this http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html?page=1' rel='nofollow' target="_blank">link was the easiest for me to understand regarding reflection and unit testing -- I'm not a java programmer but its fairly digestible.

[Jim Kring]

QUOTE (Omar Mussa @ Feb 2 2009, 06:33 PM)

Sort of, a lot of people are use 'Reflection' (google reflection and unit testing if you don't believe me ) to dynamically scope the private methods as public in order to test them (using other languages like java). I can imagine all kinds of s*!@ hitting the fan when doing that in LabVIEW since any class methods that happened to have the same name as the method you changed to public would be broken (due to not having dynamic dispatching enabled but being a part of the same class hierarchy).

Also, so far this http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html?page=1' rel='nofollow' target="_blank">link was the easiest for me to understand regarding reflection and unit testing -- I'm not a java programmer but its fairly digestible.

Well, writing tests that call private methods means that your tests are broken. Sure, you could make the methods under test public before running the tests via reflection (so that the tests aren't broken when the tests are run), but who wants to do development on VIs that are normally broken?

[Daklu]

[Edit: I see Omar chipped in and described the same issues I have with hitting the run button on class vis... only he did a much better job explaining it! There's two hours I'll never get back... )

QUOTE (Omar Mussa @ Feb 2 2009, 04:39 PM)

This is actually a design issue with your application that you should address. One of the great aspects of having LVOOP with inheritance is that you can create a parent class 'Rain.Abstract' and then have two children that inherit from it called 'Rain.Hardware' and 'Rain.Simulator'.

I actually wanted to build my application in a way that allowed me to simulate hardware, but when I designed it (before I had any real experience with LVOOP) I didn't really know how to do so. Thanks for the tip. I'll admit I'm not convinced the overhead required to maintain three classes as opposed to one class would have been worth it in this particular case*, but I'll certainly keep that in mind for the future.

(*And I certainly wouldn't get approval to go back and refactor it now.)

QUOTE (Aristos Queue @ Feb 2 2009, 05:16 PM)

Those auto tests have no business, IMHO, running the private VIs...

I fully agree with you. (In principle at least... I'll reserve judgement for the practical aspects of it.) It would probably help to differentiate between a developer testing his own checked-out code and turning over "finished" checked-in code to be tested by a third party, such as an automated nightly build. My comments are referring more to self-testing dev code, although it encroaches on the latter somewhat since I don't have software test house.

QUOTE (Aristos Queue @ Feb 2 2009, 05:16 PM)

You can always hit the run button on any VI for short, quick checks of functionality.

I do that when I can. There are a couple issues I've encountered that sometimes make that impractical:

1. One of the main data types I use in my project is a virtual base class. (I derived child classes from the virtual base class to implement vendor specific code for the family of devices I'm testing.) Many of my private sub vis have the virtual base class as an input. Running the sub vi directly instantiates the base class, which of course throws an error when a virtual method is called.
   
2. Even if I have one of the child classes, some sub vis need the input classes to be in a certain state to execute the intended code. The class default values aren't necessarily the correct ones for that sub vi.
   
3. Many sub vis need to have several test cases run against them to get good code coverage. It's a pain for me to try to remember all the combinations of inputs that need to be tested and what their outputs should be. I'd much rather just click a button to test them.
   

Those issues have led me to building many test vis for private vis that include setup, input/output comparisons, and teardown. (I'm looking forward to using the test framework!)

--------------------------------------

FWIW, in my second post bullet #2, more than any of the others, is what drives me to test private sub vis. If the public vi doesn't have a testable output, how do you verify it? Case in point: My project uses NI Motion to control a robot. Several of my public vis are broad 'action' vis, such as to execute a specific test. Buried several layers down in the private sub vis are a set of functions that take data from the public vi's front panel, the motion controller, and the device being tested to create an array of positions for the robot to move to. The public vi doesn't have any outputs other than a few pass-through inputs and an error out indicator.

From the standpoint of only testing public vis, I don't see a realistic way of making sure that set of vis is functioning correctly and creating the coordinates I expect. As it turns out, there was a bug that caused the coordinates to be off by a couple mm, but under most of my test conditions it wasn't noticable while watching the robot. I found it when I was doing some tweaking and happened to increase the spacing between each robot coordinate. Then the error became large enough for me to notice.

[Herbert]

I agree that classes in general should be tested through their public interfaces. On the other hand, I would want to design my tests so they lead me to the root cause of a problem in the shortest amount of time possible. If a "black box" test using my public interface fails, I don't want to have to dig down my VI hierarchy in order to find the root cause. Not if I know that a "white box" test inside the class could have provided me with that information without me doing anything. So I guess I want to test both the public interface and the private methods.

Obviously, black box testing is the only way of making sure that you're testing the exact behavior your class will expose to its callers. A white box test can interfere with the inner workings of a class, bearing the risk that it alters the classes behavior or otherwise produces results that couldn't occur in a black box test. So, if a black box test fails, I'll probably have to fix my code. If a white box test fails, I might have to fix the test instead. Sometimes it's worthwhile adding and maintaining a white box test, sometimes it's not ...

I strongly encourage everyone who is interested in unit testing to watch out for new releases on ni.com/softwareengineering and related content on ni.com/largeapps on Friday, 02/06/2009.

[Jim Kring]

QUOTE (Herbert @ Feb 2 2009, 10:10 PM)

Sometimes it's worthwhile adding and maintaining a white box test, sometimes it's not ...

Hi Herbert,

That's basically how I feel, too. It's nice to hear you say it

(...even if I'm wrong, in theory)

Cheers,

-Jim

[Tomi Maila]

In an ideal world, creating classes would be super lightweight and super fast. In an ideal world we would have all the abstraction tools of all most advanced programming languages together. In an ideal world, we could test all the functionality for all the parameter space of a class public API. We don't live in an ideal world.

Somewhat limited abstraction capabilities and heavyweightness of creating classes for small tasks encourages sometimes to make design decisions where class private methods implement some functionality that would in an ideal world be better off as an external abstraction unit e.g. a class. In an ideal world, you would unit test these external abstraction units, so why would you not test the same functionality implemented inside your class private methods in our real world use cases.

Sometimes (often) the parameter space of the class public API is rather wide and testing it thoroughly is impossible give the computing resources. As a result, often many uncommon parameter space combinations stay untested in real world. These untested cases pop up as bugs when someone ends up using the class in an uncommon environment. In these cases you could gain more confidence by testing the internal building blocks of the public API.

Then there are some applications where the confidence requirements are just so high that you should also test the class private methods as well. Consider for example life support systems or multi billion dollar one-shot space projects. If the life support system malfunctions someone dies. If the space craft malfunctions, billions of dollars are wasted. So when life or billions of dollars are at the stake, it would be rather stupid not to test the class private methods as well, if it could save you from trouble.

QUOTE (Aristos Queue @ Feb 3 2009, 12:54 AM)

PS: Friend classes, should they ever come into being, won't do what you just said you expect them to do, ie, give free, uncontrolled access to the private parts of the class. An object doesn't mind if friends come inside the house, but following the object into the bathroom is going too far. That's one feature that I think is poorly implemented in C++ and JAVA, and if LV ever gets it on my watch, we're going to do something just a bit different...

Can you please shed some light on your idea of implementing friend classes. I'm interested in the computer science theoretical point of view.

[Aristos Queue]

QUOTE (Omar Mussa @ Feb 2 2009, 08:33 PM)

I can imagine all kinds of s*!@ hitting the fan when doing that in LabVIEW since any class methods that happened to have the same name as the method you changed to public would be broken (due to not having dynamic dispatching enabled but being a part of the same class hierarchy).

Well, actually, you don't have to worry about that because of the known issue in LVClasses that my team still hasn't been able to fix that even when a VI is private in the parent the child class is broken if it has a VI of the same name. It was going to be fixed in LV 8.6, but in LV8.5 someone (not on the LVClass feature team) wrote some new features for another part of LV that depend upon that uniqueness of name and enshrined it. Darn software -- it grows when you're not looking.

[Omar Mussa]

QUOTE (Aristos Queue @ Feb 3 2009, 09:48 AM)

It was going to be fixed in LV 8.6, but in LV8.5 someone (not on the LVClass feature team) wrote some new features for another part of LV that depend upon that uniqueness of name and enshrined it. Darn software -- it grows when you're not looking.

Sound like you need to be involved in some more internal design reviews I don't expect these problems to be easy to solve -- if they were, none of us would be working on them! I'm glad that LAVA provides a platform for discussing them at all :thumbup:

[Daklu]

QUOTE (Aristos Queue @ Feb 3 2009, 09:48 AM)

It was going to be fixed in LV 8.6, but in LV8.5 someone wrote some new features for another part of LV that depend upon that uniqueness of name and enshrined it.

I'm really curious, what feature would break if the original bug were fixed, and can we vote on the relative value of it?

[benjaminhysell]

QUOTE (Jim Kring @ Feb 2 2009, 08:58 PM)

It seems that the general consensus is that unit testing private functionality makes the test code-base more rigid and should be avoided. That said, I like to be able to gain the benefit of violating encapsulation with tests, where needed, provided that I understand the risks and costs.

I thought I would poke this thread and see if we can get it started again...

In the past six years doing TDD I've only actually unit tested a handful of private class methods, I would say less than four. I subscribe to what I feel is the overall tone of this thread, we shouldn't be testing private methods for all of the reasons given above.

However, that said, I agree with Jim, if I know what I am getting into there should be a method to unit test private methods.

In VS2008 you can do it, however they really make you really work for it, there is a whole dance you need to go through with reflection to load the assembly and to unit test it. It is a pain, but it can be done.

I think as VI Tester goes forward it would be nice to have the option, but an option that has a cost to the developer. Inflicting this cost would make a developer think twice about wanting to actually do the unit testing this way, and may in turn cause them to refactor their code to a cleaner implementation if at all possible.

[Jim Kring]

QUOTE (benjaminhysell @ Feb 24 2009, 08:58 AM)

I thought I would poke this thread and see if we can get it started again...

In the past six years doing TDD I've only actually unit tested a handful of private class methods, I would say less than four. I subscribe to what I feel is the overall tone of this thread, we shouldn't be testing private methods for all of the reasons given above.

However, that said, I agree with Jim, if I know what I am getting into there should be a method to unit test private methods.

In VS2008 you can do it, however they really make you really work for it, there is a whole dance you need to go through with reflection to load the assembly and to unit test it. It is a pain, but it can be done.

I think as VI Tester goes forward it would be nice to have the option, but an option that has a cost to the developer. Inflicting this cost would make a developer think twice about wanting to actually do the unit testing this way, and may in turn cause them to refactor their code to a cleaner implementation if at all possible.

We've given a little thought about how we might be able to unit test private methods, but none of them are easy, to say the least. This will probably be a very low priority, in the near term.

[Matthew Zaleski]

QUOTE (Aristos Queue @ Feb 2 2009, 06:54 PM)

Hmm... Jim... I think theory and practice are closer in this case. Try this on:

You *shouldn't* test your private VIs. You should only test your public VIs. Why? Your private VIs are written to assume they are being called from within the class. Your private VIs may not be able to handle boundary conditions because the public API prevents those boundaries from ever being tested. Your private VIs may not have any error handling because their inputs are always sanitized by the public API. There may actually be logical errors in many many parts of your private VIs, but that's ok because there is absolutely no way for those ever to be invoked from the public API. Given all this, if you test your private VIs, you're going to become aware of defects that don't matter, and you'll waste development time fixing them. You may even cause slowdowns for your class by adding error checking and bounds checking code to the private parts of your class, replicating the work done by the public VIs. Or, to put it another way... Surgeon General's Warning: Testing your private parts directly may result in decreased performance, wasted development, and redundant material. Avoid testing your private parts in favor of having someone else test you through your public API.

Now, this is *theory* only. What are the thoughts from the practical experimentalists?

I'm jumping in here late and I'm still learning LabVIEW OOP (in general). I come from a background of doing practical OOP in C++, Java, Python, and Matlab. I can't speak to what features LabVIEW has or doesn't have with respect to OOP ability.

If I have 10 public methods (public API) depending on 1 private method (part of the private API) at some point or another, I want to unit test the private method. I want to guarantee that I haven't broken the private API. Othewise I'm chasing ghosts at the public API level. Depending on the complexity of your classes, it can be downright painful to exercise the private API via the public API. Sometimes it is painful in the other direction; easy to unit test the public API.

Why are we discussing ways of making the tool (LabVIEW) harder on the developer? After spending a few years with Python, I've come around to their way of thinking. We are all adults. If you tell me that touching the stove will burn me, you don't also need to encase the stove in Lexan so that I can NEVER burn myself (nor can I cook food on it easily). Python lets you easily touch the private API but the access method lets you know that you are "bypassing standard safety protocols, proceed at your own risk." It goes hand in hand with http://en.wikipedia.org/wiki/Duck_typing' rel='nofollow' target="_blank">duck typing to make a programmer's life easier.

*steps off the soap box*

[Daklu]

I had never heard of Duck typing. As I reading the article I was thinking, "this sounds like dynamic interfaces." Something like that could be very useful. Labview seems to be pretty rigid as far as typing goes. I understand that makes it easier for inexperienced programmers to build functional code, but it also leads to frustrations when trying to implement more advanced functionality. In general I think Labview places too many constraints on developers.

I'm with you as far as testing private VIs. I can't think of any reason why developers should intentionally be prevented from testing private members if they determine doing so is useful. (Although there may be technical reasons under the hood that make that unfeasable.) It appears most of the posters agree. If I were to pursue it more I'd try writing something that changes all the VIs to public before starting the test and then changing them back when the test is complete.