Integrate Mocks and Stubs into HAL - recording app actions for testings and development

[0_o]

Hi,

First, I think there should be a main topic called testing instead of TestStand or along TestStand.

I implemented a Hardware Abstraction Layer as described by NI and I think it is poorly designed for the following reasons:

The ASL (application separation layer - API for general HW access) is not OO and lacks logic, error handling and stubs/mocks for simulation.

I found that during my app development I waist a lot of time since my HAL has no good testing framework with code reuse and each test unit needs all the system to be configured and running.

Besides that, if I have no access to the hardware it is very hard to develop since I need to write code, compile it, move it to a test machine, run + debug, come back to LabVIEW and try to find the bug with no live input for LV debug tools and without being able to exactly reproduce the scenario from the test machine. This process will repeat itself until I'll build an ad hoc recording mechanism which is not reusable for other features I develop.

Thus, I wonder, do you guys have an agile solution to this bad methodology?

Is there a tool that will automatically create some Mocks/Stubs to my code (stub is a recording of the application actions and responses. It can be used to virtualize the environment while actually working on a stand alone application. Thus, testing and development time is reduces).

I wrote such a recording mechanism for some of my features but it is not general or integrated into the HAL design I copied from NI.

Thanks in advance,

GOOfy Wires.

[brian]

Thanks for the feedback. There are some things I'd like to see improved in the HAL decomposition we've put forth (http://zone.ni.com/devzone/cda/epd/p/id/6307), but some of your feedback is new.

At the risk of introducing new terminology, I prefer to talk about a "measurement abstraction layer" and a "hardware abstraction layer". The ASL is closer to the MAL, and the DSSP is closer to the HAL. The MAL can present a high-level measurement--e.g., a "stimulus/response test", or "filter characterization test"--and implement the test strategy--e.g., to use an RFSG/RFSA frequency sweep, or wideband pulse measurement. I would expect both the MAL and HAL layers to be OO.

One of the cool things about our approach is that you can choose to simulate (stub/mock) at either level of the abstraction, and it's just a matter of writing another implementation class. If you want to use simulation at the HAL layer, then we already have simulation built into our IVI and modular instrument drivers.

It sounds like you want to record real-world measurement data and then play it back. Having a HAL in place will facilitate the recording of the data, since you can connect to your actual test bench and capture the data to a file without touching the rest of your application. Then you can use a different HAL implementation class to play back the data from that file.

[0_o]

Hi Brian,

I didn't expect the developer of HAL to answer my questions here. Cool!

I would love to see an integrated version of MAL over HAL with integrated OO testing and bug fixes.

However, my question regards a different subject:

A good simulation will have to record a sequence of high level UI actions and understand which low level MAL/HAL action was taken so it could record a sequence of actions, record the relevant inputs for each action and finally play the responses for virtual UI actions.

This process should be general and not feature specific. At the moment I need to write specific record and replay for each feature. I'm searching for a general solution that will listen to the UI and add record replay to the lowest level of the DSSP and thus simulate all the high level actions at once.

All the high level (no driver/communication access vis) should be the same in the simulation mode since I'm actually trying to debug them. The lowest level should get a flag with a link to the relevant recording it should run if this is a simulation mode for a specific action that was recorded. If I need to maintain a parallel simulation class I lose all the point of having this simulation mode.

I'm trying to understand what you think I should do according to your last paragraph and I still don't understand how you are going to match together the recording to the UI action. A Stub/Mock doesn't just replay a black box, it allows a simulation of UI actions while replaying the relevant response to each specific action. Thus, if I reorder the response to move up, move down, roll over and howl I can ask in the virtual mode to do each of those actions in a different order even though some use the same DSSP vis and each time those same vis were called a different recording was run according to the action we simulate.

Thanks for the replay!

P.S. - I would pay good money for a good NI platform that will build Mocks/Stubs while adding some CI testing capabilities to a MAL/HAL design along some automatic builder in LabVIEW.

[Aristos Queue]

At the risk of causing heartburn and panic among my fellows in LV R&D, there is a feature in LabVIEW that will do what you just asked for as far as recording is concerned.

The reason it may cause panic is because it is so rarely used, so although the test suite keeps passing, it hasn't had any developers work on it in well over a decade. And yet it is still there. We kind of loathe this feature because keeping it working has required some extra complexity in some new features, and we've talked about killing it. Me advocating it as a solution runs the risk of breathing new life into it. :-) I give you this intro so that you understand: the UI is a bit rickety, but it works, at least for its intended original use case.

Right click on any subVI node. In the menu, you'll see "Enable Database". You probably have never used this menu item (I've polled huge crowds of LV users and I almost never get anyone who knows what it does unless there's a LV R&D teammate in the room). "Enable Database" will cause a "halo" to appear around the node. All the input terminals turn into output terminals, and the halo has additional terminals.

When this node is part of your application, any calls to the same subVI as the halo'd subVI get logged -- the inputs and the outputs. When the halo'd node executes, it takes as input an integer that is a call ID. This allows you to retrieve the conpane of that subVI as it was the Nth time it was executed.

I know what this feature does for static dispatch, non-reentrant subVIs. For anything else, well, I will bet that it won't crash, but I've got no idea what the defined behaviors would be. I'm 90% certain this feature does not work for dynamic dispatch VIs (I recall consciously disabling it, but someone else on my team may have hooked it up at some point). I have no idea what its behavior is for reentrant VIs.

Play around with it. See how it works. It may be slower than you want (I've got no idea what the data structure that it uses for a database looks like). It may have memory issues (I know it doesn't do any disk caching or anything like that). But perhaps it has value to you.

[0_o]

Hi Aristos Queue,

Thanks for the risk you taken here!

While searching for a solution I went over most options I could put my hands on and this was one of the winners.

It gave me some trouble in freezing a vi since it tried to record something and I didn't take it into concideration and as for dynamic vis, it misses the point for HAL which is OO and in some points reentrant.

If this feature was working correctly it could have been a solution and a very nice one too with just a little bit of a framework to wrap it up.

P.S. - if we talk about unused/depricated features I came across some property nodes for code coverage and other cool stuff like convert to vhdl that I didn't know LV could handle. I understood they are volatile but now I see even some right click features are risky.

P.S.2- now that I think about it again, I can use it to record only the lowest level that uses driver blocks and match it with the call chain and the control combination. This might just work since those vis are static non reentrant (even singletonian) !!! Thanks!!!!!

I'll post back once it is tested.

[Aristos Queue]

While searching for a solution I went over most options I could put my hands on and this was one of the winners.

It gave me some trouble in freezing a vi since it tried to record something and I didn't take it into concideration and as for dynamic vis, it misses the point for HAL which is OO and in some points reentrant.

Should be easy enough to wrap the interesting calls in a static VI wrapper.

I'm not even sure how I would define the behavior to work for dyn dispatch... should it log every call to just the one VI that is halo'ed? Every call? How about calls that happen through the Call Parent Node? If it is every call, what about calls that are explicitly to a higher level of inheritance? What about calls that are to a lower level of inheritance?

but now I see even some right click features are risky.

Not risky in the same sense. This feature works, and works well, exactly as designed. It hasn't been depricated or anything like that. It just hasn't been polished in a while and if you have questions, it means a lot of "oh, how did that go again?" research on the part of folks here at NI. :-)

[Rolf Kalbermatter]

At the risk of causing heartburn and panic among my fellows in LV R&D, there is a feature in LabVIEW that will do what you just asked for as far as recording is concerned.

The reason it may cause panic is because it is so rarely used, so although the test suite keeps passing, it hasn't had any developers work on it in well over a decade. And yet it is still there. We kind of loathe this feature because keeping it working has required some extra complexity in some new features, and we've talked about killing it. Me advocating it as a solution runs the risk of breathing new life into it. :-) I give you this intro so that you understand: the UI is a bit rickety, but it works, at least for its intended original use case.

Right click on any subVI node. In the menu, you'll see "Enable Database". You probably have never used this menu item (I've polled huge crowds of LV users and I almost never get anyone who knows what it does unless there's a LV R&D teammate in the room). "Enable Database" will cause a "halo" to appear around the node. All the input terminals turn into output terminals, and the halo has additional terminals.

Well I have been using this in the past for debugging communication drivers, using it on the Read/Write function to log all the strings sent and received to see how it went bad at some point and try to understand protocol handling issues. As such it is helpful but not exactly user friendly and I have since usually added in such cases a direct debug logger that takes the interesting parameters and writes a string line to a text file on disk. It means writing specific code for each case but is a lot easier to parse and analyze later on than manually stepping through each call iteration. If there would be a method to hook a log VI to a VI instance (or its class, I wonder if the Class Operator might be possible for that and I'm not talking directly about LVOOP but the App method App->Get/Set Class Operator) then that might be more helpful. Such a logger VI would probably receive a data structure that contains a copy/reference to the actual conpane data and would have a pre and post execution call.

The Enable Database feature has been discussed in some LabVIEW classes back in the old days (15 to 20 years ago) but has gone into the land of forgettingness since.

[0_o]

rolfk,

I just want to emphasize the different uses of a logger and a recorder.

A logger is used for debug. You log just the relevant information in order to understand what happened and for this you'll need to parse the log and follow it.

A recorder records all the input/output in the ASL level so you could simulate the operation of the system without having it at hand.

You don't need to parse a recorder. You use it to verify that the program you wrote still works as expected and develop new features assuming the system gives you a specific response you got in the past.

There is a place for logging and there is a place for recording.

I find loggers to be useful for parallel loops (if you write good parallel code there is not much use for them) while recorders are very useful since they shorten the development time and add a very easy to implement regression testing.

Aristos Queue,

thanks! I'll write back after I get the chance to test it.

Edited June 3, 2012 by 0_o