Test Module LVOOP Architecture

[John Lokanis]

I am working on an architecture to support a dynamically loaded set of test modules using LVOOP techniques. Since I am new to LVOOP, I am not sure the best way to proceed. Here are some details:

• The exe needs to be able to call a test module from the external library using only the name of the library and the module. Assume that all libraries will be stored in a fixed location relative to the exe.
• The exe will spawn multiple parallel processes that could each call the same module at the same time. The module therefore must be completely reentrant to allow this with no blocking or cross contamination of state data.
• Each of the parallel processes is allow to call multiple copies of the same module simultaneously, if the module allows for this type of operation. So, each process must allow this with no blocking or cross contamination of state data.
• The exe needs to pass information to the module in a standard format to control its execution (parameters).
• The exe must be able to determine if the module is still running (or crashed).
• The module must be able to monitor a set of Booleans controlled by the exe in order to respond to abort or error flags.
• The module must be able to pass back messages to the exe while it is still running to update its status.
• The module must be able to pass back its results to the exe and those result must remain accessible and readable after the module has completed and left memory.
• The module must be executable outside of the exe for debugging and development purposes with minimal wrapping.
• Each library of modules must be packaged into a single file that removes diagrams, type defs, etc.
• Each library of modules much include all sub-vis calls by any module in the library and those must be name-spaced to prevent cross linking with similar named Vis from other modules in other libraries that might be in memory at the same time. This way a common set of source code can be used by all modules but is frozen to the version used at compile time (of the library)
• Once a module is loaded from disk and called by the exe, a template of it will remain in memory so that future calls will not incur a disk access as long as the exe remains in memory.

My first inclination is to make a common ancestor class for all test modules. Each test module would then be a child class of this ancestor. I could then call the module by instantiating an instance of that class. The ancestor could then have must override methods for parsing the parameter inputs and for converting the modules output into a standard format for the exe to read. Not sure how to implement the flag monitoring or status updates.

I really have no idea how to package the modules into libraries. Packed Project Libraries? Right now in my non-LVOOP implementation I use the OpenG builder to create an LLB with the same characteristics.

Any ideas you can offer are appreciated!

-John

[Daklu]

The exe needs to be able to call a test module from the external library using only the name of the library and the module. Assume that all libraries will be stored in a fixed location relative to the exe.

Is there a reason the test module needs to be a single file? Can you distribute them as source code with a predefined directory structure instead? People have reported some issues with packed project libraries, so I'd probably avoid them for now unless you're okay playing around in the sandbox a bit.

The exe will spawn multiple parallel processes that could each call the same module at the same time. The module therefore must be completely reentrant to allow this with no blocking or cross contamination of state data.

Each of the parallel processes is allow to call multiple copies of the same module simultaneously, if the module allows for this type of operation. So, each process must allow this with no blocking or cross contamination of state data.

If absolute non-blocking behavior is required, mark every vi in the parent class and child classes as shared reentrant. As a practical matter you'll probably be safe if you just identify the relatively time consuming methods and mark them shared reentrant. Also, by-ref class implementations may be problematic if you're not careful with your wiring.

Important question #1: Do you expect the parallel processes to make calls to the same source code module, or the same run-time object? The former is accomplished via reentrancy settings; the latter requires creating a lot more structural code.

Important question #2: Who controls the dynamically spawned processes, the exe or the module, and why? The requirement implies the exe has to own the process, but it might be easier using an actor object pattern to let the module control itself.

The exe needs to pass information to the module in a standard format to control its execution (parameters).

Not a problem. It's just designing an api for the modules.

The exe must be able to determine if the module is still running (or crashed).

There are several ways to do this. I usually have the module send out status update messages and the exe just keeps tracks of the module's most recent status. The modules are coded so they always send a MyModule:Exiting message as the last action before exiting. You can also implement a request/response asynchronous message pair, a synchronous message, or implement a method the exe can call that checks the status of a module's internal refnum, such as the input message queue, and make sure the module releases the refnum when it exits.

The module must be able to monitor a set of Booleans controlled by the exe in order to respond to abort or error flags.

The module must be able to pass back messages to the exe while it is still running to update its status.

The module must be able to pass back its results to the exe and those result must remain accessible and readable after the module has completed and left memory.

All part of the module API.

The module must be executable outside of the exe for debugging and development purposes with minimal wrapping.

Explain. Executable apart from the main exe without using the LV dev environment? Or are you just saying you want to be able to test the modules without relying on the executable's code?

Each library of modules must be packaged into a single file that removes diagrams, type defs, etc.

Oh. Ignore my first question above. Hmm... if this is a hard requirement it sounds like you need either a packed project library or a dll, and to be honest I'm not sure how well either one will work in your situation.

Each library of modules much include all sub-vis calls by any module in the library and those must be name-spaced to prevent cross linking with similar named Vis from other modules in other libraries that might be in memory at the same time. This way a common set of source code can be used by all modules but is frozen to the version used at compile time (of the library)

Hmm... I'll have to think on that for a bit...

Once a module is loaded from disk and called by the exe, a template of it will remain in memory so that future calls will not incur a disk access as long as the exe remains in memory.

Not sure about this one either. I think LV does this automatically. If not you could just load one copy of each module and keep them in an array until the exe exits.

My first inclination is to make a common ancestor class for all test modules. Each test module would then be a child class of this ancestor. I could then call the module by instantiating an instance of that class. The ancestor could then have must override methods for parsing the parameter inputs and for converting the modules output into a standard format for the exe to read.

That's my take on it too.

Not sure how to implement the flag monitoring or status updates.

Depends on how you implement the interface between the exe's parallel processes and the modules. If you use a message-based system, the exe sends FlagUpdate messages to the modules and the modules send StatusUpdate messages to the exe. If you stick with the more direct option of just calling the parent class methods, then you need to inject boolean references (dvr) into the module when it is created and periodically check them.

[Daklu]

Each library of modules much include all sub-vis calls by any module in the library and those must be name-spaced to prevent cross linking with similar named Vis from other modules in other libraries that might be in memory at the same time. This way a common set of source code can be used by all modules but is frozen to the version used at compile time (of the library)

Hmm... I'll have to think on that for a bit...

I think you only have two options here:

1. Create custom pre-build vis that create copies of your reusable code, or

2. Build the modules into dlls. (Note - I have no idea if you can dynamically load a child class from a dll.)

Once a module is loaded from disk and called by the exe, a template of it will remain in memory so that future calls will not incur a disk access as long as the exe remains in memory.

Not sure about this one either. I think LV does this automatically. If not you could just load one copy of each module and keep them in an array until the exe exits.

AQ's timing is impeccable once again. He commented on this just this morning over here. "A class cannot leave memory until all objects of that data type have left memory."

Here's my interpretation of what that means in your situation. Creating instance 1 of the module will obviously require disk access. Creating instances 2..n before instance 1 goes idle will not require disk access. If at any time all instances of a module are idle, the module may be unloaded and you'll take a disk hit when you instantiate the module again.

[John Lokanis]

The module must be executable outside of the exe for debugging and development purposes with minimal wrapping.

Explain. Executable apart from the main exe without using the LV dev environment? Or are you just saying you want to be able to test the modules without relying on the executable's code?

The second one. I want to be able to develop modules that will be runnable in the LV environment without the presence of the exe's services. In otherwords, the module should be as decoupled from the exe as possible so that a simple wrapper can be used to allow it's execution. I need this so that many team members can create and test their modules without needing to run the main exe's code.

Looks like I just need to caches an instance of each module class to hlod it in RAM.

I will have to think about the rest of your responses and get back to you...

[Daklu]

The second one. I want to be able to develop modules that will be runnable in the LV environment without the presence of the exe's services. In otherwords, the module should be as decoupled from the exe as possible so that a simple wrapper can be used to allow it's execution. I need this so that many team members can create and test their modules without needing to run the main exe's code.

This topic is related to the recent thread on Decoupling the UI. Creating plugins that can be executed without running the main exe is pretty easy. The module itself shouldn't know or care who is asking it to do things. How complex the wrapper is depends on the module's api and how much set up work the main app needs to do to create and use the modules.

Creating plugins where no source code dependencies exist between the application and the plug ins is a little trickier, but it can be done. The solution is something called The Dependency Inversion Principle. The DIP states:

• High-level modules should not depend on low-level modules. Both should depend on abstractions.
• Abstractions should not depend on details. Details should depend on abstractions.

I have a UML diagram of one solution using packed project libraries, but can't upload it right now. AQ briefly mentioned how to do it here.

[Daklu]

(Glad Lava is back... was going through withdrawals...)

Okay, here's how I think you can accomplish your goals. I haven't actually done it yet, but based on available information I believe it will work. Your source code will be set up like this.

Once you create PlugInInterface.lvlibp, you can have different teams work on MyApp source code and concrete plug in source code in parallel. Just make sure they all have the same version of the packed library... I'm not sure what will happen if you start mixing versions. Distributing the packed library to plug in developers might be a little unusual. It's designed for a specific app, so it's not really a candidate to add to your general reuse library. Personally I'd probably store each version on a server and let the devs download the most recent version when they start a new plug in. If your scc supports externals that might be another way to do it, but I can't speak to that.

When it comes time to create an executable, I'd include PlugInInterface.lvlibp as part of the executable, as shown here.

Again, I haven't tried it and I'm not positive any of this will work, but it's the route I'd explore given your requirements.

[John Lokanis]

Any chance you can build an example project to demonstrate this?

thanks,

-John

[Daklu]

Any chance you can build an example project to demonstrate this?

Well, I built some sample project that demonstrates the concept. And it even kind of works.... sometimes.

1. I thought the exe would absorb PlugInInterface.lvlibp, but it doesn't. It appears all packed project libraries (ppl) remain external to any builds that use them.

2. If an executable depends on a ppl, a copy of that ppl will be created during the build. Oddly, the built code doesn't appear to use the copy created during the build. It links to the same copy you used in the source code. After building the PlugInInterface ppl, move it to the directory where you are going to build the exe, and link to that copy in the MyApp source code.

3. Using a release version of PlugInInterface.lvlibp while creating ConcretePlugIn.lvclass prevented me from using the wizards to override parent class methods. A Release build strips the block diagrams and it appears the wizards need the block diagrams. It might work if you manually create all the overriding methods, but I didn't explore that much. It worked after I switched over to a debug ppl build.

4. Once you add a ppl to your project you can't move it's location in the file system. You have to completely remove it from your project, move it, then add it to your project again. My main method has only one class constant and one method from the ppl, and it still irrritated me to have to delete them before moving the ppl. I tend to move things around a lot while I'm refactoring so this is a show stopper for me.

5. In general, Labview was a lot less stable while I was playing around with ppls.

6. The attached version doesn't work very well--It's not loading the concrete plug in class. It *was* loading it earlier but I haven't figured out exactly what changed yet.

How important are those particular requirements?

MyApp.zip

[Tim_S]

Well, I built some sample project that demonstrates the concept. And it even kind of works.... sometimes.

This might be what you're looking to do. I unfortunately don't recall where I found this code to give credit, but it demonstrates how to use plugins in LV.

Tim

LVOOP Plugins Example.zip

[Daklu]

This might be what you're looking to do.

John was hoping to have a single file for each plugin to simplify distribution. Given the uncertainty of working with PPLs, deploying a directory instead of a single file might be the simpler solution.

[John Lokanis]

What I really want to achieve is a single file that contains a group of plugins and all their dependent sub-vis. I call this a library of plug-ins. And I want to be able to have multiple libraries that are installed over time and as needed. I want a single EXE that can call any plugin in any of these libraries if provided with the name of the library and the plugin within.

The libraries should be able to be built from the same pool of source code sub-vis but once built, they must own their version of the sub-vis so there can be no cross linking at runtime. This way, we do not need to re-validate the EXE or older libraries when we release a new one with potentially updated sub-vis.

I can currently do this using .LLBs and the OpenG builder. I am looking for a native NI solution that uses LVOOP and lvlibs or ppls or ??. I just want to understand the methods before diving into the re-architecture.

[Daklu]

I am looking for a native NI solution that uses LVOOP and lvlibs or ppls or ??.

I think ppls *should* be able to do it, but in practice the process for getting there isn't intuitive. At least... it's not intuitive to me.

Since you want a single file for each plugin, you're limited to packaging the plugin in a dll, ppl, or llb. I have no idea what Labview code compiled into a dll looks like to other Labview code. Maybe it would work...? Using PPL to do this is finicky at best. I haven't used LLBs either. Do library VIs retain their namespace and visibility when placed in an LLB? That might end up being the best solution, though I don't think you will be able to have any parent-child relationships inside the llb.

I just want to understand the methods before diving into the re-architecture.

This is a packaging and deployment issue and shouldn't have too much impact on the overall application architecture. Knowing you want plugins is enough to get started. I'm almost positive all your stated requirements can be met--at the extreme you could wrap each plugin in code that makes it operate as a stand alone app and communicate via TCP--it's just a matter of how much extra stuff you can tolerate to get there.

[Tim_S]

John was hoping to have a single file for each plugin to simplify distribution. Given the uncertainty of working with PPLs, deploying a directory instead of a single file might be the simpler solution.

I found a NI Forums post on how to combine VI server and packed libraries. I tried it with LV 2010 and had no luck to get it work with a class.

Tim

[SteveChandler]

Creating plugins where no source code dependencies exist between the application and the plug ins is a little trickier, but it can be done. The solution is something called The Dependency Inversion Principle. The DIP states:

• High-level modules should not depend on low-level modules. Both should depend on abstractions.
• Abstractions should not depend on details. Details should depend on abstractions.

Just trying to understand. Is this like the Java Virtual Machine which decouples your code from the OS?

[SteveChandler]

Since you want a single file for each plugin, you're limited to packaging the plugin in a dll, ppl, or llb. I have no idea what Labview code compiled into a dll looks like to other Labview code. Maybe it would work...? Using PPL to do this is finicky at best. I haven't used LLBs either. Do library VIs retain their namespace and visibility when placed in an LLB? That might end up being the best solution, though I don't think you will be able to have any parent-child relationships inside the llb.

The problem with llbs is that it is a flat file structure meaning you can not have two vi's with the same name. No dynamic dispatching for one. Other than that libraries will retain their namespaces. A problem with packed project libraries is that vis they contain can not be inlined. I ran into that when trying to build the LapDog Message Library into a ppl. I have never used dlls so no comment.

One thing I am considering is a source distribution in a zipfile. My app would have an "Install new plugin" function that just unzips the file into the plugins directory. Another thought is that when the initialization sees a zipfile in the plugins directory it unzips it and removes it. Then I have all the benefits of deploying a single file while still using a run of the mill source distribution.

[Daklu]

The problem with llbs is that it is a flat file structure meaning you can not have two vi's with the same name. No dynamic dispatching for one.

In this particular case I was imagining the abstract parent class in the exe and the child class in the llb. I think you could do that. You do end up unable to create any parent-child relationships within the llb, but that may be an acceptable constraint given the circumstances.

A problem with packed project libraries is that vis they contain can not be inlined.

I hadn't considered that, but it does make sense.

One thing I am considering is a source distribution in a zipfile. My app would have an "Install new plugin" function that just unzips the file into the plugins directory. Another thought is that when the initialization sees a zipfile in the plugins directory it unzips it and removes it. Then I have all the benefits of deploying a single file while still using a run of the mill source distribution.

Yeah, that's a good idea. VIPM uses zip files for their package format. Probably have to do some pre-install processing to suck up all the dependent vis and namespace them, but I don't see a good way to get around that.