Experience with "Separate Compiled Code From Source"? (LV 2010)

[Rob Calhoun]

One of the most compelling features of LV2010 is the ability to remove the object code from VIs. This should greatly reduce the number of "recompile only" changes, which are a big headache for projects with multiple developers.

Early in the LV2010 beta this feature was applied automatically across the board when upgrading; in the final build it is disabled by default and controlled with a Project switch. The Project Properties window has a "Mark Project VIs..." button that (theoretically) can be used to split up source and object code en masse. I found the "Mark VIs" button didn't work with my (large) project in the last beta I tried, and it doesn't work in the final build that I just installed either. ("Mark VIs" opens for 5 seconds, then closes without comment. It does work with a brand new project with a single VI, but that is not much of a challenge.)

Since the approved method doesn't work, I'm considering walking the project hierarchy. We use dynamically-dispatched classes, recursion and various other modern LV complexities that mean I can't just call VI Open on App.AllVIs an expect it to work. Any recommendations on what to split? I.e., are there any types of VIs, such as dynamically---dispatched VIs, or VIs that get cloned at run time---that I should not split up?

[Shaun Hayward]

One of the most compelling features of LV2010 is the ability to remove the object code from VIs. This should greatly reduce the number of "recompile only" changes, which are a big headache for projects with multiple developers.

Early in the LV2010 beta this feature was applied automatically across the board when upgrading; in the final build it is disabled by default and controlled with a Project switch. The Project Properties window has a "Mark Project VIs..." button that (theoretically) can be used to split up source and object code en masse. I found the "Mark VIs" button didn't work with my (large) project in the last beta I tried, and it doesn't work in the final build that I just installed either. ("Mark VIs" opens for 5 seconds, then closes without comment. It does work with a brand new project with a single VI, but that is not much of a challenge.)

Since the approved method doesn't work, I'm considering walking the project hierarchy. We use dynamically-dispatched classes, recursion and various other modern LV complexities that mean I can't just call VI Open on App.AllVIs an expect it to work. Any recommendations on what to split? I.e., are there any types of VIs, such as dynamically---dispatched VIs, or VIs that get cloned at run time---that I should not split up?

I had the same weird thing with the dialog disappearing - my solution was to throw together this little VI to recursively set the property on all VIs in a folder - it's a little quick and dirty but does the trick

Mark as Source Only.vi

[oskarbosch]

I'll be using that VI.

As one of the reasons for me to move to 2010 was the separate compiled code.

how do I control where labview puts the compiled code?

(Note: have not looked into that yet either....)

[Rob Calhoun]

OK, now I have some experience with this feature and my experience has not been so good.

First the good news: you don't get dirty dots all over the place when you change a typedef. Yay! Also VIs are (a little) smaller. Yay!

The bad news:

1. The converter tool in the project doesn't work at all, but you knew that.
   
2. It makes building an application slower and less reliable. I'm not sure how much of the speed reduction is just the (more complex) LV 2010 compiler, but it takes just under 2 hours to build our product now vs about 50 minutes under LV 2009. Sometimes the builder stops working (stuck on build dialog using %0 CPU, forever), or throws a bogus error about how a "a bad VI cannot be saved without a block diagram" when in fact the VI is OK. Nuking the object cache and allowing LabView to rebuild it will fix the bogus error message, although it can come back if you build multiple products. Going through the source tree and removing busted stuff that isn't used in your application but is referenced in a library (or a referenced in a library that is referenced in a vi that's in a library that's in a vi that's in a library...I hate this behavior or libraries!) seems to help.
   
3. Things can get COMPLETELY messed up if you revert your working copy in subversion. I moved a typedef into a library with only the library open (vs all of the source code), which turns out to be a bad idea because LabView can't figure out that "mytypedef.vi" at mypath is the same as mylibrary.mytypedef at mypath. Well, OK, I can sort of understand that, so I reverted the whole thing and pulled everything in memory so the linking would propagate. Total mayhem ensued. A whole bunch of callers now believed that mytypedef was still in mylibrary, even though (since I'd reverted) neither mytypedef nor mylibrary believed this to be the case. Trying to fix it manually by selecting the VI from the file dialog (like it asked me to do) resulted in the attached error message when trying to save the caller. Yikes. The problem was that although I had reverted the source code, there was still some object cache sitting around from mytypedef's short excursion into library-land. At this point I just wanted my code back, so I quit LabView, trashed ViObjCache, reverted everything and started over. That seemed to fix everything.
   
4. I say "seemed to" because I found one block diagram constant of the typedef in question was sitting at value = 0, where it used to have value = 18. If you opened it directly from the windows explorer, or opened it with the vi server and then made a cosmetic change and reverted it (even though it was just loaded from disk and no changes had been made!) it fixed it. Trashing the object cache before launching LV also results in the correct value (18). But if you just used it (and there's no reason why I should ever look at it, given that it has not had a "real" code change in several years), the value of 0 was used. If you saved it or exported it in that stage, the bad value becomes permanent. Needless the say, the thought of LabView changing block diagram constants is pretty terrifying. We've never trusted cluster block diagram constants (that's a hard problem, updating the constant when the cluster is changed), but you'd think an atomic type (a U16 enum!) could not get screwed up in this manner. Well, I don't know exactly how it can happen, but it can happen. Yikes. I'm working with NI on this, but it's difficult to reproduce because any change to the vi (like moving it to the app engineer's computer) will generate a new viobjcache that won't be corrupt.
   

So I don't really know where to go from here. I've loved not having recompile-only changes that interfere with reasonable use of a source code control system, but the thought of block-diagram corruption is extremely worrisome to say the least. I've been thinking more about the potential problems associated with reverting a file in subversion without reverting (or deleting) the relevant file in the VIObjCache; it's actually a hard problem from LabView's perspective, because things have changed without them necessarily knowing that they've changed. I don't know what algorithm NI is using to determine whether obj cache file is up to date or not; it seems to be pretty good, but it's not perfect. Next time I do any big-time reverts (that includes major typedef changes, for example), I'm going to quit LV and nuke the object cache. I'm not sure what to do with the more common let's-just-revert-this-back-to-the-current-revision revert; blasting the whole vi obj cache will really increase the overhead of doing that.

Maybe we'll just go back to bundling the object code and source code, since then they at least get reverted together. I'm not looking forwards to that, but it beats having Labview corrupt my code.

[Jim Kring]

I had the same weird thing with the dialog disappearing - my solution was to throw together this little VI to recursively set the property on all VIs in a folder - it's a little quick and dirty but does the trick

Shaun,

You're the man! I'm starting to play around with "Separate Compiled Code From Source" and was about to create a tool just like your (and now I don't have to).

To me, it seems odd that the "Mark VIs" dialog does include this "folder on disk" functionality -- rarely do my LabVIEW project files include every VI from my project folder on disk.

Thanks!

[dannyt]

Well I am using the "Separate Source and Compile Code" system for a little while now and I am VERY happy with the results.

I agree the process or getting all your VI's updated to 2010 and the check flag set is a BIG pain and I can remember feeling sad and shouting about the change in default from the first Beta to the latter ones. I only have the top level VI's in my LabVIEW project everything else is just a dependent and the Project Method of marking VI's does not work on dependants.

Yes built times are slower in LabVEW 2010 by a factor of about x3 for us.

Yes the general 2010 IDE feels slower to me.

I did a similar VI to Shaun's, and I had real problems running it over my projects, I must have had my PC reboot on my around five times or more. Before I got it correct. It did not just fail but keep rebooting my machine.

I got it working in the end, there are a couple of things that I do differently to Shaun's one. I also set the "seperate flag" for all the VI in LLb that exist in my source code and I do not convert all the heirarchy. We keep in our source repository old file that are no longer used, just for reference (and to rebuild old versions of course) and these VI 's were not upgraded to LabVIEW 2010 and so had no "containscompilecode" property.

Why am I happy ? well doing some code change testing in LabVIEW 8.2.1 and LabVIEW 2010 I have already saved on over 200 files that would be marked as changed in LabVIEW 8.2.1 that are unaffected when doing the same change in LabVIEW 2010. That is 200 file fewer to do Quality Reviews on and track though our system..and just wait till 2011 comes out I will be able to test 2011 without checking my code base out and re saving it all.

Final thing anybody using the VI Analyser should remember to write their own analyser test to check the state of the "separate source and compile code" flag

Edited November 22, 2010 by dannyt

[Rob Calhoun]

OK, now I have some experience with this feature and my experience has not been so good....

I have more evidence of block diagram corruption as a result of using this feature. After modifying the type definition, a block diagram const array containing 11 type-defined enums was set to length 0 by LabView. I noticed, and therefore was able to recover my code by quitting LabView, trashing the object cache, and re-launching LabView. This makes the array re-appear. But if you run it in the corrupt state, you get an empty array on the wire, and (from my experience with another bug) if you modify and save a VI corrupted in this manner, you make the corruption permanent.

We have already modified our build procedure to trash the object cache before building, but I am pretty close to throwing in the towel on this feature. It corrupts my code, which is completely unacceptable.

I have more evidence of block diagram corruption as a result of using this feature. After modifying the type definition, a block diagram const array containing 11 type-defined enums was set to length 0 by LabView. I noticed, and therefore was able to recover my code by quitting LabView, trashing the object cache, and re-launching LabView. This makes the array re-appear. But if you run it in the corrupt state, you get an empty array on the wire, and (from my experience with another bug) if you modify and save a VI corrupted in this manner, you make the corruption permanent.

We have already modified our build procedure to trash the object cache before building, but I am pretty close to throwing in the towel on this feature. It corrupts my code, which is completely unacceptable.

If you're inside NI, I'm going to attached this to my previous CAR 277004.

[Rob Calhoun]

I have more evidence of block diagram corruption as a result of using this feature...

This issue is real, and I am working with LabView R&D on it.

In the meantime, I have a workaround that avoids (as far as I know) potential block diagram corruption caused by separating obj code from source code. Quit LabView, delete everything in your VIObjCache folder, and then make your VIObjCache folder unwriteable with directory permissions. LabView tries to save the obj code and fails silently (yay!); as a result, when a VI or control is loaded from disk the cached object code is never found and it is always regenerated from the source code. This ensures the object code state is always matches the block diagram state. (After the initial compile, there is no speed penalty; whether freshly recompiled or loaded from the disk cache, the obj code is held in memory during execution.)

The cost of doing this is significantly increased load times, especially for large projects. I view this as preferable to going back to recompile-only conflict heck. (And, obviously, it is much preferable to code corruption.)

[dannyt]

In the meantime, I have a workaround that avoids (as far as I know) potential block diagram corruption caused by separating obj code from source code. Quit LabView, delete everything in your VIObjCache folder, and then make your VIObjCache folder unwriteable with directory permissions. LabView tries to save the obj code and fails silently (yay!); as a result, when a VI or control is loaded from disk the cached object code is never found and it is always regenerated from the source code. This ensures the object code state is always matches the block diagram state. (After the initial compile, there is no speed penalty; whether freshly recompiled or loaded from the disk cache, the obj code is held in memory during execution.)

Hi Rob,

I am not seeing this type of situation and we am extensively using the separate source & compile code. I suppose what is really worrying me is that it is happening to me but I am not seeing it.

Is there a simple project / couple of VI's example that show this you could post so I can try and see the effect myself. I will try and spend a couple of days playing with this to see if I can see it.

The cost of doing this is significantly increased load times, especially for large projects. I view this as preferable to going back to recompile-only conflict heck. (And, obviously, it is much preferable to code corruption.)

This is very true for us as we have a large project, first load is very painful, I would really not like to have to do this yet especially as I cannot see the problem.

A question to an NI person, would a problem like this effect an executable build from code with source & compile code separated. The new executable builds are quite a bit longer than they were, I had assumed that was because when building, the source code was "freshly" build i.e. the cache was not used (if it was surly the builds would be quicker).

If build are OK, I can at least rely on our executable testing to check things are really OK.

cheers

Dannyt

[Rob Calhoun]

Is there a simple project / couple of VI's example that show this you could post so I can try and see the effect myself. I will try and spend a couple of days playing with this to see if I can see it.

I have caught this three times "in the wild" but unfortunately I do not have a synthetic example. I am still working on it, and will post one when I can.

A question to an NI person, would a problem like this effect an executable build from code with source & compile code separated. The new executable builds are quite a bit longer than they were, I had assumed that was because when building, the source code was "freshly" build i.e. the cache was not used (if it was surly the builds would be quicker).

If build are OK, I can at least rely on our executable testing to check things are really OK.

You are correct. The LV runtime does not have a compiler in it, so yes, the object code must be re-integrated into each VI/control as part of the build process. I recommend you trash the object cache before building to prevent incorporation of any corrupt object code. (There is a VI Server method to trash the VIObjCache; it works in-process and it is fast.)

We did have problems building projects that had "Disconnect Type Definitions" checked. Unchecking (which is the default for new buildspecs) and re-integrating the object code into controls programmatically before invoking the App Builder allowed us to build. (Builds failed two hours into the process with useless error messages; it was tedious to track this down.) But once built we have not seen any issues with the executables.

[Bab]

I know the thread is pretty old, but I am being struck pretty bad with this bug.

The bug (CAR 277004) is in the list of the bugs fixed in LV2011. But I am not planning to upgrade my projects to 2011 now. I have recently converted a project from 8.5.1 to 2010SP1 and another one from 2009 to 2010.

I have a simple vi which illustrates the bug:

I created it in LV821, then opened it in LV2010SP1, marked the vis a "separate compiled code", and saved them.

Now if the typedef enum is modified (item added or removed), the constant gets corrupted when it is opened again and the object cache exists. If I clear the object cache, then the constant is back to the correct value on the first open, but on the second (the cache exists again) it is wrong again.

If I create the same vi from scratch in LV2010SP1, the bug does not occur: no corruption when modifying the typedef.

Even more interesting: I delete all the code from the block diagram in the vi converted from LV821, and save it - I have an empty vi!.

If I create the same BD in this vi, with a new typedef created in LV2010SP1, the file gets corrupted in the same way as explained in the beginning of this message.

-> I conclude that the typedef itself is not responsible, but some trace of the vi with a typedef enum having been converted from an earlier version of LV to LV2010SP1.

Unfortunately these new findings do not show me oh I could "uncorrupt" my many vis with typedef enums which have been converted from earlier versions of LV. I suspect If I copy the code from the corrupted into a blank LV2010SP1 vi it would fix the vi, but it looks like a lot of work since a vi is not only a BD (FP elements, vi properties...)

Have anyone found more info on this issue?

Is there any known workaround?

Rob Calhound: how did it go with LabVIEW R&D? Did they find any other way to avoid the bug other than making the VIObjectCache read-only?

I have a support request with NI, but they don't have much info on it right now. I wish I could get in touch with the engineer who fixed this in LV2011 - he would know how to avoid or even fix this issue...

[KoBe]

Hi,

I'm using source control for a few months and since some weeks I'm not the only developer. Earlier i had all project files and library files locked by me, to avoid those "recompile save requests". But now I have locked only a portion of the code and it happens more often to me and my colleague, to loose some developed code, due to the "Don't save", because sometimes we are thinking to don't save only the recompiled code.

I read many threads on "separete compiled code from source" and I was wondering, if LV2011 is running stable with that. My NI support is quite busy at the moment to give me a clear answer about that.

Could anyone please answer my question "Should I use this feature in LV2011?" just with "Of course" without a following "but ...." ?? Our libraries are developed from LV2009 on, we did not use any earlier version.

Has anyone experience with LV-RealTime and RealTime targets and separated source code? How does that work? Any issues?

From the programming style point of view we aren't usingh neither object oriented programming, neither dynamically called VI's. Just plain LV VI-SubVI and Control typedefs.

Recompile often happen to us, just opening a VI in our Project under HOST and RT target, which use the same SubVI placed in our user.lib Just opening the calling VI of our user.lib VI once for the HOST and the another VI calling the same user.lib VI on the RT, causes a recompile of the user.lib VI. That is really a problem, because in some cases I'll be asked to save 50 or more VI's, just because of opening the MainRT after closing the MainHOST due to this recompile issue.

Any ideas? Any advice? Is the only way to resolve that to going to separate code from source?

[asbo]

Recompile often happen to us, just opening a VI in our Project under HOST and RT target, which use the same SubVI placed in our user.lib Just opening the calling VI of our user.lib VI once for the HOST and the another VI calling the same user.lib VI on the RT, causes a recompile of the user.lib VI. That is really a problem, because in some cases I'll be asked to save 50 or more VI's, just because of opening the MainRT after closing the MainHOST due to this recompile issue.

Any ideas? Any advice? Is the only way to resolve that to going to separate code from source?

You could clone your library to have RT and non-RT versions in your user lib. It might make a little more management work, but it'll prevent LabVIEW from recompiling for each platform. Separating compiled code from source won't actually fix your problem, it'll just move the problem out of your SCC.

[Jim Kring]

I'm curious whether people are using the "separating compiled code from source" feature in LabVIEW 2013 with success.  Are people still experiencing any issues? Do we trust this feature yet?

[hooovahh]

I'm curious whether people are using the "separating compiled code from source" feature in LabVIEW 2013 with success.  Are people still experiencing any issues? Do we trust this feature yet?

I've been using it almost exclusively on all code in our reuse library and several projects from versions 2011, 2012, and 2013.  I've had no issues yet.  I've also inlined just about all I can in the reuse library as well without incident.

[todd]

I jumped into the middle of two projects and converted them to subversion (I miss Hg!) and "separate". No discernible issues.

[Jim Kring]

Hi Todd, Brian,

 

I've been using it almost exclusively on all code in our reuse library and several projects from versions 2011, 2012, and 2013.  I've had no issues yet.  I've also inlined just about all I can in the reuse library as well without incident.

 

I jumped into the middle of two projects and converted them to subversion (I miss Hg!) and "separate". No discernible issues.

 

This is great to hear! Thanks for sharing your results.

 

-Jim

[lvb]

Separated compiled code has been working well for our team over the last 3 years.  There are two instances where we encountered issues:

 

• Packed Project Library build corrupting the VI Object Cache
  • The VI Object cache had to be cleared after PPL builds
    

  [*]TestStand with the LabVIEW Run-Time adapter requires uni-file code if distributed as source
  

 

-Brian

[ShaunR]

Separated compiled code has been working well for our team over the last 3 years.  There are two instances where we encountered issues:

 

• Packed Project Library build corrupting the VI Object Cache
  • The VI Object cache had to be cleared after PPL builds
    

  [*]TestStand with the LabVIEW Run-Time adapter requires uni-file code if distributed as source

 

-Brian

 

Just to expand on lvbs points.

 

If you build source plugins. i.e. plugins with diagrams so they can be recompiled on the target system when invoked from an executable. Then you must turn off compiled code separation for that VI otherwise you will get the Error 59: “The source file’s compiled code has been separated, and this version of LabVIEW does not have access to separated compiled code.

 

And just to reiterate a point that some people often forget. The global compiled code option only applies for "New Files" so checking it won't change all the existing files to use this method. You have to go and change each VIs setting individually and recompile.

 

DISCLAIMER:

The above are not "issues" as such. Just additional things that you have to bear in mind when using the option and may fit into the the grannys' eggs category. 

[hooovahh]

DISCLAIMER:

The above are not "issues" as such. Just additional things that you have to bear in mind when using the option and may fit into the the grannys' eggs category. 

I hope this is what you are talking about

 

http://en.wikipedia.org/wiki/Teaching_grandmother_to_suck_eggs

 

Never heard of it before.

[jgcode]

And just to reiterate a point that some people often forget. The global compiled code option only applies for "New Files" so checking it won't change all the existing files to use this method. You have to go and change each VIs setting individually and recompile.

 

That sounds like too much hard work - lol

You can bulk mark VIs through the project's options or you can use VI Server and script it

 

I tried this feature out recently and liked developing with it

One thing I concluded with tho is that if I ship a tool (essentially a source code dist) I want to install compiled code with it

Otherwise the tool needs to be compiled for use in its native version of LabVIEW

VIPM can do this on install, or the tool can also recompile itself, but IMHO it makes for a better UX (quicker install and first launch)

This was easily achieved using VI Server

[ThomasGutzler]

You can bulk mark VIs through the project's options

 

And that almost certainly works if you have a "small" project.

When I work with projects that started with compiled code inside the vi and were changed to separate, I occasionally open this vi from the project explorer and run it over the relevant folders:

(for the few of you who haven't coded it up themselves )

Edited March 18, 2014 by ThomasGutzler

[MikaelH]

And that almost certainly works if you have a "small" project.

When I work with projects that started with compiled code inside the vi and were changed to separate, I occasionally open this vi from the project explorer and run it over the relevant folders:

(for the few of you who haven't coded it up themselves )

 

You forgot the file pattern *.lvlcass and *.lvlib :-)

[jgcode]

Hi Mikael, I include all library files in my VI Server script as well, but I am insure as to what that option actually does. It seems the global settings (LabVIEW options or project options) override when a VI/ctl is added to a library (LV2012) making me think it is redundant or a bug. Do you know setting the library to separate compiled code does?

[MikaelH]

I assume it makes sure all new files added to the class or library will have the Compiled Code removed.