[CR] JKI Fast Mass Compile Tool

[crelf]

I hope that you find this useful and it saves you some time so that you can get back to doing the things that you love.

Let me be the first to say: Wow! Great idea and excellent execution! I really appreciate this little utility (as will the rest of my engineers who will be surely using it over the comming weeks to get their own 8.0's to 8.0.1. Funny that NI never thought of doing it this way... Other than the processor usage going to 100% when it's listing the VIs to compile at the start (this can be fixed with a 0ms wait in the recursive file list vi), this has saved me about a day - thanks sooooo much!

My 8.0 -> 8.0.1 mass compile time went from more than 7.5 hours (I actually cancelled it, and it was only 2/3 of the way through), down to 36 minutes :laugh:

I seriously owe you a :beer: at NI-Week in August - and I suggest that anyone else using this little marvel should think about putting your name down to buy Jim a :beer: as well!

[Jim Kring]

Version Update: Version 1.0.1 - Thursday, March 9, 2006 at 06:36:34 (UTC)

[FIX] Open VI Reference was not being called with option 0x01 to record modifications. This was causing the modifications bitset of all VIs to be 0 and the mass compile was not saving any VIs.

You MUST rerun the Mass Compile utility if you were using version 1.0

[Lennart]

Hi,

TNX for suppling the "compiler tool" It took me 01:26:38 to masscompile LV 8.0.1 on my DELL 610 LapTop with 1GB mem and 1.7 GHz processor

Regards Lennart

Ericsson Microwave Systems AB

Sweden

[Jeffrey Habets]

Thanx Jim,

Cool tool...

It choked on an error 43 in the Save:Instrument method though.

Did a second run where I filter out the error and display the file where the error occurs on and found that the error occured (Thus far, it is still running) on:

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FPEthernetBankConfiguration.vi and

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FieldPointEthernet.vi

[crelf]

It choked on:

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FPEthernetBankConfiguration.vi and

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FieldPointEthernet.vi

Try a normal LabVIEW mass compile on those VIs directly - you may find that they don't compile using that method either. I had the same problem (but with some personal VIs) and it turned out that they had insane objects in them.

[Mike Ashe]

Thanks Jim, great Utility and especially useful for me as my 8.0 machine is a no longer the young thoroughbred it once was.

[Jeffrey Habets]

Try a normal LabVIEW mass compile on those VIs directly - you may find that they don't compile using that method either. I had the same problem (but with some personal VIs) and it turned out that they had insane objects in them.

Yes, probably the case.. I got another error, invalid file type on a .vi. Probably a corrupt file, haven't checked it yet.

My suggestion would be to simply continue doing the mass-compile even if errors occur, so the process won't have to be started all over again.

[Jim Kring]

Version Update: Version 1.0.2 - Thursday, March 9, 2006 at 17:14:08 (UTC)

[NEW] Performance improvements of up to 50%, due to optimizations of code.

[Chris Davis]

Thanx Jim,

Cool tool...

It choked on an error 43 in the Save:Instrument method though.

Did a second run where I filter out the error and display the file where the error occurs on and found that the error occured (Thus far, it is still running) on:

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FPEthernetBankConfiguration.vi and

LabVIEW 8.0\resource\FrameWork\Providers\lvfp.llb\prefPage_FieldPointEthernet.vi

I had problems with this error too. I thought I had it setup to log the files, but apparently not, I just had it ignore that error. Perhaps Jim can change the tool to log which files had that error for everyone.

BTW, thanks Jim this has saved myself and the guys I work with lots of time!

Chris

[Norm Kirchner]

Any reason why theirs is so slow?

[ant]

I editted the tool to ignore error 1 and 43 as I had the same errors as above. My laptop has had a few knocks and as a result there are some VIs that are bad. I added an indicator to show the last VI the tool tried to edit and found the VI that kicked the program out was the same error that the NI mass compiler says is bad. So I'd suggest a tool edit to filter out 1, 43 and 4001 (as it exists now)

I'd post my edits but LV is now going through a mass compile now (lots of Tk's and modules so it is going to take a good while)

Ant

[Jim Kring]

Version Update: Version 1.0.3 - Thursday, March 9, 2006 at 20:56:32 (UTC)

[NEW] Continues through any mass compile errors and writes them to file (and opens it) after completion.

[Jeff B]

Please see here regarding 8.0.1 mass compile performance:

http://forums.ni.com/ni/board/message?boar...ssage.id=172705

And yes, of course the user VI for improved mass compile performance that I referenced is the one Jim Kring wrote. It's a cool VI, but also has some limitations, though I don't really know the best way to address them in that VI or what their consequences would be on the mass compile.

And FYI, the reason I didn't give specific credit in my post to Jim or link to this thread is mostly because he used a private VI server property, which is something we generally don't condone publishing un-password-protected. Still a cool VI though.

[Jim Kring]

Please see here regarding 8.0.1 mass compile performance:

http://forums.ni.com/ni/board/message?boar...ssage.id=172705

And yes, of course the user VI for improved mass compile performance that I referenced is the one Jim Kring wrote. It's a cool VI, but also has some limitations, though I don't really know the best way to address them in that VI or what their consequences would be on the mass compile.

And FYI, the reason I didn't give specific credit in my post to Jim or link to this thread is mostly because he used a private VI server property, which is something we generally don't condone publishing un-password-protected. Still a cool VI though.

Jeff: Thanks for info. Regarding this tool's limitations:

• Caching: The caching that I do is a sort of smart caching -- it assumes that it is safe to load all VIs that are located in the same folder, into memory at the same time. While this is not fool-proof (as there are some clever fools out there), it is a very safe assumption, IMO. Obviously, you can't have two files of the same name in the same folder, and while they might link to different files of the same name it is really unlikely and is a result of more serious problems.
  

• File types: Yes, this tool only handles VI's and CTL's. What other file types (extensions) do you recommend? I guess we could safely add VIT and CTT. And, maybe we want to to .flx (is that right?) for the FlexMotion mutants
  

[Guillaume Lessard]

File types: Yes, this tool only handles VI's and CTL's. What other file types (extensions) do you recommend? I guess we could safely add VIT and CTT. And, maybe we want to to .flx (is that right?) for the FlexMotion mutants

I can't think of why LV tries to recompile files regardless of extension... My user.lib is under subversion, and I was quite surprised to find out that LV had re-compiled the 'svnbase' files. What a wreck.

Your new mass compiler is much, much safer in that sense. I like!

[Jim Kring]

I can't think of why LV tries to recompile files regardless of extension... My user.lib is under subversion, and I was quite surprised to find out that LV had re-compiled the 'svnbase' files. What a wreck.

The built-in mass compiler is probably reading the binary to see if it is a known LabVIEW file type. Since the .svn files are VIs, the mass compiler is loading them.

[Jeff B]

Jeff: Thanks for info. Regarding this tool's limitations:

• Caching: The caching that I do is a sort of smart caching -- it assumes that it is safe to load all VIs that are located in the same folder, into memory at the same time. While this is not fool-proof (as there are some clever fools out there), it is a very safe assumption, IMO. Obviously, you can't have two files of the same name in the same folder, and while they might link to different files of the same name it is really unlikely and is a result of more serious problems.
  

• File types: Yes, this tool only handles VI's and CTL's. What other file types (extensions) do you recommend? I guess we could safely add VIT and CTT. And, maybe we want to to .flx (is that right?) for the FlexMotion mutants
  

This mass compile issue makes me think of another issue that has been on my mind for a long time. The built-in Mass Compile is a real pig. I believe that it loads each VI into memory, saves it, and unloads it -- one at a time. I can think of serveral ways to optomize this process. For example, after loading a VI into memory, save all of its subVIs, too. Keep track of which VIs that you've already saved, and do not load and save them again. I wonder how much this would speed up the process. For OpenG Commander, we built our own mass compiler that just loads the whole app into memory and saves all VIs in the hierarchy -- this saved loads of time.

So I'll go ahead and respond to this and an earlier post of yours on a different thread that I didn't have time to post anything on yesterday.

• Caching: The caching you do is quite similar to the way the built-in mass-compile's caching ends up working, albeit implemented in a somewhat different way. The built-in mass compile goes through VIs in a particular directory, and simply has a circular buffer of whatever size you specify of VIs that it will keep in memory. I don't know all the details about how subVIs are handled. For example, if you mass compiled a folder with 10 VIs, and each of those called 100 different subVIs, would that yield 1000 VIs in memory as the cache holds the 10 top-level VIs? Or would just 10 most recent top-level or subVIs with their dependencies be held in memory? I'm not sure, but I'd guess the former. Of course someone here knows, but I'm not sure it's that relevant.
  

• File types: VIT and CTT are a good start, but not particularly common as you were probably already thinking. I was thinking more of other types you can find in the "files of type" pull down on a file dialog in LabVIEW (on Windows at least). I think it would be best to also handle LVPROJ, LVLIB, XCTL, XNODE. However these aren't "instruments" that can be saved with the "Save Instrument" function, so it'd be a lot more work than just adding more types to your semi-colon-delimited list of file types. The good news is I don't *think* you take much performance penalty for loading those files non-mass-compiled like you do when loading large VI hierarchies of non-mass-compiled VIs, but I'm really not sure about this.
  

• I don't know how much this helps the end issue, but I wanted to explain a little bit about how the built-in mass compile works, as I don't think it's quite the swine you describe. The general mass compile algorithm is designed to be thorough and "as-safe-as-possible", with the built-in caching option added to improve performance in the face of the possible risk of cross-linkage dependning on what you're mass compiling. The built-in mass compile does load VIs on a VI-by-VI basis. Each time a VI is loaded, all of its subVIs are brought into memory as a result. For all VIs we just loaded, we see if they have a docmod (typically as a result of a recompile after version bump) and if so, we save them. We then close them all. Wash rinse and repeat on the next VI, etc. The longest parts of this process are the compile and save, which do not happen for VIs that have already been saved.
  

Take care,

Jeff

[Jim Kring]

Jeff,

> The built-in mass compile does load VIs on a VI-by-VI basis. Each time a VI is loaded, all of its subVIs are brought into memory as a result. For all VIs we just loaded, we see if they have a docmod (typically as a result of a recompile after version bump) and if so, we save them. We then close them all. Wash rinse and repeat on the next VI, etc.

Does the built-in mass compiler keep track of which VIs it has already saved (not just those that it has loaded, as top-level)?

> The longest parts of this process are the compile and save, which do not happen for VIs that have already been saved.

That is a very broad statement. It's probably true for vi.lib VIs that have shallow hierarchies, but probably less so for ./resource and ./project VI hierarchies.

-Jim

PS - Talk is cheap, show us the code.

[Jeff B]

Does the built-in mass compiler keep track of which VIs it has already saved (not just those that it has loaded, as top-level)?

> The longest parts of this process are the compile and save, which do not happen for VIs that have already been saved.

That is a very broad statement. It's probably true for vi.lib VIs that have shallow hierarchies, but probably less so for ./resource and ./project VI hierarchies.

The built-in mass compile doesn't keep track of what it has previously saved or what it has previously loaded as top-level. I'm not sure where you're going with this, do you have a follow-up question regarding this?

Yes, the fact that the sum of time spent compiling & saving a VI is more than that of just loading it once its already been saved in the current version is a general statement, but one that is true for just about any VI. I suppose there may be cases in which this is not true.

I've had no part in writing the code behind the built-in mass compile code (which is not written in G), and it's not my goal to try to convince you or even suggest that the built-in mass compile is in some way better than your VI approach. I originally just wanted to point out that there were differences with your approach that could be problematic in certain circumstances. Clearly this is the case with the built-in one as well - namely speed limitations, especially with no cache being used.

[Jim Kring]

The built-in mass compile doesn't keep track of what it has previously saved or what it has previously loaded as top-level. I'm not sure where you're going with this, do you have a follow-up question regarding this?

Yes, the fact that the sum of time spent compiling & saving a VI is more than that of just loading it once its already been saved in the current version is a general statement, but one that is true for just about any VI. I suppose there may be cases in which this is not true.

I've had no part in writing the code behind the built-in mass compile code (which is not written in G), and it's not my goal to try to convince you or even suggest that the built-in mass compile is in some way better than your VI approach. I originally just wanted to point out that there were differences with your approach that could be problematic in certain circumstances. Clearly this is the case with the built-in one as well - namely speed limitations, especially with no cache being used.

Thanks for answering my follow-up question(s).

> The built-in mass compile doesn't keep track of what it has previously saved or what it has previously loaded as top-level. I'm not sure where you're going with this, do you have a follow-up question regarding this?

My point about keeping track of work already done, is that there are several ways in which the built-in mass compiler can be improved, dramatically. I'm not trying to get into a pissing contest with anyone over which mass compiler is best -- I just want to work towards a mass compiling solution that is fully optomized and doesn't have any succeptibility to VI cross-linkeages. Currently the built-in mass compiler needs to be refactored and optomized -- maybe re-written in G. It sound like you have an interest in this too, so maybe you could champion that cause, from within NI.

> Yes, the fact that the sum of time spent compiling & saving a VI is more than that of just loading it once its already been saved in the current version is a general statement, but one that is true for just about any VI. I suppose there may be cases in which this is not true.

When the mass compiler is redundantly reloading large hierarchy into memory several times it wastes a huge amount of time, unnecessarily. I'm not arguing over whether compiling and saving takes more time than loading into memory, just communicating a major source of performance problems so that it can be noted and possibly improved.

Cheers,

-Jim

[Jim Kring]

Version Update:

Version 1.0.4 - Saturday, March 11, 2006 at 00:19:19 (UTC)

[NEW] Additionally, mass compiles *.vit (VI Template), *.ctt (Control Template), and *.flx (FlexMotion VIs) files.

[i2dx]

Jim, thank you for your tool!

I have posted the link to your tool in the german labviewforum.de and one user is complaining about errrors:

Error 1: Invoke Node in JKI Mass Compile__JKI_Mass_Compile.vi<APPEND>

Method Name: <b>Get VI Version</b>

The masscompile finished with about 100 (error 1)s in the log file ..

[Jim Kring]

Version Update:

Version 1.0.5 - Sunday, March 12, 2006 at 21:14:08 (UTC)

[FIX] Ignores any VI in memory with a Path property value of Not a Path, such as Express VI instances (facade VIs).

[NEW] Records the file path of any VI, CTL, etc. that causes an error.

Thank you to PJM_labview for making these modifications.

[Jim Kring]

Jim, thank you for your tool!

I have posted the link to your tool in the german labviewforum.de and one user is complaining about errors:

The masscompile finished with about 100 (error 1)s in the log file ..

You are very welcome.

Regarding the errors, please note that the latest version (1.0.5) should fix this problem. I believe that the errors did not have any impact on the mass compile -- but it might still be a good idea to mass compile again.

[i2dx]

You are very welcome.

Regarding the errors, please note that the latest version (1.0.5) should fix this problem. I believe that the errors did not have any impact on the mass compile -- but it might still be a good idea to mass compile again.

ok, thank you! I will forward this to the "labviewforum.de"