Splash screens and dynamic loading

[JamesMc86]

The example is the sample project for the actor framework. Open the new create project wizard from the splash screen and load either the actor framework template project or the evaporative cooler project (an example of how to use the template) and run these up. You will see the splash screen that AQ was referring to.

[hooovahh]

So I had an idea for a progress bar loading splash screen with a cancel option.  It's an idea that others have done before but I modified slightly.

 

There is a splash screen VI that is very small and then loads the main VI dynamically.  But how mine differs is I am able to have a progress bar loading each of the Actors, before loading the Main VI.  If you load the Main VI in the splash screen the UI will be locked until that Open Reference operation is complete because it needs to load the entire VI hierarchy.  So instead I will open a reference to each Actor Core, which is the majority of the VIs each actor will use.  During this a progress bar can be shown displaying which actor is being loaded and a cancel which would send a stop to the for loop loading each actor.  If cancel isn't pressed then after opening each actor, we then open the Main VI which takes much less time to load because each actor is already loaded.

 

This all works great but you may ask your self "How does the splash screen know what actors exist and need to be loaded without causing them to be loaded into memory already."  The only solution I came up with, is in my Build Specification I call a VI before making the EXE.  This VI will search my project for actor cores, and then write to the applications config INI file saying what the actor names are, and where in the EXE they can be found.  Then the loading splash screen reads this INI file and uses its information for the UI.

 

This all works and in practice loads quicker than just loading the Main VI dynamically, for some reason.  I tried making an example of what I described using the Actor Frame work but couldn't make it work due to my very VERY limited Actor Frame work experience.  I just used the term actor in this post so others would know what I was talking about.  But in general this could work for any producer consumer engine architecture that is broken up into sub components.  Heck this could work with anything with classes in it and just load each class before loading the Main.  Or even more generic what if we got a list of all VIs greater than a certain file size (say 50KB) then load each of them before loading the main.

Edited April 23, 2013 by hooovahh

[ShaunR]

This all works great but you may ask your self "How does the splash screen know what actors exist and need to be loaded without causing them to be loaded into memory already."

Nope Because I do it the same way

One page ago

 

The only real question is, how do you detect that the UI is initialised and ready to show since everything being loaded, doesn't necessarily mean the main FP can be shown (I use a queue or notifier for this, presumably you use an actress)?

[hooovahh]

Nope Because I do it the same way

One page ago

 

The only real question is, how do you detect that the UI is initialised and ready to show since everything being loaded, doesn't necessarily mean the main FP can be shown (I use a queue or notifier for this, presumably you use an actress)?

DO'H.  Well it's been a while since I read this whole thread so I missed that.

 

Regarding how do I know my FP is ready to be shown?  Well my Main.VI isn't the main UI you see when you run, it goes like this.

 

Launcher

- Main VI

- - UI Actor

- - DAQ Actor

- - Logging Actor

- - Other Actors

 

My Launcher is as I described, a splash screen with a progress bar, cancel button, and opening actors one at a time then opening Main VI.  Main VI has some init stuff to do (read config files) then it calls all Actors to start.  Each Actor then has its own init operation (including UI Actor).  So my Main VI sits in a loop telling the UI Actor to Show Window.  It won't be able to do this until all init has been completed.  If Main VI doesn't hear the reply from the UI Actor:Show Window within a short timeout, it will wait then call the Show Window again.  When it finally does get the reply from the UI Actor, Main VI hides it self and UI Actor shows its self.

 

In this way my Launcher is a splash screen loading the VI hierarchy, and Main is a splash screen (without progress bar and cancel) that has the same front panel as Launcher.  This has a nice effect of a progress bar increasing then when it is full it disappears, and soon after the UI Actor is shown.  There are several flaws with this but after initial debug it works well.

 

My question back at you is how do you do you know where in the EXE your VIs are so you can open a reference to them in the Launcher?  Through trial and error I noticed that the path to the file within the EXE was the same as it was on disk from the project file to the Actor Core.  I wasn't sure if this was always true, and I wasn't sure where files that aren't relative to the project file are stored.  In source I have this structure.

 

C:ProjectsExample.lvproj

C:ProjectsActor 1Actor Core.vi

 

In the EXE it appears to be this

 

<dir>My Example Program.exe

<dir>My Example Program.exeActor 1Actor Core.vi

 

What about files not relative to the project like vi.lib files?  Where are they stored in the EXE and are they subject to change?

[ShaunR]

and soon after the UI Actor is shown.

Indeed. The splash implementation I use loads the UI last which is only visible once all inits are completed and we are ready to rock 'n roll. It waits 2 secs then disappears. This means no matter how long or short the UI "show" is, the splash always hangs around for at least 2 secs (no quick blip and it's gone), the UI is always visible before it disappears and the application is fully active, configured and initialised.

 

My question back at you is how do you do you know where in the EXE your VIs are so you can open a reference to them in the Launcher?  Through trial and error I noticed that the path to the file within the EXE was the same as it was on disk from the project file to the Actor Core.  I wasn't sure if this was always true, and I wasn't sure where files that aren't relative to the project file are stored.  In source I have this structure.

 

C:ProjectsExample.lvproj

C:ProjectsActor 1Actor Core.vi

 

In the EXE it appears to be this

 

<dir>My Example Program.exe

<dir>My Example Program.exeActor 1Actor Core.vi

 

What about files not relative to the project like vi.lib files?  Where are they stored in the EXE and are they subject to change?

Well. I've never had to explicitly load vi.lib files. If they are used, they get loaded when the top level VI is loaded. But the rest of what you are saying is correct. I have a get path VI that augments the path (just a conditional structure) depending whether it is in the application or in the IDE.

I also generally have one or two directories off of the App dir and usually, they're named Core and Plugin so that I can use the dir name in the launcher window-e.g. "Loading Plugin: xyz") . The loader also has the capability to just look in a directory and load any files in there so it loads plugins dynamically as well.

A major difference is maybe that although the launcher loads the VIs. It doesn't necessarily execute them (that can be a flag in the cfg file but I prefer a specific naming convention). Therefore some VIs (modules required to be loaded but require ordered initialisation in concert with other modules) just get loaded. The launcher just obtains a ref that causes LV to load the modules hierarchy and then closes it once the main app is up and running and opened it's own. This means that the launcher only gets it in memory ready for the main app. It also has a sub-panel so things like checking connectivity can be shown in the dialogue for fancy additional indicators (scanning for RS485 devices for example).

Edited April 24, 2013 by ShaunR

[JamesMc86]

My question back at you is how do you do you know where in the EXE your VIs are so you can open a reference to them in the Launcher?  Through trial and error I noticed that the path to the file within the EXE was the same as it was on disk from the project file to the Actor Core.  I wasn't sure if this was always true, and I wasn't sure where files that aren't relative to the project file are stored.  In source I have this structure.

 

C:ProjectsExample.lvproj

C:ProjectsActor 1Actor Core.vi

 

In the EXE it appears to be this

 

<dir>My Example Program.exe

<dir>My Example Program.exeActor 1Actor Core.vi

 

What about files not relative to the project like vi.lib files?  Where are they stored in the EXE and are they subject to change?

 

I would suggest an ideas exchange to add the internal structure to the exe preview and you would have my kudos!

 

In 2009 you apparently could use the openG zip tools but that doesn't seem to work any more. That was on Jim Krings blog that showed that technique. I haven't found another one yet unless someone would like to surprise me 

[Ulf]

DO'H.  Well it's been a while since I read this whole thread so I missed that.

 

Regarding how do I know my FP is ready to be shown?  Well my Main.VI isn't the main UI you see when you run, it goes like this.

 

Launcher

- Main VI

- - UI Actor

- - DAQ Actor

- - Logging Actor

- - Other Actors

 

My Launcher is as I described, a splash screen with a progress bar, cancel button, and opening actors one at a time then opening Main VI.  Main VI has some init stuff to do (read config files) then it calls all Actors to start.  Each Actor then has its own init operation (including UI Actor).  So my Main VI sits in a loop telling the UI Actor to Show Window.  It won't be able to do this until all init has been completed.  If Main VI doesn't hear the reply from the UI Actor:Show Window within a short timeout, it will wait then call the Show Window again.  When it finally does get the reply from the UI Actor, Main VI hides it self and UI Actor shows its self.

 

In this way my Launcher is a splash screen loading the VI hierarchy, and Main is a splash screen (without progress bar and cancel) that has the same front panel as Launcher.  This has a nice effect of a progress bar increasing then when it is full it disappears, and soon after the UI Actor is shown.  There are several flaws with this but after initial debug it works well.

 

My question back at you is how do you do you know where in the EXE your VIs are so you can open a reference to them in the Launcher?  Through trial and error I noticed that the path to the file within the EXE was the same as it was on disk from the project file to the Actor Core.  I wasn't sure if this was always true, and I wasn't sure where files that aren't relative to the project file are stored.  In source I have this structure.

 

C:ProjectsExample.lvproj

C:ProjectsActor 1Actor Core.vi

 

In the EXE it appears to be this

 

<dir>My Example Program.exe

<dir>My Example Program.exeActor 1Actor Core.vi

 

What about files not relative to the project like vi.lib files?  Where are they stored in the EXE and are they subject to change?

 

You can always put a help VI in the same folder as the class you want to load. The help VI should not be part of the class and should only return the class path. When you create your splash loader, you just drop the all help VIs that you have created and execute them one-by one to get the class-path.

[hooovahh]

I haven't found another one yet unless someone would like to surprise me

This is all in the name of understanding right? I don't have it working but I wanted to share what I had to see if someone else could figure any more out.

So you can extract a newer LabVIEW EXE like Jim did and get the contents of it (I tested with 2011 EXE). However the contents may not be as useful as you'd like. There are alot of neat things like icon files, dialog information, version information etc. All of this isn't a big secret and can be viewed using Resource Hacker. You'll notice under the RCData there are a few sections and one of them should be much larger than the others. This the the main contents of your EXE and contains all the code needed to run (VIs, Controls, classes, and other files). In my two examples the large section was under "2" under the "RCDATA".

Looking at this 2 file I found a few block sections. Near the end there is the RSRC section and it mentions the "FLAG" block and "LVzp" block. The Flag block for me contained one byte that was 00 for the two EXEs I tested. LVzp contained the large majority of the "2" file and I believe this is the secret to the file's information. The LVzp block started with the same for the 2 EXEs I tested which is the following:

 

70 5A 6B D6 B4 55 AB 56 AD 5B

or in ASCII

 

pZkÖ´U«V

Now this is where I'm stuck. I'm guessing this pZk block is a compressed format of some kind but can't figure out what. Searching google for this header as a file returns nothing. I believe if we can decompress this we may have the file structure of the EXE listing out its contents.

I also did find some EXE_Miner on NI's site but it again only works with older versions of LabVIEW before this compression was done.

Edited April 24, 2013 by hooovahh

[Yair]

In source I have this structure.

 

C:ProjectsExample.lvproj

C:ProjectsActor 1Actor Core.vi

 

In the EXE it appears to be this

 

<dir>My Example Program.exe

<dir>My Example Program.exeActor 1Actor Core.vi

 

What about files not relative to the project like vi.lib files?  Where are they stored in the EXE and are they subject to change?

 

I never looked into this too deeply, but it seems to me like the structure is simple enough - the top level in the EXE is the project (or maybe if there are other files in other folders it's not the top level) and everything else is relative to that exactly like it was in source. vi.lib and other such files are saved under a folder called 1abvi3w, if memory serves. All this should be easy enough to test manually by getting a reference to a few VIs inside the app and displaying or logging their path property (or you could just write a VI which will parse the entire hierarchy and go through the data manually).

 

For a more generic solution to your current method, what you can do is statically analyze the entire hierarchy (assuming you have a simple hierarchy with only a few top level VIs) and then list it in reverse and load all the VIs in the EXE in that order (i.e. from bottom to top). I think there was a thread last year which talked about this and maybe even had some examples (I have a vague recollection of Norm participating there, so you can go through his stuff).

[hooovahh]

I never looked into this too deeply, but it seems to me like the structure is simple enough - the top level in the EXE is the project (or maybe if there are other files in other folders it's not the top level) and everything else is relative to that exactly like it was in source. vi.lib and other such files are saved under a folder called 1abvi3w, if memory serves. All this should be easy enough to test manually by getting a reference to a few VIs inside the app and displaying or logging their path property (or you could just write a VI which will parse the entire hierarchy and go through the data manually).

 

For a more generic solution to your current method, what you can do is statically analyze the entire hierarchy (assuming you have a simple hierarchy with only a few top level VIs) and then list it in reverse and load all the VIs in the EXE in that order (i.e. from bottom to top). I think there was a thread last year which talked about this and maybe even had some examples (I have a vague recollection of Norm participating there, so you can go through his stuff).

Regarding 1abvi3w, I have seen this folder being generated temporarily when building an EXE, so I think this is probably correct and I can do some testing to see.  

 

I wasn't intending on loading all VIs one at a time then loading the top (from bottom to top) I thought that would be too time consuming.  Really I think what would work best is maybe to load the 20 largest VIs (by file size I guess) then load the main VI.  This way I don't have to load all 2000-5000 VIs, my goal would be to load just enough of them so that the progress bar can keep moving.  Maybe the 20 largest VIs will load 70% of the total VIs (because they would load dependencies) and then loading the Main would use the 70% already loaded, and just load the extra 30% it needs.  Just a thought I haven't fully tested any of this.

[mje]

Maybe the 20 largest VIs will load 70% of the total VIs (because they would load dependencies) and then loading the Main would use the 70% already loaded, and just load the extra 30% it needs.  Just a thought I haven't fully tested any of this.

 

I wouldn't expect VI file size to be a good measure of the size of the dependency tree.

[ShaunR]

Regarding 1abvi3w, I have seen this folder being generated temporarily when building an EXE, so I think this is probably correct and I can do some testing to see.  

 

I wasn't intending on loading all VIs one at a time then loading the top (from bottom to top) I thought that would be too time consuming.  Really I think what would work best is maybe to load the 20 largest VIs (by file size I guess) then load the main VI.  This way I don't have to load all 2000-5000 VIs, my goal would be to load just enough of them so that the progress bar can keep moving.  Maybe the 20 largest VIs will load 70% of the total VIs (because they would load dependencies) and then loading the Main would use the 70% already loaded, and just load the extra 30% it needs.  Just a thought I haven't fully tested any of this.

 

Don't forget that the user doesn't give two hoots about what is actually in your splash screen. They just want something that shows them that when they launched the app it did indeed launch and is doing something rather than hung. Everything else is just for your benefit. If all hierarchy is loaded with main then just load main and put an animated GIF on the splash FP. The purpose of a splash is a lightweight, fast loading placebo to keep the user occupied; nothing more.

My apps are generally partitioned into sub-systems which are dynamically loaded so, apart from the UI, it just loads those then finally the UI (UI can be anywhere from 80% to 20% of the entire hierarchy....it doesn't really matter). All the "loading Plugin" is purely for me to see what I've missed or what has failed without waiting for everything. In the old days we used to just have a "VI Tree" which we would launch and the modern splash is just an extension of that. The only difference is that we can get the names and the number of those VIs and log them (if we want to).

You've done the hard part (making the splash generic and not app specific). I really wouldn't spend a lot of time worrying about how it loads your application or whether your progress bar is linear or not.

Edited April 26, 2013 by ShaunR

[hooovahh]

My apps are generally partitioned into sub-systems which are dynamically loaded so, apart from the UI, it just loads those then finally the UI (UI can be anywhere from 80% to 20% of the entire hierarchy....it doesn't really matter). All the "loading Plugin" is purely for me to see what I've missed or what has failed without waiting for everything. In the old days we used to just have a "VI Tree" which we would launch and the modern splash is just an extension of that. The only difference is that we can get the names and the number of those VIs and log them (if we want to).

Mine are too, I was just trying to make it more generic.  So that this loading splash screen with progress could be applied to applications where this dependency on sub-system architecture wasn't required.  I guess I'm satisfied with it as long as new applications also adhere to this design pattern.

[GregFreeman]

I heard an interesting idea at the CLA Summit, although I haven't tried it myself so I don't know how well it will work. Each time they launch the app they monitor change in windows memory usage from start to end and log it to a file. Then each time the application is opened, the memory usage is monitored and compared to this value from the previous time the application was run in order to calculate percentage loaded. I think this is what was done anyways, but I may be a bit off base. I supposed if you had some other process effecting your memory at the same time it could cause skewed results. But, it was an interesting concept nonetheless. 

[hooovahh]

I heard an interesting idea at the CLA Summit, although I haven't tried it myself so I don't know how well it will work. Each time they launch the app they monitor change in windows memory usage from start to end and log it to a file. Then each time the application is opened, the memory usage is monitored and compared to this value from the previous time the application was run in order to calculate percentage loaded. I think this is what was done anyways, but I may be a bit off base. I supposed if you had some other process effecting your memory at the same time it could cause skewed results. But, it was an interesting concept nonetheless. 

I like the concept but in practice you may find it difficult to update the UI while the main.vi is being loaded.  I've noticed that when I do a Open VI Reference the UI is locked and doesn't update (something about thread swapping).  So I don't know how you would update a progress bar with the new memory loaded percentage.  The reason the the Sub-System loading works, is because there isn't one Open VI Reference call, there is one for each sub-system and then one for the main, and between each of these calls there is a time where the UI can be updated.  If you can get an example working using this memory usage method I would be very interested in how it works.

[Yair]

I like the concept but in practice you may find it difficult to update the UI while the main.vi is being loaded.  I've noticed that when I do a Open VI Reference the UI is locked and doesn't update (something about thread swapping).  So I don't know how you would update a progress bar with the new memory loaded percentage.  The reason the the Sub-System loading works, is because there isn't one Open VI Reference call, there is one for each sub-system and then one for the main, and between each of these calls there is a time where the UI can be updated.  If you can get an example working using this memory usage method I would be very interested in how it works.

 

What was actually described there was a second EXE which functions as the splash screen. That second EXE saves what the RAM usage of the real EXE was after loading and then simply monitors RAM usage as the real EXE is loading and displays that as a percentage of the saved value. I'm assuming that should give a reasonably accurate answer, assuming a clean load of the app uses more-or-less the same amount of RAM (which may not actually be the case if you close the app and reopen, etc.).

[hooovahh]

What was actually described there was a second EXE which functions as the splash screen. That second EXE saves what the RAM usage of the real EXE was after loading and then simply monitors RAM usage as the real EXE is loading and displays that as a percentage of the saved value. I'm assuming that should give a reasonably accurate answer, assuming a clean load of the app uses more-or-less the same amount of RAM (which may not actually be the case if you close the app and reopen, etc.).

Oh sorry I misunderstood.  In that case yes that would work decently well but probably not perfect.  Even so any of these suggestions are better then what we have now, when a Main.vi is the startup VI and takes upwards of 20 seconds before the user even sees anything indicating the program is running.

Edited May 2, 2013 by hooovahh

[joseosorio28]

<blockquote class='ipsBlockquote'data-author="bmoyer" data-cid="84262" data-time="1300363670"><p>

<br />

I agree, this method is the way to go. All of the files are in the same exe too so it's more contained then having 2 exes.<br />

<br />

I use a VI called Startup.vi that I use to dynamically call the other VIs (designated as a "Startup VIs" in the build). In the "Always Included" section I include the Splash Screen.VI and my main program VI.<br />

<br />

Bruce</p></blockquote>

[hooovahh]

Who wants to resurrect a post  .  I have a white board in my cube and posting this example has been on my list for a while.  Attached is an example of this progress bar loading splash screen I tried to describe a year ago.  Nothing too special but I use something similar to this in my production applications.

 

Build the Normal EXE and you'll end up with a 6MB file that when ran shows no UI for about 3-5 seconds.  To make it worst add the 8th actor to the main VI along with the other 7.  I removed it because it took way too long to build an EXE.  The total VIs in memory with it added was around 4500.  If you are missing any VIs feel free to delete them, or that entire actor, it is only meant to bloat up the number of VIs in memory and they are never even ran.

 

Then build the With Progress version and you still get the 6MB file but the UI is shown within a second showing the progress of each actor being loaded into memory with an option to cancel.  Then once each actor is loaded it loads the main and runs it.

 

For this to work the Launcher is the top level VI and Main is set to always include.  And there is a Pre-Build action that gets a list of the actors, and their relative path to the VI to load and writes it to the config file used in the EXE.  This Pre-Build action is more elaborate in my production applications because not all actors sit in a folder called "Actors".  Instead it opens the project and navigates based on library names, and connector panes but the idea is the same.

EXE Loading Progress.zip

[Mefistotelis]

A lot of time passed. Many new tools are available. Possibilities have evolved.

On 4/24/2013 at 4:03 PM, hooovahh said:

The LVzp block started with the same for the 2 EXEs I tested which is the following:

Quote

70 5A 6B D6 B4 55 AB 56 AD 5B

What if I told you that it is still the same ZIP format?

The LabView Runtime Library must be extracting this somehow, right? So it shouldn't be hard to find, if you know the chunk ID is 'LVzp' (possibly backwards due to endianness) and there's zip extractor around (unzip from zlib, version from circa 2012).

Find my Github projects for details.

[Rolf Kalbermatter]

15 hours ago, Mefistotelis said:

What if I told you that it is still the same ZIP format?

Of course it is. They changed the PK0x030x04 identifier that is in the first four bytes of a ZIP stream, since when they did it with the original identifier, there was a loud scream through the community that it was very easy to steal the IP contained in a LabVIEW execuable. And yes it was easy as most ZIP unarchivers have a habit of scanning a file for this PK header, no matter where it is in a file and if they do and the local directory structure following it makes sense they will simply open the embedded ZIP archive. This is because many generators for self extracting archives simply tacked an executable stub in front of a ZIP archive to make it work as an executable.

The screaming about stealing IP was IMHO totally out of proportions, the VIs in an executable have no diagram, no icon and usually not even a front panel (unless they are set to show their frontpanel at some point). But NI listened and simply changed the local directory header for the embedded ZIP stream and all was well 😆.

The ZIP functions available in LabVIEW are a byproduct of integrating the minizip and zlib sources into LabVIEW for the purpose of compressing binary data structures inside of VIs to make the VIs smaller and of using a ZIP archive in executables rather than the old  <=8.0 LLB format used. The need to change away from the embedded LLB was mainly because with the introduction of classes and lvlibs, the VI names alone where not always unique and therefore couldn't be stored in the single level LLB anymore. They needed a hierarchical archive format and rather than extending the LLB format to support subdirectories, it was much easier to use the ZIP archive format and the ZLIB provided sources came with a liberal enough license to do that.

[Mefistotelis]

Thanks for the background info, that's good to know.

 

The ZIP format is chunked with recurring headers, so changing one might've not been enough. That's probably why the whole ZIP is xor'ed.

This is actually quite poor design - the xor goes single-byte at a time, with key depending on previous results. 8-bit ops are slow on newer CPUs (compared to 32 or 64), plus it makes it impossible to use multiple threads. This makes the decryption unnecessarily slow.

Anyone beyond student grade would now consider dividing the archive into blocks, and decrypting these blocks separately, on different CPU cores. But I guess back when the algorithm was created, that might not have been so obvious.

 

For the VIs having some blocks removed from final EXE - it actually isn't possible to secure environment such as LabView completely. Just a matter of someone having enough free time. Though the amount of time required might be really considerable here..

(not that this is wrong strategy on NIs side - after all, all the modern security algorithms are based on long time required to bypass)