Open Package Building Discussion

[Rolf Kalbermatter]

On 9/21/2018 at 5:25 PM, ShaunR said:

If VIPM isn't doing it as we would like; we can always go back to its predecessor to build in what we need (Tell us about your enhancements, Rolf )

It's nothing to fancy. I added a few things to the UI to support more features and in preparation of adding the VI renamining/relinking step that was done seperately in the OpenG DEAB tool before calling the OpenG package builder. But I never got around to really add the deab part into the package builder. It's kind of extra difficult as the DEAB compononent doesn't currently support newer features like lvclass and lvlib at all and of course no mallable VIs etc.

I can post what I have somewhere, but don't get to excited.

[ShaunR]

23 hours ago, rolfk said:

It's nothing to fancy. I added a few things to the UI to support more features and in preparation of adding the VI renamining/relinking step that was done seperately in the OpenG DEAB tool before calling the OpenG package builder. But I never got around to really add the deab part into the package builder. It's kind of extra difficult as the DEAB compononent doesn't currently support newer features like lvclass and lvlib at all and of course no mallable VIs etc.

I can post what I have somewhere, but don't get to excited.

OpenG DEAB tool?

It is quite likely I already have code for many of the features required for a package manager. Certainly the installation side since I've had my own for a few years, some of which is in the SQLite API installer (kind of precursor). What it lacks is the builder side but much of what is required exists (at least in part) as individual scripting tools. However, I'm still reliant on the JKI PM for menu creation and applying licences.

For my purposes, I find the JKI package manager insufficient for my distribution of 32 & 64 bit binaries causing me to be heavily reliant on the pre/post install VI and carefully ensuring they aren't handled by the Package Manager. A useful feature for me would be to be the ability to define slightly different configurations for 32 & 64 bit LabVIEW at the individual file level.

 

Please do post what you have.

[Rolf Kalbermatter]

Well no menu editor in what I have. I find menus quite important to use but way to infrequent to create to have felt I would want to spend lots of time into this. I did add a feature to allow for 32 bit and 64 bit shared library support but as I do actually rely on VIPM to be able to install the created package, couldn't quite go all the way as I had wanted. The VIPM format is missing a specific flag that could work to flag files to be used in a specific bitness mode and the platform string as used in OpenG and hence VIPM is to much massaged to allow that distinction. My suggestion to add such a feature to VIPM was not really considered as it was considered to obscure.

Instead I had to rely on a Post Install VI that will then rename the according shared library to the required name and delete the unneeded one.

Basically the OpenG platform consisted of several tools.

First there was the Package Builder, which as already mentioned is a glorified ZIP file builder. OpenG packages and VIPM files too are in fact simply ZIP files with one or more specific files in the root, one of them being usally an INI type spec file specifying the contents of the package and the rules to use when installing it, another one being the package bitmap and optional license text. The rest are folders corresponding to file groups that need to be installed to the target system according to the rules in the spec file. The VIPM spec file contains a few additional optional items to support some of the later VIPM features.

Then there was first an OpenG Application Builder project hat eventually morphed into the DEAB tool which was the OpenG variant of an application builder. It also allows renaming of file hierarchies to create source distributions with file prefix or postfix additions before NI really supported such a feature. This was the tool used by the OpenG libraries to namespace the library files by adding the __ogtk postfix to each filename.

Creating an OpenG package simply meant running first the DEAB tool to rename/relink the source VIs into a new location that was meant to mirror the final VI hierarchy on the target system, then running the package builder task to turn this into an OpenG package.

It seems the OpenG SVN repository contains a new sub project Builder, that is another wrapper that tries to automate the execution of the DEAB task and the package builder in a very simple UI.

My attempt was to basically extend the old Package Builder UI to allow some additional configuration options that could be used to create the necessary data structures to run the DEAB tool programmatically before invoking the package step. But in comparison to VIPM this UI is more involved and for many users probably less intuitive.

The main GUI is found in "source/GUI/OpenG_Package_Builder.vi"

OpenG Package Builder V2.0.zip

Edited October 2, 2018 by rolfk

[hooovahh]

I do love and use the Pre and Post Build/Install/Uninstall VI features.  I know it would be best to have some features be native, but having the ability to hijack the build process to make it do custom things really adds so much flexibility.

[Rolf Kalbermatter]

Ohh I'm not saying that the pre/post build steps should be removed. Just that it shouldn't be necessary to use them to install 32 or 64 bit shared libraries depending what LabVIEW version you install into.

Edited October 2, 2018 by rolfk

[ShaunR]

18 hours ago, rolfk said:

The VIPM format is missing a specific flag that could work to flag files to be used in a specific bitness mode and the platform string as used in OpenG and hence VIPM is to much massaged to allow that distinction.

A few years ago, I created an installer with a view to replacing VIPM in its entirety Partly because of this and a few other features that I wanted, but mainly because VPIM broke compatibility with VIPM 2014. I got as far as an installer which had categories, 64/32 bit support and some other features such as code signing, which was based around SQLite databases. The ini file feature of the SQLite API for LabVIEW was created specifically so that I could ingest VIPM spec files for this installer to install them instead of VIPM. It could also build packages (a SQLite database file instead of an INI  file and a less sophisticated zip file) for its own format but I stopped short at the VIPM format as it was too restrictive (for the reasons we have discussed) and I didn't want to step on JKIs toes by releasing a competitive product. It has remained in my private toolbox ever since.

Now that we have two installers, the NI and JKI ones; I can envisage one manager doing both, which IMO the NI one should have done from the get-go. If that is the case, we can kind of ignore the VIPM package format for building (but still able to install and convert them) and subsume both package managers. If we decided that we wanted to support the VIPM build standard, then that is possible too, with the same caveats as VIPM as regards bitness etc or by extending the format. However. I don't see much utility in supporting the VIPM build but I do see a lot of the code in the OpenG Package builder being useful since there is nothing inherently wrong with the zipped file format used by VIPM for distribution.

Do you know anything about the NI Package Manager formats?

[hooovahh]

Having NIPM install VIPM packages seemed like great starting point...but that likely isn't going to happen.  NI really should have just bought JKI's IP, and turned it into NIPM, building off of what has worked well in current LabVIEW.  Oh and another thing to muddy up the waters, there is already a 3rd package manager aimed at LabVIEW, G Package Manager.  Haven't used it yet but it was presented on at the CLA Summit in a session I didn't get to see.  I wouldn't be surprised if all of these use a zip or some compression as their file format.

EDIT: NIPKG can be opened in 7-zip and is listed as a "Ar" Type, looking similar to a Debian package with gzipped tar balls.

[ShaunR]

@hooovahh: Thanks for moving the posts. No need to clutter the other thread with this.

1 hour ago, hooovahh said:

Oh and another thing to muddy up the waters, there is already a 3rd package manager aimed at LabVIEW, G Package Manager.  Haven't used it yet but it was presented on at the CLA Summit in a session I didn't get to see.  I wouldn't be surprised if all of these use a zip or some compression as their file format.

If that is the the cmd line one, it's of no interest to me. I can't imagine how you would use a command line to create a palette. I would also suspect it doesn't install existing VIPM and NIPM files and, if that is true, it is doomed.

Edited October 3, 2018 by ShaunR

[hooovahh]

Well G Package Manager appears to be more project oriented.  So install OpenG Array 1.0.0 for this project, but then be able to install OpenG Array 2.0.0 for a different project, or not have OpenG Array available at all in another project.  Lots of duplication but moving closer to sandboxing projects.  Because of this I'm unsure how palettes would be effected by VIs installed for a project and not the IDE.  But yes it does seem to be command line heavy, and not intended for installing VIPM or NIPM packages.

[smithd]

NIPM just uses the opkg format, like their linux distribution. The limitation there is that its geared towards system installs, which is why you have to point it at the specific labview version (eg in NIPM they might have NI Scope support for labview 2015, NI scope support for labview 2016, etc..).

GPM is as you said geared towards a more sane project-oriented development workflow vs a global code repo. 

 

[ShaunR]

5 hours ago, smithd said:

NIPM just uses the opkg format

That's useful. IIRC, zlib supports gzip and the control files look straight forward enough.

[Rolf Kalbermatter]

Well, what I have posted is really mostly just to install libraries and geared towards application specific installs rather than project specific ones. So it is not quite the tool you guys are looking for for sure. It was developed at a time, LabVIEW would not have projects for around 10 more years, so that is at least an excuse. :-)

I was toying at some point with the idea to simply just call the DEAB tool after massaging the various paths and settings into the necessary format to perform the necessary steps, but the fact that DEAB is really predating many of the newer features after LabVIEW 8.0 made me hesitate and eventually other things got in the way.

As to the VIPM format it's pretty much the same as ogpm with a few additions for things like the license manager and a few other niceties. At least a few of them I even added to the OGPM version I posted. As it is an INI file it is really trivial to add more things, only the installer component needs to support them too, and I really did not feel like building that part too.

I only briefly looked at NIPM. I was expecting to see something which supports the current NI installers for Windows for all other platforms, having been led to believe that it should be eventually used for that, as well as a full featured VIPM replacement on steroids. I found it to be a half baked replacement for the installers and way to lacking to even support basic VIPM features. Of course being fully aware that creating a good package manager and installer is a project that many have attempted and only few really got to a decent working thing, and I'm not just talking LabVIEW here, I wasn't really to surprised .

I would be interested to have something better but have to admit that my motivation to create such a thing is not high enough to do the work for it

[ShaunR]

3 hours ago, rolfk said:

Well, what I have posted is really mostly just to install libraries and geared towards application specific installs rather than project specific ones. So it is not quite the tool you guys are looking for for sure. It was developed at a time, LabVIEW would not have projects for around 10 more years, so that is at least an excuse. :-)

I have thought about this before (project based, sandboxed).

I work on projects on a different drive to LabVIEW which isn't a problem when working on individual toolkits; as they are self contained. But it got much harder once I started needing the Encryption Compendium in everything I always needed to have it installed in the LabVIEW vi.lib directory and any time I updated it had to reinstall for the other toolkits to amke sure they still worked as expected. So I played around with symlinks placed in the vi.lib directory which worked pretty well and could keep the menus in the IDE. Just replacing the symlink to different versions was also OK and worked but I couldn't figure out a way to stop some VIs recompiling (so just lived with it).

Thinking off-the-cuff. We could probably have an IDE helper to select different VI.lib toolkit versions on-the-fly.

Edited October 4, 2018 by ShaunR

[Francois Normandin]

21 minutes ago, ShaunR said:

... I always needed to have it installed in the LabVIEW vi.lib directory and any time I updated it had to reinstall for the other toolkits.

Shaun, can you elaborate on why you needed to reinstall the other toolkits when you update your encryption library? If the project-specific librairies all pointed to vi.lib, why did their linkage break?

Edited October 4, 2018 by Francois Normandin

[ShaunR]

7 minutes ago, Francois Normandin said:

Shaun, can you elaborate on why you needed to reinstall the other toolkits when you update your encryption library? If the project-specific librairies all pointed to vi.lib, why did their linkage break?

I didn't need to update the other toolkits. I had to keep reinstalling the new Encryption Compendium if the other toolkits broke because of a change (until I had fixed it.)

[Francois Normandin]

got it.

[ShaunR]

41 minutes ago, ShaunR said:

Thinking off-the-cuff. We could probably have an IDE helper to select different VI.lib toolkit versions on-the-fly.

In fact. It could probably read the current project dependencies (from the install database) and switch the symlinks to the ones required. Hmmmm.

Edited October 4, 2018 by ShaunR

[Rolf Kalbermatter]

 

12 hours ago, ShaunR said:

In fact. It could probably read the current project dependencies (from the install database) and switch the symlinks to the ones required. Hmmmm.

Might be the incentive to finally finish the LVZIP library update. I have file helper code in there that supports symlink support for Linux and Windows (and likely Mac OS X though haven't really looked at that yet). The main reason was that I want to support the possibility to add symlinks as real symlinks into the ZIP archive rather than pulling in the original file as the LabVIEW file functions will do. There are some semantical challenges such as what to do with symlinks that are not pointing to other files inside the archive but something entirely different and how to reliably detect this without a time consuming prescan of the entire hierarchy.

But the symlink functionality to detect if a path is a symlink, to read the target it points to and to create a symlink is basically present in the shared library component. Under Windows there is the additional difficulty about how to handle shortcuts. Currently I have some code in there that simply tries to treat them like symlinks but that is not really correct as a shortcut is a file in its own with additional resources such as icons etc. that get lost in this way, on the other hand it is the only way to keep it platform independent except to choose to drop shortcut support entirely.

Edited October 5, 2018 by rolfk

[ShaunR]

1 hour ago, rolfk said:

 

Might be the incentive to finally finish the LVZIP library update. I have file helper code in there that supports symlink support for Linux and Windows (and likely Mac OS X though haven't really looked at that yet). The main reason was that I want to support the possibility to add symlinks as real symlinks into the ZIP archive rather than pulling in the original file as the LabVIEW file functions will do. There are some semantical challenges such as what to do with symlinks that are not pointing to other files inside the archive but something entirely different and how to reliably detect this without a time consuming prescan of the entire hierarchy.

But the symlink functionality to detect if a path is a symlink, to read the target it points to and to create a symlink is basically present in the shared library component. Under Windows there is the additional difficulty about how to handle shortcuts. Currently I have some code in there that simply tries to treat them like symlinks but that is not really correct as a shortcut is a file in its own with additional resources such as icons etc. that get lost in this way, on the other hand it is the only way to keep it platform independent except to choose to drop shortcut support entirely.

That sounds extremely useful. When building the encryption Compendium, I have to create dummy dll files for the OpenSSL binaries then delete them and create symlinks in the Post Install VI (NI refused to put them in the LabVIEW search path). I have a VI for creating symlinks which would suffice for an IDE helper but build support would be great.

As regards the shortcuts. While nice to have for completeness, I don't think it should hold up a release. I would be tempted to leave it out entirely and see who complains and what they expect to happen. Then make a judgement call for the behaviour a in later release, if at all.

Edited October 5, 2018 by ShaunR

[ShaunR]

On 10/5/2018 at 0:16 PM, rolfk said:

There are some semantical challenges such as what to do with symlinks that are not pointing to other files inside the archive but something entirely different and how to reliably detect this without a time consuming prescan of the entire hierarchy.

For the use case I outlined earlier (OpenSSL) the ideal behaviour would be to include the symlink and the file.

Reasoning:

When I distribute the library, the symlinks generally link to existing OpenSSL binaries in the 'shared' folder. These pretty much always exist and just a symlink would suffice. However. For the general case, a binary may not already exist or, if it does, if may be an older version; the latter of which also applies to my use case. By including both the symkink and the file itself, the application has the option to write out the file if it does not exist or update the file to the newer version (after backing up the original file of course).

Caveat:

This use case is fairly straight forward on Windows. However. If the external binary is, say, a Linux binary such as libstd.so.6 then we are in trouble with this naive process. Anecdotally. LabVIEW doesn't seem to 'see through' symlink chains consistently on Linux. For example. I found It is not enough to link to libstd.so. Instead one must link to the binary libstd.so.6 the suffix of which which can be distro dependant.

Maybe I'm just not understanding a fundamental characteristic of Linux symlinks, and it has been a while since I had to confront these quirks after dropping its support. But could you to expand on your thoughts of how to handle symlinks to external binaries in a platform agnostic way?.

Edited October 7, 2018 by ShaunR

[Rolf Kalbermatter]

17 hours ago, ShaunR said:

This use case is fairly straight forward on Windows. However. If the external binary is, say, a Linux binary such as libstd.so.6 then we are in trouble with this naive process. Anecdotally. LabVIEW doesn't seem to 'see through' symlink chains consistently on Linux. For example. I found It is not enough to link to libstd.so. Instead one must link to the binary libstd.so.6 the suffix of which which can be distro dependant.

Maybe I'm just not understanding a fundamental characteristic of Linux symlinks, and it has been a while since I had to confront these quirks after dropping its support. But could you to expand on your thoughts of how to handle symlinks to external binaries in a platform agnostic way?.

While I have seen some difficulties to get a shared library recognized in Linux (or NI Linux RT) in a way that it could be loaded in a LabVIEW application through the shared library node, I can't really imagine how the symlink on Linux could fail in itself if it points to the valid file, even if it is through intermediate symlinks. The Linux libc library handles the symlink resolution transparently through the kernel unless an application uses specifically the link aware variants of the API functions.

e.g. calling stat() on a link will return the file information of the final target file even across multiple symlins and if you want to know the details about the symlink itself you have to call lstat() instead, with lstat() returning the same as stat() for non symlink filepaths. I don't think LabVIEW does something special when not specifically trying to find out if a path is really a link so it should not matter if a symlink is redirected over multiple symlinks. What could matter is the access rights for intermediate symlinks which might break the ability to see the final file.

[ShaunR]

14 hours ago, rolfk said:

What could matter is the access rights for intermediate symlinks which might break the ability to see the final file.

Possibly. This is why "Real men run as root"

Anyhoo. I think any attempt to create a [universal] builder/installer would be best envisaged on Windows first to prove the concept, so these kinds of quirks could be ironed out later. The success of such an endeavour would (I think) be very dependant on symlinks as they are kind of like a silver bullet to allowing different library versions and LabVIEW versions of the libraries to exist side-by-side and swap between.

The conclusion I came to with my investigation was that a separate repository (in "Program Data") of installed toolkits, under each LabVIEW and toolkit version, enabled very quick and easy uninstall and version change by simply creating and deleting symlinks in the vi.lib directory pointing to them. Once that is achieved, then project based installs become a simple matter of choosing which links are present for each project whilst still maintaining menus in the palettes.

Edited October 8, 2018 by ShaunR

[hooovahh]

On 10/8/2018 at 6:03 AM, ShaunR said:

The conclusion I came to with my investigation was that a separate repository (in "Program Data") of installed toolkits, under each LabVIEW and toolkit version, enabled very quick and easy uninstall and version change by simply creating and deleting symlinks in the vi.lib directory pointing to them. Once that is achieved, then project based installs become a simple matter of choosing which links are present for each project whilst still maintaining menus in the palettes.

Oh I like this.  Have a single place where all packages, and versions of packages get installed (extracted), and then switch in the package and version "Installed" by setting up the symlinks.  I could see arguments that these links should link to under vi.lib, or user.lib or wherever, or I could see arguments for them being in folders relative to the project.  I think having to uninstall and reinstall packages (delete and recreate links) between projects isn't too big of a deal and would reduce the likelihood of cross linking issues where one developer opens it and if finds the reuse library in a different place than expected from another developer.

[ShaunR]

2 hours ago, hooovahh said:

I could see arguments that these links should link to under vi.lib, or user.lib or wherever, or I could see arguments for them being in folders relative to the project. 

You want the symlink to be in the LabVIEW search path (vi.lib, User.lib etc) to force LabVIEW to also search the other locations transparently, if required. This is another reason why it is usually a good idea to have sandboxed projects to ensure LabVIEW uses relative paths instead of absolute paths when linking (projects and lvlibs).

By the way. You can do all this manually now, just by moving the toolkits and creating the links. Although I'm not sure, off-hand, what the different flavours of SCC do with symlinks. Then all you'd need is a simple IDE helper to make it trivial choose which versions you wanted.....It only takes ms to delete and create a symlink.

Edited October 9, 2018 by ShaunR

[Rolf Kalbermatter]

I definitely like the idea to use symlinks to map specific libraries from a fixed location into LabVIEW installations and/or projects. The only problem I see with that is that if you work with multiple LabVIEW versions, as they will keep wanting to recompile the libraries and if you don't write-protect them, you sooner or later will end up with "upgraded" libraries that then fail to load into another project that uses an older LabVIEW version. Other than that it's probably one of the best solutions in managing libraries for different projects on a demand base. 

If you want to get really fancy you could even create a project provider that allows to setup project specific library reference specifications that then can be applied with a single right click in the LabVIEW project 

Edited October 10, 2018 by rolfk