Open Package Building Discussion

[ShaunR]

12 minutes ago, rolfk said:

The only problem I see with that is that if you work with multiple LabVIEW versions, as they will keep wanting to recompile the libraries and if you don't write-protect them, you sooner or later will end up with "upgraded" libraries that then fail to load into another project that uses an older LabVIEW version.

This is why I suggested

On 10/8/2018 at 10:03 AM, ShaunR said:

under each LabVIEW and toolkit version

Each LabVIEW version has its own installed toolkits and links to the repo are pre-pended with the LabVIEW version

e.g.

C:\ProgramData\_LV_Repo\9.0.1\Encryption Compendium for LabVIEW\1.0.0

[ShaunR]

27 minutes ago, rolfk said:

If you want to get really fancy you could even create a project provider that allows to setup project specific library reference specifications that then can be applied with a single right click in the LabVIEW project 

That's some LabVIEW voodoo I have intentionally steered well clear of

I would actually prefer a completely automated (no click) solution for users but with the ability to modify the project links for maintainers.

[Rolf Kalbermatter]

On 10/10/2018 at 0:56 PM, ShaunR said:

That's some LabVIEW voodoo I have intentionally steered well clear of

I would actually prefer a completely automated (no click) solution for users but with the ability to modify the project links for maintainers.

My idea was a project provider for the specific file type such that once you right click on the package file you can "install" it either globally for the LabVIEW system in question or locally in a project specific location. The idea would be that the files are stored in a hidden location for the current LabVIEW version and bitness and then symlinks are added in the necessary locations to make the package directories available to the LabVIEW or project in the correct locations.

Edited October 17, 2018 by rolfk

[ShaunR]

9 hours ago, rolfk said:

or project in the correct locations

I think this would cause issues.

The packages would already be stored "globally" with links in the vi.lib. Two reasons why the vi.lib.

1. It's on the Labview search path.
2. Menus are auto-populated in addons.

Putting the library in a project directory effectively puts the library "out of view" for LabVIEW and you could potentially have different version in the vi.lib and in various projects. The result of the latter would mean linking would boil down to whatever was opened first - from the palettes or from any one of the projects. I think we'd be back to square one.

 There would be nothing to stop people from adding a symlink in the project directory pointing to the symlink in vi.lib which, I think, would achieve the same goal, but with only one, static, repository. Is that what you are suggesting?

Edited October 17, 2018 by ShaunR

[ShaunR]

OK. So what sort of features would we want in a new package manager?

Here are a coulple that I've long hoped for:

1. The ability to export a package dependancy list (Just the names and versions? Or more than that?).
2. The ability to export licences of a package and all its dependancies.
3. Automatic detection of 32 and 64 bit binaries.
4. Support installation of VIPM, OGPM, LVPM, executables (like setup.exe) and zipped packages.
5. Import from public online repositories (like Github, Bitbucket etc).

What else?

[Rolf Kalbermatter]

I think it sums it more or less up. I would NOT bother about license installation at this time. The NI 3rd party license manager app is pretty outdated and Windows only, so not a good solution anyways. And anyone recently interested to have that working can always add some batch file script or custom post install hook to do that anyways if needed.

There definitely should be a way to support custom repositories in some way, preferably through some kind of plugin interface to make it flexible for different services.

FYI: I'm making some progress on the new ZIP ibrary with transparent support for (sym)links. Still need to figure out some issues but it starts working. Soon there will need to be some testing done to make sure it works on more than just my own systems :-).

Edited October 30, 2018 by rolfk

[ShaunR]

4 hours ago, rolfk said:

I think it sums it more or less up. I would NOT bother about license installation at this time. The NI 3rd party license manager app is pretty outdated and Windows only, so not a good solution anyways. And anyone recently interested to have that working can always add some batch file script or custom post install hook to do that anyways if needed.

Well. Having to use TPLAT I have various tools for, how should I say, un-cocking its cock-ups

No compromises. Either it installs stuff properly with a click or two or it will be relegated to "just another installer that doesn't really do what I need"

4 hours ago, rolfk said:

FYI: I'm making some progress on the new ZIP ibrary with transparent support for (sym)links. Still need to figure out some issues but it starts working. Soon there will need to be some testing done to make sure it works on more than just my own systems :-).

Sweet. What licence is it?

(Please don't say, GPL, please don't say GPL, please don't say GPL)

Edited October 30, 2018 by ShaunR

[ZwodderZ]

2 hours ago, ShaunR said:

Well. Having to use TPLAT I have various tools for, how should I say, un-cocking its cock-ups

No compromises. Either it installs stuff properly with a click or two or it will be relegated to "just another installer that doesn't really do what I need"

Sweet. What licence is it?

(Please don't say, GPL, please don't say GPL, please don't say GPL)

Yeah I would too like to know. What license is it ?

[smithd]

17 hours ago, ShaunR said:

OK. So what sort of features would we want in a new package manager?

Here are a coulple that I've long hoped for:

1. The ability to export a package dependancy list (Just the names and versions? Or more than that?).
2. The ability to export licences of a package and all its dependancies.
3. Automatic detection of 32 and 64 bit binaries.
4. Support installation of VIPM, OGPM, LVPM, executables (like setup.exe) and zipped packages.
5. Import from public online repositories (like Github, Bitbucket etc).

What else?

Having spent this morning trying to deal with vipm I have two additional suggested requirements:

1-Dont require a GB of RAM to unzip a few files

2-Dont dick around with the windows API every few seconds in an attempt to get the user's attention 

3-while performing long running operations, don't queue up 10M instances of a 'refresh' message so when the long running operation completes your user is still sitting there for five minutes watching the screen refresh.

Edited October 30, 2018 by smithd

[ShaunR]

48 minutes ago, smithd said:

Having spent this morning trying to deal with vipm I have two additional suggested requirements:

1-Dont require a GB of RAM to unzip a few files

2-Dont dick around with the windows API every few seconds in an attempt to get the user's attention 

3-while performing long running operations, don't queue up 10M instances of a 'refresh' message so when the long running operation completes your user is still sitting there for five minutes watching the screen refresh.

You forgot the /rant tag

[smithd]

9 hours ago, ShaunR said:

You forgot the /rant tag

Indeed

[Rolf Kalbermatter]

On 10/30/2018 at 4:19 PM, ShaunR said:

Sweet. What licence is it?

It's simply an update to the OpenG ZIP library. As such the LabVIEW VIs are BSD licensed and the shared library code is LGPL. Shouldn't matter for anyone unless you want to modify the shared library somehow yourself. If someone has a problem with that please elaborate what is the problem but don't tell me "I don't like anything that sounds like GPL" 

The shared libary really is not meant to be modified by others and in the 20 years that that library exists I have not received one single patch, fix or other suggestion for that part. 

If your lawayer claims that a LGPL licensed shared library component is not possible in a commercial app or whatever let them think again. They are talking out of their neck.

Edited October 31, 2018 by rolfk

[Michael Aivaliotis]

Using the concept of links. In Windows there are hard links and symbolic links. How does this difference affect the behavior of this proposed solution? Also, creating these links required Admin rights. Of course this seems somewhat expected in the world of installers.

[ShaunR]

1 hour ago, rolfk said:

It's simply an update to the OpenG ZIP library. As such the LabVIEW VIs are BSD licensed and the shared library code is LGPL. Shouldn't matter for anyone unless you want to modify the shared library somehow yourself. If someone has a problem with that please elaborate what is the problem but don't tell me "I don't like anything that sounds like GPL" 

The shared libary really is not meant to be modified by others and in the 20 years that that library exists I have not received one single patch, fix or other suggestion for that part. 

If your lawayer claims that a LGPL licensed shared library component is not possible in a commercial app or whatever let them think again. They are talking out of their neck.

I was specifically talking about GPL but some of it probably applies to LGPL as some people (like Centos) have re-licenced to older versions due to "Licence compatibility issues".....so here we go

1. Noone can decide if dynamic linking is a derivitive work or not.
2. The burden of making available source code for a few years.
3. Noone can decide what derivative licences are compatible, not comptible or if any code monkey, like me, is in violation.
4. It's unclear, to me, what happens when a customer makes their application; what burdens are imposed on them. As I undertand it, they have the same distribution burdens that I would have (which they might not be aware of) and that could open them up to a legal threat.

The problem with GPL is that everyone (including lawyers) argue about what does and doesn't constitute a violation. It is the uncertainty that means it is unusable, especially in cases where the original licences (like ZLib) are so permissive and are not surrounded by these complications.

Edited October 31, 2018 by ShaunR

[ShaunR]

1 hour ago, Michael Aivaliotis said:

Using the concept of links. In Windows there are hard links and symbolic links. How does this difference affect the behavior of this proposed solution? Also, creating these links required Admin rights. Of course this seems somewhat expected in the world of installers.

Practically speaking. If you delete a hard-link, you delete the target. If you delete a soft-link, you only delete the link. There's a little more to it than that, but not much. Windows file APIs are link-aware. The in-built LabVIEW ones are not.

[Rolf Kalbermatter]

1 hour ago, Michael Aivaliotis said:

Using the concept of links. In Windows there are hard links and symbolic links. How does this difference affect the behavior of this proposed solution? Also, creating these links required Admin rights. Of course this seems somewhat expected in the world of installers.

Actually creating symlinks are as of the latest Windows 10 build not anymore a privileged operation.

Then you have Junction Points, Hardlinks and Soflinks. In total Microsoft tradition they made it unneccessary complicated that one only works for directories while the other only works for files. There are some implications. Symlinks like under Unix are not verified to point to a valid location, so they can be created for non-existing files and directories as well as end up pointing into nirvana when the actual file or directory is deleted.

Junction Points only work for directories and must be absolute and point to a local vollume. They can specifically not point to remote locations.

Hardlinks only work for files and they are simply addiional directory entries pointing to the actual file content in the filesystem. In fact every directory entry in an NTFS volume is a hardlink, so each file has at least one hardlink entry on a NTFS volume. Once the last hardlink for a file is deleted the file itself gets deleted from the filesystem.

Personally I have for now decided to support reading either of them (except hardlinks as every file entry that is not one of the above reparse point types is in fact a hardlink so there is no way to decide if a file that has multiple hardlinks pointing to it, which would be possible to detect, should be treated as link or as actual file) and storing them as a link entry in the ZIP file and on restoring them to create at least under Windows always symlinks.

[Rolf Kalbermatter]

27 minutes ago, ShaunR said:

Practically speaking. If you delete a hard-link, you delete the target. If you delete a soft-link, you only delete the link. There's a little more to it than that, but not much. Windows file APIs are link-aware. The in-built LabVIEW ones are not.

Actually only if you delete the last occurrence of a hardlink pointing to that file. The file object itself maintains information about how many hardlinks exist for it. The built-in LabVIEW file APIs also simply use the Win32 APIs so saying that Windows APIs are link aware and LabVIEW APIs are not is a bit misleading.

However LabVIEW path handling has some particularies that always resolves the path to the actual file even if the original path was pointing to any kind of link including the good old Windows 95 shortcut and even over multiple redirections. The problem lays therein that LabVIEW functions always operate on the actual file or directory entry rather than the link or shortcut. So I had to also implement file directory listing and other file functions in order to be able to operate on the actual link file rather than the target of it.

[Michael Aivaliotis]

So if LabVIEW is not trying to manipulate the linked files (delete, rename, etc) and is just reading or saving them, then symbolic link should be enough. However if LabVIEW tries try to manage these files then it's better to go with hard links. In this context I mean the LabVIEW IDE, not the package manager written in LabVIEW.

But I think the two link types might not matter to the package manager written in LabVIEW. Since it will always be working with the installed files in the Application Data folder. All of the "work" on the files would be done there and not under the LabVIEW IDE folders. Some work to unzip the files and some other work to create the links which would be a bunch of Admin-level command line calls.

Am I understanding this right?

@rolfk As far as the zip changes your making. Explain the use-case of supporting "hard" or "soft" links in the first place. How do you use this in the context of zip?

Edited October 31, 2018 by Michael Aivaliotis

[Michael Aivaliotis]

49 minutes ago, ShaunR said:

I was specifically talking about GPL

This is not GPL.

JKI figured out a way with VIPM. So it can be done.

[Rolf Kalbermatter]

2 hours ago, Michael Aivaliotis said:

So if LabVIEW is not trying to manipulate the linked files (delete, rename, etc) and is just reading or saving them, then symbolic link should be enough. However if LabVIEW tries try to manage these files then it's better to go with hard links. In this context I mean the LabVIEW IDE, not the package manager written in LabVIEW.

But I think the two link types might not matter to the package manager written in LabVIEW. Since it will always be working with the installed files in the Application Data folder. All of the "work" on the files would be done there and not under the LabVIEW IDE folders. Some work to unzip the files and some other work to create the links which would be a bunch of Admin-level command line calls.

Am I understanding this right?

@rolfk As far as the zip changes your making. Explain the use-case of supporting "hard" or "soft" links in the first place. How do you use this in the context of zip?

The main reason that I want to support syminks comes from the fact that under Linux when installing shared libraries you normally are creating various symlinks all pointing to the actual shared library in order to have a versioning scheme. Without support for symlinks in the package itself you have to do some involved Münchhausen trick by using post install hooks to create them through the command line, which is also OS and even OS version specific sometimes. Also a shared library on OS X is in fact a whole directory structure with various symlinks pointing to each other for the resource and version information and the actual binary object file. Without support for this you have to zip the shared library package up, add the zip file to the OpenG package and on install unzip it again to the right location.

Support for links under Windows was mostly just a fun to have addition to make the symlink functionality work on all LabVIEW platforms but the practical benefit under Windows is fairly limited in terms of support for install packages. And in hindsight the effort to implement it under Windows was pretty heavy. But it does allow to test the library also under Windows without special exceptions.

[ShaunR]

8 hours ago, Michael Aivaliotis said:

This is not GPL.

JKI figured out a way with VIPM. So it can be done.

Have they? The zlib library isn't GPL or LGPL and the Vis are BSD. I assume that Rolf is intending to make an intermediary library to be LGPL and that means compliance with, now, 3 licences - the most onorus being LGPL.

8 hours ago, Michael Aivaliotis said:

But I think the two link types might not matter to the package manager written in LabVIEW. Since it will always be working with the installed files in the Application Data folder. All of the "work" on the files would be done there and not under the LabVIEW IDE folders.

That's not my vision and would argue that it would be no better than under vi.lib. I think we both see the value of install but for uninstall you don't want to touch the target packages - just delete or recreate the links.

[Rolf Kalbermatter]

1 hour ago, ShaunR said:

Have they? The zlib library isn't GPL or LGPL and the Vis are BSD. I assume that Rolf is intending to make an intermediary library to be LGPL and that means compliance with, now, 3 licences - the most onorus being LGPL.

No. All my submissions to OpenG have originally been licensed as LGPL just as everything else in OpenG. Jim Kring wanted at some point to relicense all OpenG libraries to BSD, because of community feedback and maybe also becuase he already envisioned something like VIPM to be created. He contacted every submitter and we discussed it and I recognized the difficulties about LabVIEW VIs having to be build into an application in such a way that they could be dynamically linked to the application in order to comply with the LGPL if you did not want to release the application itself as (L)GPL.

As a result I decided to relicense the VI part of all libraries under the BSD license just as all the other submitters agreed to. Since the shared library in those projects (there are other libraries such as the Port IO, LabPython, Pipe IO) all are  naturally dynamically linked to any application using them, it did not seem necessary to change that license too, so I left that as LGPL.

And as long as you do not make any modifications to the shared library, the fact that the source code is freely downloadable from the sourceforge OpenG project site really should satisfy any obligation any user of that library might have to make the source code available to any other potential user. I consider that a more long term safe option than what many Alliance Members could guarantee if they would host the source code themselves somewhere, since Alliance Members have known to go out of business in the past, and then nobody will honor source code requests from potential users of such an application. And yes I consider that part of the LGPL in fact the real benefit, the fact that any user of an application with that library in there has the right to see the source code for that library, together with the fact that if someone modifies the shared library and publishes an application of it he is obligated to make the modifications available to the community. If someone does not like that the solution is simple, don't modify the library!

If someone wants to use the LGPL as a means to not use a software library because of perceived problems to comply with it, he is free to do so. I don't see the problem and don't intend to change my opinion about that. In my opinion it is more often than not just a legalize excuse for the "not invented here" syndrome or some variant of that. My understanding is that if you add a statement to the fact that the application includes OpenG libraries and a link to the sourceforge project site, either in a readme file in the installation directory, the application documentation or the About dialog in your application, you really did everything necessary to comply with the LGPL license for the shared library part. And that statement needs to be there somewhere even for the BSD licensed code! Even if I wanted to, which I don't, I can not remove the code from the sourceforge site anymore. I could delete it from the repository yes, but it is still there in the form of the entire SubVersion history for anyone to grab and publish somewhere else if he or she decided so.

If there are specific situations where someone wants to discuss a different license for whatever reason with me for any of these libraries, I'm willing to discuss that. But as a community submission it will remain as it is for the foreseeable future.

Edited November 1, 2018 by rolfk

[Rolf Kalbermatter]

I see your point about lack of proper documentation of the LGPL used for the shared library part. It is contained in the source but as I have never been involved with the actual packaging itself so far it got somehow swept under the floor. I will make sure to remedy that in the next release by adding an according LICENSE file to the package that states the actual situation. And I will also add a link to the OpenG Sourceforge project site to that file, so anyone adding that file to his application will be in full compliance then  .

Including the entire build environment for all possible targets (11 last time I counted) really is not an option. That simply would totally explode the package. I don't even have them all on one machine. Some are in Virtual machines that I only connect specifically for compiling and testing these libraries.

Edited November 1, 2018 by rolfk

[ShaunR]

4 minutes ago, rolfk said:

I see your point about lack of proper documentation of the LGPL used for the shared library part. It is contained in the source but as I have never been involved with the actual packaging itself so far it got somehow swept under the floor. I will make sure to remedy that in the next release by adding an according LICENSE file to the package that states the actual situation. And I will also add a link to the OpenG Sourceforge project site to that file, so anyone adding that file to his application will be in full compliance then  .

You're not liable, though. JKI is. That is my point. You gonna sue?

Edited November 1, 2018 by ShaunR

[Rolf Kalbermatter]

10 minutes ago, ShaunR said:

You're not liable, though. JKI is. That is my point. You gonna sue?

Should I contact the Software Freedom Conservancy? 

Considering the recent discussions about GNU Licensing in the kernel and the Linux Maintainers wanting rather to help people to comply than getting them before court, I would say JKI is fairly safe. 

I totally agree with their sentiment. (L)GPL is not a means to sue people but to protect software to remain open source rather than being inappropriately claimed by big companies for their own evil plans. 

Edited November 1, 2018 by rolfk