VIPM LAVA Repo

[Porter]

Wouldn't it be nice to have a public LAVA repository for VI package manager?

I'm not sure how to make this happen. Just throwing the idea out there.

[hooovahh]

That is a good idea, and I suppose we could.  All we'd really need is a Windows based server running somewhere with a professional version of VIPM, serving up to a publicly accessible URL.  That being said I'm fairly certain this site isn't ran on anything that would allow us to do that here.  And then there is the main restriction I can think of which is that only those with VIPM Pro will be able to add this LAVA repository.  The free version of VIPM doesn't allow for adding repositories to monitor.  I still don't have a good picture on the VIPM Pro penetration in the community and wouldn't want to go through the effort of doing this if there are only a few who would find it useful.

I've decided to open a poll to gauge interests but having never done that before I'm not certain I did it right...

[Porter]

The projects that I post on LAVA are typically developed on my spare time. They are solutions to problems that I encounter at the office but don't have time to work on during office hours. I post them here because this community is home to some of the most competent LabVIEW architects. What better place to get a decent LabVIEW code review? The result is very good code but not quite polished enough to pass the tools network requirements. The extra time and effort to bring the project through the tools network certification process is just not worth it for me. I'm not making a product for sale. It would be nice though to share these projects with a wider audience.

Perhaps in a later version of VIPM, JKI could consider adding support for the LAVA repository (if/when it exists) in their free version. Maybe have an enable checkbox hidden away in a config window. Once enabled, they can display a big warning about how code from LAVA doesn't go through the same certification process as JKI and LabVIEW tools network.

Edited August 15, 2018 by Porter

[drjdpowell]

We'd have to deal somehow with projects that are in the CR and on the Tools Network.  Personally, I sometimes have a newer version in the CR (and update the Tools Network less often when the changes have been more real-word tested).

[Porter]

I think that when a package is hosted in multiple repos, VIPM displays the entry from the most recent package version. So if you release a new version on the LAVA repo, it will show up as an update for the LVTN version. This is definitely not what we want to do.

A solution could be to make separate LAVA and LVTN packages. For the LAVA package, you could append _LAVA to the product name and have the LVTN package listed as incompatible. For the LVTN package, have the LAVA package listed as incompatible.

[jacobson]

What exactly would be put in this repo? Would the LAVA CR essentially become this repo? Also, would there be any gatekeeping process?

I mostly want to understand what the quality of code I could expect from the repo would be. If anyone could submit anything they wanted I would probably not ever take a look.

[Porter]

The idea would be to include the LAVA CR except for non-vip packages and the uncertified section.

On LAVA, we used to submit our code to the uncertified section first. Then after some discussion, testing, debugging and polishing, we would report to the moderator that our code is ready to be certified and placed in the appropriate category in the CR. I'm not sure if this is being enforced anymore.

The how-to is from 2009: 

 

Edited August 15, 2018 by Porter

[hooovahh]

For the sake of discussion I envisioned it would be done like Porter said.  We already have a process for uncertified and certified code.  I would just intend on taking all the packages made that are in the certified section and put them into a VIPM repo.  We could think about making the uncertified code into another repo, but I'm not sure how often someone making uncertified code would go through the trouble of making a package.  I mean part of the point of uncertified code is it is semi-unfinished, or an alpha state, so spending the time to setup a palette and build a package could be someone not done until it becomes certified.

[jacobson]

24 minutes ago, hooovahh said:

I mean part of the point of uncertified code is it is semi-unfinished, or an alpha state, so spending the time to setup a palette and build a package could be someone not done until it becomes certified.

If it's something meant to go in the Tools menu, a QD shortcut, or right-click menu then you might want to make a package. Other than that I would agree that the menus and packaging of an API would be the last thing I would do.

[Francois Normandin]

This is a good idea, but I think it would require a more thorough editorial review of submissions to put them on the such a repository.
Other than the obvious code reviews, "LAVAG" would need to examine for possible license issues, consistency across versions, develop more precise guidelines for submissions, etc.

The hardest part would be to maintain the same standards over the years, as people's commitment to their "review committee duties" will undoubtedly fluctuate over time.

As for the actual repository, you don't need a server running. The package publisher needs VIPM Pro to modify and manage the destination folder. The content of the repository is stored as files on any accessible drive, so it's a matter of adding it to the list of network URLs. (requires VIPM Pro as well, as mentioned above, unless on LVTN or JKI Package Network)

Edited August 17, 2018 by Francois Normandin

[hooovahh]

1 hour ago, Francois Normandin said:

This is a good idea, but I think it would require a more thorough editorial review of submissions to put them on the such a repository.
Other than the obvious code reviews, "LAVAG" would need to examine for possible license issues, consistency across versions, develop more precise guidelines for submissions, etc.

Isn't this already handled by VIPM anyway?  You include the license, and version constrictions in the package.  When you upload code to LAVAG.org you are already authorizing LAVA to store and distribute your code.

[Bryan]

Judging by the poll results thus far, it looks like there is a LOT of interest in having a LAVA code repository for those that are not using VIPM Pro.  That's to be expected though.

If a public repository were to be set up somewhere, I'm assuming that JKI would have the option of adding it to the list of canned repositories for the Pro and Free versions if they so choose.

Edited September 24, 2018 by Bryan

[Francois Normandin]

5 hours ago, hooovahh said:

Isn't this already handled by VIPM anyway?  You include the license, and version constrictions in the package.  When you upload code to LAVAG.org you are already authorizing LAVA to store and distribute your code.

I meant that LAVAG should verify that the code it hosts does not infringe on licenses, or at least have a process to summarily verify.

You are right that VIPM handles the actual license from the user's perspective.

Edited August 17, 2018 by Francois Normandin

[Rolf Kalbermatter]

I don't think it can be LAVA's responsibility to verify licenses infringement, copyright and whatever issues of such code. Obviously the code review process should take care about glaring problems, but it can not be the idea that LAVA could and should be responsible about guaranteeing that no such issues exist. If we require that we can stop right now because we can't do that, even if we hire several full time lawyers .

By submitting code for the CR each submitter basically needs to state that he is providing the code according to the the license requirements that he selected and that to the best of his knowledge there are no copyright or license issues. Anything more than that is simply not workable.

As to providing a LAVA repository I don't think we would need a special OS/hardware system other than some internet accessible FTP/HTTP storage, which I'm sure the existing LAVA server could do, except for the additional server space that might require. I'm not sure about the exact internal workings of VIPM and it seems that even packages released on the OpenG Sourceforge site don't automatically are picked up, though it shouldn't be very difficult to do that. (Yes I did some package update in the last year and posted it to the OpenG repository but it still doesn't show up in VIPM). It seems that even for that repository there is some manual step involved from JKI to add a package to some list that VIPM then will be able to see. or maybe I did something wrong when posting the package on sourceforge.

If JKI would then add this LAVA repository as another default repository besides the Tools Networks and OpenG sourceforge repository, we would be all set even without owning a Pro license of VIPM. And I might be even persuaded to get our company to commit to a few VIPM Pro licenses then .

[Michael Aivaliotis]

I've only just seen this thread.

So the mechanism for creating a repo is simple and the current LAVA server can do this no problem. The issue is file size which would would eat up disk space and is what costs money. However, there are ways around this. For example, as @rolfk pointed out, the OpenG packages are served up from sourceforge but distributed from the JKI repo. So workarounds are possible. I don't want to get into that right now as this is not the main issue.

There are 2 main VIPM Pro bottlenecks that have to be addressed for a LAVA repo to become a reality. First is the Repo management. This needs a Pro license. However, since this function only needs to be done be a handful of people and infrequently, it could be acceptable if LAVA purchases a license and assigns some administrators for this. Second bottleneck, and the biggest hurdle, is that in order to subscribe to a repo you need a Pro license. This is the main thing preventing widespread adoption of the VIPM repo model. If JKI removed this barrier, then it would make sense to not only have a LAVA repo, but a number of repos. That would mean that anyone with a free license of VIPM could connect to any repo and get the tools they need from within VIPM, from the repo they care about. Companies could create internal repos and use that to distribute code to their developers. Or special repos for their customers. ?The possibilities are endless ?.

Ok, back down to reality.

Maybe we can use NIPM? I was browsing this content and it seems NIPM allows public feeds:

https://forums.ni.com/t5/NI-Package-Management/Introduction-to-NIPM-Feeds/ta-p/3808155

What's the issue with NIPM exactly?

[hooovahh]

13 minutes ago, Michael Aivaliotis said:

What's the issue with NIPM exactly?

How much time do you have?  Packages are version specific at the moment (so a String package needs a new release for every version of LabVIEW supported, String 2017 1.0.0, String 2018 1.0.0, etc).  There is no palette editing so you'll need to edit your own palette, make your own MNU files and include them.  There are no Pre/Post Build actions, and in fact all Post/Pre actions are EXEs with no option for calling VIs.  Only Post-Install and Pre-Uninstall actions are allowed.  Package dependencies seem to only be checked between the current feed.  So unless I'm mistaken all external dependencies need to be copied to the new feed with all the dependencies brought over.  No dependency scanning.  No package of packages.   No offline single file with everything included (but an installer might make that possible in the future).  Packages can't be loose VIs, they must be in something like an EXE, or source distribution first.

Basically NI has been saying that NIPM at the moment should be thought of as an installer alternative, and using it for reuse distribution is going to be very difficult.  That can change in the future but for now if you want to deploy an application to multiple machines, Skyline can do that because NI Packages are like little installers.  Until NI makes NIPM act more like VIPM, reuse and toolkit distribution using NIPM will be much more difficult.

[Michael Aivaliotis]

Assuming all you claim is true as of this writing. Then yes, I can see several things that are limitations. Let me look at each one:

1. Palette editing. You can edit palettes in LabVIEW and grab the menu file. I know, it would suck, but not a deal-breaker.
2. Package release for every LabVIEW version. Wow, this seems like NI took the one built-in feature of LabVIEW (automatic up-conversion) and threw it out the window. Way to go NI?
3. Pre-Post build actions. I would have to look into this further. I thought NIPM had a command-line interface (or at least this is what I was told). You could build a wrapper around that and create a VIPM-like experience, could you not? Then include your pre-bost build steps? Perhaps include palette editing (see item 1).
4. Only Post-Install and Pre-Uninstall actions are allowed. - Ya, more flexibility is needed. Not a show-stopper though.
5. Package dependencies seem to only be checked between the current feed. - That could either be a fundamental flaw, or a benefit. Not sure. However, it would be nice to have the choice. In VIPM there is no line drawn on this, and it's sometimes useful to limit yourself to a certain repo. Seems Ni took the safe approach.
6. No dependency scanning. - This is a useful feature, but I'm not sure in what context you are referring. Do you mean, during package building, to make sure the dependancies are declared? Manually declaring dependencies might be ok for now.
7. No package of packages.- Meh. Nice to have but there are other possible solutions to the same problem.
8. No offline single file with everything included - VIPM doesn't have this, unless you are referring to VIPC files.
9. Packages can't be loose VIs, they must be in something like an EXE, or source distribution first. - Well, VIPM creates a source distribution behind the scenes when it builds a package. But I get your point. NIPM could hide this complexity. - Not a showstopper though.

I think it's a combination of all of these issues that make NIPM a bad choice right now for packaging VIs.

However, If I imagined a world where VIPM did not exist and all I had was the current version of NIPM. And I was 15 years younger... I would probably figure out how to create a VIPM-like interface to the NIPM core. Fixing all the inadequacies. Not saying it's possible, but I would be working on that right now. Because it would be a fun thing to do. After all, it's how VIPM was built in the first place.

The core of it though is that NI is betting on this horse and is putting resources into it. I suggest we guide it in the right direction and make sure it's a winner and does what we want it to do. If not, then we should figure out a way to work with it, augment it and fix it. The more we do that, the more NI will make it better and modify it to remove limitations. - sigh, at least I hope.

Check out the NIPM API

[smithd]

1 hour ago, Michael Aivaliotis said:

 

1.  
2. Package release for every LabVIEW version. Wow, this seems like NI took the one built-in feature of LabVIEW (automatic up-conversion) and threw it out the window. Way to go NI?
3. Pre-Post build actions. I would have to look into this further. I thought NIPM had a command-line interface (or at least this is what I was told). You could build a wrapper around that and create a VIPM-like experience, could you not? Then include your pre-bost build steps? Perhaps include palette editing (see item 1).
4.  
5.  
6. No package of packages.- Meh. Nice to have but there are other possible solutions to the same problem.
7.  
8. Packages can't be loose VIs, they must be in something like an EXE, or source distribution first. - Well, VIPM creates a source distribution behind the scenes when it builds a package. But I get your point. NIPM could hide this complexity. - Not a showstopper though.

(2) I think I've mentioned this elsewhere, but its more accurate to say that NIPM has no knowledge of labview. Its simply a system-level package manager, akin to opkg, apt-get, chocolatey, etc. You can make the same source distribution in say, Labview 2015, but since its an OS-level package manager, you have to make a different package for each of Labview 2015, 2016, ..., 2015 x64, ...

One comment on that, is if you look at the NI installers, they often just make one package and install the VIs anyway. I only have 2017 installed, and yet I have a labview 2014, 2015, and 2016 folder for whatever reason. So nothing stopping a more lazy package developer to make a single package that deploys the same source distribution to 12 different directories.

(3) I believe NIPM can technically do this through a custom XML file, pointing at a simple batch or at cmd.exe: http://www.ni.com/documentation/en/ni-package-manager/latest/manual/instructions-xml-file-packages/#GUID-29BE2213-93C1-4281-8570-7CE1338AEB4A

http://www.ni.com/documentation/en/ni-package-manager/latest/manual/assemble-file-package/

(6) Yeah, an otherwise empty package with dependencies is the traditional fix for this

(8) Actually this is the part I like better about the NIPM build process -- its much more explicit and it seems easier to me to figure out when stuff goes wrong.

 

 

Also, a good landing page for NIPM is: https://forums.ni.com/t5/NI-Package-Management/NI-Package-Management-Portal-READ-THIS-FIRST/ta-p/3805952

Edited November 16, 2018 by smithd

[Michael Aivaliotis]

45 minutes ago, smithd said:

Also, a good landing page for NIPM is: https://forums.ni.com/t5/NI-Package-Management/NI-Package-Management-Portal-READ-THIS-FIRST/ta-p/3805952

Ya, I saw that and am planning to read through that to be better knowledgeable on what I'm complaining about. But for starters, there are 3 questions on that doc and nobody from NI has responded. Stuff like that irritates me.

There's no NIPM forum though. None that I could find anyway.

[TomOrr0W]

3 hours ago, Michael Aivaliotis said:

Second bottleneck, and the biggest hurdle, is that in order to subscribe to a repo you need a Pro license.

Has JKI ever mentioned why they chose to have this restriction?

I would ask the question on their forums, referencing this thread, but I don't really want to keep track of another forum account just to ask one question.

[Michael Aivaliotis]

8 hours ago, TomOrr0W said:

Has JKI ever mentioned why they chose to have this restriction?

Isn't it obvious? Charging for desirable features is one way of creating revenue. Deciding where to draw the line between free and paid features is always a challenge.

[Rolf Kalbermatter]

I think it would be possible for JKI to add besides the JKI package repository and the NI Tools Network repository one more well known repositorey such as a LAVA URL to the free edition. That would not be a big deal to add I'm sure.

Another option I would definitely consider, is a paid version of VIPM that costs a small fee but then allows to add custom repositories to that client. You still would need a full pro license to create new repositories but not to access them. The current license model is impossible to defend in a company as ours where you not only need a pro license to create your own company wide repository, but anybody wanting to access that repository needs that license too in order to add it to the locations VIPM will check for packages.

So something like the free community edition as now with additional LAVA repository added.

Then an intermediate Developer Client edition that allows to subscribe to custom repositories that everybody in a company could use.

And last but not least the full featured VIPM Pro as it is now.

The old OpenG package manager could more or less do the two first editions already, it is just outdated and doesn't handle new LabVIEW file tyypes properly anymore, that were coming out with LabVIEW 8 and later.

EDIT: About 2 years ago we were evaluating in our company how to manage internal libraries and the distribution thereof. VIPM was considered too but the fact that every developer would need a Pro license killed that idea very quickly. If there would have been a Developer Client edition that normal LabVIEW developers could use and could have been purchased as a site license for a feasable fee, I'm pretty sure we would have been able to get that solution approved together with a few Pro licenses.

Edited November 17, 2018 by rolfk

[hooovahh]

VIPM years ago did have an Enterprise version (I think they called it) which was more or less what you described.  It didn't have all the features of Pro but could subscribe to feeds.  At the time it seemed like it was too confusing.  There was free, pro, enterprise, and I think a 4th option.  In the early days of package management, people (myself included) were confused about what features were good for what and it was hard to know what was needed.  VIPM was the first package manager I ever used and several concepts about feeds, dependencies, and use cases just weren't well understood.  JKI simplified this with a Free, and Pro making it clear when you would want one over the other.

[Rolf Kalbermatter]

I could see how this might have been confusing. Enterprise has since a long time always seemed to be even more than Professional, especially if you look at Microsoft. So that Enterprise would add the feature to subscribe to custom repositiories that you need Professional to create is kind of weird. Unless Enterprise would have been a kind of site license that everyone in a company could install, but was it that?

[smithd]

I find it amazing how convoluted licensing gets..considering thats literally how all money comes in. And I don't say that to pick on JKI -- I say that to pick on a company I saw today which has no less than 7 core software packages, plus for 3 of them you have to buy an add-on for support, plus there are several add-ons compatible with all base versions, plus one specific add-on only compatible with the upper 2 tiers, plus an additional add-on cost per seat which varies based on which base model you selected. It made me ?

NI vision is another example -- not quite as bad, but I still have no clue which imaq functions are included with what license.

 

Edited November 21, 2018 by smithd