Code Sharing Best Practices

[John Lokanis]

One of the main topics of the 2018 CLA Summit was the need to improve access to open source code in the LabVIEW community.  This is something that I have tried to do in the past with limited success.  After hearing what others are doing and discussing the issues, I am inspired to take on the task of getting as much of my code that is shareable out into the open for others to use, improve, learn from and critique.  So, the point of this thread is to figure out how best to do that.

I have tried posting code to forums in the past.  I have even posted to the code repository here on LAVA.  I have used code posted here and via the tools network and VIPM in my own projects.  But I am not sure if any of those avenues are the right path forward for me.  There was much discussion about different open source repositories on the interwebs that we could leverage.  There was also some discussion about how to help others discover the code you shared.  What I did not hear was any definitive conclusions on how best to do this.

So, the point of this thread is to try to solicit feedback on code sharing and come to some sort of consensus on the best options out there.  If you have an opinion on this please join the conversation and share what you think is the best solution.   Here are some questions I am trying to answer:

1. Where should we share code?  What system works best for LabVIEW code and is user friendly enough to not discourage people from using it?  Please share links and how-to documents for your preferred site/system.

2. How should we license code?  I heard some discussion about the various type of licenses.  I am not interested in retaining any rights to code I share and do not want to put any burdens on those who want to use and learn from any code I share.  What licence is accepted in the open source community that supports this kind of sharing?

3. Once we post, how do we make our code discoverable?  Do we need to post links all over the place or is there a better way?  Here is one attempt at making that better you should check out if you have not already: 

 

I am not just interested in putting the code out there, but also trying to explain why I think it is worth your time to take a look.  I am willing to post on forums, create a blog, even produce some vlogs on YouTube if it is the best option.   Please let me know what format would motivate you to take the time to learn about the open source code out there.

Either way, thanks for taking the time to read this thread and contribute what you can, even if it is just to follow the discussion and learn from others like I am trying to do.

-John

[jacobson]

1 hour ago, John Lokanis said:

2. How should we license code?  I heard some discussion about the various type of licenses.  I am not interested in retaining any rights to code I share and do not want to put any burdens on those who want to use and learn from any code I share.  What licence is accepted in the open source community that supports this kind of sharing?

You might want to give https://tldrlegal.com/ a look although as I am also not a lawyer I can't really verify the correctness of any information they give. I would start by looking at the most popular but the ones I hear about most are BSD, MIT, and Apache. The NI DCAF tools are all under Apache 2 which lets users do basically anything they want to do with the code but doesn't hold you liable (again, not a lawyer).

1 hour ago, John Lokanis said:

3. Once we post, how do we make our code discoverable?  Do we need to post links all over the place or is there a better way?  Here is one attempt at making that better you should check out if you have not already: 

I don't know of a good way myself. GPM (https://gpackage.io/) looks promising but of these things very much rely on a network effect. GPM also looks much more focused on project libraries and not editor enhancements such as quick drop and right-click plugins or even templates like TLB' or DQMH. Because there isn't really an easily searchable centralized repository of LabVIEW code that I know of, I think you have to be a good self promoter if you want people to use any LabVIEW software. I think Fabiola (DQMH), Q (QControls), and Derek (Solution Explorer, GPM, MGI Stuff) all promote their stuff well if you want to see what they are doing (I think it's a mix of quality products, quality documentation, and continued promotion of the product)

Also make sure to check out the Distributed Control and Automation Framework (DCAF) when you are starting your next embedded control application.

[Ryan.Sheppard]

Github has this site for some ideas. It is focused on a single project. https://opensource.guide

Were the thoughts at the CLA Summit about sharing random bits of code? Or sharing projects? I know NI Systems Engineering group uses Github to share their open source projects.

For open source we would want the users to be able to clone a repository and be able to merge could back into the repository.

Edited September 21, 2018 by Ryan.Sheppard

[John Lokanis]

That is a good point.  The code I would want to share includes one-off examples, reuse libraries and full frameworks.  Not sure how to deal with all the dependancies.  It seems like you would have to first share all the reuse components before you could share the higher level projects that rely on them.

Also, how should we be packaging the reuse libraries?  VIPM packages?  Source code distributions?  Zip files?  And what level of documentation and testing is going to be expected and accepted?  I think we need to come up with a baseline of what shared code should include so it has some value and is not too hard to use.   But I guess we could let the market decide that.  I would still like to hear what everyone thinks is needed at a minimum.

[Mark Balla]

LV Champion Derek Trepanier has developed a Git solution and did 2 presentation at the CLA summit

 

I am still working on getting all the CLA videos edited and uploaded. The new Tecnova video sharing site is a week away from going live.

So to keep the momentum going I have posted Derek's presentation on the link below so others can see his perspective.

 

https://drive.google.com/open?id=17mjooun62caeL6EcsCG2g1arpg-OoRhk

[Mark Balla]

I am not sure what the solution would look like but I would like to somehow combine Dereks solution with the LAVA Code repository.

Ideally we would use LAVA for discussion, ranking and finding and Git for code management and collaboration.

 

[smithd]

16 hours ago, John Lokanis said:

2. How should we license code?  I heard some discussion about the various type of licenses.  I am not interested in retaining any rights to code I share and do not want to put any burdens on those who want to use and learn from any code I share.  What licence is accepted in the open source community that supports this kind of sharing?

MIT/BSD is the traditional answer but the real one is "you should license your code". That means at minimum having the full legal text (not just writing "BSD" in the VIPM field) of the license in your repository/package/distribution, with a proper name (not [enter name here] as I've seen sometimes), and ideally adding it to each separate file.

Github provides https://choosealicense.com/ as a helpful guide

 

14 hours ago, jacobson said:

GPM (https://gpackage.io/) looks promising but of these things very much rely on a network effect.

OMFG this is great.  to MGI. I've often thought about this, specifically the per-project install but the amount of effort I would have to personally invest was too high. I may very well be suuuper disappointed after spending time looking at the details, but its exciting just that someone is trying

 

---

 

The other big unique labview pain point is versioning. As long as NI insists that a 2018 VI is different than a 2017 VI, theres going to be the potential for problems.

Edited September 21, 2018 by smithd

[smithd]

2 hours ago, John Lokanis said:

3. Once we post, how do we make our code discoverable?  Do we need to post links all over the place or is there a better way?  Here is one attempt at making that better you should check out if you have not already: 

A semi-common way of doing this is to make github pages with lists -- these github pages are themselves git repos and thus can be modified by anyone, just like a wiki.

example:

https://github.com/rust-unofficial/awesome-rust

https://github.com/bulutyazilim/awesome-datascience

https://github.com/node-opcua/awesome-iot

meta: https://github.com/sindresorhus/awesome

Obviously a lot of these are poorly organized, including the data science one, but...

 

 

[ShaunR]

Quote

You can easily copy the gpm cli command straight from the website. Now open up your favorite command line tool

Quote

To create a package using Package Manager, you gather the relevant components into a single directory and build the package using the command line interface.

What is this? The 1990s?.

 

One of the major problems with search and distribution is verification. By that I mean verified as in tested, robust code that is fit-for-purpose. One reason why many packages aren't on the Tools Network is this overhead. It's also why we have the Lava CR vetting.

Edited September 21, 2018 by ShaunR

[ShaunR]

Quote

1. Where should we share code?  What system works best for LabVIEW code and is user friendly enough to not discourage people from using it?  Please share links and how-to documents for your preferred site/system.

VIPM is the sure-fire method. It has it's quirks but ticks most boxes. There are only a couple issues that mean it doesn't necessarily suit all people, one of which is getting your repository listed.

Quote

2. How should we license code?  I heard some discussion about the various type of licenses.  I am not interested in retaining any rights to code I share and do not want to put any burdens on those who want to use and learn from any code I share.  What licence is accepted in the open source community that supports this kind of sharing?

That should be up to the developer. You can argue all day and night about which licences are better or worse, but the developer is the arbiter of his creation. One of the other deficiencies of VIPM is not having a way of documenting all the licences of a package with all it's dependencies. That could be addressed fairly straightforwardly, but so far hasn't. For your particular conundrum, I would suggest multi-licencing. List multiple licences (e.g. CC-0, Public domain, MIT, BSD) then tell people they can use any one of them to their liking. Public domain is the "free"est but it is not recognised in some countries (Germany?).

Quote

3. Once we post, how do we make our code discoverable?  Do we need to post links all over the place or is there a better way?  Here is one attempt at making that better you should check out if you have not already: 

If it's in VIPM that's not an issue. Getting it in there is.

I really thought the NI Package Manager (which I always said they should have done instead of using the JKI one) would address all this. They could even have had native SCC integration. But from what I have seen it is a poor replacement for VIPM targeted at NI products rather than addons.

Edited September 21, 2018 by ShaunR

[hooovahh]

Wow lots of good discussion.

23 minutes ago, ShaunR said:

I really thought the NI Package Manager (which I always said they should have done instead of using the JKI one) would address all this. They could even have had native SCC integration. But from what I have seen it is a poor replacement for VIPM targeted at NI products rather than addons.

Totally agree.  I intend on presenting on reuse and package strategies, and in researching and discussions with NI about NIPM I think they got the messaging wrong.  By even calling it NIPM, where many LabVIEW developers have only known one package manager their whole career, means we already think NIPM is to replace VIPM.  But looking at the design choices NI made it doesn't look great.  One major current limitation is that a package is made for a specific version of LabVIEW.  This means if I want to distribute a toolkit I need to make a NXG 1.0 package, NXG 2.0 package, NXG 2.1 package, NXG 3.0 package, etc.  This is because the installation path is to a specific folder which changes with each version.  Then there is the lack of pre and post VI calls.  There is an option for calling EXEs.  Needless to say I'm not thrilled about having two (at least) package managers to think about, manage, and deploy with.

As for code sharing, I really like LAVAs low barrier to entry, flexibility with license, and automatic discussion page creation.  I like Git, and Bitbucket's version control and collaboration features.  I like NI's larger overall traffic, with potential high exposure.  And I like the tools network for ease of access for the developer.  Maybe instead of us trying to come up with the answer to "How do we share code?" we should just have a list of options, and reasons you'd use each, then talk about how to use each option.

[smithd]

 

3 hours ago, ShaunR said:

What is this? The 1990s?.

I thought I saw a UI in the video, but the main point is that this is intended for people with automated processes...so a CLI is the correct answer.

2 hours ago, hooovahh said:

Totally agree.  I intend on presenting on reuse and package strategies, and in researching and discussions with NI about NIPM I think they got the messaging wrong.  By even calling it NIPM, where many LabVIEW developers have only known one package manager their whole career, means we already think NIPM is to replace VIPM.  But looking at the design choices NI made it doesn't look great.  One major current limitation is that a package is made for a specific version of LabVIEW.  This means if I want to distribute a toolkit I need to make a NXG 1.0 package, NXG 2.0 package, NXG 2.1 package, NXG 3.0 package, etc.  This is because the installation path is to a specific folder which changes with each version.  Then there is the lack of pre and post VI calls.  There is an option for calling EXEs.  Needless to say I'm not thrilled about having two (at least) package managers to think about, manage, and deploy with.

Yeah I certainly had hope that NIPM would replace VIPM. I've occasionally heard complaints about NI stomping on products alliance partners have developed and I'm sure that does suck, but it amazes me that something so fundamental to NI's software platform is the place where they decide to draw the line.

Its technically possible to build a source distro and then convert that into an NIPM package (basically what VIPM does as I understand it) but palettes are impossible. So far as I can tell VIPM's greatest gift is not the package part, but the actually decent palette editor (something else I'd expect to be core to NI's software platform, as the palette is one of LabVIEW's killer features for learning to write code).

I've also provided that feedback on the post nipm install calls to the systemlink people. They want to do automated deploy of packages, but there is no obvious way to stop/start windows services?  I don't particularly care if its a VI call, but I do think at least having a command line call is needed.

 

Edited September 21, 2018 by smithd

[ShaunR]

2 hours ago, hooovahh said:

Maybe instead of us trying to come up with the answer to "How do we share code?" we should just have a list of options, and reasons you'd use each, then talk about how to use each option.

We already know how to share code. The problem is it is fragmented, de-integrated, and unsearchable across the myriad of different platforms and websites. The LVTN is supposed to alleviate that by having a central listing/repository and VIPM is the UI to administer it. What VIPM doesn't have (at least in the Community Edition) is a submission process - build, upload & publish. There is nothing to stop publishing being on Github, BitBucket or any other repository either- could be all of them and have the best of both worlds.

If VIPM isn't doing it as we would like; we can always go back to its predecessor to build in what we need (Tell us about your enhancements, Rolf )

[David_L]

Personally, I think that VIPM would cover most of the needs of code sharing if it wasn't for one HUGE limitation which is a public, free, open repository.  The LabVIEW Tools Network repository is great for sharing code that has been vetted and approved by NI's engineers and processes.  However there's obviously a huge need for a public repository that has little or no barrier to entry.  And even if somebody created a repository like this somewhere (LAVA maybe?) nobody could access the repo unless they pay for a Pro license of VIPM per-user.  

Now what Derek/MGI have started with GPM is an awesome first step in solving this, but I think there's one big disadvantage: there is now one additional tool for installing G Code (reminiscent of https://xkcd.com/927/).  If I have a project that uses some NI toolkits, some LVTN toolkits and some Open Source GPackage toolkits, I have 3 different tools needed for getting a system set up.  I also have three different places (not counting the numerous GitHub repos, LAVACR pages, internal code repos, etc) to go looking for the code I need.

What we need is a single tool that will install everything from all sources in the same standard.  I would also want some way to differentiate what's an official NI product, whats a validated LabVIEW Tools Network product, and what is a use-at-your-own-risk open source ...thing...  It would also be great if users could rate and review all these things so you could have more information on which of the 7 JSON toolkits is the best for your needs.  

I also realize the fallacy of pointing out problems without solutions, but as I'm not in the business of making package management tools, it's hard for me to prescribe others what to do with their time and money.  Hopefully this discussion bubbles up to the right people who ARE in this business so we can get the solution the community needs.  

 

[Edit: Darn, in the time that I was typing this, it looks ShaunR posted almost the same ideas, just more succinct... Glad we're on the same page]

Edited September 21, 2018 by David_L

[ShaunR]

On 9/21/2018 at 3:13 PM, smithd said:

so a CLI is the correct answer.

Let's agree to disagree

On 9/21/2018 at 3:13 PM, smithd said:

So far as I can tell VIPM's greatest gift is not the package part, but the actually decent palette editor

Very much so. But lets not forget the great job VIPM does with versioning, dependencies and applying licences.

On 9/21/2018 at 3:13 PM, smithd said:

but there is no obvious way to stop/start windows services?

That's just not true. What is true is that it is a hack at best to create a LabVIEW service. Perhaps this is what was meant.

On 9/21/2018 at 3:13 PM, smithd said:

but I do think at least having a command line call is needed.

Edited September 22, 2018 by ShaunR

[smithd]

1 hour ago, ShaunR said:

Let's agree to disagree

Tell you what, we can chat about this again when Jenkins puts out its first plugin specifically to work with GPM or VIPM. Until that point, you need a CLI or, even better, what people used to do -- a batch file which wraps labview 

1 hour ago, ShaunR said:

That's just not true. What is true is that it is a hack at best to create a LabVIEW service. Perhaps this is what was meant.

You can put some steps if you want into your install process that tell your user to "open services.msc, then look for the service 'blah co's sweet blah blah app', right click, select stop, and wait for it to finish, then return to this wizard". For me, I'd prefer to put in my package a system exec call "sc stop mysweetapp" and be done with it.

As for making a windows service, its not great, but its not super hacky and I now have a dozen or so services running using this pattern:

• In your application:
  • Open up a self-reference so closing the panel doesnt kill the app
  • add in a panel close event -- NOT the panel close? but panel close itself
  • When the panel close event is generated, do your safe shutdown and then close the self-reference and exit labview
• Use NSSM to create a service wrapping your application
  • On the shutdown page only check WM_CLOSE and Terminate process with a reasonable timeout specified.
  • WM_CLOSE will trigger the panel close

My solution to the NIPM limitation was to bundle NSSM with each package.

Edited September 22, 2018 by smithd

[ShaunR]

33 minutes ago, smithd said:

You can put some steps if you want into your install process that tell your user to "open services.msc, then look for the service 'blah co's sweet blah blah app', right click, select stop, and wait for it to finish, then return to this wizard". For me, I'd prefer to put in my package a system exec call "sc stop mysweetapp" and be done with it.

I do neither

[smithd]

Seems harder than my version

 

[John Lokanis]

Back to the original topic:

What is the best way to share reuse code that other code depends on?  My reuse code is deployed in user.lib so that it appears in the pallet and can be found by quick drop.  I have seen some suggestions that reuse code should be in the project folder for each project.  This seems to be more portable but then leads to multiple copies of the same code on your system, making maintenance difficult.

Another issue is using SCC with code in the user.lib.  You don't really want your SCC system to be placing files in your program file directory.  So, you end up with storing the code elsewhere for maintenance and then needing to package and deploy it after each edit.  This make adding new functions and fixing bugs cumbersome.

So, what solutions have others used?  What are the tradeoffs?  And what process should we adopt for reuse libraries we wish to share online?

I would like to find a solution where we can get the benefits of the code being in user.lib without having to keep it under the LabVIEW installation folders.

[gregoryj]

On 9/21/2018 at 5:58 AM, hooovahh said:

Wow lots of good discussion.

Totally agree.  I intend on presenting on reuse and package strategies, and in researching and discussions with NI about NIPM I think they got the messaging wrong.  By even calling it NIPM, where many LabVIEW developers have only known one package manager their whole career, means we already think NIPM is to replace VIPM.  But looking at the design choices NI made it doesn't look great.  One major current limitation is that a package is made for a specific version of LabVIEW.  This means if I want to distribute a toolkit I need to make a NXG 1.0 package, NXG 2.0 package, NXG 2.1 package, NXG 3.0 package, etc.  This is because the installation path is to a specific folder which changes with each version.  Then there is the lack of pre and post VI calls.  There is an option for calling EXEs.  Needless to say I'm not thrilled about having two (at least) package managers to think about, manage, and deploy with.

As for code sharing, I really like LAVAs low barrier to entry, flexibility with license, and automatic discussion page creation.  I like Git, and Bitbucket's version control and collaboration features.  I like NI's larger overall traffic, with potential high exposure.  And I like the tools network for ease of access for the developer.  Maybe instead of us trying to come up with the answer to "How do we share code?" we should just have a list of options, and reasons you'd use each, then talk about how to use each option.

I was actually surprised when I stumbled across this page where NI makes the following recommendations:

For LabVIEW 20xx:

• Continue using VIPM for distributing source/development code
  (e.g. reuse libraries, development tools, Tools Network, ...)
• Use NIPM for distributing and updating deployed applications (e.g. componentized applications, application EXE, PPLs, application plug-ins, NI drivers and software, LabVIEW Run-Time Engine, …).

The goal with NXG would be to have NIPM handle source code as well...

1 hour ago, John Lokanis said:

Back to the original topic:

What is the best way to share reuse code that other code depends on?  My reuse code is deployed in user.lib so that it appears in the pallet and can be found by quick drop.  I have seen some suggestions that reuse code should be in the project folder for each project.  This seems to be more portable but then leads to multiple copies of the same code on your system, making maintenance difficult.

Another issue is using SCC with code in the user.lib.  You don't really want your SCC system to be placing files in your program file directory.  So, you end up with storing the code elsewhere for maintenance and then needing to package and deploy it after each edit.  This make adding new functions and fixing bugs cumbersome.

So, what solutions have others used?  What are the tradeoffs?  And what process should we adopt for reuse libraries we wish to share online?

I would like to find a solution where we can get the benefits of the code being in user.lib without having to keep it under the LabVIEW installation folders.

I thought about these trade-offs quite a bit, and now I keep all code that is not installed with LabVIEW in a "project" folder. I just copy and paste any re-use code that I need into the project folder. It's not nearly as sophisticated as MGI's package manager, but I mostly develop code by myself so I don't need a very sophisticated solution. My main reason is that if something happens to me at least my co-workers could find the source code repo or the source.zip file I make with each build and have everything they need, instead of wondering where to find the dependencies.

If I do find a bug in the re-use code, then I need to update it in every project if it effects that project. User.lib is mostly easier from a maintenance standpoint, and mostly harder from a portability standpoint. If I wanted to make some change to my re-use code that added a feature but broke backward-compatibility, I actually would not want to have to go update old projects to get a feature they don't need. 

[smithd]

18 hours ago, John Lokanis said:

So, what solutions have others used?  What are the tradeoffs?  And what process should we adopt for reuse libraries we wish to share online?

I would like to find a solution where we can get the benefits of the code being in user.lib without having to keep it under the LabVIEW installation folders.

I've always done it the terrible way which is why gpm is exciting to me  -- use vipm to install your reuse code in vi.lib and then completely uninstall it to make changes, then rebuild. You can sometimes keep it simple by writing reuse code in version N while you write your applications in N+M, but that isn't always feasible if NI releases something worth upgrading for (vims).

I've tried subtrees in source control, but I seem to always reach a diamond pattern which breaks the subtree.

I don't see maintenance as being particularly hard with the gpm style. Also, while I'm not a web developer I've certainly downloaded and tried out web applications, and NPM is pretty amazing for providing me, the clueless end user, with a working system. Its kind of like a VM in that way -- it grabs exactly the versions the developers used and it doesn't touch anything else in my system.

I'm curious if anyone has thoughts on downsides to the gpm style besides the palette issue...because I think thats solvable (if annoying)

18 hours ago, John Lokanis said:

And what process should we adopt for reuse libraries we wish to share online?

For a distribution situation it matters a bit less. Like, I'm sure the developer has to deal with managing consumption of their own packages and development of those packages, but for everyone else we just download the latest version and install.

Edited September 25, 2018 by smithd

[hooovahh]

Continuation of package building with OpenG and DEAB tools moved to here.

[QueueYueue]

Hey, Derek here. Stoked to see GPM came up.

Couple of things I want to mention.

1. The current GPM release is still a beta release. If you find bugs or want to request features, add them to the issue tracker. https://gitlab.com/mgi/gpm/gpm
2. @ShaunR Theres a CLI and a GUI. The CLI will probably be used by CI setups, the GUI will be used by people. Eventually (though not currently) anything that you can do in one you can do in the other.(the "commands" that the CLI and GUI execute are just two different ways of executing the exact same business logic.)
3. I know it needs to support distributing things like quickdrop plugins. I haven't quite decided how I'd like to do it, but I definitely realize it's something that's needed before a full on release. https://gitlab.com/mgi/gpm/gpm/issues/18
4. Yeah, @David_L I agree, +1 to the number of package managers sucks. Sorry :/

Back to the OP's topic:

One of GPM's goals was to answer this question. If you have code that you want to give out for free, I think it should be trivially easy to do so. Using GPM, you just fill out the meta data, create an account on https://gpackage.io/ and then click publish. Once the command has finished executing you're good to go. It's published on the internet so anyone can download it. Someone who wants to use your code just needs to install it using GPM. If your code depends on other packages they'll be installed at the same time. It should be super easy (let me know if it's not!)

Additionally, GPM package meta data has all of the fields needed to properly index your stuff and link back to your repo. This makes it easy for people to find your code, and contribute to it as needed.

Happy to answer any more questions (or argue about design decisions =P)