Git submodules (or Svn externals), the fork-pull model, and LV versioning

[Stobber]

A sage LV developer recommended that I stop building my internal LV reuse libraries into VI packages and just link to them from my project repos as git submodules. Sounds interesting, especially given how much work I put into creating, testing, distributing, and documenting VIP and VIPC files. I poked around with svn externals a little bit years ago and thought they were neat. So I went snooping around the blogosphere for awhile to see what git submodules are all about.

 

Submodules in git are pointers to a specific commit in a repository at a specific URL. This is evidently different than externals in svn, but both are intended to solve the same set of issues.

 

http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer

[T]here are a number of use cases for [submodules], and they all center around nested git modules that are much more static in nature and use.

One - you have a component or subproject where the project is undergoing extremely rapid change or is unstable, so you actually want to be locked to a specific commit for your own safety. Or, perhaps a project is breaking backwards compatibility of their API and you don’t want to have to deal with that till they stabilize their code. In this case, git submodules, being reasonably static, are protecting you because of that static nature. You don’t have to clone the outside component and switch to a specific branch or go to any of that hassle - things just work.

Two - you have a subproject or component that you’re either vendoring or isn’t being updated too often, and you just want an easy copy on hand. To provide an example, in my dot files, if there’s a vim plugin I want I can just add it as a git submodule, and it’s done. I don’t care about the history. I don’t need to be at the latest version. I don’t plan on doing a lot of work on that code myself. Since this entire workflow is static, things work fine.

Three - There’s a part of your repository that you’re delegating to another party. Let’s say you’re paying someone to write a plugin for a project you’re using, and you need to develop on the main codebase. In this case, the plugin repository is chiefly developed by the plugin developers, so they own the repo and periodically they’ll tell you when to update submodule commits. Submodules are great for dividing responsibilities like this, assuming that there’s not frequent updating.

 

Right on! #2 seems right up my alley.

 

However, everyone in the git universe abandoned them over the last couple of years because of several issues with brittleness and complex git workflows.

 

https://github.com/cristibalan/braid

Vendoring allows you take the source code of an external library and ensure it's version controlled along with the main project. This is in contrast to including a reference to a packaged version of an external library that is available in a binary artifact repository such as Maven Central, RubyGems or NPM [ed: or VI Package Network].

Vendoring is useful when you need to patch or customize the external libraries or the external library is expected to co-evolve with the main project. The developer can make changes to the main project and patch the library in a single commit.

The problem arises when the external library makes changes that you want to integrate into your local vendored version or the developer makes changes to the local version that they want integrated into the external library.

 

Still, I can't help but wonder if these complaints would be minimized in my use case. Then a really important, LabVIEW-specific issue hit me:

 

LV code is compiled to its version, and it can't be opened by an older version of the LV dev environment. I wouldn't be able to push changes made in a local submodule back to the remote library repo unless my code were back-saved to the remote's version of LV. (For example, I made a pull request against NI's STM repository recently but had to leave my code in a later version than theirs.)

 

How the heck do we use tools like submodules/externals and the fork-pull workflow without being forced to upgrade all our LV code to the latest version (and re-test it, often against RT or FPGA hardware) every year when NI releases a new runtime? That's a dealbreaker for me because of the risk involved in upgrading existing projects to new drivers, runtimes, etc. (I've seen bugs introduced with a new version of LV that broke previously working code.)

 

[ShaunR]

LV code is compiled to its version, and it can't be opened by an older version of the LV dev environment. I wouldn't be able to push changes made in a local submodule back to the remote library repo unless my code were back-saved to the remote's version of LV. (For example, I made a pull request against NI's STM repository recently but had to leave my code in a later version than theirs.)

 

How the heck do we use tools like submodules/externals and the fork-pull workflow without being forced to upgrade all our LV code to the latest version (and re-test it, often against RT or FPGA hardware) every year when NI releases a new runtime? That's a dealbreaker for me because of the risk involved in upgrading existing projects to new drivers, runtimes, etc. (I've seen bugs introduced with a new version of LV that broke previously working code.)

 

Why are you removing diagrams from the VIs? If you don't remove the diagrams, then you can use the externals/submodules  features quite happily in all [forward] LabVIEW versions. The issue with LabVIEW is re-linking, If a  VI changes or is a different version, then LabVIEW usually wants to recompile the entire hierarchy including those you pulled out from other projects. The separation of compiled code from the diagrams has eased but not eliminated that aspect.

Edited April 7, 2015 by ShaunR

[Stobber]

Why are you removing diagrams from the VIs? If you don't remove the diagrams, then you can use the externals/submodules  features quite happily in all [forward] LabVIEW versions. The issue with LabVIEW is re-linking, If a  VI changes or is a different version, then LabVIEW usually wants to recompile the entire hierarchy including those you pulled out from other projects. The separation of compiled code from the diagrams has eased but not eliminated that aspect.

 

I'm not removing diagrams. If I take a VI saved with LV 2014 SP1 and try to open it with LV 2013 SP1, I'm greeted with an error dialog that says I can't open it. So if modifications to my reuse library are made and saved in the latest version of LV, any existing project that wants to import those mods will have to be upgraded to the latest version. This can be catastrophic for some of my long-lived projects that are still running on 2012SP1 or even older.

[ensegre]

Rambling on my recurring need for backsaving (e.g. quick fix of an old vi on a deveolpment machine with newer LV, deploy on some older LV), and academically thinking at whether someone had already bothered delivering a tool for mass backporting VIs/llbs/projects etc., I ran into these, for the record:

Save an LLB to the Previous Version of LabVIEW
 
Programmatically Save VIs for Previous Version

 

list of LV versions backsaving to which other previous

About this latter, I wonder if there is some property working in an arbitrary LV version, providing the list of previous versions for which backsaving is possible.

 

Also: a typical clumsy conversion case for me is the following: two main VIs using several but partially distinct subvis from a given reuse llb, need to backsave both, I start with the first and forget that the resulting backsaved llb contains only the dependencies of the first main and not of the second one.

 

And, anyone still remembers the catastrophic bug crashing LV when backsaving anything containing an event frame? I think it was 8.6->8.0.

[Stobber]

Seems like a commit hook would have to run that back-saves to the repo version, then runs unit tests on a test server (using the repo version of LV) to make sure the back-saved VIs still work as intended. What a pain!

Edited April 7, 2015 by Stobber