Jump to content

Recommended Posts

Who uses Git Submodules?  I know Greg Payne does and have watched his talk, but does anyone else have experience.   I'm especially interested in experience with projects with multiple developers and multiple products (that might need to use different versions of the subprojects).  

Link to post
Share on other sites

I do not. Not yet hopefully.
I work in a team of 4 devs, I'm pretty sure submodules - if well mastered - could help us improve the way we collaborate but even after watching Greg's talk I'm not feeling confident enough to dive in and drag my 3 co-devs in this. Yet.

Link to post
Share on other sites

I have started to experiment with them a few months ago. I have a new RT actor framework I am developing that I include as a submodule in the project that is consuming it. So far so good, but it is only a single developer single project.

Link to post
Share on other sites

I use git submodules. Do yourself a favor and don't use them. There are many pitfalls, here are just a few examples:

  • If a submodule is placed in a folder that previously existed in the repository (perhaps the one you moved into the submodule), checking out a commit from before the submodule existed will cause an error because "files would be overwritten". So you'd have to first delete the submodule and then check out the old commit. Of course, after you are finished you'd have to checkout master again and update your submodule to get it back to the current version.
    rm submodule/ -rf
    git checkout <old-commit>
    ...
    git checkout master
    git submodule update
    This is hard to remember and annoying to use. Not to mention the difficulties for novice git users and the lack of support by most UI tools.
     
  • Everyone on the team needs to remember to occasionally do git submodule update to pull the correct commit for all submodules. Just imagine shipping an application with outdated code because you forgot to run this command...
     
  • Branching submodules while branching the main repository, or as I call it: "when you spread the love".

    Imagine this situation:

    Colleague A creates a branch in the main repository, let's call it "branch_a". Now they also create a branch on the submodule, let's call it "submodule_branch_a". Of course they'll update the submodule in the main repository to follow their branch. Then they begin changing code and fixing bugs.

    In the meantime, colleague B works on master (because why not). Of course, master uses the master branch of the submodule. So they add some exciting new feature to the submodule and update the commit in the main repository. All of this happens in parallel to colleague A.

    Eventually colleague A will finish their work, which gets merged into master.

    Unfortunately, things aren't that easy... Although "branch_a" followed "submodule_branch_a", it will not do so after merging to master. Because colleague B changed the commit of the submodule in parallel to colleague A, whichever change happened last will be merged to master. This is an exciting game, where you spread the love, flip a coin and hope that you are the chosen one.

I actually changed our main repository (shared among multiple developers) to submodules a few years ago, only to realize that they introduce problems that we simply couldn't fix. So I had to pull the plug, delete all recent changes and go back to the state before submodules. That was a fun week...

That said, we now use subtrees. A subtree is somewhat similar to a submodule in that it allows you to maintain code in separate repositories. However, a subtree doesn't just store the commit hash but actually pulls the subtree into your repository, as if it was part of that repository in the first place (if you want, even with the entire history). With a subtree, you can simply clone the main repository and everything is just there. No additional commands. However, you'd want to avoid changing any files that belong to submodules, so that you essentially only pull subtrees and never push (or merge). I simply put them in a "please don't change" directory (anyone who changes files in that directory will have to revert and do their work over).

Atlassian has a nice article on git subtrees if you are interested: https://www.atlassian.com/git/tutorials/git-subtree

Link to post
Share on other sites

Typically, people tend to use 'repo' from Android to make a project from multiple git repos. It's a lot more flexible than submodules.

With 'repo', you create a 'manifest' (or multiple manifests) which contain all repositories to be included, with indication on branch and commit to be used.

It's intuitive, and anyone who worked with Android already knows how to use it.

 

Link to post
Share on other sites
On 8/28/2020 at 3:40 PM, LogMAN said:

Everyone on the team needs to remember to occasionally do git submodule update to pull the correct commit for all submodules. Just imagine shipping an application with outdated code because you forgot to run this command...

In our case it's the opposite problem; shipping code for a test station that has not been tested for that station.  We have multiple stations, that cannot all be tested at the same time, so changes to a shared component, made in developing one station type, should only be pulled into another type explicitly, not be a general "update all" thing.  Admittedly, I would much rather update common components asap, and I'm hoping splitting the common components into submodules will allow most to be pulled often.  Currently, the different stations get branched, with all shared components, which is a merging nightmare.

Link to post
Share on other sites
On 8/28/2020 at 3:40 PM, LogMAN said:

Branching submodules while branching the main repository, or as I call it: "when you spread the love".

I am hoping that the need to branch the submodule repo will be a big improvement on the current situation.  Needing to make an explicit commit/branch in the submodule will give the developer a point to think if they really meant to change something in the common component.  If they do, they will be making a commit message describing that change.  And I, if I'm the prime developer for that submodule) will notice this commit by another developer and can go and talk to them and deal with the issue (I will follow an aggressive merge-branch-quickly policy in common components).

Link to post
Share on other sites
On 9/1/2020 at 11:42 AM, drjdpowell said:

We have multiple stations, that cannot all be tested at the same time, so changes to a shared component, made in developing one station type, should only be pulled into another type explicitly, not be a general "update all" thing.

What you describe sounds very similar to our situation, except that we only have a single top-level repository for all stations. If you look at a single station repository of yours, however, the structure is almost the same. There is a single top-level repository (station) which depends on code from multiple components, each of which may depend on other libraries (and so forth).

* Station
	+ Component A
		+ Library X
		+ Library Y
	+ Component B
		+ Library X
		+ Library Z
	+ ...

In our case, each component has its own development cycle and only stable code is pulled in the top-level repository. In your case there might be multiple branches for different stations, each of which I imagine will eventually be merged into their respective master branch and pulled by other stations.

* Station (master)
	+ Component A (master)
		+ Library X (dev-station)
		+ Library Y (master)
	+ Component B (dev-station)
		+ Library X (dev-station)
		+ Library Z (master)
	+ ...

In my opinion you should avoid linking development branches in top-level repositories at all costs. Stations should either point to master (for components that are under development) or a tag.

* Station A (released)
	+ Component A (tag: 1.0.0)
	+ Component B (tag: 3.4.7)
* Station B (in development)
	+ Component A (tag. 1.2.0)
	+ Component B (master) <-- under development
* Station C (released)
	+ Component A (tag: 2.4.1)
	+ Component B (tag: 0.1.0)
On 9/1/2020 at 11:48 AM, drjdpowell said:

I am hoping that the need to branch the submodule repo will be a big improvement on the current situation.

Not sure if I misunderstand your comment, but you don't actually have to branch a submodule. In fact, anyone could simply commit to master if they wanted to (and even force-push *sight*). Please also keep in mind that submodules will greatly impact the git workflow and considerably increase the complexity of the entire repository structure. Especially if you have submodules inside submodules...

In my opinion there are only two reasons for using submodules:

  • To switch branches often (i.e. to test different branches of a component at station level).
  • To change code of a component from within the station repository.

Both are strong indicators of tightly coupled code and should therefore be avoided.

We decided to use subtrees instead. For every action on a subtree (pull, change branch, revert, etc.) there is a corresponding commit in the repo. We have a policy that changes to a component is done at component level first and later pulled into the top-level repository. Since the actual code of a subtrees is included in the repository, there is no overhead for subtrees that include subtrees and things like automated tests also work the same as for regular repositories.

On 9/1/2020 at 11:48 AM, drjdpowell said:

Needing to make an explicit commit/branch in the submodule will give the developer a point to think if they really meant to change something in the common component.  If they do, they will be making a commit message describing that change.

You have the right intention, but if any developer is allowed to make any changes to any component, there will eventually be lots of tightly coupled rogue branches in every component, which is even worse that the current state. Not to forget that you also need to make sure that changes to a submodule are actually pushed.

This is where UI tools become handy as they provide features like pushing changes for all submodules when pushing the top-level repository (IIRC Sourcetree had a feature like that).

To be fair, subtrees don't prevent developers from doing those changes. However, since the code is contained in the top-level repository, it becomes responsibility of the station owner instead of the component owner.

On 9/1/2020 at 11:48 AM, drjdpowell said:

And I, if I'm the prime developer for that submodule) will notice this commit by another developer and can go and talk to them and deal with the issue (I will follow an aggressive merge-branch-quickly policy in common components).

In my experience it's a good idea to assign a lead developer to each component, so that every change is verified by single (or a group of) maintainer(s). In theory there should only be a single branch with the latest version of the component (typically master). Users may either pull directly from master, or use a specific tag. You don't want rogue branches that are tightly coupled to a single station at component level.

Edited by LogMAN
  • Thanks 1
Link to post
Share on other sites

I had the same issue with svn and externals. One app pointed to branch of another component. It was a mess when managing this in long run.

Now when we do transition to git, i try to find alternative solution. So far the most reasonable is pulling component binaries to project that needs them via nuget package or g package manager. From what i see NIPM installs package globally for PC, not per project. 

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Similar Content

    • By Francois Normandin
      As I've reported in the UI Tools support page, I've started migrating the open source code I still have on bitbucket (Mercurial-based repos) to Github.
      I didn't think that it might be worth a specific topic until @LogMAN mentioned it. Personally, I'm moving my code to Github in the process. I know there are some reports of Hg-to-Git transitions not going so well when using sub-repositories, so please share your migration experience if you've had to jump into some hoops to get it done!
      For all of you who still use Mercurial and host your open source and/or enterprise repos on Bitbucket, this blog post is worth reading:
    • By odoylerules
      All,
       
      Recently I moved from SVN to GIT as my source control and revision manager.  I'm still getting used to everything so i don't consider myself an expert yet, but i really missed the Project Provider options that I used for SVN source control within the project window.  
       
      I'm releasing an open source beta project provider for TortoiseGit Integration.  This addon is very similar to the current SVN options but was developed entirely from scratch.  Currently this provider requires TortoiseGit to perform most of the Source Control Actions.  I know a lot of you on here have recently switched to sourcetree, however, unlike TortoiseGit they do not currently offer a command line interface yet for windows that i could find.  I've tried to make this project flexibile and if they offer a command line for windows eventually it should be fairly quick to re-factor for that.  
       
      The project can be found here: https://bitbucket.org/jed_d/lv_tortoisegit
       
      Addon Features
      Tortoise Git Icon Overlays within Labview IDE Tortoise Git Commands from within Labview IDE Ability to directly reload a project from within the Project Window, as well as prompts to reload the project when performing actions that require a reload. Open Git Bash from project window

       
      I'm hoping some of you will have a chance to try this out and let me know of any issues.  I would like to track issues on bitbucket if possible so that they are all in one place, but i will also be checking these post for issues as well.  
       
      Feel free to fork the project and hack on it yourself.  If you come up with something decent and don't mind sharing, send a pull request.  
       
       
      One initial issues i'm hoping the community can help with is building the package for older versions of LV.  Currently i only have access to 2013 so i am only able to build the package for this version.  If anyone has a change to pull down the repo and build it for older versions please do.
       
      Another question i have for the community is the license i currently have on the project.  This is my first open source project of any kind so i wasn't sure what license to put on it.  I did not want to be too restrictive so i went with a BSD license.  If any of you see an issue with this please let me know as I'm open to suggestions.
    • By PaulL
      Here is the set-up:
       
      I have a component project library (StepperMotor.lvlib) that I have moved to the userlib in order to facilitate reuse over multiple projects.  Within the library I have
      1) a strict typedef (StepperMotor.ctl) cluster that contains a numeric control for expressing the maximum output step rate.
      2) the StepperMotor.lvclass, an abstract class that contains effectively virtual methods (they do nothing but wire inputs to outputs) and LabVIEW-generated accessor methods for the two items in the class private data.
      This is LabVIEW 2013 SP1 on Windows 8.1.
       
       
      OK, so I start with everything clean (git status reports clean, LabVIEW opens and closes the library without prompting for changes).
      1) Then I open the StepperMotor.ctl directly from Windows Explorer (hence not opening the library), change the range of the numeric from 2 kHz to 4 kHz, and save it.
       
       
      2) Then I open the library, make no further changes, and attempt to close the library.  LabVIEW correctly prompts me to save changes.  I opt to close the project without saving changes, however.
       
      3) Next I discard the change to StepperMotor.ctl using git.  (I have been using Discard in SourceTree, or a git reset --hard command.)  Then git status returns a clean status.
       
      4) I open the project and attempt to close it.  Since I have reverted all changes, LabVIEW does not prompt to save changes.
       
      So far, so good.  This is expected behavior.
       
      Now, I repeat the steps, except that in step
      1) I open the library first, open the typedef control from the project, make the same change, save the control (without applying the changes), and close the project (without saving).
      2) is the same as above (OK).
      3) is the same as above.  Again, git status shows me everything is clean.
      4) I perform the same action as above, but this time LabVIEW prompts me to save all the accessor methods on the class, citing a changed typedef (even though if I open the typedef the range is indeed back to its original value).  This is unexpected behavior.
       
      Worse yet, even if I perform a git reset --hard and verify with a git status that everything is clean, if I open and immediately attempt to close the project, LabVIEW still prompts to save, citing "Type Definition modified".  This is indeed problematic.
       
      (Further note: If I let LabVIEW save the changes, git status still shows clean, and LabVIEW opens and closes the project without prompts.)
       
       
       
       



    • By PaulL
      Has anyone actually configured Git to work with LVCompare or LVMerge?  (I've seen some threads on various boards saying it "should be something like" <x>, but it isn't quite <x> and I haven't found anywhere that someone has actually described it successfully.)
       
      Notes:
      I know how to configure TortoiseSVN to work with LVCompare and LVMerge.  My new employer is moving to Git (some good, some bad in that, in my opinion, but that is another topic).
       
      I have been using Git for a few months now with reasonable success and decided I should try to set up graphical differencing.  [OK, in practice I've rarely used graphical differencing for any practical purpose, and graphical merging probably never for any actual code.]  Anyway, I think I've learned some things trying to figure out how to get this to work--but I don't have it working yet.
       
      I use the Atlassian SourceTree client at present (and sometimes the shell).  For the purposes of this discussion it might be more helpful to share the relevant text in the .gitconfig file (and any associated scripts, if necessary).
      As others have mentioned, Git itself (SourceTree similarly) does not distinguish which diff tool to call according to the file extension.  This seems like it could be a significant drawback.
       
      What I have tried:
      I added this (and variants of it) to my .gitconfig file:
       
      [difftool "LVCompare"] path = C:Program Files (x86)National InstrumentsSharedLabVIEW CompareLVCompare.exe keepBackup = false trustExitCode = false[difftool "LVCompare"] cmd = "C:Program Files (x86)National InstrumentsSharedLabVIEW CompareLVCompare.exe" "$LOCAL" "$REMOTE"[diff] tool = LVCompare SourceTree reports errors on start-up, so don't use this!  Lol!
       
      Maybe I need to add a script, too.  I'm not sure that I'm really all that close, honestly.
       
      [bigger question: Why is this difficult in the first place?]
    • By JeremyMarquis
      I am playing with GitHub as a possible replacement for our current SCC tool.
       
      Do you guys use Git or GitHub?  What client app do you use?  Any recommendations?
       
      I am specifically leaving the topic scope open, so feel free to praise/rant/headscratch.
×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.