VCS / Development Toolchains in 2020

[JB_1592]

So let's pretend for a moment that Git hasn't taken over the world. Assuming you had a choice, what is your toolchain of choice these days? This is more about total platform than specific VCS implementation.

I work with a small team with a badly overdue need for SCC. I find myself in a position to change some things, but, unfortunately, I've spent most of my career here and I'm a bit lacking in experience myself, so I've been looking into available options for a few days. The more I research, the deeper down this rabbit hole I go. Maybe a bit of discussion will help solidify the tools and features I'm looking for, and this seemed a good place for it since "works well with LabVIEW" is a factor.

Ramblings from here out...

Current state: Madness and chaos.🤪

Desired state:

• VCS repository management including either on-premises or cloud storage.
• Project management/issue tracking.
• CI/CD might be worth exploring?
• Documentation. A wiki or similar that can document the applications would be great.
• Due to the industry we are in, user management is sort of important. Ideally I'd like to use SSO and enforce dedicated accounts.

Thoughts on possible solutions...

• Gitlab
  • Seems the most likely Git contender. Bonus points for the fact that it can be self-hosted, which mitigates some of the user management concerns.
  • That said, see drjdpowell's recent thread for the type of discussion that makes me nervous about rolling Git out to an inexperienced team, especially under my not-exactly-expert guidance.
• Perforce + Jira + Jenkins
  • I believe our software engineering group is using something along these lines. (Not sure what their CI pipeline is.) This loses a some of the appeal of a single application, but I do think I would prefer the idea of a centralized VCS for our situation.
• SVN + ???
  • Maybe my Google-fu is off, but I'm not turning up a ton of modern guidance on a full tool chain. Helix Teamhub or Assembla along with Jenkins maybe? (Except I just noticed Assembla says their Jenkins plugin only works with Git... Hmm.)
• Azure DevOps
  • Can anyone share experience with TFVC or Pipelines? I haven't actually tried any of these tools yet and I'm not sure if it's worth pursuing, but the website is looking... surprisingly not bad. (Although even they seem to be pushing users toward Git.)

Also tried Github and Bitbucket. Found myself leaning towards Gitlab over Github, and prefer not to talk about Bitbucket. 🤢

[Bryan]

I haven't used many VCS systems aside from the very few variants that have existed at my current and previous employers where I had access for LabVIEW development.  Since some of those places didn't view LabVIEW as a programming language, I wasn't often granted permissions to have access to their VCS systems.  So, my experience is limited.

I spent a lot of time in Visual Source Save, but that's not going to help you here.  However, I was able to grasp SVN + TortoiseSVN pretty quickly.  I've also (for a very short period of time) had a little experience with Trac, though not enough to be a resource for knowledge on it. 

Additionally, I recall that VisualSVN Server was easy to install and manage an SVN server.

I've tried to grasp Git - but having been using TortoiseSVN + SVN for so long that I struggled to quickly grasp Git, so I abandoned it.

Edited July 9, 2020 by Bryan

[pawhan11]

@JB_1592 what was wrong with Bitbucket? Now I am in migration process from svn to git and still considering options.

[JKSH]

I suggest spending most of your energy on choosing a VCS for your team, as this is the part that takes the most effort for your team to adopt. I'd say that the other aspects (project management/issue tracking, documentation) have a lower barrier to entry.

 

18 hours ago, JB_1592 said:

I do think I would prefer the idea of a centralized VCS for our situation.

Can you elaborate on which parts of the centralized model are most important to you?

 

18 hours ago, JB_1592 said:

Maybe my Google-fu is off, but I'm not turning up a ton of modern guidance on a full tool chain.

I'm guessing it's because a distributed VCS (DVCS) can do just about everything that a centralized VCS can, but the converse is not true (hence my previous question). As a result, the online community (which is far more visible than corporations) are moving to a DVCS.

The 2 reasons I can think of for a company to stick to a centralized VCS instead of switching to a DVCS are:

• Inertia. If the developers are already familiar with an existing tool, there is a cost to switch.
• They want an extremely high level of control and security over the code base. A centralized VCS makes it a bit harder for a rogue employee to make off which the whole commit history (but it doesn't stop them from taking the current snapshot).

 

18 hours ago, JB_1592 said:

This loses a some of the appeal of a single application

If the separate applications interface with each other well, how important is it to still have a single application/platform?

 

18 hours ago, JB_1592 said:

I believe our software engineering group is using something along these lines. (Not sure what their CI pipeline is.)

Does your team have any existing interactions with the software engineering group? Can you get any support from them? Do you anticipate your team working with them in the future? If so, then the best choice for your team might be whatever the software engineering group is already using. That provides a lower barrier for collaboration between both groups.

If you expect to be completely isolated from the software engineering group, then I'm guessing there is not much difference between the possible solutions you have listed. All of them will come with an initial learning curve; the important thing is to pick one, get everyone on board, and get familiarized together.

 

18 hours ago, JB_1592 said:

user management is sort of important. Ideally I'd like to use SSO and enforce dedicated accounts.

I believe all modern hosting platforms support this.

 

18 hours ago, JB_1592 said:

Maybe a bit of discussion will help solidify the tools and features I'm looking for, and this seemed a good place for it since "works well with LabVIEW" is a factor.

Be aware that none of the common VCSes were designed to work with something like LabVIEW; they were all designed to work with text-based code.

So, regardless of which VCS you choose, LabVIEW devs must learn to take a bit more care to avoid triggering conflicts (and learn to handle the conflicts once they occur).

 

18 hours ago, JB_1592 said:

CI/CD might be worth exploring?

...

Azure DevOps

How "big" are your team's projects? How often do you produce a new release? Are  there parts of your release process where you go, "Man, this part is tedious and error-prone... It would be great to automate this"?

• CI is most useful when you have a lot of people working on the same code base and/or you have teammates who churn out commits at lightning speed. It can still be beneficial for small teams, but the impact is less pronounced (and the cost-to-benefit ratio is higher)
• CD is most useful when you want to release often, and/or your release process is tedious.
• DevOps is most useful for a large organization who wants better collaboration between their developers and their operators, and who want to make deployment more efficient.

As you described yourself as a "small team with a badly overdue need for SCC", I suspect these are lower priority for you right now. Again, getting SCC in place first will probably be the most helpful; the automations can always be added after you've tamed the chaos.

Edited July 10, 2020 by JKSH

[JB_1592]

9 hours ago, pawhan11 said:

@JB_1592 what was wrong with Bitbucket? Now I am in migration process from svn to git and still considering options.

 

Before I go into that, may I ask why you are migrating away from SVN? I'm looking for perspective on other VCS options.

 

As to your question, I think it mostly boils down a lot of little UX frustrations. So far I've had more trouble finding my way around the Bitbucket UI than any other. That said, after spending a little more time with it today... my view is softening a little.

I also find the way Atlassian separates everything in different tools to be a little confusing. For example, to get project management, repository storage, and user management, I would need to subscribe to Jira, Bitbucket, and Atlassian Access. Before I added it all up, I also found this was giving the illusion of a higher cost when moving beyond the free tier of any of this services. Maybe that's just me, and it does turn out not to be true. All the separate parts are low cost enough that they combine to still be a decent value compared to the competition.

Looking at the more common three Git platforms.

• Repo management
  • Gitlab allows creating groups and subgroups. Each repo is part of its own project, which can be placed in groups.
    • So I believe the way this would work is I would create a top level group for my organization, then use subgroups to further divide projects into categories.
  • Github allows creating an organization, which can then have teams and subteams. Repos belong to the organization and all share the same namespace. Repos can be assigned to multiple teams.
    • This feels like it would be more useful if I was more interested in controlling access vs. organizing repositories. The UI for navigating this just doesn't seem all that useful.
  • Bitbucket allow creation of a workspace, which can have projects. Similarly to Github, repos belong to the workspace and have a single namespace. Honestly, this feels pretty useless as the projects literally just group repos together and serve no other purpose. As far as I can tell, these "projects" don't provide any actual project management or access controls. (Am I rushing to judgement again?)
    • Jira allows adding multiple repositories to a project, which feels slightly more useful as it at least provides issue tracking, boards, etc.
• Project Management
  • Gitlab has an issue tracker on each project. Issues propagate and aggregate at the group level. I sort of like this. The downside, which might be something that can be overcome (or ignored) with proper configuration, is that tags used for status automation are board specific. (Although I think I may have just figured out how to fix that.)
  • Github has issue tracking on every repo, and allows creation of projects on repos, teams, and organizations. An issue can be placed assigned to multiple projects. Seems more flexible, but also like it might be more complicated to aggregate status.
  • Bitbucket on its own just has a pretty simple issue tracker on each repo that doesn't seem to have much to it. On the other hand, it also feels like a token effort because they prefer you to go to Jira, which is definitely feature rich. That said, this is also part of the UX issues I mentioned.
    • The method to get Jira issues to show up on the page in Bitbucket is apparently to create a commit that references one of the issues. I did that several times and it didn't work. The issues got hyperlinked on the commits page, but the Jira issues page still said there were no linked projects. I eventually realized this was because (I think) I didn't have Git configured on my computer with the same email as my Atlassian account. On the commits graph it says "This user cannot be matched to an Atlassian account," but apparently that doesn't stop me from being able to commit changes and reference issues... it just won't trigger a link to the project on the issues page. I edited the readme.md file from the web interface and created a commit that was linked to my Atlassian identity and it worked.
    • The integration the other way is simpler; you just click Add Item, Repository, and paste in the address. Unfortunately, that doesn't form the link in Bitbucket. That said, I'm coming to realize the more I play with this that I'd probably be looking to do project management tasks from the Jira project page, not on Bitbucket.
• Documentation
  • Yeah... they pretty much all offer per repo Wikis.
• User Management / Security
  • Gitlab
    • SAML enforcement is available at the Silver tier of the cloud application.
    • Self-hosting is available even at the free tier of service.
  • Github
    • SAML enforcement is availabe at the Enterprise tier of the cloud application.
    • I think you can also use Enterprise Server to self host for the same cost? They don't really seem to advertise this very well.
  • Bitbucket
    • Requires subscribing to Atlassian Access.
    • All of their applications can also be self-hosted. (And apparently for next to nothing with a small number of users.)
• CI
  • Gitlab - I know of others using Gitlab runners with LabVIEW, so that should work.
  • Github - They offer self-hosted runners... Haven't seen anyone specifically using them with LabVIEW, but probably doable?
  • Bitbucket - They actually get some points here for being the easiest to setup a pipeline to build a simple .Net Core console app. (Basic Hello World! thing.) That said, I don't think they offer self-hosted runners or other options that would work for LabVIEW. If we did get to the point of wanting this, I think we'd need to use Jenkins... or maybe Bamboo would work because of course there is another Atlassian product that does this. 🤣

[JB_1592]

2 hours ago, JKSH said:

I suggest spending most of your energy on choosing a VCS for your team, as this is the part that takes the most effort for your team to adopt. I'd say that the other aspects (project management/issue tracking, documentation) have a lower barrier to entry.

 

Yes. To that end, the thing that mostly drew my attention to Azure DevOps last night was the support for TFVC. They advertise Git repos so heavily that I actually didn't realize they (still?) supported that VCS until I took a closer look.

You are right, I probably should have limited my question to TFVC and not mentioned Pipelines as it certainly isn't the priority.

Maybe tomorrow I'll find a little time to install Team Explorer and play around with it... but when I went looking for the download link I found a MS rep's blog post that basically said MS wasn't going to release a standalone Team Explorer with recent versions of Visual Studio until enough customers asked for it. That makes me wonder how much they even intend to support TFVC going forward and if it might get dropped at some point soon.
 

2 hours ago, JKSH said:

Can you elaborate on which parts of the centralized model are most important to you?

 

Mostly you already mentioned them...

• "none of the common VCSes were designed to work with something like LabVIEW; they were all designed to work with text-based code"
  • Yes, which is why I'd prefer to use a lock/edit/commit workflow vs. a branch and merge one.
• "They want an extremely high level of control and security over the code base"
  • Define "extremely high level". A lot of Git workflows encourage rewriting or condensing history in ways I'm not entirely a fan of.

Can we simply not create a lot of branches (if any), never fork, and hope for no collisions without locks? Yes... and it will probably work fine. That's why I'm not completely ruling it out and looking at the rest of the platform.

(Sidenote: I'm actually a little surprised that, unless I'm missing something, Github seems to be the only platform that allows forking to be disabled at the organization level.)

 

3 hours ago, JKSH said:

Does your team have any existing interactions with the software engineering group? Can you get any support from them? Do you anticipate your team working with them in the future? If so, then the best choice for your team might be whatever the software engineering group is already using. That provides a lower barrier for communications between both groups.

If you anticipate to be completely isolated from the software engineering group, then I'm guessing there is not much difference between the possible solutions you have listed. All of them will come with an initial learning curve; the important thing is to pick one, get everyone on board, and get familiarized together.

 

This is a loaded question... Define "interaction". They, and especially the managers that would have to approve, tend to be a little... insular. (Trying to be nice here.🤣)

This research is mostly backup plan and personal learning... I hope.I do think using what we already have setup and that IS already knows how to support is the best strategy and I've already brought up wanting access to what they use. ... I just won't be surprised when it gets shot down.

 

2 hours ago, JKSH said:

As you described yourself as a "small team with a badly overdue need for SCC", I suspect these are lower priority for you right now

 

Yes, absolutely. SCC solution is the first priority, but trying to form some vision for what we might grow into can't hurt in making an informed choice. I was also generally like being aware of what's out there and what larger teams are doing.

 (Besides... I'm off this week so I've got time to research and play around with some free tier accounts. 😅)

Here's my thoughts on the parts of CI that might be nice for us...

• Automating versioning
• Automating testing
• Automating zipping the build and other artifacts and moving the packed files to a NAS location.
• Ensure the build occurs in a clean environment.
  • That way I know I won't have to go hunting down random dependencies that were in someone's vi.lib folder years later.
• Ensure the source of the build was checked in and record the commit info.

💭 Granted... I suppose I can do most of that in some form using the Application Builder API and some custom build scripts, and I suppose I could distribute those in project templates... Hmmm, maybe we wouldn't gain much... 💭

[pawhan11]

I want to leave svn behind as soon as possible because current repo structure that was tailored many years ago in my organisation. We have one svn server with one svn project and inside that project people created folders for multiple projects... i once found there labview iso image. After 2 years working with that i still have no idea where to find something.  🙂

I guess I could migrate to another svn and do it right this time. I want git because local repo is big benefit for offline work that is more common these days. Another reason would be to keep up with technology progress. All sw companies that i know use git.

For project management we have Jira and it will probably stay that way. Ability to define custom workflows solves many problems for other teams, like projects for equipment calibration, maintenance, production support work really well. We do not have make custom tailored software to manage this. I tried to do this in gitlab and failed.

[LogMAN]

On 7/9/2020 at 12:14 PM, JB_1592 said:

I work with a small team with a badly overdue need for SCC. I find myself in a position to change some things, but, unfortunately, I've spent most of my career here and I'm a bit lacking in experience myself, so I've been looking into available options for a few days. The more I research, the deeper down this rabbit hole I go. Maybe a bit of discussion will help solidify the tools and features I'm looking for, and this seemed a good place for it since "works well with LabVIEW" is a factor.

Whichever system you choose will require a lot of time and effort to implement by the whole team. Don't think that you can simply choose a tool that "feels right" and expect everyone to accept your opinion. I made that mistake and had to rethink my strategy after a disaster of a start (only one of us adopted the new system - me ). You also don't want to be responsible for everything, so it's best to have the majority on your side before you enforce a new system.

Your main focus should be on your workflow and your requirements to decide which tools are right for you.
For example,

• do you have small projects or big?
• how many developers work on a single project?
• are your projects deep (multiple projects in separate repositories), wide (multiple projects managed in a single repository), or flat (single repository)?
• do you always have access to your servers when developing?
• do you have a project leader, or is each project managed by a single head-developer?
• who are your stakeholders and which kind of reports, documents, and access rights to they need?
• how are bugs reported to your team?
• do you need to give access to your source code and bug trackers to third-parties (external)?
• what kind of development cycle is used for your projects?
• do you need to integrate additional platforms or tools (i.e. CRM)?

While it is kind of easy to state those questions, the answers are not that simple to give. I still struggle to answer some of those (among other) questions and simply made gut-decisions where necessary.

Also, don't forget to include your team in this!
Simple questions like "Where do you struggle the most?" do wonders if you care to listen.

Here is our tooling for reference:
We use

• Git for source control,
• Jira for bug tracking, planning, and reporting, and
• Confluence for the wiki.
• Our developers are free to use any client they want (i.e. Sourcetree), as long as they adhere to the commit guidelines.
• We still don't have a working CI/CD solution, so each developer builds their own project (or in case of larger projects, the head-developer). Unfortunately LV is exceptionally slow in this area, especially for large projects. We are actually at the edge of what we consider acceptable.

In the past we used ZIP files, later SVN. I don't think I need to explain why we went away from ZIP files. SVN, however, we discarded because it wasn't the right fit. Occasionally we have to change programs on-site, without server access, and want to be able to commit changes. SVN simply is too much effort for such a workflow. You need to remember to checkout and lock files before you travel and hope that nobody breaks the lock. Lessons learned from using SVN: Don't trust your colleagues 😱

[Bryan]

@pawhan11 That sounds exactly how SVN was set up where I currently work - Including having LabVIEW ISOs/Installers stored within. (I wonder if we work at the same company.)

One of the drawbacks you've mentioned and I've found with SVN is the ability to search for directories/files/etc.  There are tools out there I believe to do this, but, since I'm not an administrator, I can't implement anything myself.  Something I've done is use the SVN commands to generate a text file listing of ALL of the SVN contents... then if I need to search for something, I use Notepad or Notepad++ to search for key words in the file.  Once found, I then I have the mapping to find it in the SVN Repository. 

Something I would want to do if it were me - would be to set up a new repository (or reconfigure the current - if able) to use the proper multi-project structure that SVN recommends. 

[Mads]

On 7/10/2020 at 8:29 AM, pawhan11 said:

I want to leave svn behind as soon as possible because current repo structure that was tailored many years ago in my organisation.

I am sure it is possible to mess up any type of SCC system that way 🙃

The main complaints I have with SVN really is the slowness - mainly related to locking (can be sped up *a lot* if you choose to not show the lock status in the repo browser though), and the occasional need for lock cleanups... (When someone has been checking in a whole project folder and it did not contain all of the necessary files for example...).

Edited July 14, 2020 by Mads

[joerghampel]

For those of you interested in exploring CI/CD with git (and especially with GitLab), please take 5 minutes to visit our Release Automation Tools for LabVIEW website.

At the very least, you'll get an impression of (and maybe some inspiration through) what we've been doing for ourselves and for some of our customers for quite some time now. A ready-made solution might save you a lot of development time. If you want more details, I can share recordings of webinars we did a while back. I'd also be happy to show you around myself, too. 

Spoiler(?) alert: This is a commercial tool.

Edited August 12, 2020 by joerghampel

[Rolf Kalbermatter]

I have a fair amount of experience with SVN and a smaller amount with GIT. At our company we still use SVN, not because it is super perfect but because it simply works. I have not managed to get myself into a real mess with SVN. The worst that can happen in my experience is that some operation doesn't terminate properly and you have to do a manual cleanup to be able to continue.

Enter GIT and that changes dramatically. It's very easy to do just about anything in GIT, including mistakes. And I can not count the number of times in which I spend several hours of unraveling some mess I caused by selecting the wrong GIT action for some reason. The fact that the naming of the different GIT actions is sometimes rather ambiguous and the parameters can be mind boogling complex doesn't hel that either.

The few times I did something wrong in TortoiseSVN I simply went into the command line and entered a few simply enough commands through svn.exe and all was well. TortoiseGIT is very easy to do things wrong and the GIT command line, .... well it feels like having to do a one year study to even understand the basics of it. 😀

[pawhan11]

Did You guys have any problems with labview CLI when setting up CI? Or it is just me doing something wrong somewhere. So far I ended up with powershell script that based on params triggers builds, runs caraya tests and creates simple zip package with binaries. What I see is that sometimes build fails one time and when retried works fine when no change to code was made, sometimes cli can not connect to laview. Another problem that i encountered is fact that labview build destination is absolute path, not relative and has to be updated in lvproj xml files. I fear that when we spin off dedicated vm to run those builds it will fail due to potential LV 'problems'.