OpenG Library Exported to Github

[Michael Aivaliotis]

I've exported the OpenG sources from Sourceforge SVN to Github.

• It's located here: https://github.com/Open-G

I'm hoping this will encourage collaboration and modernization of the OpenG project. Pull requests are a thing with Git, so contributions can be encouraged and actually used instead of dying on the vine.

 

[Antoine Chalons]

Cool!

What tool have you used to import the whole SVN repo with all the revision into the Git repo?

Is it a feature in Github?

Edited October 3, 2019 by Antoine Chalons

[Michael Aivaliotis]

I followed these instructions:

• https://www.atlassian.com/git/tutorials/migrating-overview

I did that because my goal initially was to export to bitbucket, which I did. Then I changed my mind and decided that Github would be better for a community project like this. So then, since it was already in a Git format it was simple from within the Github website to select "import" and just point to the Bitbucket URL.

Note: Those instructions work best if executed from a linux machine. I quickly spun-up an Ubuntu VM to do this.

[LogMAN]

It is good to finally see some movement in OpenG.
Git and GitHub are also the right choices (Bitbucket would probably also work). Even novice programmers will be able to participate this way. 👍

That said, the current repository has a few problems:

• No tags
• No branches
• All projects in one repository
• Changed commit messages (the links in the commit messages are non-functional)

It is possible to transform an SVN repository into a Git repository while maintaining all tags and branches and updating the committers (because Git uses email addresses and SVN doesn't). Here are some instructions I used in the past for such jobs (instructions are for Linux of course): https://epicserve-docs.readthedocs.io/en/latest/git/svn_to_git.html

For OpenG this process is a bit more complex because of the way the repository is structured (i.e. tags inside folders for each project), so the scripts must be adjusted to take this into account.
I also suggest splitting the project into multiple repositories during this process to improve maintainability (unless there is a reason why it needs to be one repository).

I could prepare the scripts to automate this process if you are interested.

[Michael Aivaliotis]

Well, I was hoping someone would continue the discussion, so great! We can redo the conversion. But is it really that critical to migrate the history intact? I question the need for that. If not we can start fresh.

• Authors - The conversion I went through had the ability to add email addresses to the author names. I just don't know what the email addresses are for the authors in Sourceforge. I've attached the author list if anyone wants to help flesh out the email addresses.Then we can rerun the conversion.
• Branches and tags. - I looked at the SVN repo and there's only one branch, SVN does not due branches well, so I'm not surprised nobody used that feature. There are a few tags and those are very old, circa 2007. Not sure if anybody cares about those. It seems the tagging procedure (if any) was dropped long time ago. You should be tagging with every release but that does not appear to have happened.
• One repo - The original SVN was a single repo, this is why i kept it the same. The conversion is a lot simpler. Breaking up a single SVN repo into multiple GIT repos and keeping the history intact seems complicated if you have commits that include files that cross library virtual boundaries. If you can think of a way to do this, that would help.
• Commit messages. - The commit messages are all there. It's the URLs inside the messages that are not pointing correctly, but I'm not sure what they should be point too and how to fix that during the conversion. Also, what if SourceForge changes the URL structure later? Again, is this important?

Well, this is a good starting point. The alternative is to start fresh and create multiple GitHub repos, with the latest revision of the source. Then the SVN repo can be an archive if anyone wants to get at it.

I welcome your help if you can create scripts to solve some of the above problems.

 

authors.txt

[LogMAN]

59 minutes ago, Michael Aivaliotis said:

Authors - The conversion I went through had the ability to add email addresses to the author names. I just don't know what the email addresses are for the authors in Sourceforge. I've attached the author list if anyone wants to help flesh out the email addresses.Then we can rerun the conversion.

The simplest way is to add "@users.noreply.sourceforge.net" to make them valid addresses that don't collide with any existing GitHub user (GitHub and SF both support these). It should be sufficient for this task. Here is my version: https://gist.github.com/LogMANOriginal/c4109873a5d524387d3fb46f5b83aa0a

59 minutes ago, Michael Aivaliotis said:

Branches and tags. - I looked at the SVN repo and there's only one branch, SVN does not due branches well, so I'm not surprised nobody used that feature. There are a few tags and those are very old, circa 2007. Not sure if anybody cares about those. It seems the tagging procedure (if any) was dropped long time ago. You should be tagging with every release but that does not appear to have happened.

I agree. From what I can tell the reason it didn't work correctly is because each project has its own branch/tag subfolder. None of the standard systems support these (including git-svn). Find below a solution that can handle them.

59 minutes ago, Michael Aivaliotis said:

One repo - The original SVN was a single repo, this is why i kept it the same. The conversion is a lot simpler. Breaking up a single SVN repo into multiple GIT repos and keeping the history intact seems complicated if you have commits that include files that cross library virtual boundaries. If you can think of a way to do this, that would help.

Done, see below.

59 minutes ago, Michael Aivaliotis said:

Commit messages. - The commit messages are all there. It's the URLs inside the messages that are not pointing correctly, but I'm not sure what they should be point too and how to fix that during the conversion. Also, what if SourceForge changes the URL structure later? Again, is this important?

Sorry, my explanation wasn't very clear. My point is, that the original commit message actually doesn't have these URLs inside (see SF vs. GitHub). It was probably added during the Atlassian conversion process.

59 minutes ago, Michael Aivaliotis said:

The alternative is to start fresh and create multiple GitHub repos, with the latest revision of the source. Then the SVN repo can be an archive if anyone wants to get at it.

I welcome your help if you can create scripts to solve some of the above problems.

Here you go: https://gist.github.com/LogMANOriginal/fa1e59703c41e27758bcb935f15bea21

The script must be placed in an empty folder next to 'authors-transform.txt'. It must be marked as executable (runs in bash, not sh). svn and git must be installed of course. Then start the script via './openg_import.sh' and it should work (unless it breaks of course).

It does a few things:

• Lookup all projects, branches and tags
• Import all projects into dedicated Git repositories
• Cleanup the Git repository (i.e. tags are imported as branches, so they need to be fixed)

The output are 39 bare git repositories, one for each project with full history, branches, tags and commits. Enjoy 😉

Edited October 3, 2019 by LogMAN

[Michael Aivaliotis]

Thanks @LogMAN. After I get paying work done, I will try this out. For anyone else following along. If you are currently an OpenG developer ( @Rolf Kalbermatter?) and want to be added to the OpenG team on GitHub, send me a PM.

[_Mike_]

Great news @Michael Aivaliotis! @LogMAN thanks for supporting the process and publishing transformation code. 

Should we then wait with potential forking an merge requests for all the git-housekeeping of the repo?

[Rolf Kalbermatter]

13 hours ago, Michael Aivaliotis said:

Thanks @LogMAN. After I get paying work done, I will try this out. For anyone else following along. If you are currently an OpenG developer ( @Rolf Kalbermatter?) and want to be added to the OpenG team on GitHub, send me a PM.

Well I happen to have committed code in the last weeks to the lvZIP library including yesterday evening. Do I have to redo that to get it in the new GitHub stuff?

And I'm still busy trying to add support for symlinks and utf8 filenames to the lvZIP library. The basic work to support this is mostly done. Just need to properly integrate it in the lvZIP library and then do some corner case testing on the non-mainstream platforms like Mac OSX, Pharlap and VxWorks. MacOSX because it is different to Unix in some ways and Pharlap and VxWorks since they do not support symlinks and multibyte encoding at all.

My SF account name is labviewer.

Edited October 4, 2019 by Rolf Kalbermatter

[LogMAN]

2 hours ago, Rolf Kalbermatter said:

Do I have to redo that to get it in the new GitHub stuff?

I suggest informing all active contributors, that way everyone involved has a chance to push their changes before import.

2 hours ago, Rolf Kalbermatter said:

And I'm still busy trying to add support for symlinks and utf8 filenames to the lvZIP library.

The import script supports branches. If you upload your current status as a branch (inside the /branches/lvzip/ folder), it will be detected and imported. Once the repository is uploaded to GitHub you can simply continue working on the branch using Git.

2 hours ago, Rolf Kalbermatter said:

My SF account name is labviewer.

You have a chance to match it with your GitHub account (see 'authors-transform.txt'). This is only possible during transformation, however.

[Michael Aivaliotis]

@Rolf Kalbermatter, I will be redoing the export today based on the new scripts, if they work. So if you committed code recently, it will get included. I think you are the only one working on any code at the moment. I will add you as a collaborator on the repo so you don't need to fork. Once the conversion is complete:

• Make a branch on the git repo.
• Checkout the branch
• Do a file diff between your local SVN and the Git branch.
• Copy over the changed code from SVN to Git.
• Commit the branch.

@LogMAN, unfortunately GitHub does not have a good way to group related repositories, so they visually seem to belong together. The only distinction is organizations. There's the Project feature, which I think just helps with development workflows. Having numerous OpenG repositories will make this a bit messy. But I guess there's no way around that. It seems GitLab and Bitbucket have a slightly better approach with this.

[JKSH]

3 hours ago, Michael Aivaliotis said:

GitHub does not have a good way to group related repositories, so they visually seem to belong together... Having numerous OpenG repositories will make this a bit messy.

A poor man's way of doing that is to prefix all of the repository names with "OpenG". Users can then use the "search" feature within the Organization to find all repos related to OpenG. For example, https://github.com/ni?q=niveristand vs. https://github.com/ni

There's also GitHub Topics, although it crosses organization boundaries: https://help.github.com/en/articles/classifying-your-repository-with-topics

• e.g. https://github.com/search?q=topic%3Averistand+org%3Ani

[Michael Aivaliotis]

@LogMAN, Is it possible to remap a source forge username to a GitHub username?

[Michael Aivaliotis]

I've started adding repos:

https://github.com/Open-G

Edited October 9, 2019 by Michael Aivaliotis
Changed repo URL

[LogMAN]

@Michael Aivaliotis You are right, organizations are the only way to group projects. Just as @JKSH said, a prefix is very helpful in finding things.

On 10/5/2019 at 9:00 PM, Michael Aivaliotis said:

@LogMAN, Is it possible to remap a source forge username to a GitHub username?

On 10/5/2019 at 9:43 PM, Michael Aivaliotis said:

I've started adding repos:

https://github.com/Open-G

Looks like you figured it out

[Michael Aivaliotis]

3 minutes ago, LogMAN said:

Looks like you figured it out

Sort of. I think Github uses the email address, not the username. I've asked @Rolf Kalbermatter for his GitHub email, because it didn't work with his username.

I decided to leave the repos under GCentral and use the OpenG prefix as suggested.

[LogMAN]

1 minute ago, Michael Aivaliotis said:

I think Github uses the email address, not the username.

They do: https://help.github.com/en/articles/setting-your-commit-email-address

[Michael Aivaliotis]

I've migrated most of the code to Github.

[Porter]

Thanks @Michael Aivaliotis for taking the time to do this.

[jacobson]

I noticed all of the repos have their own ToDo.txt. Many are empty but for those that are not, you might want to create an issue for each item.

I don't know if you can create a wiki for an organization but I would also want to be able to find information on how to contribute and some specifics like what LabVIEW version is being used for development, expectations on testing, and build/release process.

[Michael Aivaliotis]

9 minutes ago, jacobson said:

I noticed all of the repos have their own ToDo.txt. Many are empty but for those that are not, you might want to create an issue for each item.

I don't know if you can create a wiki for an organization but I would also want to be able to find information on how to contribute and some specifics like what LabVIEW version is being used for development, expectations on testing, and build/release process.

Good points. You can't create a wiki for an organization. I've setup an OpenG team, but even there, no wiki support. Limitations... However, each repo has its own wiki. Which is good for content related to that specific repo. It's possible now with separate repos, to have each one have it's own LV version and build/release process if needed. However, if not, then each wiki can simply have a link to some global page that covers the overarching policies. This could be on LAVA, but I think the LabVIEW Wiki could provide support here. There could be a single page there covering everything, which is also community editable.

Creating issues can be done by anyone. So if you want to do that, go for it. If they are invalid, they can be closed later. We should also search here son LAVA and create issues for stuff reported by people in the past.

[JKSH]

34 minutes ago, Michael Aivaliotis said:

You can't create a wiki for an organization.

You can create a GitHub-hosted website for an organization: https://pages.github.com/

[Michael Aivaliotis]

6 minutes ago, JKSH said:

You can create a GitHub-hosted website for an organization: https://pages.github.com/

Cool. I'll look into it.

[LogMAN]

Here are some common files that could be helpful:

• The contribution guidelines are shown for pull requests: https://help.github.com/en/articles/setting-guidelines-for-repository-contributors
• LabVIEW versions, build instructions and the release process are commonly placed in a README file: https://help.github.com/en/articles/about-readmes
• The license can be managed in a similar fashion: https://help.github.com/en/articles/licensing-a-repository

All of these files, if they exist, add to the GitHub experience. For example: https://github.com/microsoft/vscode

[Michael Aivaliotis]

10 hours ago, LogMAN said:

All of these files, if they exist, add to the GitHub experience

I agree.