SVN Repository Stucture

[Shaun Hayward]

Hi All,

I know that a lot of you guys here are already using svn with LabVIEW, and I am currently in the process of trying to sort out a company wide roll out.

We are an integration company with 3 geographical locations (and are planning on having a server in each). However, we are trying to work out whether each server should have, for example, a separate repository for each customer we work with, or one single repository per site.

Amount the many thoughts we had were the following (and these are all maybes):

* having one per site could facilitate reusable components (ie a previous project carried out with a different customer could be branched) and make updating reusable components with applications easier

* having one per site could be more risky in terms of corruption (although backups will be made)

* Would there be any performance hit of having everyone working on the same repository (and would the associated large revision numbers be a problem)

* Are there any other issued affecting one system or the other that could be relevant?

How do you all split up your repositories? and do you have any feed back on our thoughts?

Any help is greatly appreciated,

Shaun

[PJM_labview]

Hi Shaun,

Personally, I do not see the benefit to have one server in each location. I am assuming that you will have each IT department maintain each server location. If you really want to be in charge on the server (as opposed to use svn hosting) one will do just fine. You can have your user accessing the server over https.

QUOTE

Amount the many thoughts we had were the following (and these are all maybes):

* having one per site could facilitate reusable components (ie a previous project carried out with a different customer could be branched) and make updating reusable components with applications easier

I am not sure that this is the best way to promote reuse. Reuse components should be accessible to every developer (ie from the palette) when they are working on a project.

QUOTE

...a separate repository for each customer we work with, or one single repository per site.

In term of repository; you should consider having only one. You can have granular control over who access which part of the repository through defining the access right.

QUOTE

* Would there be any performance hit of having everyone working on the same repository (and would the associated large revision numbers be a problem)

I never noticed any performances issues, but I don't think I ever worked on one with possibly more than 10 concurrent access.

Good Luck (and welcome to LAVA)

PJM

[Shaun Hayward]

Thanks for such a prompt reply! and thank you for the welcome

The reason we wanted to keep the servers for the different sites is that we do not have the speediest connections between sites, and therefore when you are in your home office, you would be able access the repositories at the full speed of the local network, regardless of whether your office is the HQ. Also, as far as concurrent users go, we a little over 20 engineers right now (and are trying to find more) so we would want something that is scalable for the future.

I know that pallets are good way for reusable VIs, but in this instance we were more thinking about application / system frameworks - we have the same basic software package on several systems and just pick and choose the modules that are appropriate before customizing it to suit the individual customer.

Have you ever had any reliability issues with svn? (especially with the all-eggs-in-one-basketrepository approach)?

QUOTE (PJM_labview @ Sep 19 2008, 01:06 PM)

Hi Shaun,

Personally, I do not see the benefit to have one server in each location. I am assuming that you will have each IT department maintain each server location. If you really want to be in charge on the server (as opposed to use http://expressionflow.com/2008/09/17/subversion-hosting-a-simple-path-to-labview-source-code-control/' target="_blank">svn hosting) one will do just fine. You can have your user accessing the server over https.

I am not sure that this is the best way to promote reuse. Reuse components should be accessible to every developer (ie from the palette) when they are working on a project.

In term of repository; you should consider having only one. You can have granular control over who access which part of the repository through defining the access right.

I never noticed any performances issues, but I don't think I ever worked on one with possibly more than 10 concurrent access.

Good Luck (and welcome to LAVA)

PJM

[jdunham]

QUOTE (Shaun Hayward @ Sep 19 2008, 10:34 AM)

The reason we wanted to keep the servers for the different sites is that we do not have the speediest connections between sites, and therefore when you are in your home office, you would be able access the repositories at the full speed of the local network, regardless of whether your office is the HQ. Also, as far as concurrent users go, we a little over 20 engineers right now (and are trying to find more) so we would want something that is scalable for the future.

I know that pallets are good way for reusable VIs, but in this instance we were more thinking about application / system frameworks - we have the same basic software package on several systems and just pick and choose the modules that are appropriate before customizing it to suit the individual customer.

Have you ever had any reliability issues with svn? (especially with the all-eggs-in-one-basketrepository approach)?

I think it would be a disaster to try to maintain separate SVN repositories at different locations if they shared data. With all the time that you will waste, it would be better to invest in more network bandwidth. Also remember you are not on the network all the time. You only really hit the network a few times per day, and SVN's deltas make network transfers very efficient.

You might want to think about separate repositories for each customer in case there are issues with customer proprietary data, since it would be easy to archive a project and remove it from your server some time after the contract ends. But I would still run them all on the same server machine. However, Philippe is right that everything will probably be easier in a single repo.

Don't be afraid of big numbers, they won't hurt you.

Overall, I wouldn't worry abou the reliability/scalability issues. Your team is much smaller than many others which are successfully using SVN. Good luck.

[Shaun Hayward]

QUOTE (jdunham @ Sep 19 2008, 02:10 PM)

Don't be afraid of big numbers, they won't hurt you.

...that's what they always say... :ninja:

...seriously though, I guess it was more a question of does the system cope with very high numbers of revisions, or does it start to break down at a certain point, but looking at yours and PJM's answers, I'm getting the feeling that it is not a problem

QUOTE (jdunham @ Sep 19 2008, 02:10 PM)

I think it would be a disaster to try to maintain separate SVN repositories at different locations if they shared data. With all the time that you will waste, it would be better to invest in more network bandwidth. Also remember you are not on the network all the time. You only really hit the network a few times per day, and SVN's deltas make network transfers very efficient.

Thats a good point, I'll have to talk to the upper echelon's about their current plan with the servers...

QUOTE (jdunham @ Sep 19 2008, 02:10 PM)

You might want to think about separate repositories for each customer in case there are issues with customer proprietary data, since it would be easy to archive a project and remove it from your server some time after the contract ends. But I would still run them all on the same server machine. However, Philippe is right that everything will probably be easier in a single repo.

I'm with you - no matter what level the repo's were going to be at, we were definitely going to put them on the same server (whether site dependant or not) - and I had not considered the ease of archive possibilities, thats a definate + for repos per customer.

QUOTE (jdunham @ Sep 19 2008, 02:10 PM)

Overall, I wouldn't worry abou the reliability/scalability issues. Your team is much smaller than many others which are successfully using SVN. Good luck.

Thanks

[Chris Davis]

QUOTE (Shaun Hayward @ Sep 19 2008, 03:02 PM)

...that's what they always say... :ninja:

...seriously though, I guess it was more a question of does the system cope with very high numbers of revisions, or does it start to break down at a certain point, but looking at yours and PJM's answers, I'm getting the feeling that it is not a problem

I'm at over 3200 on a single repository and have had no problems with svn.

[crelf]

QUOTE (Shaun Hayward @ Sep 19 2008, 04:02 PM)

I guess it was more a question of does the system cope with very high numbers of revisions, or does it start to break down at a certain point.

Maybe it does, but after 10's of thousands of revisions here, we haven't noticed a slow down.

[Shaun Hayward]

QUOTE (crelf @ Sep 19 2008, 05:37 PM)

Maybe it does, but after 10's of thousands of revisions here, we haven't noticed a slow down.

...or a repository per project, or even one single repository for everything.

yeah, its hard to find out what would be the best as I think in reality they all could work, although we have decided that the ability to a customers earlier project to form the basis of a new project is too valuable to have 1 repo per project. What do you do?

[eaolson]

QUOTE (Shaun Hayward @ Sep 19 2008, 04:43 PM)

yeah, its hard to find out what would be the best as I think in reality they all could work, although we have decided that the ability to a customers earlier project to form the basis of a new project is too valuable to have 1 repo per project.

Why not? You could always just create a branch of the first project for the second project. Or, you could also export a copy of the first project and use that as the starting point for your second.

[Mellroth]

QUOTE (Shaun Hayward @ Sep 19 2008, 11:43 PM)

...although we have decided that the ability to a customers earlier project to form the basis of a new project is too valuable to have 1 repo per project. What do you do?

I would go for one repository per project (or closely related projects, e.g. per customer), using svn:externals to include revisioned parts from other projects/repositories.

By having one repository per project, each project can set the layout exactly as they need. Thus making the layout within each repository consistent.

I also feel more safe, in terms of access control, if the customer projects are separated from other customers as well as our internal developments.

/J

[Tomi Maila]

QUOTE (JFM @ Sep 20 2008, 10:26 AM)

I would go for one repository per project (or closely related projects, e.g. per customer)

I second JFM here. It is commong that when working on a project, you end up developing reusable code. I think it is a good practice to separate reusable parts of project to their own repositories, when they mature. Then you should build reusable libraries some sort of installable packages and install them into into vi.lib or user.lib to be used in projects. I think currenlty http://jkisoft.com/vipm/' rel='nofollow' target="_blank">JKI Soft VIPM is the best tool for managing reusable libraries. This way you can limit your customers or partners not to have access to the source code of all your reusable libraries but only to the source code relevant to their project. This also forces you to think more of the reusability of software modules and you'll end up working more efficiently in the long run.

[Shaun Hayward]

Thanks to everyone for such useful input on this thread - It definately seems that there is no hard and fast advantage of one method over the other, but many good talking points have been raised. We have our next planning meeting mid October so I'll be sure to let you all know what (and why) we end up going with

Shaun

[Mellroth]

QUOTE (Shaun Hayward @ Sep 25 2008, 04:22 PM)

...there is no hard and fast advantage of one method over the other, but many good talking points have been raised...

One thing that convinced me to have separate repositories was that the revision graphs documentation states, that it will fetch all log messages from the repository root, to generate the graph. It also says that it will decrease in performance with higher count of revisions. Therefore separate repositories are much more likely to stay reasonable in size, and in turn be more responsive in terms of the revision graph output.

You can find a similar problem in ClearCase when you let your VOBs grow, having to many label types within one VOB slows down the ApplyLabel operation badly.

(I recently did a comparison between CC and SVN/TSVN for my company)

/J

[lavezza]

QUOTE (JFM @ Sep 25 2008, 11:26 AM)

One thing that convinced me to have separate repositories was that the revision graphs documentation states, that it will fetch all log messages from the repository root, to generate the graph. It also says that it will decrease in performance with higher count of revisions. Therefore separate repositories are much more likely to stay reasonable in size, and in turn be more responsive in terms of the revision graph output.

You can find a similar problem in ClearCase when you let your VOBs grow, having to many label types within one VOB slows down the ApplyLabel operation badly.

(I recently did a comparison between CC and SVN/TSVN for my company)

/J

JFM,

Can you share your findings from your comparison. We are in a situation where our company is trying to get us to move to CC (our corp standard), but we'd rather move to SVN. We're using SourceSafe right now. I'd be interested in what you found out, especially any LabVIEW specific items that we should be aware of.

Thanks,

Pat

[Mellroth]

QUOTE (lavezza @ Sep 26 2008, 07:48 AM)

...Can you share your findings from your comparison...

I can share as much as I remember right now, and I'll see if I can post some more info next week when I'm back in office.

Our group have happily used ClearCase for the last 12 years, but are now moving over to SVN to have the same tools all over the company.

What I did was to see if the methodology we used with ClearCase could be transferred to SVN without too much problems.

To summarize; Yes! but with some changes in how we use labels and also some changes to our CM-plans

Have a nice weekend

/J

The Pro's and Con's I remember right now;

Pro' Clearcase

* We really like the ConfigSpecification files, since they make it easy to backup to a specific revision, and still keep track of all linked module versions

* Very Robust, we have never had any problems whatsoever

* Dynamic views can be great, once you know how to use them, e.g. I can setup a specific view for a build. At the same time I apply my cs-file, all other users will see the same code as I do.

* Labels are easy to apply, and easy to find. The user don't have to specify where to store a specific label, c.f. tagging in SVN

* Very easy to switch between labeled versions, as this is merely a change in the cs-file

* Easy to give directions to a fellow worker, on which versions we have are currently using

* Very easy to check what revision and/or rule (e.g. label) a file/folder is loaded with

* One installation gives you all tools you need, Browser, Context support, Label browser, etc. etc.

* Easy to check which versions another users is actually using, since we can see other users cs-files.

Con's clearcase

* Slooooow, at least compared to SVN. (pretty close to a Near Death Experience, as I think of it )

* It is slow both in the CC-browser and in the Windows environment.

* Costs a lots of money for a license

* If you have several projects loaded in a Snapshot view, they will all use the same version of linked modules.

* if network goes down, and you are working in a snapshot view, you might have trouble just browsing the folders (depends on your settings)

Pro's SVN

* Fast

* Very light weight

* Easy to introduce to customers

* Easy to learn

* Good interface (Tortoise SVN)

* Easy to setup a simple repository

* Different projects in a working copy, can use different versions of the same module (specified in svn:externals)

Con's SVN

* There is not one tool that gives you all the functionality that you get from ClearCase

* Depends a lot on that rules are followed by the users, unless you create server hooks to check user actions (but that is not in the base install), E.g. tagging/Branching is merely copies, so it is up to the user to select where a specific tag is stored. If a user accidentally stores a Tagged version in the wrong place, others might having a hard time finding it.

* Not as easy as CC to find what tag a specific file belongs to, since this has to be determined from the path

* Linked modules are linked with absolute paths :headbang: , which means that it is not just to move a repository from one server to another (as all svn:externals must be fixed)

* You'll have to install a few software packages, at least if you need the CLI support as well as the support from TSVN

* Doesn't handle renames as well as ClearCase, a rename in SVN is a delete-add operation, which means that revision history is lost (I might be wrong on this)

[crelf]

QUOTE (JFM @ Sep 26 2008, 12:06 PM)

Con's SVN

* Doesn't handle renames as well as ClearCase, a rename in SVN is a delete-add operation, which means that revision history is lost (I might be wrong on this)

You are - history is presevered (although the way Tortoise shows the operation could lead you to thinking it was a delete/add operation).

[PJM_labview]

QUOTE (JFM @ Sep 26 2008, 09:06 AM)

* You'll have to install a few software packages, at least if you need the CLI support as well as the support from TSVN

If by CLI you mean Command Line Interface, then do you know that you can automate TSVN by passing command to tortoiseproc.exe?

See http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-automation.html' rel='nofollow' target="_blank">here for more info.

PJM

[Mellroth]

QUOTE (crelf @ Sep 26 2008, 07:26 PM)

You are - history is presevered (although the way Tortoise shows the operation could lead you to thinking it was a delete/add operation).

Sorry, I meant copy-delete, and I was wrong in that the history is preserved, but there are still issues with the rename operation.

http://subversion.tigris.org/issues/show_bug.cgi?id=898

QUOTE (PJM_labview @ Sep 26 2008, 07:33 PM)

If by CLI you mean

C

ommand

L

ine

I

nterface, then do you know that you can automate TSVN by passing command to tortoiseproc.exe?

Yes I knew that, but as I understand, not all SVN commands are exported in tortoiseproc.exe (and will most likely never be).

Apart from that, I prefer to have the CLI commands to be the same on all platforms.

Don't get me wrong, I voted for a switch to SVN, but I think all tools (CC as well as SVN and others) have some problems/issues.

Depending on what platform you are used to, you will probably see different problems, just because you have a certain work flow in that old tool.

/J