Backup Utility, Software, Method, etc

[ASTDan]

Hello,

I was wondering what people use for their backup method, software, etc.

Right now I use the Windows Backup utility. It works for me right now, however I am moving over to SVN. I am wondering if this will cause a problem with my repositories?

What do other people use for their backup method? What software do you use? Why do you use it? What are the pros and cons? What backup medium do you use and why? Are there any backup best practice guidelines?

Thanks

Dan

[Justin Goeres]

QUOTE (ASTDan @ Nov 3 2008, 07:29 AM)

I was wondering what people use for their backup method, software, etc.

Right now I use the Windows Backup utility. It works for me right now, however I am moving over to SVN. I am wondering if this will cause a problem with my repositories?

What do other people use for their backup method? What software do you use? Why do you use it? What are the pros and cons? What backup medium do you use and why? Are there any backup best practice guidelines?

As to whether a backup system will cause problems with your repositories, the answer is not if you do it right .

I run my personal (non-JKI) SVN system on an Ubuntu Linux box in my office. For backups I use http://backuppc.sourceforge.net/' rel='nofollow' target="_blank">backuppc. It backs up all the data I need backed up on my server, and also backs up my MacBook Pro via an SSH tunnel (which, in principle, would even work over the internet when I'm on travel).

On the Linux box, one of the directories I point backuppc to is the one where SVN keeps its database. SVN is really just a frontend to a database, and the database is where the filesystem of your repository lives. As such, if you make sure to backup the whole database then you can also just restore the whole database in the future after a catastrophe. Of course, I also back up /etc, which is where SVN keeps its config. So basically, my advice (regardless of platform) for backing up your SVN repository is to (A) make sure you back up the whole repository, which is pretty easy because it's contained under a single directory, and (B) also back up whatever directory contains your configuration for SVN, so if you have to recover you can reset to a good config, too (otherwise you might have to remind SVN where its repo is, who its users are, etc.).

It goes without saying (but I'll say it anyway ) that once you start backing up your repository (which you should!) you do not need to backup your checkouts. In fact, I personally don't backup my checked-out SVN directories specifically to encourage myself to commit frequently .

EDIT: I do not know much about using backuppc with Windows machines. In my case, any files that I need backed up in Windows are stored on network shares from the fileserver, which is backed up. Backuppc supposedly works with Windows, but I've never bother to try it .

[Justin Goeres]

I wanted to answer the rest of your question more fully .

QUOTE (ASTDan @ Nov 3 2008, 07:29 AM)

What do other people use for their backup method? What software do you use? Why do you use it? What are the pros and cons? What backup medium do you use and why? Are there any backup best practice guidelines?

When I was working independently, I took some time to figure out a backup solution for my business. It's really worth the peace of mind to do it right, and it's not that hard considering the value of the information you're protecting (whether it's personal financial data & family photos or company IP). Here's a summary of what I did. This served me well while I was in my own business, and now is doing an equally bang-up job for my personal & family data.

Level 1 / Windows Workstations: All project work (this includes work projects and any "personal projects" where rev control is useful) is checked out from an SVN repository on the Server. Non-project work (like Invoices, Expense receipts, etc.) is stored on network shares provided by the Server, and are mirrored locally using the built-in Windows Offline Files feature (this makes it reasonably easy to carry work with you when you travel, and sync changes when you come back, as well as providing redundancy).

Level 1 / Mac & Linux Workstations: All project work is checked out from SVN as on Windows, and all systems use the same network shares as the Windows machines. However, because Windows Offline Files doesn't exist for the Mac, some non-SVN user documents are stored locally. All user data & system configuration data is backed up nightly to External Backup (see below) via SSH.

Level 2 / Server: A Linux box (but it could be any OS) providing an SVN repository and SAMBA (SMB) network shares. Each user is provided with a "home" directory, and I also have "family" and "public" shares with permissions set for family members and visitors/friends. The main data storage on this machine is a 600GB RAID5 array (3 drives of 300GB each). SVN & Windows Offline Files provide redundancy for the data served to the workstations, and the RAID5 array provides resistance against data loss due to drive failure. FYI, I built this machine for about $600 two and a half years ago, and that even includes buying a super-fancy case (which was totally worth the money). It took a few days to get it built & working, but I learned a lot doing it. This machine (and the network router) are plugged into a small UPS, that provides about 20 minutes of power in case of blackouts.

Level 3 / External Backup: Data on the Server itself is backed up (using backuppc) to an external Firewire drive attached to the server. All user data, fileshares (including SVN repo) & OS configuration data is backed up each night. 7 previous incremental backups and the last 2 full backups are stored. This provides another level of redundancy for the data on the server, in the unlikely event that the RAID5 would ever fail. It also provides a brief historical archive, to guard against accidental destruction of files that aren't under SVN control.

Note that this system provides 3x data redundancy (2x for non-SVN data on the Mac ) for all user & project data. That's the critical data you create that, if you lost it, would be right up there with having your car stolen. Except that you can probably buy another car. You can't buy replacements of your wedding photos or your hundred-year-old great grandma. There should really probably be a Level 4 / Off-site, but for the time being I lock the doors of my house and just pray it doesn't burn down. And if it burns down, I'll have other problems equal to or larger than granny's missing pictures.

[ASTDan]

QUOTE (Justin Goeres @ Nov 4 2008, 09:46 AM)

I wanted to answer the rest of your question more fully .

Wow thank you for sharing you journey. I just have an external drive I back up all my critical data and store it in a waterproof/fireproof safe.

You have given me a lot to think about. :worship:

[Justin Goeres]

QUOTE (ASTDan @ Nov 4 2008, 08:00 AM)

Wow thank you for sharing you journey. I just have an external drive I back up all my critical data and store it in a waterproof/fireproof safe.

You have given me a lot to think about. :worship:

No problem. Data backup has become sort of a pet issue for me, because I think far too many people are really living on the edge and taking risks with their important data. It's really, really worth the time to set up a backup system. Think of it like buying insurance, but for your data. And if you do it right and get something reasonably automated, you can stop worrying about it. That's the real hidden benefit .

Actually, the fireproof safe is a good idea! I have a safe already and could buy a new drive. I think I could create another backuppc task to another drive, and it would only run when that drive is connected. Then I'd just have to plug the new drive in every month or so, leave it overnight, and put it back in the safe. :ninja:

EDIT: BTW, when I started making notes & buying parts for this, I got a little bit of the old "OMG what's this new contraption?" thing from my wife. I sold her on it entirely based on the we never have to worry about data backup again angle, shortly after she lost some of her old PhD. work in a USB key accident .

[ASTDan]

You have peaked my interest in a server.

What are some good resources to look into this? I know absolutely nothing about a server, so I need something on my level. We're talking single cell organism here.

Dan

[Chris Davis]

I wanted to comment on this thread as well before it went away. Justin mentioned that he doesn't have an "off-site" level of backup protection. I found a solution to the off-site level that I think meets most personal needs. First off you need to choose an off-site location. I choose a company called IXWebhosting (www.ixwebhosting.com) they provide web site hosting services with "unlimited

hard drive space, at a nominal cost (2 years of service is $100). I then setup a location on that web server that is not visible from the internet, only visible via FTP. I then use an FTP program called Transmit from Panic which provides me the capability to mirror directories on an FTP server. I use an Automator script that is triggered at various times during my work day to backup parts of my hard drive. This works best for me because when I and my wife are at work, and my kids are at school, no one is using my cable modem. This idea depends on having a broadband connection with a somewhat decent upload pipe. I have about 512kb avaliable to me, and when I make lots of changes to my family photos with iPhoto, it can take a couple of days to recover. This isn't acceptable in a work situation, but in a work situation, I'd be paying more for a better upload pipeline to the internet. As an added benefit the web site that IXWebhosting provides gives you lots of emails, ftp accounts, and lots of space to publish your own personal projects / web pages.

QUOTE (ASTDan @ Nov 4 2008, 01:56 PM)

What are some good resources to look into this? I know absolutely nothing about a server, so I need something on my level. We're talking single cell organism here.

On a simple level, a server is just another computer. Like Justin mentioned, you don't have to spend a fortune, but you'll want to make sure you know what you are buying. Most people who setup their own server run some form of linux on it. If you choose to run the program Justin mentioned, backuppc, you'll have to run some form of linux or unix. All of the major linux distributions have tried to make it very easy to make sure your hardware of choice runs fine with their operating system decisions. As such, they usually provide a "Live CD" which lets you try out a version of linux that you want without actually installing it onto the computer in question, everything runs from the CD. Makes a nice way to "try before you buy". Your choice of hardware will determine how much you spend, but deals like this computer and three of these hard drives would allow you to build a server with 3 500 GB hard drives in RAID 5 for around $310 before tax and shipping. That is assuming you have a spare CD drive laying around. As Justin also mentioned, there is a little bit of a learning curve when setting up your own linux server, but it can be good practice to learn the in's and out's of the setup process so that you have a good idea of how good your hardware will perform. Finding a place in your home to store this computer getting the appropriate power and ethernet connectivity could be another problem.

Justin's mention of backuppc has got me thinking about building my own server to take advantage of this program to perform incremental backups of my Music/Movie/TV Show and Picture selection.

[crelf]

QUOTE (ASTDan @ Nov 4 2008, 10:00 AM)

I just have an external drive I back up all my critical data and store it in a waterproof/fireproof safe.

Wouldn't it be sweet to have a waterproof/fireproof safe with a little PC in it that wirelessly did your backups? It could have a UPS to shut down nicely. The only issues would be how to get data and power to it, and heat dissapation...

[hooovahh]

QUOTE (crelf @ Nov 8 2008, 01:40 AM)

Wouldn't it be sweet to have a waterproof/fireproof safe with a little PC in it that wirelessly did your backups? It could have a UPS to shut down nicely. The only issues would be how to get data and power to it, and heat dissapation...

Ok so lets say you have a waterproof/fireproof safe, which has a thermally conductive but not electrically conductive liquid in it, then submerge the PC into it so the heat will be dissipated through the water, and then the heat could dissipate through the metal safe. Then safe may get too hot and start a fire...or it might not get that hot. I've heard about people submerging their pc into this kind of fluid and it seems to work well.

Also if it is fireproof/waterproof I assume the casing would be very heavy stuff, not sure how well wireless signals would get through it. I guess revert to some slow wireless signal with lots of redundency. But yeah getting power to it may be tricky...I guess it would just be easier to have an off-site backup, the odds of both places buring/flooding at the same time would be minimal.

[crelf]

QUOTE (hooovahh @ Nov 10 2008, 04:02 PM)

...then submerge the PC into it so the heat will be dissipated through the water...

So, intentionally flooding a flood-proof safe

[Chris Davis]

QUOTE (crelf @ Nov 8 2008, 01:40 AM)

Wouldn't it be sweet to have a waterproof/fireproof safe with a little PC in it that wirelessly did your backups? It could have a UPS to shut down nicely. The only issues would be how to get data and power to it, and heat dissapation...

I thought of that too, but when I talked to a fireman buddy of mine, he said that the inside of a fireproof safe would get to hot during a fire for plastic to really survive. This http://www.networkgarage.com/2005/10/fireproof_safe_backup_1.html' rel='nofollow' target="_blank">site seems to lean that way too. Offsite backup still seems like the way to go to me, but current broadband connections can hamper people from taking advantage of the capability of offsite backups.

[ASTDan]

QUOTE (crelf @ Nov 8 2008, 01:40 AM)

Wouldn't it be sweet to have a waterproof/fireproof safe with a little PC in it that wirelessly did your backups? It could have a UPS to shut down nicely. The only issues would be how to get data and power to it, and heat dissapation...

Here is a link to a fireproof/waterproof external hard drive.

http://www.keenzo.com/showproduct.asp?M=SE...5107&ref=GB

QUOTE

Offsite backup still seems like the way to go to me, but current broadband connections can hamper people from taking advantage of the capability of offsite backups.

I would agree that off site would be safer from fire or some other natural disaster. However the main failure mode I am worried about is my hard drive crashing. In my opinion recovering from an external hard drive or a personal server would be faster/eaiser than an off site backup facility in the event of a hard drive crash.

It is also comforting to have my data where I can see it. Kinda like keeping your cash under your mattress.

[crelf]

QUOTE (ASTDan @ Nov 11 2008, 10:16 AM)

Here is a link to a fireproof/waterproof external hard drive.

Very cool! I wonder if it's upgradable? Can I replace the internal hard drive with something bigger?

[ASTDan]

QUOTE (crelf @ Nov 11 2008, 01:17 PM)

Very cool! I wonder if it's upgradable? Can I replace the internal hard drive with something bigger?

I don't know.

Just make sure if you go down this path make sure it is waterproof. Hopefully some good natured folks would put out the fire, and they use water.

When I was looking into this I found a lot of fireproof safes, but not a lot of water and fire proof safes.

[hooovahh]

I have several hard drives floating around my house which don't have a home. Most of them are IDE, and I've ran out of IDE channels on both my PCs. So my plan is to hook these drives up to my PC using an IDE to USB adapter once a month or so and do an incremental backup. After the backup is complete I can then put the drives into a safe location. A waterproof/fireproof safe would be nice, but for now their just in a lock box, so if the PC is stolen the data will still be safe.

The software I'm using for my backup is a simple one, a batch file that calls robocopy. It's a very useful command line utility. You basically say here's my source directory, here's my destination, and if files in the destination already exist don't overwrite them. It also does updates to files if they are the same name but different.

On top of an incremental backup I could setup some kind of Raid 1, but I've ran out of hard drives.

[crelf]

QUOTE (hooovahh @ Dec 1 2008, 03:01 PM)

The software I'm using for my backup is a simple one, a batch file that calls http://en.wikipedia.org/wiki/Robocopy' rel='nofollow' target="_blank">robocopy.

I don't know what OS(es) you need to support, but why not just use the default backup software that comes with your OS?

[mje]

My home system's backups are managed by a batch file with robocopy calls as well: for Vista that is an entire solution based exclusively on the tools provided by the OS. You can add in scheduling if desired, but I don't since I keep the backup disconnected from the system. Works wonderfully, and is even network-friendly should my ISP decide to ever stop sucking. My backups are intended to prevent hard disk failures only: they're all stored at home, so any disaster would leave me out of luck.

I can't say I believe any of this fireproof nonsense. In a fire, a disk will get really hot. Put it in a fireproof safe and it might not burn, but I don't see how it can be kept cool enough to maintain data integrity.

[hooovahh]

QUOTE (crelf @ Dec 1 2008, 03:24 PM)

I don't know what OS(es) you need to support, but why not just use the default backup software that comes with your OS?

I started looking at the NTBackup software that comes with XP, and the advanced features are pretty useful. It's just for me it was easier to write a 4 line batch file to backup some media folders. If I ever want to backup more OS features (like settings and such) I will definatly keep NTBackup in mind.

[ASTDan]

QUOTE (crelf @ Dec 1 2008, 03:24 PM)

I don't know what OS(es) you need to support, but why not just use the default backup software that comes with your OS?

I use the Windows Backup Utility. I like it.

[crelf]

QUOTE (ASTDan @ Dec 2 2008, 11:44 AM)

I use the Windows Backup Utility. I like it.

Me too - it comes with the OS and it does everything that I^* need, including scheduled backups.

^*I say this not as a representative of V I Engineering, Inc: I use the Windows backup utility at home - I have no idea what our IT department uses in the depths of their multiple server rooms...

[dhchmn]

just launched -- www.mediavaulthd.com/

[crelf]

QUOTE (crelf @ Dec 2 2008, 01:09 PM)

Me too - it comes with the OS and it does everything that I need, including scheduled backups.

Since I upgraded my media centre to Vista Home Premium (Vista media center is awesome, and it works great with my extender) the backup capabilities have been limited, so I'm now using Genie Backup Manager. It's very easy to use, has lots of features, and also integrates with an online vault (paid service).

[Phillip Brooks]

I started playing around with DropBox before Thanksgiving. While I was at my parent's house, I scanned some old family photos and uploaded them them via the web interface to my 'drop box'. I configured my home computer to mount DropBox in the "My Documents" folder and my wife asked me how I found time load the pictures into the computer when we got home

It's a very simple interface, and offers 2 GB storage for free. I haven't experimented with it yet, but they also support revisions for files. It is available for Windows, Mac and Linux, and

http://www.getdropbox.com

[Phillip Brooks]

I started playing around with DropBox before Thanksgiving. While I was at my parent's house over Christmas, I scanned some old family photos and uploaded them them via the DropBox web interface to my 'drop box'. I had already configured my home computer to mount DropBox in the "My Documents" folder and my wife asked me how I found time load the pictures into the computer when we got home

It's a very simple interface, and offers 2 GB storage for free. I haven't experimented with it yet, but they also support revisions for files. It is available for Windows, Mac and Linux.

http://www.getdropbox.com

I learned about DropBox from this blog.

[ASTDan]

Well it happened.

My hard drive crashed and I got to experience how good my backup strategy was.

I was able to recover all my data that I backed up on my external hard drive no problem. I used the windows back up utility.

The problem I did have was my laptop hard drive crashed while I was traveling. It was working intermittently so I was able to back up everything when I got back to my office, and recover.

What I learned is to have a backup strategy while traveling. So what I am going to do know is get a flash drive (32 GB or so) to back up to while I am traveling. An off-site storage solution would also work, however if you are in areas with limited access to the internet (which I was) what do you do?

I wanted to pass this use case along.

Dan