[Sean Webster]

Sean's Data Backup Guide

First off, data is not backed up up unless it is in 3 or more separate locations! (with one being offsite)

Backup is a pretty simple topic. You have your precious data such as photos, videos, documents, music, and even just program files and game saves on your PC and you don't want to lose it right? Well, if so then you should always backup of your data in some sort of way.

There are two categories of backup: system backup and file backup. A system backup is when you copy everything verbatim on the OS hard drive onto an external storage medium, including the operating system, applications and data. If the drive fails, the computer can be restored to its former state.

File backup copies only data files onto an external storage medium. If the drive fails, the OS and applications have to be re-installed.

I like to do a mix of both of these to cover my butt in case of anything. You will see what I do exactly in the following steps.

RAID is not a Backup solution

RAID is not designed to replace a proper backup solution, it's there to protect against drive failures with minimal to no down time. I don't understand why some people think that just because different RAIDs such as RAID 5 or RAID 6 can take a drive loss and still maintain data integrity that their data is backed up. It simply is not. Data needs to be in multiple places to actually be backed up, not just a single RAID volume in a single PC/location.

RAID arrays do NOT protect against:

• Boot Sector Corruption
• Human error (deleting files by mistake)
• RAID controller/software failure
• Fire, flood, or other calamity such as an EMP in your PC...
• Theft, hacker attack, or other offensive action
• Multiple disk failures and Unrecoverable Read Error
• Data corruption
• Virus'

However, using RAID IN your backup is totally different then using RAID AS your backup. A cool thing you can do is use a box with hot swap bays to hold your drives to easily swap out a drive if one fails for large arrays. Also, you can set up drives as hot spares which means they are just in the system's RAID array sitting waiting for a failure so it can automatically integrate into the array so you don't have to do it manually.

Getting started:

The way I see it there are three steps in this.

1. Organize and choose what you need to backup
2. Plan out our backup routine
3. Choose a program or programs that will help you accomplish your plan, configure as needed, and test your backup routine

Organize:

Okay, so to get started you should first make sure your data is nice and organized so everything will be easier to backup and track down throughout the process. I've seen countless systems where there are hundreds of folders all over the place and nothing is organized one bit. It is the most annoying thing to deal with for me when I have to backup a system as I am used to having all my stuff organized neatly.

The way I have my data set up does as follows:

• For the OS drive I have a 128GB SSD. On the OS drive is just my OS, drivers, and programs. No actual important data except for the program settings and what not.
• For the rest of my data, it is on various drives in my system. My secondary drive basically holds all my data, it is a RAID 0 array. I have a single folder on this drive. I call it "Main Data." Within it I have my main sub-folders.
  
  Secondary drive layout:
  • My Computer, "D:\" ->
    
    • "Main Data" folder ->
      
      • Backup (Main storage for OS backup images here so when I do my file backup my OS backup is backed up too.)
      • Desktop
      • Downloads
      • Dropbox (This also contains my Documents folder, Dropbox backups my important docs via cloud storage all the time.)
      • Miscellaneous (Just where I have old data, program installers, ISOs, etc.)
      • Music
      • Pictures
      • Videos
      • Saved Games
      • SkyDrive
      • Steam
• Next, I have my 1TB scratch disk for my temp storage and random stuff. I have the same basic layout as my secondary drive. I have a main folder called "Scratch," which I have branch out into other sub-folders for what ever I want.
• Next, I have my 256GB SSD for my Virtual Machine storage. I have the same basic layout as my other drives. I have a main folder called "VMs," which then branches out into other sub-folders for my virtual machine data.
• Finally I have my Backup drive in my system. This drive is where all the organization pays off. I have a main folder called "Backup. This folder contains a few different sub folders for my various backup needs.
  
  Backup drive layout:
  • My Computer, "Y:\" ->
    
    • "Backup" folder ->
      
      • Bookmarks (Bookmarks from FireFox in my Appdata folder sync to here)
      • Main Data (Main Data folder syncs to here)
      • VMs (VM folder syncs to here)
      • Scratch (Sratch drive folder syncs here)
      • Lightroom (All my lightroom settings in my Appdata folder sync here)
      • Sticky Notes (The sticky notes file in my Appdata folder saves here for backup)
      • VirtualBox (The ".VirtualBox" folder in my User folder syncs here to keep my settings backup)

How to plan out a good backup routine:

So now that you've seen how my system is organized go ahead and organized your stuff how you like and we can begin out next step, building a backup routine. I like to do a simple automated backup routine for my data. I don't ever have to manually backup my stuff ever again! No more copying, pasting, or deleting bunches of folders back and forth, just do a one time setup and I'm good to go from then on and my mind is at ease.

Backup vs Redundancy vs Archiving:
The technology overlap between backup, redundancy and archiving can often lead to confusion, but each has a different role to play in streamlining and safeguarding data. Backups essentially create a second copy of data at specific points in time, ideally keeping multiple historic copies. Redundancy establishes a straight copy of an entire system, ready to take over if the original system fails. Backup offers a certain level of redundancy, and redundancy a basic level of backup, but neither are stand-alone solutions.

Archiving makes a primary copy of selected data with the aim of retaining data in the long-term. Not all of the data contained in a backup will ultimately end up in an archive so archiving is rarely an adequate backup solution in itself but as a complementary approach, it can considerably optimize the data storage process.

Most backup strategies rely on a combination of backup, redundancy and archiving. An important factor to bear in mind when planning a backup schedule is prioritization of data. Not all data is created equal and a tiered backup strategy that restores the most critical applications first will get you back in business faster and cut data storage costs.

There are various locations to backup your data to. You can backup to:

• Drive in your system
• External or portable drive
• A dedicated NAS or Server
• Cloud storage
• CDs
• Tape

Figure out what will work for you. The easiest thing to do is to just back a second drive, internal or external, and use that for your backup storage. Make sure it is larger than you need so you don't run out of space any time soon!

I have my system automatically do a:

• Full system backup of my OS drive bi-monthly keeping the last four system images and archive the original system image from when I just installed Windows
• File backup every night to my secondary drive in my PC, my documents folder is also synced with DropBox 24/7
• Bi-monthly file backup to my server's RAID 5 array
• Monthly backup of important files to an external away from my house...I swear it is not excessive!

How to accomplish your backup plan:

Now that you have all your stuff organized, your plan on backup set up and ready you need to figure out what ways you are going to accomplish your goal, what programs are you going to use and how are you going to set them up?

There are a ton of programs out there. Feel free to try them and see how you like 'em. Personally I use FreeFileSync for my folder backup and Acronis True Image for my OS backup now.

 

[Sean Webster]

Programs and Guides

Here is a list of some free:

• Freefilesync
• Microsoft SyncToy
• Rsync
• Easus Todo back up
• Easus Disk Copy
• Clonezilla
• AceBackup
• FBackup
• GenieTimeline Free
• PureSync
• Toucan
• Paragon Backup & Recovery (Advanced) Free
• SyncBackSE
• Macrium Reflect
• Windows Backup and Restore
• GFI Backup
• Back It Up!
• Cobian Backup
• Comodo Backup
• Recuva

Here is a list of some Paid for:

• Acronis True Image
• Norton Ghost

More info on partition backup, restore, migration, etc. here: (link)

FreeFileSync: (FREE)


What is it?
FreeFileSync is a folder comparison and synchronization tool providing highly optimized performance and usability without a needlessly complex user interface. This is my favorite synchronization software for backups and folder transfers. Basically, I just set it to the mirror option, select the source and destination folder and compare and run. You can also make a batch file to have task manager run when you are away or the PC is asleep in the middle of the night.

Spoiler: Warning: Spoiler!

Download here: (link)

Just download and install.

Select the folders you want to sync:


Top right green gear icon is the sync options button, click it and select how you want to do your sync method, I just do mirror, you can even customize for any folder you want individually as well.


Then save the file as batch or something if you want to do that same sync multiple times without resetting it up. Then just click synchronize to run it.

How do I program it to run on a schedule?

Spoiler: Warning: Spoiler!

This is very simple and easy to do.

Create the batch file:

1. First of all you need to create a backup batch file. This is really easy. Just setup your folders you want to backup as you normally would picking the source and destination.
2. Once set, at the top go to Program > Save as batch job...
3. Tinker with your settings how you like. Under Batch settings I set Error handling to Ignore, uncheck show progress dialog, and have it generate a log.
4. Once done save as what ever you want. I call mine Backup.
5. Now you need to go into Task Scheduler to set it to a schedule.

Set Schedule:

1. Open the Start Menu, type "Task Scheduler," and press Enter
2. Now go to the top left next to File click "Action," then from the menu click "Create Task"
3. It will bring a window up where you can create your task.
   1. Now name it whatever you like. I named mine "FreeFileSync."
   2. Go to the Triggers tab and click the button at the bottom labeled "New..."
   3. Now you can customize how often you want to back up. I run mine daily at 5 A.M.
   4. When you're done setting up your schedule click Ok.
   5. Now go to Actions tab and click the button at the bottom labeled "New..."
   6. Under settings click "Browse"then select the FreeFileSync.exe, located: "C:\Program Files\FreeFileSync\FreeFileSync.exe"
   7. Where it says Add arguments put the location of your batch file you created earlier, for example mine is: "D:\Data\Documents\MyStuff\Scripts\Backup.ffs_batch"
   8. That's it basically. If you want you can customize it more under the "Conditions" and "Settings" tabs.
4. Now to finish click Ok and your schedule will be set.

Acronis True image: (FREE for people with WD, Seagate, and Intel drives/Not free otherwise)


What is it?
It is a great backup program that has many options for backup including email, cloud, partition, and file backup. This is what I use to backup my OS every week. It is great, I just set it on a schedule to do weekly full system images and delete anything older than the last 4 images on its own while keeping the original. I can easily recover back to an image via the program in the OS or from a bootable media environment that you can create with the program. This and FreeFileSync together complete my backup loop.

Spoiler: Warning: Spoiler!

To make a system image with Acronis True Image:

1. Select backup data/partitions.
   
   
2. Check off all the partitions on your drive(s) you want to backup.
   
   
3. Choose your image destination.
   
   
4. Change the backup scheme to however you like, I like full backups.
   
   
   
   
5. Set up your options however you like.
   
6. Have it validate your image after creation.
   
   
7. To finish just click Backup now.
   

EaseUs ToDo Backup: (FREE)


What is it?
This is a great alternative to Acronis True Image. It can do nearly everything Arconis can do and the best part is that it is free!

Robocopy: (FREE)


What is it?
Robocopy is a Robust File Copy Utility. It is very similar to xcopy/xxcopy. It is used to make copies of your data to where ever you want it to go. Thus, awesome back up feature built into windows 7!!! Though I prefer FreeFileSync over this a lot more.

Spoiler: Warning: Spoiler!

Why use it?

• 100% Free!
• 1:1 BACK UP!
• You edit it the way you want and schedulable it to run when you want.
• It updates only the changed files in the folders you set, so it allows for faster backups.
• It uses very little resources
• Multithreaded copying
• Network backup

Before we begin!

1. Besure you have all your data backed up to 2 locations in case you accidently do something wrong; here is how I back up my data: http://cdn.overclock.net/a/a3/a3e28527_b3f001eb_Mydatabackup.png
2. Test your code and verify it if you have any questions; just post in the thread!
3. If you have windows xp you will need to download this: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17657

How to program it!

Spoiler: Warning: Spoiler!

The basic syntax:

Code:

robocopy  <Destination> [<File>[ ...]] [<Options>]

Open "Notepad" and create your command.

Here is an example of my back up command:
Note: This will copy everything from my documents folder to a duplicated documents folder on one of my backup drives. Any data in the destination folder will be mirrored from the source folder and anything in the destination folder that isn't in the source folder will be deleted.

Code:

Robocopy "D:\Documents" "E:\Back Up\My Documents" /E /V /PURGE /R:5 /W:15

• "D:\Documents" is the source.
• "E:\Back Up\My Documents" is the destination.
• "/E /V /PURGE /ETA /R:5 /W:15" are the options I choose for my backups.
  • /E - Copies subdirectories. Note that this option includes empty directories
  • /V - Produces verbose output, and shows all skipped files
  • /PURGE - Deletes destination files and directories that no longer exist in the source
  • /R:5 - Specifies the number of retries on failed copies. The default value of N is 1,000,000 (one million retries)
  • /W:15 - Specifies the wait time between retries, in seconds. The default value of N is 30 (wait time 30 seconds)
  Another ex. - The /MIR switch tells Robocopy you want to mirror the two folders, copying all folders (even empty ones) from the source directory and purging folders from the destination if they no longer exist on the source.
  
• Note: Here is where you can find all the options for it: http://technet.microsoft.com/en-us/library/cc733145%28WS.10%29.aspx

For multiple locations just add another command under your first line.

Code:

Robocopy "D:\Documents" "E:\Back Up\My Documents" /E /V /PURGE /R:5 /W:15
Robocopy "D:\Pictures" "E:\Back Up\Pictures" /E /V /PURGE /R:5 /W:15

If you want to create a log follow this example:

Code:

Robocopy "Source" "Destination" /log:C:\Desktop\ZeOtherStuffRoboLog.txt

That will allow for a log file to be created. Within it shows what has been copied and what hasn't been.
Quote:

Originally Posted by GasMan320
One tip I wanted to add, if you do multiple backups (ie daily/weekly/etc) you may want to use the /LOG+ option instead of simply /LOG so that it will append to your existing log file instead of overwriting it.

Save it!

Spoiler: Warning: Spoiler!

Now that now made the script in notepad you need to save it as a .bat file where ever you like. I have mine saved in my documents in a folder called "My Scripts"

Test it out!

Spoiler: Warning: Spoiler!

Now that you have the .bat file you can go ahead and run it by double clicking it and it will automatically start to back up your source to the destination.

How do I program it to run on a schedule?

Spoiler: Warning: Spoiler!

This is very simple and easy to do.

1. Open the Start Menu, type "Task Scheduler," and press Enter
2. Now go to the top left next to File click "Action," then from the menu click "Create Task"
3. It will bring a window up where you can create your task.
   1. Now name it whatever you like. I named mine "Robo Backup."
   2. Go to the Triggers tab and click the button at the bottom labeled "New..."
   3. Now you can customize how often you want to back up.
   4. When you're done setting up your schedule click Ok.
   5. Now go to Actions tab and click the button at the bottom labeled "New..."
   6. Under settings click "Browse"then select your .bat file you created earlier.
   7. That's it basically. If you want you can customize it more under the "Conditions" and "Settings" tabs.
4. Now to finish click Ok and your schedule will be set.

Further reading:

Spoiler: Warning: Spoiler!

http://en.wikipedia.org/wiki/Robocopy
http://technet.microsoft.com/en-us/library/cc733145(WS.10).aspx
http://www.xxcopy.com/xxcopy30.htm
http://www.upway2late.com/projects/winrobocopy
http://theether.net/download/Microsoft/Utilities/robocopy.pdf

Extra Info!

How to make the the CMD window hidden:

Spoiler: Warning: Spoiler!

1. Open up notepad
2. Copy and paste this:

Code:

var WindowStyle_Hidden = 0
var objShell = WScript.CreateObject("WScript.Shell")
var result = objShell.Run("cmd.exe /c BackUpAll.bat", WindowStyle_Hidden)

Instead of typing "BackUpAll" type in the name of your .bat file you made.
3. Now save as a .js file. So for example "BackUpAll Hidden.js"
4. Run it and see that it works.

Note 1: Both the .js and the .bat need to be in the same folder for it to work.
Note 2: The .bat file you made must not have any spaces in the name. If it does then the .js script you just made will NOT run.

Now you can go ahead and make the task scheduler run the .js script instead of the .bat file and you won't see the CMD window pop up!

Locked file backup and sleep:

Spoiler: Warning: Spoiler!

Quote:

Originally Posted by GasMan320;15319445
Great script Sean!

Ok so I've been messing around with Robocopy and up until yesterday I was sort of annoyed that if I used Robocopy it would not be a 100% sure thing since it was not able to backup any locked or open files. For example, if you use Outlook for email and forgot to close the program before your backup program ran, it would completely skip that file and you would have no emails backed up.

Well I kept looking around on Google for a solution or alternative and ended up figuring out a really cool way to have my cake and eat it too!

So, like I said, its true that Robocopy doesn't handle open/locked files, BUT if we use Robocopy and wrap around it another program, called ShadowSpawn, we can then duplicate the same backup style that Windows Backup uses itself -- creating a temporary shadowed copy of all the files that you want to backup so that if you end up opening or using any of the files that you're copying, it will still work!

Ok, so this is how I set it up -- anyone who is reading this, feel free to customize your setup as you wish. I wanted an automated solution that would wake my computer up every night, create a shadow copy, back up my files, and then put my computer to sleep after it was done.

What I use:
--Windows 7 Task Scheduler (included in Win7)
--ShadowSpawn 0.2.1-x64
--Robocopy (included in Win7)

Ok, so the first step is to create a task in Task Scheduler. Give your task a name, mine is simply called Custom Backup. Make sure it is set to "Run with highest privileges". On the Triggers tab, set up your schedule. I have my backup run every night at midnight except for on Saturday nights (on Sat night I have the system do a full virus scan instead).

Then on the Actions tab select New, choose "Start a Program" as the action, and browse to your ShadowSpawn.exe file. Select that file and then in the "Add arguments (optional)" field, enter in your Robocopy script. MAKE SURE you change your Robocopy script to work with ShadowSpawn. It can't just be your regular Robocopy script. Instead, what you need to do is setup a drive letter (which isn't being used) as the Temporary Drive where all your files are shadowed while your backup is being made.

In my example below, I am backing up all the contents of the directory 'Stuff' on my hard drive (D to my external hard drive (Z but I am using the drive letter Q: as my temporary shadow drive. So D:\Stuff is shadowed to Q:, and then Robocopy takes everything from Q:\ and copies it to Z:\Backup\Stuff. You can put this all in as one command with your robocopy flags at the end.

Code:

D:\Stuff\ Q: Robocopy Q:\ Z:\Backup\Stuff /V /MIR /R:5 /W:5 /TEE /LOG+:Z:\Backup\BackupLog.txt

Be super careful about syntax errors. When you specify your fake drive, you don't include a backslash (just Q, and when you use the Robocopy command, then you do include a backslash (Q:\).

So this will now do a shadowed backup of my D:\Stuff directory to my Z:\Backup\Stuff directory. Now, I like having my computer sleep whenever its not being used, so I want to make sure it wakes up every night to run this program. Under the Conditions tab in my task, I choose "Wake the computer to run this task" and uncheck all the other boxes on that page. You can of course choose to do it however you want.

Ok so that takes care of the computer waking up, and running the backup with the shadow drive. But one thing I want is for the computer to immediately go to sleep after it has finished! I could wait for the regular idle timeout to take place before it goes to sleep, but....why wait?

So, the next step is this. Open up notepad, copy the following text into it:

Code:

%windir%\System32\rundll32.exe powrprof.dll,SetSuspendState

and then save it as sleepnow.bat. If you saved it correctly, the icon of the file should change from a regular text file. Put this file in a safe place and on the Actions tab of the same Task you just created, click on New and browse to the sleepnow.bat file you just created. Make sure that this action shows up BELOW the ShadowSpawn/Robocopy action in your action sequence.

And there you go!

Now your computer will wake up, back itself up, and go to sleep, all on its own, AND you don't have to worry about files not being copied because they were in use!

Let me know if this helped you out, I sure hope it did!

That's all there is to it! If you have any questions or extra info for me to add let me know.

 

[andrewmchugh]

Nice Tut Sean
rep++

 

[Sean Webster]

http://www.basiccomputerinformation.ca/computer-back-up/
http://data-backup-software-review.toptenreviews.com/which-backup-solution-is-best.html
http://www.debianadmin.com/backup-basics-and-different-types-of-backup.html
http://www.pcmech.com/article/guide-to-pc-backup-options/
online/cloud
->external
->internal
->server
->cd
->usb
->tape
etc

 

[munaim1]

this is definitely handy, nice one sean +rep

 

[Shooting Star]

Nice tutorial! rep+

 

[banging34hzs]

this is great. + rep

 

[asus6983]

Nice one! +rep

 

[ddietz]

Good intro to Robocopy. I also use Robocopy for my backup strategy.

I would love to improve the basic technique though. I've asked this before without much success. I think I'll throw it in here to try again.

Basically, I think Robocopy is perfect if I could get it to archive the purge to a separate "deleted" folder (based on date) to prevent accidental deletion. I just don't know enough code to make it work. The idea is you have a folder structure:

Root
-a
--a1
--a2
-b
--b1
--b2
etc

Each folder contains other folders and/or files. I would like to have the backup process create a sub-folder "deleted_date" and move any purged files there. This moves the last possible source of data loss to corruption. The ideal solution to me is a backup solution that uses checksums to check for data corruption and that then mirrors your folder structure while maintaining the purged data for some specified period.

Anyone want to help code for a backup system that moves purged data to a sub folder within the final destination folder?

 

[Sean Webster]

Hmmm, I'm not sure on that, but there is a way to log your backup to show what has changed.

Ex.

Robocopy "Source" "Destination" /log:C:\Desktop\ZeOtherStuffRoboLog.txt

That will allow for a log file to be created. Within it shows what has been copied and what hasn't been.

 

[ddietz]

Its been a little while since I played with this but what I almost had working was:

copy source to destination only new or changed files w/o purge
if files in dest not in source then create dated_deleted folder in each subfolder in source (best if only created for that subfolder having orphan files)
move dest files not in source to folder dated_deleted

This involves if/then loops which I just can't seem to figure out given the minuscule amount of code I know.

I thought it would be really cool to use this in conjunction with ZFS so that you are protected against both accidental deletion and bit level corruption.

 

[t-ramp]

Wow, nice guide.

 

[Sean Webster]

Quote:

Originally Posted by t-ramp Wow, nice guide.

Thanks!

 

[FreekyGTi]

will have to try this when i do my next backup...thanks for the tip Sean

 

[subassy]

I love robocopy...I backup my steam games then run a script to sync them over to a USB drive. I also do that for updated copies of Minecraft, GW and WoW. My Firefox portable as well but I'm just weird that way. I have a separate batch for each thing I want to sync over. I like the skip-on-error feature.

I did find setting the time out and number of retries (on fail or error) to something sane eventually helped me about quite a bit.

I also applaud the tutorial.

 

[Pigeon]

Gonna try this! Using Areca backup and Synkron right now, i'll get around to this in a little bit

 

[Sean Webster]

Quote:

Originally Posted by FreekyGTi will have to try this when i do my next backup...thanks for the tip Sean

Np, glad to help!

Quote:

Originally Posted by subassy I love robocopy...I backup my steam games then run a script to sync them over to a USB drive. I also do that for updated copies of Minecraft, GW and WoW. My Firefox portable as well but I'm just weird that way. I have a separate batch for each thing I want to sync over. I like the skip-on-error feature. I did find setting the time out and number of retries (on fail or error) to something sane eventually helped me about quite a bit. I also applaud the tutorial.

Thanks for sharing your experience.

Quote:

Originally Posted by Pigeon Gonna try this! Using Areca backup and Synkron right now, i'll get around to this in a little bit

Good luck!

 

[Sean Webster]

Bumpidy lolz

 

[Sean Webster]

I just made an uber script!

 

[GasMan320]

Quote:

Originally Posted by SeanWebster;15286075
I just made an uber script!

What's it do? I am setting Robocopy up right now and following your tutorial as we speak!

One tip I wanted to add, if you do multiple backups (ie daily/weekly/etc) you may want to use the /LOG+ option instead of simply /LOG so that it will append to your existing log file instead of overwriting it.

Interested in knowing what your script does...

So far I think I'm going to do the following: Have Win7 do a daily system image, but not backup any individual files. Then have Robocopy do a complete copy of my storage drive. The only thing I'm afraid of is what if it is erroring out from time to time and if nobody checks the log file, there's no way of knowing.

EDIT: Ok so after reading about Robocopy some more, I'm wondering if I really should be using it or not. I read that it cannot backup open or locked files (since it doesn't use Windows' Volume Shadow service). Not sure if it is risky to be copying my entire drive if it will be skipping some things depending on what is happening on the system at the time. For example, if someone is using Outlook at that time, the outlook mailbox file will be locked and none of it will be copied over.

Sigh...the search for an automated easy backup solution continues I guess.