Overclock.net › Forums › Components › Hard Drives & Storage › General Data backup questions
New Posts  All Forums:Forum Nav:

General Data backup questions

post #1 of 7
Thread Starter 
I have a few general questions about backing up data.

First, I have a Data HDD and a backup data HDD as internals in my PC. My board does not support any raid and when I set this computer up I did not know enough about software raid to even consider that.

My current method is data is written to the main Data drive (G. I use robocopy in Win7 to mirror to a second drive (B. I have a .bat file on my desktop that I can click whenever I feel backup is necessary. I will also sometime run the .bat with a command line shutdown (shutdown -s) if I want to backup and have the computer shut off afterwards. This works great for me and takes very little time for me.

I also will backup manually to an external drive whenever I feel the need, using a different .bat file. Simple enough.

I believe I am fairly safe in terms of disk failure.

I am concerned about unknown data corruption and transferring that to my back-ups.

Question 1: Is there anything I can do to check or repair corruption before backups happen?

Important data I can easily write to all drives as necessary, however mirroring automatically negates the safety of this. I like mirroring the main drive as it allows all deleted material to be also removed from the backups.

Question 2: I am currently in the process of building a image and video editing machine for my father in law. Backup is obviously important and the same issues apply. I feel protecting against disk loss is easy but protecting against corruption is hard. We can provide him with a RAID setup if that will solve issues. Obviously, I'd like to know this before we buy disks and cards.
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
post #2 of 7
Thread Starter 
I'm also wondering what it is like to externally backup a raid 5 setup? Can one simply mirror a raid 5 to an external using software, robocopy, etc?

Thanks!
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
post #3 of 7
Thread Starter 
<cricket sounds> Anyone? (/cricket sounds>
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
post #4 of 7
Quote:
Question 1: Is there anything I can do to check or repair corruption before backups happen?
Under Windows, without some form of RAID, not really. This is actually one of the hottest areas in OS development - as drives get bigger, it's going to become even more important. File systems like zfs and brtfs can do this, but they aren't supported under Windows.

Your options are basically:
1)Use RAID, which includes checksums for your data, and manually 'scrub' the array at frequent intervals - say, every week. Unfortunately, scrubbing can take a long time to complete, and if errors on too many drives have occurred, it could render your array invalid.

2)Use an IDS like OSSEC (http://www.ossec.net/) to create a checksum of every file on your drive and see which checksums have changed over time. However, there's no way for the IDS to tell whether the file was changed deliberately or the change was the result of data corruption.

Quote:
I'm also wondering what it is like to externally backup a raid 5 setup? Can one simply mirror a raid 5 to an external using software, robocopy, etc?
As far as user programs are concerned, the RAID array is just one big drive. You can use Windows volume shadow copy to make a consistent snapshot of the array at a particular point in time, then back that up.
post #5 of 7
Thread Starter 
So how does robocopy mirror a folder/drive/etc? It does not replace each file with a new but only those that have changed since the last update. Is it using some sort of checksum for this?

So there is nothing really you can do about corrupted data? Am I understanding correctly that RAID can check for bad sectors but whether data changes are intentional or accidental/malicious there is no way to tell.

Is there a way to limit mirroring to changes occurring only within a certain period of time? I'm not exactly sure how to word this.... My issues and concerns are currently, I do not have a RAID capable board and I'm not at this time interested in a raid card. As I mentioned, I use robocopy .bat files to mirror specific folders. I consider my personal data important (music, videos, images, document, word, excel, etc) but OS/programs are, to my mind, expendable. If I loose my OS drive, I am out time, which for me is free in that respect. If I loose my personal data, I loose my memories, this is bad.

t seems like the biggest "risk" after drive failure (which is easy to protect for) is corruption. If I open a file and change it, then mirror that file, those changes are mirrored. But, if I don't open a file yet it is changed, there might be an issue. I wonder if that can be checked for? I'll do some snooping.
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
post #6 of 7
Quote:
So how does robocopy mirror a folder/drive/etc? It does not replace each file with a new but only those that have changed since the last update. Is it using some sort of checksum for this?
Nope. Just the file modification date.

Quote:
So there is nothing really you can do about corrupted data? Am I understanding correctly that RAID can check for bad sectors but whether data changes are intentional or accidental/malicious there is no way to tell.
Any form of raid that keeps checksums (raid5/6, basically) will absolutely detect data corruption. But it takes a long time for the check to complete (basically the same as repairing the array) and there is always the possibility that by the time you get around to doing a check, there is enough bad data that the raid is simply invalid and poops out (more of a problem for raid 5 than raid 6).

Furthermore, you can't use raid 5 with Win 7, and you can't use software solutions like flexraid to boot windows either, though you can use them for data.

Quote:
Is there a way to limit mirroring to changes occurring only within a certain period of time? I'm not exactly sure how to word this...
Yes...you can use the maxage/minage robocopy flags.


Quote:
It seems like the biggest "risk" after drive failure (which is easy to protect for) is corruption.
I agree 100%...that is why I store all my data on zfs using naxentacore, and most of my work in a VCS repository that guarantees atomic file operations and includes a cryptographically secure hash for the entire modification history of every file. (I use Bazaar, but many people prefer Mercurial). Unfortunately, this is an area of active research in OS development, and Microsoft will always be ~5yrs behind the other OSes in adopting new features.

It seems to me that the easiest route for you might be to use flexraid. It's not very easy to set up, but it can keep checksums your data.
Edited by thefreeaccount - 12/23/10 at 9:24pm
post #7 of 7
Thread Starter 
Thanks for all the info so far, its appreciated!

Rereading your post this morning, I think it just dawned on me how all this works. The checksum allows the system to be aware of corruption as any change will create a mis-match between the stored checksum and the current size. What I still don't get is how the system/controller knows the data change was intentional or otherwise? Or does it not? If a file is opened and changed, its checksum changes but if a virus or power issue (like BSOD) adds/removes data to a section, its checksum still changes. Is that right?


backing up an OS is not that important to me. Any downtime for me is not critical. I do have a full image of the OS on two separate locations but I could also re-install the OS/programs w/o that and would not be too grumpy.

I have been considering building a HTPC and/or a media server/backup. If I just go server/backup, FreeNAS sounds good as it allows many of the things you mentioned earlier. I do need to do some more reading on this as its all new to me. IF I make an HTPC it will be Win 7 or a linux dist. and will also include some means of backup. Regardless, my main rig and the HTPC or server will serve and each other's backup. By that I mean important documents from other computers in the house will go on the htpc/nas as backup. Media files, which may be on other computers, will be hosted from the htpc/nas for use by all computers. Backups of the htp/nas will also be made externally by USB or eSATA, with the device unpluged when not in use.


Regardless, if you where willing to give a brief summary of say ZFS, how exactly checksums work for data protection, etc, I'd appreciate it. I'll certainly be doing my own reading and such but sometimes its nice to start with a simple explination. I can figure out complicated things and I'm not asking for spoon-feeding but if you have a few minutes, feel free to get me started

Again, thanks for the help. It definitely gives me a starting point!
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
Alpha v.2
(13 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 740be @ 3.63 (16.5 x 220) ASUS M3A78-CM XFX HD4850 Crucial 2x2GM DDR2-800 6-6-6-18 
Hard DriveOptical DriveOSMonitor
WD Black 500GB OS/Programs, WD 1Tb + 800 GB data Sony DVD-RW Win 7x64 Professional Dell 22" 
PowerCase
Antec Earthwatts 650 Antec 300 w/5 fans 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Hard Drives & Storage
Overclock.net › Forums › Components › Hard Drives & Storage › General Data backup questions