Overclock.net › Forums › Software, Programming and Coding › Other Software › Duplicate file finder
New Posts  All Forums:Forum Nav:

Duplicate file finder

post #1 of 5
Thread Starter 
Does anyone have a recommendation as far as a utility which will read the MD5 hash of each file on your drive and give you a report of which are duplicates?

Thanks.
Das Rig, Ja?
(12 items)
 
  
CPUMotherboardGraphicsRAM
AMD Ryzen 1700 Asus Crosshair VI EVGA 1080Ti SC2 2x16G GSkill RGB 3200 
Hard DriveCoolingOSMonitor
500 GB 960 EVO Enermax T50A-BVT Windows 10 Pro 27" Asus 
KeyboardPowerCaseMouse
Logitech K350 EVGA 1600G2 Fractal Define C Rosewill M55 RGB 
  hide details  
Reply
Das Rig, Ja?
(12 items)
 
  
CPUMotherboardGraphicsRAM
AMD Ryzen 1700 Asus Crosshair VI EVGA 1080Ti SC2 2x16G GSkill RGB 3200 
Hard DriveCoolingOSMonitor
500 GB 960 EVO Enermax T50A-BVT Windows 10 Pro 27" Asus 
KeyboardPowerCaseMouse
Logitech K350 EVGA 1600G2 Fractal Define C Rosewill M55 RGB 
  hide details  
Reply
post #2 of 5
This does what you want. It supports CRC32/checksum matching as well as MD5.
Tiny Fun
(8 items)
 
4P Folding Rig 1
(16 items)
 
4P Folding Rig 2
(11 items)
 
CPUMotherboardGraphicsRAM
i7-4770K @ 4.0GHz MSI Z87I EVGA GTX 660 Ti Corsair Vengeance 16GB 
Hard DriveCoolingOSCase
Samsung 840 Pro 256GB Swiftech Apogee Drive II Windows 8 Enterprise x64 Cooler Master Elite 120 Advanced 
CPUCPUCPUCPU
AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core 
MotherboardGraphicsRAMHard Drive
SuperMicro H8QGi+-F Matrox G200eW 16x2 (32GB) Crucial Ballistix DDR3 1600MHz Crucial C300 64GB 
OSPowerCase
Linux Mint 15 Cinnamon 64-bit Enermax Galaxy Evo 1250W Spotswood Tech Tray 
CPUCPUCPUCPU
AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz 
MotherboardGraphicsRAMHard Drive
Supermicro H8QGL-IF+ Matrox G200eW Kingston Server 16x1GB DDR3 Corsair Nova 32GB 
OSPower
Linux Mint 15 Cinnamon 64-bit Cooler Master Silent Pro 1000W 
  hide details  
Reply
Tiny Fun
(8 items)
 
4P Folding Rig 1
(16 items)
 
4P Folding Rig 2
(11 items)
 
CPUMotherboardGraphicsRAM
i7-4770K @ 4.0GHz MSI Z87I EVGA GTX 660 Ti Corsair Vengeance 16GB 
Hard DriveCoolingOSCase
Samsung 840 Pro 256GB Swiftech Apogee Drive II Windows 8 Enterprise x64 Cooler Master Elite 120 Advanced 
CPUCPUCPUCPU
AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core AMD Opteron 6274 ES 2.2GHz 16-Core 
MotherboardGraphicsRAMHard Drive
SuperMicro H8QGi+-F Matrox G200eW 16x2 (32GB) Crucial Ballistix DDR3 1600MHz Crucial C300 64GB 
OSPowerCase
Linux Mint 15 Cinnamon 64-bit Enermax Galaxy Evo 1250W Spotswood Tech Tray 
CPUCPUCPUCPU
AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz AMD Opteron 6176 SE 12-Core 2.3GHz 
MotherboardGraphicsRAMHard Drive
Supermicro H8QGL-IF+ Matrox G200eW Kingston Server 16x1GB DDR3 Corsair Nova 32GB 
OSPower
Linux Mint 15 Cinnamon 64-bit Cooler Master Silent Pro 1000W 
  hide details  
Reply
post #3 of 5
^^ Agree. Been using this one for years. It's simple and lightweight, and does the job.
AquaAnima
(19 items)
 
  
CPUMotherboardGraphicsRAM
i7 3820 ASUS Sabertooth X79 EVGA GTX 1070 FTW 24 GB G.Skill Ripjaws Z DDR3 1600 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 840 Pro 256 GB SSD WD Vraptor 600 GB WD Black 5TB ASUS Lightscribe 
CoolingOSMonitorMonitor
Swiftech H220X AIO Win7 Pro x64 ASUS PG278Q ROG Swift 27" GSync Dell 2407 
MonitorKeyboardPowerCase
Dell U2412M Corsair K95 RGB Corsair AX1200i Corsair Carbide 500R 
MouseMouse PadAudio
Corsair M65 Pro Razer Scarab Schiit Magni + Modi2 
  hide details  
Reply
AquaAnima
(19 items)
 
  
CPUMotherboardGraphicsRAM
i7 3820 ASUS Sabertooth X79 EVGA GTX 1070 FTW 24 GB G.Skill Ripjaws Z DDR3 1600 
Hard DriveHard DriveHard DriveOptical Drive
Samsung 840 Pro 256 GB SSD WD Vraptor 600 GB WD Black 5TB ASUS Lightscribe 
CoolingOSMonitorMonitor
Swiftech H220X AIO Win7 Pro x64 ASUS PG278Q ROG Swift 27" GSync Dell 2407 
MonitorKeyboardPowerCase
Dell U2412M Corsair K95 RGB Corsair AX1200i Corsair Carbide 500R 
MouseMouse PadAudio
Corsair M65 Pro Razer Scarab Schiit Magni + Modi2 
  hide details  
Reply
post #4 of 5
Or if you had Microsoft Access you could make a 'find duplicates query' but that would take forever to do lol.
post #5 of 5
Thread Starter 
Thanks for the suggestions.

I tried to download the app and use it, but apparently it HATES network shares. I think it quoted me on 13 hours to sort through a directory containing ~1500 files and 50 GB worth of data. Upon further inspection I found the program "fdupes" in the Ubuntu repository. It was able to identify duplicates in about 2-3 minutes.
Edited by beers - 1/12/11 at 11:03am
Das Rig, Ja?
(12 items)
 
  
CPUMotherboardGraphicsRAM
AMD Ryzen 1700 Asus Crosshair VI EVGA 1080Ti SC2 2x16G GSkill RGB 3200 
Hard DriveCoolingOSMonitor
500 GB 960 EVO Enermax T50A-BVT Windows 10 Pro 27" Asus 
KeyboardPowerCaseMouse
Logitech K350 EVGA 1600G2 Fractal Define C Rosewill M55 RGB 
  hide details  
Reply
Das Rig, Ja?
(12 items)
 
  
CPUMotherboardGraphicsRAM
AMD Ryzen 1700 Asus Crosshair VI EVGA 1080Ti SC2 2x16G GSkill RGB 3200 
Hard DriveCoolingOSMonitor
500 GB 960 EVO Enermax T50A-BVT Windows 10 Pro 27" Asus 
KeyboardPowerCaseMouse
Logitech K350 EVGA 1600G2 Fractal Define C Rosewill M55 RGB 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Other Software
Overclock.net › Forums › Software, Programming and Coding › Other Software › Duplicate file finder