Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Windows › Multi-core aware Archiver for Windows
New Posts  All Forums:Forum Nav:

Multi-core aware Archiver for Windows

post #1 of 13
Thread Starter 
Hello,

I'd like some advices on a good multi-core aware archiver. I am using 7zip as of right now and it uses only one core out of 4 (though the 7zip benchmark cam apparently use all 4). I have also tried winzip, also single threaded.

I have some ratrher huge multi GB archives I need to create from time to time, they take a huge amount of time and using more than 25% of my CPU would help get the job done faster thumb.gif

Thanks for any and every insights
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
post #2 of 13
7-Zip.

If it's only using one core, you are using the wrong settings.
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.3/3.6GHz core/uncore, 1.225/1.2v Gigabyte X99 SOC Champion (F22n) 2x Sapphire R9 290X Tri-X OC New Edition (10036... 4x4GiB Crucial @ 2667, 12-11-12-27-T1, 1.37v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.05) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Cooler Master Nepton 280L 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Reference R9 290X w/Stilt's MLU 1000e / 1375m E... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.3/3.6GHz core/uncore, 1.225/1.2v Gigabyte X99 SOC Champion (F22n) 2x Sapphire R9 290X Tri-X OC New Edition (10036... 4x4GiB Crucial @ 2667, 12-11-12-27-T1, 1.37v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.05) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Cooler Master Nepton 280L 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Reference R9 290X w/Stilt's MLU 1000e / 1375m E... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
post #3 of 13
Thread Starter 
I have looked, as it's pretty curious it can bench on 4 cores (tested & works) and only archive on one - I mean it jumps a thread from one core to the other but the CPU usage per the 7zip process is pegged @or bellow 25%

If it's about the settings - I have two files in the install directory: "7zip file manager" and "help". I go on the file manager and in Tools/Options I can't find any "use X core" or anything related to threading or performance. Any idea where the setting might be?

Cheers
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
post #4 of 13
Try LZMA2 compression method setting. However, more cores does not necessarily mean better compression. 1 or 2 cores may have better compression ratios.
post #5 of 13
Thread Starter 
...some more details

Upon creating an archive (right click - add to 7zip archive) I can and do select 4/4 CPUs - it still uses only one core or 25% in task manager. To test I have set the process affinity to only one core and the speed of archiving in MB/s is unchanged initially, then it accelerates on only one core (Turbo kicking in). I'm using the latest 9.20

It is completely CPU limited since it archives from/to SSD
Quote:
Originally Posted by Riou View Post

Try LZMA2 compression method setting. However, more cores does not necessarily mean better compression. 1 or 2 cores may have better compression ratios.

Ok smile.gif I'm not that interested in 5-10% better compression as long as it moves faster. I hope to avoid having to partition into 4 archives so that I can occupy one core with each.
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
post #6 of 13
Yeah, you want to use LZMA2, and manually set your threads to at least four. It's also true that compression ratios tend to get worse at higher threads, but compression goes much faster.

Anyway, a demonstration:

375
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.3/3.6GHz core/uncore, 1.225/1.2v Gigabyte X99 SOC Champion (F22n) 2x Sapphire R9 290X Tri-X OC New Edition (10036... 4x4GiB Crucial @ 2667, 12-11-12-27-T1, 1.37v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.05) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Cooler Master Nepton 280L 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Reference R9 290X w/Stilt's MLU 1000e / 1375m E... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
Primary
(15 items)
 
Secondary
(13 items)
 
In progress
(10 items)
 
CPUMotherboardGraphicsRAM
5820K @ 4.3/3.6GHz core/uncore, 1.225/1.2v Gigabyte X99 SOC Champion (F22n) 2x Sapphire R9 290X Tri-X OC New Edition (10036... 4x4GiB Crucial @ 2667, 12-11-12-27-T1, 1.37v 
Hard DriveHard DriveHard DriveCooling
Plextor M6e 128GB (fw 1.05) M.2 (PCI-E 2.0 2x) 2x Crucial M4 256GB 4x WD Scorpio Black 500GB Cooler Master Nepton 280L 
OSMonitorKeyboardPower
Windows 7 Professional x64 SP1 BenQ BL3200PT Filco Majestouch Tenkeyless (MX Brown) Corsair RM1000x 
CaseMouseAudio
Fractal Design Define R4 Logitech G402 Realtek ALC1150 + M-Audio AV40 
CPUMotherboardGraphicsRAM
X5670 @ 4.4/3.2GHz core/uncore, 1.36 vcore, 1.2... Gigabyte X58A-UD5 r2.0 w/FF3mod10 BIOS Reference R9 290X w/Stilt's MLU 1000e / 1375m E... 2x Samsung MV-3V4G3D/US @ 2000, 10-11-11-30-T1,... 
RAMHard DriveHard DriveHard Drive
1x Crucial BLT4G3D1608ET3LX0 @ 2000, 10-11-11-3... OCZ (Toshiba) Trion 150 120GB Hyundai Sapphire 120GB 3x Hitachi Deskstar 7k1000.C 1TB 
CoolingOSPowerCase
Noctua NH-D14 Windows 7 Pro x64 SP1 Antec TP-750 Fractal Design R5 
Audio
ASUS Xonar DS 
CPUMotherboardGraphicsRAM
i7-6800K @ 4.3/3.5GHz core/uncore, 1.36/1.2v ASRock X99 OC Formula (P3.10) GTX 780 (temporary) 4x4GiB Crucial DDR4-2400 @ 11-13-12-28-T2, 1.33v 
Hard DriveHard DriveCoolingOS
Intel 600p 256GB NVMe 2x HGST Travelstar 7k1000 1TB Corsair H55 (temporary) Windows Server 2016 Datacenter 
PowerCase
Seasonic SS-860XP2 Corsair Carbide Air 540 
  hide details  
Reply
post #7 of 13
Manual: http://docs.bugaco.com/7zip/MANUAL/switches/method.htm#ZipMultiThread
Quote:
mt=[off | on | {N}]
Sets multithread mode. If you have a multiprocessor or multicore system, you can get a speed increase with this switch. This option affects only compression with BZip2, Deflate and Deflate64 methods and decompression of BZip2 streams. Each thread in the multithread mode uses 32 MB of RAM for buffering. If you specify {N}, 7-Zip tries to use N threads.


What type of files are you archiving?

Would a robocopy or incremental back up make more sense?
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
post #8 of 13
Thread Starter 
With LZMA2 it does load my CPU up to 50-60%. Twice as good as before! If I tweak the compression ratios I could probably boost this even more. Thanks!
Quote:
Originally Posted by DuckieHo View Post

Manual: http://docs.bugaco.com/7zip/MANUAL/switches/method.htm#ZipMultiThread
Quote:
mt=[off | on | {N}]
Sets multithread mode. If you have a multiprocessor or multicore system, you can get a speed increase with this switch. This option affects only compression with BZip2, Deflate and Deflate64 methods and decompression of BZip2 streams. Each thread in the multithread mode uses 32 MB of RAM for buffering. If you specify {N}, 7-Zip tries to use N threads.
What type of files are you archiving?
Would a robocopy or incremental back up make more sense?

Just saw this post. I'll try the switch. They're mainly 3D sym data, meshes and some solved compressed files - in large majority they're small files with a few larger ones coming on tow. The reason for archiving is to upload one compact file in stead of a couple of hundred of files redface.gif
Edited by dragosmp - 5/4/12 at 9:14am
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
Centurion
(14 items)
 
  
CPUMotherboardGraphicsRAM
Phenom II 940 @3.3GHz 1.25V Gigabyte 790X-DS4 XFX 5850 8GB OCZ DDR2-800 
Hard DriveCoolingOSMonitor
Corsair Force 3 TRUE Win 7 Pro x64 ACER 23" TN  
PowerCase
400W Silverstone Strider Antec P182 
  hide details  
Reply
post #9 of 13
7zip max ultra setting only uses 2 cores because it is faster that way, you can create 2 task to use 4 cores

also you could try peazip, it has more options than 7zip using the same 7z app
pote reload
(14 items)
 
  
CPUMotherboardGraphicsRAM
intel 3570k@4.4@1.25 gigabyte z77z-ud3h gigabyte gtx460 768mb oc 850/2100 8gb ddr3 1600 corair vengeance blue 
Hard DriveOptical DriveCoolingOS
samsung 500gb hd502hj, 2x crucial m4 64gb raid 0 sony optiarc 24x hyper 212+ win8 x64 
MonitorPowerCaseMouse
envision lcd wide 19 + lg tv lcds 32" 32ld330 1... corsair vx 550w cheapo steelseries ikari laser 
  hide details  
Reply
pote reload
(14 items)
 
  
CPUMotherboardGraphicsRAM
intel 3570k@4.4@1.25 gigabyte z77z-ud3h gigabyte gtx460 768mb oc 850/2100 8gb ddr3 1600 corair vengeance blue 
Hard DriveOptical DriveCoolingOS
samsung 500gb hd502hj, 2x crucial m4 64gb raid 0 sony optiarc 24x hyper 212+ win8 x64 
MonitorPowerCaseMouse
envision lcd wide 19 + lg tv lcds 32" 32ld330 1... corsair vx 550w cheapo steelseries ikari laser 
  hide details  
Reply
post #10 of 13
Quote:
Originally Posted by dragosmp View Post

Just saw this post. I'll try the switch. They're mainly 3D sym data, meshes and some solved compressed files - in large majority they're small files with a few larger ones coming on tow. The reason for archiving is to upload one compact file in stead of a couple of hundred of files redface.gif
Yeah, you don't want to break those types of files up.... back up as a whole.
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Windows
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Windows › Multi-core aware Archiver for Windows