Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › I/O optimization - C++
New Posts  All Forums:Forum Nav:

I/O optimization - C++

post #1 of 3
Thread Starter 
Basically I have a program that will have a thread that copies data from an HDD to a SSD. I'll probably have two or three SSDs in RAID 0 and the goal is to do a kind of prefetch from a traditional mechanical hard drive to a SSD to decrease the latency of accessing that data. I need this because I'm working on a scientific application and the main bottleneck is the need to read few TB of data from the HDD, and the SSD would work as a kind of cache maintained by a thread...

But my problem is to copy the data from the HDD to SSD with maintaining the data aligned (and other things that SSD take advantage of, please enlighten me) to get the most performance possible out of the SSD, in C++ code.

Any suggestions or some guidelines to where I should start? And is this the best option to decrease the gap from DRAM memory to HDD?

Thanks!
Rubbish
(15 items)
 
  
CPUMotherboardGraphicsRAM
E7400 @ 4ghz 1.376v Asus P5Q3 Sapphire HD6870 Kingston 4GB DDR3 1333MHZ 
Hard DriveHard DriveOSOS
Samsung Spinpoint F3 WD Green Windows 7 64bit Lubuntu 64bit 
PowerCaseMouse
OCZ ModXStream 700W Cooler Master 690 II with side window Razer Deathadder Black Edition 
  hide details  
Reply
Rubbish
(15 items)
 
  
CPUMotherboardGraphicsRAM
E7400 @ 4ghz 1.376v Asus P5Q3 Sapphire HD6870 Kingston 4GB DDR3 1333MHZ 
Hard DriveHard DriveOSOS
Samsung Spinpoint F3 WD Green Windows 7 64bit Lubuntu 64bit 
PowerCaseMouse
OCZ ModXStream 700W Cooler Master 690 II with side window Razer Deathadder Black Edition 
  hide details  
Reply
post #2 of 3
Async read/write an option? Call some read requests, when they complete do some write requests. That's the best overlap you can probably achieve, unless there is some processing the data needs to go through. The disk read speed of the HDD's is your hard limit here, which likely is so much slower than the SSD that it might not justify the time to retool the code to do it this way, unless you can also overlap some CPU cycles as well.

How often is the program doing disk IO? Could you just "flip" the disk handle in the code after the first read?
Code:
void DiskIO::do_InputOutput(char* stuff){
  if(mFirstHDDRead == true)
   {
     //read/write from HDD
     mFirstHDDRead = false
   }
   else
   {
     //read/write from SSD
   }
}

What's your filesystem setup? Are your HDD's all raided into one big block or can you access each individually?
Callisto
(13 items)
 
  
CPUMotherboardGraphicsRAM
2500K 4.7ghz @ 1.37v MSI P67A-GD65 EVGA GTX 680 Mushkin Enhanced Redline 8GB (2 x 4GB) DDR3 1866 
Hard DriveOSMonitorPower
Intel SSD 40GB; Seagate Cavier Green 1TB Scientific Linux 6.3/Windoze7 22" Samsung SyncMaster 2232BW Corsair TX850 
  hide details  
Reply
Callisto
(13 items)
 
  
CPUMotherboardGraphicsRAM
2500K 4.7ghz @ 1.37v MSI P67A-GD65 EVGA GTX 680 Mushkin Enhanced Redline 8GB (2 x 4GB) DDR3 1866 
Hard DriveOSMonitorPower
Intel SSD 40GB; Seagate Cavier Green 1TB Scientific Linux 6.3/Windoze7 22" Samsung SyncMaster 2232BW Corsair TX850 
  hide details  
Reply
post #3 of 3
Thread Starter 
Yes I can access them individually. What I was thinking (conceptual model, since I know little about this subject) is a design where I had few HDDs and two SSD. First I fetch the data I need from the HDDs to the SSD1. CPU/GPU (it's an hybrid application) starts working on the data on the SSD1. Meanwhile, I have another thread, working in parallel, that would fetch the next dataset that I would need to SSD2. Then, when the application finished working on the first set, I would replace the data on the SSD1 with the next dataset, and so on.

One problem I think would occur is when I need to replace the data, I need to copy the results (they're not that big) before copying the new data into the SSD, and that would create some delay...

Edit: I have like 8 SSDs + around 500€ to spend on hardware that the departement lets me use to do some experiments with this.
Rubbish
(15 items)
 
  
CPUMotherboardGraphicsRAM
E7400 @ 4ghz 1.376v Asus P5Q3 Sapphire HD6870 Kingston 4GB DDR3 1333MHZ 
Hard DriveHard DriveOSOS
Samsung Spinpoint F3 WD Green Windows 7 64bit Lubuntu 64bit 
PowerCaseMouse
OCZ ModXStream 700W Cooler Master 690 II with side window Razer Deathadder Black Edition 
  hide details  
Reply
Rubbish
(15 items)
 
  
CPUMotherboardGraphicsRAM
E7400 @ 4ghz 1.376v Asus P5Q3 Sapphire HD6870 Kingston 4GB DDR3 1333MHZ 
Hard DriveHard DriveOSOS
Samsung Spinpoint F3 WD Green Windows 7 64bit Lubuntu 64bit 
PowerCaseMouse
OCZ ModXStream 700W Cooler Master 690 II with side window Razer Deathadder Black Edition 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › I/O optimization - C++