Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Application Programming › [C++] Storing part of the object data in a file
New Posts  All Forums:Forum Nav:

[C++] Storing part of the object data in a file

post #1 of 7
Thread Starter 
I have a GUI application which works with point cloud data and a quadtree data structure behind it to handle the data. As the point format I'm working with has changed recently I had to modify my point class to hold the new attributes, which cause Point objects to grow in size significantly and in effect reducing the performance of my quadtree. Some of this attributes are not needed for displaying and processing the data, but they still need to be preserved in the output. This is roughly how my point class looks at the moment:
Code:
class Point {
    public:
        /* ... */
    private:
        /* Used data members */
        double x;
        double y;
        double z;
        double time;
        int attr1;
        int attr2;
    
        /* Unused data members */
        int atr3;
        double atr4;
        float atr5;
        float atr6;
        float atr7;
    }

When the data is loaded from the file the Points are stored in a Point* array and then handled by the quadtree. Similarly when they are savedan array of points is passed from the quadtree and saved to a file. Note that Point objects I'm using in my quadtree are different fomr those stored in the file, but I'm using a library that provides reader and writer objects which I use to create my points. Here's an example:
Code:
    int PointLoader::load(int n, Point* points) {
    
        Point temp;
        int pointCounter = 0;
    
        /* reader object is provided by the library and declared elsewhere */        
        while (pointCounter < n && reader->read_point()) {
            temp = Point(reader->get_x(), reader->get_y(), reader->get_z(), /* ... */ )
            
            points[pointCounter] = temp;
            ++pointCounter;        
        }
        return pointCounter;
    }

Now, my idea is to reduce size of the Point class, and store unused attributes in another class (or struct) called PointData on the hard drive. This is necessary because the data usually doesn't fit in memory and there's a caching system in place, which again would benefit from smaller point objects. So given the example it would look something like this:
Code:
    int PointLoader::load(int n, Point* points) {
    
        Point temp;
        PointData tempData;
        int pointCounter = 0;    
                
        while (pointCounter < n && reader->read_point()) {
            temp = Point(reader->get_x(), reader->get_y(), reader->get_z(), /* ... */ )
            pointData = (reader->get_attr3(), reader->get_attr4(), /* ... */)

            temp.dataHandle = /* some kind of handle to the data object */
            points[pointCounter] = temp;

            /* Save pointData to file to retrieve when saving points */

            ++pointCounter;        
        }
        return pointCounter;
    }

Then when I save my modified points I'd simply used the dataHandle (file offset? an index in memory mapped array?) to retrieve the pointData of each point and write it back to the file.

Does that sound like a good idea? What would be the most sensible approach of achieving this?
buka
(17 items)
 
  
Reply
buka
(17 items)
 
  
Reply
post #2 of 7
Quote:
Originally Posted by poroboszcz View Post

Now, my idea is to reduce size of the Point class, and store unused attributes in another class (or struct) called PointData on the hard drive. This is necessary because the data usually doesn't fit in memory and there's a caching system in place, which again would benefit from smaller point objects.
Does that sound like a good idea? What would be the most sensible approach of achieving this?
I think you are on the right track. essentially, you need to create a very simple file system with a file offset/ lookup map. the key could identify the PointData object, and the value would be the index. then fseek to index*sizeof(PointData). obviously you should create a separate class to handle read/write operations.

i hope this is the easiest and most intuitive way to accomplish your goal. I like to think so because i used those ideas to create my own FAT file system for a school project. biggrin.gif
Fractal Design
(15 items)
 
775 4 life
(15 items)
 
 
CPUMotherboardGraphicsRAM
Intel i7 2600K Biostar TP67XE NVidia GTX 570 Crucial Ballistix 
Hard DriveHard DriveCoolingOS
Crucial C300 RealSSD SDD Samsung F4 2TB Noctua NH-D14 Windows 7 Professional x64 
MonitorMonitorKeyboardPower
Asus VH202T 20'' 1600x900 Acer P244W 24" 1920 x 1080 Apple Keyboard with Numeric Keypad SeaSonic M12II 620W 
CaseMouseAudio
Fractal Design Define XL Titanium Grey Razor Abyssus Creative Sound Blaster X-FI Xtreme Gamer 
CPUMotherboardGraphicsRAM
Intel X3350 3.2Ghz @ 1.25v Gigabyte-GA-P35-DS3L (rev 2) XFX 4870 1GB 4GB OCZ Reaper PC2-6400 
RAMHard DriveHard DriveOptical Drive
2GB Corsair XMS2 PC2-6400 Crucial C300 64GB SSD 2TB Samsung Spinpoint F4 Sony Super Multi 
OSMonitorPowerCase
Windows 7 Professional x64 SP1 Asus VH202T 20'' 1600x900 SeaSonic M12II 620W Cooler Master Centurion 5 
Mouse
Razor Abyssus 
CPUMotherboardGraphicsRAM
Core i5-520M Lenovo 2522BF3 NVIDIA® Quadro® NVS3100M  Ramaxel Technology 4Gb DDR3 
Hard DriveOptical DriveOSMonitor
Samsung SSD 128GB 1.8" Micro SATA  hl-dt-st dvdram gu10n Windows 7 Enterprise (64-bit) 14.1" WXGA (1280x800) display, anti-glare, LED ... 
Power
9-cell plus Slice battery 
  hide details  
Reply
Fractal Design
(15 items)
 
775 4 life
(15 items)
 
 
CPUMotherboardGraphicsRAM
Intel i7 2600K Biostar TP67XE NVidia GTX 570 Crucial Ballistix 
Hard DriveHard DriveCoolingOS
Crucial C300 RealSSD SDD Samsung F4 2TB Noctua NH-D14 Windows 7 Professional x64 
MonitorMonitorKeyboardPower
Asus VH202T 20'' 1600x900 Acer P244W 24" 1920 x 1080 Apple Keyboard with Numeric Keypad SeaSonic M12II 620W 
CaseMouseAudio
Fractal Design Define XL Titanium Grey Razor Abyssus Creative Sound Blaster X-FI Xtreme Gamer 
CPUMotherboardGraphicsRAM
Intel X3350 3.2Ghz @ 1.25v Gigabyte-GA-P35-DS3L (rev 2) XFX 4870 1GB 4GB OCZ Reaper PC2-6400 
RAMHard DriveHard DriveOptical Drive
2GB Corsair XMS2 PC2-6400 Crucial C300 64GB SSD 2TB Samsung Spinpoint F4 Sony Super Multi 
OSMonitorPowerCase
Windows 7 Professional x64 SP1 Asus VH202T 20'' 1600x900 SeaSonic M12II 620W Cooler Master Centurion 5 
Mouse
Razor Abyssus 
CPUMotherboardGraphicsRAM
Core i5-520M Lenovo 2522BF3 NVIDIA® Quadro® NVS3100M  Ramaxel Technology 4Gb DDR3 
Hard DriveOptical DriveOSMonitor
Samsung SSD 128GB 1.8" Micro SATA  hl-dt-st dvdram gu10n Windows 7 Enterprise (64-bit) 14.1" WXGA (1280x800) display, anti-glare, LED ... 
Power
9-cell plus Slice battery 
  hide details  
Reply
post #3 of 7
You need a database, like SQLite. Assign an ID to each Point (or don't if there cannot be two sets of coordinates with the same values), and then use that unique info to select the data from the database.

You can implement the equivalent of this, but using a database will probably be easier. You should look for a good SQLite (or whatever db you use) wrapper instead of using it directly.

Also, why are you using an array and using a pointer to refer to it, instead of a STL vector? You're writing in C++, but your code looks like C.
Edited by Coma - 3/22/12 at 5:06pm
Akiyama Mio
(13 items)
 
  
CPUMotherboardGraphicsRAM
E6420 @ stock, 0.98v Asus P5N-E SLI Gainward GTX 460 1GB @ 800/1600/1900 2x2GB Kingston @ 800MHz 5-5-5-15 2T 
Hard DriveOptical DriveOSMonitor
WD 250GB, 320GB SATA/3, 16MB Cache, Seagate 1TB LG GSA-H62N 18x SATA Ubuntu 9.10 x86 & Win7 x86 Asus VW222U 
KeyboardPowerCase
Logitech Classic Corsair 650HX NZXT Apollo Black 
  hide details  
Reply
Akiyama Mio
(13 items)
 
  
CPUMotherboardGraphicsRAM
E6420 @ stock, 0.98v Asus P5N-E SLI Gainward GTX 460 1GB @ 800/1600/1900 2x2GB Kingston @ 800MHz 5-5-5-15 2T 
Hard DriveOptical DriveOSMonitor
WD 250GB, 320GB SATA/3, 16MB Cache, Seagate 1TB LG GSA-H62N 18x SATA Ubuntu 9.10 x86 & Win7 x86 Asus VW222U 
KeyboardPowerCase
Logitech Classic Corsair 650HX NZXT Apollo Black 
  hide details  
Reply
post #4 of 7
Thread Starter 
Quote:
Originally Posted by Coma View Post

You need a database, like SQLite. Assign an ID to each Point (or don't if there cannot be two sets of coordinates with the same values), and then use that unique info to select the data from the database.
You can implement the equivalent of this, but using a database will probably be easier. You should look for a good SQLite (or whatever db you use) wrapper instead of using it directly.
Also, why are you using an array and using a pointer to refer to it, instead of a STL vector? You're writing in C++, but your code looks like C.

I guess this could work, but I'm looking for possibly the most efficient and small solution. There's no reason to involve databases.
Quote:
Originally Posted by travesty View Post

I think you are on the right track. essentially, you need to create a very simple file system with a file offset/ lookup map. the key could identify the PointData object, and the value would be the index. then fseek to index*sizeof(PointData). obviously you should create a separate class to handle read/write operations.
i hope this is the easiest and most intuitive way to accomplish your goal. I like to think so because i used those ideas to create my own FAT file system for a school project. biggrin.gif

That's roughly what I ended up doing. Added a static map object to my loader class to keep paths to temporary files I create to store pointData objects and added an offset to Point class to locate objects in files. Seems to work ok for now. It's kind of temporary solution anyway and makes my program usable until I come up with better solution for the quadtree.
buka
(17 items)
 
  
Reply
buka
(17 items)
 
  
Reply
post #5 of 7
The simplest solution is one where you write as little code as possible. It is also often technologically better.
Akiyama Mio
(13 items)
 
  
CPUMotherboardGraphicsRAM
E6420 @ stock, 0.98v Asus P5N-E SLI Gainward GTX 460 1GB @ 800/1600/1900 2x2GB Kingston @ 800MHz 5-5-5-15 2T 
Hard DriveOptical DriveOSMonitor
WD 250GB, 320GB SATA/3, 16MB Cache, Seagate 1TB LG GSA-H62N 18x SATA Ubuntu 9.10 x86 & Win7 x86 Asus VW222U 
KeyboardPowerCase
Logitech Classic Corsair 650HX NZXT Apollo Black 
  hide details  
Reply
Akiyama Mio
(13 items)
 
  
CPUMotherboardGraphicsRAM
E6420 @ stock, 0.98v Asus P5N-E SLI Gainward GTX 460 1GB @ 800/1600/1900 2x2GB Kingston @ 800MHz 5-5-5-15 2T 
Hard DriveOptical DriveOSMonitor
WD 250GB, 320GB SATA/3, 16MB Cache, Seagate 1TB LG GSA-H62N 18x SATA Ubuntu 9.10 x86 & Win7 x86 Asus VW222U 
KeyboardPowerCase
Logitech Classic Corsair 650HX NZXT Apollo Black 
  hide details  
Reply
post #6 of 7
In java this solution is really easy..... just make the class serializable, you can then just write objects of that class to disk and load them back at will.....
 
Tragbar
(14 items)
 
 
CPUMotherboardGraphicsRAM
Ryzen 1800X Asus Crosshair VI Hero Nvidia Geforce Titan X F4-3600C15D-16GTZ 
Hard DriveHard DriveHard DriveHard Drive
Combined: 6.13 TB of space Samsung SSD 960 EVO 500GB Crucial_CT1050MX300SSD1 M4-CT128M4SSD2 
CoolingCoolingCoolingCooling
EK-Supremacy EVO White Edition PrimoChill PrimoFlex Advanced LRT EK-RES X4 250 (R2.0) Reservoir EK-CoolStream XE 360 
CoolingCoolingCoolingOS
EK-Vardar F4-120ER (2200rpm)  EK-Ekoolant EVO Liquid Coolant EK-XTOP Revo D5 PWM Pump Windows 10 Pro 
MonitorKeyboardPowerCase
Asus VG278H Ducky YOTG Keyboard Corsair AX1200 Corsair Obsidian 800D 
MouseAudioAudioAudio
Logitech G700s Schiit Gungnir Multibit Schiit Mjolnir 2 Schiit LISST 
Audio
Audeze LCD 2 - pre fazor 
CPUMotherboardGraphicsRAM
intel i7 4770K ASRock Z87E-ITX Nvidia Geforce GTX Titan X GSkill F3-2400C10D-16GTX R 
Hard DriveOptical DriveCoolingOS
Samsung 840 Pro 512GB Silverstone 12.7mm Slot Load Blu-Ray Burner Noctua NH-L12 Windows 8 Pro 
MonitorCaseAudioAudio
LG PA75U Slim LED Projector Silverstone SG08 Schiit Bifrost Schiit Asgard 2 
AudioOther
AKG Q701 Headphones Pelican 1510 Green Case  
  hide details  
Reply
 
Tragbar
(14 items)
 
 
CPUMotherboardGraphicsRAM
Ryzen 1800X Asus Crosshair VI Hero Nvidia Geforce Titan X F4-3600C15D-16GTZ 
Hard DriveHard DriveHard DriveHard Drive
Combined: 6.13 TB of space Samsung SSD 960 EVO 500GB Crucial_CT1050MX300SSD1 M4-CT128M4SSD2 
CoolingCoolingCoolingCooling
EK-Supremacy EVO White Edition PrimoChill PrimoFlex Advanced LRT EK-RES X4 250 (R2.0) Reservoir EK-CoolStream XE 360 
CoolingCoolingCoolingOS
EK-Vardar F4-120ER (2200rpm)  EK-Ekoolant EVO Liquid Coolant EK-XTOP Revo D5 PWM Pump Windows 10 Pro 
MonitorKeyboardPowerCase
Asus VG278H Ducky YOTG Keyboard Corsair AX1200 Corsair Obsidian 800D 
MouseAudioAudioAudio
Logitech G700s Schiit Gungnir Multibit Schiit Mjolnir 2 Schiit LISST 
Audio
Audeze LCD 2 - pre fazor 
CPUMotherboardGraphicsRAM
intel i7 4770K ASRock Z87E-ITX Nvidia Geforce GTX Titan X GSkill F3-2400C10D-16GTX R 
Hard DriveOptical DriveCoolingOS
Samsung 840 Pro 512GB Silverstone 12.7mm Slot Load Blu-Ray Burner Noctua NH-L12 Windows 8 Pro 
MonitorCaseAudioAudio
LG PA75U Slim LED Projector Silverstone SG08 Schiit Bifrost Schiit Asgard 2 
AudioOther
AKG Q701 Headphones Pelican 1510 Green Case  
  hide details  
Reply
post #7 of 7
Quote:
Originally Posted by rush2049 View Post

In java this solution is really easy..... just make the class serializable, you can then just write objects of that class to disk and load them back at will.....
you would still need some way to keep track of the objects on the disk. he doesn't want to load all the data into memory and then search for an item.
Quote:
This is necessary because the data usually doesn't fit in memory and there's a caching system in place.
Fractal Design
(15 items)
 
775 4 life
(15 items)
 
 
CPUMotherboardGraphicsRAM
Intel i7 2600K Biostar TP67XE NVidia GTX 570 Crucial Ballistix 
Hard DriveHard DriveCoolingOS
Crucial C300 RealSSD SDD Samsung F4 2TB Noctua NH-D14 Windows 7 Professional x64 
MonitorMonitorKeyboardPower
Asus VH202T 20'' 1600x900 Acer P244W 24" 1920 x 1080 Apple Keyboard with Numeric Keypad SeaSonic M12II 620W 
CaseMouseAudio
Fractal Design Define XL Titanium Grey Razor Abyssus Creative Sound Blaster X-FI Xtreme Gamer 
CPUMotherboardGraphicsRAM
Intel X3350 3.2Ghz @ 1.25v Gigabyte-GA-P35-DS3L (rev 2) XFX 4870 1GB 4GB OCZ Reaper PC2-6400 
RAMHard DriveHard DriveOptical Drive
2GB Corsair XMS2 PC2-6400 Crucial C300 64GB SSD 2TB Samsung Spinpoint F4 Sony Super Multi 
OSMonitorPowerCase
Windows 7 Professional x64 SP1 Asus VH202T 20'' 1600x900 SeaSonic M12II 620W Cooler Master Centurion 5 
Mouse
Razor Abyssus 
CPUMotherboardGraphicsRAM
Core i5-520M Lenovo 2522BF3 NVIDIA® Quadro® NVS3100M  Ramaxel Technology 4Gb DDR3 
Hard DriveOptical DriveOSMonitor
Samsung SSD 128GB 1.8" Micro SATA  hl-dt-st dvdram gu10n Windows 7 Enterprise (64-bit) 14.1" WXGA (1280x800) display, anti-glare, LED ... 
Power
9-cell plus Slice battery 
  hide details  
Reply
Fractal Design
(15 items)
 
775 4 life
(15 items)
 
 
CPUMotherboardGraphicsRAM
Intel i7 2600K Biostar TP67XE NVidia GTX 570 Crucial Ballistix 
Hard DriveHard DriveCoolingOS
Crucial C300 RealSSD SDD Samsung F4 2TB Noctua NH-D14 Windows 7 Professional x64 
MonitorMonitorKeyboardPower
Asus VH202T 20'' 1600x900 Acer P244W 24" 1920 x 1080 Apple Keyboard with Numeric Keypad SeaSonic M12II 620W 
CaseMouseAudio
Fractal Design Define XL Titanium Grey Razor Abyssus Creative Sound Blaster X-FI Xtreme Gamer 
CPUMotherboardGraphicsRAM
Intel X3350 3.2Ghz @ 1.25v Gigabyte-GA-P35-DS3L (rev 2) XFX 4870 1GB 4GB OCZ Reaper PC2-6400 
RAMHard DriveHard DriveOptical Drive
2GB Corsair XMS2 PC2-6400 Crucial C300 64GB SSD 2TB Samsung Spinpoint F4 Sony Super Multi 
OSMonitorPowerCase
Windows 7 Professional x64 SP1 Asus VH202T 20'' 1600x900 SeaSonic M12II 620W Cooler Master Centurion 5 
Mouse
Razor Abyssus 
CPUMotherboardGraphicsRAM
Core i5-520M Lenovo 2522BF3 NVIDIA® Quadro® NVS3100M  Ramaxel Technology 4Gb DDR3 
Hard DriveOptical DriveOSMonitor
Samsung SSD 128GB 1.8" Micro SATA  hl-dt-st dvdram gu10n Windows 7 Enterprise (64-bit) 14.1" WXGA (1280x800) display, anti-glare, LED ... 
Power
9-cell plus Slice battery 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Application Programming
Overclock.net › Forums › Software, Programming and Coding › Coding and Programming › Application Programming › [C++] Storing part of the object data in a file