New Posts  All Forums:Forum Nav:

Next Big Technology

post #1 of 4
Thread Starter 
What do you guys think is possible for the next big Technology? I had my own theory of one that could be a massive paradigm shift for several industries should it come to pass.

File Sizes and File Translations
I had a side hobby year ago looking into compression algorithms while working with C compilers and looked up all the popular methods used in Pkzip, Tar, LZW, Huffman, etc. What would a quantum leap in compression for large files assist with?

Nearly two years ago I decided to try making my own method.

Consider that any text file in binary can be translated to a binary/hex number or alternately a 3D address location (by a number of ways).

I wondered if I could take a large source file say 1Gig for example (1's and zero's), and convert to a Number(extremely large) or an address in 3D Space.

And then decode that Number or 3D space location to rebuild the entire file Lossless down to the exact bit.

This experience taught me many things about the limits of current computers, file systems, compilers that add their own garbage when writing binary strings.

I have a list of items to further overcome but I don't think it's impossible by no means. My biggest problem is time and how much I devoote to further research.


It might be an eyeopener if one could give someone a "#" or 3D address to use to rebuild say a whole file several tens of Gigabytes insize lossless...in a reasonable amount of time.

Whole industries would weep...Hard drive companies, Media Studios etc, and cable companies might lose their cash cow of charging by bandwidth.

It's a thought lol.
post #2 of 4
Just a heads up, in order to be unique that number would have to be the binary representation of the file itself, so a 1GB file would have to be on the scale of 2^8589934592, which is so large it would take 1GB of memory just to store.

Lossless compression works by looking for repetitions in the data. eg, if you notice 1100101100110010101100110 appears 1000 times in your file, could replace it with "1" and have a dictionary that says "whenever you see a 0001, replace it with 110010110011001010110011. You're taking a 24 bit number and replacing it with a 4 bit number, and you're doing it 1000 times, so you're compressing the data by approx. 2.5kb.

(Note: I'm glossing over a ton of details here, since my understanding of compression algorithms is a bit limited)


If you want to drastically improve compression write an algorithm that does this, but for much longer strings. Maybe even look into jumbling bits in the source file so that its easier to compress. eg, have a fixed scrambling algorithm that will essentially encrypt the source file based on a given seed. It could be that compressing the original source file gives you a compression ration of 0.8, but encrypting it using a seed of 8 gives you a compression factor of 0.5. It would take a lot longer to compress, since it would have to try each scramble function until it found the optimal one, but decompression would be fairly fast.
    
CPUMotherboardGraphicsRAM
Core i5 4670k ASUS Maximus VI Gene Gigabyte GTX 460 1GB Kingston Hyper-X 
Hard DriveHard DriveHard DriveHard Drive
Samsung 830 OCZ Vertex 3 WD6401AALS WD5000AAKS 
CoolingOSMonitorMonitor
Noctua NH-D14 elementary OS Dell Ultrasharp U2312HM LG W2442PA-BF 
KeyboardPowerCaseMouse
Microsoft Sidewinder X4 Corsair HX750W Corsair Graphite 600T Logitech G700 
Audio
ASUS Xonar DG 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Core i5 4670k ASUS Maximus VI Gene Gigabyte GTX 460 1GB Kingston Hyper-X 
Hard DriveHard DriveHard DriveHard Drive
Samsung 830 OCZ Vertex 3 WD6401AALS WD5000AAKS 
CoolingOSMonitorMonitor
Noctua NH-D14 elementary OS Dell Ultrasharp U2312HM LG W2442PA-BF 
KeyboardPowerCaseMouse
Microsoft Sidewinder X4 Corsair HX750W Corsair Graphite 600T Logitech G700 
Audio
ASUS Xonar DG 
  hide details  
Reply
post #3 of 4
Thread Starter 
I hear you, thanks for the response. Your mention of the number representation was one of the items I uncovered of course when researching this. It's why i'm looking at other solutions now.

Talk about a tough challenge lol. Ive also been studying some of the Wolfram mathematica work which gives me some alternate ideas on how to maybe go down a different discreet path.

I'm currently researching large numbers and a patterned approach to arrive at a 3D or multi-dimentional arrayed coordinate by only one discreet path dependant on a binary tree. It needs to be one way in and one way out of each address.

Like I said earlier I only have so much bandwidth to research this stuff...and once SWTor comes out everything falls to the wayside.

I think I have 3 months of productivity left then I'm gonna lose a couple years...again.
post #4 of 4
For advanced compression, you really need a doctorate. You don't just make stuff up at this point since all the low lying fruit have been picked. You basically think of a concept, write equations, and apply against datasets.

There is also an inherent limit in compression... if can't create a pattern, you can't compress it much more.
Edited by DuckieHo - 8/2/11 at 8:18pm
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
Once again...
(13 items)
 
  
CPUMotherboardGraphicsRAM
i7 920 [4.28GHz, HT] Asus P6T + Broadcom NetXtreme II VisionTek HD5850 [900/1200] + Galaxy GT240 2x4GB G.Skill Ripjaw X [1632 MHz] 
Hard DriveOSMonitorKeyboard
Intel X25-M 160GB + 3xRAID0 500GB 7200.12 Window 7 Pro 64 Acer H243H + Samsung 226BW XARMOR-U9BL  
PowerCaseMouseMouse Pad
Antec Truepower New 750W Li Lian PC-V2100 [10x120mm fans] Logitech G9 X-Trac Pro 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Coding and Programming