Force folders to be written contiguously?

You know thats an interesting question, I know of no way to do that. Since you're theorising a fair bit, I'll join in ..

Lets say you have a file stored at the start of the drive, its a music file, lets say a FLAC file. Now lets say you want to have all your music stored sequentially after this first music file, so that the music player can switch between files very quickly (this is just an example). How will your computer know "Oh, I am eventually gonna have 10GB of uncompressed music stored in this location, I better keep this 10GB of space free" ... well unless you tell it so, it won't know to do that. So it just deals with writes on an immediate basis, aka "real-time"... Thats how I understand it anyway.

So its a valid suggestion, and I know no way on how to do it like you said, but I thought I'd join in your theorising. Then again, with an SSD, its probably a moot point. There is defrag software that puts all your System files together on the drive, so maybe if you could specify some kind of new attribute.. eg. "Music File"... you could get the same effect. Be interesting to test out access times with a "normal use" defragmented drive and one that has all related files stored together.

Interesting topic.

Linux actually rates some filesystems by their ability to keep data contiguous and maintain high sequential performance. Ext2 was a bit better than NTFS, hence getting its reputation for not requiring defrags. Ext3 improved upon it further, and Ext4 has added more improvements. (like extents, which let programs specify they need xxxxx contiguous space if possible for writing to; they can even pre-allocate it without consuming drive I/O) All this puts those filesystems leagues ahead of Windows, and allows Linux to run many more applications without I/O related performance degradation.

For desktop use it's unlikely that Ext4 will ever require a defrag. I'm not even sure if they have a defrag tool written. It would be nice to be able to move certain files (like a database) to the very front of the drive, for maximum sequential performance... but linux has other ways around that too. It has no practical partition limit, for example - so you could partition your drive into dozens or hundreds of partitions, then mount them all as folders which certain programs run out of. (Probably not a good idea; I'm betting Ext4 can do it better than manually segmenting your drive, and with less wasted space. But if you had *one* performance critical app, you could toss it in its very own partition right on the edge.)

NTFS is a totally different beast. It doesn't have any such advanced features. If you want to preallocate a 400MB file, you write zeroes to a drive for about 5 seconds. Or if the drive was already busy, 30 seconds. (We all know how HDDs hate to multitask) Then after that you run contig on the file to read it back into memory, then write it contiguously somewhere else on the drive. (Embarassing, right? )

NTFS's philosophy is to leave gaps everywhere to facilitate low seek times and high random write performance. Windows just loves to throw files down wherever it can, and in however many fragments it requires - usually that's fine and gets the job done, but after deleting files and writing more, then deleting more, then writing more.. and repeating several times... you'll be left with a very fragmented filesystem, where downloading a few files leaves you with this:



That is never the case with Linux. For downloads like these, I'd be surprised if you could find a fragment smaller than 64MB in an Ext4 filesystem. All the files would probably be saved in just 1 fragment - no defrag required.


Zero - to answer your first question, there is a way for defrag programs to do exactly what you want, but it's up to that particular software. Most defrag software does not bother, because it knows folder sizes are constantly changing. Instead they do a quick check on the size of the drive, and then try to get files fairly close (within 5-10% of the drive's capacity if possible), and don't bother with storing all related files totally sequentially.

The two Defrag programs that I'd look into are Defraggler and MyDefrag. (formerly JKDefrag) MyDefrag had one of the best defrag algorithms - I don't know if it's still the best, since it hasn't been worked on in years, but everyone used to agree that it improved performance over more stupid/basic defrag software. Defraggler... it's been making some algorithm improvements lately, and has a very friendly UI that also reports a drive's health. That said, they don't go into any specifics on how their Defrag algorithms choose to organize files or folders.

Interestingly, Linux has a way to force a program or folder to be stored sequentially - Linux filesystems have full support for mounting zip files of all sorts. It wouldn't work well for a database or other active data, but if you zip up all the Firefox or OpenOffice crud in one big file (or even all of your favourite music), then that file can be loaded into memory completely sequentially. Program start times are definitely going to drop, and your music would be streamed in more sequentially and with less seeks.

Actually, now that I think about it - that's exactly what tar files are for. They store everything sequentially (as long as the .tar file isn't fragmented), and then often they are compressed. (.tar.gz, .tar.bzip2, .tar.7z, .tar.zip, etc.

Windows has no equivalent to Linux mounting. Probably the closest thing is Daemon Tools, but that's for mounting ISOs as DVDs. Or RAMdisk software, but that consumes a lot of memory. I haven't found any software that does the same thing for Windows and is stable. It would be nice to be able to zip up folders completely transparently to the programs running out of them, but sadly we cannot.

Thats a fantastic answer, very comprehensive.. +rep. I realised that I'd assumed that just because Windows "just loves to throw files down wherever it can", that all computers in general did the same thing. I didn't think about different file systems/Linux that might do things differently. Thanks!

Not to the level you want. Defraggler and MyDefrag do a far better job of getting related data close together, but they don't follow that rule to the letter - if something doesn't fit, it ends up elsewhere. Forcing it to do it perfectly is impossible while your OS is running, so defragmenters generally make a "best effort" attempt.
It results in fragmentation extremely quickly. That's what plagues Java (heap fragmentation; time for a garbage collect!) and SSDs. NTFS's solution is to leave gaps all over, then fill them up as it needs to. Data usually gets pretty close, but it does get fragmented as it lands. Even small files can be dumped into a dozen or so pieces.

Basically, if you prioritize low fragmentation you'll have trouble getting the data down quickly; the drive may have to seek further. (we're talking a few extra milliseconds here, but if you're going after TPM/TPS benchmarks, I suppose that matters) If you prioritize getting it down faster (like NTFS), you end up with more fragments. (Hopefully on the same track, which don't affect access times much; but eventually NTFS spirals out of control and fragments end up splattered all across the drive.) Ext4 manages to do both well because of the extents, which provide it with more information about where to place files. Even that isn't perfect - no filesystem is perfect - but I'm fairly confident that Ext4 is superior to NTFS.
I'm pretty sure I've seen Defraggler doing that. But again, it doesn't follow the rule to the letter - if it thinks something is better off somewhere else, it puts it somewhere else.
Actually, your proposal results in extremely rapid fragmentation. Writes are constantly happening, so it's quite possible that every future write will result in many fragments being created. That's because NTFS tries to get data down fast, which means it doesn't care much about where the data lands on the platter. Right beyond your folder with music in it, there will be... log files, a patched DLL, some web browser cache, more music, more other crud, etc.; if it tried to get data closer together like Linux filesystems do, that would not be the case, but then you'd have to sacrifice seek times. (Linux generally puts data near other data that is related or in the same folder; this can cause larger seeks than Windows as your drive fills up; but because it has less fragmentation and better caching algorithms, it generally maintains a higher performance level.) Anyway, what I'm trying to say is, because of how NTFS actually works your defrag idea will hurt fragmentation performance. (The drive will fragment faster) If NTFS were smarter, it would not, but NTFS is not that smart.
But very little is actually user created content on most computers. The vast majority is system files, programs, etc.; Microsoft has studied them and found that most programs only read files in their own folders sporadically/randomly rather than sequentially. Dumping all ~30GB in ProgramFiles in sequence will result in much slower access than monitoring which files are most used and dumping those files together on the very edge (with gaps to accomodate being written to.) So that's what the Windows Defrag tries to do.

I don't believe Defraggler has access to the same info, so it tries to prioritize keeping similar data together, and keeping folders together. (I read that on a forum post several years back; no idea if it's still true.) Also, it does a better job than Windows defrag at keeping fragmentation low; something to do with the gap sizes it chooses to leave open, and where it places them.

I think the solution lies in providing the filesystem with more input, rather than trying to design smarter algorithms. (which ultimately fail in one use case or another)

Microsoft recently added the ability to customize a folder view. (always show thumbnails for pictures/videos, never do, etc.) This is on a folder by folder basis - why not have some filesystem options available for folders? Maybe make it a powertoy so newbies don't screw up their systems? The info could be read by defragmenters and the filesystem, to help them make smarter decisions.

Here's some example ones... although I'd recommend giving it more than 5 minutes of thought before implementing such a thing.

[x] Keep folder contiguous when possible.
[x] Allocate in 64MB fragments when possible.
[x] Allow spare/wasted space after folder.

Option 1 is what you want if you've got music or pictures in a folder, and you're playing/viewing them in sequence. With all pictures stored sequentially, you could have SSD-level responsiveness when viewing picture folders. It would not be good for video data. (too large to be contiguous on your drive) Microsoft would have to set a sane limit, like any file over 32-64MB is exempt.

Option 2 tells it to prioritize writing to large gaps, because there's very active data in that folder which will expand further. If you were just dumping pictures off a camera, this would be less important - but if you have a video editing program rapidly creating dozens of mini thumbnails for sections of video (and then deleting or overwriting them), it would be highly important.

Option 3 would tell it that it's fine to keep lots of empty space after the folder, and to avoid using it for other stuff. (like log files) In conjunction with Option 1, it would prioritize keeping that folder's data close together with newly written data.


More input is the best way for filesystems to become smarter - and if Microsoft insists on making the algorithms better rather than relying on input... do both. It doesn't take a genius to write a daemon that detects picture folders and flips some options so they load faster.

Anyway, there's my thoughts for the day.

Ever since before Vista. "LongHorn" was going to have it.
It doesn't seem to be. Defraggler unfortunately slows down as you collect massive amounts of files in a single partition. With about 1.5 million files (some of them fairly large) I had a good amount of time to see how it sorts files. I have a downloads folder with thousands of subfolders and tens of thousands of installers in it (many being different versions of the same software) - I noticed that (for example) it would be working on defragmenting defraggler or ccleaner installers, jumping from one version to the next. Since I had about 30 different versions of each, I had a few seconds to watch it defragmenting them - the coloured squares in Defraggler's graph indicated that it was sticking them close together (perhaps even one directly after the other), probably because they were in the same folder.

But I also noticed that it was storing Ubuntu isos all over the price. Perhaps they are too big to matter? It must have a cutoff at a certain size.
I'd try it again. It has a boot-time defrag now, and the latest version is "smarter & faster". (whatever that means) I like the GUI and how nicely it presents HDD Health info. (SMART data)
Thanks. Usually I try not to flatten people with walls of text, but sometimes a topic warrants it. It's nice to know it's appreciated.