Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Windows › Windows 7: File list caching?
New Posts  All Forums:Forum Nav:

Windows 7: File list caching?

post #1 of 7
Thread Starter 
So I've been trying to find out about this on my own, but I've been somewhat hampered by not knowing the correct terminology for some of this stuff...

I'm currently running a W7 PC as a file server, which I can access across a gigabit network or wi-fi using other PCs, my netbook or my HTC.

Problem is, my Music folder currently contains 100,000 files within ~10000 folders. Needless to say, it takes an age for that file list to come up (particularly on my HTC). I would like some method of browsing these folders and files which doesn't require the RAID0 array in my server to trawl the files as I'm browsing - it's just not fast enough. Preferably this solution would be implemented server-side, rather than using a different browser client-side.

I'm looking for what I think ought to be called file tree caching or file name index or something: A solution where I'm simply browsing a list until I actually choose a file to stream...

Can anyone help me?
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
post #2 of 7
    Possibly.  Where exactly is the bottleneck?  Is it the client computer having difficulty assembling the list, or is the server computer taking a while to load the list of files from the HDD?  In the latter case, the HDD LED in the server will run a lot while you're listing folders.
 
Edited by Techie007 - 5/26/13 at 5:39am
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
post #3 of 7
Thread Starter 
I have no HDD LED tongue.gif
Two secs, I'll pull up task manager and see smile.gif

Ok, so notably Disk 1 (D:) where my music is stored isn't accessed at all. Must be accessing the Master File Table which is on C:?

Also worth noting that the list took comparatively little time to populate on this attempt. I usually attempt to access it from the kitchen, which runs off a different wireless router. The one I'm connected to right now is closer to the server in terms of network architecture. I think there's more to this...

If the list is indeed populating off the C: drive, I'll switch it out for an SSD. Reckon that would help?

Oh, but the disk access is hardly constant. I guess the HTC is just too slow :/


Edited by zsolmanz - 5/26/13 at 6:04am
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
post #4 of 7
    First of all, each partition has its own MFT, so upgrading C: to a SSD will not speed up filename retrieval on D:.  Secondly, when listing folder contents, the filenames aren't read from the MFT, they're read from B-tree structures.  Each folder has one, and each B-tree structure lists the contents of the folder.  These files can get VERY fragmented, especially if the folder contains many files written over a long period of time.  If this is the bottleneck, I would expect that the first time you list folder contents, it would take a while, but if you closed Explorer and then went right back to the same folder, it would be much faster.  If this is the case, I recommend defragmenting D: (and C: while you're at it) with MyDefrag.
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
post #5 of 7
Thread Starter 
Thanks for the explanation!
Quote:
Originally Posted by Techie007 View Post

If this is the bottleneck... the first time you list folder contents, it would take a while, but if you closed Explorer and then went right back to the same folder, it would be much faster

This is an exact description, yes. Though surely this also occurs if the folder contents are temporarily stored in the client PC memory, as I believe Windows has a tendency to do?

Regardless, a MyDefrag Monthly Data Disk script is currently running smile.gif (Though Windows Degragmenter insists the disk is 0% fragmented).
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
Tony
(17 items)
 
  
CPUMotherboardGraphicsRAM
i5 2500k Z68XP-UD3P 6950 w/ ASUS 6970 BIOS @ 950MHz 4x4Gb Corsair Dominator LP Black 1600MHz 
Hard DriveHard DriveCoolingOS
Corsair F60 Seagate Maxtor Noctua NH-D14 Windows 7 (7600) 
MonitorMonitorKeyboardPower
Dell ST2220L 1080p eMachines 1440x900 Logitech G11 Corsair TX650W 
CaseMouseMouse PadAudio
Unknown - modded Logitech G5 £3 job Creative Soundblaster X-Fi Titanium 
Audio
Roccat Kave 
  hide details  
Reply
post #6 of 7
Quote:
Originally Posted by zsolmanz View Post

Thanks for the explanation!
This is an exact description, yes. Though surely this also occurs if the folder contents are temporarily stored in the client PC memory, as I believe Windows has a tendency to do?

    Yes, Windows does cache file reads, including B-tree file reads; and that is why it will always be faster the second time.  However, when the B-tree structures are fragmented/scattered everywhere the difference between the first time and the second time can be much greater than if they were defragmented and placed in at the beginning of the disk.

    I don't know what to make of Windows' defragmenter's 0% fragmented claim.  Either way, we'll see if it is faster after MyDefrag is done or not. thumb.gif
    If the bottleneck is the client end, you may notice a speedup when browsing the files directly on the server computer but not from remote devices, where data transmission speeds and CPU power (rendering the lists) could possibly be the limitation.
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
post #7 of 7
    If you want to read more about NTFS, this page gives a good overview right away: http://wiki.sleuthkit.org/index.php?title=NTFS.  Each folder that contains a number of files (I haven't experimented to find the exact number) will have an external B-tree file sitting next to the folder with the name "::$INDEX_ALLOCATION" appended to the name.  Practically all file browsing software hides these files, as well as a number of other NTFS metafiles—thankfully so, because it would be so confusing with all those extra metafiles in there!
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
My desktop PC
(14 items)
 
  
CPUMotherboardGraphicsRAM
Intel Core i7-3770K Gigabyte P67A-D3-B3 NVIDIA GeForce 8400 GS  1x Corsair 8 GB 
Hard DriveHard DriveHard DriveOS
Kingston SV300S3 WesternDigital WD10EZEX Samsung HD154UI Windows 7 Ultimate SP1 x64 
MonitorMonitorKeyboardPower
Daewoo L947BK Gateway FPD1530 HTK-2001 Dynex DX-400WPS 
MouseAudio
Kensington K72400 Realtek ALC889 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Windows
Overclock.net › Forums › Software, Programming and Coding › Operating Systems › Windows › Windows 7: File list caching?