Overclock.net › Forums › Software, Programming and Coding › Networking & Security › Distributed Web-Crawling?
New Posts  All Forums:Forum Nav:

Distributed Web-Crawling?

post #1 of 5
Thread Starter 
Hey folks,

I'm trying to find a use for my older PCs. F@H is out of the question, as is most of the other @H distros. So I found an article talking about Disributed Web-Crawling.

http://yacy.net/

&

http://grub.org/

to name a few.

Has anyone ever set anything like this up before? Documentation is a little scarce and doesn't really drive home what to do. Especially since Yacy's primary language isn't english. I'd be really interested in putting some of my older Socket 462 and PIII servers to work for an OpenSource Web-Crawler. I just want some feedback.

Thanks!
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
post #2 of 5
Interesting...
post #3 of 5
Thread Starter 
Indeed!
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
post #4 of 5
Interesting indeed. I can imagine it causing lag when you're playing video games though.

Also consider a music server (Ampache), NAS (FreeNAS), maybe a local DNS server (ubuntu server)?
    
CPUMotherboardGraphicsRAM
Q6600 SLACR @ 3.6 GHz Asus P5E Deluxe MSI 6950 2 GB + 9800GT (PhysX) 4 GB White Lake DDR2-800 
Hard DriveOptical DriveOSMonitor
Hitachi 500 GB Sata iHas 120 Windows 7 Pro x64 u2711 (27", 2560x1440, H-IPS) 
KeyboardPowerCaseMouse
Generic Dell Combat Power 750W Aerotech PGS Bx-500 Logitech Rx300 
Mouse Pad
Desk 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Q6600 SLACR @ 3.6 GHz Asus P5E Deluxe MSI 6950 2 GB + 9800GT (PhysX) 4 GB White Lake DDR2-800 
Hard DriveOptical DriveOSMonitor
Hitachi 500 GB Sata iHas 120 Windows 7 Pro x64 u2711 (27", 2560x1440, H-IPS) 
KeyboardPowerCaseMouse
Generic Dell Combat Power 750W Aerotech PGS Bx-500 Logitech Rx300 
Mouse Pad
Desk 
  hide details  
Reply
post #5 of 5
Thread Starter 
Doubtful I'd notice anything as I have a dedicated line for hosting. I have several websites on a server.

I already have a NAS, 10TB, so that's out. I also use m0n0wall as my firewall, so it's my DNS as well. Not much for steaming music either.

Thanks though.
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
    
CPUMotherboardGraphicsRAM
Intel Overdrive 486DX4 100Mhz Digital Venturis 466 S3 Trio 32 1MB 68MB 72 Pin SIMMs 
Hard DriveOptical DriveOSMonitor
1.2Gb & 270Mb 32X CD-ROM Windows 98 SE LG 23" Flatron 
KeyboardPowerCaseMouse
Microsoft PS/2 Lite-On Digital Venturis 466 Logitech PS/2 
  hide details  
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Networking & Security
Overclock.net › Forums › Software, Programming and Coding › Networking & Security › Distributed Web-Crawling?