multi tasking firefox to the extremes?

467 Views 7 Replies 5 Participants Last post by Transhour, Mar 16, 2009

uberjon

Discussion Starter · Mar 15, 2009

long confusing story short.... i have a task for a game i play, that i wish to perform at least once daily.. i find myself putting it off, doing part of it.. etc due to some issues on windows...

this task involves opening around 3,000 tabs within about a 25 minute period of time.. each tab is read with greasemonkey, and uses the HTTP system in GM to hidden-like open a new tab for each of the previous 3000 (this ones are just to send data to a server, using $_GET the page it downloads is usually blank..

so in total, around 6,000 tabs within 25 minutes.. first of all, firefox on windows isn't multi-threaded.. (unsure if it is on *nix?) and the "highly stable" version of firefox on windows is 32bit.. the 64bit does the task a bit better.. but still not up to par..

the rig performing this task will be my sig rig.. does anyone have any suggestions on what flavor of linux or certain settings to use to get my task to be as easy as possible?? (in the long run?)

ideally i would like to only take a few minutes to do the entire task.. all without my system being laggy stalling etc.. on windows, it really bogs down after the first 300 or so tabs, and then wants to crash at about 800-1300.. then the fact it doesn't seem to be able to max out my ISP.. it seems to be software limiting..

Save

1 - 8 of 8 Posts

DigitalSonata

· #2 · Mar 15, 2009

What are you doing exactly? Does it need to be FF or could it be done with wget/curl?

Save

uberjon

Discussion Starter · #3 · Mar 15, 2009

Quote:

Originally Posted by DigitalSonata

What are you doing exactly? Does it need to be FF or could it be done with wget/curl?

being as it uses greasemoneky (javascript) it would be rather tricky to do if not impossible with wget/curl.. i attempted to do it with PHP but my knowledge in the specific area needed is lacking..

it doesn't just "send the page to the server" it 'splits' out certain data from certain parts of the page, and then sends that data..

an example of what it does, would be like.. for your "post"

it would strip out your name/post total/rep/system info and parse it into a link to send to the server. like www.domain.com/file.php?name=DigitalSonata

from there alot of php work is done to insert it into a database.. i just need a better way to load those 6000 pages with firefox. and im going to guess that some nix will do the job better than windows..

See less See more

Save

newbie1911

· #4 · Mar 15, 2009

do you really need to keep the tabs open? you could just go to the page and get redirected to the other page then repeat the whole process. you can use imacros ( firefox add-on ) to do the job.

Save

DigitalSonata

· #5 · Mar 15, 2009

Well you may be able to write a bash script to do it, or use python. I don't think FF is able to handle that many pages at once. I'm pretty sure FF isn't multithreaded either.

Save

uberjon

Discussion Starter · #6 · Mar 15, 2009

Quote:

Originally Posted by DigitalSonata

Well you may be able to write a bash script to do it, or use python. I don't think FF is able to handle that many pages at once. I'm pretty sure FF isn't multithreaded either.

currently not looking to rewrite what ive done, or change anything but the operating system.

the way i have works well, minus for firefox not wanting to respond etc. if i "take my time" it works. just slower that way.

currently downloading debian 64bit 5.0 will report back after i give it a try.

See less See more

Save

error10

· #7 · Mar 16, 2009

Doing 3000 of anything at once is going to bog down any system, regardless of OS.

It sounds like you're scraping a page and then resubmitting the data you scraped to another page. Sound about right? There are page scraping tools out there which you can probably use to build a custom solution, which would likely work much better.

Save

Transhour

· #8 · Mar 16, 2009

may i ask what you intend to do with this information? or is it just a game of programming skills?

a friend of mine and i do something similar to this, we both set out on writing a script, using what ever is needed to get it done, then when we are both done, we use creation time as a gauge to who's script is faster...

my guess would be a very minimalist linux flavor....

if i had a better of idea what you wanted to do, and how the urls are fed into the app and to scrape what type of information is mean to scrape for, php script would more than likely be way faster than opening up 6k tabs.

Save

1 - 8 of 8 Posts

This is an older thread, you may not receive a response, and could be reviving an old thread. Please consider creating a new thread.

Top Contributors this Month