I need help with modyfing a script that i have.
What i have today is a parser/scraper where i enter an adress like:
[url removed, login to view]
The parser/scraper then goes to that site, and grabs the links, comments and type for each link and imports it to my database.
The problem today is that i have to do one URL at a time, i want to speed thing up as i have 6148 links that i need to parse.
The 2nd problem with my script is that it times out on some links.
I belive that it is because it has more then 4-5 pages.
The list with the links has been attached.
So what i need fixed is:
1. Letting the script read all the lines from a i.e textfile so i dont have to manualy add the links and hit "collect". I want this fully automated by just uploading the .txt-file and hit "collect". It should then go thru ALL the 6148 links and get the info.
2. Fixing the time-out - i need to get all the links, even if it has 400 pages for a model.
I will give you the source to my script at soon as i have selected a winner.
This is pretty urgent, so if you don't have time to work with it right away, you're not the one i'm looking for!
My budget is limited, so the lowest bidder will most likely recive the project!