PHP scraping script to MySQL

I need an efficient, well written php script that will complete the following requirements:

1) Index through a list of websites and download key information to a MySQL database. The websites URLs are sequential, so they can be called from a loop function (ie. [url removed, login to view], [url removed, login to view], etc). There are approximately 10,000 pages the script will need to parse.

2) Parse data from each website. There are 10 products returned on each page in a standard <TD> <TR> html structure. Each page has identical table structure. The parser would need to pull Product name, ID#, price, image link, and product link. Each product will be on its own row in the database.

3) The script will need to present itself like a user agent (ie. googlebot).

4) The script should multi-thread, working on multiple URL's in parallel. I would like the ability to modify the code to set how many threads at a time it will process in parallel, with the default being 5 pages at a time.

5) There script should allow for "polite downloading", such that I can put in a time requirement between page downloads and prevent the host from assuming it is an attack on their site.

6) The script should clear the database of all content each time before executing.

Taidot: PHP

Näytä lisää: php scraping script, scraping script mysql, script to pull data from a website, scraping com, process data structure, link list in data structure, key data structure, index data structure, downloads site html 5, download php 5.3, data structure link list, c++ parse html 5, scraping mysql, database scraping scripts, php scrape script, scraping script, user agent parser, how to pull data from a website, index php id, download mysql, website scraping, user agent, url scraping, price script, php multi

Tietoa työnantajasta:
( 13 arvostelua ) Millis, United States

Projektin tunnus: #395521

Myönnetty käyttäjälle:



$80 USD 2 päivässä
(9 Arvostelua)

5 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


Can provide excellent solution. Thanks!

$250 USD 3 päivässä
(385 arvostelua)

I can do this work. Thanks, Suresh

$150 USD 4 päivässä
(662 arvostelua)

Hello,Please refer your [login to view URL] you.

$100 USD 3 päivässä
(48 arvostelua)

I can do this for you. I am PHP expert.

$100 USD 3 päivässä
(17 arvostelua)