Need extremely fast web scraping solution

I require you to help me implement a solution that will allow me to scrape and process a huge amount of data per minute.

The end product should support scraping of approximately 1000 random webpages per minute.

We will assume that these pages are from random websites on the internet and take approximately 3-5seconds to load and a further 2 seconds to process (extract patterns and insert into database). You however, will only be required for the Sever / Language recommendation part and some basic programming to show me how it all fits together.

Ideally I would like to work with PHP/Multi-Threading/PHP-SIMPLE-DOM but I have a strong feeling this is to resource intensive for what I require, hopefully someone can prove me wrong. What's the fastest way we can get this done?

You know exactly what is needed, now you need to sell yourself to me! Answer these questions:

How much RAM would we need?

How much CPU would we need?

How many server instances?

Approximate monthly server costs?

What language would you do it in?

Is multi-threading supported in this language and if so, how does it work?

No point bidding and not telling me what you're plan is, so please, no copy&paste replies.

Just be honest with your ideas and answer my questions in full and you'll be more likely to be chosen!

Taidot: PHP, tietojärjestelmäarkkitehtuuri

Näytä lisää: what you need to know for programming, what is web programming, web scraping solution, web scraping process, threading programming, software scraping, simple scraping software, sell yourself, scraping web for ideas, scraping the internet, scraping a server, programming patterns, php programming patterns, need help with php programming, is php web scraping, how to know programming language of a software, how does programming work, fast web scraping, fast web programming language, fast web, fastest programming language, dom programming, do it yourself websites, architecture recommendation, what is data scraping

Tietoa työnantajasta:
( 39 arvostelua ) NY, United Kingdom

Projektin tunnus: #4302371

6 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

$199 USD 4 päivässä
(218 arvostelua)

Please check private message box.

$100 USD 2 päivässä
(32 arvostelua)

please check inbox

$220 USD 30 päivässä
(2 arvostelua)

Hello, I'm very experienced with crawlers/spiders, in the past 10 years i've made at least 10-20 spiders/year. Check my private message for my opinion.

$250 USD 7 päivässä
(1 arvostelu)

consider it done . !!! check pm.

$250 USD 4 päivässä
(9 arvostelua)

Hello! Ready to work on this project/ Check PM.

$200 USD 3 päivässä
(0 arvostelua)