Need extremely fast web scraping solution

I require you to help me implement a solution that will allow me to scrape and process a huge amount of data per minute.

The end product should support scraping of approximately 1000 random webpages per minute.

We will assume that these pages are from random websites on the internet and take approximately 3-5seconds to load and a further 2 seconds to process (extract patterns and insert into database). You however, will only be required for the Sever / Language recommendation part and some basic programming to show me how it all fits together.

Ideally I would like to work with PHP/Multi-Threading/PHP-SIMPLE-DOM but I have a strong feeling this is to resource intensive for what I require, hopefully someone can prove me wrong. What's the fastest way we can get this done?

You know exactly what is needed, now you need to sell yourself to me! Answer these questions:

How much RAM would we need?

How much CPU would we need?

How many server instances?

Approximate monthly server costs?

What language would you do it in?

Is multi-threading supported in this language and if so, how does it work?

No point bidding and not telling me what you're plan is, so please, no copy&paste replies.

Just be honest with your ideas and answer my questions in full and you'll be more likely to be chosen!

Taidot: PHP, tietojärjestelmäarkkitehtuuri

Näytä lisää: need know programming, threading programming, sell yourself, programming patterns, php programming patterns, need help php programming, know programming language software, fast web programming language, fast web, fastest programming language, dom programming, architecture recommendation, need programming help fast, multi threading, multi part php, php multi part, basic programming database, require someone extract, need product sell someone, php dom, copy data internet database, know product help sell, 1000 questions copy paste, software architecture plan

About the Employer:
( 39 reviews ) NY, United Kingdom

Projektin tunnus: #4302371

6 freelanceria on tarjonnut keskimäärin 203 $ tähän työhön


I can help in your project, please check PMB and our ratings/reviews to get idea of our experience. Please let me know if you have any queries.

199 $ USD 4 päivässä
(218 arvostelua)

Please check private message box.

100 $ USD 2 päivässä
(32 arvostelua)

please check inbox

220 $ USD 30 päivässä
(2 arvostelua)

Hello, I'm very experienced with crawlers/spiders, in the past 10 years i've made at least 10-20 spiders/year. Check my private message for my opinion.

250 $ USD 7 päivässä
(1 arvostelu)

consider it done . !!! check pm.

250 $ USD 4 päivässä
(9 arvostelua)

Hello! Ready to work on this project/ Check PM.

200 $ USD 3 päivässä
(0 arvostelua)