Debug and improve my web crawler!



Hi everyone!

I have a PHP based web crawler that is hosted on AWS. The crawler runs every 24 hours, scans a list of websites that I have inputted, and emails me if it notices a change in any of those websites.

There are two problems:

1) The crawler believes there are web page updates even when there are none. Not for all websites, only some.

2) The crawler is VERY expensive to run on AWS. I believe the code could be improved to make it less costly to run on an EC2 server (see screenshot).

I'd like two tasks completed:

1) Help me debug the crawler to make sure it only emails me when real changes are made to webpages.

2) Explain and help me configure the code or the AWS instance to not be so expensive.

This project should be completed within 24 hours given that all the code is currently written, and just needs to be improved.


Taidot: PHP, tietojärjestelmäarkkitehtuuri

Näytä lisää: python web crawler source code, python web crawler github, python web crawler tutorial, scrapy vs beautifulsoup, python web crawler beautifulsoup, web crawler in python pdf, scrapy python 3, python scrapy example, web search on your chrome new tab please read carefully by clicking the button above and installing the gamingwonderland chrome, heroicplay provides these features and web search on your chrome new tab please read carefully by clicking the button above and, gamingwonderland provides these features and web search on your chrome new tab please read carefully by clicking the button abov, web page security disable copy paste, free web banner javascript html copy paste, web crawler local copy website, copy paste free codes complete project, copy paste list web excel, web site coding copy paste, word open copy paste web page vba, web excel entry ajax copy paste, web form copy paste address

Tietoa työnantajasta:
( 16 arvostelua ) Los Angeles, United States

Projektin tunnus: #15814194

Myönnetty käyttäjälle:

$11 USD / tunti
(0 Arvostelua)

7 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_18% %project_currencyDetails_sign_sub_19%/tunti tähän työhön


Hi, Carefully gone through your [login to view URL] have few queries so i request you to initiate chat so that we can discuss further. We are a team having experience over 10+ years in PHP and it's Framework and Mobil Lisää

$7 USD / tunti
(79 arvostelua)

Nice to meet you. I 've read your post and so very interested in your job. I 'm working in part of coding and software developing for over 6 years. I have experiences that scrap over 1000 sites. I have also much ex Lisää

$11 USD / tunti
(91 arvostelua)

.................................. Please send me details. .........................................

$5 USD / tunti
(36 arvostelua)

I am expertise in php, core php, CMS, HTML5, CSS3, bootstrap, WordPress, OpenCart, Magento, PSD to HTML, Angular JS, Node JS photoshop, OPencart, Codeignitor, Laravel, .NET, Java and many other technologies. I have 10 Lisää

$4 USD / tunti
(77 arvostelua)

Hello Hiring Manager, Looking to Develop an App for your Business? We develop apps like (Uber/airbnb/upwork/kickstarter) With 100% Customisable option according to your Business!! Development & Design Skills And Lisää

$7 USD / tunti
(2 arvostelua)

we have dedicated team for software [login to view URL] will provide you qualityuality work.

$4 USD / tunti
(0 arvostelua)