We have two scrapers that do not work consistently. We need them fixed.
Both scrapers are written in Python 2.7 using Scrapy. Both use Crawlera as the proxy to avoid getting banned.
One is for [login to view URL] and the other is for [login to view URL]
If possible, we would like the scrapers to retrieve the data directly from the HTTP response rather than rendering the page and then scraping it.
Both spiders are always running on a Google Cloud Platform Debian Virtual Machine (Google Compute Engine) and write all results to a MySQL 5.7 2nd generation database also on Google Cloud Platform.
When you submit your proposal, tell me how much experience you have with scrapy so that I know you read this.
Sometimes they work correctly, and retrieve data and write it to the database. Other times, they appear to get stuck on something and do not retrieve any data and write 0 records to the database. This project is to fix this inconsistency and to make the scrapers reliable, robust, and running continuously without problems, errors, or issues.
22 freelancers are bidding on average $137 for this job
How are you? As you can see from my profile, I am Python expert, And I have a good experience in Trading Bot project. I would like to discuss in detail via chat. Thanks
Hi there, please leave a message on my chat so we can discuss the budget and deadline of the project. I have read your project description and i'm confident i can do this project for you perfectly. Thanks . .