Suljettu

Fix Existing Scrapers

We have two scrapers that do not work consistently. We need them fixed.

Both scrapers are written in Python 2.7 using Scrapy. Both use Crawlera as the proxy to avoid getting banned.

One is for [login to view URL] and the other is for [login to view URL]

If possible, we would like the scrapers to retrieve the data directly from the HTTP response rather than rendering the page and then scraping it.

Both spiders are always running on a Google Cloud Platform Debian Virtual Machine (Google Compute Engine) and write all results to a MySQL 5.7 2nd generation database also on Google Cloud Platform.

When you submit your proposal, tell me how much experience you have with scrapy so that I know you read this.

Sometimes they work correctly, and retrieve data and write it to the database. Other times, they appear to get stuck on something and do not retrieve any data and write 0 records to the database. This project is to fix this inconsistency and to make the scrapers reliable, robust, and running continuously without problems, errors, or issues.

Taidot: Tietojen kaavinta verkosta, tietojärjestelmäarkkitehtuuri, Scrapy, Python, Data Scraping

Näytä lisää: fix browser alignment issues, fix browser compatibility issues, fix server configuration issues, fix css mouse issues, fix java scripting issues, fix stickam reconnecting issues, fix existing code, fix stickam connecting issues, fix browser incompatibility issues dreamweaver, fix browser compatibility issues dreamweaver file, dreamweaver fix browser compatibility issues, fix browser compatibility issues firefox, fix existing, fix existing auction site, fix ie6 css issues service, java script fix ie6 css issues, fix expanding box issues, fix website compatibility issues dreamweaver, fix cross browser issues, fix existing css

Tietoa työnantajasta:
( 3 arvostelua ) Riverton, United States

Projektin tunnus: #20556520

22 freelancers are bidding on average $137 for this job

zekovicm

Hi there, As an web scraping expert i can get it done for you. Please feel free to contact me to discuss further details. Best regards Miljan

$166 USD 4 päivässä
(146 arvostelua)
7.6
DevStar925

How are you? As you can see from my profile, I am Python expert, And I have a good experience in Trading Bot project. I would like to discuss in detail via chat. Thanks

$140 USD 7 päivässä
(68 arvostelua)
7.3
C3guru

Hello How are you? I'm very interested in your project I have built many python scripts So I'm sure I can complete your script Please share me more details over chat If you assign the job to me, I can start right now Lisää

$140 USD 7 päivässä
(42 arvostelua)
7.0
Mickelson

Hi Nice to meet you. I have enough experience in python script. Below the libraries are I used in past project. selenium, pandas, matplotlib, lxml, beautifulsoup, scipy, and other useful libraries. I have written some Lisää

$140 USD 7 päivässä
(117 arvostelua)
6.8
liveexperts123

Hi there, I have read your project description and i'm confident i can do this project for you perfectly.I still have a few questions. please leave a message on my chat so we can discuss the budget and deadline of the Lisää

$155 USD 3 päivässä
(66 arvostelua)
7.3
umg536

Hi there, please leave a message on my chat so we can discuss the budget and deadline of the project. I have read your project description and i'm confident i can do this project for you perfectly. Thanks . .

$155 USD 3 päivässä
(21 arvostelua)
6.2
sky930320

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spide Lisää

$140 USD 7 päivässä
(24 arvostelua)
6.3
kkc264043kkc

have an 3+ years experience in the web scraping and crawling with python Scrapy Can fix the scraper. want to know more about it. want to look at the script. These are my skills related to web scraping and crawling Lisää

$200 USD 2 päivässä
(58 arvostelua)
6.2
brianconey

Hello How are you I have checked your description I am a python expert and I am sure i can fix the scraper it is possible python request and beautifulsoup I can execute your script on google cloud platform It is th Lisää

$140 USD 1 päivässä
(42 arvostelua)
6.2
kunitsynartem

Hello! I have 2 years of experience in web scraping using Python and Scrapy and I'm interested in your project for your scrapers fixing. Is it possible to see the code of your scrapers? Thanks in advance for your answe Lisää

$140 USD 7 päivässä
(39 arvostelua)
5.9
Itblast

I am familiar with multiple programming languages, though most of my experience lies in scripting languages like python and object-oriented languages such as C# and Java. I have also worked a fair amount with various t Lisää

$155 USD 3 päivässä
(10 arvostelua)
5.2
NIKE9

Thanks for your job posting. I am a full-stack web developer with 5+ years of experience and I am very interested in your project. My main skill is Python, PHP, Ruby and I have the relevant javascript skills such Lisää

$155 USD 3 päivässä
(12 arvostelua)
5.4
shivansh121

Hi Employer, I can  extract any data from any [login to view URL] I have scraped the data from these websites & stored scraped data into CSVs and excels as per my clients requirement & I have made desktop applications u Lisää

$88 USD 3 päivässä
(15 arvostelua)
4.8
oswaldoodavidal

Hello, I have experience in web scraping with Python. My experience with Scrapy goes from extracting data from simple websites that doesn't require JavaScript rendering to extracting data and automating processes with Lisää

$250 USD 10 päivässä
(7 arvostelua)
4.5
innovationlight

Hi, I am python/php web scraping expert with more than 5 years experience in the field. Pls find my portfolios and good reviews. I have experience scraping recaptcha powered google site including amazon, dice, airline Lisää

$100 USD 3 päivässä
(11 arvostelua)
4.1
seocloaker81

Hi sir, I have 4 years of experience in Scrapy & 7 years of experience in data extraction, & I am proficient in using tools such as Selenium, as well as an expert in creating Python/.Net scrapers. Regards Navpreet Sing Lisää

$45 USD 3 päivässä
(5 arvostelua)
3.9
emely001

Hello, This happens to me sometimes, when I do scraping with scrapy that stays hung. This solved by configuring its timeout, or using threads in this way if some thread doesn't receive data another thread will do it. Lisää

$140 USD 2 päivässä
(6 arvostelua)
3.0
rutujaww

Hello, I have read your project description. I have done about 4 projects with scrapy with selenium and geckodriver. I have experienced some difficulties with these and was able to solve some of them. If I can take a Lisää

$171 USD 7 päivässä
(2 arvostelua)
3.0
sevenstarservice

Hello, I'm a full time freelancer and am free now. I have read your project description I am very interested in this project. I have a lot of experience in this type of work. I'm sure I can complete it on time w Lisää

$155 USD 3 päivässä
(2 arvostelua)
2.7
HamzaZahid5

Hi I have more than 5 years experience in Python, Web scraping, Database (SQL), Html 5, CSS, PHP(Laravel), Excel, Google Sheet, Python, Data Entry, Software Engineering, Data management, SQAE and Data Analysis. If poss Lisää

$150 USD 3 päivässä
(2 arvostelua)
0.5