Resolve Scrapy Crawler Errors: Python / Scrapy / Scrapyd / MySQL / Splash

Developer needed with Python / Scrapy / Scrapyd / MySQL / Splash.

The code repository is in bitbucket.

1. Task One: Video information not recording to page_video table. Our crawler is working but not saving this data to this table.

2. Task Two: Resolve Crawler Errors

We have added the ability to crawl javascript with Scrapy by using Splash. We have a few errors that we need resolving.

a. Prevent a request to [login to view URL] of splash server from happening.

b. Do not forward head requests to splash

c. There's a lot of errors like this

2018-01-29 07:56:07 [[login to view URL]] ERROR: Error downloading <POST [login to view URL]>

Traceback (most recent call last):

File "/usr/local/lib/python2.7/site-packages/twisted/internet/[login to view URL]", line 1299, in _inlineCallbacks

result = [login to view URL](result)

File "/usr/local/lib/python2.7/site-packages/scrapy/core/downloader/[login to view URL]", line 53, in process_response


File "/usr/src/app/InnoCrawler/downloadermiddleware/[login to view URL]", line 177, in process_response

request, response, spider

File "/usr/src/app/InnoCrawler/downloadermiddleware/[login to view URL]", line 156, in _convert_from_splash

for row in data['history']:

KeyError: 'history'

Taidot: Linux, PHP, Python, tietojärjestelmäarkkitehtuuri, Tietojen kaavinta verkosta

Näytä lisää: python software developer needed, php mysql google app engine developer needed, python developer needed, php/mysql developer needed, mysql and php developer needed, php mysql web developer needed, php mysql developer needed jaipur full time, python boolean mysql, php mysql developer needed, python tornado developer needed, php mysql splash page, python create mysql table csv file, python twitter mysql, python web mysql admin, python csv mysql, python extjs mysql, python django developer needed, crawler script python, web crawler extract description php mysql, perl spiders crawler cron jobs reseach mysql

Tietoa työnantajasta:
( 107 arvostelua ) Chicago, United States

Projektin tunnus: #16535481

Myönnetty käyttäjälle:


Hi. I do have experience with scrapy and spash for large crawling projects. I can take a look at the project and see more details. First of all I would like to have access to the project and check it structure and h Lisää

$833 USD 5 päivässä
(111 Arvostelua)

18 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


Hello Dear! I am very interested in your project. I have high skills in Python. Specially, I have rich experience in Scrapy/Scrapyd. If you contact me, we can discuss on your project. Thansk

$400 USD 5 päivässä
(90 arvostelua)

Hello, My name is Miguel Febres and you can locate my resume and certifications in the following links: [login to view URL] [login to view URL] In my 10+ years of e Lisää

$555 USD 10 päivässä
(91 arvostelua)

Hello? How are you? My name is GuangJun I have read your brief so i can discuss the project with you. Hope to work with you. you can see my reviews. Thanks.

$555 USD 10 päivässä
(100 arvostelua)

Hi there,I am Miljan,Python(Scrapy is my favourite library) expert from Bosnia & Herzegovina,Europe. I have carefully gone through with your requirements and I would like to help you with this job ! I can start immedi Lisää

$388 USD 3 päivässä
(78 arvostelua)

hi, sir. i am a python expert. so if you give me this project, i can finish it in time. the price may be negotiable. thanks.

$444 USD 10 päivässä
(49 arvostelua)

Hi there..... Warm Greetings We came along with your request for Resolve Scrapy Crawler Errors: Python / Scrapy / Scrapyd / MySQL / Splash and we reviewed your project description. We'd like to help you with conf Lisää

$600 USD 10 päivässä
(70 arvostelua)

Hello I'm experienced Python developer and I'm very interested with the task. Let's discuss it in detail over the chat. Regards, Alex.

$750 USD 7 päivässä
(44 arvostelua)

I believe that my 9+ years of programming experience makes me highly qualified for this project. I believe that I would be an asset to this project. Linux, PHP, Python, Software Architecture, Web Scraping,

$555 USD 4 päivässä
(23 arvostelua)

A proposal has not yet been provided

$555 USD 5 päivässä
(17 arvostelua)
$555 USD 10 päivässä
(58 arvostelua)
$277 USD 2 päivässä
(15 arvostelua)

I am available to discuss more on your project, I will provide 24/7 support and quick response guaranteed! So feel free to contact me at any time! If you need any further information, please feel free to contact me. We Lisää

$277 USD 7 päivässä
(6 arvostelua)

Hello dear, I hope this proposal find you well, I saw your job posting online, I would like to request you please have a look into my previous works. I am a full-time freelancer with over 5 years of vast experience i Lisää

$750 USD 3 päivässä
(20 arvostelua)

Hi Sir, I can do the project with ease, i just need the bitbucket repository access, and I am ready to fix those errors. I am proficient with splash, scrapy, scrapyd, and Mysql. I can easily fix Task1 but when it com Lisää

$500 USD 5 päivässä
(5 arvostelua)

AI have 6 year experience Freelancer,up work,Fiverr & 99design market place I have seen your project that i can to do easily because I have many experience to Graphic Design,Webdesign,Web Develop & programming .So I co Lisää

$555 USD 10 päivässä
(1 arvostelu)
$250 USD 10 päivässä
(0 arvostelua)

Greetings of the day! About us:- Have a look at [login to view URL] We can execute our services to your concern for you at a much lower cost than what it might be in-house. We have a three ti Lisää

$555 USD 10 päivässä
(0 arvostelua)