Simple Scrapy Crawler

I need a simple Scrapy crawler.

The crawler will hit the homepage of a domain, and look for several different things on the front page, such as 1. is there a phone number on the home page? (RuleName in the Google Sheet below: 'Has Phone Number')

2. Is there a careers page? (RuleName in the Google Sheet below: 'Has Careers Page')

3. Is there a contact page? If yes ==> navigate to the contact page and then check if there is a phone number on the contact page. (RuleName in the Google Sheet below: 'Contact Page Has Phone Number')

In total, there are 18 rules the crawler needs to check. The 18 rules are in the Google Sheet file below; see the tab called "Rules to Implement"

Every single rule needs to run on each domain, and then output a boolean indicating if the item/link was found on the domain.

As for the output, the spider will output a CSV file. The structure of the CSV file will match the tab called "Example CSV Output" in the Google Sheet linked below.

The CSV format is as follows:

Domain - the domain name

URL - the URL; for most of the rules, this is the same value as the domain column. However, when you have to nagivate to another page (for example, the contact page) this column will have the full URL of that page.

RuleName - the name of the rule that was run

RuleResult - a boolean column indicating if the information was found on that page or not (1=yes; 0=no).

RuleMatch - if RuleResult=1, this column contains the match string that made RuleResult true; otherwise, this column is blank.

CrawlDateTime - a yyyy-mm-dd hh:MM:ss timestamp of when the page was crawled

For courtesy's sake, I also provided a third tab in the Google sheet called "Input URLs from Example CSV sheet", it is a list of domains to test against. It is the list of domains on the example CSV tab.

[login to view URL]

The deliverable of this project is thee Scrapy project code. This should be done in Python 3.6+

Taidot: Python, Tietojen kaavinta verkosta, Scrapy, Data Scraping, Web Crawling

Näytä lisää: simple web crawler, simple web crawler perl cgi, simple python crawler windows, simple web crawler design, simple web crawler perl, simple python crawler, simple url crawler linux, simple web crawler script code php, simple website crawler api, source code design simple web crawler, simple site crawler, simple web crawler php, simple web crawler script, simple web crawler programming, simple web crawler perl code, simple web crawler excel, simple scrapy script

Tietoa työnantajasta:
( 8 arvostelua ) Austin, United States

Projektin tunnus: #26400156

34 freelanceria on tarjonnut keskimäärin 139$ tähän työhön


Hello, I can help you with your project - Simple Scrapy Crawler I have gone through your job posting and become very much interested to work with you. I am an expert in this field. I have already completed several p Lisää

$200 USD 5 päivässä
(57 arvostelua)

Hey there, currently working in the scrapy project where have written the crawler similar to the one you have mentioned. Have many years python programming experience so lets connect and discuss this more. Thanks. Re Lisää

$250 USD 7 päivässä
(29 arvostelua)

Hi, How are you. I am very good at scraping using python and node js. I am using scrapy, bs4, selenium in python and in node, I am using pupperteer. Please see my reviews and you will notice that. [login to view URL] Lisää

$200 USD 5 päivässä
(53 arvostelua)

Hello, how are you? Python crawling expert is here. I experienced too many scrapping tasks, so this task can be completed within this weekend. Your requirements are clear for me. Hope to hear from you soon. Regards.

$250 USD 2 päivässä
(26 arvostelua)
(72 arvostelua)

Hello client. Hope you are doing well Over 9 +years experience writing almost exclusively web scraping code. I've done it all.I can scrape all LinkedIn profile and i  have great experience of extracting information fro Lisää

$30 USD 5 päivässä
(17 arvostelua)

Hello, my name is Puru. I have 6+ years experience in providing integrated development solutions including web automation and web scraping with industry-grade expertise in python, bs4, scrapy, selenium. I have scraped Lisää

$140 USD 3 päivässä
(19 arvostelua)

Dear Hiring Manager! I'm very confident of your project. I can complete your project in 1 day. 1. My scraping works and experience I have 9 years of experience in web scraping and scraped so many many websites(around Lisää

$99 USD 1 päivässä
(72 arvostelua)

Hello Thanks for your job posting. I read your job description and understand you need. I have rich experience like your job. Will give you unlimited revisions until you are 100% satisfied, and will give you 1 month f Lisää

$120 USD 1 päivässä
(29 arvostelua)

Hi, I'm a senior scraping expert. I've already done a lot of projects to scrape data like ads, contact, product, Betting and other special info with python, php, javascript, java. Also using Multithreading, Proxy, Scra Lisää

$140 USD 7 päivässä
(9 arvostelua)

Hello Potential Client. I am professional software developer with 7 years experiences of Pandas, WebScraping, Data Mining and Python. I have developed many project including scraping amazon, youtube and twiter. I can s Lisää

$100 USD 2 päivässä
(12 arvostelua)

Hello. I am a python script developer specializing in web scraping and indexing web pages. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, bots, spiders. working examp Lisää

$222 USD 7 päivässä
(11 arvostelua)

Hello Sir, Please provide the access to the file, thank you! 100% fluent in English. I am scrapy specialist & Django Expert. I have been working in scrapinghub for quite a while. Working experience in Django (cheetay.p Lisää

$150 USD 4 päivässä
(5 arvostelua)

Hi, I have 5+ years of python experience and can easily perform this for you. I am familiar with web scraping frameworks like Scrapy. I can provide data in google sheet as requested above. Kindly accept my bid and we Lisää

$30 USD 2 päivässä
(12 arvostelua)

Dear Employer, You've specified that you need a Web Scraper. I am a Web Scraper & Internet Researcher with 5+ years of experience. I can assure you that I will provide you high-quality data. I have worked on several s Lisää

$100 USD 2 päivässä
(3 arvostelua)

I guess I'm bit [login to view URL] if you haven't found what you are looking for - let me know by simply pinging 'Hi'. I will provide you the FREE DEMO without any condition . My work will speak for itself . Actually I'm not ha Lisää

$140 USD 7 päivässä
(4 arvostelua)

Only 100USD for a day. Hi, I am a python web scraping expert. So I can use requests, BeautifulSoup and Selenium for the project. I can start the job now and finish in 10 hours. If it is ok, let's go head.

$100 USD 1 päivässä
(3 arvostelua)
(2 arvostelua)

Dear Employer, I am Swarnali, living in Australia, available online for more than 16 hours. Recently, I have completed two python projects that main goals were to scrap web data using Selenium Webdriver, Beautiful So Lisää

$100 USD 3 päivässä
(3 arvostelua)

Hello there I usually create a scraping engine for my current company and I am familiar with regex and my language to create the tools to use python (requests, selenium) and for the result, it's possible to insert to Lisää

$140 USD 7 päivässä
(5 arvostelua)