Scraping Automotive Websites

What we need; Development, implementation & support of the Website and the Crawlers

what we ask; fully commented code, open source solutions, documentation & support.

General consideration; We want the software/site in English.

A. Develop and implement 5 web crawlers to run continuously as a batch job in background updating the mysql db and tracking information changes from sites List;

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

[url removed, login to view]

The scraper should retrieve ALL the information about the listing and store it directly on our DB. The script should also be able to detect duplicate autos and avoid them.

We prefer Python (Scrapy) scripts for the crawlers but this could be implemented in some other language. In the ideal scenario the script should never stop running.

A second script for each crawler should be also developed and implemented in order to clean the DB and remove un-published listings .

You will also have to create a [url removed, login to view] where a site admin may be able to see the scrapper progress, restart it, see logs, stop, etc

1st Requirements:

Experience to support Crawlers on Amazon Web Services.

Apache and MySQL DB

2nd Requirement:

Experience scraping/crawling websites using Python

3es Requirement:

Must have strong PHP and MYSQL DB background

3rd Requirement:

Need some one who will be able to support Crawler DB and is interested in a part time position. Please let me know your hourly rate in PM If you are not able to support crawler on Amazon Web Services please don't bid.

Taidot: Amazon Web Services, MySQL, PHP, Python, Tietojen kaavinta verkosta

Näytä lisää: www edmunds com, www cars com, we develop websites, websites for develop, websites develop, web scraping solutions, web scraping python 3, web scraping part time, web develop on python, software websites, software scraping, software development websites, software development hourly rate, scraping com, rate list of website development, python language for web development, part time job python, list of websites development, job web scraping python, job web crawler, job listing websites, ideal websites, hourly rate python, edmunds com, edmunds

Tietoa työnantajasta:
( 7 arvostelua ) Fairfield, United States

Projektin tunnus: #1706949

13 freelancers are bidding on average $858 for this job


Dear Sir, We have developed extensive experience in working with auto industry web data mining from Japanese used cars auctions. We can even crawl secured sites and merge data and capable to handle thousands of listing Lisää

1000 $ USD 25 päivässä
(80 arvostelua)

Hello Mate, We have 6 years of experience of core development and we have Experienced team. Please check our PMB for brief details.

750 $ USD 15 päivässä
(24 arvostelua)

I specialize in similar projects. Please check PM for more details.

1500 $ USD 30 päivässä
(29 arvostelua)

Hello. I can do this work.

750 $ USD 30 päivässä
(48 arvostelua)

Interested in this project.

600 $ USD 10 päivässä
(11 arvostelua)

please check PM

750 $ USD 15 päivässä
(2 arvostelua)

By having expertise on crawling and handling HTML/XML files through background code, we claim to be right fit.

600 $ USD 15 päivässä
(3 arvostelua)

Hi Sir, I can do this project and i meet all your requirements.

550 $ USD 20 päivässä
(10 arvostelua)

[url removed, login to view] Go through My Resume.I am Also Sending You the Sample of My Previous Work. 1) [url removed, login to view] 2) [url removed, login to view] 3) [url removed, login to view] 4) http://richestsoft Lisää

700 $ USD 14 päivässä
(1 arvostelu)

Hi, I am expert in web scraping & python & db & create web site. I ever implement some crawler for my own website, using Scrapy - python framework at web scraping. I am expert at MySQL and PostgreSQL and the Lisää

2000 $ USD 20 päivässä
(0 arvostelua)

Custom software development - <b><i>Removed by Admin</i></b>

750 $ USD 1 päivässä
(0 arvostelua)

Hi vroombe, I have a few years of experience in developing crawling and python development in general. I took the liberty to create a very basic proposal that I am attaching as private message. Regards.

960 $ USD 8 päivässä
(0 arvostelua)

My automated web spider can handle this. i can modify it for your needs

250 $ USD 7 päivässä
(0 arvostelua)