Peruttu

Web scrape of 5-7 online retailers using Ruby Mechanize/Watir/Nokogiri (repost)

I run a data analytics consultancy in Sydney, Australia. I have a requirement to setup weekly scrapes of at least 5 large (50,000+ SKUs) online retail websites. The project will be split to scrape a single website to start with, extending to the others assuming satisfaction with the work completed.

Technical requirements:

- Scrapes must be written in Ruby using Mechanize/Nokogiri or Watir as appropriate

- Where possible, avoiding JS and scrape via direct HTTP requests using Mechanize - i.e requires strong skills in tracing/debugging HTTP requests/responses

- Alternatively, load page using Watir webdriver then pass off cookies to Mechanize

- Last resort would be brute force Watir webdriver approach (scroll, point, click, etc)

- Implement utilities for use of proxy / IP rotation where required (access to pool of anonymous IP addresses will be provided)

- Scheduling via cron job to run at least weekly

- Must use object oriented approach and design in a DRY manner so that code can be leveraged for future scrapes of additional sites and ensure future code manageability

- Must use Activerecord to persist records scraped

- Must implement logging (production logging to file + verbose debug option to STDOUT)

Personal requirements:

- Must be a great communicator both in terms of coding concepts and project management

- Good at clearly laying out the workplan, milestones and progress

- Must be available for 2x daily stand-up calls with Sydney AUS (timing TBC at mutually agreed time)

- Must be able to demonstrate prior experience with ruby web scrape projects similar to above

- Code must be well-structured and well-documented / commented

Opportunity for ongoing Ruby backend and RoR web app build if this project is a success.

Taidot: Django, Python, Ruby, Ruby on Rails, Tietojen kaavinta verkosta

Näytä lisää: Web development using Ruby on Rails, 8 5 x 7 5 print online brochure, web scrape 5 sites, quick app site web scrape using proxies, simple web application using ruby, web scraping using ruby rails, online sharing using web service, creating portal using ruby, web scrape asp login, online bookstore using php, flash web scrape, example online form using php, create online admission using php javascript html, php web scrape script, online questionnaire using asp

Tietoa työnantajasta:
( 0 arvostelua ) Australia

Projektin tunnus: #15865779

21 freelanceria on tarjonnut keskimäärin 37 $/tunti tähän työhön

bestit4u

Hello,sir I am very interested in your project, because I have much experience in such projects. I have good skills with the program language including C/C++, C#, java, php, asp.net, python, VB.NET. So I have expert Lisää

$41 AUD / tunti
(47 arvostelua)
6.1
yingli1987

Hello, I read your job posting carefully and I'm interesting in your job. My major language for scraping is python with some packages(scrapy, scrapyD, lxml, BS, selenium packages). I can make scraping script as Lisää

$41 AUD / tunti
(16 arvostelua)
5.4
londonlance

We are a London (Shoreditch) based rails studio. Please get in touch to discuss further. Following are some of our recent projects; [url removed, login to view] a freelancing marketplace built using Rails 4.2 a Lisää

$36 AUD / tunti
(1 arvostelu)
5.6
kchg

As the TOP 5th freelancer of this site, I can deliver the good result in time. Developed some kind of data scraping using Ruby on Rails and PHP. With the wonderful experience and enough knowledge, I can make the su Lisää

$35 AUD / tunti
(11 arvostelua)
6.0
zixy

We have worked on the large scale scrapers which work in the same way as you described in the project requirements. Are you available to discuss the details now? Talk soon, Zoran

$40 AUD / tunti
(2 arvostelua)
5.3
arvindkushwah9

Dear, I have 5+ years experienced in Web Development. I'm full stack Ruby On Rails developer, I have experience in developing the application like you want with, i can do this job very well. Specialties: Ruby on Lisää

$27 AUD / tunti
(35 arvostelua)
5.6
andriphone0

Hello, I am scraping expert, I have did too many similar projects, Please check my feedback then you will know. Can you tell me more details? Then I will provide demo data for you.

$25 AUD / tunti
(48 arvostelua)
5.3
TenStar718

I have good experience for web scrap app development. I can setup an app that scrapes at leat 5 large online retail websites. Let's discuss more in PM.

$39 AUD / tunti
(4 arvostelua)
4.6
abhilashtv

Hi, ➲ 7+ years of full-time experience in Ruby and Python with 50,000+ Upwork hours billed and 50+ successful Python projects ➲ Upwork Top 10 Certification for Ruby and Pythn ➲ Guaranteed Results Policy: Pay only if Lisää

$27 AUD / tunti
(11 arvostelua)
4.8
BizVenture

I've special skills in web scraping using Python and Ruby. Generally use direct HTTP requests, but if website asks javascript running on client, no problem by using selenium and webdriver. Have scraped on-demand, soc Lisää

$44 AUD / tunti
(2 arvostelua)
4.1
DarinX

Web design&development project Relevant Skills and Experience Mobile: Java, Swift, Objective-C Design: graphic design, web design, UI/UX design (Photoshop, Sketch, Illustrator, Axure, Balsamiq) Web: (front end) Angu Lisää

$30 AUD / tunti
(2 arvostelua)
3.1
leadconcept

Hi there, I have read everything and our team is very experienced in Ruby on Rails, so please share complete requirements or let me know from which websites, we have to scrap the data? Do you want to display the dat Lisää

$38 AUD / tunti
(1 arvostelu)
3.0
mike199

Hi, I’m a Web Designer/Developer from the UK. My name is Mike. Your project description sounds interesting to me and I do have skills & experience that are required to complete this project. Let's have a quick chat whe Lisää

$41 AUD / tunti
(1 arvostelu)
3.2
zdesign77

Hey, how is it going today? My name is Michael, I'm a web designer/developer from Boston. I've had a look at your project description and feel that my skills match your requirements perfectly. In fact, I did a very sim Lisää

$41 AUD / tunti
(0 arvostelua)
0.0
AzaLess

Hi, We are focusing on Data analytics and Rails development as well. So it should be a perfect fit. Here are the case studies about the proects where we worked with Text Processing and scraped a lot of data: Lisää

$41 AUD / tunti
(0 arvostelua)
0.0
paultitov

Thank you for your job posting. I am a talented python developer with rich experience of data scrapping and web programming(django) for 8+ years. About your task: After reading your job posting, I feel I am good Lisää

$41 AUD / tunti
(0 arvostelua)
0.0
ftp2012

Hello!, do you need a script or my work? I have scraped Amazon 150000 URL (Title, Price, Bullets, Pictures). look please at my portfolio.

$41 AUD / tunti
(0 arvostelua)
0.0
suhaa18

Dear Sir! Your project is very interesting to me. I want to discuss with you about your project and participate in your project. I have 5+ years of Python and Ruby development experience and have a good knowledge of th Lisää

$41 AUD / tunti
(0 arvostelua)
0.0
becritical

Hey there, I will develop a web scraper for online retailers using Ruby in combination with Watir. I will implement advance proxy rotation in order to achieve maximum performance. I have a very complex scraping Lisää

$37 AUD / tunti
(0 arvostelua)
0.0
$37 AUD / tunti
(0 arvostelua)
0.0