Puppeteer JS Site Crawler(How much experience do you have with Puppeteer?)

I’m looking to build a crawler to perform a few meta and performance checks across multiple similar websites. This crawler should be built with Puppeteer and with clear unit tests.

The intent is to crawl the homepage of a site and its site map (well defined), and then a long list of sub pages which are defined on that site map (and linked sub site maps). These pages are grouped into 3 well defined categories and are generally the same. We will then run a specific list of checks across each of these page types, and another set of checks across every page globally.

The checks that I have defined and will share at the beginning of the project are quite simple, such as “does a <meta> description tag exist” and “does the page title contain some specific string”. The reason I am interested in Puppeteer (or Foxr) specifically is because I’d also like to measure things like page load time (Largest Content Paintful) and track heavy resources (such as images and scripts).

To summarize, I need a web crawler that, with an original sitemap url, will crawl all sub-sitemaps and then the pages listed within those (strictly 3~ levels deep). Once on each of those pages, we will run a series of checks (some global, some category specific) and return details on the result (sometimes a boolean, sometimes and integer).

I have fully documented this internally and will share upon beginning of the project. I’m also technical and will be reviewing your code and unit tests. To show that you’ve read this description entirely, please include the word “dinosaur” in your response. I will be happy to jump on a call with you during development to answer any questions you may have. Thank you!

Taidot: JavaScript, Web Crawling

Näytä lisää: web site crawler, site crawler perl, site crawler php, site crawler, site crawler working, site crawler import drupal, site crawler script, site crawler code python, wordpress site crawler, site crawler mac, getting data site crawler, web site crawler linux, automated site crawler, betting site crawler, simple php site crawler, sample web site crawler, site crawler asp, write web site crawler php, site crawler multithreaded, site crawler multithread

Tietoa työnantajasta:
( 12 arvostelua ) BALIK PULAU, Malaysia

Projektin tunnus: #29127417

7 freelanceria on tarjonnut keskimäärin $583 tähän työhön


Hi there, I'm Matt Sergei (do call me Matt, please) and would like to help you - yet am currently busy probably till Thursday, Feb 11th. I've read your requirements and have experience with Puppeteer - just recently s Lisää

$750 USD 12 päivässä
(39 arvostelua)

Hi there in Malaysia!!! I'm a software developer with many years of hands-on experience. I've completed many small and medium sized projects over those years. My recent accomplishments is a server that integrates 6 thi Lisää

$750 USD 10 päivässä
(9 arvostelua)

"dinosaur"Hi Muhammad Solihin. Thanks for your job posting. I just read your main idea carefully and it catches my eye. I found that I fit best for you because my skills and experiences are exactly fit to your require Lisää

$250 USD 7 päivässä
(2 arvostelua)

Dinosaur Hi I know puppeter and worked in .net so I can implement this. I am a technical architect by profession having 10+ of experience in below technologies: SharePoint Online / SPFX / Powerapps / Workflows ASP Lisää

$700 USD 8 päivässä
(3 arvostelua)

I've built several projects with Puppeteer including a twitter bot, automated tests and I can confidently accept this project! btw I'm not a bot I'm a "dinosaur".

$650 USD 10 päivässä
(1 arvostelu)

dinosaur. I have more than 4.5 years of relevant experience in the IT industry. Please feel free to chat with me for further details. Have worked with puppeteer before and can work in this project.

$730 USD 7 päivässä
(0 arvostelua)

dinosaur (T-Rex). I'm strictly Node.js developer for many years. I've used Puppeteer and Selenium to host a browser game ([login to view URL]) with some management enhancements. As you may know they require a great deal of Pr Lisää

$250 USD 7 päivässä
(0 arvostelua)