Development of WebCrawler for job offers

The outcome of the development project hast o be a WebCrawler for websites (mainly company sites) containing job offers which allows the extraction of content and stores it in a database.

Extraction of content

The content of the websites containing the job offers (job description) has to be extracted in a text format and exported in a mysql database whereas the format should be “onlinedate, offlinedate, id, category, url, jobtitle, jobdescription, e-mail (if existing), phone (if existing), contactperson (if existing)”.

Validity of jobs

The software also has to verify if jobs are taken offline by a company and are not online anymore. The jobs than has to be marked offline in the database with the corresponding date the job has been taken offline (date of crawling process where the software has identified that the job is not online anymore).

Definition / Configuration of URLs

The configuration of the software must allow, that the URLs of the websites which have to crawled including SUB-URLs can be defined. The URLs are mainly company webpages whereas the software has to identify on which pages job offers are published.

Please do not bid on the project if you are not familiar with the technology of web crawlers.

Configuration of Keywords

Besides the URLs also keywords can be defined whereas a website will only be extracted if predefined keywords are on the site. Keywords can be grouped in a category so that a job which has been found can be put in a certain job category (e.g. consulting, banking,…).

Progress and statistics

A small progress and statistic module of the software always has to show the progress of a crawling process as well as the result (websites crawled, new jobs found, jobs updated, jobs taken offline). The data has to be provided for each new crawling process.

Error Messages

In order to identify problems with a crawling process an error log has to be written which allows the identification of the problems which have occurred during a crawling process.


The WebCrawler must ensure a fast crawling process.

PLEASE ONLY bid on the project if you are a very experienced developer and are familiar with crawling technologies.

Taidot: C-ohjelmointi, tietojenkäsittely, Java, Visual Basic

Näytä lisää: written jobs, well published, websites problems, websites for online jobs, websites development company, website development technologies, web site development process, website development online company, website development offers, website development jobs online, website developer online jobs, web jobs development, web job online, web development technology, web development online jobs, web development online job, web development jobs online, web development description, web development company websites, web development company description, web developer online jobs, web developer jobs online, web developer database problems, webcrawler software, technology job

Tietoa työnantajasta:
( 4 arvostelua ) Hamburg, Germany

Projektin tunnus: #437220

3 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


for a bit high budget,i can help you

$1500 USD 50 päivässä
(8 arvostelua)

Hello, please check PMB

$1500 USD 30 päivässä
(2 arvostelua)

Hi, please check PMB.

$950 USD 21 päivässä
(1 arvostelu)