Development of WebCrawler / bot for job offers

The outcome of the development project hast o be a WebCrawler for websites (mainly company sites) containing job offers which allows the extraction of content and stores it in a database.

Extraction of content

The content of the websites containing the job offers (job description) has to be extracted in a text format and exported in a mysql database whereas the format should be “onlinedate, offlinedate, id, category, url, jobtitle, jobdescription, e-mail (if existing), phone (if existing), contactperson (if existing)”. The software also has to be able to crawl job portals like [url removed, login to view], [url removed, login to view], [url removed, login to view],...

Example Pages:

[url removed, login to view]://[url removed, login to view]://[url removed, login to view]

[url removed, login to view]

[url removed, login to view]


It has to be considered that on some of the company websites you need a search engine in order to get to the jobs.

Validity of jobs

The software also has to verify if jobs are taken offline by a company and are not online anymore. The jobs than has to be marked offline in the database with the corresponding date the job has been taken offline (date of crawling process where the software has identified that the job is not online anymore).

Definition / Configuration of URLs

The configuration of the software must allow, that the URLs of the websites which have to crawled including SUB-URLs can be defined. The URLs are mainly company webpages whereas the software has to identify on which pages job offers are published.

Please do not bid on the project if you are not familiar with the technology of web crawlers.

Configuration of Keywords

Besides the URLs also keywords can be defined whereas a website will only be extracted if predefined keywords are on the site. Keywords can be grouped in a category so that a job which has been found can be put in a certain job category (e.g. consulting, banking,…).

Progress and statistics

A small progress and statistic module of the software always has to show the progress of a crawling process as well as the result (websites crawled, new jobs found, jobs updated, jobs taken offline, websites a job was found). The data has to be provided for each new crawling process.

Error Messages

In order to identify problems with a crawling process an error log has to be written which allows the identification of the problems which have occurred during a crawling process.

Dublicate Check

The software has to identify if job offers are posted on different sites. For example on a company home page as well as on a job portal. This information has to be stored in the database.


The WebCrawler must ensure a fast crawling process.


The software has to be documented in a proper way (functions, methos, etc.) so that a third party is able to understand it.


The payment amount will be put in an escrow agreement. After the software has been finished, delivered and quality assured the escrow payment will be released.

Taidot: .NET, C-ohjelmointi, tietojenkäsittely, Java, Visual Basic

Näytä lisää: www monster jobs . com, www monster jobs, www monster com, www monster, www job search com, www job search, www escrow com is it a proper company, written jobs, where you search order for software development, well published, websites search development, websites problems, websites for online jobs, websites development company, website for online development with html, web site development process, website development online company, website development offers, website development jobs online, web search jobs online, web project documentation format, web portal development software, web page for software development company, web page development sites, web page development online

Tietoa työnantajasta:
( 4 arvostelua ) Hamburg, Germany

Projektin tunnus: #437267

17 freelanceria on tarjonnut keskimäärin 1182 $ tähän työhön


Dear friend please kind view PMB for more details thanks

1000 $ USD 15 päivässä
(53 arvostelua)

Hi, Thanks for giving an opportunity to place a [url removed, login to view] kindly go through PM. Regards, Bhavik

1500 $ USD 20 päivässä
(8 arvostelua)

I done a lot of similar projects. Please check PMB for demo.

1350 $ USD 30 päivässä
(7 arvostelua)

fake bid for PMB

750 $ USD 1 päivässä
(25 arvostelua)

Hi, Greeting from Astinsoftech.!!! We are ISO 9001:2000 Certified Software Development Company I have studied your requirements in detail and based on our assessment I am very confident that we will execute t Lisää

1100 $ USD 15 päivässä
(3 arvostelua)

We are twelve team members who have professionally mastered the skills of Web Design and Development. We have provided excellent quality service within our clients’ budget to their utmost satisfaction. We offers an Lisää

750 $ USD 35 päivässä
(2 arvostelua)

We possess extensive experience of developing numerous high-end websites and are highly organized and adept at meeting tight deadlines that are so common in this industry. Please see PMB for more details.

1500 $ USD 30 päivässä
(1 arvostelu)

Please, check PMB

1500 $ USD 0 päivässä
(2 arvostelua)

can do [url removed, login to view]

1200 $ USD 40 päivässä
(2 arvostelua)

In your project i have to work on java for better security purpose of database so i bidding $ 1050.

1050 $ USD 15 päivässä
(1 arvostelu)

Hi, I am system analyst and have 5+ years of experience in .NET technology. I worked on Windows application, Windows service and Web application. I also worked on WCF and WF. I am really intrested in this. I hope, Lisää

1000 $ USD 30 päivässä
(0 arvostelua)

Please check PMB

751 $ USD 30 päivässä
(0 arvostelua)

Plz check your PMB for further details..

1250 $ USD 25 päivässä
(0 arvostelua)

Hi,Please check PM. Thanks.

1200 $ USD 30 päivässä
(0 arvostelua)

Hi, I have experience of working on WebCrawler bot. I have been working for a WebCrawer boot which download data from hundreds of site. Please see my PM. Thanks, Thuc

1500 $ USD 20 päivässä
(0 arvostelua)

HI,Check PM please.

1200 $ USD 20 päivässä
(0 arvostelua)

Hello, We are Software Development Company from India provides IT services, having a great experience in development and designing of complex solutions. We have well trained and fully experienced developers and prog Lisää

1500 $ USD 35 päivässä
(0 arvostelua)