Modify a Perl Webcrawler to find specific links!


I have an existing crawler (see bottom for more details)*

This crawler crawls webpages/directories and collect links together with metatags description text to store the results into a text file

I want the Crawler to be modified to search a list of Directories and find my submitted domains on those directories and put the results into a text file like

TARGETDOMAIN - DIRECTORYDOMAIN - DIRECTORYPAGE (where the targetdomain is listed)


[url removed, login to view] - [url removed, login to view] - [url removed, login to view]

After the crawler found the Target Domain / Target Domains it will stop crawling and moves on to the next directory and then to the next etc!


I do a lot of directory submissions and those bastards never send you a confirmation email when they list you (well some do) with this crawler I am able to see which directory is going to list me or never listed me at all!


- Directories have usually a high number of pages up to 100.000 and more - the crawler should be able to remember which pages it already

crawled to save time and resources and speed up the process!

- Crawler should be able to crawl a huge number of Directory in Batches! (read from a .txt file)

- Crawler should be able to pick up on the last status in case it has to be restarted!

- At the moment the browser window has to stay open - it will refresh itself after a given amount of time but it would be nice if the script could do the job without the browser window being open and

just send an email when the job is done!

- Is it better to check all the directories for only one Target Domain or is it more economical to search for several Target Domains????????

(in case I already submitted several domains to the directories)

THE Crawler

the crawler is written in perl and was designed to crawl webpages/directories and collect links together with metatags description text to store the results into a text file and let you use this kind of text/links via "includes" as content on your pages! - basically its a kind of text/content scraper


- The script has already build in features which could be beneficial like:

Max. Number of Parallel Requests:

Max. CPU Time (seconds):

Delay Between Requests (seconds):

Password Protected Admin Area!


##Money will be paid after I did a successful test with 100 Directories!-

Important please put CRAWLER in your reply - so I knnow you actually read the description ;-)))##

NOTE: I can grant you access to the crawler so you can check the source code and then decide if you are able to modify it successfully! Just send me a pm!

Budget is around 150$

Taidot: Perl, Tietojen kaavinta verkosta

Näytä lisää: without being paid, where to find a job, web scraping process, web crawler job search, web crawler features, web content protection, use case includes, source code protection, scraping web content, save a lot, pick a job for me, perl search script, password find my email, job web crawler, huge things, http www webcrawler com, find the password, find the job, find perl, find directories perl, find code, find a password, features of a web crawler, directory web scraping, c++ find

Tietoa työnantajasta:
( 11 arvostelua ) Barcelona, Spain

Projektin tunnus: #524164

13 freelanceria on tarjonnut keskimäärin %project_bid_stats_avg_sub_26% %project_currencyDetails_sign_sub_27% tähän työhön


We can help in your project, please check PMB to see our related experience.

$250 USD 4 päivässä
(44 arvostelua)

I am willing to work on the project

$200 USD 3 päivässä
(47 arvostelua)

Check the PM pls.

$150 USD 3 päivässä
(57 arvostelua)

I have worked on similar projects. I am ready to go.

$200 USD 10 päivässä
(5 arvostelua)

Kindly check PM for more details.

$149 USD 7 päivässä
(4 arvostelua)

CRAWLER I believe I can help you with your project. I put $150 as the bid, but that may be negotiable depending on the difficulty to modify the source code that you have. Spiders aren't too complicated, but modifying Lisää

$150 USD 3 päivässä
(1 arvostelu)

Check PM for details.

$100 USD 5 päivässä
(2 arvostelua)

Greetings we can complete perl scrapper modification for you no problem. It will probably only take a few hours but we bid more just to be safe. Please let us help you.

$150 USD 5 päivässä
(0 arvostelua)

Hello, I am highly motivated to analyze the source. I will get back to you with more details then. I am well skilled in Perl.

$100 USD 10 päivässä
(0 arvostelua)

I can do it very fast. Please see my PM.

$150 USD 3 päivässä
(0 arvostelua)

crawler Give me a chance to execute this project

$120 USD 4 päivässä
(0 arvostelua)

Please check PM

$100 USD 3 päivässä
(0 arvostelua)

Hi Greetings ..............!!!! Your project sounds very interesting, and immediately I would say that it is something I can help you with. I am completely clear with the requirement and very much interested to w Lisää

$200 USD 10 päivässä
(0 arvostelua)