Peruttu

Modify a Perl Webcrawler to find specific links!

DESCRIPTION:

I have an existing crawler (see bottom for more details)*

This crawler crawls webpages/directories and collect links together with metatags description text to store the results into a text file

I want the Crawler to be modified to search a list of Directories and find my submitted domains on those directories and put the results into a text file like

TARGETDOMAIN - DIRECTORYDOMAIN - DIRECTORYPAGE (where the targetdomain is listed)

Example:

[url removed, login to view] - [url removed, login to view] - [url removed, login to view]

After the crawler found the Target Domain / Target Domains it will stop crawling and moves on to the next directory and then to the next etc!

REASON:

I do a lot of directory submissions and those bastards never send you a confirmation email when they list you (well some do) with this crawler I am able to see which directory is going to list me or never listed me at all!

THINGS TO CONSIDER:

- Directories have usually a high number of pages up to 100.000 and more - the crawler should be able to remember which pages it already

crawled to save time and resources and speed up the process!

- Crawler should be able to crawl a huge number of Directory in Batches! (read from a .txt file)

- Crawler should be able to pick up on the last status in case it has to be restarted!

- At the moment the browser window has to stay open - it will refresh itself after a given amount of time but it would be nice if the script could do the job without the browser window being open and

just send an email when the job is done!

- Is it better to check all the directories for only one Target Domain or is it more economical to search for several Target Domains????????

(in case I already submitted several domains to the directories)

THE Crawler

the crawler is written in perl and was designed to crawl webpages/directories and collect links together with metatags description text to store the results into a text file and let you use this kind of text/links via "includes" as content on your pages! - basically its a kind of text/content scraper

ALREADY BUILT IN FEATURES:

- The script has already build in features which could be beneficial like:

Max. Number of Parallel Requests:

Max. CPU Time (seconds):

Delay Between Requests (seconds):

Password Protected Admin Area!

SMALL PRINT:

##Money will be paid after I did a successful test with 100 Directories!-

Important please put CRAWLER in your reply - so I knnow you actually read the description ;-)))##

NOTE: I can grant you access to the crawler so you can check the source code and then decide if you are able to modify it successfully! Just send me a pm!

Budget is around 150$

Taidot: Perl, Tietojen kaavinta verkosta

Näytä lisää: without being paid, where to find a job, web scraping process, web crawler job search, web crawler features, web content protection, use case includes, source code protection, scraping web content, save a lot, pick a job for me, perl search script, password find my email, job web crawler, huge things, http www webcrawler com, find the password, find the job, find perl, find directories perl, find code, find a password, features of a web crawler, directory web scraping, c++ find

About the Employer:
( 11 reviews ) Barcelona, Spain

Projektin tunnus: #524164

13 freelanceria on tarjonnut keskimäärin 155 $ tähän työhön

SigmaVisual

We can help in your project, please check PMB to see our related experience.

250 $ USD 4 päivässä
(44 arvostelua)
6.5
srinichal

I am willing to work on the project

200 $ USD 3 päivässä
(47 arvostelua)
6.3
Mindon

Check the PM pls.

150 $ USD 3 päivässä
(57 arvostelua)
6.1
edatawiz

I have worked on similar projects. I am ready to go.

200 $ USD 10 päivässä
(5 arvostelua)
3.5
yosif4444

Kindly check PM for more details.

149 $ USD 7 päivässä
(4 arvostelua)
2.6
geekyone

CRAWLER I believe I can help you with your project. I put $150 as the bid, but that may be negotiable depending on the difficulty to modify the source code that you have. Spiders aren't too complicated, but modifying Lisää

150 $ USD 3 päivässä
(1 arvostelu)
1.5
InnoConsulting

Check PM for details.

100 $ USD 5 päivässä
(2 arvostelua)
1.0
grx3

Greetings we can complete perl scrapper modification for you no problem. It will probably only take a few hours but we bid more just to be safe. Please let us help you.

150 $ USD 5 päivässä
(0 arvostelua)
0.0
OnyxSoft

Hello, I am highly motivated to analyze the source. I will get back to you with more details then. I am well skilled in Perl.

100 $ USD 10 päivässä
(0 arvostelua)
0.0
ajinkya314

I can do it very fast. Please see my PM.

150 $ USD 3 päivässä
(0 arvostelua)
0.0
sreeiit

crawler Give me a chance to execute this project

120 $ USD 4 päivässä
(0 arvostelua)
0.0
logicsaw

Please check PM

100 $ USD 3 päivässä
(0 arvostelua)
0.0
pearlveeram

Hi Greetings ..............!!!! Your project sounds very interesting, and immediately I would say that it is something I can help you with. I am completely clear with the requirement and very much interested to w Lisää

200 $ USD 10 päivässä
(0 arvostelua)
0.0