A simple web crawler
- Collects the links from the websites
- Save the requested information in a mySQL database
- It must be possible to run mutiple crawlers
- The program must use threads for speed!
- Respect the robots exclusion standard
- Crawling depth
- Number of threads
- User agent of the crawler
- List of domains that will be crawled
Information to collect:
- Website Title:
- Meta Description:
- Meta Keywords:
15 freelanceria on tarjonnut keskimäärin 206 $ tähän työhön
I have been creating such bots from a long time now. I assure you quality work and 100% satisfaction guaranteed. I've read your requirements and easily write such a crawler in java. Please check your PMB.
"SCRAPER AND CRAWLER HERE". We are new to [url removed, login to view] so don't have reviews here. But you can check our reviews in elance, odesk and scriptlance. Please check your PMB for details.