My current Bot needs the following features added.
- Doesn't click through gateway pages. Example: Enter or Exit and Yes I’m over 18 or No I’m not over 18 pages
- Doesn't scrape from I-frame pop ups contact pages
- Inconsistencies when scraping from whois. Example: copies registration service providers email instead of administrative.
- Specifying page depth limit to go in search, right now we don’t know how deep the scraper goes. We might need it to go very deep or only top 100.
- Add blacklist to avoid specific websites ([url removed, login to view], Wikipedia.com..ect)
- Add ability to add “look for” AIM users (ICQ,MSN,Skype) and scrape that info as well (we are targeting Adult websites and most these guys use some sort of AIM service)