My current Bot needs the following features added.
- Doesn't click through gateway pages. Example: Enter or Exit and Yes I’m over 18 or No I’m not over 18 pages
- Doesn't scrape from I-frame pop ups contact pages
- Inconsistencies when scraping from whois. Example: copies registration service providers email instead of administrative.
- Specifying page depth limit to go in search, right now we don’t know how deep the scraper goes. We might need it to go very deep or only top 100.
- Add blacklist to avoid specific websites ([url removed, login to view], Wikipedia.com..ect)
- Add ability to add “look for” AIM users (ICQ,MSN,Skype) and scrape that info as well (we are targeting Adult websites and most these guys use some sort of AIM service)
Integrated Proxy Scrapper
Integrate HMA/heroic VPN
Send Email After searching email
Add limit to the Bot (Number of sites to scrape)
Add option to Scrape AIM or not
Add a UI to the Bot
Add search in Google Places to find local businesses.
Add Select Search engine option (Bing,Yahoo,Google)
Export File via Email On completion.
Export Files in Excel Format(Site name , Email , AIM)
Scrape Phone Numbers (If available)
Scrape Contact Name
If contact Form Available Instead Fill That(With User Permission)
Take Decisions For bot Via sending email replys.