I need a crawler made for the social network bebo. I want it to crawl the site obtaining usernames and saving them into a .txt file. i dont want it to navigation away from bebo.com.
I think the best way for it to work would be to start at [url removed, login to view], then go to one of the random profiles under 'People'. Save their username, then go to a random profile on their 'top 16' friends module. Save that username and so on.
CAPTCHA codes will show up after 1000 clicks so i will need a message saying this so i can go onto bebo and type it in.
The username for a profile is contain between < and > on their main page. eg. brucey666 dot bebo dot com it will say "Brucey <brucey666>" so brucey666 is the username.
Some profiles are set to private. In that case the crawler will need to go back and select a different profile.