Simple .Net sitemap generator
-----
IMPORTANT Please write why you are the right coder for this project and what experience you
have that is similar to this project
-----
Want a simple .Net sitemap generator which must have 1 input text file with
1 url (example: [login to view URL]).
Then program spiders all php, htm and html pages in that domain (max 3000 urls) and makes
an output text file with 2 columns (TAB DELIMITED) :
column 1 contains title-tag of the url (max 60 first characters from title tag)
column 2 contains the url
Example of out format:
Houston Texans Kareem Jackson Jersey team color blue [login to view URL]
Program must be multithread, 15 threads, 3 retries, 45 seconds timeout. Spider max 3000 urls.
that's it :)