Simple .Net spider program(repost)

IMPORTANT It is important that you write what specific similar projects you have made before. Below are the specs --------------------------------------- Program gets input list with 1 sub-page url (like, [[login to view URL]][1]) per line. Program then change the sub-page url into a base-domain (like, [[login to view URL]][2]) and then spiders up to 3 levels down (index page is level 1 - and any other internal webapges that it poitns to is level 2 - so, the internal pages that it points to are level 3). Then program makes 3 output files: [login to view URL] [login to view URL] and urls_where_DIDNT_find_page_at [login to view URL] ----- Program has another input text file which contains 1 word/phrase per line. Program will then spider urls or text hyperlinks link first that contain any of these phrases. Program must max spider 500 urls per site and with 2 threads per site and timeout after 30 seconds with 3 retries. Program must spider 10 sites at a time (so, it must be multithreaded). Enclosed are the 2 input files for the project specs described above so you can easily test it. 1) in input url file then some urls are with or without "http://" (to be sure that program can understand both with and without "http://" on a given url-line. 2) the other input file contains a few words that program - after it has read index page - that it should try to spider internal pages that contain these words first in either text hyperlink or as part of their url. ## Deliverables 1) Complete and fully-functional working program(s) in executable form as well as complete source code of all work done. 2) Deliverables must be in ready-to-run condition, as follows (depending on the nature of the deliverables): a) For web sites or other server-side deliverables intended to only ever exist in one place in the Buyer's environment--Deliverables must be installed by the Seller in ready-to-run condition in the Buyer's environment. b) For all others including desktop software or software the buyer intends to distribute: A software installation package that will install the software in ready-to-run condition on the platform(s) specified in this bid request. 3) All deliverables will be considered "work made for hire" under U.S. Copyright law. Buyer will receive exclusive and complete copyrights to all work purchased. (No GPL, GNU, 3rd party components, etc. unless all copyright ramifications are explained AND AGREED TO by the buyer on the site per the coder's Seller Legal Agreement). ## Platform .Net