I am working on porting static content from one site to another. I have chosen to use HTML Agility Pack for my scraping, but I need a crawler that can be configurable. I'd like to use a crawler that lets me use a library instead of a tool and I want the language to be in C# or .NET. The things I need to configure are basically staying within the domain and whitelisting certain pages based on a url substring. A plus would also be a crawler that does not follow any links that have been already traversed. Another plus would be if its a packet I can get via the Visual Studio suite.
I really enjoy Scrapy in Python and am hoping for a version that is most similar to that.