I am working on porting static content from one site to another. I have chosen to use HTML Agility Pack for my scraping, but I need a crawler that can be configurable. I'd like to use a crawler that lets me use a library instead of a tool and I want the language to be in C# or .NET. The things I need to configure are basically staying within the domain and whitelisting certain pages based on a url substring. A plus would also be a crawler that does not follow any links that have been already traversed. Another plus would be if its a packet I can get via the Visual Studio suite.

I really enjoy Scrapy in Python and am hoping for a version that is most similar to that.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.