I would like to monitor several websites with 100K's of pages of content for changes on their individual pages. What is the most effective strategy to do this?

I have currently implemented a scraper that looks for changes on the pages and triggers a workflow if the pages have new content. The problem I am facing is that this scraper is round-robin based and in most cases there is a significant delay between the time the update is published on the site to when I actually crawl it. I am using a proxy network to do this large scale crawling.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.