I would like to monitor several websites with 100K's of pages of content for changes on their individual pages. What is the most effective strategy to do this?
I have currently implemented a scraper that looks for changes on the pages and triggers a workflow if the pages have new content. The problem I am facing is that this scraper is round-robin based and in most cases there is a significant delay between the time the update is published on the site to when I actually crawl it. I am using a proxy network to do this large scale crawling.