I am writing scrapers for around 100 websites using alltheplaces, built on Scrapy. My main problem is the sites keep changing. I am extracting data either from JSON files directly or by parsing them out of html, but when the html changes I am constantly having to update my code. I am scraping websites for retail businesses and am interested in information such as store name, address, latitude/longitude and opening hours.
My current planned solution is to create automated checks that throw warnings when a scraper breaks or returns surprising output so that they can be manually fixed.
Could anyone recommend a particularly robust scraping tool? Something that is less likely to break than Scrapy, or am I asking too much?
My machine is running on Windows 10.
the scraping tool is breaking. It is behaving as expected if the page which it is scrapping removes or renames aclassorid. I think that you just have to modify your code to detect when this happens and alert you, and then you will have to manually update your script. It is not really clear what you are askigjn for - should the tool make a guess for you?