I am looking for a web scraper, preferably FOSS and a standalone software. If it is a library, then I should be able to accomplish my objective with minimum amount of code. My preferred language is Python, but other languages are also welcomed. The requirements are below, in the order of most important to least important, as I guess most softwares are only able to satisfy a subset of these requirements:
- Traverse the webpages recursively, which means for each webpage visited, it should be able to also visit all the hyperlinks on that webpage. Then, download both the webpages and the attachments that the webpages link to.
- I should be able to limit the webpages that I want to scrape. Specifically, I could set requirements such as "only scrape the domain of example.com" or "only scrape the subdomains of example.com". Additionally, I should also be able to only recursively scrape the webpages of e.g. example.com, but also download the attachments from e.g. examplestatic.com that these webpages link to.
- It should be able to handle HTTP redirects.
- The scraping process should be paralellized as I have to scrape a lot of webpages. It could be either async or multi-threaded.
- A list of URLs traversed, whether they have been successfully scraped, and whether they are redirects should be provided in the output.
- For attachments downloaded, it should be able to correct use the filename specified in the
Content-Dispositionheader. - Login with SAML 2.0/Shibboleth SSO authentication.
- Some webpages I scrape have download buttons implemented in JavaScript. The scraper should be able to click these buttons and download the files. More specifically, these webpages are Box Drive file share links.
A project I found to be able to satisfy 1, 2, 4, and 5, is Steven Hé (Sīchàng)'s recursive scraper.
