I need an open source system that can crawl websites to gather PDF documents and then index them.
1 Answer
Have you tried httrack to copy website data locally. I think you will be able to change the parameters of the httrack config and download the PDF only files as you have mentinoed here.