I need an open source system that can crawl websites to gather PDF documents and then index them.
-
What format should the index take? An HTML page? Something else? Should the app only look at document titles (and PDF extension), or also examine the contents? – Mawg says reinstate Monica Jul 31 '19 at 13:22
Have you tried httrack to copy website data locally. I think you will be able to change the parameters of the httrack config and download the PDF only files as you have mentinoed here.
