I am looking for some ideas (solutions would be even better) for a task I want to run on my local network.
I would like to save every web page I open in my browsers and be also able to search for them later, based on relevant keywords. As it would run as a proxy, I would also like to use it for fetching and automaticaly storing my rss feeds from my news reader.
Basicaly I'want to make something like fetching.io but running it on my hw, for privacy and availability.
I was thinking of running some sort of proxy that would intercept browser requests, locally store whole pages (html, xml...) on the disk, index them periodicaly and provide a search engine interface.
Ideally it would run on arm, debian based linux server, use some sort of web proxy (polipo, squid) and search engine (yacy, searx...).
I test ran Polipo proxy but was not able to figure out how to set it up to save complete web pages. I must have missed some settings as the pages saved only had hearders readable, the rest of the pages were not in clear text. Is Polipo not able to do this at all? Is not, is there any other proxy or web cache program that is able to do this?
What about the "search engine" part? I was looking at yacy as it also provides a function of indexing local pages for use on local network. Any suggestions or pointers in the right direction? :)
Does anyone have any advice or experiance on how to go about setting this thing? Anyone ever done something similar? Or does any software solution already exists and is not a cloud solution?
Please ask, if there is something I forgot to mention anything about the desired setup.
Every help or nudge in the right direction will be greatly appreciated.
Regards, B. K