I am part of a team of 5 active civilians doing public policy research research. We mainly scrutinize the Govt's policies, identify facts in its claims, assess feasibility etc. We mostly depend on our Government websites for our research on public policy.
The problem is that their policy documents, information and data related to policies, press releases, orders etc disappear as quickly as possible. They willfully take down such information to save themselves from the scrutiny of civil society.
Ours being a small team we focus on one policy at a time and by the time we get to other policy, relevant documents/websites altogether disappear and at times we need to refer to something that was present on their website few months or even few years back.
Locally archiving websites is the best possible solution. Something like wayback machine is ideal for our purpose. Is there an opensource alternative to their wayback machine?
Requirements:
- Lets me configure list of desired websites to archive
- Periodically checks websites and archives them only if there are some changes on the website
- Lets me browse through the archives like this
I am a former decent level programmer, I can write some small scripts if needed (language doesn't matter).