Software to archive desired websites like wayback machine of Internet Archive?

Question

I am part of a team of 5 active civilians doing public policy research research. We mainly scrutinize the Govt's policies, identify facts in its claims, assess feasibility etc. We mostly depend on our Government websites for our research on public policy.

The problem is that their policy documents, information and data related to policies, press releases, orders etc disappear as quickly as possible. They willfully take down such information to save themselves from the scrutiny of civil society.

Ours being a small team we focus on one policy at a time and by the time we get to other policy, relevant documents/websites altogether disappear and at times we need to refer to something that was present on their website few months or even few years back.

Locally archiving websites is the best possible solution. Something like wayback machine is ideal for our purpose. Is there an opensource alternative to their wayback machine?

Requirements:

• Lets me configure list of desired websites to archive
• Periodically checks websites and archives them only if there are some changes on the website
• Lets me browse through the archives like this

I am a former decent level programmer, I can write some small scripts if needed (language doesn't matter).

Answer 1

The wayback machine is open source:

https://github.com/internetarchive/wayback

It is implemented in Java and uses hadoop. It looks like both the web crawler and the web front end (the very one you cite in your question) are there.

However, the wayback machine is a website, not a project that has been reused by many people and carefully honed to be an open source "product" than anybody can just pick up and use without significant effort. Customizing it, or any other similar tool, to do exactly what you need may turn out to be more work than just writing a few scripts. However, you may be able to get something valuable going with relatively little effort even if it's not really adapted to your needs.

I imagine the people at the Internet Archive will feel positively towards your goals, so perhaps they can point you in the right direction? The maintainer(s) of the wayback machine project in particular might be open to carefully considered questions about how to implement modifications.

Answer 2

I would suggest python + scrapy to scrape the web sites on a regular basis - you may also need the requests library for some pages - and storing the results in a directory structure that is version controlled with Mercurial or possibly using MongoDB.

• Python is Free, Open Source and fast to develop with.
• Scrapy is very good at scraping static pages, (requests may be need for pages that are dynamically created) and is quite easy to use. Scrapy is Free and Open Source.
• Mercurial will let you revert pages/sites to any point in the past quickly and easily - it is also very good at not marking as changed files that are no different from last time. Mercurial is written in python, Free and Open Source.
• MongoDB could also provide a compact storage for the web data - a nice example is here. MongoDB is an open-source, document database designed for ease of development and scaling.

All of the above tools, in addition to being Free, Gratis & Open Source are also cross platform.