9

It is sometimes necessary to be alerted when a change is made to a web-page.

Currently, I have some simple scripts on OpenBSD that download the web-page with lynx -width 120 -dump, get relevant parts of it with grep -v and -B3 -A3 etc, save it to a special folder under a date-based filename, keep track of the latest two versions with ln -fs, and finally perform the diff, all run from cron.

The scripts work, however, they look a bit ugly, fragile and may not necessarily handle some edge cases. Is there perhaps some existing software that can do this task better and more streamlined?

Improve this question
1
  • I googled an equivalent of your question today and came here. You didn't award the solution mark, have you stuck with your own method ever since, or found something else? Would you be so kind to describe the complete setup from your OP, without the "etc", as an edit or a self-answer? If it matters, it's for this
    – pateksan
    Nov 20, 2020 at 21:00

6 Answers 6

Reset to default
5

I found the free Android app Web Alert very useful for this case. It is also one of the very few that can work with webpages that require you to log-in beforehand or submit some form sheet.

Improve this answer
4

I have been happily using https://www.changedetection.com/ for over 10 years now. Give it a URL and it will email you when any part of the page changes.

"Since 1999 ChangeDetection.com has been providing page change monitoring and notification services to internet users worldwide. Anyone can use our service to monitor any website page for changes. Additionally, webmasters use our service to improve their sites and increase traffic.

Anyone can use our free service to track the changes to any page on the net. You will be notified when the pages you are tracking have been changed and you will be able to see a concise list of the edits made to each page over time. "

Improve this answer
1
  • Someone just upvoted this, three years later. Alas, it has now been bought out & gone commercial. I personally have stopped using it. It has now moved here. You can monitor two pages for free otherwise, you have to pay :-( Wait I take that back, it all depends how often you check for changes, from 2 free checks daily to 14 weekly to 62 if you only want them checked once per month. I can live with the latter, and might use it again.
    – Mawg says reinstate Monica
    Nov 25, 2018 at 14:14
3

I've been using two tools for that purpose so far:

Initially I used urlwatch as a cron job which sends mail upon website changes. But it caused too many false positives for my taste and ignoring parts of web pages required plugin programming.

What I use now is specto despite it's no more maintained, but it does the job quite well. It sits in the system tray of your desktop and the icon changes its color from blue to orange in case one of the web pages changed.

Improve this answer
1

Best approach is to write a script to get the page via cURL, run a hash function on it and compare it to the previous hash for that page.

Depending on the elements on the actual page itself, you may need to do some processing in-between the cURL call and the hash function, to remove dynamic elements (such as headers, blocks containing ads etc..) - you could use tools such as FiveFilter's Full-Text RSS API to extract the relevant data from your webpages and hash just that.

If you're interested in file changes rather than content updates - i.e.: for example, you want to monitor changes in your file system to protect against a code injection - then you're better off versionning your code in Git or Mercurial and running a git status | grep "nothing to commit (working directory clean)" | grep -v grep | wc -l and if the script finds any modified files, alert you via email.

Improve this answer
2
  • 1
    Page hashing fails if the site has something like "Page generated in x ms".
    – Avamander
    Sep 30, 2017 at 11:42
  • @Avamander Filter that part then. Works.
    – prosody-Gabe Vereable Context
    Nov 25, 2018 at 4:29
0

Just started a project of my own. Hope someone finds it useful

sitestalker - Bulk domain/website change monitoring tool w/ screen capture, gallery, and email notification

Improve this answer
0

Huginn seems to be popular on Github and probably does what you are asking.

https://github.com/huginn/huginn

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.