2

I want to mirror a website regularly. Right now I use httrack started by cron.

The problem is that sometimes the website has a database problem and says just that (HTTP status still remains "200" in those cases). Then my last working mirror is overridden by a copy just saying “Database problem”.

I could write a script that does something like

curl website.com
if not “Database problem”:
   httrack website.com

I can not believe that a mirror program doesn't have such a feature. Is there a tool that will only mirror on such an inline condition?

3
  • 1
    What HTTP status code does the website return on "database error"? If other than 2xx, that might be helpful (I could imagine a 5xx code here: 500/503 would match that situation; if you can configure that, a 304 ("not modified") could be triggered as well). I rarely use mirroring scripts, but could imagine them configurable to act on conditions like this. wget e.g. skips content on error unless --content-on-error is set to on. – Izzy Sep 12 '14 at 8:49
  • 1
    Btw: What OS must the tool run on? – Izzy Sep 12 '14 at 8:50
  • 2
    Status code is still 200 and I can't change that. Program has to run on Linux. – t2d Sep 12 '14 at 9:55
1

Might I suggest using rsync instead? Like curl it can handle encrypted connection and works fine with cron. It was written for mirroring purposes rather than for just downloading and should do what you want. I have used it for years without any issues. Rsync can push or pull a site so you can write a script to control it then decide to send.

Could you explain what this database problem is and what program is causing/reporting it? That would be important if it still needs a workaround.

2
  • 1
    I think rsync is not an option because I'm talking about a dynamic website (PHP) and just want to store the resulting html files. The problem is the temporarily unavailability of the database server to the webserver. Thus PHP reports this error. – t2d Sep 12 '14 at 7:47
  • I am also using PHP with rsync, using mySQL as the database. In the future, please post stats on your project with the question that will help people assist you. – Jeff Clayton Sep 12 '14 at 12:29
-1

So I hacked it myself. quick & dirty, but works for my use case:

chttrack (at Github)

It's a little module written in Python, so you will need to have Python installed in order to use it. If anyone wonders about the name: the "c" stands for "conditional" ;)

2
  • 1
    As written currently, I'd expect it to never update any page containing the words "failed" or "error" (having put that in a comment here, you no longer can mirror this page, for example). So it really needs some more work, but might be a good start. (note: the downvote is not mine). – Izzy Sep 12 '14 at 11:57
  • So you are saying you have found a solution to your problem on your own then. – Jeff Clayton Sep 12 '14 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.