1. Site with with some useful data can go offline. It can be shot down by government, it can run out of money, it can go down by million other reasons. The result -- data is lost.
  2. Site can be changed, like one day there is some article, open file share, or picture, but the other day it can be removed or redacted.

  3. There still places with no internet.

  4. Security. Like you can download data with more secure, but less "realtime" or just slow networks, and then view it from offline cache without lags.

What I want:

  1. Efficient snapshots. Like vmware does to virtual machine disk -- incremental, without storing one page twice.
  2. Flexible scheduler and crawler. To make versions by demand, and on schedule, with flexible crawler rules.
  3. Search. Really need offline search on downloaded pages and documents.
  4. At least basic parsing. To not store "casino" and "enlarge your pencil" ads.
  5. Changes tracking. If you can shedule downloading site, if you parse it, which is also needed to make good seach, if you have incremental versions, why not make function of tracking changes on site, useful and obvious idea.
  6. Interface. Some browser plug-in should be brilliant, like make new versions of pages or sites, when you access them; access cached pages first with ability to fast choose "snapshot" you want or choose access network to load fresh version of some page. May be force browser to NOT ACCESS internet to load some resource for security concerns(like when you are using untrusted network). Browser extension also can help in parsing and crawling making user available to choose forms to save or to not save or to track changes on (ublock like gui).
  7. Open Source. Really. It is about security and flexibility.

So, complicated needs.

There many projects that may be related.

  1. Httrack -- first what comes in mind when you talk about site copying and offline browsing. Still it not so good at offline seaching, may be there is versionising functionality, I am not sure about this.
  2. Yacy -- decentralized search engine. It should have some parsing and caching functionality in it. But it's wiki is mainly German, poor, covering only most obvious use cases. May be there some good blogs or resource? May be I should read code, but really don't have so many time.
  3. Searx -- another open source search. Not really dived in it, but it stated that it has open source server, that means it should have some abilities for my needs.
  4. Elastic search -- Don't know much, but as I can get it, it is more like library for making tools. Not the tool itself. Includes nosql database, for fast full text searching. May be it should be used to write my own tool, but I really want to avoid writing large project by myself.
  5. Caching proxies. I read that it should "cache". But it still instrument for not actually my needs.
  6. Adblock, security proxies -- may be used with httrack or something else or strip unwanted elements, but still don't solve problems by themselves.

All that thoughts, as I believe appears not only in my mind. So it should be ready solutions(Maybe not exactly what I want but close to that.), or at least guides to make some composition of tools, or blogs/resources on that subject.

Sorry for long text. Any help or comment's appreciated.

The short Question is -- What good resources about web scraping you I should read?

  • man wget specifically the mirroring options.
    – ivanivan
    Apr 8, 2019 at 2:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy