I would like to download a website and all of its first degree sub-urls, as txt or html files. For example, I would like to download:

google.com and google.com/10001 and google.com/10002 etc. but not google.com/1001/1002

HTTrack is not slow at downloading, but before downloading, it scans all urls which is very very slow.

Software must run on Windows and be gratis.

I need to bypass robot.txt


1 Answer 1


Shouldn't HTTrack allow to set the recursion level? See this:

httrack THE_SITE_URL -O /tmp/shoesizes -r50

In this example, we limit the directlry depth to 50 levels deep. As a general rule, web sites don't go much deeper than 20 levels or so, and if you think about it, if there are only 2 subdirectories per directory level, a directory structure 50 deep would have about 10 trillion directories. Of course many sites have a small number of files many levels deep in a directory structure for various reasons. In some cases, a symbolic link will cause an infinite recursion of directory levels as well, so placing a limit may be advisable.

-----Otherwise--- VisualWget should help you do this. Setting the depth/level of the recursion should help you.

VisualWget third-party manual

However, the link to VisualWget is broken in that manual. Google a bit and you should find the actual one. I will update this a little later. (I did use it a yesterday at home.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.