Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.
Sign up to join this community
I would like to download a website and all of its first degree sub-urls, as txt or html files. For example, I would like to download:
google.com and google.com/10001 and google.com/10002 etc. but not google.com/1001/1002
HTTrack is not slow at downloading, but before downloading, it scans all urls which is very very slow.
Software must run on Windows and be gratis.
I need to bypass robot.txt
Shouldn't HTTrack allow to set the recursion level? See this:
httrack THE_SITE_URL -O /tmp/shoesizes -r50
In this example, we limit the directlry depth to 50 levels deep. As a general rule, web sites don't go much deeper than 20 levels or so, and if you think about it, if there are only 2 subdirectories per directory level, a directory structure 50 deep would have about 10 trillion directories. Of course many sites have a small number of files many levels deep in a directory structure for various reasons. In some cases, a symbolic link will cause an infinite recursion of directory levels as well, so placing a limit may be advisable.
-----Otherwise--- VisualWget should help you do this. Setting the depth/level of the recursion should help you.
However, the link to VisualWget is broken in that manual. Google a bit and you should find the actual one. I will update this a little later. (I did use it a yesterday at home.)
