1

I'm trying to download/scrape about 3000 urls and save them as HTML locally (I actually only need the contents of a certain div from each URL but the whole HTML works to). No images, js, css etc. are required from the sites.

They are all on the same domain. (domain.com/001/ domain.com/002/ domain.com/003/ etc.) The site needs authentication.

What I've tried:

ScrapeStorm : Fails to recognize the data I want to scrape.

Httrack : Doesn't work because when I include the authentication cookie (XSRF Token) and my useragent in the HTML header httrack tells me the header is too long.

I've searched but couldn't find any solution that is a)free and b) requires no coding. Any help would be greatly appreciated.

OS can be either Linux or windows. The maximum budget would be around $50 since this is really just a personal project.

Thank you.

0

1 Answer 1

1

It's not obvious exactly what you want, but would this work (Linux bash shell):

$ URL="https:…"
$ lynx -dump -source "$URL" >tempfile

If that's good enough, then:

$ URL="https:…"
$ lynx -dump -source "$URL"  \
    #sed or awk command to extract the required <div>  \
>tempfile

And if that works okay, then:

$ while read URL
do
    lynx -dump -source "$URL"  \
        #sed or awk command to extract the required <div>  \
    >dump-dir/"$URL"
done <file-of-3000-urls

I don't know what authentication is required, but lynx has -auth=ID:PASSWD.
(Be aware that everyone on the system can see that line using ls.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.