Does anybody know a tool with which I can download files from a Amazon S3 Requester-Pays Bucket for Linux? (I want to bulk download arxiv.org)
The tool should be gratis.
Software Recommendations Stack Exchange is a question and answer site for people seeking specific software recommendations. It only takes a minute to sign up.Sign up to join this community
Tim Kay's Perl wrapper with support for S3, including the requester pays feature works well. You can run it in parallel using something like xargs -P 32 to speed things up. When requester pays is turned on for an S3 bucket that means that the requester pays for out of Region data egress from that bucket. If you move data to an EC2 instance in the same region as the S3 bucket, there is no charge.
Listing an S3 bucket would look like this. ./aws ls -1 --requester bucketName/
As stated on this page: http://arxiv.org/help/bulk_data_s3
you can use "s3cmd" to bulk download arxiv.org. (s3cmd is free and open source)
If you use the last versions (> 1.6.0), you don't even need to apply the patch that is described in the help page. You just need to use the "--requester-pays" command line flag of the tool.
You can find the last version of the tool on github: https://github.com/s3tools/s3cmd or you can download a not too old version of it from most linux distribution repositories.
The solution is pretty simple (on Linux systems, at least):
$ pip install s3cmd $ s3cmd --configure $ s3cmd get --recursive --skip-existing s3://arxiv/src/ --requester-pays
One nice property about this command is
--skip-existing. You can stop the command (Ctrl+C) at any time and continue it later.