Amazon S3 Tool to download parts of Requester-Pays Bucket

Question

Does anybody know a tool with which I can download files from a Amazon S3 Requester-Pays Bucket for Linux? (I want to bulk download arxiv.org)

The tool should be gratis.

Answer 1

Tim Kay's Perl wrapper with support for S3, including the requester pays feature works well. You can run it in parallel using something like xargs -P 32 to speed things up. When requester pays is turned on for an S3 bucket that means that the requester pays for out of Region data egress from that bucket. If you move data to an EC2 instance in the same region as the S3 bucket, there is no charge.

http://timkay.com/aws/

Listing an S3 bucket would look like this. ./aws ls -1 --requester bucketName/

Answer 2

As stated on this page: http://arxiv.org/help/bulk_data_s3

you can use "s3cmd" to bulk download arxiv.org. (s3cmd is free and open source)

If you use the last versions (> 1.6.0), you don't even need to apply the patch that is described in the help page. You just need to use the "--requester-pays" command line flag of the tool.

You can find the last version of the tool on github: https://github.com/s3tools/s3cmd or you can download a not too old version of it from most linux distribution repositories.

Answer 3

Get Amazon credentials

• https://console.aws.amazon.com/s3
• Click on your name in the upper right corner, then on "Security Credentials"
• Click on Access Keys (Access Key ID and Secret Access Key)

Use s3cmd

The solution is pretty simple (on Linux systems, at least):

$ pip install s3cmd
$ s3cmd --configure
$ s3cmd get --recursive --skip-existing s3://arxiv/src/ --requester-pays

One nice property about this command is --skip-existing. You can stop the command (Ctrl+C) at any time and continue it later.