4

I am looking for open source software that can be used for expose a file system tree as a service with an S3 (subset) compatible API.

I need something that allows software (i.e. GenomeSpace which has has S3 client capability) to remotely read and write files from the file system of my server as if my server was a full-blown S3 server.

Redundancy, etcetera is not a requirement. This is not full-function Object Storage: it is simply making files and directories look like object storage to an S3 client running on another machine (probably in the same data centre).

I should have mentioned that my server will run some flavour of Linux.

3
  • Might be useful to add some more details: if "software" (ready-to-use), what OS? If library (which I take from your mentioning "compatible API"), what programming language?
    – Izzy
    Mar 23 '15 at 16:41
  • Internet Archive has a S3 interface, maybe you can borrow some of their code (much of it is in GitHub).
    – Nemo
    Oct 5 '15 at 20:34
  • I haven't actually used this but you might try: s3ninja.net It sets up an S3 API for development and testing.
    – user22153
    Mar 11 '16 at 16:00
1

If you want to create a private S3-like service atop your data / file system, then you need a software stack such as Eucalyptus that implements an AWS-alike set of services atop your private infrastructure.

The specific Eucalyptus component you'd need is called "Walrus."

-1

You can try s3fs for Linux, or possibly wins3fs for Windows (though its last update is forever old).

There are also commercial alternatives like ExpanDrive, but these fail your "open source" criteria.

I would caution you to test whatever solution you pick rather carefully. I have used both open and proprietary approaches to "S3 as a file system," and have never been satisfied with their reliability. Trying to couple what fundamentally is a remote, high-latency resource with a software system (the file system) that expects a very local, low-latency resource--well, the two often don't match up very well. I've found that I can copy files beautifully to and from S3...and then a few minutes later, when I try to copy more files, the connection is somehow broken. This is an an environment with 50 Mbps network connections, in which software like ssh and scp have no problems whatever. My experience is that trying to treat S3 as a file system seemed like the very simplest and most fool-proof approach...but in practice, it often did not work robustly, and I spent more time restarting my S3 file system connection than it was worth. I have since switched to command-line tools like s3cmd which do not suffer the same drop-outs. YMMV.

1
  • 1
    You have misunderstood my requirements. I don't want to make S3 look like a file system. I want to make a file system look like S3
    – Stephen C
    Mar 24 '15 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.