Library for file management. Two backends: filesystem, s3

Question

I am dreaming of a Python library which abstracts the file handling of my application.

The application should run in two different configurations:

1. No storage server. All file operations get done on the local disk.
2. With storage server. All file operations should get done via s3.

I would like to do separation of concerns.

The application code should not care which configuration gets used. Choosing the right configuration (with or without storage server) gets done via configuration management.

I don't need all file operations which I can do via os and os.path. I just need all operations which can be done via s3.

Other required features:

• Open source: BSD or LGPL, not GPL
• Support for Linux. Other operating systems are not important in this context.

Distinction / out off scope

I don't want all file operations (like os.walk()). I just need the fundamental storage APIs of s3, but without a running storage server.

Answer 1

I suggest you give Siilo a try.

It is a storage abstraction layer for Python.

Siilo gives you a base API with operations like open and delete, that you can use without worrying about what is being used below.

It supports:

• Local filesystem
• Amazon S3
• Apache Libcloud
• CMIS on fmerges's branch

Minus points:

• It seems that very few operations are implemented. If I understand the code correctly, if you need write you will have to implement it yourself (in that case please send a pull request)
• No documentation nor code samples
• Community is small and very quiet

Plus points:

• Open source (MIT)
• Very lightweight

Answer 2

There is a Python library to mock out the boto (s3) library:

https://github.com/spulec/moto

Moto is a library that allows your python tests to easily mock out the boto library

It can run in standalone server mode, if you want to access it from a different programming language