0

I want to implement a client+server which allows uploading of big files over very slow and faulty networks.

This means the upload needs to be interruptible.

Example: If 80% of the upload data was already transferred, then (after the tcp connection was lost and created again) the second request should only transfer the missing 20%.

In my case the client-server communication needs to use https.

An upload can last up 12 hours.

Client and server will be implemented with Python.

Of course I could invent my own protocol on top of http. I guess this would be simple.

But, I would like to implement a standard/spec (if there is any).

Are there open source tools which already implement this?

0

I did this before using BTsync which was free when it was in beta but became resilio and is now commercial software. Basically it's an encrypted private torrent engine. I was able to move hundreds of gigabytes of virtual machines from a data center that was going out of business and had very intermittent internet connectivity across the country without any issues. It checked MD5 of all files and ran like rsync over BitTorrent. There's a lot of projects on GitHub if you are going to code your own you could build off of for python. Please release on GitHub and post here if you pull it off, could come in handy for many people.

| improve this answer | |
0

There is https://tus.io/

Quoting:

People are sharing more and more photos and videos every day. Mobile networks remain fragile however. Platform APIs are also often a mess and every project builds its own file uploader. There are a thousand one-week projects that barely work, when all we need is one real project. One project done right.

We are the ones who are going to do this right. Our aim is to solve the problem of unreliable file uploads once and for all. tus is a new open protocol for resumable uploads built on HTTP. It offers simple, cheap and reusable stacks for clients and servers. It supports any language, any platform and any network.

It may seem to be an impossible dream. Perhaps that is because no-one has managed to solve it yet. Still, we are confident and we are going to give it our best shot. Join us on GitHub and help us make the world a better place. Say "No!" to lost cat videos! Say "Yes!" to tus!

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.