2

First post, hope it helps someone else storing large amounts of data online.

I use a cloud service, Bitcasa (NOT recommended) to serve my media library and for personal backup. They've recently radically changed their business model, and given a 3 week notice of a 12x price increase part way in to a year long subscription on beta customers like myself who've been helping as paying customers for several years while they iterate their product. All this to say I recommend staying away from them. I've had it, will never trust them after breach of contract, so I'm in a rush trying to download and verify over 7TB of data.

Having already downloaded 7TB from Bitcasa, it appears some is corrupted. It's not clear whether the original data was compromised in original transfer, or in their database servers, or errors occurred in download. Courts ordered an extension to Nov 20th for customers to DL data, however I don't have long before the data must be DLed before the Bitcasa service change. Most of the data is DLed, but I don't have confidence in the transfer.

In short, what's a good, fast, and reliable software/service to verify TBs of online data against my downloaded copy in a Mac Yosemite environment? My ISP has at best a 100/10 transfer rate. I'm not afraid of command line. File count is likely in the hundreds of thousands, mix of large and small, .txt to GB size media. Given time constraints, the verification should be multithreaded if possible. Any suggestions welcome!

Cheers!

Improve this question
6
  • Your question is off-topic here because it is not asking for software recommendations. I suggest you try it at SuperUser.com
    – user416
    Commented Nov 18, 2014 at 21:58
  • 1
    @JanDoggen I think it could be reworded to be more on topic. Something like "A software to verify a large amount of data with an online cloud service"
    – Tymric
    Commented Nov 18, 2014 at 22:27
  • 1
    I second Timmy. The re-phrased question is fine and should stay. // @thepen: that's one of the reasons I'd never trust any cloud service with (especially the sole) copy of my data: you never know about 1) policy changes, 2) the price changes, 3) the company closes (all data gone), 4) your data get corrupted, 5) your data get compromised. Sad experiences all. Good luck with your switch-over to a (hopefully more relyable) service!
    – Izzy
    Commented Nov 19, 2014 at 12:52
  • @Izzy Yup, lesson learned. Now all is local with some in the cloud. Bitcasafilelister-master is great for automagic upload to Gdrive. I used this to DL most data. Awesome.
    – thepen
    Commented Dec 7, 2014 at 2:21
  • You should take a look at git-annex (see answers here and here and here). I've not yet tried it, but sounds awesome: your local, open-source "dropbox" can connect with cloud services as well ;)
    – Izzy
    Commented Dec 7, 2014 at 10:45

1 Answer 1

Reset to default
1

If the online service offers or allows the computation on the servers a reasonable quality hashing function, (e.g.: MD5, SHA-1, etc), you could use a python script to both fetch the online value and compute the value in your download for each file and then either report the problem file or start a fresh download. Libraries to look at are URLLib2, hashlib and OS.walk.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.