2

First post, hope it helps someone else storing large amounts of data online.

I use a cloud service, Bitcasa (NOT recommended) to serve my media library and for personal backup. They've recently radically changed their business model, and given a 3 week notice of a 12x price increase part way in to a year long subscription on beta customers like myself who've been helping as paying customers for several years while they iterate their product. All this to say I recommend staying away from them. I've had it, will never trust them after breach of contract, so I'm in a rush trying to download and verify over 7TB of data.

Having already downloaded 7TB from Bitcasa, it appears some is corrupted. It's not clear whether the original data was compromised in original transfer, or in their database servers, or errors occurred in download. Courts ordered an extension to Nov 20th for customers to DL data, however I don't have long before the data must be DLed before the Bitcasa service change. Most of the data is DLed, but I don't have confidence in the transfer.

In short, what's a good, fast, and reliable software/service to verify TBs of online data against my downloaded copy in a Mac Yosemite environment? My ISP has at best a 100/10 transfer rate. I'm not afraid of command line. File count is likely in the hundreds of thousands, mix of large and small, .txt to GB size media. Given time constraints, the verification should be multithreaded if possible. Any suggestions welcome!

Cheers!

  • Your question is off-topic here because it is not asking for software recommendations. I suggest you try it at SuperUser.com – Jan Doggen Nov 18 '14 at 21:58
  • 1
    @JanDoggen I think it could be reworded to be more on topic. Something like "A software to verify a large amount of data with an online cloud service" – Tymric Nov 18 '14 at 22:27
  • 1
    I second Timmy. The re-phrased question is fine and should stay. // @thepen: that's one of the reasons I'd never trust any cloud service with (especially the sole) copy of my data: you never know about 1) policy changes, 2) the price changes, 3) the company closes (all data gone), 4) your data get corrupted, 5) your data get compromised. Sad experiences all. Good luck with your switch-over to a (hopefully more relyable) service! – Izzy Nov 19 '14 at 12:52
  • @Izzy Yup, lesson learned. Now all is local with some in the cloud. Bitcasafilelister-master is great for automagic upload to Gdrive. I used this to DL most data. Awesome. – thepen Dec 7 '14 at 2:21
  • You should take a look at git-annex (see answers here and here and here). I've not yet tried it, but sounds awesome: your local, open-source "dropbox" can connect with cloud services as well ;) – Izzy Dec 7 '14 at 10:45
1

If the online service offers or allows the computation on the servers a reasonable quality hashing function, (e.g.: MD5, SHA-1, etc), you could use a python script to both fetch the online value and compute the value in your download for each file and then either report the problem file or start a fresh download. Libraries to look at are URLLib2, hashlib and OS.walk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.