-1

My team is currently trying to get our compressed json data in an cloud storage, so that we can start pre-processing it using our google VM instance. We have the tar file uncompressed, but after it was uncompressed, each json file was found to be compressed through bzip2. We have a method to unzip iterativley again, but we need a place to store all this json data so that we can access it's contents to pre-process it. We're currently using Google Cloud Platform's Cloud storage to store the bzip2 files. Two sample file paths are as follows: 2012/07/00/00/01.bz2 , 2012/07/31/23/59.bz2. The Path file contains year/month/day/hour/minute, and there are multiple 00.bz2 files due to there being 31 days worth of data.

0

Python includes as standard two modules that you might be interested in: bz2 and tarfile both of which can help here.

If you insist on storing the files somewhere as .bz2 files then the bz2 library provides methods for accessing the contents either by decompressing the files to new files or by allowing incremental, standard file like, access to the decompressed contents.

You could, however, simply store the original tar format files and use the python tarfile module to open them, (specifying that they are .tar.bz2 files), and then you can open the json files within them without decompressing first.

Better yet since python also understands json via the json library you could probably perform your entire processing chain in python, (most Google VMs have or can have python installed). Note that Google has examples on how to access files on the Cloud Storage from Python.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.