3

I have a input file(s) which can have size up to 25 GB. The file type may be a image, video, text, binary, etc. I want to know if I there's a cross-platform library that provides a way to split/join files.

Or if there's a class/function in C++ that provides me this kind of utility.

  • 1
    Why not use any of C++'s usual I/O systems? Read in the file and write it out in different files to split; read back the output files and concatenate to join. Or call the GNU coreutils split and cat. – Kodiologist Mar 11 '18 at 21:07
  • 1
    Look at the source for split and join in Debian's coreutils package – ivanivan Mar 12 '18 at 16:03
1

As @Kodiologist says this isn't too hard to do from first principles:

Edit: Simplify code. Unfortunately the added complexity from being able to support any stream type is not helpful in this context.

#include <fstream>
#include <memory>
#include <sstream>
#include <vector>

const int size1MB = 1024 * 1024;

std::unique_ptr<std::ofstream> createChunkFile(std::vector<std::string>& vecFilenames) {
    std::stringstream filename;
    filename << "chunk" << (vecFilenames.size() + 1) << ".txt";
    vecFilenames.push_back(filename.str());
    return std::make_unique<std::ofstream>(filename.str(), std::ios::trunc);
}

void split(std::istream& inStream, int nMegaBytesPerChunk, std::vector<std::string>& vecFilenames) {

    std::unique_ptr<char[]> buffer(new char[size1MB]);
    int nCurrentMegaBytes = 0;

    std::unique_ptr<std::ostream> pOutStream = createChunkFile(vecFilenames);

    while (!inStream.eof()) {
        inStream.read(buffer.get(), size1MB);
        pOutStream->write(buffer.get(), inStream.gcount());
        ++nCurrentMegaBytes;
        if (nCurrentMegaBytes >= nMegaBytesPerChunk) {
            pOutStream = createChunkFile(vecFilenames);
            nCurrentMegaBytes = 0;
        }
    }

}

void join(std::vector<std::string>& vecFilenames, std::ostream& outStream) {
    for (int n = 0; n < vecFilenames.size(); ++n) {
        std::ifstream ifs(vecFilenames[n]);
        outStream << ifs.rdbuf();
    }
}

void createTestFile(const std::string& filename) {
    std::ofstream ofs(filename, std::ios::trunc);

    std::unique_ptr<char[]> buffer(new char[size1MB]);

    int i = 0;

    for (int n = 0; n < 1024; ++n) {
        for (int m = 0; m < size1MB; ++m) {
            buffer[m] = 'a' + (i++ % 26);
        }

        ofs.write(buffer.get(), size1MB);
    }
}

int main()
{

    // Create test file

    std::string filenameBefore = "before-big.txt";
    createTestFile(filenameBefore);

    // Split

    std::ifstream ifs(filenameBefore);
    std::vector<std::string> vecFilenames;

    split(ifs, 100, vecFilenames);

    // Join

    std::string filenameAfter = "after-big.txt";

    std::ofstream ofs(filenameAfter, std::ios::trunc);
    join(vecFilenames, ofs);

    return 0;
}

This builds for me in Visual Studio 2015. No reason why it shouldn't in any C++11 compiler (but I can't promise you won't have to make minor adjustments).

Here's a quick sanity check that the split and joined file is the same as the original

compare md5 files

The kind of error I'm checking for here is exactly the reason I'd prefer to rely on tried and tested code, but I'm not convinced coreutils is convenient to grab and use as a library. To be honest, I'd probably end up just executing the split and join binaries as child processes from within my main program.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.