Program that zips a group of files into sized chunks, each openable individually

Question

I am looking for a Windows program that can zip a large group of files up and save them to multiple zip files, each (say) no more than 100 MB.

It is critical however that any single zipfile can be opened on its own, that you don't need the other zipfiles to open a particular one.

This is to save me from manually trying to select 100MB worth of files (or a little bit less), zip em up, and repeat.

Again, the normal way that programs like WinZip handle this will NOT work for me, because I believe it creates a set like fileset.zip, fileset.z01, fileset.z02, etc and it is my understanding that you cannot open a file like fileset.z02 alone without having all the others in the set. Compression is NOT necessary, I am just trying to package them, not necessarily compress them. Compression would be great, but the true purpose is to collect the files in manageable chunks.

Do any zip utilities do what I need?

Answer 1

Interesting problem! I don't know any software to achieve what you want, but how about something like this?

#!/usr/bin/env python

import os, operator, sys
limit = int(sys.argv[1])
dirpath = os.path.abspath(sys.argv[2])
# make a generator for all file paths within dirpath
all_files = ( (os.path.join(basedir, filename) for basedir, dirs, files in os.walk(dirpath) for filename in files))
all_file_sizes = ((os.path.getsize(fname), fname) for fname in all_files)
sorted_files = sorted(all_file_sizes) 


def yield_groups(size_filenames, limit):
    cumulative_size = 0
    file_list = []
    for file_size, file_name in size_filenames:
        new_size = cumulative_size + file_size
        #print "%r" % new_size, "%r" %limit,  new_size > limit, file_name
        if new_size > limit:
            #print "bigger"
            yield file_list
            file_list = [file_name]
            cumulative_size = file_size
        else:
            file_list.append(file_name)
            cumulative_size += file_size

    yield file_list


for i, group in enumerate(yield_groups(sorted_files, limit)):
    print ("tar -cvf /tmp/group-%03u.tar %s" % (i, ' '.join(group)))

It seems to work reasonably well for me:

➜ chunks>du -hs *
1,0M    file1
1,0M    file2
1,0M    file3
1,0M    file4
1,0M    file5
2,0M    file6
2,0M    file7
3,0M    file8
3,0M    file9

Now requesting groups of 3MB:

➜ chunks>python /tmp/sizezip.py 3145728 /tmp//chunks/
tar -cvf /tmp/group-000.tar /tmp/chunks/file1 /tmp/chunks/file2 /tmp/chunks/file3
tar -cvf /tmp/group-001.tar /tmp/chunks/file4 /tmp/chunks/file5
tar -cvf /tmp/group-002.tar /tmp/chunks/file6
tar -cvf /tmp/group-003.tar /tmp/chunks/file7
tar -cvf /tmp/group-004.tar /tmp/chunks/file8
tar -cvf /tmp/group-005.tar /tmp/chunks/file9

Or rather 5MB:

➜ chunks>python /tmp/sizezip.py 5242880  /tmp//chunks/
tar -cvf /tmp/group-000.tar /tmp/chunks/file1 /tmp/chunks/file2 /tmp/chunks/file3 /tmp/chunks/file4 /tmp/chunks/file5
tar -cvf /tmp/group-001.tar /tmp/chunks/file6 /tmp/chunks/file7
tar -cvf /tmp/group-002.tar /tmp/chunks/file8
tar -cvf /tmp/group-003.tar /tmp/chunks/file9

Note that this is quick and dirty code. It does not even attempt to solve the Knapsack problem.

Answer 2

Another option, with or without compression, is the following:

• Copy the entire tree you want to package

• Implement a loop going through the entire copy tree doing:

  • If current file to be added is above file size limit, do something... Flag it, terminate, throw exception, ...
  • Add file (with or without compressing) to a temporary archive
  • If temporary archive is not above file size limit, remove the currently added file
  • If temporary archive file size is above limit, move it out of the way, and start a new temporary archive (thusly not deleting the lastly added file)

• Do whatever you want with your newly created chunk archives

• (optional) Build a list based on each chunk archive to keep a list of which files went into which archive

There are some advantages to this procedure:

1. Since you're working on a copy of the original tree, you can delete files already archived/handled
2. It can be implemented with rather simple tools (i.e. windows command/batch files) without having access to dictionaries (or similar tools to keep track of which files are still to be handled)
3. If needed/wanted, you can run extra tools to remove unneccessary files from the copy tree before the archival process, i.e. remove too large files, remove backup versions, remove other uninteresting files, ...

The procedure (or any tool you choose) might however fail if you have single files which is above your file size limit. But if they are moved out of the way, you could handle those afterwards in a manual process.

PS! This is similar to the python approach, but you don't need python, and it can be used either automated or actually also manually without too much hassle. I've done a similar run with a few thousand files, where we removed some intermediate files and stuff, and working with a copy of the original data made everything a lot easier
PPS! A copy of the data, does however require that you have room to duplicate your data temporarily

Answer 3

I have the same requirement and did research and found spinzip.exe. However the issue it has is that it keeps the folder structure in the zip file which is not good for my requirements. The author said they don't know how to fix it. http://www.donationcoder.com/forum/index.php?board=298.0

Your question is over a year old. So if you have found a solution, please post it as a comment here so I can get an email notification. Thanks