Let's say I have a build process where one of the steps is applying a transformation to a bunch of files, such that one source file equals one target file. This could be something like compression, encryption, image scaling, etc.

Because doing it every time for all source files takes a long time, I'd like to do it incrementally, so a file is only processed if it's newer than the corresponding target file.

It seems to me that this is a very generic process and condition, so I'm wondering, is there an existing tool that can take care of this part of the process? So that all I need to implement myself is a command that takes an input and output file as parameters (or uses stdin/out) - if it doesn't exist yet.

Linux and/or Windows solutions welcome. Optional bonus features:

  • A single source file results in multiple target files (e.g. scaling an image to multiple sizes)
  • Multiple source files result in less target files (e.g. archiving bunches of files together)
  • Basing the "convert or not"-decision on a hash instead of modification date (more reliable). This would require the transformer to support storing the source hash in the target file, and reading it.

Is this perhaps something that make can do?


You can use make, scons, ant, etc. all of which take basically a rule of the form:

target_file depends on input(s)

and an operation:

create target by performing step(s)

The build tool will look at all of the relationships and perform the build, including where the output file of one process is one of the dependencies of one or more of the others and will correctly just rebuild those targets whos ancestors have changed.

make works purely on timestamps. scons uses a combination of timestamps, sizes & md5s to check what is changed.

N.B. Software developers have been using these tools for years but there is no reason that they cannot be applied to any processing providing it can be done from the command line or scripted. Several tools will even try to parallel process so if a -> A and b -> B but A+B -> C then the transformation of the first two can be done in parallel but C cannot be created until both have finished.


Use python. I don't understand your dilemma though : you check your files time stamp[1], and folder contents, you check if a _001.zip file exists[2] / or high_res.jpg exists and if not, create it[3][4]. you apply different scaling parameters to ffmpeg for example and 7zip[5][6].

[1] https://stackoverflow.com/a/237084/1889814
[2] https://stackoverflow.com/a/82852/1889814
[3] https://stackoverflow.com/a/24745969/1889814
[4] https://stackoverflow.com/a/14569017/1889814
[5] https://trac.ffmpeg.org/wiki/Scaling%20(resizing)%20with%20ffmpeg
[6] https://sevenzip.osdn.jp/chm/cmdline/syntax.htm

  • 1
    Time stamps are not really reliable. Use a hash instead. May 9 '16 at 11:25
  • 1
    My "dilemma" is not that this is particularly hard, I could probably implement it rather quickly. But I don't wish to keep reinventing this wheel. May 9 '16 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.