4

I'm tired of long build times …

The #1 programmer excuse of legitimately slacking off: 'My code's compiling.'

I'm interested in a distributed build system, either at the lower-level of running the equivalent of make, and not knowing anything about anything other than dependency rules, or at the higher level of something like CMake or SCons. The system must (make an effort to) utilize all cores on a group of machines.

Example

Suppose I have the following sources, intermediary targets and final target:

and suppose building each of this takes a significant amount of time, and I have two machines.

The build system will have one machine build b0 and a second machine build b1, then one machine build b2 and another machine build b3 - and it will make sure only the relevant sources are copied/made available on relevant machines as soon as possible, not just after previous intermediaries are done. And it will determine these dependencies by looking at the source file and minimal additional information I provide - similarly to CMake+make.

Requirements

Must:

  • not be limited to a single programming language (e.g. not distcc or something living in a distributed JVM).
  • not require root access on any of the machines.
  • support on x86_64 machines
  • support (at least) modern distributions of Linux
  • have a command-line interface.
  • support on-demand builds.
  • be gratis.

Should:

  • be sort-of language-agnostic, in the sense that Make or CMake are (so, have mechanisms to handle different languages, recognize depndencies and so on without assuming targets are in any specific language or any language at all).

May or may not:

  • require partial or complete uniformity of the system configuration, software-wise.
  • integrate with version control
  • have a GUI
  • be web-accessible
Improve this question
13
  • 1
    Browser based interface, or command line? Continuous Intergration, or on demand? Integration with Version Control?
    – Mawg
    Commented Aug 24, 2016 at 8:04
  • @Mawg: Fair enough questions. I had sort of assumed people had some defaults in mind when I say "distributed equivalent of make/cmake" but I guess I can be more specific. See Edit.
    – einpoklum
    Commented Aug 24, 2016 at 8:36
  • 1
    @Mawg: Integration is great, I don't mind it being possible - but I'm being very modest here. Also, I've written I need it to support Linux.
    – einpoklum
    Commented Aug 24, 2016 at 11:38
  • 1
    @atp9: Does it do what I'm interested in?
    – einpoklum
    Commented Aug 24, 2016 at 18:15
  • 1
    @atp9: Please make an answer explaining how to make Jenkins use several machines to build something faster than it would have done on a single machine. Thanks!
    – Nicolas Raoul ♦
    Commented Aug 25, 2016 at 6:12

2 Answers 2

Reset to default
1

Sounds like you might actually need several tools for example:

Pick a command line tool to automate builds that get run when you type "./build.sh" (linux) or "build.cmd" (windows)

  1. ant - Not just for Java, but really a powerful ecosystem with many plugins, almost any build and deployment task that you need to automate can be automated with it. Doesn't provide a web UI, or a scheduling service so you use it inside something else.

  2. make or cmake - for systems level people with a lot of dependencies and at least some C or C++ - for old school folks with at least some C or C++.

  3. "RemObjects Train":https://github.com/remobjects/train - for people who want to build Linux, Mac and windows products and use javaScript to do it.

Now pick a web based continuous integration tool to schedule the build or run it automatically when someone checks in code:

A. jenkins - The big guy out there. Also there's hudson which is what Jenkins is a fork of, but I think hudson is basically dead. Free.

B. GitLab CI - A newcomer, but really handy if you want your automatic builds to occur whenever someone pushes to a git repo, and you want to have a choice of hosting the git server yourself or having it be on the internet, this is a great choice. And free.

Now create jenkins jobs for shared components. Now distribute those artifacts to the secondary stages. Find parallel build opportunities yourself and build your jenkins tasks so there is a "pipeline". Distributed builds take human ingenuity to design and do not self optimize and self-discover and self-distribute binary intermediates.

Again, what's wrong with your request in my view is that you need to think of the web thing as a "layer" of your build system, and your command line build tool as another layer.

Also don't limit yourself to only one build tool. Maybe you will need CMake so you can rebuild libffmpeg and you will need ant so you can rebuild some jars, and you will need python and bash scripts to do some stuff that seemed best to do that way.

There is no one build tool to rule them all just as in large systems there is seldom only one language, compiler or editor in use. Smaller, composable tools make better systems than "one tool to rule them all" tools.

Improve this answer
6
  • The solution you describe does not build something faster than it would have done on a single machine, right? If my project contains (among other things) two independent libraries, then a distributed build system would build each one on a separate machine. That's the point of the question.
    – Nicolas Raoul ♦
    Commented Aug 25, 2016 at 6:22
  • So, will Jenkins be able to have some of the intermediary make targets built on each machine, then bring them together on different machines in order to build additional intermediary targets? It doesn't sound like it from your description. Especially since you describe it as a "layer". I don't want another layer, I want to replace the existing layer with something that's distributed.
    – einpoklum
    Commented Aug 25, 2016 at 9:20
  • The problem of distributing the work is up to you. Sounds like you want magic. There is no such thing as an automatically-distributed multi-language distributed build system. You have to divide the work into pieces yourself, like every other build system designer always does. If you expect a magic build system to first co-ordinate who does what, and guarantee no overlap and guarantee it's faster than a single system build, then you're posing an optimization problem with multiple conflicting constraints which are essentially impossible to solve simultaneously. Several layers of NP-hard.
    – Warren P
    Commented Aug 25, 2016 at 11:04
  • What sane engineers do is engineer the shared library as a piece of a pipeline, and build stages of their work out as part of their build system design. They do this in Jenkins, and when a shared artifact completes, they build the secondary stages that depend on the library themselves. It is up to YOU as a build system designer to create and assign the tasks. Jenkins "slaves" do as you command, they do not self organize and auto assign tasks.
    – Warren P
    Commented Aug 25, 2016 at 11:09
  • Note that the way to speed up the build is to split it up into a number of tasks and use Jenkins "Slave" instances to do the actual building. Intermediate results can be passed between machines as needed. wiki.jenkins-ci.org/display/JENKINS/Distributed+builds is worth a read.
    – Steve Barnes
    Commented Dec 23, 2016 at 17:39
0

Not a perfect solution, but CruiseControl's Distributed Plugin allows a rudimentary form of build distribution:

[...] allows a master build machine to distribute build requests to other physical machines on which the builds are performed and to return the results to the master.

Your build script will look like this:

<distributed>
    <ant antscript="ant.bat"
        antworkingdir="C:/cruise-control-agent/checkout/BasicJavaProject" >
    </ant>
</distributed>

The Ant task will not be executed on the machine that runs CruiseControl ("master"), but instead will be executed by one of the available agent machines (separate machines running a specific agent software).

You can filter agents so that a particular tasks is only executed on a Linux 64bit agent, for instance.

Supported sub-build commands: ant, maven, maven2, nant, phing, rake, exec, pipedexec, xcode

Weak point: Dependencies are not resolved by the system. You have to find them by yourself and then write your build script accordingly.

Gratis, open source.

Improve this answer
2
  • "it won't allow me to distribute my CMake/make build": Your question does not ask for that either. Your question asks for a distributed build system, and does not require it to be compatible with your existing Cmake/make scripts.
    – Nicolas Raoul ♦
    Commented Aug 25, 2016 at 9:32
  • Let me rephrase. It seems like it won't allow me to distribute the building of intermediary targets in a C/C++ project to different machines, which would then be copied to different machines for building additional intermediary targets etc. But let me edit my question to clarify this.
    – einpoklum
    Commented Aug 25, 2016 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.