Parallel Cons

Parallel Cons

This page is a central location for information about past (and future) attempts at creating a version of the Cons software construction utility, that can build multiple targets in parallel (the sort of functionality supported by the make -j option). This includes copies of the relevant code, email exchanges, and editorial comments (see the bottom of the page). I hope that sharing this information will provide the right insight or inspiration for someone to complete the job.

There have been (to my knowledge) four significant attempts at creating a parallel version of Cons. None of these versions has yet provided a complete solution. So that we have some common terminology, I'm calling them, in order, multi, cons-p, pcons, and pcons-2.3.0:

multi	This is Bob Sidebotham's original attempt at parallelizing Cons. Bob did most of this work in 1996, and then set it aside. Bob created `job::serial` and `job::parallel` classes to manage job creation, and tries to enumerate all the dependencies of a target before starting any builds by doing a separate recursive descent. This is a nice idea, but can't accomodate dynamic dependencies (that is, it loses if you have to generate a `.c` file before you can scan it for `#include` lines).	multi.tar.gz	Bob's original tar file he sent me (and another person on the cons-discuss mailing list) containing all his work on parallelizing Cons.
		multi.email	Bob's original email explaining what's in the multi.tar.gz file.
		cons.multi	Bob's parallel version of Cons, already extracted from the tar file for easy viewing. (This is slightly different from the version in multi.tar.gz; I pasted back in an enumerate subroutine that was left out.)
		multi.diff	My `diff -c` of cons.multi and the original version of Cons on which it was based, so you can see the changes Bob made to implement his parallelism.
cons-p	This is my attempt at picking up and finishing Bob's work on multi. This took place in early 1999 (February through April). I took a lot of wrong turns, among them: merging Bob's two job classes into one (I thought it would be clever to handle serial jobs as an N=1 case of parallelism, but it's much more efficient to keep them separate); breaking signature handling; spending too much energy on a complicated scheme to enumerate dependencies up-front and still handle dynamic dependencies... I added some documentation, though, and extended Bob's job classes to handle multi-line commands.	cons-p.tar.gz	An archive of the last three versions of cons-p, plus the test cases (shell scripts in the manner of versions 2.X of the cons-test suite) for which I was trying to get cons-p to work.
		cons-p.email	Some fairly lengthy email exchanges among Bob, Rajesh and myself as I struggled to get cons-p working.
		cons-p.diff	`diff -c` output of the baseline version of Cons I used (1.6b1) and the last version of cons-p I was working on before setting it aside.
pcons	This is John Erickson's parallel Cons from August 1999. John took a different approach from Bob's, spawning parallel jobs within the recursive dependency descent for a target. This leads to an elegantly concise set of changes that don't give ideal parallelism (you can't keep N compiles running simultaneously if they're spread across multiple targets), but do an excellent job of keeping each target as parallel as possible. The only other drawback is that John didn't keep multi-line commands working; you had to separate commands with `&&` instead of `\n`.	pcons.announce	John's original announcement to the Cons mailing list about pcons, with his attached patch.
		pcons.diff	John's original patch, as attached to his announcement mail.
		pcons	John's patch applied to Cons 1.6b1 (the current version at the time).
		pcons.email	An email exchange between John and myself about pcons, differences between his and Bob's approaches, and how we might take it to the next step. We also discussed the idea of using threads.
pcons-2.3.0	This is Thomas Gleerup's merging and updating of John Erickson's pcons change with Cons version 2.3.0. This has the same minor drawback as Erickson's (very useful but slightly imperfect parallelism), and half-supports multi-line commands by automatically converting newline separators into `&&`. Note that the Cons 2.3.0 version on which this is based has known problems with multi-target dependencies, and the `Install`, `InstallAs`, and `Install-Local` methods.	pcons-2.3.0.announce	Thomas's original announcement to the Cons mailing list about his work.
		pcons-2.3.0	Thomas' version of pcons, based on Cons 2.3.0. Note that this is actually slightly updated from the version in his announcement, adding the automatic conversion of `\n` command separators to `&&` in multi-line commands.
		pcons-2.3.0.diff	`diff -c` output of pcons-2.3.0 against the 2.3.0 version of Cons that on which it was based.

EDITORIAL COMMENTS:

After re-familiarizing myself with these attempts, I think each has some good ideas that could be pulled together into a good implementation:

First and most importantly, I now think the pcons approach of parallelizing individual targets, instead of parallelizing across targets, is probably the most fruitful approach for Cons' current structure. So what if it's not ideal? It's practical, it works, it will help people.
multi's job classes are more flexible than the way pcons did it, especially as we Cons gets ported to more operating systems and we have other process models to support. Add the pieces from cons-p that handle multi-line commands.
cons-p can provide some template documentation, although its text that crows about the ideal world of parallelize across targets would have to be rewritten.

Feel free to send me something at [email protected].

Last modified: 8 November 2001