a generation lost in the bazaar - acm queue

8/11/2019 A Generation Lost in the Bazaar - ACM Queue

1/24

RELATED CONTENT

BARBARIANS AT THE GATEWAYS

High-frequency Trading and ExchangeTechnologyJacob Loveless

ONLINE ALGORITHMS IN HIGH-FREQUENCY

TRADING

The challenges faced by competing HFTalgorithmsJacob Loveless, Sasha Stoikov, RolfWaeber

THE ESSENCE OF SOFTWARE ENGINEERING:

THE SEMAT KERNEL

A thinking framework in the form of anactionable kernelIvar Jacobson, Pan-Wei Ng, Paul McMahon,Ian Spence, Svante Lidman

MANAGING TECHNICAL DEBT

Shortcuts that save money and timetoday can cost you down the road.Eric Allman

BROWSE THIS TOPIC:

DEVELOPMENT

QUEUE ON SLASHDOT- More Encryption Is Not the Solution- The Antifragile Organization- Realtime GPU Audio

QUEUE ON REDDIT

- Nonblocking Algorithms and ScalableMulticore Programming- FPGA Programming for the Masses - There's just no getting a round it: Youare Building a Distributed System

A Generation Lost in the Bazaar

Quality happens only when someone is responsible for it.

POUL-HENNING KAMP

Thirteen yea rs ago , Eric Raymond's book The Cathedral and the Bazaar(O'Reilly Media, 2001) redefined ourvocabulary and all but promised an end to the wa terfall model and big software companies, thanks to thenew gras s-roots open source softwa re development movement. I found the book thought provoking, butit did not convince me. On the o ther hand, being deeply involved in open source, I couldn't help but thinkthat it would be nice if he was right.

The book I brought to the beach house this summer is also thought provoking, much more so thanRaymond's (which it even mentions rather positively): Frederick P. Brooks's The Design of Design(Addison-Wes ley Professional, 2010). As much as I find myself nodding in agreement and as much as I enjoyBrooks's command of language and subject matter, the book also makes me sad and disappointed.

Thirteen years ago also marks the apogee of the dot-com euphoria, where every teenager was a Webprogrammer and every college dropout had a Web startup. I had genuine fun trying to teach some of

those greenhorns about the good old-fashioned tricksof the tradetest-restoring backups, scriptingoperating-system installs, version control, etc. Hindsight, of course, is 20/20 (i.e., events may have be enless fun than you remember), and there is no escaping that the entire dot-com era was a disas ter forIT/CS in general and for so ftware qua lity and Unix in particular.

I have not seen any competent analysis of how much bigger the IT industry became during the dot-comyears. My own estimate is thatcounted in the kinds of jobs tha t would untilthen have been behind thelocked steel doors of the IT departmentour trade grew by twoordersof magnitude, or if you prefer, bymore than 10,000 percent.

Getting hooked on computers is easyalmost anybody can make a program work, just as almost anybodycan nail two pieces of wood togethe r in a few tries. The trouble is that the market for two pieces of woodnailed togethe rinexpertlyis fairly small outside o f the "proud grandfather" segment, and getting fromthere to a decent set of chairs or fitted cupboards takes talent, practice, and education. The extra 9,900percent had neither practice nor educationwhen they arrived in our trade, and before they ever had thechance to acquire it, the party was over and most of them were out of a job. I w ill charitably assume thatthose who managed to hang on were the most talented and most skilled, but even then there is noescaping that a s IT professionals they mostly sucked because of their lack of ballast.

The bazaa r meme advocated by Raymond, "Just hack it," as opposed to the carefully designed cathedralsof the pre-dot-com years, unfortunately did, not die with the dot-com madness, and today Unix is rapidly

sinking under its weight.

I updated my laptop. I have been running the development version of FreeBSD for 18 years straight now ,and compiling even my Spartan w ork environment from source code takes a full day, because it involvestrying to make sense and a rchitecture out of Raymond's anarchistic software bazaar.

At the top level, the FreeBSD ports collection is an attempt to create a map of the bazaa r that makes iteasy for FreeBSD users to find what they need. In practice this map consists , right now, of 22,198 filesthat give a summary description of each sta ll in the bazaara couple o f lines telling you roughly wha t thatstall offers and where you can read more about it. Also included are 23,214 Makefiles that tell you what todo w ith the softwa re you find in each stall. These Makefiles also try to inform you of the choices you shouldconsider, which options to choose, and wha t would be sensible defaults for them. The map alsoconveniently comes w ith 24,400 patch files to smooth over the lack of craftsmanship of many of the w aresoffered, but, generally, it is lack of portability that creates a need for the se patch files.

Finally, the map helpfully tells you that if you want to have ww w/firefox, you will first need to get devel/nspr, security/nss, databa ses/sqlite3, andso on. Once you look up those in the map and find their dependencies, and recursively look up their dependencies, you will have a shopp ing listof the 122 packages you will need before you can get to w ww/firefox.

Modularity and code reuse are, o f course, A Good Thing. Even in the most trivially simple case , however, the CS/IT dogma of code reuse is totallyforeign in the bazaar: the software in the FreeBSD ports collection contains at leas t 1,342 copied and pasted cryptographic algorithms.

If that resistance/ignorance of code reuse had resulted in self-contained and independent packages of softwa re, the price of the code duplicationmight actually have been a good trade off for ease of package management. But that was not the case: the packages form a tangled web ofhaphazard dependencies that results in much code duplication and waste.

Here is one example of an ironic piece of waste: Sam Leffler's graphics/libtiff is one of the 122 packages on the road to www /firefox, yet theresulting Firefox browser does not rende r TIFF images . For reasons I have not tried to uncover, 10 of the 122 packages ne ed Perl and se venneed Python; one of them, devel/glib20, needs both langua ges for reasons I cannot even imagine.

Further down the shopping list are repeated a pplications of the Peter Principle, the idea that in an organiza tion where promotion is based onachievement, success, and merit, that organ ization's members will eventually be promoted beyond their level of ability. The p rinciple is commonlyphrased, "Employees tend to rise to the ir level of incompetence." Applying the principle to software, you will find that you need three differentversions of the make program, a macroprocessor, an assembler, and many other interesting packages . At the bottom of the food chain, so tospeak, is libtool, which tries to hide the fact that there is no s tandardized way to build a shared library in Unix. Instead of standardizing how todo that across all Unixensomething that would take just a s ingle flag to the ld(1)commandthe Pe ter Principle wa s applied and made itlibtool's job instead. The Pete r Principle is indeed s trong in this casethe s ource code for devel/libtool weighs in at 414,740 lines. Half that linecount is test cases, wh ich in principle is commendable, but in practice it is just the P eter Principle at work: the tests e laborately explore thefunctionality of the complex solution for a p roblem that should no t exist in the first place. Even more maddening is that 31,085 of those lines arein a s ingle unreadably ugly she ll script called configure. The idea is tha t the configure script performs approximately 200 automated tes ts, so thatthe user is not burdened with configuring libtool manually. This is a horribly bad idea, already much criticized ba ck in the 1980s when it appeared,as it allows source code to pretend to be portable be hind the veneer o f the configure script, rather than actually having the quality of portability

ACM QueueArchitecting Tomorrow's Computing

A Generation Lost in the Bazaar

Tweet 431

COLUMNS > THE BIKESHED

view issue

by Poul-Henning Kamp | August 15, 2012

Topic: Development

Like 113 people like this. Sign Upt

A Special Offer to Join ACM Why Join ACM?
http://queue.acm.org/listing.cfm?item_topic=Development&qc_type=theme_list&filter=Development&page_title=Development&order=deschttp://queue.acm.org/detail.cfm?id=2389616http://queue.acm.org/detail.cfm?id=2534976http://queue.acm.org/detail.cfm?id=2536492http://www.facebook.com/campaign/landing.php?campaign_id=137675572948107&partner_id=queue.acm.org&placement=like_plugin&extra_1=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&extra_2=UShttps://twitter.com/intent/tweet?original_referer=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&text=A%20Generation%20Lost%20in%20the%20Bazaar%20-%20ACM%20Queue&tw_p=tweetbutton&url=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&via=ACMQueuehttp://twitter.com/search?q=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrsshttp://queue.acm.org/index.cfmhttp://queue.acm.org/whyjoinacm.cfmhttp://www.acm.org/joinacm2?ref=queuehttp://queue.acm.org/whyjoinacm.cfmhttp://queue.acm.org/whyjoinacm.cfmhttp://www.acm.org/joinacm2?ref=queuehttp://www.facebook.com/campaign/landing.php?campaign_id=137675572948107&partner_id=queue.acm.org&placement=like_plugin&extra_1=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&extra_2=UShttp://queue.acm.org/listing.cfm?item_topic=Development&qc_type=theme_list&filter=Development&page_title=Development&order=deschttp://queue.acm.org/issuedetail.cfm?issue=2346916http://twitter.com/search?q=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrsshttps://twitter.com/intent/tweet?original_referer=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&text=A%20Generation%20Lost%20in%20the%20Bazaar%20-%20ACM%20Queue&tw_p=tweetbutton&url=http%3A%2F%2Fqueue.acm.org%2Fdetail.cfm%3Fid%3D2349257%26ref%3Dfullrss&via=ACMQueuehttp://queue.acm.org/whyjoinacm.cfmhttp://queue.acm.org/index.cfmhttp://www.reddit.com/r/compsci/comments/1e05y4/theres_just_no_getting_around_it_you_are_building/http://www.reddit.com/r/programming/comments/1960w7/fpga_programming_for_the_masses_the/http://www.reddit.com/r/programming/comments/1g9zsa/nonblocking_algorithms_and_scalable_multicore/http://hardware.slashdot.org/story/13/05/10/1819224/realtime-gpu-audiohttp://developers.slashdot.org/story/13/07/02/2231225/the-simian-army-and-the-antifragile-organizationhttp://it.slashdot.org/story/13/07/31/2133216/more-encryption-is-not-the-solutionhttp://queue.acm.org/listing.cfm?item_topic=Development&qc_type=theme_list&filter=Development&page_title=Development&order=deschttp://queue.acm.org/detail.cfm?id=2168798http://queue.acm.org/detail.cfm?id=2389616http://queue.acm.org/detail.cfm?id=2534976http://queue.acm.org/detail.cfm?id=2536492


2/24

to begin w ith. It is a travesty that the configure idea survived.

The 1980s saw very different Unix implementations: Cray-1s with their 24-bit pointers, Amdahl UTS mainframe Unix, a multitude of more or lesscompetently executed SysV+BSD mashups from the minicomputer makers, the a lmostbut not quiteUnix shims from vendors such as DataGeneral, and even the genuine Unix clone Cohe rent from the paint company Mark Williams.

The configure scripts back then were w ritten by hand and did things like figure out if this was most like a BSD- or a SysV-style Unix, and thencopied one or the othe r Makefile and maybe also a .h file into place. Later the configure scripts became more ambitious, and as an almostpredictable application of the Peter Principle, rather than standa rdize Unix to eliminate the need for them, somebody wrote a p rogram, autoconf,to write the configure scripts.

Today's Unix/Posix-like operating systems, even including IBM's z/OS mainframe version, as seen with 1980 eyes a re identical; yet the 31,085lines of configure for libtool still check if and exist, even though the Unixen, wh ich lacked them, had ne ither sufficientmemory to execute libtool no r disks big enough for its 16-MB source code.

How did that happen?

Well, autoconf, for reasons that have ne ver made sense, was written in the obscure M4 macro language , which means tha t the actual tests looklike this:

## Whether `make' supports order-only prerequisites.

AC_CACHE_CHECK([whether ${MAKE-make} supports order-only prerequisites],

[lt_cv_make_order_only],

[mkdir conftest.dir

cd conftest.dir

touch b

touch a

cat >confmk /dev/null 2>&1; then

lt_cv_make_order_only=yes

else

lt_cv_make_order_only=no

fi cd ..

rm -rf conftest.dir

])

if test $lt_cv_make_order_only = yes; then

ORDER='|'

else

ORDER=''

fi

AC_SUBST([ORDER])

Needless to say, this is more than most programmers would ever want to put up with, even if they had the skill, so the input files for autoconfhappen by copy and paste, often hiding behind increasingly bloated s tandard macros covering "standard tests" such as those mentioned earlier,which look for compatibility problems not seen in the pas t 20 years.

This is probably also why libtool's configure probes no fewe r than 26 different names for the Fortran compiler my system does no t have, and thenspends another 26 tests to find out if each of these nonexistent Fortran compilers supports the -g option.

That is the so rry reality of the bazaar Raymond praised in his book: a pile of old festering hacks, endlessly copied and pasted by a cluelessgeneration o f IT "professiona ls" who wouldn't recognize sound IT architecture if you hit them over the head w ith it. It is hard to be lieve today,but unde r this embarrassing mess lies the ruins of the be autiful cathedral of Unix, deservedly famous for its s implicity of design, its economy of

features, and its elegance of execution. (Sic transit gloria mundi, etc.)

One o f Brooks's many excellent points is that quality happens only if somebody has the responsibility for it, and that "s omebody" can be no morethan one single personwith an exception for a dynamic duo. I am surprised tha t Brooks does not cite Unix as an example of this claim, since wecan pinpoint w ith almost surgical precision the moment that Unix started to fragment: in the early 1990s when AT&T spun off Unix tocommercialize it, thereby robbing it of its architects.

More than once in recent years, others have reached the s ame conclusion as Brooks. Some have tried to impose a kind of sanity, or even to laydown the law formally in the form of technical standa rds, hoping to bring order and structure to the ba zaar. So far they have all failedspectacularly, because the gene ration of lost dot-com wunderkinderin the bazaar has never seen a cathedral and therefore cannot even imaginewhy you would want one in the first place, much less wha t it should look like. It is a sad irony, indeed, that those w ho most need to read it mayfind The Design of Designentirely incomprehensible. But to anyone who has ever wonde red whethe r using m4 macros to configure autoconf towrite a shell script to look for 26 Fortran compilers in order to build a Web browser was a bit of a detour, Brooks offers well-reasone d hope thatthere can be a better way.

LOVE IT, HATE IT? LET US KNOW

[email protected]

Poul-Henning Kamp([email protected]) has programmed computers for 26 years and is the inspiration behind bikeshed.org. His softwa re hasbeen w idely adopted as under-the-hood building blocks in both open source and commercial products. His most recent project is the Varnish HTTPaccelerator, which is used to speed up large Web sites such as Facebook.

2012 ACM 1542-7730/12/0800 $10.00

Originally published in Queue vol. 10, no. 8see this item in the ACM Digital Library

POUL-HENNING KAMP ([email protected]) is one of the primary developers of the FreeBSD operating system, which he has worked on from thevery beginning. He is widely unknown for his MD5-based pas sword scrambler, which protects the passwords on C isco routers, Juniper routers,and Linux and BSD systems. Some people have no ticed that he wrote a memory allocator, a device file system, and a disk encryption method thatis actually usable. Kamp lives in Denmark with his wife, his son, his daughter, about a dozen FreeBSD computers, and one of the world's mostprecise NTP (Network Time Protocol) clocks. He makes a living as an independent contractor do ing all sorts o f stuff with computers and networks.

For additional information see the ACM Digital Library Author Pa ge for: Poul-Henning Kamp

COMMENTS
http://dl.acm.org/author_page.cfm?id=81100230469mailto:[email protected]://portal.acm.org/citation.cfm?id=2349257http://bikeshed.org/mailto:[email protected]


3/24

William Payne | Mon, 20 Aug 2012 14:31:12 UTC

So should we then flee to the relative oasis of sanity that is Plan 9?

phd_student_doom | Mon, 20 Aug 2012 15:10:25 UTC

Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the 'chaos'

is what makes programming fun.

Matt Welsh | Mon, 20 Aug 2012 15:13:22 UTC

Nice article. I think we all predicted this would happen when the open source "movement" started to gain momentum a decade ago.

To be fair, a lot of good engineering goes into open source projects, although it's often hard to see because of the smoke

screen of amateur contributions and longwinded threads on the email lists. It's not all hackers working in their spare time:Google, for example, contributes a lot of code back to open source projects (e.g., Guido van Rossum works at Google). There

also seems to be a great deal of entropy at work: Witness all of the failed efforts to "clean up" BSD and Linux that never went

anywhere. Chaos is king.

Poul-Henning Kamp | Mon, 20 Aug 2012 15:25:27 UTC

Matt, My point is not that there are no cathedrals today, but that people don't recognize them as such or see any value in

them.

Ken Stox | Mon, 20 Aug 2012 15:48:13 UTC

Like many other "good" ideas, the Bazaar has been over applied, and used in situations where it is a bad solution. Give a man a

hammer... That being said, there are situations where the Bazaar is a great path to take.

I would argue that UNIX began to fragment in the late 1980's when AT&T's licensing terms became more onerous than the customers

were willing to accept, leading to the creation of OSF. AIX and HP/UX forked off the System V.2 codebase. The only reason SUN

veered from BSD to System V.4 was because AT&T owned a substantial portion of them at the time.

I suspect that we will oscillate around the strange attractor that is the Cathedral and the other that is the Bazaar. After

all, isn't that what Chaos is all about?

iain | Mon, 20 Aug 2012 15:48:55 UTC

Contained within this rant are some good points:

Code reuse is good

Removing unneeded dependancies is good

Fixing up libtool to not check for fortran would be brilliant

However, surrounding these good points are a lot of non-sequiturs and nostalgic romanticised thinkings harking back to the good

ol' days when all you needed was a terminal and a |. I'm sure it was lovely back then, but these days some of us like being

able to plug in our USB devices and have them automatically detected, being able to just click on the wifi network we want to

connect to.

So libtiff is an implicit dependency for Firefox even though Firefox cannot display tiff images, and this is an example of how

badly designed the whole system is. Is it not more likely that someone somewhere (possibly the author, in his haste to show us

how hardcore he was because he compiles everything from source) had forgotten to pass a --disable-tiff to one of the other 121

packages?

Surely even the complaint that Firefox uses 122 other packages shows that code reuse, rather than being "totally foreign in the

bazaar" is actually quite a common occurance?

The bazaar (I'm assuming this is to mean the free software world) is anarchistic? From all the software projects I've worked on

over the past 15 years in Free Software, the one thing they have in common is a very strong sense of hierarchy so that clearly

can't be an anarchy, can they? Large distributions have smart people who are thinking about the best way to put them together,

people can't just fling their code into them, so these aren't anarchies either.

There is also a fair greater sense of reverence to Eric Raymond and his ramblings (the man has written articles about how he is

the embodiment of the god Pan, for goodness sake) in this article than I have seen in the Free Software world. The man is a

hack, anyone who has ever had to look at his code would be able to tell you he isn't very good at it. He stopped being relevant

to the Free Software community circa 1999.

This rant seems to be the ranting of an out of touch old hacker, who's a bit annoyed technology has got too complicated for him

to understand, but rather than seeing that as the inevitable progress of technology (a modern car is considerably more complex

than Henry Ford's, the modern aeroplane is more complicated than the one the Wright Bros used, and the modern computer OS is

more complex than the operating systems of the early 80s), he concludes that it was better in the old days, and everyone today

is a moron.

Kevin Vilbig | Mon, 20 Aug 2012 15:54:38 UTC

What are the cathedrals?

philip andrew | Mon, 20 Aug 2012 15:54:51 UTC

There are problems when building a Cathedral, a small problem in one part of the code has a tendency to be difficult to

discover and fix when the system gets very large. Well, from my experience on a large C++ project, 400,000 lines of code, we

found a few problems.

One, its very hard to debug some problems even with the core dump from the customer with 100 threads dumped, looking through

the threads call stacks.

Two, its very hard to understand the whole thing.

Three, the dependancies all had to be compatible with each other.

Four, when something broke, the whole system crashed as it was one single process.

We ended up breaking the large program up into many small programs which communicated via Corba, you could insert JNI here or

any other communications method. The small programs were then understandable, maintainable, testable. What they depended upon


4/24

was relative small, one program did Authentication, one did Authorization, one did Document Storage.

Basically the architecture changed from a single large C++ program to a broken up system.

A Cathedral could work, communism could work if everyone followed the rules, if everything worked well and worked according to

good rules. However, in our real world, nothing works the way we expected. Software has bugs, bugs you have never even through

of. Things fail. People fail.

At least in the Bazzar model I can understand the many small programs. In the Cathedral model, I cannot, at least with C++, it

reaches a complexity limit for the human brain, thats also why we don't program in data-flow. With some other language and

tools and rules perhaps it can be done. Perhaps ADA, that was the idea. (Well I prefer Haskell for the perfect world).

Mark dugger | Mon, 20 Aug 2012 15:57:16 UTC

I respectfully disagree. The Cathedral model only works if one is capable of defining the scope of the problem completely from

the outset of the program. In systems programming, hardware doesn't change nearly as much as an organic customer. Therefore

development processes need to change given what the code is interfacing.

Poul is an excellent developer and I've enjoyed his contributions to FreeBSD and varnish, but his criticisms of the bazaar

model do not hold water. Poul argues that code quality can only be assured if quality can be accounted for easily, I would

argue that code quality can only be assured if the users of the code had leverage. In free software, the currency that the

users yield is popularity and reputation, which developers can flip into fame and respect. Code quality is hidden from the

user most of the time, but the incentive that the programmer can provide the user is useful features that are released often,

which may not heed the best architectural design. In this case, the incentive for developers to write code first fulfilled the

needs of the user first before engineering concerned were addressed.

What he says is true when he described the computing life as simpler once upon a time. People once used computers for

computers to do work for the most part, and now modern open source unix has to be built in a manner where computers are used

both for computers and people to do work directly. While code was simpler, many users of UNIX found the platform(s) wanting.

By the early 90s commercial UNIX was certainly written in a Cathedral model. All of them, most certainly by virtue of a market

that lacked choice given that hardware and OS were tethered, were found wanting.

His thesis of the Cathedral model being superior holds true if and only if the Cathedral in question reacts quickly and

effectively to the demands of the user. Unfortunately for the Unix world, before the rise of the bazaar model, said Cathedrals

in the Unix world have only cooperated enough to turn into an oligopoly. Before the rise of Linux and *BSD, the world had to

suffer through Unix as its detractors described in the Unix Haters Handbook. To wit, the question shouldn't be whether or notwhich development process gives one better code, but which one ships better products. The Bazaar model is the development

process gives one access to better user feedback as it welcomes development contributions directly, and henceforth ships

successful products more often.

Phil Regnauld | Mon, 20 Aug 2012 16:04:28 UTC

Poul-Henning, spot on. But it's an onset of different factors, not just the "open source" movement. This perfect storm: a lack

of vision, the internet, combined with a penchant for instant gratification ("TL;DR" mindset) has brought this about. You only

have to read the second comment to convince yourself ("the 'chaos' is what makes programming fun"). No chance I'll be flying a

plane where someone like this has contributed code :) But I think you may have another article to write on quality and

responsibility. Cf. Dan's excellent talk on software testing (Flying Linux - www.usenix.org/event/lisa04/tech/talks/klein.pdf)

Jeremy Huffman | Mon, 20 Aug 2012 16:04:49 UTC

Maybe instead of writing a rant, you could start building the replacment for the autotools - encompassing a more declaritive

shared library management system? If it worked, if there was a clear migration from the current tools and if it presented clear

advantages while being simpler to use then it would eventually be adopted. Its a massive challenge; anyone who has tried to

bite off a small piece of it has just made the problem worse so far, but that doesn't mean it cannot be done.

KP Freds | Mon, 20 Aug 2012 16:10:45 UTC

phd_student_doom, I honestly can't tell if your post is satire or not.

Assuming it's not, go hang up your keyboard and stick to the amateur leagues. In the same way that an amateur woodworker should

not be building houses, an amateur programmer should not be writing software.

Poul Henning-Kamp doesn't address this, but a side-effect of the IT boom -- including the current VC bubble -- was that anyone

that could remotely do the job was hired. This lead to inexperienced engineers having a voice and role grossly disproportionate

to their abilities, and ultimately, lead to 15+ years of wheel re-invention and history repetition amongst junior engineers who

lacked any adult supervision.

In fact, companies such as ycombinator are fueled by and intentionally encourage youthful naivety, as they require a steady

stream of heady grist for the VC mill. All it takes for them to turn tremendous profits is one successful exit out of a hundred

failures.

Your distasteful lack of respect for expertise and experience is the result of this lack of adult supervision.

metageek | Mon, 20 Aug 2012 16:11:54 UTC

This is a typical engineering point of view. Engineers like to think that the world could be designed, however Nature shows the

opposite. Any sufficiently complex system cannot be built out of design, it has to be *evolved*. Evolution is messy

(anarchic?): pieces that work get reused, pieces that do not are lost. There must be something good about autoconf that has

enabled it to survive and eliminate those awful "designed" makefiles that needed manual tweeking even though they did not check

for 15 different Fortran compilers (the status quo of the 1980s). Does it matter that autoconf looks for those now extinct

compilers? No, but it gives it robustness which is why it has survived. Someday a new, less bloated, alternative to autoconf

will appear and it will slowly overtake autoconf. In the meantime we are doing very well with it, thank you very much.

Software is the most complex construction that mankind has so far created. It is so complex that it can only be evolved much

like Nature's products are. Engineers can continue living in their nice little artificial world of linear systems and planned

devices. Those that venture in the real-world of complexity and are successful will take up Nature's ideas.

Goodbye Unix of the 80s. Linux is here to stay

rdm | Mon, 20 Aug 2012 16:15:22 UTC


5/24

1. I agree that Eric Raymond's characterization of good software development processes leaves something to be desired.

2. But I also agree with the author, here, that the "destruction of the cathedral" predates Eric Raymond's prose

3. I agree that modern systems have many inefficiencies and inadequacies, and that if someone were responsible for fixing them

that they would get fixed.

I think, though, that if we want to understand why these inefficiencies last as long as they do, we need to look at their

costs: the incremental costs are trivial, for the people currently responsible for propagating them.

(The systemic risks are much higher -- these kinds of inefficiencies are what would be called a "business opportunity" in some

circles. But there are also other "business issues" lurking in the wings. For example: the incremental value of a lot of

software is approximately the same as a [perhaps badly printed] marketing flyer, and the incremental production costs are much

lower.)

drsoong | Mon, 20 Aug 2012 16:32:25 UTC

> Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the

'chaos' is what makes programming fun. -- phd_student_doom

From this attitude, I can see that you're never going to be a good programmer. Other people are going to end up having to fix

your crap. Typical attitude for "rockstar" programmers who lack the skill, intelligence, or foundation to be decent software

engineers. You'd better find a different line of work.

Fazal Majid | Mon, 20 Aug 2012 16:38:59 UTC

I have my own set of makefiles and patches, grown organically over the last 12 years to do something similar to FreeBSD's

ports, except on Solaris and OS X. One of the reasons why it takes a full day for PHK to rebuild is the shocking lack of

parallelism. Only 215 out of 578 packages build properly when GNU make's -j flag for parallel compiles is used, and even then,

more often than not, more than half the total build time is spent in the non-parallelizable "configure" step, limiting your

opportunities to parallelize by Amdahl's law.

The open source ecosystem cannot be a soviet-style planned economy, some duplication of features by competing projects is

inevitable, and probably even desirable as competition fosters innovation. The question is when too much is too much, as withthe proliferation of crypto libraries or crackpot build automation tools.

KP Freds | Mon, 20 Aug 2012 16:47:20 UTC

metageek, sir, that is a load of bollocks: a simple-minded excuse for compounding ill-considered decisions.

You don't have to look far for better examples of engineering decision making.

As a side note, this may be the first time I've seen someone defending autoconf in the past 15 years.


@Iain: "The bazaar (I'm assuming this is to mean the free software world) [...]" This clearly proves that you are one of the

lost generation: You have no idea what a cathedral is in the first place.

@philip andrew, @Mark and others: I'm not arguing that cathedrals is a better solution, they are certainly not without their

own kind of problems. What I'm pointing out that is people like iain don't even know what they are in the first place.

@Jeremy Huffman: I have only a small calendar and must live with it.

@metageek: You're _totally_ missing the point.

Jacob Halln | Mon, 20 Aug 2012 16:54:10 UTC

The *BSD distributions are legacy code. To maintain an operating system distribution you need a massive effort. While nobody

does it perfectly, there are some organizations in the Linux sphere that do quite a good job. Debian/Ubuntu for instance.

The fact that your Firefox build-depends on 122 different packages is not strange. To build some graphics libraries that are

prerequisites for building Firefox, you will need TIFF support, even though Firefox doesn't support TIFF.

The way around this problem for a user is to not build the entire software suite, but instead use binary packages from a

trusted source.

To think that the problems of distribution management at this scale can be solved by up-front design is very nave.

David Kastrup | Mon, 20 Aug 2012 16:56:12 UTC

At Mark Dugger: "I respectfully disagree. The Cathedral model only works if one is capable of defining the scope of theproblem completely from the outset of the program. In systems programming, hardware doesn't change nearly as much as an

organic customer. Therefore development processes need to change given what the code is interfacing."

You confuse the Cathedral/Bazaar dichotomy, which is about continuity of the executive, with working to plan, which is about

continuity of targets.

In fact, combining changing targets with changing responsible programmers is a recipe for cruft: they don't have the knowledge

which code pieces have been tailored to different targets and should be replaced, rewritten, or removed. So some idioms may

stay around when they are no longer a good match to reality.

For example, take a look at the original fork system call: why was this a sensible design choice? Because it had a very

elegant implementation sitting well with the memory subsystem: fork just swapped out the current process but proceeded without

marking the in-memory copy invalid. In the time of demand-paging, this idiom makes rather little sense as a primitive.

Tom Alexander | Mon, 20 Aug 2012 17:13:21 UTC

I'd also recommend reading this much shorter rant from him about Autotools. https://www.varnish-

cache.org/docs/2.1/phk/autocrap.html


6/24

metageek | Mon, 20 Aug 2012 17:14:44 UTC

@Poul-Henning Kamp: I thought the point was whether you can plan the entirety of the construct versus building it in an

evolutionary way. My argument is:

1) it is impossible to plan the entirety of a complex system at the outset,

2) most software is indeed sufficiently complex to fit point 1

3) evolutionary construction is the solution: smaller parts of the system are assembled through objective criteria

4) code bloat that is not harmful will last for a long time

if someone cared to remove the fortran checks, autoconf would not work much more efficiently than it does now, so leaving it in

is fitness neutral (which is why it is still there, since it costs energy to remove it)

Apologies if I am still missing the point ;)

@KP Freds: I am not defending autoconf, just explaining why it has so far dominated. I am not making excuses for not believingthat classical engineering approaches do not scale up to large and nonlinear constructions. Nature keeps showing how it is

done, and while it is messy and chaotic, it is incredibly efficient and robust.

I have cursed against autoconf many times too, but the energy to go around its problems is orders of magnitude less than to

make an "efficient" alternative.

maht | Mon, 20 Aug 2012 17:26:33 UTC

Talking of plan9, we have a saying, coined by Boyd Roberts, "this is not Linux, there are rules".

Dave Presotto also graced us with "Linux, by amateurs, for amateurs".

Which, I think, summarises the spirit of your post.

Plan9, where even #ifdef is eschewed.


metageek: cathedrals can evolve, many of them did, some of them have changed major style three times or more. Some of them

evolve to this day. And those are built out of stone :-)

My point really is that Brooks has written a wonderful book, but a lot of people are simply not equipped with the necessary

mental models and background knowledge to understand what The Old Man is talking about.


If plan9 had been available in anywhere near runnable form in 1992, I wouldn't have spent two decades on FreeBSD. I actually

asked and couldn't get a license.

Steve | Mon, 20 Aug 2012 17:36:12 UTC

@Matt Welsh: Funny you should mention Google. They are far from innocent, having built their own Makefile generator known as

"gyp", which have in turn their own portability problems, and often bundling dependencies along with their source, not updating

software to work with newer versions of those dependencies. These things happen out of necessity though. By definition, "hacks"

are things that accomplish the short term goal in the short term. PHK rightly laments the long term cost. Whether you aretalking about software or business, the short term goal is usually the winner. Twas ever thus...

Shawn Garbett | Mon, 20 Aug 2012 17:48:31 UTC

I view the problem is that the chance of project failure increases with scope. This causes most "cathedral" designs to fail.

Address this key issue and the bazaar will move into the background. It'll be like the discovery of steel impacting on the

architecture of buildings if you succeed. Till then the market consists mostly of yurts and tents.

This said, I wouldn't run national security, a mars probe or space shuttle on bazaar code. I.e., how important is the project?

Important enough there will be an investment of time/money to solve the design issues?

Eric S. Raymond | Mon, 20 Aug 2012 17:52:05 UTC

There are a number of ways in which this rant is defective. One of them is mistaking a local problem for a global one; yes,

autoconf is a pile of festering crocks, but the step where this demonstrates that the bazaar model is a failure is just a

handwave and an assertion.

But the largest failure is in the unstated assumption that there is an alternative - that at the scale of Linux or *BSD you canhave anything *but* what Kamp dismisses as "chaos". It is worth remembering that these artifacts are orders of magnitudes

larger in LOC and other measures of complexity than the successful cathedrals of programming legend. They are not just

operating systems in the traditional sense but application suites of unprecedented size.

Nothing this vast can ever have a single person responsible for overall quality, because that task is beyond the cognitive

capability of any individual human being. The implicit alternative Kamp is proposing is impossible - it doesn't scale.

iain | Mon, 20 Aug 2012 17:53:09 UTC

@Poul-Henning Kamp: Great come back. I was going to leave it, but no, I won't. I know perfectly well what the cathedral is,

however none of your complaints stem from any difference between the two models, and rather stem from your belief that all

systems should be simple enough for you to understand and build in under a day. For example, I'm pretty certain your libtiff

complaint was your own fault and if you had passed the correct options to the configure file you would have not needed to build

it.

None of the good points in your rant are anything new, people have been asking why libtool needs to check for fortran for

years, people have been reusing code and trying to reduce dependencies for years.


7/24


@ESR: Your reading of my piece is highly defective :-)

I'm not saying the bazaar method is defective, I'm saying it is so "successful" that most people don't even know what a

cathedral is anymore.

Try checking the comments in hackernews or reddit, people are forcefully making that point for me :-)


@iain: I am pretty sure my piece includes this text: "This is a horribly bad idea, already much criticized back in the 1980s

when it appeared," :-)

RORVI | Mon, 20 Aug 2012 18:07:41 UTC

Let me ask you something ....Why do you think these "cathedral breaking programmers" appeared ? Is it because people like you

oftenly refuse to answer a "dumb question" ? Or is it because books like "Teach yourself C++ in 24 hours" appeared ? When was

the last time you answered a question of one of these "dumb programmers" ? The lack of consistency in coding nowadays can also

be the result of a very fast growing industry,that needs more,better code that is written very fast.Also,nobody mentors

nowadays.I have yet to find somebody to spend some of their time to answer a question or two and give some advice.(i myself am

18,and i started programming 2 years ago !).So one of the most painful questions is : where are the mentors,the experienced

programmers and ,for example,a website where they can select beginners in order to mentor them(this should be done for free,or

for symbolic prices,not for 200$ a course,since in some countries the average salary is low-in my country is 300 $).So,in the

conclusion,i would like to point out the fact that you ask for better,but you don't do anything to help shape the experts that

will create great software.

Andre | Mon, 20 Aug 2012 18:11:00 UTC

"Patches welcome."

The sound of crickets that come after those words is the problem. Not the Baazar, Cathedral, Developer, Engineer or Hack.

David F. Skoll | Mon, 20 Aug 2012 18:14:35 UTC

Your article was interesting and thought provoking. You are right: Open-source systems like Linux are a mess for developers.

They are insanely inconsistent.

But like democracy, the system is the worst system except for all the others. Proprietary systems like Microsoft Windows have

plenty of API inconsistencies, weird API warts, magical and mysterious registry settings, etc.

I think these system simply reflect human nature, which is to be wildly creative but (alas) inconsistent, lazy and messy.


@RORVI:

I think you pinpoint a lot of the auxillary causes correctly.

Please notice that I don't blame all the dot-com wunderkinds for grabbing opportunity, I merely note that it was a catastrophyfor CS/IT as a discipline that they got the opportunity in the first place.

With respect to mentoring, I have mentored a fair number of people myself, most of them FreeBSD committers, and I have always

done it for free.

But I have to make a living, so I can't mentor everybody who wants me to.

I also have to be pretty ruthless to my mailbox, and I hate that, but I cannot spend 2-3 hours a day on email that doesn't put

bread on my table. You'd be surprised at the number of emails I get from students whenever a professor gives homework related

to code I have written...

So all in all, I don't feel that your claim that "I don't do anything" is justified.

But I can certainly understand your frustration: I started out in a back-water country, with no net-access, and had to teach

myself everything from books I had to order home from abroad, at great expense. Mentors were nowhere to be had.

rayohauno | Mon, 20 Aug 2012 18:28:56 UTC

@metageek beautiful words !!! I fully agree.

anon | Mon, 20 Aug 2012 18:42:23 UTC

"as opposed to the carefully designed cathedrals of the pre-dot-com years"

Right there in paragraph 6 is the flaw in your thesis.

Neither Cathedral nor Bazaar entails Quality.

CuriousWayne | Mon, 20 Aug 2012 18:44:50 UTC

It's been awhile, but some cobwebs shook loose while reading this post. The Cathedral and the Bazaar was presented as a

metaphor for centralized control of code in the Data Center (AKA the priesthood of Data Processing, hence, a Cathedral) versus

distributed programming (AKA Linux).

From the first reading, I sensed a weakness in the metaphor; while the Data Center as Cathedral works OK, Linux was not a

Bazaar. Linus Torvalds was King of Linux and dictated what got in and what did not for the longest time. Linus was a single

programmer scratching his own itch for a usable, affordable OS (the most effecient and effective programming team possible).


8/24

Furthermore, the DotCom bubble was not a Bazaar. Bazaars are competitive but predictable; it is highly unlikely for someone

with too much money to come into a Bazaar and buy out all of the vendors' stuff.

Yet that was exactly how a Silicon Valley VC, after the bubble burst, described to me why the bubble happened in the first

place; too much money was chasing too few opportunities, and the VC community lost their quality controls on their investment

process.

I have respect for Mr. Kamp and his perspective on software engineering. I have respect for the modern paradigm of Web

programming that breaks a simple idea like a website into a billion moving, dependent parts with specialists running all over

the place. The two perspectives are somewhat mutually exclusive (or similar) in that they are solving different technical

problems using coding techniques.

Having worked in other engineering disciplines, none have the challenges or the leverage that software has. Software has no

universal and unambiguous design language, nor is it likely to have one anytime soon (stuff changes too fast; the design

language is more dependent on the makeup of the team than the problem).

By contrast, any junior engineer in any other discipline (anyone know of an exception to this?) can spec a design and get itmade on the other side of the planet by people that share nothin else than a universal design language.

Here's what I think is a more apt metaphor for today vs legacy than the cathedral and the bazaar: if you had a performance

problem in legacy days, you might get fired if you 'threw hardware at it'. If you have a performance problem today, you might

get fired if you don't, because hardware is now often cheaper than paying to tune code.


@anon:

Where do you see "quality" in the text you quote ?

Just because something is carefully designed doesn't mean that it is good.

See "galloping gertie" for just one of many examples of the opposite.


Damn, I hate it when I remember an even better example seconds later:

@anon: Best example: The Hubble telescopes main mirror: The mirror that was most precisely ground wrong, ever.

Tommy McGuire | Mon, 20 Aug 2012 18:53:45 UTC

PHK is said to have been programming computers for 26 years. I would have thought he would have remembered the '80's and '90's

at least as well as I do, but I could be wrong. I also thought that the whole point of the BSD effort was to build software

that anyone could do anything they wanted do with; certainly IBM did that, when they created an ld(1) that required a good deal

more than one flag to build a shared library, among other things.

Should standards have done something about that? Sure, but standards are not handed down by the Almighty, or even by Fred

Brooks. They're built by people with many agendas, including ensuring that the standard does not inconvenience their market

share. (This might make PHK smile; it does for me: the memory of the environment variable POSIX_ME_HARDER.)

Now, I suppose you could blame the hideous environment that resulted in the terrible solution of the autotools on the bazaar,

in that AT&T did commercially license Unix in the early '90's and thereby allow more people to soil it's pristine beauty. But

to say that was "robbing it of its architects" seems to me to be a bit too much. I knew some of the people working on AIX at

IBM at that time, and they were all very good, and all as dedicated to the beauty of the cathedral as anyone you could putinside any fortified wall you want to build. But, heck, if I really wanted to pinpoint the moment Unix began to fragment, I

might have to go back to the seventh edition, which did not have the odd proliferation of system calls that includes socket(2).

So, if any wunderkinder really deserve blame, they are probably not kinder any more. And since the bazaar is so bad, remember

that there are cathedrals out there for you to worship at; they must be better. Right?

Paul W. Homer | Mon, 20 Aug 2012 18:57:00 UTC

You are right. The stuff we have has been degenerating for years, and each new wave of kids blows us off because they lack the

knowledge to really understand the technologies that they are wielding or creating. I also think that it's gotten significantly

worse in the last decade, that we've right in the middle of a 'dark ages' that is dragging us backwards.

At first I just ranted but nobody wants to listen, they're too busy re-writing the same wheels over and over again. These days

when I blog I try to be more positive, but not always:

http://theprogrammersparadox.blogspot.ca/2011/05/software-virgin.html

It's all very disappointing, and the trend has been for it to get worse. I think it's going to be a long time (generations)

before the winds change, people tend to wear blinders when exposed to too much complexity, which forms a vicious cycle.

Paul.

PS. I miss Unix too, my favorite version was a pure, clean, Sys V from AT&T in the early nineties.


@Tommy McGuire:

I have it from first hand, that people at Bell Labs found the socket API hideous, and the SVID IPC and STREAMS was even less to

their liking.

With respect to standards, yes: The UNIX-vendors shot themselves (and us) in the foot bigtime, by photocopying the SVR manuals

into "the UNIX standard" (how ever many there were of those), rather than look at what the future would require of them.

That's why we still call N system calls just to open a TCP socket connection to somewhere, rather than a convenient single

function in a well designed API.

But the lost dot-com generation is an entirely different issue: They have essentially wiped out all CS history before 1990, by

never bothering to read any of it, and that means we have to slowly relearn what we once knew about the craft of IT.


9/24

And the particular fact that the do not even recognize the architected cathedral, is a particular expensive bit of amnesia.

Justin Alan Ryan | Mon, 20 Aug 2012 19:10:58 UTC

Cathedrals are built by some of the greediest, yet least financially stable companies in the world.

I also find it really interesting that the author insists on using FreeBSD on his laptop, but clearly knows less about free

software and dependencies than I know about my Ubuntu hosts.

There are some sticky points when it comes to dependencies. A cathedral would provide us with less choices. Why does glib

depend on both Perl and Python? Because glib is a library that many people use from Perl and Python, and it's set to build

those bindings.

Buy a fucking mac, install XCode, and tell me about the beauty of the cathedral.


Dear Justin,

Thanks for illustrating my point about the ignorance of the lost generation so forcefully.

You clearly have no idea about what a cathedral is in this context, you clearly did not bother to find out and you spew random

insults at the author, without even bothering to check if you were about to make a total ass of youself thereby.

The reason I run FreeBSD is that I wrote a very significant part of it and therefore I probably know more about dependencies

than you ever will. Building the perl/python bindings as part of glibc is ass backwards in my book. As for your "fucking

mac", last I checked, my name was 35 times in the kernel of the open source version ("Darwin") of the OSX kernel.

Peter Lindhard | Mon, 20 Aug 2012 19:43:38 UTC

Perhaps it could be a general point in OSS projects, that contributors are reminded/persuaded to reread their parts at a later

date. I honestly found it disturbing, to read what seems like disappointment that newcomers don't/can't do 20 years of code-

clean up.

Author seems blind to new challenges/interests, that new developers face/embrace. To be very honest antisocial ranting apparent

in several publications by this author, makes for yet another terrible read.

mcur | Mon, 20 Aug 2012 19:50:47 UTC

Mark Williams wasn't a paint company.

http://en.wikipedia.org/wiki/Mark_Williams_Company

Sven Trpe | Mon, 20 Aug 2012 19:54:50 UTC

You are free to go to a restaurant rather than to the bazaar. You seem, however, to insist on preparing your meal yourself.

Yes, this involves a variety of imperfect tools, and it makes a mess that you'll have to clean up befor you're done. But such

was your choice, you wanted to prepare a meal and not go to a restaurant where the chef would have done it for you. In the

restaurant, you install Firefox by dragging an orange icon over to the Applications folder, that's it. Somebody else takes care

of the 122 ingredients for you.

Rick Russell | Mon, 20 Aug 2012 20:26:14 UTC

Mr. Kamp, your credentials are without fault, and I certainly am not qualified to question any of the specifics you discuss.

But despite all these perfectly legitimate observations, a good Linux distribution like Ubuntu is not only easy to install, but

it is easy to maintain. The issues of package dependency, whatever they may be, are effectively hidden behind a rock-solid

package management system that seems to have very few failure modes, at least to this 80s UNIX user and occasional programmer.

I remember installing software on AIX, SunOS, Solaris and IRIX systems back in the late 80s & early 90s, before the Linux

revolution and I would say that, subjectively, it was much more difficult to get working software back then. I don't just mean

open source, although open source was part of it. Perhaps it was because the dependency mapping was not as complete -- you'd

install a piece of commercial software on your Solaris environment and find out that one part of the software required a

version of Motif you didn't have, and then you find out that version of Motif isn't compatible with the X terminals your labs

use, and then... ?

There may be errors in the dependency tree for some applications in FreeBSD, but at least the dependency tree *exists* in some

form. If individual packages are claiming unnecessary dependencies, then maybe the people who write that software need to clean

up their act. In the bazaar, you have to tolerate such sloppiness. In the cathedral, that software might not be available at

all because it never sees the light of day. Neither solution is optimal, but the bazaar seems preferable, to me, since at leastI have a shot at getting it to work.

Elizabeth Marston | Mon, 20 Aug 2012 20:30:09 UTC

Great article. It's a shame how much ignorant ageism is in the comments --- winning an argument on reason rather than emotion

is, ironically, an application of rationality at which engineers are often poor.

What cathedrals have going for them is that they are beautiful. What bazaars have going for them is life.

FOSS didn't win the mindshare wars through propaganda and hazy hippie ideas about sharing; it won by being better; by building

more slowly; by being inclusive. And inclusivity always means mess, because not everyone has the same intentions or tastes.

And how is UNIX dying? Please support this claim. From where I sit, it looks like it's winning by default. Code I write for

Linux tends to resist bitrot. Unlike, say, the code I've written for Windows-land, where MS effectively says "and now for

something completely different" every four years (R.I.P., C#) and thereby generates enormous waste.

It's that careful, incremental, inclusivity-loving bazaar mentality that allows us to bring the full force of our UNIX and C

heritage to bear on any problem, and is the essence of code reuse.


10/24

The results speak for themselves. My apartment has a single, old, moldering Windows box and a slew of iOS, OSX, Android and

Linux devices. That UNIX:Windows ratio has been constant for about ten years. What's changed is that my retired parents' place

is now approaching that ratio as well.

Put more clearly: software that has no unnecessary parts is software that can never be put to any purpose except those foreseen

(and permitted) by the designer. It is as fragile and as beautiful as a snowflake, and tends to last about as long before

someone has a better, entirely incompatible idea. It's what killed Esperanto, and it's what will kill Windows (and many other

commercial projects.)

Top-down design works great for as long as everyone shares (or at least respects) the intentions of the designer. But that

necessarily means quashing other voices. Perhaps some of those voices aren't as clear, or those speaking have impoverished

mental models of how the OS functions; but others will simply be different opinions about an underdetermined solution space.


(Quick clarification on my last post: -- the phrase 'what will kill Windows' isn't clear in context, as Windows has many

unnecessary parts :3 What will kill Windows is the _anxiety_ that its developers have about the unnecessary parts, and their

repeated attempts to deprecate whatever isn't cool this week.)

Hector Correa | Mon, 20 Aug 2012 20:53:47 UTC

I find it interesting that even systems owned by a single company (like MS Windows) are also messy.

Microsoft has done a great job of simplifying the installation of Windows (particularly compared to what you describe for your

system!) but as David F. Skoll mentioned in a previous comment the API is equally messy if you look behind the scenes. Heck,

there were even some well publicized stories about some of the hacks buried in the Windows source code to account for wacky

scenarios (e.g. http://www.kuro5hin.org/story/2004/2/15/71552/7795)

Even more interesting, is that none of these systems (Windows or Unix) are typically maintained by "casual software developers"

that just took on development overnight during the dot-com era.

I think the mess inside these systems is just the nature of large systems, maintained by a large number of people over a long

time period.

Mike Smith | Mon, 20 Aug 2012 21:16:28 UTC

Poul,

Thanks for putting this together so cogently. Your rants are always thought-provoking and a good read.

Your point about design had me wondering; why is "good design" currently trendy when it comes to UX, but uncool when it comes

to software (and many other engineering disciplines). Is it really just a socialisation problem?

= Mike


Isn't that simply because "civilians" see UX, but not source code, and there are more people with a sense of style, sthetics

and architecture amongst the "civilians" ?


Poul: With a background that straddles both UX and low-level dev, I've been wondering this myself. At first I thought it was an

aesthetic thing. But now I think that it's just straight-up conflict of interest, like we're used to seeing elsewhere in the

public sphere. Developers can't advocate for users because they're _developers_ -- their best interests are in long-term

architectural stability and elegance, not in ease-of-use or rapid production. Developers are intrinsically lazy, and do not

wish to make work for themselves; and in every other context, this is a good thing. Do not expect a developer to agitate

passionately against the best interests of developers, as a UX specialist must often do.

Jonathan Abbey | Mon, 20 Aug 2012 21:45:25 UTC

I hope you will follow this column with one that more fully illustrates the nature of these cathedrals, ideally with examples

for us to enjoy.

David F. Skoll | Mon, 20 Aug 2012 22:01:29 UTC

Oh, some more random thoughts... I've been programming professionally for about 18 years and have participated in my share of

both open-source and closed-source projects.

In my experience, no matter how careful you are and how good your intentions are, almost all large software projects tend to

have a few ugly dark nuggets at their core. Perhaps it was done that way to work around limitations of the development

environment, or perhaps the original design was found lacking and it was easier to patch things up than do a major redesign.

But we're only human: We can't predict everything and sometimes a workaround (aka a hack) is more efficient and more productive

than a redesign. Sometimes the perfect really is the enemy of the good.

Again writing from personal experience, I'm much more likely to try to exorcise the ugly dark nuggets from an open-source

project than from a closed-source one. In the open-source project, my peers can read my code and I might be subject to their

derision. In a closed-source project, everything can be swept nicely under the rug.

(I realize that "open source" does not mean "bazaar", but "closed source" usually does mean "cathedral.")

Kartik Agaram | Mon, 20 Aug 2012 22:09:57 UTC

@Mike Smith, @phk: Reading your conversation it occurred to me that for all the benefits of commandlines that we geeks like to

rant about, a GUI has one key benefit. Anybody who looks at it can instantly spot ugliness and cruft to some extent. By


11/24

contrast, a commandline-oriented stack is a lot less discoverable, and encourages the formation of various 'commons' like

dotfiles, extra commandline flags, open file handles. Since nobody is monitoring these areas for cruft, we developers dump

indiscriminately on them.

This point is independent of how savvy the reader is, I think. My visual cortex instantly shows me what's crufty about a UI.

Dotfiles on the other hand I have to deliberately go dig around in to criticize.

Anton Striklov | Mon, 20 Aug 2012 22:11:16 UTC

As much as I like FreeBSD one cannot equate the success / failure of the whole bazaar ecosystem to it. FreeBSD is awesome in

some aspects and terrible at others, and while many projects lag behind it, plenty far surpass it. It's package management

system for example if far inferior to the one present in Gentoo. But why stick with Unix OS's for analysis ? The sheer volume

of non-OS related successful projects developed under the bazaar model is staggering !

ford p | Mon, 20 Aug 2012 23:54:18 UTC

This article is horseshit. Total flamebait and founded on the obviously false assumption that you ever need to compile a whole

distro from source. Was this article written in the 1980s and you just got around to UUCPing it off from paper tape?

Bernd Jendrissek | Tue, 21 Aug 2012 02:54:11 UTC

Much of your rant seems predicated on the badness of the autotools. While some of your criticisms are certainly valid (even

though I might not necessarily agree what the response should be), you seem to be burning the sources of inelegant software in

effigy, by attacking the autotools. The creators of the autotools are not the ones responsible for the problems which the

autotools were designed to address!

Secondly, some of your criticisms (here and in your autocrap rant) against the autotools are again misplaced: you seem to want

to hold the autotools' creators responsible for the misuses to which others put them, sometimes due to innocent ignorance,

other times due to an arrogant sense of superiority (poor translation: I want to convey the german word "besserwisserisch").

As others have pointed out, I'm not surprised either that there are features in a large dependency graph that go into a black

hole, like the TIFF support you mention. My curiosity is somewhat piqued to know how this irony resolves. And specifically, is

this a run-time dependency or a build-time dependency?

But here's another irony: you don't recognize the deliberate architecture to which the autotools were built. I wasn't around at

the time to know if they were originally built cathedral-style (their development now seems very bazaar-like), and you might

not appreciate the architectural style used, but boy, you'd better believe it, much of your interaction with them was

*designed*. In my experience most of the "friction" in interacting with the autotools is due to the very sort of meta-ignorance

you write of: not even knowing what the tools' raison d'tre are, or "disagreeing" with them (as if that is possible). I like

to say: Those who do not understand the autotools are doomed to reinvent them, poorly. Please understand that I mean a

philosophical understanding; I'm sure you know very well what AC_CHECK_LIBS does, for example. And yes, I think I can agree

with your criticism of keeping around 20-years-superseded feature checks.

P.S. I know I'm conflating cathedral-as-designed-artifact with cathedral-as-home-of-a-priesthood.

mike mccafferty | Tue, 21 Aug 2012 03:19:19 UTC

The dilution of a profession's core skills due to population explosion strikes me as an interesting twist on Naomi Klein's

"Shock Doctrine." Dotcom-era opportunities create a virtual Gold Rush, people with wildly varying degrees of of technical

skill rush into a space, and the IT population growth rate is such that experienced hands simply cannot transmit best practices

fast enough to keep pace with the population increase. Soon the notion of "state of the art" has changed, because a large

percentage of the population doesn't know where to look, and doesn't know who to ask for pointers. It's the "telephone" game,

played out on the clock and at a velocity that prevents error correction.

Thanks for recommending out the book. I know that I will benefit from the lessons contained within.

Terry Lambert | Tue, 21 Aug 2012 03:51:25 UTC

When Sun switched over to SVR4, it became impossible to to get BattleMap to run on any decent hardware. Now it's sold by

McCabe software as McCabe IQ. It's down from $50,000 a seat to $45,000 for three floating licenses.

I think one of the issues with the Bazaar model is that younger programmers, particularly the ones Poul-Henning identified as

coming out of the .bomb era as being hired out of their second year in CS coursework to be cubicle warmers at startups which

went nowhere, wouldn't recognize a decent CASE tool if it bit them on the ass.

Complexity is manageable through abstraction, and through good tools, and anyone who doesn't understand that is doomed to

recreate their own version of the autotools mess in their own projects. Eventually I will probably run into them in McDonalds

while they are asking me "Do you want fries with that".

Also, I take umbrage at Phillip Andrew's "break everything into tiny programs" model of development; an inability to manage

complexity is no reason to drink the Daniel J. Bernstein Kool-Aid that has djbdns _still_ unable to do zone transfers, and

synchronizing primary and secondary DNS servers "is an exercise left for the student". The attack surface on 50 small programsis way larger than the attack surface on one large program. It's why we still build monolithic kernels.

Poul-Henning and I haven't always seen eye to eye on everything, but his post here is spot on. Good on you, PHK.

Max Kanat-Alexander | Tue, 21 Aug 2012 05:33:19 UTC

First, let me say that this is an excellent article. For many years, I was the Chief Architect of the Bugzilla Project and

heavily involved in the open source community as a whole, and I can assure all readers here that there will never be a model of

software development in which developer responsibility can be neglected. In every model that will ever exist, developer

understanding and responsibility will be the key component to quality. There are methodologies that make quality hard and

methodologies that make quality easy, but there are no models that create quality code without guidance from some unified

viewpoint and individual responsibility for software design down to the lowest levels of development.

The bazaar has all the problems that anarchy has--if a few good people step up to accomplish and lead things, then they go

well. If nobody steps up to be responsible for quality, then even though things get built, they can't be nice things.

The cathedral has all the problems of a dictatorship. If the dictator is wise and benevolent, then the project prospers and is

beautiful. If the dictator is ignorant or possessed of motives beyond the good of the project, then the project can fail and be


12/24

disastrous.

Our modern economy throws in one additional crowbar to the cathedral as well--that most people making centrally-controlled

projects are creating them under the banner of organizations that do not want external contributions and which do not which to

collaborate with others. The cathedral has been creating just as much duplication of effort in the past many decades as has the

bazaar.

And finally, an even more salient point to touch on is that perhaps "the bazaar" is itself a myth. The best open-source

projects are, in reality, cathedrals in which anybody is allowed to build pieces, but only after they have passed the approval

of the central control. We could certainly image a "bazaar" to exist--some set of projects that have somebody birthed

themselves out of anarchy. But there's no situation in which that's true. Software is conceived, created, and guided by people,

and when those people have great understanding and all take strong responsibility, excellent work is done. When they neglect

their code or assume that "somebody else will take care of it," then ugly disorder ensues.

-Max

Justen | Tue, 21 Aug 2012 05:45:25 UTC

Sounds like your problem is a crap package management system more than anything. I can't believe you have to actually manually

research and add dependencies. What decade are you living in?

I'd also like to note that development methodology has little do to with the problem of quality. The internet is swimming with

horrific windows shareware that is not only *not* package managed, and full of bugs and compatibility issues even though they

depend on an essentially monolithic platform and set of libraries, but still inexplicably expects us to pay for "full"

features. One of the greatest joys I discovered in moving to Linux was all the free, relatively bug-free, fully functional

basic utility software of the kind that every computer needs to be useful - all in a central packaging system that not only

installs software in one click (along with all its dependencies) but keeps track of when it needs to be updated too! And don't

even get me started on Mac, a platform so terrified of free (as in beer or as in freedom) and open source that I was just asked

to pay $8 for a script that finds and consolidates duplicate music files in iTunes. Are you *serious*? How is that not a built-

in feature of every media library manager in existence, let alone one of the keystones of the Apple experience.

Nah man, I think you're looking at this through too narrow a lens. And maybe you need to get off FreeBSD, the red headed

stepchild of of ghetto *nixes.

Justen | Tue, 21 Aug 2012 06:43:22 UTC

After reading the further discussion in the comments, I have a greater understanding of what you're actually trying to complain

about here. But you've turned what ought to be a simple lamentation about cruft into a jermiad. There are reasons why this

cruft sticks around: because somewhere out there, someone wants to install a bit of *nix software on a truly ancient machine

where it will actually matter; because a few thousand clocks is infinitesimal in cost to any given user compared to the cost to

a developer to carefully excise it; because the vast majority of users don't know it's there and wouldn't even know why it's

bad if they did; because most of us use binary package systems, only supplementing with compiled-from-source binaries when they

are too obscure to merit inclusion in the repositories.

The *nix ecosystem is incredibly massive. If it's true that you can only get a cathedral from continuity of executive, and that

a single virtuoso, or rarely a dynamic duo, must manage an entire project with an iron fist in order to achieve cathedraldom;

then where is this godlike ego you propose to manage all of *nix? Or shall we commit the even bigger sin of tossing it all out

and starting over so that, this time, we can do it "right"? Even when standards are proposed, or when they emerge from common

practices, there will always be the outliers (*coughbsdcough*) who insist on doing it differently and they'll still want to run

common *nix software. Inevitably some monster like autoconf and friends will arrive on the scene to facilitate that.

Now as to M4, I have no apologies for that. Ugh. I thought Perl was opaque until the first time I tried to configure sendmail.

Pete | Tue, 21 Aug 2012 07:10:25 UTC

There are some good points here, but there's a couple glaring problems.

The first half of the article laments the mass of "college dropouts" who entered the technology field around the turn of the

millennium. The second half of the article laments the mess of legacy technical decisions that we are stuck with. But for the

most part, the former did not cause the latter!

For example, you spend a while talking about the disaster that is autoconf. (I think this is one of those tar-babies that most

people don't bother talking about any more, because everybody knows how bad it is.) But it was about the furthest thing from a

dot-com-era college project: it predates that party by about a decade. It was created to solve a real problem (hand-editing

configure scripts), back in the day.

This leads straight into the second problem. It's perhaps not fair to complain about autoconf itself, since it solved a real

problem back in the early 1990's, but it is fair to complain about modern projects like Firefox continuing to adopt it, rather

than being more portable from the start. But Firefox, for the most part, *works*. It exists. It delivers what people want.

Imagine a web browser built with more solid software engineering principles. Does it give users anything they don't have

today? (Firefox is pretty stable, and it does everything I think I need.) Does it make life better for its developers?

(Apparently there are developers willing to work on Firefox as it is today for free, but I don't see anyone doing the same for

a serious alternative.) What tangible benefit would it actually have, except to make some of us feel better about it? We have

working, free, open-source web browsers, which are faster and more capable than ever. I wish they had a better architecture,but it's hard to complain too much when things are so much better than 10 years ago.

Finally, you also dish out against Unix and autoconf quite a bit, but I'm not sure if this is because other operating systems

are any better, or because it's simply the most public one, so we see all the problems.

holger | Tue, 21 Aug 2012 10:22:05 UTC

The true cause indeed seems to be the explosion of IT "professionals" we saw in the years leading up to and during the dot com

boom. Things just had to get done (and quickly) because otherwise someone else could have been first to market. Creating an

incredible demand for IT "somethings" and attracting lots of people with little or no formal education in the field. But the

problem is not only lack of education: today the core problem applies to cathedral and bazaar software in exactly the same way.

Because what coincided with the dot com boom (of not being the cause of it) was that hardware advanced faster than ever before

in those years. Meaning you could pretty much get away with less then optimal code (and coding practices) most of the time.

Almost nobody cares about efficient design (much less code efficiency) anymore, it's all about quick design and development

efficiency. Once something (mostly) works, it's good enough. Needless to say, it usually isn't,

but that isn't easily noticable on the developers machine. But deploy that code to a hundred thousand servers or a couple

million users, and the overall balance shifts - inefficient code translates to wasted computing resources and wasted energy on

a very large scale. And to make matters worse, object oriented languages, while being in invaluable design tool to the


13/24

programmer who is experienced with them, make it unbelievably easy to write horribly inefficient and bloated code without even

noticing if you lack said experience. (Not to mention to include more dependencies than you can count if code reuse means

"someone else probably already wrote something similar, so i'll just google" to you).

Someone actually suggested using the libgmp bindings for bit manipulations in some language that apparently didn't readily

offer them in a comment on thedailywtf.com because he googled and found out that it had something similar (for bignums if I

remember correctly, but hey, you can always convert and back, right?). That was one of my largest wtfs i ever saw there.

And even the better programmers out there often end up adding layers over layers of abstraction to make an overly complex

problem somewhat tractable without ever questioning the underlying design once there is one that seems to work "good enough".

In a bazaar style project, there's at least the off chance someone is going to revisit things. In a cathedral style project,

design decisions tend to be set in stone once the first release is out "that's just the way things are done around here".

Alkonaut | Tue, 21 Aug 2012 10:41:13 UTC

Don't confuse the "bazaar" with Linux. Linux is a perfect example of a cathedral. Complete with its high priest. The linux

*distros* (e.g. GNU/Linux) shows the real "chaos" of the bazaar.

foljs | Tue, 21 Aug 2012 10:42:18 UTC

@phd_student_doom

"""Yawn, I can't wait till these type of people retire. I've only been alive 2/3 of the time he's been programming but the

'chaos' is what makes programming fun."""

Well, speaking as someone of your age, sure, it will be a very "fun" IT landscape indeed when ignorants like you take over

(even more so considering this BS non-response was written by a "phd student").

This "chaos" is what makes programming fun?

Spoken like someone who doesn't understand accidental and essential complexity at all, and cherishes trivial bullshit tinkering

over actually making something, much less making something beautiful.

Glenn French II | Tue, 21 Aug 2012 12:32:15 UTC

I figured that I would chime in once I read all of the other unprepared and ill-advised comments.

On another note, thanks for all of the contributions.

David MacKenzie | Tue, 21 Aug 2012 13:34:45 UTC

As the creator of Autoconf, I'd like to apologize to everyone for using m4. I chose it to minimize dependencies, as it was on

every Unix-like system already. Many of us were on 28kbps dialup and people would complain about the time needed to download a

big package like Perl. Then I hit so many Unix m4 limitations and bugs that I ended up requiring the GNU version anyway. I

probably should have started over at that point with another approach. It's true that I did most of that work before I'd earned

my CS degree, but no one more experienced deigned to tackle the practical problems I needed to solve, so I had to. (Larry Wall

had half-tackled them.)

When I did the initial design of Automake a few years later, Perl had become widespread enough that I could get away with

specifying it as the implementation language. (I wasn't involved with libtool.)

The core GNU project (not to be confused with the ports or other packages like X11) does have a quality assurance dictator of

sorts in Richard Stallman and the GNU Coding Standards. A lot of the work I and other maintainers did on GNU was bringing otherpeoples' code up to those quality standards, so that our versions of POSIX commands and libraries had fewer arbitrary

limitations and buggy corner cases than the Unix versions that originated in Bell Labs. Our source code was also supposed to be

readable and well documented. We had the advantage of another decade or two in processing power and RAM at our disposal

compared to the original Unix developers, who by then were working on Plan 9 which was elegant but never really became

available to use until it was itself a dead project.

When I started Autoconf, there were no full Unix-like operating systems with all the source code available to everyone, so we

had to bootstrap ours on the variety of existing systems and accomodate their variations.

It doesn't solve any practical problems to say that applications like Firefox should be written in a more portable way so there

isn't a need for something like Autoconf. Unless you want to reduce the whole global software industry to a single company or

government regulation, communist-style, you can't control what other people are going to do. And so you need to work around the

diversity.

The dependencies-of-dependencies tree can indeed get large when you're leveraging general-purpose tools. And keep in mind that

although you need to build some seemingly unrelated things in order to end up with Firefox, those things might end up getting

used for something else you're also building. Most people just download binary distributions, anyway.

What I haven't seen so far in this discussion is a workable real-world proposal for what to do differently other than a few

incidental details about particular software. The complaint seems to be about the big-picture approach, but I suspect EricRaymond is right that no cathedral can be large enough to encompass a whole industry. And that's effectively what the FreeBSD

ports system contains.

Frank Greco | Tue, 21 Aug 2012 14:48:55 UTC

Brilliant article. I've been thinking of the same stuff since the Cathedral book. It seems in other endevours there exists

quick spike, multi-creator mediocrities and high-priest design masterpieces. From building architecture to music to paintings

to TV shows to finance, et al. Apparently software is no different.

Poul-Henning Kamp | Tue, 21 Aug 2012 17:58:42 UTC

@David MacKenzie:

I don't dispute that autoconf was a pragmatic solution to the madness of UNIX vendors back in the 1980'ies and 1990'ies. It

certainly saved time for me back then.

But today it no longer serves a useful purpose, other than add incomprehensible onamentation and woodoo incantations to open


14/24

source software, for no credible gain in overall quality or portability.

I am certainly not looking for "The One Unix To Rule Them All", but I would really love to see some intelligent standardization

fill the worst pot-holes, so we don't have to waste so much time while autoconf checks for and libtool looks for 26 fortran

compilers, twice.

The one thing cathedrals is better at than bazaars, is throwing junk overboard, that's probably the main reason why I wish we

had more cathedral and less bazaar.

As for M4: Apology accepted, but can we please loose it now ?

Marty Fouts | Tue, 21 Aug 2012 19:09:26 UTC

Mr Kamp is right, but it is clear from the responses that he is not well understood.

I have programmed professionally since 1972. At that time the computer industry was already twenty years old, but those of us

who entered it studied its history and learned from it. Sometime between 1980 and 1990 that willingness to learn lessons from

past projects disappeared, while a sense of hubris entered the industry, epitomized by the conversion of our job titles from

"programmer" to "engineer" that took root in the 1980s.

Now, the computer industry is always twenty years old, and those who come to the profession no longer learn the lessons of the

past. Recently, an algorithms "expert" I interviewed, admitted to having no idea who Don Knuth was, or why they should be

aware of his work. Such interviews are becoming more, not less, common. I can point at hundreds of such examples of dumbing-

down in the industry: OS kernel

a generation lost in the bazaar - acm queue

Documents