Posted by: drew | June 15, 2006

Broken Windows Theory

Anzev at Slashdot passes along an interesting blog entry written by a Microsoft Team Manager about why the Windows Vista release date really slipped to 2007.

Both the article and the ensueing comments make for an interesting read. The general consensus seems to be that the problems the author describes are common to most, if not every software development company to a certain degree. I'd say this concurs with my admittedly brief experience as well.

The usual suspects:

  • Windows code is too complicated. It’s not the components themselves, it’s their interdependencies. An architectural diagram of Windows would suggest there are more than 50 dependency layers (never mind that there also exist circular dependencies). After working in Windows for five years, you understand only, say, two of them. Add to this the fact that building Windows on a dual-proc dev box takes nearly 24 hours, and you’ll be slow enough to drive Miss Daisy.
  • Windows process has gone thermonuclear. Imagine each little email you send asking someone else to fill out a spreadsheet, comment on a report, sign off on a decision – is a little neutron shooting about in space. Your innocent-seeming little neutron now causes your heretofore mostly-harmless neighbors to release neutrons of their own. Now imagine there are 9000 of you, all jammed into a tight little space called Redmond. It’s Windows Gone Thermonuclear, a phenomenon by which process engenders further process, eventually becoming a self-sustaining buzz of fervent destructive activity.

More below

Cultered to Slip

After months of hearing of how a certain influential team in Windows was going to cause the Vista release to slip, I, full of abstract self-righteous misgivings as a stockholder, had at last the chance to speak with two of the team’s key managers, asking them how they could be so, please-excuse-the-term, I-don’t-mean-its-value-laden-connotation, ignorant as to proper estimation of software schedules. Turns out they’re actually great project managers. They knew months in advance that the schedule would never work. So they told their VP. And he, possibly influenced by one too many instances where engineering re-routes power to the warp core, thus completing the heretofore impossible six-hour task in a mere three, summarily sent the managers back to “figure out how to make it work.” The managers re-estimated, nipped and tucked, liposuctioned, did everything short of a lobotomy – and still did not have a schedule that fit. The VP was not pleased. “You’re smart people. Find a way!” This went back and forth for weeks, whereupon the intrepid managers finally understood how to get past the dilemma. They simply stopped telling the truth. “Sure, everything fits. We cut and cut, and here we are. Vista by August or bust. You got it, boss.”

You know that the cancer is running all the way up the food chain when those below are afraid to tell the truth.

Micromanagement and Arbitrary Design Decisions

Micromanagement, though not pervasive, is nevertheless evident. Senior vice presidents sometimes review UI designs of individual features, a nod to Steve Jobs that would in better days have betokened a true honor but for its randomizing effects. Give me a cathedral, give me a bazaar – really, either would be great. Just not this middle world in which some decisions are made freely while others are made by edict, with no apparent logic separating each from the other but the seeming curiosity of someone in charge.

I've personally encountered this problem and can attest to the counterproductive effect it can have.

There was one particular reader comment that caught my eye further down the page:

RetiredOldCoot said:

Based on my observation of Microsoft over the years, having used its products, having competed against it, having read your assesment of Why Vista is Late, I'd say you got somethings right, somethings wrong and missed some more. I'd like to address some of the things you missed.

First is the complete absense of business ethics, as evidenced by the dollar value of settlements and judgements against Microsoft. How does this affect the ship date? I suppose every employee recognizes, at least unconsciously if not openly, that his employer will screw him as well as competitors to win at all costs. I would further imagine that this weight adversely affects productivity by making him practice defensive measures to protect his job, be a YesMan and to stay out of the way of thrown chairs. From those I know currently working at MS, all have "sold their souls" to the Beast of Redmond. All would rather be somewhere else, doing something else, but the market won't support it right now, so they sold out. They have to make a living.

Read about Bill Gates, Steve Balmer in Stout, "The Sociopath Next Door". That's what got Windows into trouble.

Next is that Windows has achieved the Bathtub Effect. That is: the level of complexity of the product, the level of coupling of components in particular has gotten to the point that nobody can understand enough of it to be able to work on it safely. Safely is defined as being able to change it without breaking it.

Windows achieved its level of complexity be going on a huge API grab in its early years. Every week you'd see some new announcement from MS for an API to do this, that or the other thing. Shortly thereafter, you'd see rants and raves from prospective users of those APIs often saying that the API sucks because … Often they were right. I can also imagine, and I believe others will attest, that the implementation of those APIs was undertaken without regard to security, usability, or any number of other factors influencing its long term stability. The job was to do the API Landgrab first.

Decisions like making Internet Explorer part of the operating system itself were done, I would imagine, as part of the Kill Netscape strategy. Without thinking through, or maybe it was thought through and the risk taken anyway, integrating IE into the OS caused almost complete failure of security to Windows in its entirety. I can only imagine the herculean effort required to extract IE from the OS. … imagine! A web browser as a critical component of an operating system, complete with unrestrained memory access to everything in the box! … what were they thinking?

Finally, ASP. If IE were not enough on its own, ASP was the straw that broke the camel's back. By giving the BadActors of the world a tool guaranteed to cause remote scripting compromises (remember, IE is part of the OS withe carte blanche to write whereever it wants) MS sealed its fate. Windows is unmaintainable, unsustainable, unrecoverable. It's history.

The ultimate root cause of Why Vista is Late is Greed. Microsoft is truly a world class phenomenon. From nothing to one of the biggest ($) companies in the world in a little over 20 years. Would I work there? Not on a bet. Would I buy its products? Not on a bet. Will I cheer when it falls? You bet.

Interesting perspective and the Bathtub effect reference is interesting.

I wouldn't agree that everyone in Redmond has sold their soul, but the old axioms that "Money is the root of all evil" and "Absolute power corrupts absolutely" will tend to apply to organizations which monopolize a market and rake in metric #$%@-tons of cash. The management layers multiply exponentially, managers fall in love with their processes, and the truth is a casualty of the money making machine.

Microsoft has forgotten that their OS is an OS and have created a monster that tries to be everything to everyone in their obsession to eliminate any and all competition. The fact is, however, that Windows is the standard OS and IE is the browsing standard for most of the developed world at this point.

Competitors like Firefox are chipping away at the megalith, but as long as the large PC manufacturers continue to ship with Windows, as long as computer programs and games are written for Windows, and as long as hardware drivers are written for Windows, the OS and IE will continue to monopolize the market.

Trust me, I'd love to switch to Linux distros like Ubuntu Dapper and Open Office. However, given that I'm a gamer I'm pretty much stuck with Windows unless I want to run two systems or reboot into Windows to play WoW.

Having said that, I'm looking forward to giving Windows Vista Beta 2 a spin. While that may sound hypocritical, it's not that, it's just that in addition to being an information-addict, I'm also a technology-addict who voraciously devours any software or hardware that my budget allows. In this case it's free.

[edit: the author once again pulled down the blog entry from the MSDN site. Thanks to Google cache you can find it again. I'm posting it in it's entirety below. I understand that the author pulled the entry because it appeared to fuel MS detractors.

My opinion is that the problems he describes are inherent in the system, not just at MS. It's a "no shit Sherlock" kind of thing, but I feel that the article is a valuable resource for software developers and managers to be able to read. Dunno if I'll get flak for reposting it, but as it's available via Google cache I don't see how it really matters.

Hey, at least I left the annoying bolded text in…]

Broken Windows Theory

Vista. The term stirs the imagination to conceive of beautiful possibilities just around the corner. And “just around the corner” is what Windows Vista has been, and has remained, for the past two years. In this time, Vista has suffered a series of high-profile delays, including most recently the announcement that it would be delayed until 2007. The largest software project in mankind’s history now threatens to also be the longest.

 

Admittedly, this essay would be easier written for Slashdot, where taut lines divide the world crisply into black and white. “Vista is a bloated piece of crap,” my furry little penguine would opine, “written by the bumbling serfs of an evil capitalistic megalomaniac.” But that’d be dead wrong. The truth is far more nuanced than that. Deeper than that. More subtle than that.

 

I managed developer teams in Windows for five years, and have only begun to reflect on the experience now that I have recently switched teams. Through a series of conversations with other leaders that have similarly left The Collective, several root causes have emerged as lasting characterizations of what’s really wrong in The Empire.

 

Useless Trivia Sidebar: Broken Windows Theory

 

The original broken windows theory, first coined by Wilson and Kelling, describes the purported phenomenon whereby an abandoned warehouse with no broken windows is mostly left alone, but as soon as one window is broken, it acts as an open invitation to passers-by that it’s open-season for rock-throwing.

 

This was generally accepted for many years as being true, but is recently coming under fire from different angles. We won’t delve into those here, since we mostly commandeered the phrase because it sounded good, not because it actually has anything at all to do with our subject matter.

 

 

The Usual Suspects

 

Ask any developer in Windows why Vista is plagued by delays, and they’ll say that the code is way too complicated, and that the pace of coding has been tremendously slowed down by overbearing process. These claims have already been covered in other popular literature. A quick recap for those of you just joining the broadcast:

 

  • Windows code is too complicated. It’s not the components themselves, it’s their interdependencies. An architectural diagram of Windows would suggest there are more than 50 dependency layers (never mind that there also exist circular dependencies). After working in Windows for five years, you understand only, say, two of them. Add to this the fact that building Windows on a dual-proc dev box takes nearly 24 hours, and you’ll be slow enough to drive Miss Daisy.
  • Windows process has gone thermonuclear. Imagine each little email you send asking someone else to fill out a spreadsheet, comment on a report, sign off on a decision – is a little neutron shooting about in space. Your innocent-seeming little neutron now causes your heretofore mostly-harmless neighbors to release neutrons of their own. Now imagine there are 9000 of you, all jammed into a tight little space called Redmond. It’s Windows Gone Thermonuclear, a phenomenon by which process engenders further process, eventually becoming a self-sustaining buzz of fervent destructive activity.

 

Let’s see if, quantitatively, there’s any truth to the perception that the code velocity (net lines shipped per developer-year) of Windows has slowed, or is slow relative to the industry. Vista is said to have over 50 million lines of code, whereas XP was said to have around 40 million. There are about two thousand software developers in Windows today. Assuming there are 5 years between when XP shipped and when Vista ships, those quick on the draw with calculators will discover that, on average, the typical Windows developer has produced one thousand new lines of shipped code per year during Vista. Only a thousand lines a year. (Yes, developers don’t just write new code, they also fix old code. Yes, some of those Windows developers were partly busy shipping 64-bit XP. Yes, many of them also worked on hotfixes. Work with me here.)

 

Lest those of you who wrote 5,000 lines of code last weekend pass a kidney stone at the thought of Windows developers writing only a thousand lines of code a year, realize that the average software developer in the US only produces around (brace yourself) 6200 lines a year. So Windows is in bad shape – but only by a constant, not by an order of magnitude. And if it makes you feel any better, realize that the average US developer has fallen in KLOC productivity since 1999, when they produced about 9000 lines a year. So Windows isn’t alone in this.

 

The oft-cited, oft-watercooler-discussed dual phenomenon of Windows code complexity and Windows process burden seem to have dramatically affected its overall code velocity. But code can be simplified and re-architected (and is indeed being done so by a collection of veteran architects in Windows, none of whom, incidentally, look anything like Colonel Sanders). Process can be streamlined where inefficient, eliminated where unnecessary.

 

But that’s not where it ends. There are deeper causes of Windows’ propensity to slippage.

 

 

Cultured to Slip

 

Deep in the bowels of Windows, there remains the whiff of a bygone culture of belittlement and aggression. Windows can be a scary place to tell the truth.

 

When a vice president in Windows asks you whether your team will ship on time, they might well have asked you whether they look fat in their new Armani suit. The answer to the question is deeply meaningful to them. It’s certainly true in some sense that they genuinely want to know. But in a very important other sense, in a sense that you’ll come to regret night after night if you get it wrong, there’s really only one answer you can give.

 

After months of hearing of how a certain influential team in Windows was going to cause the Vista release to slip, I, full of abstract self-righteous misgivings as a stockholder, had at last the chance to speak with two of the team’s key managers, asking them how they could be so, please-excuse-the-term, I-don’t-mean-its-value-laden-connotation, ignorant as to proper estimation of software schedules. Turns out they’re actually great project managers. They knew months in advance that the schedule would never work. So they told their VP. And he, possibly influenced by one too many instances where engineering re-routes power to the warp core, thus completing the heretofore impossible six-hour task in a mere three, summarily sent the managers back to “figure out how to make it work.” The managers re-estimated, nipped and tucked, liposuctioned, did everything short of a lobotomy – and still did not have a schedule that fit. The VP was not pleased. “You’re smart people. Find a way!” This went back and forth for weeks, whereupon the intrepid managers finally understood how to get past the dilemma. They simply stopped telling the truth. “Sure, everything fits. We cut and cut, and here we are. Vista by August or bust. You got it, boss.”

 

Every once in a while, Truth still pipes up in meetings. When this happens, more often than not, Truth is simply bent over an authoritative knee and soundly spanked into silence.

 

 

The Joy of Cooking

 

Bundled with a tendency towards truth-intolerance, Windows also sometimes struggles with poor organizational decision-making. Good news is that the senior leaders already know this and have been taking active steps to change the situation.

 

There are too many cooks in the kitchen. Too many vice presidents, in reporting structures too narrow. When I was in Windows, I reported to Alec, who reported to Peter, to Bill, Rick, Will, Jim, Steve, and Bill. Remember that there were two layers of people under me as well, making a total path depth of 11 people from Bill Gates down to any developer on my team.

 

This isn’t necessarily bad, except sometimes the cooks flash-mob one corner of the kitchen. I once sat in a schedule review meeting with at least six VPs and ten general managers. When that many people have a say, things get confusing. Not to mention, since so many bosses are in the room, there are often negotiations between project managers prior to such meetings to make sure that no one ends up looking bad. “Bob, I’m giving you a heads-up that I’m going to say that your team’s component, which we depend on, was late.” “That’s fine, Sandy, but please be clear that the unforeseen delays were caused by a third party, not my team.”

 

Micromanagement, though not pervasive, is nevertheless evident. Senior vice presidents sometimes review UI designs of individual features, a nod to Steve Jobs that would in better days have betokened a true honor but for its randomizing effects. Give me a cathedral, give me a bazaar – really, either would be great. Just not this middle world in which some decisions are made freely while others are made by edict, with no apparent logic separating each from the other but the seeming curiosity of someone in charge.

 

In general, Windows suffers from a proclivity for action control, not results control. Instead of clearly stating desired outcomes, there’s a penchant for telling people exactly what steps they must take. It’s creating a generation of McDevs, few of whom enjoy the monotony. (For more on action control vs. results control, read Kenneth Merchant’s seminal work on the subject – all $150 of it, apparently).

 

 

Uncontrolled? Or Uncontrollable?

 

We shouldn’t forget despite all this that Windows Vista remains the largest concerted software project in human history. The types of software management issues being dealt with by Windows leaders are hard problems, problems that no other company has solved successfully. The solutions to these challenges are certainly not trivial.

 

An interesting question, however, is whether or not Windows Vista ever had a chance to ship on time to begin with. Is Vista merely uncontrolled? Or is it fundamentally uncontrollable? There is a critical difference.

 

It’s rumored that VPs in Windows were offered big bonuses contingent on shipping Vista by the much-publicized August 2006 date. Chris Jones even declared in writing that he wouldn't take a bonus if Vista slips past August. If this is true, if folks like Brian Valentine held division-wide meetings where August 2006 was declared as the drop-dead ship date, if general managers were consistently told of the fiscal importance of hitting August, if everyone down to individual developers was told to sign on the dotted line to commit to the date, and to speak up if they had any doubts of hitting it – mind you, every last one of those things happened – and yet, and yet, the August date was slipped, one has to wonder whether it was merely illusory, given the collective failure of such unified human will, that Vista was ever controllable in the first place.

 

Are Vista-scale software projects essentially uncontrollable by nature? Or has Microsoft been beset by one too many broken windows? Talk amongst yourselves.

Advertisements

Responses

  1. thanks for posting some excerpts – it looks like the original MSDN blog post has been removed.

  2. well, looks like Google cached it:

    http://tinyurl.com/otdal

  3. Excellent. Thanks for the google cache copy. I’m going to post the entire blog entry in my post.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: