Monday, April 17, 2006

Estimating Projects: A History of Failure

When I lived in Crested Butte, Colorado, I sometimes hiked across the mountains to Aspen. There were five hiking trails that I knew of, all of them former toll roads from the old gold-mining days when the only way to reach Aspen was through Crested Butte. As I struggled to breathe going up these 13,000+ foot passes, I often wondered why there were five toll roads. But the time I'd hiked all five, I knew the answer. There were five toll roads because none of them were very good.

Consultants are frequently given the task of estimating, or helping to estimate, projects. To start with, they're asked to estimate how long it's going to take to solve their client's problem—which is usually undefined. Although there have been billions of words written on dozens of methods of estimating projects, most of them are, in my opinion, useless or nonsensical. Or both. Like toll roads to Aspen, there are so many because none of them are very good.

Why do I say they're not very good? Well, why were the toll roads to Apsen not very good? Mostly, I think, because if you're walking from Crested Butte to Aspen, you have to cross an extremely high barrier. We might want to be able to hike over a 13,000 foot pass without breathing hard, but it's impossible (at least for most of us).

And that's the primary reason that all these estimating methods are not very good is exactly the same: they are trying to do something that's impossible—predict the future. Sure, we'd all like to do this, but we can't.

People have been trying to predict the future as long as people have been around, so I thought it would be useful to read about methods from the past, so perhaps you could relate them to some of the failures of today. The following post is from dgc, a friend who has worked for a long time in the software industry, mostly as a contractor and consultant—long enough not to take himself too seriously, a good lesson for all of us consultants.

Some Estimating Methods

The fabulous Wizard of Oz
Retired from his racket because,
What with up-to-date science,
To most of his clients
He wasn’t the Wizard he was.
- Anonymous

Since estimation seems to have much in common with the mantic arts (i.e., the arts of divination), I turn to the writings of an expert in that field: Dreamas Gentilharte Cheynelokk, a nearly thousand-year-old wizard who writes about the mysteries of CMMI (Comprehensive Magistrate of Magical Integrity) certification.

For those unfamiliar with this form of CMMI, Dreamas briefly describes its five levels of magical competence as follows (the translation of this text is by Dr. Devin G. Kettenschloss in "Paduan Vellums, Volume VI, Item lvii- A Translation" following the seminal work of the late Dr. Blanche V. Foote-Falles):

"(1) Erotic [1]: animalistic, sensual, like kindling consumed in a flame, uncontrollable, never twice the same, exciting

(2) Telestic [2]: ritualistic, repeated, like an unbending oak planted in the earth, formal, delimited and limited capacities, routine

(3) Mantic [3]: prophetic, governable, like a ship sailing home on uncharted seas, divinatory, observed and proven by trial, directed

(4) Poetic: rhapsodic, meted, like a falcon riding steady the rolling air [4], quantifiable, sound and proportioned in structure, balanced

(5) Ecstatic [5]: euphoric, evolving, like the shining cosmic quintessence [6], heavenly transcendence married to earthly immanence, sustained."

Dreamas goes on to describe hundreds of mantic techniques useful across all aspects of software development. But for the topic of estimation, the following description of the art of "Alectryomancy" seems relevant when speaking about making time/schedule estimates for a project, especially since the technique used differs based on the CMMI level of the organization (again quoting from the previously cited translation).

"Alectryomancy [7]: Divination by the Actions of Poultry Pecking at Grain

So much software development goes awry for lack of sufficient numbers of dedicated top-breed poultry!

For those functioning at the CMMI Level 1 (Erotic), any hen or rooster of any kind will do. Simply put out a handful of grain and let the chicken loose for one minute. At the end of one minute, guess how many grains the chicken ate and multiply by 3. This calculation will yield the needed number of hours, days, weeks, months or years it will take to complete the project.

Those groups at CMMI Level 2 (Telestic) laugh at that naiveté. The wizards at this level realize it is not that easy. First, more chickens are needed, at least three but never more than nine. Second, before being let loose on the grain the chickens properly motivated with loud chanting and the waving of gleaming, sharpened axes. Third, the grain for each bird is placed in a line running east to west and the first grain eaten by each bird is the estimate. That is, if the 4th grain from the east is the first eaten by a hen, then the estimate from that hen is 4. Finally, after all chickens are done, the lowest estimate is picked with any multiplicative factor determined by the wizards using Capnomancy (divination by smoke) or Catoptromancy (divination by mirrors placed under water).

But groups at CMMI Level 3 (Mantic) realize estimation is a daring, daunting and dangerous undertaking. So in their practice, they only use roosters. Each rooster is allowed to make its estimate by pecking a grain in its own sealed off arena. A variety of techniques for deriving the numerical estimate can be employed (counting grains, counting volume, which grain is first picked, which is last picked, etc.) with skilled wizards selecting in advance the appropriate technique by means of Gyromancy (by whirling until dizzy). After each rooster has produced his estimate, all the roosters are placed in one arena for a cockfight {SIDE NOTE FROM 'dgc': I realize the cruelty of this technique may offend some people's sensibilities, but that is what the original text says} with the winning cock's estimate being adopted [8] and everyone being held to that estimate under threat of the hatchet.

Groups at CMMI Level 4 (Poetic) with more refined sensibilities bemoan the waste of possibly good estimators by the Mantic level wizards as well as the blatant discrimination against hens. Therefore, they eschew the wastefulness of cockfights and use both hens and roosters; but they also carefully record and plot age, breed, and accuracy of auguries over time. In addition, no matter what anyone says about chickens and lips, they firmly believe in the importance of using Labiomancy (divination by lip reading) as part of their process.

Finally, wizards at CMMI Level 5 (Ecstatic) realize that the work of estimation is never done. So they use the greatest number of poultry and have them estimating and reestimating often using dozens of techniques and inventing new ones whenever needed. At this level, the chicken coop and yard is a constant bustle of activity yielding untold numbers of eggs. After all, in the end, it is all about the real egg production."


Footnotes (From the translation)

[1] From the Greek meaning "sexual love" as associated with Eros, who is not the cute cupid of modern times but is rather the passionate young god, son of Venus, as best described in Apuleius' story of Psyche and Eros in "The Golden Ass" (Chapters 7-9).

[2] Very rare in modern English. Derives from the Greek "telestes," meaning a "hierophant or expounder of sacred mysteries."

[3] The Romans, especially Cicero in his treatise "De Divinatione," clearly preferred the word "divination," which derives from the Latin "divinus" and indicates a deific origin, to Plato's "mantic," which is equivalent to the Latin word "furor" meaning "madness, lunacy, or raving."

[4] Echoing Gerard Manly Hopkins' "The Windhover" (http://www.bartleby.com/122/12.html).

[5] There is a delicious irony in the CMMI master's choice of the word "ecstatic" for their highest and ideal level. From the writings of mystics throughout the ages, ecstasy is often associated with displays of frenzy and agitation bearing strong sexual overtones. See, for example, the writings of the Spanish mystics St. Teresa of Avila ("Interior Castles") or St. John of the Cross ("Dark Night of the Soul"). Given these strong sexual associations, it is possible it might be somewhat difficult to see how the Ecstatic level differs from the lowest level, Erotic.

[6] Here, Dreamas means the ancient and medieval concept of the perfect material of which the stars were made: an exact blend of all other types of matter (fire, earth, water, and air).

[7] An alternate spelling is "Alectromancy."

[8] Note how in many ways this is similar to the Delphi technique of estimation some speak of in software engineering.

4 comments:

Unknown said...

I have my say here: Software Project Estimation Techniques For Different CMMI Levels

Morbridae said...

ok, so I'm at CMMI 1 level (Erotic estimation method). But, since my boss normally says "Oh, no! That's unacceptable. Divide it by 20. It doesn't matter that we don't have 20 developers!", I believe that the rooster fight method is more appropriate... even if my rooster has its legs tied and its eyes blinded!

willjud2009 said...

Yeah, Hi Gerry - its nice to hear from you. When I first started consulting, I read your book, 'The Secrets of Consulting'. That was, what - almost 20 years ago. I especially liked your rasberry jam theory, 'The wider you spread it, the thinner it gets!

I'll take your dialogue a little farther...the truth of the matter is, for the most part, deadlines and schedules are determined without project estimates, and the IT team is expected to meet ‘Time to Market’ dates, or when management thinks a project should be delivered. I’m actually in the midst of writing a book myself – keep your eyes open for ‘The Problems with IT - and Recommended Solutions’
Gerry, you’re a nice guy….but I’m going to be a little more blunt – hey kids – there is no Santa Claus. Project Estimation is more of an art than a science. Furthermore, Project Estimation is but one of the branches in your Project Management suite. What I discuss in my book is not just that project estimates are poor, but why they are often poor. For example, if your team is comprised of a Business Requirements team, a Data Team (DAs and/or DBAs), do you have a ‘symbiotic trio’ or a ‘dysfunctional family’?
But let’s talk about project estimation. The biggest problem with an estimate is threefold:

a) The team is requested to provide an estimate for something that they've never done before

b) The practice of re-estimating (once the Business Analysis and initial Design is done) is never practiced

c) There's a translation process that has to done to numbers provided by the

Allow me to discuss these issues separately—

a) When working on a project that involves an area of the business that the team is not intimately familiar, the team has to be honest with management. Let them know that the best you can do is provide a ‘framework timeframe’ for the first release of the project. Let them know that what exactly will be in a specific version of release of the project will be determined well in advance of the deadline. Just don’t be overly optimistic and promise things that you’re not sure you can deliver – not in the initial estimation phase. If your feet are put to the fire, talk to the Developers and figure out the things that are easiest to do (and make sure that they’ve at least done something similar) within the context of the target deliverables…only promise the items that are ‘slam-dunks’

b) Make sure your management is aware that the Business Area Analysis and initial System Design is intended to provide clarity pertaining to what is needed and how to go about meeting your requirements. If you perform your Business Area Analysis properly, you will have a good Data Architect and a good Business Analyst (or people that can do both). As the Business Area Analysis documentation (sometimes called Business Requirements Document) is being completed, the DA should be refining his Data Model to meet the requirements and should have a good handle on the data. Moreover, it is PARAMOUNT that the BA and the DA used the same terms, they should collectively be forging a Business Lexicon that is common to all. Once the model, BA doc, and initial draft of the System specs is published, provide management with a ‘re-estimation’ of the project. If your estimate has now changed, let your management know this ASAP. If your management balks, then let them know that you can still serve the meal, but it might not be ‘soup-to-nuts’…

c) Now I’m going to let all of you in on a secret….Over the last 20 years I’ve discovered something – call it ‘Will Estimating Corollary”. Developers speak in ‘dog-years’. If a Developer says it will take an hour to develop something, that doesn’t equate to it takes an hour to get it done. Basically, there are two estimate notions, the short task estimate (1-3 hours) , and the long task estimate (1 or more units of 8 hrs each….e.g., ‘5 days’). The Translation that you can probably feel safe to use as a guide are as follows-

Developer Provided Estimates and what you should put on your Project Plan:
1-3 Hours – anytime a developer gives you an estimate for something that they think will take more than an hour (an up to 3 hrs), you should put an estimate of 1 day for that task
4-8 Hour estimate  you might as well consider these as a ‘Day Estimate’
‘Day Estimates’ – I take each task that the Developer says will take 8 hrs and multiply is by 2.

Once I complete my plan, I provide the Developers with a the anticipated dates. I don’t let them know which tasks have been stretched from a time perspective.

Keep in mind that you should always be updating your plan. In addition to the uncertainties that I’ve mentioned above, there will be unplanned meetings, issues that will need to be resolved, and other stop gaps. If you encounter unanticipated external constraints, always make sure that you update your project plan. Remember, there should be no such thing as ‘make the time lost up by….’ – lost time is lost

Udi Dahan said...

I employ the "2 day rule" with the teams I consult.

If the estimate for any task is greater than 2 days, the task has to be broken down to subtasks. If that requires some architecture/design/business analysis work - so be it.

Quite frankly, any time a developer estimates something at 5 days or a week, they don't have a clue how to even begin let alone when it's going to end. The thing is nobody ever told them that "I don't know" is a valid answer when someone asks you for an estimate. They see everyone else in the room providing numeric estimates so they think they have to as well.

The "2 day rule" works to undo the damage of those misunderstandings.

You may also like this post on my blog:

Estimate invidually - fail globally