How to Prevent IT Project Failures

The Issue

For the past year I have been doing research on the causes of IT project failures and on how to prevent them. Here is a summary of what I have leaned. More information is available on my web site: www.ghco.biz.

More than forty years after the advent of third generation computing, the IT profession overall still doesn’t know how to execute IT projects predictably and successfully. 70% of all IT projects fail, a number that has been documented in numerous surveys conducted by many organizations in many countries. This number is startling, and a bit frightening when you recall that IT expenditures constitute more than half of all capital expenditures by American businesses.

Maybe we are too prone to use sports as metaphors for business. If a baseball player consistently bats .300, he is likely to get a multi-year, multi-million dollar contract. Why shouldn’t a CIO who bats .300 (which is the same as saying that 70% of his projects fail) also get a multi-year, multi-million dollar contract?

Several possible explanations occur to me. One is that the baseball player’s career is very short and he should be compensated for a lifetime during the few years he plays. On the other hand, is a CIO’s career really any longer? Current statistics say that it is not. Or maybe it is because anyone interested can see for himself how good a job the batter is doing while it is almost impossible for a spectator to see how good a job the CIO is doing. Or maybe because the ball player is more fun to watch. Resolving this compensation anomaly would be interesting, but we have a better chance of increasing the CIO’s batting average, so let’s look at how we could to that.

The Costs

The costs of these failures are substantial. Here is an analysis of a hypothetical company based on industry averages.

Revenue	$100 MM
IT budget @ 5% of revenue	$5 MM
Application development budget @ 20% of IT budget	$1 MM
Total budgets of failed projects @70%	$700,000
Budget overruns @ 50%	$350,000
Average project length	2 years
Average schedule over-run of project @30%	0.6 year
Average planned ROI of projects	25%/year
Foregone profits of late projects @ 0.6 x 0.25 x $700,000	$105,000
Total annual losses because of budget and schedule overruns	$455,000

Looked at another way, preventing project failures can increase the productivity of a system development department by 45%.

Root Cause Analysis

What does it mean to say an IT project is a failure? A project is a failure if the relevant business executives say it is. Here is a reasonably comprehensive list of things that might cause a business executive to make that judgment. Call them indicators of failure.

1. End product of the project does not meet real business need.
2. End product of the project does not support organizational strategy.
3. Project implementation was chaotic and disruptive.
4. Project was not implemented or was soon abandoned.
5. Degree of project success or failure is unclear.
6. Success or failure of future projects is unpredictable.
7. Project exceeded budget.
8. Project was finished late or not at all.
9. Final product lacks originally planned functionality.

Items 1-6 above have to do with the business effects of the project; iems 7-9 are more focused on the execution of the project by IT.

My root cause analysis of IT project failures identified these seven generic root causes:

• Incomplete project planning and evaluation.
• Project plan misses non-technical issues.
• Roles and responsibilities of users, senior. management, and IT are not well understood and agreed to by all parties.
• Bad or non-existent communications among interested parties.
• Inadequate project governance.
• Lack of post audit procedures and project archive
• Inconsistent application of good practices.
  

In the course of the root cause analysis, several things became clear.

• Most of the project failures – but certainly not all of them – have non-technical causes.
• Many of these causes exist outside of the IT function and hence beyond the direct control of IT.
• The relationships between the indicators of failure and the root causes are many and complex. For example, “end product does not meet real business need” might be caused by bad project planning, or by inadequate project governance. Similarly, each root cause may influence more that one bad result. For example, inadequate project governance may lead to budget overruns, schedule overruns and/or chaotic or disruptive implementation.
• Although it may be expedient to attack the root causes one at a time, you should remember these interactions and not be surprised when fixing one root cause does not cure the specific problem you are addressing.

Preventing Project Failures Before They Occur

A program to prevent these failures should include these elements.

• A careful analysis of the “as is” situation. This analysis will almost certainly show that some of the generic root causes have already been taken care of, and it may also identify company specific root causes not included in the generic list.
• An educational program for all the relevant constituencies, including senior corporate management and senior business unit management as well as the IT community.
• A business process redesign of the project management function to address the needs identified by comparing the “as is” situation with the results of the root cause analysis.
• Institutionalizing the new business process by coaching the project team through at least one project.

This may sound like a complex and onerous process, but it really is not. It will be a little awkward for the first project, but the payoff will be substantial.

<><><><>

Innovation and Chaos

One of the basic ideas we will pursue in this blog is that many diverse kinds of organizations have characteristics that are broadly similar to one another, and that these similarities can provide useful guides to analysis and change. Our approach is rather like the self-similarity concept in fractal geometry. Or, for those of you who prefer more established ideas, it is like general systems theory, wherein a system is characterized by its relationships with other systems rather than by the way it functions internally. Things that appear to be quite different from one another often have similar structures ‘under the hood’. Our goal is to exploit these similarities as we seek to improve the organizations that are the substance of our daily lives: political entities, private organizations, the IT organizations which are components of these larger entities.

We in the IT world have been trying for decades to create comprehensive information systems: systems intended to meet all of the information needs of an organization, present and future. Call it a global MIS (management information system). Every one of these attempts has failed, despite the valiant efforts of talented and dedicated IT professionals and the expenditures of huge sums of money. This quest is intrinsically futile. We will never succeed and we should stop wasting our resources trying. This is no loss. In fact, having such a system is basically a bad idea because it would be so hard to change that it would never be flexible enough to respond to changing organizational needs.

For a time some people believed that ERP systems were the answer. But the vision of ERP as a global MIS has vanished as the vendors have moved from offering integrated solutions to offering individual modules that are (presumably) easy to interconnect, using such infrastructure tools as CORBA and dotNet. Another MIS panacea will be proposed, and it too will fail. It may be web-enabled software, although I doubt it; fresh memories of ERP will prevent a new fiasco for at least a couple of years.

I wrote a white paper about ERP (enterprise resource planning) a couple of years ago, which you can find at http://www.ghco.biz/silverbullet.htm. It reviews the history of attempts to build the global MIS. In it, I conclude that while having a single, integrated information system meeting all of the information needs of an entire organization (ERP is one version) is an attractive idea to executives and a seductive one to IT professionals, no one has ever succeeded in building one, and no one ever will. Organizational and human constraints will prevent it even if all the technical difficulties are overcome.

My instinct and experience told me a long time ago that global MIS solutions are not the right thing to attempt, but I wasn’t clear about why until I read “The Competition Solution” by Paul A. London (ISBN 0-8447-4204-X). Mr. London, a sociologist rather than an economist, examines the history of monopolies and oligopolies as they have disintegrated in the United States over the past few decades. Some of these disintegrations have been the direct result of deregulation: airlines, telecommunications, certain segments of financial service, electrical power generation. Others have been the result of antitrust decisions by the courts. But the progression from constrained markets to free markets has the same in every industry.

Start with a monopoly or oligopoly, either government imposed or created by private interests.
End the monopoly by deregulation or court order.
Chaos is predicted by the opponents of change, who are usually the beneficiaries of the current system.
Some chaos actually occurs, but it quickly decreases to a level much less than predicted.
A new equilibrium is reached with some of the chaos remaining but with substantially lower costs to consumers.
The nay-sayers point to the residual chaos as evidence that deregulation failed, but the consumers know better.

In every case that London studied, the competitive environments have generated enough savings and innovation to pay the costs of the residual chaos and also decrease prices to the consumer.

Look at the progression described above from the bottom up – from a competitive environment to a monopolistic one. You begin with some chaos and high costs. You think that consolidation will provide enough economies of scale to make up for the loss of focus and effectiveness inevitably associated with centralized operations. But if you go too far toward a single system, you end up with monopoly and high costs. (This is not to say that consolidation is always a bad idea; rather, it says don’t go to the extreme of trying to combine everything. Remember Pareto: 80% of the benefits come from 20% of the effort.)

Translated into the IT context of trying to meet the information needs of an entire enterprise, this means that the competitive marketplace for system modules and capabilities creates enough savings to pay the costs of dealing with the residual chaos of building module interfaces and some extra data transmission, and still yields net cost savings.

The residual chaos is the primary source of innovation. The challenge for IT management is to create and manage this residual chaos in an IT department and reap the innovation that it generates. IT professionals – like most other professionals – abhor chaos. It’s not in our genes. We work with machines and algorithms that are completely deterministic. (There are cynics who do not believe this, but that is a topic for another day.)

The challenge to the CIO is this: how do you promote the chaos that will generate the innovations you will need to prosper or even to survive? How do you organize and manage it? How do you motivate and reward the corporate mavericks you will need to staff it? How do you keep the organizational immune system from rejecting it?

Some companies have organized research groups within IT, often called new technology groups. Their life-span is usually two budget cycles, by which time cost pressures and long lead times catch up with them. Two things seem to work when done in a very low key way. One is to encourage developers to investigate new technologies as a routine part of the design process. The other is to identify a few people who are passionately devoted to technology and innovation and give them part time assignments to watch, learn, and innovate. These activities are best hidden from prying eyes.

* * *

Another book shows that this insight – that global systems can’t work – applies in the broader sphere of international relations. In “The Case for Sovereignty” by Jeremy A. Rabkin (ISBN 0-8447-4183-3), Prof. Rabkin examines the concept of national sovereignty in the context of non governmental organizations (Greenpeace, Amnesty International), the United Nations and its offspring (World Health Organization, World Bank) and treaty-based organizations (the World Trade Organization, NATO). His political point is that the first two types (NGOs and UN-based) are antithetical to our sovereignty and our freedoms (and everyone else’s) because they are not instruments of defined political constituencies and thus operate independent of any legitimate political mandate and without meaningful political control. He thinks – and I agree – that we (the United States) should oppose these attacks on our sovereignty.

Along the way, he makes a point relevant to our discussion: sovereign nations are competitive with one another in ways that benefit citizens of all countries, ways that could not happen under supranational organizations such as the UN and the European Union. We have seen many examples: the EU badgers Ireland to “harmonize” its taxes with those of the EU. What this means, of course, is that Ireland should raise its taxes to EU rates, which would undermine Ireland’s remarkably successful economy. Who would benefit? The high cost economies of the continent, not the Irish people or the Irish economy. The Irish understand this very clearly and have refused to join the EU’s tax cartel.

The world has a rich history of government economic planning, from FDR’s National Recovery Act to the USSR’s notorious five year plans, along with many others. They share many characteristics, one of which is this: they did not work. The NRA did not end the Great Depression; the five year plans did not help the USSR’s economy. These planning regimes failed for many reasons, not the least of which is that no small group of people is smart enough or can possibly have enough information to create a plan as good as the result of individual decisions based on individual self interest. Conclusion, stated in terms of this paper: global planning and global governance systems don’t work and can’t work. The UN’s efforts to morph into a world government will fail.

The chaos that central planners abhor is our freedom.

* * *

This is demonstrated vividly in a new book by Rodney Stark: “The Victory of Reason: How Christianity Led to Freedom, Capitalism, and Western Success” ISBN 0812972333) describes the birth of capitalism in the erroneously named Dark Ages. He identifies the Roman Catholic Church as the primary enduring institution during this period. For geographical and historical reasons, the Church was much less centralized than the Church we know today. (The same was also true for some of the political entitles of the time.) Local monasteries and bishoprics ran their own affairs subject only to taxes and broad theological guidance from the center.

Chaotic from the point of view of Rome? Of course. But very innovative and productive. Monasteries developed from entities subsisting on products they made for themselves (cloth, food, etc.) to entrepreneurial organizations selling products to the public at large.

The evolution of political entities paralleled the evolution of the Church, but with a major difference that makes our point about chaos very forcefully. The smaller political entities – independent city-states or cities answerable only to a distant king – developed innovative capitalist economies, while the entities subject to strong central controls developed the feudal system. For people subject to the feudal system, the age was very dark indeed.

* * *

In summary, chaos as the generator of innovation is our first example of similar characteristics within disparate organizations. Chaos was never planned in either the church or in politics. But it happened, and it spawned great waves of innovation and prosperity. Our challenge as executives is to deliberately create chaos in our own organizations or support it as it develops in order to obtain the innovations we will need to survive.

You may have noticed that I have not provided any kind of definition so chaos. I will address this in a future blog.

<><><><><>