Skip to main content
 
Making Smart IT Choices: Understanding Value and Risk in Government IT Investments



Chapter 2. The analysis and evaluation process

Phase 3. Evaluate alternatives and make smart choices

The final analysis of alternatives is mainly an opportunity to integrate and compare information. Your work group has looked at the problem and potential solutions from many points of view using several tools. Now is the time to step back and ask what the entire body of data has to say. What are the main findings and conclusions that emerge? This final analysis helps you see what can and cannot be done to achieve your goals. Often, at this point you have several reasonable approaches to consider. This is the stage where you compare and contrast them along important dimensions such as risk, cost, and acceptability to key stakeholders. A more detailed description of the tools that can help you evaluate alternatives are available in Part Two.

Usually one or two of the possible solutions seem better to you than the others. The analysis you conduct at this stage needs to use the full range of data to test these preferences and justify why a particular course of action is preferable.

The most important limitation of any analysis is the quality and completeness of the data on which it is based. Weak data can't be improved by powerful analytical tools or fancy presentations. Be sure to pay attention to assumptions, estimates, and just plain guesses, and be honest with yourself and others in revealing what part they play in your analysis. Remember, too, that everyone has biases. Try to identify yours and counter balance them with solid analysis and reasonable alternative interpretations of the data.

How to make smart IT choices

Compare risks
In this phase you will make an explicit analysis of the risks associated with various tested alternatives. Notice that you have been identifying and dealing with risks from the outset. Opposing stakeholders represent risks, new technologies bring risk, and so on. Risk analysis can be very formal and quantitative or more qualitative. Scenario building is a technique in which you describe a hypothetical situation in a way that helps you predict the consequences of decisions and actions. It helps you see what could go wrong in different situations. You could also model these situations on paper or in simulations to understand how different actions or decisions might lead to unexpected results. This kind of analysis improves the confidence level of decision makers. These realistic projections of risk will help them understand the possible consequences of different choices.


Compare costs and expected performance improvements
Costs, of course, are critical to the final analysis of alternatives. You begin to identify and specify them much earlier in the process, just as you did with risks. This is the time to examine them in more detail to be certain that all cost factors are accounted for and that future costs are considered along with the costs of design and implementation. In projects that involve more than one organization or level of government, be sure to assess the costs to all players. Avoid the common mistake of costing out only the direct investments that your own organization must make.

Cost estimates can be obtained from historical data such as budgets or spending records, feasibility studies, or from outside consultants or agencies that have attempted similar projects. A cost-benefit analysis can be a simple comparison of costs and projected savings, or it can be a more detailed financial model. You can also conduct a robust cost-benefit or cost-performance analysis by considering all the players and effects of the system over time. Whatever you choose, the results have to be a convincing aspect of the business case you will develop.


Make and explain final choices
At this point you may have a clear best choice and need go no further to consider other options. If you still have at least two feasible alternatives, though, some selection tools will be useful. These are designed to compare alternative courses of action in a structured and explicit way.

The multi-attribute utility model or MAU model is a versatile tool for comparing alternatives. Its academic-sounding name makes it seem forbidding, but if you have been involved in any major procurements, you have probably already used it. The MAU model rests on the specification of weighted evaluation criteria. These might be such items as total cost, convenience to users, time to completion, reliance on contractors, or a host of others. Each criterion is weighted relative to the others, often by distributing 100 points among them. Then each alternative solution is scored on all criteria. Simple math gives each alternative a total score. Sometimes the scores demonstrate a clear best choice, sometimes they narrow the field, and other decision- making processes are needed. Occasionally the numerical scores do not "fit" the intuitive assessments of the evaluation group or decision maker. Often this means that an important criterion is not being made explicit and should be added to the model.

SWOT (for "strengths, weakness, opportunities, and threats") analysis presents another way to compare alternatives. This and other prioritizing and choice-making tools to consider will be described in the tools installment.

If you have done a thorough job of working through the three phases of analysis and evaluation, you will now be in a good position to make "smart" choices. You will have a clear understanding of the project objective(s) and will know what your stakeholders think about it. Relevant business processes will have been identified and improved. You will have information about alternative ways of approaching your goal, including the experiences of others who have done similar work. These alternatives will have been tested in some way with stakeholders and users, and the costs and risks of each one will have been identified. If more than one approach makes sense, you've conducted a comparison process that reveals their relative strengths and weaknesses. Now you are ready to make your recommendation for action in the form of a business case.

Get ready to prepare a business case
Gather together the findings from each stage of your analysis and review them all. Even if you were involved in all parts of the project, you may find that new information and insights emerge only when you step back and look at all the evidence together. Look for patterns, reinforcing information, conflicts, and gaps. Try to state the main findings in a few sentences or bulleted key phrases to create a framework for the final analysis and recommendations.

Get all the people on your team together to discuss the results and to decide how best to present your key findings and recommendations. Your business case will need to answer several questions in a direct and convincing way in order to garner support from decision makers. Try to state the answers to the following questions in a few powerful sentences and then organize your detailed data to back them up.

  • What are the characteristics and dimensions of the problem to be solved or initiative to be undertaken? (Review your service objectives.)
  • What are the expected benefits of solving this problem or taking this initiative? (Review the strategic framework and features and functionality analysis.)
  • What alternatives are available and how do they compare in terms of cost, feasibility, effect on stakeholders, and other key criteria that matter to decision makers? (Draw from cost-performance and stakeholder analyses.)
  • What course of action do you recommend and why? (Use the results of tests and evaluations.)
  • What are the one-time and ongoing costs of implementing your proposed solution? (These are part of the cost analysis.)
  • What are the drawbacks in your recommendation? (Refer to the risk analysis.)
  • What are the next steps?
Chapter 3 presents the essential elements of a business case for IT investment along with recommendations for presenting the case to various audiences of decision makers and stakeholders.