"It [ROI] will let us begin to make assessments and decisions about funding a project or developing a new service based upon some true data. That moves you from having emotional debates about projects to having factual discussions."

Gerry Wethington, Missouri CIO¹

Government decision makers must make the most of scarce resources and at the same time respond to ever-increasing demands for improved performance and new technology. Thus the need for wise investment in information technology continues to grow. Growing demand in the face of scarce resources generates hard questions and close scrutiny of proposals for new investments. What’s more, the dismal failure record of many government IT investments raises legitimate concerns about the value of these investments and why they so often fail to live up to expectations, or even to work at all. As a result, IT planning processes often include, or even require, a rigorous business case to justify new IT investments. These include ways of assessing the costs and returns to be expected from that investment, that is, return on investment (ROI) analysis. This Guide is designed to help government executives who need to design, direct, conduct, or work with the results of such an analysis to make the most of their investments in information technology.

Growing interest in assessing returns on IT investments has spawned wide interest in methods of return on investment analysis. However, there is little agreement about best practices or specific methods for assessing ROI. Professional publications and consultant white papers present quite a variety of possible approaches. As a result, government executives and decision makers have difficulty choosing or designing a return on investment analysis that is both feasible and appropriate to their needs. To help administrators and decision makers with these choices this guide presents an overview of the purposes and concepts of ROI along with an introduction to basic methods and example cases. It also includes links to other resources for those who wish to explore some subjects in greater depth.

This guide treats ROI analysis as part of the overall decision making process for IT investment. The planners and designers of an IT project can use ROI analysis to help persuade decision makers to support the project. Decision makers can use an ROI analysis, indeed may even require one, as part of an IT investment proposal to aid them in evaluating it. In either case, an ROI analysis will be shaped by the situation in which it is designed and carried out.

Decisions about what sort of return on investment analysis to do, or whether to do one at all, will usually depend on a variety of factors. ROI analysis may or may not fit the larger context of investment decision making. Decision situations driven by very short deadlines or highly specific policy directives may rule out extensive analysis. Scale matters as well. An elaborate ROI analysis would hardly be justified for a small-scale, low-risk project that requires a fast decision. By contrast, large complex projects are typically high- risk propositions for which the added time and cost of an extensive ROI analysis would be fully justified. Even though justified, in some environments ROI analysis is not used at all in favor of best practice reviews or benchmarking to evaluate investment possibilities. Current practices vary considerably. In Iowa, for example, the state government has a standard framework for all agencies submitting IT project proposals, including business case and ROI analysis requirements.² The US Office of Management and Budget has imposed similar requirements on federal agencies. Agencies and local governments in other states may develop their own internal business case and ROI requirements.

Given the diversity of practice in the IT investment world, this guide takes an eclectic, non-prescriptive approach. It treats the subject of ROI analysis from the point of view of a curious but uncertain decision maker. The key issues facing this decision maker are whether to do an ROI analysis and, if so, how. The guide does not advocate engaging in ROI analysis under all circumstances, nor does it favor any particular technique. Instead it presents an approach to understanding a range of purposes and methods for ROI analysis that can assist that decision maker to move forward with wise and effective IT investment choices.

Decisions about how to use ROI analysis depend on understanding the nature of the methods themselves and how they relate to the business setting. ROI analysis in general is a rather diverse collection of methods, skills, tools, activities, and ideas. They all may be useful for assessing the relative value over time of some investment. These methods are not, however, a single formula or predetermined calculation that will yield a simple yes-or-no answer to the question of how to invest. ROI is not a silver bullet. Actually designing and carrying out any kind of ROI analysis requires making many choices among the ideas and methods available and conducting an analysis appropriate to the decision situation. Different choices will produce different results.

Consequently, a meaningful analysis of returns on investment in information technology is far easier said than done. Choices about how to conduct an ROI analysis should be based on critical understandings about:

• the strategic objective(s) of the analysis,
• the place of the proposed IT investment in the overall enterprise³,
• exactly how the analysis should be done (i.e., what data and methods of analysis are best suited to those objectives), and
• how the ROI analysis fits in the overall decision context for IT investments.

This guide will introduce you to these four basic understandings and provide resources for deeper investigation of each.

Your understanding of the strategic objectives of an ROI analysis will determine how the analysis is ultimately done and used. The matter of strategic objectives has two related parts. One deals with the objectives and context of the proposed investment. The second deals with the objectives and context of the ROI work itself. An adequate understanding of the first part of the strategic objective must include answers to these key questions detailed below.

The value of an investment is directly related to the programmatic goals and business process employing the new technology. For example, the strategic goal of investing in a new financial management system would be to improve the quality of financial decision making and control of resource flows, not simply to produce faster or more complex accounting reports. Attention to the strategic objectives keeps attention focused broadly on the kinds of benefits sought and how to measure them, rather than narrowly on the technology. Focus on the strategic and business objectives will also lead to a clearer understanding of the full range of benefits to be expected from the investment.

Stakeholders are critical players in the success of new projects. Ignoring or underestimating the importance of their role puts the success of an investment at great risk and can lead to substantial unanticipated costs or missed benefits. One large organization recently developed an extensive new customer help desk application without the participation of the staff who handled customer phone calls. Those staff members learned of the new system the day before it went live. The result was poor performance and a serious blow to staff morale—lower returns and higher costs.

An ROI analysis can itself be a substantial investment. Extending its scope, level of detail, or complexity beyond what is needed for effective decision making is a waste of resources. Attempting an ROI analysis that is beyond the resources or capacity of the organization will also waste resources. Choosing an appropriate scope for the analysis is therefore a critical part of the process. These choices determine the details of the ROI analysis itself: kinds of data used, whether hard or soft estimates will suffice, the kinds of projections, quantitative or qualitative data that are needed; which financial or non-financial outcomes (customer/user satisfaction, social or political outcomes, improved equity) are all important. The answers to these questions then influence the kinds of personnel, tools, and other resources needed.

Consideration of risk should always be a part of the ROI analysis. Risk factors arise from the nature of the project itself (complexity, scale, novelty), its organizational setting (conflict, resource constraints, top-level support, time pressures), and the larger environment (political turbulence, crises, and policy shifts). Risk analysis, discussed in some detail later in this chapter, examines the likelihood of risk factors affecting the project and what elements of the project or its results are likely to be affected.

Table 1. IT ROI Questions from the CFO to the CTO⁴

If an ROI analysis were just one simple thing, then there would be one simple way to measure the costs, returns, and benefits. In practice, however, there can be many different questions asked of an ROI analysis requiring different measurement approaches to fit those questions. How to identify these measures and apply them to analysis are the next parts of the puzzle.

Solving that puzzle requires understanding the differences among the questions asked of an ROI analysis. There are four different but related types: financial, effectiveness, efficiency, and impact questions.

Answering these questions requires information about costs and returns in terms of the monetary value of the resources used (inputs), as measured and recorded by standard financial factors. In its simplest form, an ROI analysis based on this kind of question would calculate the return in terms of the expected savings and revenue increases (if any) compared to the dollar cost of all expenditures on the new system. The costs, savings, and revenues might be projected over a multi-year time span in order to show a payback period or to estimate the present value of future returns.

Answering effectiveness questions requires information about the costs of the project in relation to how much it contributes to achieving program goals and producing the desired results. The metrics will be more complex, involving unit cost or activity cost calculations. The measurement of returns will be expanded beyond cost savings or revenue increases to include levels of performance relative to program or project goals.

Answering efficiency questions requires information about whether the project will produce the greatest possible value relative to its costs. Efficiency questions pose serious analysis problems. Establishing that a particular result is the best of all possible results requires either examining many alternatives or simulating performance in some way that gives a valid picture of what is possible. This can be done for some kinds of systems with sufficient resources and data, but can substantially increase the cost and complexity of the analysis.

Answering impact questions requires information about the larger social and economic benefits and costs of a project. These questions pose two tough problems. Measuring the broad social and economic costs of an investment requires data far beyond what typical financial systems provide. Measuring, or even identifying the full range of social and economic benefits from some government IT project can be even more daunting. The idea of figuring a cost/benefit ratio is appealing, but seldom feasible. Though not impossible, the breadth and complexity of this kind of analysis means it is rarely found in IT investment planning.

The four types of measurement questions and approaches differ in several ways. The choice of relevant metrics is one critical difference. Some approaches are based on strictly financial metrics (costs or returns in dollar terms), others include production output metrics, such as quality of goods or services, client or customer satisfaction. Metrics may extend to organizational factors such as morale or to social and political outcomes. These can include impacts on quality of life, social equity, social or human capital, or political support. These differences are summarized in Table 2 (page 11).

This guide views any IT investment project as embedded in its organization’s enterprise architecture and in a context with three major elements: relevant business processes, the organizational setting, and the external environment.⁵ These are illustrated in Figure 1 on page 10. The immediate context of any IT project is found in the current business process(es). Some of the costs and returns of the project will be directly linked to business processes, such as training costs for staff involved or the improved efficiency of the overall process resulting from project implementa- tion. Other costs and returns will be linked to the organization, for example through changes in resource flows, performance changes and changes in workflow and internal relationships. Linkages with the external environment may be significant as well. Resources may be committed from that environment to support the project, and additional costs may be imposed on external persons or organizations by changes in the way services are delivered or other business is conducted.

For example, an agency implementing EDI (electronic data interchange) to support purchasing transactions may be imposing costs on vendors who wish to do business with the agency and have to invest in developing their own EDI capabilities. The way Figure 1 below represents investment costs and returns as part of the same context is an important way of looking at ROI. Neither the costs nor the benefits of an IT project begin and end at the project’s boundaries. Financial allocations to a new project mean fewer resources somewhere else in the organization or its environment.

Changes in one part of a business process may impose costs on other parts that have to adjust activity, retrain staff, or modify other systems. Increased efficiencies in one business process can make resources available elsewhere in the organization, but may also result in changes in other linked processes within the enterprise.

Whether you see something as a cost or benefit of an IT project may depend on whose perspective you take. The expendi- ture on a new personnel management application is a cost to the agency that pays for it, but it is a benefit to the vendor, who is an external stakeholder. These links make it clear that an analysis of costs and benefits of an investment can require attention extending well outside the project itself into the organization and its environment. Business processes are the critical connection between the project and the rest of the organization. So attention to business process linkages with the project is an important part of the overall ROI analysis.

Table 2. Approaches to Cost and Return Measurement

Measurement Question	Measuring Costs	Measuring Returns/Benefits
Can we afford this and will it pay for itself?	Financial metrics; defined by policy and accepted accounting principles ; reporting and control-oriented; standards- based or consistent; not linked to business process; ignores important cost factors; short time frame; data routinely collected/reported	Savings as measured in accounting categories; narrow in focus and impact; increased revenues, reduced total costs, acceptable payback period
How much ‘bang for the buck’ will we get out of this project?	Financial and outcome/quality metrics; operations and management oriented; defined by program and business process; may or may not be standardized; often requires new data collection; may include organizational and managerial factors	Possible efficiency increases; increased output; enhanced service/product quality; enhanced access and equity; increased customer/client satisfaction; increased organizational capability; spillovers to other programs or processes
Is this the most I can get for this much investment?	Financial and organizational metrics; management and policy oriented; non-standard- ized; requires new data collection and simulation or analytical model; can reveal hidden costs	Efficiency increases; spillovers; enhanced capabilities; avoidance of wasteful or suboptimal strategie
Will the benefits to society (our state, our city, etc.) justify the overall invest- ment in this project?	Financial, organizational, social, individual metrics; individual and management oriented; nonstandard; requires new data collection and expanded methods; reveals hidden costs; potentially long time-frame	Enhanced capabilities and opportunities; avoiding unintended consequences; enhanced equity; improved quality of life; enhanced equity; enhanced political support

Investment decisions in the public sector, whether they involve IT or any other expenditure, take place in a context of political and policy influences. No matter how solid or technically sophisticated an ROI analysis may be, it will not likely be the sole determinant of an investment decision. It is useful in deciding how to prepare and present an ROI analysis, therefore, to take into account possible political and organizational factors. Such a consideration of external factors may help shape the style, emphasis, or presentation strategies employed to introduce ROI analysis into decision processes. Such considerations are discussed below and may also help in recruiting support for the conclusions of the analysis and guiding how the analysis process is described or defended.⁶ All of these considerations can be classified as different types of risks.

Public exposure of failure or error. Government’s business is public business. Most new ideas are implemented in full public view. Any investment-gone-wrong risks not only dollars, but the credibility of an agency and its leadership with legislators, executive officials, and the public. So government tends to reduce risks by relying on the "tried and true." Failure risk can be mitigated by taking care not to over promise the benefits of new projects and to ensure that there is adequate strategic planning to reduce the probability of failure. Undue caution can also risk a different kind of failure: missed opportunities for successful projects.

Divided authority. Public executives seldom have a clear line of authority over agency operations. Their decisions are circumscribed by existing laws, budget and financial controls, civil service systems, political constraints, and a variety of regulations imposed by both legislatures and the courts. These restrictions impede managing the complexities of multi-million dollar IT projects in a rapidly changing technical environment.

Multiple stakeholders. Stakeholders typically have competing goals. Customers, constituents, vendors, service providers, elected officials, professional staff, and others all have some stake in IT projects. Understanding how different choices may affect and be affected by each stakeholder group helps to prevent unexpected problems.

Annual budgets. Government budget cycles increase the uncertainty about the size and availability of future resources. This diminishes government agencies’ abilities to adopt or sustain new IT innovations successfully, especially those that have long development periods.

Highly regulated procurement. Regulations in the typical competitive bidding process are ill suited for the experimentation and learning that is often essential for successful IT investments. While promoting integrity and fairness, procurement regulations are often a source of problems and delays. These are especially troublesome when agencies write requests for proposals (RFP’s) that depend on the limited information they have been able to gain from inadequate experience and research.

Complex program networks. Government programs are connected in many complex ways to other programs in the same or other agencies, or to non-governmental entities. Sometimes the connections are explicit and formal. Often they are informal or unintended. Changing programs can have unintended consequences for several others, producing additional costs and problems for the investment project.

Misalignment of goals. Some parts of an organization may see the goal of an IT project in narrower, possibly conflicting terms. For example, an IT unit may become enamored with a database or office automation project because of its technical elegance. The end users of the system, however, may want capabilities that are not compatible with the new technology. Without some goal alignment a project is on the path to failure.

Lack of leadership support and organizational acceptance. Top management support for a technology initiative is critically important. Similarly support and acceptance throughout the organization, especially among the people who will use the technology or its products, are equally important, and often more difficult to achieve. Understanding and enhancing support reduces or limits risks.

Reality of the process. IT inventions are embedded in business processes. Failure to understand and account for this reality in project design and implementation introduces major risks. Systems are often created that do not serve business needs, are too expensive for the small productivity gains they provide, or are not flexible enough to meet changing demands.

Rapid change. Obsolescence is a risk as soon as a project chooses its technology. The pace of technological change makes it difficult for planners to keep up with the details of new developments and to understand comprehensively how each new technical tool works, or may be superceded. Technology choices guided by long term strategies and strong linkages to business goals can mitigate many of the risks produced by rapidly changing technology.

Technology interactions. A basic concept of system analysis is "you can’t do just one thing." New technologies interact with old technologies and work processes. The interactions may enhance the value of the older and newer technology, or interfere with both. A careful analysis of these interactions will identify risky situations and provide insights for avoiding problems and errors.

Scale and complexity. Risks increase directly with the scale and complexity of IT projects. Planning an incremental process of development, with careful contingency plans, can mitigate some of these risks and will avoid problems that cannot be anticipated on the path from small to large, complex systems.