If an ROI analysis were just one simple thing, then there would be one simple way to measure the costs, returns, and benefits. In practice, however, there can be many different questions asked of an ROI analysis requiring different measurement approaches to fit those questions. How to identify these measures and apply them to analysis are the next parts of the puzzle.

Solving that puzzle requires understanding the differences among the questions asked of an ROI analysis. There are four different but related types: financial, effectiveness, efficiency, and impact questions.

Answering these questions requires information about costs and returns in terms of the monetary value of the resources used (inputs), as measured and recorded by standard financial factors. In its simplest form, an ROI analysis based on this kind of question would calculate the return in terms of the expected savings and revenue increases (if any) compared to the dollar cost of all expenditures on the new system. The costs, savings, and revenues might be projected over a multi-year time span in order to show a payback period or to estimate the present value of future returns.

Answering effectiveness questions requires information about the costs of the project in relation to how much it contributes to achieving program goals and producing the desired results. The metrics will be more complex, involving unit cost or activity cost calculations. The measurement of returns will be expanded beyond cost savings or revenue increases to include levels of performance relative to program or project goals.

Answering efficiency questions requires information about whether the project will produce the greatest possible value relative to its costs. Efficiency questions pose serious analysis problems. Establishing that a particular result is the best of all possible results requires either examining many alternatives or simulating performance in some way that gives a valid picture of what is possible. This can be done for some kinds of systems with sufficient resources and data, but can substantially increase the cost and complexity of the analysis.

Answering impact questions requires information about the larger social and economic benefits and costs of a project. These questions pose two tough problems. Measuring the broad social and economic costs of an investment requires data far beyond what typical financial systems provide. Measuring, or even identifying the full range of social and economic benefits from some government IT project can be even more daunting. The idea of figuring a cost/benefit ratio is appealing, but seldom feasible. Though not impossible, the breadth and complexity of this kind of analysis means it is rarely found in IT investment planning.

The four types of measurement questions and approaches differ in several ways. The choice of relevant metrics is one critical difference. Some approaches are based on strictly financial metrics (costs or returns in dollar terms), others include production output metrics, such as quality of goods or services, client or customer satisfaction. Metrics may extend to organizational factors such as morale or to social and political outcomes. These can include impacts on quality of life, social equity, social or human capital, or political support. These differences are summarized in Table 2 (page 11).