Government IT innovation is risky business. Evidence from many surveys and cases suggests that the threats and other challenges of IT projects often overwhelm the capabilities of the developers and implementers. Data from the private sector tells a similar story. For this paper, we focus our attention not on the full range of risks to government IT project success, but concentrate on the public return aspects of that problem.

The distinction between the public return and more generic aspects of risk analysis is not, however, a simple one. The approach is the same: identify and evaluate threats, develop and evaluate response methods, and produce a summary analysis and mitigation strategy. To identify and evaluate threats, we return to the overall value proposition schematic presented earlier (Figure 12).

Figure 12. Public Return on Investment Value Chain

In this figure, we identified two kinds of risk: 1) development risk and 2) benefit risk. Development risk, simply put, is risk that the development and implementation of the IT will fail outright or will not perform as designed and intended. Benefit risk applies to whether the IT investment will fail to produce the envisioned benefits in spite of being successfully developed and implemented. For the public value framework we focus on benefit risk. Of course any threat to the development of the IT is an indirect threat to public returns. However, several of the ROI methods described below include adequate analysis methods for development risk. These methods do not deal adequately, however, with the additional threats and issues in benefit risk.

The benefit risks associated with the creation and assessment of public returns come from threats to the creation of the returns and to their detection. And there appear to be two main sources of those threats: one is what we will call "theory failure" and the other is "exogenous factors." In theory failure, the underlying assumptions or theory on which the project is based are flawed or simply wrong. One such theory failure caused the US Department of Education to abandon a multi-million dollar pilot project for online college student financial aid administration. Developed without significant participation from college financial aid officers, the system did not attract supporters and generated much stakeholder resistance.(14) It is important to use both sources and types of benefits threats in a full risk analysis, as suggested by the two-by-two array in Table 6.

The risk analysis process can then use the threats identified in this way to estimate the potential loss or cost that each threat represents and the likelihood of the threat materializing. These estimates should be based to the extent possible on input from stakeholders, analysts, users, developers, managers, and policy makers. This will provide a basis for accurate estimates and concentrating attention on the threats with the highest combined loss-likelihood estimates. In some cases, the value, cost, and loss-likelihood estimates can be quantified to yield decision tools for moving ahead with an investment. The displays in Figure 13 and Figure 14 below, show such a result from the US Federal CIO Council's Value Measurement Method, which employs voting and other methods to prioritize and estimate quantitative values for cost and their related risks. The decision boundaries shown in the figures come from management decisions or some other deliberative process.(15)

The threats due to difficulty in detecting indirect or second-order effects deserve some added discussion. Public value outcomes can extend beyond those involved in the direct interaction with government. For example, drivers are the direct beneficiaries of an online license renewal system. However, the scope of possible indirect value outcomes and beneficiaries can be very broad. Residents in the neighborhoods of the license renewal agencies will benefit to some degree from reduced traffic congestion and pollution. Shopkeepers in that neighborhood may lose business for the same reasons. Second order effects may be even more diffused and difficult to detect. Learning to trust the online process for license renewal, for example, can result in more use and greater benefits from using other online services.

Since risks are tied both to stakeholders and possible value outcome variables, pursuing this line of reasoning can lead to a very large, and likely infeasible, list of risk analysis factors and tasks. To work within resource constraints will require limiting the risk analysis to the most important value variables and stakeholders. Setting priorities for this kind of analysis will therefore be essential, and must be based on the goals of the project and stakeholder value estimates.