This framework is designed to assist government IT executives and analysts in understanding and measuring the value to the public of government IT investments. The main goal is to produce an assessment of public returns that is credible, persuasive, and highly relevant to the investment decisions faced by the government. We use value here, rather than return to emphasize the broad scope of the framework. Most methods for assessing return on investment focus on financial or economic metrics; this approach includes a much broader view of how IT investments can produce results of value to citizens or to the society as a whole.(4) This concept of value includes more than the usual financial or economic metrics common in ROI analysis. It is a new and expanded way of understanding the results of government IT expenditures.

We call this set of ideas a framework to indicate that it is more than a particular method for public value assessment. It is broad in scope so that it can be applied to virtually any government IT investment, from simple Web sites to governmentwide information systems and architectures. It is broad in scope because this range of investments requires a comparable range of assessment methods. Our framework, therefore is a way of thinking about and organizing the analysis of a family of problems that can encompass many methods. A spreadsheet, for example, is a framework for working on a broad class of problems or analytical tasks. Any particular spreadsheet may include specific methods, such as scenario analysis or a net present value calculation. Instead of guiding the assessment process in terms of calculations in a matrix, however, this framework provides an analysis process that starts with a high level view of the IT investment and then drills down through successive steps to identify the specific measures and methods that will reveal and document public value.

In this way, the framework offers both a systematic way of thinking about public value and a way to apply that thinking to particular IT investment decisions. The drilling down process is necessary to tailor a specific public value assessment to the nature of a particular investment decision. The framework shows how to take into account how public value can change across the many interests of citizens and groups in interacting with governments. In the morning, for example, an executive doing business with the government may think about how IT speeds payments on her government contracts, in the evening while helping with homework she may observe how computers improve the quality of schools, and while watching the news on TV at night she might learn how a new crime mapping system makes the neighborhood safer. These ways of thinking about public returns include both easily measured value, like improved financial flows, as well as highly subjective ones like public safety, service quality, or government integrity. A framework for public value assessment must provide a way to deal with these many perspectives and possible measures of public value.

Not every aspect of public value is relevant for a particular IT investment. The Washington State Digital Archives project, for example, had no particular public safety related goals, but is of considerable value to genealogists and local historians. Our framework begins the process of narrowing and focusing by starting with the three basic elements of analyzing public value: the investment, the government operations affected, and the stakeholders. At the beginning of a public value assessment, an analyst often knows that there are many connections among theses three elements, but those connections are poorly specified and understood. The situation might be described by the overlapping parts of Figure 5 below. The public value is to be found by unpacking, so to speak, the area of overlap. The task of the assessment, therefore, is to identify the connections and gather data about how the IT investment produces value for the relevant stakeholders.

Close attention to all three elements is essential. The connections among these elements are the keys to a fully informed public value assessment. The framework provides a way of describing these connections to show how public value is generated and the risks involved therein. Focusing on one or two elements alone cannot reveal the necessary scope of public values involved in an IT investment or how they can be assessed. The risks of slighting one or another of these elements in an assessment can be substantial. In addition to missing significant public returns, such limited thinking can lead to stakeholder resistance, flawed technology decisions, or poor integration with or disruption of business processes.

Analyzing those connections can, in principle, start in any part of the problem. Ultimately, of course, these three elements must be considered together in order to ensure that the value assessment for any particular investment project or system is tailored to the specific value context, investment situation, and the beneficiaries involved. In practice, however, the assessment must begin somewhere. The framework presented below describes the logic and methods that guide the assessment process.

The sequence of activities shown for an assessment is not designed or intended, however, to suggest how IT investment decisions should be made.

Figure 5. The Basic Elements in the Public Value Framework

The framework describes public value assessment as a series of steps to gather specific data and use it to answer questions that lead to public ROI results. The results can include measures and documentation of public returns, risk analysis of the threats to achieving those results, and suggestions for presenting results to decision makers. Each component of the framework deals with a different set of questions and possible data to use in subsequent steps, and so forth. By working with the ideas and methods presented in the framework, the analysis can produce a public ROI assessment that is broad in scope, yet tailored to the particular investment of interest.

The steps are not intended to be used in a lock-step manner. The questions and inquiry involved in one part of the framework may be obvious and easily answered in some cases and quite complex and difficult in others. It may be necessary to explore later steps before preceding ones are finished, and then cycle back. In general, however, the sequence follows the path shown in Figure 6 below.

The steps illustrate how the process of value identification and measurement carries through from the goals of an IT investment, through the value generating mechanisms of the business processes, connection with stakeholders, to specific data and reporting. These are the main components of the framework.

By starting with a deliberately broad scope, then narrowing to specific questions for each project, this approach addresses one of the shortcomings of previous work on ROI for government IT, namely the narrow scope of possible value questions considered. By expanding the initial scope of analysis to include a broad view of public returns, the framework can lead to a more comprehensive result. The framework thus does not replace or supersede the existing methods developed by others for assessing returns to government IT investments. Rather the framework shows how these methods can be used in assessment, what additional value issues should be addressed, and how additional assessment methods may be used.

Figure 6. Steps in Applying the Public Value Framework

The question of how an IT investment can impact public value can be answered initially in terms of three impact mechanisms. These three, alone or in combination, describe the general ways IT investments can connect with the public, as revealed in our research. Identifying these linking mechanisms is an important step in the analysis process because it leads to a more detailed examination of impact pathways than might otherwise occur. IT investment can enhance public value in many ways, some of which are only indirectly related to the performance of a particular IT system. Attention to these indirect and more complex mechanisms can help ensure a full picture of public value results is developed.

Direct service impacts (Figure 7) occur when IT is embedded directly in a service delivery process, generating service changes that enhance value to the citizen or official involved in the service process. The net public value is just the difference between the value of the new and the old. Value in this sense, and in the other mechanisms as well, is assessed from the point of view of the public participant in the transaction, not the government. The total value may be a composite of several separate measures, such as lower cost to the citizen, plus increased satisfaction with quality of service, plus enhanced trust in government. The specific kinds of value involved are described in more detail in a later section.

Figure 7. Type 1: Direct Service

The e-government investments in our case studies and much of the literature on e- government provide many examples of this kind of investment. These include systems for fee and tax payments, license application and renewals, obtaining information, filing forms, etc. The public value propositions for these investements go beyond the important but obvious cost and time savings to include attention to service quality, access, equity, and the full range of values described below.

Indirect service impacts (Figure 8) occur when back office or infrastructure investments produces changes in a government business process. As in Type 1 mechanisms, the value to the public is reflected in the changed interaction or transaction with a government business process. However, the process change results from an IT investment at least one step removed from interaction with the public. Because of this indirect route, it may be more difficult in Type 2 cases to trace the links from the IT investment clearly and unambiguously to the public. Risks to the effectiveness of the investment are higher as well, due to dependence on business process changes that are independent of the IT itself. The IT investment may have potential results spread over many business process and may interact with other technologies, further obscuring the impact linkages.

Figure 8. Type 2: Indirect Service or Impact

The ERP system implementations in Austria, Israel, and Pennsylvania that were examined in our case studies are excellent examples of this kind of investment. The primary impact of these systems is on the internal, back office operations of the government. How to identify and describe these linked back office operations is discussed in a later section of the framework in terms of business reference models (see Figure 10). The impacts on these internal operations may be very positive and extensive, and still remain hard to trace to many kinds of value gains for individual citizens or organizations. This is an important part of the public value problem dealt with in detail in the value types discussion to follow. ERP systems also contribute to public value generation through the Type 3 mechanism discussed below.

In this third type of impact mechanism (Figure 9 below), the links to public value are more complex. The direct value mechanism shown here as Vd, is the same as a Type 1 or Type 2 mechanism, producing a value increment for a citizen, official, or group (shown in a simplified way). In this type of system, the new IT is also linked to changes in the environment and relationships between the direct beneficiary and other entities (persons, groups, organizations) in the public arena. This indirect mechanism can result in additional public value flowing from interactions outside the government, between other persons or organizations. The changes in the environment may also produce direct value beyond the direct transactions.

Figure 9. Type 3: Mixed Direct, Indirect Service & Environmental Impact

There are many examples of this type of value mechanism in our cases and in other settings. In the Washington State Digital Archives case at least two such impacts occurred. The ability of the county auditors to shift responsibility for preserving digital records to the state archive produced savings in cost and workload at the county level (Vd). This allowed county auditors to improve services to their citizens in other areas (Vi). In addition, the accessibility of the government records online increases the overall environment of transparency for the government, benefiting all citizens. In the Service New Brunswick case, Web access to company registrations provided direct value to citizens such as accountants, lawyers, financial institutions, and the general public searching for such business information (Vd). In addition, the decision by CGI to locate its Global E-Government Headquarters in Fredericton, New Brunswick, due in large part to its strong partnership with Service New Brunswick, benefits the economic development of the city of Fredericton and surrounding areas in the province (Vi). In a different way, putting some forms of information online, such as the Toxic Release Inventory in the US, enable citizens to obtain benefits from other transactions, such as lawsuits against polluters (Vi).(5)

Just identifying these operational mechanisms in general terms, however, does not tell the full value story. Each mechanism can involve more than one kind of value generator. These mechanisms show how IT investments can link to increased public value, but not how that increase is produced. Each mechanism has the potential to generate more than one kind of public value increase, depending on the details of the situation. The framework recognizes four basic kinds of public value generators, listed below, each with a different range of measurements and implications for assessment:

• Increases in efficiency — obtaining increased outputs or goal attainment with the same resources, or obtaining he same outputs or goals with lower resource consumption. In our Austria and Pennsylvania case studies, for example, new ERP systems helped achieve substantial efficiencies in financial management and other core administrative functions of government.
• Increases in effectiveness — increasing the quality and/or quantity of the desirable thing. Our case study of Service New Brunswick, for example, reports how an online registry for land data can contribute to improvements in property tax administration.
• Enablement — providing means or allowing otherwise infeasible or prohibited desirable activity, or preventing or reducing undesirable events or outcomes. In our Washington State Digital Archives case study, for example, putting birth and marriage records online enabled research by local historians and genealogists.
• Intrinsic enhancements — changing the environment or circumstances of a stakeholder in ways that are valued for their own sake. For example, our Israel case showed how enhanced financial accounting and reporting in the Merkava ERP opened government financial decision making to greater transparency.

The examples above illustrate public returns that accrue in addition to, and largely independently of, internal efficiency gains that may accompany the IT investment. These value generators can also operate together, increasing the overall return. Research by the local historians mentioned above was enabled by access to previously inaccessible records; online access to these records also made research much more efficient compared to paper records. Access to crime mapping information by citizens could also improve the efficiency of choosing a place to live or locate a business.

This analysis of public value generation requires identifying how the IT investment project links to government goals and activities. IT investments generate value in relation to the policy and administrative setting in which they operate. The goals and business processes provide links between how the technology operates and the interactions with stakeholders that generate value. The analysis, therefore, includes linking the investment to the relevant government goals, operations, and business processes.

This linking process is more complex than it may appear, requiring a comprehensive and reasonably detailed picture of government goals and operations. Such a picture will ensure that all the relevant links between the technology and business processes will be identified. Many government IT investments have potential links across many agencies and processes. The Washington State Digital Archives, for example, collects records from hundreds of state and local agencies. The Merkava project in Israel will eventually involve all government agencies. The Pennsylvania IES supports human resources management, budgeting, and other administrative functions for all executive agencies in the Commonwealth. The relevant business process setting of an IT investment, therefore, can be quite extensive. To deal with this contingency, this part of the framework is based on just such a comprehensive picture. This is a useful analysis strategy because it helps draw attention to value generating aspects of the investment that might be missed because they result from indirect effects or complex interactions across many agencies or processes.

The first part of the linking takes advantage of the comprehensive integrated descriptions of government goals and processes that are found in the enterprise architecture work currently underway for US and other governments. For our purposes here, the most useful comprehensive description of government processes is in the US Federal Enterprise Architecture (FEA) Business Reference Model (seen in Figure 10).(6) With some minor modifications, the components in this FEA model can be used to identify the business operations and general goals of an IT investment in most government agencies. Local governments, for example, do not typically have defense and national security operations. The supporting documents that form part of the Business Reference Model contain descriptions of each of the components. These descriptions can be used to clarify meaning and help identify which components are linked to a particular IT investment.

While comprehensive, this FEA model is not the only enterprise architecture model that may be used. There are a number of comparable models or business process frameworks that could work as well, such as those developed for the EU, the UK, Hong Kong, and Malaysia.(7) The National Association of State Chief Information Officers (NASCIO) has an enterprise architecture model, adapted for US state governments, that may also be used. The European Commission also supports additional framework activity through the Athena Project, which has published similar documents.(8)

Using this model, the analysis proceeds by identifying where the IT investment links to government business. This linking can be based on the new technology's stated goals, organizational location, and intended operational profile. The detailed descriptions of each Business Reference Model component are useful in identifying these links. If needed, those responsible for the analysis can enlist a team of IT and operational experts to help ensure all relevant links are identified.

Figure 10. Elements in the FEA Business Reference Model

To illustrate how to use the model, we marked several of the components in Figure 10 with asterisks. The asterisks mark an example of the business links we identified for an IT investment project in the Merkava case study used for this framework: an online reverse auction system for procurement. The agency conducting the online auction sets the terms of a procurement, publishes them, and qualifies vendors as potential bidders. At a predetermined time, a Web site is opened for bidding. All qualified vendors can then submit authenticated price bids to a public space on the site, visible to all other vendors. The bidding continues until the low bid remains unchanged for a predetermined time (e.g., five minutes), closing the auction. The low price bidder wins the contract. The entire auction takes place in public view and is recorded in detail.

For an illustrative example, consider the procurement of new police radios for a law enforcement agency. Based on the nature of the e-procurement system and the component descriptions, we see it as linked to the five marked components in Figure 10: law enforcement, financial management, supply chain management, direct service to clients (vendors), and planning and resource allocation.

Making further links to business processes means connecting the larger goal or function from the Business Reference Model to specific activities identifiable as business processes. For this step, picking the appropriate level of detail is important. Choosing too general or too fine-grained a process analysis will tend to obscure stakeholder interactions that are necessary to identify public value. For this step in the analysis, we recommend another enterprise architecture tool: the Zachman Framework.(9) The descriptions in the first two rows of the Zachman Framework (Scope and Business Model) are a good guide for the most useful level of detail.

This method identifies several business processes for the e-procurement example. They include:

• determining communication needs for the law enforcement agencies
• budgeting for the procurement
• recruiting and qualifying vendors
• conducting the auction
• paying the vendors

The list could be extended for further analysis by adding deployment of the radios, training for users (human resource management), and evaluation of the impacts of the new radios use in the agencies and business processes where installed. To keep this example manageable, however, we are limiting the analysis to the processes listed below. The components identified from Figure 10 are linked to the business processes in the rows of Table 1.

Identifying the business processes leads to two key questions: (1) How does a business process generate increased public value? and (2) For whom? A way of answering the first question is recorded in the third column of Table 1. Our analysis of public value generating mechanisms is shown in the discussion of value impact mechanisms below.

Identifying the linkages between business processes and public value generators described above carries the analysis a critical step farther: from how the IT changes a business process to the impact of those changes for a particular public or stakeholders. That step is accomplished by looking to the value of the impact in terms of the interests of one or another stakeholder. So the framework includes an analysis scheme for taking into account interests that can apply to the full range of stakeholders. The framework employs these six basic kinds of interests:

• Financial — impacts on current or anticipated income, asset values, liabilities, entitlements, and other aspects of wealth or risks to any of the above
• Political — impacts on personal or corporate influence on government actions or policy, role in political affairs, or influence in political parties or prospects for current or future public office
• Social — impacts on family or community relationships, social mobility, status, and identity.
• Strategic — impacts on economic or political advantage or opportunities, goals, and resources for innovation or planning
• Ideological — impacts on beliefs, moral or ethical commitments, alignment of government actions or policies or social outcomes with beliefs, or moral or ethical positions
• Stewardship — impacts on the public's view of government officials as faithful stewards or guardians of the value of the government itself in terms of public trust, integrity, and legitimacy

This last interest—stewardship—is deliberately not included in the ideology category, though it could logically fit there. Setting it off separately emphasizes its importance in the overall public value framework. Just as corporate managers and directors are responsible for stewardship of a corporation's integrity and assets on behalf of stockholders, public managers and elected officials have a parallel responsibility for government on behalf of the public. Recent financial scandals in the US have demonstrated how lack of stewardship in major corporations (e.g., Enron, WorldCom) can destroy their financial value. The government officials in our cases expressed a similar stewardship responsibility in terms of, for example, using IT to increase transparency and accountability. Public surveys of trust in government show that citizens have similar stewardship expectations of government.(10)

Clearly this list goes well beyond the internal efficiency or service quality impacts of an IT investment. Of course, attending to this expanded concept of value raises many problems of definition and measurement. However, our case research and results from many other kinds of inquiry show the importance of this more comprehensive treatment of value and interests. Many of the measurements and data implied by these interest types are found in existing ROI methods as well. However, none of the methods we reviewed for this white paper cover the full range represented here. Working with this expanded range of value types represents one important contribution of this framework.

It is also clear from this way of describing public value that it supplements rather than replaces methods that assess the efficiency gains or savings returns captured by government agencies internally. For ex- ample, it seems safe to say that a Department of Motor Vehicle's internal cost savings from putting license renewals online are invisible and largely irrelevant to drivers. These savings would have no detectable impact on overall tax burdens or benefits resulting from a shift of government resources to some other service. The information about the savings, however, is another matter. If government officials make some political use of the cost savings information, that would represent strategic value to those officials or to their political allies. The public learning about the savings might also improve their opinion of the government's stewardship, fairness, or other ideological value.

Completing the analysis of value impacts requires identifying those with an interest in the value generating process: i.e., the stakeholders. The kind of stakeholder analysis required for this task has three main parts: (1) identifying the persons or groups (including organizations) whose interests are potentially affected, (2) identifying what their specific interests may be, and (3) assessing their role and potential influence in the delivery of public value. The first two parts are necessary for the analysis of value impacts and possible assessment strategies and so are discussed here. The third part is more relevant to the risk analysis discussed in more detail in section III.H below.

Identifying stakeholders for the framework will depend on in-depth knowledge of the context of the investment and the agencies involved in its development and use. Typically, stakeholder analysis engages a group of participants with extensive knowledge of the political and organizational setting of the investment and, hopefully, with experience in this kind of analysis. Because the analysis is so context-dependent, there are no standard processes to follow. There are consistencies, however, among the many versions of stakeholder analysis methods found in the management, planning, and assessment literature. These include:

• Involving multiple participants with wide knowledge of the stakeholder environment
• Looking widely to identify all relevant stakeholders through brainstorming and related methods to stimulate divergent thinking and include multiple opinions and information sources
• Identifying multiple stakeholder roles, internal and external to the organization setting (e.g., internal user, customer, vendor, developer, manager, oversight, politician, taxpayer, analyst, advocate, etc.)
• Identifying stakeholder expectations, influence potential, past and future participation possibilities, and level of interest

In this kind of analysis, continuing reference to the nature of the investment and impact mechanisms will help produce the needed focus and detail.(11)

The results of a stakeholder analysis have implications for value assessment and for understanding their potential to influence the investment project. These results can be presented in ways that show the multiple dimensions used in the analysis. An example of results from a hypothetical stakeholder analysis is shown in Figure 11.

Figure 11. Stakeholder Analysis Matrix (12)

This figure combines data for 14 stakeholders' roles (SA = Stakeholder A, etc.) and times of possible influence in a project's lifecycle. This kind of display illustrates some of the complexities of stakeholder analysis and the possibilities for interactions among different stakeholder groups. A more complete analysis would include estimates of the stakeholder's influence capabilities and specific interests in the project development and outcomes. These issues are discussed in more detail in the section on risk analysis (section III.H).

To focus on stakeholder interests, the illustration in Table 2 below shows how the results of a simple stakeholder analysis can be linked to the business process, value types, and mechanisms.

The business processes identified in Table 2 link to a partial list of stakeholder types in the columns: citizens at large, vendors, and elected officials. An actual assessment would include a more detailed list of stakeholders. The value types most likely to map onto each stake- holder type are entered in the cells. The basis for this mapping is linking the nature of the value as understood in that context with detailed knowledge of the stakeholder groups. This part of the assessment can include data collection about the interests and expectations of stakeholders through interviews, surveys, or focus groups if needed. The entries in the column for the citizens at large, for example, are based on the assumption that their interests in government stewardship would be served by the e-procurement system. While a reasonable assumption, this could be confirmed by collecting opinion data from a sample of citizens, or relying on previous opinion research. Similarly, the entries in the column for government IT staff are based on assumptions about what advances the interests of those staff members. A successful procurement system would provide political value and some strategic advantage to elected officials, as well as enhance their reputation as good stewards of public resources.

Using the framework up to this point has identified many kinds of data relevant to assessing public value. The next step is to choose the specific variables or points of observation for collecting the assessment data. This is as much a practical problem as an analytical one. Many variables may be relevant for each combination of stakeholder and value type. A few variables that would be relevant to this example are shown in Table 3.

This table illustrates two important aspects of its role in the framework and assessment process. First, it is highly unlikely that applying the framework to any IT investment would lead to variables in all, or even most of the cells of this matrix. This matrix can be thought of as an heuristic device, prompting questions about what might be useful and available variables for each row and column without expecting to fill them. The second is that identifying a specific variable relies on combining information about stakeholder interests, the value type, the impact mechanisms, and the context. This is a complex and demanding process. This section offers additional guidance on choosing the best public return variables for a given assessment. In addition, the other ROI methods described below include many variables and additional methods that can be helpful in that task. However, the more the choice of variables can be tailored to the specific public value context, the more valid and persuasive the assessment is likely to be.

An actual public value assessment should be based, of course, on the best available information. But actual assessments take place in practical settings of limited resources and access to data, plus being part of the additional work needed for internal returns and costs. The priority setting described in the risk analysis section can narrow the field to only the most important public value outcomes. The section below describes additional strategies for choosing the appropriate variables and analysis methods.

The choices of variables and analysis methods for the empirical parts of a public value assessment should be considered together. In terms of basic measurement and analytical methodology, what constitutes an appropriate analysis depends, in part, on the types of data and variables involved.

The scope of this framework, as applied to a particular IT project, could encompass a very wide range of data types. Many kinds of quantitative data from financial sources, operations research, and surveys are appropriate for statistical analysis, modeling, and simulations. Many of the social, political, and intrinsic value returns can be expressed in normalized scales, or may best be revealed in qualitative terms or in simple dimensions that are not suitable for much quantitative manipulation. To help guide the assessment, given the very large number of possible public return data types and variables, the framework provides two kinds of resources. The first is a general scheme suggesting variable types and sources for different value types (see Table 4). The second consists of summaries of the approaches and variables available in a range of existing ROI methods that can be of value in completing an assessment (see Table 5).

A wide range of possible public value data can be identified by the methods presented here. The framework approach is based on the assumption that virtually any kind of data can be useful in describing public value creation, from the most precise quantitative figures available from financial or physical measurements, to material as diverse as the content of blogs or observation of user or customer behavior. A conclusion about public value creation requires an inference, since value does not stalk about wearing a label. Valid inferences about value can be formed from qualitative as well as quantitative data, content analysis as well as statistics. Taking these four principles into account, choices made about how a specific analysis proceeds should be based on three criteria: 1) What constitutes the best data? 2) What kind of analysis is appropriate to that type? 3) Who will be the audience for the conclusions reached? The best kind of data available will be specific to the operational and stakeholder context. The kinds of analysis appropriate to various data types are shown in summary form in Table 4.(13)

Beyond these general considerations, the choice of variables and analyses for an assessment can draw on a volume of existing work on ROI methods for guidance. These methods, summarized in Table 5, vary widely in the number and type of variables used, the scope of public value considered, and the level of analytical detail and technique included. Some are intended for use prospectively, in planning for and justifying an investment. Others are aimed primarily at showing impacts of investments after the fact. They also vary in terms of the degree to which they deal with both internal and public value results of the investment, and whether they are designed specifically for IT or government investments generally. The summary of these methods or models in Table 5 can help in the selection of variables and analysis to fit the IT project.

The SROI (Social Return on Investment) model, the only private sector oriented one in the summary, is included for its special features. Its private orientation refers not to the commercial sector but to a private philanthropic orientation. The method was developed by a San Francisco foundation to assess its social and economic development programs. It illustrates not so much how to assess an IT investment but rather how to deal with personal and community impacts in a systematic and comprehensive way.

Government IT innovation is risky business. Evidence from many surveys and cases suggests that the threats and other challenges of IT projects often overwhelm the capabilities of the developers and implementers. Data from the private sector tells a similar story. For this paper, we focus our attention not on the full range of risks to government IT project success, but concentrate on the public return aspects of that problem.

The distinction between the public return and more generic aspects of risk analysis is not, however, a simple one. The approach is the same: identify and evaluate threats, develop and evaluate response methods, and produce a summary analysis and mitigation strategy. To identify and evaluate threats, we return to the overall value proposition schematic presented earlier (Figure 12).

Figure 12. Public Return on Investment Value Chain

In this figure, we identified two kinds of risk: 1) development risk and 2) benefit risk. Development risk, simply put, is risk that the development and implementation of the IT will fail outright or will not perform as designed and intended. Benefit risk applies to whether the IT investment will fail to produce the envisioned benefits in spite of being successfully developed and implemented. For the public value framework we focus on benefit risk. Of course any threat to the development of the IT is an indirect threat to public returns. However, several of the ROI methods described below include adequate analysis methods for development risk. These methods do not deal adequately, however, with the additional threats and issues in benefit risk.

The benefit risks associated with the creation and assessment of public returns come from threats to the creation of the returns and to their detection. And there appear to be two main sources of those threats: one is what we will call "theory failure" and the other is "exogenous factors." In theory failure, the underlying assumptions or theory on which the project is based are flawed or simply wrong. One such theory failure caused the US Department of Education to abandon a multi-million dollar pilot project for online college student financial aid administration. Developed without significant participation from college financial aid officers, the system did not attract supporters and generated much stakeholder resistance.(14) It is important to use both sources and types of benefits threats in a full risk analysis, as suggested by the two-by-two array in Table 6.

The risk analysis process can then use the threats identified in this way to estimate the potential loss or cost that each threat represents and the likelihood of the threat materializing. These estimates should be based to the extent possible on input from stakeholders, analysts, users, developers, managers, and policy makers. This will provide a basis for accurate estimates and concentrating attention on the threats with the highest combined loss-likelihood estimates. In some cases, the value, cost, and loss-likelihood estimates can be quantified to yield decision tools for moving ahead with an investment. The displays in Figure 13 and Figure 14 below, show such a result from the US Federal CIO Council's Value Measurement Method, which employs voting and other methods to prioritize and estimate quantitative values for cost and their related risks. The decision boundaries shown in the figures come from management decisions or some other deliberative process.(15)

The threats due to difficulty in detecting indirect or second-order effects deserve some added discussion. Public value outcomes can extend beyond those involved in the direct interaction with government. For example, drivers are the direct beneficiaries of an online license renewal system. However, the scope of possible indirect value outcomes and beneficiaries can be very broad. Residents in the neighborhoods of the license renewal agencies will benefit to some degree from reduced traffic congestion and pollution. Shopkeepers in that neighborhood may lose business for the same reasons. Second order effects may be even more diffused and difficult to detect. Learning to trust the online process for license renewal, for example, can result in more use and greater benefits from using other online services.

Since risks are tied both to stakeholders and possible value outcome variables, pursuing this line of reasoning can lead to a very large, and likely infeasible, list of risk analysis factors and tasks. To work within resource constraints will require limiting the risk analysis to the most important value variables and stakeholders. Setting priorities for this kind of analysis will therefore be essential, and must be based on the goals of the project and stakeholder value estimates.

The framework is a combination of individual analysis steps combined into an overall process or plan for public ROI assessment. A summary of that overall process is shown in Figure 15. As shown there, the process follows the arrows, starting with a clear understanding of the larger context of public value generation. The investment goals that emerge from the investment plans and understanding of the context link to the business processes and public value impact mechanisms. Analysis of the impact mechanisms and public return types leads to specific impacts listed in the stakeholder interest analysis. Identifying the interests and impacts will then guide choosing public value variables, analysis methods, risk analysis, and ultimately, reporting tools and techniques.

For simplicity, this process diagram is somewhat idealized. It shows only the basic linear path through the various steps. In practice, the actual path is rather more likely to have backtracking and loops. Learning at one step can easily lead to revisions in conclusions or data collected at preceding steps. This is particularly so for the risk analysis steps in the process. It is not feasible to complete the risk analysis for achieving any particular public value return until the details about that return and how to assess it are known. Loops can occur in the process when a step produces unexpected or challenging results. For example, identifying all related business processes for a large, complex project may produce too many results to carry forward in the analysis. That would require looping back to reduce the scope. So the potential user should view this process summary as a rough guide rather than a lock-step procedure.

The process also represents a comprehensive approach to public value assessment. We argue that all the steps are needed to ensure that the full range of public value returns will be included in any assessment and report. Some small or very narrow projects, of course, may have limited public value potential and not require such an extensive review. But the research and analysis leading to this framework have clearly demonstrated that the scope of public value possible from government IT investments can far exceed what an internally focused, financial assessment will reveal. Unless assessments use a much broader view, as represented in this framework and other models cited above, the full value of government IT investments will not come to light. It also seems likely that this same kind of analytical framework would be useful for other forms of government investments. We did not, however, explore that possibility for this report.

This framework for public value assessment presents both problems and opportunities for summarizing and reporting. The problems arise from the large number and types of results that the assessment can produce. For presentation to policy makers and non-technical audiences, the results should be as simple and accessible as possible. Simple charts and summary tables are best for this purpose. For multiple stakeholders and value variables, the number and complexity of charts may become a problem. This section discusses some of the specific issues and alternative methods available.

For qualitative variables, the presentation of results can show the presence of a value result, and information about magnitude and direction, if relevant and available. Using the information in Table 3, a summary display similar to Table 7 below can present the types and direction of results, with the estimated relative magnitudes as well. The down arrow in the stewardship row for vendors indicates a potential loss to vendors due to transparency increases that diminish opportunities for some forms of influencing procurement. Whether this is a positive or negative result overall may not be clear.

For the more quantitative results, where the public value variables lend themselves to calculation and statistical analysis, many presentation and summary methods are available. Clear and simple visuals are generally preferable to tables of quantitative data for non-technical audiences. For cross sectional data, a column chart, such as in Figure 16, can be used to present the same comparative public value data converted to a ten-point index or scale.

Figure 16. Stakeholder Impact Column Chart

Such a multidimensional chart can be difficult to interpret, however, and some ROI methods use the so-called radar chart for the same data, shown in Figure 17. This kind of chart provides a clearer image of the pattern of results for each stakeholder type and value dimension. However, this kind of display becomes much more difficult to interpret if the number of axes or stakeholders is large.

Figure 17. Stakeholder Impact Radar Chart

The use of indexes can also be used to compare the relative risk and value of alternative investments on a value scale if the variables can be combined into a single index. The Demand and Value Assessment Model (Australia) provides guidance on how to produce such a public value index and use it for comparison purposes. An example of that kind of result is shown in Figure 18 below.

Figure 18. Portfolio Risk and Value Comparisons - DVAM

For cross sectional assessments, more elaborate presentations are available, particularly if the value variables are indexed or based on quantitative data. The Accenture Public Service Value Model (PSV) can use historical data about government program effectiveness and costs to show changes in performance over time.16 The model is based on the principle that public value is created when both outcome results and cost effectiveness increases. An example of this form of results presentation is shown in Figure 19 for the Arizona Department of Revenue. Overall, the performance shown is improving from 1999 to 2001 and 2002 to 2003 when the organization is creating increased outcomes and doing so more cost effectively.

Figure 19. Historical Performance Change Model (PSV - Accenture)

Documentation of results through this kind of chart or visual device should include background material about methods and measurement issues. This framework advocates the use of a wide range of data and analytical styles, many of which are considered controversial or suspect in some environments. Decision makers and analysts often have deep-seated biases about the validity of qualitative data or social statistics, or other non-traditional material for ROI analysis. Those performing a public value assessment must, therefore, be thorough in providing rationale and supporting material for all results. They must be attentive to the issues of interpretation and validity that may affect how key members of their audience respond to the assessment results.

The principles for SROI analysis used by the REDF organization provide valuable guidance for the conduct of the kind of assessment described here. These principles can be applied to traditional ROI analysis as well, but seem particularly well suited to the public value issues involved with this framework and related methods. The principles make an appropriate bridge from the general ideas and methods presented here to the difficult work of carrying out public value and ROI assessment in practice.

SROI Design Principles: (17)