Appendix: The Research Basis for the Public Value Framework
A. Consultative Workshop
On July 13, 2005, the Center for Technology in Government, in collaboration with SAP, hosted a consultative workshop on assessing public return on government investments in IT. A select international group of 22 experts from government, academia, and the private sector were invited to meet with researchers from CTG to discuss the core issues and themes that make assessing public ROI for IT investments such a complex and difficult problem. The workshop results were used to inform CTG's next steps for the project. The workshop results were used to help develop a preliminary framework for conducting ROI analysis in the government sector and design the case studies research approach. This approach was employed to identify and select the five case studies, which were used to refine the preliminary framework and develop the white paper. The lists below summarize the key issues that emerged from the workshop that characterize the complexity of the issue. Workshop participants provided their recommendations in the form of key themes for CTG to explore in the follow-on research.
1. Key Issues Characterizing the Complexity of Assessing
Public ROI for Government IT
- Lack of incentives to assess public ROI. There may be no consequences for absence of ROI or other demonstration of results.
- Lack of historical perspective and data.Government tends to be prospective (not retrospective), so it tends to focus on what should be done, but not on what has already been done.
- Governments have trouble harvesting savings, which often get moved around the budget.
- There is no straightforward quantitative bottom line value measure for ROI in public sector.
- Government is multidimensional. Non-linear, complex interactions among benefits—hard to measure results and link to specific programs or technologies, i.e., proving the causal relationship between the two.
- ROI requires advanced project management and portfolio management skills that are often lacking.
- Comprehensive ROI analysis can greatly increase transparency of government decisions and investment results. This level of transparency is a firiskfl in and of itself, increasing the probability for embarrassment and criticism leading to loss of support.
- ROI is done in a vacuum. Not focused on the strategic investment.
- Lack of longer-term tracking and assessment makes it difficult to build a measurement model.
- It is hard to evaluate IT ROI elsewhere in the government enterprise because the outcome frameworks (intersectors) aren't established.
2. Themes to Explore in the Research
- Value and impact measurements should take into consideration the cost impacts on other business processes, by elimination or changes in the way we work together.
- Look for efficiency and cost-reduction in non-IT areas. IT is an enabler, and the purpose of IT is to enable other things, including changing the culture of organization, strategies, etc. Thus, it is important to see how IT is enabling returns in other areas.
- Assessment should include attention to how particular IT investment enables IT elsewhere. ROI analysis must get beyond evaluating things on a more traditional basis to include questions of outcomes for broader range of beneficiaries or stakeholders.
- Social cost-benefit analysis and political considerations involve different people making different decisions. It is important to understand how this may impact the IT aspect of ROI calculations.
- Need more attention to risks as well as benefits. Assessment should include attention to particular government sensitivity to risk factors.
- Method development should include system analysis that engages a broad scope of operations. Analysis should include questions about leadership, feasibility, political support as well as results/outcomes, how constituents and opponents will react.
- Focus assessment on question of getting ROI on the programs, not the IT; IT doesn't deliver the outcomes.
- Portfolio management is a big theme in current discussion of IT management. As applied to public ROI, implies looking at the overall picture, not just a slice of the project.
- ROI methods must accommodate or provide for shared outcomes that cross over existing stovepipes.
- While benefits are measured in terms of traditional (financial), political and social factors, risks are measured based on other factors: (1) technical,
(2) organizational, (3) time, and (4) political.
- Find relationships between inputs and outcomes. Once indicators are devel- oped cause and effect relationships can be explored using econometric methods.
- Value of IT investment may be enabling change
- Find ways to ensure follow-through in delivery and assessment of government programs.
- Explore what are the attributes of a good public ROI model: measurable in different ways, creates expectations of performance and assessment at both IT and program levels, is integrated with the budget process, and is included in a yearly review.