On July 13, 2005, the Center for Technology in Government, in collaboration with SAP, hosted a consultative workshop on assessing public return on government investments in IT. A select international group of 22 experts from government, academia, and the private sector were invited to meet with researchers from CTG to discuss the core issues and themes that make assessing public ROI for IT investments such a complex and difficult problem. The workshop results were used to inform CTG's next steps for the project. The workshop results were used to help develop a preliminary framework for conducting ROI analysis in the government sector and design the case studies research approach. This approach was employed to identify and select the five case studies, which were used to refine the preliminary framework and develop the white paper. The lists below summarize the key issues that emerged from the workshop that characterize the complexity of the issue. Workshop participants provided their recommendations in the form of key themes for CTG to explore in the follow-on research.

• Lack of incentives to assess public ROI. There may be no consequences for absence of ROI or other demonstration of results.
• Lack of historical perspective and data.Government tends to be prospective (not retrospective), so it tends to focus on what should be done, but not on what has already been done.
• Governments have trouble harvesting savings, which often get moved around the budget.
• There is no straightforward quantitative bottom line value measure for ROI in public sector.
• Government is multidimensional. Non-linear, complex interactions among benefits—hard to measure results and link to specific programs or technologies, i.e., proving the causal relationship between the two.
• ROI requires advanced project management and portfolio management skills that are often lacking.
• Comprehensive ROI analysis can greatly increase transparency of government decisions and investment results. This level of transparency is a firiskfl in and of itself, increasing the probability for embarrassment and criticism leading to loss of support.
• ROI is done in a vacuum. Not focused on the strategic investment.
• Lack of longer-term tracking and assessment makes it difficult to build a measurement model.
• It is hard to evaluate IT ROI elsewhere in the government enterprise because the outcome frameworks (intersectors) aren't established.

• Value and impact measurements should take into consideration the cost impacts on other business processes, by elimination or changes in the way we work together.
• Look for efficiency and cost-reduction in non-IT areas. IT is an enabler, and the purpose of IT is to enable other things, including changing the culture of organization, strategies, etc. Thus, it is important to see how IT is enabling returns in other areas.
• Assessment should include attention to how particular IT investment enables IT elsewhere. ROI analysis must get beyond evaluating things on a more traditional basis to include questions of outcomes for broader range of beneficiaries or stakeholders.
• Social cost-benefit analysis and political considerations involve different people making different decisions. It is important to understand how this may impact the IT aspect of ROI calculations.
• Need more attention to risks as well as benefits. Assessment should include attention to particular government sensitivity to risk factors.
• Method development should include system analysis that engages a broad scope of operations. Analysis should include questions about leadership, feasibility, political support as well as results/outcomes, how constituents and opponents will react.
• Focus assessment on question of getting ROI on the programs, not the IT; IT doesn't deliver the outcomes.
• Portfolio management is a big theme in current discussion of IT management. As applied to public ROI, implies looking at the overall picture, not just a slice of the project.
• ROI methods must accommodate or provide for shared outcomes that cross over existing stovepipes.
• While benefits are measured in terms of traditional (financial), political and social factors, risks are measured based on other factors: (1) technical, (2) organizational, (3) time, and (4) political.
• Find relationships between inputs and outcomes. Once indicators are devel- oped cause and effect relationships can be explored using econometric methods.
• Value of IT investment may be enabling change
• Find ways to ensure follow-through in delivery and assessment of government programs.
• Explore what are the attributes of a good public ROI model: measurable in different ways, creates expectations of performance and assessment at both IT and program levels, is integrated with the budget process, and is included in a yearly review.

Some government IT initiatives focus narrowly on a specific technical problem, like enabling mobile data communication or Web-based transactions. Others are driven by a broader, more ambitious goal, to use IT as a tool to transform government. The Merkava Project in the Government of Israel (GOI) is most certainly one of these transformative efforts. It is transformative in its own right, as an effort to restructure the financial, logistics, and human resource components of governmentwide administra- tion into an integrated ERP system. Merkava is also part of a comprehensive e-govern- ment initiative that includes the ERP system as one of five layers of new technologies and operational systems for enhanced internal operations and improved benefits and services to citizens. These layers are part of a multi-year strategic plan, described in 2002 by Prime Minister Sharon: "The e- government project is an expression of the government goal to provide state-of-the-art strategic systems, while affecting social and economic national targets. We believe that this infrastructure constitutes a better tomorrow and that tomorrow is already here."

From its beginnings in 1999 to the present, the Merkava ERP project has been a very large undertaking. It is described by the Ministry of Finance as the largest single IT project in Israeli history. It has been implemented or is in the roll-out process in 30 government units (out of 100), and currently supports approximately 2,300 users. The 2005-2007 IT master plan calls for the Merkava system to be implemented in 90% of the government's offices during the planning period. Work on the other four layers in the overall eGov initiative has been progressing in parallel with the Merkava implementation. The integration provided by the Merkava ERP provides an important part of the infrastructure for operations in the three upper layers of the initiative and contributes to the returns that they deliver as well. The success of implementation efforts to date and the growing record of returns, both internal to the government and the public, suggest that the remaining roll-out will continue as planned and the accumulation of returns will continue to grow.

How a government obtains value from its citizens and external public environment and how it returns value to them are linked directly to its financial management systems. These systems are crucial links in the flows of revenues to the government and the flow of expenditures and services back to the public. Improving financial management, therefore, has the potential to produce significant returns in terms of both greater internal efficiencies and enhanced public returns. These were the goals of the Austrian Federal Budgeting and Bookkeeping System project initiated in 1997 by the Minister of Finance and supported by the Chancellor (Prime Minister). The goal of the project was to redesign and integrate the electronic workflow of the federal government's budget and bookkeeping processes. The strategy they chose was to implement a single ERP software standard throughout the federal government, along with the adoption of necessary legal authority. The result was that, by 2005, the Ministry of Finance successfully consolidated 85 bookkeeping units across the federal government into one federally owned, but privately operated, agency.

The consolidation and integration produced immediate and tangible benefits in terms of internal efficiencies. These resulted from the implementation of a new standardized work process for accounting and budgeting throughout the federal government, with reduced work process steps and processing time. As of 2005, the legal consolidation of the numerous bookkeeping departments into one agency, along with the technical and organizational implementation of the ERP, has resulted in annual savings of approximately 30 million. In terms of these returns to the government itself, the project is clearly a resounding success.

In terms of broader public returns, the project goals went beyond internal financial management efficiencies. The aim of restructuring the Austrian Federal Accounting and Bookkeeping system was much more than simply an fiIT investmentfl from the very beginning. The bookkeeping system was part of a larger effort to imple- ment the SAP ERP technology throughout the Austrian Federal Government as part of a governmentwide public management reform effort. The ERP technology was one part of a comprehensive strategy that included legislative reform, governmentwide reorganization and consolidation, and implementing a modern accounting and budgeting standard across the government. Though the specific ERP technology was just one element of a larger approach, involving legal and organizational strategies, the ERP technology played a very important enabling role vis-à-vis the legal and organizational strategies.

Washington State's investment in digital archiving for government records provides a highly focused and successful example of pursuing public value through information technology. The job of collecting, preserving, and providing access to the records of government is central to the mission of Washington's Office of Secretary of State. That mission recognizes the fundamental importance of government record keeping in a democratic society. That is also the foundation of the public value proposition guiding the Digital Archives program: that the state has the constitutional and statutory mandate to preserve and provide access to records of enduring legal and historical significance.

The growth of electronic records in government agencies in the 1990's pre- sented a challenge to the State Archives' ability to fulfill its mission, since it lacked an effective program and technology to deal with records in these new formats. The Washington State Digital Archives (WSDA), a program within the Office of Secretary of State, is the response to that challenge. It was initiated by the Office of the Secretary of State, with initial planning begun by the then State Archivist in March of 2000. The initiative was taken up in 2001 as a priority by the newly elected Secretary of State Sam Reed, and included in the Secretary of State's 2001-2007 Strategic Plan. The project was subsequently supported by the state legislature and included in the State of Washington's 2001-2003 Capital Budget. Construction of the physical hub of the WSDA in Cheney, Washington, began in January 2003.

Beginning in mid-2001, the WSDA team began exploring a wide range of technologies and techniques for collection, access, and preservation. The results led to the custom development of a Web interface and database design that blended the latest technologies with traditional archival theory to create a first-of-its-kind digital records repository for state government. The goal of the program was to make the historical electronic records of Washington's state and local governments easily accessible, from anywhere, at anytime. The initial vision and value proposition were carried through a complex political and technical process to a functioning digital archiving program and facility delivering the promised public value. The WSDA project team began with a clear vision of the expected value of the Digital Archives to both the government and citizens. In addition, it successfully identified the benefits that it needed to communicate to the state and local government agencies that were keepers of public recorders in order to mobilize their support and participation. The WSDA project demonstrates a strong connection between the initial high-level public value proposition that motivated the project and its realization in the performance of WSDA itself.

From the beginning, the Commonwealth of Pennsylvania's Integrated Enterprise System was much more than a technology project. The ERP implementation continued through three gubernatorial administrations with consistent top level executive support; eventually putting in place the technical infrastructure and enterprise standards for core administrative functions with improved public value. Between early 2001 and mid-2004, the ERP implementation for the five business functions was completed for 53 Commonwealth agencies including all 49 of the agencies under the governor's jurisdiction.

Immediate returns in the form of improved government operations were realized soon after implementation and continue today. However, this infrastructure also provides the Commonwealth with capability that can be further leveraged to support additional improvements in government operations that go well beyond direct improvements in core administrative functions.

The Commonwealth has begun such efforts. The IES infrastructure provides public returns in the form of direct improvements in the efficiency and effectiveness of core administrative functions. This infrastructure also provides the basis for improvements in the back office operations of other service areas which, in turn, offer improved services to the public. The Commonwealth has recently taken steps to move in this direction by implementing the necessary institutional structures and policies to take fuller advantage of this enterprise-level asset.

Service New Brunswick (SNB) is well-known internationally for its expertise in providing multi-channel fisingle windowfl citizen access to government services as well as for developing and maintaining geographic information databases. SNB's award- winning approach provides one-stop- shopping for different government services on behalf of provincial and municipal government agencies, but also provides a linkage to the Canadian Federal Government in a fijoined upfl government model. SNB was launched in a time of high pressure from citizens for improved service delivery; today it serves the province through its award winning service delivery model, and also and maybe more importantly in the long run, through its innovations in economic development.

The numbers behind SNB speak for themselves; customer satisfaction numbers are the highest in Canada going from about 50% in late 1980's to 92% in 2005. These numbers require that responsible examinations of public value include SNB. But there is more to this story than customer satisfaction ratings. The essential part of the public value story in this case is Service New Brunswick (SNB) as a quasi- governmental organization and its evolution as an integrated service provider and economic development innovation. SNB is a public corporation with a single shareholder - the government of New Brunswick. This shareholder has guided SNB with this model throughout its existence: make public service good business.