Learning to be “up front” with information technology investment decisions

IT investment decisions are risky business. They are costly, rife with complexity, and just plain error-prone. Here is how one agency is learning to meet the challenge.

• Introduction
• The place of IT investment in DOT
• Changes in IT decision making and planning
• The importance of "up front" work
• IT investment decisions demand much more than technical information
• Standards and formal procedures help with some tasks
• It takes more than rules and procedures
• For best results, invest in the investment process
• A strategy that links business needs to information to funding to action
• And it's not over yet

Introduction
Decisions to invest in new information technology (IT) are some of the toughest ones any administrator or agency faces. When the agency is a very large and diverse one, like the New York State Department of Transportation (DOT), the challenges multiply. Over the past two years, DOT improved both the quality and amount of information available for its IT investment decisions. Their story focuses in particular on what the Department came to call the "up front" parts of planning and information gathering, the parts that relate to the business needs that IT will support. Their strategy also included a major shift in decision-making policy and practice. As a result, the roles of the main participants in investment decisions changed in ways that complemented the improvements in the data and the process. The overall result is a major revamping of the entire IT project review and selection process that is better aligned with the agency's program/business goals and directly connected to its budget decisions.

Read full case

The place of IT investment in DOT
The scope of information technology needs in DOT is very broad, due to the size of the Department's operations and the diverse activities involved. DOT's responsibility for the state's transportation network is enormous. It includes:

• A state and local highway system which annually handles 100 billion vehicle miles. This total system encompasses over 110,000 highway miles and 17,000 bridges.
• An extensive 5,000 mile rail network over which 42 million tons of equipment, raw materials, manufactured goods, and produce are shipped each year.
• 456 public and private aviation facilities through which more than 31 million people travel each year.
• Over 130 public transit operators, serving more than 5.2 million passengers each day.
• 12 public and private ports which handle more than 110 million tons of freight annually.
• Appropriations for the 2000-01 fiscal year of just under $10 billion

The Department operates out of a headquarters in Albany, 11 regional offices, and 68 county offices. It uses IT for a wide range of work, from routine administration and finance to project management, engineering, materials research, mapping, and complex contracting and procurements. Consequently, the information needs of IT planners and decision makers are highly demanding and dynamic.

Read full case

Changes in IT decision making and planning
DOT's current emphasis on front-end planning and improved decision making has its roots in changes made during late 1998 and early 1999. One key change was in the creation of a new Department-level group responsible for selecting IT investment projects. In late 1998 the Department created a new IT Council, replacing the previous MIS Steering Committee. The Steering Committee was active during the development and management of a Management Information Systems (MIS) Plan and members were high level executives who were primarily interested in setting policy and were somewhat removed from the operational and technical issues of IT projects. The IT Council (ITC), by contrast, is composed of program and regional managers whose perspective is much closer to the operational goals and impacts of IT projects. Their involvement is based on more programmatic and business process concerns.

A second major change involved the development of a new process to review IT projects as long-term investment decisions, not just short-term cost factors. The Department produced a new Interim Information Technology Policy and Procedure in January 1999 that reflects the investment point of view. The adopted Select-Track-Evaluate overarching framework includes a three-step selection process described as Screen, Score, and Select. The initial Screen step involves an IT Triage Committee, composed of selected IT, finance, program, and analysis staff. This Committee's job is to work with project sponsors to do a preliminary review and send the project proposal down one of three possible paths:

1. if small (less than 500 hours of development) and viable, it can be approved and moved to implementation or further scoping and analysis,
2. if non-viable, it is sent back to project sponsors with recommendations for additional work or it is referred to the IT Council for final decision, or
3. if large and viable, it would move forward to the Score/Select stages.

A major portion of these stages involves the development of a Business Case which includes a business alignment section, a financial analysis section and a comprehensive risk self-assessment of the proposed project.

The IT investment process includes a Technical Architecture Review by the Information Systems Bureau (ISB) technical managers to determine if the proposed project IT architecture is consistent with Department standards and IT infrastructure. The results are sent to the IT Council for consideration in the Select phase. This Technical Review also evaluates the impact of proposed projects that would use non-standard technical elements and also reports these findings to the IT Council.

The Business Case, the results of the financial analysis, the scores from the risk self-assessment, and the results of the Technical Architecture Review all go to the IT Council for consideration in the Select phase. In this Select phase each project sponsor completes a presentation on the proposed project. The IT Council then uses this information along with the other collected data to rank/prioritize all the project proposals and either moves them forward for funding recommendations or sends them back to their sponsors for additional work. This review process involves a standardized way of collecting and presenting data about the proposed investment project. The process is based on the Information Technology Investment Process(ITIP) developed for the federal government and has much in common with processes used in other states to select IT projects.

A third major shift in decision making is more recent, involving the budget procedures for the 1999-2000 fiscal year. Previously, there was no comprehensive, all-inclusive Department-wide budget for IT projects. Instead projects were funded from program allocations and often did not represent a Department-wide view of IT requirements and needs. Under the new system, there is a separate IT budget and IT spending plan for the Department, administered by the IT Council, resulting in a broader, more comprehensive view of the total IT needs of the Department.

A fourth change involved the creation of the in-house title of Business Account Manager (BAM). These positions are responsible for providing the link between the business units and the IT functions. One of their primary functions is to assist the business units in developing IT initiatives and coordinating the technical support aspects of project development. All full-time Business Account Managers are drawn from the program areas, so they know the business side of the programs and are also familiar with, but not necessarily expert in, the technical issues. These positions provide a key link between the business functions and the IT support activities and are an important ingredient in making the IT investment process work in DOT.

A final major change was in the increased concern for the planning and analysis of the business problem involved in the IT investment. As one participant described it, the old approach was, "Here's the problem. Here's the solution. Let's start coding!" In the newer perspective, more attention to the underlying business process and strategic context of the problem gets first attention; it becomes the "up front" work central to the final investment decision. To support that kind of work the Information Services Bureau (ISB), which supports the entire IT investment process, created new guidelines for IT project planning and business case analysis, and instituted a pilot training project for staff.

Read full case

The importance of "up front" work
"Up front" work is what the ISB staff began calling the new kinds of analysis they felt were needed to improve the quality of IT project initiatives. "Up front" work is what's necessary to understand the business problem before proposing any particular IT solution. In the old decision-making process, units within DOT developed proposals to describe and justify a particular IT procurement or solution. The proposals usually focused on the costs and technical functionality of the solution. But the proposals rarely took into account the Department-wide view necessary for sound IT investment planning and were produced with little analysis of the underlying problem the investment was intended to solve or its relationship to the rest of the unit's or agency's work. That made it very hard to judge how well the proposed solutions would really work, making it difficult for the Department executive management staff to choose which proposals to support. Because Department management and ISB technical staff wanted more "up front" information about the target problem or need and its relationship to business processes, the proposal process had to change.

The change was based on a two-part strategy. One part was to develop new tools and supporting materials to help proposers generate and organize better information about their IT investment ideas with particular emphasis on clearly identifying and prioritizing business needs and problems. For this goal the Information Services Bureau created a new, comprehensive guide for preparing proposals and organized a pilot training program for a sample of agency staff. Their goal was to improve investment decisions by:

• expanding the types of the information needed to justify an initiative
• improving methods for collection and processing that information
• enhancing staff skills with new methods of developing and interpreting information

The other element of the strategy was to engage in a training process for agency staff that brought the business and process questions to the forefront. The training approach resulted from the ISB staff's familiarity with the "Making Smart IT Choices" publication and related training activities of the Center for Technology in Government.

Read full case

IT investment decisions demand much more than technical information
The primary data needed for these investment decisions comes from the proposals submitted by the units requesting the new investment. They might come from engineering or any of a dozen other program units within DOT. To help the IT Council and agency executive management better understand the business impact of initiatives and make better investment decisions, the staff reviewers wanted to improve the quality of information in these proposals. One major improvement they sought would result from adding new kinds of analysis to the proposal preparation process: the "up front" part of the work. This idea was prompted in large part by the experiences of some Information Services Bureau staff who participated in a workshop based on the CTG program of Making Smart IT Choices. This workshop emphasized careful attention to the strategic context and the business problem for any IT investment. ISB staff reasoned that if those who prepared the proposals had similar training, they could then expand the scope of analysis and improve the resulting information provided in the proposals. They expected such training would prompt many new questions and data not previously required or normally present in the old-style proposals. These included questions about:

• the service or business objectives involved in the new technology
• the identity and interests of stakeholders, including possible benefits and costs to each
• how to describe the business processes involved
• the strategic linkages and likely partners in the implementation
• performance measures for the new system and processes
• the organizational and political risk factors that could influence the effectiveness of the project

Answers to these and related questions would result in new information becoming available to the proposal reviewers and the ultimate investment decision makers.

Standards and formal procedures help with some tasks
The Information Services Bureau staff initiated another approach to improving data that worked along with the training and the need for expanded front-end analysis. They also adapted the information structure and analysis approach from the Information Technology Investment Process (ITIP) which is an IT investment approach being used in the federal government and a number of states. This approach describes the documentation that project sponsors are ultimately responsible for providing to the IT Council to justify an IT investment initiative. The documentation includes five profiles covering the main components of the project:

• business profile, including the business alignment case and business process analysis opportunities and efforts
• risk profile, including risk assessment and mitigation plans
• financial profile, including ROI and cost/benefit analysis
• technology profile, including compatibility with existing infrastructure and systems
• management and planning profile, including both project and acquisition plans

The profiles provide a formal framework of standardized data elements for evaluating IT investment proposals.

Read full case

It takes more than rules and procedures
Much of the information and insights needed to complete the profiles and scoring in assessing IT investments comes from or is based on good "up front" analysis. That is, a highly structured and formal approach to facilitate proposal presentation is no better than the basic understandings and skill of the people using it. Improving the quality of information and analysis going into investment decisions depends heavily on the skills, knowledge, actions, and relationships of a wide range of staff doing the work. The training program the ISB offered was a start in developing and transmitting this crucial message to the proposal preparation teams. It was a beginning in changing attitudes and organizational culture to support the new rules, procedures and process. The training provided the tools and techniques to create and analyze the more complex and diverse information that underlies the IT proposals. The result is a more comprehensive and useful mix of information, better suited to the complexities of the decision making situation.

The training component of the new approach had other features that facilitated the acceptance of new methods and techniques by the participants. There is typically some resistance in any organization to the introduction of new processes and ways of doing business. The participants in this training had an opportunity to interact with each other and the trainers. That interaction provided an opportunity to discuss content and clarify uncertainty. These experiences reduce resistance to change and provide useful feedback for the trainers. The training brought together staff from different areas of the Department. By working together they were exposed to a variety of different perspectives on how business problems are analyzed and how this analysis can affect IT project initiative development. As one participant said,

… I find myself, now that I've taken this class, spending much more time with the regional and the residency personnel, listening to them and ensuring that we can meet their needs, trying to develop alternatives of how they do business. So it has kind of opened my eyes to the point that there is more than just myself involved in this, and that other people's interests are just as important.

Perhaps the most important effect of the training is that the participants developed skills and insights that sharpened their focus on information about the business problems and organizational processes involved. This training approach led the participants to define problems to be solved in terms of the business processes and goals of the agency program, not in terms of technology. In describing their response to the training, the participants emphasized the importance of clearly understanding the program need or business problem. As one training participant put it:

I run into a lot of people who are really not that familiar with technical IT systems, and they try to focus on the part that is technical instead of the real world part. So what they do is they look at a system developer and they say, "Solve my problems" without them actually saying what their problems are.

This focus on "up front" information was institutionalized in the practices of the Triage Committee, the group charged with recommending which IT project proposals would go forward for full IT Council review. This Committee began requiring "up front" business process analysis information as a routine part of proposals. This included suggestions for stakeholder analyses to show the impacts on a wide range of possible stakeholders. In addition, the Committee members put their "up front" planning questions into a routine that carried problem and business process-related questions forward from one meeting to the next. This change in information requirements initially met with some resistance from some units presenting proposals but this attitude is changing as the investment process and the need for this kind of front-end analysis work becomes institutionalized. As a filter of proposals moving forward to the IT Council full review stage or in its role as helping sponsors develop better business aligned initiatives, the Committee was in a strong position to convince the sponsors of the need for this kind of front-end analysis and documentation.

The training participants emphasized how their perspectives had been changed. They came to see the importance of information about the program or business perspective in all parts of IT planning. They often repeated the desire for program managers to be included in the training and to develop the same perspective. It should not be limited to technology specialists, Information Services Bureau staff, or administrative staff. One technology specialist described how this training for program managers would help them take a broader view of IT projects and developments. These managers, she reported, often define IT projects to solve a narrow, highly specific problem in a way that is not integrated with larger program issues and linkages. If the program managers took a broader view, IT projects could be designed in an integrated way that produced greater benefits or avoided conflicts or incompatibilities with other IT systems.

The training experience also changed some views about the information needs across a variety of projects and units. The participants expected the training to be highly structured, focusing on generic skills, tools, and analyses. They thought information needs would be more consistent across projects and units of the agency. After the training they expressed an appreciation for the diversity of problems, business processes, and information needs across projects. They recognized that the basic approach, involving questions about the business process and nature of the underlying problem, does remain fairly consistent. But each project and organizational setting requires its own analysis. In the training, participants noted the importance of involving program managers and other specialists in future training to be sure their perspectives are part of the analysis.

Read full case

For best results, invest in the investment process
The primary cost of this transition to a new investment process is staff time. The expanded requirements for proposal preparation have two cost components. The requirements call for more information than in the past, demanding more complicated gathering and analysis techniques. Eventually it will be necessary to involve larger numbers of program and IT staff in training to prepare them for these expanded analysis requirements. Considerable training will be necessary to bring all staff involved in IT investment development up to speed. The pilot training activity for this case involved only 10 DOT staff directly. But it nonetheless took 10 person-days of preparation and delivery for the trainers, over 20 person days of DOT staff time in the training and follow-up activities, and an unknown amount of additional time for early attempts to integrate the new techniques into the IT development process. When extended to the whole department, these become very significant costs, even though they may not have an obvious budget line. These could be considered extra costs, but DOT chooses to view them as part of the new investment process itself. They expect this additional staff effort will be rewarded in better choices and more successful and cost-effective systems.

Read full case

A strategy that links business needs to information to funding to action
In less than two years the Department has significantly changed IT investment decision making. These changes combine new information requirements with fundamental changes in governance. The pilot training component addresses the importance of staff skills and attitudes in improving information use and analysis. The new proposal review procedures and information standards address the need for higher quality, comparable information to evaluate proposals. The changes in the governance structures and policies complete the link between the substance of IT investment decisions and organizational action. DOT used a multi-part strategy to improve both the availability of information for decision making and the decision-making process itself. This strategy was operationalized in a new planning document called Instructions for the Development of a Technology Investment Business Case: A Standard for Information Technology Projects. This standard calls for a detailed business assessment as part of IT investment proposals. The business assessment guide gives operational meaning to the expectation for better "up front planning." The planning document also calls for a detailed self-assessment of risk that includes a wide range of factors for the planners to take into account. By requiring the use of a standardized format for collecting, evaluating, and transmitting IT investment data, the ISB staff established a new and consistent information framework for a large portion of proposal content. This helps them evaluate and compare proposals more objectively. Placing budget allocation authority in the IT Council is the third critical piece. Since the IT Council consists of program and regional managers, their budget recommendation decisions reflect concern for the Department-wide programmatic and operational value of IT investments. And since they review and prioritize IT proposals as a group, these budget decisions can also result in better coordinated and integrated IT investment across the whole Department.

Read full case

And it's not over yet
This has been an incremental process, using multiple tactics and adjustments along the way. The relatively modest exposure of ISB and other program staff to new ways of doing "up front" work has already had a major impact, largely because the ISB took full advantage of its strategic position in the overall Departmental approach to IT investment decisions and because of the involvement and cooperation of the program staff. The new approaches to "up front" work came along with new decision structures and policies, all working together. But the overall process is far from finished. The Information Services Bureau staff continues to seek improvements in information resources and proposal development processes. There is active consideration of extending the training in "up front" analysis to all program staff who are actively involved in developing IT initiatives or concepts, to enhance their abilities in problem definition and business process analysis. They are considering implementing the Information Technology Investment Program(I-TIPS) application, which is a software package developed for the federal government which serves as a management tool and repository for IT investment information, to further support proposal development, project tracking and evaluation. Rather than a one-time reform effort, DOT is engaged in a continuous improvement process. This approach can help ensure that as the IT investment decisions get tougher and more complex, the quality of information available for decision making can keep pace and ultimately result lead to better project and business results.

Read full case