Learning
to be “up front” with information technology investment decisions
IT investment decisions are risky business. They are costly, rife
with complexity, and just plain error-prone. Here is how one agency is
learning to meet the challenge.
Introduction
Decisions to invest in new information technology (IT) are some of the
toughest ones any administrator or agency faces. When the agency is a
very large and diverse one, like the New York State Department of Transportation
(DOT), the challenges multiply. Over the past two years, DOT improved
both the quality and amount of information available for its IT investment
decisions. Their story focuses in particular on what the Department came
to call the "up front" parts of planning and information gathering,
the parts that relate to the business needs that IT will support. Their
strategy also included a major shift in decision-making policy and practice.
As a result, the roles of the main participants in investment decisions
changed in ways that complemented the improvements in the data and the
process. The overall result is a major revamping of the entire IT project
review and selection process that is better aligned with the agency's
program/business goals and directly connected to its budget decisions.
The
place of IT investment in DOT
The scope of information technology needs in DOT is very broad, due to
the size of the Department's operations and the diverse activities involved.
DOT's responsibility for the state's transportation network is enormous.
It includes:
- A state and local highway system which
annually handles 100 billion vehicle miles. This total system encompasses
over 110,000 highway miles and 17,000 bridges.
- An extensive 5,000 mile rail network
over which 42 million tons of equipment, raw materials, manufactured
goods, and produce are shipped each year.
- 456 public and private aviation facilities
through which more than 31 million people travel each year.
- Over 130 public transit operators,
serving more than 5.2 million passengers each day.
- 12 public and private ports which handle
more than 110 million tons of freight annually.
- Appropriations for the 2000-01 fiscal
year of just under $10 billion
The Department operates out of a headquarters
in Albany, 11 regional offices, and 68 county offices. It uses IT for
a wide range of work, from routine administration and finance to project
management, engineering, materials research, mapping, and complex contracting
and procurements. Consequently, the information needs of IT planners and
decision makers are highly demanding and dynamic.
Changes
in IT decision making and planning
DOT's current emphasis on front-end planning and improved decision making
has its roots in changes made during late 1998 and early 1999. One key
change was in the creation of a new Department-level group responsible
for selecting IT investment projects. In late 1998 the Department created
a new IT Council, replacing the previous MIS Steering Committee. The Steering
Committee was active during the development and management of a Management
Information Systems (MIS) Plan and members were high level executives
who were primarily interested in setting policy and were somewhat removed
from the operational and technical issues of IT projects. The IT Council
(ITC), by contrast, is composed of program and regional managers whose
perspective is much closer to the operational goals and impacts of IT
projects. Their involvement is based on more programmatic and business
process concerns.
A second major change involved the development
of a new process to review IT projects as long-term investment decisions,
not just short-term cost factors. The Department produced a new Interim
Information Technology Policy and Procedure in January 1999 that reflects
the investment point of view. The adopted Select-Track-Evaluate overarching
framework includes a three-step selection process described as Screen,
Score, and Select. The initial Screen step involves an IT Triage Committee,
composed of selected IT, finance, program, and analysis staff. This Committee's
job is to work with project sponsors to do a preliminary review and send
the project proposal down one of three possible paths:
- if small (less than 500 hours of development)
and viable, it can be approved and moved to implementation or further
scoping and analysis,
- if non-viable, it is sent back to
project sponsors with recommendations for additional work or it is referred
to the IT Council for final decision, or
- if large and viable, it would move
forward to the Score/Select stages.
A major portion of these stages involves
the development of a Business Case which includes a business alignment
section, a financial analysis section and a comprehensive risk self-assessment
of the proposed project.
The IT investment process includes a Technical
Architecture Review by the Information Systems Bureau (ISB) technical managers to determine if the proposed
project IT architecture is consistent with Department standards and IT
infrastructure. The results are sent to the IT Council for consideration
in the Select phase. This Technical Review also evaluates the impact of
proposed projects that would use non-standard technical elements and also reports
these findings to the IT Council.
The Business Case, the results of the
financial analysis, the scores from the risk self-assessment, and the
results of the Technical Architecture Review all go to the IT Council
for consideration in the Select phase. In this Select phase each project
sponsor completes a presentation on the proposed project. The IT Council
then uses this information along with the other collected data to rank/prioritize
all the project proposals and either moves them forward for funding recommendations
or sends them back to their sponsors for additional work. This review
process involves a standardized way of collecting and presenting data
about the proposed investment project. The process is based on the Information
Technology Investment Process(ITIP) developed for the federal government
and has much in common with processes used in other states to select IT
projects.
A third major shift in decision making
is more recent, involving the budget procedures for the 1999-2000 fiscal
year. Previously, there was no comprehensive, all-inclusive Department-wide
budget for IT projects. Instead projects were funded from program allocations
and often did not represent a Department-wide view of IT requirements
and needs. Under the new system, there is a separate IT budget and IT
spending plan for the Department, administered by the IT Council, resulting
in a broader, more comprehensive view of the total IT needs of the Department.
A fourth change involved the creation
of the in-house title of Business Account Manager (BAM). These positions
are responsible for providing the link between the business units and
the IT functions. One of their primary functions is to assist the business
units in developing IT initiatives and coordinating the technical support
aspects of project development. All full-time Business Account Managers
are drawn from the program areas, so they know the business side
of the programs and are also familiar with, but not necessarily expert
in, the technical issues. These positions provide a key link between the
business functions and the IT support activities and are an important
ingredient in making the IT investment process work in DOT.
A final major change was in the increased
concern for the planning and analysis of the business problem involved
in the IT investment. As one participant described it, the old approach
was, "Here's the problem. Here's the solution. Let's start coding!"
In the newer perspective, more attention to the underlying business process
and strategic context of the problem gets first attention; it becomes
the "up front" work central to the final investment decision.
To support that kind of work the Information Services Bureau (ISB), which
supports the entire IT investment process, created new guidelines for
IT project planning and business case analysis, and instituted a pilot
training project for staff.
The
importance of "up front" work
"Up front" work is what the ISB staff began calling the new
kinds of analysis they felt were needed to improve the quality of IT project
initiatives. "Up front" work is what's necessary to understand
the business problem before proposing any particular IT solution. In the
old decision-making process, units within DOT developed proposals to describe
and justify a particular IT procurement or solution. The proposals usually
focused on the costs and technical functionality of the solution. But
the proposals rarely took into account the Department-wide view necessary
for sound IT investment planning and were produced with little analysis
of the underlying problem the investment was intended to solve or its
relationship to the rest of the unit's or agency's work. That made it
very hard to judge how well the proposed solutions would really work,
making it difficult for the Department executive management staff to choose
which proposals to support. Because Department management and ISB technical
staff wanted more "up front" information about the target problem
or need and its relationship to business processes, the proposal process
had to change.
The change was based on a two-part strategy.
One part was to develop new tools and supporting materials to help proposers
generate and organize better information about their IT investment ideas
with particular emphasis on clearly identifying and prioritizing business
needs and problems. For this goal the Information Services Bureau created
a new, comprehensive guide for preparing proposals and organized a pilot
training program for a sample of agency staff. Their goal was to improve
investment decisions by:
- expanding the types of the information
needed to justify an initiative
- improving methods for collection and
processing that information
- enhancing staff skills with new methods
of developing and interpreting information
The other element of the strategy was
to engage in a training process for agency staff that brought the business
and process questions to the forefront. The training approach resulted
from the ISB staff's familiarity with the "Making Smart IT Choices"
publication and related training activities of the Center for Technology
in Government.
IT
investment decisions demand much more than technical information
The primary data needed for these investment decisions comes from the
proposals submitted by the units requesting the new investment. They might
come from engineering or any of a dozen other program units within DOT.
To help the IT Council and agency executive management better understand
the business impact of initiatives and make better investment decisions,
the staff reviewers wanted to improve the quality of information in these
proposals. One major improvement they sought would result from adding
new kinds of analysis to the proposal preparation process: the "up
front" part of the work. This idea was prompted in large part by
the experiences of some Information Services Bureau staff who participated
in a workshop based on the CTG program of Making Smart IT Choices. This
workshop emphasized careful attention to the strategic context and the
business problem for any IT investment. ISB staff reasoned that if those
who prepared the proposals had similar training, they could then expand
the scope of analysis and improve the resulting information provided in
the proposals. They expected such training would prompt many new questions
and data not previously required or normally present in the old-style
proposals. These included questions about:
- the service or business objectives
involved in the new technology
- the identity and interests of stakeholders,
including possible benefits and costs to each
- how to describe the business processes
involved
- the strategic linkages and likely
partners in the implementation
- performance measures for the new system
and processes
- the organizational and political risk
factors that could influence the effectiveness of the project
Answers to these and related questions
would result in new information becoming available to the proposal reviewers
and the ultimate investment decision makers.
Standards
and formal procedures help with some tasks
The Information Services Bureau staff initiated another approach to improving
data that worked along with the training and the need for expanded front-end
analysis. They also adapted the information structure and analysis approach
from the Information Technology Investment Process (ITIP) which is an
IT investment approach being used in the federal government and a number
of states. This approach describes the documentation that project sponsors
are ultimately responsible for providing to the IT Council to justify
an IT investment initiative. The documentation includes five profiles
covering the main components of the project:
- business profile, including the business
alignment case and business process analysis opportunities and efforts
- risk profile, including risk assessment
and mitigation plans
- financial profile, including ROI and
cost/benefit analysis
- technology profile, including compatibility
with existing infrastructure and systems
- management and planning profile, including
both project and acquisition plans
The profiles provide a formal framework
of standardized data elements for evaluating IT investment proposals.
It
takes more than rules and procedures
Much of the information and insights needed to complete the profiles and
scoring in assessing IT investments comes from or is based on good "up
front" analysis. That is, a highly structured and formal approach
to facilitate proposal presentation is no better than the basic understandings
and skill of the people using it. Improving the quality of information
and analysis going into investment decisions depends heavily on the skills,
knowledge, actions, and relationships of a wide range of staff doing the
work. The training program the ISB offered was a start in developing and
transmitting this crucial message to the proposal preparation teams. It
was a beginning in changing attitudes and organizational culture to support
the new rules, procedures and process. The training provided the tools
and techniques to create and analyze the more complex and diverse information
that underlies the IT proposals. The result is a more comprehensive and
useful mix of information, better suited to the complexities of the decision
making situation.
The training component of the new approach
had other features that facilitated the acceptance of new methods and
techniques by the participants. There is typically some resistance in
any organization to the introduction of new processes and ways of doing
business. The participants in this training had an opportunity to interact
with each other and the trainers. That interaction provided an opportunity
to discuss content and clarify uncertainty. These experiences reduce resistance
to change and provide useful feedback for the trainers. The training brought
together staff from different areas of the Department. By working together
they were exposed to a variety of different perspectives on how business
problems are analyzed and how this analysis can affect IT project initiative
development. As one participant said,
… I find myself, now that
I've taken this class, spending much more time with the regional and the
residency personnel, listening to them and ensuring that we can meet their
needs, trying to develop alternatives of how they do business. So it has
kind of opened my eyes to the point that there is more than just myself
involved in this, and that other people's interests are just as important.
Perhaps the most important effect of the
training is that the participants developed skills and insights that sharpened
their focus on information about the business problems and organizational
processes involved. This training approach led the participants to define
problems to be solved in terms of the business processes and goals of
the agency program, not in terms of technology. In describing their response
to the training, the participants emphasized the importance of clearly
understanding the program need or business problem. As one training participant
put it:
I run into a lot of people who
are really not that familiar with technical IT systems, and they try to
focus on the part that is technical instead of the real world part. So
what they do is they look at a system developer and they say, "Solve
my problems" without them actually saying what their problems are.
This focus on "up front" information
was institutionalized in the practices of the Triage Committee, the group
charged with recommending which IT project proposals would go forward
for full IT Council review. This Committee began requiring "up front"
business process analysis information as a routine part of proposals.
This included suggestions for stakeholder analyses to show the impacts
on a wide range of possible stakeholders. In addition, the Committee members
put their "up front" planning questions into a routine that
carried problem and business process-related questions forward from one
meeting to the next. This change in information requirements initially
met with some resistance from some units presenting proposals but this
attitude is changing as the investment process and the need for this kind
of front-end analysis work becomes institutionalized. As a filter of proposals
moving forward to the IT Council full review stage or in its role as helping
sponsors develop better business aligned initiatives, the Committee was
in a strong position to convince the sponsors of the need for this kind
of front-end analysis and documentation.
The training participants emphasized how
their perspectives had been changed. They came to see the importance of
information about the program or business perspective in all parts of
IT planning. They often repeated the desire for program managers to be
included in the training and to develop the same perspective. It should
not be limited to technology specialists, Information Services Bureau
staff, or administrative staff. One technology specialist described how
this training for program managers would help them take a broader view
of IT projects and developments. These managers, she reported, often define
IT projects to solve a narrow, highly specific problem in a way that is
not integrated with larger program issues and linkages. If the program
managers took a broader view, IT projects could be designed in an integrated
way that produced greater benefits or avoided conflicts or incompatibilities
with other IT systems.
The training experience also changed some
views about the information needs across a variety of projects and units.
The participants expected the training to be highly structured, focusing
on generic skills, tools, and analyses. They thought information needs
would be more consistent across projects and units of the agency. After
the training they expressed an appreciation for the diversity of problems,
business processes, and information needs across projects. They recognized
that the basic approach, involving questions about the business process
and nature of the underlying problem, does remain fairly consistent. But
each project and organizational setting requires its own analysis. In
the training, participants noted the importance of involving program managers
and other specialists in future training to be sure their perspectives
are part of the analysis.
For
best results, invest in the investment process
The primary cost of this transition to a new investment process is staff
time. The expanded requirements for proposal preparation have two cost
components. The requirements call for more information than in the past,
demanding more complicated gathering and analysis techniques. Eventually
it will be necessary to involve larger numbers of program and IT staff
in training to prepare them for these expanded analysis requirements.
Considerable training will be necessary to bring all staff involved in
IT investment development up to speed. The pilot training activity for
this case involved only 10 DOT staff directly. But it nonetheless took
10 person-days of preparation and delivery for the trainers, over 20 person
days of DOT staff time in the training and follow-up activities, and an
unknown amount of additional time for early attempts to integrate the
new techniques into the IT development process. When extended to the whole
department, these become very significant costs, even though they may
not have an obvious budget line. These could be considered extra costs,
but DOT chooses to view them as part of the new investment process itself.
They expect this additional staff effort will be rewarded in better choices
and more successful and cost-effective systems.
A
strategy that links business needs to information to funding to action
In less than two years the Department has significantly changed IT investment
decision making. These changes combine new information requirements with
fundamental changes in governance. The pilot training component addresses
the importance of staff skills and attitudes in improving information
use and analysis. The new proposal review procedures and information standards
address the need for higher quality, comparable information to evaluate
proposals. The changes in the governance structures and policies complete
the link between the substance of IT investment decisions and organizational
action. DOT used a multi-part strategy to improve both the availability
of information for decision making and the decision-making process itself.
This strategy was operationalized in a new planning document called Instructions
for the Development of a Technology Investment Business Case: A Standard
for Information Technology Projects. This standard calls for a detailed
business assessment as part of IT investment proposals. The business assessment
guide gives operational meaning to the expectation for better "up
front planning." The planning document also calls for a detailed
self-assessment of risk that includes a wide range of factors for the
planners to take into account. By requiring the use of a standardized
format for collecting, evaluating, and transmitting IT investment data,
the ISB staff established a new and consistent information framework for
a large portion of proposal content. This helps them evaluate and compare
proposals more objectively. Placing budget allocation authority in the
IT Council is the third critical piece. Since the IT Council consists
of program and regional managers, their budget recommendation decisions
reflect concern for the Department-wide programmatic and operational value
of IT investments. And since they review and prioritize IT proposals
as a group, these budget decisions can also result in better coordinated
and integrated IT investment across the whole Department.
And
it's not over yet
This has been an incremental process, using multiple tactics and adjustments
along the way. The relatively modest exposure of ISB and other program
staff to new ways of doing "up front" work has already had a
major impact, largely because the ISB took full advantage of its strategic
position in the overall Departmental approach to IT investment decisions
and because of the involvement and cooperation of the program staff. The
new approaches to "up front" work came along with new decision
structures and policies, all working together. But the overall process
is far from finished. The Information Services Bureau staff continues
to seek improvements in information resources and proposal development
processes. There is active consideration of extending the training in
"up front" analysis to all program staff who are actively involved
in developing IT initiatives or concepts, to enhance their abilities in
problem definition and business process analysis. They are considering
implementing the Information Technology Investment Program(I-TIPS) application,
which is a software package developed for the federal government which
serves as a management tool and repository for IT investment information,
to further support proposal development, project tracking and evaluation.
Rather than a one-time reform effort, DOT is engaged in a continuous improvement
process. This approach can help ensure that as the IT investment decisions
get tougher and more complex, the quality of information available for
decision making can keep pace and ultimately lead to better project
and business results.