Skip to main content
Chapter 2. The analysis and evaluation process

In this chapter, we present an analytical process that can be used in any project that applies IT to a service delivery or administrative goal. The process begins by defining a problem or purpose within its environmental context. It then goes on to identify and test possible solutions. The third phase focuses on the evaluation of alternatives and the selection of a preferred approach. For simplicity, we present these phases as sequential steps, but in action they are often iterative. What you learn in one phase may prompt you to return to an earlier one to refine your thinking before moving forward again.

Although this process of feedback and learning takes extra time, it is critical to comprehensive understanding, and it allows you to build a business case that reflects all the essential risks and options. The final steps described in chapters three and four are the preparation and presentation of that business case. Each phase builds on the perspective gained from earlier ones. The result is a multi-faceted analysis of a proposed project that has a high likelihood of accurately predicting real costs and outcomes. Figure 1 illustrates the entire process.

How to make smart IT choices

First things first: Choose a "good" problem
The research, testing, and evaluation that go into this analysis are labor intensive processes. They can be applied to any problem, but not every problem deserves this level of attention. This is a process that is worthwhile for complex, mission-critical, information-intensive problems. We call these "good" problems because their solution has high positive impact. For these problems, you need to pay close attention to the political, economic, legal, and organizational environments as well as the technologies involved. Not every IT decision has these characteristics or warrants the investment of resources that this process entails. Here are some administrative problems and service delivery questions that have successfully used this process:

  • How effective are current service packages in helping homeless families achieve independence?
  • How can the complete history of relationships between a state agency and local governments be captured for ongoing use in current and future technical assistance?
  • How can information about the health and well being of children be made readily available to professionals and individuals concerned with the design of children's service programs?
The first question represents an information sharing and program evaluation problem. The homeless services system involves not only state agencies, but local governments, and scores of nonprofit shelter and service providers, all with different kinds of information maintained in different systems and formats. These organizations also respond to different funding streams and to a variety of legal requirements.

The second problem involves the central office of a major administrative agency and its regional offices distributed around the state. Each regional office is responsible for monitoring and advising local governments in its geographic area on their financial affairs. Each region collects information and does things in its own way. There is no statewide repository of information that can provide context, history, best practices, or overall performance information about this important aspect of municipal government.

The third example does not focus on operational concerns, but on the availability and usability of statistical information about children. In this case, the information comes from 13 state agencies, is compiled once a year into a printed book by a central coordinating agency, and is used by hundreds of municipalities, nonprofit service agencies, and research organizations. The data is collected according to different time periods, using different definitions, and covering different geographic distributions.

Despite their differences, these problems have some similarities. First, they are mission critical to the agencies that sponsor them. The homeless services project is deeply embedded in the core services and values of all involved organizations. The financial health of municipalities is one of a handful of overriding mission goals for the administrative agency. And a visible public focus on the needs of children is the entire reason for the existence of the coordinating agency.

Second, these problems are information-intensive situations. The solution to each problem depends, in large part, on the quality, timeliness, and accessibility of information. In most cases, some information systems are already in place and need to be taken into account in any new approach.

Third, they all exist in an environment of high complexity. Both the homelessness and children's data projects involve many independent organizations, each with its own practices, values, and rules. Similarly, the municipal affairs project covers the whole state and must deal with local diversity ranging from huge sophisticated cities to tiny rural towns and villages. All three projects must deal with public opinion, public budgeting and legislative cycles, and legal requirements. Civil service rules circumscribe staffing assignments and compensation. Organizational rules, traditions, and structures set boundaries. Many different business processes already in place will need to be understood, and may need to be changed.

In projects with these characteristics, opportunities abound for wrong assumptions, premature decisions, and dangerous oversimplification. This kind of complexity seems overwhelming, and a common reaction is to try to cut through to the part of the project that is more concrete and manageable - the technology. This is almost always a mistake. Projects like these demand a careful analysis that works through and manages the complexity at every level, from the larger environment, to the organizational considerations, to the work processes and data needs, to the technology choices.