Skip to main content
 
Return on Investment In Information Technology: A Guide for Managers



Chapter Two: Methods of ROI Analysis for IT

Methods for understanding the business process

IT investments of any sort must be viewed as part of the business processes in their organizations. Both the costs and returns of the investment are tied to how the new technology fits with and how it alters business activities. The notion of what constitutes IT must be expanded to include not only the chips, wires, and software, but also the activities and interactions that generate the costs and value that results from using the IT (i.e., the enterprise architecture). Failing to map and understand the enterprise invites a badly flawed understanding of how the IT investment will work, and can be a short cut to failure. For example, the U.S. Department of Education recently invested millions of dollars in a Web site for college students to apply for financial aid. The site generated very little use and was closed down. It failed because it ignored the important . role of college financial aid officers in the overall process of students seeking and receiving aid. The designers did not understand or ignored the overall business process of financial aid administration.

Business process analysis tools and methods range from simple flow and GANTT charts to sophisticated formal and mathematical modeling. Simple flow charts are sufficient to identify the basic elements of most business processes and some of the more obvious dependencies. How far business process modeling needs to go beyond simple diagrams depends on the questions to be answered and the resources available for the work.

The basic components of a business process are activities, flows, controls, and dependencies. Analyzing the business process therefore means identifying the various kinds of activities, what kinds of flows they generate and support, and how the activities and flows are controlled and linked to produce some particular collection of goods or services. Flows can consist of information, persons, resources, control inputs, and work products moving from one activity to another. The analysis consists of gathering information about the activities and flows in as much detail as necessary. Depending on the kinds of modeling to be used, this information can include descriptions of activities and flows in strictly qualitative terms or detailed measurements of resources used, flow metrics, and outcome measures.

Descriptive models

Approaches to process modeling can be grouped roughly into three levels or types: descriptive, analytical, and dynamic. Descriptive models are the most fundamental. They describe the elements of a business process and the relationships among them. A flow chart that describes how travel expense reimbursements are processed, for example, might show how the request originates, how it moves from one work process to another, what actions or decisions take place at each point, and the elapsed time for each step. Descriptive models may involve some measurements, such as personnel time, expenditures, other resources used, and outputs. The flow chart in Figure 2 below illustrates the actual billing information flows for a nonprofit provider of services to mentally retarded and developmentally disabled clients. It shows how information about services delivered is converted into bills submitted to the various government sources of funding for these client services.7 This flow chart was produced to help understand how a change in the reporting and processing technology for one of the funding agency’s would affect operations among the providers who deal with multiple agencies. The IT project was planned by one state agency, but the planners recognized that the business process involved a range of other agencies and local practices. Therefore their analysis of the costs and potential benefits of the project had to include attention to the overall business process, not just what happened within one state agency.

Figure 2. Example of Process Flow Chart

Figure 2. Example of Process Flow Chart

The level of detail used for such a model is a matter of judgment, in this case dictating a model at an intermediate level of detail. For some purposes, a more highly aggregated model may be sufficient, such as the following.

Highly Aggregated Model of Process Flow Chart

It might be necessary to examine a much finer level of detail, such as how administrators review service records before approving billing in Figure 2 on page 17. The level of detail depends on the questions that guide the overall effort. A descriptive model influences the choice of what is to be measured and whether more formal modeling will be needed.

Use of analytical or formal models

Analytical or formal models go beyond description to represent the performance of a business process or project in some quantitative or mathematical form. This kind of model has many advantages for analysts and decision makers, along with some substantial problems and limitations. There are at least four kinds of advantages.
  • Formal models require making the implicit explicit, and to do so in precise and systematic ways.
  • Many formal modeling methods allow exploring "what if?" questions and work with alternative scenarios by simulating the results of using the model of the new system. The planner can thus test ideas and alternatives without risk to real systems, operations, or data.
  • The analysis yielded by the modeling can reveal insights into the behavior of persons or organizations that would otherwise be difficult or impossible to detect.
  • The modeling process, if done in a collaborative way, produces a shared understanding of the work among managers and stakeholders that can lead to improved decisions about the project and its ultimate implementation.
Because of these advantages, the use of these techniques for process improve- ment and project planning has grown substantially and spawned an industry producing software applications, consulting, and training in one or more of these methods (see resources in Appendix D).

These methods are not without problems or limitations, the most important of which is that they raise the costs of analysis in several ways. These models typically require more detailed and extensive information, particularly performance and outcome measures, than simpler descriptive methods. In addition, the construction and use of these modeling methods requires considerable time and expertise. Organizations that do not have staff prepared to do the modeling work themselves would have to invest either in extensive training or employing consultants. It is important to review carefully whether the complexity of the project justifies the investment in formal modeling before taking that course of action. The review of alternative methods below is intended to help in that regard.

Types of formal modeling approaches

Agent-based modeling. The basic idea of agent-based modeling is that complex behaviors of collectives (social groups, members of an organization, population) can be modeled by simple computational rules. The rules treat the members of the collective as autonomous agents that follow simple rules of behavior, rules that can be modeled on a computer. What happens to the collective is the result of the individual agents interacting with each other according to the rules. Changes in the rules change the overall results, sometimes in predictable and sometimes unpredictable ways.

One early example of agent-based modeling in the social arena was a model of residential segregation developed in the late 1960’s.8 The model starts with a hypothetical city with a mixed race population randomly distributed over the residential area. Each household either stays or moves according to two simple rules: a preference for living near persons of the same race, and deciding to move if the population in surrounding areas has too low a proportion of such households. There is no rule involving dislike for households of different race or desire to avoid them. Nonetheless, the action of these rules over time produces a city of racially segregated neighborhoods.

In agent-based modeling, the rules are the model or theory. Thus they make explicit the assumptions and motives of the agents and what governs their actions. The value of the model depends on the degree to which the rules produce results that match the actions and outcomes for the actual collective of interest. If rules are developed that reproduce the results of interest, they can provide a powerful tool for understanding how those results occurred and for testing ideas about how to change or improve them. This kind of modeling may be particularly appropriate for information technologies that will involve the activities of large numbers of persons engaged in the same type of activity, such as large numbers of citizens seeking the same kinds of information on a Web site. A valid model of how people will use a system can be a valuable tool for exploring what costs and benefits to expect.

Agent-based modeling is not for the faint of heart, however. It is mathematical, often requiring advanced knowledge of algorithms and computing to implement. There are special modeling languages for developing the computer programs to pursue agent-based work, but they require experienced users or a considerable investment in training. If the rules are sufficiently complex or abstract they may be difficult to communicate to participants in the planning or to key stakeholders.

Operations research and statistical modeling. If the factors that influence a business process can be identified and measured, modeling that process by operations research and statistical methods may be feasible. These modeling approaches use mathematical equations to represent the business process. This requires measurements of the business process itself and of the factors that are thought to influence how the process works. Then the modeling equations are chosen to fit the way the significant elements of the business process are conceptualized.
  • If the process is seen as a series of activities that consume resources and occur in particular interdependent sequences, project description tools such as PERT charts or critical path analysis may be used.
  • If the sequence of events in the business process is uncertain, or if multiple paths or outcomes are possible, probabilistic methods such as Markov chain models may effectively represent the process.
  • If the sequence or path of influence is not known, but some measure of their general effects is available, linear modeling and regression methods may be useful to represent the overall process or predict the results of changes.
  • If particular combinations of influences are thought to work together to affect the flow and/or performance of the business process, then scenario analysis may be useful.
These modeling methods are similar to agent-based methods in that they require measurements. They differ in that they are based less on specific theories or assumptions about the activities that make up the business process. Rather, they use simplifying assumptions about the underlying cause and effect relationships that drive the process. That makes the models easier to use in some respects. Little original theory building is needed. In addition, the methods are supported by commonly available applications, and training in many of the particular methods is typically part of management education programs. Choosing among these alternatives is not a straightforward task and a reasonably high level of expertise is needed. The same applies to interpreting the results and understanding the limiting assumptions that apply to each.

System dynamics modeling. Many business processes involve feedback, that is, chains of effects in which what happens at one point in the process has an impact elsewhere that in turn influences (increases or decreases) the original effect. For example, a new database application may produce a new kind of report that users find particularly valuable. Seeing one possible new outcome prompts the users to request additional kinds of reports that produce more valuable results, and more new requests, and so forth. Any kind of process that can be represented as stocks or supplies that flow from one place to another with either reinforcing or negative feedback can be represented using the methods of system dynamics modeling. These models can provide both a concep- tual and mathematical representation of the processes. The conceptual or graphical representation is usually referred to as a causal loop model. These models can be built by analysts working with the staff involved in the process, making use of staff knowledge and insights. The image created is a potentially powerful tool to evoke knowledge about the process and to move the participants to a shared understanding. An example of such a diagram is shown in Figure 3 (page 21). This particular model represents a working theory about how work progresses. Tasks can exist in one of the four following states.
  • Work to do
  • Undiscovered rework
  • Known rework
  • Work really done
At the start of work, all tasks are in the stock of work to do. As participants perform work, correctly done tasks move to the stock of work really done, with a probability of 1- error rate (a fraction). The error rate is based on the idea that it is impossible to perform all tasks correctly on the first try. Tasks performed incorrectly require rework, so as work proceeds some propor- tion of the tasks enter the accumulation undiscovered rework, according to the probability of an error occurring, error rate. Similarly, problematic tasks can be redone correctly (moving from the accumulation of known rework to work really done) or incorrectly (returning to undiscovered rework), based on the error fraction. The concreteness of instructions and transformability of the tasks themselves are shown as affecting the ability to do work right in the first place or to correct errors. With appropriate assumptions and values for the variables and parameters in such a model, a mathematical representation can be built that reproduces the behavior of the process itself. Analysts, who should have appropriate expertise in system dynamics modeling, can change assumptions or values and explore the consequences, yielding new insights into the possible costs and performance of a project.

Figure 3. Causal Loop Diagram of Project Management Processes

Figure 3. Causal Loop Diagram of Project Management Processes

Unified Modeling Language (UML) and Use Case Modeling. UML is a programming and modeling language that provides a system for representing and documenting object-oriented software development and the business process in which it resides.9 It is particularly tailored to analyze the application development process and for use by development teams in that environment. The overall conceptual scheme for use of the UML includes what is referred to as Use Case Modeling as a component. A Use Case is a guide for software design based on a behavioral case study of how an application is used in an actual business process. This differs from traditional approaches that specify software requirements in terms of technical needs and goals. For IT projects that are primarily application development, UML can be a very useful and powerful tool for modeling the project itself for management purposes, as well as for its linkages to the business process. However, the UML is highly technical and requires considerable study and programming experience to use effectively.

Investment in the use of UML may be justified, especially for large application development projects that involve multiple teams. As the UML documentation states:

"In the face of increasingly complex systems, visualization and modeling become essential. The UML is a well-defined and widely accepted response to that need. It is the visual modeling language of choice for building object–oriented and component-based systems.10"

Workflow modeling. The workflow modeling approach is similar to UML but involves a different focus and approach. Instead of using a programming language to model work processes, workflow methods use generic models of work processes and the relationships among them. The components of these generic models represent business activities, resources, dependencies, and controls. The design of the models is aimed particularly at enhancing coordination by identifying dependencies, failure modes, and problems of handling exceptions. For complex business processes, a flow chart will not reveal the consequences of errors at critical points, unanticipated resource shortages, or missing factors during some point in the process. Workflow models allow these situations to be explored and provide analysis of where processes are particularly vulnerable to failure or waste.

Workflow modeling technologies and commercial products may be useful to analyze an existing business process or re-engineer one. But they are designed primarily to develop automated work processes. These depend on computer- based controls, typically where the flow of information or materials involves networked links among the activities. However, the basic concepts of dependencies, exception handling, and error modes may be applied regardless of the level of automation in a workflow. So the conceptual tools of workflow analysis may contribute to a process model concerned with costs and returns to a new system.

7 For example, Medicaid billing goes to the Medicaid Management Information System (MMIS), education related billing goes to the State Education Department, and so forth.
8 Thomas Schelling, (1971) "Dynamic Models of Segregation," Journal of Mathematical Sociology 1, 143-186.
9 "The Unified Modeling Language (UML) provides system architects working on object analysis and design with one consistent language for specifying, visualizing, constructing, and documenting the artifacts of software systems, as well as for business modeling." Object Management Group, p. xi.
10 Object Management Group. OMG Unified Modeling Language Specification, Version 1.3. Framingham, MA: Object Management Group Headquarters, 1999, p. 1-4. 13 Office of the New York State Comptroller. Accounting and Reporting Manual. Albany, NY (no date), p. 17