The following are ways to collect quality information before making an IT investment.

Many information management problems and proposed IT solutions are not unique. Organizations and project teams can benefit greatly from the experience of others in both government and in the private sector. In addition, researchers have conducted and documented many studies that can offer insight into your project and ways to approach it. Information about all these kinds of experiences is published in periodicals, books, journals, or databases.

Systematic searches of print and electronic publication sources. The purpose of these searches is to identify, review, analyze, and evaluate material that might inform thinking about a problem facing your organization.

Library-based searches. A quick consultation with a reference librarian at the outset will help you focus the search, identify the most relevant print and electronic access tools, and be sure that you aren't overlooking any new information sources. You may want to access the library's magazine holdings, databases, newspaper articles, or government documents section in addition to looking a professional journals and books.

Subscriptions. Many professionals subscribe to periodicals or news services that help them stay on top of developments in their fields. Electronic news services are becoming increasingly popular and deliver daily or weekly summaries to you e-mail in box, often with links to more complete stories.

Identifying best practices. A good review of relevant publications should help you uncover some best practices, as well as analytical and theoretical frameworks.

Learning more about possible solutions. Your review should reveal something about current "state of the art" solutions that may help you solve your agency's problem.Research journals in particular should present objective evaluations of the performance of a given management strategy or technology.

Identifying potential expert advisors. There are probably a number of organizations and individuals whose experience you can draw on for advice or consultation. Use your library and document review to identify these possible advisors. Also look for organizations that may allow you to see in operation the particular technology, or process , that you are considering.

Avoiding mistakes. The more you learn about the potential pitfalls in system design or implementation, the easier it is to avoid them.

Identifying potential partners and vendors. Use this kind of research to identify locate partners in government and academia, as well as potential vendors and consultants.

Need to define correct scope, key words. One of the keys to effective and efficient literature reviews is identifying an appropriate search scope and key words. Starting too narrowly is often more effective than starting too broadly. If the scope is too narrow and nothing can be found, you can easily make your search less specific. This is far preferable to wading through hundreds of potentially unrelated documents in hopes that one or two relevant items will surface.

Time lags. There is often a substantial time lag between the completion of a document or report and its appearance in print. For scholarly paper-based journals, this can be as long as 24 to 36 months between submission of the finished manuscript and its publication. Electronic peer-reviewed journals involve a much shorter time lag.

Basic research skills are needed. The organization of indexes, abstracting journals, library catalogs, etc. is complex, and the inexperienced searcher can easily overlook relevant material or sources.

Gerstenfeld, S. "Chapter 6: Literature Review," in "Handbook for IQP Advisors and Students". http://www.wpi.edu/Academics/Depts/IGSD/IQPHbook/ch6.html [accessed June 11, 2003]

The Internet can be a powerful tool in your effort to develop and implement IT solutions for your business problems. Whether you want news about the latest privacy policy or e-commerce applications, a comparison of available software packages, or contacts in other government agencies, the Internet is the place to look. Developing your ability to quickly and easily search the Web will allow you to tap into its vast resources. Search the Internet to identify other organizations 0that are solving problems similar to yours. Try to take these examples apart to see if the way you are thinking is on track. Compare the example with your project. Look for dissimilar organizations with similar problems, as there is often a lot to be learned here. Finally, think about the technologies or management methods that might be useful in your project and search for organizations that are known to use them well. This kind of investigation is designed to leverage your research effort with known experience from a variety of other places.

Using search engines. Web searching involves using Internet search engines to locate pertinent information about a given topic. Using various key words and phrases, these sites search the Internet for Web sites, publications, listservs, and other resources about your topic. The Internet is also a valuable source and may actually yield more useful information than a library search in terms of identifying people and organizations working on IT implementations or specific business problems. Information found on the Internet may also be more current than those journal references that have gone through a lengthy review process. Many Web pages often provide direct e-mail access to the source of the information, making it easy to contact them. The search capabilities of these engines are constantly enhanced. Take advantage of what technology has to offer and let the Web work for you.

Listserv searches. A listserv is a subscription-based electronic mailing list where individuals who share similar interests can post and respond to messages. Relevant listservs and their respective subscription instructions can be identified through Web-based searches. A posting to a listserv asking for information on a specific business problem or technology is a very effective and low-cost way of reaching a large targeted audience. If you are unfamiliar with the codes of conduct in using listservs, either subscribe to the list and watch for a while before your first posting, or refer to one of the many resources available that discuss "netiquette."

Finding best/current practices. Use the Web to find out how other organizations dealt with problems similar to the ones you want to solve. Groups like to publicize their successes. Online magazines, newsletters, and journals often contain success stories and case studies.

Identifying contacts. The Internet is a good way to locate other government and IT professionals who can be great sources of advice and inspiration. Use search results to identify knowledgeable people you should talk to. Most web sites include contact information. Get in touch with these professionals by e-mail or phone and interview them about their experiences.

Information technology options. All kinds of hardware and software are available to fix business problems. By conducting a Web search, you can get information about all the different technology options available to you. You can arm yourself with the knowledge to make informed choices.

Write a good query. The information you get out of a Web search is only as good as your query. You must write a targeted and well-defined search query in order to locate relevant resources.

Variability of results. Since search engine databases vary in size, frequency of updates, and search capabilities, using different engines can lead to results that vary considerably.

Information quality and accuracy. Anyone can create a Web site and fill it with whatever content they wish. It's important to make sure you know the source of information you access on the Internet. Try to find data from recognized and respected institutions, such as universities, corporations, government agencies, media publications, and professional organizations.

Formulate your query. Take a few minutes to think about your problem or goal. Try to break it into key elements or sub-topics. Write down a series of words or phrases that describe your topic using different synonyms. Also think about issues that are associated with your goal. This will help you develop a more comprehensive picture of the issue that has some reasonable limits.

Conduct a broad Internet search. Start by accessing some of the major search engines and use different keywords and word combinations. Once you have searched a topic, narrow your results by searching within the search results or formulating a more advanced query. Remember how easy it is to follow link after link as you pursue a lead until you have forgotten where you started. Have a pencil and paper handy to keep track of the sites you visit. Use the bookmark feature of your browser to easily return to the most useful ones later.

Search within discussion groups. Don't forget to search the discussion groups of main search engines to see if the issues you are interested in have been debated within these groups.

Join listservs. Look for a listserv corresponding to the subject you are researching. Subscribe to it and use this forum to ask who knows about or has had experience with your issue. Regularly monitor lists that seem to provide good ongoing discussion of your topic.

The results you get from a search on the Web are only as good as your query. By using the following tips to write a query, you may get more targeted and useful results.

Quotation marks. If you are looking for an exact phrase or group of words, such as American Marketing Association for example, make sure to use quotation marks: "American Marketing Association." Otherwise, the engine will search each word separately.

AND. Use AND (uppercase) when you want to make sure your results contain two terms. If you use AND in your query, the search engine will retrieve only documents that contain both words.

OR. Search your subject using different synonyms to maximize your chances of getting results. Using OR (uppercase) will allow you to enter several synonyms for the search engine to use as it will retrieve any of the words you typed.

+Sign. Some search engines will ignore short words included in a query, such as: in, of, a, out, with. Using the plus sign will ensure that all words are searched. For example if your search is: women in government, make sure to type: +women +in +government.

- Sign. Use the minus sign if you want to make sure your results won't contain a certain word. For example, if you are looking for information on marketing strategy but do not want to get results from consulting companies, you can type: +marketing +strategy --consulting.

* Sign. You can use the star sign to truncate a word. For example, if you are looking for marketing consultants or marketing consulting companies, you can type: +marketing +consult* and the search engine will retrieve results with any word starting with consult.

Combination. You can use a combination of signs or words to do your search by putting terms in parenthesis. For example, you can type (American or U.S.) AND presidency.

A detailed overview of how to conduct Internet research: http://www.tbchad.com/resrch.html [Accessed June 27, 2003]

We've found the following search engines to be particularly helpful:

Alta Vista: www.altavista.com

Yahoo: www.yahoo.com

Google: www.google.com

Hotbot: www.hotbot.com

Teoma: www.teoma.com

The following metasearch engines, which provide more extensive coverage by searching simultaneously in several of the largest search engines' databases, have proven to be useful:

Debriefing: http://www.debriefing.com

Dogpile: http://www.dogpile.com

Profusion: http://www.profusion.com

Ask Jeeves: http://www.askjeeves.com

When you need structured information from a large number of people, surveys are the way to go. They allow you to fairly quickly collect data from many people in different locations. They can be used to inform project direction and focus of effort and to build empirical support for a hypothesis.

Surveys usually employ short, simple questions. They generally offer a series of answers from which participants choose. For example, a survey could ask respondents how strongly they agree or disagree with statements about the potential impact of a new project. Or you might offer multiple-choice questions that ask people to choose among alternative responses.

Structured questionnaires. Surveys may be self-administered questionnaires that are distributed by hand, mail, fax, or e-mail. They can also be administered face-to-face or over the phone. Regardless of what form they take, surveys are designed to gather information from a representative group of people.

Methods for getting information from a defined population. Sampling is an important part of survey planning and design. For example, if you want to learn what people over 65 think about your new statewide initiative, you could try to find and ask every person in this age group. However, this is not likely to be logistically or financially feasible. Instead, you will probably ask a random sample of the population of seniors. If your sample is drawn with statistical precision (a topic much too complicated for this note), you can then generalize the results from this smaller group to the entire population of people in that age group.

Sometimes the population you want to know about is accessible and small enough to be surveyed directly. For example, you may want the opinions of all the people in your bureau about their usage of email. In this case, you would send a survey to every person -- a 100% sample.

Gathering opinions and demographics. Surveys allow you to assess user, client, or stakeholder opinions and evaluations. You can also collect demographic descriptions of these groups.

Supporting qualitative analysis. You can obtain quantifiable results for statistical analysis of such subjective material as attitudes, opinions, and values.

Low cost data collection. Surveys require little time, cost, and commitment from the participants.

Keeping respondents anonymous. Like some other methods of information gathering, surveys allow you to preserve the anonymity of respondents. You're more likely to increase the number of responses when people know their opinions are anonymous.

Careful design and field testing. The questions must be carefully designed and field-tested. You must know what questions to ask and how to word them. Even slightly imprecise wording in the question or the answer choices can give results very different from the ones you seek.

The cost of design and analysis. While the cost of a survey is low for participants, a good survey design, execution, and analysis may be expensive. Questions must be carefully designed and the answers thoughtfully analyzed, both of which take expertise and time.

No chance to discuss answers. Surveys provide little or no opportunity to discuss, elaborate, or explain answers. Respondents who do not understand a question may answer improperly or not at all.

Potential problems with the sample. Biases or flaws in a sample can make it difficult or inappropriate to generalize about the larger population.

Absence of sampling. If you don't use statistical sampling, you can't generalize your findings beyond the group you actually contacted. This is fine when the group is relatively small and well known. For example, if you want to know what every person on your staff thinks about the new proposal, you can ask them all, and report your results without worrying about statistical validity.

American Association for Public Opinion Research (AAPOR). "Best Practices for Survey and Public Opinion Research." http://www.aapor.org/default.asp?page=survey_methods/standards_and_best_practices/ best_practices_for_survey_and_public_opinion_research [Retrieved June 9, 2003]

Council of American Survey Research Organizations (CASRO). "CASRO Guidelines for Survey Research Quality." http://www.casro.org/guidelines.cfm [Retrieved June 9, 2003].

Babbie, Earl R. (1973) Survey Research Methods. Belmont, CA: Wadsworth.

The Survey System had an overview for beginners of survey and questionnaire design at http://www.surveysystem.com/sdesign.htm [Retrieved July 7, 2003]

When you want to gather detailed information about people's impressions, experiences, ideas, and attitudes, interviews are often the best method. Face-to-face or telephone conversations yield rich information, and are a good way to understand complex topics.

Conversations with a person or group. There are several types of interviews. All of them can take place with individuals or groups. You may conduct a structured interview with a series of pre-set questions and answers from which the participants choose. A semi-structured interview uses pre-set questions, but allows people to answer them in their own way.

Unstructured interviews rely on a general idea of the types of questions you want to ask, but use no predetermined phrasing or order. A computer-mediated conversation, such as an online chat, is another form of interview.

Ways to gather and record opinions and experiences. Having a record of the conversation allows for thorough and accurate analysis and interpretation. This may involve the interviewer taking notes by hand or on laptop computers. The interview sessions may also be recorded with audio or video equipment. It is good practice to ask participants before the interview if you may record the interview.

Collecting and recording complex responses. Engaging in a conversation with one or more people is an effective way to gather information about their experiences, opinions, attitudes, needs, and ideas about your project.

Encouraging full discussions and explanations. Interviews allow the interviewer and participant to have full discussions of the questions being asked and answers being provided. This give-and-take yields rich, detailed information.

Promoting active involvement. Effective interviews encourage participation, establish rapport, and capture observations of non-verbal behavior.

Greater time and expense. Interviews are often more expensive and time consuming to conduct than other information-gathering processes.

High level of involvement. Interviews require considerable commitment and involvement of the participants.

Skilled interviewer needed. Training is required to conduct good interviews. Unstructured interviews, in particular, demand special skills.

Higher cost of analysis. Transcription and analysis of interview data can be complex, time consuming, and expensive.

Doyle, J. K. "Chapter 11: Introduction to Interviewing Techniques," in "Handbook for IQP Advisors and Students". http://www.wpi.edu/Academics/Depts/IGSD/IQPHbook/ch11.html#11 [Retrieved June 11, 2003]

Fowler, F. J. (1990). Standardized Survey Interviewing: Minimizing Interviewer-Related Error. Newbury Park, CA: Sage.

Goldman, A. and S. McDonald (1987). The Group Depth Interview: Principles and Practices. Englewood Cliffs, NJ: Prentice-Hall.

Rubin, H. and I. Rubin (1995). Qualitative Interviewing: The Art of Hearing Data. Thousand Oaks, CA: Sage.

Seidman, I. E. (1998). Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences, 2nd ed. New York: Teacher's College Press.

Steinar, K. (1996). Interviews: An Introduction to Qualitative Research Interviewing. Thousand Oaks, CA: Sage.

Survey Research Center (1983). General Interviewing Techniques: A Self-Instructional Workbook for Telephone and Personal Interviewer Training. University of Michigan, Ann Arbor: Survey Research Center.

The essential purpose of an experiment is to learn about what influences the way some process or activity actually works. The data are typically a result of direct observation of behavior, albeit in a contrived and controlled situation. Experiments put you one step closer to understanding what might happen in a natural setting. The natural setting involves a combination of many interacting influences that make it very difficult to sort out the independent effects of one factor or another. So an experiment is designed to control enough of the factors to allow an assessment of the impacts of the specific ones that are of greatest interest or importance.

For an IT system or prototype, an experiment can become part of testing or evaluating system performance. The experimental design would have to provide for the system or prototype to function in an essentially natural way. For computing systems, these experiments often take the form of running a set of highly standardized and tested procedures or software routines that simulate actual use in a controlled way. The experimenters can apply the same procedures under systematically varied conditions, such as running the same simulation on varying hardware configurations. Experiments may also involve hypothetical work or service delivery situations. In such an experiment, carefully selected persons perform a standardized set of actions on a system under controlled conditions. The experimenter can observe and record the results of realistic work behaviors or client transactions. If well designed, such experiments can yield highly useful data for assessing systems and prototypes.

Ways to study what impacts performance. Experiments areartificially constructed and controlled situations designed to study what affects the performance of some system or process. On occasion, a so-called "natural experiment" can be useful as well, as when a change in the natural setting occurs which works in the same way as a deliberate experimental manipulation of the situation.

For example, if an organization changed a work procedure, but kept the workers, technology, incentives, and work setting constant, a comparison of productivity before and after the procedural change would be a natural experiment.

Direct observations of a situation under controlled conditions. Experiments allow you to directly observe and/or measure a situation, such as service delivery or system performance, under controlled conditions. Experimental controls can eliminate or account for the influence of all but the most important components of a system. This allows direct testing and evaluation of these high priority components.

Observing and measuring activity. TThe activity of users, clients, and system components can be observed and measured under realistic, controlled conditions. These include: assessing how system performance may be affected under conditions of significantly increasing scale of operations, providing benchmark data for use in evaluating system performance in natural settings, and repeating activities and assessing performance under consistent conditions to test system reliability, stability, and performance.

Assessing system influences. Conducting an experiment on your system will allow you to assess the influence on performance or system behavior, including critical components or operational factors. By controlling for, or eliminating the effects of, other low-importance factors, an experiment can illuminate the role of the most critical components in overall performance.

Evaluating reliability and stability. Experiments also allow you to assess a system's performance under low-frequency or extreme conditions. You can apply varied tests or operations systematically to evaluate performance under a pre-determined set of circumstances.

Can be costly. Experiments can be expensive to design and conduct. The construction of realistic, controlled conditions may require extensive laboratory facilities, equipment, or similar resources. Materials and protocols must be carefully designed. Participants must be recruited and prepared. The observation, recording, and analysis of experimental data may be very complex and time consuming as well.

May require unrealistic assumptions. When conducting experiments, you may have to make unrealistic assumptions in order to accomplish the necessary controls. These can compromise the validity of the resulting observations. For example, experiments often call for participants to assume particular roles, such as business owner, or teacher, so as to include the necessary range of transaction or clients. The ability of the participant to accurately play that role may be quite limited, and the resulting behavior may not be truly typical of people in that occupation.

May have ethical constraints. The actions that can be taken in an experiment may be governed by ethical or policy constraints. For example, in some designs may be prohibited because they involve unacceptable costs or risks for participants, such as divulging sensitive or potentially damaging information, or being subjected to highly stressful conditions.

Validity depends on the controls. In any experiment, the validity of the data depends directly on the effectiveness of the controls. All potential influences on the outcomes must be taken into account or provided for in effective ways. This requires detailed and extensive knowledge of the processes involved, and all the components of the experiment itself.

Babbie, Earl (2004). The Practice of Social Research, 10th Edition. Belmont, CA: Wadsworth Publishing.

Cook, T. D. and Campbell, D.T. (1979) Quasi-Experimentation: Design and Analysis for Field Settings. Chicago: Rand McNally College Publications.

Morgan, D., ed. (1993). Successful Focus Groups: Advancing the State of the Art. Newbury Park, CA: Sage Publications.

Morgan, D., Krueger, R., and King, J. (1998). Focus Group Kit. Thousand Oaks, CA: Sage.

In the world of IT planning and design, the word "system" is often used to refer to a technical system -- a collection of electronic components that function together. Another use of the word "system," however, refers to the collection of human actions and interactions that create the social and managerial systems within which technical systems operate. These social and managerial systems are often characterized by a high degree of complexity, variability, and uncertainty.

System simulations provide a structured approach to analyzing and understanding how complex social and managerial systems give rise to problem behaviors, as well as what types of solutions might be applicable to those problems.

Ways to describe and understand feedback. System simulations are structured, computer-based, analytic approaches to describe and understand the feedback complexity in social and/or managerial systems. They show, for example, how a policy influences behavior which in turn influences costs. System simulation models are constructed using specialized simulation software that help users understand these complex, often hidden, relationships.

Time plot simulations support to planning. System simulations explicitly model delays in system functioning as well as feedback loops. This can help you predict and plan for the future behavior of a system under a wide variety of circumstances.

Examining complex systems. System simulations help create models of a problem or a solution when the level of social or managerial complexity surrounding a system is high.

Identifying problematic behaviors. Simulations allow you to identify patterns of inter-agency or inter-organizational decision making that are causing or contributing to a problematic or undesirable pattern of behavior.

Understanding how a technical system fits into social and managerial system. System simulations can help your project team understand how a technical system will fit into, and function within, a complex social or managerial system. Often, the simulation can help to identify forces within the social system that will result in a system that doesn't solve the basic problem. This can lead to redesign of the technical system or to a redesign of the business processes that support the managerial system.

Considering causes and possible solutions to problems. Your project is more likely to succeed if you understand the root causes of the problem, and how one or more proposed solutions may (or may not) act over time to solve or reduce it.

Understanding your part. These models will help members of your management team understand how their decisions contribute either to the creation of problem behavior within a system or to the resolution of it.

Expense. System simulations can be expensive and time-consuming to construct. They require specialized expertise and tools to build and analyze.

Mixes hard and soft data. Because system simulation models integrate a diversity of data sources, ranging from hard data about time to softer and more qualitative estimates of human factors, the output of such models can be confusing and hard to interpret.

Requires high involvement of key actors. Usually all of the key actors in a system need to be involved in the construction of a system simulation. Bringing all the key people together for the amount of time that is required to build a system simulation as a group exercise may be logistically impossible or prohibitively expensive.

Because system simulations require assistance from a skilled modeler, and because they are more complicated and expensive than some other analytic processes, it is useful to have some criteria for knowing when such modeling efforts are warranted. Here are some indicators of when the benefits of such an exercise will probably outweigh the costs:

• when complex feedback is involved,
• when actors from different parts of the system don't readily agree on the root cause of the problem(s) or the possible solution,
• when the predicted impacts of a system on organizational performance are uncertain or when serious risks are involved,
• when the costs of making a mistake in implementation are unacceptably high, and
• when the problem is an important one that has attracted the interest of top management.

Morecroft, J. (1988). System Dynamics and Microworlds for Policymakers. European Journal of Operational Research 35 (3): 301-320.

Roberts, N., D. Andersen, R. Deal, M. Grant, and W. Shaffer (1983) Introduction to Computer Simulation: A System Dynamics Modeling Approach. Reading MA: Addison-Wesley.

Senge, P. (1990). The Fifth Discipline. New York: Doubleday.

Senge, P. (1990). "Catalyzing Systems Thinking Within Organizations" In Advances in Organization Development, ed. F. Masarik. Norwood NJ: Ablex.

Wolstenholme, E. (1993). Evaluation of Management Information Systems. Chichester; New York: J. Wiley.