Tying a Sensible Knot: A Practical Guide to State-Local Information Systems:

Chapter 3. Best Practices

Review and evaluate performance

Performance-based accountability is a real issue for managers and agency directors, and systematic evaluation and review activities can help administrators and IT staff determine if their systems are achieving their intended goals. Every system should begin with clear purpose and goals. After it has been up and running for a period of time (say six months to a year), it makes sense to revisit them in a formal way to see if the system is performing as intended.

The system evaluation assesses how well the information system is working to support the purpose and goals of the project. Moreover, a solid evaluation of your work establishes credibility that goes a long way in establishing the support you need for the next project. A comprehensive evaluation plan is attractive to funders, policy makers, and taxpayers alike. A project without an evaluation plan is like a stage performance without any audience reaction. Program managers need to know if the system is making a difference. Designers need to know how well the system is working in order to modify and adjust it. Often a new information system is also the impetus for improvements in business processes and an evaluation helps measure these as well.

An evaluation can sometimes be as simple as a self-administered customer survey, phone interview, or focus group with users. It can also be a more formal and elaborate program review, cost-benefit study, or other analysis conducted by evaluation experts. The method needs to be matched to the goals the project was designed to achieve. Each method has its advantages and disadvantages with resources and time crunches being major considerations. Whatever format is used should address the following kinds of questions:

Service outcomes: how well does the system meet pre-defined customer needs?
Programmatic outcomes: how well does this system contribute to integrated service delivery or other service system goals?
Operational outcomes: how well does the system meet time-savings, streamlining, and other operational improvement goals?
Financial outcomes: how well does the system meet cost-savings or revenue goals?
Return on investment results: Considering what it cost to create and operate, how cost-effective is this investment?

The answers to these questions lead to decisions about changes, improvements, refinements, and lessons for future initiatives.

Immunization Information: Measuring up to a set of ideals

The NYS Immunization Information System (NYSIIS) project has a number of measurable goals that lend themselves to formal evaluation. One set focuses on the functional requirements of an ideal immunization information system. Each of the four regional demonstrations now underway has great latitude to choose methods and features for its systems, but all are trying to measure up to these functional ideals:

Each child’s immunization record must contain a unique identifier that could be reconstructed by the parent from information they have readily at hand.
The system will use the National Immunization data standards to assure compatibility and usability of data across different regions
A registry must be able to identify all immunizations due for every child in the registry and generate reports of immunizations due
The system will capture only necessary data elements and automatically generate reports and reminders based on that data set
Local systems and users, with appropriate safeguards, will be able to update and query regional systems
Immunization coverage level reports will be available for children at various ages from birth to 24 months of age.

The second set of measurable goals is focused on the public health outcomes associated with immunization:

By 1997, have at least 120,000 immunization records of children under age 5 (i.e., 15% of such children outside New York City) captured in the NYSIIS
By 2000, have at least 400,000 immunization records of children under age 5 (i.e., 50% of such children outside New York City) captured in the NYSIIS

To monitor progress over time, these questions are being asked about each regional effort starting in 1997:

How many immunizers are electronically exchanging immuniza- tion records with a regional IIS server?
How many individuals have their immunization records in a regional IIS server?
Of the immunization records captured, what percent are accurate and complete?

These and other evaluation activities are reported and discussed by all the participants in annual working group meetings where each site shares its progress and problems with other public health officials from around the state.

Annual Financial Reports: Looking back for future success

The NYS Office of the State Comptroller (OSC) wanted to design its automated annual reporting system to be useful to localities for as long as possible. OSC recognized that the longevity of the system would depend not only on how well it was initially designed, but on how well it could continue to meet local needs. This meant that communication channels with the customers would need to be maintained, and that periodic reviews of the system would have to take place so that changing environmental factors and functional requirements could be accommodated. OSC let its local customers know that it is committed to the longevity of the system by identifying and correcting problems as soon as they become visible. Customers were encouraged to keep in contact with OSC staff, and to report any problems right away. In turn, OSC keeps in constant communication with system users by informing them of system changes and by demonstrating by their responses to requests that they are listening. The result? A system that is not only accepted but appreciated by its users, with the potential for being around for a long time.

< Previous |

| Next >