Each year, more than 33,000 people receive business permit assistance over the phone from the Office of Regulatory and Management Assistance (ORMA), now called the Governor's Office of Regulatory Reform. To answer the variety of questions posed by callers, Permit Coordinators rely on a database describing nearly 1,200 permits issued by more than 40 different New York State agencies.
This report presents the results of a project that developed a prototype voice response system that would use more sophisticated voice response technology to meet the needs of its customers. The project assessed technical feasibility, cost-effectiveness, business process, and customer service implications of fitting the system into existing operations.
Reviewing the Performance of ORMA's Voice Response System for Automated Business Permit Information: Integrating Technical, Cost-Based, and Customer-Oriented Evaluations of System Performance
Wed, 01 Mar 1995 >Download PDF
Wed, 01 Mar 1995 >Download PDF
This report presents the results of the Center for Technology in Government’s formal efforts to evaluate a New York State Office of Regulatory and Management Assistance (ORMA) prototype voice response system for automated business permit information.
This report has four specific objectives: (1) to review the original research objectives of the ORMA project as it was proposed by ORMA; (2) to document how those original and rather narrowly focused research objectives were eventually expanded to include a broader set of questions; (3) to summarize a multi-method research approach that has been used by CTG to evaluate this entire project; and (4) to present answers to each of the research questions posed. These answers draw from threads of investigation taken from the several methods that tested the overall research approach.
With a small staff and limited fiscal resources, ORMA could not call together special project teams from within or redirect its budget resources without seriously impacting its services. This project required experts in voice information and response technologies, a full time Project Coordinator, and assistance in acquiring the necessary hardware and software to serve as a prototype platform for the voice response applications. The Center for Technology in Government provided project coordination, while ORMA provided its business expertise. A corporate partner, Precision Systems, Inc., provided the technical capabilities. The combination proved valuable to all participants. The lessons learned from the project include the following:
The prototype successfully demonstrated a voice response system that can be easily integrated into its existing phone response system. The project showed that speaker-independent voice recognition (SIVR) is an effective alternative input method for callers who do not use a touch tone telephone.
The complex information needed to obtain business permits can be automated within the context of a voice response system and delivered to callers by both voice messages and fax-back delivery of hard copy.
In the experimental evaluation, both those who called the prototype automated system and those who called ORMA and spoke with Permit Coordinators were generally satisfied with the service they received.
Business permit information is complex and the process by which information is transferred from the agency to its customers is at least as important as the accuracy and completeness of the information content. Regardless of the method used during the experimental evaluation (prototype system, ORMA Permit Coordinator, or unassisted search), the accuracy and completeness of the information transfer process was low. In addition, under all tested conditions, callers had difficulty identifying and completing all necessary referrals to other agencies. We believe the fax-back feature (which was not available during the experiment) will help alleviate this problem. However, it remains an area of uncertainty and should receive continued attention as ORMA moves forward with this initiative.
When the project began, ORMA managers wanted to use an advanced voice processing system to process all permit-related inquiries. During the project, ORMA staff learned how to achieve a higher return on investment by automating only the most routine and most numerous inquiries. They also developed an appreciation for the complicated dynamic links among call volume, lines, ports, staff, and software that will continue to shape their ability to deliver high quality services.
Development of the menu scripts for the prototype system required detailed analysis of the information solicited from a caller by the Permit Coordinators when responding to an inquiry. When this analysis was completed, ORMA staff realized that not only had they produced the menu scripts, but that they could use the logical structure of the scripts as the basis for in-house training materials to orient new staff in the most efficient ways of soliciting information from callers.
ORMA was aware that, in most information systems projects, the future costs of application maintenance could be quite large, particularly in a field where changing regulations and permit requirements would require frequent modification of the application. Working with the project corporate partner, Precision Systems, Inc., staff learned how advanced software development tools can help reduce labor costs associated with modifying and maintaining the application to accommodate changes in procedural and substantive regulatory content.
Due to the knowledge gained through participation in this project, the agency was able to commence procurement of a new system much earlier than would otherwise have been the case.
The cost effectiveness of voice-response technology cannot be calculated or understood by a simple formula. It depends on complex interactions among call volume, telephone lines and ports, number of staff, and the specific automated information products. Such systems should be dynamically modeled before development so that these relationships are well-understood before investments are made.
Much work remains to be done in exploring the factors influencing the accurate and complete reception of information disseminated by an automated voice processing system. In this case we learned that although the prototype proved effective in transmitting information to the experiment participants, there was a much lower degree of accuracy and completeness in their recall and understanding of the information. Any agency considering such a system should take special care to address this issue.
Finally, the project produced some indicators of interest to state policy-makers as they seek to make New York a more hospitable environment for business development.
The existing pattern of state regulatory requirements offers substantial opportunities for cross-agency coordination and process reengineering. It appears that ORMA's business assistance mission does help business entrepreneurs understand and navigate through the regulatory environment. However, half or more of the participants in the experiment said that doing business in New York was more complicated than they expected. The data also suggest that citizens are dissatisfied with the large amount of personal effort it takes to meet regulatory requirements once identified.
Clients may be willing to pay substantial fees for higher levels of service. Participants in the experimental evaluation indicated a uniform willingness to pay fees for more personalized assistance with permit requirements. These hypothetical services all reflected some level of integration among a variety of currently separate requirements.