This assessment report was prepared by the Center for Technology in Government (CTG) (see Appendix A) under a contract with the NYS Office of Children and Family Services (OCFS). The purpose of the work was to assess the performance of mobile technology deployed in a pilot test program with child protective service (CPS) workers. The mobile technologies were deployed to a sample of CPS workers for use in their field work and reporting responsibilities. The pilot was conducted in three Local Departments of Social Services (Local Districts): the New York City Administration for Children’s Services (NYC/ACS), Westchester County Department of Social Services, Family and Children's Services, and Monroe County Department of Human Services, Child and Family Services Division. OCFS engaged the Center for Technology in Government to conduct this assessment and provide a report to the Commissioner of OCFS to assist in decision making and planning for possible further deployment of these technologies.

This assessment covers technologies deployed in initiatives developed by these three Local Districts. Although they come under the oversight of the OCFS, these Local Districts were responsible for administering their programs according to local conditions and policies. As a result, each Local District developed their own mobile technology strategy and determined the length of their respective testing periods. The NYC/ACS pilot ran May-August, 2006, the Monroe County pilot began in September and is scheduled to run through December of this year, and the Westchester County pilot ran July-September, 2006. The assessment therefore examined each initiative in part as a separate activity and also collected additional data about overall work flow. Despite the difference across the three initiatives, the results provide useful insights into technology use and evidence of positive impacts on performance.

The assessment included a review of research on current practices throughout the United States (see Appendix B), which showed that New York is one of only a few states leading in the testing and deployment of mobile technologies for CPS workers. New York State’s approach is unusual in testing several different mobile technologies. Because this is a new area of technology deployment and use, there is much uncertainty about effective mobile technology strategies. Thus a pilot strategy like the one reported here can be a useful way to test several technology alternatives and gather lessons to better inform decision makers about further deployments. These lessons are needed because technology innovations of this kind face can face significant organizational and technical challenges.

The assessment results come in part from surveys of CPS professionals who used the mobile technologies. They also participated in a series of assessment workshops and interviews across the three Local Districts. The results reveal much about the way participants used the technologies and how the devices performed. In addition, data from the central database was analyzed and provided some evidence of technology impacts on work flow. Within the constraints of the pilot conditions, the results do provide evidence of the value of mobile technologies for CPS field work as well as important lessons and guidance for further technology deployment and testing.

Overall results are linked to the particular technologies used in the three Local Districts:

• Laptop computers with wireless network capabilities (NYC/ACS District) - The overall results for laptops are positive in terms of user ratings and increased work output. Positive laptop ratings were based on: (1) value for reporting and documenting required only small adjustments in normal work practices, (2) mobile connectivity to the central database expanded opportunities to complete reporting requirements, and (3) access to the central database and other online resources made workers more productive. Reporting volume to the central database increased substantially during the pilot period compared to the previous month, as did the rate of closing older cases. Deployment and use issues included the desire for more reliable connectivity and easier logon arrangements, which would increase the value of laptop use as well, and review of caseload and work policies related to the new capability to work outside the office and over extended hours.
• Voice recognition software and digital pens (Monroe County) - The initial experience with both voice recognition and digital pens include positive results and challenges. Over the short time for learning to work with these tools, some workers were able to increase their reporting efficiency. Others found dictation difficult due to lack of experience and that the transfer of text from the digital pen and the recognition software to the database involves too many steps to improve efficiency. This leaves much uncertainty about the potential value of these technologies to efficiently substitute speaking and writing for typing reports directly into the central database.
• Telephone dictation and translation services accessible from cell phones (Westchester County) The dictation technology did show overall positive impacts in terms of volume of reporting increases and worker acceptance. The volume of progress notes entered per month during the pilot increased by over 90% compared to the prior month. User ratings of the dictation service for effectiveness and general use were high as well, although low dictation skills did remain an issue with some workers. One of the positive results was high worker ratings for the value of cell phones, both for the dictation service and for other communication needs in field work. Some of the challenges included: Connectivity and reliability issues early in the pilot, which were largely resolved by its end, and the system does not allow dictating directly into the central database, requiring users to cut and paste text from the transcription service.

The evidence of mobile technology’s effectiveness boils down to three key observations:

• The effectiveness of any mobile technology strategy appears to depend on a combination of worker preferences, work practice demands, the capabilities of the various devices and systems deployed, and organizational support. More than one technology may have value for CPS workers. Some provision for individual worker preferences should be a part of future strategies. And much more attention is needed to training, technical support, and adaptation of work policies to support the mobile CPS worker.
• The testers most often rated the technologies as effective, in spite of challenges in the early deployment and use, and with few exceptions recommended continued deployment. The testers also reported sophisticated and nuanced assessments of the strengths and weaknesses of the various devices. One clear lesson is that the users’ experiences and judgments must be an important part of any future tests and decisions about wider deployment.
• The two mobile strategies that had the longest tests—the telephone dictation system and connected laptops—showed evidence of improved timeliness and overall greater work productivity. These conclusions deal with the recording of progress notes, but not the outcomes of the related cases. Further data collection and analysis regarding case outcomes is needed before firm conclusions can be made about technology effectiveness.

The promise of mobile technology to improve child protective services seems clear. But rapid progress toward achieving that promise will require strategies that include significant attention to training and change management, and ongoing evaluation. The success of any new technology depends on human and organizational conditions at least as much as the devices and systems themselves. The ability to work with the laptops outside the office at any time raised concerns about impacts on caseload and work policies that require further attention.

Based on the full range of assessment results and the current practices research, the report includes recommendations for further technology deployment and evaluation. The following statements present the highlights of the recommended next steps, more detail is available in the report.

More focused technology assessment: The most significant potential for improving CPS work was using laptops in the field. Continue to evaluate laptop use and connectivity enhancements.

Test additional combinations of dictation and laptops: Additional improvement in CPS work may be available by using a dictation device coupled with the laptop’s functionality

Take a broader look at caseload management: The introduction of mobile devices provides a partial mechanism to address productivity and quality issues. Attending to improved caseload management from a technological perspective only may limit the potential for success.

Focus on change management and overall support: An understanding of the implications of change in any process for users, support staff, and executives is essential for any large-scale deployment.

Provide training, training, and more training: Investments are needed in training for all mobile devices deployed in every district.

Develop additional measures for improvement: These assessments should include expanded indicators of improvements in CPS work outcomes.

Address policy issues related to a more mobile workforce: Policies that govern work schedules and compensation need to be re-examined so that they reinforce rather than work against the goals associated with the use of new mobile tools.

Align wireless security provisions with the guidance of oversight agencies: All security policies should be developed in accordance with the NYS Office for Technology and NYS Office of Cyber Security and Critical Infrastructure forthcoming wireless security policies.