The Westchester County Department of Social Services tested two technologies at the same time: Santrax, a third-party telephonic dictation service (the system), and cellular telephones. The pilot project spanned thirteen weeks, July 1 to September 30, 2006. Approximately thirty-four child protective service employees and preventive services/foster care caseworkers participated in the pilot from two areas within the district: White Plains and Mt. Vernon. Additional employees from Peekskill began using the technology while the pilot was in progress, however the length of time for each of these participants varied. Twenty-eight responded to the baseline survey and 16 responded to the post- pilot survey, and about 14 were involved in the information gathering sessions and the Final Assessment Workshop, organized by CTG.

The primary technology tested was a telephonic dictation system, Santrax. A service that provides a way for the user to use a cellular or landline telephone to enter data and dictate narrative progress notes. Caseworkers using Santrax call into a dedicated number where they are asked to enter some demographic information about the case using number prompts, and then are instructed to dictate their narrative into the system. The voice recordings are then transcribed by persons that are contracted by Sandata. Within twenty-four hours, the caseworker can access the narrative text in digital form through a secure Santrax Web site. Caseworkers can then cut and paste the text from the Santrax Web site into CONNECTIONS. The secondary technology, the cellular phone, was key in enabling users to access the Santrax system in the field or otherwise out of the office. The full use of both technologies requires CPS employees to connect to a computer (either at their home, office, or by laptop) in order to complete the entry of their progress notes in CONNECTIONS. CTG evaluated both the telephonic dictation system and the cellular telephone separately, though they are complementary technologies.

A variety of technical and use characteristics are associated with each technology and impact individuals’ acceptance and use of the technology to do their work. Through surveys and workshops we gathered users’ ratings and opinions about cell phones including: size, weight, portability, battery life, readability, durability, and quality of cellular connection. We also asked about the specific characteristics of the system including: connectivity to the system, readability and accuracy of transcribed notes, and reliability of turn-a-round (i.e., the ability of the dictation service to produce digitized progress notes in 24 hours).

Participants told us that the most important characteristics overall were:

• Reliability of cellular connection
• Quick turn-around of digitized notes

Table 6 below provides a summary of the average participant’s ratings for cell phone technical characteristics. The size of the cell phone was appropriate for most individuals, as was the weight, portability, and durability of the device. The battery life and quality of wireless connection was rated closer to poor (a mean of 2.93 and 3.36 respectively on a 7-point scale). Many participants commented on how frequently the cellular service dropped calls or they encountered “no service” messages throughout their territories. The poor quality of the wireless connection was often cited as frustrating and bothersome when using the cell phone. One participant said, “I was in the middle of a removal and the call dropped on me, I had to go outside of the house and try to use it there. I was in a situation where I needed the assistance of a coworker, and I had no service.”

Table 6 - Average participant ratings of technical characteristics: Westchester

Technical Characteristics	Cell Phone (n)
size	4.57 (14)
weight	3.67 (15)
portability	3.64 (14)
battery life	2.93 (14)
readability	3.29 (14)
quality of wireless connection	3.36 (14)
durability	3.93 (14)

Notes: Size was rated on a 7-point scale (1 = “Too large” and 7 = “Too small”). Weight was rated on a 7-point scale (1 = “Too heavy” and 7 = “Just right”). Portability, Battery Life, Readability, and Quality of wireless connection were rated on a 7-point scale (1 = “Poor” and 7 = “Excellent”). Durability was rated on a 7-point scale (1 = “Not at all durable” and 7 = “Durable”).

Quick turn around of digitized notes is extremely important to child protective service work. One participant emphasized the importance of a reliable dictation system because of time pressures in completing work stating, “We have deadlines, where we have to do certain things in a certain amount of time, which includes dealing with paperwork.” Early during the pilot period the Local District experienced uneven turn around of digitized notes within 24 hours. One participant said she dictated seven case notes on a Friday, but on the following Monday the notes still were not digitized (though the voice recording was there). She had to spend that morning typing the same progress notes directly into CONNECTIONS. Initially, these service glitches impacted some caseworkers’ ability to meet deadlines, while the effect also dampened enthusiasm for using the new technology. The Local District immediately took action and worked with the third party service to ensure delivery of digitized notes in 24 hours. The vendor implemented an email notification mechanism that informed the implementation team when recorded notes approached the 24-hour limit without being transcribed. Caseworkers expressed that by the end of the test period digitized notes were coming back reliably within the time period.

The system’s transcription accuracy was high. The Local District took the imitative to sample dictated voice recordings and compare them to the transcribed notes. The transcription error rate was less than three percent. In addition, many participants acknowledged that overall, the transcription was accurate. Approximately eight participants used the Santrax system to dictate progress notes in Spanish and had them transcribed and digitized in English. Some commented that this worked well for them, while others had more difficulty with reliable transcriptions because of their accents.

Caseworkers’ ability to read the digitized notes was very important. In the beginning, readability of the digitized notes from the dictation service took some getting used to. The dictation service transcribers had difficulty with punctuation, paragraph divisions, and initially refused to transcribe obscenities included in notes by caseworkers to provide an accurate account of the client’s statements. Initially, caseworkers found that the service provided notes in one continuous narrative and the worker had to break into the appropriate sections. This added additional time to editing progress notes before cutting and pasting them into CONNECTIONS. The Local District worked with the vendor throughout the pilot and the system was refined to include a separation in the digitized notes which highlighted multiple dictation sessions for a given caseworker on a that day and to ensure the obscenities were included.

Different use characteristics are associated with the two technologies. Table 7 below summarizes the average participant ratings for usability in the post-pilot survey. The telephonic dictation service and the cell phone were rated as relatively easy to use and participants were relatively comfortable with the technology after they used it for some time (ratings averaging near 5 on a 7-point scale). Overall, many commented that accessing the Santrax system by phone and the secure Web site was easy and straight forward.

Table 7 - Average participant ratings of mobile technologies use characteristics: Westchester

Interaction Characteristics	Santrax (n)	Cell Phone (n)
ease of use	4.94 (16)	5.27 (15)
comfort with technology	4.75 (16)	5.38 (13)
quicker data entry mode	3.75 (16)	2.40 (10)
encountered technical problems	5.00 (13)	4.43 (14)

Notes: Ease of Use, and Comfort with Technology were rated on a 7-point scale (1 = “Very difficult” and 7 = “Very easy”). Quicker Data Entry, and Technical Problems were rated on a 7-point scale (1 = “Strongly Disagree” and 7 = “Strongly Agree”).

Some participants were less likely to believe that using the mobile technology was a more efficient way to enter data into the system (ratings near the middle of the 7-point scale). The speed of this process depends on workers’ abilities and preferences. In addition, the system is not one continuous process, it includes intermediary steps of retrieving digitized notes, editing, and cutting and pasting.

Participants described in detail the process to get progress notes into CONNECTIONS. After 24 hours, the workers would access the dictated notes on the secure Santrax Web site. They would then edit or add to the notes, then cut and paste them into CONNECTIONS. Some workers avoided the system and continued to type their notes into CONNECTIONS directly because they were fast typists. Others did not mind the process and were able to talk fast, and cut and paste easily.

Participants reported they encountered some technical difficulties with the technologies in the early stages of the pilot. For example, Santrax initially was set up with a three-second pause rule, i.e., that the system disconnects if there is a pause in the dictation of greater than three-seconds. This was a surprise to most caseworkers and also caused some initial frustration. Some participants reported that they could not tell that the call was disconnected, and so continued dictation without recording. Partway through the pilot, the Local District worked with the service provider to extend the pause period to five-seconds. This allowed caseworkers additional time to collect their thoughts. This change was looked at favorably by participants. Some calls were also disconnected because of the low volume of the speaker’s voice and was interpreted as a pause. The need for a higher voice volume limited the number of places that caseworkers could use the service, due to privacy concerns.

These difficulties caused frustration for some participants. For instance, some dictated narratives multiple times because of the dictation service or the cell service dropping their calls. Many suggested it was important to have some type of back-up method for gathering your thoughts (i.e., on a piece of paper) in case there was a dropped call. Some participants reported using landlines to call the service provider to alleviate the problems.

Others were frustrated with the perceived process inefficiencies such as the template for capturing demographic data. It was considered bothersome by some and they either got used to it or developed a work around. Overall, the Local District addressed many of these concerns by working with the dictation service to extend the pause time, develop a prompt that enabled the caseworker to end a call, and requested that the voice component be available within minutes of dictation.

Participants expressed the importance of technical support. Throughout the testing period, the implementation team provided contact information for the service’s technical support, consultation on problems during weekly meetings, and were available by phone or email to resolve user needs. Still, some caseworkers mentioned they did not know who to turn to in order to get the issues resolved. Some went to supervisors, while others decided not to use the system.

The participants also expressed a desire to have the system allow them to dictate notes on multiple families without hanging up and calling back. The current system allowed dictation for one case per call. Caseworkers disliked this constraint because they often have multiple visits in one day and may not dictate their notes until the end of the day. Calling back into the system four or five times is seen as tedious and time consuming. The Local District has asked the service to devise a solution.

How well a technology fits with various locations and modes of transportation is key to its effectiveness. The mode of transportation affects whether people can successfully use mobile devices. To increase the mobility of the dictation service, using a cell phone is optimal. But the service can also be used at the office or at participants’ personal residences through a landline telephone. Factors such as data privacy or confidentiality are important with the system because participants are dictating sensitive case notes. Dictation will not work in all public spaces. Therefore we asked participants where and how they used each of the technologies. Their answers, based on survey and workshop data, appear below.

The system was used primarily for progress notes. Some were able to dictate supporting materials, such as arrest reports, court reports, statements from clients, and medical reports into the system for easy cutting and pasting into CONNECTIONS at a later time. The cell phone, while used in conjunction with Santrax, was also used for other necessary CPS work activities. The cell phone was described as a good tool to use in the field for staying in touch with supervisors, co-workers, and clients, as well as providing a sense of security when visiting homes, and for making collateral contacts while out in the field. Several expressed that they felt connected. Managing contacts was an important benefit of having a cell phone. One cell phone user was described by peers as a “whiz kid.” When asked what the whiz kid did with the cell phone, she remarked that it was used to program clients’ phone numbers into the cell phone for easy access, schedule alarms for appointments, and to listen to messages about appointments (voice mail). Not all participants were able to use the cell phone in this way, admitting that they did not know how.

The most frequent location for using the system and cell phone was in the field in a county-issued vehicle. Cell phones were used in any location where a signal was available, including the client’s home, court, or in their cars while moving from one visit to another. Hands-free accessories which included headsets were provided for privacy while dictating notes. While the Local District did not suggest that caseworkers use the hands-free accessories or headsets while driving, some caseworkers found this to be an effective way to dictate notes from one visit to the next. Some stated that they did not feel comfortable using the cell phone with hands-free accessories while driving, expressing that it was bulky and they had difficulty driving while dictating notes. Therefore, they would pull over to the side of the road, in a parking lot, or down the street from the client’s home. Others reported that they did not have a cell phone or that the locations where they traveled had poor cell phone connection, so they used the system at home or in the office.

The participants unanimously selected the courthouse as the number one location where they hoped to use the mobile technology. They also reported that dictating in the courthouse was difficult because there was no space where they could call into the system in private.

The system is available at any time, anywhere, but the time available to the worker to use the system is limited. The reported time periods they logged on to the system varied from 5 to 25 minutes. There are times when the travel times between visits are lengthy and others much shorter. The non-routine flow of each day impacts the times when they can dictate notes.

Overall, the advantages to using the telephonic dictation system and cell phones were:

• completing progress notes right after a visit
• having the flexibility to use it anywhere at anytime
• keeping in constant contact with supervisors or co-workers creating the opportunity to work outside the office
• dictating in different languages (i.e., Spanish for translation to English)

Overall, the disadvantages to using the telephonic dictation system and cell phones were:

• loss of work, or interruption of work due to failed connectivity or transcription failure slow learning curve that caused some backlog and job stress
• lack of skill and training in dictation for some people
• useful for narrative parts of reports, but not for data entry not connected directly with CONNECTIONS
• editing and copying/pasting is time consuming

Adjusting to a New Technology

The introduction of new technologies or ways of working often is accompanied by an adjustment period or learning curve. Participants expressed that initially, it was time consuming to learn to dictate, master the specialize commands associated with the system, and learn how to retrieve and work with the digitized notes. At first it was a disruption in the way they did work. Several participants concluded it was not natural for them to dictate. By the end of the testing period, some had mastered dictation and others still rejected this way of working. One participant said, “At the beginning it takes time to memorize the commands, but now it is easy.” While another person stated they “there is a learning curve and job change that goes along with it.”

Documentation and Reporting – Progress Notes

Progress notes are important casework documentation, so we examined how the technology use interacted with casework duties, such as improving recall of details, entering notes during down time, or changing work routines. The ratings shown in Table 16 provide a summary of participants’ views about timeframes for preparing and entering progress notes.

The ratings are mixed. All of the characteristics’ average ratings were slightly below the midpoint of the scale. Participants were evenly split over whether they were able to prepare progress notes during down time (43 percent disagreed; and 43 percent agreed) and when asked if they usually entered progress notes all in one sitting (37 percent disagreed; and 37 percent agreed).

Table 8 - Average participant ratings for progress notes timeframes: Westchester

Progress Notes Characteristics	Mean Prior to Pilot (n)	Mean During Pilot (n)
was able to prepare my progress notes during down time	3.81 (27)	3.71 (14)
usually entered progress notes all in one sitting	3.96 (27)	3.75 (16)
usually entered progress notes during regular working hours	3.85 (27)	3.75 (16)

Notes: Progress Notes Characteristics were measured using a 7-Point Scale (1 = “Strongly Disagree” and 7 = “Strongly Agree”).

The dictation capability may have improved the completeness of notes. Many stated that they were able to recall more details when dictating notes right after a visit. One participant said, “My field time was more productive, it’s good to be able to record, while issues are very fresh in my mind.” This advantage did not apply unless caseworkers had the chance to enter notes immediately after a visit. Others stated that the service did not help them because they were already efficient at doing progress notes or that they just did not feel comfortable with or liked using the technology.

Participants reported that the quality of progress notes were not impacted by using the technology, other than being able to recall more details if used after a visit. They said that what constitutes quality in notes varies among supervisors. Some like long, very detailed notes, while others prefer concise notes. Therefore, it was difficult to assess whether the quality of notes were improved by using the technology. In addition, the type of case and type of note determines how much detail is needed; progress notes can vary from as few as 30 to over 2000 words.

Time Savings

Participants’ ratings of the efficiency of these technologies are found in Table 9 below. The results are mixed. For example, the average participant ratings across all three questions is just above the middle of the 7-point scale. More than half (56 percent) reported that they perceived time savings with the device while 44 percent disagreed. During the workshops, some participants described the system as a terrific addition to their tool kits, while others felt that it did not work for them at all. One participant said, “[The] ability to have someone else do the typing saved me very valuable time to do other things that I had to do before and had limited time to do it.” While still another participant expressed, “I have always been up-to-date [with progress notes], I did not need additional tools to complete my job.”

Table 9 -Average Participant Ratings of Devices for Efficiency: Westchester

Efficiency Impacts	Santrax (n)	Cell Phone (n)
the device saved me time	4.06 (16)	4.00 (12)
was a more efficient way to work	3.67 (15)	4.29 (14)
the device allowed me to accomplish other tasks	3.88 (16)	3.83 (12)

Notes: Time Savings, and Accomplishment of other tasks were rated on a 7-point scale (1 = “Strongly Disagree” and 7 = “Strongly Agree”). More efficient way to work was rated on a 7-point scale (1 = “Much Less Efficient” and 7 = “Much More Efficient”).

The mixed pattern is present for cell phones also. While more participants believed that the cell phone saved them time (50 percent), another 42 percent did not. In addition to time savings, many participants commented on the increases in communication that occurred. One participant said, “I like having clients being able to reach me when I am in the field, it saves a lot of time and makes planning much easier.”

Ratings of using the system or cell phones as a more efficient way to work were mixed. While the average participant ratings for telephonic dictation were near the midpoint of the scale (3.67 on 7-point scale) and the average participant ratings for cell phones was higher (4.29 on a 7-point scale), about 47 percent reported that telephonic dictation was an efficient way to work, while 57 percent reported that the cell phone was a more efficient way to work.

There was a split response in participants’ rating that the device allowed them to accomplish other tasks. Of those that stated that it allowed them to do other work, some remarked that they were able to make phone calls or spend more time with client families, or close cases, but this impact is moderate.

Stressors

Overall, normal work routines provide caseworkers with job stress (means above the mid point on a 7- point scale). The introduction of technology added some additional job stress. Many participants reported that when the system goes down or does not work properly, it causes them a lot of job stress. The problems caseworkers faced with progress notes not being digitized in 24-hours or missing were very frustrating because it put caseworkers behind schedule, they had to repeat work, and they were not sure who they had to call to fix the problem. Since management and the public law requires certain tasks to be performed by certain deadlines, technical problems that were out of the caseworkers’ control caused additional job stress. The Local District attended to many of these technical issues. The data collection period did not allow CTG to determine the long term effects of job stress.

Table 10 - Average participant ratings of work-related stress during the pilot: Westchester

General Impacts on Work	Mean Value (n) During the Pilot
normally I was under a lot of work-related stress	4.69 (16)
open cases caused me a lot of stress	4.38 (16)

Notes: Work-related stress, and Stress due to open cases were rated on a 7-point scale (1 = “Strongly Disagree” and 7 = “Strongly Agree”).

Changing Working Habits and Location

Many participants suggested that the use of mobile technologies changed the location in which they could work (i.e., out in the field) and the time their work got done. About two-fifths never work from home, but approximately one out of five participants work from home a few times a week. More than ten percent work from home daily.

The pattern changed for participants working after normal business hours. Almost half of participants worked after normal business hours at least a few times a week. One caseworker said, “We are paid to do a job in 7.5 hours. If you are behind, then it is your fault, if it spills over into your personal time, that is your problem.”

Caseworkers did express concern that the current district policies and administration practices were not currently set up to deal with caseworkers working from home. But they were optimistic that using telephonic dictation could help reduce the amount of time they work after normal business hours.

We asked participants to rate the extent to which they were satisfied with the mobile technologies for doing their work. Overall, satisfaction with both the dictation system and the cell phone were slightly below the midpoint of the seven point scale (Table 11). Some participants were very satisfied, while others were not at all satisfied. Participants reported that they would recommend the device to do child protective (means above 5 on a 7-point scale) and the same is true for their overall opinion of the device (means above 4 and 5 on a 7-point scale). One case worker said, “the technology is helpful if it works well.” This quote expresses what we have heard throughout the evaluation, and that is the need for reliable cellular service, coupled with reliable dictation and 24-hour return of notes to win the full support and acceptance of the technologies from caseworkers.

Table 11 - Average participant ratings in overall satisfaction and recommendation of devices: Westchester

Overall Evaluation	Santrax (n)	Cell Phone (n)
overall satisfaction with device	3.93 (14)	3.77 (13)
would recommend device to be used to do child protective work	5.21 (14)	5.46 (13)
overall opinion of device for your work	4.30 (10)	5.45 (11)

Notes: Overall Satisfaction was rated on a 7-point Scale (1 = “Not at all Satisfied” and 7 = “Very Satisfied”). Recommendation of Mobile Technology was rated on a 7-point Scale (1 = “Not at all Recommend” and 7 = “Strongly Recommend”). Overall opinion was rated on a 7-point Scale (1 = “Low” and 7 = “High”).

In their overall opinion, approximately eight out of ten participants would continue to use the technologies in the field. In addition, all of the participants who answered the question would recommend the devices to their coworkers for use in the field.

Deployment

A district wide or multiple district deployment of the telephonic dictation system and cell phones needs to consider several factors such as 1) the reliability of cellular infrastructure across regions and the state, 2) third-party service quality, 3) improved process and product, 4) the need for substantial training, and 5) change management.

The current statewide cellular infrastructure is well established and several quality service providers are available. Even with relatively well established infrastructures, there are still areas without reliable cellular connections. Since the mobility benefits of using telephonic dictation are optimized on cellular phone service, assessing the reliability and networks of different carriers and their service guarantees is very important. In addition, back up plans and procedures should be evaluated and communicated to employees in case of any major cellular outage.

A system dependent on a single service provider is at risk. If that vendor fails, alternatives are needed. Planning should include assessment of the scalability of the service to hundreds, if not thousands of caseworkers and thousands of cases. Similarly, reliable user support will be important to overall success. Quality and service guarantees would need to be established for working with the third-party vendors to ensure they can deliver reliable products.

Many participants said that the system and way of working is not a turn-key process. Therefore, adequate training in dictation, the use of the system, and basic computer skills (i.e., cutting and pasting) should be provided and coordinated to ensure that caseworkers have the skills necessary to realize process improvements. One caseworker said it would be desirable to have templates that would allow caseworkers to automatically cut and paste information into CONNECTIONS, but there are no plans to provide this enhancement to the system.

The implementation team mentioned that caseworkers’ willingness to accept the technology was influenced by other coworkers. When one had difficulty, others did not try it or just stopped using it. It is possible to establish work groups that help each other learn the technology.

Security

The Local District experienced some security concerns with using a third party service provider, namely: 1) the extent to which information would be kept confidential by vendor employees transcribing notes, 2) the policies related to disposal of the information, voice recording, and records in general, 3) the security of the transmission of the notes through the Internet and, 4) the security of the server and Web interface used to access the digitized notes. Another issue mentioned, but not of major concern, was the level of privacy available when dictating progress notes in public spaces.