Monroe County tested a voice recognition software, Dragon Naturally Speaking (DNS) and digital pens. The pilot project began in mid-September with the distribution of DNS and is scheduled to end in December 2006. The test included 47 Child Protective Services (CPS) case workers and supervisors in the pilot. Of these, 23 responded to the post-pilot survey, 26 participated in the baseline survey, and 20 participated in the information gathering sessions and the Final Assessment Workshop, organized by CTG.

The two technologies differ in three main aspects: the required skill sets, the physical and operational characteristics of the technology, and mobility. The voice recognition software is a technology that requires the users to dictate their notes. Digital pen users recorded their notes by writing on special- purpose paper. The voice recognition software is an application that is installed on a computer, while digital pens are physically separate devices. The voice recognition software can be a mobile technology if used with some other device, such as a digital recorder or is installed on a portable computer. The digital pens are naturally mobile, but require a PC (desktop or laptop) to extract and interpret the writing stored in the pen. Both the voice recognition software and the digital pens collect analog data and convert it into text in digital format for the user to store and manipulate.

Caseworkers using the voice recognition software dictate their notes into a MS Word document and can edit the content at the same time. After completing this process, users cut and paste the dictated notes into CONNECTIONS. The voice recognition software was capable of allowing the caseworkers to dictate notes directly into CONNECTIONS, but this capability was not intended for direct entry and therefore was not fully functional and did not receive high ratings for efficiency. Caseworkers who used the digital pens wrote their notes on the special-purpose paper, storing a copy of the notes on the internal memory of the digital pen. Upon returning to the office, caseworkers downloaded the stored data onto the desktop PC, which converted the image of each page from the pen to a separate MS Word file. Caseworkers could then copy the notes into a single document, edit the recorded notes, and then paste the notes into CONNECTIONS.

This section reports on how people rated characteristics and interactions with the technologies used during the pilot. Technical characteristics are physical characteristics or technical features. Interaction characteristics reflect how users rated the technology when using and interacting with it.

Participants felt that the data entry methods used by the voice recognition software were slightly problematic and unnatural, and that the technology was not as portable as desired. Overall, participants were satisfied with the digital pen technology. Participants believed that the weight, portability, battery life, and durability of the digital pen were very good. However, ratings on the size and data entry method were slightly below average. While size, weight, battery life, and durability were not applicable to the voice recognition software, the ratings illustrate that participants consider the portability and data entry to be poor.

Table 12 - Average participant ratings of mobile technologies technical characteristics: Monroe

Technical Characteristics	Dragon Naturally Speaking (n)	Digital Pen (n)
size	NA	3.38 (13)
weight	NA	5.15 (13)
portability	2.20 (20)	5.46 (13)
battery life	NA	5.27 (11)
data entry	3.09 (22)	3.25 (12)
durability	NA	5.36 (11)

Notes: Size was rated on a 6-point scale (1 = “Too small” and 6 = “Too large”). Weight was rated on a 6-point scale (1 = “Too Heavy” and 6 = “Just right”). Portability, Battery Life and Data Entry were rated on a 6-point scale (1 = “Poor” and 6 = “Excellent”). Durability was rated on a 6-point scale (1 = “Not at all durable” and 6 = “Durable”).

During the information gathering sessions, participants identified advantages and disadvantages to using the mobile technologies. By using DNS, participants were able to enter information directly into CONNECTIONS without the need to copy and paste. Since the software was not intended for direct entry, technical issues prevented this process from working efficiently. Participants considered the digital pens to be helpful in ending the duplication of basic notes, since they can digitize the notes they are already taking. However, participants identified some disadvantages using the digital pen: they are highly sensitive to handwriting styles and were not able to interpret shorthand notes, which led to caseworkers needing to adjust the way they write. Finally, caseworkers felt that the small size of the digital pen was a liability, making it easy to lose.

The voice recognition software was used for progress notes, e-mail, and case summaries. Caseworkers noted that dictating directly into CONNECTIONS was possible, but quite slow. In contrast, digital pens were used for general note taking in meetings and field contacts. The pens were unobtrusive and could be used with clients. CPS workers could also practice with the pens at home. Pens were used for progress notes, safety assessments, and investigation notes. In general, CPS workers complained about the cumbersome task of cutting and pasting from MS Word into the central database.

Table 13 - Percentage of workers using the mobile technology and mean time in weeks: Monroe

Mobile Technology		Use (n)	Mean Time Spent (n)
The voice recognition software		100% (23)	3.05 (20)
	Version 8	100% (17)
	Version 9	0% (9)
digital pen		48% (23)	1.40 (10)

All caseworkers who participated in the survey used DNS version 8. On average, they used DNS three weeks, with a range from one to six weeks. About half of the participants used digital pens for an average of about one and a half weeks, with a range from one week to three weeks.

Use and Interaction Characteristics

Regarding the interaction characteristics, DNS was not very easy to use and people did not feel very comfortable with it. Participants encountered some problems that were out of their control. They disagreed that it was quicker to enter/input/dictate their progress notes using the voice recognition software. As for the digital pen, the analysis shows that participants found it relatively easy to use and they felt comfortable with it. However, they also encountered important technical problems and thought it was not a more efficient way to enter/input/dictate their progress notes.

Table 14 - Average participant ratings of mobile technology use characteristics: Monroe

Interaction Characteristics	Dragon Naturally Speaking (n)	Digital Pen (n)
ease of use	3.62 (21)	4.00 (12)
comfort with technology	3.36 (22)	4.00 (12)
quicker data entry mode	2.74 (19)	2.60 (10)
encountered technical problems	3.62 (21)	3.91 (11)

Notes: Ease of Use was rated on a 6-point scale (1 = “Very Difficult” and 6 = “Very Easy”). Quicker Data Entry Mode, Comfort with Technology, and Encountered Technical Problems were rated on a 6-point scale (1 = “Strongly Disagree” and 6 = “Strongly Agree”).

Work Location

The voice recognition software is not a mobile technology without a digital recorder or a laptop. Therefore it was used mainly at the office. The digital pen is an unobtrusive device and can be easily used for many types of activities and in many different locations. Caseworkers felt comfortable using it on home visits. During the information gathering sessions, participants also identified one key advantage to using digital pens for CPS work in general: the digital pens are good for writing when waiting in court, which is when a caseworker would normally be documenting their cases.

Types of Activities

Slightly more than three quarters of participants used the voice recognition software for their progress notes and about half used it for Microsoft Office applications. About one-third of them used it for investigation conclusion, and only a few participants used it for the other uses shown in Table 15 (below). About a third of the participants used a digital pen for their progress notes and Microsoft Office applications. A few participants used it for investigating conclusion, e-mail, and safety assessment.

Table 15 - Percentage of Caseworkers using the Mobile Technology: Monroe

Task	Percentage of Caseworkers using the Mobile Technologies (n)
	Dragon Naturally Speaking	digital pen
progress notes	83 % (23)	35 % (23)
adding/changing case worker or planner	4 % (23)	0 % (23)
demographics	4 % (23)	0 % (23)
safety assessment	9 % (23)	4 % (23)
risk assessment profile	9 % (23)	0 % (23)
Investigation conclusion	30 % (23)	9 % (23)
Microsoft office applications	44 % (23)	30 % (23)
e-mail	17 % (23)	9 % (23)
schedule organizer	4 % (23)	0 % (23)

Some caseworkers reported using DNS to complete progress notes faster, being able to dictate faster than type. For faster typists, the DNS system did not have this effect. Increased efficiency, or the lack of it, may be related to the complexity of the DNS system, which in the beginning, requires training. In fact, the time and effort needed to become adept may be high, which produces frustration for some of the caseworkers.

Digital pens also had some impacts on overall work, but caseworkers had high expectations that were not always met. Some expected the pen output to look like a finished product, but it did not. Others wrote more and with more complete sentences, so that the notes would be closer to a finished product and more easily placed in CONNECTIONS. Some users wrote brief notes and incomplete sentences and then modified the text prior to entering it into CONNECTIONS. They also noted the potential to use the pens during opportune moments during field work to develop finished products for uploading.

Impact on progress notes

The participants rated several characteristics of their progress notes before and during the pilot (see Table 16). According to these ratings, the length of the progress notes improved during the pilot. Caseworkers also reported being able to enter their progress notes during regular office hours. In contrast, most of the other progress note characteristics worsened during the pilot. This may be a result of the problems of learning a new technology and the complexity of incorporating it in their daily work.

Table 16 - Characteristics of Progress Notes before and During the Pilot: Monroe

Progress Notes Characteristics	Mean Value (n)
	Prior to the Pilot	During the Pilot
about right length	5.15 (27)	5.24 (21)
completeness	5.30 (27)	4.81 (21)
adequate details	5.70 (27)	5.00 (21)
up-to-date	4.44 (27)	4.43 (21)
able to prepare of progress notes out of the office	3.81 (27)	1.89 (19)
able to enter progress notes into CONNECTIONS in the field	3.84 (25)	1.15 (20)
able to enter progress notes all at one sitting	3.96 (27)	3.86 (21)
able to entered progress notes during regular work hours	3.85 (27)	4.14 (21)

Notes: Progress Notes Characteristics were measured using a 7-point scale (1 = “Strongly Disagree” and 7 = “Strongly Agree”).

Participants reported that neither the voice recognition software nor the digital pens saved them much time. Participants generally disagreed that the voice recognition software or the digital pens aided in accomplishing other tasks. In addition, they concluded that using the technology is only moderately efficient for doing their work.

Table 17 - Average Participant Ratings of Devices for Efficiency: Monroe County

Efficiency Impacts	Dragon Naturally Speaking (n)	Digital Pen (n)
the device saved me time	2.52 (21)	2.25 (12)
was a more efficient way to work	3.53 (19)	3.36 (11)
the device allowed me to accomplish other tasks	2.32 (19)	1.82 (11)

Notes: Time Savings, and Accomplishment of other tasks were rated on a 6-point scale (1 = “Strongly Disagree” and 6 = “Strongly Agree”). More efficient way to work was rated on a 6-point scale (1 = “Much Less Efficient” and 6 = “Much More Efficient”).

Participants mentioned one major disadvantage of using a digital pen: they produce separate files for each page of notes written on the special-purpose paper. They are thus required to cut and paste from individual files into a single MS Word document, and then from the MS Word document into CONNECTIONS. The inability to have multiple windows open at the same time in the central data application makes this problem even worse. Note entering takes extra time as a result of this cumbersome process.

Impacts on job stress

The pilot test experience appears to have had a small impact on stress levels. Prior to the pilot, participants rated their stress levels only slightly higher than during the pilot period. The survey responses also showed that stress due to open cases was rated slightly lower during the pilot period.

Overtime issues

Prior to the pilot only a small percentage of participants worked from home on a daily basis. Only one-third of the respondents indicated they did so a few times a week, while slightly over one-fourth rarely worked from home, and nearly two-fifths stated they never worked from home. There was an overall decrease in the number of participants working from home during the pilot. No participants reported working from home on a daily basis, and only one in ten worked a few times a week. Nearly one-third of respondents rarely worked from home, and than half never worked from home. The number reporting never working from home rose from 37 percent prior to the pilot, to 62 percent during the pilot, though this may be in part a result of fewer respondents to the later survey.

Table 18 - Percentage of caseworkers that work from home or after hours to complete progress notes – Monroe County

General Impacts on Work	Prior to the Pilot					During the Pilot
	N	Daily	A few Times/ Week	Rarely	Never	N	Daily	A few Times/ Week	Rarely	Never
worked from home to complete progress notes	27	4%	33%	26 %	37%	21	0 %	10%	29 %	62%
stayed in the office after normal work hours to complete progress notes	27	26%	41%	15 %	19%	21	0 %	57%	33 %	10%

In general, the technology use seems to have produced a shift: reduced extra work time spent at home and increased after hours work in the office. Prior to the pilot slightly more than one-fourth of the participants worked after hours at the office on a daily basis, less than half a few times a week, about one in ten rarely worked after hours, and about one in twenty never worked after hours. During the pilot, the daily after-hours work dropped to zero, but the occasional rate rose to 57 percent. Only one- tenth of the participants never worked after working hours. This may be a consequence of the desktop installation of DNS and the unavailability of the digital recorders.

Dragon Naturally Speaking

The most common advantage mentioned during the information gathering sessions was related to how well dictation fit with workers’ preferred way of recording information. Those caseworkers who could dictate faster than type preferred the voice recognition system. The workers also reported that dictating helps to relieve users’ wrists and hands, thus limiting the risk of repetitive stress injuries.

Caseworkers also identified disadvantages of DNS. They did not like the inability to multitask while using DNS (i.e., making phone calls while typing). The requirements for training were considered too time consuming for caseworkers’ current work-schedule. They also described DNS as often difficult to use in cubical office environments due to ambient noise and dictation often being audible across many workplaces.

Digital Pens

Participants identified several potential advantages with digital pens. The technology could be used to complete local forms. In fact, the software has a tagging capability that when used with special forms it can put the data directly into a Word document. However, these special forms would need to be ordered from the manufacturer at an additional cost.

Some participants mentioned that the digital pens are useful for taking notes or numbers, but not long narratives such as progress notes. For instance, they were useful for an administrator and an IT person to use for short meeting notes. Therefore, using the technology and the pads could be beneficial if the data captured is routine, rather than narratives. Finally, supply of special-purpose pads is also perceived as a potential problem due to its relatively high cost.

Overall Satisfaction

Table 19 shows that participants were only mildly enthusiastic about these technologies overall. Their inclination to recommend these systems for further use was not strongly positive. However when asked for a simple “yes” or “no” about future use or recommending the technologies, the users were mostly positive, though more for the digital pens (72% yes), than for the DNS (65% yes).

Table 19 - Average participant ratings in overall satisfaction and recommendation of devices: Monroe

Overall Evaluation	Dragon Naturally Speaking (n)	Digital Pen (n)
overall satisfaction with technology	3.19 (21)	3.08 (12)
would recommend device to be used to do child protective work	3.57 (21)	3.75 (12)
overall opinion of technology for your work	3.42 (19)	3.55 (11)

Notes: Overall Satisfaction was rated on a 7-point scale (1 = “Not at all Satisfied” and 7 = “Very Satisfied”). Recommendation of Mobile Technology was rated on a 7-point scale (1 = “Not at all Recommend” and 7 = “Strongly Recommend”). Overall Assessment was rated on a 7-point scale (1 = “Low” and 7 = “High”, with a “No opinion” option).

The deployment issues for DNS are typical of new workplace technologies. There was not adequate training or change management provisions included in the pilot test. The hardware did not appear to consistently meet the DNS system’s minimum requirements.⁴ DNS is a relatively complex tool and dictating is not natural for many workers nor was it a common work skill prior to the pilot project. In addition, OCFS does not currently support DNS, resulting in possible lack of technical support. Although DNS does have the advantage of being a stand alone product, rather than a system, thus requiring no ongoing expense other than periodic upgrades.

Deployment of the digital pens involves similar problems, aside from dictation skills. Further development of input capabilities would depend on support for future forms development. The full potential of digital pens will depend on investments in forms applications for easier input to CONNECTIONS.

Using a digital recorder in conjunction with the voice recognition software can pose some security concerns. The contents on a digital recorder cannot be encrypted, hence the loss or theft of a device containing notes could result in loss of confidential information. However, this problem is similar to the potential loss of paper notes which are no more secure, but in common use. In addition, dictating in public would seldom be possible, so caseworkers would need private places to use the recorders for dictation.

The digital pens present similar security concerns. The main potential issue was that there are no easy ways to secure the pens, and therefore, loss of the device would result in loss of confidential information.