The effectiveness of the technologies examined in this assessment depend in large part on their overall capabilities and how well they fit with the users’ normal work practices. As the summary in Table 1 shows (below), these technologies differ markedly in how they accomplish two main tasks: (1) converting the users’ words into digital text, and (2) entering that text into the central database. Laptop activities are essentially the same as working at an office PC, except for connection and logon procedures. The other technologies involve additional steps to accomplish the analog-to-digital conversion and complete the database entry process. These additional steps add to the complexity of the work and introduce additional points of possible user error or technical malfunction. These use characteristics of the technologies are discussed in more detail below.

Table 1: Comparison of Technology Use Activities

Device	Data Entry Process
	Analog-to-Digital Text Conversion	Intermediary Steps or Modifications	Placement into CONNECTIONS
Connected Laptop	type and edit at keyboard	none	simultaneous with typing
Digital Pen	write on special-purpose paper and download through software	compile and edit text	cut and paste
Voice Recognition Technology with Digital Recorder	speak with handheld recorder and download through software	edit text	cut and paste
Telephone Dictation Service	speak with cell phone to third party transcription service	retrieve and edit text	cut and paste

The technologies also differ in their capabilities, both as tested and as developed fully. Those differences are shown in Figure 1 (below).

 

The comparison is based on the use of the dictation and digital pen technologies in the field without a laptop computer. The data entry, retrieval, and interactivity capabilities are the ones described by the participants in the assessment. The figure shows the much higher overall capability of the connected laptop computer. Without a laptop or desk top PC, the other devices are effective only for the recording step of analog-to-digital conversion in the field. More specifically, in using Dragon Naturally Speaking (DNS) with a digital recorder, the only step that can be completed in the field is recording. Any conversion must take place in an office when connected to a PC. This is also true with a digital pen. The only variation that would extend the capability to perform functions in the field would be to use DNS or the digital pen in conjunction with a laptop with a wireless connection, but these capabilities were not tested in these initiatives.

The overall results for laptop use, based on the NYC/ACS experience, are positive in terms of user ratings of device performance and support for increased work output. The case workers used the laptops primarily for documenting and reporting, plus for access to information resources both in the central database and on the Web. It was clear from the survey results and workshop comments that the positive ratings of the laptops were based primarily on three factors:

• using the laptop for reporting and documenting required only small adjustments in work practices, since laptop use is very similar to working with a desktop machine
• connectivity to the central database from outside the office provided new or increased opportunities for completing their reporting and documenting requirements
• access to the central database and other online information resources and email made workers more productive.

It was also clear, however, that case workers distinguished among the various locations for use and preferences among the laptop models. They generally did not use the laptops for recording or working in client contact situations, due to concerns that using the device would interfere with rapport or pose a security risk. The workers clearly preferred smaller, lighter laptop models as well, and recognized differences in connectivity among the models. However, the overall ratings for efficiency and recommendations for future use were quite high and very similar across the four models tested. Only the tablet PC and Blackberry device received no or low ratings - the tablet PC due to the fact that it was not equipped to convert handwriting to digital text (recognizing or handling handwriting is also beyond the current capabilities of CONNECTIONS), and the Blackberry’s restriction to email only.

The evidence presented below on timeliness and work product performance of the NYC/ACS technology suggests that there were productivity gains. However, there are too many other factors involved to attribute these gains solely to laptop use. The evidence on timeliness of progress notes entry (Figure 3), volume of notes entered per month (Figure 4), and clearing of case backlogs (Figure 5 and Figure 6) does point to a positive impact of laptop use. However, the case workers reported that because they used the laptops at home, the work time per day was increased and generated overtime pay. They had also been instructed to make an extra effort to clear backlogs during the “crisis” period. So we can say that the laptops supported more productive time during a work day (or week), but not necessarily that the laptops increased efficiency.

The dictation-based and digital pen technologies tested in Monroe and Westchester counties share sufficient characteristics and use issues to be considered together. Both rely on multi-step processes during which the devices transform the users’ analog input (speaking or writing) into text in a digital format. The user then accesses the digital text files for review and cuts and pastes the text into the central database at a desktop PC. These could be called semi-mobile technologies, since they include small transportable devices for mobile analog input, but rely on desktop devices to complete the work process.²

The Monroe County results, involving both the voice recognition system and digital pens, reveal the close relationship among work practices, work situations, and technology use. The roughly one third of participants that used the digital pens found them to be useful for the following:

• note-taking in the field;
• other reporting tasks and occasional input for word processing;
• taking meeting notes that could be easily stored in electronic files, eliminating scattered paper notes;
• possibly completing locally designed forms, to be developed on the specialized paper.

Workers who preferred writing, used the pen in work situations where writing was an established and natural act. They did not, however, generally report pen use as particularly efficient or adaptable for a wide range of tasks

The main efficiency problems with pens appeared to result from the cumbersome process for transferring the text files created by the pen system into the central database, often requiring many intermediate steps. Some pen users also reported inaccuracies with writing recognition and some dissatisfaction with using them for long narrative notes. Pen users also expressed some concern for the cost and availability of the special paper needed for pen use. The users reported good results when the work situation, worker’s preference for writing, and the pens unobtrusive nature aligned, however overall ratings of digital pens were not highly positive.

The telephone dictation system received a mix of high and moderately positive ratings and did contribute to increased work output in the following ways:

• users rated the system moderately efficient and usable;
• most users recommended keeping and expanding deployment of the service;
• several users found the cell phones valuable for other work-related tasks.

The workers also reported problems of poor cell phone connectivity and the complexity of data entry (cut & paste, etc.), which may have tempered the overall ratings. Some workers found dictation difficult and thus slower than typing, which could be mitigated by additional training. Problems with cell phone connectivity were also common, and some workers reported delays in the system processing of their dictation. These difficulties may account for the apparent drop in timeliness for the Westchester pilot seen in Figure 3. However, other measures of timeliness for the three initiatives presented in the following section show different patterns.

Dictation methods presented the same multi-step issues for moving the notes into the central database. Both the transcription service and Dragon voice recognition systems required users to cut and paste the text generated by the system into their reporting database, in one case from the transcription service Website and in the other from the PC running the Dragon application. In principle, the Dragon system could be used to dictate directly into the reporting database. But that would only be possible at a desktop PC with connections to the database, since no laptops with the Dragon system were deployed in the Monroe County test. In this context, the Dragon system by itself is not a mobile technology. The digital recorders intended for mobile recording were not deployed in time to include in this assessment.

The Westchester County pilot test, with slightly over 13 weeks of dictation system use, provided an opportunity to examine how learning to use a new technology might have affected workflow. The other pilot tests did not include data for a sufficient length of time with each new device to show much in the way of learning effects. If there was a learning process affecting progress note production in Westchester, plotting the number of notes entered each week would show a gradual increase, with relatively lower production in the early period, then the volume of notes accelerating as skill and familiarity increase. In Figure 2 (below) we see little evidence of such learning, with an uneven pattern indicating little systematic variation in progress note entry from week to week, though there is some acceleration in the last week. This may be the result of a slow learning process, showing up only in the last week or two. However, it seems more likely from the user’s comments that adaptation was very rapid for those that found the system easy, and much slower or nonexistent for those that essentially rejected its use altogether. The combination of these two effects would result in the same pattern.

 

There is some indication that the types of progress notes entered during the Westchester pilot were different from the pre-pilot period (see Figure 7 below). Our analysis treated all progress notes alike, so the trend in the above Figure may miss important productivity or learning effects.

In addition to use for dictation, some workers employed the cell phones for other work activities. They reported using the phones for setting up client meetings, following up voice mail while in the field, contacting the office for needed information, and even as a date book and appointment calendar. For these workers, an apparently good fit of a mobile technology’s capabilities with a worker’s needs and preferences can be a very effective combination. By contrast, other workers used the same phone for virtually no dictation or other tasks.

The test of the Dragon dictation system in Monroe County provided some data about the system’s effectiveness and use, but none about its mobile potential, since the digital recorders were not deployed in time to include them in this evaluation. We concentrated our evaluation instead on the evidence for efficiency gains, though the time period was too short for firm conclusions. The participants’ ratings of their work on progress notes during the pilot were uniformly lower than for the pre-pilot period. They also rated the Dragon system in the low ranges for improving efficiency and versatility. The timeliness and progress note volume data show virtually no difference between the pre- pilot and pilot period (Figure 5 and Figure 6 below). A longer period of use will be necessary before the effectiveness of the Dragon system can be adequately assessed.

A longer time period will also allow for adjustments to the use problems described by the workers. These included the complexity of the Dragon application itself, the need for additional training, lack of experience and skill with dictation, and the problems of using a dictation system in the office environment. Training and additional experience may mitigate the complexity and dictation skill problems. However, dictation with the Dragon system constrains multi-tasking (typing while on the phone, etc.), which was described as a common work practice. The open office environment also means dictation can be disrupted if the system picks up extraneous noise. The sound of many users dictating simultaneously may also be distracting and disruptive to other work. These problems appear to be reflected in the participants’ low overall satisfaction with the system. They did see value, however, which was reflected in majority of testers who recommended continued use.