The data extracted from the central database provided some insight about three questions related to possible technology impacts: (1) Does timeliness of reporting change? (2) Are workers more or less productive with respect to progress notes and reporting? and (3) Does technology use affect the kind of reporting activities undertaken by the CPS workers? We were able to find some partial answers to these questions.

The indicator of timeliness we used was the number of days between an event and the entry of progress notes concerning that event. Progress note records in the central database include the date a note is entered, the type of note, the related event date, and other case information. We focused on the number of days between each note and its related event. We then plotted the percent of all notes entered for each district by days from the related event. The higher the proportion of notes in the earlier days, the more timely the overall reporting process. The results for all districts, both prior to and during their pilot tests are shown in Figure 3 (below).

The differences in timeliness across the three districts fit well with our understanding of the pilot test situations. The pre-pilot versus pilot indicators for Monroe County are essentially the same— apparently no substantial technology impacts on timeliness. This would be expected, given the very short time and incomplete deployment of the technologies for this pilot test. There is an approximately 10 percent improvement in timeliness for the NYC/ACS, which is consistent with the reported effectiveness of the laptops used and the “crisis” period management instructions to the workers. There is a decrease in the timeliness indicator for Westchester County, which is consistent with the mixed reports on effectiveness and acceptance of the telephonic dictation technology, as well as with technical problems reported with the system and cell phones. After the first two days following an event, the timeliness indicators for the three districts are much closer together and are virtually indistinguishable over the longer term.

 

A different indicator of timeliness was also examined: the average elapsed time, in days, between event and progress note entry. This measure shows a different pattern altogether. The average number of days between an event and progress note increased for all three districts: from two to four days for Monroe, from two to over six days for NYC/ACS, and from two to over nine days for Westchester. For NYC/ACS, this rather large difference may be the result of what appears to be time devoted to completing documentation for older cases during the pilot period (described below). This does not, however, apply to the other districts. The Westchester County participants entered higher volume of notes during the pilot periods, compared to the pre-pilot period (described below). That larger volume may also include cases with much longer delays between event and note entry, which would raise the average event-to-note delay. There are also outliers—entries that occurred many days after the event—that have a disproportionate effect on the average.

Possible work flow effects of the technologies appear in the different rates of progress note entry. There was a marked increase in the number of progress notes per month during the pilot test periods for NYC/ACS and Westchester, shown in Figure 4 (below). For both of these districts, the increase is substantial and not likely attributable to seasonal fluctuations in case load. The ACS increase may be caused in part by the “crisis” situation as well as by the technology use. It is also possible that the so- called Hawthorne Effect may account for some of these large increases in progress note entries.³ The difference for Monroe County is too small to be meaningful, as would be expected for the short pilot time period.

 

The increases in efficiency reported for the technologies may provide the workers with the capacity to reduce case backlogs. If so, the age of cases closed during the pilot period should be higher, on average, than during the prior period. So we compared the proportion of cases closed before and after the 60 day closing requirement for the pre-pilot v. pilot test period. Those comparisons are shown in Figure 5 and Figure 6 below.

Only the ACS data show evidence of backlog clearing, with the proportion of older cases closed during the pilot at more than 65%, compared to just over 40% in the prior period. Just the opposite shift appears in the Monroe and Westchester data (Figure 6). The increase in the newer case closing in Monroe and Westchester can be seen as evidence of productivity increases as well, since the actual rate of case closing increased for these two counties during the pilot.

The percentages shown above are useful for comparison across districts, but they do not tell the whole story. The NYC/ACS case load during the pilot test period included a much larger number of cases open over 60 days (261), compared to 83 in Monroe and 66 in Westchester. Therefore, we would expect to see a substantial increase in older cases closed in the NYC/ACS district, given the combined impact of a substantial backlog to work on, new technology, and increased incentives and resources.

New technology is seldom neutral with respect to work practices. We would therefore expect the introduction of these new technologies to result in changes in what kind of work is done, as well as the speed or quantity. The descriptions of work impacts in the three districts provide a qualitative picture of some impacts. The data from CONNECTIONS shows some differences in types of impacts as well for Westchester and to a lesser degree for NYC/ACS. The patterns for the three districts are shown in Figure 7 (below). The figure shows the proportion of four kinds of notes: (1) attempted contacts, (2) contacts, (3) collateral contacts, and (4) summary notes both prior to and during the pilot test periods. These are only a few of the many types of notes, but the numbers of other note types were too small for meaningful comparisons.

The main technology impacts appear to have occurred in Westchester County. The increased number of notes seen for Westchester in Figure 4 above appears to come predominately from increased numbers of field contacts as well as increased documentation. There was a proportionate reduction in collateral notes. This could be the result of increased time for field work due to the mobile devices, reducing the need for travel to and from the office for note entry. In contrast to Westchester County, the ACS pilot period differed less compared to the prior month. There were proportional reductions in contact notes relative to collateral notes, which is likely related to the large increase in work on older cases in NYC/ACS. As expected, the Monroe County comparisons show very little pre-versus post pilot period differences. Larger differences may emerge over a longer test period.