While there were overall productivity gains for the pilot test period, these gains were not consistent across all 20 districts. That lack of consistency prompted us to examine whether or not variations in the test conditions could account for different productivity gains. Because of the small number of districts and the many variations in test conditions, it was not possible to statistically isolate or measure the independent influence of any particular factor. However it is possible with this number of districts to explore whether there are groupings or clusters of districts that correspond to differences in one factor or another. Therefore we used a statistical clustering technique (K-Means analysis) to see if productivity results appeared to be related to two kinds of test conditions: the availability of overtime compensation for the workers outside normal hours, and the favorability of technology conditions (connectivity, access to laptops). That is, the analysis tests to see if districts could be grouped such that high or low measures of productivity were connected with favorable or unfavorable test conditions.

To perform the analysis, each district was rated as favorable or unfavorable for overtime conditions and technology conditions (see Appendix F for a description of coding for overtime and technology conditions). The K-Means analysis then forms clusters of districts to maximize the differences of the averages (means) across the clusters putting the districts that had higher average productivity gains with one test condition (favorable or unfavorable), and lower gains with the other. If districts with favorable conditions cluster with appreciably higher productivity gains that is evidence of a relationship.

The results below come from separate analyses of increases in case closing and progress note entry clustered separately with overtime and technology conditions. Of the four possible results, three showed a substantial relationship between test conditions and productivity gains in the expected direction, and one less so. Those results are shown in Figure 7 through Figure 10 below. It is important to bear in mind that these results are based on examining only one possible influence on productivity. Therefore, the results do not establish that improving overtime or technology conditions will cause improved productivity, but only that a relationship may exist that deserves further attention.

The analysis results in Figure 7 below show evidence of a relationship between higher case closings performance and more favorable overtime conditions. Case closings in districts clustered with favorable overtime conditions were approximately 25% greater than those in the less favorable overtime conditions. The districts are divided almost equally between the clusters as well, suggesting that the possible relationship is more general across the districts.

 

The evidence of a relationship between overtime conditions and progress note improvement does not appear as strong as for case closings. The analysis seen in Figure 8 below shows only a modest 3% advantage of the favorable overtime cluster versus the unfavorable. Also the distribution of districts between the clusters is quite uneven, suggesting that the possible relationship in this instance is less generally important.

 

Differences in technology conditions appear to be more strongly related to productivity results than the overtime analyses above. For the increases in case closings shown in Figure 9 below, the favorable technology cluster performed about 10% better than the unfavorable one. For this comparison, the districts were evenly divided between the clusters, indicating a rather consistent pattern across the districts.

 

A similar but even larger difference is shown in the analysis of progress note entry in relationship to technology conditions. The results in Figure 10 below show a 20% gap in performance between the favorable and unfavorable technology clusters. Though the distribution of districts between the clusters is not quite even, the size of the difference is strong evidence of a connection between the technology conditions and progress note entry.

 

Taken together, the results over all analyses present a predominately positive picture of productivity gains during the pilot period. In terms of the overall volume of work, comparisons between the pre- pilot and pilot test periods show substantial increases. Timeliness of case closing improved, even with an increase in the overall number of cases closed over the two periods. Only the timeliness indicators for progress notes and safety assessments show decreases for the pilot test period. The progress note decrease appears to be accounted for by work on closing a higher proportion of older cases during the pilot period, not by an actual slowdown in the documentation process.

With any new technology implementation we would expect significant interactions with the normal work processes. That seems to be the most likely mechanism at work here. In the absence of a measurement effect, our best interpretation of this timeliness impact is essentially the same as for progress notes, i.e., work on a backlog of cases needing both progress notes and safety assessments. That kind of work pattern would shift the overall proportion of timely and late safety assessments for the pilot test period. This issue may be resolved with examination of more work process data than was available for this assessment.