There were four streams of data collection throughout this project. Online surveys of pilot participants and data from the central OCFS CONNECTIONS database provided quantitative data that were used to assess questions concerning productivity, satisfaction, and timeliness, along with patterns and locations of use of the mobile device. This data was supplemented by qualitative data gathered via workshops with CPS workers in both participating boroughs and telephone interviews with CPS supervisors and managers. Each method is described in greater detail below.

Facilitated workshops with CPS staff from both Manhattan and Staten Island, were conducted on August 27th and August 28th , approximately four weeks after the start of the pilot. Specifically, CTG conducted two three-hour workshops with CPS caseworkers from Manhattan and three one-and-a-half-hour workshops with CPS caseworkers from Staten Island. Supervisors II and managers were purposely excluded to allow for an open discussion with the CPS caseworkers. The difference in length of time and the number of workshops was driven by limitations of physical space at the Staten Island offices, which allowed only for 10-12 interviewees per session. In total, 59 caseworkers attended the workshops representing approximately 42% of all caseworkers testing the laptops in both locations. The workshops in Manhattan were attended by 28 participants, Staten Island workshops were attended by 31 participants. Each workshop was facilitated by two CTG investigators and was designed to elicit information about participants’ experience with the laptop, as well as general information about the overall fit of mobile technology into CPS work.

In order to capture experiences of supervisors II and managers whose work practices differ from CPS caseworkers and supervisors I, CTG conducted a series of telephone interviews with supervisors II and managers who participated in the mobile technology pilot and who were also supervising caseworkers who were assigned a laptop. The telephone interviews took place from October 11th through October 18 and ranged in length from 25 to 45 minutes. In total, CTG conducted five interviews with six people. Namely, we interviewed two supervisors II from Staten Island, 1 supervisor II from Manhattan, two managers from Manhattan and one manager from Staten Island. All interviews were conducted by 2 CTG investigators and were designed to gather information regarding the personal experience of each interviewee with the mobile device as well as their observations of the effects of the laptop computer on the work practices of their subordinates.

The overall objective for using the CONX data was to measure the effect of the use of mobile technologies on CPS work practices by using data from the central database. The CONNECTIONS dataset (i.e., the central database) contained information on case records and caseworkers’ progress notes. The information contained within each of these records include: Stage ID, time-related information the investigation stage (Intake Start Date, Investigation Stage Start Date, Investigation Stage End Date, progress notes information (Progress Notes ID, Progress Notes Event Date, Progress Notes Time, Progress Notes Entry Date, Progress Notes Types, Progress Notes Purposes) safety assessments (Safety Submit Date, Safety Approval Date) logged by caseworkers (Person ID), and districts (District Name). The CONNECTIONS data included progress notes entered April 30th 2007 and October 19th 2007. The data used during this phase of the study included 38,392 entries within 2,402 unique Investigation Stages, from 135 caseworkers. The data was used to compare two time periods: a pre-pilot period (April 29th 2007 – July 21st 2007) and pilot period (July 29th 2007 – October 19th 2007).

A total of 91 cases that were closed between July 22nd 2007 and July 28th 2007 were excluded from the analyses. These cases fell into a period between the end of the pre-pilot and start of the pilot. In addition, there were 43 entries that took place after October 20th 2007 that were not included in the analyses. These entries were created after the end of the pilot period. Finally the data were checked for consistency, and only one entry was deleted as it was for an entry date prior to the actual event date.

The primary objective for the use of surveys was to collect the perceptions and attitudes regarding testers experiences using the laptops. Two separate surveys, a baseline and post-pilot survey, were administered to assess the effect of mobile technology use in Child Protective Services. The surveys were developed over a period of a few months. During this time, the surveys were modified based on the project team’s past-knowledge of the initial pilot study and its understanding of the objectives set forth in this extended pilot study. An online survey was developed using commercial software (Survey Monkey).

The surveys were administered to caseworkers and supervisor I. The names, email addresses, and titles of participating CPS caseworkers were collected from the two participating field offices (Williams Street in Manhattan, and Staten Island). Personalized survey invitations were emailed to participants. This measure was necessary to ensure that only the intended participants were in fact those taking the survey. Both surveys were sent to a total of 135 CPS caseworkers and supervisors I, of which 70 were from the Williams Street field office and 65 were from the Staten Island field office.

The baseline survey was administered prior to the test period and the deployment of laptops to CPS caseworkers and Supervisors I. The baseline survey was open for eight weeks; starting on June 25 2007 and ending on July 20 2007. The survey was designed to collect data from the following thematic categories: CPS work practice, CPS work time, demographic information, mobility, skill and stress levels, technology acceptance, training, and use of technology. The overall response rate for both field offices was 86% (n = 116). William Street’s response rate was 83% (n = 58), whereas Staten Island’s response rate was 89% (n = 58).

The post-pilot survey was administered three months following the deployment of laptops. The survey was open for one week; starting on October 15 2007 and ending on October 23, 2007. Data was collected from three new thematic categories, namely the impact of laptops on CPS Caseworkers and Level-I Supervisors’ daily activities, mobility-related issues, and technical difficulties experienced during the pilot. The post-pilot survey also collected data to compare participants’ pre- and post. The overall response rate for both field offices was 78% (n = 105). Williams Street response rate was 76% (n = 53), whereas Staten Island’s response rate was 80% (n = 52). After both surveys were closed, participants’ responses were downloaded and a data quality check was performed by choosing randomly-selected surveys. Following the data quality checks, the data variables and responses were relabeled and coded.