Chapter Two: Findings and Lessons Learned
Nature, strengths, and weaknesses of available technology
There is an important technology story in the WTC response. Technology failures, technical experiments and innovations, and professional expertise and ingenuity all played noteworthy roles.
Telecommunications infrastructure
"[The loss of the ] Verizon central office was very much a single point of failure for most of our network and most of the City's network... in fact it continues to be an issue today that -- regardless of your vendor, for example, for long distance telephone or data communication service -- they basically lease space or run over Verizon’s network."
The WTC event highlights the importance of defining what constitutes network redundancy and how it should be incorporated into network infrastructure planning. Clearly, having separate service providers who use the same physical infrastructure does not guarantee redundancy or high levels of network availability. Communications networks that were thought to be separate were actually running on the same infrastructure. The Verizon central office at 140 West Street, extensively damaged and rendered inoperable on September 11, was a major telecommunications hub for the City and some of the surrounding region. Although many organizations in the City bought their telephony and Internet services from different providers, nearly all providers ran their services over the same physical infrastructure. The protection organizations expected from separate service providers was therefore never realized. In addition, to a large extent, cellular telephone service depended on antennas atop the Trade Towers. When these were lost in the collapse, neither cellular nor land line telephone services were available for most of lower Manhattan.
A NYC official described the impact on his agency in this way: "We lost phone connectivity and data communications connectivity to all of our downtown locations . . . south of Canal Street. I would say that approximately 2000 to 2500 of our 6000 employees [and] about 2000 of our 4000 network users are south of Canal Street. So we identified that 27 of our 43 locations city-wide were without wide area connectivity . . . .
The impact was not restricted to those locations south of Canal Street but various other sites throughout the city which in some manner or form tied back to the Verizon central office . . . So that's why our impact was much greater than just downtown Manhattan. For the majority of these sites, not only did we lose T1 connectivity but we also lost our ISP and ISDN backup . . ."
Because most telecommunications and utilities in the United States, including New York City, are privately owned and operated, little information about them is shared or coordinated and no comprehensive picture of their relationships exists. Although these systems are often thought of as independent, 9/11 vividly demonstrated that they are actually highly interdependent with each other and with other services. Given this interdependence, the scope and nature of restoration expands in unpredictable ways depending on the seriousness of the attack. Thus, restoring services in an expeditious manner required an ability to quickly determine whether, how, and where these systems could be untangled to be reactivated independently. To coordinate these processes, managers need to have available—in a form accessible in an emergency—inventories of systems and capabilities, plus contact information about employees with the knowledge and skills to do the job. In this case, those charged with restoring services had to work with very inadequate information, setting and re-setting priorities as they learned the extent and nature of the entangled systems. For state agencies alone, more than 2000 circuits serving 40 agencies were damaged or destroyed. Service for public health and safety agencies took top priority, but the process was arduous and unpredictable, and adjusted frequently according to new information gathered in the process of recovery.
Despite these extreme problems, network restoration was still remarkably good under the circumstances for at least three reasons. First, large data networks are designed to be resilient and re-routing could be accomplished for many circuits within hours or days. Second, key operators (primarily Verizon) and government officials were motivated to get the system working as quickly as possible, for obvious reasons. They used every business and knowledge asset available from their own resources and their suppliers to accomplish this. Third, clever means were found to circumvent and compensate for losses of regular network connectivity. The prime example of this third reason may be the City’s use of the Ricochet system, a defunct wireless Internet access system that was temporarily restored to operation to provide networking capacity for responders. The Nextel cellular telephone network, which combines cell and radio capabilities, proved to be particularly resilient and useful as well.
Back up and restoration of data and IT services
"We had the data but we did have a problem with being able to utilize it, because we didn't have a redundant set of hardware."
For all organizations, information technology capacity and redundancy are particularly important assets in an emergency. These can be classified in terms of hardware, software, networks, and the physical locations of facilities. On the hardware and software side, for instance, the New York City Office of Emergency Management’s (OEM) IT functionality was lost when 7 World Trade Center, which contained the EOC, collapsed. OEM retained its software and data, and was able to load that material on machines and networks installed by the NYC Human Resources Administration (HRA) at Pier 92. This effort demonstrated that a loss of hardware is less important than the loss of software and data if two conditions are met: software and data are properly backed up and maintained off site and it is possible to obtain the requisite hardware on short notice.
Nevertheless, the destruction of IT resources in the WTC created two related challenges. One was the need to replace the functionality of the technology lost in the collapse or otherwise made unusable due to restricted access to many buildings. For example, some government agencies lacked back-up hardware on which to restore lost data. Consequently, even when they had data and software readily available to restore operations, they could not quickly resume business. One NYC official recounted the problems of recovering from the loss of access to the agency’s buildings: ". . . the data was backed up off-site. We had immediate access to the data, but we didn't have the hardware or the alternative data center that we could restore this data to, to continue operations. So this impacted [continuation of our regular] programs . . . In some cases, our staff walked up 25flights of stairs in buildings without power, to retrieve hardware so that we could bring them to alternative sites and get them going again."
The second challenge was to identify and put into operation the new facilities and services (such as mapping) needed to respond to the demands of the emergency itself. The GIS operation in the City had not been a central feature of the emergency operations plan, but after only a day or two it became very apparent that spatial data analysis could be extremely valuable for many aspects of the response and recovery effort. The creation of the Emergency Mapping and Data Center on Pier 92 occurred rapidly with government leadership, volunteer effort, and private donations augmenting a cadre of City staff experts. In fact, the entire IT infrastructures of Piers 92 and 94 were built in this way.
Internet
"Are schools going to be open? Are the subways running? ...NYC.gov became an important source of information for that."
From the beginning, the Internet worked when other networks failed. The World Wide Web and Internet telephony were critical in the early hours after the attack when both wired and cellular telephone service failed massively. The Internet provided telephone and text messaging service to key City officials, was used extensively to keep citizens informed of progress, and was the basis for emergency management applications that allowed workers in different locations doing different jobs to collect and transmit information to shared emergency management applications. The Internet technology was used for internal communications in the EOC as well, which set up its telephone system using voice over IP (Internet Protocol) equipment. Text messaging and e-mail were also critically important to communications among the involved organizations. Experts credit the decentralized nature of the Internet with much of its resilience. However, they also caution that electrical failures and interdependencies between Internet and telephone infrastructures pose ongoing risks for future emergencies.
NYC.gov, the City’s public Web portal, was an important source of information for the public during the crisis. Although access to it was interrupted by the collapse of the towers, re-routing of the network was accomplished within hours. Usual applications were replaced with information to keep the public informed of the situation and its effect on their daily lives. Despite tremendous user demand, the site operated smoothly. NYC.gov, a relatively new resource, had been built with a capacity much larger than was necessary for the normal flow of traffic. Therefore, during the crisis, it routinely handled a tremendous volume of news-seeking traffic, which reached four times its peak of usage prior to the attack.
Wireless, mobile, and remote technologies
"We got a line of sight . . . We put up two antennas and they were back [in operation] because we were shooting across the airways. It worked so well, we kept it. It’s now our backup."
Wireless computing and communication capabilities were essential although not widespread. While "line of sight" is a major obstacle in a high-rise city, wireless networks were used effectively to bring some City agencies back online. The connectivity achieved in these cases was excellent and has since been adopted as a backup technology by at least one major City agency. Thanks to this experience, the use of wireless communications, a technology most had not yet worked with, has been greatly expanded since 9/11.
Much equipment used in disasters is mobile – carried by individuals or installed in trucks, buses, and other vehicles. Use of global positioning systems (GPS) has become more widespread since 9/11 as emergency response organizations have come to understand how this satellite-based technology helps them deal with the age old problem of knowing where their equipment and staff are located at any point in time, as well as where they are needed and what routes they might take to get there. Unmanned mobile technology also played a part. Robotic surveying equipment was deployed at Ground Zero to collect data about the stability and condition of the debris pile in areas that were too dangerous for humans to enter.
Remote sensing was judged to be extremely useful but was not advanced enough to be employed in as many ways or places as were needed. For example, the extremely valuable remote imaging that resulted from the fly overs had to be processed on the ground in Albany and then driven by car to the City each day by the State Police. Several respondents described how much more useful it would have been to have technologies in place that would not only collect the data but also send it directly to analysts and users. Given its great potential, remote sensing research has increased in visibility and funding in academic, government, and commercial venues since 9/11.
Retrofitting and adapting existing technology
"We had been working on a project to automate inventory tracking of evidence in the forensics lab . . . and the light bulb came on that maybe they could use it down there in the medical examiners’ office."
Flexible and adaptive use of existing or emerging applications allowed quick response to unexpected situations. For example, a severe weather advisory application for use on the Internet had recently been completed by the City's Department of Information Technology and Telecommunications (DOITT). Designed to give residents directions to shelters and other safe locations, it was quickly modified to notify City residents of changes in transportation systems and availability of housing, water, and electricity. A recently built New York State Police application for keeping track of forensic evidence and DNA samples was immediately installed and adapted for use by the NYC Medical Examiner for the arduous process of identifying victims.
Some technologies were quickly implemented that had never been used in the City before. For example, identification of human remains was a grim task that began as a cumbersome, error-prone manual process in which descriptions and location of body parts was written down by hand and later transcribed and entered into a data base. After a few days, NYC officials searched for and adopted a wireless, hand-held technology that used global positioning and pocket PCs with scanners that could withstand extreme environmental conditions. This allowed data to be collected once, in electronic form, for use in a variety of applications including maps and forensics. Another example is E-team, a collaborative software system for emergency management, that uses the Internet to transmit information collected in a variety of places by different responders and supporting staff. The information is analyzed and integrated and then made available for access by these same workers, giving them a more comprehensive picture of changing conditions and available resources. NYC OEM had already contracted to purchase E-team but the contract had not been completed by September 11. The event prompted immediate deployment with expert assistance from the vendor and experienced users from Florida and other states.
"Hidden" technologies
". . . at the moment when a [military] technology could be of maximum benefit for a civilian population . . . please don’t be bashful about showing up and telling us how you could be of benefit."
Surprisingly, helpful and available resources unknown to the responders were not always offered by those who had them. While many organizations in New York, the nation, and the world spontaneously offered technical and humanitarian assistance, some tools that would have been extremely useful remained unknown to the emergency response teams. These tools for working in hazardous areas, such as technologies that could produce clear images despite thick smoke and haze, had been developed for the US military but were simply unknown to the civilian government agencies dealing with the crisis. Formal emergency response protocols rest on a series of official requests to activate various forms of assistance. Eventually, these resources were discovered because, while touring the EOC weeks after the attack, those who had them asked why they had never been requested. Respondents pointed out the consequent need for a readily activated assistance connection across civilian and military lines that would not depend on knowledge of specific resources as the trigger mechanism for communication.
Technology expertise
"Nothing we did . . . over the whole duration of the recovery was entirely new to us."
The expertise and capacity of IT professionals were diverse, widely available, and readily deployable. Both government and businesses were able quickly to supply this expertise along with equipment and software. Staff of the City's HRA built the internal networks on Pier 92 (the EOC) and Pier 94 (the Family Assistance Center) in two to three days, drawing on the expertise and products of its long-time suppliers and its own finely-honed experiences of dealing frequently with smaller technology crises in its many far-flung service locations. The Pier 92 and 94 experiences showed that it does not matter where the IT capacity exists as long as it can be identified, mobilized, and deployed. Thus, as mentioned above, OEM had software and data but needed hardware and networks; HRA provided that equipment and the expertise to install it. Other organizations, like the IT unit of the State Police, were similarly well prepared to act, discovering that the demands made on them were the same ones they routinely encounter in their regular mission, although much larger in scope and duration.
The private sector response in terms of IT products and expertise was enormous and immediate. Many IT equipment, services, and consulting firms assisted affected organizations in all three sectors – public, private, and nonprofit. This response included equipment, software, systems design and programming, loaned facilities, temporary office space, and project management. Among the most important were the case management system in the Family Assistance Center, imaging applications used by the Medical Examiner, and an umbrella data services organization to serve nonprofit service organizations.