Skip to main content
Chapter One: The Testbed Methodology

Prototype, pilot, testbed

Prior to the 1980s, developers had few tools to test or simulate an application in a user’s environment. In the early 1980s, prototyping was developed as a way to gain user acceptance or establish technical feasibility by constructing a partial system to demonstrate an intended system’s behavior. During the same period, designers used pilots to provide a trial run of an application and correct any problems before implementation or large scale use.

However, prototypes and pilots did not look beyond the application to the social and organizational challenges raised by a new technology or workflow change. While technically feasible, systems still failed because social and organizational issues became barriers to implementation. In previous projects and research, CTG found that looking at all the issues—technical, organizational, and managerial—mitigated many of the barriers to system implementation. As a result, CTG developed a comprehensive prototyping approach that examines not just technical factors, but larger organizational issues as well.

Photograph of XML Testbed Work Session
For this project, comprehensive prototyping was employed within a larger testbed methodology that offered training, guidance, and a “safe environment” in which participants could examine the feasibility of using XML for Web site management within their specific situation. This testbed approach was taken because many agencies confront obstacles to the adoption and implementation of XML-based Web site management despite their recognition of its benefits. CTG wanted to determine if the problems were due to the technology or the social and organizational issues surrounding innovation. The XML Testbed provided not only technical training but also an exploration of the organizational impact and workflow changes that the implementation of XML would potentially cause. The project was designed to help prospective agencies investigate their capability for such an implementation. As part of the Testbed model, agencies had to demonstrate leadership buy-in not only for participation in the Testbed, but also for the potential organizational changes their prototypes might produce.