Prior to the 1980s, developers had few tools to test or simulate an application in a user’s environment. In the early 1980s, prototyping was developed as a way to gain user acceptance or establish technical feasibility by constructing a partial system to demonstrate an intended system’s behavior. During the same period, designers used pilots to provide a trial run of an application and correct any problems before implementation or large scale use.

However, prototypes and pilots did not look beyond the application to the social and organizational challenges raised by a new technology or workflow change. While technically feasible, systems still failed because social and organizational issues became barriers to implementation. In previous projects and research, CTG found that looking at all the issues—technical, organizational, and managerial—mitigated many of the barriers to system implementation. As a result, CTG developed a comprehensive prototyping approach that examines not just technical factors, but larger organizational issues as well.

For this project, comprehensive prototyping was employed within a larger testbed methodology that offered training, guidance, and a “safe environment” in which participants could examine the feasibility of using XML for Web site management within their specific situation. This testbed approach was taken because many agencies confront obstacles to the adoption and implementation of XML-based Web site management despite their recognition of its benefits. CTG wanted to determine if the problems were due to the technology or the social and organizational issues surrounding innovation. The XML Testbed provided not only technical training but also an exploration of the organizational impact and workflow changes that the implementation of XML would potentially cause. The project was designed to help prospective agencies investigate their capability for such an implementation. As part of the Testbed model, agencies had to demonstrate leadership buy-in not only for participation in the Testbed, but also for the potential organizational changes their prototypes might produce.

The XML Testbed involved a series of presentations, training sessions, workshops, and discussions to support the development of agency-specific XML prototypes and business cases:

• Two training sessions were focused on XML programming and its advantages as well as the challenges when used for Web site management.
• Another two-day training session provided the basics of project management, including defining a problem, identifying stakeholders, analyzing workflows and business processes, and developing cost-benefit analysis. Each team developed a written business case for their prototype as a result of this training.
• A final training class was devoted to project management issues such as work breakdown structures, challenges of project management, competing priorities, and how to garner executive sponsorship.
• Six workshops were organized for participants to present their progress in both the development of their business cases and their XML prototypes. The teams were assigned specific tasks to complete between workshops. In addition, each workshop featured a presentation by an industry expert, a Webmaster with relevant expertise, or a university professor that highlighted some potential applications, capabilities, and limits of XML. Several workshops fostered discussions between and within the teams. Opportunities were also provided for people in similar roles (content providers, content reviewers, Web developers) to exchange ideas and concerns.
• Finally, communication among team members was encouraged for solving common questions and problems through the use of emails, office hours with the CTG project team members, and individual project team meetings.

To participate in the Testbed, an agency needed to meet specific requirements established by CTG. The team members had to represent all the roles involved in the Web publishing process from content creation to publishing on the Web. Through the development of their business cases, the agency teams needed to gain the necessary knowledge about their business problem in detail through investigating the workflow and business process currently followed, identifying relevant stakeholders, garnering executive support not only for the prototype but also for the potential future project, and identifying where resistance to change could arise within the organizational units involved in the process. They also had to detail their ideal state—how the process would change by using XML as a Web management tool.

Agency teams were trained in the use of XML and were asked to develop a prototype based on their original agency proposals. Throughout this project, the project teams had the opportunity to further their knowledge about XML for content management and then apply it immediately to an aspect of their environment. Their learning occurred in the context of their organizations. They were able to identify the potential enablers and constraints facing an XML project. The comprehensive prototyping experience not only allowed participants to understand a technology, but also to understand a specific technology application (XML for Web site content management) immersed in a specific context (their state government agency).

The key factor in the Testbed was that participants were engaged and focused on a real project that directly impacted their organization and were asked to work on the project as if they were going to implement it at their site.

The goal of the Testbed was to see if the benefits experienced by CTG in its transition to XML were replicable within state agencies. Because CTG’s organizational environment was different from NYS agencies, its policies and practices in adopting XML could not be simply transferred to other organizations. Furthermore, each NYS agency has its own unique culture and capabilities, so CTG wanted to examine how XML implementation would play out across a spectrum of different agencies.

Initially, CTG developed a one-day Introduction to XML for Web Site Content Management training class that was delivered to over 65 different individuals (primarily Webmasters and IT managers) from over 30 different NYS agencies in 2004. However, follow-up with students indicated that they could not move past the training to implementation. They found it hard to apply what they learned in class into their work place. They were often unable to explore new skills due to competing priorities of their normal work load. And they felt isolated with no community of practice to help support their experimentation.

Training was effective in increasing knowledge of XML, but participants were still missing a supportive environment in which to experiment and apply what they learned. The adoption and implementation of XML involved much more than understanding a technology. It impacted workflows and business processes along with a new approach to managing Web site content. The class attendees realized they were only one piece of a much larger organizational puzzle.

The Testbed was designed to address these questions:

• What barriers prevented these students from applying what they learned in class back at their agencies?
• What additional items besides technical training were needed to successfully explore XML for Web content management?
• If training was not enough, what else was needed?

The Testbed provided the teams with a combination of training and practice within a safe environment. The teams themselves were chosen based on criteria that demonstrated commitment to the process. Each agency team had to provide a proposal for consideration for selection. Within their proposals they were required to discuss their current environment and their goals for participating in the Testbed. Tables 1 and 2 provide a brief summary of each agency and the challenges they hoped to address by investigating XML for Web site management.

The Testbed also required the project team to represent the organizational spectrum from content creators to Webmasters. Since the Testbed design considers adoption and implementation of a technological innovation as more than a technical issue, expertise in XML was not as important as the fact that all roles within the process were represented. Some project team members had previously attended the one-day XML training; others were slightly familiar with the potential benefits that XML could provide. And still others were only familiar with XML from a data exchange perspective. Many had access to information technology expertise within their organization, while others had only minimal support.

The inherent design of the Testbed purposely addressed the shortcomings of traditional training alone. The Testbed allowed for experimentation and exploration. It provided the necessary community to share ideas and explore designs. It also required executive sponsorship for participation, a key aspect that students stated was missing when they returned to their offices after the single-day XML training.

Table 1. Testbed Team Profiles

Agency		# of FTEs1	Environment	XML Web Content Prior Knowledge	IT Dept / Web Team
Department of Civil Service		570	ColdFusion Studio Dreamweaver	Yes	Yes
Office of the Prevention of Domestic Violence		30	Dreamweaver DCJS Hosts Web site	No	No – 1 person
Higher Education Service Corporation		700	Lotus Notes Domino	No	Yes
Office of Cultural Education	Library	170	NYS ED	No	Yes
	Archives	140			Yes
Division of Housing and Community Renewal		940	.NET	Yes	Yes

Table 2. Challenges to be Addressed by XML

	Department of Civil Service	Office for the Prevention of Domestic Violence (OPDV)	Higher Education Services Corporation	Office of Cultural Education	Division of Housing and Community Renewal
Duplication of effort	X	X	X		X
Limited Document Formats		X	X		X
Resource Constraints (time, people)	X	X			X
Accessibility			X	X	X
Cross Platform Compatibility			X
Separation of Content from Presentation	X	X	X
Print		X	X		X
Ease of update or conversion	X				X
Lack of standardization	X		X		X