Skip to main content
photo
 
Evaluation Findings

We evaluated these two experiments for three purposes:
  1. To assess their effectiveness in creating or enhancing long-term international research relationships.
  2. To determine their effect on participants’ individual careers, international and cultural awareness, and scholarly development.
  3. To identify replicable practices and strategies.
To do this, we collected and analyzed several kinds of evaluation data. For the working groups, we surveyed all participants, observed working group meetings, conducted periodic phone interviews with co-chairs, and organized a two-day refl ection workshop involving representatives of all three groups. For the iGov Institute, we conducted exit surveys at the conclusion of each program and follow up surveys for 1-2 years after each program. The full methodology is presented in Appendix A

WORKING GROUPS

The working group (WG) approach comprised a seven-element strategy to foster substantive group accomplishments, increase individual participants’ awareness of and expertise in international work, and encourage scholarly activity. We evaluated each of these objectives

Impacts: Group accomplishments

Each group achieved substantive results in its area of inquiry. The range of scholarly products included conference panels, journal articles, a jointly-authored book, software, case studies, short courses, dissertations, grant applications, and newly funded work.

The E-consultation group convened fi ve times in cities in the US, UK, and France between 2007 and 2009. Each meeting was divided between the group’s research and a related scholarly event open to the public. This group’s main goal was to assess how the process and evaluation of a specific consultation should be tailored to legal, political, and cultural contexts. Connecting Democracy: Online Consultation and the Future of Democratic Discourse, recently published by MIT Press, presents a multi-disciplinary and international look at online consultations and draws on the individual and collective experience of the group in 18 chapters coauthored by group members. The group also developed curricula and presented papers, panels, and posters at international conferences.

Meeting in Philadelphia
Members of the North American International Working Group brainstorming during their first meeting in Philadelphia in May of 2007.
The North American group convened four times (twice in the US and once each in Mexico and Canada) during 2007-2009. In each meeting, local and national government offi cials gave presentations on key issues and new initiatives and discussed their research potential. After two meetings, NADGWG organized two topical subgroups based on participant expertise and interest. One focused on border regions and began working to understand the information sharing and interoperability issues in the border regions of Canada, Mexico, and the United States. Specifi c issues included transnational business processes and collaborative cross-border initiatives. The second subgroup investigated the roles of government policy, trust, and information and communication technologies in North American distribution networks for goods such as organic and fair trade food. The group eventually proposed and was awarded a multi-year NSF grant to pursue these questions. At the same time, NADGWG continued to develop a more comprehensive North American digital government research agenda and successfully applied for funding from both Mexico and Canada. That funding now supports comparative research on “smart cities” involving research teams in seven cities and four countries. In addition, group members described the emergence of a “North American” identity that helped to secure funding for these projects.

Between 2007 and 2009, the Hotspot Informatics group held six extended meetings in the US and India that included not only research efforts, but also classroom instruction and various forms of engagement with local and national government officials and academic institutions. The group focused on the practical challenges of watershed management in rural India where it involved university faculty and students as well as public offi cials and civil society organizations. This group also leveraged other research projects to pull together fi ndings, tools, and other resources that could be re-used by the working group in a set of software tools and a case book. Through this work, the group had extensive and replicable practical impacts on watershed management in India. It also involved the local community in watershed planning activities, and worked with local partners who developed an interpretive drama to inform local citizens of these activities and their implications for quality of life.

Impacts: Personal competencies, international awareness, and career development

The survey results indicate that the WG experiment had a strong positive impact on participants’ self-reported personal and professional international competencies. As shown in Table 2, respondents rated all but two items higher than 4.0 on a 5-point scale, including being introduced to new ideas, improved understanding of practical digital government challenges, contribution to professional goals, and fostering a sense of community and long lasting professional relationships. Likewise the experience increased interest and opportunities in international, collaborative, and multidisciplinary DG research. Even the lowest scoring items, increasing cultural awareness in teaching and influencing supervision of students, were rated above the mid-point of the scale.

These positive views hold up across different groups although the effects are stronger for some types. For example, on 6 of 32 measures, junior (untenured) scholars (n=11) perceived the experience to be more highly positive than senior scholars (n=24). These differences are statistically significant (t-test, p<05). The differences include increased interest in DG research generally, in international DG research, in interdisciplinary work, in transnational and comparative studies, in the value of practitioner involvement, and in the value added by the different meeting locations. All of these factors are less likely to be accessible to early career scholars which may explain the higher value they placed on these elements of the WG experience.

table 2
Table 2.
At the same time, it appears that the more experienced DG researchers, those with six or more years of general DG experience by the end of experiment (n=47), felt better able to take full advantage of the working group opportunities for enhancing their research range, skills, and networks. For this group, three differences were statistically signifi cant (t-test, p<05) The value of the overall experience and the likelihood of having established long-lasting professional relationships were rated higher, and the value of online collaboration software was rated lower.

Scores were more consistent across levels of international research experience and US vs. non-US participants. The mean scores of non-US participants tended to be higher overall, but none of these differences was statistically significant.

table 3
Table 3.
Impacts: Scholarly products and engagement

The working group experiment provided complete freedom to the groups to choose goals and work products. Tables 3 and 4 present research productivity and scholarly engagement outputs. These were measured by participant reports of products in-progress or completed as a direct result of group participation. We focused these measures on traditional products such as journal articles, scholarly visits, or jointly developed conference panels, software, or curricula. As shown in both tables, despite the fact that no funding was provided for research activities or salaries, the total number of outputs reported is more than three times the number of survey respondents, indicating substantial productivity and a high rate of collaborative activity over the three-year span of the experiment.

Of particular note is the number of respondents reporting serving on dissertation committees (17), writing joint research proposals (18), or engaging in long scholarly visits (33), or joint curricula development (16). These kinds of activities indicate more intensive and longer relationships than would be expected from less involved activities such as jointly authored journal articles and conference proposals.

table 4
Table 4.
Effective Practices

Given the positive results of the working group experience described above, we next explored the qualitative data from observations, interviews, and the reflection meeting to identify the practices that contributed to successful group formation, operation, and results. While each group had its own character and dynamics, several practices appeared to be instrumental across all three groups.

Optimizing face-to-face meetings

Although the working groups organized their face-to-face meetings differently, members consistently said these events made the biggest impact on the overall experience. The specifi cs of the meetings varied but all included a “business meeting” focused on their substantive goals, plus informal social events, and immersion in the local academic and practice settings. Immersion activities included visiting government organizations, bringing local practitioners into the meetings, and moving the meetings among different countries with local hosts. Time together was clearly a precious commodity and co-chairs, local hosts, and group members all worked to make these events as well-planned, managed, and productive as possible.

Openness and flexibility regarding goals and strategies

The working group strategy enabled participants to work in non-traditional ways, developing new relationships, working in different cultural settings, and engaging across different disciplines. These opportunities encouraged fl exibility and creativity that members found stimulating. However, because the working groups were explicitly not tied to predefi ned research projects, they presented ambiguity that could be difficult to manage and frustrating for individual members. These dynamics created challenges for both group leaders and individual participants. Each person and the group as a whole had to find a comfort zone where risks and opportunities could be balanced. Co-chairs were instrumental in figuring out how to move the groups toward mutually benefi cial and realistic working agendas and goals. All three groups made conscious choices to veer away from or redefine their initial goals. For all groups, it became apparent by the second meeting that adjustments in their original plans and activities were necessary for the group to succeed. After face-to-face discussion about their underlying ideas and their practical limitations, all three groups deliberately adjusted their agendas, setting somewhat different (usually more modest) goals or approaching their goals with revised strategies.

Negotiating institutional constraints and competing expectations

While working group activities took less than 5 or 10 percent of most members’ time, these activities still needed to contribute to their home obligations. Groups need to “fi nd the sweet spot” where members’ participation in the group also supported expectations at their home institutions and did not conflict too much with academic calendars or core conferences. In some cases, members’ home institutions do not support activities that fall outside of defi ned areas of work, especially if they cut into teaching responsibilities. Some members connected their unfunded working group activities to existing projects as a way to generate synergy, leverage resources and show the signifi cance of the work. Having one’s institution “host” a face-to-face meeting was also helpful because hosting gave recognition to their university for its connection to an international partnership affiliated with NSF. Practitioners found it diffi cult to participate because they could not self-fund the travel to meetings. Groups dealt with this common problem by hosting the meetings in different locations so that nearby practitioners could attend without traveling.

Identify and support multiple value propositions

Senior and junior faculty, students, and practitioners have different performance expectations based on career stages, disciplinary emphasis, and professional trajectories. In addition, their different home settings place different value on international work and on the balance between research, teaching, and service. Every group offered the potential for traditional academic outputs, but because of these differences, they also needed to support a collection of activities that offered some particular benefi t to every member. For example, graduate students who independently reached out to join working groups had to sell the idea and demonstrate its value to their advisors by emphasizing how participation enhanced their research capabilities with exposure to new methods, theories, and research settings, as well as a wider network for future work. For some established researchers, new settings offered access to different data and a wider scope of problems to study. Some took advantage of the diversity in their groups to develop new multi-disciplinary grant proposals. Student members took advantage of teaching opportunities and junior faculty joined proposal writing teams with senior colleagues and mentors.

Multiple forms of leadership

Working group co-chairs and senior faculty provided important cohesion in working groups. Co-chairs took on the bulk of organization and management needs, while senior faculty provided insight on topics, brought experience from past collaborative projects, and mentored junior faculty and graduate students. Widespread trust in co-chairs’ ability to make this work was a major factor in group sustainability and cohesiveness. Likewise, junior faculty were often described as the “work horses” of the groups, executing literature reviews and grant proposals, and taking the lead in keeping work going between meetings.

Network building and network access

The working groups actively connected senior, junior, student and practitioner networks providing benefi ts to all members and creating the conditions for sustained future relationships. The multi-dimensional nature of the groups provided entry points to explore or enlarge multi-disciplinary, cross-generational, and international research networks. Senior faculty participation was an important mechanism for connecting junior faculty and graduate students to leaders in relevant fi elds. Senior faculty also have more credibility and well-established relationships with practitioners and funding institutions based on their reputations. They therefore eased access to these resources for other members.

Table 5
Table 5.
Impacts: Linking the WG strategy to the results and prospects for replication

The results as described were assessed in several ways. First, participants rated the value of the stuctural elements of the strategy. As shown in Table 5, respondents rated the value of all elements higher than 4.0 on a 5.0 scale, including the value of face-to-face meetings in different locations; mixing senior, junior, and student scholars and practitioners; and the contribution of resources aside from the limited travel funds provided by NSF.

Second, we conducted linear regression analyses to explore the degree to which the WG design accounted for the differences in overall experience (Model 1), perceived individual career effects (Model 2), and perceived growth in participants’ international awareness (Model 3). The independent variables in the model are the set of requirements in the ‘Call for Proposals’. The analysis tests the degree to which these variables account for variations in working group experience, individual career effects, and growth in international awareness (Table 6).

Model 1 uses ‘Overall experience’ as the dependent variable. Models 2 and 3 use composite scores as the dependent variables. Model 2 uses ‘Career effects’ – representing 17 variables ranging from increased interest in the fi eld, increased opportunities to do research, new engagement with practitioners and the emergence of long-lasting professional relationships. Model 3 uses ‘International awareness’ – representing 6 variables ranging from increased understanding of the field’s practical challenges, awareness of cultural factors in teaching and research, and fostering a sense of international community.

These analyses confirm that the package of WG design elements was highly effective. The regression models account for roughly 70 percent of the variation in the three dependent variables (overall experience, career effects, and international awareness). Moreover, one variable, face-to-face meetings, was the most important factor. The value of these meetings was positive and statistically signifi cant in all three models, indicating its substantial contribution to all three desired outcomes: positive working group experiences, positive career effects, and growth in international awareness.

In Model 1, the value of online collaboration tools is signifi cant, but negative indicating that the use of such tools decrease participant’s positive experiences. Model 2 indicates the positive contribution of the NSF name and practitioner engagement to participants’ overall career effects. We interpret these to mean that association with NSF enhances individual scholarly reputations and engagement with practitioners offers opportunities for a broader context. In Model 3, in addition to the face-to-face meetings, requiring travel support from other organizations positively affected participants’ international awareness. International travel was necessary to engage in the groups, but members from institutions outside the US had to fi nd their own travel funding. While this was sometimes difficult, one international participant explained how this requirement brought them to the table as equals, despite the fact that each one had access to different kinds and amounts of resources. These findings also point out the intangible but signifi cant value of NSF’s reputation for supporting high quality research, and the value of access to research venues, such as the leadership levels of government, that are often closed to individual scholars.

These results are consistent with prior research. Cummings and Kiesler (2005) found that the work arrangements that make collaborations possible require a deliberate strategy for coordination, where face-to-face supervision and engagement were an important factor in sustainability. In addition, while electronic tools for communicating across distance have been hailed as a breakthrough in distributed work, research indicates mixed effects. Our fi ndings show that online collaboration tools, even in combination with personal interaction, contributed little to relationship building and group productivity. Every group tried some form of collaboration platform and all gave them up and reported that email was the only electronic tool that was consistently useful.

table 6
Table 6.

NOTES FOR TABLE 6: Model 1: Dependent Variable: Rating of Overall Working Group Experience (Very negative to Very positive); Model 2: Dependent Variable: Rating of Perceived Career Effects (Very negative to Very positive); Model 3: Dependent Variable: Rating of Perceived Impact on International Awareness (Very negative to Very positive). T-statistics are in parentheses under coefficient values. Significance of coeffi cients are indicated by * (10% level), **( 5% level), and *** (1% level).

iGOV INSTITUTE

We evaluated the iGov institute strategy through successive surveys of student participants as well as observations of the annual program, correspondence with participants, and document reviews.

Effectiveness of the Institute design

A few weeks after each iGov program, all students were surveyed about the design, structure, and value of the experience. The response rates were consistently 100 percent. In these exit surveys, all four cohorts rated the overall quality of the institute as excellent (an overall mean score of 4.47 on a 5- point scale, n=72). Table 9 shows that all programmatic elements received high positive ratings, including providing opportunities for informal meetings (4.82), participating in small groups (4.72), making good use of the physical location as an integral part of the program (4.65), structuring time to discuss student research (4.55), having junior faculty as mentors (4.55), engaging in practitioner sessions and site visits (4.35), encouraging student and faculty interaction (4.30), and lectures from faculty (4.22). The year to year ratings show that the program effectiveness continuously improved with a generally upward trend in student opinions on all items (Table 7).

table 7
Table 7
Students also appreciated that the program was relatively short making it fully accessible despite competing academic, employment, and family obligations. Fifty-three percent agreed that one week was ideal, while another 21 percent recommended 10 days as an ideal length.

Students expressed how all of these elements came together in open-ended comments stating, “Being injected into a new setting was very valuable to me. While I do interact with practitioners and scholars, it was a new and exciting experience to interact with them in a foreign setting.” Another noted that the institute “provides a unique opportunity to interact with these diffi cult to reach government officials and bridge the practitioner-researcher divide.” They also expressed appreciation for the one on one time with faculty that provided opportunities for personal reflection and mentoring.

The value of participating in small groups was consistently rated well above 4.0 on a 5-point scale The small groups met over a three day period of the seven-day program. The experience was designed to introduce students to the challenges and opportunities of a cross-disciplinary, crosscultural research team.

In the first year, students were pre-assigned to small groups designed to maximize diversity and were asked to develop their topics, approach, and presentation. Even with senior faculty mentors, most found this quite diffi cult. Starting in the second year, topics were suggested and recorded throughout the fi rst few days based on student reactions to lectures and site visits. Then students winnowed the list down to three topics in a plenary discussion and selfselected themselves into one of the groups. Each group was assigned a junior faculty mentor. Judging from the group results, student comments, and the survey results, this approach was more effective and more satisfying. The assignment given to the groups was very loosely structured to give students maximum freedom to defi ne their topic, approach, and presentation style. All students found this challenging, but most agreed that it helped foster awareness of cultural factors in research, highlighted differences in language and terms used, and enhanced their ability to work across cultural and disciplinary lines. They also gained an appreciation for the diffi culty of not only designing but executing international research. By the third and fourth years, the small groups were the highest rated element of the program. One student described the frustration and the benefit stating, "[Personally, I felt] the group project, though diffi cult and seemingly impossible at times, was also helpful. You forced six PhD students from various disciplines and countries to decide on a topic, do some research, and prepare a presentation. [In addition] to it being a bonding ritual, it was also a crash course on international and cross-disciplinary collaboration."

Immediate personal and professional effects

Personal competencies, international awareness, and career effects

The annual exit surveys confi rmed that the iGov Institute infl uenced young academics in the short term (see Table 8). Across all four cohorts, respondents strongly agreed that the institute’s design and content fostered a sense of intellectual community (4.49), improved their understanding of practical international DG challenges (4.38), and introduced them to useful ideas outside their main fields (4.28).

These strong positive perceptions applied to all types of students but some differences were evident. For example, students who were either U.S. citizens or enrolled in U.S.-based institutions perceived greater improvement in their understanding of practical international DG challenges and were more strongly affected by the introduction to ideas outside of their main fi elds of study. These differences were statistically signifi cant (t-test, p<.05). In addition, students enrolled in U.S.-based institutions said iGov would infl uence their dissertation topics more than students being educated outside of the U.S, although in general the institute had less effect on dissertations than on more fundamental scholarly competencies and international awareness. As most students were well into their doctoral programs when they attended, we expect that their dissertations were already started and therefore not open to substantial change in topic or approach.

Lastly, students who were citizens of developing countries perceived the experience to be even more highly positive than students from developed countries. Students from developing countries recorded higher mean scores for fostering a sense of intellectual community, contributing to research or professional goals, and prompting future consideration of comparative or transnational DG research. These differences are statistically signifi cant (t-test, p<.05). We suspect that all of these opportunities are less available to students in developing countries which may explain the higher value they placed on them.

Long-term personal and professional effects

Table 8
Table 8
The one- and two-year follow up surveys showed that iGov’s reported positive impact on students’ career development, interest in international research, and international awareness actually increased over time. For example, students reported that the iGov Institute continued to increase their awareness and ability to conduct international investigations and to include multi-cultural aspects in their research and teaching (See Table 9, next page). In addition, the institute’s impact on participants’ research or professional goals, inclination to do future comparative or transnational DG research or the ability to work across cultures continued to be positive and generally showed a continuing upward trend over time.

The follow-up surveys also showed that participants remained very interested in doing international research, but perceptions of the opportunities available to them were slightly more modest. The perception of being able to foster long-lasting professional relationships was highest right after the institute and was more modest, but sustained at a positive level, over time. We surmise that dissertation pressures for most students left less time to cultivate these relationships. One student noted in a follow-up survey, "Once I’ve gathered my dissertation data and I am in the writing phase, I will be in a better position to reach out to fellow iGov participants to inquire about joint publications or conference papers. I feel as though I don’t have enough data now to pursue further collaboration with them, but will soon." In addition, sustained international awareness effects also appear to have been realized. At exit, participants reported iGov increased their awareness of cultural factors in their teaching, research, and everyday life (all scoring above the mid-point, Table 11, p. 26). One and two years later, perceptions of iGov’s contribution to increased cultural awareness in teaching, research, and everyday life continued to increase.

Table 9
Table 9
Scholarly products and engagement

Tables 10 and 11 (next page) present the research productivity and scholarly engagement effects of iGov as measured by reports of specifi c products that participants’ view as a direct result of attending. Table 12 shows the collaborative activity in the years following iGov. Short scholarly visits, joint research proposals, and joint conference panels were the most common collaborative activities over the four years. Table 13 shows the total number of outputs reported is more than double the number of survey respondents, indicating infl uence on research productivity, even at this early career stage.

Overall impact and replicability

The program overall appears to stimulate participants’ individual creativity, scholarly productivity, and professional networks, while broadening their appreciation for work that investigates internationally important topics and involves not only multidisciplinary but multicultural teams. All of these effects will enhance the quality, versatility, and creativity of future digital government researchers.

Table 10
Table 10
While the survey results suggest consistently positive effects, the results do have limitations, mainly related to self-reported perceptions. However, these findings are at least anecdotally supported from other sources. For example, we know that several doctoral advisors recommended additional students for admission to the Institute in successive years based on their satisfaction with the results experienced by students who attended earlier. Other supervisors told us informally that their students who attended especially benefited from the research-practice connection, which is not often emphasized in traditional doctoral programs. Alumni were also eager to return as junior faculty mentors and continued to respond to successive surveys in high numbers. Finally, based on its reputation within the DG research community, a number of our international colleagues volunteered to join the faculty or host the program in future years.

Table 11
Table 11

Manchester, 2008
iGov Institute, Manchester, UK, 2008.