TeleCommons Development Group (TDG)

512 Woolwich Street

Guelph, Ontario,

Canada

N1H 3X7

Abstract: This evaluation framework was tested using three NGO case studies in Uganda, East Africa as part of the author’s M.Sc. thesis research. A report on the results of that research is expected to be available in early 1999. Until recently, Northern development agencies had virtually ignored the development of information technology (IT) in rural areas in the South (Zijp 1994). Recent literature evaluating Internet use in developing countries fails to recognize how such technology impacts those without Internet access: the "unconnected" rural stakeholders. Moreover, the literature ignores the validity of "the unconnected" as being considered stakeholders in IT projects and processes at all. Given this scenario, stakeholders without Internet or email connectivity have been as excluded from IT research as they have from IT expansion.

This paper proposes a new evaluation framework to examine the extent to which Internet use by non-governmental organizations (NGOs) benefits the unconnected stakeholders that they serve. It utilizes the notion of a two-way information sharing process associated with the theory of Communication for Development. The framework measures the efficiency, effectiveness, and impacts of NGOs’ ability to receive, disseminate, and share information with rural communities.

This framework for evaluating the impacts of NGOs with Internet connectivity on rural communities represents an attempt at addressing the shortcomings of previous ICT evaluations. This paper is not meant to be a definitive study of how to examine rural communities’ information sharing with NGOs with Internet connectivity, but it does propose a framework from which to build upon, improve, and make as relevant as possible an understanding of the benefits and impacts of the Internet for those who lack connectivity in rural areas of developing countries. Only through application in the field can such frameworks be tested, assessed, and improved upon. It is hoped that this framework represents a fresh approach to understanding the complex issues and networks involved in information sharing and ICT use.

Much of the literature on the use of Information and Communication Technologies (ICT) concerns the potential that such investments offer for the developing world. Yet despite the rhetoric of how ICT can bridge the gap between the information haves and the have-nots, there is little research available which demonstrates that the use of ICT are in fact able to bridge that gap. Moreover, the focus of ICT research is currently fixated on the institutional level, which is composed of Internet service providers (ISPs), policy makers and governments. While the value of such research is not challenged here, the fact remains that very little investigation has been conducted concerning the impacts of ICT on those who have the most to gain through the expansion of ICT in the developing world: the rural stakeholder communities.

The benefits of expanding ICT to major urban cities has been well-documented, and the results of such urban investment can be seen in both the expansion of ISPs and the increasing connectivity throughout the cities of the developing world. The impacts on rural stakeholders and rural communities, however, are less understood (Richardson 1996; Sirimanne 1996; Wresch 1996; Zijp 1994). Until recently, Northern development agencies had virtually ignored the development of information technology in rural areas in the South (Zijp 1994: 14). Furthermore, the economics involved in development agencies’ and corporations’ willingness to expand Internet connectivity into rural areas in both Northern and Southern hemispheres has proven to be a major stumbling block. In fact, rather than expansion being based upon the desire to meet the communication needs of rural communities, rural ICT expansion is primarily based on the degree to which it will be a profitable and worthwhile venture for the communication companies involved (Richardson 1996). Such sentiment is supported by Michel Menou in his book, Measuring the Impact of Information on Development:

The exclusion of rural communities and stakeholders from ICT expansion occurs in spite of the benefits which ICT is intended to bring to the isolated areas of the world. In "What About the Developing Countries?", Shamrika Sirimane writes that ICT is a tool which can be used to narrow the information and communication gap between rural communities and urban centres through, "giving rural people access to valuable information; and by transmitting indigenous information and locally produced knowledge" (Sirimane 1996: 6). Richardson (1996) writes that the Internet,

The exclusion of rural stakeholders from ICT expansion mirrors the dramatic lack of research being undertaken to measure the benefits that such investments are able to provide for rural communities. A notable exception to this situation is Kingo Mchombu’s 1996 report, "Impact of Information on Rural Development: Background, Methodology, and Progress". In it, Mchombu outlines the mid-project results of INFORD 1 and 2, a two-phased project funded by the International Development Research Centre (IDRC) which monitored six rural communities throughout Southern Africa to investigate "the provision of information to support rural development and the impact of such information on development" (Mchombu 1996: 2). This project assessed the general and specific information needs of the individual communities, and then examined the extent to which these needs were being met through various rural development activities employed in and around those communities. The evaluation’s intention was to establish, "if the information provided has led to development gains in the community"; a follow-up report has yet to be published (Ibid.: 11).

Although the Mchombu report researches the origins and impacts of information on rural communities, it does not specifically address the use of the newest form of ICT: the Internet. The irony is that recent reports focusing specifically on the Internet offer very little in terms of addressing the impacts of such technologies on rural communities. Ricardo Gomez’s report, "Electronic Agora or Disneyland Democracy?" (1997) examines the utilization of the Internet by Colombian non-governmental organizations (NGOs) connected to the Association for Progressive Communication (APC) network, and deals solely with the perceptions those NGOs have about their use of the Internet; no references are made at all to rural stakeholders in this report (Gomez 1997).

Both the Canadian International Development Agency (CIDA) and IDRC published reports in 1997 which were intended to provide indicators and results of the use of ICT in projects throughout the developing world. However, while data is available on impacts and results of ICT on groups that have connectivity, neither of these reports addresses the issues and needs of the rural stakeholders who have ties to organizations with access to the Internet. In short, those without connectivity have been as excluded from ICT research as they have from ICT expansion.

The Canadian International Development Agency’s (CIDA) 1997 report, "ICT and Development: Testing a Framework for Evaluation, volume 1", set out to address four core characteristics related to ICT use in CIDA -sponsored projects throughout Asia and Africa: information, borderless connections, timelines, and improving costs and benefits (Young et al. 1997). The report uses a framework which provides much attention to how ICT have been adopted within its projects, but places little emphasis on what the impacts of such connectivity means for those rural stakeholders without direct connectivity to the Internet. Indeed, of 30 key questions identified in the CIDA Evaluation Framework, only two address the extent to which stakeholders were involved in the projects. Unfortunately, CIDA’s stakeholder definition appears to exclude those without Internet connectivity (Ibid.:10).

Even with the inclusion of the two stakeholder questions relating to local stakeholders’ acceptance, support, and ability to assume ownership of ICT the CIDA report offers vague, insufficient analysis of the rural impacts of ICT. This is evidenced in the presentation of a case study of a Peruvian rural telephone expansion project; the report states that the project’s immediate impact is an increase in telecommunications access for 200,000 people. Beyond this statistic, however, the report was unable to identify any of the direct and indirect impacts of this project at the community level (Ibid.: 19). That CIDA’s report is deficient in its approach to rural communities and stakeholders is further demonstrated by the fact that the evaluation research was completed without the involvement of any stakeholder representatives; CIDA desk staff and NGO partners were contacted by phone, fax, and email, with no research done in the field (Ibid.: 27).

The second of the two ICT evaluation reports to emerge in 1997, "Use of Information and Communication Technologies in IDRC Projects: Lessons Learned", was written by Michael Graham and published by IDRC. This evaluation had the objective of identifying ways in which ICT might be better used to enhance people’s lives. The report raises 8 questions to address issues related to 13 IDRC-funded projects, and, like its CIDA counterpart, discusses results only in terms of those stakeholders who have connectivity; no mention is made of rural stakeholders who are without ICT access. Of the 8 Evaluation questions posed in IDRC’s report, only one concerns ICT in relation to its use and adoption by all stakeholders (Graham 1997: 8). Moreover, the IDRC report, like the CIDA report, was written without any field research being conducted and without the involvement of rural stakeholders (Ibid.: 11).

That both CIDA and IDRC, two of Canada’s strongest players in terms of funding International Development and ICT research, should neglect the presence of rural stakeholders without connectivity in their evaluation reports is both perplexing and worrisome. What is the cost of ignoring the benefits and impacts reaped by rural stakeholders that are affiliated with NGOs with connectivity? Does this indicate that international development agencies such as IDRC and CIDA believe that an organization’s potential with Internet connectivity ceases to have impacts beyond the organization itself? And if researchers overlook the participatory links being made between rural communities and NGOs with connectivity, how will the powers that control and influence ICT infrastructure and investment be made aware of whether or not the technology is indeed assisting them? As Richardson states in a recent paper, "All stakeholders must encourage development agencies to engage in collaborative efforts that lead to activities and projects focused on rural and agricultural development" (Richardson 1997a: 3).

There is a need to refocus ICT research so that the communication processes occurring with the use of the Internet are examined, rather than the infrastructure components themselves. This is supported by Menou (1993), who states, "The whole communication cycle, from generation to assimilation, has not been studied in specific institutions or communities of developing communities" (Menou 1993: 68). Don Richardson, in his paper, "The Internet and Rural Development: Recommendations for Strategy and Activity", makes the case for increasing the focus on those who live on the fringe of ICT expansion, and against focusing all research on those with direct access to the connectivity. "Research on impacts", he writes,

Through learning how communities without Internet connectivity are able to benefit from organizations that have connectivity, and through learning how the two entities can work together to ensure that information is received, shared, and put to good use, those in development can move towards guaranteeing that stakeholders’ needs are being met, and that adequate support is provided to those who need it most. If the benefits of the Internet are not made available to the rural peoples who can benefit from it the most, "then we are failing to fully leverage the large infrastructure investments involved, and we are failing to assist people in making appropriate decisions based on such valuable information" (Richardson 1997a: 5).

The concept of participatory development communication is central to examining the use of the Internet as it relates to rural community stakeholders. Its basic tenet is that, "the communication process is more important than the production of media products" (Richardson 1997b: 1). Participatory communication for development is a process that emphasizes a two-way communication process based on sharing ideas and information in order to improve the quality of life of the stakeholder communities (Bessette 1996). Unlike traditional communication methods of disseminating and transmitting information, development for communication,

The Internet is the latest in a long line of communication tools that enable a community to receive information and assistance from outside organizations. Unlike technologies such as radio, newspaper, and video, however, the Internet, "is the first media tool that allows every user to be a sender, receiver, narrowcaster and broadcaster" (Richardson 1996:9). The tremendous opportunities that such technology can offer to rural stakeholder communities must not be missed. The challenge, however, is ensuring that the Internet is utilized for two-way communication processes such as communicating local knowledge and perceptions. Existing pilot projects in Chile, Mexico, and Southern Africa exemplify rural stakeholders’ ability to successfully adapt to the opportunities offered by the Internet (Richardson 1997a: 2).

Roling (in Ramirez 1997) identifies three roles of communication which communication for development could play in natural resource management: making things visible through the creation of new perspectives, fostering policy acceptance, and facilitating platform processes involving all stakeholders (Ramirez 1997: 5). Ramirez sees this third role as, "the less explored and more promising dimension of communication", where the focus is on, "the orchestration of platforms of negotiation among multiple actors" (Ibid.). In working with these platforms of negotiation, "a true demand-capacity from the grassroots is able to match and select relevant information", thus ensuring that the rural communities’ needs can be better served (Ibid.:10).

As illustrated in the existing literature, current evaluations tend to ignore those rural stakeholders who are without connectivity. The need exists for an evaluation approach which will specifically examine the impacts of ICT on development at a grassroots level. Michel Menou writes of the importance of understanding the grassroots’ impacts from the Internet when he states that "lower level data are needed to interpret higher trends" (Menou 1993: 37). He goes further in suggesting the importance of focusing on this segment of the population when discussing the issue of equality of access to information. Such access,

Richardson, too, endorses a refocusing of research and evaluation energies related to the local levels of development when he states that,

The time to focus on the processes that accompany technological tools such as the Internet is now. Without understanding how ICT such as the Internet are impacting those at the grassroots level, we risk remaining unaware of the effectiveness of technological investments which occur at the institutional level. Given the need for such an evaluation, this author proposes a new framework to look exclusively at the extent to which Internet use by local NGOs benefits the unconnected rural stakeholder communities that they serve. This evaluation will utilize a traditional process of measuring the effectiveness, efficiency, and impacts of the NGOs’ use of the Internet in relation to the rural communities. The process can be considered traditional in the sense that without the "essential" interim measurements of effectiveness and efficiency, the measurement of impact might not be possible (Mchombu 1996: 4).

In measuring the impact of ICT on rural communities, this evaluation will focus on the process, rather than the end result, of information exchange between rural stakeholder communities and local non-governmental organizations. The research will involve case studies of between two to four local, non-governmental organizations (NGOs) in Uganda, East Africa that are working on rural issues with rural communities. The methodology will include a combination of participant observation and key informant interviews with key actors in the stakeholder communities, NGO extension workers, and NGO staff to construct a realistic picture of how rural communities are able to benefit from Internet use. Using a model similar to that proposed by Tandon (in Mchombu 1996), this research hopes to follow a process of inquiry and intervention, in which both the researcher and the subjects not only learn from one another, but also learn from the very situation that they are a part of and engaged in studying (Mchombu 1996: 5). The research will collect both quantitative and qualitative data, though the very nature of evaluating the process of information sharing will place more of an emphasis on the latter.

This section examines the extent to which the local NGO’s use of the Internet allows it to provide the greatest amount of assistance, services, and information to stakeholder communities using the least amount of input factors.

1.1 Quality of Internet Service: Background

This section examines the extent to which the organization finds its Internet system to be reliable and easily accessible. The importance of this first section of the evaluation is evidenced by Michel Menou (1993), who states that, "Lack of access to networks and databases is itself a measure of inequality ... because it prevents access to information and the chance to benefit from it" (Menou, 1993, 46). To provide further information on the degree to which the Internet is accessible to the NGO, frequency of Internet use within the organization is also measured.

Indicators for the reliability of the Internet service include assessing whether the NGO’s Internet service offers users any advantages to the staff over other means of communications such as post, fax, and telex. Indicators for the advantages to using the Internet include the degree to which less transmission time is required for messages sent via Internet than via post, fax, courier, etc; reduced transmission costs; and faster turn-around time for replies. Frequency of Internet use is quantified in terms of the number of times the individual users access the system on-line.

Open-ended interviews and participant observation is used to gather the data required for this section.

1.2 Capacity of NGO to Acquire, Disseminate, and Respond to Internet Information

This component examines the number of individuals involved in disseminating information from the Internet to the community. This component is directly linked to an organization’s ability to manage and absorb information received from the Internet (Menou, 1993; Zijp, 1994). If the organization lacks the capacity to sort, identify and utilize the information that is available to it, then the organization’s use of that system will be severely undermined.

The indicator for the efficiency of the dissemination process is the number of individuals involved in this process measured against the total number of individuals working within the organization.

Interviews and observation can be used to understand the networks involved in the dissemination process. Techniques used for Rural Appraisal of Agricultural Knowledge Systems (RAAKS), such as linkage matrices to determine which stakeholders have contact with whom, and linkage webs to represent levels of resource and decision control were used to conceptualize the links present in the dissemination process (Engel and Salomon, 1986; Rogers and Kincaid, 1981). The tremendous value of these techniques is emphasized in a 1995 FAO report on communication networks among farmers in the Philippines which stated that, "Mapping linkages in a knowledge system uncovers information exchange mechanisms" (FAO, 1995, 4).

1.3 Costs of the Internet

This component addresses costs of the Internet for the organization. Indicators for this component are the account costs associated with Internet connectivity. The annual cost of the NGO’s Internet service will also be measured as a percentage of its total communications budget, and as a percentage of its total operating budget. Through examining the values of this data, it is possible to determine the degree to which Internet service is affordable for the respective organizations.

Open-ended interviews with key NGO staff, and inspection of annual budgets were used to assist in the examination of the economic costs of Internet connectivity.

This section will examine the extent to which the Internet enables the NGO to locate and disseminate information that can be used in its programs with rural community stakeholders.

2.1 Appropriateness of Information

The importance of the appropriateness of information is emphasized by Zijp (1994), who states that, "It is essential to ensure that the information is of good quality, reliable, timely, and presented in a useful way. To do so, information system designers must clearly identify user’s goals, the information that users need to achieve those goals, and the process by which that information will be gathered and entered into the system" (Zijp, 1994, 20). This component examines whether Internet-acquired information is shared among NGO staff members, and whether the Internet-acquired information which is provided for their community stakeholders is appropriate to their needs.

The indicator for determining whether Internet-acquired information has been shared among the staff members is the degree to which staff members identify Internet-acquired information having been shared with other staff in the past. Indicators for determining the appropriateness of Internet-acquired information are: 1) the methods used by the NGO staff to identify Internet-acquired information which is relevant for their stakeholder communities; and 2), the degree to which the stakeholders, themselves, indicate that the information which has been provided to them by the NGO has met their needs.

Open-ended interviews with both NGO staff and key informants within the community assisted in the identification of communication patterns and linkages existing among the various actors in the information sharing process. Communication networks were identified through the use of linkage diagrams and webs (refer to Section 1.3 Methodology).

2.2 Dissemination of Information to Community Stakeholders

Open-ended interviews with both NGO staff and community stakeholder representatives will be used to illuminate the communication environment existing between the two entities. The identification of communication networks as outlined in sections 1.3 and 2.1 will serve to facilitate this examination.

2.3 Gender Equality

Information Technologies (IT) can open opportunities for women and men alike by increasing their access to information, providing venues for the expression and sharing of knowledge, as well as providing training in a variety of areas (Rostagno, 1997; Zijp, 1996). Gender equality, as defined by the Government of Canada in its 1995 Federal Plan for Gender Equality, "means that women and men have equal conditions for realizing their full human rights and potential and to contribute to national, political, economic, social and cultural development and benefit equally from the results" (IDRC, 1998).

This evaluation component will examine the issues of gender equality with respect to staff members' use of the Internet services within the NGO. Indicators for these factors include: 1) the numbers of male and female staff who utilize the Internet in relation to total numbers of male and female staff each organization; and 2), the numbers of male and female staff who have received Internet training within the organization.

Open-ended interviews and participant observation can be used to identify the degree to which gender equality exists within the organization.

This section will assess the success with which the NGO’s objectives for providing assistance, service, and information to rural, unconnected stakeholder communities are being met as a result of its own Internet connectivity.

3.1 and 3.2 Direct and Indirect Impacts

Using Menou's (1993) identification of the two main categories of benefits resulting from access to information, this component examines both the Direct and Indirect impacts which result from the use of the Internet in the information sharing process.

Indicators for the Direct impacts of the use of the Internet are: 1) the degree to which NGO staff members believe that their organization is meeting its goals and objectives for establishing connectivity; and 2), the degree to which staff members believe that the use of the Internet assists them in achieving the operational goals of the organization as a whole.

The indicator for measuring indirect impacts is the degree to which NGO staff members believe that the use of the Internet has effected the relationship between the NGO and its community stakeholders.

Interviews with NGO staff members.

3.3 Multiplier Effect

For the purposes of this research, Multiplier Effect is defined as the extent to which other organizations affiliated with the NGO have adopted Internet technology into their own operations, either directly through acquiring their own accounts, or indirectly through utilizing the account of the case study NGO. This concept is closely related to Narayan's (1993) definition of replicability, which is defined as, "the capacity to duplicate the processes and benefits of a set of development activities in new locations after their effectiveness has been demonstrated in limited geographic areas" (Narayan, 1993, 95).

The indicator for the multiplier effect is the number of groups and organizations identified by the NGO staff as having adopted Internet technology after having had use or exposure to their own Internet service.

Interviews with NGO staff members.

3.4 Sustainability of the Communication Process

In discussing the issue of sustainability as it relates to communication for development, Richardson (1997b) states that the ability of an organization to financially sustain its Internet technologies should not be the sole consideration with respect to sustainability. Considering the role that the Internet can play in the exchange of information between different groups and organizations of people, those with connectivity as well as those without, "it is the end result of catalyzing empowerment and sustaining people’s participation in their own development that ought to be the focus of our attention on ‘sustainability’" (Richardson, 1997b, 3). Thus, while the financial sustainability of the NGOs' Internet service is examined through the focus on costs in Section 1.3, the focus of this section will be on the sustainability of the communication process between the Internet-equipped NGO and the community that does not have Internet connectivity.

The indicator for the sustainability of the information sharing process is the degree to which the community representatives believe that they have opportunities to give feedback to and receive feedback from the NGO. The opportunity for rural community feedback is an integral component in the communication for development process (Bessette, 1996; Ramirez, 1997; Richardson, 1997b; Zijp, 1994). As Zijp (1994) states, feedback opportunities are important, "because rural people are not only recipients of information, but also creators of knowledge based on their own and other’s experience" (Zijp, 1994, 16).

Interviews with key actors in the stakeholder communities and NGO staff.

This report proposes a framework to evaluate the impacts of the Internet on rural communities without connectivity. The framework was tested by the author in the field using a sample of three NGOs working in Uganda, East Africa in 1998. It is expected that the results of that research will be made available in early 1999.

This proposal is intended to be a stepping stone towards achieving a better understanding of what the existing research on ICT use in developing countries has been unable to tell us thus far. Only through the continued field testing of evaluation methodologies such as this -- with real organizations and real, rural unconnected communities -- can the lessons learned be incorporated into more improved frameworks which yield relevant results. In discovering and understanding what the impacts of NGOs with connectivity are on rural communities that lack such technology, researchers, Internet technicians and development practitioners alike can work together to improve the networks and potential that Internet technology can offer to those who, for the time being, remain ‘the unconnected’.