Objectives: This article analyses some impact evaluation studies conducted in SSA as to why contradictions and exaggerations are manifested in some reports and what would be the future prospects of impact evaluation of agricultural extension programmes in the region.

Methods: Impact evaluation reports and results of agricultural extension programmes from 10 SSA countries were reviewed and analysed based on impact evaluation principles and theories.

Results: The results show that most of the evaluations reported positive impacts. There are also conflicting reports on extension performance. The fact that the overwhelming majority of impact evaluation reports claim positive extension impacts is not in line with the reports on agricultural productivity growth in the region. There are various reasons for overestimated impacts and contradictory results, which include use of poor impact evaluation methodologies, lack of reliable data and insufficient capacity to conduct rigorous impact evaluations.

Conclusion: Due to these challenges and the shift in agricultural research and extension approaches, it is recommended that rather than investing effort in trying to prove impact, greater attention should be given to improving impact as well as using other innovative monitoring and evaluation (M&E) and learning tools that consider the dynamic nature of agricultural development.

Présentation: En Afrique subsaharienne (ASS), l’évaluation de l’impact a été utilisée afin de déterminer si les interventions relatives au développement agricole avaient produit les résultats escomptés ou afin de déterminer les liens de cause à effet entre les interventions et les résultats obtenus. On observe cependant un certain scepticisme quant à la validité et à la fiabilité des résultats des rapports sur l’évaluation de l’impact en raison de leur caractère contradictoire et exagéré.

Objectifs: Cet article analyse des études d’évaluation de l’impact en ASS afin de déterminer pourquoi les contradictions et les exagérations sont manifestes dans certains rapports, et quelles seraient les perspectives futures des programmes d’évaluation de l’impact du développement agricole dans la région.

Méthodes: Les rapports et résultats en matière d’évaluation de l’impact des programmes d’extension agricole tirés de dix pays d’ASS ont été examinés et analysés à partir de principes et théories de l’évaluation de l’impact.

Résultats: La plupart des évaluations ont révélé des impacts positifs. On note aussi des rapports contradictoires sur la performance du développement. Le fait que la grande majorité des rapports d’évaluation de l’impact fasse état d’impacts positifs sur le développement n’est pas conforme aux rapports sur la croissance de la productivité agricole dans la région. Il existe plusieurs raisons à la surestimation des impacts et aux résultats contradictoires, et notamment le recours à des méthodologies d’évaluation de l’impact de mauvaise qualité, l’absence de données fiables et une capacité insuffisante pour procéder à des évaluations rigoureuses de l’impact.

Conclusion: En raison de ces défis et du changement observé dans les approches à la recherche et au développement agricole, il est recommandé qu’une plus grande attention soit accordée à l’amélioration de l’impact et à l’utilisation d’autres outils de suivi et d’évaluation (S&E) et d’apprentissage innovants tenant compte de la nature dynamique du développement agricole, plutôt que d’essayer de prouver l’impact.

Agricultural extension theories have shown changes over time, based on changes in agricultural development paradigms. As a result, a variety of extension approaches have been promoted in the past several decades. Early models that focused on transfer of technology characterised by ‘top-down’, ‘linear’, ‘rigid’ approaches were criticised for their reductionist perspectives and the passive role of farmers. These approaches also failed to recognise the diversity of the socio-economic and institutional environments and other factors affecting farmers (Chambers & Ghildyal 1984; Birner et al. 2006). Since the 1970s, however, there was a change of extension models due to change in agricultural development discourses over the years. Hence, extension models such as training and visit (T&V), participatory approaches, farmer field schools (FFSs), ICT-based on-line advisory services to farmers and promotion of model farms (Waddington et al. 2010) have been used in different countries.

Impact evaluations were conducted to test the success and outcomes of various extension interventions and approaches in sub-Saharan African countries (SSA). The difficulty of tracing the causal relationship between extension input and its impact is appreciated by various authors (Anderson 2007; Anandajayasekeram et al. 2008; Davis 2008). This article, based on analysis of a number of impact evaluation studies conducted in SSA, attempts to identify challenges and future prospects associated with the conduct of impact evaluations of agricultural extension programmes in the region.

Yields per ha have been stagnant in the SSA region, particularly for cereals, in contrast to substantial yield increases in other regions of the world from 1960 to 2005 (Staatz & Dembele 2007). This means that even if there is yield production growth, it is mainly due to area expansion, awhich shows limited technological change in the agriculture sector.

The impact evaluations in SSA have asserted that the effectiveness of various agricultural extension models is context specific (Appendix 1). For instance, Purcell and Anderson (1997) evaluated T&V extension models in five SSA countries and found that the model had a satisfactory effect in three of the countries (Kenya, Somali and Zimbabwe) whereas it had an unsatisfactory effect in Ivory Coast and Rwanda. The Uganda National Advisory Services (NAADS) was reported to have positive impact in terms of availability and quality of services but there was no significant difference in terms of yield growth (Benin et al. 2007). Similarly, Davis et al. (2010) found that FFS extension models had mixed results in the East African region. The model brought significant change in terms of productivity and income in Kenya and Tanzania whilst it had non-significant impact on the same variables in Uganda.

Apart from the result of the impact evaluations, which is inconsistent with the agricultural growth trend in the region, some of the impact evaluation studies are also contradictory. Bindlish and Evenson (1997) in their study of the T&V programme reported a high and significant rate of returns to extension in Kenya. However, Gautam and Anderson (1999) rejected this result, reasoning that the evaluation suffered from data and specification problems. Using the same data, they showed that the finding was invalid and the rate of return was not significant. Similarly, whilst Ayele et al. (2005) and IFPRI (2008) reported a positive and significant effect of extension in Ethiopia, another study conducted by EEA/EEPRI (2006) asserted that the Ethiopian extension programme suffers from various problems and its impact on farm income, productivity, food security and poverty is not significant.

In Mozambique, Walter et al. (2004) reported that agricultural extension had a negative and statistically insignificant impact on individual and household income. Neither did agricultural extension services lead to any perception of improved economic condition in the household. They stipulated that the absence of positive and statistically significant results on household and community variables could be due to limited access to improved inputs and location-specific adapted technologies. To the contrary, ECON Analysis (2005) rejected this finding and reported that extension services in Mozambique had positive effects on rural livelihoods through promotion of new varieties, inputs such as natural pesticides and soil conservation practices.

Due to exaggerations and contradictions in similar reports, some researchers are sceptical of the results of evaluation studies conducted in sub-Saharan Africa (Davis 2008). There are different reasons for such contradictions and exaggerations in impact evaluation studies conducted on agricultural extension programmes in the region. These are related to challenges and limitations that emanate from the nature of impact evaluation, extension science, agriculture, impact evaluation methodologies and capacity that will be discussed in the next section.

Bringing agricultural growth has been a main agenda for most sub-Saharan African countries. Investment in extension services is an important tool for improving agricultural productivity and increasing farmers’ incomes (Anderson 2007). Hence, bilateral and multilateral donors have been supporting efforts of developing countries in providing extension services. Therefore, most SSA countries have been providing agricultural extension services for the last several decades even if the modalities have shown changes over the years.

Small-scale extension initiatives in Africa have been supported by the World Bank since 1970, when it started financing integrated rural development projects. Following the shift in its emphasis from integrated rural development projects to the development of national institutions in the early 1980s, the Bank began to support reorganisation of national agricultural extension systems in Africa.

The T&V model was first adopted by several dozen countries in Africa in the eighties. In practice, the T&V system was effective in disseminating Green Revolution technology, especially in the high-potential, irrigated areas, in promoting very specific packages. T&V has been criticised for high recurrent cost and financially unsustainable (Anderson, Feder & Ganguly 2006). The extension debate shifted in the late eighties and nineties to a new model: the farmer field school (FFS) (Anderson et al. 2006). Farmer field schools were introduced into sub-Saharan African in the mid-1990s. Although many positive reports exist on the benefits of the FFS approach, some studies have called into question their overall impact and financial sustainability (Davis 2008).

Most extension models have common characteristics such as addressing farmers without restrictions, using mass communication methods, extension agent-farmers’ contacts, farmers-to-farmer extension, etc. These typical characteristics affect impact evaluation in the following ways:

Lack of comparable control group: Agricultural extension programmes mostly are run nationwide and it is difficult to establish control groups (or exclude some from the benefits in order to establish a control group) from operational and ethical considerations. Furthermore, there is always spill-over of technology, knowledge and information from programme beneficiaries to non-beneficiaries, making it hard to find an ‘uncontaminated’ counterfactual. Sometimes evaluators use non-participants as a control group. This may lead evaluators to two types of erroneous conclusions.

a. There is non-significant impact: This is referred as a false negative or type II error in the specialised literature. As information, knowledge and technology diffuse from participants to non-participants, the difference in outcome between the two groups could be similar. This could lead to an underestimation of the effective impact.

b. There is significant impact: This is referred as false positive or type I error in the specialised literature. When selecting control groups for impact evaluation, they are expected to be comparable to the treatment group. This means that both groups should be similar (on other variables), except in their access to the programme intervention and, as a result, characterised by a different use of the information, knowledge and technologies of the extension programme. Extension programmes don’t usually have restrictions and are available to every farmer. Hence, if the technology is good, most farmers will adopt it, except those farmers known as ‘laggards’. These segments of the population are not expected to perform well, even in the absence of extension programmes. That means there are other individual factors that differentiate the beneficiary and non-beneficiary groups other than the extension programme, which exaggerates the net impact.

Selection bias: When the differences in outcomes between the treatment and the control group are due to pre-existing differences between the treatment and control groups (in which case randomization did not work), the evaluation conclusions are affected by a validity threat that is normally referred to as selection bias. Peoples involvement in agricultural extension programmes is based on interest and resource availability. Additionally, the ability to try new technology requires the capacity to take risks. The better-offs and the educated are those with better information access and risk-taking capacity. These people are likely to be successful even without the extension programme intervention. This will overestimate the impact of the intervention. There is also de-selection bias which is the opposite of the above discussion.

Endogeneity in extension-farmer interaction: Extension contact, which plays a central role in the success of extension programmes, is affected by farmers’ characteristics and actions. For instance, more productive farmers tend to seek more extension contacts and get more information and technologies than others. Similarly, extension agents also may seek contacts with model farmers who are good performers even in the absence of extension contacts. In this case, estimating the impact of extension programmes could be biased upward, which is attributed to the superior characteristics and actions of farmers.

Other sources of information and technology: There are other sources of information and technology, apart from the extension programme, available to farmers. Farmers use various technologies from the market and/or information they get from other media to improve their productivity. It is not easy to differentiate between the impacts of technologies disseminated by the extension programme and from other sources. This is particularly difficult in SSA, where most farmers are uneducated and cannot easily identify the technology provider. Hence, the impact of the extension intervention under evaluation may be exaggerated.

Extension to be effective needs other support services: the success of extension programmes with the ultimate goal of improving productivity depends on relevant and appropriate technology (research), timely provision of inputs, good market outlets for agricultural produces, good transportation facilities, good credit services, etc. Success/failure of these services is related to the success/failure of extension programmes. Unless their effects are not controlled, it may lead to under or overestimation of the effects of investment in extension services.

In agricultural extension programmes, the knowledge and information acquired is an output whilst adoption of technologies is an outcome and the final impact is change in productivity. Most impact evaluation studies focus on one or two of this hierarchy of objectives but not the whole process (FAO 1997). However, according to White (2009) the job of a quality impact evaluation is to trace the causal chain from inputs to outcomes and impacts. Hence, the final outcome in extension programmes should be measured in terms of productivity. Measuring extension impact by using change in knowledge and awareness (e.g. Oleru et al. 2005) would overestimate the impact of extension programmes. Sometimes, other indicators that are not related to extension are used to evaluate agricultural extension impact. For example, in Ethiopia extension effectiveness was measured using input distributed (Spielman 2008).

These challenges and factors affect the validity and reliability of agricultural extension impact evaluation results in SSA. Hence, due to these challenges and the contemporary shift in agricultural extension paradigms, there is a need to re-examine the practice of impact evaluation on agricultural extension programmes.

This paradigm shift emanates from challenges faced by agricultural extension organisations, farmers and agriculture due to the dynamic nature of social and natural environments (Leeuwis 2004). These are related to funding, new emerging roles, development in extension theories and practices and the emergence and expansion of new communication technologies. These changes have necessitated the need for dealing with collective issues, co-designing rather than disseminating innovations, matching the technical and social dimension of innovation, catering for diverse farming and livelihood strategies, managing complexity, conflict and unpredictability, becoming learning organisations, being brokers in an era of participation, and coping with dwindling resources (Leeuwis 2004). The new agricultural extension principles include a shift from ‘transfer of technology’ to facilitation and from considering farmers as ‘beneficiaries’ to clients and stakeholders. Moreover, it recognises the importance of market demand, the private sector, pluralism and decentralisation, along with the necessary coordination and dialogue among actors (Neuchâtel Group 1999).

Such a paradigm shift in agricultural research and extension calls for a review of M&E approaches including impact evaluation. For instance, one of the developments in extension science that affects extension impact evaluation is the need to manage complexity, conflict and unpredictability. Technical and agro-ecological processes do not always behave in expected and linear ways, and at the point where the social and technical meet, many unforeseen developments may take place (Holling 1995). Furthermore, innovations are inherently messy, chaotic, complex and unpredictable (Leeuwis 2004). Hence, extension organisations need to organise their interventions in a much more adaptive and flexible way so that learning experiences and emergent developments can be incorporated in ongoing activities. This requires new forms of monitoring and evaluation including impact evaluation.

Creating causal relationships between interventions and outcomes will not be easy due to the complex, dynamic and institutional pluralism nature of the agricultural development and the new research and extension paradigm. Hence, M&E approaches which are participatory, qualitative and focused on learning would be useful. Likewise, due to the challenges related to impact evaluation mentioned above, there is a need to move from ‘proving impact’ to ‘improving impact’. The former aims at providing conclusive evidence for donors or policy-makers and the approach taken is likely to be externally led and biased towards collecting quantitative data for accountability purposes. The latter looks at how to improve practice and develop internal learning systems, with a focus on qualitative data and use of participatory tools. In line with the new development discourse, various M&E tools have been developed including outcome mapping (OM), most significant change (MSC), etc. These tools are more learning oriented, focus on contribution than attribution, participatory and flexible in nature

Attribution and estimating the counterfactual are the challenges that emanate from the nature of impact evaluation itself. The challenges related to the nature of agricultural extension programmes are: spill over effects, selection biases, endogeneity in farmers’ extension interactions, differentiating the effect of extension programme from other sources of technology or information and other complementary services such as research and the difficulty of establishing a comparable control group due to ethical and political reasons.

The complex, diverse and risk-prone nature of agriculture in sub-Saharan Africa makes it difficult to disentangle the true impact of agricultural extension services. The definition and indicators used to measure impact, application of appropriate impact evaluation design, sample size, statistical techniques used for analysis, availability of baseline data and adequate data and capacity of staff in rigours impact evaluations and availability of other resources, are other factors that affect the validity and quality of impact evaluation results.

Agricultural extension has undergone a paradigm shift, from transfer of technology (ToT) approaches to innovation systems approaches, which perceive the agricultural development process as complex and dynamic and with institutional pluralism against the conventional linear and unidirectional approaches. Hence, this new paradigm requires a shift in M&E methodologies. In line with this, impact evaluation objectives also have shown a shift from proving impact to improving impact.

Building evaluation capacity, establishing appropriate M&E systems, using impact evaluation approaches in line with the reality in SSA and a change in agricultural extensions paradigms, using easy, less costly and more flexible tools would improve shifting the objective of impact evaluation from proving to improving impact, would improve the quality, validity and usefulness of impact evaluation results in the region. For instance, outcome mapping (OM) has been reported to be a useful monitoring, evaluation and learning tool for interventions that require dynamic and complex processes and the contribution of multiple stakeholders at various levels. The tool enhances the possibility of development impact through continuously improving implementation based on feedback information and learning (Earl, Carden & Smutylo 2001).

Anderson, J.R., 2007, ‘Agricultural advisory services.’ Background Paper for World Development Report 2008, Agriculture for Development, World Bank, Washington, DC.

Anderson, J. R. & Feder, G., 2007, ‘Agricultural extension’, in R.E. Evenson & P. Pingali (eds.), Handbook of agricultural economics, vol. 3, Agricultural development: Farmers, farm production, and farm markets, p. 2343–2378, Elsevier, Amsterdam.

Anderson, J.R., Feder, G. & Ganguly, S., 2006, ‘The rise and fall of training and visit extension: An Asian mini-drama with an African epilogue?’, in A.W. Van den Ban & R.K. Samanta (eds.), Changing roles of agricultural extension in Asian nations, p. 149–172, B.R. Publishing Company, New Delhi.

Ayele, G., Alemu, D. & Kelemework, F., 2005, ‘The provisions of rural services in Ethiopia: Characterization, impacts, and farmers’ priorities’, unpublished manuscript, International Food Policy Research Institute (IFPRI), Washington, DC.

Benin, S., Nkonya, E., Okecho, G., Pender, J., Nahdy, S., Mugarura, S., Kato, E. & Kayobyo, G., 2007, ‘Assessing the impact of the national agricultural advisory services (NAADS) in the Uganda rural livelihoods’, IFPRI Discussion Paper 00724. International Food Policy Research Institute (IFPRI), Washington, DC, viewed 27 August 2013, from http://www.ifpri.org/sites/default/files/publications/ifpridp00724.pdf

Bindlish, V. & Evenson, R.E. 1997. ‘The impact of T&V extension in Africa: The experience of Kenya and Burkina Faso’, The World Bank Research Observer 12 (2), 183–201.

Birkhaeuser, D., Evenson, R.E. & Feder, G., 1991, ‘The economic impact of agricultural extension: A review.’ Economic Development and Cultural Change 39(3), 607–640.

Birner, R., Davis, K., Pender, J., Nkonya, E., Anandajayasekeram, P., Ekboir, J., Mbabu, A., Spielman, D., Horna, D., Benin, S. & Cohen, M. 2006, ‘From “best practice” to “best fit”: A framework for analyzing pluralistic agricultural advisory services worldwide’, DSGD Discussion Paper No. 37, IFPRI, Washington, DC.

Chambers, R. & B.P. Ghildyal, 1984, ‘Agricultural research for resource-poor farmers: The “farmer first and last” model’, Paper for the National Agricultural Research Project Workshop, Hyderabad, India.

Davis, K. 2008. ‘Extension in sub-Saharan Africa: Overview and assessment of past and current models and future prospects’, Journal of International Agricultural and Extension Education 15(3), 15–28.

Davis, K., Nkonya, E., Kato, E., Mekonnen, M., Odendo, M., Miiro, R. & Nkuba, J. 2010, ‘Impact of farmer field schools on agricultural productivity and poverty in East Africa’, IFPRI Discussion Paper 00992, International Food Policy Research Institute, Washington, DC, viewed 27 August 2013, from http://www.ifpri.org/sites/default/files/publications/ifpridp00992.pdf

Earl, S., Carden, F. & Smutylo, T. 2001, Outcome mapping – Building learning and reflection into development programs, International Development Research Centre (IDRC), Ottawa.

ECON Analysis, 2005), ‘Impacts of extension services in rural Mozambique’, Report 2005-015, commissioned by World Bank, Project no. 42860 Public, viewed 27 August 2013, from http://www.econ.no/stream_file.asp?iEntityId=2001

EEA/EEPRI. 2006. Evaluation of the Ethiopian agricultural extension with particular emphasis on the participatory demonstration and training extension system (PADETES), Ethiopian Economic Association/Ethiopian Economic Policy Research Institute, Addis Ababa.

FAO, 1997, Improving agricultural extension: A reference manual, FAO, Rome.

Gautam, M. & Anderson, J.R., 1999, ‘Reconsidering the evidence on returns to T&V extension in Kenya’, Policy Research Working Paper (WPS2098), World Bank, Washington, DC, viewed 27 August 2013, from http://www-wds.worldbank.org/external/default/WDSContentServer/IW3P/IB/1999/06/04/000094946_99042105315210/additional/113513322_20041117141033.pdf

Haggblade, S. & Hazell, P.B.R., 2010, ‘Successes in African agriculture.’ IFPRI Issue Brief No. 63, International Food Policy Research Institute, Washington, DC, viewed 27 August 2013, from http://www.ifpri.org/sites/default/files/publications/ib63.pdf

Haggblade, S., Hazell, P., Kirsten, I & Mkandawire, R., 2004, ‘Building on successes in African agriculture: African agriculture – Past performance, future imperatives’, 2020 Vision Focus 12 Brief No. 1, IFPRI, Washington, DC.

Holling, C.S., 1995, ‘What barriers? What bridges?’, in L.H. Gunderson, C.S. Holling & S.S. Light (eds.), Barriers and bridges to the renewal of ecosystems and institutions, p. 3–37. Columbia University Press, New York.

IFPRI, 2008, ‘The impact of agricultural extension and roads on poverty and consumption growth in fifteen Ethiopian villages’, IFPRI Discussion Paper 00840, International Food Policy Research Institute, Washington, DC, viewed 27 August 2013, from http://www.ifpri.org/sites/default/files/publications/ifpridp00840.pdf

Kahan M.H. & Kahan M.S., 1995, ‘Agricultual growth in China and sub-Saharan African counties’, The Pakistan Development Review 34(4), 429–456.

Leeuwis, C. & Van den Ban, A., 2004, Communication for rural innovation: Rethinking agricultural extension, 3rd edn., Blackwell, Oxford.

Neuchâtel Group, 1999, Common framework on agricultural extension, Neuchâtel Group, Paris, viewed 27 August 2013, from http://www.g-fras.org/fileadmin/UserFiles/Documents/Frames-and-guidelines/New-paradigms/Common-Framework-on-Agricultural-Extension.pdf

NEPAD (New Partnership for African Development), 2003, Comprehensive Africa agricultural development programme, NEPAD, Pretoria.

Oleru, J., 2005, ‘Assessing impact of contract agricultural extension system: Promotion of four priority enterprises in Arua District, north-western Uganda’, African Crop Science Conference Proceedings, vol. 7. p. 733–738.

Owens, T., Hoddinott, J., & Kinsey, B., 2003, ‘The impact of agricultural extension on farm production in resettlement areas of Zimbabwe’, Economic Development and Cultural Change 51(2), 337–357.

Purcell, D.L., & Anderson, J.R., 1997, Agricultural extension and research: Achievements and problems in national systems, World Bank, Washington, DC.

Romani, M., 2003, The impact of extension services in times of crisis: Côte d’Ivoire (1997–2000), Centre for the Study of African Economies, University of Oxford, viewed 27 August 2013, from http://www.econ.yale.edu/conference/neudc03/papers/4a-romani.pdf

Spielman, D., 2008, ‘Encouraging economic growth in Ethiopia: Perspectives on agricultural input markets, agricultural extension and advisory services, and agricultural education and training’, background paper prepared for a study on agriculture and growth in Ethiopia, viewed 27 August 2013, from http://users.ox.ac.uk/~econstd/Brief_DSpielman.pdf

Staatz, J.M. & Dembele, N.N., 2007, Background paper for World Development Report 2008, World Bank, Washington, DC, viewed 27 August 2013, from http://siteresources.worldbank.org/INTWDR2008/Resources/2795087-1191427986785/StaatzJ&DembeleN_AgriForDevtInSSA_ve19.pdf

Temu, A.E., Nyange, D., Mattee, A.Z. & Kashasha, L.K. 2005, Assessing rural services, infrastructure and their impact on agricultural production, marketing and food security in Tanzania, Arusha, Tanzania Development Gateway, viewed 27 August 2013, from http://www.tanzaniagateway.org/docs/AssessingRuralServicesIandInfrastructureonAgriculturalProduction_Tanzania.pdf

Van den Ban, A.W. & Hawkins, H.S., 1996, Agricultural extension, 2nd edn., Blackwell Science, Oxford.

Waddington, H., Birte, S., White, H. & Anderson, J., 2010, ‘The impact of agricultural extension services’, synthetic reviews, SR009 Protocol, International Initiative for Impact Evaluation, New Delhi, viewed 27 August 2013, from http://www.3ieimpact.org/media/filer/2012/05/07/009%20Protocol.pdf

WFP, 2005, Briefing note, African Union Commission Report on Food Security in Africa, AU Summit, Sirte.

Walker, T., Tschirley, D., Low, J., Pequenino Tanque, M., Boughton, D., Payongayong, E. & Weber, M., 2004, ‘Determinants of rural income in Mozambique in 2001–2002’, Research Report 57E, Ministry of Agriculture and Rural Development, Directorate of Economics, Maputo, viewed 27 August 2013, from http://ageconsearch.umn.edu/bitstream/56061/2/wps57E.pdf

White, H., 2009, ‘Some Reflections on Current Debates in Impact Evaluation’, Working Paper 1, International Initiative for Impact Evaluation, New Delhi.