The growing demand for evidence to support policy decisions, guide resource allocation and demonstrate results has elevated the need for expertise in monitoring and evaluation (M&E). Despite the mushrooming of short courses in M&E, their impact on improving the capacity to meet the demand has not been adequately and comprehensively measured or evaluated. The purpose of this article was to highlight the need for improving the measurement of evaluation capacity building (ECB) to better understand what works in building M&E capacity in Africa.
This article provides important insights into the need for empirical and rigorous measurement of ECB interventions and their role in strengthening evaluation practice.
The study was primarily a desktop review of existing literature, corroborated by a survey of a few senior representatives of organisations responsible for capacity building across the African continent.
The review found that there remains little empirical evidence that indicates whether ECB processes, activities and outcomes are ultimately effective. There is also very little empirical evidence that helps to interpret how change happens, and how this may shape ECB efforts. Training is acknowledged as only one element of ECB, and there is a need for a multi-pronged approach to ECB.
Much more empirical and rigorous research is needed to build a clear understanding of what conditions are needed in ECB in Africa to strengthen evaluation practice. This article is useful for guiding further research into measuring the effect of ECB, as well as implementing more effective models of ECB towards strengthening evaluation practice in Africa.
Evaluation capacity building (ECB) has been recognised as a critical issue for development by international agreements such as the Accra Agenda for Action, the Cairo Consensus on Capacity Development as well as the Busan High-Level Forum (Lucas
An oft-quoted definition of ECB by Stockdill, Baizerman and Compton (in Ross & Hopson
ECB involves the design and implementation of teaching and learning strategies to help individuals, groups, and organisations, learn about what constitutes effective, useful, and professional evaluation practice. The ultimate goal of ECB is sustainable evaluation practice – where members continuously ask questions that matter, collect, analyze and interpret data, and use evaluation findings for decision-making and action. For evaluation practice to be sustained, participants must be provided with leadership support, incentives, resources, and opportunities to transfer their learning about evaluation to their everyday work. Sustainable evaluation practice also requires the development of systems, processes, policies, and plans that help embed evaluation work into the way the organization accomplishes its mission and strategic goals. (p. 444)
The authors designate 10 different strategies, of which training is one, in their taxonomy of factors to consider when embarking on any ECB intervention (Preskill and Boyle
The challenge in the evaluation community is that there is still no globally ratified consensus on the guidelines for desired evaluator practice and essential evaluator competencies that is used consistently in ECB in Africa (despite the existence of legitimate competencies and standards produced by, for example, the International Development Evaluation Association, the African Evaluation Association, the Joint Committee on Standards for Educational Evaluation and others). There is also no professional body overseeing the content of M&E training material on the African continent. The quality, content and composition of training courses vary between institutions. Of the many training courses that are undertaken every year, it is not known whether these assist individuals to become better evaluators, or if they strengthen organisational evaluation practice. There is little empirical evidence available that tests whether ECB processes, activities and outcomes in general are ultimately effective (Preskill & Boyle
The problem that this review sought to unpack was the absence of adequate attempts to measure the effectiveness of ECB in strengthening M&E capacity on the African continent. The problem is compounded by the lack of a singular and agreed upon definition of what constitutes M&E capacity. This further problematises the measurement of capacity (i.e. which indicators to use, which tools can be used to measure and the level – individual or organisational – at which capacity is measured, as well as the absence of baseline indicators, which tell us what capacity was there to start with before training began).
This review therefore explores the challenges facing ECB and measuring the effect of ECB in Africa through a document study and a survey of regional institutions (including the South African Development Community, Economic Community of West African States, the African Union Commission and Economic Community of Central African States, among others) from parts of the central, east and west African ECB community. It also makes recommendations for moving beyond the status quo in the design and measurement of ECB initiatives on the African continent.
The primary purpose of this review is to explore the extent to which the effectiveness of ECB (including its contribution to strengthening evaluation practice) is adequately measured on the African continent.
Capacity building initiatives in Africa have traditionally been driven by donor agencies since the inception of development aid, and take various forms. Most commonly, they include the training of individuals as a basic tenet, supported by the provision of technical support. ECB has experienced rapid growth within public sector organisations on an international scale (Naccarella et al.
It is important to interrogate the effects of capacity building interventions in Africa, especially in light of the mushrooming of M&E training offerings across the continent. The questions that remain unanswered include:
What role does training play in ECB in Africa?
How do we ascertain that the training to build evaluation capacity is yielding the intended results?
What are the challenges to measuring the effect of training on building evaluation capacity and how might they be addressed?
In response to the widespread agreement in the evaluation sector that there is a research gap in what constitutes evaluation ‘capacity’, and how the various factors that constitute ECB inter-relate, Taylor-Ritzler et al. (
The growing demand for good governance and accountability within the African continent demands that we embark on a path to adequately measure the effect of capacity building interventions. Considering the protracted global economic downturn and the impact on the availability of financial resources within developing countries (in particular the continent of Africa) for the myriad development challenges facing it, there is a need to ensure that any expenditure on interventions results in improvements in development results.
The specific objectives of the review were to explore:
The various approaches to defining and measuring ECB.
The extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity.
Challenges experienced in the measurement of the effect of training on ECB in Africa, and recommendations for how these may be addressed.
The literature review focused on the following key concepts: evaluation capacity; ECB and the measurement of these. It further explores the aspect of competencies in evaluation, and how this interfaces with the concepts of capacity and capacity building, which form the back-drop to an exploration of the efforts at, and complexities of, the measurement of ECB interventions.
Significant resources have been invested in building evaluation capacity (particularly by international donor agencies) in Africa; however, it is not clear whether these efforts are yielding results (Tarsilla
The ability to sustain the capacity to use evaluative knowledge in decision-making.
Aligning the organisation’s functional elements in a way that is coherent, and supports effectiveness.
Allowing evaluative knowledge in its varied forms to be used in all phases of the policy cycle.
The ability to use evaluation knowledge at several levels of practice and decision-making to improve organisational effectiveness.
This study challenged the notion that evaluation capacity is limited to the ability to conduct evaluations and use the findings appropriately (Nielsen et al.
Brinkerhoff and Morgan (
the capability to commit and engage; the capability to carry out technical, service delivery and logistical tasks; the capability to relate and attract support; the capability to adapt and self-renew; the capability to balance diversity and coherence. (Brinkerhoff & Morgan
The authors further posit that capacity is a ‘latent phenomenon’, and only becomes explicit once it is exercised in order to attain a certain outcome. Defining evaluation capacity is therefore complex and multifaceted, including the provision of financial as well as other resources (Crisp, Swerissen & Duckett
Building and measuring capacity is equally complex, it can take many forms and may occur at an individual, organisational or systems level. LaFond et al. (
The lack of consensus in defining evaluation capacity and capacity building poses challenges in the measurement of ECB interventions.
The study conducted by Tarsilla (
Davies and Mackay (
Dillman (
Training therefore remains important, but not in isolation. A study conducted on ECB interventions in 13 community-based organisations revealed that ECB does increase organisational evaluation capacity (Stevenson et al.
Podems (
Training, and by extension efforts at building individual competencies, can only go as far as imparting and facilitating skills and knowledge transfer, but cannot determine the capacity to implement these. This has implications for measurement and attribution, and how the evaluation community may determine if a single training intervention has the ability to build capacity in a comprehensive sense.
Challenges in defining capacity and ECB invariably translate to challenges in
Tracer studies, if conducted empirically and designed well, could shed light on the impact of training (in particular) on knowledge acquisition, skills and practices. As very little research could be found on tracer studies in the ECB sector, ECB practitioners could learn from research conducted in other sectors, such as education, health and behavioural sciences. Studies in these sectors have shown that tracer studies ‘can be useful for gathering information that positively impacts on training and policy’ (Mubuuke, Businge & Kiguli-Malwadde
Crisp et al. (
The review was conducted through a literature review on ECB and its measurement, focusing specifically on the adequacy of training as one aspect of ECB, supported by a survey of key strategic institutions operating on the African continent and linked to ECB activities. The survey was conducted with a small convenience sample of key stakeholders (including organisations responsible for capacity building or specialising specifically in ECB across the African continent) who were invited to participate in a self-administered survey in the context of their responsibility for or linkages to ECB interventions across the African continent. The organisations were participants of a workshop, hosted by the African Capacity Building Foundation (ACBF) in October 2016, to identify the potential content, design and implementation modalities for an Africa-wide M&E capacity building programme. CLEAR-AA had also been invited to the workshop. It was a closed meeting, attended by a select number of organisations who are actively involved in regional integration, capacity building or M&E activities across the African continent. Permission was requested from the ACBF to distribute the survey instrument, and all of the organisations present were invited to complete the survey.
The survey instrument was developed using the following thematic areas as the point of departure: (1) the extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity and (2) challenges and recommendations in the measurement of the effect of training on ECB. The items on the survey included biographical information on the organisation represented by each respondent, including the countries within which it operated. This provided insight into the ‘footprint’ of the organisations on the African continent. In addition to the biographical information, the survey consisted of eight closed-ended items and three open-ended questions covering the following issues: the extent to which organisations were involved in M&E training, whether, as well as how, organisations measured the effect of their training (knowledge, behaviours, attitudes, practices) and challenges and recommendations to measuring the effect of ECB efforts on the African continent.
In total, 13 surveys were returned by senior representatives of 12 regional organisations, listed in
Survey respondents.
Name of organisation |
Countries served |
---|---|
1. CLEAR Francophone Africa | Francophone Africa |
2. East African Community (EAC) | Uganda, Kenya, Tanzania, Rwanda, Burundi |
3. Southern African Development Community (SADC) Secretariat | 15 SADC countries (including Angola, Lesotho, Malawi, Mozambique, Mauritius, Namibia, Madagascar, Seychelles, South Africa, Zambia, Zimbabwe) |
4. Arab Maghreb Union (UMA) | Algeria, Libya, Morocco, Mauritania, Tunisia |
5. African Union (AU) Commission | All African countries across the continent |
6. Economic Community of West African States (ECOWAS) Commission | Benin, Burkina Faso, Cape Verde, Ivory Coast, Guinea Bissau, Ghana, Liberia, Mali, Niger, Nigeria, Senegal, Togo |
7. Alliance for Green Revolution in Africa (AGRA) | 18 countries in Africa |
8. ACBF (African Capacity Building Foundation) | All African countries across the continent, particularly sub-Saharan Africa |
9. Senegalese Evaluation Association (SENEVAL) | Senegal |
10. African Evaluation Association (AFREA) | All African countries across the continent |
11. Economic Community of Central African States (CEEAC-ECCAS) | Gabon, Cameroon, Central African Republic, Chad and Burkina Faso |
The snap survey provided perspectives of senior staff members of key regional organisations who are extensively involved in the coordination or implementation of capacity building programmes in Africa in general and ECB specifically. Their views or responses, however, might not be representative of their entire respective organisations. Perspectives were sought around training and its effect on strengthening evaluation capacity. The results of the survey were analysed using simple descriptive statistics (tabulated in the results section of this article), due to the sample size being too small to draw any inferences to a larger population. Qualitative data were analysed using content analysis, as no preconceptions about the categories of findings were articulated ahead of the analysis. The findings emerging from the survey results were corroborated with the findings from the literature review, and the results categorised into themes and discussed in response to the stated objectives of the review.
There was no expectation to generalise or transfer the results of this review beyond the organisations represented in the sample. Convenience sampling, as a non-probability sampling method, precludes the drawing of inferences to a larger population. However, the corroboration of the survey results with the findings emerging from the literature studied allowed for the drawing of some general conclusions about the measurement of the effect of training on evaluation practice, and extrapolate this (in broad terms) across the African continent.
The survey results indicated that training remains an important component of ECB in Africa. According to
Number of respondents who conduct training in M&E.
Variable | Not applicable | Always | Often | Sometimes | Rarely | Total |
---|---|---|---|---|---|---|
Number of respondents who conduct M&E training | 1 | 2 | 3 | 6 | 1 | 13 |
Eight out of twelve of the respondents surveyed do M&E related work, while one has M&E functions which may not include a training component as a major function. Those who do not specialise in M&E capacity building (5 out of 13) listed the following as their main functions:
Regional coordination and integration
Economic integration of member states
Capacity building in policy management
M&E related services
Project monitoring
The survey did not request respondents to explain whether their training interventions were part of broader capacity building initiatives.
The survey results also revealed that very few of the respondents who conduct M&E training consistently measure the effect of their efforts.
Number of organisations who measure the effect of training.
Variable | Not applicable | Always | Often | Sometimes | Rarely | Total |
---|---|---|---|---|---|---|
Number of respondents who measure the effect of their training | 3 | 0 | 0 | 5 | 5 | 13 |
Only sometimes or rarely measure the effect of training on behavioural change (7 out of 13).
Only sometimes or rarely measure the effect of training of knowledge (8 out of 13).
Rarely measure the effect of training on attitudes (8 out of 13).
Rarely measure whether participants use the training in their day-to-day work (8 out of 13).
Respondents listed the following as the various approaches and methods used to measure the effect of their training interventions:
Questionnaires and evaluation forms
Interviews (individual and group)
Simple chi-squared test
Treatment and control (experimental design)
Pre-post-test (before-after comparison)
Most-significant change (MSC) technique
Individual follow-up
Site visits
What gets measured.
Types of effects measured | Not applicable | Always | Often | Sometimes | Rarely | Total |
---|---|---|---|---|---|---|
Effect on behavioural change | 1 | 2 | 3 | 6 | 1 | 13 |
Effect of training on knowledge | 2 | 1 | 2 | 3 | 5 | 13 |
Effect of training on attitudes | 2 | 1 | 0 | 2 | 8 | 13 |
Effect of training on practices | 2 | 1 | 1 | 4 | 5 | 13 |
Use of training in the day-to-day work | 2 | 1 | 0 | 2 | 8 | 13 |
In general, very few respondents regularly use the findings of their abovementioned pre-training or post-training measurement efforts. Only four respondents reported that they always or often use the findings of the studies they conduct to adjust their training interventions (
Whether findings are used to adjust training interventions.
Variable | Not applicable | Always | Often | Sometimes | Rarely | Total |
---|---|---|---|---|---|---|
No. of respondents who use their findings | 1 | 2 | 3 | 6 | 1 | 13 |
This could be related to the challenges highlighted by respondents around the measurement of their training programmes, which included inadequate data, limited resources as well as poor attention paid to evaluating the impact of ECB interventions. These challenges are further discussed below.
The challenges raised by survey respondents can be categorised into four broad areas: challenges related to planning and design, weaknesses in data collection and information management, resource challenges and challenges in the capacity to measure the effect of training programmes. In terms of planning and design, survey respondents highlighted the challenge of weak planning of ECB impact assessments in particular. It was emphasised that not enough attention is paid to measuring the effect or impact of ECB interventions. One reason that was provided for this is that donor interests are in what was termed ‘programmatic initiatives’, and therefore activities related to measuring the impact of the training programmes are not prioritised. At least one respondent pointed out that the selection of appropriate indicators to measure the effect of training was a challenge in measuring the effect or impact of ECB.
In relation to data collection and information management, respondents highlighted that tracing beneficiaries (i.e. training participants) in order to measure the effect of the training intervention posed a challenge. For those who managed to collect information from participants, the low response rate as well as what were held to be ‘poor/inaccurate responses to questions’ were added to the list of challenges. Inadequate monitoring data, the loss of institutional memory as well as inadequate knowledge management were also listed as areas of concern. The third category of challenges revolves around the availability of resources to support the measurement of training effects. This included both human and financial resource constraints for some respondents, where one respondent noted ‘resources in terms of skills and financial to monitor impact studies of any kind’ as a challenge. One respondent also noted that the capacity to measure the effect of training posed a challenge to their ability to do any measurement of this kind.
The survey respondents also made a number of suggestions to address these challenges. These included improving the availability of proper monitoring data as well as improved knowledge management. At least one respondent suggested that tracer studies and some form of post-training follow-up must be introduced. More rigorous methods of measuring the effect of training were also recommended, including the use of impact evaluations.
The primary objectives of this article were to: (1) explore the extent of, and ways in which, selected key institutions on the African continent measure the effect of training on strengthening evaluation capacity and (2) discuss the challenges of and make recommendations for improving the measurement of the effect of training on ECB in Africa. Each of these is discussed separately below.
The findings of the review, supported by literature (Tarsilla
Capacity, in all its complexity, is in fact hard to measure (Tarsilla
The intersecting variables (such as incentives to learn, institutional support, etc.) that play a role in the extent to which evaluation practice is strengthened in individuals and organisations contribute to this complexity. The review found that there is little empirical evidence available that tests whether ECB processes, activities and outcomes in general are ultimately effective (Preskill & Boyle
Even within the formal academic education sector, there are hardly any evaluations of changes in knowledge, attitudes and behaviour as a result of teaching and learning that the ECB sector can learn from (Burgess & Carpenter
The need for more empirical research in this area of ECB could perhaps be filled by the use of randomised control trials (RCTs, Suarez-Balcazar & Taylor-Ritzler
The tracer study conducted by CLEAR-AA in 2015 on the perceptions of participants about the effect of in-service evaluation training conducted on behalf of the South African DPME, revealed that the highest perceived gains were made in the area of knowledge transfer and less in the management of evaluations (CLEAR-AA
ECB initiatives in Africa are often poorly designed and planned (Tarsilla
Suarez-Balcazar & Taylor-Ritzler (
Poor planning and design adds to the measurement of ECB efforts as well, confirmed in the survey findings wherein respondents highlighted the challenge of weak planning of ECB impact assessments, and the lack of attention that is paid to measuring the effect or impact of ECB interventions. Even though some authors (Taylor-Ritzler et al.
LaFond et al. (
The small sample size and non-probability sampling method in this review does not allow for a generalisation of the results to the entire ECB community across the continent of Africa. Furthermore, the view of the individual respondents cannot be held to be representative of their respective organisations. Some of the institutions sampled are also hosts to other institutions who specialise in training provision and other ECB interventions, and this distinction was not made in the instrument. The survey instrument also did not assess whether institutions supplemented their training in M&E with other ECB interventions, which should form part of future research of this nature.
The findings of this review indicate some of the research gaps in ECB as well as the kinds of empirical evidence that is required to adequately measure the effect of training on changing knowledge, behaviours, attitudes and practices. Future studies of this nature would need to adopt a more robust probability sampling method and possibly employ more extensive methods of both quantitative and qualitative data collection (such as interviews and focus groups) in order to emerge with a deeper and more nuanced analysis of the conditions under which ECB (and training in particular) are most suited to delivering on development outcomes.
A number of questions remain unanswered around what is considered a ‘capacitated’ individual or institution as it pertains to evaluation, how this is measured and what interventions are required to build such capacity. In response to the objectives of this review, the following recommendations are a reflection of what institutions (especially those whose mandate involves ECB) need to focus on going forward if ECB is to have any impact on strengthening evaluation practice on the continent in a meaningful way.
As emphasised above, training forms one part of a package of interventions in M&E capacity building. The findings of the review have revealed that M&E training is often delivered as a standalone intervention in ECB, but that an integrated set of services (which may include technical assistance, coaching and other means of support) is more effective in strengthening evaluation practice. The design of ECB interventions for institutions should, ideally, include a multiplicity of strategies and activities and should (as far as possible) not constitute of training as a standalone event.
The findings have shown that capacity building is a multidimensional, dynamic and complex phenomenon, which is influenced by many elements including individuals’ existing capacity, their knowledge, behaviours and attitudes. It is also influenced by the context within which the individual or institution finds themselves. M&E capacity is not measured in a standardised manner on the African continent, and therefore it is difficult to build an empirical case for why there is a need to continue providing short course training to an ever-growing clientele. There is a critical research gap in evaluating the impact of training on changes in attitudes, behaviours and practices in the evaluation sector (as well as to some degree knowledge), and therefore more research is therefore required in this area. It is also recommended that all training interventions should include, as a matter of course, a plan for the evaluation of the effect of the training on the acquisition of evaluation competencies, which need to be agreed upon at the design phase of the ECB intervention.
There is scant use of tracer studies to track M&E training participants’ acquisition of knowledge and skills through coursework programmes in the ECB sector. One is even less likely to find studies that track whether training has improved participants’ ability to use their newfound skills in a way that improves evaluation practice. Evaluative thinking is a fundamental tenet of M&E, and it is a foregone conclusion that any interventions of this nature need to be monitored and evaluated to test the soundness, efficiency and effectiveness of the intervention logic. All institutions embarking on ECB interventions should therefore ensure that tracer studies are built into all training of this nature, so that ongoing empirical evidence may be built up over time about the effect of training on strengthening evaluation practice.
The write-up and publication of the results of those who do conduct studies would be very useful and are critical for the evaluation community in Africa to learn more about the strategies and methods of ECB (in particular training) that work well, and those that do not. There is also an opportunity to develop a database of findings on the evaluation of ECB efforts, which would be valuable to the evaluation community in general, and ECB individuals and institutions in particular, to inform the trajectory of future ECB interventions.
Much more empirical and rigorous research is needed to build a clear understanding of what conditions are ideal for the transfer of evaluation skills, competencies and knowledge, and the strengthening of evaluation practice at large.
The multiplicity of challenges in ECB in Africa hampers the measurement of capacity building interventions. These include the lack of clarity and consensus on the meaning of evaluation capacity, the various dimensions that constitute capacity, the levels at which capacity may be identifiable (e.g. individual and organisation) as well as the absence of indicators and baseline information on evaluation capacity on the continent. The absence of a variety of robust measures of evaluation capacity, and in some cases lack of knowledge around those which do exist, means that (1) more empirical research is needed on measuring the attribution of various dimensions of ECB on strengthening evaluation practice and (2) more work is needed to mainstream the use of instruments that do exists (e.g. Taylor-Ritzler et al.
ECB interventions must be better designed to achieve the results they set out to achieve, and a longer-term view of capacity building needs to be adopted by institutions who are actively involved in ECB.
This article presents the findings of a desktop review, combined with a snapshot survey of institutions from parts of the African continent, of the measurement of ECB in strengthening evaluation practice. The review explored the challenges facing the ECB sector in general, as well as the difficulties in measuring the effect of ECB interventions, in particular training, as only one aspect of ECB. The findings revealed that the complexity of defining ECB renders it almost impossible to test whether ECB processes, activities and outcomes in general are ultimately effective. The many intersecting variables that play a role in the extent to which evaluation practice is strengthened in individuals and organisations would best be measured through empirical studies on the continent, of which there are currently very few. The review also found that there is a general gap in the research in Africa on the impact of training on improving the competencies required for quality evaluations. More work needs to be done to determine, through research, what conditions are ideal for the transfer of evaluation skills, competencies (which a number of evaluation associations are currently working on) and knowledge, and the strengthening of evaluation practice at large.
ECB has become a ‘hot topic’ within the evaluation field (Preskill & Boyle
This article contributes to addressing the gap in the research around measuring the effect of evaluation capacity building interventions in general, and across the African continent in particular. It provides a snapshot of the extent of the measurement of the effect of training on strengthening evaluation competencies and practice on the continent, and provides insights into future areas of research that are needed in order to enhance evidence-based policy and development practice.
The authors declare that they have no financial or personal relationship that may have inappropriately influenced them in writing this article.
C.M. was the project leader and was responsible for the problem identification and project design. Both C.M. and M.R. reviewed relevant literature on the topic and jointly did the write-up.
The 12th organisation remained anonymous