Abstract
Background: Over the years, Communities of Practice have gained popularity as a capacity-building method among Monitoring and Evaluation practitioners. Yet, thus far, relatively little is known about their effectiveness.
Objectives: This article focuses on National Evaluation Societies as Communities of Practice that aim to contribute to the monitoring and evaluation capacity building of their members.
Method: Drawing upon a survey of 35 National Evaluation Societies in 33 low- and middle-income countries, we explore to what extent capacity building efforts have been successful and what factors explain the relative success or failure in capacity building. We rely upon Qualitative Comparative Analysis as we are particularly interested in different pathways to ensure successful National Evaluation Societies.
Results: Our findings highlight that regular face-to-face contact is a particularly important element. This does not entirely come as a surprise, as monitoring and evaluation capacity building often implies tacit knowledge that is most effectively shared face-to-face. Furthermore, capacity building in conducting and, particularly, using evaluations entails building networks among the monitoring and evaluation supply and demand side which can most easily be done through regular face-to-face interaction.
Conclusion: Our findings are not only theoretically interesting, they are also policy relevant; they hint at the fact that in an era of quick advances in technology, investing in face-to-face contact among members remains important.
Introduction
Since the turn of the century, results-based management has become a buzzword in international development, with the Millennium Development Goals and the 2005 Paris Declaration (OECD/DAC 2005) being milestones in this respect. As the effective implementation of a results orientation in policies and budgeting is conditional upon a well-functioning Monitoring and Evaluation (M&E) system, M&E capacity building has moved up the agenda during the past decade. One of the instruments of capacity building increasingly being used by various agencies and organisations are Communities of Practice (CoP), whereby M&E practitioners, academics, policy makers, and others share knowledge and practices while building networks (see e.g. the Asian Development Bank CoP in M&E; the International Development Evaluation Association CoP; the African Evaluation Association CoP).
The concept of CoP has existed since 1991 (Lave & Wenger 1991) and has evolved over time (Cox 2005; Li et al. 2009a) without arriving at a clear consensus on the exact definition (see e.g. Handley et al. 2006; Roberts 2006). However, most authors (Brown & Duguid 1991; Lave & Wenger 1991; Wenger 1998, 2000; Wenger, McDermott & Snyder 2002) agree on the fact that CoP consist of groups of peers (practitioners, but also policy-makers and academics) who share a common interest in a specific area of competence and are willing to share their experiences by interacting on an ongoing basis. CoP are considered particularly useful in transferring operational and tacit knowledge (‘knowing how’) that cannot be easily conveyed through books or ex-cathedra teaching (Duguid 2005). This obviously also holds for evaluation capacity development which is defined as ‘the process of reinforcing or establishing the skills, resources, structures and commitment to conduct and use M&E through the simultaneous stimulation of supply and demand’ (ADB n.d.; Labine et al. 2012; Nielsen, Lemire & Skov 2011). In particular, capacity to use M&E is not something that can easily be taught ex-cathedra as it rather entails learning by doing, and building interfaces and networks among actors of the M&E demand and supply side.
In sharp contrast to the innovative practices of M&E on the ground, is the lack of research on the topic. To the best of our knowledge, no research exists on the specific topic of M&E CoP; yet there has been some recent research in other sectors, including education, health and business, in which CoP have become a mainstream Knowledge Management strategy (see Meessen et al. 2011). A number of recent reviews of CoP studies (Li et al. 2009a; Ranmuthugala et al. 2011a, 2011b) have highlighted that the existing research to date is highly descriptive, with little focus on the effectiveness of CoP, which is in part due to the huge variety in CoP (see e.g. Roberts 2006) and difficulties in making the concept operational (Li et al. 2009b). Instigated by these reviews, there are a number of recent efforts at the elaboration of conceptual frameworks which feed into more systematic research on success factors and hindrances (see e.g. Bertone et al. 2013).
Our research aims to add to these recent research initiatives. In order to address internal validity problems related to the huge variety in CoP, we have selected one single type of CoP. More specifically, we focus on National Evaluation Societies (NES), which usually bring together national actors involved in M&E demand and/or supply from various arenas, including policy makers, parliamentarians, bureaucrats, donors, civil society and the private sector. In doing this, NES, among others, provide a platform for continuous exchange of localised M&E methods and practices. While NES are a booming phenomenon, particularly in developing countries (Smith et al. 2011), systematic research on the topic is largely lacking. Our 2011 online survey of 35 NES in 33 low- and middle-income countries highlights that NES consider M&E capacity building to be one of their most important goals (see Holvoet & Dewachter 2013; Holvoet, Dewachter & Gildemyn 2011). Drawing upon a review of literature on CoP, organisational performance and evaluation capacity building, our survey identifies factors that likely influence the effectiveness of NES in M&E capacity building. Our conceptual framework, which is discussed in the next paragraph, distinguishes between actor-related (i.e., financial and human resources; membership size and composition; and the interaction modalities among the members) and context-related (i.e., donor support; and Poverty Reduction Strategy Paper [PRSP] processes) conditions. To capture the complex causality between actor and context conditions and the combinations between them, we look for successful pathways to M&E capacity building by using Qualitative Comparative Analysis (QCA). QCA will be presented in a more elaborate manner in section three, while section four puts forward the results of the QCA analysis. In the final section, the main findings are discussed and summarised.
Effectively building M&E capacity among NES members
No theoretical framework currently exists that can readily be used for studying the effectiveness of NES (as one type of CoP) in building M&E capacity. To identify factors that may influence NES’ performance in capacity building, we have found inspiration in the relatively scarce literature on COP’s effectiveness (see Li et al. 2009a; Ranmuthugala et al. 2011a, 2011b), the organisational performance literature (including resource mobilisation theory, see e.g. Edwards & Mc Carthy 2004; and political opportunity structure theory, see e.g. Tarrow 1994) and the evaluation capacity building literature (see e.g. Labin 2014; Labin et al. 2012; Nakrosis 2014; Nielsen et al. 2011). An online survey in English and French was elaborated in 2011 to collect data on national evaluation societies in low- and middle-income countries. Forty NES participated in the survey out of a total of 67 identified national evaluation societies, yielding a response rate of 60%, which is relatively high for an online survey. The survey focused on the goals and activities, the perceived performance of the evaluation societies as well as on organisational features. The ‘most knowledgeable respondent’ was targeted (see Bryman 1989) resulting in the majority of the respondents (67%) being president/board member/founding member, while the others identified themselves as mere members of the evaluation societies (19%) or M&E experts (14%). Information from the survey was triangulated with other sources of information such as interviews and informal discussions with some members of the African evaluation networks and our own participant observations.
In this article we focus specifically on ‘M&E capacity building’, which NES labelled as their second most important goal of the five goals which were identified (see Holvoet, Dewachter & Gildemyn 2011) Survey findings highlight that our respondents perceive the evaluation capacity of the NES members to have changed over the years (since 2006). Only one NES indicated deteriorating evaluation capacity, 25 NES felt evaluation capacity had (much) improved, while eight NES indicated a status quo (see Figure 1).
 |
FIGURE 1: Perceived change in National Evaluation Societies (NES) members’ evaluation capacity and perceived contribution of National Evaluation Societies to this change. |
|
However, this increase in the members’ capacity is not necessarily brought about by efforts from the NES. Hence, a separate question in the survey attempts to gauge the extent to which NES have contributed to capacity building of the evaluation capacities of their members. Figure 1 shows that (at least in perception) some 35% of NES have contributed much or very much to capacity building among their members, while 65% indicated that they did so only to a limited extent or even not at all.
Secondly, survey respondents were asked to rate to what extent organisational capacity factors (human resources, financial resources, social resources, etc.) and context-related factors (political openness, government interest, donor interest, etc.) have been beneficial in increasing the M&E capacity of NES members. More specifically, the respondents were asked to indicate (on a seven-point scale, where
1 = completely insufficient and 7 = completely sufficient) the degree to which they felt the resources under review were sufficiently available to the evaluation association to strengthen the M&E capacity of the members. The factors identified by the survey respondents as the least at their disposal (= lowest score) and most available (= highest score) to strengthen M&E capacity are included in our analytical framework. For strengthening the evaluation capacity of the members, NES indicated that financial and human resources were the most important stumbling blocks. Given that 75% of the NES in our survey had no paid staff and 70% of NES had a budget of less than $10 000, the lack of financial means and staff seems to be indeed widespread among NES. Evaluation societies therefore build heavily on the work of volunteers. These findings are in line with the evaluation capacity literature in which the availability of resources (staff/time/money) is often highlighted as one of the most important organizational-level factors for capacity building (see Labin et al. 2012). Similarly, the resource mobilisation theory highlights the importance of the level of resources available to an organisation for the amount of action an organisation can undertake (see Cress & Snow 1996; Edwards & McCarthy 2004; Eisinger 2002).
In line with the literature on CoP success ingredients (see e.g. ADB n.d.; James 2010; Meessen et al. 2011), NES’ most positively rated asset was found to be their membership base (average score = 5.62). Members have very diverse professional backgrounds, ranging from government agencies and civil society organisations, to academia, the private sector, donor agencies, parliament and even the media. There is a plethora of expertise in different evaluation methodologies and skills available (quantitative, qualitative, mixed methods) among the membership. While the diversity in methodological skills enables mutual learning and cooperation among members, the width in terms of professional sector and the concomitant networks facilitates rapid diffusion and dissemination of information and, in the long run, it can build trust beyond the borders of one’s own sector. NES members with different professional backgrounds also tend to have access to different types of information and resources. Parliamentarians often have easier access to national-level policy data, and academics often have a comparative advantage in terms of analytical and evaluation tools, while civil society organisations often have easy access to information about the actual implementation on the ground and citizens’ perceptions of policy effects at local level. This allows for useful triangulation of data from various sources, and may stimulate discussion and counter-analysis, while also helping to address the policy-to-implementation and research-to-policy gap (see Holvoet, Gildemyn & Inberg 2012). Finally, the diverse membership base nurtures the creation of interfaces between M&E demand and supply-side actors, which is likely to contribute to tacit knowledge on how to use M&E.
Another critical ingredient for success that is often referred to in CoP as well as M&E capacity building literature is face-to-face contact (see e.g. Fleming & Easton 2010; Labin et al. 2012). While the upsurge of ICT has led to an increase in virtual exchange opportunities, various authors (see e.g. Duguid 2005; Meessen et al. 2011; Orlikowksi 2002) refer to the continued importance of face-to-face interaction, as ‘tacit’ knowledge in particular cannot easily be transmitted but rather needs displaying and/or exemplification (see Duguid 2005:113). In this regard, Brown and Duguid (2001) and Vaast and Walsham (2009) usefully distinguish between ‘community of practice’ and ‘network of practice’, whereby the main difference is the degree of face-to-face interaction between members, resulting in closer (community) or looser (network) relationships. In line with this, we identified different interaction modalities among NES members as a potentially important factor of influence on the effectiveness of NES’ capacity building. Our sample diversity in this regard allows one to explore whether it is enough to interconnect a large group of people with similar interests and professional challenges online and offer them the opportunities to exchange knowledge and best practices, or whether more frequent face-to-face interaction is necessary in order to build relations of trust, foster cooperation and make exchange networks sustainable.
In a final step, we focus on the context in which evaluation societies function and, more specifically, which contextual conditions are important to explain NES’ ability to nurture their members’ evaluation capacity (see also Labin et al. 2012; Nakrosis 2014). For NES, the donor setting could be an important element of a context that is more-or-less conducive to evaluation capacity building. Given the specific context of low-income countries, we have incorporated as a context variable whether or not the NES was based in a country that signed the Paris Declaration. Against the background of the 2005 Paris Declaration (and the follow-up Accra Agenda for Action, 2008, and the Busan Partnership for Effective Development Cooperation, 2011) and its five key principles of country-ownership, harmonisation, alignment, results orientation and mutual accountability, there has been a renewed interest in M&E (Holvoet, Gildemyn & Inberg 2012). More specifically, an M&E reform agenda has unfolded whereby it is expected that donors break down their own M&E and align it with recipient countries’ M&E systems which need strengthening. From this vantage point, the assumption is that there may be more incentives to invest in evaluation capacity in PRSP countries. Additionally, seeing how donors are often pushing for M&E systems to be in place and improved (at various levels), we could also hypothesise that donor-supported NES have more incentives to invest in M&E capacity building. Finally, it could also be possible that such a positive effect only works when both conditions are present, namely when an NES is donor funded in a PRSP country.
In sum, our conceptual framework specifies three actor dimensions and one context dimension, each comprising two variables (Figure 2), namely, Resources (human and financial resources), Membership (the number of members and level of diversity in terms of the sectors in which they work), Exchange modalities (the frequency of face-to-face interaction and online activity) and Context (PRSP country and donor funded).
 |
FIGURE 2: Analytical framework to study effectiveness of National Evaluation Societies’ monitoring and evaluation capacity building. |
|
As we are interested in what combinations of actor and context conditions can facilitate effective M&E capacity building, we have used a crisp-set Qualitative Comparative Analysis (csQCA) to get an insight into NES’ effectiveness in capacity building. Prior to an overview and discussion of the main findings, the next section explains the QCA method.
Research method and design
In this article we use QCA (Ragin 1987), as it is particularly well placed to deal with complex causality (Schneider & Wagemann 2012). Lewegie summarises ‘complex causality’ as follows: causal factors combine with each other to lead to the occurrence of an event or phenomenon (conjunctural causation), different combinations of causal factors can lead to the occurrence of a given type of event or phenomenon (equifinality), and causal factors can have opposing effects depending on the combinations with other factors in which they are situated (asymmetry) (adapted from Mahoney & Goertz 2006:236; Wagemann & Schneider 2010:382 in Lewegie 2013). Based on the QCA analysis we will thus distil multiple causal configurations of context and actor conditions that lead to (un)successful M&E capacity building by NES. Moreover, we will not simply assume that the inverse of the configuration leading to successful M&E capacity building will necessarily lead to unsuccessful M&E capacity building. This implies that a separate analysis is performed for the positive outcome (= successful M&E capacity building) and the zero outcome (= NOT successful M&E capacity building).
Operationalisation of the outcomes, actor and context conditions
To analyse which combinations of factors are conducive for NES capacity building and which configurations rather lead to limited capacity building, we first operationalise our analytical model. Since we use csQCA, which requires dichotomous data (Schneider & Wagemann 2010), we also identify calibration points in order to dichotomise the data (see Table 1).
| TABLE 1: Operationalisation of the analytical Qualitative Comparative Analysis (QCA) framework and calibration of the QCA conditions. |
With respect to the actor conditions, various operationalisations of the factors were possible. ‘Financial resources’ was operationalised as the availability of paid staff and dichotomised based on having at least one or not having any paid staff as a ‘natural’ cut off point. Given the fact that NES mostly build on volunteer input, taking into account only paid workforce sells NES short. Moreover, not only the number of volunteers matters, but also the skills and competencies with which they are endowed. Unfortunately, as we had no survey data on the educational profile of all the members of a NES, we used a proxy indicator that captures the overall education level in a country (i.e., secondary and tertiary level education enrolment and completion rates at national level). Admittedly, this is a distant proxy, but we felt we had to control for the overall education level as there were substantial differences among countries (e.g. between DRC and Argentina). NES in countries scoring high on two out of the three higher education indicators were labelled as ‘high overall higher education level’ and those who did not as ‘not high overall higher education level’ (see UNESCO 2016). As regards the operationalisation of ‘NES membership’, we included both the number of individual NES members and the number of different sectors represented among their members. The cut off points for ‘high membership’ and ‘inclusive membership’ were set at more than 50 members and at least five different sectors, respectively. The reasoning behind the latter cut-off point was to distinguish between truly multisectoral membership (at least five sectors of the following list: Government; PRSP M&E Unit; Parliament; National civil society organisations; International civil society organisations; Universities/Research institutes; Consultancy firms; Donor agencies; Private sector) and those that do not have such an extensive multisector membership. The final actor dimension is the type of interaction modality the NES uses, whereby we operationalise ‘Frequency of face-to-face activity’ of NES as the frequency with which the NES organised meetings. The cut-off point was put at twice a year or more, for having frequent face-to-face contact, based on our appreciation of NES practices. Standard practice among NES is to have one ‘general assembly’ a year and, as we want to distinguish those with frequent face-to-face contact from the other NES, we thus consider all NES with meeting frequencies higher than the ‘standard once a year meeting’ as having frequent face-to-face contact. Having an operational website should provide us with an indication of online NES activity and was based on a survey question (yes/no). The context conditions in this research were made operational as follows: ‘being donor supported’ was operationalised by asking whether the NES received any donor support for its budget. Similarly, ‘PRSP context’ was also easy to operationalise and calibrate; we classified countries that have produced at least one PRSP and one progress report as ‘PRSP countries’ as this implies that the PRSP process has been institutionalised at least to some extent.
Finally, the outcome, ‘NES M&E capacity building’, was operationalised by the survey question which gauges NES’ perceived contribution to strengthening members’ M&E capacity (see Figure 1). The cut-off point was set so that only those having strengthened ‘much’ and ‘very much’ are considered as having high NES performance on M&E capacity building.
Results
Analysis and findings
Firstly, a raw data table was constructed in which the original data from the survey for six actor conditions and two context conditions were dichotomised into 1 (present/high) and 0 (absent/low). Eight dichotomised conditions (0/1) generated 263 logically possible combinations of conditions, of which 31 combinations were observed in reality (10 with a positive outcome, whereas 21 conjunctural combinations were linked to non-occurrence of the outcome (i.e., no/modest contribution to strengthening members’ M&E capacity).
The data were analysed using Tosmana (Cronqvist 2011). Originally, eight conditions in total were introduced for 35 cases, which yielded a proportion of conditions to cases of 0.23, still significantly less than is suggested as the upper limit (ratio = 0.33) for conducting meaningful QCA analyses (Marx 2010), but above the upper limit of seven conditions put forward by Marx (2010) as a threshold for generating non-random meaningful conjunctural paths. Given that one of the conditions could be removed without drastically changing the outcome and that this condition – namely, educational level – was actually not measured at NES level but rather at national level, we therefore decided to remove the educational level as it was only a distant proxy for capturing the availability of human resources for NES. The probability of QCA generating a model based on random data with 35 cases and seven conditions is zero (Marx 2010). Our 35-case analysis also easily surpasses the minimum of 12 cases and, with seven conditions, it does not surpass the upper limit of seven conditions put forward by Marx (2010) as a threshold for generating non-random meaningful conjunctural paths.
As mentioned previously, QCA is asymmetrical by assuming that the factors explaining the lack of NES M&E capacity building are not necessarily the inverse of the factors explaining successful NES M&E capacity building. Hence, the following sections subsequently focus on the analysis of perceived high NES performance in strengthening M&E capacity of their members (outcome = 1) and perceived ‘not high NES performance’ in strengthening M&E capacity building (outcome = 0). For each of the outcomes, we first concentrate on the necessity analysis before moving to the sufficiency analysis.
Analysis of necessity
As suggested by Schneider and Wagemann (2010), the necessity analysis should precede the actual analysis of the sufficient conditions in a QCA analysis. Lewegie defines necessity as ‘Condition A is necessary for outcome Y if the occurrence of Y is not possible without the presence of A, but A alone is not enough to produce Y’ (Lewegie 2013:5) and suggests using 0.90 as a consistency threshold and 0.50 as a coverage threshold, when considering if a condition is or is not necessary (Lewegie 2013). He further emphasises that necessary conditions are not often identified in empirical studies (George & Bennet 2005; Schneider & Wagemann 2007).
Applying the thresholds put forward by Lewegie (2013), the findings in Table 2 show that frequent face-to-face meetings is the only necessary (but not sufficient) condition for NES M&E capacity building. This is an interesting finding, particularly given the potential that is ascribed to online exchanges of best practices, tips and knowledge for capacity building. Finally, the necessity analysis also supports our removal of education as an explanatory condition as it is also the condition with the lowest value on necessity (both in consistency and coverage).
| TABLE 2: Necessity analysis for perceived high National Evaluation Societies’ performance on monitoring and evaluation capacity building. |
Analysis of sufficiency
The next step in a QCA analysis is to compose a ‘Truth table’ (Table 3) in which the cases are grouped according to the identical causal paths displayed; that is to say, NES that have the same (dichotomised) score for capacity building and the same score on the actor and context conditions are grouped together.
QCA then applies Boolean algebra to minimise the combinations for the (non) occurrence of the outcome. Boolean algebra employs ‘+’ as OR, and ‘*’ as AND. Capital letters indicate the presence of a condition, while lower-case letters indicate the absence of a condition. Two context conditions, namely, ‘D’ (having donor support) and ‘P’ (being in a PRSP country), and five actor conditions, namely, ‘A’ (high frequency of face-to-face meetings), ‘W’ (operational website), ‘M’ (extensive NES membership), ‘C’ (diverse composition of membership) and ‘E’ (paid employee) were included in the analysis. The QCA output summarised below is based on a process of logical minimisation (Grofman & Schneider 2009; Ragin 1987).
The overall consistency value is 1.00 as there are no contradictory cases, which comfortably surpasses the minimum threshold of 0.75 suggested by Ragin (2008:118). Nine different sufficient conjunctural configurations were identified (see solution below), which indicates that all nine combinations of context and actor conditions lead to a positive outcome (i.e., perceived high NES performance on M&E capacity building). By way of example, the first causal pathway leading to high NES perceived effectiveness in terms of M&E capacity building (eAwMCdP) reads as follows: an NES having frequent face-to-face meetings (A), with many members (M) and from many different sectors (C), operating in a PRSP country (P) – even without paid employees (e), in the absence of an operational website (w) and without donor support (d), is able to – at least in the perception – contribute greatly to strengthening NES members’ M&E capacities. The full solution reads:
eAwMCdP + EAWMCD + AWMCDp + EAwmCDP + EAWmcDp + eawMCDP + eAwmcDP + eAWMcDP + eAWMcDP + eAWmCdP → CAPACITY BUILDING
Besides the full solution, Appendix 1 also provides the parsimonious solution of our QCA analysis for reasons of transparency. The parsimonious solution includes hypothetical cases (logical remainders) which are based on theoretical knowledge and is generally used in QCA analysis to minimise the outcome (Ragin & Sonnett 2004; Rihoux & Ragin 2009). Due to the lack of prior theoretical knowledge on the topic of study, we did not use the parsimonious solution in our analysis.
The above full solution highlights that each of the nine pathways is a sufficient combination of conditions for perceived high NES performance on M&E capacity building, which implies that if the combination of conditions is present, the outcome will be positive.
Different paths can thus lead to high NES performance on M&E capacity building. Table 4 highlights that the most empirically-prominent causal path (EAWMCD with a unique coverage of 0.23) is also theoretically the most likely path to lead to significant capacity building as all conditions are positive/present: the NES has a paid full-time employee, has frequent face-to face interaction, an operational website, many members from different sectors, and is receiving donor funding. The latter result seems to confirm the importance, and direction of, the hypotheses formulated in our analytical framework for explaining effective M&E capacity building. Only the condition of being in a PRSP country is not defined in either a positive or negative direction in this combination. The NES in the Philippines, Niger and Burkina Faso follow this pathway to substantial M&E capacity building. Two other different pathways have a unique coverage of 0.15 (as they each cover two cases), namely, eAwMCdP and EAwmCDP. The first pathway combines frequent face-to-face activity, with membership strengths both in size and composition, and operates in a PRSP country, with no paid employees, an operational website and no donor support (Senegal, Pakistan). The second pathway (Madagascar, Georgia) can build on a paid employee, having frequent face-to-face interaction though no operational website, a diverse but low level of membership, and being donor supported in a PRSP country. The quasi-necessity of having frequent face-to-face interaction also becomes clear again. All pathways – except eawMCDP (Ethiopia) – have A (Frequent face-to-face interaction) in the configuration. Our QCA-findings are also supported by qualitative evidence provided by several of the survey respondents. The Réseau Burkinabe de Suivi-Evaluation, for instance, emphasises the importance of their ‘Evaluation days’, during which actors from different professional backgrounds meet together to discuss ideas and current thinking, as well as practical experiences. As to trigger evaluation use, they also deliberately invite policy makers. The same holds for the case of Senegal, where the ‘Senegalese days of evaluation’ are considered an important moment for strengthening capacities through different face-to-face training sessions. As a follow-up to their evaluation days, a committee was created which met with several government authorities to inform them about the importance and benefits of evaluation. In addition, the Réseau Nigérien de Suivi-Evaluation organises face-to-face meetings between the actors of the M&E supply and demand side, which facilitates joint identification of the existing M&E capacity buildings needs while at the same time creating mutual understanding. Similarly, the representative of Pakistan Evaluation Network emphasises the importance of creating a sense of ‘togetherness’, which facilitates the exchange of information and literature among participants from local non-governmental organisations, donor and/or United Nations organisations. He furthermore points out that ‘in such settings, there is also little hesitancy among members to come up with a problem and ask peers to suggest a solution or share similar experience’.
| TABLE 4: Sufficient conditions for perceived high National Evaluation Societies’ (NES) performance on monitoring and evaluation capacity building. |
Pathways to perceived low NES performance on M&E capacity building
Similar to the above analysis for high NES performance on M&E capacity building, explaining the perceived low NES performance on M&E capacity building should start with a necessity analysis, to map potentially necessary conditions before proceeding to the analysis of sufficient conditions.
The findings in Table 5 clearly show that the absence of frequent face-to-face contact is not a necessary condition for low performance on M&E capacity building (0.55 < 0.90). Therefore, there is no perfect symmetrical causality between ‘frequent face-to-face contact’ and ‘capacity building’. Whereas having frequent face-to-face contact is a necessary condition for having high NES performance on M&E capacity building, lacking face-to-face contact in the NES is not a necessary condition for failure to excel in M&E capacity building. Put differently, even some NES that do have face-to-face contact can still have limited performance on M&E capacity building. The condition that comes closest to being a necessary condition for not having substantial M&E capacity building is the absence of paid staff, with a consistency value of 0.82, or the absence of donor funding with 0.77 (both with coverage surpassing the 0.50 threshold).
| TABLE 5: Necessity analysis for perceived low National Evaluation Societies’ (NES) performance on monitoring and evaluation capacity building. |
Sufficiency analysis
Table 6 summarises the findings from the QCA analysis that was performed with regard to the zero outcome or, put differently, perceived low NES performance on M&E capacity building. The formula for the zero outcome is:
| TABLE 6: Sufficient conditions for perceived low National Evaluation Societies’ (NES) performance on monitoring and evaluation capacity building (= outcome 0). |
ewmdP + eAWmcp + EAWmdP +AWmcdP + eawcdp + eawMCd + eaWmCP + eaMCdp + eAWmdp + EaWMCDP + EaWMcdP + eAwMCDP + eAWMCdP → capacity building (see Appendix 1 for the parsimonious solution)
The most prominent empirical path, namely, ewmdP, is also the path that is most similar to what we theoretically expect (all factors absent), except for being in a PRSP country. Similarly, eawcdP is also a pathway where the factors that were predicted to be important for capacity building are absent (except PRSP country) and that leads to the absence of capacity building. However, we must admit that many other pathways to limited capacity building are possible, which makes it difficult to distill a clear pattern.
Discussion and Conclusion
Outline of results
This study analyses the contribution of National Evaluation Societies as Communities of Practice to the monitoring and evaluation capacity building of their members.
The most compelling finding from our QCA analysis is that frequent face-to-face contact is a necessary condition for substantial M&E capacity building by NES. While there is no perfect necessity, the necessity value is above the threshold and this is quite an exceptional finding. Moreover, if we plot ‘frequent face-to-face contact’ against the outcome of ‘significant M&E capacity building’ (see Figure 3), it becomes clear that the only exception to frequent face-to-face contact being a necessary condition for capacity building, namely Ethiopia, is a borderline case that is as close as possible to also being a necessary condition.
 |
FIGURE 3: Plot of frequent face-to-face contact against National Evaluation Societies’ monitoring and evaluation capacity building. |
|
Disregarding Ethiopia, which is a borderline case, the upper left quadrant (NES able to build M&E capacity without having frequent face-to-face contact) is empty. However, the reverse is not true. While most of the NES that are unable to significantly contribute to M&E capacity building have limited or no face-to-face contact, a considerable portion of them do (bottom right quadrant).
Reverting back to the original discussion in this article on modalities of capacity building and how online communities are often praised as the way to facilitate capacity building, in light of the findings of our analysis more modesty is probably in order when it comes to presenting the potential of online networks relating to practices for M&E capacity building. To compare the effect of offline and online M&E capacity-building modalities (admittedly using crude proxy variables) and reverting to classical statistics, we find that the average score on M&E capacity building is significantly higher for NES with frequent face-to-face interaction (average capacity building [frequent face-to-face interaction] = 3.59; average capacity building [infrequent face-to-face interaction] = 2.62; p = 0.04), while we do not find a significant difference in the average scores on capacity building for NES with operational websites, compared to those without.
Practical implications and recommendations
The important policy-relevant message to retain from this analysis is to keep in mind the importance of face-to-face contact for M&E capacity building, which is most likely related to the substantial involvement of unstandardised and localised hands-on tacit knowledge. Furthermore, capacity building in conducting, and particularly using, evaluations entails building networks between the M&E supply and demand side which can most easily be done through regular face-to-face interaction. This finding is particularly important against the growing realisation that building nationally-owned M&E systems is not a stroke-of-the-pen exercise. While national M&E systems are key ingredients for the realisation of the Paris/Accra/Bhusan principles of results orientation, alignment and mutual accountability, progress on the ground turns out to be among the most challenging components of the Paris Declaration reform agenda (see OECD 2011; OECD/DAC 2008). Disaggregating scores on indicator 11 (‘building results-oriented frameworks’) into its three sub-components, namely, ‘quality of information’, ‘access to information’ and ‘coordinated M&E system’, highlights that it is particularly the latter, more institutional, component that is most difficult to implement (World Bank 2007). This does not entirely come as a surprise as forging collaboration among different staff and entities dealing with M&E is a long-term process of institutional and personal trust building (see also Bedi et al. 2006). From this vantage point, the value added by NES which offer a unique forum for repeated and informal face-to-face interaction among much of the nationally available M&E expertise scattered over different sectors and institutions, is obvious. Supporting NES in the organisation of face-to-face activities may thus be one efficient way forward for those donors who have national M&E capacity building in their mandates and face challenges in making this objective operational against the backdrop of their own persistent capacity-related weaknesses (OECD/DAC 2010). As the lack of financial and/or human resources was identified as the condition closest to being a necessary condition for poor capacity-building performance, allocating resources to enable NES to have at least a minimal level of permanent human resources at their disposal could make a significant difference in NES performance.
Finally, the findings from our QCA analysis lend support to the conceptual framework we have elaborated, which is evident from the fact that the most prominent path to high NES performance on M&E capacity building is the one which includes the actor and context conditions included in our conceptual framework.
Acknowledgements
Competing interests
The authors declare that they have no financial or personal relationships which may have inappropriately influenced them in writing this article.
Authors’ contributions
Both N.H. and S.D. worked jointly on data collection, analysis, interpretation of the results and the drafting of the paper. The article is thus the collective work of both authors.
References
3rd High Level Aid Effectiveness, 2008, Accra agenda for action, 3rd High Level Forum on Aid Effectiveness, Accra.
4th High Level Aid Effectiveness, 2011, Busan partnership for effective development cooperation, 4th High Level Forum on Aid Effectiveness, Busan.
Asian Development Bank, s.d, Community of practice in monitoring and evaluation. C-Cube: Linking to results, Concept Paper, viewed 3 July 2014, from https://wpqr4.adb.org/
Bedi, T., Coudouel, A., Cox, M., Goldstein, M. & Thornton, N., 2006, Beyond the numbers. Understanding the institutions for monitoring poverty reduction strategies, World Bank, Washington, DC.
Bertone, M.P., Meessen, B., Claryssen, G., Hercot, D., Kelley, A., Kafando, Y., et al., 2013, ‘Assessing communities of practice in health policy: A conceptual framework as a first step towards empirical research’, Health Research Policy and Systems 11(39), 1–13. http://dx.doi.org/10.1186/1478-4505-11-39
Brown, J.S. & Duguid, P., 2001, ‘Knowledge and organization: A social-practice perspective’, Organization Science 12(1), 198–213. http://dx.doi.org/10.1287/orsc.12.2.198.10116
Bryman, A., 1989, Research methods and organization studies, Routledge, London.
Cox, A., 2005, ‘What are communities of practice? A comparative review of four seminal works’, Journal of Information Science 31, 527–540. http://dx.doi.org/10.1177/0165551505057016
Cress, D.M. & Snow, D.A., 1996, ‘Mobilization at the margins: Resources, benefactors, and the viability of homeless social movement organizations’, American Sociological Review 61(6), 1089–1109. http://dx.doi.org/10.2307/2096310
Cronqvist, L., 2011, Tosmana – Tool for small-N analysis [Version 1.3.2], Trier, viewed 3 July 2014, from http://www.tosmana.net
Duguid, P., 2005, ‘The art of knowing: Social and tacit dimensions of knowledge and the limits of the community of practice’, The Information Society: An International Journal 21(2), 109–118. http://dx.doi.org/10.1080/01972240590925311
Edwards, B. & McCarthy, J.D., 2004, ‘Resources and social movement mobilization’, in D. Snow, S. Soule & H. Kriesi (eds.), The Blackwell companion to social movements, pp. 116–152, Blackwell, Malden.
Eisinger, P., 2002, ‘Organizational capacity and organizational effectiveness among street-level food assistance programs’, Nonprofit and Voluntary Sector Quarterly 31(1), 115–130. http://dx.doi.org/10.1177/0899764002311005
Fleming, M.L. & Easton, J., 2010, ‘Building environmental educators’ evaluation capacity through distance education’, Evaluation and Program Planning 33, 172–177. http://dx.doi.org/10.1016/j.evalprogplan.2009.07.007
George, A. & Bennett, A., 2005, Case studies and theory development in the social sciences, MIT Press, Cambridge.
Grofman, B. & Schneider, C., 2009, ‘An introduction to crisp-set QCA, with a comparison to binary logistic regression’, Political Research Quarterly 62(4), 662–672. http://dx.doi.org/10.1177/1065912909338464
Handley, K., Sturdy, A., Fincham, R. & Clark, F., 2006, ‘Within and beyond communities of practice: Making sense of learning through participation, identity and practice’, Journal of Management Studies 43(3), 642–653. http://dx.doi.org/10.1111/j.1467-6486.2006.00605.x
Holvoet, N., Dewachter, S. & Gildemyn, M., 2011, Strengthening national evaluation societies: An exploratory survey, University of Antwerp, Institute of Development Policy and Management, Antwerp.
Holvoet, N. & Dewachter, S., 2013, ‘Multiple paths to effective national evaluation societies: Evidence from 37 low and middle-income countries’, American Journal of Evaluation 34(4), 519–544. http://dx.doi.org/10.1177/1098214013497083
Holvoet, N., Gildemyn, M. & Inberg, L., 2012, ‘Taking stock of monitoring and evaluation arrangements in the context of poverty reduction strategy papers: Evidence from 20 aid-dependent countries in sub-Saharan Africa’, Development Policy Review 30(6), 749–772. http://dx.doi.org/10.1111/j.1467-7679.2012.00597.x
James, R., 2010, ‘Monitoring and evaluating learning networks’, viewed 3 July 2014, from www.intrac.org
Labin, S.N., 2014, ‘Developing common measures in evaluation capacity building: An iterative science and practice process’, American Journal of Evaluation 35(1), 107–115. http://dx.doi.org/10.1177/1098214013499965
Labin, S.N., Duffy, J.L., Meyers, D.C., Wandersman, A. & Lesesne, C.A., 2012, ‘A research synthesis of the evaluation capacity building literature’, American Journal of Evaluation 33(3), 307–338. http://dx.doi.org/10.1177/1098214011434608
Lave, J. & Wenger, E., 1991, Situated learning: Legitimate peripheral participation, Cambridge University Press, Cambridge.
Lewegie, N., 2013, ‘An introduction to applied data analysis with qualitative comparative analysis (QCA)’, Forum Qualitative Social Research 14(3), Art 15.
Li, L.C., Grimshaw, J.M., Nielsen, C., Judd, M., Coyte, P.C. & Graham, I.D., 2009a, ‘Evolution of Wenger’s concept of community of practice’, Implementation Science 4(11), 1–8. http://dx.doi.org/10.1186/1748-5908-4-11
Li, L.C., Grimshaw, J.M., Nielsen, C., Judd, M., Coyte, P.C. & Graham, I.D., 2009b, ‘Use of communities of practice in business and health care sectors: A systematic review’, Implementation Science 4(27), 1–9. http://dx.doi.org/10.1186/1748-5908-4-27
Mahoney, J. & Goertz, G., 2006, ‘A tale of two cultures: Contrasting quantitative and qualitative research’, Political Analysis 14(3): 227–249. http://dx.doi.org/10.1093/pan/mpj017
Marx, A., 2010, ‘Crisp-set qualitative comparative analysis (csQCA) and model specification: Benchmarks for future csQCA applications’, International Journal of Multiple Research Approaches 4(2), 138–158. http://dx.doi.org/10.5172/mra.2010.4.2.138
Meessen, B., Kouanda, S., Musango, L., Richard, F., Ridde, V. & Soucat, A., 2011, ‘Communities of practice: The missing link for knowledge management on implementation issues in low-income countries?’, Tropical Medicine and International Health 16(8), 1007–1014. http://dx.doi.org/10.1111/j.1365-3156.2011.02794.x
Nakrosis, V., 2014, ‘Theory-based evaluation of capacity-building interventions’, Evaluation 20(1), 134–150. http://dx.doi.org/10.1177/1356389013517763
Nielsen, S.B., Lemire, S. & Skov, M., 2011, ‘Measuring evaluation capacity – Results and implications of a Danish study’, American Journal of Evaluation 32(3), 324–344. http://dx.doi.org/10.1177/1098214010396075
OECD, 2011, Aid effectiveness 2005–2010. Progress in implementing the Paris declaration, OECD, Paris.
OECD/DAC, 2005, Paris declaration on aid effectiveness, OECD/DAC, Paris.
OECD/DAC, 2008, Survey on monitoring the Paris declaration. Effective aid by 2010? What will it take? OECD/DAC, Paris.
OECD/DAC, 2010, Development evaluation resources and systems: A study of network members, OECD/DAC, Paris.
Orlikowksi, W., 2002, ‘Knowing in practice: enacting a collective capability in distributed organizing’, Organization Science 13(3):249–273. http://dx.doi.org/10.1287/orsc.13.3.249.2776
Ragin, C., 1987, The comparative method: Moving beyond qualitative and quantitative strategies, University of California Press, Berkeley, CA.
Ragin, C., 2008, Redesigning social inquiry: Fuzzy sets and beyond, University of Chicago Press, Chicago, IL.
Ragin, C. & Sonnett, J., 2004, ‘Between complexity and parsimony: Limited diversity, counterfactual cases, and comparative analysis’, in S. Kropp & M. Minkenberg (eds.), Vergleichen in Der Politikwissenschaft, pp. 180–197, VS Verlag für Sozialwissenschaften, Wiesbaden.
Ranmuthugala, G., Cunningham, F.C., Plumb, J.J., Long, J., Georgiou, A., Westbrook, J.I., et al., 2011a, ‘A realist evaluation of the role of communities of practice in changing healthcare practice’, Implementation Science 6, 49. http://dx.doi.org/10.1186/1748-5908-6-49
Ranmuthugala, G., Plumb, J.J., Cunningham, F.C., Georgiou, A., Westbrook, J.I., & Braithwaite, J., 2011b, ‘How and why are communities of practice established in the health care sector? A systematic review of literature’, BMC Health Services Research 11, 273. http://dx.doi.org/10.1186/1472-6963-11-273
Rihoux, B. & Ragin, C. (eds.), 2009, Configurational comparative methods: Qualitative comparative analysis (QCA) and related techniques, Sage, Thousand Oaks, CA.
Roberts, J., 2006, ‘Limits to communities of practice’, Journal of Management Studies 43(3), 623–639. http://dx.doi.org/10.1111/j.1467-6486.2006.00618.x
Schneider, C.O. & Wagemann, C., 2007, Qualitative Comparative Analysis (QCA) und Fuzzy Sets: Ein Lehrbuch für Anwender and jene die es warden wollen, Budrich, Opladen.
Schneider, C.Q. & Wagemann, C., 2010, ‘Standards of good practice in qualitative comparative analysis’, Comparative Sociology 9, 397–418. http://dx.doi.org/10.1163/156913210X12493538729793
Schneider, C.O. & Wagemann, C., 2012, Set-theoretic methods for the social sciences: a guide to Qualitative Comparative Analysis, Cambridge University Press, Cambridge.
Smith, N., Brandon, P., Hwalek, M., Kistler, S., Labin, S., Rugh, J., et al., 2011, ‘Looking ahead: The future of evaluation’, American Journal of Evaluation 32(4), 565–599. http://dx.doi.org/10.1177/1098214011421412
Tarrow, S., 1994, Power in movement, Cambridge University Press, Cambridge.
UNESCO, 2016, UNESCO Institute of Statistics online database, last viewed 18 August 2016, from http://data.uis.unesco.org
Vaast, E. & Walsham, G., 2009, ‘Trans-situated learning: Supporting a network of practice with an information infrastructure’, Information Systems Research 20(4), 547–564. http://dx.doi.org/10.1287/isre.1080.0228
Wagemann, C. & Schneider, C.Q., 2010, ‘Qualitative comparative analysis (QCA) and fuzzy-sets: Agenda for a research approach and a data analysis technique’, Comparative Sociology 9(3):376–396. http://dx.doi.org/10.1163/156913210x12493538729838
Wenger, E., 1998, Communities of practice, Cambridge University Press, Cambridge.
Wenger, E., 2000, ‘Communities of practice and social learning systems’, Organization 7(2), 225–246. http://dx.doi.org/10.1177/135050840072002
Wenger, E., McDermott, R. & Snyder, W.C., 2002, Cultivating communities of practice: A guide to managing knowledge, Harvard Business School Press, Cambridge, MA.
World Bank, 2007, Results-based national development strategies assessment and challenges ahead, World Bank, Washington, DC.
Appendix 1
Outcome = 1 with logical remainders: several solutions (parsimonious solution)
| Ep + |
awD + |
wmD + |
AwMd + |
AWMD + |
eAWmCP → CAPACITY BUILDING (N simplifying ass = 54) |
| EAD + |
awD + |
AMp + |
cDP + |
AwMd + |
eAWmCP → CAPACITY BUILDING (N simplifying ass = 48) |
| EAD + |
awD + |
MDp + |
cDP + |
AwMd + |
eAWmCP → CAPACITY BUILDING (N simplifying ass = 48) |
| EAD + |
awD + |
cDP + |
CDp + |
AwMd + |
eAWmCP → CAPACITY BUILDING (N simplifying ass = 48) |
The choice between the various options is based on: 1) the least simplifying assumptions; 2) a high degree of similarity between all three of the remaining solutions; 3) the presence of frequent face-to-face activity in combination with having many members (Amp)
Outcome = 0 without logical remainders: several solutions (complex solution)
| ewmdP + |
eAWmcp + |
eawcdp + |
eawMCd + AWmcdP + EAWmdP + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
| ewmdP + |
eAWmcp + |
eawcdp + |
eawMCd + EAWmdP + eAmcdP + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
| ewmdP + |
eAWmcp + |
eawcdp + |
eawMCd + EAWmdP + AWmcd + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
| ewmdP + |
eAWmcp + |
eawcdp + |
eawCdP + AWmcdP + EAWmdP + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
| ewmdP + |
eAWmcp + |
eawcdp + |
eawCdP + EAWmdP + eAmcdP + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
| ewmdP + |
eAWmcp + |
eawcdp + |
eawCdP + EAWmdP + eAWmcd + |
eaWmCP + |
eAWmdp + |
eaMCdp + |
EaWMCDP + EaWMcdP + eAWMCdP + |
eAwMCDP → capacity building |
The choice between the various options for the complex solutions for the zero outcome is not so important, as our main argument is that indeed there are various pathways to the zero outcome and that it is difficult to distil one path. However, the most prominent pathway – and the theoretically most logical pathway, ‘eawcdp’ – is present in all the solutions.
|
Email this article 
Hide
Show all
