For some years, non-governmental organisations (NGOs) and civil society have become increasingly involved in the fight against the HIV/AIDS pandemic in Africa. But even though their role is well appreciated, their actions are perceived as ineffective because of a lack of monitoring and evaluation capacity.
This paper aims to describe local HIV/AIDS NGOs’ involvement in evaluation and the characteristics of this involvement.
Descriptive analysis of data collected in questionnaires completed by 34 NGO executives (one per NGO).
Most NGOs do not have the minimal conditions required for positive and effective involvement in evaluations. In addition, funding agencies’ expectations for evaluations, total human resources as well as experience as NGO are contextual factors that explain most aspects of their involvement in evaluations.
This study provides funding agencies, NGO leaders and all those interested in developing evaluation capacity in these NGOs to understand the extent of the task in this area. They must keep in mind that there is no solution for all, but that solutions must be adapted to the developmental level of each organisation.
Local non-governmental organisations (NGOs) are playing an increasing role in the fight against the HIV pandemic. Despite recent advances (ONUSIDA
This paper aims to describe local HIV/AIDS NGOs’ involvement in evaluation and the characteristics of this involvement. Widespread concerns for the HIV pandemic, as well as an increased involvement of civil society in this fight, have raised much interest in these NGOs. In the following literature review, we begin with the local context of the fight against HIV/AIDS in Benin and the role of NGOs in this fight, then we move to a conceptualisation of evaluation as an organised system of actions before discussing the characteristics of evaluation activities within these structures.
The HIV-infection epidemic is at a generalised stage in Benin (prevalence equal to or greater than 1%). The prevalence of HIV for the overall population in Benin is stabilising around 1.2% since 2002. The estimated number of adults (15–49 years) with HIV was 65 472 in 2013 and 69 164 in 2015 (Comité National de Lutte contre le Sida
NGOs are involved in the HIV pandemic on three levels: communication activities to promote outreach and behavioural change (Mamadou & Tossou
Contandriopoulos
the process of making a value judgement about an intervention by implementing a control system that can provide scientifically valid and socially legitimate information about this intervention or any of its components to the different actors involved, so as to enable them, according to their area of expertise, to take a position on the intervention and make a judgement that can translate into action. [
Not only is evaluation a major aid in decisions to undertake, pursue, amend or analyse a public health action, but it is also an appropriate means to participate in developing a collective health regulation system needed to respond to social situations. This is why White (
According to Richter
For Champagne
the inclusion of local people in the development process reflects an acknowledgement of the need to tap into the wealth of wisdom and experience of the recipients of development aid, and to work with them to move the development of their communities forward.
Similarly, the evaluator’s position with regard to decision-makers can change over time, depending on how advanced the organisation is in the process of establishing an evaluation capacity and culture. The production function provides a description of the technical process of evaluation. It is influenced by two dimensions, according to Champagne
The values maintenance function is the basis upon which evaluation is organised in NGOs. According to Champagne
by values maintenance, we mean the fact that values are synthesised within a paradigm, that they are internalised by the actors and institutionalised, within the framework of an action system, which guarantees the system a certain cohesion. [
The paradigm here is that of an evaluation culture defined by Owen and McDonald (1999), quoted by Owen (
Gilliam
The theoretical framework for this study is presented in
Theoretical model showing relationships between components of evaluation involvement and local context of action.
This framework considers evaluation to be an organised action system with four dimensions organised around two components: the evaluation involvement level and the style of involvement in evaluation. The evaluation involvement level of an NGO is defined by the quantity and quality of human and financial resources made available for evaluation (adaptation), by the existence of a defined role or function assigned evaluation, by learning opportunities and a commitment to use the evaluation results (maintenance of values and meaning) and by the organisation’s annual rhythm of evaluations (production). The style of involvement in evaluation is defined by the positions that the staff assigned to evaluation hold in the organisation’s decision-making structure (adaptation), by the extent of stakeholder participation in evaluations, by the paradigmatic position of the head of evaluation within the NGO (values), by the participation level of NGO staff in evaluation and by the responsibility this staff can assume for the technical aspects of the evaluation (production). These two components interact to influence the NGO’s capacity to attain its goal and to continuously improve its intervention effectiveness. In addition, this organised system is included in a local context described by five components: geographic area, human resources available, type and expectations of funding agencies, main source of funding, and NGO experience.
We aim to describe, using the dimensions of our concept ‘involvement in evaluation’, the population of local NGOs in HIV/AIDS.
The population studied is made up of local NGOs in Benin which are private not-for-profit associative institutions recognised by national authorities. This recognition can be either by permission of the Ministry of the Interior, like all other organisations in civil society, or by a partnership contract with the Ministry of Health.
The inclusion criteria were: (1) to have been recognised for a minimum of 3 years by the national authorities; and (2) to have worked either in counselling and screening of HIV/AIDS, or in medical or psychosocial care for people living with HIV/AIDS, for at least 3 years. The sample population was composed of 161 NGOs assumed by the Ministry of Health to be working actively in this field. From this population, 110 randomly selected NGOs (number by department was proportional to the number of NGOs in each department in the directory) were contacted either at their head office or by phone. Many could not be reached because of imprecise addresses, or were reached but could not participate because they had not worked in the field for enough years. Only 62 NGOs contacted and able to participate in the study were enlisted (38.51% of the original population). Out of this group, 34 NGOs consented to participate in the study, for a recruitment success rate of 54.84%.
Comparison of three groups of NGOs in Benin directory of HIV/AIDS NGOs.
Population characteristic | NGOs that participated ( |
NGOs that refused ( |
Other NGOs in a directory ( |
Comparison test for the three groups |
---|---|---|---|---|
• Atacora Donga | 5 | 6 | 18 | |
• Alibori Borgou | 8 | 6 | 10 | |
• Zou Collines | 4 | 4 | 9 | |
• Mono Couffo | 5 | 3 | 12 | |
• Ouémé Plateau | 6 | 3 | 19 | |
• Atlantique Littoral | 6 | 6 | 31 | |
• Rural | 19 | 13 | 47 | |
• Urban or semi-urban | 15 | 15 | 52 | |
• Prevention screening | 31 | 27 | 95 | |
• HIV/AIDS care | 3 | 1 | 4 | |
• Mean | 11.79 | 11.14 | 7.73 | F(2,148) = 9.970 |
• Standard deviation | 4.45 | 6.84 | 4.75 | |
• Minimum | 4 | 1 | 0 | |
• Maximum | 21 | 25 | 35 | |
• Q1 | 9 | 6.25 | 5 | |
• Average | 12 | 8.5 | 7 | |
• Q3 | 15 | 17 | 9 |
The study has been built around three sets of variables.
Involvement is defined by a combination of six categorical variables on an ordinal scale (values from 0 to 2):
existence in the budget of specific financial resources for evaluation (no resources; insufficient resources; sufficient resources)
existence and quality of human resources available for evaluation (no human resources; person or team with only field experience; team with graduate training)
annual frequency of learning opportunities and discussion of evaluation results in the NGO (non-existent; 1 to 3; 4 and more)
annual frequency of new monthly data being incorporated into evaluation data in the NGO (4 or less; 5 to 8; 9 to 12)
existence of an evaluation structure in the NGO’s organisation chart and its association with the conception of new programmes (non-existent; existing but without contribution; existing and operational)
level of NGO staff’s participation in evaluations (low; medium; high).
This is defined by five nominal variables:
paradigmatic position of evaluation leader (positivist; constructivist; post-positivist)
extent of stakeholder participation (selective; medium; large)
evaluation staff’s position in organisational decisions (hierarchical; consultative; decision-maker)
staff responsibility in technical aspects of evaluations (none; shared; total)
type of result transfer (open; narrow).
There are five variables in this category:
principal source of funding: discreet variable with three modalities [local funding (self-funding, public or religious funding); foreign-state funding; foreign partners]
expectations of funding agencies regarding evaluations (yes; no)
geographic location of NGO (rural; semi-urban; urban)
NGO staff numbers (in full-time equivalents)
experience of NGO (in years).
Two methods of data collection were used:
document consultations: partnership contracts between the funding structure and the NGO (self-evaluation requirement from partner), organisation charts or job description cards (evaluation role or function description), periodic reports with financial information (specific resources for evaluation in the budget)
questionnaires completed by NGO agents in charge of monitoring and evaluating programmes, with closed questions administered by the principal investigator during an interview.
These data collection tools were pre-tested in six NGOs with characteristics similar to those of the study population. The pre-test enabled the researchers to reformulate or divide some questions in order to make them more understandable for respondents.
Data collected were subjected to descriptive analysis with univariate frequencies for categorical variables, and central and dispersion parameters for continuous variables. Then bivariate analyses (chi-square test, relative risk, comparison of means by t-test or ANOVA, depending on the case) were carried out between, on the one hand, items of level and style of involvement in evaluations and, on the other, contextual variables. These analyses were performed using SPSS 17.0 software. The cut-off threshold for conclusions was
NGO distribution by items related to level of involvement in evaluation.
Variables and modalities | % ( |
|
---|---|---|
• No resources | 15 | 44.1 |
• Insufficient resources | 14 | 41.2 |
• Sufficient resources | 5 | 14.7 |
• No human resources | 3 | 8.8 |
• Person or team with field experience | 27 | 79.4 |
• Person or team with graduate training | 4 | 11.8 |
• Non-existent | 0 | 0 |
• 1 to 3 years | 12 | 35.3 |
• 4 years and more | 22 | 64.7 |
• 4 times or less | 20 | 58.8 |
• 5 to 8 | 0 | 0 |
• 9 to 12 | 14 | 41.2 |
• Non-existent | 10 | 29.4 |
• Existing without contribution | 17 | 50 |
• Existing and functioning | 7 | 20.6 |
• Low | 24 | 70.6 |
• Medium | 5 | 14.7 |
• High | 5 | 14.7 |
Level of involvement in evaluation | Mean = 5.56; SD = 2.12 | |
Minimum = 2; Maximum = 11 |
The composite index for the level of involvement in evaluation has a near normal distribution, with both the median (5) and the statistical mode (5) near the mean (5.56).
NGO distribution by items related to style of involvement in evaluation.
Variables and modalities | % ( |
|
---|---|---|
• Positivist | 2 | 5.9 |
• Constructivist | 14 | 41.2 |
• Post-positivist | 18 | 52.9 |
• Selective | 18 | 52.9 |
• Medium | 11 | 32.4 |
• Large | 5 | 14.7 |
• Hierarchical | 5 | 14.7 |
• Consultative | 3 | 8.8 |
• Decision-maker | 26 | 76.5 |
• None | 25 | 73.5 |
• Shared with external evaluator | 3 | 8.8 |
• Total | 6 | 17.6 |
• Open | 18 | 52.9 |
• Narrow | 16 | 47.1 |
More than half of the NGOs (18, or 52.9%) operate in rural areas and have local funding (public or self-funding). For exactly half of them, having a contract with a funding agency implies being involved in evaluations. The total number of personnel in NGOs (full-time) ranges from 2 to 85 with a mean of 18.03 and a standard deviation of 17 years. These NGOs have experience ranging from 4.21 to 21.16 years (mean = 12.4, SD = 4.14).
Relations between items and composite index of level of involvement in evaluations and contextual variables.
Variables | Specific financial resources for evaluation | Human resources for evaluation | Learning opportunities | New data frequency | Evaluation structure | Staff participation intensity | Composite index |
---|---|---|---|---|---|---|---|
Primary source of funding | F2/31 = 0.109 | ||||||
Ex |
RR = 1 | ||||||
95%IC = [0.245–4.083] | |||||||
Intervention area | RR = 0.714 | RR = 0.971 | |||||
95%IC = [0.173–2.954] | 95%IC = [0.125-1.999] | ||||||
Total human resources | F2/31 = 0.225 | F2/31 = 1.176 | F2/31 = 0,541 | ||||
Experience | F2/31 = 0.238 | F2/31 = 3.106 | F2/31 = 1.391 | F2/31 = 1.634 | |||
The bold values show the statistically significant link between the variables of level involvement in evaluation and contextual variables.
The composite index of the level of involvement in evaluations is statistically significantly influenced by funding agencies’ expectations regarding evaluations in NGOs (average level of involvement in evaluations for these NGOs is 6.65, versus 4.47 for NGOs for which there are no expectations of evaluation from funding agencies;
Relations between items related to style of involvement in evaluation and contextual variables.
Variables | Paradigmatic position of leader of evaluations | Extent of participation of stakeholders | Evaluation staff’s position in decisions | Evaluation staff’s responsibility for technical aspects | Type of results transfer |
---|---|---|---|---|---|
Primary source of funding | |||||
Expectations from funding agencies for evaluations | RR = 0.244 | RR = 1 | |||
95%IC = [0.041–1.449] | 95%IC = [0.260–3.845] | ||||
Intervention area | RR = 1.167 | RR = 0.778 | |||
95%IC = [0.239–5.698] | 95%IC = [0.201–3.008] | ||||
Total human resources | F2/31 = 0.321 | F2/31 = 0.091 | F2/31 = 0.348 | ||
Experience | F2/31 = 0.774 | F2/31 = 0.628 | F2/31 = 0.003 | F2/31 = 2.183 | |
The bold values show the statistically significant link between the variables of style involvement in evaluation and contextual variables.
This study demonstrates the benefits to NGOs in using programme evaluation as a tool for continuous improvement of their intervention quality. That improvement can lead to better care for their clients. The possible harm for the NGO as result of their participation can be the revival of concealed attribution conflicts between the powerful and the powerless in evaluation and accountability for the structure, as the latter do not have the means to implement their vision. For the clients, the possible harm can be linked to recollection of negative experiences with a given NGO. The main researcher was available to discuss all these problems with the people concerned and, together with them, to find suitable solutions.
Information notices and consent forms were provided to all subjects before their recruitment. The subjects’ consent was informed and free, the participation was voluntary and participants could withdraw consent at any stage of participation. Given the sensitivity of the data and records used in this study, a data confidentiality agreement was completed and signed by the main investigator and all researcher agents in the study.
Data will be preserved for 7 years in the personal computer of the main investigator in his personal account accessible uniquely by digital footprint and in a file accessible by password.
This work has obtained permission from Ethics Committee of die Faculty of Medicine at the University of Montreal (Comité d’Éthique de la Faculté de Médecine de l’Université de Montréal, CERFM) [Certificate number: CERFM 201011 (112) 4#422] and from the National Temporary Ethic Committee for Health Research in Benin (Comité National Provisoire d’Éthique de la Recherche en Santé au Bénin, CNPERS) [Notice Number 013 of 13 October 2010] before starting the research.
The data collection and quality assurance procedures enable the researchers to have confidence in the study’s reliability.
An important aspect of this study is the treble triangulation of sources, data types and analysis methods, ensuring exhaustivity and increasing its methodological rigour and its internal validity. All stakeholders’ points of view have been considered (managers and staff of NGO, public or private financing agencies, clients). Many data sources (activities report, contracts between NGO and financing agencies) have been examined in order to compare declarations with written evidence. Both quantitative and qualitative data collection and analysis methods were used in this study. Because of our time and resources difficulties, we were limited in NGO recruitment; this limits the external validity of the study even if we point out many contextual factors (like staff numbers and the funding agencies’ expectations) which can explain the generalisability of the results.
NGOs’ involvement in evaluations supposes a minimum number of conditions: having a well-educated and trained person (or team) in evaluation, having specifically designated financial resources in the organisation’s budget for M&E, and having a functional structure defined in an organisation chart. The study’s conclusion is simple: most NGOs have no human resources or have personnel with only field experience in evaluations (88.2%), most have no specific financial resources or have insufficient resources for evaluations (85%), and finally, nearly 80% of these NGOs either have no functional evaluation structure in their organisation chart or have either a structure with no defined role or a role without a structure. Despite these critical conditions not being met, it is very interesting to note that these NGOs are involved in at least minimal evaluation activities. This is why nearly half of these NGOs (41.2%) have new data which can be used as evaluation data nine months per year, nearly one-third (29.4%) have staff that participate fairly or intensively at least four times a year in learning opportunities and in discussion of evaluation results. These are important opportunities for sharing experiences and especially for putting evaluation results into practice. This indicates that there is an awareness of the importance of evaluations for the continuous improvement of interventions. Regarding the influence of context, we note that local funding is associated with the existence of a functional structure; this supports the idea that the NGOs in this study are conscious of the importance of evaluations for their activities. However, funding agencies’ expectations explain the availability of financial resources and the production of evaluations, whilst NGO staff numbers explain not only the intensity of their participation in evaluations but also the responsibility assumed for the technical aspects of evaluation and the multiple opportunities for learning and discussion around evaluation results, all of which are essential for ensuring the use of evaluation results and for improving interventions. We are, therefore, at least partially in agreement with Lomeña-Gelis (
Gibbs
Assuming that NGOs with low participation in evaluations and no responsibility for the technical aspects are at the acceptance stage, we note that this corresponds to 70.6% (24/34) of NGOs in our study sample; thus, 8.8% (3 NGOs) are in the investment stage and 14.6% (6) would then be at the advancement stage. However, if we add for the last two stages of evaluation the requirements of institutionalisation by definition of role or function and by specific budget allocation for evaluation, only two NGOs remain in the investment stage (5.8%) and only one in the advancement stage (2.9%). In Gibbs’ sampling (
Carman and Fredericks (
This study shows that NGOs operating in the same context can be at different stages of development, although most of the NGOs in our sample in the HIV/AIDS sector in Benin were in the beginners stage. Unfortunately, there is no single solution for strengthening their capacities that would fit them all (Carman & Fredericks
This study faced the usual time and resource constraints. Because of these, we were unable to enlist a sufficiently large number of NGOs in our sample to be able to detect any independent effect of involvement in evaluations on effectiveness. The main problem was inadequate operationalisation of the concept of evaluation. Indeed, in attempting to group into 10 categories the 46 types of utilisation of evaluation processes found by Courtney and Bradley Cousins (
This study shows that although there is strong awareness that the involvement of HIV/AIDS NGOs in evaluations is important for the improvement of interventions in Benin, this process is in its early stages. Most NGOs do not even meet the minimal conditions required for positive and effective involvement in evaluation. In addition, both funding agencies’ expectations and staff numbers are contextual factors that explain most dimensions of involvement in evaluations. This study can provide funding agencies, NGO leaders and all those interested in developing evaluation capacity in these NGOs to understand the scope of the task at hand. They must keep in mind that there is no solution for all, but that solutions must be adapted to the developmental level of each organisation. Future research could establish whether this portrait of NGOs’ involvement in evaluations in Benin will help make their interventions in the field more effective.
The authors acknowledge the Canadian Fellowship Programme for French-speakers (Programme canadien de Bourse de la Francophonie) for granting a PhD fellowship to the principal researcher and for sponsoring field data collection. We also thank all the NGO leaders in Benin who agreed to participate in this study and Donna Riley for its linguistic review.
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
M.T.A., as PhD candidate, was the project leader. Under supervision of his research director and co-director, he designed and wrote the study protocol, designed the data collection tools, obtained ethics authorisation and monitored the entire process, planned and implemented the research budget, conducted all interviews with NGO directors, supervised all patient satisfaction surveys, managed all electronical data records, did all processing and analyses and finally wrote the manuscript. F.C., as PhD student supervisor and director of the research, supervised the process of thesis elaboration. N.L., as PhD student supervisor and co-director of the research, also supervised the process of thesis elaboration.