This article describes the development of the national evaluation system in South Africa, which has been implemented since 2012, led by the Department of Planning, Monitoring and Evaluation (DPME, previously the Department of Performance Monitoring and Evaluation) in the Presidency. It suggests emerging results but an evaluation of the evaluation being carried out in 2015 will address this formally. Responding to dissatisfaction with government services, in 2009 the government placed a major emphasis on monitoring and evaluation (M& E). A ministry and department were created, initially focusing on monitoring but in 2011 developing a national evaluation policy framework, which has been rolled out from 2012. The system has focused on improving performance, as well as improved accountability. Evaluations are proposed by national government departments and selected for a national evaluation plan. The relevant department implements the evaluations with the DPME and findings go to Cabinet and are made public. So far 39 evaluations have been completed or are underway, covering around R50 billion (approximately $5 billion) of government expenditure over a three-year expenditure framework. There is evidence that the first evaluations to be completed are having significant influence on the programmes concerned. The big challenge facing South Africa is to increase capacity of service providers and government staff so as to be able to have more and better quality evaluations taking place outside of as well as through the DPME.
This article documents the development of South Africa's national evaluation system (NES) from late 2011 to mid 2014, a period which involved establishing many basic systems and in which 39 evaluations were started and 11 completed. The article discusses how the system developed, the approach used, some of the key systems, initial outcomes in terms of improved programmes and reflects on the lessons at this stage. It builds on a framework for institutionalising evaluation systems by Goldman and Mathe (
Major investment by government in monitoring and evaluation (M& E) started in the 2000s. In 2007, the Presidency issued the policy framework on the government-wide M& E system, which linked performance information, official statistics and evaluations and coordination of various role-players at the administrative centre of government to champion M& E practices in government. At that stage there was no formal evaluation system, although there were emerging pockets of practice in some few sectors such as the Department of Social Development (DSD), Public Service Commission (PSC), as well as donor-driven evaluations.
Leading up to elections in 2009 there had been significant changes in South Africa's economy and society. The economy emerged from a long period of stagnation, achieving growth rates of over 5% by the mid 2000s. Access to potable water rose from around 62% of households in 1994 to over 90% of households. By 2008 and 2009 over 13 million people were receiving social grants of some sort. However, inequality remained a major problem and the Gini coefficient (based on expenditure) was stubbornly high, falling slightly from 0.64 in 2000 but still 0.63 (DPME
DPME's initial efforts focused on planning and monitoring the 12 priority outcomes, a management performance assessment system and monitoring front-line service delivery (Phillips et al.
To ensure the credibility of evaluation, one needs to show the independence and quality of evaluation.
The need for different types of evaluations, standardised systems to overcome limited capacity and an annual or rolling multi-year evaluation plan.
A budget allocation for evaluation in the range of 2% – 5% of programme budgets.
A central capacity is needed to support evaluations in government, both developing policy, systems and supporting methodology and quality assurance.
Improvement plans should be developed based on the evaluations and their implementation closely monitored.
In August 2011 the group that went on the study tour met in a writeshop
Goldman and Mathe (
Analytical framework for looking at institutionalisation of monitoring and evaluation systems.
Enabling conditions | |
Key role of a powerful and capable central ‘champion’ with sustained political will for the long haul and a coalition to support. Utilisation seen as the measure of ‘success’. Substantive government demand. The importance of establishing incentives (including the ability to use hard and soft authority effectively to enforce change). Performance management and M& E system that promotes interaction and variety and is dynamic. |
|
The process | |
A clear diagnosis of the existing situation and an understanding of where delivery must improve. The reform strategy and plan defined before the structure, so a clear policy direction with a commitment to results. The process should not rely on legislation and regulations to be implemented. A clear and effective implementation strategy. A talented team to drive the system and solve problems early and rigorously. The courage to rethink processes completely. Experimentation, piloting and scaling up. A major investment in communication. Care not to over-engineer the system. Establishing the culture and capacity to analyse, learn and use M& E evidence. Role of structural arrangements to ensure M& E objectivity and quality and reliable ministry data systems. |
M& E, monitoring and evaluation.
The national evaluation policy framework (NEPF) describes evaluation as: The systematic collection and objective analysis of evidence on public policies, programmes, projects, functions and organisations to assess issues such as relevance, performance (effectiveness and efficiency), value for money, impact and sustainability and recommend ways forward. (DPME
The NEPF suggests the purposes of evaluation are (DPME
Improving performance (evaluation for learning).
Evaluation for improving accountability.
Evaluation for generating knowledge (for research) about what works and what does not.
Improving decision-making.
The M& E system in government when DPME was established was predominantly a compliance-based system, above all in terms of accountability to National Treasury and the Auditor General (see Chitepo & Umlaw in press; Phillips et al.
To start with enthusiasts are required who would be supportive and likely to implement the findings; hence, departments are invited to submit proposals for evaluations. Ownership of the evaluation is kept with the custodian department, although DPME is a full partner (and sometimes other departments).
With the limited capacity the focus had to be on important programmes or policies – either because they were large or strategic – and selection of policies or programmes to evaluate is on this basis. DPME also needed to work closely with departments in a learning-by-doing process and so the core support is from a DPME evaluation specialist who supports evaluation steering committees through the whole evaluation process, thus ensuring that standard systems are used.
DPME needed to build agreement across government around the approach so that it is not seen as imposed. Hence, an early decision was to create a cross-government Evaluation Technical Working Group (ETWG) as a sounding board and to be the advocate of the system. To start with the system was not made compulsory except for evaluations under the National Evaluation Plan (NEP); later, the system would become government-wide.
Evaluations have to be credible, enabling confidence in findings and recommendations. Except in the case of design evaluations, all evaluations are commissioned to external service providers to ensure independence and credibility. As expressed by Minister Radebe at his first meeting with DPME managers in September 2014, ‘we can’t expect departments to evaluate themselves’.
Evaluations go to Cabinet and are made public, so that results have to be faced and used.
The system does not stop with the evaluation itself but an improvement plan process follows completion of the evaluation, wherein the departments involved have to act on the recommendations.
At the same time as the NEPF was being developed the decision was taken to start some pilot evaluations. The initial concept for an outcomes evaluation and research unit to drive the evaluation system was approved in September 2011. The purpose was ‘to coordinate the evaluation function in government, ensuring that high quality evaluation and research underpins public policy and programming, so maximising the impact of government policy and services’ (Goldman
Developing and maintaining the policy framework for evaluation in government, as well as a three-year and annual evaluation plan.
Building a cross-government approach to take forward evaluation and to ensure that evaluations are used to inform plans and budgets.
Developing the technical specifications, systems and guidelines for evaluation in government.
Undertaking and supporting evaluations and research.
Oversight and quality control of the evaluation process across government.
The first pilot evaluation started in October 2011 and was completed in June 2012 (on Early Childhood Development, ECD). The 2012 and 2013 NEP was approved by Cabinet in June 2012, with 8 evaluations (DPME
Status of evaluations as of 31 July 2014.
Improvement plans being implemented | Served at Cabinet | Approved reports† | Evaluation underway | Terms of Reference approved | Preparation stage | Stuck |
---|---|---|---|---|---|---|
6 | 2 | 11 | 21 | 4 | 1 | 2 |
†, Approved by Evaluation Steering Committees.
The evaluations undertaken are implementation (26), impact (7), diagnostic (3), design (1) and economic (1) (DPME
The NEPF envisaged that after the NEP, provincial evaluation plans and departmental evaluation plans (DEPs) would be developed. In 2012 and 2013 DPME piloted provincial evaluation plans with Gauteng and the Western Cape Provincial Governments. DPME provided technical support to the provinces as well as made all guidelines, templates and training available to them. The Western Cape government provincial evaluation plan (WCG PEP) 2013/2014 – 2015/2016 was approved by the provincial Cabinet in March 2013 (WCG 2013). The Gauteng evaluation framework was approved by the Executive Council in 2012. During the 2013 and 2014 financial year, North-West, Free State, Limpopo, the Eastern Cape and Mpumalanga produced draft concept notes for PEPs (DPME
Although no formal process has started yet on promoting DEPs, an increasing number of national departments are planning or undertaking evaluations (now 15 out of 46 national departments) (DPME
Experience has shown that a top-down approach is often characterised by resistance and failure to utilise evidence from evaluations (e.g. see Mackay
As evaluations were not conducted widely in government, a number of support systems were established, including standards for the quality of evaluations, guidelines for how to conduct elements of the evaluation system, evaluation competencies, various capacity development elements, a quality assessment system and communication elements. All of the guidelines and tools are available on the DPME website.
Building a
This coalition has been used to build intergovernmental commitment to the system; as potential champions emerge in other spheres they are being included, for example involving the two first provinces in the ETWG. The explicit intent of the system was to start national, then develop provincial involvement and evaluations plans for each department. This is indeed what has happened with 15 of the 46 national departments now involved, five out of nine provinces and five departments developing DEPs. This staggered approach is to enable testing of the system, but also being realistic of the capacity within DPME to support the system. For this reason too the approach has been not to focus on local government at this stage, but to systematically start focusing on the big cities (the metro municipalities) in 2016 and 2017. As evaluations start emerging at provincial level, issues of how to iterate or align evaluations across spheres is starting to emerge.
In 2012 DPME developed a set of evaluation
One of the ways to ensure minimum standards and ensure that evaluations follow the NES is the use of guidelines. DPME has developed 18 practical guidelines and templates on various components of the evaluation process. These range from a guideline on developing evaluation terms of reference (DPME
A range of
Other aspects to improve quality are a
DPME has undertaken advocacy and promotion of evaluation as a discipline. This has involved publications and presentations in national and international conferences, to senior management of departments, to Cabinet and to Parliament. DPME has also signed a memorandum of understanding with the South African Monitoring and Evaluation Association (SAMEA) to promote M& E in South Africa. Areas of collaboration include capacity development and learning activities, dissemination of M& E, evaluation standards and competencies and professionalising evaluation. Beney
Effective
The change management approach adopted requires senior managers to see the point of doing evaluations. In order to
The proposed impact of the NES in the evaluation logframe is ‘improved performance and accountability of government programmes and policies as a result of evaluation’ and, at outcome level, ‘evaluation and research evidence [
As indicated earlier the evaluations undertaken are implementation (26), impact (7), diagnostic (3), design (1) and economic (1) (DPME
The first way to see progress in the use of evaluations is through implementation of improvement plans produced after each evaluation. As most evaluations only started in October 2012 it is early days to see concrete impacts of the evaluations (whether symbolic, conceptual or instrumental), but some initial results are indicated and an evaluation is planned for 2016 and 2017 to help understand their use. There is already evidence of significant follow-up on some of the first evaluations to be undertaken. On ECD (see Davids
Despite these encouraging results there is also evidence of delays by departments in producing and reporting on improvement plans (DPME
Beyond their use, the evaluations that have been completed to date (11) have all scored well in the independent quality assessment, averaging 3.7 out of 5 which is well above the minimum quality threshold of 3, so the quality is acceptable (Goldman
Apart from the effects of specific evaluations, a significant effect of the NES has been the elevation of the status of evaluation as an essential element of the work of government. The NES ‘forces’ the placement of evaluation on the agenda of government departments, responding to the request from DPME to government departments to submit proposals for evaluations in the NEP. Two departments that have been involved in the NES see the following benefits for government departments:
It provides a framework from which to develop departmental evaluation policies and guidelines.
It provides a mandate and an impetus to initiate evaluations.
It provides technical and budgetary support to facilitate the commissioning and management of evaluations.
It lends credibility to evaluation findings, by facilitating built-in quality assurance mechanisms.
It seeks to ensure that evaluation reports do not sit on shelves, but are used to improve interventions.
It draws attention to key policy issues by ensuring that evaluation reports are tabled in Cabinet.
It facilitates inter-departmental cooperation and coordination on issues that are cross-cutting.
Increasingly, departments such as DSD, DHET and DTI are developing and implementing DEPs (DPME
Another impact has been that as evaluations were undertaken the weakness of
The second issue is that
To address this DPME has developed with National Treasury a guideline on planning implementation programmes (DPME
This section summarises lessons about the system and the change management approach adopted, drawing from the analytical framework introduced earlier (
Lessons against the analytical framework.
Element | Importance | Degree to which it has been achieved in the national evaluation system or other cases |
---|---|---|
Key role of a powerful and capable central ‘champion’ with sustained political will for the long haul and a coalition to support. | Critical.† | DPME has played an institutional role in South Africa and similar champions in Mexico, Uganda, Benin and Colombia.† |
Utilisation seen as the measure of ‘success’. | Critical and the focus has been appreciated.† | Key focus of the system. Early indications are that utilisation is likely to happen but more evidence is required.‡ |
Substantive government demand. | The system has been designed as demand-led. This does seem to be critical.† | Formalised in the NEP. Requests coming from new sectors but some sectors avoiding. DPME responded by proposing in areas where requests were not forthcoming. Some enthusiasts emerging. Also promoting collaborative approach so departments feel they are co-owners of the system.† |
The importance of establishing incentives (including the ability to use hard and soft authority effectively to enforce change). | Incentives are critical and both soft (e.g. part-funding by DPME) and |
At the moment concentrated on soft authority to create a pull not push environment. Evidence that does increase likelihood of uptake of findings.† |
Performance management and M& E system that promotes interaction and variety and is dynamic. | The effort to make a system that is seen to be widely owned, that departments can influence and that develops seems to be contributing |
Interaction is being promoted between programme managers and M& E staff. |
A clear diagnosis of the existing situation and an understanding |
This aspect does not seem so essential if the right group of stakeholders with deep knowledge of the system comes together to design.‡ | This has happened based on a group of departments who are active in evaluations. DPME has subsequently undertaken other diagnostic exercises around M& E.† |
The reform strategy and plan defined before the structure, so a clear policy direction with a commitment to results. | It would appear to be critical to have a clear intent and approach before setting up structures.† | South Africa developed the evaluation policy before starting to implement the system which has given a strong direction. This is not true of other countries.† |
The process should not rely on legislation and regulations to be implemented. | Legislation does not appear to be necessary before starting processes. Not being legislated also allows for systems to be developed and tested before freezing them in laws and regulations.† | Appears to be valid. Not present in SA; the system has been established based on executive decisions and has survived a new administration. May benefit more going forward. Other countries such as Mexico and Colombia have more legalistic systems where legislation may |
A clear and effective implementation strategy. | Critical.† | This is present.† |
A talented team to drive the system and solve problems early |
Critical.† | As above. Also true in Mexico, Uganda, Benin and Colombia.† |
The courage to rethink processes completely. | New systems may be needed. Where there are none this is easier |
In evaluation new systems were created as there were no national evaluation systems. This has helped in trying to change the predominant compliance culture.† |
Experimentation, piloting and scaling up. | Essential.† | Visible in SA, also Uganda and Benin. Some departments emerging as early adopters. DPME working in learning-by-doing process with departments. Flexible support from PSPPD critical in getting system started as well as learning from other countries.† |
A major investment in communication. | Essential.† | Significant efforts within government but not yet to the public and wider stakeholders.‡ |
Care not to over-engineer the system. | Not clear from the examples.‡ | There has been an awareness of evaluation systems being ‘good enough’ rather than theoretically excellent but too difficult to implement. However, this being government, systems are complex.‡ |
Establishing the culture and capacity to analyse, learn and use |
Critical.† | Major efforts made at supply and demand side but major capacity deficits in SA. Long way to go. Improvement plan is the key system to ensure learnings implemented – not yet clear if the improvement plan system is working.‡ |
Role of structural arrangements to ensure M& E objectivity and quality and reliable departmental data systems. | Critical. The belief that M& E systems are objective and valid is critical |
Major efforts have been made to ensure the rigour of the findings. There is less success around administrative data which is problematic. Also issues around capacity of service providers.‡ |
†, indicates likelihood of success, ‡, where it is not sure.
M& E, monitoring and evaluation; DPME, Department of Planning, Monitoring and Evaluation; NEP, national evaluation policy; SA, South Africa; PSPPD, Programme to Support Pro-poor Policy Development; DG, directors general.
There are a number of areas where problems have emerged around the planning, budget and M& E system, and where the evaluation system and its linkages could be strengthened. These are shown in
Key challenges and how these are being addressed.
Challenge | How this is being addressed | Further action needed |
---|---|---|
Poor programme planning | Development of guideline on planning new implementation programmes approved by Cabinet, as well as design evaluations. | Do audit of implementation programmes in government. |
Not getting evaluations from some sectors | Raising gaps with Cabinet and proposing possible evaluations for them to select from. |
Cabinet to consider priorities they would like to be evaluated. |
Discuss what cross-cutting evaluations are key for local government. | ||
Some departments taking a very long time to procure | DPME to procure wherever possible. | Evaluations where departments procure not prioritised in the NEP but rather included in departmental evaluation plans. |
Inadequate supply of strong evaluators | Advocacy work at universities to encourage them to participate. | Develop course to assist researchers to understand evaluation. |
Capacity building work with service providers. | Developing training courses and briefings in 2014–2015. Undertake rating system of service providers and publicise the results. | |
Diagnostic on the supply of qualified evaluators. | Fundraising for this. | |
New call for evaluation panel in August 2014 to expand the group to draw from. | In process. | |
Inadequate data for some evaluations to be viable | Developing model for evaluability assessment and apply in |
Work to improve administrative data quality and also programme data collection. |
Encourage all first evaluations to be implementation evaluations, only after which do we consider an impact evaluation. | Departments to plan impact evaluations at programme inception. | |
Departments taking too long to take forward evaluation results, including improvement plans | Standard now being applied that DPME takes the evaluation to Cabinet. DPME is having to keep reminding departments about completing the improvement plans, and progress reports. | Cabinet to note the problem. Include this in Auditor General monitoring and Management Performance Assessment standards. |
Improve communication of evaluation findings | Testing out with next evaluations including policy briefs, seminars and development of a communication strategy. | See how this works and additional inputs needed. |
Departments slow to produce improvement plan progress reports | Repeated requests and highlighting the problem. | See whether the Auditor General can audit reporting on improvement plans. Also include in Management Performance Assessment standards. |
Additional capacity needed to support provincial and departmental evaluations | Supported two provincial evaluation plans in Western Cape and Gauteng to test the system. Now working with five other provinces. | Strengthen imperative to take forward. In 2016 and 2017 major focus on DEPs. |
DPME, Department of Planning, Monitoring and Evaluation; DEP, departmental evaluation plans; NEP, national evaluation policy.
As stakeholders become familiar with the system there is evidence that they wish to
At present DPME's capacity to support DEPs is limited, but this is intended to be an important component from 2016 to 2017, so that evaluation is internalised across government. To do this a major drive is needed to build capacity both in service providers and government. This will require a close collaboration between training providers such as universities, SAMEA and DPME.
South Africa is one of a few countries that have developed an NES – Mexico, Colombia, Canada, Chile, Uganda and Benin are the notable examples. In these countries evaluation is a means of understanding in depth what is working and not working in government programmes and policies and to see how to strengthen them. In some cases (notably Chile) there is a strong budget linkage (Mackay
In order to develop a credible system a fairly complex system has been developed in South Africa. The system is working: overall satisfactory evaluations are emerging and first evidence is that these are having a significant impact on programme design and implementation.
The South African model has drawn considerably from Mexico and Colombia in particular. Some of the particular characteristics of the South African model emerging are:
The NEPF gave clear direction to the system from the beginning (Diego Dorado, World Bank, personal communication).
The system has developed very quickly, essentially in two years. This is partly as it was able to build systematically from previous experience and has maintained strong links to other exponents.
It is led from the Presidency – so is very central (rather like Uganda and unlike Mexico, whilst Colombia has strong links with the Presidency).
It is explicitly working at the supply and demand sides of evaluation, with the latter working with DGs and parliamentarians to stimulate demand for evaluations and other forms of evidence (others are to some extent too).
It has a demand-driven approach, stimulating departments to ask for evaluations and to encourage them to use evaluation results.
It recognises the need for evaluations at different stages of the project cycle, from diagnostic to impact (like Mexico and Colombia, unlike Benin and Uganda).
It is working with national departments and provinces.
DPME has been able to generate very significant amounts of funding for performance M& E, including for the NES, and to part-fund evaluations (Kathrin Plangemann, World Bank, personal communication). DPME has an annual budget of about R250 million (around $25 million) and about 200 staff, of which R23 million (around $2 million) and 15 staff are allocated to the Evaluation and Research Unit.
It is beginning to bring together evaluation and research as contributors to evidence (Aristide Djidjoho, Benin, personal communication).
DPME are estimating that there should be a minimum 10% improvement in programmes being evaluated. This is equivalent to around R5000 million of government expenditure, for an annual expenditure on evaluation of approximately R50 million or R150 million over three years. This suggests a minimum rate of return of over 30:1. Financially this suggests a strong argument that evaluation is a good investment to ensure programme effectiveness, efficiency and sustainability.
A challenge will be upscaling so this can happen across government. Over the next 10 years major capacity development is needed both for service providers and also in government. Parallel work will be needed on programme planning and also on budgeting. Together with expenditure reviews these promise to provide major opportunities to improve government performance and accountability.
The Programme to Support Pro-Poor Policy Development (PSPPD), a partnership between the Presidency and the EU, funded the special edition of which this article is a part. The work on evaluation has received tremendous support from the Minister in the Presidency until 2014 (Collins Chabane MP, sadly now passed away), the then director general of the DPME (Sean Phillips) and the deputy DG (Nolwazi Gasa).
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
I.G. (Department of Planning, Monitoring and Evaluation, DPME) led the evaluation process from inception, as well as the development of this article. J.M. (DPME) and C.J. (DPME) joined the evaluation and research unit in DPME as evaluation directors at an early stage (2011 and 2012), with M.A. (DPME), A.H. (DPME) and M.E. (DPME) joining in 2013. H.N. (Department of Higher Education and Training), T.B. (Department of Social Development), S.N. (DPME) and M.S. (Programme to Support Pro-Poor Policy Development) were part of the team that undertook the initial study tour and wrote up the initial policy framework. All have contributed sections to the article, with I.G. undertaking overall editing.
A partnership between the presidency and the European Union focusing on evidence-based policymaking.
A workshop during which the outline content was developed together; then different authors wrote up different sections, all within the workshop environment.
These were provided by Thabani Buthelezi and Hersheela Narsee, heads of evaluation in their respective departments of DSD and DHET respectively, and co-authors of this article.
Note that this list was developed from a list circulated to international colleagues asking for their views on areas where South Africa's system was making a contribution; some endorsed suggestion by the authors and suggestions by international colleagues (who are named).