Book Title: Evaluation management in South Africa and Africa

Editors: Fanie Cloete, Babette Rabie and Christo de Coning

ISBN: 978-1-920689-50-6

Publisher: African Sun Media

Review Title: Review of ‘Evaluation management in South Africa and Africa’

Reviewer: Patricia J. Rogers¹

Affiliation:
¹Better Evaluation/ RMIT University, Australia

Email: patricia.rogers@rmit.edu.au

Postal address: RMIT University, 124 Latrobe Street, Melbourne, VIC 3000, Australia

How to cite this book review: Rogers, P.J., 2015, ‘Review of ”Evaluation management in South Africa and Africa”’, African Evaluation Journal 3(1), Art. #172, 4 pages. http://dx.doi.org/10.4102/aej.v3i1.172

Copyright Notice: © 2014. The Authors. Licensee: AOSIS OpenJournals. This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Over my more than 25 years’ experience conducting evaluations, it has become painfully clear that without effective management of evaluations, good evaluation is not possible. No matter how skilled and experienced the evaluator, if the manager of the evaluation does not effectively plan it, work in partnership with the evaluator during the conduct of the evaluation, and actively work to support the use of the findings afterwards, then the evaluation cannot succeed.

It was therefore a delight to read this new contribution to the evaluation literature, which focuses on evaluation management in an African, and particularly South African, context. The editors, Fanie Cloete, Babette Rabie and Christo de Coning, have brought together an enormous collection of thinking, as well as a discussion of practice in this important area.

Some indication of the scale of this work can be found in the dimensions of the undertaking − 12 chapter contributors and 48 case contributors producing a book that is an astonishing 661 pages. Fortunately, it is readily available as an e-book, as well as in print-on-demand hard copy.

The actual chapters are broader in scope than the title would suggest. The book covers much more than the management of evaluation, with extensive chapters on evaluation models and approaches, theories of change, and programme logic that seem more like general textbook chapters than discussions of evaluation management or guidance for evaluation managers. Chapter 11 draws strategic lessons from the preceding chapters, which would have been useful to present at the beginning of the book to help readers make sense of the huge volume of ideas, examples and frameworks presented.

The first section of the book discusses conceptual approaches to evaluation. Separate chapters discuss the context of evaluation management, the historical development and practice of evaluation, theories of change and programme logic, programme evaluation designs and methods, indicators for evidence-based measurement in evaluation, institutional arrangements for monitoring and evaluation, evaluation professionalisation, and capacity building. It includes a case study of management use of evaluation reports, and a case study of VOPEs (Voluntary Organisations for Professional Evaluation), evaluation associations and societies.

The chapter on the context of evaluation management by Babette Rabie and Ian Goldman provides an important framework for the book. After setting out the ideal, rational model of how monitoring and evaluation is intended to support planning and implementation of programs and projects, it then discusses the limitations of this simple model. It sets out common perverse effects of performance management M& E (monitoring and evaluation) systems − such as punishing good performance by setting even higher targets or reducing budgets, and acting in ways that achieve targets whilst reducing goal achievement (for example, persuading weaker students to drop out of school to keep exam pass rates high). It contrasts situations where performance measurement is easier (for example, where services are standardised and causal paths are well understood), as well as where it is harder. It discusses the M& E implications of Mintzberg’ s (1996) different organisational models − for example, the machine model where decisions are made from the top and monitoring is used to check staff compliance, as compared to the network model where professionals use monitoring to adapt their practice − and how these different models might be useful characterisations for particular types of programs. It sets out De Bruijn's (2007) five ‘laws’ of unintended negative consequences of M& E and what might be done to mitigate these − for example, the ‘Law of Mushrooming’, where M& E systems become increasingly bloated and less practical and useful. It also sets out challenges for evaluations, including unrealistic budgets and timelines, lack of ownership and inadequate data availability.

This discussion names important issues in monitoring and evaluation, and it would have been particularly helpful if these issues had been taken up in the subsequent chapters. To what extent have these ideas actually informed the planning and implementation of M& E systems in Africa? What strategies have actually been put in place to avoid the potential negative consequences? How effective have they been? What has been done to find out whether they have been effective? The later chapters do not explicitly refer to these ideas, which is a missed opportunity.

The remaining chapters in this section would have benefited from being more tightly focused on the issue of evaluation management, as well as from better coordination across the chapters.

Chapter 4 (‘Evaluation models, theories and paradigms’ by Babette Rabie) and chapter 5 (by Johann Mouton) both tackle the very difficult task of providing a conceptual framework for making sense of the cacophony of ways of thinking about and doing evaluation. Chapter 4, after discussing a range of classifications previously used, provides an overall classification scheme in three parts: the scope of the evaluation (an entire intervention or parts of it), the philosophy or theory (either theory-driven or participatory), and the evaluation design (quantitative, qualitative or mixed method). Chapter 5 discusses evaluation types in terms of purposes (improvement or judgement) and stage of the program cycle.

Having these two different classification systems in the book, without referencing each other, is likely to be confusing for readers. In both of them, some of the groupings are questionable. In chapter 4, presenting theory-driven and participatory as apparently mutually exclusive options is problematic as an evaluation can be both. It is hard to agree with the classification of goal-free evaluation as a type of theory-driven evaluation, given that its inventor (Michael Scriven) is notably sceptical of the value of theory-driven approaches, and goal-free evaluation does not include any attention to articulating the program theory − only to identifying actual impacts and comparing these to transparent and defensible evaluative criteria. Similarly, it does not seem appropriate to classify utilisation-focused evaluation as a type of participatory evaluation, given that its defining feature is to identify and engage primary intended users and uses, and to make every decision about the evaluation with reference to those − a process that does not necessarily involve active participation. In chapter 5, impact evaluation is only framed as being done for judgement purposes (summative) − without the option of using it for improvements in a later cycle − and only through the use of experimental and quasi-experimental designs.

Chapter 6 (‘Indicators for evidence-based measurement in evaluation’ by Babette Rabie) provides useful guidance on selecting and developing indicators, including setting out different types of indicators, providing a useful example of limitations of indicators (in this case, indicators of the perception of safety and security in a community), and providing a series of steps to develop relevant and credible indicators, as well as a checklist for reviewing indicators. The chapter also provides detailed examples of indicators and central government indicator initiatives.

Chapter 7 on institutional arrangements for monitoring and evaluation (by Christo de Coning and Babette Rabie) is particularly relevant to the focus of the book. It systematically discusses what is required and provides case examples to illustrate the points made. The structure of the chapter, however, is confusing, with two separate sections on institutional arrangements − each with a different list of components discussed in subsections.

Chapter 8 (‘Evaluation professionalisation and capacity building’ by Donna Podems and Fanie Cloete) provides a clear discussion of different standards for evaluation, competencies for evaluators and managers of evaluation. The chapter discusses the implications for evaluator competencies of the diverse range of evaluation types. Given the discussion of the AfrEA (African Evaluation Association) standards and the DPME (Department of Planning, Monitoring and Evaluation, South Africa) competencies for evaluators, it would have been useful to have the detailed documents available as an annex to make it easier for readers to access the detailed versions. The chapter ends by discussing and providing a link to a range of capacity-building strategies and resources − these are largely in terms of formal courses and online resources. It would have been useful if some discussion of other strategies for developing evaluation capacity − such as peer support, mentoring, coaching and learning circles − had been included.

The second section of the book discusses evaluation in the public sector in selected African countries. There is an extended discussion of the development of M& E systems in South Africa, including a discussion of challenges and future directions, and summary case studies of the state of formal M& E systems in Benin, Ghana, Kenya, Senegal and Uganda (reproduced from longer case study reports undertaken by the Regional Centers for Learning on Evaluation and Results (CLEAR). The cases show the importance of national, regional and international processes and structures in influencing the demand for M& E, including national constitutions and key policy agendas, as well as different approaches to develop and embed M& E systems. The descriptions of the various elements of the systems, and the processes that led to their development and influenced their performance, are useful. Common elements include strong management support and effective capacity development. The cases would have been even more useful if they were accompanied by an analysis across the cases in terms of the issues discussed in the earlier chapter regarding managing the risks of perverse outcomes in M& E systems, and the need to implement appropriate M& E systems for different types of programs.

The following chapters describe a number of African VOPEs: AfrEA, Cote d’Ivoire RISE, Egypt EREN, Kenya ESK, Morocco MEA, Niger ReNSE, and Senegal SenEval. VOPEs play a critically important role in developing evaluation capacity, creating a community of practice and engaging in advocacy for evaluation. Learning from these cases is therefore likely to contribute to sustained support for evaluators and evaluation managers. Common issues arise for evaluation association and societies - ebbs and flows of activity and success, depending on the demand for evaluation and the availability of committed volunteers and strategic partners to undertake the many activities required for an effective VOPE.

The third section presents case studies on particular issues in M& E: the use of ICT (information communications and technology) and software, an example of using a DfID (UK Department for International Development) program theory of change to underpin an evaluation of roads in East DRC, a World Bank completion report (ICR) on Dar es Salaam's water supply and sanitation program, a UNICEF review of the Ipelegeng program in Botswana, a UNDP report on contribution to development results in Angola, four cases from South Africa describing examples from national, provincial and local government and an NGO, and 2 additional case studies of evaluations. These cases could be useful for teaching purposes or for self-study, and could have been enhanced by the provision of discussion questions and explicit links to relevant chapters.

This section also has a chapter by Michael Bamberger, Jim Rugh and Fred Carden on alternatives to the conventional counterfactual (drawn from notes from a think tank session at the American Evaluation Association conference in 2009). Whilst this is an interesting list of ideas, it is not clear how it is intended to relate to the rest of the book, especially in terms of the earlier chapters that discussed attribution only in terms of conventional counterfactuals.

There is also a chapter on ‘Made in Africa’ evaluation (‘Uncovering African roots in evaluation theory and practice’ by Bagele Chilisa and Chiku Malunga), originally published in 2012, which sets out some intriguing and potentially influential ideas. It proposes two different elements to a ‘Made in Africa’ approach to evaluation and adds these to the ‘Evaluation Theory Tree’ described in the earlier chapters. The first addition draws on ‘the evolving postcolonial indigenous paradigm’. It would include five key elements:

• A critical analysis of the history of evaluation and evaluation outcomes of past related projects in a given context.
• A critique of past and related projects from the communities’ perspective.
• Development of community owned standards for evaluation of project, integration with project blueprint standards and integration of indigenous and imported evaluation standards.
• A combination of community indigenous methods and adapted Western methods, in order to collect evidence of merit and worth.
• Dissemination of evaluation outcomes approaches that are inclusive of community indigenous dissemination approaches. (p. 549)

The second addition is relational evaluation, which draws on African perceptions about the nature of being. This would include valuing and using community knowledge, valuing the community as knowers and evaluators, and ensuring that evaluators and agents of funders establish long-lasting relationships with communities.

Some further exploration of these ideas, both in theory and in practice, would have been enormously helpful − either in terms of examples of their implementation or learning from difficulties in implementing them in evaluations in Africa. Bringing this chapter, and some further development of it, into the initial section of the book would have increased the likelihood of it providing a framework for the later chapters.

Whilst there is much to commend in the book, there are also areas where its content, structure and format could have been improved. Some of the chapters are awkwardly structured or cover content that has already been addressed in previous chapters. For example, the evidence-based policy movement is discussed in sections 1.6.2 (‘Evidence-based policy-making and management’) and 2.6.2 (‘Evidence-based policy making’). It would have been better to have a single section on this, or for the different chapters to be better coordinated to remove redundancies, and to explicitly discuss any differences. Several chapters discuss definitions of evaluation and monitoring and logic models − but not in a way that references other discussions. The discussion of ‘Made in Africa’ evaluation in the history chapter does not reference the later chapter, which discusses this in more detail. The use of three-level numbering of the sections gives an indication of the awkward structure of some of the chapters and makes it harder for readers to keep in mind the logic of how the chapters are set out. Readability was also reduced by inconsistent formatting of bullet points − sometimes using dots and sometimes using lettered numbering, and in some cases indenting whole paragraphs rather than a list of points.

Some of the detail in the chapters, especially in terms of the history of evaluation in non-African countries, seems excessive, given the stated focus of the book. Whilst this might make the book more useful as a detailed textbook for evaluation coursework, it makes it less useful as an accessible resource for practising evaluators and evaluation managers.

I also had some problems with the e-book format on my desktop, which used the free Snapplify reader. The font did not display clearly, it was not possible to write notes on it and the navigation was awkward. It might work better on a tablet. The service provided was good, however, with a rapid response to fix problems with the access link.

Overall, this is a landmark publication, with much that will repay rereading. It provides an important documentation of current and recent evaluation practice and its evaluation management context, and provides useful information for evaluation managers.

De Bruijn, H., 2007, Managing performance in the public sector, Routledge, Oxford.

Mintzberg, H., 1996, ‘Managing government, governing management’, Harvard Business Review 74(3), 75–83.