Abstract
Background: This article was based on the study on the assessment of evaluation capacity in the transport sector in South Africa. The purpose of the study was to test the Six Sphere Framework (SSF), which is an innovative evaluation capacity diagnostic tool developed by the Centre for Learning on Evaluation and Results (CLEAR-AA) based in South Africa.
Objectives: The article presents the findings emanating from the study and new knowledge on how evaluation capacity assessments can be conducted in the transport sector in South Africa.
Method: A variety of methods including a survey, semi-structured interviews, document review and focus group discussions were used to collect data from primary and secondary sources. In all the cases, research questions were structured around the six components of the framework. Similarly, the presentation of the findings was arranged in themes that mirror these components.
Results: The article locates the SSF within the current evaluation capacity development literature and argues that existing evaluation capacity assessment tools are inadequate to understand pertinent issues affecting the use of evidence in the transport sector in South Africa.
Conclusion: In this regard, the framework is recommended as an innovative tool to assist evaluation practitioners and scholars to better understand evaluation capacity constraints within a broader context that involves logistical, technical, contextual, social and political dimensions. It also offers an important insight on how these components interfaced to shape the organisational value system that impacts the use of evidence in the transport sector in South Africa.
Background
The transport sector sector plays a critical role in the South African economy and has been consistently contributing about 9% to the gross domestic product (GDP) and just over 6% to the national labour force. In real terms, the sector’s revenue contribution has increased from R242 billion to R256 billion in 2012 and 2015, respectively. Much of this has been generated from the province of Gauteng, which is the economic hub of South Africa (Transport Education and Training Authority Skills Sector Plan [TETA SSP] 2016).
Despite its strategic importance to the South African economy, the sector – like many in the country – is faced with socio-economic inequalities inherited from the apartheid era. These are manifested through severe skills shortages among the black majority and women in particular, leading to racial- and gender-based economic exclusion not only in the transport sector but across the broader economic spectrum.
In response to these challenges, a democratic government which came to power in 1994 developed a national skills development programme through the Skills Development Act of 1998. The Act provides for the establishment of the Sector Education and Training Authorities (SETAs) whose mandate is to coordinate and implement the skills development programmes across all sectors of the economy. Transport Education and Training Authority (TETA) is an entity which performs these functions across the eight sub-sectors1 that constitute the transport industry in South Africa. In addition to having this huge responsibility, this entity – like many in South Africa – is faced with many challenges with respect to monitoring and evaluation (M&E). This is supported by research showing lack of M&E capacity as one of the main challenges facing the government as a whole leading to ineffective policy responses in the country (Paine Cronin & Sadan 2015; Wotela & Sidzumo 2016).
In an attempt to support governments for improved use of evidence for policymaking, the World Bank set up global Centres for Learning on Evaluation and Results (CLEAR) in 2011. The centre serving the Anglophone Africa is located in South Africa and strives to develop and advance evaluation capacity development for promotion of evidence-based policy-making in the region. In doing this, CLEAR Anglophone Africa (CLEAR-AA) works with governments at different levels, civil society organisations (CSOs) and public entities to strengthen their M&E systems. The relationship between CLEAR-AA and TETA should be understood in this context.
This relationship was dictated by the terms of reference which required CLEAR-AA to develop the research strategy and the sector skills plan. This could have easily been treated as a business transaction ending immediately after the delivery of the outputs to the client. However, CLEAR-AA adopted a developmental approach anchored in capacity building and driven by a desire to enhance transparency, accountability and better use of evidence for policy-making in Anglophone Africa.
This approach is informed by considerations of two factors. Firstly, evaluation capacity development in itself is a long process and cannot be reduced to a business transaction. An effective evaluation capacity strategy focuses on strengthening systems of management, governance, accountability and learning (African Development Bank & World Bank Operations Evaluation Departments 1998; OECD 2010). Secondly, CLEAR-AA has a mandate to develop evaluation capacity in Anglophone Africa, and hence, developmental response is built into its operations.
The first step in evaluation capacity development is to analyse the issues and challenges facing policymaking and policy-makers (Campbell et al. 2007; Paine Cronin & Sadan 2015). Therefore, the Six Sphere Framework (SSF) tool was developed and used to obtain information about factors that facilitate or inhibit effective implementation of M&E including the use of evidence for programme improvement and policy decisions in the transport sector in South Africa. Preskill, Zuckerman and Matthews (2003) identified five factors that hinder utilisation of evaluation results. This article adds to the existing knowledge by highlighting more issues affecting the use of evidence for decision-making in the transport sector in South Africa.
An overview of current evaluation capacity assessment practice
Evaluation literature indicates a growing interest in evaluation capacity building theory and practice (Carman 2007; Preskill & Boyle 2008; Taylor-Ritzler et al. 2013). However, information on evaluation capacity assessment instruments is limited. For example, out of potential 194 articles yielded through a search of peer-reviewed articles on evaluation capacity development, only seven were about evaluation capacity instruments or tools (Bourgeois, Simmons & Osseni 2015:12). The authors’ own search identified additional sources including peer-reviewed articles which applied or recommend the use of certain instruments, tools and guidelines for evaluation capacity assessment (Holvoet & Inberg 2013; Mackay 2007; Taylor-Ritzler et al. 2013; UNAIDS 2009).
There are limitations with the current evaluation capacity assessment frameworks. For example, the UNAIDS framework2 does not provide clear guidance on how the assessment should consider sociocultural and political issues. Stressing the importance of political dynamics in understanding situational context, Eyben et al. (2008), cited in Stein and Valters (2012:15), argued that ‘any model of societal change is political and value-laden and should understand and relate to the power relations therein’. The UNAIDS framework is inadequate in the context of South Africa where socio-economic and political issues remain important as the country continues to grapple with the legacy of an oppressive and exclusive apartheid system.
Similarly, the evaluation capacity assessment instrument (ECAI) developed by Taylor-Ritzler et al. (2013) does not take these factors into account. It is structured in a way that suggests all organisations are motivated and encouraged to use evidence for planning and decision-making. This approach fails to recognise that motivation to use evidence for decision-making in South Africa is affected by factors such as ‘political pressure, lack of value placed on learning, research and mistrust between political leadership’ (Paine Cronin & Sadan 2015:8).
A diagnostic report for the assessment of M&E system in Columbia shows that M&E is used for political and social control (Mackay 2007). Although the tool highlights political factors that facilitate the use of M&E information in Columbia, it does not emphasise issues such as race, political ideology and culture, which may cause mistrust between political leadership in South Africa and thereby affect the effective use of evidence for decision-making.
Finally, Holvoet and Inberg (2013) developed a tool to assess M&E systems in the health sector in Uganda and Rwanda. Although the tool covers important aspects relevant to the health sector, it does not pay attention to the context. For instance, it ignores the socio economic, political and legislative framework within which health sector policies are implemented. In sum, the studies described above are not adequate to assess a myriad of factors that affect effective use of M&E in TETA. The SSF brings innovation into the evaluation capacity assessment literature by highlighting the need to understand the overall context including socio-political factors that shape organisational values norms and practices that influence M&E practice in general and the use of evidence for decision-making in particular.
Methodology
The need for TETA to assess its evaluation capacity came at the opportune time right after the development of the SSF. This created an ideal opportunity for the researchers to use the organisation as a case study to apply the SSF to explore factors that impact effective use of M&E in the transport sector.
Sampling
The purpose of the assessment was to explore the use of a new analytical tool to understand experiences, abilities and opinions of people about M&E and how it could be used to improve decision-making in the transport sector. Purposive sampling was used to select participants including those who were part of the telephonic survey in which 300 respondents participated. These were selected from a list of 1600 companies provided by TETA. Officials from the biggest companies in terms of market size and number of employees were selected for the survey and subsequent participation in the workshops and focus group discussions. Respondents ranged from senior to mid-level officials working in human resources, corporate services and planning departments. The selection of companies was also based on the fact that large companies have the biggest skills development programmes and pay huge amounts in skills levy.
Data collection
The study adopted a mixed methods approach with a strong inclination towards qualitative methods. The reason for the heavy reliance on qualitative approach is that the study was exploratory in nature with the primary purpose of deriving lessons from the application of the tool rather than testing its validity. Semi-structured interviews structured around the six themes of the SSF were used to obtain data from programme staff (n = 10) and senior management (n = 5) across different departments of TETA. In addition, focus groups of between six and eight participants were held with each of the eight chamber boards. In total, 60 people participated in the discussions. The purpose of the discussions was to obtain strategic information critical for better understanding of the organisational value for M&E.
Another stream of data collection involved the use of a survey of officials from a variety of companies making up the eight sub-sectors of the transport industry (n = 300). The purpose of this was to obtain the perception of stakeholders on whether TETA has adequate evaluation systems and tools for M&E. The survey was conducted telephonically using a questionnaire structured around the six components of the SSF. In order to enhance validity, it was pretested internally among CLEAR-AA staff, and feedback was used to improve the quality of the questions. It was also pretested within TETA and modified accordingly.
The survey was supplemented with 16 workshops in the four major transport provinces (Gauteng, Western Cape, KwaZulu-Natal and Eastern Cape). The workshop participants (n = 40 per workshop) were drawn from the survey. The workshops were participatory and involved presentations and guided group work based on the six components of the SSF. Following each workshop, a focus group discussion of individuals (n = 10) was held for detailed discussion of key issues that emerged from the workshops.
Secondary sources were also reviewed to obtain relevant data from published and grey literature. The review focused on literature on evaluation capacity development in general and the use of evidence in decision-making in South Africa in particular. The review was guided by the six components of the SSF. The use of multiple methods is useful to triangulate information from different sources in order to enhance credibility of the findings.
Data analysis
Upon the conclusion of data collection, the SSF was used as a framework for data analysis. In this regard, data from different data sets (survey, literature review, semi-structured interviews and focus group discussions) were categorised in six themes in accordance with the dimensions of the SSF. A comparative analysis was then conducted to identify and describe patterns emerging from different data sets. Frequencies were generated from the survey data and used where necessary, to support qualitative evidence.
The SSF framework
This section provides a brief description of the SSF with particular emphasis on key elements of the six components as depicted in Figure 1.
Logistical sphere
Logistical issues relate to the assessment of how organisations arrange, mobilise and manage their resources to support the use of evidence for policymaking. These resources may include time, partnerships, staff and finances. These are critical ingredients for building a successful M&E system. For instance, the scheduling of planning, budgeting and M&E activities is important to ensure the effective use of evidence in decision-making especially in big organisations where these functions may be located in different units.
In the same manner, the location of the M&E unit within the organisation is important as it has implications on how the unit will work optimally in collaboration with planning and budget functions. The location of the unit also gives an indication of the value and the mandate of the unit within the organisation. It is this value that determines allocation of resources to enable the unit to be effective. Görgens and Kusek (2009) argued that the M&E unit functions better when it is linked to policy and planning processes although it may have less autonomy, authority and power than when it is a stand-alone unit.
The authors further show that creating strong partnerships is important for building effective M&E systems. This is more so in the context of TETA as it has the responsibility of coordinating the entire transport sector. Based on this, the tool attempts to understand internal and external partnership arrangements within TETA and the extent to which these are used strategically for promoting evaluation capacity development within the organisation and the sector as a whole.
Technical sphere
Although there are different dimensions of capacity assessment, this component focuses on technical capacity at the individual and organisational levels. These are the elements considered pertinent to a functional M&E system (Görgens & Kusek 2009). At the individual level, the assessment focuses on technical skills required for effective performance of M&E functions including general management and coordination as TETA has the responsibility to coordinate skills development programmes across the entire transport sector. Similarly, the organisational dimension is concerned with the assessment of the availability, quality and use of necessary tools and processes that facilitate effective functioning of M&E within TETA.
Contextual sphere
The importance of context in evaluation has been long highlighted in the literature (Stake 1975; Stufflebeam 1971; Weiss 1972). Despite this, various contextual factors that influence evaluations are not adequately addressed (Fitzpatrick 2012). In an effort to address this weakness, the tool clearly specifies the need for consideration of contextual factors. Context in evaluation is broadly and encompasses a range of dimensions some of which are discussed in other sections. However, in this case, the context is limited to understanding the overall design of the government M&E system and the challenges it presents to TETA in terms of reporting. The component also looks at the legislative framework within which the skills development programme is implemented and how it affects reporting and use of evidence in decision-making within the transport sector.
Social sphere
Social dynamics are hardly taken into consideration in capacity assessment studies. For example, none of the studies (Carman 2007; Preskill & Boyle 2008; Taylor-Ritzler et al. 2013) reviewed mentioned the importance of social factors in evaluation capacity assessments. Social issues are important variables for consideration in order to have a full understanding of the South African context where racism, gender discrimination and systemic exclusion of youth in the policy space prevails (Paolini, Horvath & Coutinho 2013). Furthermore, a recent study identifies lack of trust as one of the key factors impacting the effective use of evidence in South Africa (Paine Cronin & Sadan 2015). Issues of trust and commitment to change could be barriers towards addressing structural racial and socio-economic divisions inherited from the apartheid system. It is for this reason that the SSF attempts to understand how social aspects could hinder or facilitate the effective use of evidence in the transport sector.
Political sphere
The inclusion of political factors in the SSF is based on the notion that evaluation itself is a political activity that takes place within a political context (Weiss 1993). Research in South Africa further shows that evidence is often ignored at the expense of political considerations when policy decisions are made (Paine Cronin and Sadan 2015). It is, therefore, important to assess the extent to which the same argument holds in the transport sector. The main focus in this component is to first establish who the drivers of the M&E agenda are. Secondly, it is important to determine whether there is political will to facilitate better use of evidence in decision-making and whether TETA leadership shares a common vision in this regard.
Organisational value system
This sphere should not be seen in isolation but rather as a knot that ties the other aspects of the framework together. For instance, all the elements collectively shape the organisational attitude, norms, practices, beliefs and values towards use of evidence. Although these are important in shaping the attitudes and behaviour of people towards implementation and use of M&E for decision-making within organisations, they are very difficult to identify and change. This assessment tool attempts to bring these factors to the fore so as to understand how logistical, technical, contextual, social and political factors interface to shape the organisational culture and values that facilitate or hinder effective implementation of M&E.
Findings
The findings presented in this section are organised along thematic areas derived from the six components of the SSF framework. The section highlights logistical, technical, contextual and socio-political factors that influence or facilitate M&E in the transport sector. It also shows how these interact to shape organisational norms, practices and values that enhance or inhibit effective implementation of M&E and use of evidence in the sector.
Logistical factors impacting the use of evidence
The main observation is that dual implementation modality is used to finance and implement the skills development interventions across the transport sector in South Africa. This involves mandatory and discretionary funding mechanisms. Mandatory grants are paid by TETA to the employers for implementation workplace skills development programmes, which are designed by employers based on their own needs. In order to qualify for these grants, employers have to pay a skills levy and submit an application for a grant, a Workplace Skills Plan (WSP) and the Annual Training Report.
Although the workplace programmes are structured in terms of timelines governing submissions of mandatory documents, only 5% of the employers submit ATRs. The main reason being that the reporting cycle of TETA is not aligned with that of many employers it is coordinating. For example, an application for a grant is 30 April every year, but some companies’ financial year ends in June or December. This makes it difficult for such companies to apply and report for the previous year.
Companies are not the only culprits when it comes to planning and coordination. Interviews with internal TETA staff reveal that the eight operating chambers within TETA itself undertake their strategic planning meetings at different times during the planning cycle. This presents a major challenge for the M&E unit in terms of coordination leading to unnecessarily more time spent on operational rather than strategic matters.
Discretionary grants on the other hand are implemented within TETA as part of the SSP and Annual Performance Plan (APP). These programmes are implemented through service providers and monitored through a standard government service level agreement. The assessment indicates that these programmes are monitored to ensure compliance with legal agreements and there is no mechanism to monitor and evaluate for programme or policy improvement. The findings also indicate that these projects are largely subject to political interference and are often not well designed, implemented and evaluated.
The location of the M&E units within organisations has a bearing on its effectiveness depending on its primary functions. According to Görgens and Kusek (2009), if its primary purpose is accountability, the unit is best placed as an independent office outside the organisation. In the case of TETA, there are two units with similar M&E functions, namely the M&E unit and the research and knowledge management unit. As reported during the interviews with staff, having two units with overlapping responsibilities creates confusion and unhealthy competition instead of complementarity. It also results in inefficiency as the units are over stretched. The same issues were explored among recipients of mandatory grants where it is noted that all the skills programmes are located within human resources-related departments which largely play an administrative role of conducting skills audits and compiling and submitting applications and reports to TETA.
Technical issues impacting the use of evidence
Research highlights poor reporting as one of the major constraints to the effective use of evidence in decision-making in South Africa (Engela and Ajam 2010; Paine Cronin & Sadan 2015). This could be an indication of lack of individual technical capacity in many respects. It is for this reason that Görgens and Kusek (2009) identify human resource capacity development as an important strategy for a functional M&E system. With this in mind, technical capacity of M&E staff within TETA was assessed against the known competencies for M&E. However, most reported having inadequate skills to manage evaluations and ensure effective use of evaluation results for policy and decision-making.
At the organisational level, the assessment focused on the existence of tools, policies and systems for M&E. In this case, TETA has an M&E framework. However, some staff did not know about it. Furthermore, the majority (80%) of stakeholders who participated in the survey felt that TETA did not have necessary tools and systems to ensure the effective use of evidence for decision-making. These sentiments were also expressed during the focus group discussions.
Contextual factors impacting the use of evidence
The implementation of the skills development programme within the transport sector should be seen in the context of the national development plan that provides a framework for national skills development programme as outlined in the third generation of national skills development strategy. In the same breadth, its M&E should be located within the broader Government-wide Monitoring and Evaluation System (GWMES) introduced in 2005.
Increased awareness on the need to use evidence for policy decision has been reported since the introduction of the GWMES (Cloete 2009). However, research points to a plethora of problems afflicting the system (Cloete 2009; Engela and Ajam 2010). Some of these were also reported by the programme managers who were unanimous about duplication of reporting between the national treasury and the Department of Higher Education and Training (DHET), which oversees the TETA. It must also be noted that DHET has taken corrective measures by establishing one reporting template which accommodates the reporting standards set by Treasury, DHET and the Department of Performance Monitoring and Evaluation (DPME).
The legislative framework governing TETA is another factor that impacts reporting. Stakeholder’s interviews revealed that the amendment of the Skills Levies Act in 2007 has created disincentive for reporting. The amendment significantly reduced the mandatory amount payable to the employers by TETA from 50% to 20% of their total expenditure on workplace skills development activities. Furthermore, the Skills Development Act does not give TETA legislative power to compel employers to prepare their WSPs and ATRs.
Social factors impacting the use of evidence
The transport sector, like all other sector in South Africa, is affected by social inequalities inherited from the apartheid system. For example, the black population and women and youth in particular are largely excluded from the mainstream economy including the transport sector, which employs only 20% of women. Similarly, young people constitute about 12% share across all occupational groups and are equally underrepresented in the transport sector (TETA SSP 2016). The assessment focused on social variables such as gender and race based on the assumption that these could affect the level of commitment necessary to facilitate the effective use of evidence. However, no clear correlation between gender, race and ineffective use of evidence is found. The only indication that trust may affect the use of evidence is reflected in a recent study that links mistrust between political leadership to ineffective use of evidence in policy (Paine Cronin & Sadan 2015).
Political factors impacting the use of evidence
Weiss (1993) argued that evaluation cannot be separated from politics. It is against this background that the SSF includes a political dimension aimed at establishing the extent of political interference in decision-making. In this regard, the research corroborates the fact that important policy decisions are often influenced by politics even when strong evidence supporting alternative approaches exists (Paine Cronin & Sadan 2015). The majority (75%) of participants in the survey agreed that political considerations often cloud better judgement when important policy decisions are made with regard to implementation of the skills development programme in the transport sector. These views were also expressed in the stakeholders’ workshops where participants gave an example of many programmes implemented under discretionary grants as being at high risk of political meddling. Political meddling cuts across all the stages of the policy cycle but is more pronounced at the implementation phase. The respondents mostly identified this phase as where procurement processes and procedures are often manipulated for personal benefit.
Organisational value system
Oliveira and Tamayo (2004) argued that the importance attached to certain values by an organisation determines the amount of effort and resources it is willing to invest in a particular activity. In the same spirit, Görgens and Kusek (2009:227) maintained that ‘positive culture for M&E is an essential and important part of having an enabling environment for M&E’. Therefore, the assessment was intended to demonstrate the extent to which logistical, technical, contextual, social and political elements collectively influence negative or positive culture towards M&E in TETA. The findings in this particular case indicate that TETA as the organisation does not have adequate M&E tools and systems to support a culture of use of evidence for decision-making.
Discussion
Reporting is a key challenge in the implementation of programmes under mandatory grants. Although this can largely be attributed to poor individual M&E technical skills, it also shows how changes in legislative environment affect motivation for reporting on the part of the employers. Research shows that limited time allocated to policy process is another important barrier to the effective use of evidence in policy-making (Görgens & Kusek 2009). In the case of TETA, the lack of use of evidence for decision-making is not necessarily a result of limited time allocated to policy process but rather unsynchronised planning cycle.
Görgens and Kusek (2009) further pointed out that the location of the M&E unit within organisations is important to ensure its effectiveness. Although M&E deficiencies are noted within TETA, this is a result of two issues. Firstly, there is a challenge of overlapping mandates between the M&E and the research and knowledge management units. Secondly, the inefficiencies are caused by inadequate technical evaluation skills and do not necessarily relate to the location of the units within the organisation.
As indicated earlier, TETA does not have adequate M&E systems to facilitate the effective use of evidence. This finding is consistent with the views expressed by Paine Cronin and Sadan (2015) who state that the lack of common norms and standards that guide and assess good use of evidence for decision-making is a serious factor limiting the use of evidence in South Africa. The lack of strong organisational policies and guidelines for M&E in TETA may foster a negative culture towards use of M&E for decision-making.
On the political front, the results confirm literature that shows political interference as one of the key barriers on effective use of evaluation in South Africa (Paine Cronin & Sadan 2015). In this case, it has been observed that programmes implemented under discretionary grants system are more prone to political interference than those implemented under mandatory modality. The reason for this is that mandatory grants are often well structured and implemented by the employers, whereas discretionary grants are often implemented without proper diligence on design, implementation and evaluation. However, this may change over time as there is strong political will from senior management to improve M&E within the organisation.
This article is structured around the assumption that logistical, technical, contextual, social and political issues interface to influence organisational culture towards M&E and the use of evidence for decision-making. TETA is relatively weak on the logistical and technical requirements necessary to facilitate the effective use of evidence for decision-making. Although no empirical evidence shows the role of social factors in constraining or facilitating the effective use of evidence for decision-making in the organisation, there is documented evidence suggesting that trust is an important variable that impacts the use of evidence for decision-making (Paine Cronin & Sadan 2015).
Conclusion
This article reveals some limitations inherent in the current literature on evaluation capacity assessment frameworks. The authors recommend the SSF as a new tool that could be used to assess evaluation capacity in the South African context. Components of the SSF are interrelated and should not be understood in isolation. For example, the implementation modality is categorised as a logistical issue, but this is also derived from the legislative framework from which TETA draws its mandate. Similarly, the operational arrangements outlined under logistical issues are intertwined with individual technical capacities of M&E staff in TETA. Likewise, political and legislative factors do not only shape the context within which the skills development programmes in the transport sector are implemented but also interface with logistical and technical aspects to provide a better understanding of factors that inhibit the use of evidence for decision-making in the transport sector. As much as the SSF is an addition to the evaluation literature, it does not confirm previous evidence that social elements such as trust affect the effective use of evidence in the case of TETA. Moreover, the framework assumed that race and gender play a role in influencing the use of evidence for decision-making. However, this assumption appears to be not true in the case of TETA.
The results from the application of the SSF revealed commitment by senior management towards strengthening M&E within TETA. It is recommended that the research and knowledge management unit and the M&E unit be merged to create a strong and effective unit. Finally, management should consider aligning their internal planning processes and ensure that the M&E unit fully supports such processes and facilitate the use of evidence for decision-making.
Acknowledgements
The authors acknowledge the assistance of the officials of the Transport Education and Training Authority (TETA) who provided valuable information from which this article has duly benefitted.
Competing interests
The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.
Authors’ contributions
B.D.B. and K.T. were the project leaders and were responsible for the problem identification and project design. All authors reviewed relevant literature on the topic and jointly did the write-up.
References
African Development Bank & World Bank Operations Evaluation Departments, 1998, Evaluation capacity development in Africa: Selected proceedings from a seminar in Abidjan, African Development Bank, Tunis.
Bourgeois, H., Simmons, L. & Osseni, R., 2015, Building evaluation capacity in Ontario’s public health units, Cornwall, Ontario, Canada.
Campbell, S., Benita, S., Coates, E., Davies, P. & Penn, G., 2007, Analysis for policy: Evidence-based policy in practice, Government Social Research Unit, HM Treasury, London, England.
Carman, J.G., 2007, ‘Evaluation practice among community-based organizations: Research into the reality’, American Journal of Evaluation 28, 60–75. https://doi.org/10.1177/1098214006296245
Cloete, F., 2009, ‘Evidence-based policy analysis in South Africa: Critical assessment of the emerging government-wide monitoring and evaluation system’, Journal of Public Administration 44(2), 293–311.
Crawley, K., 2017, ‘The six-sphere framework: A practical tool for assessing monitoring and evaluation systems’, African Evaluation Journal 5(1), a193. https://doi.org/10.4102/aej.v5i1.193
Engela, R. & Ajam, T., 2010, Implementing a government-wide monitoring and evaluation system in South Africa, World Bank, Washington, DC.
Fitzpatrick, J. L., 2012, ‘An introduction to context and its role in evaluation practice,’ New Directions for Evaluation 135, 7–24.
Görgens, M. & Kusek, J.Z., 2009, Making monitoring and evaluation systems work: A capacity development toolkit, World Bank, Washington, DC.
Holvoet, N. & Inberg, L., 2013, ‘Taking stock of monitoring and evaluation systems in the health sector: Findings from Rwanda and Uganda’, Health Policy and Planning 29, 506–516. https://doi.org/10.1093/heapol/czt038
Mackay, K., 2007, How to build M&E systems to support better government, World Bank, Independent Evaluation Group, Washington, DC.
Organisation for Economic Cooperation and Development, 2010, ‘Towards a strategic approach for evaluation capacity development’, in Evaluation Capacity Development, draft note prepared for the OECD 11th Meeting.
Oliveira, A.D.F. & Tamayo, A., 2004, ‘Inventário de perfis de valores organizacionais’, Revista de Administração da Universidade de São Paulo 39(2), 129–140.
Paine Cronin, G. & Sadan, M., 2015, ‘Use of evidence in policy making in South Africa: An exploratory study of attitudes of senior government officials’, African Evaluation Journal 3(1), 1–10. https://doi.org/10.4102/aej.v3i1.145
Paolini, G., Horvath, A. & Coutinho, A.S.D.A., 2013, Youth social exclusion and lessons from youth work: Evidence from literature and surveys, Education, Audiovisual and Culture Executive Agency (EACEA) of the European Commission (Eurydice Unit), Europe.
Preskill, H. & Boyle, S., 2008, ‘A multidisciplinary model of evaluation capacity building’, American Journal of Evaluation 29, 443–459. https://doi.org/10.1177/1098214008324182
Preskill, H., Zuckerman, B. & Matthews, B., 2003, ‘An exploratory study of process use: Findings and implications for future research’, American Evaluation Journal 24(4), 423–442. https://doi.org/10.1177/109821400302400402
Sidzumo, S. & Wotela, K., 2016, ‘Deriving a conceptual framework for assessing utilisation of evaluation information in government departments’, SAAPAM Limpopo Chapter 5th Annual Conference Proceedings 2016, South African Association of Public Administration and Management (SAAPAM), Mokopane, South Africa, October 26–28, 2016, pp. 320–333.
Stake, Robert E. & Western Michigan Univ., Kalamazoo. School of Education, 1975, Program Evaluation, Particularly Responsive Evaluation. Paper #5 in Occasional Paper Series [microform] / Robert E. Stake, ERIC Clearinghouse, Washington, DC.
Stein, D. & Valters, C., 2012, Understanding theory of change in international development: A review of existing knowledge, JSRP, International Development Department, Houghton, London.
Stufflebeam, D.L., 1971, ‘The relevance of the CIPP evaluation model for educational accountability’, Journal of Research and Development in Education 5, 19–25.
TETA SSP, 2016, Transport SETA Sector Skills Plan 2016/17, Transport Education Training Authority, Johannesburg, South Africa.
Taylor-Ritzler, T., Suarez-Balcazar, Y., Garcia-Iriarte, E., Henry, D.B. & Balcazar, F.E., 2013, ‘Understanding and measuring evaluation capacity: A model and instrument validation study’, American Journal of Evaluation 34(2), 190–206. https://doi.org/10.1177/1098214012471421
Joint United Nations Programme on HIV/AIDS (UNAIDS), 2009, 12 components monitoring & evaluation system assessment: Guidelines to support preparation, implementation and follow-up activities, UNAIDS, Geneva, Switzerland.
Weiss, C.H., 1972, Evaluation research: Methods for assessing program effectiveness, Prentice Hall Inc., Englewood Cliffs, NJ.
Weiss, C.H., 1993, ‘Where politics and evaluation research meet’, Evaluation Practice 14(1), 93–106. https://doi.org/10.1016/0886-1633(93)90046-R
Footnotes
1. The eight sub-sectors are aerospace, freight forwarding and clearing, freight handling, rail, road passenger, maritime, road freight and taxi industry.
2. The 12 Components Monitoring and Evaluation System Strengthening Tool.
| 
Hide
Show all
Email this article (Login required)
