World Vision implemented the community-based Maternal and Child Health Transformation (MaCHT) Project from September 2010 to September 2014 in fragile-state South Sudan. To document and measure health-related activities executed by an international non-governmental organisation to sustainably strengthen the capacity of the health system in delivering essential health services to pregnant women and children under two years of age, including new-borns and infants. A range of mixed methods, including in-depth interviews, focus group discussions, observation, and uncontrolled cross-sectional before-and-after surveys using Henderson's method were carried out. The unit of analysis was mothers of children under two years of age, and community health workers (CHWs). An estimated 39 000 children under age two were attended to by CHWs. Coverage of essential maternal and child health care (MCH) increased in all single interventions, ranging from a minimum of 5% points to a maximum of 49% points during the implementation period. The capacity of the health system to deliver essential MCH services improved by building the supply and performance of the health workforce through task-shifting and in-service training. Likewise, operational linkages between community structures and local health services were strengthened. In conclusion, this program supported health system strengthening, mainly in the areas of service delivery, health workforce, and medical products, vaccines, and technologies. The project also informed policy at district and national levels and repositioned the maternal, neonatal, and child health (MNCH) agenda to further scale up these activities. An evaluation of a four year USAID-funded child survival project implemented by an international non-governmental organisation (NGO) in fragile-state context showed progress and challenges in health system strengthening for maternal health practices and community case management of diarrhoea, pneumonia, and malaria in children under five.

Half of global child deaths take place in fragile conflict-affected countries,¹ mainly due to weak health systems with consequent major disruptions of health service delivery. Despite the work of international non-governmental organisations in this type of setting, their role and value in supporting national health systems has not been adequately documented.

The evaluation of the Maternal and Child Health Transformation (MaCHT) project² examined the contribution of an international non-governmental organisation (INGO) to the efforts of the South Sudan Ministry of Health (MoH) in strengthening the health system's capacity to deliver essential community-based services in a fragile post-conflict context.

In 2005, the Government of Southern Sudan (GOSS) (later South Sudan) operationalized three levels³ of health care – the Primary Health Care Unit (PHCU), the Primary Health Care Centre (PHCC), and the Referral Hospital (GOSS 2005). However, the health system remains severely constrained by chronic and severe shortages in human resources for health with only 126 trained physicians and 211 trained midwives for the country's current population of 11.3 million and covering an area of 644 329 square km, roughly the size of France (World Health Organization 2015). Nearly 85% of health workers are provided by international and faith-based non-governmental organisations (NGOs) (Wakabi 2011). Furthermore, health indicators in South Sudan are dismal, with the world's highest maternal mortality ratio of 2054 per 100 000 live births, and an infant mortality rate of 68 per 1000 live births (HHS 2006).

INGOs are increasingly transitioning from a role of direct service provision to one of building capacity and systems strengthening. INGOs work alongside government ministries, provide input to policy and planning, engage at the local level, and advocate at the macro level in national and international settings (Ulleberg 2009). In some conflict/post-conflict settings, such as in South Sudan, INGOs provide health, education, and other services where the host government is unable to do so. Indeed, the United States Agency for International Development (USAID), in its three-year strategy for South Sudan's post-comprehensive peace agreement period, noted that:

A key consideration is striking the right balance between capacity building and timely, effective implementation,” with recognition that “development resources may need to be redirected from capacity and systems building to service delivery. (USAID Transition Strategy 2011:15)

In many low-income countries, bottlenecks to effective health service delivery include a shortage of human resources for health, poor supply chain, and limited physical and policy infrastructure to support a robust health system (Chopra et al. 2012). Shifting service delivery from clinical settings to the community can be an effective way to overcome constraints of health service delivery, whilst other strategies concurrently address the bottleneck within the clinical service delivery channel (Chopra et al. 2012). In post-conflict/conflict settings such as South Sudan, community-based human resources supported by INGOs can fill this gap by supporting immediate service delivery whilst also training health workers, enhancing infrastructure, and advocating for policy.

One of the most critical areas affecting MCH outcomes is the training of community health workers (CHWs). Haines et al. (2007) point out that CHWs play an increasingly significant role in the delivery of preventive and curative health care to populations for whom location, economic status, gender, and other factors make direct access with the formalised health system difficult. The MaCHT project used the Home-Based Life Saving Skills (HBLSS) curriculum and the community case management (CCM) approach to train home health promoters (HHPs), South Sudan's cadre of volunteer health extension workers, to provide community-based maternal, neonatal, and child health (MNCH) services.

HHPs mobilised families to prepare for birth, assisted with home births by providing a cleaner birth environment, referred dangerous pregnancy cases to a health facility, and provided essential care to new-borns. In addition, HHPs were trained to identify and treat non-severe pneumonia, diarrhoea and malaria in children per MoH protocol and refer those with serious illness. MaCHT also mobilised leaders at the boma [cluster of villages] and payam [administrative centre] levels to organise communities around health issues and promote utilisation of essential services such as long-lasting insecticidal nets (LLINs), immunisations, and antenatal care (ANC).

MaCHT did not provide routine procurement or direct service delivery, but facilitated access to, and utilisation of, MNCH-related preventive and treatment services. It organised refresher training for PHCU staff in MNCH-focused areas and coordinated with State MoH and other partners to ensure availability of drugs at PHCUs. It sought to improve community ownership of peripheral health services by developing and strengthening Health Facility Management Committees (HFMCs) and by facilitating linkages of HHPs to PHCUs.

The evaluation of the MaCHT project evaluation was conducted to measure and document the INGO contribution to rebuilding the health system's capacity for delivering essential health care to the target population of women and children. The evaluation assessed the project's achievement towards its stated goals and objectives; provides information on mechanisms used during implementation of the various interventions; and provides evidence to inform decisions for future programming. The overall objectives of the project were to: (1) Increase use of high-impact, low-cost, and feasible interventions to improve maternal, neonatal, infant, and child health outcomes; (2) Strengthen capacity of the health system to deliver essential services; and (3) build and strengthen local and national partnerships to sustain improvements in maternal, neonatal, infant, and child health.

Two main evaluation questions guided the design and implementation of the overall evaluation to document contributions in improving the capacity of the health system to deliver essential services: (1) To what extent did the project accomplish and/or contribute to targeted goals/objectives? (2) What were the key strategies and factors, including management issues, that contributed to what worked or did not work? This publication will focus on these questions and related results, and identify the contribution of the INGO's work in rebuilding the local health system.

The evaluation used a mixed-methods approach, incorporating primary quantitative data collected through knowledge, practice, and coverage (KPC) surveys carried out at baseline and endline, with qualitative data collected during the evaluation from interviews and focus groups. The WHO sampling protocol developed by Henderson for evaluating EPI immunisation coverage was used. With this method, to be accurate within 10% with a margin error of 5% would require surveying 210 children per intervention area. For greater accuracy, we surveyed 298 children at baseline and 510 at endline. It assessed improvements in outcomes; such as increased access to and use of diarrhoea, pneumonia, and malaria treatment, and other essential health interventions; the effectiveness of the project's strategy and processes for reaching its targets, and contextual factors that influenced progress.

Quantitative methods: 808 mothers of children under two were included in the sample; 298 during baseline (FY2011) and 510 during endline (FY 2014).

The selection of participants used a two-stage random selection (two-stage 30 cluster sampling). Within each selected cluster, households and respondents (mothers of children aged 0–23 months) were randomly selected and only one respondent was selected from a household to avoid overrepresentation.

A structured questionnaire was applied to selected participants during baseline and endline. Eight enumerators and four supervisors were trained and deployed to collect data in each cluster.

Data collection management and follow-up supervision was in place during data collection to minimise errors. Questionnaires were checked for completeness and consistency when they arrived at the place of data entry. Any inconsistencies observed during data entry were recorded and the feedback was given to the concerned enumerator and supervisor on a daily basis. Lessons applicable to all enumerators were also shared with them before they left for the field.

All collected data was coded and entered into Excel. Data was checked for inaccuracies and inconsistencies, and then entered into SPSS Statistical Analysis software. Data analysis was conducted in two steps. The first step consisted of the production of descriptive statistics for each variable included in the survey. The second included the calculation of p-values (using Mantel-Haenszel two-tailed tests).

Qualitative methods: In-depth qualitative interviews and focus group discussions were conducted with stakeholders including project staff, MoH, local NGOs and community-based organisations, district health teams, community and facility-based health workers, community members, community leaders and mothers. Two focus groups were conducted, one with mothers of children under two years of age and one with HHPs. Both of these were conducted in the program intervention area where operations research was done. Focus group questions explored the effects of the project, and identified barriers to project success. Fourteen interviews were conducted with stakeholders selected on the basis of their involvement with the project.

Secondary data: The evaluator reviewed project reports (e.g. detailed implementation plan; annual reports; mid-term evaluation; baseline and endline KPC surveys; monitoring reports) to assess the quality of quantitative and qualitative data and to assess the project's results in relation to its design and set targets.

Limitations: The endline survey was limited by several factors, including the distance to villages, difficult terrain, vehicle availability, and early nightfall. Due to the questionnaire's length, it was difficult for some respondents to complete and difficult to ensure accuracy of entered data. Additionally, as the data for household practices was based on mothers’ responses and not observation, over-reporting of recommended practices cannot be ruled out. The qualitative survey was limited by the number of beneficiaries and volunteers that could be interviewed for the project. Impediments to the evaluation included weather, long travel times, and limited number of days for the on-site qualitative evaluation process. One key informant from the state MoH was unable to keep the scheduled appointment and could not reschedule during the period set aside for qualitative interviews.

Warrap State is located in the northern part of the country and is home to an estimated 1.7 million people, mostly ethnic agro-pastoral communities of the Twic Dinka tribe. The state is located in flood plains with a long rainy season, and the region is endemic for malaria, and experiences frequent outbreaks of acute watery diarrhoea and measles. Within Warrap State, the project covered four of nine payams, across Gogrial East and Gogrial West counties. The total end-of-project population was estimated to be 148 899.

In the area of health, much of the human capital is provided by non-nationals. According to WHO, the country has a health professional density of 2.8⁴ which is one-eighth of the minimum acceptable level to achieve health coverage for essential health interventions. The Warrap State MoH reports that less than 3% of health professionals needed to provide adequate care in the area are currently available.⁵ Educational and capacity building projects have not yet created enough human capacity to fill all of the positions necessary to implement and deliver a full range of comprehensive health care. Human and physical infrastructure needs are great, especially roads, health facilities, nurses, midwives, and doctors. Where health facilities are available, they require major upgrades, as well as effective supervisory and quality assurance mechanisms.

Health services are delivered through hospitals (at national, state, and county levels), PHCC, and PHCUs. Each PHCC is designed to provide basic emergency obstetric care (BEmOC) services for 25 000 women of reproductive age (WRA) and is staffed by midwives and community midwives. PHCUs are outposts of PHCCs and provide basic preventive and curative services to about 15 000 people (Table 1).

HHPs provide promotion care and mobilise communities, supervised by maternal and child health workers (MCHW) and CHWs. HHPs are selected by communities and are often trained by NGO-led programs. In the project area, CHWs and MCHWs are based in the PHCUs and are appointed following nine months of training. National and international NGOs run most of the PHCCs and PHCUs around the country and provide over 80% of all health services in the country.

The project's target area has seven PHCUs offering basic prevention and promotion services and two PHCCs – both staffed with community midwives and one designated to provide basic emergency obstetric services. HHPs trained to provide case management for non-severe pneumonia, malaria, and diarrhoea, as well as referrals for more severe conditions, were not in place prior to the start of the MaCHT project.

MaCHT utilised five technical interventions and corresponding levels of effort were: Maternal and newborn care (MNC) (30%), malaria (20%), pneumonia (20%), control of diarrheal disease (CDD) (20%), and immunisation (10%). The project sought to utilise high-impact, low-cost, and feasible interventions⁶ to improve maternal, neonatal, infant, and child health outcomes at the household/community level through the training of HHPs, a newly approved cadre of CHWs in South Sudan, supported by the HFMCs and mother leader groups (MLGs). Additionally, the project sought to strengthen the capacity of the health system through investment in increased human resource capacity and improvements at health facility level.

The partners for the MaCHT project included national and state Ministries of Health (MOH and SMOH), county health departments, village and boma health management committees, and the Gogrial East Women's Association.

Maternal care: All seven maternal care-related variables showed significant and positive change over the life of project implementation (see Figure 1). Nearly one-third of mothers surveyed reported having the recommended four or more antenatal checks whilst pregnant with their youngest child, showing a statistically significant increment of 17 percentage points compared with data found at baseline. Three out of five of mothers surveyed received at least two tetanus toxoid vaccinations before the birth of their youngest child, as well as iron supplementation and intermittent preventive therapy (IPTp) representing 38, 31 and 40 percentage point differences respectively from baseline data. Skilled birth personnel (doctor, nurse, midwife, or other medical staff with midwifery training) attendance increased by 5% from baseline to endline as well as active management of third stage of labour (AMTSL).⁷ Delivering at a health facility was a common theme. One woman noted the importance of delivery at health facility, especially for the first time, ‘because the baby may not come out and you may have complications. You may be helped when something goes wrong.’ Another woman said that she felt more secure at a facility. In general, women had been well educated by the HHPs and had a good relationship with them. The percentage of mothers surveyed at endline who were able to name at least two post-partum danger signs that would require them to seek immediate medical attention was 73%, compared to 32% at baseline.

FIGURE 1: Difference between baseline and endline of maternal-related variables, Warrap State, South Sudan, 2012–2014.

Child care: Similarly, all variables related to care-seeking behaviour and treatment coverage for diseases measured showed statistically significant increases (see Figure 2); thus care-seeking and treatment for fever increased from baseline to endline with 22 and 25 percentage points respectively. Care-seeking for acute respiratory infection (ARI) increased by 27%. Fifty-two percent were taken for care within two days of onset of symptoms, compared to only 19% at baseline; a 33% increase. The amount of treated drinking water increased by 38 percentage points from baseline. Three out of every four children with diarrhoea seek care and received oral rehydration (ORS) treatment, with a 49 percentage point difference from baseline.

FIGURE 2: Difference between baseline and endline of child-related variables, Warrap State, South Sudan, 2012–2014.

The MaCHT project aimed to strengthen the capacity of the health system to deliver essential MCH services through building the supply and performance of the health workforce.

Increasing the health workforce supply was primarily achieved by a combination of recruitment and task-shifting to mitigate the acute shortage, particularly at community level (e.g. through supporting the establishment and training of HHPs). Improving the performance of the health workforce was achieved through in-service training (at health facility and community level), and performance appraisals (focused on community level service delivery).

A review of the project's monitoring data showed that over 100 women were trained as HHPs to deliver evidence-based interventions and provide education around MCH. Qualitative data revealed that women with children under two learned new information from the HHPs. One woman reported that she had learned ‘many things: that when you deliver, you set up a mosquito net around the baby, and to breastfeed immediate. They also demonstrate how to clean their homes.’ Another woman said, ‘When you deliver the HHP will come to advise about immunisations, and to have ANC monthly. The HHPs also evaluates and advises to go to facility if they have no drug. She knows more than her other village mates.’ A third woman noted that in villages where there are no HHPs; the parents do not know to take their children to someone for examination.

Another strategy was pre-service education and training, which was implemented at health facility level through midwifery training. The MaCHT project supported five community-selected candidates for a two-year midwifery training. Once trained, they were deployed to Warrap State MOH, to help fill gaps in the PHCCs and facilities without midwives. Training of health facility personnel in HBLSS and iCCM was done in ten health facilities to improve technical capacity. Likewise, project activities tangentially supported another component of the health system such as equipment and essential drugs. Obstetric delivery kits were supplied to PHCUs as part of basic infection control measures.

Due to contextual factors affecting the government's health system (violent conflict and government instability) project activities related to drug procurement and availability of transportation did not find a viable operational framework on which to integrate, leading to the creation of parallel tracks to address drug supply needs and mobilisation of health personnel. As a result, there was a lack of sustainable achievements in these areas.

The evaluation found that the project led to increased community awareness about the need for maternal care and supported the government's efforts to reduce maternal and infant mortality in Gogrial East and West counties (14 key in-depth interviews and two FGDs with project stakeholders, beneficiaries and community volunteers).

In addition, the MaCHT project has paved the way for a hospital improvement project, specifically to ensure Emergency Obstetric Care (EMOC) is available for residents of this geographical area.

In tandem with government priorities for rural areas (where HR shortage is highest), most projects’ efforts focusing on community structures were at the level of HHPs. The overall effort was devoted to increasing HHP capacity to deliver high impact essential interventions and to strengthen operational linkages with PHCU. Thus, an evidence-based medical action plan (iCCM plus algorithms) plus its educational curriculum and teaching methodology, supported by clinical and teaching equipment, was collaboratively developed and/or adapted by the relevant branches of the INGO for use in the context of rural Warrap State. Community-based transport committees were organised and linked to the eight health facilities (seven PHCU and one PHCC⁸). Likewise, within communities included in the project geographical area, mother-groups⁹ were organised around MCH.

The rebuilding of routine public health services should be the main goal of post-conflict reconstruction. Nonetheless, most often during the transition phase following a post-conflict situation, access to health services suffers a contraction due to the slow replacement of sustainable and quality health services by public health services. The MaCHT project described in this paper, and implemented by an INGO in close collaboration with the South Sudan Government, was found to be largely successful in achieving levels of essential health service coverage surpassing the overall basic health service coverage estimated at 40% in South Sudan (Canavan, Vergeer & Bormenisza 2008), and with funding at 25% of the level recommended by the 2001 Commission for Macroeconomics and Health.¹⁰ Furthermore, according to a Lives Saved Tool (LiST) analysis¹¹ of six prevention and treatment interventions (see Figure 2 for a list of the interventions), there was an overall 7.7% decline on under-5 child mortality over a three-year period (2.6% of annual rate of decline).¹²

Over 100 HHPs were trained by the project and were found to be very effective in improving knowledge amongst community members, assessing mothers and children, and initiating treatment for malaria and diarrhoea.

Qualitative results strongly suggest that the MaCHT project increased the awareness and motivation for the MoH at the state and local level to refocus its efforts on MCH projects in the area. The Health Pooled Fund project,¹³ supported by multiple donors, is actively pursuing the improvement of the Kuajok hospital (district) as a direct effect of program activities. The successful training of HHPs and the dissemination of research results pertaining to their effectiveness has garnered interest from across the health system and suggests that the project has repositioned the maternal, neonatal and child health agenda.

The quality of MCH education was improved in the project area. Over the life of the project, the components of the health system affected positively were human resources (supply and performance), service delivery, and information system. The procurement and distribution of essential drugs and other health supplies was not integrated into the national health system due to limitations associated with contextual factors outside the control of the project; resulting in the creation, as a secondary effect, a parallel system mostly supported by the NGO. Governance and financial components were not the focus of the project, and were thus not measured.

The implementation of this project was affected by the effects of at least three major contextual issues – infrastructure, political/social instability and human resource capacity. South Sudan has existed as a country for less than five years. In addition to the aftermath of a long period of war with Sudan, this newly independent country has faced numerous challenges, including gaps in health infrastructure to meet the needs of the influx of people from Sudan to South Sudan, and relief efforts taking priority over development efforts. In addition to challenges inherent to integrating with a nascent national health system in the process of building capacity at all levels, the project encountered numerous setbacks. Amongst these were chronic supply chain disruptions, including shortages of drugs, and evacuation of project staff from the project area and the national office as a result of civil unrest. The large relief efforts implemented to care for the large number of people displaced by the civil conflict shifted some of the shared resources to these more urgent issues.

Uncontrolled cross-sectional before-and-after studies are intrinsically weak evaluative designs because attribution of observed change to specific intervention is non-feasible. Likewise, over-estimation of results has been documented with this type of research design; thus inference of overall results should be done with caution. Data collection for the baseline and endline survey was affected by several factors, including the distance to villages, difficult terrain, short supply of vehicles, and early nightfall. Additionally, as the data for household practices was based on mothers’ responses and not observation, over-reporting of recommended practices cannot be ruled out. The qualitative survey was limited by the number of beneficiaries and volunteers that could be interviewed for the project. Impediments to this process included weather, long travel times, and number of days for the on-site qualitative evaluation process.

INGOs’ efforts and resources can and should be aligned with governments’ initiatives for strengthening local and district health systems. In general, these private efforts and resources should focus on four health systems strengthening components: health workforce, service delivery, health information and medical supplies and equipment. Specific areas of support include: improving partnerships and communication with the national and state MoH to build capacity and implement strategies; integrating community-based resources and activities into new projects to rapidly increase health coverage of essential interventions; addressing supply chain and infrastructure shortcomings; and filling gaps in staff shortages, turnover, and capacity.

INGOs can have an important role in complementing national health systems’ strengthening efforts. Weak health systems are a common challenge faced by many developing countries, particularly those with fragile-state conditions. Our findings suggest that community-based models supported by INGOs in fragile-state context effectively and efficiently complement government efforts in the areas of service delivery, human resources, information and, with due caveats, supply of medical products.

In South Sudan, community-based health services provided via HHPs based on iCCM are a viable model for increasing MNCH services and practices, and improving access in an overstretched system. Further comparison design research plus additional investments in packaging the model can help prepare it for scale-up.

The work presented in this article has been previously submitted to the USAID for publication, and they have a royalty-free, nonexclusive and irrevocable right to reproduce, publish, or otherwise use the work for federal purposes, and to authorise others to do so.

The authors wish to acknowledge the funding support of the United States Agency for International Development's Child Survival and Health Grants Program.

A.C.R. (World Vision, Inc.) designed operational research component, monitored project implementation, and principal writer of the paper. F.W.J.A. (Independent Evaluator) led and reported on the evaluation. D.T.C. (World Vision, Inc.) designed the project into which the study was embedded and made significant conceptual contributions to the study. E.W. (World Vision South Sudan) contributed to the research report as field project manager. J.A.H. (World Vision Inc.) provided support to operations research and reporting. R.H. (World Vision, Inc.) provided support on LIST analysis and reporting.

The authors declare that they have no financial or personal relationship(s) that may have inappropriately influenced them in writing this article.

1. Fact sheet: Progress on the Millennium Development Goals. [http://.go.worldbank.org/4TZL5TGCW0.

2. The Maternal and Child Health Transformation Project was funded through the United States Agency for International Development's Child Survival and Health Grants Program from 2011–2014.

3. Now five: referral hospitals are now tiered as county, state, and teaching hospitals.

4. http://www.who.int/gho/countries/ssd/en/.

5. Health Strategic Plan (2011-2015). Government of South Sudan Ministry of Health.

6. High impact low cost health interventions with proven evidence included in the program were: vaccination with tetanus toxoid, antimalarial intermittent preventive treatment in pregnancy, skilled birth attendance, use of impregnated bed-nets (mother and child), iron supplements, active management of third stage of labor, ORS, antibiotics for pneumonia, antimalarial, and water and sanitation (Bellagio Child Survival Study Group; Lancet 2003; 363:65–71).

7. Active management of third stage of labour involves three components: an injection to prevent excess bleeding, the birth attendant placing pressure on the mother's abdomen while pulling on the umbilical cord, and massage of the abdomen following delivery of the placenta.

8. During the life of the project, only one PHCC was linked to community-based transport committees due to budget limitations.

9. Mother groups are organised groups made up of mothers who support HHPs and PHCU outreach activities in fostering positive maternal, newborn and young child health and nutrition practices within their communities.

10. The 2001 Commission for Macroeconomics and Health stipulated $34/person/year to achieve coverage of essential health services.

11. LiST is a child survival modeling tool that uses country level under-5 mortality rates and cause of death profiles, and models the effects of change in coverage of interventions on overall and cause-specific mortality rates for children under 5 years of age (http://www.jhsph.edu/dept/ih/IIP/list/spectrum.html).

12. According to a Lives Saved Tool (LIST), analysis of results of the project (not included in this publication), the project achieved a 7.7% reduction in under-five mortality rate over a period of three years of intervention coverage.

13. Health Pooled Fund South Sudan is financing with $1.9 million over 18 moth-periods the Kuajok hospital (Warrap State) to provide emergency obstetric and newborn care.