Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
112,008 characters · 20 sections · 71 citation commands
Community Matters: Heterogeneous Impacts of a Sanitation Intervention
Public health interventions are often promoted as an instrumental driver of behavioral change and adoption of health products. But how effective are these programs in developing countries, especially when implemented at scale? We investigate this question in the context of a participatory community-level intervention that has been widely introduced by governments around the world to improve access to safe sanitation.
Comprehensive water and sanitation programmes in developed nations in the early 1900s have been dubbed the most effective public health intervention of the last century AlsanGoldin2019 due to their significant impact on infant survival and reduction of communicable diseases. The costs of poor sanitation and the disease environments they create - in terms of child health, mortality, human capital accumulation, and economic growth - are, by now, well understood Adukia2017,Alzua2018ASanitation,SpearsLamba2016,AugLes2018,Bairdetal2016.\footnote{ Low and middle income countries carry a disproportionately heavy burden for communicable diseases, such as diarrhea PrussUstunCorvalan2006. Better sanitation could, for example, prevent the majority of diarrhea-related deaths of 361,000 children aged less than 5 years each year pruss2014burden, and would reduce the prevalence of worm infections in school children, which affect educational attainment and labor market outcomes Bairdetal2016. For an overview, see also UnitedNations2016CleanMatters,WSP2012} Yet, 4.5 billion people still lack access to safely managed sanitation worldwide WHOUNICEFJMP2017. Improving access to sanitation has thus been recognized as a goal towards sustainable development by the UN. To achieve it, identifying effective policies that work in low income countries is key.
In this paper, we show the results of a cluster randomized controlled trial (RCT) we implemented to assess the Government of Nigeria's `National Strategy for Scaling up Sanitation and Hygiene'.\footnote{ Nigeria provides a suitable context to study sanitation as 34% of its population practices open defecation, toilet ownership rates have stagnated unicef2015 and the country contributes to a significant share of the global population without access to adequate sanitation WHOUNICEFJMP2017.} We evaluate the main pillar of the governmental strategy, an intervention known as Community-Led Total Sanitation (CLTS), at scale. CLTS entails community meetings and the provision of information with the aim of eradicating open defecation (OD), by triggering a desire for collective behavioral change and encouraging communities to construct and use toilets. As part of the experiment, CLTS was implemented in a random sample of 125 of 247 study clusters of rural communities, located in the Nigerian states Ekiti and Enugu. In each cluster, we randomly selected 20 households for interview at baseline in 2014 and conducted three follow-up surveys with this sample 8, 24 and 32 months after implementation. The resulting balanced panel contains more than 4,500 households.
Our results show that, on average, CLTS led to small and temporary reductions in OD among households in treated communities, i.e. a reduction by 3 percentage points (pp) over a period of 24 months. However, these estimates hide meaningful impact heterogeneity across population subgroups: we find that intervention impacts are considerably stronger among, and restricted to, the asset poorest half of the studied communities. In these communities, OD fell by three times as much as on average, i.e. 9 percentage points in the short-run (8 months post implementation), and this sanitation improvement is sustained over time. The reduction in OD is achieved mainly through household sanitation investments, i.e. the construction of new toilets. The effect cannot be ascribed to pre-treatment differences between communities in toilet ownership. Neither do we find evidence that community differences in social capital or social interactions, public goods infrastructure or leader characteristics explain our results. Yet, communities' socio-economic status clearly matters for the intervention effectiveness. Community-specific impact estimates along three alternative socio-economic dimensions of communities - geographic isolation, low population density and low average night light intensity (obtained from satellite measurements) - yield estimates of similar magnitude to those for community wealth.
Additionally we use results from the Nigerian RCT and data from five recently published CLTS trials conducted in other countries bricenoEtAl2017,Patil2014TheTrial,Cameron2019,pickering2015effect,guiteras2015encouraging to i) demonstrate that such heterogeneous impacts can also be found in other contexts, and to ii) show an inverse relationship between area-level wealth and program effectiveness beyond the Nigerian context. This provides external validity to our findings and rationalizes CLTS impact estimates across studies.
Our results have four important implications. First, we show that public health information interventions such as CLTS can indeed trigger behavioral change in (some) population groups, in line with results surveyed in dupas2011health. Second, and consistent with CLTS practitioners' experiences and priors as to where the intervention should work best kar2008handbook, we pinpoint community wealth as a key factor for the effectiveness of this participatory community-level information intervention.\footnote{Recent studies have tried to identify the underlying factors that drive heterogeneous impacts of interventions across and within studies, such as Meager2018UnderstandingExperiments,Bandiera2018SocialServices, cunha2018price.} We show that heterogeneous impacts by communities' socio-economic characteristics dominate impact heterogeneity at the household level.
Third, our results have important implications for the scale-up of CLTS. To be able to draw policy relevant conclusions regarding a wider rollout Muralidharan2017, we implemented CLTS at scale. Government officials working in WASH units (Water, Sanitation and Hygiene) were trained to deliver the intervention. We argue, based on our findings, that a targeted implementation of CLTS may yield larger policy returns than a blanket approach. Measures of wealth are readily available in standard household surveys, or alternatively nightlight intensity indices from open access satellite data. Governments can easily combine these data with findings from RCTs to develop a more effective targeting strategy for this popular sanitation program, which is currently implemented in more than 25 Latin American, Asian and African countries. As an example, we demonstrate this using the 2013 Nigerian Demographic and Health Survey (DHS), and show that the data can replicate the classification of geographical areas into poorer and (modestly) richer ones with a precision that is similar to the measures obtained from our primary study data.
Finally, we demonstrate how heterogeneous intervention impacts can help address the widely acknowledged concern that the high internal validity that characterises RCTs comes with important shortcomings in external validity Basu2014,DeatonCartwright2018,PetersEtAl2018. WangEtAl2006 and Meager2018UnderstandingExperiments emphasize context or location-specific factors as obstacles to external validity. We present local socio-economic status, particularly community wealth, as common underlying factor for CLTS effectiveness beyond the Nigerian setting of our RCT and, in fact, across various diverse contexts. The results from five recently published CLTS trials bricenoEtAl2017,Patil2014TheTrial,Cameron2019,pickering2015effect,guiteras2015encouraging range from very large impacts - an increase in toilet ownership of 30 percentage points, and an open defecation reduction of 23 percentage points - in a trial in Mali pickering2015effect, to no detectable impacts on toilet ownership in Bangladesh guiteras2015encouraging. We estimate heterogeneous impacts for each of these evaluations using a harmonised method and pair impact estimates with a proxy of local socio-economic characteristics available for all studies, namely average night light intensity at baseline. The inverse relationship between area-level wealth and program effectiveness that we find in this exercise rationalizes the existing range of CLTS impact estimates across studies. We conclude, in the spirit of BanerjeeEtAl2017Chapter4, that our result from Nigeria carries plausible external validity across various contexts.
The remainder of the paper is structured as follows. In the next section we describe the intervention and the experimental design. In Section 3, we present the empirical method and Section 4 presents our impact estimates. Section 5 compares the results of our study to those of other CLTS interventions. Section 6 concludes.
The Government of Nigeria adopted Community-Led Total Sanitation as its major approach for the development of rural sanitation within its Strategy for `Scaling up Sanitation and Hygiene', launched in 2007. This decision followed three years of piloting, conducted by the National Task Group for Sanitation in collaboration with state and local governments as well as local and international NGOs such as WaterAid and UNICEF. The effort to scale-up CLTS to the whole country began in 2008. We study the effectiveness of CLTS in the context of Nigeria's national strategy through a cluster-randomized controlled trial conducted in nine local government areas (LGAs) in the states of Enugu and Ekiti that did not have any recent experience of CLTS, or CLTS-like interventions.
CLTS is a community-level information and mobilization intervention aimed at reducing open defecation and improving toilet coverage. It is typically implemented in three steps. The firsts step focuses on mobilization: Community leaders are approached and engaged in a discussion about the negative health implications of OD\footnote{ A further key message is the importance of sanitation externalities, i.e. that all community members (particularly children) are at risk of contracting sanitation-related diseases if some residents practise open defecation.}, as well as the potential benefits of CLTS in achieving behavioral change within their communities. The aim of the meeting is to convince community leaders to arrange a community meeting. This meeting, the so-called `triggering meeting', marks the second step, and the main component of CLTS. The meeting starts once and only if a significant number of community members gathered in a predefined public space on the identified day. The first activity is typically a community mapping exercise, in which each attending community member marks their household's location and toilet ownership status on a stylized map on the ground. Community members next identify regular OD sites and mark these as well. In many cases, this exercise is used by facilitators to follow up with graphic images showing that the community lives in an environment contaminated by feces. Facilitators of the meetings further use the map to trace the community's contamination paths of human feces into water supplies and food.\footnote{ A number of other activities may follow, at the discretion of the facilitator. Examples include transect walks through the community (often referred to as `walks of shame'), pointing out visible feces in the environment to evoke further disgust and shame; medical expense calculations related to illnesses likely induced by OD practices; or graphic exercises, where facilitators might add feces to drinking water, illustrating that these are not necessarily visible to the naked eye. In the context of our study, only about 20% of triggering meetings included at least one such additional exercise, graphic illustration being the most popular one (implemented in 14% of triggering meetings) followed by expense calculations (7%).}
As a closing task, attendees are asked to draw up a community action plan to achieve community-level open defecation free (ODF) status. This aspect of CLTS seeks to foster collective action and collaboration. It includes discussions of how poor or vulnerable households can be supported to achieve ODF status. The action plan is posted in a public spot. Volunteers (so-called `natural leaders') are chosen to follow up regularly on each attendee's commitment towards implementing the plan, the thirs step of CLTS. Followup visits by the facilitators are organized. Eventually, the community might be certified for its achievements by the national Rural Water Supply and Sanitation Agency (RWASSA) and the National Task Group on Sanitation (NTGS).
CLTS does not offer any monetary incentives, subsidies or credit to finance toilet construction or reward OD reductions or ODF achievement, nor is technical assistance or hardware provided. Neither does it promote a particular toilet technology. The aim is to drive a change in sanitation practices purely by altering the perceived costs of unsafe sanitation and the perceived benefits of toilet use.
We study the effectiveness of CLTS in the context of Nigeria's national strategy through an RCT conducted in collaboration with the international non-governmental organisation (NGO) WaterAid.\footnote{ The study protocol was approved by the following IRBs: National Health Research Ethics Committee, Federal Ministry of Health, Nigeria (NHREC/01/01/2007-20-20/11/2014), University College London Ethics Committee (2168/009). The trial was registered at the ISRCTN registry (ISRCTN74165567). We note that the research project intended to also evaluate a supply intervention, Sanitation Marketing (SanMark). However, SanMark development and piloting took longer than planned and implementation had been in place not long enough at the time of the endline survey to conduct a full impact analysis. Details are outlined in the project's final report AbramovskyEtAl2018.} WaterAid worked closely with local government areas (LGAs)\footnote{ LGAs are Nigeria's second sub-division after states. LGAs are administrative divisions led by a Local Government Council.} and two local resource agencies (NGOs from Ekiti and Enugu) to implement CLTS in two of Nigeria's 36 states, Ekiti, Enugu.\footnote{ A third state, Jigawa, was dropped due to budget limitations and security concerns.} Specifically, WaterAid Nigeria and the two local NGOs conducted CLTS training sessions, one in each state. These sessions trained the LGA water, sanitation and hygiene (WASH) units, which are part of Nigeria's public service, who then conducted the CLTS mobilization, triggering meetings and followup activities. Four LGAs without recent experience of CLTS, or CLTS-like interventions, were selected in Enugu and 5 in Ekiti. Their locations are indicated in Figure (ref).\footnote{ Study LGAs in Enugu are Igbo Eze North, Igbo Eze South, Nkanu East and Udenu. In Ekiti, Ido Osi, Ikole, Moba, Irepodun Ifelodun and Ekiti South West are part of the study.}
Since CLTS is a community-level intervention, we used what we refer to as `triggerable clusters' as the unit of randomization. Triggerable clusters are groups of geographically close villages, neighbourhoods or quarters. These clusters were defined jointly by the researchers and the implementers. Hence, they do not match any of Nigeria's administrative units. Clusters were chosen with the view of reducing information spillover, e.g. to be self-contained units so that information about triggering activities would not spread to the next cluster. This implies, for example, that triggerable clusters do not share markets or large public areas. To further reduce potential contamination between experimental treatment and control clusters, `buffer' areas were introduced around triggerable clusters to ensure that no two clusters were located in close geographic proximity. No specific distance was imposed. The definition of the `triggerable clusters' and their buffer zones was driven by implementers' previous experiences from working in these areas.
In both study states, a cluster comprises of on average 1.7 villages or quarters\footnote{ The median and modal number of villages or quarters within a cluster is 1. The maximum number of villages in a cluster is 7, occurring only once.}, all of which CLTS was implemented at the same time. The treatment period during which mobilization and triggering activities took place lasted about six months -- between January and June 2015.
In total, we identified 247 `triggerable clusters'. 246 of these were randomized with equal probability into either receiving CLTS (treatment) or not receiving it during the course of the study (control).\footnote{ One cluster from the original sampling was subsequently dropped since data collection in any post-treatment survey wave was not possible due to civil unrest in the community.} Randomization was stratified by LGA. The distribution of treatment and control clusters is presented in Table (ref).
We collected five rounds of data over a time span of almost three and a half years.\footnote{Data collection was carried out by an independent data collection company, blinded to treatment status.} The sampling frame was established in October 2014 through the first round of data collection, a household census in the nine participating LGAs from Ekiti and Enugu. The census collected basic household information from 50,333 households (27,888 from Enugu and 22,445 from Ekiti).\footnote{ Initial estimates of the size of triggering clusters in the implementation plan were around 150 households. In the field, cluster sizes turned out to be larger. Budget limitations constrained the listing to a maximum of 180 households per cluster. To achieve a representative sample of households, we adopted the following approach: For each triggerable cluster, we randomly ordered the villages/quarters and started the listing exercise at the top of the list. The first village/quarter was listed completely, independent of its size (i.e. going above the 180 households threshold if needed). If less than 150 households were listed, data collection would continue in the second village/quarter on the list, again listing every household in this village/quarter. This process continued until either all villages/quarters were listed in each cluster, or until around 180 households were reached. This approach ensured that we have listing data from each of the study clusters, that our overall sample remains representative for the study area (since the ordering of village/quarter listing was randomly determined), and furthermore that whole villages and quarters were listed while keeping within budget.} Based on this census, we randomly selected 20 households from each cluster for interview. Our sample is thus a representative panel of households in the nine LGAs. This is in contrast to other studies which restricted their samples to households with children (Cameron2019,briceno2015promoting,pickering2015effect). Our final sample consists of 4,671 households in the 246 clusters, distributed evenly across Ekiti and Enugu, and covering around 9% of the population in the study area. A baseline survey was conducted during December 2014 and January 2015. The first follow-up survey (FU1) took place between December 2015 and February 2016 eight months after implementation, on average. FU2 took place 24 months after implementation (March-April 2017) and FU3 (which we also refer to as the `endline survey') after 32 months (between November 2017 and January 2018). The three followup surveys allow us to study the dynamics of CLTS impacts over time, providing insight into the sustainability of program impacts. Figure (ref) summarizes intervention and data collection timings.
Our study had low attrition rates across the three followup survey rounds: 2.53% at FU1, 8.81% at FU2 and 11.58% at FU3 as shown in Appendix Table (ref). A possible concern is that non-response was not random across treatment and control villages, which could bias estimated treatment effects. The lower panel of Table (ref) shows that there is no significant difference between attrition levels in the control and intervention clusters at the time of the endline survey, FU3, with attrition rates of 11.14 for households in control and 12.05 in treatment areas. The same conclusion holds for differential attrition by treatment status during FU1 and FU2 (Appendix Table (ref)). To investigate this more formally, we test whether treatment status can predict attrition conditional on district fixed effects and household-level baseline characteristics (Appendix Table (ref)). Overall, the coefficient of the treatment dummy is close to zero and never statistically significant in any of the three followup surveys or estimated specifications. This suggests that selective attrition is not a concern. We therefore focus our analysis on the balanced panel of 4,540 households which were successfully interviewed in each round of data collection.
As the allocation of clusters to treatment and control was random, we expect no systematic differences between both groups at the time of the baseline survey. We check balancedness in Table (ref), presenting summary statistics measured at baseline for the main characteristics of households, the outcomes we consider and characteristics of the study communities. We present the baseline mean for the control group (in the post attrition sample), the difference in means between treatment and control group and report the p-value for a t-test of equality of these means in the last column.
As expected, there are no statistically significant differences between the two groups along the presented dimensions,\footnote{ The project's baseline report supports balancedness on a wide range of additional variables Abramovsky2015BaselineNigeria.} except a small (0.27) difference in the number of household members. In an F-test of joint significance of all characteristics, we reject the null hypothesis at the 5% level (p-value=0.038), and hence include this variable as a covariate throughout our analysis. Yet, once we remove household size, the explanatory power of the remaining variables falls markedly (p-value=0.27), supporting the validity of our randomization strategy (except regarding household size).
The first panel of Table (ref) reports household characteristics. Household heads in study communities are predominantly male (64%), with a mean age of 55 years. Most completed primary school (68%) and were employed (77%). Average household size was four, with 30% living with a child under the age of six. For 45%, farming was the main activity. We also present a household asset wealth indicator, measured as the first factor of a principal component analysis based on a series of questions regarding asset ownership.\footnote{ Details of its components and their factor loadings are provided in Table (ref) in Appendix (ref).} While the number value of this variable is in itself not meaningful, it is balanced across experimental arms.
The second panel presents the main outcomes, all measured at the household level. Our primary interest lies in households' sanitation behavior, in particular open defecation practices, for which we use two measures. The first captures whether the main survey respondent (typically a woman) states that at least one household member above the age of four years performs OD. The second variable takes the value one if the respondent herself declares to perform OD.\footnote{ Both rely on the question “Where do you go to defecate?”, combined with a showcard of possible places. Urination habits are asked separately at baseline, but are not used here.} Table (ref) shows that in our study population, 62.8% of households have at least one member defecating in the open and a very similar percentage (62.4%) of women report that they follow this practice themselves.
We find that the habit of defecating in the open is closely linked to lack of toilet ownership. Ownership and use of private toilets are the most frequently discussed channels to reduce OD in CLTS community meetings kar2003. Yet, only 36.9% of households own a toilet, 36.1% own a functioning toilet of any type, and 32.4% own a functioning and improved toilet at baseline. We include all three measures in our analysis as outcomes since they capture different dimensions of interest. The first records ownership, but ignores functionality. The second additionally captures maintenance investments into the existing stock of toilets, or toilet divestment through lack of maintenance. The third accounts for quality, satisfying the stricter criteria set by the WHO/UNICEF Joint Monitoring Program regarding improved sanitation.
To validate households' reports of toilet ownership, interviewers asked respondents whether they may inspect the reported latrines at the endline interviews. If systematic measurement error was present in the treatment group as a result of CLTS, particularly due to over-reporting of “desirable” outcomes, then we would expect lower consent rates and higher rates of corrections in treatment relative to control areas at endline. However, we find this not to be the case: 24% of households withheld their inspection consent, with no difference between treatment and control groups (24.86% in control; 23.85% in the treatment group, p-value=0.693). Inspection did not yield significant discrepancies between reported and actual ownership, and, if anything the correction rate is lower in the treatment group (7.98%) than in control (9.56%). None of these measures are statistically significantly different between treatment arms. We conclude that there is no evidence of selective reporting differences between treatment and control group in either dimension - consent rates for inspection and truthful reports of ownership by consenting households. In additional sensitivity analysis, we exclude households who refused the validation of their toilets, and re-estimate average and heterogeneous impact estimates; all parameter estimates are virtually unchanged in the restricted sample.
CLTS impacts on reported open defecation practice post intervention, which the interviewers cannot monitor directly, are closely mirrored by reverse changes in (partially verifiable) toilet ownership (see results in Section 4.2). This is in line with the close relationship observed between ownership and OD behaviour at baseline: 98% of (functioning) toilet owners report that their household does not practice OD (see Table (ref)). This percentage remains very similar after the intervention (95%), and does not differ between households in treatment and control groups (p-value=0.345).\footnote{ This is different from OD habits in India for example, where toilet ownership and usage does not necessarily go hand in hand (see for example GuptaEtAl2019)}
The third panel of Table (ref) finally presents statistics for a set of community characteristics. The communities are located on average about seven kilometers from the LGA head quarters (HQ)\footnote{ This is a walk of about an hour.} and encompass 1,616 households within a five kilometer radius. Part of our analysis, described in detail in Section (ref), will focus on heterogeneity in CLTS effectiveness by community socio-economic status (SES). Both the distance to the LGA HQ and number of households within a 5k radius are often used as proxies for SES status: distance to urban or semi-urban centers is typically used as a measure of remoteness or isolation, and the number of households within a 5km radius, as a measure of population density. We present here, and use later, summary statistics of two further SES proxies, namely a community wealth index based on the median household wealth in the community,\footnote{ Detailed lists of household asset items are frequently elicited in household surveys in developing countries, as they are often more precise than measures of household income. The aggregated index is mean zero and has a standard deviation of one.} and a pre-intervention nightlight intensity index within a 5km radius, a proxy for local economic wealth and income.\footnote{ Michalopoulos2013 presents evidence that wealth and nightlight intensity are strongly correlated.} We find that average nightlight intensity in our study area is very low with a mean of two relative to the total nightlight range of 0 to 61.
The next set of community characteristics we present relate to social interactions within the community, a dimension that has also been suggested as accelerating the effectiveness of CLTS (see for example Cameron2019). We capture social interactions in three ways: a community's level of i) trust, ii) social capital, and iii) religious fragmentation. Trust is the average community score of household measures of the degree to which they trust their neighbors. Social capital is constructed similarly, based on households' participation in community events. We adapt the measures used in studies of ethnolinguistic fragmentation (ELF) to capture religious fragmentation, as our study sample is homogeneous along ethnic lines but very diverse in terms of religion. Detailed definitions of these measures and their distribution can be found in Appendix (ref). We find that religious fragmentation at 60% was higher than in 70% of the countries recorded in Alesina's 2003 fractionalization dataset, and people somewhat trusted their neighbors (i.e about 0.9 on a 0 to 2 scale).\footnote{ Mean social capital and community wealth are by construction close to zero.}
Finally, in line with household level averages, the mean toilet ownership rates in our study clusters is 36.2%. All community characteristics are balanced across treatment and control.
We estimate the impacts of CLTS on our primary outcome, open defecation practices, using an intent-to-treat (ITT) design based on cluster randomized assignment to treatment.\footnote{ Intent-to-treat designs are informative about the key parameters of interest and often carry more external validity than estimates based on treatment recipiency, since perfect take-up of interventions is rare. As abramovsky2016 discuss, while leaders in all communities were approached, triggering meetings were not held in 18 communities (14% of all treatment communities) due to insufficient number of community members coming to the planned event. In a successfully randomized scenario, as is our study (see Table (ref)), ITT designs yield unbiased estimates of the average impact of the intervention on the sample assigned to treatment. For robustness purposes, in Appendix (ref) we show that there is no evidence of selective triggering in our study. Additionally, we follow Imbens1994IdentificationEffects and Angrist1995Two-stageIntensity and instrument triggered treatment with treatment assignment. The results are very similar to the ITT estimates (see Appendix (ref)).} We compare open defecation practices $y_{ict}$ in household $i$ living in community (cluster) $c$ in period $t$ by treatment assignment:
where community-level CLTS treatment status is defined by $T_{c}$. Baseline characteristics of households and their heads, $X_{ic0}$, are included to maximize precision and account for the imbalance in household size observed at baseline. To filter out unobserved area effects and contemporaneous shocks, we include LGA and survey wave fixed effects, $\omega_{g}$ and $\delta_{t}$. The parameter of interest, $\gamma$, captures the average impact of CLTS.
Our preferred specification conditions on the baseline value of the outcome variable, $y_{ic0}$. The resulting ANCOVA estimates are more efficient than difference-in-difference and simple difference estimators in experimental contexts, when pre-treatment information is available and the outcome is strongly correlated over time (mckenzie2012beyond). Alongside, we present conventional difference-in difference estimates.
To investigate heterogeneous impacts, we expand the specification in Equation ((ref)):
We introduce a binary variable $CC_{c}$ indicating a community characteristic, say community wealth, split our sample of communities along its median, and include the interaction term $T_{c} \times CC_{c}$. $\gamma_{r}$ is the average CLTS treatment effect in the richer half of communities (i.e. those for which $CC_{c}=0$), and $\gamma_{d}$ is the difference in treatment effects between rich and poor communities (i.e. those for which $CC_{c}=1$). The communities in our sample are typically located towards the middle (4th to 7th decile) of the Nigerian community wealth distribution (see Appendix (ref)), rather than in the tails. Hence, neither communities are rich nor very poor relative to the Nigerian distribution. We simply denote the upper (lower) half of our study communities in the remainder of the text as “rich” (poor) communities for ease of description,
Since we are testing multiple hypotheses simultaneously in our analysis of heterogeneous impacts, we report p-values that are adjusted for the family-wise error rate in brackets. We compute these using the methodology proposed by romano2005stepwise. Naive or unadjusted p-values obtained from individual significant tests for each point estimate, calculated by drawing 1,000 clustered bootstrapped samples, are shown in parentheses.
We show average impact estimates on OD practices in Table (ref). The first dependent variable is a dummy equal to 1 if at least one household member above the age of four performs OD (columns 1-3)\footnote{ Note that this question was not asked at the first follow-up survey, denoted FU1.}, the second a dummy variable equal to 1 if the main respondent performs OD (columns 4-6). Columns 1-2 (4-5) present difference-in-difference estimates without (with) household characteristics. Columns 3 and 6 present the ANCOVA specification described in equation (ref), including household characteristics.
Panel A pools observations across the three followup surveys. We find that CLTS reduced OD consistently across all specifications and for both OD measures. Yet, the magnitude of behavioural change is small (around 3pp). As expected, the estimated coefficients are identical across specifications, but precision is highest when accounting for the lagged dependent variable. This being our preferred specification, we reject the null hypothesis of zero impact at the 10% level for both measures of OD: exposure to CLTS resulted in a reduction in OD by 3pp.
Due to possible divestment in toilet maintenance, as discussed in section (ref), or short-lived behavioral changes in OD, CLTS impacts may not be sustained over time. We thus investigate time dynamics in impacts, considering impacts estimates approximately 8 (FU1), 24 (FU2), and 32 (FU3) months after CLTS implementation. Panel B of Table (ref) reveals that point estimates are of similar magnitude throughout the period of study, and are statistically significant for the first and second followup waves. These estimates point towards a short-run reduction in OD eight months after CLTS. This reduction is sustained over two years after intervention implementation, but then fades out.
CLTS is designed and implemented as a participatory intervention at the community-level, with the aim of bringing about collective change. This raises the question of whether community characteristics hinder or foster intervention effectiveness. In spite of its current popularity, there is still scant experimental evidence where and under which conditions CLTS works. If CLTS is more (or only) effective in certain settings, successful targeting requires an understanding of the characteristics that best predict its effectiveness.
The CLTS Handbook, a practitioners' guide drawing on field experience from 16 countries, suggests that the impact of CLTS on sanitation outcomes may depend on the socio-economic status (SES) of treated communities kar2008handbook. It posits that successful implementation of CLTS is more likely in rural communities that are small, culturally and socially homogeneous, are located in remote areas and have a high prevalence of OD. In addition, Cameron2019 emphasize communities' social capital as a key facilitator for CLTS impact. Studies in the field of experimental economics have drawn similar conclusions, highlighting the importance of community rather than individual level differences in determinants of variations in concepts such as fairness and altruism HeinrichEtAl2006,HeinrichEtAl2001, a conclusion made also in the context of Nigeria GowdyEtAl2003.
Following these hypotheses, we define four broad indicators of local socio-economic status that may mediate CLTS impacts. In Section (ref), we introduced community wealth as a widely available, comprehensive proxy for local socio-economic status (SES). We explore whether CLTS impacts vary by this SES proxy as well as using three additional proxy measures: i) night light intensity, ii) population density, and iii) isolation.\footnote{ Details on all four measures are available in Appendix (ref).}
The pairwise correlations between the four characteristics demonstrate that poor communities, i.e. those below median wealth, are indeed often remote, less densely populated and have lower night light activity (see Table (ref)). However, some pairwise correlations are relatively low, suggesting that each measure may capture a different aspect of socio-economic conditions. For instance, while population density and isolation appear to be highly correlated (rho = -0.58), the correlation between community asset wealth and average night light intensity is lower (rho = 0.16).\footnote{ Note that the inverse relationship between isolation and density reflects that higher population density is associated with shorter distances to the nearest LGA capital.}
Additionally, we test whether the adjustment margin is higher in communities with low baseline toilet coverage, rendering them more susceptible to CLTS. As expected, Table (ref) shows that higher levels of SES measures are correlated with greater toilet coverage at the community level.
Table (ref) presents the heterogeneous impact estimates, expressed as percentage point changes in OD, calculated using the pooled sample. The outcome variable captures the main respondent's OD practice, as this outcome is measured in all survey waves. Each column presents heterogeneous impacts by one of the four community-level SES measures and baseline toilet coverage, which we discretize along the sample median. For example, in column 1 we rank communities according to their wealth score. Communities with wealth scores equal to or above the median are defined as `High asset wealth' communities ($CC_c=0$), while the rest are classified as `Low asset wealth' communities ($CC_c=1$). The table shows the regression estimates for $\gamma_{r}$ and $\gamma_{d}$ from specification (ref), as well as for the linear combination of both, for comparison purposes.
Table (ref) shows strikingly consistent heterogeneous CLTS impacts for our four proxy measures of community SES. CLTS reduced OD prevalence by 7-9pp in communities with low asset wealth, low night light intensity, low density of households, or in communities that are far away from administrative capitals (i.e. high isolation, note the scale reversal here). Impact point estimates in these communities are two to three times as large as those found on average (Table (ref)) and remain statistically significant at endline. In contrast, we find statistically insignificant impact estimates close to zero in richer communities, regardless of whether we proxy these by asset wealth, night light intensity, density or isolation (see first row, columns 1-3 and second row, column 4).
The third row of Table (ref) presents the point estimates for the difference in CLTS impacts between the two halves of the sample, i.e. the estimated parameter $\gamma_{d}$ in Equation ((ref)). We reject the hypothesis that CLTS had the same impact on communities below and above the median for wealth, and find that the difference in CLTS impacts between poor and rich communities was 10pp (see column 1). Similarly, we reject it for all SES measures according to naive p-values at the 5% level or lower. Using multiple hypothesis testing, we reject it for wealth and, marginally, for density. These results suggest that CLTS was indeed effective in a sub-sample of communities that shared underlying characteristics related to low SES.
Asset wealth is highly correlated with toilet coverage at baseline. One possible explanation for our findings in columns 1-4 of Table (ref) could therefore be that we are picking up differences in initial toilet coverage, and that the latter is the more relevant dimension of CLTS heterogeneity. This does not seem to be the case. In column 5 we find that the impact of CLTS on OD is 4pp stronger in areas with low initial toilet coverage areas. This is smaller than what we found using any of the SES proxies and is not significantly different from zero. Hence, the four SES proxies, in particular asset wealth, are a more informative measure to understand CLTS effectiveness than toilet coverage.
The significant impact on OD behavior in poor communities might be driven by either a direct change in sanitation behavior, i.e. toilet usage, or by an increase in sanitation investments. Using the detailed measures of toilet stock, flow and quality, described in section (ref), we break down sanitation investments into two components. First, CLTS may have promoted investment in new toilets, increasing toilet coverage (and its usage). Second, households who owned toilets at baseline may have invested more heavily into their maintenance and upkeep. This would increase the stock of functioning toilets, reducing the depreciation of the stock of toilets.
CLTS may also have changed sanitation behavior directly, either through increased toilet usage conditional on ownership, or through an increase in shared usage, i.e. when the main respondent declares to use a shared toilet (e.g. owned by a neighbor, public toilet or toilet at school or work).
Table (ref) presents impact estimates on sanitation investment and shared usage, showing pooled results in Panel A and heterogeneous impact estimates in Panel B. The reduction by 9pp in OD observed in poor communities (i.e. communities with low asset wealth in Table (ref)) is almost identically matched in column 1 by an increase in toilet ownership of 8pp, suggesting an increase in the stock of toilets. Ownership of functioning toilets, i.e. maintained stock, increased by 10pp. These results strongly suggest that the change in sanitation behavior is driven by increased sanitation investment, mainly due to an increase in the toilet stock (see column 2).\footnote{ To conduct further sensitivity analysis for potential measurement error in toilet ownership, we estimate a probit model of selective inspection consent by community and household characteristics. We find that if anything, measurement error would bias differences between poor and rich communities towards zero: richer households and households in richer communities are more likely to refuse consent, increasing the possibility of over-reporting of ownership in this group. Critically, there are no differences in refusal rates between treatment and control groups, even when split by wealth levels. In addition, we test the robustness of our estimates by restricting the analysis to households that give inspection consent, and find that our results above are virtually unchanged.}
In contrast, usage of existing toilets (column 3) and shared usage (column 4) in poor communities increased by much less or not at all - the differential impact of CLTS by community wealth is not statistically significant. Appendix (ref) shows that these results are robust to using the three alternative measures of communities' socio-economic status. In sum, OD reductions brought about by CLTS were due almost exclusively to increases in toilet ownership. Usage of owned toilets was high, around 80% (p-value 0.19), and not statistically different between the ones that had a toilet already at baseline and those that built a toilet after the baseline data collection.
Figure (ref) shows dynamic CLTS impacts on toilet ownership and open defecation practice in poor and rich communities up to 32 months after the baseline survey. CLTS reduced OD in the short-run in poorer communities, and impacts were sustained over time (light blue). The estimated short- and long-run impacts of CLTS on both OD (see left panel) and toilet ownership (right panel) across the three followup periods are remarkably constant in poor communities. We draw two conclusions: First, our interpretation that CLTS-induced OD reductions are realized through increased toilet ownership holds also in the dynamic context. This suggests that CLTS has had a persistent effect on OD in poor communities through toilet uptake, akin to that of a one-shot policy. Yet, given the strong link between toilet investment and improved sanitation behavior, CLTS impact in poor communities is achieved in the short-and sustained in the long-run.
In this section we conduct a number of robustness checks on our heterogeneous impact finding that wealth is a policy relevant margin for CLTS effectiveness.
We presented above differential impacts based on a discrete split into rich and poor communities. While using median values as cut-off is a standard approach , our results are qualitatively and quantitatively robust to alternative functional forms. Using for example a linear specification of community wealth rather than the discrete split into rich and poor communities, we find that treated communities that are one standard deviation poorer than the median display a 10% reduction in OD (detailed results in Appendix (ref)). Similarly, using quartiles of community asset wealth, we find that CLTS impacts are statistically significant and decreasing by wealth quartile (Figure (ref)). They are statistically significant up to median wealth, for higher quartiles the treatment effects is zero.
Our measure of community asset wealth is an aggregation of household level information. Richer (poorer) communities tend to be composed of richer (poorer) households. Yet, significant heterogeneity between household and community wealth remains: 31% of households living in poor clusters have higher asset wealth than the median, and 34% of the households living in rich clusters are below median wealth.
To understand whether our estimates are simply capturing that CLTS is (more) effective among poorer households (rather than poorer communities), we proceed in two ways. First, we run the same regression presented in column 1 of Table (ref) but use a household wealth indicator instead of the community one. Second, we split the sample to investigate the effects of CLTS on OD for poor households in rich communities and rich households in poor communities. Appendix (ref) shows that CLTS is more effective among poorer households than rich ones, but the point estimate of the difference between rich and poor is about half as large as that of the community wealth impacts and not statistically significant under multiple hypothesis testing (see Column 1 in Appendix Table (ref)).\footnote{ In Table (ref), we also show that household composition or education do not explain the impact of CLTS.} Furthermore, while in poor communities both rich and poor households reduce OD, there is no discernible effect nor a difference between poor and rich households in rich communities (see columns 2 and 3 in Appendix Table (ref)). This suggests that CLTS is more effective in poorer communities, regardless of the household's position in the wealth distribution.
Poorer and richer communities may differ along many dimensions that may be relevant for the effectiveness of a program such as CLTS. In Table (ref) we present differences between the two groups at baseline by social interactions in the community, access to infrastructure and village leader characteristics.\footnote{ We in addition present in the Table what we refer to as `Household characteristics', the share of households headed by a male and with children below the age of 6 years. Since these variables do not differ between rich and poor communities, we do not discuss them further.}
The index of social capital appears to be uncorrelated with community wealth. Poor communities, however, do exhibit higher levels of social cohesion (as measured by community-level trust and religious fragmentation) and lower levels of asset wealth inequality than rich communities.\footnote{ Details about how these measures are constructed are in appendix (ref).} We also find significant differences in access to public infrastructure, including having a local school, a hospital and paved internal roads. Finally, poor communities have less experienced and less educated leaders.
We assess whether any of these dimensions might be the main drivers behind differential CLTS effectiveness by community wealth. kar2008handbook suggest social interactions as a potential driver and Cameron2019 find that a CLTS intervention in Indonesia generated stronger effects in communities with higher social capital. However, results in Table (ref) show that the point estimates, in most cases, go in the opposite direction of what would be expected if stronger social cohesion improved program effectiveness. For example, we find slightly stronger reductions in OD in treated communities with lower social capital, high fragmentation and inequality. In all cases, differences in CLTS effectiveness along dimensions of social interactions are not significantly different from zero in our study context.
We secondly explore whether our main results could be explained by poorer communities' lower access to infrastructure which may, for example, proxy for transport costs. Further, as leaders are the initial point of contact for the implementers and help organize the CLTS meeting in their village, leaders' tenure and education may play an important role. We find no heterogeneous CLTS impacts along either of the three indicators of public infrastructure (Table (ref), columns 1-3) nor along the village leader's tenure or education (Table (ref), columns 4 and 5). \\
CLTS is a fairly standardized intervention. Nonetheless, there is the possibility that the heterogeneous impacts we observe are the result of differences in the intervention's delivery. In recent work based on the same RCT, abramovsky2016 show that CLTS triggering meetings are more likely to fail, and not be carried out at all, in areas with high population density which is positively correlated with community wealth. Is CLTS ineffective in rich communities because CLTS triggering meetings are not taking place? Using community wealth directly, we find that the difference in triggering rates is small. The share of communities assigned to CLTS in which CLTS triggering meetings were successfully run was 75% for rich communities, and 83% in poor communities. These rates are not significantly different from each other at standard levels of statistical confidence (p-value=0.301). In addition, we present in Table (ref) Appendix (ref) estimates of the impact of CLTS, instrumented by treatment assignment, and find very similar results to the ITT estimates.
Differences in delivery (and hence impacts) may also arise if meeting attendance rates are higher in poor communities. We do not find evidence for this hypothesis either. Attendance rates, measured as the number of attendees recorded by CLTS facilitators over village population, were not significantly different between rich and poor communities.\footnote{ On average, 34% of the village members attended CLTS meetings in rich communities, while 42% attended in poor communities. A community level regression of attendance rates on community level wealth group (i.e. a dummy equal to one if the community is poor) and LGA fixed effects results in a point estimate of just 3pp and a p-value of 0.662. Results are available upon request.}
Differences in the quality of CLTS delivery may also arise from different delivery agents. As described in Section (ref), WaterAid hired two NGOs with CLTS experience, one from each state, to train local government officials in the facilitation of CLTS meetings. If these intermediate delivery agents (the NGOs) differed in the quality of their training activities, we might observe state-level differences in CLTS effectiveness.\footnote{ Of course, this is not the only channel that could explain differential results by state.} We find no evidence of this (see Table (ref) Appendix (ref)). Interacting treatment status with state dummies in a specification similar to Equation ((ref)), we find that the interaction term is small and not significantly different from zero (p-value=0.326).
Taken together, our results suggest that community wealth encompasses a number of community characteristics that made CLTS more effective. We did not find evidence that any of these characteristics (such as toilet coverage, implementation or measures of social cohesion) could explain independently why CLTS worked better in poorer communities.
While the available data does not allow us to pin down the drivers of these differential impacts, the heterogeneity dimension is in line with CLTS originators' theory and practitioners' experiences as to where the intervention should work best. The importance of context for CLTS effectiveness was also highlighted in a recent cross-country study, which concludes that “[t]he impact of CLTS and subsequent sustained latrine use varied more by region than by intervention, indicating that context may be as or more important than the implementation approach in determining effectiveness” CrockerEtAl2017Sust.
In line with this observation, we will show in the next section that even without a context-specific understanding of underlying mechanisms, our heterogeneous CLTS impact can be used as a basis for more effective targeting of the intervention \textemdash within Nigeria and beyond. In other words, we will show that community wealth (or a proxy) is a precise predictor of CLTS effectiveness and can be used for policy targeting.
It is widely advocated that decisions about investment in public health interventions should be evidence-based. Due to the high cost of producing location-specific evidence, policy decisions are usually based on a limited number of studies, often conducted elsewhere. Their results are generalized to make an implementation decision in a different set of target sites. This is despite the understanding that outcomes depend on both the types of interventions and the context in which they are implemented WangEtAl2006,Meager2018UnderstandingExperiments. Or, as Angrist2004 put it `[t]he relevance or 'external validity' of a particular set of empirical results is always an open question.' Recent reviews of RCT impact evaluation studies, in the medical Malmivaara2019 as well as the development economics literature PetersEtAl2018, highlight that their external validity may be limited.
At the same time, context-specific impacts may arise due to heterogeneous responses of population subgroups, leading to diverging average treatment impacts across RCTs that sample (randomly) from populations with different underlying characteristics. As such, a better understanding of heterogeneous responses could help strengthen the generalizability to and external validity of RCTs in different target populations HotzEtAl2005,AllcottMullainathan2012,ImaiRatkovic2013,FerraroMiranda2013. Policy makers can use evidence on responsive subgroups for a more cost effective targeted implementation strategy HeckmanEtAl1997,DjebbariSmith2008.
We argue, in the same spirit, that policy-makers in Nigeria and in other countries can build on our findings, target poor geographical areas and, by doing so, avoid wasting money on implementing CLTS in low or no response communities. This is particularly relevant given recent estimates that `the true costs of participatory sanitation' can be quite substantial CrockerEtAl2017: existing cost estimates range from US\$30.34 to 81.56 per targeted household in Ghana, and US\$14.15 to 19.21 in Ethiopia. We present our argument for the external validity of our findings to other contexts in the next section.
Developed in Bangladesh in 1999, CLTS is today widely implemented and endorsed by national governments and NGOs in more than 25 Latin American, Asian and African countries. Given limited evidence on its effectiveness in its early existence VenkataramananEtAl2018, its spread was typically a reaction to enthusiastic advocacy by numerous actors, including grassroot activists, state bureaucrats and the donor community Zuin2019.
Rigorous evaluation studies emerged in the early 2010s. Yet, even if policy-makers today wanted to base CLTS implementation decisions on the available evidence, the conflicting impact estimates would not be a useful guide. To the best of our knowledge, five other studies have estimated the impact of CLTS-like interventions in developing countries based on RCTs. The World Bank's Water and Sanitation Programme (WSP) conducted three of these studies, in Tanzania bricenoEtAl2017, Madhya Pradesh, India Patil2014TheTrial, and East Java, Indonesia Cameron2019. pickering2015effect conducted a similar evaluation in rural Mali. Finally, guiteras2015encouraging carried out a cluster RCT in Tanore, Bangladesh, in which they evaluated the impact of three different policy approaches, one of which was a CLTS-style intervention.\footnote{ The other two were supply side technical assistance and subsidy provision.} The resulting evidence is inconclusive, with estimated CLTS impacts ranging from 30pp increases in toilet ownership in Mali to no statistically-detectable impacts in the studies conducted in Bangladesh and Indonesia.
In this section, we argue that community-level SES, in particular wealth, might be the factor underlying these diverging results. In Section (ref), we documented the strong heterogeneity of CLTS impacts by community wealth in our field experiment in Nigeria. If these results are externally valid, CLTS interventions in richer areas or study sites will generally have lower (or no detectable) impact than those administered in poorer areas.
Using study-specific data, we analyze how CLTS effectiveness across (and within) RCTs varied along SES status. As a first step, we use a consistent method to estimate heterogeneous impacts on toilet ownership and open defecation by community socio-economic status for each RCT where data is publicly available (Bangladesh, India, Indonesia and Tanzania). \footnote{The datasets for these studies can be found at: DVN/GJDUTV_2017 (Bangladesh), WaterandSanitationProgram2009WSP2009-2011 (India), WaterandSanitationProgram2008IndonesiaIDN_2009_2011_WSP-IE_v01_M_v01_A_PUF (Indonesia), and Briceno2012ImpactTZA_2012_SHRSBIE-EL_v01_M_v01_A_PUF (Tanzania). For the RCT in Mali (pickering2015effect), where data is not publicly available, we report the estimates presented in their paper, obtained by a means comparison of outcomes between treatment and control households at endline. The OD outcome captures whether any adult female performed OD.}
Optimally, we would compare impact estimates along a community measure of wealth that is a) consistent across studies, and b) aggregated by treatment unit, i.e. community. Unfortunately, not all studies collected data on ownership of consumer durables that is used to construct the community wealth index in our Nigerian study. In addition, the durable items underlying asset wealth indices are highly country- and context-specific Filmer2001EstimatingIndia.\footnote{Similar limitations rule out the use of population density and distance to the nearest district headquarters.} Aggregation at treatment unit is not always possible due to data confidentiality reasons which made the exact location of study clusters non-disclosable in a majority of the cited studies.
For these reasons, we use night light intensity at baseline as an alternative measure of community-level SES in estimation. Night light indices are comparable across locations, and we compute them at the lowest geographical level $d$ available in each study.\footnote{Appendix Table (ref) gives a detailed overview of the level of geographical disaggregation and the number of observed geographical units used to calculate average night light intensities. In line with our analysis in the previous sections, we exploit the within-study variation in night light intensity to calculate treatment effects separately for the poorer and richer halves of each sample. See appendix (ref) for full details and exceptions where this procedure was not possible due to data limitations. All night light indices relate to the baseline survey year.} While used in the past to proxy for GDP per capita, a measure of income, at the sub-national level in African countries Michalopoulos2013, we have shown that it is a robust proxy for community SES, and yields very similar heterogeneous impact estimates to those obtained by community wealth (see Table (ref)).\footnote{Michalopoulos2013 also presents evidence of a strong correlation between wealth and nightlight intensity.}
For estimation, we split each RCT sample into richer and poorer clusters, and then implement a relatively parsimonious ANCOVA model based on information that is commonly available across studies, leading to the following specification:
The outcome variables, $y_{icd}$, are two dummy variables, denoting i) ownership of a private, functioning toilet by household $i$ in cluster $c$ in district $d$, and ii) OD by any member of the household. $T_{cd}$ is a dummy variable equal to one if cluster $c$ from district $d$ has been assigned to CLTS, and zero otherwise. We distinguish between district and cluster here, as for data protection reasons we observe treatment assignment at the anonymised cluster level $c$, and observe community SES at the non-anonymised district level $d$.\footnote{In cases where multiple treatment arms existed, we include only the CLTS arm and the control group. We include baseline values $y_{icd0}$ for the outcome variables, except for Tanzania, where no baseline data was collected. Our study in Nigeria is the only one to include multiple post-treatment survey waves, so we restricted our sample to observations from the second followup wave (FU2), which is closest in timing to the post-intervention surveys of the other studies.} Household level controls $X_{icd}$ are gender, age and age squared of the household head, and whether a household's primary activity is farming. We include geographic fixed effects at the district level ($\omega_{d}$) where appropriate, to account for stratified randomization.\footnote{ District level fixed effects are used to estimate impacts for the case of India, Indonesia, Nigeria and Tanzania. We omit geographical fixed effects in the case of Bangladesh since the experiment was conducted in a single district.} Standard errors are clustered at the cluster level, the unit of randomization.
In Figure (ref) we report the study-specific point estimates of CLTS impacts for poorer and richer communities on toilet ownership and open defecation (right axis) along with the corresponding satellite nightlight intensity over the considered set of study areas (left axis).\footnote{The exception is Mali where we cannot produce heterogeneous impact estimates due to the lack of data access.} It illustrates our main result, that CLTS is on average more effective in poorer settings, i.e. among those with lower nightlight intensity to the left of the Figure. As night light intensity increases, point estimates decrease in magnitude for both outcomes (depicted in black circles for toilet ownership and grey diamonds for OD), and the likelihood of rejecting the null hypothesis falls. Comparing the within-RCT impact estimates (e.g. for example Tanzania-Low versus High), we find that impact estimates are always larger in the `poorer' half of communities. Note that we find no evidence of CLTS impacts in study sites with night light intensity above that of the Nigerian median (i.e with average nightlight above 0.89, marked by the horizontal dotted line just above the 0 value of Night Light Index Axis). In contrast, all four community groups with the lowest nightlight indices (and all samples with average nightlight below the poorer Nigerian communities) display statistically significant CLTS impacts. We argue that this is supportive of our argument that CLTS is more effective at increasing toilet ownership and at decreasing open defecation in relatively poorer areas.
As Figure (ref) is suggestive of an inverse relationship between community SES and CLTS impacts, we explore this further by producing a linear fit between the point estimates of CLTS on toilet ownership and open defecation by (the log of) area-specific night light index (see Figure (ref)). The R-squared of these linear fits suggests that variation in log night light can rationalise 42% (50%) of the variation in CLTS effectiveness in increasing toilet ownership (and reducing OD).
In a separate exercise, we pool the three studies for which GPS data was available and we could thus measure nightlight intensity at the cluster level (i.e. Indonesia, Nigeria and Tanzania) and estimate overall and heterogeneous CLTS impacts on toilet ownership, where reporting should be straightforward.\footnote{Please note that when using pooled estimates, we cannot rely on strict exogeneity by randomisation for identification. Yet, we include country fixed effects which will pick up sampling variation across RCT sites, but do not find that they change the impact estimate (see columns 1 and 2 in Table (ref)).} We find qualitatively similar average CLTS impacts to the Nigerian RCT, in the magnitude of a five percentage point increase in toilet ownership (see columns 1 and 2 in Table (ref)). We then estimate heterogeneous impacts for the pooled sample using three alternative functional forms to capture nightlight variation. First, we split geographic units according to whether they display zero or positive nightlight intensity, as its distribution is strongly skewed to the right. Second, we use the results from our Nigerian RCT as reference and define the Nigerian median as a split point as we found no CLTS impact estimates beyond this level in our RCT (see Table (ref)). Third, we estimate heterogeneous impacts by nightlight intensity using a more flexible split into tertiles. The results, shown in columns 3 to 5 of Table (ref), support our hypothesis that CLTS impacts vary by communities' SES, here measured through nightlight intensity. We find substantially larger impact estimates in lower SES areas, in the magnitude of 9 percentage points in areas with zero nightlight, i.e. the lowest tertile\footnote{31% of the observations in the pooled sample are located in areas with zero night light intensity.}. These results get even stronger if we split areas along the Nigerian nightlight median: in poorer areas CLTS increases toilet ownership by 12 percentage points (significant at the 1% confidence level). Similarly, we never find statistically significant CLTS impact estimates in high SES areas in any specification (be they defined via positive nightlight intensity, intensity above the Nigerian median or in the upper tertile). We further test whether the difference between the estimated coefficients in “poor” and “rich” areas differ from zero and reject the hypothesis in all specifications but one.\footnote{ The exception is the specification in column 3 where we split the sample into zero and positive nightlight areas. It is likely that this split is too coarse and puts low SES areas with very low but positive nightlights into the high SES category. } Furthermore, impacts are declining across tertiles of increasing nightlight (column 5), similar to the results in our RCT (see the quartile split displayed in Figure (ref)).
These results are only suggestive, but are consistent with our findings. A more detailed analysis would require access to the exact cluster locations in each study or more detailed data regarding household ownership of consumer durables (both unavailable), that could then be combined to separately identify cross-study variation from genuine observational differences across households as, for example, proposed by Meager2018UnderstandingExperiments.
We thus argue that the large range of CLTS impacts across studies can be rationalized by differences in the average wealth (measured through night light) of the area in which they were conducted. Our results - in Nigeria and beyond - suggest that CLTS has been more effective in poorer communities.
The implications of our findings are self-evident: Governments with restricted funds may achieve larger improvements in sanitation if they use an effective targeting strategy and such a strategy can be based on a measure of community wealth.
We argue that available surveys such as the Demographic and Health Surveys (DHS), or satellite nightlight intensity, both of which are available for all of the 60 countries across continents where CLTS is widely implemented, can form the basis of such a targeting strategy.
In Appendix (ref), we demonstrate in detail how such data can be used for CLTS targeting in the case of Nigeria. We develop a targeting strategy based on our impact estimates from the Nigerian RCT and the 2013 Nigerian Demographic and Health Survey. Even though the DHS contains a less detailed list of assets than our data, the simpler DHS index of community wealth strongly predicts the more sophisticated measure of community wealth used in our study, supporting the notion that readily available surveys collecting asset wealth information, such as the Demographic and Health Survey, are well-suited for CLTS targeting. The resulting targeting map in Figure (ref) highlights priority areas for targeting, i.e. poor areas where toilet ownership rates are low, in darker shaded areas.
The design of effective policies to address the urgent sanitation concerns in the developing world requires a nuanced understanding of households' investment choices and drivers of behavioral change. In this paper we provide evidence on the effectiveness of Community-Led Total Sanitation (CLTS), a participatory information intervention widely implemented around the world.
Our study uses a large cluster randomized experiment in Nigeria for which we collected data up to three years after treatment. Implementation of CLTS was conducted at-scale, i.e. by WASH civil servants trained by local NGOs. We show that CLTS, a participatory community intervention without financial components, had positive but moderate effects on open defecation and toilet construction overall. However, average impacts hide important heterogeneity by communities' socio-economic status, as the intervention has strong and lasting effects on open defecation habits in poorer communities, and increases sanitation investments. In poor communities, OD rates decreased by 9pp from a baseline level of 75%, while we find no effect in richer communities. The reduction in OD is achieved mainly through increased toilet ownership (+8pp from a baseline level of 24%). While this result is robust across several measures of community socio-economic status, and is not driven by baseline differences in toilet coverage, our data does not allow us to pin down why households in poorer communities are more susceptible to the programme. However, in addition to the more effective targeting strategies, highlighted in the previous section, that governments can adopt, our results have three further important implications.
First, our results provide an additional reason why scale-up of interventions is not trivial Ravallion2012,BoldEtAl2013,BanerjeeEtAl2017,DeatonCartwright2018. Discussions on why interventions may not scale-up successfully in a national roll-out have focused on general equilibrium and spillover effects, and recently on aspects of implementation and delivery. The literature has suggested that spillovers and moderating general equilibrium effects may lead to lower returns to interventions, when interventions conducted in areas with specific characteristics are being rolled out universally, e.g. in richer areas. We show that community-specific, heterogeneous treatment impacts are an additional impediment to successful scale-up in terms of effectiveness of interventions.
Second, community SES also provides plausible external validity beyond our Nigerian-based RCT. Using data from our study and five other RCTs of similar interventions, we find an inverse relationship between area-level wealth, measured by night light intensity, and program effectiveness across these studies. Thus, we have identified a characteristic that rationalizes the wide range of impact estimates in the literature.
Last but not least, we show that interventions relying on information and collective action mechanisms can have substantial impacts on households' health investments and behaviour, specifically relating to sanitation. Yet , there is an important caveat for policy-makers working towards meeting the sanitation-related sustainable development goals. CLTS achieves convergence between poor and rich communities in terms of OD and toilet coverage in our study - and thus levels the playing field. However, it is not a silver bullet to achieve open defecation free status in poor communities. Hence, more research on alternative or supplementary interventions to close the sanitation gap in low income countries is needed. These may either seek to magnify CLTS impacts (e.g. through complementary financial incentives, loans or subsidies or more intensive followup), or improve sanitation in rich communities where CLTS is ineffective, e.g. via infrastructure investment and supply side interventions.