B.C. Business Matters:
Guest Authors >>
Evidence-Based Policy and Investments in Children
This post was guest-authored by Dr. Bill Warburton. With a PhD in economics from the University of London, he has worked as an economist in government, academia and the private sector, contributing to policy debates on employment, training, income support programs, child welfare and health care.
The social, emotional and academic skills of the population are central to the health and well-being of our citizens and our society. Economists refer to these skills as human capital and have shown that they are crucial for ensuring continued prosperity. A dearth of human capital imposes costs on society and government, in the form of additional crime (Lochner & Moretti, 2004), higher welfare costs (Coelli, Green, & Warburton, 2007), worse health (Heckman, Humphries, & Veramendi, 2016) and foregone tax revenue (Rouse, 2005). An abundance of human capital contributes to economic growth directly through its impact on the marginal productivity of labour, and indirectly by, for example, facilitating skill-biased technical change, generating positive externalities, and speeding innovation (Hulten & Ramey 2017).
So how can we increase human capital and realize these benefits? There is a consensus that even though “Long-term evidence on their effectiveness is surprisingly limited,” some early interventions are very effective at increasing human capital (García, Heckman, Leaf, & Prados, 2016). But most interventions for children have a modest impact, and some are actually harmful.
When Tennessee implemented its statewide pre-K program in 2005, it did so with bipartisan support. Then Governor Bredesen said, “Quality Pre-K classrooms are one of the best investments we can make in the education of children in Tennessee.” But a recent evaluation found that by the time they reached 3rd grade, children who participated in Tennessee’s Pre-K program had worse attitudes toward school and poorer work habits than children who didn’t (Lipsey, Farran, & Hofer, 2015). And a recent large-scale rigorous evaluation of the American flagship program, Head Start (Puma, et al., 2012), concluded that “early effects rapidly dissipated in elementary school, with only a single impact remaining at the end of 3rd grade for children in each age cohort.”
Figure 1 is a histogram of effect sizes from early education interventions. Gregory Camilli and colleagues (Camilli, Vargas, Ryan, & Barnett, 2010) identified and reviewed 123 studies that made estimates of the impacts of early childhood interventions.
There are three interesting features of the graph. The first is that the average is positive, so if we implement an early childhood intervention, we will probably help the children that participate. The second is that among the successful programs, about a third have a very a small impact. The other two-thirds have impacts that vary from twice as big to more than ten times as big. The third and possibly most interesting feature of this graph is that a substantial portion of the interventions actually harmed the children, and a few did substantial harm.
Achieving “evidence-based policy” in the area of interventions for children is particularly challenging because the field is over-run with advocacy and junk science. In a famous review of the literature, the American National Research Council (Committee on Youth Employment Programs, National Research Council, 1985) found that 93% of studies of programs intended to promote youth employment were unreliable. This ratio has been sustained over the years. In 1998, Lyn Karoly and colleagues (Karoly & Levaux, 1998) reported that only nine studies provided reliable evidence on the effects of early childhood interventions. In order to sift the evidence from the mounds of junk, the US government established the What Works Clearinghouse (WWC) within the Department of Education. On the first page of drop-out prevention programs, WWC reports that of the 126 studies reviewed, 116 (92%) did not meet their standards of evidence (WWC, 2017).
Nonetheless, if the BC government decides to be guided by evidence-based policy, it has that option. It can eschew confusion and embrace rigor. It can capture the potential benefits from investments in children (and ensure we do no harm), in two steps: i) identify the children who need help the most and provide it to them; and ii) develop strong evidence on the effectiveness of interventions, so that it can gradually and carefully expand the interventions that are working well and re-tool ones that are less effective.
1. Identify children at risk and provide services to those children first
From a technical perspective, identifying children at risk is straightforward. For example, to identify the children at risk of dropping out, policy researchers can estimate the parameters of a regression equation in which the dependent variable takes the value 1 if the student graduates. In this way, data from our administrative systems can identify 4,000 ten-year-old children who have at least a 75% probability of dropping out.
Providing services to children at highest risk first has two advantages. First, it is ethical. John Rawls, in his A Theory of Justice, suggests allocating scarce resources to those who need them most, as a first priority. Second, the literature indicates that providing services to the disadvantaged first is likely to increase the cost-effectiveness of those services (Public Health Agency of Canada , 2009) (Elango, García, Heckman, & Hojman, 2015).
2. Develop strong evidence of effectiveness
Regression Discontinuity Design, which compares outcomes for those just above the cut-off for receiving services with the outcomes for those just below the cut-off, is widely regarded as producing strong evidence (Lee & Lemieux, 2010). For example, it meets the WWC standards of evidence.
Our society’s experience with programs for children shows that popularity and/or good intentions are not sufficient to ensure success. Some popular interventions have harmed the children that they were intended to help. But our society’s experience also shows that early childhood development projects have the potential to reduce crime, poverty and inequality. However, to achieve that potential, we must bring quantitative analysis and rigour to early childhood interventions. Evidence-based policy should be more than a slogan.
Committee on Youth Employment Programs, National Research Council. (1985). Youth Employment and Training Programs: The YEDPA Years. National Academies Press.
Camilli, G., Vargas, S., Ryan, S., & Barnett, W. S. (2010). Meta-analysis of the effects of early education interventions on cognitive and social development. Teachers College Record, 112(3), 579-620.
Coelli, M. B., Green, D. A., & Warburton, W. P. (2007). Breaking the cycle? The effect of education on welfare receipt among children of welfare recipients. Journal of Public Economics, 1369-1398.
Elango, S., García, J. L., Heckman, J. J., & Hojman, A. (2015). Early Childhood Education. IZA Discussion Papers, No. 9476.
García, J., Heckman, J., Leaf, D., & Prados, M. (2016). The Life-cycle Benefits of an Influential Early Childhod Program. National Bureau of Economic Research.
Heckman, J. J., Humphries, J. E., & Veramendi, G. (2016). Returns to education: The causal effects of education on earnings, health and smoking. National Bureau of Economic Research.
Hultan, C., & Ramey, V. (2017). Introduction to Education, Skills, and Technical Change: Implications for Future U.S. GDP Growth. National Bureau of Economic Research.
Karoly, L. A., & Levaux, H. P. (1998). Investing in our children: What we know and don't know about the costs and benefits of early childhood interventions. Rand Corporation.
Lee, D., & Lemieux, T. (2010). Regression Discontinuity Designs in Economics. Journal of Economic Literature.
Lipsey, M. W., Farran, D. C., & Hofer, K. G. (2015). A Randomized Control Trial of a Statewide Voluntary Prekindergarten Program on Children’s Skills and Behaviors through Third Grade. Vanderbilt University, Peabody Research Institute.
Lochner, L., & Moretti, E. (2004). The effect of education on crime: Evidence from prison inmates, arrests, and self-reports. The American Economic Review, 155-189.
Public Health Agency of Canada. (2009). Investing in Prevention – The Economic Perspective: Key Findings from a Survey of the Recent Evidence.
Puma, M., Bell, S., Cook, R., Heid, C. B., Jenkins, F., & Downer, J. (2012). Third grade follow-up to the Head Start Impact Study: Final report (OPRE Report 2012-45). US Department of Health and Human Services, Administration for Children and Families, Office of Planning. Research and Evaluation, 28.
Rouse, C. (2005). The Labor Market Consequences of an Inadequate Education. Social Costs of Inadequate Education. Columbia University.
Tennessee State Government. (2009, April 15). Tennessee Pre-K Earns National Recognition, Increases Access. Retrieved March 2, 2017, from https://www.tn.gov/news/25398.
WWC. (2015). Preview of Regression Discontinuity Design. What Works Clearinghouse.
WWC. (2017, June 10). Search Publications. Retrieved from What Works Clearinghouse : https://ies.ed.gov/ncee/wwc/Publication#/FWWFilterId:13,ContentTypeId:1,SortBy:RevisedDate,SetNumber:1.
 28/400 papers met their minimum standards of evidence. Page viii.