The Middlemiss Study Tells Us Nothing About Sleep Training, Cry-It-Out, or Infant Stress

Last week, I wrote a post about sleep training and stress, in which I argued that everything we know about stress suggests that sleep training is not harmful.

In response, many people objected that sleep trained babies continue to experience elevated cortisol and significant distress, even after they have stopped crying. In their view, sleep training teaches babies that crying does not help. They haven’t learned to self-soothe or to fall asleep on their own, they’ve simply given up.

What a heartbreaking thought. And one that surely strikes fear in the heart of many parents.

So it’s important to realize that this claim comes from a single small and deeply flawed study of 25 babies, led by Wendy Middlemiss, a researcher at the University of North Texas’s College of Education.

Here is a typical example of how her study is described in the popular press:

“The researchers found high levels of cortisol, a stress hormone, in both the mothers and the babies during the times the babies were crying. After several days, the babies learned to go to sleep without crying. Researchers found that during these quiet nights, the mothers no longer had high cortisol levels but the babies’ cortisol levels remained high. They had merely learned to remain quiet while distressed.

The researchers noted that this was the first time the mothers and babies had not been in sync emotionally. The mothers no longer had high stress levels, not realizing that their babies were still just as upset.”

I can see how many parents would read something like this and swear off sleep training.

Here’s the truth, though: Nothing in her study supports these claims.

To see why, let’s start by briefly reviewing her study’s design. Middlemiss studied 25 mother-infant pairs who were enrolled in a sleep training program at a local hospital. The babies ranged in age from 4 to 10 months.

Mothers spent the day at the hospital with their infants, helped prepare them for sleep at naptime and bedtime, and then retreated to a hallway outside the room where they could hear them, but their infants could no longer see them. The nurses put the infants down in their cribs and let them cry, without soothing, until they fell asleep. This process was repeated for 4 days.

On the 1st and 3rd nights of this program, Middlemiss measured the babies and mothers’ cortisol levels, a hormonal marker of stress. She tested their cortisol levels once right before bedtime and then again 20 minutes after the babies had fallen asleep.

So what’s wrong with her study? Well, a lot.

(1) The study lacked a control group. Without a group of control babies who were put to sleep by nurses in the hospital, but who did not experience sleep training, we cannot say whether sleep training affected infants’ cortisol levels, or whether something else about the program, like being put to sleep in an unfamiliar room in a hospital or being put to sleep by a stranger, affected infants’ cortisol levels.

 (2) She does not analyze her data correctly. Let’s take her findings one at a time.

Claim #1: Babies cortisol levels remained “high” before and after falling asleep on the first and third nights of sleep training, while mothers’ levels dropped after their babies had fallen asleep on the third night.

Problem #1. Middlemiss does not report a baseline cortisol level for the babies. We do not actually know whether infants’ cortisol levels were “high” or “low” or “normal”. She calls them high. But we have know way to know that they’re high; they stay constant throughout the study.

When Melinda Wenner Moyer, a reporter for Slate, asks why she calls their cortisol levels high, Middlemiss responds that she also assessed the babies’ cortisol levels while at home, and the levels were lower than at the hospital. But she never reports these baseline levels in her paper.

To anyone who has published a scientific paper, this is a baffling response. If you are drawing conclusions based on data you collected, you report that data. That’s the way it works.

Problem #2. She uses the wrong statistical analyses to compare before and after cortisol levels. To me, this is the most egregious problem with her research.

Here is how she describes her analyses:Screen Shot 2016-04-20 at 1.55.04 PM

In stats-speak, Middlemiss compared whether the group means before and after sleep training were significantly different from one another. This is the wrong analysis. She should have used a repeated measures analysis. Simply comparing group means is incorrect. It is also considerably less powerful (it fails to take into account that you already know something about the individuals the second time around) and thus more likely to lead to a false conclusion of no change.

Let me illustrate why using an analogy. Imagine you have a group of 25 students who enroll in an SAT prep class. You compare their test scores before the class has begun with their tests scores after the class ends. The mean test score among students does not increase. Does this imply that the class was worthless?

Well, maybe, and maybe not. What you really want to know is not whether the group mean is higher, but whether on average the students improved. And that is actually a different question. For example, if 90% of the students improved by say, 50 points, while 10% dropped by 500 points, the means would remain the same, despite the vast majority of students improving.

Problem #3 Middlemiss is missing a ton of data, and–you guessed it–she does not handle that issue correctly either.

Look back how she describes her analyses. Do you see how the number of mothers and infants in each group changes from before and after sleep training? Consider the mothers’ cortisol data on the third night. The pre-sleep group includes cortisol samples from 17 of the 25 mothers. The post-sleep group includes cortisol samples from 12 of the 25 mothers.

This raises some questions: Are these 12 mothers a subset of the first 17? Or do these 12 include mothers not included in the before group? Middlemiss never tells us.

Why is missing data a problem? Let me again use an analogy. Imagine I have a bag of 25 apples. First I pull 17 apples out and weigh them, and then put them back. Next I pull out a second set of 12 apples out and weigh those. The second set of 12 apples weigh less, on average, than the first 17. Can I conclude that apples in the bag have lost weight?

Of course not.

Now, missing a sample or two is not a huge deal, even in a relatively small study. Large studies can often handle significant amounts of missing data, provided data loss occurs more or less at random. But Middlemiss is missing samples from over half of the mothers on the third night!

At the very least, if you wanted to test whether the women’s cortisol has dropped after sleep training, you ought to compare the same mothers before and after the their infants fell asleep. But she does not do that.

So, was the babies’ cortisol high? I don’t know. Did the mothers’ cortisol drop? I don’t know. It’s impossible to tell from what she reports.

(Note that this should never be the case in a scientific publication. The whole point of a scientific publication is to make what you did clear enough that someone else could replicate your study and analyses.)

Claim #2 Mothers’ and babies’ cortisol levels were no longer correlated after the third night of sleep training. 

This is what she reports:

Screen Shot 2016-04-20 at 2.29.05 PM

Problem #4 Here again Middlemiss uses the wrong statistical test. She claims that the mothers and babies cortisol levels no longer correlated after the third night of sleep training, because the second correlation, r(10)=.422 is not statistically different from zero. This is the wrong test.

She should have tested whether post-sleep correlation of r=.422 is significantly different than the pre-sleep correlation of r=.582. A significant difference between the correlations is, after all, what she claims to have found.

And by the way, the two correlations are not significantly different from one another.

Problem #5 Even if Middlemiss had performed the correct test, we would still have a major problem, because, yet again, she’s lost over half of her sample! We have no way of knowing whether the pre-sleep mothers-infant pairs are the same, largely the same, or completely different from the post-sleep mother-infant pairs. She nevers tell us.

Problem #6 Her entire argument boils down to 10 mother-infant pairs. That’s too small of a number to tell much of anything. To see why, note that if she found a correlation of .422 in her entire sample of 25 mother-infant pairs, it would have been significantly different from zero–meaning her whole argument rests not just on the wrong analysis but on a lack of statistical power.

In sum…

What did Middlemiss and colleagues conclude from these non-findings? 

That infants continue to experience physiological distress, as measured by cortisol, despite being able to soothe themselves to sleep, and that sleep training leads to an “asynchrony” in mother and infant cortisol levels.

What do I conclude?

That this study should never have been published, at least not in its current form. Middlemiss and colleagues had no control group. They performed the wrong statistical analyses. They had huge amounts of missing data which they did not account for at all. Her key findings relied on less than half of her original sample of 25.

These are not minor, nitpicky problems. These are major, glaring problems that make interpretation of her findings impossible.

Warning women against sleep training on the basis of this study is absurd.

Advertisements

Critics of Cry-It-Out Fundamentally Misunderstand How Stress Affects the Brain

Because whether or not to sleep train can be such a fraught decision for new parents, I wanted to share my sleep training story, and to explain why, given everything we know about stress, the argument that sleep training causes long-term harm doesn’t hold water.

Sleep Training My Son

When my son was 4.5 months old, I decided to sleep train him. Even by baby standards, my son was not much of a sleeper. He’d snooze for at most 4 or 5 hours, and then wake up every hour like clockwork, wanting to nurse but not wanting milk, popping on and off my breast and screaming in frustration.

I had gone back to work a month earlier, so napping to catch up on sleep was out of the question. Worse, I was commuting an hour to the office each way.

By then, I had reached the end of my sleep deprivation rope. I was so tired I could barely string two thoughts together. I had to coach myself through even mundane tasks like checking out at the grocery store. Say hello to the cashier. Take out your credit card. Pick up the grocery bags. Leave.

I was terrified every time I got into my car to head to work that I would nod off at the wheel and kill someone, quite possibly myself. I joked with coworkers that driver’s licenses should be temporarily suspended for new parents, but the situation really wasn’t funny.

So there I was the first night of sleep training, dripping sweat as I listened to my son’s cries. Minutes ticked by, each seeming longer than the last. I pondered whether the Ferber method included soothing every five minutes just so that you would realize only five minutes had passed.

But I was determined to stick this out, to get it done. Doing it halfway was worse than not doing it at all, I reminded myself over and over. If I were to give in, I could teach my son that crying for 30 minutes was what it took to get mommy to pick him up.

That night, he woke two more times, but never again cried more than 15 minutes. The next night, he cried for 10 minutes at bedtime, conked out, and slept until morning. That morning we greeted each other with a smile, and for the first time since his birth, I really felt like smiling at his freshly woken little face.

Although by all appearances, sleep training went well for us, some critics of cry-it-out methods would contend that I was an inadequate parent who had permanently harmed my son by leaving him alone to cry.

The Cry-It-Out Controversy

“An emotionally available parent would probably not let their baby cry it out,” claims Dr. Teti, a researcher at Penn State.

Dr. Narvaez writes in Psychology Today:

“Letting babies get distressed is a practice that can damage children and their relational capacities in many ways for the long term. We know now that leaving babies to cry is a good way to make a less intelligent, less healthy but more anxious, uncooperative and alienated persons.”

When someone tells you that you have permanently damaged your child, it’s hard to shake off, no matter how much happier you and your baby seem once you start getting some solid rest.

Thankfully, as someone who has studied the effects of chronic stress in animals and in people, I knew that claims like Dr. Narvaez’s are not supported by data and instead rest on a fundamental misreading of stress research.

Studies on Cry-It-Out

Studies involving cry-it-out methods find no ill long-term effects for babies, and big benefits for parents. In one study, rates of depression in mothers dropped from 70% to 10% after sleep training.

Critics of cry-it-out methods argue that these studies are flawed–they don’t have the right measures of harm, they fail to determine whether the parents actually used cry-it-out methods to sleep train their babies, and they rely on parents’ reports instead of observation.

Admittedly, some of these criticisms of sleep training research are fair. Sleep training research is hard to do, especially since researchers can’t exactly force parents to sleep train little Johnny, or prevent parents from doing so.

That said, there’s no evidence in humans–none–to support the view that sleep training is harmful. If there were, we’d have heard about it.

So what the critics of cry-it-out argue, really, boils down to this: They know that sleep training is harmful, because they know stress is harmful to babies.

The problem with that argument? All stress is not created equal. We were designed to handle short-term stress. Where we humans, and other animals, run into trouble is when stress becomes chronic.

Short-Term Versus Chronic Stress

In terms of their effects, the difference between short-term and chronic stress is one not of degree, but of kind. Short-term stress enhances memory; chronic stress impairs it. Short-term stress boosts the immune system; chronic stress weakens it.

(Seeing short-term and chronic stress as fundamentally different is not just my own heterodox personal take. This is the view of people who study stress for a living, including the renowned stress neuroscientist and primatologist Robert Sapolsky, professor of Neurology at Stanford University, who writes extensively about this key distinction in his excellent book, Why Zebras Don’t Get Ulcers. The American Academy of Pediatrics emphasizes this distinction in its statement on early life stress. And the Harvard statement on child development, cited by Dr. Narvaez herself in her Psychology Today piece, makes a point of distinguishing between short-term and long-term stress.)

The Effects of Stress in Early Life

Short-term stress mobilizes us for action–the classic fight-or-flight response. Adrenaline and cortisol pump through our veins; our bodies brim with energy; and mentally we become hyper-focused.

But these short-term adaptations are harmful when switched on for too long, especially when we are young. Scores of animal and human studies show that early life stress, such as severe early social deprivation, leads to long-term changes in the brain, cognitive and social problems, and heightened susceptibility to anxiety, depression, and drug abuse in adulthood. Chronic stress is toxic.

But from this can we conclude that all early stress, even short-term stress, is harmful?

No. Absolutely not. In studies of short-term stress early in life, occurring within the larger context of a close caregiver-infant relationship, none of these ill effects are observed.

In fact, young monkeys exposed to early short bouts of stress, such as brief periods of separation from their mothers, become more resilient to future stresses. They are less anxious and have less extreme physiological reactions to stress later in life. This phenomenon is so consistent that researchers have labelled it stress inoculation.

My Take

So where does that leave us? A little stress, even in infancy, is fine, if not beneficial, but too much for too long is very, very bad.

Do we know exactly where sleep training fits in this spectrum? Just how much stress does a baby experience during cry-it-out?

The short answer is that we don’t know for certain. Everything we do know, however, suggests that this amount of stress, in the context of a warm, loving family, is just fine.

To see why, let’s return for a second to the American Academy of Pediatrics statement on early life stress, which provides examples of the types of stress children can withstand, provided they occur within a broader context of loving, supportive relationships. These include “the death of a family member, a serious illness or injury, a contentious divorce, a natural disaster, or an act of terrorism”. By comparison, sleep training seems pretty mild.

But I would go further. I believe that sleep training is not only not harmful, it is beneficial. Successful sleep training can decrease depression and chronic stress in the parents, and this benefits parents and their babies. Unlike sleep training, having a depressed mother during early childhood has been shown, repeatedly, to be linked with worse long-term outcomes for children.

Which brings me to what I find most troubling about the claims of sleep training opponents: Their zero-sum take on parenting. Worrying about your own sleep needs is selfish, they not so subtly imply. Any time you fail to put your baby’s needs before your own, you are potentially doing him harm.

What a narrow, cramped view of parenthood.

No one would ever dispute that parenthood entails enormous sacrifices, especially when your children are young and their need for you feels so endless and all-consuming.

But I think that because parenthood, and motherhood in particular, is so often judged in terms of self-sacrifice, we tend to forget that a primary job for parents is to be strategic.

As parents, we must weigh short-term costs against long-term harms, because our children cannot. We have to consider the risk of a few nights of stress and unmet needs against the risk of a car accident or job loss, and against the serious physical and emotional toll of chronic sleep deprivation on the entire family.

We can forget, too, that the parent-child relationship is one not only of sacrifice but also of profound mutual benefit. You being a whole, fulfilled individual with a solid relationship with your partner, meaningful social ties, and a sense of purpose enriches your world and your child’s world. You being a well-rested, healthy, and happy parent is good for you and good for your child.

References

Ashokan A, Sivasubramanian M, Mitra R. Seeding Stress Resilience through Inoculation. Neural Plasticity. 2016;2016:4928081. doi:10.1155/2016/4928081.

Center on the Developing Child (2010). The Foundations of Lifelong Health Are Built in Early Childhood. Retrieved from www.developingchild.harvard.edu.

Crofton EJ, Zhang Y, Green TA. Inoculation Stress Hypothesis of Environmental Enrichment. Neuroscience and biobehavioral reviews. 2015;0:19-31. doi:10.1016/j.neubiorev.2014.11.017.

Darcia Narvaez. Dangers of “Cry-It-Out”: Damaging children and their relationships for the longterm. Psychology Today. 2011.

Fenoglio KA, Brunson KL, Baram TZ. Hippocampal neuroplasticity induced by early-life stress: Functional and molecular aspects. Frontiers in neuroendocrinology. 2006;27(2):180-192. doi:10.1016/j.yfrne.2006.02.001.

Gunnar M. Reactivity of the Hypothalamic-Pituitary-Adrenocortical System to Stressors in Normal Infants and Children. Pediatrics. September 2, 1992;90(3):491.

Hsiao YM, Tsai TC, Lin YT, Chen CC, Huang CC, Hsu KS. Early life stress dampens stress responsiveness in adolescence: Evaluation of neuroendocrine reactivity and coping behavior.Psychoneuroendocrinology. 2016 May;67:86-99.

Lyons DM, Parker KJ, Schatzberg AF. Animal Models of Early Life Stress: Implications for Understanding Resilience. Developmental psychobiology. 2010;52(5):402-410. doi:10.1002/dev.20429.

Mindell JA, Kuhn B, Lewin DS, Meltzer LJ, Sadeh A; American Academy of Sleep Medicine. Behavioral treatment of bedtime problems and night wakings in infants and young children. Sleep. 2006 Oct;29(10):1263-76.

Parker KJ, Buckmaster CL, Sundlass K, Schatzberg AF, Lyons DM. Maternal mediation, stress inoculation, and the development of neuroendocrine stress resistance in primates.Proceedings of the National Academy of Sciences of the United States of America. 2006;103(8):3000-3005. doi:10.1073/pnas.0506571103.

Sánchez MM, Ladd CO, Plotsky PM. Early adverse experience as a developmental risk factor for later psychopathology: evidence from rodent and primate models. Dev Psychopathol. 2001 Summer;13(3):419-49.

Sanchez MM, McCormack KM, Howell BR. Social Buffering of Stress Responses in Nonhuman Primates: Maternal Regulation of the Development of Emotional Regulatory Brain Circuits. Social neuroscience. 2015;10(5):512-526. doi:10.1080/17470919.2015.1087426.