The Middlemiss Study Tells Us Nothing About Sleep Training, Cry-It-Out, or Infant Stress

Last week, I wrote a post about sleep training and stress, in which I argued that everything we know about stress suggests that sleep training is not harmful.

In response, many people objected that sleep trained babies continue to experience elevated cortisol and significant distress, even after they have stopped crying. In their view, sleep training teaches babies that crying does not help. They haven’t learned to self-soothe or to fall asleep on their own, they’ve simply given up.

What a heartbreaking thought. And one that surely strikes fear in the heart of many parents.

So it’s important to realize that this claim comes from a single small and deeply flawed study of 25 babies, led by Wendy Middlemiss, a researcher at the University of North Texas’s College of Education.

Here is a typical example of how her study is described in the popular press:

“The researchers found high levels of cortisol, a stress hormone, in both the mothers and the babies during the times the babies were crying. After several days, the babies learned to go to sleep without crying. Researchers found that during these quiet nights, the mothers no longer had high cortisol levels but the babies’ cortisol levels remained high. They had merely learned to remain quiet while distressed.

The researchers noted that this was the first time the mothers and babies had not been in sync emotionally. The mothers no longer had high stress levels, not realizing that their babies were still just as upset.”

I can see how many parents would read something like this and swear off sleep training.

Here’s the truth, though: Nothing in her study supports these claims.

To see why, let’s start by briefly reviewing her study’s design. Middlemiss studied 25 mother-infant pairs who were enrolled in a sleep training program at a local hospital. The babies ranged in age from 4 to 10 months.

Mothers spent the day at the hospital with their infants, helped prepare them for sleep at naptime and bedtime, and then retreated to a hallway outside the room where they could hear them, but their infants could no longer see them. The nurses put the infants down in their cribs and let them cry, without soothing, until they fell asleep. This process was repeated for 4 days.

On the 1st and 3rd nights of this program, Middlemiss measured the babies and mothers’ cortisol levels, a hormonal marker of stress. She tested their cortisol levels once right before bedtime and then again 20 minutes after the babies had fallen asleep.

So what’s wrong with her study? Well, a lot.

(1) The study lacked a control group. Without a group of control babies who were put to sleep by nurses in the hospital, but who did not experience sleep training, we cannot say whether sleep training affected infants’ cortisol levels, or whether something else about the program, like being put to sleep in an unfamiliar room in a hospital or being put to sleep by a stranger, affected infants’ cortisol levels.

 (2) She does not analyze her data correctly. Let’s take her findings one at a time.

Claim #1: Babies cortisol levels remained “high” before and after falling asleep on the first and third nights of sleep training, while mothers’ levels dropped after their babies had fallen asleep on the third night.

Problem #1. Middlemiss does not report a baseline cortisol level for the babies. We do not actually know whether infants’ cortisol levels were “high” or “low” or “normal”. She calls them high. But we have know way to know that they’re high; they stay constant throughout the study.

When Melinda Wenner Moyer, a reporter for Slate, asks why she calls their cortisol levels high, Middlemiss responds that she also assessed the babies’ cortisol levels while at home, and the levels were lower than at the hospital. But she never reports these baseline levels in her paper.

To anyone who has published a scientific paper, this is a baffling response. If you are drawing conclusions based on data you collected, you report that data. That’s the way it works.

Problem #2. She uses the wrong statistical analyses to compare before and after cortisol levels. To me, this is the most egregious problem with her research.

Here is how she describes her analyses:Screen Shot 2016-04-20 at 1.55.04 PM

In stats-speak, Middlemiss compared whether the group means before and after sleep training were significantly different from one another. This is the wrong analysis. She should have used a repeated measures analysis. Simply comparing group means is incorrect. It is also considerably less powerful (it fails to take into account that you already know something about the individuals the second time around) and thus more likely to lead to a false conclusion of no change.

Let me illustrate why using an analogy. Imagine you have a group of 25 students who enroll in an SAT prep class. You compare their test scores before the class has begun with their tests scores after the class ends. The mean test score among students does not increase. Does this imply that the class was worthless?

Well, maybe, and maybe not. What you really want to know is not whether the group mean is higher, but whether on average the students improved. And that is actually a different question. For example, if 90% of the students improved by say, 50 points, while 10% dropped by 500 points, the means would remain the same, despite the vast majority of students improving.

Problem #3 Middlemiss is missing a ton of data, and–you guessed it–she does not handle that issue correctly either.

Look back how she describes her analyses. Do you see how the number of mothers and infants in each group changes from before and after sleep training? Consider the mothers’ cortisol data on the third night. The pre-sleep group includes cortisol samples from 17 of the 25 mothers. The post-sleep group includes cortisol samples from 12 of the 25 mothers.

This raises some questions: Are these 12 mothers a subset of the first 17? Or do these 12 include mothers not included in the before group? Middlemiss never tells us.

Why is missing data a problem? Let me again use an analogy. Imagine I have a bag of 25 apples. First I pull 17 apples out and weigh them, and then put them back. Next I pull out a second set of 12 apples out and weigh those. The second set of 12 apples weigh less, on average, than the first 17. Can I conclude that apples in the bag have lost weight?

Of course not.

Now, missing a sample or two is not a huge deal, even in a relatively small study. Large studies can often handle significant amounts of missing data, provided data loss occurs more or less at random. But Middlemiss is missing samples from over half of the mothers on the third night!

At the very least, if you wanted to test whether the women’s cortisol has dropped after sleep training, you ought to compare the same mothers before and after the their infants fell asleep. But she does not do that.

So, was the babies’ cortisol high? I don’t know. Did the mothers’ cortisol drop? I don’t know. It’s impossible to tell from what she reports.

(Note that this should never be the case in a scientific publication. The whole point of a scientific publication is to make what you did clear enough that someone else could replicate your study and analyses.)

Claim #2 Mothers’ and babies’ cortisol levels were no longer correlated after the third night of sleep training. 

This is what she reports:

Screen Shot 2016-04-20 at 2.29.05 PM

Problem #4 Here again Middlemiss uses the wrong statistical test. She claims that the mothers and babies cortisol levels no longer correlated after the third night of sleep training, because the second correlation, r(10)=.422 is not statistically different from zero. This is the wrong test.

She should have tested whether post-sleep correlation of r=.422 is significantly different than the pre-sleep correlation of r=.582. A significant difference between the correlations is, after all, what she claims to have found.

And by the way, the two correlations are not significantly different from one another.

Problem #5 Even if Middlemiss had performed the correct test, we would still have a major problem, because, yet again, she’s lost over half of her sample! We have no way of knowing whether the pre-sleep mothers-infant pairs are the same, largely the same, or completely different from the post-sleep mother-infant pairs. She nevers tell us.

Problem #6 Her entire argument boils down to 10 mother-infant pairs. That’s too small of a number to tell much of anything. To see why, note that if she found a correlation of .422 in her entire sample of 25 mother-infant pairs, it would have been significantly different from zero–meaning her whole argument rests not just on the wrong analysis but on a lack of statistical power.

In sum…

What did Middlemiss and colleagues conclude from these non-findings? 

That infants continue to experience physiological distress, as measured by cortisol, despite being able to soothe themselves to sleep, and that sleep training leads to an “asynchrony” in mother and infant cortisol levels.

What do I conclude?

That this study should never have been published, at least not in its current form. Middlemiss and colleagues had no control group. They performed the wrong statistical analyses. They had huge amounts of missing data which they did not account for at all. Her key findings relied on less than half of her original sample of 25.

These are not minor, nitpicky problems. These are major, glaring problems that make interpretation of her findings impossible.

Warning women against sleep training on the basis of this study is absurd.


Back Sleeping During Pregnancy and the Sydney Stillbirth Study

Pregnancy can be cruel. When you’re at your most swollen, bloated, and exhausted, sleep proves frustratingly elusive. Every night, you toss and turn, trying to find a comfortable position, your back aching, and your belly pressing down on your bladder. Just as you finally start to drift off, you realize you need to pee.

And then, to add insult to injury, you read this:

“After 16 weeks of pregnancy, experts advise women to not sleep on their backs, but rather should lie on their sides, ideally the left side.” – mamalette

Pregnant women are often warned against sleeping, even lying on their backs while pregnant, but with little scientific justification.

The current alleged basis of this advice is the three studies that have linked back sleeping with late stillbirth (pregnancy loss after 28 weeks). (Interestingly these warnings predated the three studies, so they are not exactly the reason women are told to avoid back sleeping)

I described the first two studies, one conducted in Ghana, the other in New Zealand, in an earlier post, and concluded that not only did they provide no reason for alarm, they certainly do not justify blanket advice again back sleeping.

In 2015, a third study came out linking back sleeping with late stillbirth. Does it change the overall picture?

Continue reading “Back Sleeping During Pregnancy and the Sydney Stillbirth Study”

Bedsharing and SIDS: Why I Chose to Bedshare with My Second Child

How risky is bedsharing with your child? And if you choose to bedshare, how can you do so as safely as possible?

My second baby slept in bed with me, all night, every night, from the time we took her home from the hospital until she was 3 months old. At first, I was almost too terrified to fall asleep, for fear that I would roll over and suffocate her.

After all, nearly all major medical organizations warn against bedsharing, on the grounds that it increases the chances of Sudden Infant Death Syndrome (SIDS).

“The safest place for your baby to sleep is in the room where you sleep, but not in your bed. Place the baby’s crib or bassinet near your bed (within arm’s reach). This makes it easier to breastfeed and to bond with your baby,” according the The American Academy of Pediatrics.

Statements like these sound definitive. But, in fact, considerable scientific controversy surrounds the role of bedsharing in SIDS.

Continue reading “Bedsharing and SIDS: Why I Chose to Bedshare with My Second Child”

Let’s Face It: Formula-Fed Babies Sleep Better

Breastfed babies tend to arouse from sleep more easily and sleep for shorter periods of time. Nearly all babies who sleep through the night by 3 months are formula-fed.

Breastfeeding is a major battleground of the modern mommy wars. In her widely discussed piece in The Atlantic, Hanna Rosin called breastfeeding the “new sucking sound”–replacing vacuuming as the task that shackles women to the house, promotes the unequal distribution of childcare and household duties, and prevents women from reaching the upper echelons of professional success. The benefits of breastfeeding have been oversold, she claims, and–just as significantly–the costs to women’s sleep, time, and career progress have been downplayed.

On the other side of the debate, the American Academy of Pediatrics states that the benefits for the infant in terms of reduced risk of infection, adult obesity, allergies, and asthma are so great that breastfeeding must be viewed as an “investment in your child’s future” rather than a “lifestyle choice.” Some lactation consultants fall into this camp too, needing to be reminded to suppress their impulse to sigh when yet another mother complains of exhaustion and lack of sleep, for fear they alienate her–and fail to convince her to keep breastfeeding.

On both sides, well-intentioned but overzealous advocates twist the evidence on breastfeeding, cherry-picking among studies to support their preexisting views.

This is especially true when it comes to one of breastfeeding’s major downsides: disrupted sleep.

Consider the post, 5 Cool Things No One Ever Told You About Nighttime Breastfeeding, which claims that the number 1 coolest thing about nighttime breastfeeding is “breastfeeding moms actually get MORE sleep than their formula-feeding counterparts,” and concludes with the rhetorical question: “Did you ever think, when you hear your baby rouse at 2:00am, that they are actually giving you the gift of MORE sleep…?”

To which I would like to respond: No, never, not only because it does not square with my own experience, but also because the research on this topic is clear: breastfeeding moms, on average, get less sleep, not more.

Almost without exception, studies on formula feeding, breastfeeding, and sleep find that breastfed babies wake up more often than formula fed ones at night, and breastfeeding mothers therefore get LESS uninterrupted nighttime sleep.

Nighttime Wakings in Formula-Fed Versus Breastfed Babies

Continue reading “Let’s Face It: Formula-Fed Babies Sleep Better”