How Now Brown Cow: The “Chocolate Milk Study” Revisited

You’ve probably heard it before; some smug “science-loving” bloke trying to portray Americans as ignorant and unscientific by saying, “Did you know there was a study that showed that 10% of Americans Think Chocolate Milk comes from brown cows?” when an issue regarding stupidity amongst Americans comes up.

The media milked this study; pretty much every American Journalism organization wrote about this study to “show” how many Americans believe something incredibly ridiculous and “prove” to their readers that Americans are ignorant, scientifically illiterate idiots.

However, if you took a look at this fabled study, then its clear that the individuals making this lofty claim aren’t as scientifically literate and intelligent as they think they are; the study has multiple flaws that affect its validity as a proper scientific study.

This topic came to mind when discussing it with The Roundup’s Editor in Chief Constantine Lambridis in one of our episodes of Season 6 of the Colt’s Corner Podcast. I cited this study as an example of the mass ignorance of the American populace.

However, Constantine was skeptical, marking the first time I had ever heard someone question the validity of this study, prompting me to study the study. This is so meta; if only there were a word for meta-studies, get on it, Merriam Webster.

For one, the study itself isn’t publicly available, and the only way to find more information about it is to read highly sensationalized media articles. Immediately, this is a big red flag regarding the validity of the study. Most respected scientific journals and research organizations make their research available online, available to those who pay a fee or belong to an academic institution.

Because of the lack of concrete information about the study, we have no way of knowing if the sample size was a legitimate representation of all of America or if the questions were worded poorly. Sample sizes that don’t accurately represent the groups they are trying to research and confusing questions are all factors that can negatively affect the validity of a study.

What we do know is that the sample size of the study was 1000 individuals. We also have no idea who these people are, so you can’t necessarily draw an immediate parallel to the entirety of the American populace, like many media outlets have done.

The only insight to the questions asked was from an NPR interview with Jean Raglie-Carr, the President of the National Dairy Council, the group responsible for putting out the survey.

Carr stated that the responses to the question along the lines of “Where does chocolate milk come from?”(we still don’t know the exact way this question was phrased). were: Brown Cows, Black and white Cows, and “I don’t know.”

Obviously, the pre-selected responses for the participants are flawed. For one, there isn’t an option for someone who knows that chocolate milk does not come from a specific cow color and that it is just a mixture of milk and other ingredients. Its also true that chocolate milk can “come” from a brown cow, in the sense that a brown cow can produce the milk that was used to create said chocolate milk.

Also, the phrase “comes from” is not a very good choice. It could mean directly from the cow’s teat, or does it mean partially responsible for making one of the ingredients present in chocolate milk?

When we look at the actual percentages of responses, 7% of respondents stated that chocolate milk “comes from” brown cows, and another 48% did not know. While it wouldn’t be entirely implausible for a valid scientific study to show that 7% of a diverse and representative sample size representing America would believe chocolate milk comes from brown cows, the fact that supposedly “half” of America did not know the correct origins of chocolate milk, makes you question its credibility or the questions’ wording. Maybe more people did know the true origin of chocolate milk but weren’t given a good enough response to reflect that.

Even worse than how the actual study was conducted, in my opinion(it’s the opinion section, I can say that), was the way the media covered it to draw attention to headlines and draw clicks.

First of all, NONE of these organizations could even view the actual study, with one interview with a member of the NDC being the only inside look at this study. It’s pretty disingenuous for journalists to write an article about a study they didn’t even read.

Many journalistic organizations, even some very well-respected ones, sensationalized the heck out of the weak study to grab a good headline, drawing invalid conclusions that don’t make much sense but look good to readers.

For example, many articles boldly stated in the title that 16.4 million Americans believe this to be true. When we look at the study, it’s not that many. Yes, proportionally comparing the study to the entirety of America’s population, it works. Still, many of the articles failed to mention that the study only had 1000 participants, and there weren’t truly 16.4 million Americans involved. 16.4 million Americans, sounds much more shocking than 70, so it’s no surprise the articles omitted this.

The fact that many respected journalistic organizations drew weak conclusions from a poorly designed study, that the writers weren’t able to view themselves, is an example of incredibly bad journalism.

But probably the most egregious thing to come out of this situation is how it was embraced by folks who believe they are so intelligent and love to cite studies on every possible occasion, despite never understanding scientific literature themselves.

The rapid spread of this story by reputable news organizations sparked a fire in individuals who believe themselves incredibly well-read. When anything regarding the American populace comes up in conversation, these people will quote study after study to be the next Green Day and call the majority of Americans stupid and dumb, effectively.

These people staunchly believe that the rest of the American populace is so ignorant and scientifically illiterate compared to them, so when a clickbait article featuring a flawed study that paints “millions of Americans” in an ignorant light, its conformation bias city for these folk.

The study is frequently embraced and cited as an example of alarming mass ignorance by “millions”(just 70 actually) of Americans, “Did you Know 16.4 MILLION Americans think Chocolate Milk Comes from brown cows? Ugh, why are Americans so ignorant!”

While this person continues to whine about the ignorance of everyone else, they are demonstrating they themselves are also exceedingly ignorant and suffer from excessive confirmation bias. Ironically, they are the ignorant ones in this situation, peddling a flawed study that they can’t even read themselves(unless they work for the National Dairy Council).

So moral of the story, the next time someone quotes a study, especially if they are doing to make others look bad, make sure you fact check them because they probably aren’t interpreting the study correctly or drawing a flawed conclusion from poor studies that they read from a single headline.