That recent study on bullshit is bullshit.

You may have heard of a recently published paper, which claimed that “people who accept profound-sounding bullshit” like

“Hidden meaning transforms unparalleled abstract beauty.”

have lower intelligence.

The study is bullshit. Here’s why.

They had college students take a survey in exchange for credit. Right away you should suspect the study, because hey, free credit for an online quiz? Sure! Just fill it out randomly, it takes no time at all.

To their credit, the researchers included an “attention check”, which was designed to measure whether students were paying attention.

Participants were also given an attention check. For this, participants were shown a list of activities (e.g., biking, reading) directly below the following instructions: “Below is a list of leisure activities. If you are reading this, please choose the “other” box below and type in ‘I read the instructions’”. This attention check proved rather difficult with 35.4% of the sample failing (N = 99). However, the results were similar if these participants were excluded. We therefore retained the full data set.

Their sample size was just under 300, and fully 1/3 of these of these students failed to answer a question designed to test whether people followed the instructions.

In other words, this test was designed to see if the students were bullshitting the researchers. Fully one third of them were! The researchers left these bullshitters in the study, because the bullshitters’ responses looked just like the non-bullshitters’.

In other words, this study — which was based upon the idea that some people can’t tell bullshit from meaningful information — says that people who bullshit the researchers gave the same information as people who were not bullshitting the researchers.

OK.

Then, the researchers had study participants rate the output of some markov chains, in terms of “how profound” they are. This was the scale they used:

  1. Not at all profound
  2. Somewhat profound
  3. Fairly Profound
  4. Definitely Profound
  5. Very Profound.

Is there a meaningful zero for this scale? What’s the difference between “Somewhat” and “fairly” profound. Why is “definitely” more than “fairly”?
Can something even be “fairly profound?” That’s like someone saying they are “fairly outraged”, or “somewhat furious.” Who cares! This is a bullshit study, anyhow.

How did people rate the randomly generated phrases?

The mean profoundness rating was 2.6, which is in-between “somewhat profound” and “fairly profound” on the 5-point scale.

If each student picked numbers at random, you’d get a mean profoundness of 3. Is 2.6 “Fairly close” or just “somewhat close” to 3?

Then the study goes on to make fun of dumb people for believing in dumb things. Haha! Dumb people, can’t even science like us smart people who are not dumb.

Maybe this whole study is a troll. That original “bullshit line” they gave was

“Hidden meaning transforms unparalleled abstract beauty.”

Maybe the hidden meaning was that this paper is bullshit and smart people bought it because it confirmed what they wanted to believe. That’s beautiful. It’s abstract. Has anyone pulled this off before?

These results indicate that our participants largely failed to detect that the statements are bullshit.

On this, we can agree.

share your thoughts!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s