Paul Mitchinson is a part-time writer and a full-time father of two. He writes when he can. » more about me

I received a gently critical remark from a reader, and an entirely reasonable objection from Seth Roberts himself, in response to my feverish rant against Freakonomics from a couple weeks ago.

A brief response. First of all, Seth Roberts’ research is fascinating and often illuminating. But the language used by the Freakonomics authors was (typically) imprecise and misleading. Roberts, they write, “discovered” a “fix” for lifting his mood. He also “discovered two agents capable of tricking the set-point system” in his diet regime.

But this is not what Roberts was claiming in his paper.

“Scientists sometimes forget about idea generation,” he wrote in his 2004 article, “Self-experimentation as a source of new ideas: Ten examples about sleep, mood, health, and weight.” He added: “It is not easy to come up with new ideas worth testing, nor is it clear how to do so.” Roberts, in other words, wasn’t testing hypotheses — he was trying, in his own words, to “generate plausible new ideas” that might be the future object of genuine scientific research. That’s why it is so misleading for him and others to bring up the example of Barry Marshall, the Australian gastroenterologist who drank Helicobacter pylori bacteria to show that they can cause ulcers. Marshall’s self-experimentation was prepared for by years of preliminary idea generation and scientific testing. His experiment was a bold confirmation of the truth of his general hypothesis — not a way to generate a new idea that might need further testing.

I’m not convinced that Roberts fully appreciates the need for rigorous scientific testing of his hypotheses. He challenges me by asking:

how do you explain the numerous success stories posted as comments? I count ten successes and two failures as of today (24 Sep 05). [I think Roberts is referring to this — PM]

But the sample group is utterly useless for the purposes of scientific testing. These are highly motivated people anxious to discover a successful diet regime. New diets always boast fantastic successes in their early days — there are obvious psychological reasons to explain this. People who have failed in their diets rarely like to advertise their failure — even under the cloak of Internet anonymity.

So by all means try Seth Roberts’ diet. And go live in a cave if you believe that Stone-Age man had the key to happiness. But don’t try to tell me these are scientific “discoveries,” or even “theories” that have been subjected to serious scientific testing.

2 Responses to “Seth Roberts and Freakonomics Revisited”

  1. Paul,

    Thanks for the explanation. I completely agree with the distinction you draw between my work and previous self-experimentation such as Barry Marshall’s. That was a point I tried to make in my paper, that self-experimentation could be used for idea generation, not just idea confirmation or demonstration.

    I am less sure than you that comments on blogs are “utterly useless for the purposes of scientific testing.” The flaws are pretty obvious; but so are the flaws in conventional scientific tests. A “scientific” study of a weight-loss plan uses subjects who may be paid for participation or receive other benefits (such as blood tests); thus their motivation may be different from the average dieter, who just wants to lose weight. Any study will involve considerable data collection, which is time-consuming; thus the subjects must be more motivated than the average dieter. Such studies usually have high drop-out rates (such as 50%), so that it becomes even less clear how representative the final sample is.

    I am also less sure than you that “people who have failed in their diets rarely like to advertise their failure — even under the cloak of Internet anonymity.” I can’t think of a good reason that blog comments should be biassed toward success, much less heavily biassed.

    Thanks for your nice comments about my work.


    Seth Roberts

  2. Paul,

    I’m sure you’re a well meaning guy, but unfortunately in this case Seth Roberts conclusions on fructose sugar are right on the money. I’ve got 20 lbs. of weight loss so far to prove it.

    Maybe you’ll use what I’m going to say to actually bolster your argument, however I would think Seth could say the same.

    1. I eat less now that I drink the fructose during the day. Mind you, I’m a foodniks foodnik. Love tasty foods.

    2. I avoid desserts and overeating far more than before the fructose. Aha, you say, that’s why you’re losing weight. Paul, it’s the fructose taking away that hunger edge that allows me to avoid them. 8 years of trying to cut them out didn’t work, but the fructose did.

    It looks like your trying to find reasons to dis Seth’s work. Lighten up, open up your mind and I’ll tell you why.

    At some processed food company right now, they’re freaking out (no pun intended) at Seth’s work and hoping it doesn’t get around too much. At some point in the future they’re going to open up the document vault in that processed food company only to find out they knew all along what Seth knows following his experiment. It will be discovered that they knowingly used it to profit from by raising the weight set points of hundreds of millions of American’s, basically fattening them up for the proverbial slaughter of a hundred health problems.

    David Zemanek

    David Zemanek