I received a gently critical remark from a reader, and an entirely reasonable objection from Seth Roberts himself, in response to my feverish rant against Freakonomics from a couple weeks ago.
A brief response. First of all, Seth Roberts’ research is fascinating and often illuminating. But the language used by the Freakonomics authors was (typically) imprecise and misleading. Roberts, they write, “discovered” a “fix” for lifting his mood. He also “discovered two agents capable of tricking the set-point system” in his diet regime.
But this is not what Roberts was claiming in his paper.
“Scientists sometimes forget about idea generation,” he wrote in his 2004 article, â€œSelf-experimentation as a source of new ideas: Ten examples about sleep, mood, health, and weight.” He added: “It is not easy to come up with new ideas worth testing, nor is it clear how to do so.” Roberts, in other words, wasn’t testing hypotheses — he was trying, in his own words, to “generate plausible new ideas” that might be the future object of genuine scientific research. That’s why it is so misleading for him and others to bring up the example of Barry Marshall, the Australian gastroenterologist who drank Helicobacter pylori bacteria to show that they can cause ulcers. Marshall’s self-experimentation was prepared for by years of preliminary idea generation and scientific testing. His experiment was a bold confirmation of the truth of his general hypothesis — not a way to generate a new idea that might need further testing.
I’m not convinced that Roberts fully appreciates the need for rigorous scientific testing of his hypotheses. He challenges me by asking:
how do you explain the numerous success stories posted as comments? I count ten successes and two failures as of today (24 Sep 05). [I think Roberts is referring to this -- PM]
But the sample group is utterly useless for the purposes of scientific testing. These are highly motivated people anxious to discover a successful diet regime. New diets always boast fantastic successes in their early days — there are obvious psychological reasons to explain this. People who have failed in their diets rarely like to advertise their failure — even under the cloak of Internet anonymity.
So by all means try Seth Roberts’ diet. And go live in a cave if you believe that Stone-Age man had the key to happiness. But don’t try to tell me these are scientific “discoveries,” or even “theories” that have been subjected to serious scientific testing.