![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
How I Ruined Science and other stories
Nov. 8th, 2010 11:45 pmToday I came across one of those hi-larious comic flowcharts, this one about alternative medicine. Now, it's hardly new or innovative to make fun of 'alternative therapies' (though this is a fairly well-done piece of humour), but I want to draw your attention to one corner of it in particular. That is, the options for those wanting a "wholly 'natural' remedy" and who believe that "Yes, Big Pharma are the devil". The choice is then based on the "Quantity of active ingredients required". "Bugger all" leads to "Homeopathy";* "An unknown, uncontrolled & untested amount" leads to "Herbal Medicine".
This idea of testing has been at the centre of most of the more civil debates I've had or seen about herbal medicines, and it's an important one. Many arguments are marred throughout by both sides' tendency to argue as though more committed to being on a side than to striving towards Truth, no matter what they may claim. That is: typically, someone on the anti-herbal side will point out that little or no medical testing has been done for most herbal remedies. Then someone on the pro-herbal side will either bemoan the lack of funding for testing in most places -- at which point arguments usually end because the opponents see that they are on the same 'side' really, the side of scientific testing, they are just coming into it with differing hypotheses -- or else the pro-herbalist will question the validity of medical testing itself. And that is when it usually gets nasty.**
It's this sort of oppositional attitude, I think, that leads people to ridiculously extreme positions of either disregarding all scientific research, or blindly accepting it all just because it's *~*~science~*~* (though it's worth noting that the latter view seems to be much more prominent among rationalistic non-scientists than practicing scientists or especially scientific researchers). The trouble, of course, is that a lot of scientific research, and -- this excellent article in this month's Atlantic magazine leads me to believe -- medical research in particular, is often filled with methodological flaws. Some are the result of bias or fraud, but many are simply unavoidable, and probably many more are simply oversights. It is simply not healthy -- literally or figuratively -- to accept all research uncritically.
In the above-linked article, meta-researcher Dr. John Ioannidis claims, and has come up with a mathematical proof to demonstrate, that under normal conditions, most medical research turns out to be wrong. Moreover: "His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials." And yet, of course, it would be wrong to say that this is a reason to automatically distrust all medical research -- though it certainly appears to be a reason only to trust randomized trials, and even to take those with a grain of salt. It is still less reason to think we should abandon the concept of medical research altogether. It just means that we need to work to make that research better.
An example from my own life has been niggling at my conscience for years now. ( Read more... )
The good news, though, is that Dr. Ioannidis' work has been exceptionally well-received by the medical community. Yet there is apparently controversy within the meta-research community for exactly the reasons described above: some fear that seeding public doubts about scientific research will simply drive people to seek "alternative" therapies or ignore the medical establishment, or their own health, altogether. I much prefer his proposed solution. To quote the Atlantic article: "We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough."
* As well it should.
** Let us be clear: it also gets nasty because of the anti-herbal camp's tendency to lump herbal remedies together with all other "alternative" therapies, like homeopathy and crystal healing and bullshit like that, and equivocate between them in their refutations; and by the tendency of many proponents of herbal remedies to also believe in bullshit like homeopathy and crystal healing.
This idea of testing has been at the centre of most of the more civil debates I've had or seen about herbal medicines, and it's an important one. Many arguments are marred throughout by both sides' tendency to argue as though more committed to being on a side than to striving towards Truth, no matter what they may claim. That is: typically, someone on the anti-herbal side will point out that little or no medical testing has been done for most herbal remedies. Then someone on the pro-herbal side will either bemoan the lack of funding for testing in most places -- at which point arguments usually end because the opponents see that they are on the same 'side' really, the side of scientific testing, they are just coming into it with differing hypotheses -- or else the pro-herbalist will question the validity of medical testing itself. And that is when it usually gets nasty.**
It's this sort of oppositional attitude, I think, that leads people to ridiculously extreme positions of either disregarding all scientific research, or blindly accepting it all just because it's *~*~science~*~* (though it's worth noting that the latter view seems to be much more prominent among rationalistic non-scientists than practicing scientists or especially scientific researchers). The trouble, of course, is that a lot of scientific research, and -- this excellent article in this month's Atlantic magazine leads me to believe -- medical research in particular, is often filled with methodological flaws. Some are the result of bias or fraud, but many are simply unavoidable, and probably many more are simply oversights. It is simply not healthy -- literally or figuratively -- to accept all research uncritically.
In the above-linked article, meta-researcher Dr. John Ioannidis claims, and has come up with a mathematical proof to demonstrate, that under normal conditions, most medical research turns out to be wrong. Moreover: "His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials." And yet, of course, it would be wrong to say that this is a reason to automatically distrust all medical research -- though it certainly appears to be a reason only to trust randomized trials, and even to take those with a grain of salt. It is still less reason to think we should abandon the concept of medical research altogether. It just means that we need to work to make that research better.
An example from my own life has been niggling at my conscience for years now. ( Read more... )
The good news, though, is that Dr. Ioannidis' work has been exceptionally well-received by the medical community. Yet there is apparently controversy within the meta-research community for exactly the reasons described above: some fear that seeding public doubts about scientific research will simply drive people to seek "alternative" therapies or ignore the medical establishment, or their own health, altogether. I much prefer his proposed solution. To quote the Atlantic article: "We could solve much of the wrongness problem, Ioannidis says, if the world simply stopped expecting scientists to be right. That’s because being wrong in science is fine, and even necessary—as long as scientists recognize that they blew it, report their mistake openly instead of disguising it as a success, and then move on to the next thing, until they come up with the very occasional genuine breakthrough."
* As well it should.
** Let us be clear: it also gets nasty because of the anti-herbal camp's tendency to lump herbal remedies together with all other "alternative" therapies, like homeopathy and crystal healing and bullshit like that, and equivocate between them in their refutations; and by the tendency of many proponents of herbal remedies to also believe in bullshit like homeopathy and crystal healing.
