Rationally speaking, is a presentation of New York City skeptics dedicated to promoting critical thinking, skeptical inquiry and science education. For more information, please visit us at NYC Skeptic's Doug. Welcome to, rationally speaking, the podcast, where we explore the borderlands between reason and nonsense, I am your host, Massimo Yuchi, and as always with me is my co-host, Julia Galef. Julia, where are we going to talk about tonight? Tonight is a very special taping of the Russian speaking podcast.

We are here in front of a live audience in New York City at the university settlement with a fantastic guest. I am so excited to introduce Ben Goldacre. Ben is a physician and academic and award winning science writer. He writes the famous bad science column for The Guardian, as well as the author of two books, Bad Science in 2008 and more recently, Bad Pharma, which just came out a critique of the relationship between the pharmaceutical industry and the field of medicine, which we're going to have the pleasure of discussing with them tonight.

Everyone, welcome Ben Goldacre. So, Ben, I have to tell you that reading Bad Pharma made me realize a danger of ebooks that I had not previously been aware of. So throughout my life, I have occasionally ended up reading books which for one reason or another made me outraged, starting, I believe, with Matilda by Roald Dahl when I was little. And my reaction upon reading these books was always to be overcome with the strong urge to punch the book.

So several no scratch that many times throughout the course of reading the e-book version of Bad Pharma, I nearly punched a hole in my own computer. And so for our audience here and our listeners at home, maybe you could kick things off by explaining why I was so outraged while reading that pharma. What is so bad about pharma? Well. I guess to be clear, this isn't it's not like a conspiracy theory book, I don't think the evil drug companies are poisoning us with things that'll kill you.

And I don't have any vitamin pills to sell, at least not on purpose. And really, the title of this book should have been The information architecture of evidence based medicine has several interesting flaws, many of which are exploited by industry, but all of which are amenable to very low cost fixes. If anybody had the weight or the desire to crack on with it a long time title. Yeah, well, publishers are very reluctant to have sort of overinclusive titles like that.

So we have to settle for bad form. But it's some you know, I'm glad that it made you angry because the real struggle with all of this stuff is that people feel very abstracted away from the real world of flesh and blood and suffering and death when they're dealing with the abstractions of evidence and and forest plots and the fact sizes and confidence intervals. And I think that's part of the story. I think my profession have acquiesced in the face of some appalling problems, really, because we've managed to convince ourselves that they're not relevant or important.

So very basically, they're kind of three chunks to the book. Firstly, trials, which we like to imagine are always fair tests can be flawed by design in such a way that they exaggerate the benefits of treatments. Next, that we endure biased dissemination of the results of clinical trials to decision makers. But the biggest and most important problem by a very long way, is the problem of missing clinical trial data. So the best currently available evidence on this question comes from a systematic review, and a systematic review is immune to cherry picking.

So it's amazing to think actually that in the 1980s, even if you wanted to write a chapter for a medical textbook or a review article about a particular sort of problem in medicine or, you know, the treatment of depression in patients who were in hospital and also had physical illness, you would just kind of think of a few trials that you'd seen recently that you remembered. You might ask a few friends if they had any trials. You flick through the journals that you happen to subscribe to.

Maybe you'd keep your ear out at conferences and you get this very unsystematic collection of data that was vulnerable to all of the biases that you educate people about. And so conscious and unconscious people would cherry pick people would be swayed by their own subjective prejudices and so on. And then in the 1980s, we invented something called a systematic. He described the the databases that you go to, you describe the search terms you type in, and you make sure that you get all of the evidence on one question.

So I'm telling you this so that you understand that there's nothing mischievous or weird when I say that the best currently available evidence from a systematic review in 2010 shows that overall for the treatments that we use today in medicine, the best estimate is that around half of the trials from that era have not gone on to be published and that trials have positive results are about twice as likely to be published as trials with negative results. So this is no small potatoes, right?

That we cannot make informed choices as doctors and patients when half of the evidence is missing. And we've known about this for three decades, we failed to fix it. And I think, you know, people do get angry about this. And I think it's a mistake, really, to get angry at the pharmaceutical industry. It's the medics and the academics and regulators and policy people who we are most entitled to have high expectations of and who have most failed us.

None of these problems are hidden from public view. All I've done is take them from the technical academic literature and deliver what I hope is a reasonably clear explanation. So are you suggesting that that the culprit here is like the coproduced, like in murder on the Orient Express? Everybody did it, but we're all responsible here. Yeah, so and also actually that's what kind of makes it more interesting, because I'm not and I've never really written about fraud.

I don't find fraud very interesting. It's just like you made it up that you're an arsehole, like, thanks a lot for faking your results, but that's really obviously evil and wrong. What's really interesting about about this huge sort of complex, interlocking ecosystem of problems in evidence based medicine that all reinforce each other is that for the most part, the players involved probably don't regard themselves as bad people. They access to the limits of what they would think is acceptable in the kind of grey areas.

And and also we see cultural blind spots because, you know, if if if if you did a sort of basic school research project, age 50. And you deleted half the data points from that so that the line on the little graph when exactly how you think it ought to, we would all I think, agree that you were guilty of research misconduct. Right. But for some reason, we have a cultural blind spot in medicine and in academia about people deleting whole studies with the results that they don't like from the academic record, even though we know that all of the evidence is going to be synthesized together in one place to produce a summary of the evidence and whether something works or not.

And if we delete whole studies, we will distort the overall evidence base and that will result in people making the wrong decision. Although I still don't know that you mentioned that the leading data I still remember the International Congress of Genetics a number of years ago where I looked at a poster and it was this nice little graphic was a straight line through data points. And then I realized that there were two data points. And in the line, remarkably, did not fit.

It went just below one and just above the other. Wow. Anyway, so the statistics are fascinating and we're probably going to go back to that in a minute. But I want to hear a particular story like the one about our reboxetine, which is something that you actually actually use you prescribe for for patients. So so was it what was the story there just to give us a flavor of how these things actually work? Yeah. So Reboxetine is an antidepressant that I myself have prescribed, and it's an antidepressant with a fairly unusual mode of action.

So if somebody hasn't got better on one class of drug, then if you want to try something new, you might try something from a completely different class. I'm a fairly dorky doctor. It won't surprise you to hear. And so I tried to read the literature. So I read a couple of trials which showed that Reboxetine was better than a dummy placebo sugar pill. And I found a couple of trials showing that reboxetine was just as good as any other antidepressant.

In 2010, an extraordinary paper was published by an organization called Equi, which is the German Government Cost Effectiveness Agency, and they described the battle they'd had with Pfizer where they'd gotten rid of them. Yeah, yeah. And they got in touch with Pfizer and they said, look, we are the German Cost-Effectiveness agency. We're doing a summary of all of the evidence to help health insurers in Germany decide if they're going to pay for this treatment or not.

And, you know, we're aware that often not all trials get published and we're aware that unflattering data is more likely to go missing in action. So can you please tell us all of the trials and all of the results on everything that's ever been done on Reboxetine? And Pfizer said no. We said, well, it's really not good enough and we're certainly not going to prove this. But also, this is really dodgy. What's going on? Finally, Pfizer handed it over.

And what we found was that there were three studies showing that the vaccine was better than a placebo sugar pill. They were published seven studies showing that it was no better than dummy placebo sugar pill, and they weren't published. There were two studies showing that it was just as good as any other antidepressant. But data was collected in three times as many patients in a trial which showed that it was worse than other antidepressants and that data wasn't published. Now, overall, when you add all of this stuff up, depending on how you add it together, reboxetine is either completely useless, in which case it only exposes people to side effects or at best, only scathingly better than nothing, but certainly worse than other antidepressants, in which case I wouldn't have used it.

You know, I did something to my patient, which overall, in my view, probably did more harm than good, either because it was a completely useless treatment or because I gave it to them instead of what would have been a useful treatment. And actually, we shouldn't shy away from recognizing that there's an opportunity cost when you give people a treatment that's not so great because you're depriving them of a better treatment. And I think patients reasonably expects that we give them the best treatments.

In fact, actually, I think that's one of the keys to how people let themselves off the hook, because, you know, let's say those two treatments in one class, one will save lives out of one hundred one, save six hundred. Well, which one do you want? You know, if you want the one that's going to save lives out of a hundred, and if you can persuade doctors to use the treatment that only save six lives out of 100, well, the drug company people might feel okay about that because, hey, we've done something that's like we've got this useful treatment.

It saves a lot of lives at 100. But the net impact of you convincing somebody to use an inferior treatment through withholding data bias, trial design and aggressive marketing is where two lives down of every hundred people now. We shouldn't let the fact that that's a bit sort of abstract and remote and takes 40 seconds to explain, let us shy away from the reality. You know, if there were eight people laid out on train tracks in front of a very slow moving shunter, and I saved six of them.

And then I stop and I award myself a point for saving six lives. Well, there's something very slowly and gradually as I congratulate myself, crushes and kills the remaining two people on the tracks. Well, I think you would rightly regard me as a very suboptimal physician, to put it mildly. And yet, for some reason, you know, we don't seem to and we don't seem to regard that as an issue. And I think we've been very unambitious in medicine.

You know, we invented the basic principles of evidence based medicine three or four decades ago, much more recently than we feel comfortable admitting. And then we we sort of swanned around imagining that we have instantiated them, that we are competently implementing evidence based medicine, and we're very simply not. And these imperfections, they're not just real. They are very dangerous. They result in unnecessary loss of life. And it should be okay to talk about that. You know you know, Ben, the parts that so there are parts of your book that that really gave me this sort of feeling of of dejection and and sort of, you know, existential crisis about the futility of it all.

And I could point out that Julia is a pretty optimistic person. I'm pretty chipper down. You did a pretty good job in general. And those parts were the parts that you are alluding to a few minutes ago about the sort of systemic problems where, you know, individuals are all acting, maybe not benevolently, all of them, but but not maliciously. And the sort of the way the systems are set up combined to produce this, that's really, really suboptimal outcome.

Sort of the same feeling I got watching the wire. Actually, just this is horrible. You know, people like like failing because of institutions and the way the institutions interact with each other. I don't. And this is much easier to fix than, like, social inequality. Bold claim. Well, yeah. No, I. Right. So so I felt I felt depressed at the parts about the systemic hurdles. And then I felt sort of frustrated and maybe annoyed at the parts in which the you described how the pharmaceuticals themselves, like Pfizer, were sort of obstructing this process of of us collectively trying to provide the world with the best care possible.

But the person that really riled me up were the parts where the you described how entities that should have been that I thought were in the business of of improving things like the regulatory agencies were actually obstructing the process, maybe not as much as the pharmaceuticals, but to a much larger degree than I would have imagined possible. Can you describe that phenomenon? Well, yeah. I mean, it's truly extraordinary, really. The people have let us down. The most of the people who we should have been able to expect were on our side and the European Medicines Agency, for example.

And when when Cochrane I would hope that Cochrane is like famous to you. Cochrane is an international non-profit, academic body of maybe 14000 academics around the world who produce gold standard summaries of all the evidence for various different, you know, pretty much all of the relevant and interesting questions in medicine and. In about 2000 and seven, the Nordic Cochrane Collaboration, we're doing a systematic review on to weight loss drugs from Orlistat and Rimonabant, and they had this brainwave write.

They went, guys, you know, isn't it? Isn't it funny how we do these systematic reviews? But we know that half the data is missing and we know that there's publication bias. And what we see is the more positive results. And then they thought, well, hang on. Actually, regulators have much more information than we do regularly to see a lot of this stuff. They don't necessarily have absolutely all of it, but they have a lot more than we do.

So why don't we ask the European Medicines Agency? So they wrote a letter to the European Medicines Agency saying, I will cooperate. You might have heard of us international nonprofit, 14000 people making these gold standard summaries that are used almost universally in medicine. And can we have all of the evidence that you've got on the risks and benefits of Orlistat? Rimonabant AMA wrote back and said no. And this triggered a three and a half year long battle between Ukraine and the European Medicines Agency, which resulted in the single most damning finding of maladministration by the European Ombudsman, the people who work in Europe say they've ever seen.

You know, the findings said that the EMA, the European regulator, had failed to give an adequate or even consistent account of why they were withholding the stuff. They kept saying that it had commercially confidential information in it. But, you know, finding out that your drug isn't so great, I can understand that might have commercial implications, but it's not really what it should be. It's not ridiculously confidential. Exactly. And they were saying that it had an individual patient data in it.

And that's kind of true in the full clinical study report. But they're in separate appendices and the administrative burden is like, you know, the ring binder and you have to click the things and you did the thing over and you get to the right tongue and you take that out and put it on the table and then you put the things back and you close the thing. This is not electronic. This accurate binders, OK, depending on how far back you go.

Yeah, well, there's a story I can't tell you about paper archives for about two weeks, but it will make you will get you later on. Yeah. And so, you know, they made all of these claims, which the European Ombudsman said were just absurd. And eventually, after three and a half years, European Ombudsman said, you know, you are in breach of the Declaration of Helsinki, which is the moral code that came out of the trials at Nuremberg that covers all trials, research.

And I just said, look, you know, everybody involved in research has an obligation to make research findings public. You're exposing patients to harm by withholding this information. Your brief public summaries are incomplete, inaccurate and truly extraordinary ruling. And what's the purpose of telling that story is and since 2010, since this damning finding of maladministration by the European Ombudsman, the European Medicines Agency has been forced to share the clinical study reports that they hold. So this isn't all of the clinical study reports on all of the trials for all of the medicines that we currently use.

It's a tiny, tiny subset. They only have them going back a couple of years. They don't even have all the modules for all of the ones that they do have an. But they are now releasing this stuff and they expect to be awarded some kind of certificate for awesomeness and transparency, and it's just extraordinary because it only came about because of the most damning finding of maladministration that people who work in Europe say they've ever seen. And so it's episodes like that that I think are very disappointing.

And and, you know, we can talk about the revolving door between regulators and industry and regulatory capture and all of that stuff. I'm not sure I care about the people stories, I care about what they do, and we should be entitled to expect that regulators have patient's best interests at heart and that that the way that they behave has historically not been very reassuring on that front. Maybe it really will change the AMA making big promises. We will see.

Well, so not surprisingly, you've ruffled a few feathers with this with this approach. And let me let me bring up a couple of quotes from one of the people you chat. Yeah. So one of the people you apparently upset is Stephen Whitehead, who is the Association of the British Pharmaceutical Industry CEO. And you wrote an open letter to you. And I'm just going to read a couple of things just to give people an idea of what it is and then you can comment about it.

The advantage to me. Well, this was published. Yeah. Now, this was published. You did. You responded to it. Oh. So at least I hope this is like a letter that he posted on this was published in the new statement. Oh, yeah, it's exactly that one. So just to give our audience a sort of a taste of what's what's going on here, why did that you seem to be stuck in a bygone era where pharmaceutical companies wine and dine doctors in exchange for signing on the dotted line.

The pharmaceutical industry has been responsible for the development of 90 percent of medicines in the world and through incremental innovation, has helped. The management of many illnesses will need to go back to that minute. And then they do not seek. We mean the industry does not seek trial data. Do I trial data. It is already best practice within the industry to publish all data positive and negative. You don't buy any of that, do you? Well, OK, so I mean, Stephen Whitehead has behaved truly astonishingly over this book.

And, you know, I, I, I'm not sure I fully identify as a sceptic in the sense that I don't really care about God. I'm not an atheist. I'm an atheist. I just find it really boring either way. And I'm lucky not to live in a country that has lots of reactionary Christians in it. And I am not very interested in psychics or ghosts. Quackery, sort of funny to me, and a useful tool for explaining the basics of evidence based medicine.

But in talking to people who write about quackery, it seems to me that. The reason why they don't write about about problems in evidence in mainstream medicine is partly they feel a bit intimidated by the complexity, although it's basically the same thing, but just slightly more sophisticated and bad behavior with the evidence. But also, I think the reason why people write more about outand out quacks is that they behave so badly when you criticize them that they just create fun narratives, right.

They will say that black is white and 90 day they will issue lurid, conspiratorial denunciations. They'll threatened to sue a 12 year old blogger. You know, they'll say, like I thought, what's his name? Who was it? I was Rees-Mogg. And in England, you know, they sent a school kid pictures of his house. I mean, you know, quacks will behave appallingly badly. And but actually, why I've discovered, to my true amazement, is that.

The pharmaceutical industry, 700 billion dollar global industry will actually behave in a very, very similar fashion. So when the book came out, they put out a press release saying that. All of the issues in this book are historic and they have all been fixed, and that's an extraordinarily sober book, which like a lot of it. Especially like the badly designed trials, but that basic undergraduate medical degree stuff, right, like I teach that to medical students, trials are very frequently flawed by design.

There's nothing weird or conspiracy theorist about that. That's like you can't get a medical degree without understanding that stuff, like anybody who believed Stephen Whitehead would fail to qualify as a doctor. And the problem of missing, maybe we should be worried about. Well, the problem of missing clinical trial data, which Stephen Whitehead says has is all historical, has long been addressed. And it's best practice. Well, you know, the best currently available evidence from a systematic review shows that half of all trials for the medicines that we use today.

Have not been published, that positive results are about twice as likely to be published as negative ones, and not just that, but. I think that would be bad enough because that's the evidence base for what we have today, but everything that people claim is fix that problem hasn't just failed. It's actually delivered false reassurance and perpetuates the problem. So, for example, everybody said that the clinical trials dot gov legislation could fix this problem forever. So the FDA Amendment Act 2007, I mean, you could be out sort of smoking crack and injecting heroin and hanging out with models in New York.

But you're listening to me talk about the FDA Amendment Act 2007. This reflects very badly on you. So the FDA Amendment Act 2007 said all trials are conducted and completed with at least one site in the U.S. or for a marketing authorisation, have to post their results on clinical trials within one year of completion. And everybody said, oh, fantastic, this has been fixed forever. No public order. When a public audit was finally conducted and published in one of the top five medical journals in the world in 2012, it found that the rate of compliance with this law, which everybody said had fixed the problem, was 22 percent for trials out of five failed to comply with this legislation.

I would naively say that that's low. Well, yeah, I'd say that's pretty poor and actually. Right. Even if it had worked perfectly, it still would have failed to fix the problem because it only required you to post the results of trials that are conducted and completed after October 2008. Now. Out of all the prescriptions issued over the past year, 80 to 85 percent of them are for medicines that came on the market more than 10 years ago, generic medications.

Now, we don't need the results of clinical trials from 2008. We need the trial results from 2003, 2001, 1998, 1993. And this stuff sold out there. You know, it's in it's in dry documents, storage archives in the chalk hills of Cheshire and decommissioned nuclear bunkers in the Arizona desert or whatever. And we just need that stuff, you know, getting us all of the trials that are conducted and completed after October 2008 that would help improve the evidence based medicine in about 2033.

But I'm feeling quite impatient about this problem right now. I want to make sure I have the evidence base to make informed decisions about my patients now today in 2013. So the notion that this problem has been fixed is is I mean, it's utterly fallacious and. I wouldn't want to make any comment about the motives of any individual, because you have to be careful what you say about this, right. But you could be sued. Well, but I will say as a general point about the pharmaceutical industry as a whole, there's a famous quote from the tobacco lobby.

Right, the leaked memo. Doubt is our product or we have to do is spread a little bit of doubt in people's minds because people don't want to believe that tobacco causes cancer, just our product. Just so a little bit of downright. For industry, there is nothing more important than. Maintaining their ability to withhold unflattering data about the results of clinical trials, the marketing staff, they can take it or leave it the way that the paid advocates, gifts, trips, any of that stuff they can live without.

What matters to them more than anything else is to continue being able to withhold evidence about the effectiveness of treatments, and they will defend that viciously to the very end. And delay is their products. And their main objective is to firstly persuade people that this problem doesn't exist because it's so obviously preposterous as soon as you're given a clean, clear explanation. Half of all clinical trials for the treatments we used today have not been published, and positive results are twice as likely to be published.

Everybody just goes, well, that's obviously absurd, right? What do you mean evidence based medicine? How can you possibly be serious, selective, evidence based medicine? You know, and, you know, I genuinely believe future generations will look back on the way that we've tolerated this enormously. And I look back at us in the way that we look back at medieval blood letters. Right. They'll say you spent tens of millions of dollars, sometimes hundreds of millions of dollars on one clinical trial sometimes.

And you obsess over making it show that it was as free as possible from bias. You used all the sophisticated statistical tools to try and detect very, very modest differences, sometimes between one treatment group and another. And you went to all of this huge effort to produce this perfect evidence. And then after that, the final furlong, you just let half of it get erased from history and not even any of but you let half of the less flattering half be selectively erased.

I it's so ridiculous or shredded in one case. Right. Well, shredded in a in a year. Another future story that I will tell you in about three weeks time. Right. So but this this brings so much delay is their product. Every single thing, like everything that you see from industry is about delaying us fixing this problem. And they will fight on that harder than they will fight on anything else. They will do everything they possibly can to continue the current system where they are able to hide unflattering light so nothing matters more to them.

You've used tonight several times the term evidence or evidence based. Now, one of our friends, Steve Novela, who does, among other things, the Skeptic Guide to the Universe podcast, makes a big deal of making a distinction between evidence based medicine and science based medicine, when in this case, these theories that sort of science is a little higher standard than evidence. The problem being that evidence, it's a little generic. You know, anecdotal evidence is still evidence unreliable, but it is evidence, selective evidence is still evidence.

You can cite it and all that sort of stuff, but clearly wouldn't pass, as you were pointing out a minute ago, or shouldn't pass muster even with an undergrad student in training in medical medical research. So there is this distinction now that we do hear a lot of web about evidence based or evidence based, that I guess one of the problems we're getting at here is that evidence is not a panacea. It's not like, oh, well, I got evidence for it.

Therefore, you should believe me. It really does. It's a question of the quality of the evidence in this case, even the quality of the evidence that has been withdrawn from from the public. But generally speaking, I mean, you can do a sloppy study just because it is in fact, it's not sloppily done, period, without actually any intention of hiding anything. There's been some interesting medical studies of a lot of medical research that have appeared over the last several years that have shown exactly that, that that often the problem is simply with with suboptimal experimental design, even when it's done by non pharmaceutical researchers are not actually funded by the pharmaceutical industry.

Now, one of the things you do point out is that when the researchers are funded by the pharmaceutical industry, somehow the results, the positive results all of a sudden increase significantly is that that's you're not implying that there is necessarily a conscious bias there or on the part of the researchers? Well, there's certainly a you know, industry wants to get positive results. And and, you know, that will that will drive behavior. And, you know, I'm not I don't know what I would call it, whether I'd say there's evidence based medicine, science based medicine.

But the reality is there are you know, there's good evidence and there's bad evidence. And, you know, we've you've got to make sure that you don't allow people to cherry pick data to use badly designed trials. But actually, I mean, we fall down on some really basic stuff. For example, regulators often have this idea that it's okay for them to make decisions behind closed doors. And actually, Roshe have now started saying because I mean, the one thing I'm sort of reasonably pleased with is by moving this discussion ahead in the UK, at least, where the book came out a couple of months ago.

There's now a parliamentary select committee inquiry into missing clinical trial data. There've been questions in parliament, a little bit of. TV and radio stuff here and there, and so we started flushing out people from industry and also elsewhere who will try to defend withholding data. And one thing that a lot of them say is, look, you know, regulators should make these decisions behind closed doors about what works and what doesn't. And that is a really, really interesting betrayal of the scientific method and failure to recognize how evidence and science should work because.

You don't trust it in science because you wear a white coat or you've got lots of letters after your name. Or you should or you shouldn't be, you know, science in as far as it has any authority that derives from transparency, it's about showing your methods and results. Sharing your work in the Royal Society and in London has its motto, like a like a TV show, like a sitcom as its motto, like chipped in stone. And it's millions in Verber on the word of No.

One. Show me a working now. Regulators like to make decisions behind closed doors, and that suits a lot of people. The problem is sometimes regulators make the wrong call. If you look at some of the big problems in medicine, Vioxx, for example, wasn't spotted by regulators, problems that were spotted by independent academics and doctors, problems with rosiglitazone, Avandia not spotted by regulators, but by independent doctors and academics. Similarly, problems with the evidence base for Tamiflu are not spotted by regulators, spotted by independent academics and researchers.

And it's not because regulators are stupid people. They're not. It's because these are very difficult problems. And like all difficult problems, they benefit from many eyes and withholding information about how trials were conducted and what their results were and the full clinical study reports and all of the detailed information by withholding that from doctors and academics, you're just restricting the number of people who can see it and the number of people who can spot problems. And it's okay for people to have discussion and debate.

And actually, you know, I can tell you that regulators that I've spoken to who say, look, if this stuff was freely available, there'd be panic in the streets. Right. What do you mean? What what exactly are you withholding? And actually, I don't think I don't think they mean because people will find out the horrors that we've hidden, Wolf. Well, hopefully not, although in some cases here and there. You know, but but what they mean is there will be public disagreement, and that's completely normal and fine.

I mean, what could be more normal in science than people publicly disagreeing? And it's it's just such a strange call. Well, it's very paternalistic, actually. It's it's a lot like, you know, doctors in the 50s, maybe in the 70s, maybe even now if you're lucky, you know, we would routinely lie to our patients. We say, oh, shut up, I'm not going to tell you that I can shut up and take the pills.

Right. And make your work easier. Well, cuz I and doctors these days, I'm not sure it does actually, but doctors these days. Don't do that. Hopefully, I mean, we teach our medical students to work together with the patient towards an optimum health outcome according to the communication skills and that and and that's real. That's important stuff. You know, and to be fair, the regulators are treating doctors like doctors used to treat patients.

It's shut up. We know best. And that's just preposterous. Yeah, that is a bad idea, it seems. But but to be fair, a little bit on the other side of the same argument that you're making is in terms of access of the public by the public to do this kind of information in terms of transparency. Well, we have had and we still have a lot of controversies in the United States. The most famous one is probably the one about vaccines and autism, where, you know, published studies are available to the public and the public makes a mess out of it because by and large, the public tends to be uninformed about science, unsophisticated about how to interpret science has a nasty tendency to follow celebrities who have a nasty tendency to talk about things they don't understand.

So, I mean, I can see the point of saying, well, yeah, and they also MMR they also like vaccines. But all of those scare stories happened. They they happen anyway. It's it's my hunch that there is like a fixed volume of stupid news stories about health and that fixed volume will be filled with whatever people can find to fill it. So if I just write like two thousand articles about the negative effects of some random plant somewhere, then we're done for the year.

We just have good science for the rest of the year. If we can deliver on in the first week of the year, that would be ideal. And and also, you know, secrecy actively ferments conspiracy theories. And once things are open, people get bored. In the UK, there's something called the Queen's database, which was opened up to the public last year. Tory government have actually been very good about opening up government data and the queen's databases.

Every receipt in the whole of government, over six pounds or something, nine million. And the first week that the Queen's database was published, obviously the newspapers are absolutely stuffed full of it and really vicious stuff like, you know, is this the most wasteful civil servant in Britain? We found that the admin secretary who spent more in florists than anyone else. Yes, really nasty, silly stuff. And then once that week was over, there has not been a single coin's database story ever since.

It's just boring. People go to receipts, go who wants to go through receipts that really boring. And it's the same with clinical study reports. There's no there's no more opportunity to create mischief in CSR than there is in in in straight like brief summaries of clinical trials in academic journals. You know, it's it's an excuse. And and what's really disturbing is to see the lines put out by industry being parroted by people in regulators and by people in government.

And that's scary. You know, Pulcher, junior health minister in the UK, recently said to the the bill committee on the Health Research Authority, serious that you should be hanging out with like models in a punk basement club somewhere. Anyway, the bill committee for the Health Research Authority. Golove, your conception of our outside options tonight? Yeah, we should say that those options, of course, are available only for people were actually physically here, not the people who can't see this at home, but almost everyone here is naked.

They're covered in tattoos, you know, piercings through parts of the body. I don't even know the name of, um, but they're all really captivated by my description of what happened with the junior health minister at the Bell Committee for the new health research. And for some reason, yeah, it's bizarre, isn't it? It's you know, it's true what they say about science being the new rock and roll. And so Health Research Authority, really awesome.

The person who's in charge of it, Janet, wisely, really muscular, strong possibility to it. Right. She's saying maybe Ethics Committee should actually start saying to people, you've got to publish the results of chance that you do. And you've also perhaps even got to prove that you're not withholding the results of any previous clinical trials before you were allowed to touch any more patients, which I think is actually very sensible because it actually wasn't anything I'd say.

I mean, apart from the fact that you're distorting evidence and therefore harming patients by withholding data as if that wasn't bad enough. It's also a betrayal of the participants in the trials, because people who participate in clinical trials as patients in the belief that what they're doing is going to generate new knowledge that will improve the treatment of other people in a similar predicament to them in the future. And when patients find out that half of all trials don't get published, they are quite correctly, really angry and annoyed, as if we've got a bunch of patients in our campaign group, all trials Donette, who've been writing to people saying, can you fix this?

We're trial participants, we're annoyed about this. And an industry had the gall to say, well, if you say this to patients, then people will be less reluctant to participate in trials. It's very sad to stop people participating in chance you go. Right. You want me to to maintain your lie, to avoid frightening people. Why don't we just fix? The problem is utterly bizarre. Behavior. Anyway, Janet Winsley, health researcher, he's going to be muscular in the bill committee that sets up this authority, they're talking about putting that in legislation.

And Pulcher, the health minister, also a doctor, said, well, we're going to be very careful. People might you know, pharmaceutical companies might leave if we do this, you think, oh, my God. I mean, what an amazingly candid, low opinion you have of drug companies that you very openly say they want to hide data and mislead doctors and consequently harm patients. What next? And we better let them do that. Otherwise they leave and go somewhere else.

Exactly. So we went and met the the other health minister with responsibility for the pharmaceutical industry and how everybody in England is an all or ignored, by the way. Yeah, I noticed. Know, um, and, you know, he said, are we going to be very careful because, you know, they're already angry about, um, about the fact that we're not paying them very much for medicines in through the NHS because the NHS sets prices and we've been keeping prices very low, blah, blah, blah.

And you socialists in the socialist utopia of the National Health Service. Yeah. And just the day before the head of the FBI, Stephen Whitehead, the man who says that none of these problems exist, I mean, how that man still has his job is a mystery to me. And he can't even competently deny things or smear. You know what I mean? That's what I find insulting anyway. And the day before, he gives a talk where he said, you know, if if government doesn't pay enough for these pills, we're going to take jobs away.

I mean, it's possible that people take jobs away and people in government were complaining about this as if it was somehow unjust or unreasonable or unfair. Weird. And actually, I completely disagree. This is the normal bread and butter might give us a regional development grant to put a car factory in in Detroit and some tax breaks or we'll take jobs away. This is normal. This is bread and butter. This is normal business money. And we'll give you some jobs.

That's what happens. Right. And what should never be on the table is allow us to withhold clinical trial results, mislead doctors and harm patients, or we will take our jobs away. Right. That that shouldn't be part of the transaction. That there would be like saying in the case of the card industry, you give us the money or we will make cars that are going to break down and kill people, allow us allow us to put like lots and lots of little flaws in the seatbelts or we will take jobs away from, you know, for that matter.

I mean, talking about the incentives that the pharmaceuticals respond to, you made the the striking point in the book that the penalty is that they actually when they when they actually are taken to task for obstructing the, you know, obstructing evidence or for like withholding evidence that that showed that their product was dangerous there. When they are penalised, it's nothing compared to the revenue that they get from their drugs. And it's like, I think you said a parking ticket.

Why why can't they why can't government set penalties higher so that pharmaceuticals actually have an incentive not to put out drugs that hard to bring about? Well, when you say harm, I mean, we have to be absolutely clear. I think it is fairly unusual that drugs come on the market which do more harm than good. It happens. But I think it's really unusual. The harm that comes is mainly because people are deprived of the best currently available treatment, because we are misled about the relative benefits.

So I know we covered that earlier with the six and the eight people being crushed under the diesel locomotive. But yeah, you're right, the penalties are trivial. I mean, GSK, who we perhaps have to be a little nicer about because they just signed up to all trials and we should talk moment. And GSK fined three billion dollars for acts of criminal and civil fraud relating to the mis selling of medicines and withholding clinical trial that just last year, three billion dollars.

But the sum total of revenue just for the drugs covid just for the period covered by the of criminal and civil fraud is twenty seven billion. So it was like 10 percent of revenue during that period. The fines for non-compliance with clinical trials dolgov that law that fixed everything, according to people like Stephen Whitehead, but which has actually been ignored by four trials out of five. That law, even if even if even if penalties had been enforced and the FDA has never, ever find anyone for non-compliance with this law, even if that fines have been enforced, it is ten thousand dollars a day.

Now, I hope that that would be a struggle for any of you to pay. Otherwise, I feel like my life is gone horribly wrong. But three and a half million dollars a year to a company that's making billions of dollars a year to a big research physician, it's a it's a parking ticket subsidy. I've shaken the hands of individuals who earn more than three and a half million dollars a year. Not very many of them around, but I have touched them.

So they exist, you know, and and and, yes, the penalties are are utterly, utterly trivial. But, you know, more than anything else, I find it with. We get preoccupied with the idea of like we have to find ways of forcing people to hand this stuff over because actually we haven't really tried asking. That is the that is the extraordinary extent of the failure of ambition and vision on the part of my own profession and regulators that the clinical trials registers that exist are these kind of small, unsatisfactory, ad hoc, non overlapping or partially overlapping lists.

Clinical trials dot gov is a list of some of the trials that have been conducted in America quite a lot since 2007, but none of the trials conducted elsewhere. Very, very few of them. The European Clinical Trials Register is all the trials conducted in Europe for the past few years. But that's not a useful list of trials. We need a list of all the trials that have been conducted on all of the medicines that are currently in use. And this is a very easy thing to ask for and it should be very straightforward.

Dear X, congratulations. Your medicine is currently available for prescription in the European Union. As you know, we have the European Clinical Trials register. This is a list of all the trials that have been done on any use of any medicine that is currently available for prescription in the European Union. Here are the forms. Just let us know. Fill them out and we'll just pop them all up on the European clinical trials, which those have done that, you know, like that.

It sounds easy, but. Well, but we could try asking, you know what I mean, in a systematic and comprehensive way. And if people started playing silly buggers, then, well, then we could start to think about it. But we actually you know, people talk about evidence based medicine. And I'm a huge Berlei biggest advocate. You come across for this. Right. But. Where's the list of trials like we we cannot even produce a list of all of the evidence that we use in medicine, we don't have a list of all the trials that have been done.

I find that completely bizarre. So, again, however, that doesn't if you think about it in the sort of the broader, broader picture of, you know, just not just medical research, but science generally, that's not really that unusual or surprising. I mean, it shouldn't it should be worrisome. But the trouble in fact, the idea that, you know, I've been done working in evolutionary biology and ecology and I can tell you that a lot of the the the papers that get published are probably, you know, 10 percent or 20 percent of the stuff that actually gets done.

Most of the stuff is never published in languages in sort of some some people's literally drawers now. Now hard drives and things like that. And of course, as a result, you do get these very skewed perspective on a number of issues. Now, of course, when it comes to the, you know, ecology of a local plant species nobody's heard of, well, who cares when it comes to your health, that's a little more worrisome. Exactly.

And, you know, this is a problem that affects everything. If you if you look at the functional brain imaging literature, there are more positive findings than the number of observations should support, which strongly suggests that it's a problem. Yeah. And if you look, there was a fantastic paper in March 2012 in Nature where they just said, OK, 53 basic molecular biology papers that are suggestive of a potential target for for cancer treatment. Let's see how many we can replicate.

The answer was six at 53. And they recognize that the reason why they weren't able to replicate stuff was that people were publishing flix and their recommendation was we should be better at providing vehicles for people to publish negative findings. Now, that's just not sexy. Well, they're not sexy maybe in molecular biology, but they're vitally important in clinical medicine. And it's extraordinary to see. I mean, there are people like John Lamattina, the previous head of R&D at Pfizer, who's now writing, sort of trying to say, oh, well, when a trial has a negative result, that's not really important because it's a failed trial is not fail trial.

It's a really useful piece of information showing that a drug didn't work. Sure, there's a design flaw, you know about that before you do try and you don't find out that there's a design flaw at the end when it doesn't give you the answer that you want. Absolutely. But if you're talking about the numbers we're talking, I can imagine as as an academic, you know, going up for tenure and say here I got 20 publications, 17 of them showed that absolutely nothing happened or nothing interesting happened.

But here I got it published anyway, because their negative result in 2013 is going to look at it some way and ways of avoiding it. I mean, remember, what you talking about is only an issue in in basic science, in clinical trials. It is there are no barriers to publishing negative results. In fact, even when people are obliged to under the law, just by posting a brief summary on clinical trials that they fail to do so four times out, fine basic science research.

That was another thing I was surprised by that. You mentioned in the book that journals actually are not as resistant to publishing at these medical journals, not as resistant to publishing negative results as I thought they were, or even as the researchers themselves thought the journals would be. So I wonder if just better dissemination of information would make researchers more willing to submit their articles that had negative results in them for publication so that the best currently available evidence from the systematic review from 2010 shows that generally discrimination against negative results is not the bottleneck in publishing.

Now, I don't have a hugely high opinion of academic journals. They publish utter dross and they also let people get away with all kinds of analytic crimes. They let people switch the primary outcome in their trial. They let them deal in stupid ways with missing data. They let them get away without paying attention to tree analysis and so on. So there are huge problems with journals. They also have massive conflicts of interest and around accepting studies with positive results and studies that are industry sponsored.

And but I just don't think it's I don't think it's realistic any longer to say that an. That trialist feel that there's there's no outlet for their creative work. And so that's why people are going, oh, yeah. And so that's one problem. You think it's been. That's right. But also, I mean, you know, the real problem that hasn't been fixed is that we haven't invested enough in the sort of supporting stroma of scientific evidence.

We spend tens and hundreds of millions of dollars on just one trial, but we haven't invested in the kind of the basic information architecture of evidence based medicine. We don't even have a list of all the trials that have been done, matching protocols against publications and spotting the differences and the gaps. Now, you know, if you put me in charge of NIH for a year and let me assure you, this is reasonably unlikely to happen any time soon.

And, you know, I would cancel all trials, research for a year. I we can live without another trial comparing ovarian cancer treatments head to head just for one year. And I'm sorry if that disturbs the relationship, but just for one year, I would spend every single penny that we have on better systems for collating the evidence that we already have, systematically putting it all together, summarizing it, disseminating it to decision makers, doctors, patients, health care providers.

And I would add and I'd invest in monitoring the outcomes and and and appraising the benefits and finding the best systems for doing that. And at the moment, this is regarded as a sort of cheap. Second cousin is sort of the cottage projects, you know, every now and then you see a pharmacist in a district general hospital does a project in which they're doing a notes review. He stops patients in an old age ward having more than five drugs at the same time.

But, you know, we need to invest massively in making sure that we properly understand and use the evidence that we already have. And I you know, it's a classic problem in medicine and in science generally. If if your project requires a lot of money and it's sort of glamorous and involves machines that go ping, you sort of accrue there's a halo effect if you see it in psychology departments, you know, the people who do functional brain imaging experiments, well, that requires machines that cost millions of dollars and staff to run them and engineers and stuff like that.

And they kind of accumulate a glamour. And the psycho physicists who just do pen and paper and stopwatch stuff on verbal fluency, they're still getting nature papers, but all they cost is their salary and maybe some notebooks. Right. And and they don't have that same kind of glamour. And that's the problem with the world of clinical trials, research, you know, trials of big admin, heavy projects. They require a lot of manpower. They bring in grant money, they've got overheads and people accrue power around that.

But actually, the projects of evidence synthesis just don't have the same umph and prominence. But, you know, it's the bit that we've neglected. It's it's the bit that we failed on the procedure. The machine beep is definitely across the science. Again, first of all, about the MRI scans, one of the things that always I found always curious is that there is no way you can almost no way you can fail that kind of research. Because you know what?

You're going to show that while you're thinking about something, you're some part of your brain is active. Well, it's worse than that. You know, I used to do functional brain imaging research and I've seen some extraordinary things. I mean. SBM, which is the analytic software that people use. There are so many opportunities to have a second go, you know, you can go, oh, well, nothing lit up, right? Well, maybe we should we should re do I think we should rerun the alignment because I think the head move, maybe if we put in some different alignment parameters and we say, well, we have to convert, every brain is a bit different anatomically, but we normalize them all to the Telerate internal Atlus.

Maybe we should renormalize the brains. Maybe, maybe that'll make things light up nicely. And that's one of the many ways in which you managed to get more positive findings out of the functional brain imaging literature than than the than the data you've collected should allow. So you understand the basic idea that, you know, out of so many experiments with a statistical cutoff of whatever, you should only be able to find a certain number of findings, a positive signals.

And if you're finding a weirdly large number of positive findings, then either either the entire field is full of really lucky people or something's gone wrong. Yes, I'd like to yeah. I remember years ago in the National Science Foundation asking a bunch of ecologists, including myself, about this this issue that, you know, ecology is a low tech discipline and we really need to agree not to go to Congress and get some money. We really need to get some big science like so that we're talking about some expensive.

Yeah, exactly. So we're talking about the equivalent of a tennis ball through the equivalent of a telescope or, you know, or a space probe or something. And that's they they they brought a bunch of us into a room for an entire weekend asking us. So let's say you invent a completely superfluous expense. Yeah, it's pretty much. And so they can they kept telling us. So if I gave you a hundred million dollars, what would you do with them?

And the best and most of my colleagues, including myself, could come up with, well, I could hire a thousand postdocs and do a lot of field work. No, no, no, that's wrong. We want some big one big piece of equipment. And, of course, you know, at some point people got creative and said, well, I could do my research in the in the rainforest with helicopters that help gold plated helicopters got to spend that money.

You know, I mean, there probably been about maybe 300000 trials ever done. Give me give me a million quid, maybe a little bit more. Give me two million quid and I. What's the change exchange rate between the quid and the dollar? Give me give me give me two or three million dollars and I will give you a list of all of the trial registries all linked to matched up and all of the publications. And I will give you a list of which drugs are the worst for missing child.

I see which companies are the worst missing child data. Which companies otherwise missing child data, which countries are the worst for missing child data? Which disease area is the worst missing child data? I'll be able to tell patient groups, hey, you know, you get very upset when people mention that you take money of industry and you want to protest that you're independent. Well, here's your opportunity of the drugs taken by your members. Here's a list.

Here's a list of all the trials that are missing. Here are the data controllers for that. For those trials, you should be writing. You know, the National Rheumatoid Arthritis Society should be writing to the companies that make drugs that their members take and saying we can see from all trials doctor list that the following four trials for treatments that are taken by our members have not been published in seven hundred patients worth of data. Please, can you tell us where it is published to get two or three million quid, two or three million dollars and whatever it is that is you, we can do that at once.

Then before we have to wrap up, would it be possible for you to leave our our audience and our listeners with some of the hacked as well, with some barring that, with some practical advice for how to evaluate medical questions in the meantime, before we reach the happy stage of, you know, actually science based medicine and, you know, true transparency and so on, like, are there some 30, whatever it was? Well, so I'm thinking of something like, is there a particularly reliable source of these systematic reviews on particular issues?

Are there some sub fields of medicine or kinds of drugs that you can trust the evidence on? More like I got the sense from your book, although you didn't address this question explicitly, that drugs for depression were like the studies about drugs for depression where, you know, less reliable on average than studies for other kinds of drugs, although I could be wrong about that, correct me if need be. So I'm really militant on this. I don't do like top five checklists and I don't do readers health advice.

And I, I don't say which things are good in which things are bad. I'm obsessed and sanctimonious about this. I don't think that you can solve any of these problems by identifying like a good drug or a good or good corner of the evidence sphere, and these are systemic failures. And the only way to fix it is by fixing the system's failures so that the thing that if anybody wanted to help after the book came out and we realized that there was a sort of head of steam growing in the U.K., so we set up all trials on that, which was me and some friends.

I said friends are in the British Medical Journal and the Centre for Evidence Based Medicine in Oxford University, in a sense about science for fun and awesome and all trials done that is campaigning for and just to know about the existence of all the trials that have been conducted and completed. The brief summary results in full clinical study reports, minus identified patient data where it's available, and that's supported now by eighty five patient groups, the great and the good of independent academic research funding in the UK.

It quick the German Cost Effectiveness Agency who did the reboxetine story I mentioned earlier. Also GlaxoSmithKline GSK, amazingly. And and if you've got if you're a member of any kind of professional body, whether it's a Royal College or the AMA, it's a conservative organisation, from what I understand, or any patient group or anyone, you know, get them to sign up like nothing. Nothing can work in medicine until we have all of the clinical trials, like everything that we do, everything pretend to be doing is completely broken when half the data is missing and fixing that has to come before everything else.

And and that's where the most fun would be. Right. You know, industry will fight viciously on this. They'll they'll still roll over and they'll make concessions on things like money for doctors, marketing and all of that stuff. But if you try and stop them hiding data, they will fight vindictively and viciously. And that's how you know how much it matters to them. And that's fixing. That's the most important thing. And after this, I challenge anybody here to go to a pharmacy, you know, I mean, that's the real that's the real difficulty, isn't it?

You know, it's still the only game in town. And my judgment and your judgment and my judgment from looking at a lot of these stories is that overall, it's quite rare that there are treatments that actively do more harm than good and that we're mostly misled around what's the best treatment out of what's available. OK, I'll take that as an optimistic, and that's the best I'm probably going to get. So but I do encourage everyone to pick up a copy of Bad Pharma, because Ben has, in addition to the stuff that made me want to punch the book, there is several inspiring lists of things that you can do to start improving the situation.

So pick up a copy of Bad Pharma, maybe a paper copy, not an e-book in case you feel like punching something that's not your computer. And we are now out of time. So, Ben, it's been a pleasure having you as a guest on rationally speaking. Thank you. Thanks for having me. Sorry to be so depressing. This concludes another episode of rationally speaking. Join us next time for more explanations on the borderlands between reason and nonsense.

I'd like to take this moment to remind our listeners that if you're a fan of the rationally speaking podcast, you'll definitely enjoy this year's Northeast Conference on Science and Skepticism, which will be held in New York, New York the weekend of April 5th through 7th, 2013. Go to NextG now to get your tickets there on sale in addition to Masimo. And you'll also find a lineup of great speakers, including the SGU, Simon Singh, Michael Shermer and our keynote speaker, physicist Leonard Mladenov, author of The Drunkard's Walk.

Next story. That's an easy asphaug. Go get your tickets now.

The rationally speaking podcast is presented by New York City skeptics for program notes, links, and to get involved in an online conversation about this and other episodes, please visit rationally speaking podcast Dog. This podcast is produced by Benny Pollack and recorded in the heart of Greenwich Village, New York. Our theme, true by Todd Rundgren, is used by permission. Thank you for listening.