G. Miller
May 1, 2014
Citations
1
Influential Citations
41
Citations
Journal
Toxicological sciences : an official journal of the Society of Toxicology
Abstract
Over the past few months there has been considerable discussion in scientific circles regarding reproducibility of data, or more specifically, the lack thereof (Nature, 2012, 2013). This is a very serious issue for science, including the discipline of toxicology. The director and deputy director of the National Institutes of Health (NIH) in the United States, Dr Francis Collins and Dr Lawrence Tabek, have outlined several steps that the NIH will be taking to improve reproducibility, including additional review of grant applications and greater access to raw data (Collins and Tabek, 2014). The authors point out that this is not an issue of research misconduct, but rather an issue of carelessness at multiple levels of the scientific enterprise. As a leading outlet of research in the field of toxicology, Toxicological Sciences can and will take steps to improve the reproducibility of the research published in our journal. Most of this discussion has focused on preclinical research related to drug development. Lack of reproducibility can prompt unnecessary research, lead a start up company down a fruitless path, or stall a clinical trial of a promising new drug. Although depriving sick patients of innovative treatments is a major concern for many fields, toxicology has the opposite problem. In studying the adverse actions of environmental chemicals and pharmaceuticals, we are trying to identify potentially dangerous unintended effects and stop them from occurring. As toxicology is attempting to prevent a future injury, illness, or adverse effect, poorly designed and executed studies and a lack of an attention to the detail, i.e., low reproducibility, can result in unnecessary harm to citizens and damage to our planet. Toxicology also plays a major role in establishing guidelines, such as reference doses, no adverse effects levels, and safety margins for thousands of chemicals. The integrity of our field depends on the quality of data used in making those decisions. Toxicology must be a data-driven science not an agendadriven science. This requires self-examination of our own biases in advance and the careful design of experiments that negate or control for these biases. For example, many experiments suffer from confirmation bias. An experimenter thinks something to be true and sets off on a course of study to confirm it. It is better to evoke the spirit of Karl Popper and attempt to disprove what we believe to be true. Efforts to disprove our own hypotheses are often rewarded with the comfort that our intuition was correct and our thinking stands up to challenges. More rarely, and perhaps more excitingly, is when our attempts to disprove our hypotheses succeed. Although initially crushing (“the slaying of a beautiful hypothesis by an ugly fact” Thomas Huxley), the revised hypothesis will be stronger and get us closer to the truth we seek. How many graduate students have actually tried (and I mean really tried) to disprove their dissertation hypotheses? Self-preservation makes such attempts difficult and uncomfortable, but the result of such exercises can be rewarding. Either you demonstrate the soundness of your ideas or you realize that you are basing your future on a weak or tenuous premise and hopefully have time to change course. We should welcome robust debate and challenges to our own research. If an organization requests to see our raw data we should not cower as if under an audit, but welcome the close examination of our scientific process and the results that are obtained. As toxicological findings can have implications for restrictions, regulations, bans, and economic losses, it is essential that the data be robust and verifiable. Some view the vigorous defense of products by corporations as merely an attempt to protect the bottom line.While this may be amajor motivation, these industries have the right to refute poorly designed studies. In fact, they have the obligation to question the quality of the data that may impact their industries. However, bias does not originate from industry alone. Biases in toxicology can be viewed as a bell-shaped curve. On the tails we have companies so defensive of their products that they are unwilling to examine data that have raised safety concerns about a product, although in most cases there is a genuine effort to make products that contribute to an improved quality of life without inflicting harm on humanity. On the other end of the continuum, we have those who believe that every exogenous or man-made chemical is bad, including vaccines, or the academic scientist on a mission to see a particular compound banned regardless of what the data show. The zealotry from both ends of the curve is harmful to the credibility of the scientific community. Fortunately, most of us are somewhere within a standard deviation or two on either side of the mean, but that does not mean we are free of bias. One of the ways of improving reproducibility is to introduce a bit more noise into the experimental system. It is not uncommon for a laboratory to order 80 animals, split them into four, six