Pew’s Scientific Credibility is Questionable

Many of us have concerns about the rapid changes occurring in food animal farming. Meat production has changed significantly in the last 30 years due to reductions in the number of farmers, the amount of land available, economics, and increases in modern technology. These changes have generated enthusiasm by leaders of the change and anxiety by those left out, as these changes affect everyone.  As a result of increasing anxiety, several research synthesis reviews have been published, including by the Pew Charitable Trusts.  These reports have addressed the public health risk of antibiotic use, the social impacts of industry consolidation, and the negative health effects of Industrial Farm Animal Production (IFAP) sites.

Accepted methods for Review Literature

Synthesis reviews are an important tool for public policy decision making because they combine evidence from multiple scientific publications. Over the years, scientists have worked to develop standards by which research papers can be evaluated for inclusion in these synthesis reviews.. The two messages from these scientists, including the World Health Organization, are that the scientific quality of each publication should be evaluated, and the overall body of evidence should be estimated from the combination of the disparate publications. In other words, lack of a clear message from all the higher quality research infers lack of public health risk.

The value or credibility of health effects research should be evaluated by the following criteria:

• Study design of the primary research – Epidemiologic (observational) studies are weaker than clinical trials.
• Evidence of bias in the primary research studies – Were study participants chosen to give a predetermined result? Are those complaining of illness evaluated in same manner as the controls (not ill)?
• Directness or external validity – How well does the study population used in the primary research apply to the specific question or affected people?
• Precision – Are there a sufficient number of participants to evaluate the statistical power of the study, or could the effects have happened by chance?
• Publication bias – Does the review include all possible published papers? Does the topic lead to publication of those papers with a negative causal effect only or visa versa?
• Consistency – Do most of the studies report a similar causal effect?
• Is there evidence for a “dose-response”, i.e. do participants closer to the hazard (e.g. IFAP sites) have more negative effects than those further away?

Example errors from Pew reports

On the topic of risk from farm animal antibiotic use, the Pew reports in the past have been emphatic: “One of the most significant public health issues associated with IFAP is its contribution to the increasing crisis of antimicrobial resistant infections worldwide.”  A careful review by the scientists on behalf of the American Veterinary Medical Association, however, found the Pew methods were unscientific and biased:

“Both in substance and in approach,…the Pew report contains significant flaws and major deviations from both science and reality. These missteps lead to dangerous and under-informed recommendations about the nature of our food system—and shocking recommendations for interventions that are scarcely commensurate with risk.

The report is, in many ways, a prolonged narrative designed to romanticize the small, independent farmer, while vilifying larger operations, based simply upon their size.”

Summary

The application of selected scientific research to a political question does not make good policy. Research synthesis reports combining a body of scientific evidence and can be useful for making informed policy decisions; however, formal methods of assessing bias and error in a study and bodies of scientific evidence are required to correctly translate research to policy. Failure to accurately consider the entire message and to evaluate the credibility of the evidence can lead to misunderstanding and bad policy. Careful scientific critique of these Pew reports strongly suggests they have “cherry-picked” selected papers to make a politically motivated point. Pew purposefully not giving the whole story is misleading to the consumer and is misinforming them about the facts of animal agriculture as a whole. (Although, misleading marketing tactics do seem to be all the rage these days.)