Anesthesiology News - The Boldt Affair: A Quandary for Meta-Analysts

Anesthesiology News - The Boldt Affair: A Quandary for Meta-Analysts

Meta-analyses of randomized clinical trials provide the most rigorous evidence by which to judge the effectiveness and safety of therapeutic interventions. Fraudulent data pose a threat to the validity of meta-analyses and should be excluded. This might be straightforward in some cases. In June 2012, for instance, a Special Investigation Committee for the Japanese Society of Anesthesiologists announced its finding that 172 papers of Yoshitaka Fujii, MD, had been fabricated (with a subsequent report putting the number at 183). A meta-analyst would not hesitate to exclude those papers.

The case of German anesthesiologist Joachim Boldt, MD, PhD, at one time a prolific and preeminent authority on the perioperative use of intravenous fluids, is more problematic, because the scope of fraud has been neither conclusively established nor fully investigated. In March 2011, 88 of his publications were retracted because Dr. Boldt allegedly failed to secure approval from an institutional review board (IRB) for his studies.¹ In announcing the retractions, a consortium of journal editors stated that lack of IRB approval “does not mean that the research results per se are fraudulent.” Accordingly, Dr. Boldt’s retracted studies continued to be included in meta-analyses.^2-4

Such a course is understandable, since meta-analysts are reluctant to exclude studies without sound reason. A key safeguard against bias in meta-analysis is comprehensiveness. The meta-analyst undertakes to identify and include, without exception, all trials fulfilling prespecified eligibility criteria.

A Specter of Doubt

Yet, Dr. Boldt’s research misconduct clearly went beyond failure to obtain IRB approval. After suspicions were raised by anomalously low variability in interleukin-6 (IL-6) measurements, one of his papers was determined to have been fabricated.⁵ At least 10 articles contained false data, according to the findings of an independent investigational committee for Klinikum Ludwigshafen, announced last August.

The identities of the articles were not specified, however, and the final report has not been made public. The committee did reveal that records were either missing or incomplete for the large majority of studies; indeed, only 8% of study subjects could be identified. These findings suggested fabrication on a grand scale, but the committee did not state any conclusions about fabrication. In effect, they left the validity of the 88 papers in limbo.

Furthermore, the committee examined only papers that Dr. Boldt published since 1999. His earlier work, comprising nearly 40% of his total clinical studies in fluid management, may never be investigated. A shadow of doubt hangs over all of Dr. Boldt’s work.⁵

So, with the validity of specific papers unresolved, meta-analysts face a quandary. Which, if any, of Dr. Boldt’s studies can still be taken at face value? Some meta-analysts have chosen to include his data, even if retracted for lack of IRB approval, while also presenting sensitivity analyses to determine their impact on overall results.^2-4 The drawback of this approach is that it may lend credibility to spurious data and inflate the apparently available breadth of valid evidence on a particular clinical question.

Moreover, publicly available evidence beyond that released by the retracting editors and Klinikum Ludwigshafen reveals that Dr. Boldt engaged in rampant and long-standing fabrication and data manipulation. For example, two additional cardiac surgery trial reports of his described extracorporeal circuit priming with albumin.^6,7 However, no albumin was used for priming at Klinikum Ludwigshafen during that time period.⁵ Hence, those studies presumably must have been fabricated. Six additional trials of his in cardiac and major abdominal surgery were evidently fabricated because of anomalously low IL-6 variability.^8-13

Meta-analyses^14,15 have cast doubt on seven more papers describing trials of hydroxyethyl starch (HES) for fluid management in sepsis and trauma.^16-22 These trials showed unusually low variability in mortality.¹⁴ They also were inconsistent with studies from other investigators.¹⁵Whereas the Boldt trials suggested a 24% relative reduction in the odds of mortality attributable to HES, 10 trials from other investigators suggested a 30% increase. The difference was statistically significant (P=0.024). Thus, Dr. Boldt’s data introduced major bias and obscured the true effect of HES on patient survival.

For two of the seven papers, underlying data were available in a doctoral dissertation.²³ The dissertation indicated that in the articles, a single trial was misrepresented as two separate trials, and data were altered to conceal the double publication. Furthermore, gastric intramucosal pH data were manipulated to falsely suggest the superiority of HES over albumin in restoring splanchnic perfusion in sepsis patients.

Dissertations^24,25 also revealed double publication and concealment involving four more papers dating as far back as 1990.^26-29 In one of these papers, data were manipulated to show incorrectly that blood loss and transfusion of red blood cells and fresh frozen plasma were significantly lower in patients receiving HES 130/0.4 (Fresenius Kabi/Hospira) than in patients receiving a competing HES product.

Deciding how to handle Dr. Boldt’s data is of immediate importance. A topic closely associated with his work, the safety of HES solutions, is now in the spotlight of international attention. Dr. Boldt had been a tireless and highly visible champion of HES.³⁰ Thanks in large part to his clinical research papers and review articles, HES 130/0.4 alone has been administered to at least 30 million patients worldwide, according to the manufacturer. New clinical trial data indicate that many of those patients suffered organ failure, bleeding complications and death as a result.^31,32Regulatory actions are currently under consideration by both the FDA (www.fda.gov/​downloads/​AdvisoryCommittees/​CommitteesMeetingMaterials/​BloodVaccinesandOtherBiologics/​BloodProductsAdvisoryCommittee/​UCM325690.pdf) and the European Medicines Agency (www.ema.europa.eu/​ema/​index.jsp?curl=pages/medicines/human/referrals/Hydroxyethyl_starch-containing_solutions/human_referral_prac_000012.jsp&mid=WC0b01ac05805c516f). Previous meta-analyses have been instrumental in assessing HES safety, and new meta-analyses likely will contribute to the present decision-making process.

The available evidence reveals a pattern of pervasive research fraud spanning 20 years. This pattern suggests that, in the absence of evidence to the contrary, all of Dr. Boldt’s data should be viewed as unreliable and excluded from meta-analyses. Such a policy of categorical exclusion has been implemented in our recent meta-analysis of postoperative bleeding in cardiac surgery patients receiving HES.³³

The Klinikum Ludwigshafen, where Joachim Boldt, MD, PhD, worked until 2010, could not validate the findings in many of his published studies.

The continuing contemporary relevance of the Boldt issue has been dramatized further by a new meta-analysis assessing outcomes after HES exposure.³⁴ No overall effect of HES on survival could be demonstrated. However, after exclusion of the seven unretracted sepsis and trauma studies by Dr. Boldt’s group discussed above,^16-22 significantly increased mortality was shown among HES recipients. Results from Dr. Boldt and his collaborators altered the fundamental conclusion of the meta-analysis, even though objective evidence has existed since a 2004 dissertation²³ and a 2006 meta-analysis¹⁴indicating that those data were not valid.

According to an editorial accompanying the new meta-analysis,³⁵ “hydroxyethyl starch continues to be available and widely used, in part because of uncertainty surrounding both the magnitude of scientific misconduct by Boldt and the influence of this misconduct on summary measures of clinical efficacy and harm.” This uncertainty will not be dispelled until the scientific validity of all Dr. Boldt’s fluid management studies has been definitively determined and publicly disclosed.

The percentage of retractions for fraud is on the rise,³⁶ and the issues raised by the Boldt affair are likely to be recurring. What lessons can be learned for the future? Determining the scientific validity of studies under suspicion should be a high priority. Patient care may depend on it.

In the cases of Drs. Fujii and Boldt, the two researchers perpetrated fraud over decades and at different institutions. All institutions concerned need to participate in the investigation. The findings of the investigation should be promptly disclosed, publicly and in full. As this may not occur, editors should be willing to act on available evidence themselves. finally, including data in a meta-analysis amounts to a vote of confidence in their validity. Meta-analysts should err on the side of caution before including suspect data.

—Mahlon M. Wilkes, PhD, and Roberta J. Navickis, PhD

---

Drs. Wilkes and Navickis are principals and co-founders of Hygeia Associates, a biomedical consultancy specializing in evaluation, statistical analysis and publication of biomedical research data with a focus on meta-analysis. They have received unrestricted research funding from Baxter, CSL Behring and Grifols, but not for the preparation of this article.

References

1. Rasmussen LS, Yentis SM, Van Aken H, et al. Editors-in-chief statement regarding published clinical trials conducted without IRB approval by Joachim Boldt. Minerva Anestesiol. 2011;77:562-563.
2. Bunn F, Trivedi D. Colloid solutions for fluid resuscitation. Cochrane Database Syst Rev.2012;6:CD001319.
3. Gattas DJ, Dan A, Myburgh J, et al. Fluid resuscitation with 6% hydroxyethyl starch (130/0.4) in acutely ill patients: an updated systematic review and meta-analysis. Anesth Analg. 2012;114:159-169.
4. Perel P, Roberts I. Colloids versus crystalloids for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2012;6:CD000567.
5. Shafer SL. Shadow of doubt. Anesth Analg. 2011;112:498-500.
6. Boldt J, Brosch C, Ducke M, et al. Influence of volume therapy with a modern hydroxyethylstarch preparation on kidney function in cardiac surgery patients with compromised renal function: a comparison with human albumin. Crit Care Med. 2007;35:2740-2746.
7. Boldt J, Brosch C, Röhm K, et al. Is albumin administration in hypoalbuminemic elderly cardiac surgery patients of benefit with regard to inflammation, endothelial activation, and long-term kidney function?Anesth Analg. 2008;107:1496-1503.
8. Lang K, Suttner S, Boldt J, et al. Volume replacement with HES 130/0.4 may reduce the inflammatory response in patients undergoing major abdominal surgery. Can J Anesth. 2003;50:1009-1016.
9. Boldt J, Ducke M, Kumle B, et al. Influence of different volume replacement strategies on inflammation and endothelial activation in the elderly undergoing major abdominal surgery. Intensive Care Med.2004;30:416-422.
10. Boldt J, Schölhorn T, Mayer J, et al. The value of an albumin-based intravascular volume replacement strategy in elderly patients undergoing major abdominal surgery. Anesth Analg. 2006;103:191-199.
11. Boldt J, Brosch C, Röhm K, et al. Comparison of the effects of gelatin and a modern hydroxyethyl starch solution on renal function and inflammatory response in elderly cardiac surgery patients. Br J Anaesth.2008;100:457-464.
12. Boldt J, Suttner S, Brosch C, et al. The influence of a balanced volume replacement concept on inflammation, endothelial activation, and kidney integrity in elderly cardiac surgery patients. Intensive Care Med. 2009;35:462-470.
13. Boldt J, Mayer J, Brosch C, et al. Volume replacement with a balanced hydroxyethyl starch (HES) preparation in cardiac surgery patients. J Cardiothorac Vasc Anesth. 2010;24:399-407.
14. Ioannidis JPA, Trikalinos TA, Zintzaras E. Extreme between-study homogeneity in meta-analyses could offer useful insights. J Clin Epidemiol. 2006;59:1023-1032.
15. Zarychanski R, Turgeon AF, Fergusson DA, et al. Renal outcomes and mortality following hydroxyethyl starch resuscitation of critically ill patients: systematic review and meta-analysis of randomized trials.Open Med. 2009;3:E196-E209.
16. Boldt J, Heesen M, Welters I, et al. Does the type of volume therapy influence endothelial-related coagulation in the critically ill? Br J Anaesth. 1995;75:740-746.
17. Boldt J, Heesen M, Müller M, et al. The effects of albumin versus hydroxyethyl starch solution on cardiorespiratory and circulatory variables in critically ill patients. Anesth Analg. 1996;83:254-261.
18. Boldt J, Heesen M, Padberg W, et al. The influence of volume therapy and pentoxifylline infusion on circulating adhesion molecules in trauma patients. Anaesthesia. 1996;51:529-535.
19. Boldt J, Mueller M, Menges T, et al. Influence of different volume therapy regimens on regulators of the circulation in the critically ill. Br J Anaesth. 1996;77:480-487.
20. Boldt J, Müller M, Heesen M, et al. Influence of different volume therapies on platelet function in the critically ill. Intensive Care Med. 1996;22:1075-1081.
21. Boldt J, Müller M, Heesen M, et al. Influence of different volume therapies and pentoxifylline infusion on circulating soluble adhesion molecules in critically ill patients. Crit Care Med. 1996;24:385-391.
22. Boldt J, Müller M, Mentges D, et al. Volume therapy in the critically ill: is there a difference? Intensive Care Med. 1998;24:28-36.
23. Papsdorf M. Auswirkungen einer Volumenersatztherapie auf Makro- und Mikrozirkulation und deren Regulatoren: Ein Vergleich von Hydroxyethylstärke und Humanalbumin beim kritisch kranken Patienten. Abteilung Anaesthesiologie, Intensivmedizin, Schmerztherapie. Gießen: Justus-Liebig-Universität Gießen; 2004: 1-113.
24. Herold AC. Hyperosmolarer Volumenersatz in der Herzchirurgie. Abteilung für Anaesthesiologie und Operative Intensivmedizin. Gießen: Justus-Liebig-Universität Gießen; 1992: 1-101.
25. Schellhaaß A. Untersuchungen über den Einfluss von Volumenersatzmitteln auf die Blutgerinnung in der Abdominalchirurgie - Vergleich zweier kristalloider und zweier kolloidaler Volumenersatzmittel. Klinik für Anaesthesiologie und Operative Intensivmedizin des Klinikums der Stadt Ludwigshafen am Rhein.Gießen: Justus-Liebig-Universität Gießen; 2003: 1-82.
26. Boldt J, Kling D, Herold C, et al. Volume therapy with hypertonic saline hydroxyethyl starch solution in cardiac surgery. Anaesthesia. 1990;45:928-934.
27. Boldt J, Zickmann B, Thiel A, et al. Hyperosmolarer Volumenersatz in der Herzchirurgie. Anaesthesist.1990;39:412-419.
28. Boldt J, Haisch G, Suttner S, et al. Are lactated Ringer’s solution and normal saline solution equal with regard to coagulation? Anesth Analg. 2002;94:378-384.
29. Boldt J, Haisch G, Suttner S, et al. Effects of a new modified, balanced hydroxyethyl starch preparation (Hextend^®) on measures of coagulation. Br J Anaesth. 2002;89:722-728.
30. Reinhart K, Takala J. Hydroxyethyl starches: what do we still know? Anesth Analg. 2011;112:507-511.
31. Myburgh JA, finfer S, Bellomo R, et al. Hydroxyethyl starch or saline for fluid resuscitation in intensive care.N Engl J Med. 2012;367:1901-1911.
32. Perner A, Haase N, Guttormsen AB, et al. Hydroxyethyl starch 130/0.4 versus Ringer’s acetate in severe sepsis. N Engl J Med. 2012;367:124-134.
33. Navickis RJ, Haynes GR, Wilkes MM. Effect of hydroxyethyl starch on bleeding after cardiopulmonary bypass: a meta-analysis of randomized trials. J Thorac Cardiovasc Surg. 2012;144:223-230.
34. Zarychanski R, Abou-Setta AM, Turgeon AF, et al. Association of hydroxyethyl starch administration with mortality and acute kidney injury in critically ill patients requiring volume resuscitation: a systematic review and meta-analysis. JAMA. 2013;309:678-688.
35. Antonelli M, Sandroni C. Hydroxyethyl starch for intravenous volume replacement: more harm than benefit.JAMA. 2013;309:723-724.
36. Fang FC, Steen RG, Casadevall A. Misconduct accounts for the majority of retracted scientific publications.Proc Natl Acad Sci U S A. 2012;109:17028-17033.