Journal of Pathology Informatics Journal of Pathology Informatics
Contact us | Home | Login   |  Users Online: 142  Print this pageEmail this pageSmall font sizeDefault font sizeIncrease font size 

Table of Contents    
J Pathol Inform 2011,  2:52

An open-source software program for performing Bonferroni and related corrections for multiple comparisons

1 Faculty of Medicine, Bachelor of Health Sciences Program, Room G503, O'Brien Centre for the BHSc, 3330 Hospital Drive N.W. Calgary, Alberta T2N 4N1, 2, Canada
2 Departments of Pathology and Laboratory Medicine, University of Calgary and Calgary Laboratory Services, C414, Diagnostic and Scientific Centre, 9, 3535 Research Road NW, Calgary AB Canada T2L 2K8, Canada

Date of Submission07-Sep-2011
Date of Acceptance18-Nov-2011
Date of Web Publication26-Dec-2011

Correspondence Address:
Christopher Naugler
Departments of Pathology and Laboratory Medicine, University of Calgary and Calgary Laboratory Services, C414, Diagnostic and Scientific Centre, 9, 3535 Research Road NW, Calgary AB Canada T2L 2K8
Login to access the Email id

Source of Support: None, Conflict of Interest: None

DOI: 10.4103/2153-3539.91130

Rights and Permissions

Increased type I error resulting from multiple statistical comparisons remains a common problem in the scientific literature. This may result in the reporting and promulgation of spurious findings. One approach to this problem is to correct groups of P-values for "family-wide significance" using a Bonferroni correction or the less conservative Bonferroni-Holm correction or to correct for the "false discovery rate" with a Benjamini-Hochberg correction. Although several solutions are available for performing this correction through commercially available software there are no widely available easy to use open source programs to perform these calculations. In this paper we present an open source program written in Python 3.2 that performs calculations for standard Bonferroni, Bonferroni-Holm and Benjamini-Hochberg corrections.

Keywords: Bonferroni correction, software program, type I error

How to cite this article:
Lesack K, Naugler C. An open-source software program for performing Bonferroni and related corrections for multiple comparisons. J Pathol Inform 2011;2:52

How to cite this URL:
Lesack K, Naugler C. An open-source software program for performing Bonferroni and related corrections for multiple comparisons. J Pathol Inform [serial online] 2011 [cited 2022 Jul 7];2:52. Available from:

   Background Top

When multiple hypotheses are tested in a single experiment, the risk of type I error is increased and with it the risk of promulgating spurious "significant" findings. [1],[2],[3] The likelihood of obtaining a false positive result increases proportional to the number of tests performed. For example, the probability of obtaining at least one false positive result when performing 10 tests is given by

where P(A) is the confidence level of the test.

Although the problems associated with multiple testing are well known, numerous studies still fail to correct their reported P-values. For instance, Bennett et al. found that only between 60% and 74% of the neuroimaging articles published in several major journals corrected for multiple comparisons. [4] Similarly, a study performed by Austin et al. also demonstrated that the failure to account for multiple testing resulted in statistically significant, yet implausible results. [5] In both cases the results were no longer significant after correcting for multiple testing.

The lack of attention paid to this problem in the pathology literature stands in stark contrast to its recognition in other fields such as ecology where there has been intense interest for over two decades since the seminal publication by Rice. [6] That being said, even within the field of ecology this topic still engenders debate. [7] A systematic exploration of this problem in the pathology literature has not been undertaken; however we have previously reported on a convenience sample of 800 publications from the pathology literature in 2003, of which 37 presented multiple comparisons. Twenty one of these 37 did not attempt to control for increased type I error due to multiple comparisons. [8]

One means of reducing the type I error from multiple testing is the Bonferroni correction, which controls the family-wise error rate (FWER). The FWER is the probability of type I error among the entire set of hypotheses.

The Bonferroni correction is calculated as follows:

where n is the number of hypotheses tested. There is a lack of consensus as to what actually represents a "family" of statistical tests; however it has been suggested that if it is appropriate to place multiple P-values in the same table, it may be appropriate to correct all values in that table for multiple comparisons. [6]

Because the Bonferroni correction is conservative with regard to statistical power, other methods of correcting for multiple testing have been developed. Another method that controls for the FWER is the Bonferroni-Holm correction. [9] The Bonferroni-Holm correction is calculated as follows:

where n is the number of hypotheses tested, and k is the ordered rank of the uncorrected P-values (from smallest P-value to largest P-value).

Rather than controlling for the probability of one or more type I errors in the entire experiment, some of the more recent approaches to the multiple testing problem have focused on controlling the false discovery rate (FDR) in the experiment. By controlling the proportion of type I errors, this has the advantage of further increasing the statistical power of the algorithm, and is especially suitable when conducting numerous hypothesis tests. [10],[11] The Benjamini-Hochberg method [12] is a commonly used way to control the FDR of an experiment. It is calculated as follows:

where n is the number of hypotheses tested, and k is the rank of the uncorrected P value.

Several commercial statistical software packages are capable of performing one or more of these corrections as well as at least one open-source program (GNU R); however the cost of the commercial packages, and the learning curves involved, may discourage researchers from using these programs. Online tools are also available (e.g., but are limited in scope and available options and rely on continued access to the publisher's website.

"Bonferroni Calculator" software

Using the open-source programming language Python v 3.2, we developed a program capable of performing Bonferroni, Bonferroni-Holm, and Benjamini-Hochberg corrections for any number of P-values. The user is prompted for a set of P-values and the desired significance (alpha) level. From the main menu the user may choose to display the results of the desired correction to the screen, or to export the corrected P values to the hard disk (text and csv file types). The source code is available free as a supplementary file to this article (which may serve as a literature reference for the program). A copy of the source code may also be obtained by email from the corresponding author. The program requires the free programming language Python 3.2 which is capable of running on Microsoft Windows, MAC OS, and Linux/Unix operating systems. It may be downloaded from

The program is available for free by emailing the senior author at [email protected] Detailed instructions and a FAQ are available at To use the Bonferroni Calculator software, place the files "Bonferroni" and "Lesack and Naugler.txt" in a folder on your hard drive. In windows, the program will run from the command line by double clicking on the "Bonferroni" icon; however the preferred method is to right click on the icon and select "Edit with IDLE" from the dropdown list. Press F5 to run the software, and then maximize the size of the window. Follow the instructions on the screen. If the option is selected to save the results to files, these will be found in the same folder as the "Bonferroni" icon. The program is also available from the authors as a stand-alone executable file.

   References Top

1.Koch G, Gansky M. Statistical considerations for multiplicity in confirmatory protocols. Drug Inf J 1996;30:523-33.  Back to cited text no. 1
2.Bender R, Lange S. Adjusting for multiple testing-when and how? J Clin Epidemiol 2001;54:343-9.  Back to cited text no. 2
3.Karr A, Young SS. Deming, data and observational studies. Significance 2011;8:116-120.  Back to cited text no. 3
4.Bennett CM, Baird AA, Miller MB, Wolford GL. Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: an argument for multiple comparisons correction. J Serendipitous Unexpected Results 2010;1:1-5.  Back to cited text no. 4
5.Austin PC, Mamdani MM, Juurlink DN, Hux JE. Testing multiple statistical hypotheses resulted in spurious associations: a study of astrological signs and health. J Clin Epidemiol 2006;59:964-9.  Back to cited text no. 5
6.Rice WR. Analyzing tables of statistical tests. Evolution 1989;43:223-5.  Back to cited text no. 6
7.Nakagawa S. A farewell to Bonferroni: the problems of low statistical power and publication bias. Behav Ecol 2004;15:1044-5.  Back to cited text no. 7
8.Zheng Z, Naugler C. Type I error in pathology papers, prevalence and effect on publication citations. Poster Presentation, Canadian Association of Pathologists Annual Scientific Meeting, Montreal, PQ, Jul 11-15 2010.  Back to cited text no. 8
9.Holm S. A simple sequentially rejective multiple test procedure. Scand J Stat 1979;6:65-70.  Back to cited text no. 9
10.García LV. Escaping the Bonferroni iron claw in ecological studies. Oikos 2004;105:657-63.  Back to cited text no. 10
11.Wit E, McClure J. Statistics for microarrays: Design, Analysis, and Inference. 1 st ed. Hoboken, New Jersey: John Wiley and Sons; 2004. p.195.  Back to cited text no. 11
12.Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Series B 1995;57:289-300.  Back to cited text no. 12

This article has been cited by
1 Skeletal muscle undergoes fiber type metabolic switch without myosin heavy chain switch in response to defective fatty acid oxidation
Andrea S. Pereyra, Chien-Te Lin, Daniela Mesa Sanchez, Julia Laskin, Espen E. Spangenburg, P. Darrell Neufer, Kelsey Fisher–Wellman, Jessica M. Ellis
Molecular Metabolism. 2022; : 101456
[Pubmed] | [DOI]
2 Alterations in sphingolipid composition and mitochondrial bioenergetics represent synergistic therapeutic vulnerabilities linked to multidrug resistance in leukemia
Kelsey H. Fisher-Wellman, James T. Hagen, Miki Kassai, Li-Pin Kao, Margaret A. M. Nelson, Kelsey L. McLaughlin, Hannah S. Coalson, Todd E. Fox, Su-Fern Tan, David J. Feith, Mark Kester, Thomas P. Loughran, David F. Claxton, Myles C. Cabot
The FASEB Journal. 2022; 36(1)
[Pubmed] | [DOI]
3 Core Oligosaccharide Portion of Lipopolysaccharide Plays Important Roles in Multiple Antibiotic Resistance in Escherichia coli
Jianli Wang, Wenjian Ma, Yu Fang, Hao Liang, Huiting Yang, Yiwen Wang, Xiaofei Dong, Yi Zhan, Xiaoyuan Wang
Antimicrobial Agents and Chemotherapy. 2021; 65(10)
[Pubmed] | [DOI]
4 Executive Functions in Neurodevelopmental Disorders: Comorbidity Overlaps Between Attention Deficit and Hyperactivity Disorder and Specific Learning Disorders
Giulia Crisci, Sara Caviola, Ramona Cardillo, Irene C. Mammarella
Frontiers in Human Neuroscience. 2021; 15
[Pubmed] | [DOI]
5 The Structure of Relationships between the Human Exposome and Cardiometabolic Health: The Million Veteran Program
Kerry L. Ivey, Xuan-Mai T. Nguyen, Daniel Posner, Geraint B. Rogers, Deirdre K. Tobias, Rebecca Song, Yuk-Lam Ho, Ruifeng Li, Peter W. F. Wilson, Kelly Cho, John Michael Gaziano, Frank B. Hu, Walter C. Willett, Luc Djoussé
Nutrients. 2021; 13(4): 1364
[Pubmed] | [DOI]
6 Intrinsic OXPHOS limitations underlie cellular bioenergetics in leukemia
Margaret AM Nelson, Kelsey L McLaughlin, James T Hagen, Hannah S Coalson, Cameron Schmidt, Miki Kassai, Kimberly A Kew, Joseph M McClung, P Darrell Neufer, Patricia Brophy, Nasreen A Vohra, Darla Liles, Myles C Cabot, Kelsey H Fisher-Wellman
eLife. 2021; 10
[Pubmed] | [DOI]
7 Genome-wide bioinformatic analyses predict key host and viral factors in SARS-CoV-2 pathogenesis
Mariana G. Ferrarini, Avantika Lal, Rita Rebollo, Andreas J. Gruber, Andrea Guarracino, Itziar Martinez Gonzalez, Taylor Floyd, Daniel Siqueira de Oliveira, Justin Shanklin, Ethan Beausoleil, Taneli Pusa, Brett E. Pickett, Vanessa Aguiar-Pulido
Communications Biology. 2021; 4(1)
[Pubmed] | [DOI]
8 Global population genetic structure of the sequential hermaphrodite, dusky grouper ( Epinephelus marginatus )
Jussara Oliveira Vaini, Rodrigo Rodrigues Domingues, Bruno Lopes da Silva Ferrette, Eric M. Hallerman, Kenneth Gabriel Mota, João Pedro Barreiros, Alexandre Wagner Silva Hilsdorf
Aquatic Conservation: Marine and Freshwater Ecosystems. 2021; 31(8): 2119
[Pubmed] | [DOI]
9 Global phylogeography of the smooth hammerhead shark: Glacial refugia and historical migration patterns
Bruno Lopes da Silva Ferrette, Rui Coelho, Victor Marten Peddemors, Jennifer R. Ovenden, Bruno Alexandre De Franco, Claudio Oliveira, Fausto Foresti, Fernando Fernandes Mendonça
Aquatic Conservation: Marine and Freshwater Ecosystems. 2021; 31(9): 2348
[Pubmed] | [DOI]
10 General dental practitioners’ knowledge and attitudes on children’s pain and pain management—A questionnaire survey
Larisa Krekmanova, Stefan Nilsson, Magnus Hakeberg, Gunilla Klingberg, Agneta Robertson
Paediatric and Neonatal Pain. 2021; 3(2): 87
[Pubmed] | [DOI]
11 Synergy of endoplasmic reticulum aminopeptidase 1 and 2 (ERAP1 and ERAP2) polymorphisms in atopic dermatitis: Effects on disease prevalence
Wanda Niepieklo-Miniewska, Lukasz Matusiak, Joanna Narbutt, Aleksandra Lesiak, Piotr Kuna, Andrzej Wisniewski, Karolina Piekarska, Izabela Nowak, Piotr Kusnierczyk
Human Immunology. 2021; 82(2): 121
[Pubmed] | [DOI]
12 Subcellular proteomics combined with bioenergetic phenotyping reveals protein biomarkers of respiratory insufficiency in the setting of proofreading-deficient mitochondrial polymerase
Kelsey L. McLaughlin, Kimberly A. Kew, Joseph M. McClung, Kelsey H. Fisher-Wellman
Scientific Reports. 2020; 10(1)
[Pubmed] | [DOI]
13 Novel approach to quantify mitochondrial content and intrinsic bioenergetic efficiency across organs
Kelsey L. McLaughlin, James T. Hagen, Hannah S. Coalson, Margaret A. M. Nelson, Kimberly A. Kew, Ashley R. Wooten, Kelsey H. Fisher-Wellman
Scientific Reports. 2020; 10(1)
[Pubmed] | [DOI]
14 Serum exosomal microRNA transcriptome profiling in subacute spinal cord injured rats
Shu-Qin Ding, Yu-Qing Chen, Jing Chen, Sai-Nan Wang, Fei-Xiang Duan, Yu-Jiao Shi, Jian-Guo Hu, He-Zuo Lü
Genomics. 2020; 112(6): 5086
[Pubmed] | [DOI]
15 Serum exosomal microRNA transcriptome profiling in subacute spinal cord injured rats
Shu-Qin Ding, Yu-Qing Chen, Jing Chen, Sai-Nan Wang, Fei-Xiang Duan, Yu-Jiao Shi, Jian-Guo Hu, He-Zuo Lü
Genomics. 2020; 112(2): 2092
[Pubmed] | [DOI]
16 Truncating the Structure of Lipopolysaccharide in Escherichia coli Can Effectively Improve Poly-3-hydroxybutyrate Production
Jianli Wang, Wenjian Ma, Yu Fang, Hailing Zhang, Hao Liang, Ye Li, Xiaoyuan Wang
ACS Synthetic Biology. 2020; 9(5): 1201
[Pubmed] | [DOI]
17 Determinants of motor, language, cognitive, and global developmental delay in children with complicated severe acute malnutrition at the time of discharge: An observational study from Central India
Navneet Khandelwal, Jagdish Mandliya, Kamna Nigam, Vandana Patil, Aditya Mathur, Ashish Pathak, Marzia Lazzerini
PLOS ONE. 2020; 15(6): e0233949
[Pubmed] | [DOI]
18 Ablation of Sirtuin5 in the postnatal mouse heart results in protein succinylation and normal survival in response to chronic pressure overload
Kathleen A. Hershberger, Dennis M. Abraham, Juan Liu, Jason W. Locasale, Paul A. Grimsrud, Matthew D. Hirschey
Journal of Biological Chemistry. 2018; 293(27): 10630
[Pubmed] | [DOI]
19 FleQ regulates both the type VI secretion system and flagella in Pseudomonas putida
Yuzhou Wang,Ye Li,Jianli Wang,Xiaoyuan Wang
Biotechnology and Applied Biochemistry. 2017;
[Pubmed] | [DOI]
20 Physiological and parasitological implications of living in a city: the case of the white-footed tamarin (Saguinus leucopus)
Iván Darío Soto-Calderón,Yuliet Andrea Acevedo-Garcés,Jóhnatan Álvarez-Cardona,Carolina Hernández-Castro,Gisela María García-Montoya
American Journal of Primatology. 2016;
[Pubmed] | [DOI]
21 Statistics Commentary Series
David L. Streiner
Journal of Clinical Psychopharmacology. 2016; 36(1): 5
[Pubmed] | [DOI]
22 Gene expression signatures, pathways and networks in carotid atherosclerosis
L. Perisic,S. Aldi,Y. Sun,L. Folkersen,A. Razuvaev,J. Roy,M. Lengquist,S. Åkesson,C. E. Wheelock,L. Maegdefessel,A. Gabrielsen,J. Odeberg,G. K. Hansson,G. Paulsson-Berne,U. Hedin
Journal of Internal Medicine. 2016; 279(3): 293
[Pubmed] | [DOI]
23 Customizing Laboratory Information Systems
Peter Gershkovich,John H. Sinard
Advances In Anatomic Pathology. 2015; 22(5): 323
[Pubmed] | [DOI]
24 The impact of ginsenosides on cognitive deficits in experimental animal studies of Alzheimer’s disease: a systematic review
Chenxia Sheng,Weijun Peng,Zi-an Xia,Yang Wang,Zeqi Chen,Nanxiang Su,Zhe Wang
BMC Complementary and Alternative Medicine. 2015; 15(1)
[Pubmed] | [DOI]
25 Linking GABA and glutamate levels to cognitive skill acquisition during development
Kathrin Cohen Kadosh,Beatrix Krause,Andrew J. King,Jamie Near,Roi Cohen Kadosh
Human Brain Mapping. 2015; 36(11): 4334
[Pubmed] | [DOI]
26 Population Genetic Structure of Southern Flounder Inferred from Multilocus DNA Profiles
Verena H. Wang,Michael A. McCartney,Frederick S. Scharf
Marine and Coastal Fisheries. 2015; 7(1): 220
[Pubmed] | [DOI]
27 Integrated Metabolomic and Proteomic Analysis Reveals Systemic Responses ofRubrivivax benzoatilyticusJA2 to Aniline Stress
Md Mujahid,M Lakshmi Prasuna,Ch Sasikala,Ch Venkata Ramana
Journal of Proteome Research. 2015; 14(2): 711
[Pubmed] | [DOI]
28 Anterior-posterior cerebral blood volume gradient in human subiculum
Pratik Talati,Swati Rane,Samet Kose,John Gore,Stephan Heckers
Hippocampus. 2014; : n/a
[Pubmed] | [DOI]
29 A novel compression garment with adhesive silicone stripes improves repeated sprint performance – a multi-experimental approach on the underlying mechanisms
Dennis-Peter Born,Hans-Christer Holmberg,Florian Goernert,Billy Sperlich
BMC Sports Science, Medicine and Rehabilitation. 2014; 6(1): 21
[Pubmed] | [DOI]
30 Short-term effects of a modified Alt-RAMEC protocol for early treatment of Class III malocclusion: a controlled study
C. Masucci,L. Franchi,V. Giuntini,E. Defraia
Orthodontics & Craniofacial Research. 2014; 17(4): 259
[Pubmed] | [DOI]
31 Treatment and posttreatment effects induced by the Forsus appliance:A controlled clinical study
Giorgio Cacciatore,Luis Tomas Huanca Ghislanzoni,Lisa Alvetro,Veronica Giuntini,Lorenzo Franchi
The Angle Orthodontist. 2014; 84(6): 1010
[Pubmed] | [DOI]
32 Characteristics of cognitive deficits and writing skills of Polish adults with developmental dyslexia
Katarzyna Maria Bogdanowicz,Marta Lockiewicz,Marta Bogdanowicz,Maria Pachalska
International Journal of Psychophysiology. 2013;
[Pubmed] | [DOI]
33 Investigation of genetic risk factors for chronic adult diseases for association with preterm birth
Nadia Falah,Jude McElroy,Victoria Snegovskikh,Charles J. Lockwood,Errol Norwitz,Jeffey C. Murray,Edward Kuczynski,Ramkumar Menon,Kari Teramo,Louis J. Muglia,Thomas Morgan
Human Genetics. 2013; 132(1): 57
[Pubmed] | [DOI]
34 Sleepiness and nocturnal hypoxemia in Peruvian men with obstructive sleep apnea
Charles Huamaní,Jorge Rey de Castro,Edward Mezones-Holguín
Sleep and Breathing. 2013;
[Pubmed] | [DOI]




   Browse articles
    Similar in PUBMED
   Search Pubmed for
   Search in Google Scholar for
 Related articles
    Access Statistics
    Email Alert *
    Add to My List *
* Registration required (free)  

  In this article

 Article Access Statistics
    PDF Downloaded1251    
    Comments [Add]    
    Cited by others 34    

Recommend this journal