Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Strategies to Prevent or Reduce Gender Bias in Peer Review of Research Grants: A Rapid Scoping Review

  • Andrea C. Tricco,

    Affiliations Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada, Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

  • Sonia M. Thomas,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Jesmin Antony,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Patricia Rios,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Reid Robson,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Reena Pattani,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Marco Ghassemi,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Shannon Sullivan,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Inthuja Selvaratnam,

    Affiliation Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada

  • Cara Tannenbaum,

    Affiliation Institute of Gender and Health, Canadian Institutes of Health Research, Montréal, Quebec, Canada

  • Sharon E. Straus

    sharon.straus@utoronto.ca

    Affiliations Li Ka Shing Knowledge Institute, St. Michael’s Hospital, Toronto, Ontario, Canada, Department of Geriatric Medicine, University of Toronto, Medical Sciences Building, Toronto, Ontario, Canada

Abstract

Objective

To review the literature on strategies implemented or identified to prevent or reduce gender bias in peer review of research grants.

Methods

Studies of any type of qualitative or quantitative design examining interventions to reduce or prevent gender bias during the peer review of health-related research grants were included. Electronic databases including MEDLINE, EMBASE, Education Resources Information Center (ERIC), PsycINFO, Joanna Briggs, the Cochrane Library, Evidence Based Medicine (EBM) Reviews, and the Campbell Library were searched from 2005 to April 2016. A search for grey (i.e., difficult to locate or unpublished) literature was conducted and experts in the field were consulted to identify additional potentially relevant articles. Two individuals screened titles and abstracts, full-text articles, and abstracted data with discrepancies resolved by a third person consistently.

Results

After screening 5524 citations and 170 full-text articles, one article evaluating gender-blinding of grant applications using an uncontrolled before-after study design was included. In this study, 891 applications for long-term fellowships in 2006 were included and 47% of the applicants were women. These were scored by 13 peer reviewers (38% were women). The intervention included eliminating references to gender from the applications, letters of recommendations, and interview reports that were sent to the committee members for evaluation. The proportion of successful applications led by women did not change with gender-blinding, although the number of successful applications that were led by men increased slightly.

Conclusions

There is limited research on interventions to mitigate gender bias in the peer review of grants. Only one study was identified and no difference in the proportion of women who were successful in receiving grant funding was observed. Our results suggest that interventions to prevent gender bias should be adapted and tested in the context of grant peer review to determine if they will have an impact.

Introduction

Despite parity between the number of women and men completing undergraduate and graduate training in biomedical and health sciences worldwide [1, 2], women continue to be underrepresented as researchers in these domains and tend to receive less research funding than their male counterparts [35]. For example, an analysis of health services and policy research funding in Canada over the past decade found that female researchers under the age of 45 years had significantly lower success rates than age-matched male researchers [6]. Results from the 2015 2nd Pilot competition at the Canadian Institutes of Health Research (CIHR) indicated that mid-career and senior-career female researchers were less likely to be funded in the newly launched Foundation Grant program, which emphasizes track record and provides up to 7 years of funding for the pursuit of potentially high-impact research programs [7]. A similar study of the National Institutes of Health (NIH) in the United States has shown that women received larger individual grant awards but men held more grants than women at any point in their careers [5].

It is unclear whether the discrepancy between the rate of successful male and female grant applicants is a sign of systemic bias. It is important to consider if there is unconscious gender bias in the grant peer review process because career advancement in academic settings is often contingent on the ability to obtain research funds. An unconscious bias is an implicit attitude, stereotype, motivation, or assumption that can occur without one’s knowledge, control, or intention [8]. Forms of unconscious bias include gender bias, racial bias, and ageism, with gender bias representing one of the most frequently investigated biases associated with grant peer review [923].

In a study conducted in Sweden, female applicants had to be 2.5 times more productive (in terms of higher volume of publications or publication in journals with a higher impact factor) than male applicants in order to achieve the same “competence” scores on their grant applications [24]. A 2015 study identified gender bias in the peer review process through a linguistic analysis of NIH R01 reviewer comments on applications between 2007 and 2009. They found that successful applications submitted by women received more positive descriptors and praise. Furthermore, experienced female investigators received more references to competence than their male counterparts. Nonetheless, female applicants received similar scores to male applicants, despite the discrepancies in word choice used by the reviewers [25]. In a subsequent study of NIH R01 applications between 2010 and 2014, Kaatz et al. 2016 reported that despite more standout adjectives (e.g., outstanding) and references to ability being used in female applications than male applications, peer reviewers were more likely to assign statistically significantly worse priority, approach, and significance scores to female than male investigators [26].

In response to these concerns of inequity in the grant review process, several funders have implemented strategies to narrow the gender gap. In the United States, the NIH has implemented programmatic strategies to promote and support careers of women in biomedical science such as changing the wording in their grant announcements to ensure gender neutrality and establishing strategies to monitor equity across grant programs [27, 28]. However, their impact is unclear because these programs were delivered at the same time that individual universities were making attempts to address gender bias [29]. In the United Kingdom, the National Institute for Health Research made funding contingent on the candidate organizations receiving at least a Silver Award from the Athena Scientific Women’s Academic Network (SWAN) Charter; an award that signifies institutional attempts to advance gender equality [30]. To inform efforts at the CIHR, we completed a rapid scoping review of strategies to prevent or reduce gender bias in the peer review of research grants.

Methods

We developed a protocol using the scoping review methods proposed by Arksey and O’Malley [31] and further refined by the Joanna Briggs Institute [32]. To provide the CIHR with a timely answer in less than 2 months, we completed a rapid review [33]. This review was registered through the Open Science Framework (https://osf.io/qdzpt/). Although the PRISMA statement [34] has not been modified for scoping reviews, we used it to guide our reporting where possible (S1 Appendix).

Eligibility criteria

Our eligibility criteria were defined using ‘Population, Intervention, Comparison, Outcomes, Study designs, Timeframe’ (PICOST) components, with input from the knowledge user (CT).

Population

Peer reviewers (including researchers or knowledge users) or grant applicants for any type of research grants (e.g. clinical, health services, education) and any sex/gender. For descriptive studies, the participants were those who submitted a grant proposal to peer review.

Intervention

Any strategy to prevent or decrease gender bias in the peer review of research grants.

Comparators

Any other intervention or no intervention; studies without a comparator were also eligible for inclusion.

Outcomes

Any outcome that assessed or measured gender bias (including awareness, knowledge, attitudes) or its potential impact (such as proportion of funded research projects by women).

Study designs

All qualitative or quantitative study designs.

Timeframe

Due to the rapid nature of this review, we restricted the timeframe to papers published from 2005 to April 2016.

Other

We limited documents to those published in English due to the rapid nature of this review.

Information sources and search strategy

The protocol for comprehensive literature searches were developed by an experienced information specialist (JM) in consultation with the research team and completed by a library technician (AE). First, we searched MEDLINE, EMBASE, Education Resources Information Center (ERIC), PsycINFO, Joanna Briggs, the Cochrane Library, Evidence Based Medicine (EBM) Reviews, and the Campbell Library from 2005 to April 2016. Second, we searched for grey literature (i.e. difficult to locate or unpublished material) using the Canadian Agency for Drugs and Technologies in Health Grey Matters checklist [35]. Specifically, we searched Google and websites of funding agencies such as the CIHR, UK MRC and NIH. Third, we asked experts in the field to identify any additional potentially relevant articles. We limited the search to articles published in English. The final search strategy for MEDLINE is presented in S2 Appendix and additional search strategies are available from the corresponding author upon request.

Study selection

Search results were imported into our proprietary software for screening citations (i.e. titles and abstracts) and full-text articles [36]. The inclusion criteria were also imported into this online software and used for screening citations during the screening of titles and abstracts (i.e., level 1 screening) and full-text articles (i.e., level 2 screening).

We completed a series of calibration exercises prior to each stage of screening to ensure reliability across reviewers. Inter-rater agreement for study inclusion was calculated using percent agreement and when it reached >75% across the research team, we proceeded to the next stage. If the percent agreement was ≤75%, the inclusion criteria were clarified and another pilot test occurred. For level 1 screening (title and abstracts), one pilot test of 50 citations was conducted with all team members and we achieved 92% agreement. Subsequently, two reviewers independently reviewed all titles and abstracts for inclusion. For full-text screening, one pilot test of 13 full-text articles was conducted with all team members, and we achieved 85% agreement. Following this calibration exercise, two reviewers screened the full-text of potentially relevant articles to determine inclusion. The results of the grey literature search were initially screened by a single reviewer, and the full-text of any potentially relevant grey literature identified was then assessed by two reviewers. All discrepancies between reviewers were resolved by a third reviewer consistently (SMT or JA).

Data collection

We abstracted data on study characteristics (e.g. year of study conduct, country, setting, type of publication, focus of the study), population characteristics (e.g. % female, % new investigators), and quantitative (e.g. % successful applicants) outcomes. Due to the small number of included studies identified, a pilot-test was not conducted for data abstraction. The data abstraction form was developed and modified as required based on feedback from the team. Each study was abstracted by two team members and any discrepancies were resolved by a third person (SMT).

Methodological quality appraisal

We did not appraise quality or risk of bias of the included articles, consistent with accepted scoping review methods [31, 32] and scoping reviews on health-related topics [37].

Synthesis and data charting

We charted the data quantitatively to identify the number of relevant publications according to types of participants, interventions, comparators, and outcomes, and summarized these findings using descriptive frequencies. We obtained additional data from the authors of one study and included this information in the analysis [38].

Consultation

We provided the results to the CIHR knowledge user (CT) as well as 3 experts in the field of gender bias for their review. Their comments were used to inform the presentation of our results. Patients were not engaged with the conception or conduct of this review.

Results

Literature search

We retrieved 4,798 citations from the electronic database search (Fig 1). Of these, 140 citations were potentially relevant and their full-texts were reviewed. Subsequently, 1 article met our eligibility criteria from the database search and was included [38]. We identified 726 records from the grey literature search. Of these, 30 records were potentially relevant and their full-texts were reviewed, however none of these were deemed to be relevant for inclusion from the grey literature search [3941].

thumbnail
Fig 1. Study Flow Diagram.

Breakdown of the number of studies identified in the literature, assessed for eligibility, and finally included in the rapid scoping review.

https://doi.org/10.1371/journal.pone.0169718.g001

Publication and participant characteristics

The included study was an uncontrolled before-after study conducted in Europe examining gender bias using an intervention [38]. The gender-blinding experiment was briefly described in Ledin 2007 [38], and additional details were obtained by contacting the corresponding author (S3 Appendix). For this study, 891 applications to the European Molecular Biology Organization (EMBO) Long-Term Fellowship programme were sent to 13 peer reviewers to be scored, after all references to the applicant’s gender had been removed. Forty-seven percent of the grant applicants, and 38% of the peer reviewers were female.

Interventions to potentially mitigate gender bias in peer review

Ledin and colleagues (2007) assessed the success of female applicants before (i.e. 2002 to 2005) and after (i.e. 2006) gender-blinding of the applications [38]. A detailed description of their intervention and data is provided in the S3 Appendix. From 2002 to 2005, 14.8% of applications led by women were successful compared with 18.1% of applications led by men (S3 Appendix). In 2006, they eliminated references to gender from the applications, letters of recommendations, and interview reports that were sent to the committees for scoring. The proportion of applications that were successful and led by women did not change after gender blinding (14.7%) although there was a slight increase in the successful applications that were led by men (20.7%). Given that lists of publications were provided by the applicants, there was nothing prohibiting the peer reviewers from conducting an internet search to identify the applicant’s identity, however the study authors felt that this was unlikely to occur given the number of applications each reviewer was assigned. Nonetheless, because it was perceived that gender blinding of the applications was not effective, this strategy was discontinued in 2007. From 2007 to 2014, there was a slight decrease in the proportion of funded applications that were submitted by women (13.8%).

Discussion

Several studies, conducted internationally, have identified that women are less likely than men to be funded in research grant competitions and this has not changed substantively over time [57, 38, 42, 43]. We identified only one study that assessed a strategy (gender-blinding of applications) to mitigate the risk of gender bias in grant peer review. The results of this study were negative [38]. However, it was uncontrolled and reviewers may have drawn conclusions about the applicants based on the de-identified information provided, including lists of prior publications. As well, some studies have found that letters of recommendation for men had a greater proportion of standout adjectives (e.g., excellent, superb, outstanding, unique) than women [44], which may also have influenced the results of the Ledin et al. study. Other recommendations to mitigate gender bias in peer review may entail ensuring appropriate representation of men and women on peer review panels, training peer reviewers in unconscious bias, and ensuring evaluation criteria are consistently applied [40, 43]. Self-awareness and scrutiny of the language and descriptors used to qualify under-represented populations may be effective for gender stereotypes. These recommendations are limited by a lack of studies that have evaluated their true impact.

Other fields in academia outside of grant peer review have likewise explored gender bias and potential strategies to minimize it, and this work can inform future steps in improving grant peer review within science. For example, a 2009 systematic review of interventions that impact gender bias in hiring examined employment for all types of positions and found a negative bias towards women being evaluated for positions that are traditionally or predominantly held by men [4547]. This earlier review identified strategies to mitigate this bias that could be considered potentially relevant to grant peer review, including using structured evaluations versus unstructured evaluations and implementing training workshops to make peer reviewers aware of common biases and how to overcome such biases [45]. A more recent study found that a 20-minute workshop providing education on implicit biases and strategies for overcoming them changed participants’ perceptions of bias; this type of workshop could be considered for grant peer reviewers [46]. In another relevant study, a cluster-randomised trial of faculty from 92 departments (including medicine) at one university showed an increase in self-efficacy to engage in gender-equity promoting behaviours following a 2.5 hour workshop [47]. When more than 25% of department members attended this workshop, there was an increase in self-reported activity to promote gender equity at 3 months [47]. Furthermore, in contrast to the uncontrolled before-after study included in our review, anonymous review of applicants without disclosing the investigator’s name or gender was found to be effective in other settings [48], suggesting that it might be useful to test this strategy using a more rigorous design, such as in symphony orchestras and the technology industry [4951]. Finally, when preparing peer review reports, peer reviewers may wish to ensure that they spend equal time on positive and negative aspects of the grant [52] and that they are aware of potential unconscious bias on gender differences [44].

There are several limitations to our rapid scoping review that should be considered. First, our search was limited to the period from 2005 to 2016 because it was a rapid review. As a result, we may have missed publications that might have contributed meaningfully to our findings. However, our discussions with experts in the field suggested that we did not miss any large, landmark studies published before 2005 that may have influenced our results. Second, we limited the articles to those published in English for feasibility reasons. Most of the studies were conducted in high-income countries similar to Canada, highlighting the fact that the results are applicable to our national funding agency, CIHR. Third, because this was a scoping review, we did not conduct a risk of bias assessment of the included study.

We believe a high quality systematic review of the literature focusing on exploring the potential for gender differences in research grant applications also be considered, given this is currently lacking. And if this systematic review indicates that a prospective study is necessary, we would encourage funding agencies to fund an experimental study. In particular, studies of NIH data show mixed results [1013], while data from the Wellcome Trust and UK MRC show no difference [14] in success rates between male and female applicants. Similarly, data from the Australian Research Council [15] did not show a difference in success rates. Data from the European Molecular Biology Organisation showed female applicants had lower success rates than men over the period 1996–2001 [16]. The European Council Research grants found women had a lower success rate than male applicants from 2008–2013 [17]. Data from the Netherlands Organisation for Scientific Research showed evidence of gender differences in success rates [18]. Data from the landmark Swedish Medical Research Council showed a gender gap [19]; Hallsten [20] conducted a review of data from the sub-council for Medicine within the Swedish Research Council and found that applications from men without an affiliation with a reviewer received lower scores than female applicants. A recent unpublished study shows that women were less successful than men in receiving funding from European Research Council grants, even after for controlling for factors such as the number of publications and grants received by the applicant [43]. And, several studies from various countries have shown that on average, women receive fewer and smaller grants than men; the reasons for this are not clear [4, 21, 22].

Conclusion

This review represents the first rapid scoping review of gender bias in grant peer review. It has highlighted substantial gaps in this area and the need for funding agencies to evaluate the impact of initiatives they implement to mitigate gender bias in peer review. Randomised trials of different training strategies for grant peer reviewers including, completion of the Harvard Implicit Association Test or participation in online education about how to break gender stereotyping and promote gender equity, may help strengthen the evidence for granting agencies to change current practice. Furthermore, the implementation of the Athena SWAN initiative in the UK provides an opportunity to explore its impact on grant success of female researchers. Recent data from a realist evaluation show that the Athena SWAN program is perceived to have an important impact across participating universities but is associated with a heavy workload to fulfil its requirements. Further complicating this issue is that the workload is largely being shouldered by women, unintentionally reproducing gender inequity through its very enactment [53]. Establishing best practices for grant peer review to reduce gender bias will benefit from more supporting evidence. Active strategies are needed to address potential gender bias in grant peer review to ensure that the creativity and innovation offered by our diverse population is not lost.

Supporting Information

S3 Appendix. Detailed description of the gender-blinding intervention and additional data.

https://doi.org/10.1371/journal.pone.0169718.s003

(PDF)

Acknowledgments

We thank Jessie McGowan for designing the search and Alissa Epworth for conducting the database search and obtaining full-text articles. We thank Dr. Gerlind Wallon for sharing her unpublished data and reviewing our manuscript. We also thank Jessica Comilang for helping with initial preparation of the manuscript for submission and Jane Aubin and Danika Goosney for comments on an earlier draft of the manuscript.

Author Contributions

  1. Conceptualization: ACT RP CT SES.
  2. Data curation: SMT.
  3. Formal analysis: SMT SES.
  4. Funding acquisition: ACT CT SES.
  5. Investigation: ACT SMT JA PR RR RP MG SS IS.
  6. Methodology: ACT SES.
  7. Project administration: SMT.
  8. Supervision: ACT SMT SES.
  9. Validation: SMT JA.
  10. Visualization: SMT IS.
  11. Writing – original draft: SES ACT.
  12. Writing – review & editing: ACT SMT JA PR RR RP MG SS IS CT SES.

References

  1. 1. Ovseiko PV, Greenhalgh T, Adam P, Grant J, Hinrichs-Krapels S, Graham KE, et al. A global call for action to include gender in research impact assessment. Health research policy and systems / BioMed Central. 2016;14(1):50. Epub 2016/07/20.
  2. 2. Council of Canadian Academies Expert Panel on Women in University Research. Strengthening Canada's research capacity: the gender dimension. 2012.
  3. 3. Beery AK, Zucker I. Sex bias in neuroscience and biomedical research. Neuroscience and biobehavioral reviews. 2011;35(3):565–72. pmid:20620164
  4. 4. Head MG, Fitchett JR, Cooke MK, Wurie FB, Atun R. Differences in research funding for women scientists: a systematic comparison of UK investments in global infectious disease research during 1997–2010. BMJ open. 2013;3(12):e003362. Epub 2013/12/12. pmid:24327360
  5. 5. Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW. Sex Differences in Application, Success, and Funding Rates for NIH Extramural Programs. Academic Medicine. 2011;86(6):759–67. pmid:21512358
  6. 6. Tamblyn R, McMahon M, Girard N, Drake E, Nadigel J, Gaudreau K. Health services and policy research in the first decade at the Canadian Institutes of Health Research. CMAJ open. 2016;4(2):E213–21. Epub 2016/07/12. pmid:27398366
  7. 7. Canadian Institutes of Health Research. 2016 Foundation Grant results [Web Page]. Canada: Canada.ca; 2016. http://www.cihr-irsc.gc.ca/e/49854.html.
  8. 8. Banaji MR, Roediger H III, Nairne J, Neath I, Surprenant A. Implicit attitudes can be measured. The nature of remembering: Essays in honor of Robert G Crowder. 2001.
  9. 9. Mutz R, Bornmann L, Daniel HD. Does Gender Matter in Grant Peer Review?: An Empirical Investigation Using the Example of the Austrian Science Fund. Zeitschrift fur Psychologie. 2012;220(2):121–9. Epub 2012/01/01. pmid:23480982
  10. 10. Ley TJ, Hamilton BH. Sociology. The gender gap in NIH grant applications. Science (New York, NY). 2008;322(5907):1472–4. Epub 2008/12/06.
  11. 11. National Institutes Of Health. Funding NIH Data Book, Data by Gender 2016. [November 2016]. https://report.nih.gov/nihdatabook/.
  12. 12. National Institutes Of Health. Data By Gender RO1-Equivalent Grants: Success Rates, By Gender and Type of Application 2015. [November 2016]. https://report.nih.gov/NIHDatabook/charts/Default.aspx?chartId=178&catId=15.
  13. 13. Pohlhaus JR, Jiang H, Wagner RM, Schaffer WT, Pinn VW. Sex differences in application, success, and funding rates for NIH extramural programs. Academic medicine: journal of the Association of American Medical Colleges. 2011;86(6):759–67. Epub 2011/04/23.
  14. 14. Grant J, Burden S, Breen G. No evidence of sexism in peer review. Nature. 1997;390(6659):438. Epub 1997/12/11. pmid:9393992
  15. 15. Marsh HW, Jayasinghe UW, Bond NW. Improving the peer-review process for grant applications: reliability, validity, bias, and generalizability. The American psychologist. 2008;63(3):160–8. Epub 2008/04/02. pmid:18377106
  16. 16. Gannon F, Quirk S, Guest S. Searching for discrimination. Are women treated fairly in the EMBO postdoctoral fellowship scheme? EMBO reports. 2001;2(8):655–7. Epub 2001/08/09. pmid:11493590
  17. 17. European Research Council. Gender Statistics 2014. [November 2016]. https://erc.europa.eu/sites/default/files/document/file/Gender_statistics_April_2014.pdf.
  18. 18. van der Lee R, Ellemers N. Gender contributes to personal research funding success in The Netherlands. Proceedings of the National Academy of Sciences of the United States of America. 2015;112(40):12349–53. Epub 2015/09/24. pmid:26392544
  19. 19. Wenneras C, Wold A. Nepotism and sexism in peer-review. Nature. 1997;387(6631):341–3. Epub 1997/05/22. pmid:9163412
  20. 20. Hallsten M, Sandstrom U. [Distribution of medical research support: men without contacts are treated unfairly]. Lakartidningen. 2009;106(30–31):1872–3. Epub 2009/09/11. pmid:19739437
  21. 21. Waisbren SE, Bowles H, Hasan T, Zou KH, Emans SJ, Goldberg C, et al. Gender differences in research grant applications and funding outcomes for medical school faculty. Journal of women's health (2002). 2008;17(2):207–14. Epub 2008/03/07.
  22. 22. Bedi G, Van Dam NT, Munafo M. Gender inequality in awarded research grants. Lancet (London, England). 2012;380(9840):474. Epub 2012/08/07.
  23. 23. Svider PF, D'Aguillo CM, White PE, Pashkova AA, Bhagat N, Langer PD, et al. Gender differences in successful National Institutes of Health funding in ophthalmology. Journal of surgical education. 2014;71(5):680–8. Epub 2014/04/30. pmid:24776863
  24. 24. Wold A, Wennerås C. Nepotism and sexism in peer review. Nature. 1997;387(6631):341–3. pmid:9163412
  25. 25. Kaatz A, Magua MW, Zimmerman D, Carnes M. A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution. Academic medicine: journal of the Association of American Medical Colleges. 2015;90(1):69.
  26. 26. Kaatz A, Lee YG, Potvien A, Magua W, Filut A, Bhattacharya A, et al. Analysis of National Institutes of Health R01 Application Critiques, Impact, and Criteria Scores: Does the Sex of the Principal Investigator Make a Difference? Academic medicine: journal of the Association of American Medical Colleges. 2016;91(8):1080–8. Epub 2016/06/09.
  27. 27. Philogene S, Lal B, Chaturvedi R, Zhu A, Huges MB, Shipp S, et al. FY 2004–2008 NIH Director’s Pioneer Award Process Evaluation—Comprehensive Report. Final Report. January 2010. Prepared for the National Institutes of Health. Washington DC: Institute for Defense Analyses, Science and Technology Policy Institute, 2010.
  28. 28. Carnes M, Geller S, Fine E, Sheridan J, Handelsman J. NIH Director's Pioneer Awards: could the selection process be biased against women? Journal of women's health (2002). 2005;14(8):684–91. Epub 2005/10/20.
  29. 29. Plank-Bazinet JL, Bunker Whittington K, Cassidy SK, Filart R, Cornelison TL, Begg L, et al. Programmatic Efforts at the National Institutes of Health to Promote and Support the Careers of Women in Biomedical Science. Academic medicine: journal of the Association of American Medical Colleges. 2016;91(8):1057–64. Epub 2016/05/19.
  30. 30. Davies DSC. Women in Science letter from Professor Dame Sally C Davies [online] London, United Kingdom2011. [August 2016]. http://www.medschools.ac.uk/SiteCollectionDocuments/Letter_from_Dame_Sally_Davies_-_Women_in_Science.pdf.
  31. 31. Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology. 2005;8(1):19–32.
  32. 32. The Joanna Briggs Institute Reviewers’ Manual 2015 Methodology for JBI Scoping Reviews [Internet]. 2015. http://joannabriggs.org/assets/docs/sumari/Reviewers-Manual_Methodology-for-JBI-Scoping-Reviews_2015_v2.pdf.
  33. 33. Tricco AC, Antony J, Zarin W, Strifler L, Ghassemi M, Ivory J, et al. A scoping review of rapid review methods. BMC medicine. 2015;13:224. Epub 2015/09/18. pmid:26377409
  34. 34. Moher D, Liberati A, Tetzlaff J, Altman DG. P`rred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Annals of internal medicine. 2009;151(4):264–9. pmid:19622511
  35. 35. Canadian Agency for Drugs and Technologies in Health (CADTH). Grey Matters: a practical tool for searching health-related grey literature [Web Page]. Canada: CADTH; 2015. https://www.cadth.ca/resources/finding-evidence/grey-matters.
  36. 36. Knowledge Translation Program. synthesi.SR Toronto: Li Ka Shing Knowledge Institute, St. Michael's Hospital. [cited 2016]. http://www.breakthroughkt.ca/login.php.
  37. 37. Tricco AC, Lillie E, Zarin W, O'Brien K, Colquhoun H, Kastner M, et al. A scoping review on the conduct and reporting of scoping reviews. BMC medical research methodology. 2016;16:15. Epub 2016/02/10. pmid:26857112
  38. 38. Ledin A, Bornmann L, Gannon F, Wallon G. A persistent problem. Traditional gender roles hold back female scientists. EMBO reports. 2007;8(11):982–7. Epub 2007/11/02. pmid:17972895
  39. 39. Kamdin. Mind the Gap, A Statistical Approach to Understanding Gender Inequality in the Physical Sciences. USA2014. p. 1–44.
  40. 40. Gutknecht-Gmeiner M LL, Stöger E, de Ridder W, Strahm P, Strahm E, Koski L, Stalker B, Hollstein R, Allulli G, Bech Kristensen O. European Peer Review Manual for initial VET. Austria: 2007.
  41. 41. Kaatz A, Gutierrez B, Carnes M. Threats to objectivity in peer review: the case of gender. Trends in pharmacological sciences. 2014;35(8):371–3. Epub 2014/08/05. pmid:25086743
  42. 42. Hewitt T. Open Letter to University Presidents from the Canada Research Chairs Program: Canada Research Chairs; 2016. [August 2016]. http://www.chairs-chaires.gc.ca/whats_new-quoi_de_neuf/2016/letter-lettre-eng.aspx.
  43. 43. Craig N. ERC awards biased against women, preliminary study says: Research Professional, Research Research Ltd; 2016. [December 2016]. http://www.researchresearch.com/news/article/?articleId=1364434.
  44. 44. Trix F, Psenka C. Exploring the color of glass: Letters of recommendation for female and male medical faculty. Discourse & Society. 2003;14(2):191–220.
  45. 45. Isaac C, Lee B, Carnes M. Interventions that affect gender bias in hiring: a systematic review. Academic medicine: journal of the Association of American Medical Colleges. 2009;84(10):1440.
  46. 46. Girod S, Fassiotto M, Grewal D, Ku MC, Sriram N, Nosek BA, et al. Reducing Implicit Gender Leadership Bias in Academic Medicine With an Educational Intervention. Academic medicine: journal of the Association of American Medical Colleges. 2016.
  47. 47. Carnes M, Devine PG, Manwell LB, Byars-Winston A, Fine E, Ford CE, et al. Effect of an intervention to break the gender bias habit for faculty at one institution: a cluster randomized, controlled trial. Academic medicine: journal of the Association of American Medical Colleges. 2015;90(2):221.
  48. 48. Mainguy G, Motamedi MR, Mietchen D. Peer review—the newcomers' perspective. PLoS biology. 2005;3(9):e326. Epub 2005/09/10. pmid:16149851
  49. 49. Goldin C, Rouse C. Orchestrating Impartiality: The Impact of" Blind" Auditions on Female Musicians. The American Economic Review. 2000;90(4):715–41.
  50. 50. Smith J. Why companies are using 'blind auditions' to hire top talent 2015. [September 2016]. http://www.businessinsider.com/companies-are-using-blind-auditions-to-hire-top-talent-2015-5.
  51. 51. Khosla V. Blind audition helps remove gender bias in recruitment 2016. [September 2016]. http://economictimes.indiatimes.com/jobs/blind-audition-helps-remove-gender-bias-in-recruitment/articleshow/51786855.cms.
  52. 52. Bauer CC, Baltes BB. Reducing the effects of gender stereotypes on performance evaluations. Sex Roles. 2002;47(9–10):465–76.
  53. 53. Caffrey L, Wyatt D, Fudge N, Mattingley H, Williamson C, McKevitt C. Gender equity programmes in academic medicine: a realist evaluation approach to Athena SWAN processes. BMJ open. 2016;6(9):e012090. Epub 2016/09/10. pmid:27609850