Abstract
Context
As the number of medical school graduates continues to outpace the available residency training positions, applying for residency in the United States has become a highly competitive process, often associated with a low rate of selection and invitation for interview. The National Resident Matching Program (NRMP) Program Director survey provides data assessing factors considered by Program Directors (PD) in selecting and inviting candidates for interview. Assessing the evolution of these factors over time is efficacious to inform and guide prospective applicants toward improving preparation for residency application.
Objectives
We aim to synthesize NRMP data showing factors that PDs reported and rated as important in their decision to select and invite applicants for interview.
Methods
Data from residency PD surveys from 2008 to 2021 were accessed, but after applying inclusion/exclusion criteria, only the data from 2016 to 2020 were reviewed and analyzed. The NRMP survey reports provided two metrics that characterized PDs’ evaluation of the residency factors for interview, namely, “percent citing factor” and “average rating” on a 0 to 5 Likert-type scale. These two metrics were combined into an aggregate measure of importance (AI), and another measure of relative importance (RI) was constructed from normalizing the AI of each individual factor to the sum of the AI within each survey year.
Results
The top ranked factors were United States Medical Licensing Examination (USMLE) Step 1/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1, Letter of Recommendation (LOR) in the specialty, Medical Student Performance Evaluation (MSPE/Dean’s Letter), and USMLE Step 2 Clinical Knowledge (CK)/COMLEX Level 2 Cognitive Exam (CE) score, any failed attempt in USMLE/COMLEX, and perceived commitment to specialty. Factors rising in importance were Audition Elective/Rotation Within Your Department, Personal Statement (PS), Perceived Commitment to Specialty, Perceived Interest in Program, LOR in the Specialty, Other Life Experience, and Personal Prior Knowledge of the Applicant. Factors with declining importance were Interest in Academic Career, Awards or Special Honors in Basic Sciences, Graduate of Highly Regarded US Medical School, Awards or Special Honors in Clinical Clerkships, Lack of Gaps in Medical Education, Awards or Special Honors in Clerkship in Desired Specialty, and Consistency of Grades. Compared to the 2021 PD survey, our findings show continued predictive consistency, particularly related to specialty and program commitment.
Conclusions
The factors identified for the selection of medical school graduates for interview into a residency program reveal that PDs move toward a more integrated approach. Specifically, PDs are placing increasing emphasis on factors that border on subjective qualities more so than the more traditional, quantitative, and objective metrics. Medical students and educators need to continually apprise themselves of the NRMP data to inform students’ preparation endeavors throughout medical school to strengthen their application portfolios and enhance their competitiveness for the matching process.
Residency is an essential step in the training of physicians. From medical school, students must advance to and complete postgraduate training (residency) to qualify and practice as physicians. With the increased need for physicians, many new medical schools are coming on board both within and outside (international medical graduates) the United States. Hence, the number of medical students has been growing in recent years without attendant increases in residency slots at the same pace. Consequently, the number of students applying for residency consistently surpasses the number of postgraduate training positions available. In 1976, there were 16,112 first-year residency (PGY1) positions available with 16,728 active applicants—this represented a deficit gap of about 3.8 %. As of 2021, these numbers have grown to 35,194 and 42,508, for available positions and active applicants, respectively [1]. This represented a deficit gap of approximately 20.8 %, compared with 3.8 % in the earlier decades. Increasingly, the transition process has become highly competitive and ever-exhausting, and the intensity of the process and the accompanying disappointments it engenders in many applicants have become concerning [2].
In response to pressure from this rising competition and to boost the likelihood of getting into a residency, graduates are submitting an increasing number of applications over time [3]. Relative to the year 2005, the average number of applications submitted by US medical students has increased by approximately 130 %, that is, from 30.3 to 72.8 per applicant in 2021 [3]. The perceived competition has resulted in many applicants hedging a successful match by submitting larger numbers of applications. Although this strategy may feel intuitive and attractive, it can be costly and unaffordable to less financially privileged applicants [4]. Moreover, this tactic may have diminishing benefits, and success may vary by specialty [5].
With an ever-increasing number of applications and already limited resources and time for programs to adequately review applications [4], Program Directors (PDs) are forced to implement filters to distill the number of applications before a full review. Despite this practice, more than half of PDs spend less than 10 minutes reviewing each of the resultant applications even after the filtration [6]. Therefore, for applicants, obtaining interviews in a congested system can be a difficult endeavor.
To our knowledge, although there is growing literature guiding PDs in this evolving process [7], [8], [9], [10], comparable resource or study is limited for prospective applicants. The National Resident Matching Program (NRMP) publishes data from surveyed PDs informing students on factors considered for interview selection. Although this provides valuable information, it is limited as a snapshot of the present status and warrants a longitudinal synthesis that would provide more comprehensive information to guide student application decisions. To give a time-dependent, longitudinal perspective to residency applicants, we conducted a retrospective analysis of national surveys of PDs [11, 12] over the period 2016 to 2020 to identify prevailing trends and dominant factors that PDs considered for the invitation of applicants for residency interviews. By being well informed of the specific factors impacting selection decisions, early-career students would be able to tailor their preparation and application accordingly to obtain interviews in an increasingly competitive atmosphere.
Methods
This study was a retrospective review of secondary data from publicly available datasets. The data were obtained from the biannual publication of PD surveys by the NRMP. They came as de-identified and reported in aggregate, so no human subject research concerns were applicable to qualify for an Institutional Review Board (IRB) review.
The PD surveys for the period 2016 to 2020 on factors considered during the interview selection process for residency were accessed from the NRMP’s website and reviewed. PD response rates computed as (number of respondents)/(total number of surveys sent), declined with rates of 1,435/3,599 (39.9 %), 1,333/4,546 (29.3 %), and 924/5,143 (18 %) for the years 2016, 2018, and 2020 [1, 12], respectively. Data collection and flow chart analyses for this study are shown in Figure 1. Over the period under review, 33 factors were at play for consideration by the PDs in assessing the applicant pool and making their final decision for invitation for interview.
Data analysis flow chart. National Resident Matching Program (NRMP) Program Director (PD) surveys from 2016 to 2020 were collected, and the data were utilized to calculate aggregate importance (AI; see Figure 2). Utilizing calculated AI values, the relative importance (RI) of each factor was derived. RI was then utilized to identify factors that showed a significant increase or decrease in importance (see Figure 3).
To analyze the perceived importance of the factors as a composite of the two PD survey metrics (“percent citing factor” as important and “average rating on a Likert-type scale”), a measure called Aggregate Importance (AI) for each individual factor within their respective surveys was created. Each AI was calculated by multiplying the percentage of PDs citing the use of each factor by its corresponding average Likert rating (scale: 1=not at all important, through 5=very important) serving as a weighted score. This was done to ensure that variations from the two different measures reported for each factor were balanced out as a single aggregate index. For example, the measure of importance of a factor assigned a low percentage citation, yet had a relatively high Likert score or vice versa. Mean AI of each factor among them was then calculated, allowing us to determine the order of importance of the AIs based on the magnitude of the means (Figure 2).
Order of aggregate importance (AI) for factors considered during interview selection process. Mean values of AI for each factor listed from highest to lowest *: Utilizes the USMLE data for 2020 in the calculation given that the COMLEX’s use is less frequently cited relative to its USMLE counterpart suggesting prior to 2020. Because PDs were likely assigning value primarily based on the USMLE, this allowed us to equate the USMLE in 2020 to that of the combined USMLE/COMLEX factor in prior surveys.
Trend analyses were performed to identify appreciating and depreciating factors over the study period. To adjust for time-dependent trends, each year’s data was internally normed/modified to offset year-to-year variations, likely due to fluctuations in the PD response rates. For this norming, the relative importance (RI) for each factor was obtained by dividing the individual factor’s AIs by the sum of all factors’ AI in the survey year, multiplied by 100 %. To observe meaningful time-dependent changes, the RIs were subjected to a linear regression analyses to identify possible increasing or decreasing trends in the slopes. We identified rising and declining factors as those with a slope >0.1 and <−0.1, respectively, and with a corresponding R2>0.8. Figure 3 shows the factors and their corresponding time-dependent changes.
Factors considered among residency program directors trending in increasing (A) and decreasing (B) importance. Factors demonstrating increasing and decreasing importance over time can be seen in panels (A) and (B), respectively. The mean and slope for each factor is indicated next to their respective plotted line.
A follow-up content analysis was conducted on the resulting subset of factors from the initial 33 into logical categories that were underpinned by a common defining characteristic. Those were factors found to be significant from the quantitative analysis.
Results
Rank of aggregate importance for factors considered for interview selection
Utilizing the AIs of possible values of 0 through 5, values greater than or equal to 3 signified higher importance with the rating index. Figure 2 illustrates the overall ranking of all factors from 2016 to 2020. Specific factors with AI≥3.0 are:
USMLE Step 1/COMLEX Level 1
Letter of recommendation (LOR) in Specialty
Medical Student Performance Evaluation (MSPE/Dean’s Letter)
USMLE Step 2 CK/COMLEX Level 2 CE score
Any failed attempt in USMLE/COMLEX
Perceived commitment to specialty
These factors could be categorized as underpinning a construct of “academic performance, endorsements, and commitment to specialty.” Following this group of the most highly valued factors, eight factors have AI≥2.5, suggesting moderately higher importance:
Grades in required clerkships
Personal statement
Evidence of professionalism and ethics
Personal previous knowledge of the applicants
Grades in clerkships in desired specialty
Audition elective/rotation within your department
Class ranking/quartile
Leadership quality
Most of these factors could be categorized to “professional characteristics, and performance and commitment to specialty/program.” In general, this group echoes the theme, “commitment to specialty” identified in the AI≥3.0 group.
Time-dependent trends
From the linear regression, utilizing the RIs of all 33 factors, time-dependent trends for rising and declining factors were obtained. Seven factors with a slope>0.1 and R2>0.8 showed a rising trend (see data table in Figure 3A), and another seven factors with a slope<−0.1 and R2>0.8 exhibited a declining trend (see data table in Figure 3B).
Based on the shared innate characteristics, these 14 factors can be grouped into four general categories (Table 1): Category I includes factors reflecting “Relationship of Individual to Specialty,” Category II “Relationship of Program to Individual,” Category III “Innate Personal Characteristics,” and Category IV “Academic Performance.”
Rising and declining factors identified for the period, 2016–2020.
| Category I Relationship of individual to specialty |
Category II Relationship of program to individual |
Category III Innate personal characteristics |
Category IV Academic performance |
|---|---|---|---|
|
|
|
|
-
LOR, letter of recommendation.
Category I (“Relationship of Individual to Specialty”) was constituted by the two factors, perceived commitment to specialty and LOR in the specialty, which relate to demonstrating one’s dedication toward the applied specialty. Category II (“Relationship of Program to Individual”) may relate to program preference toward inviting known candidates and those whose applications suggest that they are likely to rank their program highly. Category III (“Innate Personal Characteristics”) consists of factors that offer additional insights or information that may relate to Categories I/II, depending on one’s unique background and the content of one’s Personal Statement. Meanwhile, Category IV (“Academic Performance”) relates to academics, specifically experiences from within the field of medicine and evidence of superior scholastic abilities.
In general, Categories I–III are constituted by factors that cannot be easily quantified utilizing standardized metrics, instead relying on relatively subjective assessments as interpreted by individual faculty within a program and as interpreted between programs. These factors rely largely on characteristics existing outside of a candidate’s scholastic control, with reliance on intrinsic qualities and experiences. Together, these describe personal characteristics and other knowledge of the applicants.
Rising factors
Among the seven factors displaying a rising trend (see Figure 3A and its associated data table), Audition elective/rotation within your department had the sharpest increase based on slope, followed closely by Personal Statement. It is also noted that, with the exception of Other Life Experiences, all other six factors among this group have stronger mean RI of >3.0, with the LOR in the specialty having the highest mean RI. These metrics suggest that these seven rising factors also have a greater weighted importance among the 33 factors. In categorization, all seven factors fall into Categories I–III (Table 1).
Declining factors
Seven factors showed a declining trend (see Figure 3B and its associated data table). Consistency of grades showed the sharpest decline based on slope, followed by Awards or special honors in clerkship in desired specialty. In addition, these seven factors collectively have weaker mean RIs, ranging from 1.03 (the lowest importance: Awards or special honors in basic sciences) to 2.98 (Consistency of grades). Together, these metrics suggest these seven declining factors also have a lesser weighted importance among the 33 factors. In categorization, all seven factors fall into Category IV (Table 1).
Discussion
Utilizing the aggregated analyses considering Percent Citing Factor with their respective Average Rating released from NRMP PD surveys, we determined over the period covered by this study (2016–2020) the factors that PDs considered in inviting applicants for residency interviews. Our ranking order analyses identified factors that are most valued by the PDs in the initial selection process, which were those providing a construct of academic performance, endorsements, and commitment to specialty.
The time-dependent trending studies identified increasing and decreasing trends among the 33 factors that PDs considered for inviting applicants during 2016–2020. We showed seven factors of an increasing trend with Audition elective/rotation within your department and Personal statement have the highest rate of increase. Furthermore, among the seven increasing factors, Perceived commitment to specialty and LOR in the specialty had the highest RI. Overall, these factors were associated with an underlying construct of applicant personal characteristics and knowledge that could be described as not easily amenable to the traditional, objective, and quantifiable metrics for assessment. This is in marked contrast with the characteristic of factors that demonstrated decline in our study. Seven factors were identified to have a declining trend, with Consistency of grades exhibiting the sharpest decline. In fact, all these declining factors are related to academics, including mostly recognition of superior scholastic abilities. In other words, the factors in decline are thematically opposite to those seen rising in valuation and provides evidentiary support for one another. Together, we observe a theme consistent of evaluating and selecting an applicant in a more holistic approach [13].
Among all the trending factors, it is not surprising that LORs have increasingly served to engender a more personalized perspective of an applicant. Such attestation provides independent yet focused testimony that may not be otherwise discerned [14, 15]. Although selecting LOR writers that will speak highly of an applicant is important, it may be wise to limit the number of writers from outside of the applied specialty to intentionally create a networking effect within a desired specialty and signal commitment. Additionally, a chair letter may be an additional opportunity to demonstrate commitment, which is a factor not yet measured by the NRMP. Reliance on indirect appraisals can be circumvented via audition rotations (ARs). As one of the fastest rising factors, it allows a more comprehensive and direct evaluation by programs, which is particularly important for osteopathic students [16]. Furthermore, ARs can offer applicants not only an opportunity to express explicit interest in particular programs but also may communicate geographical preference and specialty commitment.
While carrying out this study, our team also substantiated the disparity in PDs assessing USMLE and COMLEX results from applicants. At the beginning of 2020, the factors pertaining to the USMLE/COMLEX were assessed separately for the first time by the NRMP. Prior to 2020, the “USMLE Step 1/COMLEX-USA Level 1” was the most reported factor that the PDs considered. Meanwhile, in 2020, while Step 1 remained the most cited, Level 1 was ranked number 18. Therefore, parcellation of these factors seen in 2020 onwards yields valuable insight on the inequitable weighting of these examinations by PDs and an inadvertent measurement bias in prior surveys. This is despite the American Medical Association’s (AMA) unanimous vote in 2018 to officially recognize these examinations as equal, promoting their acceptance at all US residency programs [17, 18]. Despite this inequity, for PDs who do take it into consideration, similar values of importance are assigned [12].
Although osteopathic students should be viewed now (and always) as equals to their allopathic colleagues, this may not be so and may be slow to change without a catalyst. Despite tremendous progress, a glass ceiling may exist for osteopathic candidates as allopathic candidates appear to be given preference in competitive specialties, even at present [19]. Accordingly, among all specialties, 37 % of PDs ‘never’ or ‘seldom’ consider an osteopathic senior for an interview, a figure approaching 100 % in some fields (Figure 3 Summary of Program Interviewing and Ranking Activities. Sub figure: Percentages of Programs Interviewing Candidates by Applicant Type, 2021) [1]. Given the ongoing concerns for systemic discrimination [20], osteopathic medical graduates should be considered within programs’ Diversity, Equity, and Inclusion protection.
Limitations
While many interesting insights may be derived from this study, there were several limitations. Each year, the response rate to the PD surveys was relatively low, resulting in populations of PDs (responding vs. nonresponding) with potential bias due to lack of representativeness of the respondent sample. As a survey, it may be riddled with inherent selection bias. Participating PDs may be more inclined to favor some factors than nonrespondent PDs. Additionally, factor emphasis may differ by specialty and should be a topic of future research to benefit students with interests in specialty areas.
Conclusions
The factors that residency Program Directors consider as most important for their decision to invite applicants to interview evolve over time. Although some important factors maintained their traditional recognition, increasingly many nontraditional factors are rapidly gaining recognition of importance. Evolving values will continue to shape the future and attendant shifts in the importance of these factors in postgraduate medical education. Understanding these changes will be imperative to maximize a medical student’s success in obtaining an interview as a first step toward entry into residency (postgraduate medical education).
Acknowledgments
The authors are grateful to Rex A. Dwyer, PhD, for the discussion of data analysis and for performing linear regression.
-
Author contributions: All authors provided substantial contributions to conception and design, acquisition of data, or analysis and interpretation of data; all authors drafted the article or revised it critically for important intellectual content; all authors gave final approval of the version of the article to be published; and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
-
Competing interests: None reported.
-
Research funding: None reported.
References
1. Results and Data: 2021 Main Residency Match. National Resident Matching Program. https://www.nrmp.org/match-data-analytics/residency-data-reports/ [Accessed 14 May 2021].Search in Google Scholar
2. Gruppuso, PA, Adashi, EY. Residency placement fever: is it time for a reevaluation? Acad Med 2017;92:923–6. https://doi.org/10.1097/ACM.0000000000001468.Search in Google Scholar PubMed PubMed Central
3. ERAS Statistics. Association of American Medical Colleges. https://www.aamc.org/data-reports/interactive-data/eras-statistics-data [Accessed 21 December 2021].Search in Google Scholar
4. Chang, CW, Erhardt, BF. Rising residency applications: how high will it go? Otolaryngol Head Neck Surg 2015;153:702–5. https://doi.org/10.1177/0194599815597216.Search in Google Scholar PubMed
5. Mann, S. How many residency applications? AAMC.org Web site updated 2019. https://www.aamc.org/news-insights/how-many-residency-applications [Accessed 4 July 2022].Search in Google Scholar
6. Dunleavy, D, Geiger, T, Overton, R, Prescott, J. Association of American Medical Colleges. Results of the 2016 program directors survey: current practices in residency selection; 2016. Available from: http://www.r2library.com/resource/title/0000901083.Search in Google Scholar
7. Katsufrakis, PJ, Uhler, TA, Jones, LD. The residency application process: pursuing improved outcomes through better understanding of the issues. Acad Med 2016;91:1483–7. https://doi.org/10.1097/ACM.0000000000001411.Search in Google Scholar PubMed
8. Puscas, L. Viewpoint from a program director they can’t all walk on water. J Grad Med Educ 2016;8:314–16. https://doi.org/10.4300/JGME-D-16-00237.1.Search in Google Scholar PubMed PubMed Central
9. Hartman, ND, Lefebvre, CW, Manthey, DE. A narrative review of the evidence supporting factors used by residency program directors to select applicants for interviews. J Grad Med Educ 2019;11:268–73. https://doi.org/10.4300/JGME-D-18-00979.3.Search in Google Scholar PubMed PubMed Central
10. Sinnott, C, Johnson, N. “First come, first served” serves no one: a call for change to residency interview invitation practices. J Grad Med Educ 2019;11:255–7. https://doi.org/10.4300/JGME-D-19-00015.1.Search in Google Scholar PubMed PubMed Central
11. Stone, C, Zwerling, CS. A retrospective analysis of contemporary trends from the NRMP program director surveys: a call to action. OP-ED Column. Ann Rev Res 2019;5:55566. https://doi.org/10.19080/ARR.2018.04.555664.Search in Google Scholar
12. Match Data and Report Archives. National Resident Matching Program. https://www.nrmp.org/match-data-analytics/archives/ [Accessed 14 May 2021].Search in Google Scholar
13. Association of American Medical Colleges. Holistic review. AAMC.org; n.d. https://www.aamc.org/services/member-capacity-building/holistic-review [Accessed 29 Jan 2023].Search in Google Scholar
14. Kominsky, AH, Bryson, PC, Benninger, MS, Tierney, WS. Variability of ratings in the otolaryngology standardized letter of recommendation. Otolaryngol Head Neck Surg 2016;154:287–93. https://doi.org/10.1177/0194599815623525.Search in Google Scholar PubMed
15. Garmel, GM, Grover, CA, Quinn, A, Lotfipour, S, Langdorf, MI, Cheaito, MA, et al.. Letters of recommendation. J Emerg Med 2019;57:405–10. https://doi.org/10.1016/j.jemermed.2019.04.020.Search in Google Scholar PubMed
16. Reeves, RR, Burke, RS. Perception of osteopathic medicine among allopathic physicians in the deep central southern United States. J Am Osteopath Assoc 2009;109:318–23. 109/6/318 [pii].Search in Google Scholar
17. Status-implementation of resolutions and report recommendations AMA house of delegates interim meeting – November 10–13, 2018; 2019.Search in Google Scholar
18. National Board of Osteopathic Medical Examiners. American medical association officially recognizes COMLEX-USA’s equality with USMLE. December 3, 2018. https://www.nbome.org/news/american-medical-association-officially-recognizes-comlex-usas-equality-with-usmle/ [Accessed 14 May 2021].Search in Google Scholar
19. Craig, E, Brotzman, E, Farthing, B, Giesey, R, Lloyd, J. Poor match rates of osteopathic applicants into ACGME dermatology and other competitive specialties. J Osteopath Med 2021;121:281–6. https://doi.org/10.1515/jom-2020-0202.Search in Google Scholar PubMed
20. Gevitz, N. The progress and consequences of the ACGME merger: a call for action. WWW.AACOM.gov. September 2016. https://www.aacom.org/docs/default-source/hpf-2018/2018-02_seminar/progress-and-consequences-of-acgme-merger.pdf?sfvrsn=69e92097_2 [Accessed 14 May 2021].Search in Google Scholar
© 2023 the author(s), published by De Gruyter, Berlin/Boston
This work is licensed under the Creative Commons Attribution 4.0 International License.
