EDUCATION/RESIDENTS PERSPECTIVE

Does the Multiple Mini-Interview Address Stakeholder Needs? An Applicants Perspective

Andrew W. Phillips, MD, MEd; Gus M. Garmel, MD
0196-0644/$-see front matter Copyright 2014 by the American College of Emergency Physicians. http://dx.doi.org/10.1016/j.annemergmed.2014.01.021

[Ann Emerg Med. 2014;-:---.]

INTRODUCTION
The multiple mini-interview is an interview format originally created for medical school admissions to provide an objective, validated, and predictive measure for selecting applicants.1 It is essentially an objective structured clinical evaluation that measures noncognitive characteristics imprecisely dened in the literature to date.2 Applicants participate in 3 to 12 stations of one-on-one interviews, each lasting 7 to 10 minutes. Each station has 1 unique question that the applicant answers.1,3-5 For example, at one station a candidate may be asked to consider the ethical problems that exist in circumcision, and at another describe possible implications of health care reform.2 It is important to recognize the multiple mini-interview as an interview format (much like a standardized, written test is a type of test format), rather than a specic tool (such as the Medical College Admissions Test). The success of this new format depends on the complex needs and interactions of the interview process stakeholders, dened in the literature as the applicants, interviewers, and admissions teams.6 One of this articles authors (AWP) recently experienced the multiple mini-interview as an applicant at an emergency medicine program that adopted the multiple mini-interview for its resident selection process. His background in medical education research allowed him the unique perspective to understand both the programs goals for the interviews (reliable outstanding resident selection, as published in previous multiple mini-interview studies7) and an applicants goals (seeking the program in which he could most excel as a physician in training). He was impressed that the program took the bold step to change its resident selection process. To his surprise, however, he completed the multiple miniinterview with the perception of a guarded, almost stoic, community of residents and faculty. This contrasted starkly with the creative and supportive community he experienced during 2 traditional interviews at the same program that day. The unidirectional experience left him without a sense of the interviewers or the people they represented, which made it more difcult to make a nal assessment of the programs personality. This was true anecdotally for other applicants with whom he spoke as well.
Volume
-,

According to previous studies with medical student and residency applicants in other specialties, the multiple mini-interview may benet the emergency medicine residency selection process by mitigating interviewer biases and quality variability,8 reliably predicting licensing examination performance1,9,10 and providing a more fair and objective interview format.2,6 However, studies also suggest that the same interview format may provide an articial barrier for applicants and interviewers to cross and leave them disenchanted with the process.6,11 This combination of anecdotal experiences and literature from other elds presents the question, how well would the multiple mini-interview address all stakeholders needs in the emergency medicine residency application process?

THE MULTIPLE MINI-INTERVIEW FOR MEDICAL SCHOOL ADMISSIONS

The majority of multiple mini-interview research examines its efcacy for medical school admissions, which may serve as an indirect predictor for its role in the emergency medicine resident selection process and is worthy of review in the context of resident selection. Nursing and dentistry programs also recently started exploring multiple mini-interview adoption, but there is currently minimal literature available.12,13 Exactly what the multiple mini-interview evaluates (content validity) depends on the station questions an institution chooses, such as a behavioral question (eg, A patient verbally threatens you; what would you reply?) or an ethical question (eg, Explain your choice to invest in vaccines for the future or therapies for the present).8 Studies evaluating the content validity as a whole for the medical student multiple mini-interview suggest that it measures noncognitive qualities associated with professionalism but not emotional intelligence.14,15 Many of the questions in the initial study were behavioral, which differed from other studies specically addressing content validity.2 The content validitywhat the multiple mini-interview questions are truly measuring in candidatesremains relatively unclear in the literature.11 The most consistent nding is the multiple mini-interviews predictive validity. It signicantly predicted preclinical and clinical objective structured clinical evaluation performance, whereas grade point average (GPA) did not in 2 Canadian studies.1,10 Moreover, the multiple mini-interview and undergraduate GPA were equally signicantly predictive of
Annals of Emergency Medicine 1

THE APPLICANT EXPERIENCE

no.

2014

The Multiple Mini-Interview overall Canadian Part 1 licensing examination performance. The multiple choice question section scores were predicted only by GPA, whereas the clinical decisionmaking section scores were predicted only by the multiple mini-interview.10 The multiple mini-interview also predicts objective structured clinical evaluation scores during medical school, but not preclinical examination grades.1 A recent review concluded that the multiple mini-interview provides incremental validity over the GPA and Medical College Admissions Test (MCAT).16 Another feature of the multiple mini-interview for medical student selection is that preparatory coaching may actually be detrimental to scores.17 Furthermore, the cost of the multiple mini-interview is approximately the same as that of traditional interviews.4 Implications of the multiple mini-interview from the perspectives of applicants and interviewers have also been examined.6 Surveys found almost uniformly positive remarks by both applicants and interviewers, including benets of one-onone interviews, reduced sex and cultural interviewer biases, fewer total person-hours required for the interviewers, and a greater general18 sense of fairness compared with traditional interviews.6,11,19 Additionally, most medical school applicants reported that they enjoyed the multiple mini-interview experience over the traditional interview and that the multiple mini-interview improved their opinion of the school.6,19 In contrast, the only study for medical school applicants that pooled purely descriptive data in stakeholder study interviews identied applicant and interviewer concerns not observed in survey studies. In that descriptive study, applicants and interviewers alike reported that the single-question format and short station time restricted the scope of conversation and limited the quality of interaction between applicant and interviewer.11

Phillips & Garmel native to the country did not prefer one interview format to another.22, 23 The multiple mini-interview has strong to excellent statistical reliability in the small number of studies to date, with generalizability coefcients ranging from 0.6 to 0.8.20,24,25 However, there are conicting data in the 2 studies that assessed predictive validity with respect to licensing examinations. One study found moderate predictive validity, whereas the other found no statistically signicant predictions.20,24 Such a one-time, highly reliable assessment of applicants noncognitive abilities may be more relevant to selecting medical students than residents. Medical school applicants have various amounts and circumstances of clinical experiences, and there is not a standard method of demonstrating their clinically relevant noncognitive abilities to medical schools.26 In contrast, residency applicants must provide their clerkship evaluations to residency programs.27 These evaluations account for noncognitive skills during the course of multiple clerkships in a relatively standardized array of specialties (eg, internal medicine, surgery) for at least 1 year. Thus, medical schools may have a greater need than residency programs for a reliable, one-time assessment of clinically relevant noncognitive skills, such as the multiple miniinterview. With regard to the multiple mini-interviews effect on recruiting, only 8% of respondents in a large study of 484 participants reported that the multiple mini-interview negatively affected their decision to apply to the program. However, candidates were not asked whether the multiple mini-interview affected their nal program selection.25 The potential recruiting effect remains otherwise poorly described.

THE MULTIPLE MINI-INTERVIEW FOR RESIDENCY APPLICANTS

There is comparatively little known about the multiple miniinterview for resident selection in any specialty. As of 2013, only 6 original, peer-reviewed articles to our knowledge directly addressed the topic. None were from the United States or in emergency medicine. Programs studied included family medicine, obstetrics/gynecology, internal medicine, physical medicine and rehabilitation, radiation oncology, and pediatrics in Canada or the United Kingdom.20-25 Residency candidates generally appear to be comfortable with the concept of experiencing a novel interview format with regard to information and mental preparation for the change.23 Two articles that specically examined the multiple mini-interview as an objective measure of all international medical graduate applicants for their programs found that these applicants preferred the multiple mini-interview to the traditional interview, more so than applicants who graduated from medical school from the native country of the program,20,22 likely because the international medical graduate applicants described the multiple miniinterview as a more fair interview format. Residency applicants 2 Annals of Emergency Medicine

A large study by Love et al28 of all applicants to emergency medicine programs in the United States who used the match system found the interview to be the single most inuential factor within the programs control. The most important factor overall was location. A much smaller emergency medicine study at a single West Coast program found that the program director interview was the second most important aspect in applicants nal decisions, behind overall feel or personality of the program.29 In contrast, faculty interviews ranked sixth, behind resident interviews, although the sample size and scale make these nuances difcult to compare. The interviews prominent role in the applicants selection process is also present in other specialties, although it is also not reported as the single most important factor in those studies.30,31 Given that location is consistently the most inuential factor to emergency medicine residency applicants,28,32 one may question whether spending nancial and personnel resources on improving resident recruitment, including the interview format, is worthwhile. However, the costly meals, outings, and faculty
Volume
-,

THE POTENTIAL ROLE OF TRADITIONAL AND MULTIPLE MINI-INTERVIEW INTERVIEWS IN EMERGENCY MEDICINE APPLICANTS FINAL PROGRAM SELECTION

no.

2014

Phillips & Garmel and resident time spent on recruiting suggest that even the less inuential factors are nonetheless perceived as important and worthy of resources. According to the aforementioned emergency medicine residency studies,28,32 the interview should be given recruiting importance equal to if not greater than that of other factors that are within programs control. The programs interpretation of how the interviews should be conducted for the multiple mini-interview is critical to fullling a recruiting role. An overly structured interview risks a conversation from which the applicant cannot glean much information about the programperhaps what our colleagues experienced and interpreted as distant or guarded interviews. An overly unstructured interview risks a reduction in statistical reliability. The need to accept a balance between semi- and fully structured interviews for the multiple mini-interview format is a known but only recently published concept.33 Therefore, an applicants impression of a program that uses the multiple mini-interview may strongly depend on the extent of interview structure. An applicants program selection is a complex, varied, and individual process. If interviews inuence emergency medicine applicants selection process, any interview formatincluding the multiple mini-interviewwill need to fulll a recruiting role.

The Multiple Mini-Interview Widespread use of the multiple mini-interview in the emergency medicine residency selection process will likely be determined by a programs ability to nd the proper balance between selecting and recruiting residents. An additional consideration may also be whether programs can supply interviewers who are properly trained to administer the multiple mini-interview, although this has not been a reported difculty in the medical student selection multiple mini-interview literature.2,8,16 The effect of the multiple mini-interview on emergency medicine recruiting may be further complicated by the differences in specialty competitiveness and the inherent role of the interview in the different environments. The ultracompetitiveness of radiation oncology, for example, may allow the interview to be less directed at recruiting compared with the interview recruiting expectations of emergency medicine residents. Simply put, if a specialty is intensely competitive, the applicants may tend to accept whatever encounter they are offered.

CONCLUSION
The multiple mini-interview is a statistically reliable and wellvalidated interview tool to select medical school applicants, but its role to select residency applicants remains in its infancy. This is especially true in emergency medicine. At present, the multiple mini-interview appears to provide reliable interview scores and may predict some licensing examination scores in other specialties. However, its effect on resident recruiting is unknown and must be studied with particular attention to the effect of interview structure for any future customization of the multiple mini-interview to emergency medicines needs. Program directors interpretation of the unquantiable recruiting risk versus the benet of reliability and possible predictive validity will likely determine emergency medicines adoption of the multiple mini-interview in the near future for resident selection. Even more fundamentally, the needs for the entire spectrum of stakeholders in the emergency medicine resident selection process, especially program leadership and applicants, remain incompletely described. These requirements must rst be clearly established before any future research can determine whether the multiple mini-interview can meet emergency medicine resident selection stakeholders needs. For now, emergency medicine residency programs will likely continue their current approach, using traditional interviews without strong evidence or incentive for making signicant changes in their selection processes. However, if the multiple mini-interviews reported success in medical student selection studies is a reasonable predictor of outcomes in future emergency medicine residency studies, it might eventually become as commonplace in emergency medicine resident selection as it has in medical schools around the world.
Supervising editors: Stephanie A. Eucker, MD, PhD; Debra E. Houry, MD, MPH Author afliations: From the Stanford/Kaiser Emergency Medicine Residency Program, Stanford, CA (Phillips, Garmel); and Kaiser Permanente Medical Center, Santa Clara, CA (Garmel). Annals of Emergency Medicine 3

THE FUTURE OF THE MULTIPLE MINIINTERVIEW IN EMERGENCY MEDICINE

The statistical reliability and predictive validity of the multiple mini-interview will likely remain strong in residency studies, given their consistency in the medical school studies. Additionally, both medical school and residency studies use licensing examination scores as outcome variables by which to measure predictive validity. The multiple mini-interview for residency applicants of any specialty is essentially a delayed medical student multiple mini-interview, and applicants may be expected to perform similarly over time, although no study to date has assessed this. The statistical reliability and predictive validity are likely attractive to residency programs that are seeking to reduce the inherently subjective nature of interviews. The complexity with which the multiple mini-interview and other subjective factors interact to inuence emergency medicine applicants decisions is not yet fully understood. This is clear in the conicting data for medical school admissions between study methods and in the ambivalent responses from residency applicants in other specialties.6,11,19 Unfortunately, there are no data to our knowledge specically evaluating the multiple miniinterview for emergency medicine recruiting, Additional descriptive studies to elucidate fundamental stakeholder perspectives may be necessary before further Likert scale survey studies are undertaken. The notable difference in ndings between Likert and narrative descriptive studies to date suggests that the restricted answers of scaled surveys may prevent important insights of which survey authors are unaware, such as the nuances described in the qualitative study by Kumar et al.11 Whether the multiple mini-interview inuences recruiting appears highly dependent on the degree of interview structure.
Volume
-,

no.

2014

The Multiple Mini-Interview

Funding and support: By Annals policy, all authors are required to disclose any and all commercial, nancial, and other relationships in any way related to the subject of this article as per ICMJE conict of interest guidelines (see www.icmje.org). The authors have stated that no such relationships exist. Address for correspondence: Andrew W. Phillips, MD, MEd, E-mail warejko@stanford.edu.

Phillips & Garmel

15. Jones PE, Forister JG. A comparison of behavioral and multiple miniinterview formats in physician assistant program admissions. J Physician Assist Educ. 2011;22:36-40. 16. Siu E, Reiter HI. Overview: whats worked and what hasnt as a guide towards predictive admissions tool development. Adv Health Sci Educ Theory Pract. 2009;14:759-775. 17. Grifn B, Harding DW, Wilson IG, et al. Does practice make perfect? the effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust. 2008;189:270-273. 18. Reiter HI, Lockyer J, Ziola B, et al. Should efforts in favor of medical student diversity be focused during admissions or farther upstream? Acad Med. 2012;87:443-448. 19. Dowell J, Lynch B, Till H, et al. The multiple mini-interview in the UK context: 3 years of experience at Dundee. Med Teach. 2012;34: 297-304. 20. Hofmeister M, Lockyer J, Crutcher R. The multiple mini-interview for selection of international medical graduates into family medicine residency education. Med Educ. 2009;43:573-579. 21. Finlayson HC, Townson AF. Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview. Am J Phys Med Rehabil. 2011;90:330-335. 22. Hofmeister M, Lockyer J, Crutcher R. The acceptability of the multiple mini interview for resident selection. Fam Med. 2008;40:734-740. 23. Humphrey S, Dowson S, Wall D, et al. Multiple mini-interviews: opinions of candidates and interviewers. Med Educ. 2008;42:207-213. 24. Eva KW, Reiter HI, Trinh K, et al. Predictive validity of the multiple miniinterview for selecting medical trainees. Med Educ. 2009;43:767-775. 25. Dore KL, Kreuger S, Ladhani M, et al. The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions. Acad Med. 2010;85(10 suppl):S60-S63. 26. AAMC. AMCAS References. Washington, DC: American Association of Medical Colleges; 2013. [Cited December 18, 2013.] Available at: https://www.aamc.org/students/applying/amcas/amcasresources/. Accessed February 9, 2014. 27. AAMC. About ERAS. Washington, DC: Amercian Association of Medication Colleges; 2013. [Cited December 18, 2013.] Available at: https://www.aamc.org/students/medstudents/eras/about/. Accessed February 9, 2014. 28. Love JN, Howell JM, Hegarty CB, et al. Factors that inuence medical student selection of an emergency medicine residency program: implications for training programs. Acad Emerg Med. 2012;19:455-460. 29. DeIorio NM, Yarris LM, Gaines SA. Emergency medicine residency applicant views on the interview day process. Acad Emerg Med. 2009;16:S67-70. 30. Milne CK, Bellini LM, Shea JA. Applicants perceptions of the formal faculty interview during residency recruitment. Acad Med. 2001;76: 501. 31. Sledge WH, Leaf PJ, Sacks MH. Applicants choice of a residency training program. Am J Psychiatry. 1987;144:501-503. 32. Mahler SA, Wagner MJ, Church A, et al. Importance of residency program web sites to emergency medicine applicants. J Emerg Med. 2009;36:83-88. 33. Eva KW, Macala C. The effect of station type on multiple mini-interview test characteristics. Med Educ. In press.

REFERENCES
1. Eva KW, Reiter HI, Rosenfeld J, et al. The ability of the multiple miniinterview to predict preclerkship performance in medical school. Acad Med. 2004;79(10 suppl):S40-S42. 2. Eva KW, Rosenfeld J, Reiter HI, et al. An admissions OSCE: the multiple mini-interview. Med Educ. 2004;38:314-326. 3. Brownell K, Lockyer J, Collin T, et al. Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility. Med Teach. 2007;29:394-396. 4. Rosenfeld JM, Reiter HI, Trinh K, et al. A cost efciency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ Theory Pract. 2008;13:43-58. 5. Eva KW, Reiter HI, Rosenfeld J, et al. The relationship between interviewers characteristics and ratings assigned during a multiple mini-interview. Acad Med. 2004;79:602-609. 6. Razack S, Faremo S, Drolet F, et al. Multiple mini-interviews versus traditional interviews: stakeholder acceptability comparison. Med Educ. 2009;43:993-1000. 7. Prideaux D, Roberts C, Eva K, et al. Assessment for selection for the health care professions and specialty training. Med Teach. 2011;33:215-223. 8. Harris S, Owen C. Discerning quality: using the multiple mini-interview in student selection for the Australian National University Medical School. Med Educ. 2007;41:234-241. 9. Roberts C, Zoanetti N, Rothnie I. Validating a multiple mini-interview question bank assessing entry-level reasoning skills in candidates for graduate-entry medicine and dentistry programmes. Med Educ. 2009;43:350-359. 10. Reiter HI, Eva KW, Rosenfeld J, et al. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007;41:378-384. 11. Kumar K, Roberts C, Rothnie I, et al. Experiences of the multiple miniinterview: a qualitative analysis. Med Educ. 2009;43:360-367. 12. Callwood A, Allan H, Courtenay M. Are current strategies for preregistration student nurse and student midwife selection t for purpose from a UK perspective? introducing the multiple mini interview. Nurse Educ Today. 2012;32:835-837. 13. McAndrew R, Ellis J. An evaluation of the multiple mini-interview as a selection tool for dental students. Br Dent J. 2012;212:331-335. 14. Yen W, Hovey R, Hodwitz K, et al. An exploration of the relationship between emotional intelligence and the multiple mini-interview. Adv Health Sci Educ. 2011;16:59-67.

4 Annals of Emergency Medicine

Volume

-,

no.

2014