SCORE Usage Correlates with ABSITE Percentile in Surgery Residents

Introduction

The American Board of Surgery In-Training Examination (ABSITE) is a multiple-choice examination administered to surgery residents in the United States since 1975 [1]. The ABSITE is offered annually and is designed to measure the progress attained by residents in their knowledge of applied science and management of clinical problems related to surgery. Following completion of residency, the General Surgery Qualifying Exam (QE) is one of two exams taken to obtain board certification. ABSITE scores have been shown to correlate with future performance on the QE [2]. Therefore, ABSITE scores are of upmost importance during surgical residency. The ABSITE exam is standardized and considered reproducible [3]. Thus, surgical residency program directors utilize ABSITE scores as one of the measures to assess resident performance [4]. According to the American Board of Surgery, the ABSITE is furnished to program directors as a formative evaluation instrument to assess resident progress [5]. Program directors of surgical residencies therefore utilize ABSITE scores to evaluate clinical competency, medical knowledge, overall progress, and appropriate promotion within the residency program [4].

The Surgical Council on Resident Education (SCORE) was established in 2004 and is a consortium of seven US based surgical organizations [6]. The goal of SCORE was to develop a standardized, competency-based curriculum to provide residents and residency programs high-quality educational materials structured for self-directed learning. The SCORE portal, a web-based online platform, was created in 2009 and made available by purchased subscription. By 2012, 96 percent of general surgery residency programs subscribed to the SCORE web portal to support resident education [6]. Early analysis of initial subscriptions to the portal showed considerable improvement in mean American Board of Surgery Qualifying Examination scores for residents in programs with subscriptions to SCORE [7]. In 2014, the ABSITE was aligned with the SCORE Curriculum Outline for General Surgery, with ABSITE content directly corresponding to that of SCORE [8]. Overall, approximately 72% of the ABSITE questions address SCORE Patient Care topics while 24% cover SCORE Medical Knowledge topics. National resident use of SCORE has been described and the ABSITE is now a prominent factor stimulating SCORE usage [9, 10, 11]. Other institutions have shown that time spent on electronic-based resources, including SCORE, predict future ABSITE performance [12]. With extrapolation, use of SCORE may therefore predict future performance on the QE. We hypothesized that surgery resident SCORE portal time investment would correlate positively with ABSITE scores.

Methods

This was a single institution, retrospective review of all postgraduate (PG) year 2, PGY-3, and PGY-4 general surgery residents for two academic cycles (July 1, 2017 through January 24, 2019) in a large, urban based training program. Residents were reviewed individually during each academic cycle. The first academic cycle began July 1, 2017 and ended with completion of the ABSITE. The second cycle began July 1, 2018 and similarly ended with completion of the ABSITE. All residents were encouraged to use the SCORE portal by the program director. Each resident was provided an individual logon for the SCORE portal. Residents were informed that time investment on the portal would be tracked and recorded.

The SCORE portal provided the number of times and the minutes spent during each logon for the included residents. Sessions extending over 180 minutes were excluded from the analysis and total time used in minutes was calculated for each resident per academic cycle. ABSITE percentiles for the same time period were used for comparison. Residents who scored in the 30th percentile or below were identified as requiring formal remediation. The data was analyzed with descriptive statistics, student’s-t test, and ANOVA. A p-value <0.05 was considered significant.

Results

There were 51 residents included in the study between July 1, 2017 and January 24, 2019. Of the 51 residents, 16 (31%) were PGY-2s, 17 (33%) were PGY-3s, and 18 (35%) were PGY-4s. Overall, the average time spent using the SCORE portal for all included residents was 2124 minutes. The average time spent using SCORE per academic cycle was 1780 minutes for PGY-2s, 2557 minutes for PGY-3s, and 2021 minutes for PGY-4s. There was no significant difference in SCORE usage (p=0.648) between the different PGYs (Table 1).

The average percentile was 37th for the PGY-2s, 51st for the PGY-3s, and 48th for the PGY-4s. There was no significant difference in the ABSITE percentile score (p=0.353) between the different postgraduate years (Table 2).

Discussion

Increased use of the SCORE portal was associated with higher performance on the ABSITE. The SCORE portal has become an ubiquitous learning tool among general surgery residents in the United States, with specific reliance towards ABSITE preparation [10]. The vast majority of surgical residencies carry subscriptions to the portal and therefore allow residents to access the curriculum to promote self- directed study. Many surgical residency programs have also formally incorporated components of the portal into structured education. Within our program, the importance of SCORE is emphasized and certain milestones within the curriculum are now annually required.

With the ability to track individual SCORE usage and completion of modules, we have the ability to ensure what we deem as appropriate time spent on the curriculum. This Week in Score (TWIS) is series of quizzes for specific topics on the SCORE portal. Our program monitors TWIS completion and quiz scores as a component of the Clinical Competency Committee (CCC) assessment of residents. The CCC is an essential component of resident evaluation required for program accreditation. Our intent is to provide adequate and comprehensive knowledge which should be reflected in ABSITE scores given the similar topics and learning objectives. Ultimately, the goal is to see this translated to the General Surgery Qualifying Exam and subsequent Certifying Exam (CE) for board certification.

While mastering surgical technique and achieving clinical competency are essential goals for a graduating surgical resident, of equal importance is obtaining board certification by passing the QE. This exam is administered by The American Board of Surgery and is designed to assess a surgeon’s knowledge of general surgical principles and applied science [5]. The content comprised by the QE is considered necessary for the practicing general surgeon and a vital component for success. The correlation between ABSITE performance and subsequent pass rates of the QE is well-known and established, which reinforces the importance of achieving a high ABSITE score [2, 13]. A multi-institutional study by de Virgilio et al. demonstrated that ABSITE scores of less than the 35th percentile at any time during residency was associated with failure of the QE [14]. Another study by the same authors showed ABSITE scores of less than the 30th percentile once, or less than the 35th percentile more than once during residency is correlated with failure of the QE [15]. An even lower score on the ABSITE of less than the 25th percentile was associated with failure of both the QE and the CE [14].

Because ABSITE exam questions are mapped to the SCORE curriculum, one would expect that greater time on the SCORE portal improves ABSITE scores. We are not the first authors to describe this relationship. Hancock et al. identified high scoring residents (>80th percentile) spent significantly more minutes on SCORE per day [12]. Perez et al. conducted a single institution retrospective review of general surgery resident use of two electronic study resources, including SCORE and TrueLearn [12]. TrueLearn is an independent company that provides question banks to training physicians for preparation of licensure exams in medicine. Distinct differences in time spent on the electronic study resources were identified in residents with higher ABSITE scores. Longer online sessions (15 minutes per day or 35 minutes per session) and increasing use prior to the ABSITE exam was seen in the higher scoring residents. Interestingly, high ABSITE performers were more likely to access SCORE later at night and on weekends, generally outside of regular work hours [12].

Because of the importance of the QE and the correlation with ABSITE, it is necessary to identify residents at potential risk for failure. Our institution has adopted an ABSITE remediation program called Accelerated Clinical Education in Surgery (ACES) for residents scoring below the 30th percentile, as this is consistent with a higher QE failure rate [16]. This program was initially developed prior to the SCORE portal and instead utilized weekly reading and quizzes from the Surgical Education and Self-Assessment Program (SESAP) and other sources. With the development of SCORE, we have maintained our ACES program but have restructured it with SCORE modules and SCORE TWIS questions. We have observed more significant improvements in ABSITE scores of remediating residents following the adoption of SCORE. In our remediation experience, the improvement in ABSITE score is greatest in those residents who spend longer times on the SCORE portal and use it throughout the academic year.

This discussion raises questions of exactly how residents prioritize study topics and also how free time is designated for education. There have been limited studies evaluating precisely what is studied and when. Regarding free time, residents theoretically have more time for education in the context of a restricted 80-hour work week. Studies have shown that regular study habits, good conference attendance, amount of sleep prior to examination [17, 18], regular reading [19, 20], and practice questions [21, 22] increase test performance. There is limited evidence supporting that the above behaviors are better practiced with fewer clinical work hours. We are unaware of studies demonstrating increased reading time resulting from increased work hour restrictions and ABSITE scores have been unchanged since the implementation of the 80-hour work limit [23, 24]. What has been shown, however, is exponential SCORE usage peaking just prior to the ABSITE, with a significant decline upon completion. In theory, if SCORE usage were maintained consistently throughout the year, improvements in ABSITE scores and board certification would been achieved. Incorporation of SCORE assignments into our ACES program has allowed for a more regular usage of the curriculum in addition to overall increased time. Gamification of SCORE usage should also be considered [25].

There are several limitations to our study. This is a single institution experience with a small sample size. SCORE portal sessions exceeding 180 minutes were excluded as these likely represent unattended open sessions. There may, however, be residents who genuinely use the curriculum for this amount of time. In addition, only the total time spent on the SCORE portal was evaluated. The SCORE portal includes 800 module topics, TWIS quizzes, chapters from multiple textbooks, thousands of multiple-choice questions, and narrated operative videos. The number of times the portal was opened, the length of individual sessions, the topics covered, question bank use, or the amount of time on each module was not evaluated. Lastly, residents may use alternative resources for ABSITE preparation which is not reflected in total SCORE time.

In conclusion, we have shown that the utilization of the SCORE curriculum improves ABSITE scores. Further studies are needed to determine exactly which aspects of the portal lead to improved scores. This study provides additional evidence supporting the importance of the SCORE portal in achieving higher ABSITE scores and the fundamental knowledge required for surgeons.