Changing Perceptions of Graduating Students on a Decade of Criterion-Referenced Assessment and Grading System Implementation
Article information
Abstract
Yonsei University College of Medicine (YUCM) adopted a criterion-referenced assessment (CRA) system in 2014. Six cohorts have graduated over the past decade under CRA. Positive impacts on student wellness and academic achievement were observed during the system’s initial years, and this study therefore aimed to analyze the changing perceptions of graduating students who experienced CRA to reflect YUCM’s experience of CRA implementation and suggest future directions and insights. Utilizing self-reporting graduation survey data of 621 graduates from 2018 to 2023, eight items regarding perceptions of CRA were examined with analysis of variance to explore changes in perceptions among the graduates, and short answers were also reviewed to investigate keywords and main themes. The CRA grading system positively influenced students’ learning motivation, level of class participation, cooperative attitudes, and self-directed attitudes. In particular, students’ perceptions of cooperative attitudes significantly improved from 2018 (3.97) to 2023 (4.46). The system also had a positive impact on students’ interest in conducting research from 2018 (3.04) to 2023 (3.56). Students’ perceptions of validity significantly increased from 2018 (3.40) to 2023 (3.92), and the perceptions of reliability also increased from 2018 (3.38) to 2023 (3.65), although the inter-year changes were not statistically significant. This study on graduating students’ changing perceptions of the CRA at YUCM aligned with the intended goals and context of its introduction and implementation. Ongoing improvements in fairness and reliability are required to enhance students’ perceptions of the system’s credibility, necessitating ongoing education and improvement efforts. Further studies on the positive and negative factors influencing perceptions of CRA are required to derive insights for system improvement.
Introduction
The introduction of competency-based medical education (CBME) proposed a paradigm shift toward outcome-based, desired-goal-oriented education [1,2]. This shift in medical education aims to address the health needs of the population through the desired outcomes and competencies of education and training [3]. CBME intends to enhance the mastery level of educational achievement of all learners by providing inspiration and guidance for learning while ensuring public safety through identifying incompetent candidates [4,5].
As the purpose of CBME is to ensure the achievement of competence, criterion-referenced assessment (CRA), often referred to as pass/fail grading, is considered an appropriate match for CBME evaluation [2,6]. Unlike norm-referenced assessments, CRA evaluates the students’ performance against predefined outcomes. CRA complements CBME by supporting mastery learning. Since its development [7], Hambleton et al. [8] have identified the use of CRA as assigning students’ mastery states, determining whether the examinee’s score meets a standard that indicates mastery at the mastery level. Medical schools utilize CRA to assess whether individuals achieve the mastery level required to become competent physicians.
Although CRA has been employed by numerous medical schools for decades, concerns have been raised about decreased motivation and subsequent academic performance among undergraduate students. In the United States during 2021–2022, approximately 94.8% of medical schools applied the CRA either partially or fully within their curriculum [9]. However, precedent studies have shown that academic performance, including United States Medical Licensing Examination step 1 scores under pass/fail grading in preclinical years, was not inferior to traditional 5-interval grading [10-13].
In addition to enhancing academic achievement and competence, CRA promotes students’ well-being by reducing competition and time pressures, thereby alleviating burnout [14,15]. The stressful atmosphere of medical school is particularly intense during the first year, with a significant increase in academic workload, pressure to achieve high academic success, and the challenge of mastering extensive information alongside equally motivated and intelligent peers [16]. This often leads to a higher prevalence of depression compared with the general population, due to the competitiveness [17,18]. The CRA, otherwise, has the potential to reduce anxiety and promote stronger group cohesion among medical students by alleviating unnecessarily excessive competitiveness [19].
Based on these plausible effects of CRA, Yonsei University College of Medicine (YUCM) in South Korea first adopted CRA in its grading system. South Korea has 40 medical schools, including public and private institutions. Like other medical schools in South Korea, YUCM employed a norm-referenced assessment system until 2013. With the introduction of a new undergraduate curriculum in 2013 titled the Curriculum Development Project (CDP) 2013, YUCM adopted CRA for the first time in South Korea. In CDP2013, the educational content and curriculum were organized based on the main principles of student-centered learning, outcome-based learning, research-oriented learning, and integrated learning, named “SORI” combining the first letters of each principle. With these principles, learning-centered educational programs, CRA, research programs, and organ system-based integrative curricula were introduced; these programs not only support the main principles but also complement each other toward CBME. To implement CRA through CDP2013, YUCM organized a committee in 2010 and started preparing for the new system. Since 2013, the grading system has shifted from a five-leveled letter-graded system (A, B, C, D, and F), which was decided proportionally according to the ranking, to CRA with pass with honor, pass and non-pass grades, and has been applied fully across all subjects in undergraduate medical courses. YUCM applied Ebel method to establish the minimum academic competence for borderline students. Students who exceed this threshold are designated as ‘pass’; however, if further reinforcement is deemed necessary, they are assigned a ‘non-pass’.
A decade after the initial implementation of the CRA, a sixth cohort of students graduated in February 2023. YUCM is currently at a pivotal moment as it reviews its system in light of the curricular renovation that began in 2023. This involves reassessing the overall curricular structure while maintaining the CRA. Additionally, there is pressure to set an exemplary standard for the CRA as other medical schools in Korea gradually implement it. Although there are studies regarding students’ well-being and mental health as an effect of CRA [20], a study of YUCM students’ perception of CRA has not yet been performed.
Thus, this study aims to reflect on YUCM’s experience with the CRA from the perspective of graduating students who experienced it, by analyzing changes in their perceptions over time regarding learning, research, and the validity and reliability of the CRA since its implementation. Specifically, this study investigates three areas. First, it examines changes in learning perceptions aligned with the objectives of introducing the CRA at YUCM, focusing on four aspects: learning motivation, level of class participation, cooperative attitude, and self-directed learning attitude. Second, as CRA considers various outcomes besides academic achievement, this study explores how its introduction has influenced students’ perceptions of research and its impact on career development. Third, to assess changes in students’ perceptions of CRA, the study investigates graduating students’ views on the validity and reliability of the system, which are key components of the assessment.
Methods
1. Data sources
YUCM annually conducts a “Graduation Survey” for graduating students at the end of the entire program. This survey primarily includes items related to perceptions of curriculum and associated systems, including CRA. This study utilized 6 years of data from these surveys from 2018 to 2023, analyzing specific relevant items. After excluding those who provided unreliable responses, 621 students were selected for the final data analysis. Except for 2022, which had a low survey response rate owing to coronavirus disease 2019 (COVID-19), approximately 110 individuals participated in the survey each year. The gender ratio of the entire dataset was approximately 70% men and 30% women, with an average response rate of 85.2% across all 6 years (Table 1).
Descriptive characteristics of the data
The institutional review boards exempted the need for approval due to the retrospective nature of the study and the anonymized date for this non-interventional study (no., 4-2024-0125).
2. Variables and research methods
1) Variables
The graduation survey at YUCM comprises approximately 200 questions, including sub-questions. We analyzed changes in the perceptions of graduating students through eight questions that assessed their awareness of the CRA (Table 2). The analysis was conducted across three domains: learning perception, perception of research, and validity and reliability of the CRA. Learning perception was divided into four sub-domains: “learning motivation,” “level of class participation,” “cooperative attitude,” and “self-directed learning attitude.” Perception of research was divided into “research interest” and “research-related career development.” The validity and reliability of the CRA were categorized into “validity” and “reliability.” All subdomains were measured with a single question, and each question followed a self-report format using a 5-point Likert scale (strongly agree=5, agree=4, neither agree nor disagree=3, disagree=2, strongly disagree=1).
Survey questions for graduating students
2) Research methods
To examine the mean differences in participants’ perceptions across the 6 years, we conducted analysis of variance and used F tests to assess the significance of annual differences. Subsequently, we conducted Scheffé’s post-hoc tests to determine whether there were significant differences in the means between any two specific years. Scheffé’s method is particularly suitable for analyzing unbalanced sample-sized data and is insensitive to assumptions of normality and homogeneity of variances, thus making it appropriate for post-hoc analysis [21]. We employed listwise deletion to handle cases with missing data. Finally, we examined the factors influencing participants’ perception changes by extracting keywords from the responses to open-ended questions.
Results
1. Leaning perception
We examined the annual changes in perception to determine whether the CRA positively influenced participants’ “learning motivation.” The analysis revealed that the perception of “learning motivation” scored 3.84 in 2018, 3.85 in 2019, and significantly increased to 4.24 in 2021. Regarding “level of class participation,” the scores were 3.70 in 2018, 3.77 in 2019, and significantly increased to 4.18 in 2021. This suggests that the CRA fosters more profound engagement in classes rather than merely being regular class. Furthermore, students’ perception of “cooperative attitude,” which aligns with the objective of the CRA, scored 3.97 in 2018, 4.03 in 2019, and significantly increased to 4.46 in 2023. This suggests that introducing the CRA shifted the learning culture from a competitive learning environment to cooperative team-based learning. Lastly, the “self-directed learning attitude” score significantly increased from 3.78 in 2018 and 3.88 in 2019 to 4.18 in 2023 (Table 3).
Participants’ perceptions of criterion-referenced assessment (learning perception)
Subsequently, we explored the factors influencing changes in perception based on the qualitative responses to open-ended questions. These questions were designed to ask participants for the reasons behind their answers. First, regarding “learning motivation,” some students responded that the CRA positively impacted them by “reducing the sense of disparity among students” and “decreasing feelings of defeat.” Conversely, some students responded negatively, mentioning that the CRA led to “complacency,” “an atmosphere of not studying hard,” and “reduced motivation.” Second, regarding the “level of class participation,” students provided mixed responses. Some noted that the CRA encouraged a more active attitude in class, while others disagreed. Third, regarding “cooperative activities,” many students observed that the CRA contributed to “increasing collaboration and cooperation among peers” and “creating a supportive learning culture,” which they felt aided their cognitive and affective growth. Lastly, concerning “self-directed learning attitude,” some students responded that the CRA helped them enhance their autonomous learning attitude.
2. Perception of research
We examined annual changes in perception to determine whether the CRA positively influenced participants’ “perception of research.” The analysis revealed that the CRA positively impacted students’ interest and engagement in research. Specifically, the level of interest in research among students who first experienced the CRA was 3.04 in 2018 and significantly increased to 3.61 in 2021 and 3.56 in 2023, showing a general upward trend. Although there was a slight decline in 2022, this was attributed to the reduced number of respondents owing to the COVID-19 pandemic.
Additionally, the CRA positively influenced the diversity of career development for graduating students. Specifically, the score significantly increased from 3.17 in 2018 to 3.67 in 2023 (Table 4). These results suggest that the introduction of the CRA provided opportunities to assess various forms of learning outcomes, considering individual interests beyond simple academic achievement.
Participants’ perceptions of criterion-referenced assessment (perceptions of research)
The study examined the proportion of students who perceived a positive impact of the CRA on their coursework and clinical practice, specifically in the research domain. Approximately 16.2% of graduating students in 2018 and 2019, and 20.3% in 2022 and 2023, responded affirmatively regarding CRA’s positive influence on “research.” Specific responses included statements such as, “I was able to focus on research after the CRA was implemented,” “I had adequate time to invest in research,” and “I could allocate time for both research and personal development.”
3. Validity and reliability of the CRA
Through survey questions asking whether students believed academic competencies were appropriately evaluated according to learning objectives under the CRA, we examined changes in perceptions regarding the validity of the CRA. The analysis revealed positive shifts in perceptions regarding the validity of evaluations under the CRA, with scores significantly increasing from 3.40 in 2018 to 3.85 in 2021 and 3.92 in 2023. Furthermore, we explored participants’ perceptions of the reliability of assessment outcomes under the CRA but did not find significant changes in their perceptions regarding the consistency and reliability of assessments (Table 5).
Participants’ perceptions of CRA (validity and reliability of the CRA)
Although significant results were not obtained, we investigated, through open-ended questions, whether graduating students perceived the CRA in their past courses and clinical practice as reliable, and explored the reasons. Graduating students who perceived the CRA as reliable mentioned “consistency in outcomes,” “introduction of clear learning objectives and objective evaluation criteria,” “inclusion of diverse assessment factors beyond ranking,” and “trust in professors.” Conversely, students who did not perceive CRA results as reliable mentioned “discrepancies in evaluation criteria across courses” and “skepticism towards the Honor evaluation criteria,” calling for improvements to the system.
Discussion
In this study, the overall perception improved over time compared to the initial stages. This improvement may be attributed to the doubts and concerns initially held by students regarding the CRA system, which were likely reflected in the perceptions of the first cohorts. Over time, as the system became internalized among stakeholders and these doubts and concerns were alleviated, perceptions gradually improved, supported by the observable positive effects of the CRA system. The graduating cohort of 2021 displayed an exceptionally positive perception, which may have been partially influenced by external factors such as the COVID-19 pandemic or the 2020 residents’ and medical students’ strike. Further analysis is necessary to confirm the impact of these factors. The trends in graduating students’ perceptions of the CRA over the last decade met the aim of introducing a new system to YUCM in the following points.
First, the CRA promoted a positive shift in students’ attitudes of learning perceptions toward individual mastery learning. To enhance the atmosphere of mastery learning and promote learning motivation, YUCM provides “a pass with honor” for some students who meet the criteria for each course. “Pass with honor” would acknowledge students’ remarkable academic achievements and encourage them to deepen their learning. Setting up the criteria for passing with honor was dedicated to each course director. CDP2023 will provide the “advanced course” to the “pass with honor” students.
In YUCM, the Learning Community (LC) has played an important role in aiding the implementation of the CRA by providing guidance to the students and facilitating communication between the school and students regarding the system. Along with the CRA, YUCM launched LC program in CDP2013. The faculty members in charge of each LC group were supervisors concerned with the curricular progress and serving as mentors throughout the medical school years. This LC program with supervisors strengthened the self-directed learning and enhanced the overall learning environment supporting the CRA implementation [15,22].
Second, students have started exerting their effort in other important aspects to be good physicians. As seen from the students’ ideas in open-ended questions, the atmosphere of excessive and blindly consuming study effort in norm-referenced testing has diminished, while students have tended to spend their time on research, with an increase in interest in scientific and medical research. According to data from a graduating student survey, the number of published studies has increased since the CRA was applied. In the CDP2013, a Research Education Course (REC) was also implemented, expecting the students to have more interest and time in clinical and basic scientific research during their undergraduate years, along with the change in the grading system. The YUCM sets a perception of research as one of the core outcomes to be achieved through the undergraduate curriculum. As research constitutes a crucial component of medical doctors’ academic studies and clinical practices, competent physicians require fundamental research skills and the cultivation of research-related attributes [23]. Based on the analysis, students’ perception of research showed a favorable shift due to the change to the CRA, as integrated with REC in the CDP2013. CDP2023 has a revised research curriculum comprising a scholarly immersion course that allows students to focus their whole semester on research after completion of three semesters of preclinical clerkship courses (phase 1) and two semesters of common core clinical clerkship courses (phase 2). The synergistic integration of the CRA with this research immersion course would enhance the perception of research, indicating students’ perceptual improvement and the quantitative and qualitative advancement of the school’s competency from the perspective of school policy. Further research on the perception of research and associated outcomes is required as the CDP2023 proceeds.
Third, there should be a common sense of understanding among faculty members, students, and schools regarding the CRA. In the CRA, the criterion for judging students’ achievement is whether the student has reached the “borderline.” Therefore, faculty members should consider what competencies students must demonstrate to “prove” that they have reached the borderline and plan to evaluate accordingly. Moreover, faculty members must design courses and create items that allow students to demonstrate their competencies. Furthermore, providing accurate feedback to students is crucial for enhancing their learning, which can drive the integration of the curriculum, teaching, and assessment. Proactive support from schools is required to improve school members’ perceptions of the CRA to accomplish this. The school administration should plan and conduct training programs for assessor development.
Planning and conducting training programs for faculty members and producing and distributing a guidebook containing fundamental concepts of the CRA, the literal meaning of “borderline,” and examples of appropriate items at the borderline level are crucial. This approach aims to stabilize and activate the CRA. Additionally, students should be informed that the CRA focuses on supporting the learning process and providing information for self-evaluation and self-assessment, shifting from “assessment of learning” to “assessment as learning.” Furthermore, it is important to encourage students to recognize that the primary beneficiary through assessment tools is neither the teachers nor the school but the students themselves. These processes will help the system members understand the validity and reliability of the pass/fail grading system.
Ultimately, as this is Korea’s first case of implementing CRA in medical school with “pass/non-pass” grades, the consideration of the system with residency match is now becoming more important than ever in the CRA as no one had imagined residency matching without grade point average and ranking in Korea. The first cohort of graduating students with the CRA in 2018 expressed a high level of anxiety about how the evaluation outcomes could be utilized in the selection process for internships and residency matches. Our analysis shows that concerns have decreased recently. According to previous studies, there was no significant difference in academic performance between the norm-referenced tiered grading system and the CRA with pass/fail grading; consequently, no significant difference was found in resident placement [24]. Additionally, regarding the long history of the CRA with pass/fail grading and the utilization of the grading in resident selection processes, a survey conducted in the United States found that the residency program directors perceived that the performance in residency program from both the tiered grading system and the pass/fail grading system showed no significant difference [25].
As the YUCM was the first medical school to implement the CRA in South Korea, utilizing the evaluation results in residency matching is critical and could be exemplary. Most of the graduates from YUCM were matched at Severance Hospital, managed by the Yonsei University Healthcare System (YUHS), which YUCM is a part of. Therefore, the grading system and its results in residency selection have agreed well between YUHS and YUCM so far. Other medical schools in South Korea have begun implementing the CRA, and more schools are expected to follow this trend. Therefore, shared standards and criteria should be established among medical schools for residency placement.
1. Limitations
This study examined annual differences in prospective graduates’ perceptions of CRA using self-reported items. According to previous research, students in higher grades are more likely to respond positively to the overall school system; conversely, higher academic performance has been associated with a more positive perception of the school system [26,27]. However, this study did not control for confounding variables. Therefore, future studies should explore the positive and negative factors influencing the perceptions of CRA to provide insights for system improvement.
2. Conclusion
In conclusion, the changing perceptions of graduating students suggest that the implementation of the CRA system at YUCM aligns with the institution’s initial aims. The new curriculum, CDP2023, with its emphasis on research and bespoke advanced courses, is expected to further enhance the aimed CBME in YUCM, along with the CRA. A follow-up study examining the effects of the new curriculum within the CRA system and its influence on student outcomes should be considered.
Notes
Conflict of interest
Shinki An serves as an editor-in-chief of the Korean Medical Education Review, but has no role in the decision to publish this article. Hanna Jung serves as an assistant editor of the Korean Medical Education Review, but has no role in the decision to publish this article. Youkyoung Oh serves as a deputy assistant editor of the Korean Medical Education Review, but has no role in the decision to publish this article. Except for that, no potential conflict of interest relevant to this article was reported.
Authors’ contribution
Conceptualization: HJ, SA. Methodology: EL, YO. Data curation: EL, YO. Formal analysis: EL, YO. Investigation: EL, YO, HJ, SA. Writing–original draft: EL, YO. Writing–review & editing: EL, YO, HJ, SA. Project administration: EL. Validation: YO. Supervision: HJ, SA. Final approval of the version to be published: all authors.