TY - JOUR
T1 - Using an Enhancement-Focused Framework Model to Improve the Quality of High-Stakes Assessments
AU - Shaban, Sami
AU - Regmi, Dybesh
AU - AlMahmoud, Tahra
AU - El Zubeir, Margaret Ann
AU - Branicki, Frank
AU - Govender, Romana
N1 - Publisher Copyright:
© Common Ground Research Networks, Sami Shaban, Dybesh Regmi, Tahra AlMahmoud, Margaret ElZubier, Rank Branicki, Romona Covender. All Rights Reserved.
PY - 2020/6/30
Y1 - 2020/6/30
N2 - Producing a robust multiple-choice question examination (MCQ) for high-stakes assessment requires significant time and effort to ensure reliability and validity. The enhancement-focused framework recommends stepwise processes to produce a high-quality assessment. The processes, consisting of blueprinting, vetting, standard setting, external review, and student orientation, were used to develop and administer the final year medical school multiple-choice question examination. Item analysis was performed, question psychometrics were interpreted, and the locally developed examination was compared with the external International Foundations of Medicine Clinical Science Examination. The examination cut score was 55.08 percent and the mean score was 62.04 percent (SD ±8.27). Internal consistency (Cronbach’s Alpha Reliability measure) was 0.759. 57 percent of questions had a discriminatory index > 0.2 and more than 85 percent had a difficulty index > 0.30. Comparison of the locally developed MCQ examination with the external examination resulted in a Pearson’s correlation coefficient of 0.81 validating the locally developed MCQ examination as a high-quality assessment tool. Despite availability of multiple contemporary assessment methods, MCQs remain a popular, valid, and reliable summative assessment tool in higher education. The enhancement-focused framework provides a strong foundation to develop a high-stake MCQ examination. However, continued rigorous scrutiny of all test items and the refinement of the individual processes such as blueprinting, vetting, standard setting, and external review is imperative to enhance their quality, and this process is a continuous cycle enriching locally produced question banks.
AB - Producing a robust multiple-choice question examination (MCQ) for high-stakes assessment requires significant time and effort to ensure reliability and validity. The enhancement-focused framework recommends stepwise processes to produce a high-quality assessment. The processes, consisting of blueprinting, vetting, standard setting, external review, and student orientation, were used to develop and administer the final year medical school multiple-choice question examination. Item analysis was performed, question psychometrics were interpreted, and the locally developed examination was compared with the external International Foundations of Medicine Clinical Science Examination. The examination cut score was 55.08 percent and the mean score was 62.04 percent (SD ±8.27). Internal consistency (Cronbach’s Alpha Reliability measure) was 0.759. 57 percent of questions had a discriminatory index > 0.2 and more than 85 percent had a difficulty index > 0.30. Comparison of the locally developed MCQ examination with the external examination resulted in a Pearson’s correlation coefficient of 0.81 validating the locally developed MCQ examination as a high-quality assessment tool. Despite availability of multiple contemporary assessment methods, MCQs remain a popular, valid, and reliable summative assessment tool in higher education. The enhancement-focused framework provides a strong foundation to develop a high-stake MCQ examination. However, continued rigorous scrutiny of all test items and the refinement of the individual processes such as blueprinting, vetting, standard setting, and external review is imperative to enhance their quality, and this process is a continuous cycle enriching locally produced question banks.
KW - Assessment
KW - Blueprinting
KW - High-Stakes Examination
KW - Multiple-Choice Questions
KW - Standard Setting
UR - http://www.scopus.com/inward/record.url?scp=85212864258&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85212864258&partnerID=8YFLogxK
U2 - 10.18848/2327-7920/CGP/v27i01/27-43
DO - 10.18848/2327-7920/CGP/v27i01/27-43
M3 - Article
AN - SCOPUS:85212864258
SN - 2327-7920
VL - 27
SP - 27
EP - 43
JO - International Journal of Assessment and Evaluation
JF - International Journal of Assessment and Evaluation
IS - 1
ER -