ABOM EXAM DEVELOPMENT

Exam items are written by members of the ABOM item writing committee. All members of the ABOM Item Writing Committee are physician Diplomates of the American Board of Obesity Medicine. Each member has undergone in-depth training on best practices in exam item writing delivered by doctoral level psychometricians and are guided by experienced item writers.

The members of the item writing committee are individually selected to ensure adequate representation and perspective from a wide spectrum of residency and fellowship training backgrounds, as well as practice types, practice settings, and practice locations throughout the country. Each item writer has demonstrated excellence in knowledge of obesity medicine and is actively practicing medicine.

The content of the assessment is determined by a table of specifications called a blueprint. Creation of the assessment blueprint is informed by job/task analyses that are repeated periodically, and validated by response from ABOM-certified physicians, considering the relative importance of each task, knowledge, skill or ability. The blueprint contains the target percentage of items on each exam that are mapped to each content domain and task. Each year, the ABOM exam is assembled according to the specifications outlined in the exam blueprint, ensuring that each exam measures the same depth and breadth of knowledge.

Exam items are in the form of multiple-choice questions with a single best answer, as there is literature to support that scores on multiple choice questions positively track with superior clinical performance. Most exam items include a clinical stem that grounds the item in clinical practice and reflects the clinical reasoning used in the daily practice of medicine. Some items may not include a clinical stem if not relevant to the concept or principle being addressed.

Every year, a gap analysis is performed to identify the areas in which new questions need to be written to ensure that the overall assessment can be assembled according to the assessment blueprint. Based on their clinical expertise, the item writing committee also identifies areas of scientific and medical advancement for which new questions may need to be written. Item writing assignments are then provided to item writers, who write new questions to address the concepts (mapped to the blueprint) that need to be assessed. Each item assesses knowledge or skill that the examinee would be expected to know without the assistance of other resources or references, with the goal of differentiating those that do have the knowledge necessary for certification.

After a question is written, it undergoes extensive evaluation by a convened group of physician subject matter experts trained in item writing best practices with support from a psychometric professional. If needed, edits are made to ensure each item adheres to all item writing best practices including fairness, clinical relevance, adequate scientific references, and inclusion of a clear single best answer choice.

ABOM works closely with psychometricians to assemble a new certification exam each year, ensuring adherence to the exam blueprint and completing further review of each item that is placed on the exam. Statistical methods are used to ensure that exam forms are comparable year to year. The ABOM exam is administered annually for new ABOM candidates as well as ABOM diplomates that chose to recertify via the exam recertification pathway.

The process of identifying a passing exam score is called standard setting. ABOM uses a well validated statistically sound approach (modified Angoff method) for standard setting, as appropriate. In this procedure, panels of representative ABOM-certified physician subject matter experts are guided in discussion and internalization of the ability of examinees that are near the threshold for the level of expertise needed for certification. The panelists then evaluate the difficulty of test questions and estimate the proportion of examinees close to the threshold that will correctly answer each item. As such, the passing score for the exam is identified. Year to year, statistical equating is used to ensure that the passing score remains appropriate. Of note, pass rates, or the number of examinees that pass the exam may change while a passing score stays constant.

Given that the field of obesity medicine advances quickly, ABOM completes a key validation process with the assistance of expert psychometricians after all candidates/recertifying diplomates have taken the exam and before scores are released to candidates. This process ensures that each test question remains accurate and valid with one single best answer choice, not affected by any interim medical field advances. Items that are identified as needing review by statistical or other methods are assessed by a key validation committee composed of trained physician subject matter experts. If an item is identified as no longer accurate, the item is removed from scoring and no longer used in the future.