Modified Angoff Method
The Physician Assistant Certification Council of Canada (PACCC) Test Committee uses the Modified Angoff Method of Standard Setting to set the standard or cut score for the Entry to practice PA certification exam (PACE).
The process involves subject matter experts (SMEs) estimating the likelihood that a minimally competent candidate would answer each test item (question) correctly.
Here is the step by step explanation of how the Modified Angoff Method typically works:
Selection of SMEs: A panel of SMEs is selected. These individuals are all certified PAs or Physicians who are members of the PACCC Test Committee. The SMEs work under the guidance of a Psychometrician.
Item Review: The SMEs review each exam item individually. They are asked to estimate the probability (typically on a percentage scale) that a minimally competent candidate would answer each item correctly.
Discussion and Consensus: After each SME independently rates each item, the Test Committee holds a group discussion. SMEs then justify their estimates and discuss any discrepancies in their ratings. The goal is to reach a consensus on the expected probably of the minimally competent candidate answering each item correctly.
Calculations of cut scores: Once consensus is reached on the probability for each item, these estimates are averaged across all SMEs for each item. The resulting percentage is the score expected to be achieved by the minimally competent candidate.
Setting the Standard: Finally, based on the calculated cut score, a decision is made on what constitutes the passing score or standard for that exam. The PACCC Test Committee recommends the Pass Mark to PACCC for approval after that year’s exam has been written.
Key considerations in the Modified Angoff Method include:
- Ensure that SMEs have a clear understanding of the what constitutes a minimally competent candidate.
- All estimates for each item is based on the content and difficulty of the items rather than on their personal opinions or biases.
The Modified Angoff Method is considered more reliable than simply setting cut score based on statistical methods alone, as it directly involves SMEs related to the content and context of the assessment.
The Modified Angoff Method is based on the content of the exam and not on group performance. There is no applied distribution scoring curve and it is feasible for all candidates writing the exam to be successful.