The COVID-19 pandemic presented unique challenges for licensure exams. This article is based on a 2020 Annual Meeting presentation by Lorin Mueller and Kimberly Swygert.
The COVID-19 pandemic has affected every part of our lives, and high-stakes licensure exams are no exception. The Federation of State Boards of Physical Therapy and the National Board of Medical Examiners detail how they met these new challenges and what may lie ahead.
In mid-March, it became clear to FSBPT that they needed to shut down the April testing window. This resulted in an unprecedented testing disruption. There were 5,922 displaced candidates. For context, over the last eight years, in total, FSBPT had forty-three candidates displaced in the month of April. FSBPT determined early on that the focus should be on working with Prometric to get candidates back into centers when that was feasible. Every day FSBPT went through a similar cycle that involved talking with Prometric, gathering information, assessing gaps, getting feedback, and providing accurate information to candidates. As the information was constantly shifting, this cycle continued repeatedly.
In terms of communications, FSBPT needed to balance speed with accuracy. FSBPT was constantly evaluating the benefits of distributing information promptly with the risks that that information may change, causing a candidate to go down the wrong path.
Logistically, under normal circumstances, registration between Prometric and FSBPT is a well-oiled machine. However, many of those systems were not set up to deal with this type of disruption. FSBPT and Prometric’s IT teams worked to address these issues as expeditiously as possible. As with most organizations, FSBPT staff also needed to transition swiftly to remote work, including the call center.
Test center openings were constantly shifting. There were myriad components involved in opening a center, ranging from state and local rules to center-specific issues such as staffing, staff training, personal protective equipment, and social-distancing setups and processes. Center-specific demand also shifted as students dispersed away from campuses back to their hometowns.
To respond to this evolving situation, FSBPT increased communications via multiple channels: emails, the website, social media, and webinars. FSBPT was committed to ensuring its stakeholders were in the loop. FSBPT also frequently talked with similar health care testing organizations; international counterparts; and our partners, APTA, CAPTE, ACAPT, PTA SIG, and ELP.
While FSBPT was very proactive, there were still pain points for candidates and stakeholders. Prometric displaced more than 450,000 customers over two months. As they juggled the moving parts of this pandemic, they needed to consider all of their partners—not just FSBPT—along with safety issues and jurisdictional laws. They decided to emphasize essential services programs in May—however, initially, some of the messaging around that was confusing. Additionally, even as they decided to re-open, there were multiple reasons a center or an appointment could be canceled, which caused additional confusion for candidates.
For example, there was confusion over which centers Prometric opened as the list on the website was not always up to date. Even open centers could not open at full capacity as they needed to recognize social distancing standards and jurisdictional limits on gatherings. Prometric canceled many appointments at open centers. After initial cancellations, they conducted an audit, which resulted in late cancellations. Finally, some cancellation notices from Prometric had erroneous or outdated information that misinformed NPTE candidates.
Prometric has been an FSBPT partner for more than twenty years. Surveys show that candidate satisfaction for Prometric is usually very high. Additionally, they came to solutions that addressed FSBPT’s needs relatively quickly, given the circumstances.
Fortunately, the situation improved. FSBPT decided early on to open a testing window instead of fixed dates, as they usually do. Spaces opened up and 99.99% of April exam candidates who decided to take the exam in May were able to complete the exam in May. Some candidates decided to wait until July and the vast majority of those candidates were also able to test. In July, FSBPT also opened a window while strongly encouraging candidates to test on the fixed date if possible. Fortunately, the July exams went much more smoothly than the May window. Both FSBPT and Prometric were able to work out many of the kinks that this new COVID-19 era introduced. Candidates have backed this up with much more positive evaluation responses.
As always, FSBPT is constantly concerned about test security. FSBPT used Caveon Analysis to keep an eye on test security issues. Fortunately, the windows do not seem to be introducing new security issues.
The pandemic has also affected exam development. FSBPT canceled all its in-person 2020 meetings, including Exam Development Committee meetings and Item Writer Workshops. FSBPT had to work quickly to move these activities online in a secure way to continue exam development activities.
Volunteers have provided feedback on these changes. Their most frequent comment is that they miss being onsite in Alexandria, Virginia, and the networking opportunities that in-person meetings provide. Additionally, a long virtual meeting can feel like a slog in a way that weekend-long in-person meetings do not. Volunteers also have some issues working from home, such as distractions and lack of resources (e.g., textbooks). However, volunteers also expressed that there are benefits to joining the meetings and workshops while staying home, and they are glad to still participate and volunteer.
FSBPT responded to this volunteer feedback by supplying better access to electronic resources and additional equipment and software when appropriate. FSBPT also redesigned agendas to make them more conducive to virtual formats. FSBPT may leverage virtual meetings even once it is safe to meet in person, but they will not completely move away from in-person meetings as those provide a strong value to volunteers and the overall process.
FSBPT is also working on transitioning Prometric-based Law exams to JAMS that can be taken anywhere to have less reliance on in-person testing for that type of assessment. FSBPT will continue to have some flexibility when it comes to fixed dates as needed.
The National Board of Medical Examiners also had unprecedented disruptions to testing. They also dealt with Prometric centers closing, as well as challenges to test security and stakeholder communication. However, the pandemic also affected other aspects of testing. For example, exams that were already set up for delivery at medical schools were impacted by the shutdown of those schools. NMBE also had to move their staff to full-time remote work very quickly and rescheduled in-person committee meetings to be remote meetings.
NMBE provides several different exam programs. The USMLE series involves licensure testing. USMLE Step 1, Step 2 CK, and Step 3 are normally administered at Prometric centers, while the USMLE Step 2 CS is administered as an in-person standardized patient performance assessment at five test centers across the United States. Subject Examinations are normally administered via the web within medical school computer labs with in-person proctoring. For a short time, the administration of all of these exams ceased.
NBME immediately prioritized their tasks. The most pressing issue was to develop a new process for the Subject Exam administrations. Students were no longer on campus, but there was still a strong need to offer these exams. For this, NBME moved to a proctoring-by-web-conference scenario with a Zoom or Microsoft Teams conference where a medical faculty member was proctoring. The reason this was feasible is because NMBE already had a strong relationship with schools and faculty. Many elements were already in place, and the goal was “service over psychometrics.” While this does not mean that psychometrics was not still highly valued, NBME knew it needed to pivot quickly to get these exams running.
NBME also wanted to quickly develop multiple options for the USMLE administrations that could operate alongside or in place of Prometric administration. This made communications more challenging as it created more pathways. NBME worked with medical schools that could provide socially distanced regional testing centers.
NBME had to make a quick decision about Step 2 CS, as that exam required in-person tasks. NBME decided to suspend this exam until no sooner than June 1, 2021. However, restarting this exam is not as simple as just bringing it back once the pandemic ends. With the pandemic, there have also been changes in telehealth. As the profession changes, the assessment tools used to regulate the profession must also change. Therefore, NBME is in the process of re-evaluating what to measure in an observational physician licensure exam and investigating the technologies required to support this type of assessment.
NBME also needed to acknowledge examinees’ situations with new policies and public announcements, and actively set expectations concerning the assessments before any changes. Examinee communications became even more crucial as new steps evolved and different routes became available.
NMBE expanded the FAQs on the website to help candidates. They also adjusted policies to address the situation, such as extending automatic eligibility extensions. Staff and volunteers also shifted rapidly to the new remote environment and have done a phenomenal job with this transition.
Accessibility was a challenging issue before. COVID-19 has only acerbated accessibility now that testing is often home-based. Social media is absolutely amplifying existing criticisms. Sitting back and waiting for stakeholder criticisms to arrive via tweets is not the smart move. Moving forward, NBME will continue having strong communication with candidates and other stakeholders. They will also re-enforce why they administer tests and why these tests have value. As others have brought up, if NBME can get by without having an exam temporarily, some will question why we need these exams at all.
Some states have made decisions on legal bar exams ranging from requiring remote bar exams (California) to allowing new graduates to practice without the exam (Louisiana). The Medical Council of Canada moved to remotely proctoring their initial qualifying exam and has had success with that. However, the American Board of Surgery’s experience with remote proctoring was less successful. The Federation of State Medical Boards is providing a map of COVID-19–related legislation efforts that affect physicians, ranging the gamut from telemedicine, to fee waivers for licenses, to license reciprocity, to allowing continued practice by residents without a licensing exam if an exam is not available. Test security, as always, is a pervasive issue. NBME, as an organization, may have to consider a permanent shift to types of assessment administrations that are known to have more security challenges than previous methods. Therefore, NBME may need to invest in new data forensic methodologies, updated policies, and advanced tracking methods (e.g., dashboards for evaluation clusters of performance). It may also be necessary to create more exam content by greater leveraging of cognitive modeling and artificial intelligence tools. The old ways of developing content may not hold in this new normal.
The world is rapidly changing to respond to COVID-19, and some of those changes may be lasting and may affect the message about the value of assessment for practice in various fields. All assessment teams need to continually re-evaluate what they measure and why they measure it.
Lorin Mueller, PhD, is the Managing Director of Assessment at the Federation of State Boards of Physical Therapy. He has over twenty years of experience in assessment development and behavioral research. He holds a doctorate in Industrial and Organizational Psychology from the University of Houston.
Kimberly Swygert, PhD, FCPP, has over twenty years of experience as a psychometrician. She is Director, Test Development Innovations, at the National Board of Medical Examiners, where she oversees the teams responsible for NBME’s test construction, multimedia, and innovations in test development. She has extensive research and applied experience and publications across multiple domains, including test scoring, test development, innovative item development, test construction, and test security. She is currently expanding her skill set in natural language processing, machine learning, and application development. She received her doctorate in quantitative psychology from the University of North Carolina at Chapel Hill.