Developing Content Validity

Practice Analysis to Test Content Outline

Forum, Volume 21, Number 1

What types of knowledge need to be covered on the National Physical Therapy Examination (NPTE) in order to determine if an individual is minimally competent to work as an entry-level PT or PTA? It's a key question that ultimately is answered through a process referred to as a practice analysis. A practice analysis is a study that systematically determines these responsibilities or "activities," and the knowledge and skill requirements (KSRs) for performing the activities. The results of a practice analysis are useful for informing decisions about the test content outline and for providing evidence of content validity.

Purpose of a Practice Analysis

• Verify current entry-level practice in the physical therapy profession;
• Determine the knowledge and skill requirements (KSRs) required to perform at an entry level; and
• Maintain the content validity of the NPTE by ensuring that the test content outline continues to measure entry-level knowledge and skills important for public protection.

The practice analysis conducted by the FSBPT is not intended to encompass the entire physical therapy profession or what it “should be.” It also is not intended to be a curriculum outline or synopsis of physical therapy education, an evaluation of advanced knowledge and skills for physical therapy, or to express an opinion or a position on physical therapy.

A Standard Approach

The practice analyses conducted by the Federation reflect best practices and meet criteria set forth in the Standards for Education and Psychological Testing developed by the American Education Research Association, the American Psychological Association and the National Council on Measurement in Education for defensible methods of establishing content validity in licensure and certification testing. The frequency with which a practice analysis should be conducted depends on how rapidly job requirements change; for the physical therapy profession, practice analyses are typically conducted every five years.

Previous and Future Practice Analyses

The Federation conducted a study in the United States and Canada in 1995-1996, with individual content outlines for PTs and PTAs in each country. The last study in the United States was completed in 2002. The Federation has just begun its 2006 Analysis of Practice of physical therapists and physical therapists assistants.

Overview of a Practice Analysis

Job responsibilities differ for PTs and PTAs. As a result, a different examination is developed for each occupation. The practice analyses for physical therapist and physical therapist assistant NPTEs also are parallel but separate processes. The Federation first develops and pilots surveys to obtain information on activities of entry-level PTs and PTAs and the knowledge and skill requirements (KSRs) for these activities. The pilot surveys are then revised and distributed to a nationally representative sample of PTs and PTAs. To the Federation, that means sending surveys to physical therapists and physical therapist assistants in all 53 licensure jurisdictions.

The survey data are analyzed to determine the set of activities and KSRs critical for entry-level practice. The results are used to update the PT and PTA NPTE test content outlines, which ensures that the tests continue to measure important information in the right proportions.

Phases of a Practice analysis

• Conduct a literature review of current physical therapy practice documents to develop a list of potential activities and KSRs for the surveys
• Develop and pilot surveys for entry-level PT and entry-level PTA activities and KSRs (knowledge and skill requirements)
• Develop final surveys based on data from pilot surveys
• Distribute final surveys to a nationally representative sample of PTs and PTAs
• Analyze survey data to determine critical activities
• Link critical activities to KSRs
• Use the findings to update the content outlines for the PT and PTA NPTE

The Oversight Panel and Task Forces

The Federation solicits nominations from member jurisdictions and professional physical therapy groups and sections in order to ensure the oversight panel and task forces represent the profession in terms of physical therapy practice settings, ethnicities, ages, lengths of practice and regions of the United States.

The Practice Analysis Oversight Panel is appointed to oversee the practice analysis process and task forces. Members of the panel are familiar with the NPTE development process, its content outlines and current practice issues in physical therapy. The panel reviews the pilot surveys, activity lists and KSRs, providing advice and guidance to the task forces and staff throughout the process.

Two task forces are appointed, one for the PT practice analysis and one for the PTA practice analysis. They produce the pilot and final surveys and use the survey results to build new content outlines for the PT and PTA examinations. The task forces participate in two assignments: (1) developing lists of job activities and knowledge and skill requirements; and (2) reviewing preliminary survey and linkage results. The results of those assignments are used to update the PT and PTA test content outlines.

Survey Development

Both pilot and final surveys ask for demographic and professional background information from the individual taking the survey. They include lists of activities and KRS, which will be rated on scales to assess their importance to entry-level practice. Typical scales include: 1) acquisition level, 2) consequence of incorrect performance, and 3) frequency of performance. The pilot survey also asks general questions about the survey itself such as "Did we forget an activity that was essential?" and "Was some activity not physical therapy?"

Acquisition Level: At what level of practice are the knowledge requirements and skills necessary to independently perform this activity typically required?

Consequence of Incorrect Performance: When considering the risk of unnecessary complications, impairment of function or serious distress to patients, how much physical or psychological harm will the incorrect performance of this activity most likely cause the patient?

Frequency of Performance: How often do you perform this activity?

Task force members review the pilot survey findings and revise the survey as suggested by the respondents. Revisions included clarifying confusing areas, condensing the survey and including the amount of time needed to complete the survey in the request to participate.

Survey Participation

The survey of activities is representative, meaning that the number of pilot and final survey participants are based on the total number of PTs and PTAs in jurisdictions. The ideal survey subjects have up to five years of experience so they are not too far from entry-level practice. They represent the same diversity as the profession in key areas such as gender, age, ethnicity, region and various clinical settings. All have passed the exam and are licensed or certified.

A sophisticated distribution plan is used to elicit as many responses as possible. For instance, an alert letter may be offered the survey on a web link. Non-respondents receive a follow-up letter with a paper survey. This communication may be followed up with a postcard, then a second follow-up letter with another paper survey.

The typical respondent to the last practice analysis survey (done in 2000) was female, white, licensed or certified between 1996 and 2000, full-time/salaried, working in direct patient care (especially ambulatory/outpatient and acute care) and more likely to report obtaining an MPT or MSPT credential.

Survey Analysis

Ratings from the activity survey are combined into a single index of criticality, with the entry-level activities having consequence for public protection receiving the most weight. Knowledges and skills required to perform these important entry-level activities are linked to the critical activities and then structured into a preliminary content outline for each exam. The task forces are reconvened to review and finalize the test content outlines. It should be noted that activities may be dropped from a content outline if they are part of advanced practice or too infrequently performed to warrant inclusion. Once the test content outline is finalized, new forms of the NPTE are assembled to meet the updated test content outlines.

A Status Report on the 2006 Practice Analysis

To date, the Practice Analysis Oversight Panel and task forces have been appointed. They have generated a list of activities and KSRs, which are currently being used to develop the pilot surveys. The pilot surveys will be sent in late spring this year and the national survey will be sent this summer. We expect to finalize test content outlines by early fall.