Purpose: The purpose of this study was to investigate how physician assistants (PAs) in major focused areas ofpractice perform on each of the National Commission on Certification of Physician Assistants (NCCPA)recertification examinations. In addition, the relationship between certification—the Physician AssistantNational Certifying Examination (PANCE) and recertification—the Physician Assistant National RecertifyingExamination (PANRE) and Pathway II—scores was assessed. Design: The sample consisted of the scores of3,402 PAs who took PANRE between 2000–2001 and 1,206 who took Pathway II between 1999–2001. InitialPANCE scores were also obtained for each member of the sample. Examinees’ focused areas of practice includedgeneral/family practice (53%), surgical (25%), emergency medicine (11%), internal medicine (8%), andpediatrics (3%). Results: Results indicated consistency in both PANRE and Pathway II performance acrossdifferent practice areas, even after controlling for length of time examinees had spent in their declared practicearea. Results also showed no statistically significant correlation between the average amount of time spent seeingpatients with a particular organ system condition and the number of questions answered correctly in that organsystem of the recertification exam. Finally, PANCE scores significantly correlated with PANRE scores (r2=.31)and to a lesser extent with Pathway II scores (r2=.07) for both general/family practice and surgical PApopulations. Conclusions: PAs across different major practice-focus areas of medicine perform similarly on therecertification examinations, regardless of how long they have spent in their practice area. PAs who on averagespend more time seeing patients with a specific organ system condition do not do better in that area of therecertification exam. Additionally, PAs who had higher scores on PANCE also tended to have higher scores onthe recertification exams, notably PANRE. The findings of this study are consistent with the training andpractice model adopted by NCCPA to describe the philosophy behind the PA recertification examinations.

2004 ¶ Perspective on Physician Assistant Education ¶ Volume 15, Number 1 47

NCCPA Spec ial I s sue

IntroductionThe National Commission on

Certification of Physician Assistants(NCCPA) recertification programs aredesigned to assess physician assistants’(PAs) general medical knowledge and skill

in primary care. However, data from the2000 NCCPA stakeholders’ study1 andconsiderable anecdotal evidence indicatethat there is a perceived need for somechanges to the NCCPA recertificationexamination and/or process. The impetusfor these changes was based on concerns ofthe growing number of PAs who aremoving away from general practice intofocused areas of practice. Their contentionis that the NCCPA recertification examina-tion and its test specifications are no longerappropriate for them because of its gener-alized content across areas of medicine. Inresponse, the NCCPA Board of Directorsbegan to explore additional or modifiedprocesses for recertification by taking intoaccount focused areas of PA practice.

As the NCCPA Board of Directorsexamined the recertification process, italso initiated an empirical researchprogram to support the validity of currentexamination scores. For instance, theimpact of the practice-focus transition onNCCPA recertification exam performancewas recently investigated by Hess andSubhiyah (in this issue).2 Using factoranalysis methods, they discovered that thecurrent structural design and scoringmodel for the Physician Assistant NationalRecertifying Examination (PANRE) wasconsistent across general/family practiceand surgical PAs. Only a single, generalability factor primarily explained variationin responses to questions for each contentblueprint dimension. It was concluded

Brian Hess, PhD; Raja Subhiyah, PhD

A Closer Look at Physician Assistants’Performance on the NCCPARecertification Examinations

(Perspective on Physician Assistant Education 2004;15(1):47-50)

Brian Hess was the manager of Test Development &Research, National Commission on Certification ofPhysician Assistants, Norcross, Georgia. Raja Subhiyahis the associate vice president, Scoring Services, NationalBoard of Medical Examiners, Philadelphia,Pennsylvania.

Correspondence should be addressed to:

Brian Hess, PhDAmerican Board of Internal Medicine510 Walnut Street, Suite 1700Philadelphia, PA 19106E-Mail: bhess@abim.org

that recertifying surgical PA examinees donot appear to be at a disadvantage whentaking PANRE, despite the fact that theexam measures PAs’ general medicalknowledge in primary care.

Whether or not the practice-focustransition across the PA profession hasplaced other major practice groups at adisadvantage when taking the NCCPArecertification exam has not been investi-gated. Thus, the purpose of the presentstudy was to take a closer look at how PAsin major focused areas of practiceperform on the current NCCPA recertifi-cation examinations. The relationshipbetween examinees’ PANCE scores andtheir recertification test (PANRE orPathway II) scores was also investigated.

Method

SampleThe data used in this study included

responses of 3,402 candidates who tookPANRE in 2000 and 2001 and 1,206candidates who took Pathway II in 1999,2000, and 2001. All candidates’ recordsrepresented first attempts to pass each testin their last recertification cycle. In addi-tion, PANCE scores (from PAs’ firstattempt to gain initial certification) werealso used for each subject. In this study, thefinal standard scale scores were reported;the standard scale used converted rawscores so that the mean was 500 and thestandard deviation was 100 (standard scalescores for the NCCPA examinationsranged from 200 to 800). These scores areequivalent across administrations andforms.3 Pass rates or percentages were alsoreported. The passing score for all examswas 350 and was based on previous stan-dard-setting procedures conducted byNCCPA and National Board of MedicalEducation (NBME) staff using a modifiedAngoff method and several separate expertpanelists.

Determining PA Practice FocusPAs in several different major focused

areas of practice were represented in thesample. Each examinee’s practice focuswas determined according to the specialtyreported category in longitudinal data

obtained from the American Academy ofPhysician Assistants (AAPA)’s PA censussurvey from 1991 to 2001. For this study,the length of time examinees spent in adeclared practice-focus area was definedas the number of past consecutive yearsreported in their practice focus at thetime of the recertification exam. PAswho had spent less than two consecu-tive years in their practice focus at thetime they took the recertification exam-ination were eliminated from the analy-sis. Additionally, only a few majorpractice-focus groups were able to bedetermined from the data set and thusare represented in this study. The samplesizes of various other medical and surgi-cal specialties/subspecialties were toosmall to extrapolate adequate statisticalcomparisons and inferences. Thus, themajor practice-focus areas representedwere general/family practice (53% ofthe sample), surgery (25%), emergencymedicine (11%), internal medicine(8%), and pediatrics (3%). Finally,information from the NCCPA practicesurvey was used to determine whetherPAs who, on average, spend most oftheir time seeing patients with certainorgan system conditions obtain higherscores in that organ system area of therecertification exam.

Data AnalysisAn analysis of covariance (ANCOVA)

procedure was used to assess group differ-ences in mean scores. Subsequently, a χ2

test was used to compare pass rates acrossgroups. The overall level of statisticalsignificance was set at P<.05. However,because large sample sizes were used inthis study, we reported and interpretedmeasures of effect size. Similarly, whenwe reported correlations, we alsoreported and interpreted the squaredcorrelation coefficient (r2) effect size,which represents the percentage of vari-ance shared between the two variables.

Results

PANRE PerformanceAn ANCOVA was conducted with

mean PANRE score as the dependent

variable, type of practice focus as theindependent variable, and length of timein a practice focus as a covariate. Resultsshowed that the length of time in practicefocus, while statistically significant, didnot account for a meaningful amount ofvariance in PANRE scores. The effect sizeη2 value of .002 indicates that only 0.2 %of the variance in PANRE scores wasexplained by length of time in practice.However, we retained this variable as acovariate in order to compare practice-focus mean scores, controlling for lengthof time spent in a practice focus.

Results indicated a statistically signifi-cant difference in mean scores betweenthe major practice-focus groups, F(4,3397)=72.81, P<.05, η2=.07. However,the effect size indicated that only 7% ofthe variance in PANRE scores is attrib-uted to differences in type of practicefocus, after controlling for length of timespent in that practice focus. While the dataindicated that surgical PAs on averagescored lower than the other practice-focusgroups, the size of the difference wassmall. When comparing pass rates acrossthe practice-focus areas, results of a χ2 testindicated a statistically significant differ-ence in pass rates, χ2

(4, N=3,402)=36.23,P<.05, rc=.09. However, the contin-gency coefficient effect size (rc) value of.09 indicated that only 9% of the variabil-ity in pass rates is attributed to differencesin the type of practice focus. Table 1presents the overall PANRE performanceof PAs in major focused areas of clinicalpractice.

48 2004 ¶ Perspective on Physician Assistant Education ¶ Volume 15, Number 1

A Closer Look at Physician Assistants’ Performance on the NCCPA Recertification Examinations

Table 1

PANRE Performance of PAs in Major FocusedAreas of Practice

Practice area N M (SD) Pass %

General/Family 1947 533 (81) 98.7

Surgical 760 481 (81) 95.4

Emerg Med 331 529 (79) 99.1

Internal Med 292 551 (81) 99.3

Pediatrics 72 528 (74) 98.6

Total 3402 522 (80) 98.1Note: Group means are adjusted after controlling forlength of time examinees spent in their practice focus.

In summary, there were very minordifferences in PANRE scores across thedifferent practice-focus areas, control-ling for length of time spent in a prac-tice focus. Pass rates for all groups wereat or above 95%.

Pathway II PerformanceResults of an ANCOVA similar to

the one conducted for PANRE indi-cated no statistically significant differ-ences in mean scores among thedifferent practice-focus groups aftercontrolling for length of time spent ina practice focus (P>.05). Whencomparing pass rates, results of a χ2

test indicated no statistically significantdifference in pass rates (P>.05) amongPAs in different practice-focus groups.In summary, Pathway II performancewas consistent across the differentpractice-focus areas after controllingfor length of time spent in a practice-focus area. Table 2 presents the overallPathway II performance of PAs infocused areas of clinical practice.

Relationship Between PANCE andRecertification Exam Scores

Table 3 shows the correlation ofPANCE scores with PANRE andPathway II scores. As expected, PANCEscores (first attempt) predicted perfor-mance on PANRE better (r=.56) thanfor Pathway II (r=.27). Both PANCEand PANRE are proctored tests, whilePathway II is a take-at-home test that, by

design, measures different skills andconcepts. Squared correlations (r2),representing the amount of variability inPANRE scores explained by PANCEscores, was 31%. In anticipated contrast,the Pathway II score variance explainedby PANCE scores was only 7%.Correlations of PANCE scores withPANRE scores were the same forgeneral/family practice PA and surgicalPA examinees.

Interestingly, surgical PAs who tookPANRE also tended to score slightlylower on their first attempt at PANCE(for initial certification), as shown inTable 4. However, as with PANRE, thisdifference was not meaningful.

Organ System PerformanceCorrelations between the average

amount of time spent seeing patientswith a particular organ system conditionand number of questions answeredcorrectly in that organ system of theexam were calculated. Results showed nostatistically significant correlations forany of the 13 organ systems (correlationswere all less than .04 for all the organsystems). These results were the same forPANRE and Pathway II examinees.

DiscussionThe purpose of the present study was

to take a closer look at how PAs in differ-ent focused areas of practice performedon each of the current NCCPA recertifi-cation examinations. The relationshipbetween PANCE and recertificationscores was also investigated.

Very small differences were found inPANRE scores across major practice-focus areas; however, these differenceswere not meaningful, as they accounted

for only a small portion of the variancein PANRE scores. No significant differ-ences were found across these groups onPathway II. No differences were foundacross length of time spent in theirdeclared practice area in either PANREor Pathway II. Finally, pass rates acrossall groups represented in this study werewell above 90%. This suggests that PAsacross major practice-focus areas ofmedicine generally perform well on theNCCPA recertification examinations—regardless of how long PAs spend intheir practice-focus area.

While concern has been raised bysome PAs in specialty areas about theircontinued ability to pass a generalistexamination, it does not appear that themajor practice-focus groups representedare at risk for lower performance on theNCCPA recertification examinations.Further, these results are consistent withHess and Subhiyah2 in that the perfor-mance of surgical PAs on PANRE doesnot indicate that they are at a disadvan-tage when taking the PANRE. Moreover,our data are consistent with the trainingand practice model adopted by NCCPAto describe the philosophy behind the PArecertification exams.

2004 ¶ Perspective on Physician Assistant Education ¶ Volume 15, Number 1 49

A Closer Look at Physician Assistants’ Performance on the NCCPA Recertification Examinations

Table 2

Pathway II Performance of PAs in MajorFocused Areas of Practice

Practice area N M (SD) Pass %

General/Family 475 531 (85) 97.7

Surgical 406 524 (90) 94.4

Emerg Med 168 532 (92) 95.2

Internal Med 84 540 (99) 97.6

Pediatrics 73 518 (93) 94.5

Total 1206 529 (89) 96.0Note: Pathway II data is aggregated across the 1999through 2001 administrations. Group means are adjustedafter controlling for length of time examinees spent intheir practice focus.

Table 3

Correlation Between PANCE and NCCPA Recertification Examination Scores

PANCE w/PANRE PANCE w/Pathway IIGroup N r N r

General 1947 .56 438 .31

Surgical 760 .56 374 .21

Total 2707 .56 812 .27Note: Data is from examinees’ first attempt to pass each exam.

Table 4

PANCE Performance of General/FamilyPractice and Surgical PA Examinees

PANCE

Specialty N M (SD) Pass %

General 1947 527 (75) 92.9

Surgical 760 515 (76) 90.1

Total 2707 523 (76) 92.1

Note: Data are from PANRE examinees only.

50 2004 ¶ Perspective on Physician Assistant Education ¶ Volume 15, Number 1

A Closer Look at Physician Assistants’ Performance on the NCCPA Recertification Examinations

Similarly, when PAs did well overallon the recertification exams, theyperformed well across all organ systemareas of the exam; when PAs did notperform well overall, they did notperform well across all organ systems. Infact, PAs who reported (at the time ofrecertification) spending on averagemore time seeing patients with a partic-ular organ system condition did not tendto correctly answer more questions inthat organ system area on the exam. Thismay suggest that the NCCPA recertifica-tion (or indeed, the certification) examsare not specialized enough to pick up onthe differences between PAs who workmost often in one or another area. Inother words, the NCCPA examinationsmeasure core knowledge that is sharedby these areas. This is consistent withthe NCCPA’s philosophy that theseexams are constructed for the generalpractitioner. They are not targeted to theeclectic specialist. Thus, this finding isconsistent with the PA “generalist” train-ing and practice model and lendssupport to the validity of the NCCPArecertification examination scores.

Finally, the results of this study indi-cated that PAs who scored higher onPANCE also tended to score higher onthe recertification exams, particularlyPANRE. These correlations were consis-tent across general/family practice andsurgical PAs. This suggests that PANCEscores have respectable predictive valid-ity vis-à-vis PANRE scores. The amountof shared variance between PANCE andPANRE scores (31%) may be attributedto the similarity of the two tests informat, method, and content. This isexpected as most certification examscores across the professions typicallycorrelate fairly highly with recertificationscores when the same methods are used.Conversely, because the Pathway II is atake-at-home test, it measures (bydesign) a largely different set of skills ina largely different operating environ-

ment. Hence, weaker correlations areexpected and duly obtained.

The predictive potential of PANCEof recertification exam scores maypartially explain why surgical PAs tendedto score on average a bit lower on boththe NCCPA certification and recertifica-tion exams. While these differences werevery small, this pattern suggests thatsomething other than the structure ofthe NCCPA examinations themselvesmay be contributing to their slightlylower mean scores.2

It may be of value to investigate howPAs in different practice-focus areasprepare for the NCCPA recertificationexams. For example, the leadership ofthe AAPA Surgical Congress has previ-ously expressed the need for a substan-tial review of generalist content by theirconstituents in order to prepare for theNCCPA examinations. They suggestedthat continuing medical education(CME) should be directed at this goal(in addition to activities that target theirfocused area of practice, which typicallyresult in the accumulation of an excess ofthe hours needed for certification main-tenance). This type of preparation mayexplain the consistency in performancebetween groups seen in the presentstudy. Furthermore, future studies couldalso examine the type and amount ofCME undertaken to delineate the impacton recertification performance. Otherpreparation characteristics, such as self-study, might be examined as well.

LimitationsNot every medical specialty/subspe-

cialty was uniquely represented. Forexample, surgical PA examinees in thisstudy were identified via AAPA surveydata as any PA practicing in any one ormore surgical specialty or subspecialty.PAs were not identified as belonging to avery specific surgical area such as ortho-pedics, cardiovascular surgery, etc. Thismay pose a limitation in that uniqueness

of any one surgical specialty or subspe-cialty could not be investigated.However, because these specialized areasof surgery share a great deal in common,it made sense for the purpose of thisstudy to identify and include one overallsurgical category. Finally, we definedlength of time spent in a practice-focusarea as the consecutive number of yearsPAs worked in their declared practicearea at the time of recertification. Thismay be confounded by the mobilitywithin the PA profession (ie, movingback to one practice focus after severalchanges may not accurately reflect theexperience accumulated over one’sprofessional career).

ConclusionPAs across major practice-focus

areas of medicine perform consistentlywell on the NCCPA recertificationexaminations, regardless of the lengthof time they spent in their practicefocus. PAs who on average spend moretime seeing patients with a specificorgan system condition do not performbetter in that area of the NCCPA recer-tification tests. Additionally, PAs whoperform higher on PANCE also tend toscore higher on the PANRE. The find-ings obtained in this study are consis-tent with the training and practicemodel adopted by NCCPA to describethe philosophy behind the PA recertifi-cation examinations.

References1. National Commission on Certification of

Physician Assistants. Assessment of the Currentand Future Uses and Value of NCCPACertification. NCCPA: Norcross, Ga; 2000.

2. Hess B, Subhiyah R. Confirmatory factoranalysis of the NCCPA Physician AssistantNational Recertifying Examination. Perspectiveon Physician Assistant Education. 2004;15:55-61.

3. Subhiyah R, Hess B, Arbet S. Building acertification examination program, Part II:Scoring and standard setting. Perspective onPhysician Assistant Education. 2004;15:21-24.