8

Standards Engineering

January/February 2014

BackgroundThere was a time when a person would

learn a skill that would prepare him or her for a lifetime of work. These skills could be learned through a guild or apprenticeship, by working for a master, at a vocational school, in a private training facility, or on the job. The worker would then enter the world of work and would spend their entire lifetime working on that job. Similarly, there was a time when a professional studied (either at a university or other institution of learning) and then entered the profession for an entire career of practice in that field. The consumers went to these persons with the confidence that, by virtue of the title issued to them— blacksmith, car-penter, doctor, accountant, etc.— they were competent and would do a good job.

Those days are gone. Today both work-ers and professionals enter jobs through many different avenues (apprenticeship, on- the- job training, education, etc.); some are legitimate, some are not. And to complicate things, the odds are that the person won’t stay in that job or profession throughout his or her lifetime. The United States Depart-ment of Labor, Bureau of Labor Statistics released a report in 2012 that showed that “The average person born in the latter years of the baby boom (1957- 1964) held 11.3 jobs from age 18 to age 46”.1 A significant number of the job changes are also career changes which will require the person to learn a new skill or acquire new knowledge to do a different kind of job or profession.

Additionally, the globalization of the workforce has created an environment where workers and professionals who have the same background on paper (university degree, ex-perience, etc.) may actually not have equiva-lent qualifications. The medical degree in a particular specialty issued by a university in one country may or may not be equivalent to the medical degree in the same specialty is-sued by a university in another country. Or the

Verifying the Competence of Personsby Cynthia Woodley, Ed.D

The International Organization for Standardization (ISO) published a new conformity assessment standard related to certification of persons programs (ISO/IEC 17024: Conformity Assessment – General requirements for bodies operating the

certification of persons) in July of 2012. Recently gaining the attention of many Technical Committees and Working Groups, this standard is being used as the basis for various sectors and industries to develop their own schemes for certification bodies for persons.

By following the requirements of ISO/IEC 17024, a personnel certification body can better create quality certification programs.

five years of experience working as an elec-trician in one country may or may not mean the same thing as the five years of experience working as an electrician in another country (particular with differing electrical systems).

With so many different avenues for entering an occupation or profession, so many differences between qualifications, and so many job changes throughout persons’ careers, how does the employer know that the person is competent to practice? Moreover, how does the public know they are hiring a competent person? Verifying the competence of persons is becoming a very important activity as the world labor market changes.

One attempt to provide some informa-tion regarding the competence of a worker or professional is by the development of cre-dentials. Recently the number of credentials has significantly increased. The proliferation of credentials has led to a market where credentials can be very confusing to many employers, consumers, and even other pro-fessionals. And these days, some of them do nothing to ensure the competence of the person and are not legitimate.

A credential is an attestation by a body (governmental agency, certification body, school, etc.) that says something about the person who holds it. Credentials can be is-sued by a second party body such as a school or university or they can be issued by a third party body who has evaluated the compe-tence of the individual before issuing the credential. The most common credentials are licenses, certifications, and certificates. Each of these will be further discussed.

License to PracticeA license is a credential issued by a

regulatory body (federal, state, or local government) to an individual that allows the individual to practice a profession or occupation. Physicians, nurses, social work-ers, nail technicians, and real estate brokers,

among others, are required to be licensed in order to practice in their field. The license is required and one may not engage in the profession or occupation without the license. Licenses by definition are designed to pro-tect the public so that someone not qualified or competent to practice does not enter the profession or occupation. This is analogous to a driving license. One may not drive a car until one has obtained a driver’s license and the purpose of that license is to ensure that the driver is competent enough to not be a danger to others on the road.

In licensure, common requirements can include graduation from an appropriate educational/training program, experience, a criminal background check, and an assess-ment of competence through an examina-tion. Until the governmental regulatory body is convinced that the person has the knowledge and skills to be a competent worker and not harm the public, that person may not work in that profession.

Professional CertificationA certification program is a program

operated by a third- party certification body for persons. Certification bodies award certi-fication to individuals who demonstrate that they have met the criteria (the standard) for the certification. These criteria/standards of-ten include prerequisites such as educational requirements and experience requirements but always must include an assessment or examination of the knowledge and skills required for competent performance in the profession or occupation. Certification differs from licensure in the following ways:• certification is usually controlled by the

profession itself, not a governmental regulatory body;

• certification is usually not mandatory to practice;

• certification may be at different levels (it may be at the entry level similar to

January/February 2014 9

Standards Engineering

(Continued on page 10)

licensure, but often it is at an advanced level indicating a higher level of exper-tise and knowledge).

For example, a licensed physician might also choose to become a Board Cer-tified Surgeon. The board certification is awarded to the physician by a professional organization such as the American Board of Surgery after the physician demonstrates advanced knowledge and skills in surgery. This demonstration includes meeting the prerequisite requirements: education in surgery, experience as a surgeon, and pass-ing several examinations that measure the surgeon’s knowledge and skills associated with surgery. The physician did not have to obtain the Board certification to practice as a surgeon (it was not mandatory to practice) but by obtaining it, he or she communicates to the public that he or she has a higher level of expertise in surgery than the physician who does not have a Board certification.

Licensure and certification have com-mon elements. Both are based on an as-sessment of candidate competence by a governing body (either a governmental body in the case of licensure, or a certification body in the case of certification). Both are designed to communicate information to the consumer, the public, or the employer about the competence of the certified person.

Certificate ProgramsA certificate program is a credentialing

program awarded as part of a training or edu-cational program. An individual who com-pletes the training program and accomplishes the intended learning outcomes is awarded a certificate. Certificates are generally awarded for life and cannot be taken away. They say nothing about the continued competence of the person, or even the initial competence of the person, just that the person success-fully completed the program and learned the content presented in the program.

Certificates are valuable credentials that indicate the professional has attended an educational or training institution and com-pleted the requirements. As a consumer, this tells you that the individual was educated or trained. In some circumstances, this is very important. We want our teachers and dentists to have attended a program that taught them how to be a teacher or dentist. Certificate

programs also serve very important roles for the professions as they are often pre-requisites for certification programs. For example, a dentist would not be allowed to sit for the dental examination without first providing evidence that he or she graduated from a dental program or a medical doctor would not be allowed to sit for the medical license exam without first providing evi-dence of completion of a medical program.

The weakness in certificate programs is that there may or may not be an assessment or examination to verify learning. It is possible to progress through some programs merely by attendance and to obtain the certificate with-out any real learning. And the other weakness is that if an examination is given during the certificate program, it is usually based on the course content and is designed to verify that the candidate learned the content during the course. This may or may not specifically relate to the job the person is going to do.

Certificates versus CertificationsIn today’s environment, we value

graduation from a certificate or diploma program; but we often do not trust that the certificate or diploma program alone was sufficient to ensure competence of the pro-fessional. Remember that “seat time” in a course or training program or completion of a certificate program is no guarantee that the worker has actually learned anything related to competence on a job. While there is value in learning, it is really the outcomes of that learning (the obtained knowledge and skills) that are of greatest interest to employers and the consumers. Licensing and certification are competency- based indicators that ensure that the candidate has acquired the requisite knowledge and skills.

That is why in most professions where public harm (both financial and physical) may occur, there is often a license or certi-fication requirement after graduation from the education/training program before the professional is allowed to practice. Thus the dentist, physician, pilot, insurance salesper-son, cosmetologist, building contractor, and teacher may have graduated from a great educational or training program, but we still require that they pass an independent examination that measures their knowledge and skills to be competent before we let them practice on the unsuspecting public.

It is important to distinguish certification programs from certificate programs. Unlike certificates, certification cannot be awarded for life. A certification must be renewed to remain valid. Most licenses and almost all certifications require renewals, meaning that the license or certification expires after some number of years. The worker/professional must then provide evidence that he or she continues to have the knowledge and skills to practice competently. Evidence may include continuing education, continued practice in the field, or re- examination.

Quality certification programs are re-ferred to as third- party assessments of the knowledge and skills required for competent performance because they are conducted by bodies that do not have a stake in the creden-tialing outcome of the candidate (for example, they do not employ them or train them). Training a candidate and then certifying that candidate is considered a conflict of interest. While it is possible for a certification body to also provide training, when these two activi-ties are housed within one body, they must be separated physically, administratively, and financially to ensure that the certification program remains confidential and impartial.

Ideally, certifications are administered by neutral third- party certification bodies that have not trained the candidate or other-wise have an interest in the outcome of the certification. Unlike certification programs, certificate programs are not independent of the instruction and therefore not a neutral, third party assessment of competence. Most certificate program examinations are teacher/instructor made examinations and therefore can be influenced by the teacher/instructor.

ISO/IEC 17024With so many different certification

programs out there, a need to develop a stan-dard to address the requirements for bodies operating certification of persons emerged. ISO/IEC 17024: Conformity Assessment – General requirements for bodies operat-ing the certification of persons was revised and published in July of 2012. The original version was published in 2003 and was based on a European Standard, EN 45013. ISO/IEC 17024 is a conformity assessment standard, meaning that it is designed to be used to verify that a body is operating in conformance with its requirements. This can

10

Standards Engineering

January/February 2014

be done in a number of ways, but the highest confidence will be given to a certification of-fered by a certification body for persons that has been accredited by an accrediting body such as the American National Standards Institute (ANSI) as meeting ISO/IEC 17024. Just as we would trust the competence of a worker who was certified by a neutral third party certification body, we would trust a certification body which was accredited by a neutral third party accrediting body.

ISO/IEC 17024 was developed through a consensus process by an ISO CASCO work-ing group representing a number of countries and interests (accrediting bodies, certification bodies, industries, etc.) Since its publication, ISO/IEC 17024 has quickly become one of the more well known and used ISO confor-mity assessment standards. ISO/IEC 17024 contains a number of requirements that help to ensure that a certification body for persons develops and maintains certification programs that provide value to the public, employers, and other interested parties. The standard is organized into the following components which should be the major functional areas of a certification body for persons:• Structural requirements (organizational

structure and separation of any per-sonnel certification body activities in training candidates from certification activities);

• Resource requirements (including con-ditions for personnel and examiners);

• Records and information requirements (including the non- disclosure and secu-rity of confidential information);

• Development and maintenance of the certification scheme;

• Development and maintenance of the assessment and evaluation process/ instruments;

• Suspending, reducing, or withdrawing certification;

• The use of certificates, logos, and marks;• Appeals and complaints; and• Management systems.

The standard provides the specific requirements that—if followed correctly—will result in a certification program for persons that is fair, valid, reliable, and le-gally defensible. The terms fair, valid, and reliable are important components addressed by ISO/IEC 17024, which states that “Ap-

propriate methodology and procedures shall be documented and implemented in order to reaffirm, at justified defined intervals, the fairness, validity, reliability, and general performance of each examination, and that all identified deficiencies are corrected.”2

Structural RequirementsISO/IEC 17024 has requirements as-

sociated with the structure of the certifica-tion body. This is important to ensure the impartiality and validity of the certification process. This includes a clear separation between certification activities and any other activities the organization may do including offering training and/or serving as a membership organization. The reason the impartiality of the process is so important is because the main purpose of certification is to attest that the certified person has the knowledge and skills for competent practice. What we don’t want to happen is for com-mercial or any other interests to interfere with the pure mission of awarding certifica-tion to the competent persons. It would be too easy for purposes to be mingled. For example, if the certification body is also a membership organization, it would be too easy to compromise the certification pro-cess by making the examination easier or cheaper for members than for others or even to require membership in order to obtain the

certification (which is not allowed by ISO/IEC 17024).

ISO/IEC 17024 also has structural requirements that specifically address the separation of training/education from testing. An examination is by design sup-posed to be a random sample of the body of knowledge or skills that the certified person is supposed to have. We can’t test every single concept in one exam, but we can require that the candidate study all of the knowledge and skills that may be covered in the examination and be prepared to answer whatever question we give him or her. The exam will then contain a set of questions that he or she doesn’t know will be asked (a random sample of the content from each of the content domains). If a trainer or educator is exposed to the examination content, there is a danger that the trainer will teach to the test, rather than teach all that is needed to pass the examination. Teachers by nature are caring individuals who want to see their students succeed. And if success is passing a licensing or certification examination, teachers may inadvertently focus more on the concepts that the teachers know will be covered in the examination. This could result in a candidate passing the examination who, in fact, does not know the wide body of knowledge and skills required for the job. The candidate would only know the content associated with the examination.

Similarly, if teachers or educators are used to author the examination questions, there would be a natural propensity to write questions that are associated with the con-tent that the teacher teaches. So instead of having a big bank of examination questions associated with the full range of knowledge and skills associated with the certification area, the certification body could end up with a question bank that has questions specifically focused on the training cur-riculum. This is just as much of a danger as allowing the educator to know the content of the examination. Thus it is important to be sure that subject- matter educators are not involved in the development or administra-tion of the certification program.

Finally ISO/IEC 17024 requires the structure to be impartial and include the par-ticipation of all interested parties. We have all heard the comment, “There is strength in numbers.” The same can be said about

(Continued from page 9)

January/February 2014 11

Standards Engineering

certification programs. A certification program developer should be sure that the structure has provided for the involvement of appropriate experts as well as input from others who will be impacted by the certifica-tion (employers, consumers, certified people, etc.). Questions written by five content experts will be stronger than questions writ-ten by only one content expert because one content expert probably can’t be counted on to have in- depth knowledge of all the con-cepts being examined. Further, if the ques-tions written by multiple content experts are reviewed by several more groups of experts, the examination will improve over time.

Resource RequirementsISO/IEC 17024 contains requirements

associated with resources of the certifica-tion body, including personnel. Personnel requirements cover all personnel involved in certification activities including examiners and invigilators (examination proctors) as well as outsourced personnel. It is important for a certification body to ensure that all per-sonnel, both internal and outsourced, have agreed to keep confidential all secured data and records, and have agreed to comply with the policies of the certification body, includ-ing those associated with conflict of interest and impartiality. Finally, requirements also address other resources associated with the certification process including examination equipment and so forth.

Records and Information RequirementsISO/IEC 17024 contains requirements

associated with records and information. It addresses the information that must be made public and the information that must be kept secure. Candidate data as well as any examination data is confidential and must be kept secure. However there is a re-quirement that the certification body reports the status of a certified person if asked and provide information regarding the scope of the certification and the certification process to potential candidates.

Development and Maintenance of the Certification Scheme

ISO/IEC 17024 is very specific in terms of the components that must be included in the development and maintenance of the certification scheme for persons. ISO/IEC

17024 requires a scheme for persons to contain the following (Clause 8):• Scope of certification• Job and task description• Required competence (knowledge and

skills to achieve intended results)• Abilities (if applicable)• Prerequisites (when applicable)• Code of conduct (when applicable)• Criteria for initial certification and re-

certification• Assessment methods for initial certifi-

cation and recertification• Surveillance methods and criteria (if

applicable)• Criteria for suspending and withdraw-

ing certification• Criteria for changing the scope or level

of certification (if applicable)• The involvement of appropriate experts• The use of an appropriate structure that

fairly represents the interests of all par-ties significantly concerned without any interest predominating

• The identification and alignment of prerequisites, if applicable, with the competence requirements

• The identification and alignment of the assessment mechanisms with the competence requirements

• A job or practice analysis that is con-ducted and updated to:

{ Identify the tasks for successful per-formance

{ Identify the required competence for each task

{ Identify the prerequisites (if appli-cable)

{ Confirm the assessment mechanisms and examination content

{ Identify the recertification require-ments and interval

One of the issues associated with ISO/IEC 17024 is that a number of ISO Technical Committees (TCs) and Working Groups (WGs) are taking ISO/IEC 17024 and going further to develop competence schemes for their professions. Examples of this include welders, non- destructive testing professionals, food auditors, and so forth. Some of these TCs and WGs are utilizing ISO/IEC 17024 as a normative reference and then, in the development of their schemes, including requirements that conflict with

the requirements of ISO/IEC 17024. This is becoming an increasing problem to be addressed by ISO CASCO and its Technical Interface Group (TIG). The exact method of addressing the concern is currently being discussed; however the critical point is for scheme developers to know that ISO/IEC 17024 contains an entire clause (Section 8) that defines how a scheme is to be developed if it is going to meet ISO/IEC 17024.

Development and Maintenance of the Assessment and Evaluation Process/Instruments

ISO/IEC 17024 provides requirements associated with how the assessment and the examination are developed and maintained. This is one of the most important aspects of certification activities that must be addressed by a certification body for persons. If the examination or assessment process isn’t mea-suring what it is supposed to be measuring, the certification body could be awarding cer-tification to someone who is not competent.

Clause 9.3.5 of ISO/IEC 17024 states that “Appropriate methodology and pro-cedures (e.g. collecting and maintaining statistical data) shall be documented and implemented in order to reaffirm, at justi-fied defined intervals, the fairness, validity, reliability and general performance of each examination, all that all identified deficien-cies are corrected.” Validity and reliability are two important measurement concepts re-quired of a certification scheme for persons. In order to be useful, an examination must provide inferences about people by differ-entiating between desirable and undesirable personal characteristics in a given situation. In other words, the examination must be able to correctly establish that an individual has the necessary competencies for the job. This requirement is known as validity.

Validity consists of the evidence that supports the interpretation or outcome of examination scores. Since the objective is to prove that an individual is prepared to practice safely and effectively, the evidence for validity should establish a relationship between the issuance of a credential and readiness for practice. Reference to the va-lidity of an assessment says that the process measures what it is intended to measure. It is very easy for constructs other than the content to be measured to affect the examination

(Continued on page 12)

12

Standards Engineering

January/February 2014

(Continued from page 11)

process and influence the assessment process. For example, if an examination uses lengthy written questions with detailed scenarios to measure knowledge of welding concepts, is it measuring the subject’s reading skills instead of knowledge of welding?

Although validity is a prime consid-eration, a good examination must also be reliable. This means that it must measure a person’s abilities in a consistent manner. Without validity, it does not matter if the test is reliable because it does not measure the correct skill sets. For example, if a group of candidates is asked to take an examination one day and then retested a week later with-out having learned anything in the interim, would they achieve the exact same scores? If the examination is reliable they would. But if the exam is measuring the wrong thing, having it be consistent doesn’t matter. ISO/IEC 17024 requires that reliability be demonstrated. There are many methods by which reliability estimates can be calculated.

Other ClausesISO/IEC 17024 also contains require-

ments associated with suspending, reduc-ing, or withdrawing certification; the use of certificates, logos and marks; appeals and complaints; and management systems. One of the important concepts related to the use of certificates, logos, and marks is that for a certification program, the certificates belong to the certification body and can be withdrawn. The certification is supposed to communicate competence of the certified person; if the certified person does not abide by all of the requirements of the certification body, including competence requirements, recertification requirements, or a code of ethics, the certification body must withdraw the certification. ISO/IEC 17024 has specific requirements for enforceable agreements pertaining to the withdrawal of certification.

Psychometrics and Measurement TheoryOne of the problems for certification

bodies for persons is how to develop schemes for persons according to ISO/IEC 17024 and how to ensure that examinations are fair, valid, and reliable. The main issue is that most certification bodies, professional as-sociations, and organizations do not have the expertise to understand, develop, and manage these schemes and examination programs.

There exists an entire academic discipline of study (psychometrics) that deals with how to develop and administer examinations and assessments that are fair, valid, reliable, and legally defensible. Psychometrics is defined as the field of study associated with the theory and techniques of psychological measure-ment which includes the measurement of knowledge, skills, abilities, and attitudes. The field is primarily concerned with the construction and validation of measurement instruments such as examinations, and other assessments. To create a valid and reliable certification scheme for persons (a require-ment of ISO/IEC 17024), one must begin with developing the certification scheme in a sound manner using an appropriate process.

While it isn’t a requirement to do so, most US certification bodies for persons utilize outsourced vendors who specialize in psychometrics when developing their certification for persons programs. A number of these vendors have familiarity with ISO/IEC 17024. Most professional testing ven-dors have staff expertise in psychometrics, examination development, and certification program development. This is not to imply that a body can’t develop a certification pro-gram for persons on their own without the use of experts. There have been some who have successfully done so. But it is a little like saying one can’t build one’s own home because one does not have a contractor doing the construction. It can be done, but requires quite a bit of effort and care to be sure it is completed correctly. Assuming the certifica-tion body can understand the clauses, utiliz-ing ISO/IEC 17024 will help. By adhering to the requirements in the standard, certification bodies are taking big steps towards develop-ing quality certification programs.

ConclusionsThe proliferation of credentials (licenses,

certificates, and certifications) has led to confusion in the quality of these credentials. When comparing the three, certifications have the most rigor because, by definition, there is an assessment of competence of the certified person conducted by an impartial third party. ISO/IEC 17024 is a standard developed by ISO to describe the requirements for certifica-tion bodies for persons. By adhering to ISO/IEC 17024, a certification body for persons may be able to better develop a certification

scheme for persons and a certification pro-gram for persons that is fair, valid, reliable, and legally defensible and one in which em-ployers and consumers can have confidence.

SES CertificationThe world's only certification program

for standards professionals!

Become a

Certified StandardsProfessional (CStd)

or an

Associate StandardsProfessional (AStd)

Go to www.ses-standards.org

or contactSES Certification Program

SES – The Society for Standards Professionals

1950 Lafayette RoadBox 1

Portsmouth, New Hampshire 03801tel: +1 603 926 0750

fax: +1 603 610 7101admin@ses-standards.org

References1 July 2012, US Department of Labor, BLS,

available at: http://www.bls.gov/news. release/pdf/nlsoy.pdf

2 ISO/IEC 17024, Section 9.3.5