cms measurement report 2011

8/3/2019 Cms Measurement Report 2011

1/58

CMSC 2011 Measurement Study Report

27th Annual

Coordinate Metrology

Systems Conference

Phoenix, Arizona

July 2011

2011 CMSC Measurement Study Report

How Behavior Impacts Your Measurement


2/58

Page 2


TABLE OF CONTENTS

Introduction and Scope .......................................................................................Page 4

Project Overview .................................................................................................Page 4

2010 Measurement Study ....................................................................................Page 4

2011 Measurement Study ....................................................................................Page 5

Measurement Workshop Activity ........................................................................Page 6

Measurement Tasks and Participant Profiles .......................................................Page 8

Observations ........................................................................................................Page 16

Results .................................................................................................................Page 35

Conclusion ...........................................................................................................Page 42

Appendix A ..........................................................................................................Page 44

Appendix B ...........................................................................................................Page 48

Appendix C ...........................................................................................................Page 57

Acknowledgements .............................................................................................Page 58

Cover Photography: Courtesy of the Coordinate Metrology Society, Photographer: Dirk Dursharme


3/58

Page 3


COORDINATE METROLOGY SOCIETY REPORT:

How Behavior Impacts your Measurement

REPORT AUTHORS:

Keith Bevan, National Physical Laboratory (NPL)

Trevor Toman, Coventry University

Keith Bevan serves on the Certification Committee of the Coordinate

Metrology Society. Heis theTraining Product Development Manager,

Knowledge Services, at the National Physical Laboratory. NPL is the UKs

National Measurement Institute, and one of the worlds leading science

and research facilities developing and applying the most accuratestandards, science and technology available.

Hampton Rd | Teddington | Middlesex | UK | TW11 0LW

Tel: +44 20 8977 3222 |

Email:[email protected]| Web:www.npl.co.uk

Trevor Toman serves as the Academic Committee Chair for the

Coordinate Metrology Society. Heis the Manager of Dimensional

Metrology,Faculty of Engineering and Computing, at Coventry University.

Tel:+44 2476 236327Email:[email protected]| Web:www.cad.coventry.ac.uk

CERTIFICATION COMMITTEE, COORDINATE METROLOGY SOCIETY:

Randy Gruver serves as the Certification Committee Chair for the

Coordinate Metrology Society, and has served on the Committee for

three years. Randy is an Employee Development Specialist at The Boeing

Company in Seattle, WA.Email:[email protected]
mailto:[email protected]:[email protected]:[email protected]://www.npl.co.uk/http://www.npl.co.uk/http://www.npl.co.uk/mailto:[email protected]:[email protected]:[email protected]://www.cad.coventry.ac.uk/http://www.cad.coventry.ac.uk/http://www.cad.coventry.ac.uk/mailto:[email protected]:[email protected]:[email protected]:[email protected]://www.cad.coventry.ac.uk/mailto:[email protected]://www.npl.co.uk/mailto:[email protected]


4/58

Page 4


INTRODUCTION AND PROJECT OVERVIEW

In 2009, the Coordinate Metrology Society (CMS) formed a Certification Committee to explore the

need for personnel certification in portable metrology. Their first actions were to research existingcertifications, survey the CMS membership at the 2009 Conference, and develop a preliminary Body of

Knowledge (BOK). An Established Charter was developed to define the committee membership,

establish the reporting responsibility, and update the CMS bylaws to make the Certification Committee

a standing committee.

Their original objectives were to: 1) justify the need for certification, 2) determine if equipment

operators and data processors were the target audience, 3) investigate a partnership in administering

certification, and 4) determine if training would be provided by 3rd party organizations such as

manufacturers, service providers, academia, and national institutes.

CONCLUSIONS DRAWN FROM 2009CERTIFICATION COMMITTEE WORK

Based upon marketing and internal research, the Certification Committee determined a properly

structured certification program would be of definite value for the metrology community. Currently,

equipment must be calibrated, but the operator, the greatest potential source of error, is not required

to be certified. The research also established:

Certification should be multi-level to delineate degree of capability and responsibility.

Certification should indicate mastery of a core body of knowledge with additional certifications

for equipment/software.

Hardware/software certification should demonstrate appropriate technical knowledge as well

as proficiency.

There should be certified examiners for each hardware group.

There may be areas where certification would be application specific.

CMSC2010STATISTICAL STUDY ON SKILLS DEVELOPMENT

In 2010, the CMS Certification Committee performed a statistical study at the annual CMSC

conference. The study was developed to identify skill gaps in the general metrology community, and

was held as an open measurement workshop for conference delegates. In addition to this event, other

workshops held at CMSC would relate content-to-data developed in the study.

The basis of this first principle measurement study was focused on a variety of hand tools used in

dimensional measurement. The objective of the study was to observe participant behavior when

acquiring measurements, and examine the importance of:

Core measurement principles

Implementing the right measurement strategy

A questioning culture


5/58

Page 5


The measurement study was conducted over two-day period in two separate areas using a range of

defined first principle tasks. The criteria of the tasks were modified to allow for various training and

assessment methods to be undertaken, such as questioning techniques, practical task monitoring and

demonstration, and assessment of prior learning and experience. More than 200 sets of measurementswere gathered, and many people participated in all four tasks. Please reference Appendix A for an

overview of the data.

Upon completion of the study, the following eight areas were identified as core knowledge and

understanding vital to making informed decisions during the measurement process.

1. Standards

2. Traceabilty

3. Calibration

4. Measurement Uncertainty

5. Understanding Design Requirements

6. Repeatability and Reproducibility

7. Questioning

8. Attitudes and Behavior

2011CMSCMEASUREMENT STUDY

Inspired by the success of the 2010 measurement study, the CMS commissioned a follow-up study for

CMSC 2011 based on coordinate metrology equipment. The research project would collect

measurement results and focus on core knowledge, understanding, and behavioral requirements. The

event was held over a 3-day period and conference delegates were encouraged to participate in the

study entitled How Behavior Impacts Your Measurements. On days 1 and 2, attendees performed 3D

coordinate measurements while being observed. On the third day, the participants were invited to the

measurement workshop where the study group presented their findings.

Figure 1:

Photorealistic

Rendering of the

2011 Measurement

Study Work Area.


6/58

Page 6


2011SCHEDULE OF MEASUREMENT WORKSHOP ACTIVITY

Day 1 (Tuesday)

On the first day, 3D measurement tasks were carried out with little or no instruction from the studygroup. Without guidance, participants determined the requirements to take measurements or

instructed the operator as to which measurements were needed. The resultant data was stored for

later analysis.

Day 2 (Wednesday)

On the second day, 3D measurement tasks were conducted with instructions or procedures, and the

participant was required to take measurements or instruct the operator with a method to collect the

needed measurements. The resultant data was stored for later analysis.

Day 3 (Thursday)

In the afternoon of the third day, a formal Workshop was held to present the observations and data

from the measurement study to conference delegates.

MEASUREMENT STUDY CATEGORIES

For the measurement study, key characteristics were identified and observed during the data

collection period. An operator and observer were available to monitor those key characteristics for

every participant on every task in these categories:

1. Pre-measurement

2. Measurement planning

3. Measurement

4. Post-measurement

STUDY PARTICIPANTS

The experience level of each participant was

recorded during the study. Their metrology

skills fell into one of the following categories:

A beginner to measurement

A beginner to portable measurement

Some experience in portable

measurement

Experienced in portable measurement


7/58

Page 7


As stated, no formal procedure or planning was devised or available to the participants on the first day

of the measurement study. The following table was used by the operator and observer to assist in the

data acquisition process.

TABLE 1: Observer/Operator Guidelines

Beginner to Measurement Beginner to Portable Coordinate Measurement

Measurement Study Categories

Pre-Measurement

Measurement planning

Measurement

Post Measurement

Give the participant the titles above and ask theiropinion of each area, and its relevance to the task

they are performing.

Guide the participant by explaining the task, but

let them take measurements at their own speed

using their own method.

If they are probing, inform them of the number of

points in/on the feature, but do not be specific.

Provide help if needed!


Pre-Measurement


Measurement

Post Measurement

Observe if the participant asks the general questionsas per the check sheet

Guide the participant, but let them take the

measurements.

Some experience in Portable Coordinate

Measurement

Experienced in Portable Coordinate Measurement


Pre-Measurement


Measurement

Post Measurement

Ask participant if they are comfortable in takingmeasurements, but urge them to explain the

software operative, and articulate what they

want to achieve from the software.

Let participants take measurements and notice if

they ask about results. They may have asked the

question at the pre-measurement stage.


Pre-Measurement


Measurement

Post Measurement

No guidance let the participant tell the observerwhat they want from the software and allow them

to use the equipment. Do not interrupt unless there

is a problem or they are not using a device properly.

Check to see which categories above are mentioned

by the participant.


8/58

Page 8


On day two, formal procedures were introduced for each task encompassing the four categories in

Table 1 on page 7. The operator and observer noted if participants questioned the procedures

associated with the measurement study categories and demonstrate the relevant behavior linked to

good practices.

Were best practices and the right behavior instilled?

Did the participant question and plan the measurement process around the task requirements?

MEASUREMENT TASKS DEFINED

In preparation for the study, three distinct measurement tasks were devised and incorporated

coordinate metrology equipment as follows:

TASK 1 - An automobile door panel

measured using a combination of a

Laser Tracker, retro reflectors, and

inspection software

TASK 2 - An engine compartment

measured using a combination of an

articulated arm, probe and inspection

software

TASK 3 - A vehicle measured using a

Laser Tracker, combined with probing

and laser scanning systems, and

inspection software


9/58


PARTICIPANT PROFILES

Each participant of the study was asked a series of questions about their industrial sector, job role, department, experience,

frequency of taking measurements, if they participated in the 2010 measurement study, and if they use or operate metrology

equipment. This information was used to generate a participant profile. Once the Q & A session was completed, the study group

would monitor the participants measurement behavior and its application to each task performed. This demographic information

will be presented in the following section. Profile data and observations have been assessed and separated into the graphical

representations showing the actual number of participants.

Industries Represented: Aerospace, Nuclear, Automotive and Other

Page 9


10/58


Job Roles in Industry: Engineer, Metrologist, Scientist, Management and Other

Page 10


11/58


Company Department: Design, Quality, Inspection, Manufacturing, Calibration, and Other

Page 11


12/58

Page 12


Measurement Experience: 0 3 Years, 3 7 Years, More than 7 Years


13/58

Page 13


Measurement Frequency: Daily, Weekly, Occasionally or Never


14/58

Page 14


Participated in the CMSC 2010 Measurement Study: Yes or No


15/58

Page 15


Uses/Operates Metrology Equipment


16/58


OBSERVATIONS

In the following section, behavioral and good practice observations were collated and categorized in the graphical representations.

Each measurement task is identified and separated into day 1 and 2. It is important to note the values in the following graphs are

percentages in relation to the observations listed in the graph.

Each observation title was identified, questioned, or commented upon relative to the measurement category. For example, in the

Pre-measurement category of the articulated arm, 6% of the participants asked about the temperature in the measurement area,

19% asked about the calibration status, 6% mentioned the product was located on a carpet.

Day 1: Articulated Arm / Engine Compartment - Pre-measurement Category

Page 16


17/58

Page 17


Measurement Planning


18/58

Page 18


Measurement Process


19/58

Page 19


Analysis


20/58

Page 20


Day 1: Laser Tracker / Door - Pre-measurement Category


21/58

Page 21


Laser Tracker / Door - Pre-measurement Category (Continued)


22/58

Page 22


Laser Tracker / Door - Pre-measurement Category (Continued)


23/58

Page 23


Analysis


24/58

Page 24


Analysis (Continued)


25/58

Page 25


Day 1: Laser Tracker / Vehicle - Pre-Measurement Category


26/58

Page 26


Day One: Laser Tracker / Vehicle Pre-Measurement Category (Continued)


27/58

Page 27


Day 2: Articulated Arm / Engine Compartment Pre-Measurement Category


28/58

Page 28


Day 2: Articulated Arm Measuring / Engine Compartment Pre-Measurement Category (Continued)


29/58

Page 29


Day 2: Articulated Arm Measuring / Engine Compartment Pre-Measurement Category (Continued)


30/58

Page 30


Day 2: Articulated Arm / Engine Compartment Pre-Measurement Category (Continued)


31/58

Page 31


Day 2: Laser Tracker / Automobile Door Pre-Measurement Category (Continued)


32/58

Page 32




33/58

Page 33


Day 2: Laser Tracker / Vehicle Pre-Measurement Category


34/58

Page 34




35/58


MEASUREMENT RESULTS

During the two days of observations and data collection for all three tasks, the number of sets

of measurements collected were as follows:

Day 1 - 54

Day 2 94

The split of participants across each task was very similar, although it was noted that nearly double the

amount of sets of measurements were taken on Day 2.

The results for each task were obtained as follows:

On Day 1, each participant was required to tell the equipment operator the measurement

strategy for both the alignment and measured features of the part. Guidance would be given if

requested.

On Day 2, each participant was required to follow set procedures developed by the equipment

operator to determine the alignment and the measured feature data.

Page 35


36/58


GENERAL OBSERVATIONS OF THE RESULTANT DATA

Task 1: Door

The following graphs are a sample of data taken from numerous features measured. The measurement units are in inches.

The variability in the results on Day 1 ranges from 1.3799 to 1.7007 inches, and Day 2 varies from 0.0051 to 0.0093 inches.

The graphs identify outliers but consistency seemed to be more apparent from Day 1 to Day 2 when incorporating good practice,

asking questions of participants, encouraging questions from the participants to the operator about the pre-measurement,

measurement planning, measurement and post-measurement requirements.

Day 1 Day 2

Page 36


37/58

Page 37


Task 2 - Engine Compartment

The following results show that the consistency in results from Day 1 to Day 2. On Day 1, it was observed that the measurement

strategy chosen by the participants varied thus giving deviation in the results. Whereas on Day 2 with a specific procedure and

measurement strategy being in place, the variation was greatly reduced. Additional equipment such as the use of adaptors were

made available as an option on Day 1, but were required on Day 2. The following two graphs show the variability in the alignment

routines between Day 1 and 2.

Day 1In this graph,

both the Z and Y

values are not

zero as they

should be, and

the Y values are

not balanced.

The ranges of

the values are

very similar

because on Day

1 almost all

users used the

best fit

alignment

method.


38/58

Page 38


Day 2

In this graph, you will notice the Z values for all 3 reference points show a zero deviation, the X values for points 1 and 3 also show a

zero deviation, and the Y values for points 1 and 3 are mirrored and equally opposite as they should be. This proves a much more

controlled and functional alignment, thereby making the subsequent measurements more reliable. This was due to a controlled

alignment being implemented in the program.


39/58

Page 39


The following 2 graphs give a snap shot of the typical observation between the results on Day 1 and Day 2. Even though the number

of reading vary, the range on Day 1 varies from 0.0223 to 0.1503 inches, and on Day 2 from 0.00589 to 0.0357 inches.

Day 1


40/58

Page 40


Day 2

See Appendix B for further results


41/58

Page 41


Task 3 - Vehicle

The following graphs are a sample of data taken from numerous features measured. The measurement units are in millimeters and

the variability in the results on Day 1ranges from 5.36 to 8.198 millimeters, and Day 2 varies from 0.926 to 1.59millimeters. The

graphs identify outliers, but consistency seemed to be more apparent from Day 1 to Day 2 when incorporating good practices, asking

questions of participants, encouraging questions from the participants to the operator about the pre-measurement, measurement

planning, and post-measurement requirements. See Appendix B for further results.

Day 1 Day 2


42/58


CONCLUSION

After completion of the measurement study, the same key areas identified in the 2010 study were also

acknowledged as important during the 2011 study irrespective of the equipment.

The importance of the requirements for pre-measurement, measurement planning, obtaining the

measurement, and the post measurement criteria was emphasized on both days.

Knowledge and understanding of the following factors is essential for a sound measurement strategy

incorporating best measurement practices and using the right behavior to reduce the variation and

uncertainty about the results.

Standards

Traceabilty

Calibration

Measurement uncertainty

Understanding design

requirements

Repeatability and

reproducibility

Analysis and output

requirements

Functionality

Questioning

Planning

Attitudes and behavior

In conclusion, whether the metrologist uses hand tools or the most sophisticated coordinate

measuring system, questioning and planning all the requirements of the measurement help to reduce

the possibility of generating poor measurements.

The keys to informed measurement results are: 1) not taking things for granted, and 2) implementing

behavioral questions and techniques for the measurement process.


43/58

Page 43


SIX GUIDING PRINCIPLES FOR ACHIEVING GOOD MEASUREMENT RESULTS

1. Make the Right Measurement

Define and understand the measurement to be made. Routinely repeated measurements should

follow a pilot study.

2. Choose the Right Instruments

Appropriate instrument should be in a good state of repair and calibrated. Use according to

instructions of owner or manufacturer

3. The Right People

Human error is major source of poor measurement quality. An operator needs to receiveinstructions and training. Where a group of operators is involved, their individual roles and

responsibilities need to be understood.

4. Regular Review

Follow a written schedule to check performance of instruments at regular intervals.

5. Demonstrable Consistency

A measurement result is only valid if it can be reproduced consistently. Local factors need tobe taken into account. Important or difficult measurements should be compared with the same

measurements acquired by other laboratories/operators.

6. The Right Measurement Procedures

Follow appropriate written measurement procedures, including health and safety guidelines.

Review the procedures regularly.


44/58

Page 44


APPENDIX A

2010 Measurement Study

The 2010 The Coordinate Metrology Society hosted an open measurement workshop inviting

conference delegates to participate in a first principle measurement study based on a variety of

hand tools used in dimensional measurement. The objectives were to examine the importance of:

o Core measurement principles

o Instilling the right measurement strategy

o Observing behavior when dealing with measurements

o Instilling questioning

The measurement studies were undertaken in a two day period in two separate areas using various

defined first principle tasks. The criteria of the tasks were modified to allow for various training and

assessment techniques to be undertaken such as:

o Review of prior learning and experience

o Questioning techniques

o Practical task monitoring

o Demonstration

Participant Profile

Diverse Industries

o Aerospace

o Nuclearo Automotive

o Science

o Woodworking

Varied Job Roles

o Quality

o Scientist

o Management

o Measurement

o Supplier Experience

Measurement Experience

o Ranging from newcomers to veterans with over of

15 years of experience


45/58

Page 45


Observations During Measurement Study

Checked jaws for parallelism

Checked calibration status Used gauge block to check vernier

Cleaned before using

Made multiple measurements

Made multiple re-zeros

Checked vernier zero

Cleaned jaws

Misunderstanding of scale and units

Check for damage

Locked jaws using thumbscrew

Measured with jaws not parallel to

the workpiece

Used internal jaws for depth

measurement

Used depth bar and checked zero

Spent effort to find minimum value

Used gauge block as a comparator

Poor lighting didnt help reading scale

Cleaned the artefact

Checked the zero

Felt for dirt on the surface table

Used without zeroing difference between

2 values

Base upwards / part downwards

Cleaned surface plate

Checked using gauge block Used direct comparison against gauge block

During the study, key characteristics were identified and observed during the data collection period. An

operator and observer monitored those characteristics for every participant on every task in these categories:

pre-measurement, measurement planning, measurement, and post-measurement.


46/58

Page 46


Resulting Data

Four measurement tasks were executed with and without instruction that covered diameters, length,

depths and gauge block wringing.

Task 1: Without procedures 55 readings ranged from 0,155 to 0,191mm

Task 2: Without procedures 43 readings ranged from 0.0015 to 1.21 inches

Task 3: With procedures 39 readings ranged from 0,009 to 0,057mm

Task 4: Without procedures 28 readings varied by 0.0045 inches

Key observations and behavior across a wide range of expertise:


47/58

Page 47


Resulting Data (Continued)


48/58


APPENDIX B

Appendix B shows further results for data obtained during the study across all 3 measurement tasks.

Day 1 Door: Examples of data obtained on Day 1 that were not toleranced features.

Page 48


49/58

Page 49


Day 2 Door: Examples of data obtained on Day 2 that were not toleranced features.


50/58

Page 50


Day 1 Door: Examples of data obtained on Day 1 that were toleranced features.


51/58

Page 51


Day 1 Door: Examples of data obtained on Day 1 that were tolerance features.


52/58

Page 52


Day 1 Door: Examples of data obtained on Day 1 that were out of specification.


53/58

Page 53


Day 2 - Door: Examples of data obtained on Day 1 that were out of specification.


54/58

Page 54


Day 1 Engine compartment: The capability index example for the engine compartment. The ideal CP is >1.


55/58

Page 55


Day 2 Engine Compartment: The CP of 99.0 is due to there being no deviation of the data.


56/58

Page 56


Day 1 Vehicle: Examples of data obtained on Day 1 that were tolerance features.

Day 2 Vehicle: Examples of data obtained on Day 2 that were toleranced features.


57/58


APPENDIX C

Flyer Example

Inviting you to the CMSC 2011 Interactive Measurement Study

How Behavior Impact Your Measurement

After the success of the Gage R&R study in 2010 the measurement study returns for 2011...

We invite you to booth 506 to participate in a study that will be coordinated by the National Physical Laboratory

(UK), and assisted by members of the CMS Certification Committee. All attendees are encouraged to participate

in this daily data collection activity, which will provide hands-on experience with large volume measurement

systems.

The purpose of the workshop is to explore the measurement strategies and behavior of coordinate

metrologists.

Each station in Booth 506 will allow the participant to take measurements using a Laser Tracker, Portable Arm

and Optical System. You will have the option to either perform a series of prescribed measurements withassistance or alternatively instruct the appropriate operator on how to take the required measurements. We will

ask you to share your level of experience and background in metrology. Data collected during the first two days

(Tuesday and Wednesday) will be automatically stored and compiled. The anonymous information collected will

then be examined for measurement variability. The Measurement study outcomes will be presented by the

National Physical Laboratory on Thursday in the FLW Ballroom I-J at 2.30pm. The criteria of the measurement

tasks will enable various training and assessments techniques to be analyzed, such as the evaluation of prior

learning and experience, questioning techniques, and practical task monitoring and demonstration.

Your cordial invitation to Booth 506 Tuesday & Wednesday


58/58


ACKNOWLEDGEMENTS

The success of the measurement study would not have been possible without the support of the

CMS Certification Committee, the booth sponsors and volunteers, the metrology equipmentmanufacturers, and lastly, all of the participants.

A special thanks goes to the following individuals for their dedication to the study and contributions to

this report:

Bertrand Gili Metrologic

Nathalie Blanco Metrologic

Mark Denham Metrologic

Dennis Martin Metrologic

Ben Hughes National Physical Laboratory

Figure 2:

Photorealistic

rendering

of the 2011

Measurement

Study Work Area.

cms measurement report 2011

Documents