NG BB 25 Measurement System Analysis - Attribute

Download NG BB 25 Measurement System Analysis - Attribute

Post on 21-Jan-2015

2.781 views

Category:

Education

1 download

Embed Size (px)

DESCRIPTION

 

TRANSCRIPT

<ul><li> 1. UNCLASSIFIED / FOUO UNCLASSIFIED / FOUONational Guard Black Belt Training Module 25MeasurementSystem Analysis (MSA)Attribute DataThis material is not for general distribution, and its contents should not be quoted, extracted for publication, or otherwiseUNCLASSIFIED / FOUO copied or distributed without prior coordination with the Department of the Army, ATTN: ETF.UNCLASSIFIED / FOUO</li></ul><p> 2. UNCLASSIFIED / FOUOCPI Roadmap Measure 8-STEP PROCESS 6. See 1.Validate2. Identify 3. Set4. Determine5. Develop 7. Confirm8. StandardizeCounter-the Performance ImprovementRoot Counter- ResultsSuccessfulMeasuresProblem GapsTargetsCause Measures &amp; ProcessProcessesThroughDefineMeasure Analyze ImproveControl TOOLSProcess MappingACTIVITIES Map Current Process / Go &amp; SeeProcess Cycle Efficiency/TOC Identify Key Input, Process, Output Metrics Littles Law Develop Operational Definitions Operational Definitions Develop Data Collection PlanData Collection Plan Validate Measurement System Statistical Sampling Collect Baseline Data Measurement System Analysis Identify Performance Gaps TPM Estimate Financial/Operational Benefits Generic Pull Determine Process Stability/CapabilitySetup Reduction Complete Measure Tollgate Control ChartsHistogramsConstraint IdentificationProcess Capability Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive.UNCLASSIFIED / FOUO 3. UNCLASSIFIED / FOUO Learning Objective Understand how to conduct and interpret a measurement system analysis with Attribute Data Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 3 4. UNCLASSIFIED / FOUO Attribute Measurement Systems Most physical measurement systems use measurement devices that provide continuous data For continuous data Measurement System Analysis we can use control charts or Gage R&amp;R methods Attribute/ordinal measurement systems utilize accept/reject criteria or ratings (such as 1 - 5) to determine if an acceptable level of quality has been attained Kappa and Kendall techniques can be used toevaluate these Attribute and Ordinal MeasurementSystemsMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 4 5. UNCLASSIFIED / FOUO Are You Really Stuck With Attribute Data? Many inspection or checking processes have the ability to collect continuous data, but decide to use attribute data to simplify the task for the person taking and recording the data Examples: On-time Delivery can be recorded in 2 ways: a) in hours late or b) whether the delivery was on-time or late Many functional tests will evaluate a product on acontinuous scale (temperature, pressure drop, voltagedrop, dimensional, hardness, etc) and record the resultsas pass/failStrive to get continuous data!Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 5 6. UNCLASSIFIED / FOUO Attribute and Ordinal MeasurementsAttribute and Ordinal measurements often rely on subjective classifications or ratings Examples include: Rating different features of a service as either good orbad, or on a scale from 1 to 5 with 5 being best Rating different aspects of employee performance asexcellent, satisfactory, needs improvement Rating wine on a) aroma, b) taste, and c) after tasteShould we evaluate these measurement systems before using them to make decisions on our CPI project?What are the consequences of not evaluating them?Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 6 7. UNCLASSIFIED / FOUO MSA Attribute Data What methodologies are appropriate to assessAttribute Measurement Systems?Attribute Systems Kappa technique which treat all misclassifications equallyOrdinal Systems Kendalls technique which considers the rank of the misclassification For example, if we are judging an advertising service on ascale from 1 to 5 and Inspector A rates the service a 1 whileInspector B rates it a 5. That is a greater misclassificationthan Inspector A rating it a 4 while Inspector B rates it a 5. Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 7 8. UNCLASSIFIED / FOUO Data Scales Nominal: Contains numbers that have no basis on which to arrange in any order or to make any assumptions about the quantitative difference between them. These numbers are just names or labels. For example: In an organization: Dept. 1 (Accounting), Dept. 2 (Customer Service), Dept. 3 ( Human Resources) In an insurance co.: Business Line 1, Line 2, Line 3 Modes of transport: Mode 1 (air), Mode 2 (truck), Mode 3 (sea) Ordinal: Contains numbers that can be ranked in some natural sequence. This scale, however, cannot make an inference about the degree of difference between the numbers. Examples: On service performance: excellent, very good, good, fair, poor Salsa taste test: mild, hot, very hot, makes me suffer Customer survey: strongly agree, agree, disagree, stronglydisagree Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 8 9. UNCLASSIFIED / FOUO Kappa Techniques Kappa is appropriate for non-quantitative systems such as:Good or badGo/No GoDifferentiating noises (hiss, clank, thump)Pass/failMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 9 10. UNCLASSIFIED / FOUO Kappa Techniques Kappa for Attribute Data:Treats all misclassifications equallyDoes not assume that the ratings are equally distributed across the possible rangeRequires that the units be independent and that the persons doing the judging or rating make their classifications independentlyRequires that the assessment categories be mutually exclusive Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 10 11. UNCLASSIFIED / FOUO Operational Definitions Thereare some quality characteristics that are either difficult or very time consuming to define To assess classification consistency, several units must be classified by more than one rater or judge Ifthere is substantial agreement among the raters, there is the possibility, although no guarantee, that the ratings are accurate Ifthere is poor agreement among the raters, the usefulness of the rating is very limitedPoor attribute measurement systems can almostalways be traced to poor operational definitions Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 11 12. UNCLASSIFIED / FOUO Consequences? What are the important concerns?What are the risks if agreement within and between raters is not good? Are bad items escaping to the next operation in theprocess or to the external customer? Are good items being reprocessed unnecessarily? What is the standard for assessment? How is agreement measured? What is the Operational Definition for assessment? Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 12 13. UNCLASSIFIED / FOUO What Is Kappa? KPobserved Pchance K1 PchancePobserved Proportion of units on which both Judges agree = proportion bothJudges agree are good + proportion both Judges agree are badPchance (expected) Proportion of agreements expected by chance = (proportion JudgeA says good * proportion Judge B says good) + (proportion JudgeA says bad * proportion B says bad)Note: equation applies to a two category analysis, e.g., good orbad Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 13 14. UNCLASSIFIED / FOUO Kappa Pobserved PchanceK 1 Pchance Forperfect agreement, P observed = 1 and K=1 As a rule of thumb, if Kappa is lower than 0.7, themeasurement system is not adequate If Kappa is 0.9 or above, the measurement system isconsidered excellent Thelower limit for Kappa can range from 0 to -1 For P observed = P chance (expected), then K=0 Therefore, a Kappa of 0 indicates that the agreement isthe same as would be expected by random chanceMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 14 15. UNCLASSIFIED / FOUO Attribute MSA Guidelines When selecting items for the study consider the following:If you only have two categories, good and bad, you should have a minimum of 20 good and 20 badAs a maximum, have 50 good and 50 badTry to keep approximately 50% good and 50% badHave a variety of degrees of good and badIf only good items are chosen for the study, whatmight happen to P-chance (expected)?Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 15 16. UNCLASSIFIED / FOUO Attribute MSA Guidelines (Cont.) If you have more than two categories, with one of the categories being good and the other categories being different error modes, you should have approximately 50% of the items being good and a minimum of 10% of the items in each of the error modes You might combine some of the error modes as other The categories should be mutually exclusive or, if not, they should also be combinedMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 16 17. UNCLASSIFIED / FOUO Within Rater/Repeatability Considerations Have each rater evaluate the same item at least twice Calculate a Kappa for each rater by creating separateKappa tables, one for each rater If a Kappa measurement for a particular rater is small, that rater does not repeat well within self If the rater does not repeat well within self, then they will not repeat well with the other raters and this will hide how good or bad the others repeat between themselves Calculatea between-rater Kappa by creating a Kappa tablefrom the first judgment of each rater Between-rater Kappa will be made as pairwise comparisons(A to B, B to C, A to C) Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 17 18. UNCLASSIFIED / FOUO Example: Data Set = Attribute Ordinal.mtw An educational testing organization is training five new appraisers for the written portion of the twelfth-grade standardized essay test The appraisers ability to rate essays consistent with the standards needs to be assessed Each appraiser rated fifteen essays on a five-point scale (-2, -1, 0, 1, 2) The organization also rated the essays and supplied the official score Each essay was rated twice and the data captured in the file Attribute Ordinal.mtw Open the file and evaluate the appraisers performance Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 18 19. UNCLASSIFIED / FOUO Minitab and Attribute Measurement SystemsStat&gt;Quality Tools&gt;Attribute Agreement Analysis Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 19 20. UNCLASSIFIED / FOUO Minitab Dialog Box1. Double click on theappropriate variableto place it in therequired dialog box: Attribute = Rating Samples = Sample Appraisers = Appraiser2. Click on OKMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 20 21. UNCLASSIFIED / FOUO Within Appraiser PercentThis output represents the percent agreement and the 95%confidence interval around that percentage Date of study : Assessment Agreement Reported by : Name of product: Misc: Within A ppraisers100 95.0% C IP ercent8060Percent4020 0Duncan Hayes Holmes Montgomery SimpsonAppraiser Measurement System Analysis - AttributeUNCLASSIFIED / FOUO 21 22. UNCLASSIFIED / FOUO Within Appraiser Session Window OutputThis output is the same information contained in the graphwith the addition of a Between-Appraiser assessment Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 22 23. UNCLASSIFIED / FOUO Lets Do It AgainStat&gt;Quality Tools&gt;Attribute Agreement AnalysisMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 23 24. UNCLASSIFIED / FOUO Introducing a Known Standard 1. Double click on the appropriate variable to place it in the required dialog box(same as before) 2. If you have a known standard (the real answer) for the items being inspected, let Minitab know what column that information is in.3. Click on OKMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 24 25. UNCLASSIFIED / FOUO Appraiser vs. Standard Date of study :Assessment Agreement Reported by : Name of product: Misc: Within AppraisersAppraiser vs Standard 100 95.0% C I100 95.0% C I P ercent P ercent 90 90 80 80 70 70 PercentPercent 60 60 50 50 40 40 30 30anes esryonan es esryon ncaylmmeps nc aylmmepsDuHHogo SimDu HHogo Simont ontM M Appraiser AppraiserMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 25 26. UNCLASSIFIED / FOUO Within Appraiser In addition to the Within-Appraisergraphic, Minitab will give percentages Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 26 27. UNCLASSIFIED / FOUO Each Appraiser vs. StandardSome appraisers will repeat their own ratings well but may not match the standard well (look at Duncan) Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 27 28. UNCLASSIFIED / FOUO More Session Window OutputThe session window will give percentage data as to howall the appraisers did when judged against the standardMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 28 29. UNCLASSIFIED / FOUO Kappa and MinitabMinitab will calculate a Kappa for each (within) appraiser for each categoryNote: This is only a part of the total data set for illustration Measurement System Analysis - AttributeUNCLASSIFIED / FOUO 29 30. UNCLASSIFIED / FOUO Kappa vs. StandardMinitab will also calculate a Kappa statistic for eachappraiser as compared to the standardNote: This is only a part of the total data set for illustration Measurement System Analysis - AttributeUNCLASSIFIED / FOUO 30 31. UNCLASSIFIED / FOUO Kappa and MinitabMinitab will not provide aKappa between a specificpair of appraisers, but willprovide an overall Kappabetween all appraisers foreach possible category ofresponseHow might this output help us improve our measurement system? Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 31 32. UNCLASSIFIED / FOUO What If My Data Is Ordinal? Stat&gt;Quality Tools&gt;Attribute Agreement AnalysisMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 32 33. UNCLASSIFIED / FOUO Ordinal DataIf your data isOrdinal, youmust also checkthis boxMeasurement System Analysis - Attribute UNCLASSIFIED / FOUO 33 34. UNCLASSIFIED / FOUO What Is KendallsKendalls coefficient can be thought of as an R-squared value, it is the correlationbetween the responses treating the data as attribute as compared to ordinal.The lower the number gets, the more severe the misclassifications were.Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 34 35. UNCLASSIFIED / FOUO KendallsKendalls coefficient can be thought of as an R-squared value, it is the correlation between the responses treating the data as attribute as compared to ordinal. The lower the number gets, the more severe the misclassifications were. Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 35 36. UNCLASSIFIED / FOUO Kendalls (Cont.)Measurement System Analysis - Attribute UNCLASSIFIED / FOUO 36 37. UNCLASSIFIED / FOUO Exercise: Seeing Stars Divide into teams of two One person will be the rater and one the recorder Have each rater inspect each start and determine if it is Good or Bad (Kappa) Record the results in Minitab Mix up the stars and repeat with same rater 2 more times Compare results to other raters and to the known standard Take 30 minutes to complete the exercise and be...</p>

Recommended

View more >