ng bb 20 data collection
TRANSCRIPT
National GuardBlack Belt Training
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Module 20
Data Collection
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
CPI Roadmap – Measure
Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive.
TOOLS
•Process Mapping
•Process Cycle Efficiency/TOC
•Little’s Law
•Operational Definitions
•Data Collection Plan
•Statistical Sampling
•Measurement System Analysis
•TPM
•Generic Pull
•Setup Reduction
•Control Charts
•Histograms
•Constraint Identification
•Process Capability
ACTIVITIES• Map Current Process / Go & See
• Identify Key Input, Process, Output Metrics
• Develop Operational Definitions
• Develop Data Collection Plan
• Validate Measurement System
• Collect Baseline Data
• Identify Performance Gaps
• Estimate Financial/Operational Benefits
• Determine Process Stability/Capability
• Complete Measure Tollgate
1.Validate the
Problem
4. Determine Root
Cause
3. Set Improvement
Targets
5. Develop Counter-
Measures
6. See Counter-MeasuresThrough
2. IdentifyPerformance
Gaps
7. Confirm Results
& Process
8. StandardizeSuccessfulProcesses
Define Measure Analyze ControlImprove
8-STEP PROCESS
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
3
Learning Objectives
Determine what to measure and why
Prepare plans to collect output, process and/or input data
Apply sampling techniques, as needed
Construct forms and test data collection procedures
Refine data collection
Implement data collection plan
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
4
What Is a Measure?
A quantified evaluation of characteristics and/or level of performance based on observable data
Examples include:
Length of time (speed, age)
Size (length, height, weight)
Dollars (costs, sales revenue, profits)
Counts of characteristics or “attributes” (types of customer, property size, gender)
Counts of defects (number of errors, late checkouts, complaints)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
5
Why Measure?
Establish the current performance level (baseline)
Determine priorities for action – and whether or not to take action
Substantiate the magnitude of the problem
To gain insight into potential causes of problems and changes in the process
Prevent problems and predict future performance
To gain knowledge about the problem,process, customer or organization
National GuardBlack Belt Training
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Determine What to Measure
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
7
What Do We Need to Know?
The first step in the creation of any data collection plan is to decide what you need to know about your process and where to find measurement points
What data is needed to “baseline” our problem?
What “upstream” factors might affect the process/problem?
What do we plan to do with the data after it has been gathered?
Do we have a balance between Output and Input/Process measures?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
8
Deciding “What and Where”
Preparing the SIPOC diagram and a more detailed process map can help a team select its measures
Choosing good measures requires a clear understanding of the definitions of and relationships between Output, Process, and Input measures
Input Output
Process
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
9
Overtime
Score
Output Input/Process
Y = f ( X1 + X2 + X3 + . . . . . . . . . Xn )
Customer Satisfaction
= Front Desk
Courtesy+
Check In
Ease+
Room
Comfort+
Room
Service
Check Out
Ease+
Loan Process Cycle Time =
Application
Data EntryTime
+ +Credit &
Collateral Check Time
Risk Assessment
Time
+Review &
Approval Time
Loan Service
Time+
Final Score in
Basketball
Game
=First
QuarterScore
+Second
QuarterScore
+Third
QuarterScore
+Fourth
QuarterScore
+
“X” and “Y” Variables
Generally, you can influence some of the Xs but not all. CPI projects will generally address those Xs which can be influenced
and which have the greatest impact on Y.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
10
Measuring Business Processes
Time Per Task
In-Process Errors
Labor Hours
Exceptions
How well do these (Xs)… …predict this (Y)?
X - PREDICTOR (Leading) MEASURES
Y - RESULTS (Lagging) MEASURES
Input OutputProcess
• Customer Satisfaction
• Total Defects
• Cycle Time
• Cost Profit
• Arrival Time
• Accuracy
• Cost
• Key Specs
(X) (Y) (X)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
11
Categories of Performance Metrics
Developing Input, Process and Output metrics around the Voice of the Customer (VOC) and Voice of the Business (VOB) process performance needs is a good starting point for determining what to measure
Quality
Product or Service Features, Attributes, Dimensions, Characteristics Relating to the Function of the Product or Service, Reliability, Availability, Taste, Effectiveness - Also Freedom from Defects, Rework or Scrap (Derived Primarily from the Customer - VOC)
CostProcess Cost Efficiency, Prices to Consumer (Initial Plus Life Cycle), Repair Costs, Purchase Price, Financing Terms, Depreciation, Residual Value (Derived Primarily from the Business - VOB)
SpeedLead Times, Delivery Times, Turnaround Times, Setup Times, Cycle Times, Delays (Derived equally from the Customer or the Business – VOC/VOB)
Serviceand Safety
Service Requirements, After-Purchase Reliability, Parts Availability, Service, Warranties, Maintainability, Customer-Required Maintenance, Product Liability, Product/Service Safety
StewardshipEthical Business Conduct, Environmental Impact, Business Risk Management, Regulatory and Legal Compliance
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
12
Output Measures
Referred to as “Y” data. Output Metrics quantify the overall performance of the process, including:
How well customer needs and requirements were met (typically Quality & Speed requirements), and
How well business needs and requirements were met (typically Cost & Speed requirements)
Output measures provide the best overall barometer of process performance
Focus on one Primary Output (Y) metric at a time. Use Secondary Y metrics to “keep you honest”
Example: If the Primary Y is to improve cycle time, the Secondary Y could
monitor defects to make sure they also improve or at least don’t get worse!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
13
Typical Output Measures
Product/Manufacturing
AmmoMetal chemistry/thickness/propellant weight/ballistics
Dining-inCeremony
Number of missing/incorrect place cards, seating time, delivery time, accuracy (food/beverage order)
Service/Transactional/Administrative
Re-enlistmentPapers
Cycle time, accuracy (# of errors), completeness (# of items missing)
Anthony’sPizza
Delivery timeliness, accuracy, temperature
OutputProcess TypePossible Output
(Y) Measures
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
14
X and Y Metrics
Suppliers Inputs Process Outputs Customers
• Delivered Invoice
• Billing Dept. staff• Customer
database• Shipping
information• Order information
Billing Process
Input Metrics Process Metrics Output Metrics
• System responsiveness• Accuracy of order info.• Accuracy of shipping
info.
• Rework % at each step • Invoice accuracy
Quality
Speed
Cost
• Time to receive order info.• Time to receive shipping
information
• # of process steps• Time to complete invoice• Time to deliver invoice• Delay time between steps
• Invoice cycle time
• # of process steps • Cost/invoice
• Accuracy of database info.
• Staff expertise• System up-time
• # of billing staff
Other Metrics• Invoices
processed per month and variability
National GuardBlack Belt Training
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Develop Data Collection Plan
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Exercise: Data Collection
Collect Height Data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
17
Types of Data
Continuous / Variable – Any variable measured on a continuum or scale that can be infinitely divided into recognizable parts. Primary types include time, dollars, size, weight, temperature, and speed. Any metric that can be continuously divided by 2 and the metric still makes sense is a continuous metric. Continuous Data is always preferred over Discrete or Attribute Data.
Discrete / Attribute – A count, proportion, or percentage of a characteristic or category. Service process data is often discrete.
Continuous/Variable
• Cycle time
• Cost or price
• Length of call
• Temperature of rooms
Discrete/Attribute
• Late delivery
• Gender
• Region/location
• Room type
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
18
The Objective: Data Collection Plan
Let’s see how a Data Collection Plan is developed
Data Collection Plan
12 3 4 5 6
How will data be used? How will data be displayed?
Examples: Examples:
Developed
earlier
Identification of Largest Contributors
Identifying if Data is Normally Distributed
Identifying Sigma Level and Variation
Root Cause Analysis
Correlation Analysis
Pareto Chart
Histogram Control Chart
Scatter Diagrams
Performance
Measure Operational
DefinitionData Source
& Location Sample SizeWho Will
Collect Data
When Will
Data Be
Collected
How Will
Data Be
CollectedStratification Factors
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
19
Step 1. Stratification Factors
Data Stratification - Capturing and use of characteristics to sort data into different categories (also known as “slicing the data”)
Used to:
Provide clues to root causes (Analyze)
Verify suspected root causes (Analyze)
Uncover times, places where problems are severe (“vital few”)
Surface suspicious patterns to investigate
What are the ways you need to look at the data?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
20
Stratification Factors
If you do not collect stratification factors “up front,” you might have to start all over later. On the other hand, seeking too many factors makes the data more difficult and/or more costly to collect.
What Complaints, Defects
When Month, Day
Where Region, City
WhoDepartment,
Individual
Factors Examples
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
21
Stratification Matrix
Key Steps
Fill in the Output measure Y
Fill in the key stratification questions you have about the process in relationship to the Y
List out all the levels and ways you can look at the data in order to determine specific areas of concern
Create specific measurements for each subgroup or stratification factor
Review each of the measurements (include the Y measure) and determine whether or not current data exists
Discuss with the team whether or not these measurements will help to predict the output Y, if not, think of where to apply the measures so that they will help you to predict Y
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
22
Stratification Matrix
(Output Y)
1
2 3 4
5
6
Questions About Process Stratification factorsX Variables
Measurements
Will these measurements help to predict
Y? (Y/N)
Does data exist to support
these measurements
?(Y/N)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
23
Stratification Matrix Example - Checkout
23 4
5
6
Total adjustments
at checkout
Does the number
adjustment vary over time?
# adjustments / day
Is there a difference by
type of employee?
Is there a difference by
type of customer?
Does the amount of
adjustments vary from one
location to another?
By time period
% of adjustments / associate
# of adjustments by new
vs. exp. Employees
By employee
By type
# adjustments by room size
# adjustments by
customer segment
By location# adjustments in North East
# adjustments in South
# adjustments in Midwest
# adjustments last year
2 3 4
1
(Output Y)
Questions About Process Stratification factorsX Variables
Measurements
Will these measurements help to predict
Y? (Y/N)
Does data exist to support
these measurements
?(Y/N)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
24
Step 2. Developing Operational Definitions
Operational Definitions apply to MANY things we encounter every day. For example, all the measurement systems we use (feet/inches, weight, temperature) are based on common definitions that we all know and accept. Sometimes these are called “standards.”
Other times, our operational definitions are more vague. For example, when someone says a loan is “closed” they might mean papers have been sent, but not signed; another person might mean signed but not funded; a third person might mean funded but not recorded.
While here we are focused on operational definitions in the context of measurement, the concept applies equally well to “operationally defining” a customer requirement, a procedure, a regulation, or anything else that benefits from clear, unambiguous understanding
Learning to pay attention to and clarify operational definitions can be a major side benefit of the CPI process
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
25
Defining “Operational Definitions”
What it is...
A clear, precise description of the factor being measured
Why it is critical...
So each individual “counts” things the same way
So we can plan how to measure effectively
To ensure common, consistent interpretation of results
So we can operate with a clear understanding and with fewer surprises
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
26
Developing Operational Definitions
From General to Specific:
Step 1 – Translate what you want to know into something you can count
Step 2 – Create an “air-tight” description of the item or characteristic to be counted
Step 3 – Test your Operational Definition to make sure it is truly “air-tight”
Note: Sometimes you will need to do some “digging” up-front to arrive at good operational definitions. It is usually worth the effort!!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
27
Step 3. Data Sources
Key Question: Does the data currently exist?
Existing Data – Taking advantage of archived data or current measures to learn about the Output, Process, or Input
This is preferred when the data is in a form we can use and the Measurement System is valid (a big assumption and concern)
New Data – Capturing and recording observations we have not or do not normally capture
May involve looking at the same “stuff,” but with new Operational Definitions
This is preferred when the data is readily and quickly collectable (it has less concerns with measurement problems)
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
28
Key Considerations: Existing vs. New Data
Existing vs. New Considerations
Is existing or “historical” data adequate?
Meet the Operational Definition?
Truly representative of the process, group?
Contain enough data to be analyzed?
Gathered with a capable Measurement System?
Cost of gathering new data
Time required to gather new data
The trade-offs made here, I.e. should the time and effort be taken to gather new data, or only work with what we have, are significant and can have a dramatic impact on the project success
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
29
Step 4. How will Data Be Collected?
Check Sheets
The workhorse of data collection
Enhance ease of collection
Faster capture
Consistent data from different people
Quicker to compile data
Capture essential descriptors of data
“Stratification factors”
Need to be designed for each job
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
30
Data Collection Forms – Check Sheets
Check sheets are convenient for gathering data
Data sheets allow:
Faster, more accurate capture
Consistent data from different people
Quicker, easier compilation
Capture essential descriptors of data
Designed for each different data gathering situation
The data may then be analyzed
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
31
Get Data You Can Use
As you set up Check Sheets...
Prepare a spreadsheet to compile the data
Think about how you will do the compiling (and who will do it)
Consider what sorting, graphing, or other reports you will want to create
Continuous or Discrete Data?
Adequate level of discrimination and accuracy?
Adjust check sheet as needed to ensure usable data later
But do not make data harder to collect
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
32
Constructing Check Sheets
1. Select specific data and factors to be included
2. Determine time period to be covered by the form
Day, Week, Shift, Quarter, etc.
3. Construct form
Be sure to include:
Clear labels
Enough room
Space for notes
4. Test the form!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
33
Check Sheet Tips
Include name of collector(s) (first and last)
Reason/comment columns should be clear and concise
Use full dates (month, date, year)
Use explanatory title
Consider lowest common denominator on metric
Minutes vs. Hours
Inches vs. Feet
Test and validate your design (try it out)
Do not change form once you have started, or you will be “starting over!”
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
34
Types of Check Sheet: Frequency Plot
Frequency of RepairsJuly
1
2 X X X X X X X
3 X X X X X
4 X X X X X
5 X X X X
6 X X
7 X X X
8 X
9 X X X X X X
10 X X X X
11 X X X X
12 X X X X
13 X
14 X X X
15
16 X X X X X X
17 X X X X X
18 X X X X X X X X
19 X X X X
20 X
21 X X X X X
22
23 X X X X X X X X X
24 X X X X X X X
25 X X X X X X
26 X X X X X X
27
28 X X X X X
29 X X
30 X X X X X X X X
31 X X X X X X
Shows “distribution” of items or occurrences along a scale or ordered quantity
Helps detect unusual patterns in a population –or detect multiple populations
Gives visual picture of “average” and “range”
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
35
Types of Check Sheets: Standard
Week of: 6/26 Collected by: Kevin Regan
TV Smk Det Thrmstat RemCon Shower Window
30-Jun 8:00a EJS X X 10 min
28-Jun 8:15a MWT X 1 hr
27-Jun 7:00p MWT X 15 min
26-Jun 6:30p KLC X 2 hrs
28-Jun 5:45p PP X 30 min
30-Jun 6:00a KR X 40 min
1-Jul 8:15p DRT X 4 hrs Replaced part
1-Jul 8:20p ECS X 2 hrs Not in stock
28-Jun 9:35a MWT X 1 hr
29-Jun 9:40a KLC X 30 min
29-Jun 5:15p EJS X 45 min
29-Jun 5:20p KR X 15 min
Repair
TimeNotes
Repair ComplaintCall Date Call Time Initials
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
36
Types of Check Sheets – Traveler
Traveler Checksheet
Awards Approval Process
Awardee: __________________________________________________
Award type: □ PCS □ Other ___________________________
Proposed award date: ________________________________________
Recommender’s division:
□ G-1 □ G-2 □ G-3 □ G-4 □ Other __________
Process stepTime begun; Time
completedDefects found
Fill out forms
Approve
recommendation
Schedule presentation
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
37
Types of Check Sheets – Confirmation
Example: Power Steering project tracking
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
38
Types of Check Sheets – Location
Defect location Check Sheet for rotor blade voids
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
39
Check Sheet Takeaways
A check sheet is an easy way to collect data in order to observe trends and identify improvement priorities
Mistake-proof data collection by using check boxes, tallies, or choices that can be circled (reduce any writing to an absolute minimum – or none at all!)
Remember to include those who understand the process and those who will actually use the check sheet in the design of the check sheet. This is very important for success!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
40
Step 5. Who Will Collect the Data?
Considerations:
Familiarity with the process
Availability/impact on job
Rule of Thumb – If it takes someone more than 15 minutes per day it is not likely to be done
Potential Bias
Will finding “defects” be considered risky or a “negative?”
Benefits of Data Collection
Will data collection benefit the collector?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
41
Preparing Collectors
Be sure collectors:
Give input on the check sheet design
Understand operational definitions (!)
Understand how data will be tabulated
Helps them see the consequences of changing
Have been trained and allowed to practice
Have knowledge and are unbiased
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
42
Step 6. Sampling
Sampling is using a smaller group to represent the whole population (the foundation of “statistics”)
Benefits:
Saves time and money
Allows for more meaningful data
Simplifies measurement over time
Can improve accuracy
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
43
Sampling Considerations
Time
Cost
Accuracy
Units ProcessedPer Day
Cost to CollectData
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
44
Population – Drawing from a fixed group with definable boundaries. No time element.
Customers
Complaints
Items in Warehouse
Process – Sampling from a changing flow of items moving through the business. Has a time element.
New customers per week
Hourly complaint volume
Items received or shipped by day
Sampling Types
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
45
Population or Process Sampling
Of primary importance in a Lean Six Sigma measurement effort is to clarify if you are engaged in Population or Process sampling
Most traditional statistical training focuses on sampling from populations – a group of items or events from which a representative sample can be drawn. A population sample looks at the characteristics of the group at a particular point in time.
Quality and business process improvement tends to focus more often on processes, where change is a constant
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
46
Population or Process Sampling
In process sampling, you measure characteristics of things or characteristics as they pass through the process, and observe changes over time
Any data you collect that has “time order” included can be examined as either a population or a process – however, the size of the sample analyzed might need to be different
Given a choice, process data gives more information, such as trends and shifts of short duration. Process sampling techniques are the foundation of process monitoring and control.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
47
Sampling Biases
Self-selection
Self-exclusion
Missing key representatives
Ignoring “non-conformances”
Grouping
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
48
Sampling Methods/Strategies
The big pitfall in sampling is “bias” – i.e., select a sample that does NOT really represent the whole. The sampling plan needs to guard against bias. Different methods of sampling have different advantages and disadvantages in managing bias.
Judgment
As it sounds – selecting a sample based on someone’s knowledge of the process, assuming that it will be “representative.” Judgment guarantees a bias, and should be avoided.
Convenience
Also just like it sounds – sampling those items or at those times when it is easier to gather the data. (For example, taking data from people you know, or when you go for coffee.) This is another common (but ill-advised) approach.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
49
Sampling Strategies
Best Methods:
Random
Best approach for population situations. Use a random number table or random function in Excel or other software, or draw numbers from a hat.
Systematic
Most practical and unbiased in a process situation. “Systematic” means that we select every nth unit, or take samples at specific times of the day. The risk of bias comes when the timing of the sample matches a pattern in the process.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
50
Sampling Strategies Considerations
Should we stratify first? ...
Focus on one group within the process or population?
Ensure adequate representation from various segments of the population or process?
Does it “feel right?”
Sampling needs to fit common sense considerations
Confront and manage your biases in advance
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
51
Key Sampling Terms/Concepts
Sampling Event – The act of extracting items from the population or process to measure
Subgroup – The number of consecutive units extracted for measurement at each Sampling Event (A “subgroup” can be just one!)
Sampling Frequency – Applies only to process sampling; the number of times per day or week a sample is taken (i.e., sampling events per period of time)
These are the key elements to be included in the sampling plan: what we will “extract,” how many we will take at a time, and how often we will take a sample.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
52
Population Sampling Steps
Building the “Sampling Plan”
1. Develop an initial profile of the data
2. Select a sampling strategy
3. Determine the initial sample size
4. Adjust as needed to determine minimumsample size
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
53
Sampling – Initial Data Profile
Population size (Noted as “N”)
As you begin preparing the Sampling Plan, you first need to determine the rough size of the total population
Stratification factors
If you elect to conduct a stratified sample, youneed to know the size of each subset or stratum
What precision result do you need?
Next, you need to define the level of precision needed in your measurement. Precision notes how tightly your measurement will describe the result. For example, if measuring cycle time, your sample will be affected by whether you want precision in days (e.g. estimate is within +/- 2 days) or hours (estimate is within +/- 4 hours). Precision is noted by the variable “d” or D. The sample size goes up very rapidly as the precision is tightened.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
54
The last step in your initial profile is to estimate the variation in the population
Continuous data requires an estimate ofthe “standard deviation” of the variablebeing measured
Continuous data: How much does thecharacteristic vary? (estimated standard deviation)
Discrete data requires an estimate of “P,” the proportion of the population that contains the characteristic in question
Discrete data: What proportion contains the characteristic?
Sampling – Initial Data Profile
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
55
Sampling – Sampling Strategy
Random or systematic?
How will we draw the sample?
Who will conduct the “sampling event?”
How will we guard against bias?
Most representative vs. time, effort, and cost
No differences between what you collect and what you do not collect
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
56
Sampling
Some Final Tips ...
When you want to ensure representation from different groups or strata, prepare a separate sampling plan for each group
Be sure to maintain the time order of your samples/subgroups so you can see changes over time
Common sense is a useful tool in sampling
Help is available if you need it!
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
57
Test, Refine and Implement
Ensuring “Quality” Measurement
Measurement is rarely perfect – especially at first
Even good measurement can go “bad”
As you use data, lessons might include ...
How to simplify measures
Other stratification factors needed
Ways to improve collection forms
Other measures to investigate
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Input, Process, Output Metrics Template
Suppliers Inputs Process Outputs Customer
Start
Step1Step 2 Step 3
Step 4Step 5
Input Metrics Process Metrics Output Metrics
Quality
Speed
Cost
VOC/
VOB
Required Deliverable
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Operational Definitions Template
Define each of the Key Input, Output, Process Metrics from your SIPOC that you are going to collect data on (via the Data Collection Plan) as well as any other terms that need clarification for the data collectors and everyone else on the team.
Examples:
Award Process PLT: The time from when a Director submits the Award recommendation to the time when the employee is presented the Award in a ceremony.
Number of Claims Processed: The number of Claims processed per weekday (M-F).
Total Hours Worked: The total number of hours worked in the facility including weekends and holidays.
Number of Personnel: The total number of military and civilian personnel working (not including contractors).
Include other unique terms that apply to your project that require clear operational definitions for those collecting the data and for those interpreting the data.
Required
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
Data Collection Plan TemplatePerformance Measure
Operational Definition
Data Source and
Location
How Will Data Be Collected
Who Will Collect Data
When Will Data Be
Collected
Sample Size
Stratification Factors
How will data be used?
Ability to update projects and build tollgate reviews
X1
– Steps to update projects
In DEPMS By counting steps Name ASAP 1 None To find VA, BNVA, NVA
Ability to update projects and build tollgate reviews
X2
– Tollgate template slides that match POI
In DEPMS By determining % of activity steps identified in “Introduction to _____” modules in POI that are adequately addressed in templates
Name ASAP 40 None To determine consistency with POI
Easy Access to LSS tools and references
X3 – Availability of
LSS tools and references
In DEPMS By determining the percentage of tools, with their references, listed on DMAIC Road Map slides that can be found in PS
Name ASAP 63 None To determine availability of tools and references
Easy Access to LSS tools and references
X4
– Steps required to find tools and references
In DEPMS By counting # steps required to find the tools and their references
Name ASAP 37 None To find VA, BNVA, NVA
Required Deliverable
- Example -
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
61
Exercise: Data Collection
Objective
Create a data collection plan for the GGA's Budget Department
Instructions
Include:
1. Key input, process and output metrics
2. Operational definitions
3. Data collection methods
Time = 30 Minutes
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
62
Takeaways
Know what to measure and why
Create a plan to collect output, process and/or input data
Construct forms and test data collection procedures using appropriate data sampling methods
Refine data collection
Collect the data
Analyze the data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
What other comments or questions
do you have?
National GuardBlack Belt Training
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
AppendixSample Size Calculations
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
65Determine What to Measure and Data Collection
How Many Do We Need to Count?
Factors in Sample Size Selection:
Situation: Population or Process
Data Type: Continuous or Discrete
Objectives: What you will do with results
Familiarity: What you guess results will be
Certainty: How much “confidence” you need in your conclusions
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
66
Three Factors Drive Sample Sizes
Three concepts affect the conclusions drawn from a single sample data set of (n) items:
Variation in the underlying population (sigma)
Risk of drawing the wrong conclusions
How small a Difference is significant (delta)
Risk
Variation Difference
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
67
Three Factors: Variation, Risk, Difference
These 3 factors work together. Each affects the others.
Variation: When there’s greater variation, a larger sample is needed to have the same level of confidence that the test will be valid. More variation diminishes our confidence level.
Risk: If we want to be more confident that we are not going to make a decision error or miss a significant event, we must increase the sample size.
Difference: If we want to be confident that we can identify a smaller difference between two test samples, the sample size must increase.
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
68
Minimum sampling size from a population or a stable process can be estimated from the following formulas:
Continuous Data Sample SizeFor continuous data:
Where: n = minimum sample size requireds = estimate of standard deviation of the
population or process data
D = level of precision desired from the sample
in the same units as the “s” measurement1.96 = constant representing a 95%
confidence interval
Determining Minimum Sample Size
296.1
D=
sn
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
69
Benefits of Continuous DataUsually requires a smaller sample
More information for stratification and root cause analysis
Determining Minimum Sample SizeDiscrete Data Sample SizeFor discrete or proportion data:
Wheren = minimum sample sizeP = estimate of the proportion of the population or process which is defective
D = level of precision desired from the sample in units of
proportion1.96 = constant representing a 95% confidence interval
The highest value of p(1-p) is 0.25 or p=0.5
)1(96.1
2
PPn
D=
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
70
Formula for Small Populations
Making adjustments in the minimum sample size required/needed for small populations:
Both sample size formulas assume:
a 95% confidence interval
a small sample size (n) compared to the entire population size (N)
If n/N is greater than 0.05, the sample size should be adjusted to:
The proportion formula should only be used when:
N
n
nn finite
+
=
1
5nP
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
71
Formula for Small Populations
Example: Processing CAC applications
Given:
The sample size formula shows that you need a minimum sample size of 289
You have only processed 200 units
Solution: The correct minimum sample size would be:
=
+
=
+
=
200
2891
289
1N
n
nn finite 118.2 or 119 - minimum sample size required
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
72
Minimum Sample Size – Continuous Example
Example: Sample Size Calculation – Continuous
A Lean Six Sigma team samples a contracting process to determine the average processing time and wishes to estimate the average time within one day. Based on previous sampling, the team has estimated the standard deviation of the current contract process time as 4 days.
What is the minimum sample size required to be able to estimate the average with the required precision?
296.1
D=
sn
contractsn 621
496.12
=
=
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
73
Minimum Sample Size – Discrete Example
Example: Sample Size Calculation – Discrete
Another Lean Six Sigma team determines the minimum sample size required for the proportion of DPW, Department of Public Works, service contracts that require rework at the approval meeting. From interviews, the team has concluded that approximately 25% of the contracts contain errors and require rework. They wish to determine the % requiring rework within 5%.
)25.1(25.05.
96.12
=n
n =(1536.64)(.1875) = 289 contracts
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
74
Exercise:
Sample Size
Objective:
Determine the appropriate sample size
Instructions:
Use the pizza delivery example. The pizza is scheduled for the time the customer requests delivery.
The customer requirement is +/- 10 minutes from the scheduled delivery time
Estimated s = 7.16 minutes and D = 2 minutes
Estimated number of defects is 30% ( P = 0.30; D =5%)
Determine the minimum sample size for both continuous and discrete data
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
75
Exercise:
Sample Size Answers
Discrete
5024.492
03.14
2
16.7*96.196.1222
=
=
=
D=
sn
Continuous
32369.32221.0*2.39)70.0(30.005.0
96.1)1(
96.1 2
22
==
=
D= PPn
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
76
Exercise:
Sample Size
Objective:
Determine the appropriate sample size
Instructions:
Select one output indicator for your process
Determine the type of data (continuous / discrete)
Continuous - estimate “s” and D
Discrete - estimate D and P
Determine the minimum sample size required
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
77
Exercise:
Sample Size FormulaObjective:
Determine the appropriate sample size formula to use
Instructions:
At your tables determine the right formula (proportion/discrete or continuous) to use and calculate the sample size for each situation
1.Estimate the average cycle time within 2 hours. The estimated standard deviation is 8 hours. What is the minimum number to sample?
2.A team collected 100 observations to determine the proportion defective. They found 20% to be defective. How accurately can they estimate the proportion defective?
3.You have a customer survey with 2 categorical questions and 8 interval statements. You estimate that at least one option of a categorical question will be answered by approximately 50% of the respondents and you want to be able to detect a difference within ± 5%. For the continuous statements you want to be able to detect a difference of at least ½ a point. The highest estimated standard deviation for any of the statements is 1.2. You expect the response rate to be 25%. How many surveys do you have to send out and how many completed surveys do you need returned?
UNCLASSIFIED / FOUO
UNCLASSIFIED / FOUO
78
1. Continuous
2. Discrete/Proportioned
3. Discrete Calculation
Continuous
Must send out 4* minimum sample or 4*385 = 1,540
622
)8(96.196.122
=
=
D=
sn
Answers to Sampling Exercise
)1(96.1
2
ppn
D=
)2.1(2.96.1
100
2
2
D=
%808. =D or
385)5.1(5.05.
96.12
=
=n
235.
)2.1(96.12
=
=n