copyright 2002-2006 minnesota council for quality revision 08/21/06 1 minnesota quality award 2006...

Copyright 2002-2006 Minnesota Council for Quality Revision 08/21/06

1

Minnesota Quality Award2006 Site Visit Training


2

Today’s Agenda

• Site Visit Introduction• Site Visit Process

– Advanced Preparation– Final Preparation– Conducting the Site Visit– Interviewing Techniques– Site Visit Products

• What Happens Next


3

Baseline Questions

• Narrative or Baldrige-Express?• Which Criteria?• Length of site visit? (Usually 3 days.)• Anything unusual?


4

Purpose of the Site Visit

• Understand key organizational factors• Verify and clarify facts and site visit issues• Prepare feedback report• Score the items• Assess commitment, sustainability, and

momentum• Prepare and present exit presentation


5

Elements of a Successful Site Visit

People

Process Products

Roles,Ground rules,

Communication

Assignments,Controlled agenda,

Consensus building

Exit presentation,Feedback report,Judges’ presentation


6

Presentation to Judges

Information Flow

+

=


7

Evaluator Role & Responsibilities

• Conduct interviews to gather information• Manage and track resolution of site visit issues• Revise and update comments• Backup other team members as needed• Review and comment on all feedback comments• Lead scoring discussion for assigned category• Provide input to exit presentation and judges

presentation

Remember that you represent the council!


8

Team Lead Role and Responsibilities

• Lead site visit planning meetings and caucuses• Coach team members• Interface with applicant representative and executive• Monitor progress of category owners in closing

issues and updating comments• Oversee process for team refinement/approval of

final comments• Oversee preparation and delivery of exit presentation• Deliver final feedback report to council


9

MCQ Role and Responsibilities

• Ensure communication is occurring between team and applicant

• Participate in opening presentation• Monitor team progress & provide logistical support• Monitor applicant satisfaction and concerns• Resolve process issues that may arise• Moderate the exit presentation


10

Applicant Rep. Role & Responsibilities

• Contribute information for findings• Arrange availability of interviewees and materials• Help team complete its agenda• Provides:

• List of key contacts• Organization chart• Facility layout• Optional requested material

• See Site Visit Preparation Checklist for Applicants


11

Today’s Agenda





12

Site Visit Sequence

Advance Preparation

Final Preparation

Site Visit

1b.Exit

Presentation

1a.Executive Preview

2.Feedback

Report

3. Presentation

to Judges


13

Advance Preparation• Evaluators - prepare Site Visit Issue worksheets• Team leads:

– Compile list of documents to be reviewed– Compile list of individuals to be interviewed

• Team leads and applicant representative– Draft site visit agenda and interview schedule

• Finalize & confirm logistics for:– Hotel and meeting rooms– Security requirements– Emergency contact number 612-462-3577– Name tags (as needed)– Expense accounting


14

Site Visit Issues

• Information that is missing or vague • Deployment of the practices • Cross-cutting issues or themes• Site Visit Issues should be written at the basic

or overall level • Limit SVI’s to 3 – 4 per item • The total SVI’s need to be broad enough to verify

or clarify all the comments in the consensus report


15

Example Site Visit Issues

Verify and Clarify:• Role and activities of senior management (driving customer focus,

empowerment, innovation, learning)

• Degree of involvement and empowerment of employees

• Reliable and accessible data and information

• Use of facts in decision-making

• Focus on customers

• Training effectiveness and on-the-job reinforcement

• Use of strategic objectives, action plans, and related measures to align work at all levels—links to performance feedback and employee rewards

• Evidence of evaluation and improvement

• Integration of all processes—both core and support

• Supplier involvement in performance improvement activities

• Unclear or missing results and continuing trends


16

Site Visit Issues Worksheet Site Visit Issues Worksheet

Item Ref. 5.2-1 Category: 2 and 5 Item Ref #: 2.2 and 5.2

Issue: Verify the content of the Human Resource Plan (2.2) and clarify its use in identifying the strategic training needs for the organization (5.2)

Strategy (evidence sought, person to interview, sample questions):

Person(s) to Interview: HR Manager

Document(s) or Information to Review: Human Resource Plan

Sample Questions: Tell me about HR plans you create. How do you decide what should be included in the plans? How do you use these? What is an example of training that was conducted in the past year based on the plan?

Walk-around Question:

How is it decided what training you receive each year?

Findings:


17

Site Visit Issues Strategies• Acceptable techniques

• Reviewing data, reports, documents, presentations (hard-copy or electronic) as provided directly by the applicant

• Interviewing individuals and teams• Asking “walk-around questions” to check deployment and

communication

• Unacceptable techniques• Conducting focus groups, surveys, or interviews with

customers, suppliers, or dealers• Anything that disrupts work processes• Interviewing sources or collecting data beyond that provided

by the applicant • This includes conducting library or internet research and

visiting web sites


18

“Typical” Walk-around Questions

• What are the values, mission, and vision of the organization?

• What are the goals of your work unit and what role do you play in achieving them?

• What kind of training do you receive?– Is it useful to your work? – How is training reinforced on the job?

• How are you empowered to take actions needed to carry out or improve your work?

• What is recognized or rewarded here?• Who are your customers and how do you know what

they need?


19

Site Visit Interview Matrix

Individual to Interview Site Visit Issue Ref.

Sheila Knight, Marketing 1.1.1, 2.2.1, 3.1.1, 3.1.2, 3.1,3, 3.2.1, 3.2.4, 4.1.1, 4.1.3, 4.2.1, 4.2.2, 7.1.3

Ron McDonald, Administration 1.1.1, 1.1.3, 1.2.1, 1.2.2, 7.2.3

Mary Stewart, Employee Development 1.1.1, 2.1.1, 5.1.1, 5.1.3, 5.2.1, 7.3 all

Alex Hamilton, Operations 1.1.1, 2.1.1, 6.1.3, 6.2.1, 6.2.2, 7.5 all

John Jones, Finance 4.1.4, 4.2.1, 4.3.2, 7.2 all

Leader/manager of Process Improvement Team & members of PI Team

6.1.5

Individual responsible for coordinating leadership communication

1.2.5, 2.2.2

Individual responsible for Support Services Process Improvement Meetings with Leadership

6.2.5


20

Today’s Agenda





21

• Final prep meeting held evening before site visit– Review SV issue worksheets– Review and update documentation list– Review and update interview matrix– Set up Key Themes Storyboard– Establish and schedule final walk-around questions– Finalize the agenda, assignments, pairs

• Evaluators bring copies of site visit issue worksheets, to prepare a composite set for all team members

• Handoff issues or questions as needed• Applicant representative meets with the team

• Aid in planning the time on site• Share any last minute schedule conflicts

Final Preparation


22

Key Themes Storyboard

• A key theme is a Strength or OFI that is common to more than one Item or Category, is especially significant in terms of the applicants Key Factors, and/or addresses a Core Value of the Criteria.

• Provides key points and an overall summary of the team’s evaluation of the applicant.

• Divided into three sections to address three questions concerning important Strengths, significant OFIs, and key Results

• Will evolve throughout the site visit• Will be used to help prepare the exit presentation


23

Site Visit Schedule Template

7:45 - 8:00 TRAVEL TO SITE

8:00 - 8:15 Park, GET ACCESS PASSES, ETC.

8:15 - 9:30 INTRODUCTIONS & APPLICANT PRESENTATION

9:30 - 9:40 TEAM CAUCUS

9:40 - 11:00 FIRST ROUND OF INTERVIEWS

Deputy for Programs/Proj. Mgmt. Chief, CPAC Chief, Engineering Div.

Judith L.A. DesHarnais Linda Kruger John Bailen

Craig & Sarah Terry & Angie Mike & Mary

11:00 - 11:10 TRAVEL 11:10 - 12:10 SECOND ROUND OF INTERVIEWS

Chief, Proj. Mgmt. & Development Chief, Proj. Mgmt. Branch Outreach ManagerChuck Spitzack Lisa Hedin Russel SnyderMike & Sarah Angie & Craig Terry & Mary

12:10 - 12:40 LUNCH12:40 - 1:00 TEAM CAUCUS1:00 - 2:00 THIRD ROUND OF INTERVIEWS

Dep. Chief, Contracting Div. Chief, Public Affairs Office Acting Chief, IM Office & Knowledge ManagerMaj. Willie Green Mark Davidson Dave HimmerickMary & Terry Angie & Craig Sarah & Mike

2:00 - 2:30 TRAVEL, BREAK, CAUCUS2:30 - 3:30 FOURTH ROUND OF INTERVIEWS

Chief, Office of Counsel Chief, Real Estate Div. Chief, Resource Mmgt. OfficeEd Bankston Mark Nelson Randy BrunetMike & Angie Craig & Mary Terry & Sarah

3:30 - 4:00 TRAVEL, BREAK, CAUCUS4:00 - 5:00 FIFTH ROUND OF INTERVIEW

Category 6 Application Lead A group of Account Training CommitteeJames Mosner Managers MembersSarah & Mike\ Craig & Mary Terry & Angie

DAY ONE - MONDAY, AUGUST 25


24

Today’s Agenda





25

Site Visit Schedule

• Day 1– Opening meeting

• Introductions to the applicant (MCQ/Team Leads)• Applicant presentation (up to one hour)

– Facilities tour (optional)

• All Days– Interview meetings – Walk-around interviews of employees– Team discussions (caucuses)– Meeting with applicant representative and executive at end of

day

• Final Day– Executive preview and exit presentation


26

Typical Daily Progress (Day 1)

• Advanced level of understanding of business factors• Close some Site Visit Issues (typically Cat. 1 and 2)• Establish remaining agenda• Confirm the Day 2 schedule will closeout all remaining

SVI’s• Brief applicant on progress and outstanding issues• Category Owners begin editing their comments


27


• Confirm key themes• Close 100% of remaining Site Visit Issues• Identify any remaining information/interviews needed• Finalize plan for Day 3• Brief applicant on progress, outstanding issues, and

plans for Day 3• Finalize 100% of the feedback comments


28


• Close remaining issues (if any)• Prepare and deliver Executive Preview and Exit

Presentation• Complete the feedback report • Identify any Pretty Good Practices• Score items• Assess commitment, sustainability, and momentum• Complete logistics (hotel checkout, return of applicant

materials, collection of shredding, 360 feedback)


29

Evaluator Team Caucuses

• Review progress against the schedule• Report important findings• Handoff notes to category leaders• Hand-off any outstanding questions• Update SVI worksheets and key themes

storyboard• Review and update feedback comments• Be sure to schedule enough time for

caucuses plus some personal break time


30

Managing Issues and Handoffs

• Create 7 flip charts, one for each category.

• Ask Evaluators to use yellow sticky notes to record issues they have to hand off to the category owner in the “open” area and red for new issues.

• When the information has been collected, record the answer and move the sticky to “closed” portion.

• This way everyone can keep track of open and closed issues and effectively manage handoffs.

Category 1

Open

Closed

Verify deployment

between shifts

Ask CEO about employee focus (4)


31

Today’s Agenda





32

Interview Structure

• Begin the interview with:– Introduce yourself and why you’re there

• We will respect confidentiality of information• We will be looking for both Strengths and OFIs

– Ask if they have questions about the interview– Explain why you are taking notes– Describe the time limits

• End the interview with:– Do you have any questions for me?– Thank them for their time and honesty


33

Effective Interviewing Tips

• Ask open-ended questions, probe when necessary, bring it to a conclusion

• Script your lead and follow-up questions• Maintain eye contact and relaxed body language• One person interviews; backup takes notes• Backup may clarify or ask missed point• Refocus when they ramble – remind them of time

limits• Be frank and respectful

– Use the applicant’s terminology– Actively LISTEN to the applicant

• Be patient with shy interviewees


34

Asking QuestionsUse open-ended questions

•Tell me about …

•How do you …

•Help me understand …

•Please walk me through …

Avoid Leading questions.

• Why don’t you –

• At my office we do it this way. What do you do?

•Have you always done it this way?

•Why did you change?

•How did you know what to change?

•Don’t you think it would be a good idea to re-engineer your manufacturing process?

•How do you determine how frequently to check this process?

•Why do you only audit this process every two years?

•How do you know customer requirements are correctly defined?

•Why don’t you involve customers in your design-build teams?


35

Today’s Agenda





36

Prepare Exit Presentation

• Review the Key Themes Storyboard– Significant Strengths, including Pretty Good

Practices– Significant Opportunities for Improvement

• Prepare the PowerPoint presentation using the Exit Presentation Template

• Identify presenters


37

Conduct Executive Preview

• Applicant’s option• Team Leads only• Council representative also present• Summary review of Exit Presentation content• Opportunity for clarifying questions• Expression of thanks for cooperation and

participation• Followed by final modification of the Exit

Presentation if required


38

Deliver Exit Presentation

• One hour maximum (often shorter)• Moderated by MCQ representative• Audience determined by applicant• Review key themes

– Significant Strengths and Pretty Good Practices– Significant Opportunities for Improvement

• Clarifying Q&A• Thanks


39

Complete the Feedback

• Category owners prepare final wording of comments

• Team reviews and signs off on all comments• Team scores the items • Team identifies Pretty Good Practices• Team reviews judges’ questions

– Commitment– Sustainability– Momentum

Be sure to follow all writing rules!


40

Score Items

• Assign scoring facilitator• Determine order for scoring items• Allow 5-10 minutes maximum per item• Review scoring guidelines, terminology (see Clarifying

Requirements for Basic-, Overall-, and Multiple-Level Scoring)

• For each item– Category Owner reviews key points of comments– Team members individually determine scores– Team displays scores on facilitator’s signal– Team negotiates variances until consensus reached– Be mindful of item point values - Don’t bog down over minor

points!

• Record percent scores on appropriate scoring matrix Remember--It’s not over till it’s over


41

Scoring Calibration

• Remember from evaluator training?– Few: <15%– Some: 15% < 30%– Many: 30% < 50%– Most: 50% < 80%– Nearly All: 80% <100%– All: 100%


42

Scoring Diagram


43

Finalize the Feedback Report

• After leaving site (within one week)– Team Leads write Executive Summary based on Exit

Presentation using Key Themes Worksheet– Feedback editor:

• collates comments into the appropriate feedback report template (Business, Healthcare, Education)

• incorporates Executive Summary and Scoring chart

• distributes report to team for feedback

– Team lead forwards finished feedback report to lead judges

• Next steps (approximately two weeks)– Judges review and edit final report and forward to Council– MCQ forwards finished report to applicant


44

Judges’ Presentation Agenda

• Key Factors – 2 minutes• Key Themes: Process Strengths – 3 minutes• Key Themes: Process OFIs – 3 minutes• Key Themes: Results – 6 minutes• Scoring Summary – 2 minutes• Significant Differences (if any) – 1 minute• Commitment, Sustainability, and Momentum – 3 minutes• Questions – 10 minutes


45

Judges’ Presentation - Commitment

The extent to which leaders are willing to invest time, resources and energy in making the journey

An example of low commitment: leaders spend little time on quality or planning for improvements – few resources are made available for improving quality systems and approaches

An example of high commitment: leaders consistently encourage and lead efforts to improve the organization – leaders plan for high levels of performance – leaders allocate resources to the effort


46

Judges’ Presentation - Sustainability

The extent to which the organization is likely to continue making progress on its journey

An example of low sustainability: The quality journey may be seen as just a management fad – “something the boss wants us to do.” If the leader of the quality effort were to disappear from the scene, the journey would stop.

An example of high sustainability: “Managing quality is everyone’s job. We need to excel as an organization in order to survive and prosper. It’s not just the boss’ thing.” If there were a change in senior leadership, progress would continue.


47

Judges’ Presentation - Momentum

The direction and rate of progress on the quality journey

An example of low momentum: There may be slow or no progress overall from year to year, although some pockets of rapid or steady growth may exist. The organization may be little aware of its opportunities for improvement or how to close the gaps. Perhaps, improvements in the past may have been lost and are no longer apparent.

An example of high momentum: Innovation and improvement in business practice is frequent and pervasive. Baldrige scores may have been rising each year through multiple cycle of self-assessment and improvements.


48

Today’s Agenda





49

What Happens After the Site Visit

• Judging to assign a recognition level– Team leads present and respond to judges’ questions– All team members invited to attend and participate– Judges deliberate privately and assign award level

• Applicant receives feedback report• Improvement Planning Session

– Facilitated by council– Team leads walk applicant through report– All team members invited to attend and participate

• Celebration!


50

Other Tools to Help with Site Visit

• See the Evaluator Resource Center at: http://www.councilforquality.com/assess_eval_resource.cfm

• See the Site Visit Packet provided by the council prior to site visit

http://www.councilforquality.com/assess_eval_resource.cfm


51

Thank you for all your time and effort! The MQA couldn’t exist without you!

copyright 2002-2006 minnesota council for quality revision 08/21/06 1 minnesota quality award 2006...

Documents

minnesota council

quality revision

minnesota quality award

site visit trainingcopyright

length of site visit

team lead role

materialshelp team

exit presentation1a