training evaluatio sixth level

www.institutomvc.com.br

Beyond the Fifth LevelThe Forgotten Dimension

Session Number TU 3155/18/2010 4:00 PM - 5:30 PM


Speaker’s profileJOÃO BAPTISTA VILHENAPartner of the Instituto MVC, a Brazilian consulting company. Twenty nine years of professional experience in the areas of T&D, Management, Marketing, Negotiation, Sales & Distribution. Author of 5 books. Speaker of ASTD in 2006 and 2007. [email protected]

LUIZ AUGUSTO COSTACURTA JUNQUEIRAPartner of the Instituto MVC, a Brazilian consulting company. A Management Education Consultant with thirty five years of experience. One of the brazilian pioneers in Management Negotiation . Author of 8 books. Speaker of ASTD in [email protected]


Objectives

SHARE with you a different approach to evaluate training programsDISCUSS how to go beyond the traditional ROI analysis

Our intention is not toDeemphasize the importance of the traditional 5 levelsIgnore the importance of the participants of the training

program


Agenda• The challenge of evaluating something• Factual data• How to know the root cause of your client• Process design• Evaluating the boss opinion• Wrap-up


In teams of 5 or 6 (closest to you)

You have 5 minutes to put yourselves in the audience shoes and discuss 3 criterias to evaluate a samba school

parade


First of all, keep in mind that

To evaluate something it is important• Know why you will evaluate (objective)• Know what you will evaluate (criterias)• Know how you will evaluate (tools)


Agenda

• The challenge of evaluating something• Factual data• How to know the root cause of your client• Process design• Evaluating the boss opinion• Wrap-up


Samba schools evaluating criteriasEnredo (plot or theme)

This is the general theme that motivates the whole presentation of a School. It has to be clear and understandable. Enredos change from year to year.

Samba-enredoSamba is a general denomination for a type of rhythmic music derived from the African influence in Brazil and typical of urban Rio. It's meant to be sung and danced. Samba-enredo is a samba-song that tells a story, a plot derived from the main theme of the School.

Bateria (percussion)It's the pulse, the beating of the School. It has to sustain the rhythm that drives the School's song and dance. Each Samba School has from 250 to 350 percussionists.


Samba schools evaluating criteriasPorta-bandeira e mestre-sala (the flag bearer and the master of ceremony)

This couple is of great importance for the School. The School's flag bearer is a woman and her male scort is the master of ceremony. She carries the flag and he shows his dance abilities. Their special choreography must present all their lightness and nimbleness, sometimes almost ballet-like.

Comissão de frente e alas (front commission and wings)Traditionally a School's board of directors marches at the head of the parade, often accompanied by TV stars or sports idols. Next, comes the comissão de frente, a group of 10 to 15 people in costume which opens open the parade introducing the theme of the School and performing an original choreography. The bulk of the other members comes behind them, distributed in different alas (wings).Each wing is composed by hundreds of people wearing the same costume, which is related to a point of the Samba plot.


Samba schools evaluating criteriasHarmonia (harmony)

Refers to the degree of synchronicity among the bateria (percussion), the singing of all members and the dancing of the thousands of passistas (samba dancers). Sambar means "to dance to the samba music".

Evolução (evolution)Refers to the quality of the dance and to the way it evolves in tune with the rhythm of the music and the beat of the percussion. This is translated by the choreography, the passistas (samba dancers) skills and the general enthusiasm of all School members.

Alegorias e adereços (allegories and ornaments)The gigantic and richly decorated floats carry some of the figuras de destaque, that is, celebrities or personalities to which the School pays homage. Usually they are famous TV people or sports idols invited to wear the Schools costumes. There can't be more than 8 floats per School. Floats are judged by their beauty and creative concept.


Samba schools evaluating criteriasAla das baianas (bahian ladies wing)

Traditionally each School has one. It usually comes in the middle or the end of the parade. It is a wing with hundreds of women, usually elderly ladies, dressed in round large skirts in remembrance of the Bahian former slaves, who introduced many of the traditions of Rio Carnival. During the year these ladies work for the Samba Schools sewing and making the costumes.

Fantasias (costumes)Costumes are judged by their originality and beauty. Their colors are as usual the traditional ones adopted by each School.


See the chart below

Inputs Outputs

Inputs Some kind of performance gapProcess T&D programs and evaluationOutputs Performance gap solution

The Training Flow

The boss The boss

Training program

5 levels

T&D


Agenda• The challenge of evaluating something

• Factual data• How to know the root cause of your client• Process design• Evaluating the boss opinion• Wrap-up


The first questionHow to be sure you know exactly

what is the boss’s problem?• People knowledge

» Desire Classical T&D problems» Power

• Process• Structure T&D is not able to solve this

kind of problems

• Strategy


Agenda• The challenge of evaluating something• Factual data

• How to know the root cause of your client• Process design• Evaluating the boss opinion• Wrap-up


If you are sure you have a T&D problem, the next step is design

the solution

• T&D people loves to design process• T&D people are used to think the process is the

solution (because they believe in people transformation)

• T&D people prefer the easier way• T&D people knows how to deal with the target

audience


So we have finished the training program. Now it’s time to verify if

we have achieved the objective• The 5 levels of evaluation (reaction, learning,

behavior aplication, business impact and ROI) always focus on the participant

• We won’t say never but it’s not common T&D people came back to verify if the boss is pleased with the training program (perhaps they don’t know, exactly, how to deal with bosses)


As you probably have realized• The sixth level is ask if the boss had achieved his

objetives using the training program as a tool (or how important has the T&D program been?)

• To ask this question you have to be sure you have done the right thing (and be prepared to listen that it had not been enough)


Hello Houston, we’ve get a Problem!• And what if the boss isn’t a person? (let’s suppose

your task was to create a sense of social responsability)

• Which will be the best criterias?


How do you know the root cause of the boss?

• Define clearly who is the boss (the one who will evaluate your job)

• Analyse cause X effect• Finding the root cause• Avoid generalizations: everybody, nobody, many, few• Check with the boss if you have really put your finger

in the wound• Be sure you know exactly what is the desired output

of the process


Designing the process

• Don’t take it for granted T&D is the best solution• Avoid loop holes• Be sure the participant knows what is he doing here• Be prepared to use 5 level’s model to analyse process

results


Agenda• The challenge of evaluating something• Factual data• How to know the root cause of your client

• Process design• Evaluating the boss opinion• Wrap-up


Evaluating the boss opinion or Going beyond the fifth level

• The ceteri paribus principle• Isolating variables• From general to specific• Using the 2H5W method: one possible way


Agenda• The challenge of evaluating something• Factual data• How to know the root cause of your client• Process design

• Evaluating the boss opinion• Wrap-up


Ceteri paribus principle• Ceteri paribus is a latin expression. In a free

translation it means isolating other variables• How to do it?


Methods to isolatethe effects of

training

Using Control Group

Trend Lines

Participants Estimate

Supervisors of participants estimates


Control groups• A control group arrangement can be used to isolate

the training impact• With this strategy, one group receives training, while

another similar group does not receive training• The difference of performance of the two groups is

attributed to the training program.


Trend lines• Trend lines are used to project the values of specific

variables if training had not been undertaken• The projection is compared to the actual data after

training and the difference represents the estimate impact of training


Trend projection

Sales after the training

Training programs

Volume of sales

time


Participants and Supervisors of participants estimate the

training’s impact• This method rests on the assumption that participants (and

their supervisors) are capable of estimating how much a performance improvement is related to the training program.

• Because their action have produced the improvement, participants (and their supervisors) may have very accurate input on the issue.

• They should know how much of the change was caused by applying what they have learned in the program.


Typical Questions to Estimate• What percentage can this improvement be attributed

to the application of skills/techniques/knowledge gained in the training program?

• How reliable are these estimates expressed as a percentage?

• What other factors have contributed to performance improvement?


Isolating the effects of Training


From general to specificSources of barriers to training success

• Lack of reinforcement on the job• Reentry problems (environment, ex.)• Nonsupportive organizational culture• Participants perception of impratical training

programs• Lack of commitment of the trainer with the

program


The boss roles• Collect baseline performance data• Transform training topics into performance standards• Involve supervisors and participants in needs analysis

process and program planning• Brief participants on the importance of training• Review instructional content and materials• Plan to participate in training sessions• Encourage participants to attend all sessions


Agenda• The challenge of evaluating something• Factual data• How to know the root cause of your client• Process design• Evaluating the boss opinion

• Wrap-up


Wrap-up• It’s not enough to evaluate the process• Roles: the boss, T&D, participants• Be sure you’ve “put your finger in the wound”• Don’t take it for granted that T&D is the only solution• Use 5 levels to evaluate the process• Choose a way to evaluate the boss satisfaction (which

means the problem has been solved)• Never forget the chicken principle


THANK YOUSee you in BRAZIL

If you wish to have a copy of this presentation, please access www.institutomvc.com.br

training evaluatio sixth level

Documents