11

Handling Conflict of InterestHandling Conflict of Interestin Utility Evaluation Studiesin Utility Evaluation Studies

Handling Conflict of InterestHandling Conflict of Interestin Utility Evaluation Studiesin Utility Evaluation Studies

Presentation/Discussion TopicCALMAC Meeting – October 19, 2005

Douglas Mahone, Heschong Mahone Group, Inc.with support from: Matt Brost, RLW

Richard Ridge, Ridge & Assoc.Kathleen Gaffney, KEMA

22

Common Points of AgreementCommon Points of Agreement

• Evaluation must be credible, disinterested, Evaluation must be credible, disinterested, independent and professional, and this can’t independent and professional, and this can’t be done when there are conflicts of interest.be done when there are conflicts of interest.

• Important that the evaluation studies Important that the evaluation studies for utility administrators be done rightfor utility administrators be done right

• For evaluators, this is central to their For evaluators, this is central to their credibility and professional standingcredibility and professional standing

33

Issues for CALMAC DiscussionIssues for CALMAC Discussion

• Review CPUC Decision 05-01-055*Review CPUC Decision 05-01-055*• Types of evaluation studiesTypes of evaluation studies• Firewall between evaluators & implementersFirewall between evaluators & implementers• Rules for utility evaluation studiesRules for utility evaluation studies

• Review proposal: conflict of interest scoringReview proposal: conflict of interest scoring• Explicitly address in contractor selectionExplicitly address in contractor selection• More fair, transparent, fact-based than simple firewallMore fair, transparent, fact-based than simple firewall

• Decide course of actionDecide course of action• Adopt, reject, amend, leave alone, etc.Adopt, reject, amend, leave alone, etc.

*Interim Opinion on the Administrative Structure for Energy Efficiency: Threshold Issues, January 27, 2005 *Interim Opinion on the Administrative Structure for Energy Efficiency: Threshold Issues, January 27, 2005

44

CPUC’s “Impact Studies” TypeCPUC’s “Impact Studies” Type

• For PY 2006 and beyondFor PY 2006 and beyond• Energy Division manages & contractsEnergy Division manages & contracts• EM&V studies used to:EM&V studies used to:

• Measure & verify savingsMeasure & verify savings• Generate data for savings & cost effectivenessGenerate data for savings & cost effectiveness• M&E achievements for “performance basis”M&E achievements for “performance basis”• Evaluate whether goals are metEvaluate whether goals are met

• Firewall: These evaluators cannot do Firewall: These evaluators cannot do “program delivery work”“program delivery work”

55

Utilities’ “Program & Market” Studies TypeUtilities’ “Program & Market” Studies Type

• CPUC terminology: “Program design evaluation CPUC terminology: “Program design evaluation and market assessment”and market assessment”

• Purpose: Provide information feedback Purpose: Provide information feedback • to administrators and implementersto administrators and implementers• to improve program performance to improve program performance

• Excluded from the firewall – “much less relevant” Excluded from the firewall – “much less relevant” for these types of studiesfor these types of studies

• But… ED makes final selection of contractors, But… ED makes final selection of contractors, consulting with “ad hoc” committeeconsulting with “ad hoc” committee

66

Why Our Concern?Why Our Concern?

• Despite Decision, firewall may be applied Despite Decision, firewall may be applied anywayanyway• Twice, PG&E has started to apply it Twice, PG&E has started to apply it

incorrectly, andincorrectly, and• Twice has corrected itselfTwice has corrected itself• ““Safe” way to goSafe” way to go

• People with CA market experience would People with CA market experience would be unavailable to utilities for market studiesbe unavailable to utilities for market studies

• Need to formalize an alternative approach, Need to formalize an alternative approach, and apply it consistentlyand apply it consistently

77

Ambiguous Conflict Scenarios (just a sample)Ambiguous Conflict Scenarios (just a sample)

• University scenarioUniversity scenario• Mechanical engineering professor consults on a A/C Mechanical engineering professor consults on a A/C

efficiency program (implementation)efficiency program (implementation)• Statistics professor consults on sample design for impact Statistics professor consults on sample design for impact

study of same (EM&V)study of same (EM&V)• Firewall says this is a conflict (but it’s probably not)Firewall says this is a conflict (but it’s probably not)

• Vendor scenarioVendor scenario• Entity performs EM&V studiesEntity performs EM&V studies• Same entity sells electric metering equipment or billing Same entity sells electric metering equipment or billing

software to utilities (not implementation)software to utilities (not implementation)• Firewall says this is not a conflict (but maybe it is)Firewall says this is not a conflict (but maybe it is)

• CEC PIER contractor scenario – CEC PIER contractor scenario – is it implementation?is it implementation?

88

Types of ConflictsTypes of Conflicts

• Direct Financial ConflictDirect Financial Conflict – direct $$ stake – direct $$ stake in evaluation outcomein evaluation outcome

• Indirect Financial ConflictIndirect Financial Conflict – future business – future business advantage based on evaluation outcomeadvantage based on evaluation outcome

• Personal, professional, political biasPersonal, professional, political bias - bias - bias or interest in evaluation outcomeor interest in evaluation outcome

99

Dealing with ConflictsDealing with Conflicts

• EverybodyEverybody has bias, and some degree of conflict has bias, and some degree of conflict of interestof interest

• Firewall only addresses:Firewall only addresses:• Direct Financial ConflictDirect Financial Conflict• One variety of Indirect Financial Conflict One variety of Indirect Financial Conflict

(hope of future implementation work)(hope of future implementation work)

• Other types of conflicts ignoredOther types of conflicts ignored• American Evaluation Association – Guiding American Evaluation Association – Guiding

Principles for Evaluators - addresses conflictsPrinciples for Evaluators - addresses conflicts• Conflicts must be disclosed and Conflicts must be disclosed and managedmanaged

1010

RecommendationsRecommendations

• Full Disclosure – require entities to fully disclose Full Disclosure – require entities to fully disclose all potential conflictsall potential conflicts• Should even extend to proposal review committeesShould even extend to proposal review committees

• Model on CPUC RFP (see attachment)Model on CPUC RFP (see attachment)• Automatically disqualify if direct financial Automatically disqualify if direct financial

conflictconflict• Include in scoring of proposalsInclude in scoring of proposals

• highly experienced with insignificant conflict highly experienced with insignificant conflict could outscore could outscore inexperienced with no conflictinexperienced with no conflict

1111

Open, Transparent ScoringOpen, Transparent Scoring

• Establish Conflict Guidelines - Establish Conflict Guidelines - see AEA Guiding Principles to startsee AEA Guiding Principles to start

• Case-by-Case Review – part of normal Case-by-Case Review – part of normal contractor selection processcontractor selection process

• Dispute Resolution – Open, consistent, Dispute Resolution – Open, consistent, timely process to address conflict & biastimely process to address conflict & bias

1212

CPUC Disclosure LanguageCPUC Disclosure Language

• Rationale – Conflicts may: Rationale – Conflicts may: • Prevent “impartial, technically sound, objective Prevent “impartial, technically sound, objective

assistance and advice”assistance and advice”• Result in a biased work productResult in a biased work product• Result in an unfair competitive advantageResult in an unfair competitive advantage

• Required disclosure of:Required disclosure of:• Current & prior contracts/financial relationshipsCurrent & prior contracts/financial relationships

($$ involved, duration, nature of service, past 3 yrs)($$ involved, duration, nature of service, past 3 yrs)• Any current business (including active proposals)Any current business (including active proposals)

• Signed statement on disclosureSigned statement on disclosure

1313

CPUC Scoring (example)CPUC Scoring (example)

• 20% - Experience with technical content20% - Experience with technical content• 20% - Work plan20% - Work plan• 10% - Resources available10% - Resources available• 10% - Ability to execute10% - Ability to execute• 10% - Experience with utility projects10% - Experience with utility projects• 10% - Familiarity with public purpose progs10% - Familiarity with public purpose progs• 15% - Potential for conflicts of interest15% - Potential for conflicts of interest• 5% - Education and credentials5% - Education and credentials

1414

Third EM&V Type: “Research & Analysis” StudiesThird EM&V Type: “Research & Analysis” Studies

• Grey areas:Grey areas:• Technical Potential StudiesTechnical Potential Studies• DEER Database DEER Database • Net-to-Gross Ratio AssessmentsNet-to-Gross Ratio Assessments• Best PracticesBest Practices

• Unclear how firewall rule applies to theseUnclear how firewall rule applies to these…or how much of an issue it would be…or how much of an issue it would be

• But… will be managed by EDBut… will be managed by ED• Recommend conflict scoring, not firewallRecommend conflict scoring, not firewall

1515

…so what should CALMAC do?…so what should CALMAC do?

• Unofficial body, but respected evaluation Unofficial body, but respected evaluation advisory groupadvisory group

• Only sitting body with insight and Only sitting body with insight and knowledge to address this issueknowledge to address this issue

• Recommendation will help utilities contract Recommendation will help utilities contract for their evaluation servicesfor their evaluation services

• Statewide consistent method will be better Statewide consistent method will be better than many different approachesthan many different approaches