We Have Measurement Standards, Now What?

VCU Social Media Institute July 20, 2015

Katie Delahaye Paine, CEO

Paine Publishing

PainePublishing.com

@queenofmetrics

measurementqueen@gmail.com

@queenofmetrics

Paine Publishing: Providing communications professionals the knowledge and

information they need to navigate the journey to good measurement.

Books

Measure What Matters Measuring the Networked

Non-Profit

Newsletters The Measurement Advisor Paine Publishing Insider

Tutorials

Consulting

Free resources

@queenofmetrics

First Came The Barcelona Principles

1. Importance of Goal Setting and Measurement

2. Measuring the Effect on Outcomes is Preferred to Measuring Outputs

3. The Effect on Business Results Can and Should Be Measured Where Possible

4. Media Measurement Requires Quantity and Quality

5. Earned Media Value/AVEs are not the value of Public Relations

6. Social Media Can and Should be Measured

7. Transparency and Replicability are Paramount to Sound Measurement

@queenofmetrics

Then came…

A published specification that:

establishes a common language;

contains a technical specification or other precise criteria;

is designed to be used consistently, as a rule, a guideline, or a definition.

All standards are available on http://painepublishing.com/standards-central-2/

Social Media Measurement

Standards: The Conclave

• Council of Public Relations Firms (http://prfirms.org/) • Global Alliance for Public Relations and Communication

Management (www.globalalliancepr.org/) • Institute for Public Relations (http://www.instituteforpr.org/) • International Association for Measurement and Evaluation of

Communication (http://amecorg.com/) • Public Relations Society of America (http://www.prsa.org)

Traditional Media Standards: The Coalition

@queenofmetrics

Why we need standards

48.94%

68.31%

88.64%

0.00% 20.00% 40.00% 60.00% 80.00% 100.00%

SDL

UberVu

NetBase

% Agreement with human coding

SDL UberVu NetBase

33.12%

58.00%

7.84%

11.95%

26.53%

13.00%

0.32%

9.46%

40.35%

30.00%

92.11%

78.00%

SDL

Beyond

NetBase

UberVu

% positive %negative %neutral

@queenofmetrics

3

Testing validity of criteria

@queenofmetrics

The Coalition has published, tested, and

validated Standards for:

Definition of an Item of Content 1

Circulation 2

Mentions 3

Sentiment 4

Quality Elements 5

For detailed definitions & methodology go to: http://painepublishing.com/wp-content/uploads/2014/03/Standard-Codebook.pdf and

http://painepublishing.com/wp-content/uploads/2014/03/7-Jackson-Sharpe.pdf

@queenofmetrics

The Conclave has published Standards for:

Content Sourcing & Methods 1

Reach & Impressions 2

Engagement 3

Influence & Relevance 4

Opinion & Advocacy 5

Impact & Value 6

For details go to: www.smmstandards.com

@queenofmetrics

Content Standard

All social media measurement reports should include a

standard “content sourcing and methodology” table that

helps clients know “what’s inside” the product for full

transparency and easy comparison (like a food nutrition

label).

@queenofmetrics

#SMMStandards – Sources & Methods Transparency Table www.smmstandards.com

Timeframe Analyzed

Research Lead(s)

Channels Analyzed

Data/Content Sources

Analysis Depth ☐ Automated ☐ Manual ☐ Hybrid ☐ All Content Reviewed ☐ Rep. Sample

Source Languages

Search Languages

Sentiment Coding ☐ Automated ☐ Manual ☐ Hybrid ☐ Manual Sampling: _____________________ ☐ 3-pt scale ☐ 5-pt scale ☐ Other scale ☐ At entity level ☐ Paragraph/doc level

Spam/Bot Filtering ☐ Automated ☐ Manual ☐ Hybrid ☐ Includes news releases ☐ Excludes releases

Metrics Calculation and Sources

-- Reach

-- Engagement

-- Influence

-- Opinion/Advocacy

Proprietary Methods

Search Parameters See full search string list on page ___ of this report

@queenofmetrics

#2 Standards for Reach & Impressions

All impression numbers are flawed for a variety of

reasons.

Multipliers should never be used.

A divider is more appropriate because it is less

than 5% of what is posted is actually seen.

The term “Potential Impressions” is preferred and

must be specific to a particular

channel – i.e.

For Twitter OTS is the number of first line

followers

For Facebook it is the number of likes to a

page

@queenofmetrics

#2 Items, Reach, and Impressions

ITEM= a post, micro-post, article, or other instance appearing for the first

time in a digital media.

MENTION (not recommended) = refers to a brand, organization, campaign,

or entity that is being measured.

REACH represents the total number of unique people (deduped) who had an

opportunity to see an ITEM or a valid reproduction of that ITEM across any

digital media.

IMPRESSIONS represent the gross number of items that could have been

seen by all people, including repeats. The term “displayed” applies across

channels, browsers, devices, and other methods by which an individual might

see an item.

12

@queenofmetrics

#3 Standards for Engagement

Engagement = some action beyond exposure…in response to content on

an owned channel – i.e. when someone engages with you.

Conversation = online or offline discussion by customers, citizens,

stakeholders, influencers or other third parties about your organization.

Any measure of Engagement and Conversation must be tied to the goals

and objectives.

Engagement and Conversation occurs offline and online -- both must be

considered.

Engagement should be measured by the % of your audience that is

engaged, and the % engagement for each item published.

@queenofmetrics

Standards for Engagement (continued)

Engagement counts such actions as: likes, comments, shares, votes, +1s,

links, retweets, video views, content embeds, etc. Engagement types and

levels are unique to specific channels but can be aggregated for cross-

channel comparison.

Engagement should be measured by the percentage of your audience

engaged by day/week/month; and the percentage of engagement for each

item of content your organization publishes.

Conversation counts such items as blog posts, comments, tweets,

Facebook posts/comments, video posts, replies, etc. Conversation types

and levels are unique to specific channels but can be aggregated for cross-

channel comparison.

14

@queenofmetrics

#4 Influence & Relevance

Adhere to WOMMA Standards

“Influence” is the ability to cause or contribute to a change in opinion or behavior

Influence cannot be expressed in a single score or algorithm

Should include some combination of the following elements:

Reach

Engagement around individual

Relevance to topic

Frequency of posts around the topic

If an individual scores a 0 on one element, they aren’t influential

@queenofmetrics

#5 Opinion & Advocacy

Sentiment is the feelings the author is trying to convey, often measured

through context surrounding characterization of object.

Opinion is a view or judgment formed about something, not necessarily

based on fact or knowledge. It is articulated and associated to the speaker.

Advocacy (noun) vs (verb) is publicly stated support for or

recommendation of a particular cause or policy. Advocacy requires a level

of expressed persuasion.

The key distinction between “advocacy” and “opinion,”

is that advocacy must have a component of

recommendation or a call to action embedded in it.

@queenofmetrics

#5 Standards for Measuring Tone/Sentiment

Whatever process is defined and applied, it must be used

consistently throughout any analysis.

Sentiment reliability varies by vendor and approach –

therefore coding definitions, consistency and transparency

are critical.

Opinions, recommendations, and other qualitative measures

are typically more valuable than raw sentiment and

increasingly measurable:

Opinions (“It’s a good product”)

Recommendations (“Try it” or “Avoid it”)

Feeling/Emotions (“That product makes me feel happy”)

Intended action (“I’m going to buy that product tomorrow”)

@queenofmetrics

#6 Impact & Value

Impact: The effect of a social media campaign,

program or effort on the target audience

Value: The impact expressed in either cost savings

or revenue incurred

ROI: Return on Investment. A financial

performance measure. To calculate ROI, the

benefit (return) of an investment is divided by the

cost of the investment; the result is expressed as a

percentage or a ratio

Any measure of Impact & Value must be tied to the

goals and objectives for your organization, brand,

or program

@queenofmetrics

Important Numbers to Remember

The percent of conversation that happens OFF

LINE 90%

The amount of conversations generated by

bots, spammers and pay-per-click sites 40%

The percent of on-line conversations that are

public 10%

The percent of Facebook & Twitter posts that

are actually seen < 5%

The percent of online ads that are ignored 82%

The number of times per hour Digital Natives

switch media—every 2.2 minutes 27

@queenofmetrics

We need a new Model

Assumed ROI

Other Paid Marketing

Digital/

Online Media

Buy

Print/TV Media

Buy

Real ROI

Digital/ Online

Media Buy

Print/TV Media Buy

Earned

Media/

PR

Owned

Media

Earned

Media/PR

Owned

Media

@queenofmetrics

Eyeball counting

HITS Outcomes

MSM

Online

Social Media

Impressions are not awareness.

Where’s the “So What?”

21

“Counting impressions is like counting

sperm…”

@queenofmetrics

6 steps to standards-compliant measurement

Step 1: Define your goal(s).

What outcomes is this strategy or tactic going to

achieve?

What are your measurable objectives?

Step 2: Define the parameters

Who are you are trying to reach? How do your

efforts connect with those audiences to achieve the

goal?

Step 3: Define your benchmarks

Who or what are you going to compare your results

to?

Step 4: Define your metrics

What are the indicators to judge your progress?

Step 5: Select your data collection tool(s)

Step 6: Analyze your data

Turn it into action, measure again 22

6 Steps

to Success

1

2

3

4

5

6

@queenofmetrics

Why do we communicate?

7/21/2015 23

Outtakes

(Intermediary Effects)

• Awareness

• Knowledge/Education

• Understanding

Outcomes

(Target Audience Action)

• Engagement

• Advocacy

• Revenue/Cost Savings

Activities How does what you do

contribute to the bottom line?

@queenofmetrics

Goals, Actions, and Metrics

24

Goal

Increase

Marketable

universe

Increase

preference

Action

Conduct Media

Event

Increase social

and online

message

penetration

Outcome Metric

% increase in marketable

universe

% increase in favorable

perceptions & awareness of

brand attributes

Activity Metric

% of items containing

key messages

% increase items

containing one or more

key messages

7/21/2015

@queenofmetrics

Step 2:Get everyone on the same page

Be inclusive

Include anyone who will see your

reports

Be clear about definitions:

Engagement?

Success?

ROI?

Don’t reinvent the wheel

Understand what data already

exists.

Make it interactive

Follow up

25

@queenofmetrics

Step 3: Establish benchmarks

Past Performance Over Time

Think 3

Whatever keeps your

leadership up at night

26

@queenofmetrics

Photo of the Week, SOWC Drive

Comments & Retweets

SOWC2012

27

@queenofmetrics

Step 4: Why you need a Kick-Butt Index

The Perfect KBI

Is actionable

Is there when you need it

You become what you measure, so

pick your KBI carefully

Continuously improves your processes

Gets you where you want to go

28

@queenofmetrics

The Path to the Mission

High-quality media coverage

Increase in intent

Change belief

More “heads in beds”

29

@queenofmetrics

Step 4: Matching Goals to Metrics

KBI

Increase on-message media

presence

Increase in intent to visit

Increase consideration as a fun

place to go with friends

Metric

% increase in KBI (via media

analysis)

% increase in downloads of

visitor guide

% increase in perception of AC

as a fun place to go with friends

30

@queenofmetrics

Kick Butt Quality Score

Desirable Criteria Score Undesirable Criteria Score

Contains a key message

3.50 Negative message -2.0

Contains a desirable visual

1.00 No call to action -2.0

Contains a third-party recommendation

2.50 Contains an undesirable visual -1.0

Contains a call to action

1.00

A story or a headline that leaves

the reader less likely to visit -3.0

The story or headline leaves a reader more

likely to visit 2.00

Recommends visiting somewhere

else -2.0

Total

10.00 -10.0

31

@queenofmetrics

Typical Social Engagement Index

Action Score

Like/Follow/Opens/+1/Favorite 1

Shares content (including retweets, forwards, shares, etc.) or shares a link to a

website or page 2

Generates sign-up to receive email or other content or follows a link to content 2

Share or comment 2

Positive comment (leaves a reader more likely to support ) 1

Views a complete video 2

Total 10

32

@queenofmetrics

Step 5: Selecting a measurement ool

Objective

Increase inquiries,

web traffic, recruitment

Increase

awareness/preference

Engage

Volunteers

Communicate

messages

KBI

% increase in traffic

#s of click thrus or downloads

% of audience understanding your messages

% increase in engagement index

Total opportunities to see key messages

Cost per opportunity to see key messages

Tool

Web Analytics: Google Analytics,

Omniture, Web trends

Survey: SurveyMonkey, or Mail

Web analytics or Content

Analysis: Facebook Insights, Convio,

Omniture, Google Analytics

Media content analysis, Survey Research

33

@queenofmetrics

Step 6: Be Data Informed, not Data Driven

Rank order results from worst to best

Ask “So What?” at least three times

Put your data into an overall framework

consistent with C-Suite expectations

Find your “Data Geek”

Compare to last month, last quarter, 13-

month average

34

@queenofmetrics

Photo Event

High Quality Content

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Level of Engagement

Ver

y H

igh

Med

ium

Hig

h

Low

High Resources

Low Quality Content

Low Resources

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

What gets the most bang for the buck?

Resource Use

Low

Hig

h

Med

ium

Ver

y h

igh

Ver

y H

igh

Med

ium

Hig

h

Low

Webinar

Status update

Link

Ultimate Road Trip

Google + Chat

Media Day

Corporate Video

@queenofmetrics

ACA has made a demonstrable difference in the

quality of media coverage

If an ACA or one of its programs

was mentioned in a media story,

it was:

More likely to contain an

endorsement

More likely to be positive

More likely to contain key

messages

Less likely to be negative

0.22

0.00

0.62

1.42

4.61

0.08

0.08

0.01

-0.61

-0.66

Endorsements

Positive Visual

Negative Visual

Tone

MessageCommunication

Average of OCS

Differences in Quality of Coverage with and without

ACA

Atlantic City Atlantic City Alliance

@queenofmetrics

ACA programs drive higher OCS scores, which

correlate highly with web visits

0.41

0.44

0.47

AC items

ACA Items

ACA OCS Scores

Correlations between Web Visits and PR Metrics

Pearson r. value

@queenofmetrics

Free entertainment generated the highest OCS

Scores in Q2

5.26

5.73

6.36

6.54

6.71

7.15

7.31

7.36

7.50

7.67

8.00

8.10

8.68

9.30

Miss America

Meet AC

DO AC

July 4th Fireworks

Air Show

Blake Shelton

Miss'd America Pageant

Sand Blast

Hello Summer

Boardwalk Hall Light Show

Challenge Triathlon

Sand Sculpting World Cup

Lady Antebellum Concert

Free Entertainment

Top Programs By OCS

@queenofmetrics

ACA maintains high OCS score with proactive

program coverage Lady Antebellum,

Non-Gambling Fun, Hello Summer, GMA segment

14 44 31

93 87 75 75

40

95

28 7

27 17 18 12

107 122

254

5.5

7.18

6.3

5.02

6.24 6.42

7.42 7.21

6.54

5.27

4.25

5.08

6.47

5.94

5.33

4.75

6.87

6.41

0

1

2

3

4

5

6

7

8

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May June

0

50

100

150

200

250

300

OC

S Sc

ore

Nu

mb

er o

f m

enti

on

s

ACA Mentions vs. OCS Over Time

ACA Mentions ACA OCS

Miss America Statue, ACA

new campaign, Blake Shelton, World Poker

Championship

@queenofmetrics

14 39

23 36

133

235

8 7 4 2 10 15 8 11 1 2 9 2 24

70

21

73 88 110

13

104 74

138

87

48

Jan Feb Mar Apr May Jun

Qtr1 Qtr2

2014

Atlantic City has something for everyone

Atlantic City is a clean and safe place to visit

Atlantic City is a year-round destination spot

Atlantic City is making a comeback

Atlantic City offers more than just gaming

Communication of the "Making a Comeback" &

"Something for Everyone" messages increased as

ACA announced new programs

@queenofmetrics

PR is having a positive impact on preference and

perceptions

Based on the results of the most recent brand tracking

survey, it appears that people who say they have recently

heard news about Atlantic City are very likely to perceive

AC as fun – a key driver of preference.

Respondents who remember seeing news reports about

Atlantic City are also very likely to associate key messages

and positioning statements such as “place I am excited to

go to” and “fun place to hang out with friends.”

The same respondents are also more likely to recommend

Atlantic City to friends.

The largest percentage of respondents who remember

seeing news about Atlantic City don’t remember where

they saw it. If they do remember, most saw news in

newspapers and online media.

@queenofmetrics

Thank You!

For more public relations and social media measurement education,

check out our Measurement Classroom

Also check out The Measurement Advisor , our bi-monthly newsletter

Email me: measurementqueen@gmail.com

Find me on Twitter: @queenofmetrics

Call me: 1-603-682-0735

7/21/2015 42