analytics academy 2017 presentation slides

WELCOME TO ANALYTICS ACADEMY!

A/B Testing and Creating a Culture of Experimentation

A/B experiments show users different versions of your site and then compare results

TEST FIRST: FAST

● Can often mock up a feature in the testing tool first, without involving a tech queue

● Measuring results isn’t affected by seasonality, or other marketing efforts, or changes to the consumer mood because you are testing one group randomly divided, so all these factors are controlled for

● Results are statistically tested and validated

BUILD FIRST: SLOW

● Higher up-front investment: Have to invest in building the feature without knowing if it will work

● Hard to measure results: you roll out the feature and compare conversion before (5-7% for last two months) to after (5.5-6.7% for month after). Is this an improvement or normal variation? Is it affected by seasonality? Did the email campaign that went out last week affect this rate?

Benefits of A/B testing

Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more

learning-centered phrase will produce more click throughs

What’s the result?

Experiment 1: Tools in StoreHypothesis: ● Replacing "Tools" with a more

learning-centered phrase will produce more click throughs

What’s the result?● Learning Tools click rate up 16%● Teach Yourself click rate up 27%

Hypothesis: ● Changing to one year would have

no negative effect on subscription conversion

What’s the result?●

What we think●

Experiment 2: One year vs 10 issues

Hypothesis: ● Changing to one year would have

no negative effect on subscription conversion

What’s the result?● One year variation subscription

rate up 9%

What we think● People understand the value of a

year more than number of issues

Experiment 2: One year vs 10 issues

Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would

increase product purchase

What’s the result?●

What we think●

Experiment 3: Formatting on product pageHypothesis: ● Improving formatting would

increase product purchase

What’s the result?● No statistical difference in

product purchase

What we think● Was formatted description too

long? Should we have short text and preview page?

A/B testing can inspire cultural change● Practice with A/B tests builds experimentation muscles

○ People practice the steps to build a good experiment so they start to feel obvious

○ A/B tests require good methodology: you are forced to pick a goal to measure; you automatically have a control group; the software collects and reports on the results

○ The benefits of the speed/clarity from these experiments increase demand for similar speed/clarity in areas outside the website

Ideal state for all business stakeholders for all questions: always ask, “Can this be an experiment?”

Basic human nature makes this hard

● Short term, it feels easier to make a decision based on gut feel, or defer to highest paid person’s opinion (HiPPO), or just try something and see what happens without formalizing a hypothesis or measuring the result (but still call it an experiment)

What makes it hard to experiment?

In the long run, it is actually a LOT easier to run an experiment

● Fail fast: have an idea? Experiment with a minimum viable product to see if the idea deserves further development--or not

● Decisions vs more discussion: The organization can move a lot faster when there’s certainty around a course of action. When there’s uncertainty, healthy discussion can sometimes sour into multiple meetings and prolonged debates, or, perhaps worse, unspoken doubts sap the momentum for the group moving forward

Experimentation done well becomes self-reinforcing, as people see how much easier/faster they can work

A/B testing shows experiments are easier

● New product development: set out hypotheses about the market (e.g. “managers want to buy HBP materials to help their direct reports”) and then test with customers (e.g. customer interviews where we learn that there’s an equally large market from coaches). Key is to build and test in stages, so you validate hypotheses along the way

● Email testing: split list as randomly as possible and send different emails to the two groups

● Before and after testing: Create an insider newsletter and compare subscriber engagement before and after

● A/B testing: use a formal A/B testing platform on the website

Some ways that HBR experiments

Change is as good as a rest: Sometimes a change tests well just because it is a change and gets people’s attention. Change a button from red to blue, you may get higher click throughs; effect diminishes over time, then six months later change it back to red and get higher click throughs

Focus on the big picture: don’t look at a change in isolation; look at the total impact. Adding a newsletter widget that gets clicks is good, but does it increase the total newsletter signups (or just cannibalize the clicks you are getting from other widgets)? Does the additional visual clutter lower overall engagement (higher bounce, lower time on site)?

Follow some best practices

But remember we are not a labHaving a culture of experimentation does not mean that your group transforms into a medical lab where we need 98% certainty and huge sample sizes to make a decision

Perfect is the enemy of the good: It’s better to have 20 good experiments than 3 perfect ones

Hurdle: is the experimental results better information that what you would have used otherwise? (e.g. better than gut instinct?)

Questions?

ResourcesGoogle analytics experiments: FREE! (but anecdotally pretty hard to use)

Optimizely: easy interface, great training resources to help you get acquainted with testing (we started here)

VWO: may be cheaper than Optimizely

Adobe Target: more robust, integrates with Adobe Analytics in a very powerful way (we moved here in January)

Facebook Ads:Audiences and ImpactJoseph Casciano, Harvard Public Affairs and Communications

How much to spend?

Why are we spending it?

● Low-value objective● Larger audience

● High-value objective● Smaller audience

Problems with Interest Targeting

● Often inaccurate, so serves your ad to the wrong people.

● Hard to get a precise, smaller, well-targeted audience.

● To big audiences, Facebook serves your ad to whoever is cheap and easy.

Business Manager

↓

Assets

↓

Audiences

↓

Create Audience

↓

Custom Audience

Metric Time

Relevance Score

CPM (cost per 1,000 impressions)

Results and Reach

Custom audiences make metrics matter

● Don’t provide contextless numbers about faceless masses.

● Tell concrete, true stories about your valued audience.

● So: “We reached half of our email subscribers on Facebook, half of who watched the video for more than three seconds.”

● Not: “We reached 132,674 Malaysian bots who couldn’t even theoretically fly into Cambridge for our symposium.”

● Use custom audiences○ To guide your budget○ To make metrics matter

Questions?

[email protected]

Google Analytics Tips Tricks

Elizabeth Brady, EWB Analytics

IntroElizabeth Brady, Founder & Principal Web Analyst - EWB Analytics LLC - launched

March 2010

Specialties: Google Analytics and Google Tag Manager implementations, site audits,

web analytics support during site re-launch, measurement strategy and ongoing

analysis

Harvard groups I have collaborated with since 2012: Digital Communications, Harvard

Alumni, Harvard Admissions, Harvard Innovation Lab, Kennedy School, Harvard

Library, Harvard Learning Portal, HWPI, Ash Center

Contact: [email protected]

Keep It Clean

Filters Can Help PreventInternal Traffic

m 1

Maintain ‘exclude’ filter of

known internal IP

addresses

Ghost Spam

Maintain ‘include’ filter of

valid site hostnames

Crawler Spam

Maintain ‘exclude’ filter of

list of known spam

referrers

Filter: Exclude Internal Traffic by IP AddressInternal traffic inflates conversions & conversion rates. Check

current IP address by visiting whatismyip.com

IP AddressesUse regular expressions for a range of IP

addresses (ask IT for office IP range)

Dynamic IP Addresses

(residential)

Verify/update regularly

Test ActivityUse custom dimensions to track test users

even when not on internal network

https://www.whatismyip.com/

Filter: Test Accounts by Custom Dimension● Use a custom dimension to

identify a test visitor who

visits a specific internal page

(webadmin, test, etc)

● Set the custom dimension at

the ‘user’ level

● Create a filter for any traffic

with that custom dimension

value

Ghost referrer spam:● Never actually visits your site● Sends data via the ‘measurement profile’

randomly to your GA account (became an issue only with Universal Analytics)

● Sends data with a missing (not set) or inaccurate hostname

● Can be prevented with a valid ‘hostname’ (include) filter

Prevent Ghost Referrer Spam

Filter: Valid Hostname(s)

Language Spam - New Spam in 2016Language Spam

● Rather than referrers, the spamming

sites inserted spam messages into the

‘language’ reports

● Most do not use valid hostnames so

this would also be prevented with a

‘hostname’ include filter

● Additional exclude filters can be added

to address language spam

Lowercase Filters● Especially when starting a new

view, lowercase filters can avoid

capitalization inconsistencies

● Recommended for - page,

campaign

(medium/source/campaign),

search term (on site search)

Query String Cleanup● Google Analytics includes any query string

parameters (after the ‘?’ as part of the URL)

● Leads to multiple versions of the same ‘page’

and a challenge aggregating data

● Parameters to exclude can be identified in a

list in the view settings, or you could ‘go

nuclear’ and exclude them all with this filter

on the right

● Full URL with query strings (or just query

strings) can be captured as a custom

dimension to be viewed when needed

Check Your Setup

Referrer Exclusion List (Property Level)● Make sure your site subdomain/s is

included (new properties set up with

Universal Analytics will have this set

up on creation but any older site rolled

over to Universal Analytics did not

automatically have this configured)

● Include any off-site flows (login

validation, back-end sites like

pin1.harvard.edu) to prevent triggering

a new session

● Do not set up harvard.edu in the

exclusion list (that will prevent any

other Harvard sites showing up as

‘referrals’) - they will be ‘direct traffic’

Check ‘Exclude Bots & Spiders’ (View)● Excludes traffic from sites on the IAB

(Interactive Advertising Bureau) list of

known bots & spiders

● Sometimes these can be contracted

services like site response time (like

Gomez) that execute javascript and

would otherwise show up in reports

● It is recommended to leave this

unchecked for the unfiltered view

Link Google Search Console (Property) for Organic Search Trends

● Google Analytics no longer has much

insight into Google organic search

keywords

● Link your site’s Search Console

(formerly ‘Webmaster Tools’) account

for impressions/clicks on Google

Remember Campaign ‘Timeout’ (Property) is Configurable

● Standard campaign setting is 6 months

(sessions and conversions will be

credited to the last campaign in the

past 6 months)

● This explains why you may see

traffic/conversion for ‘old’ campaigns

● Your business group may decide you

need a shorter or longer campaign

timeout

Data Import (File Upload) Can Extend Analysis

● Data import lets you append

data to any dimension (standard

or custom) you collect

● The actual import can be a

simple text file upload

● Some uses might be to add

details around campaigns, add

authors or other details to

content pages, or group

information differently than

they way it is grouped in Google

Analytics

Reporting & Analysis

Custom Reports

Don’t dig for your data!

404s

Social Media Details

Top Pages by Type

Top Events by Type

Deep dive into a certain source of

traffic (ex: email campaigns)

Tracking 404’s

● No extra tagging needed

● Report Filter: Page title contains ‘Page Not Found’

● Dimensions: URL (page requested), Previous Page (might be entrance),

source/medium (more important for entrance pages to understand source of

traffic)

404 Error Report

● Monitor 404 volume over time

● Monitor broken links

● Set up 301 redirects where needed

● This report is very helpful after a site re-launch

Social Media Details

● Report Filter: Channel = Social

● Dimensions: Medium, Social Network (or Source)

● ‘Social’ = tagged social campaigns, ‘referral’ = organic social traffic (no camppaign

tags)

Pages by Type

● Report Filter: Page contains <URL identifier for type of content>

● Dimensions: Page

Possible content: blogs, story pages, article pages, FAQ’s

Events by Type

● Report Filter: Event category = ______________

● Dimensions: Event label, event action

Possible events: document downloads, offsite links, navigation links, carousel clicks

‘Unique’ Metrics - Pageviews● To report the number of sessions that

viewed a page, use ‘unique pageviews’

● GOTCHA - do NOT combine page

with sessions as a custom report (GA

WILL let you set this up, but sessions

are ONLY associated with the entry

page)

‘Unique’ Metrics - Events● To report the number of sessions that recorded a certain event, use ‘unique

dimension combinations’

● This shows the sessions with the event for whatever dimension combination is

presented in the report

Custom Segments● Create a custom segment to filter any report by sessions/users meeting specific

criteria

● Some common segments include:

○ Sessions from a specific campaign

○ Sessions that viewed a specific page

○ Sessions that registered a specific ‘event’

● Then apply a segment to a basic or custom report, for example:

○ Geographic reports

○ Traffic (source/medium) reporting

○ Technology: device/browser/OS reporting

Custom Segment - Viewed the homepage●

Custom Channel Groupings● CUSTOM channel groupings give you

the flexibility to roll up the data the way

you want to see it

● They are retroactive, but are specific to

the user account where they are created

but can be shared like other assets

● For the Gazette, we break out

Harvard.edu referrals, other Harvard

referrals, and non-Harvard referrals as

separate ‘Channels’

Custom Channel Groupings● CUSTOM channel groupings are a

view-level setting - be sure to find the

‘custom’ groupings rather than the

‘channel groupings’ (changes to the core

channel groupings will not be retroactive

and will only impact data collection

moving forward)

Helpful Toolkit

Google Tag AssistantChrome Extension

● Quickly check the status of Google

Analytics and Tag Manager code on

any page

● Red/yellow warnings identify tagging

problems

EditThisCookieChrome Extension

● View cookies set on a site

● Delete selected, or all, cookies on the site

without having to clear all of your cookies

for other sites

Google Data StudioGoogle’s new dashboard Tool

now offers free, unlimited

dashboards, with great

integration with Google

Analytics and other Google

products

Features: interactive filters,

flexible formatting, multi-page

dashboards

datastudio.google.com

Takeaways - Top Tip From Each Topic!1. Filters - maintain data integrity by collecting the cleanest data you can in your

production a view (a non-filtered view should also exist), slides 4-12.

2. Settings - key settings to check include referral exclusions (include your own

harvard SUBdomain/s) and make sure bots/spiders are excluded.

3. Reporting - remember to use ‘unique’ metrics when reporting the number of

sessions with a specific page/event.

4. Tools - download ‘Google Tag Assistant’ for very user-friendly feedback on tag

set-up and data collection.

[email protected] - Feel free to reach out with specific questions!

mailto:[email protected]

mailto:[email protected]

Copyright © President & Fellows of Harvard College

Responding to Analytics with SEOMarcus Dandurand - March 30th, 2017

How to Measure Traffic from Search EnginesIn Google Analytics: Source/Medium = Google/OrganicIn Adobe Analytics: Marketing Channel = Natural Search

2

Are we getting enough Search traffic?

3

Optimization is never done!

4

➢ Benchmark against yourself

➢ Compare traffic Year-Over-Year

➢ Try optimizing existing pages

➢ Create new pages to target new keywords

➢ Think beyond “branded” keywords

Search Traffic Year-Over-Year

5

Wait… What?!Traffic is down!!! Is it something we did? Did Google change its algorithm?Will it fix itself?What can we do?

6

First, a few SEO myths...

7

1. We don’t know what Google wants

2. The algorithm changes too often

3. SEO is an attempt to “game the system”

4. SEO is a job for IT

5. My CMS has SEO built-in

What do search algorithms care about?

8

Relevance

Performance

Authority

What is page “Relevance”?Your page content closely matches a keyword search phrase.

9

A few ways to improve a page’s relevance:

10

➢ Keyword Research - Relabel content using “outside voice”

➢ Breakup Content - Each important idea should have its own landing page

➢ Accurately describe all page components∙ Page Titles/Meta data∙ Navigation links∙ Section Headers∙ Etc.

What is page “Performance”?Pages are useful, Pages load fast, Site is accessible to humans & robots.

11

A few ways to improve a page’s performance:

12

➢ Reduce page load time

➢ Make sites mobile friendly

➢ Improve click-through rate in search engine results

➢ Make sure websites can be crawled/indexed properly by search engines

What is page “Authority”?Every link acts as an endorsement of a page’s credibility.

Both External and Internal links!

13

Authority began as Google “PageRank”

14

A few ways to improve a page’s authority:

15

➢ Create resources that people will share (inbound linking)

➢ Use 301 redirects (site cleanup)

➢ Restructure website navigation to distribute authority to your most important pages

Distributing Authority:Are the important links on your page 1/10 or 1/100?

Working Knowledge Website Updates:SEO lessons learned the hard way.

16

Mid August 2015

17

Working Knowledge has a search traffic problem!

18

Working Knowledge search traffic improves!

19

Working Knowledge search traffic improves!

20

➢ Sept – Nov 2015: down 43% vs. Prior Year

➢ Sept – Nov 2016: up 46% vs. Prior Year

So, what did we fix?

21

1) New dynamic landing pages

22

➢ Hundreds of Topic landing pages were not indexed by Google. New browse page was behaving like a single dynamic page. (Performance)

➢ Each page had the same Title & Meta Description (Relevance)

2) Deleted the Working Knowledge Archive

23

➢ Several articles were still very popular for search traffic. (Relevance)

➢ Many pages were still cited and linked to by important sources. (Authority)

Without proper redirects, the authority passed back to the home page was lost.

Other optimization for Working Knowledge

24

➢ Created customized Titles & Descriptions for each page in the CMS. (Relevance)

Other optimization for Working Knowledge

25

➢ Created new “display descriptions” visible on-page. (Relevance)

Are You Making A Major Website Update?Please consider the following...

26

#1) Navigation Links Transfer Page Authority

27

➢ Primary navigation create backlinks from every page on your site.

➢ Try to put your important pages in your primary navigation (but only if it makes sense).

➢ Try to remove links that are nice to have, but not critical.

➢ Adding new links will dilute the authority of existing links.

#2) All URL Changes Need 301 Redirects

28

➢ 404 errors create a bad user experience and waste page authority.

➢ 302 redirects are “temporary,” so Google keeps the old page in the index. No authority is passed!

➢ 301 redirects are “permanent,” so the authority of old pages are passed.

#3) Avoid Duplicate Content

29

➢ Each page should have a unique Title & Description.

➢ You should not be able to see the same page via two different URLs.

Pop Quiz:

30

Which of the following URLs below are exactly the same as: www.hbs.edu/mba

A. www.mba.hbs.eduB. hbs.edu/mbaC. http://www.hbs.edu/mbaD. https://www.hbs.edu/mbaE. All of the above

Free Tools for SEO Analytics

31

Free Tools for SEO Analytics

32

➢ MozBar (Browser Extension) ● Page Authority, Inbound links

➢ SiteImprove - Free through HUIT!● Missing Page Titles, Descriptions, Broken Links, 302 redirects

➢ Google Search Console - Formerly “Google Webmaster” ● 404 errors, Organic Keywords, XML sitemap, Mobile issues

➢ Link Redirect Trace (Browser Extension) ● Follow the path of multiple redirects

Thank You!

33

From Big Data to Insights in Massive Open Online CoursesA Traveler’s Guide

Daniel SeatonHarvard UniversitySr. Research ScientistVPAL Research Team

[email protected] Academy - March, 2017

• Consortium of Institutions creating MOOCs

• Maintain Open-Source Platform• Host Courses/Content• Lead Outreach• Maintain “https://www.edx.org”

• Partners from Higher Ed / Industry / Government / High Schools

• Create Courses/Content• Manage Courses in Open Online

(MOOC) and On-Campus (residential) settings.

• Perform Research into Teaching and Learning

Consortium Members

What is edX?


Data from Dec. 2016

http://harvardx.harvard.edu/Harvard University’s MOOC Organization:

• Partners with faculty to create open online courses

• Supports initiatives to use MOOC content beyond open online models


Data from Dec. 2016

Harvard University’s MOOC Organization:

• Partners with faculty to create open online courses

• Supports initiatives to use MOOC content beyond open online models

http://harvardx.harvard.edu/


Research Timeline and Perspective

2012

What are learners doing?

Who and where are our learners?

Besides open online, how else can we use MOOC platforms and content?

Why are learners taking courses?

2013 2014 2015 2016

Single MOOC

Transforming Advanced Placement High School Classrooms Through

Teacher-Led MOOC ModelsSeaton, Hansen, Goff, Houck, Sellers

Many MOOCsContext around

MOOC enrollments

Alternative MOOC Models


Single MOOC


• 6.002x: Circuits and Electronics (first MOOC from MITx - now edX)

• Over 100K enrollees• Over 7K certified users• Over 100GB of data from clickstream• Limited profile information

• MITx now a member of edX: ~ 100 open access courses


We started with “what” people are doing in 6.002x

Transition between resources

Nodes are resources(size ~ time spent)

Edges are transitions (size ~ weight)

Who does what in a Massive Open Online Course?Seaton, Bergner, Mitros, Chuang, Pritchard ( Comm. of the ACM - 2014)

Analyzed learner interactions with all aspects of 6.002x. Particular focus on time-on-task and resource-use during problem solving.

Measurements• Time-on-Task• Resource Interactions• Daily/Weekly Progress• Transitions between resources

during problem solving


Many MOOCs +

Context Around Enrollments


• HarvardX and MITx Working Paper #1

• Now had access to all course data from MITx and HarvardX

• Addressed “what” people were doing, and “who” they are, across 17 MITx and HarvardX courses

Key Takeaways:

1. Courses are very different.

2. Registrant diversity is immense compared to residential.

3. Participation greatly varies.

Ho, et al. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).


What are learners doing across MITx and HarvardX?

% G

rade

% Chapters Accessed0

100

100



What are learners doing across MITx and HarvardX?



Cross course surveys launched in 2014 addressing broad issues across MITx, but teaching experience was central issue.

Results from 11 spring 2014 MITx MOOCs:

• 28.0% (9451) self-identify as past or present teachers (navy).

• 8.7% (2847) current teachers (orange).

• 5.9% (1871) teach/taught the topic (gray).

On average across courses, ~ 8% (1 in 12) of comments are from current teachers.

For teachers that teach/taught the topic, the average across courses is ~6% (1 in 16).

PercentCommentsin Forum

Did not take

survey

Surveyed

SurveyedTeachers

Non-Teachers

43.8%

22.4%

33.8%


Why are we still not talking about course structure/design?+

Visualizing Course Design


Goals and Motivations

• Support HarvardX by collecting relevant stats on course structure.

• From a research perspective, identify canonical patterns in course development and better understand how those patterns affect behavior and outcomes.

Human beings, viewed as behaving systems, are quite simple. The apparent complexity of our behavior over time is largely a reflection of the complexity of the environment in which we find ourselves.

- Herbert Simon, “The Science of the Artificial”

Practical Motivation

Abstract Motivation

http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research


Visualizing Course Design

Key point: Use these visualizations to look across courses.


http://vpal.harvard.edu/blog/exploring-course-structure-harvardx-new-year%E2%80%99s-resolution-mooc-research


Alternative MOOC ModelsAP High School Content/Courses


• Boston Public School students• Take MOOC online during school• Commute to BU weekly for

labs/recitations with TAs/Faculty

Of 34 regular and charter schools serving 16,165 students, 2 high schools offer algebra based AP® Physics 1. Only 60 BPS students took the AP® Physics 1 exam during the 2014-2015 school year.

BU Project Accelerate

• Open-online and teacher-led/flipped • All content open on edx.org• Special instances for teachers to

use content in classrooms• Showed 0.08 added to AP exam score per

hour usage above class average

http://vpal.harvard.edu/blog/complementary-models-mooc-instruction-advanced-placement%C2%AE-high-school-courses


Davidson Next - AP content for use by teachers and students

Program at Davidson College:

• Supplemental content for 14 Challenging Concepts in each AP subject.

• Challenging concepts determined using College Board exam data from 2011 to 2013. Piloted with Charlotte-Mecklenburg School System in 2014-2015 school year.

• Modules designed for each concept meant to facilitate use both in classrooms, and open online. Real AP Teachers from developed content with Davidson faculty.

• Courses released on edX.org and through a new Custom Course tool (CCX).


Bubble Charts for Detecting Daily Activity• Made these to help monitor teacher use of Davidson Next in Charlotte High

Schools

• Full time assessment coordinator worked with teachers on implementation and efficacy of content (collected district data and AP exam scores).

Transforming Advanced Placement High School Classrooms Through Teacher-Led MOOC Models

Seaton, Hansen, Goff, Houck, Sellers (MIT LINC Conference - May 2016)

Pilot program in North Carolina High Schools

Massive Open Online Courses via edX.org

Exam score residuals are then correlated with student usage relative to class median indicating 0.08 points per hour spent (p<0.05).

http://edx.org


Building Community Around Analytics…


Open Source Tools for edX Data

• Harvard and MIT already share resources and code for analytics

• https://github.com/mitodl/edx2bigquery

• https://github.com/mitodl/xanalytics

• Open-Source Repos• Python + Google

BigQuery for aggregation of edX data.

• Dashboard via Google App Engine


edX Data Workshop Summer 2016

Meeting of data analysts and engineers in institutional roles responsible for edX data; 16 attendees from 11 institutions.

Goals for meeting:

• Discuss broader aspects of data sharing and analytics.

• Standup the Harvard/MIT edX Data Pipeline.

• Happy to report that each participant completed this task.

• Next workshop summer 2017?

• Hoping to broadly release workshop documentation in the spring.

http://news.harvard.edu/gazette/story/2016/07/moocs-ahead/


Visit our blog: http://vpal.harvard.edu/blog


Many collaborators to thank before dicussion!

HarvardXAndrew Ho, Dan Levy, Jim Waldo,

John Hansen, Sergiy Nesterko, Justin Reich, Tommy Mullaney

Miki Goyal, Gabe Mulley,Carlos Rocha, Victor Schnayder,

Olga Stroilova, Brian Wilson

Julie Goff, Aaron Houck, Kristen Eshleman, Pat Sellers, Noelle Smith

Yoav Bergner, Cody Coleman, Isaac Chuang, Curtis Northcutt,David Pritchard, Saif Rayyan

VPAL Research TeamAndrew Ang, Glenn Lopez, Brooke Pulitzer, Yigal Rosen, Dustin Tingley, Selen Turkay, Jacob Whitehill, Joseph Williams

Lydia Snover, Jon Daries, Mark Hansen

Research and

Analytics

Ditch the spreadsheet and tell the story

Katie HammerOffice for Sustainability and Harvard Public Affairs and Communications

How do you get your team to understand your analytics story?

MATERIALS AND CONTENT CREATED:

GHG Landing page (OFS)

4 Page Climate Report PDF

Community-wide message from President Faust

Wide-format Gazette Story and Graphics

8 #HarvardClimateStories instagram profiles

Video targeted at social media

Custom social graphics for 12 Schools + departments

harvard.edu/climate modules

Community-wide email sent by President Faust

52% open rate867 clicks

Gazette Story in the Daily Gazette 22% open rate637 clicks

Social Promotion (Harvard & OFS)

● Video with paid boost● Twitter & Twitter Moment● #HarvardClimateStories

Instagram Campaign

● 108,000+ video views● Almost all Schools shared

news w/ graphics● 17,694 likes; 84 comments

Inclusion in OFS December email newsletter

25% open rate507 clicks

Feature on Harvard.edu 3,010 clicks

External Press (Crimson, Harvard Magazine Story,NY Times, Boston Globe, WGBH etc.)

DISTRIBUTION EFFORTS:

OFS Goal Page: 335

Gazette Story: 210

HUCE Site: 301

Clicks

Try something new?Flaunt it.

FACEBOOK AD CAMPAIGN: OFS

OFS Ad Spend: $170Duration: December 8 - 12

Audience: Targeted students and alumni (where Harvard was listed as School and age was 18+); OFS email list; People who liked our Facebook page

Reach (number of people that saw the post):● 42,019 total people reached● 16,624 people reached as a result of paid● 17,000 total video views; ● Cost per 1,000 people reached $10.23● $.05 per 10 second video view

Engagement (reactions, comments, shares):● 7,696 total actions● $0.27 per engagement ● 23 link clicks● Post generated 55 new GreenHarvard

Facebook page likes

Context:● Typical GreenHarvard video is viewed ~500

times

Note: Many comments did include mention of divestment; however about half were positive and congratulatory.

Qualitative data matters too.

TWITTER STRATEGY:

● Worked with Facility teams to create custom graphics optimized for social for Schools to use

● Partnered w/ HPAC to share on the @Harvard accounts● Outreach in advance to all digital counterparts at Schools/Depts● Created a Twitter Moment to capture various influencer

and School tweets about announcement

RESULTS:

● Initial tweet: Retweeted 66 times; Liked 102 times, Clicked 34 times● Moment tweet: Retweeted 33 times, Liked 92 times, Clicked 90 times● Retweets and original tweets from internal “influencers” like HBS,

HSPH, HAA ● Almost all 12 Schools promoted us in some way, in addition to the

Museums, Libraries, and various departments

And they all livedhappily ever after...

Lessons learned and opportunities

● Targeted outreach to Schools and Depts works; Schools/Depts find easier to promote when can link data/anecdotes back to them (social graphics received well).

● While School/Dept outreach worked and we did have some external influencer tweets (Climate Registry, USGBC), we should develop a more solid plan for faculty and social influencers in the future.

● #HarvardClimateStories campaign a success; 8 profiles in a month was ambitious; for future campaigns could start earlier and conduct interviews and shoots further in advance.

● Important to consider different outreach methods for different audiences; for example the social video was extremely brief but gave an external audience the message they needed “Harvard set an ambitious goal and they met it.”

● We should consider allocating time more evenly across a wide range of projects, considering goals, audiences, and reach (PDFs, videos, social campaigns). For example, though PDF a considerable amount of our time, the social/web reach was minimal.

Keys to a telling a good story

FormatChoose a vehicle that’s relatable (even if that means powerpoint). Keep it simple.

StyleUse language that seems right for your story (and for your client).

SettingSet your story by bringing in context to explain the why. Remember you control this!

ThemesLet the themes of your data shine by weaving them throughout your story.

IllustrationsImages, examples, and visual cues only add to your story.

ConclusionEnd your story with lessons learned and opportunities that leave the reader ready for your next story!

Thanks!Any questions ?◉ [email protected]

Strategize, Synthesize, and JAZZERCIZE®

your Analytics Dashboards and Reports

with your host Aaron David Baker

Remember Jazzercize®?

Problem:

Non-dancers are taking Jazz Dance classes because it is a great workout but aren’t interested in all the work on form and technique. They just wanna have fun while exercising.

Solution:

Create a fun jazz dance-style fitness class that’s interesting and fun!

The power of FUN

The problem was neither exercise nor jazz dance class

Exercise is a chore; we make it fun to get it done.

It’s the same with creating/consuming analytics data

Give them what they want!

The people who consume your reports want them to be engaging.

They don’t always have to contain charts and graphs.

Good document design also conveys professionalism.

Don’t just report on the content you produce or see, share screenshot highlights to give context.

Great reports are persuasive and can change attitudes

Introducing Scoop

HPAC’s analytics dashboard, built around public APIs to Google Analytics, Facebook, Silverpop, and more to come

Presents consistent, up-to-date performance data per story, post, and mailing

Makes comparative metrics possible with benchmarking and visualizations

Introducing Scoop

We have a Strategy

● Identify and capture metrics that matter in one convenient place

We have Synthesis

● Stats from many different platforms in one place for easy reporting

We are working on that Jazz

● Make it beautiful and fun

Strategize

The official Harvard Style Guidelines & Best Practices site has an updated Analytics with resources and setup information.

harvard.edu/guidelines

Use these best practices to ensure your site is up-to-date with the latest analytics code and tracking practices.

First, check that everything is in order

Second, define key stats (these are just some examples)

Health stats are these

● Users/Sessions/Pageviews ● % New Users over time● Page speed● 404s

Strategic metrics are these

● Content performance (pageviews)● Content engagement

○ Time on page○ Scroll depth

● Users by○ New/returning○ Geolocation○ Content sections they visit○ Frequency of visits

● Content dimensions○ Content category/section/tag○ Content length

● Acquisition paths○ Search keywords

Choose goals and metrics that make sense for your organization!

Report on what’s exceptional and on what’s important

What’s exceptional

● Top content in terms of pageviews and/or engagement metrics (time and scroll)

● Large number of people reached or high number of impressions

● Social activity (likes, comments, shares, retweets)

● Unusual spikes in traffic* or unknown sources

What’s important

● Key initiatives○ President Faust’s priorities○ Special Gazette features

● Things you spent money on○ Paid social, AdWords, etc.○ Extra money gives you extra

metrics● Experiments

○ A/B testing ○ SEO

● Conversions

Here we’ve chosen to highlight pageviews by channel and accumulated pageviews over time

● Helps us visualize content distribution and sources of traffic.

● Benchmarking is our own comparative metric—average daily pageviews of all stories in Scoop.

Scoop Example

Synthesize

The chore of reporting

● Go here● Find stats● Copy/paste into doc● Go there● Find stats● Copy/paste into doc● …

Wouldn’t it be nice if the stats gathered themselves?

Scoop pulls in stats from

● Google Analytics● Facebook● Silverpop

With plans to incorporate

● Twitter● Instagram● YouTube● Etc.

Anything with an API can be consumed.

Scoop Example

Synthesis isn’t just about automation

Automation is nice and does save a lot of time.

Linking things by a common element (like URL or topic) can make finding the stats easier.

Synthesis is really about telling the whole story.

Automated reporting cannot speak for you.

Talk about the why in your reports.

My Weekly Report

Ultimate synthesis of what happened last week

Mostly highlighting what’s exceptional

Occasionally mentioning what is bizarre

Always as interactive as possible

● links go to actual online posts or to Scoop itself

Jazzercise!

Back to Jazzercize®

Our reports are working, and people come to us for information, but how can we analytics reports more enjoyable?

Design and data visualizations:

● Beautiful, clean, contemporary, and inviting design

● Display visually our data’s trends, patterns, and correlations

● Provide content creators and distributors with intuitive, at-a-glance insights about performance of published work

● Be designed with interactive development in mind

The plan for Scoop

Your turn to talk:

How do you add jazz to your reports?

Thanks y’all!