renee thompson, techtarget, optimizely experience

TESTING TIPS AND LESSONS LEARNED

Renée Thompson, Senior Director, Analytics and Conversion Optimization, TechTarget

August 6, 2014

TECHTARGET: WHO WE ARE

• Online global B2B media company with >150 technology-focused websites (SearchSecurity.com, SearchCIO.com, etc.)

• We deliver targeted technology content to IT and business users

• We provide technology marketers media programs via custom advertising, branding and lead generation solutions

OUR TESTING BACKGROUND

• TechTarget has been actively focused on A/B and multivariate testing for over 6 years

• We formerly used Test & Target

• In the past year we switched to Optimizely

• “Always Be Testing” is our mantra

OUR TESTING TEAM• Small team – myself and one dedicated member (Brian DeLoach)

• I generally manage communications, test plan documentation, success and best practice documentation, etc.

• Brian manages designs, test setup, test coding and conducts user research

• We are looking to hire someone else to work under Brian

• We work closely with our team at large on ideas, brainstorming

TIPS

6

TIP 1: DON’T STOP AT ONE TEST WAVE

• Multiple iterations can compound success

• Results can inform additional ideas/iterations

Example: Registration Testing

Test 1: 30% lift in conversions

Test 2: 60% lift in conversions

Test 1: Control Experience

Test 1: Winning Experience

• Updated copy• Elimination of side rail• Modified button call to action • Larger form labels• Slight form field re-ordering• Removal of ‘required’ explanation

30% lift in conversions

Test 2: Dynamic Display

60% lift in conversions

TIP 2: USE QUALITATIVE DATA TOO

• Quantitative only tells part of the story (what, not why)

• Combine with quantitative data to run more successful tests

• Test your “hunches”

Example: TechTarget Redesign

UserTesting.com

Qualaroo

Verify

• KPI metrics comparison before/after showed positive results for most engagement metrics (pages per visit, clicks, mobile engagement, etc.)

• We conducted a series of qualitative tests using UserTesting.com, Qualaroo and Verify

• Most qualitative data jived with quantitative/analytics data

• On Definition pages (a major source of organic traffic), users were very favorable about new “Continue Reading” section, which jived with click data we were seeing

HOWEVER…

• Users like the modern design/look but are frustrated by the lack of navigation

• After analyzing click data on home page, we noticed many users were clicking the drop-down next to the logo

• Most users expected this to be a search box or a navigation menu

• As a result, we are planning more qualitative tests • Do users understand the content labeling methodology?

• And we are planning the following tests:• Navigation addition

• “Continue Reading” optimization and content

TIP 3: VIEW MANY DIFFERENT METRICS

• Each test has a main success metric

• Important to also look at other health metrics in context with the test

• Is a positive result worth a negative trend somewhere else?

• Metrics to watch will depend on your business

Engagement,

Navigation UsageAd CTR

Affiliate Clicks

Site Engagement

TIP 4: INTERNAL PR IS CRUCIAL

• Does the business at large know about the great things you’re doing?

• Are you having trouble getting resources?

• Are other people/groups doing “testing” on their own?

Socialize your testing program!

• Talk to other groups about your process, tests you’ve run, results you’ve seen

• Invite them to share ideas or bring problems to you where you might be able to help

• Develop case studies / one-pagers highlighting your success; document best practices to be shared

• Serves as a proof point for your team

• Ensures everyone knows what you’re doing

• Positions you/your team as expert!

• We hold monthly update meetings with a key group of stakeholders, and quarterly updates with a larger group (including executives)

• We publish success sheets and best practices to share with various groups

HAPPY EXPERIMENTING!

27