cultivating a culture of experimentation
TRANSCRIPT
Speakers
Cultivating a Culture of Experimentation
Bryan Berger, Product Design Lead, General Assembly
Taylor Gilbert, Partnerships & Alliances, Optimizely
Today’s Speakers
Bryan Berger
Taylor Gilbert
Housekeeping
• We’re recording!• Slides and recording will be
emailed to you after the webinar is complete
• There will be time for questions at the end
Agenda
• ‘Culture’ of Experimentation
• General Assembly’s Approach
• How to Energize Your Team
• Some Things We Tested
‘Culture’What does that
mean?
Turns out… culture is a
big deal.
of Fortune 500 companies from 1955 are still in business today12%
¼ companies fell off the list in the last year alone
The Organization of the Future…. Arriving Now
1. Culture has never been more important, experimentation is the vehicle.
2. Your executives are struggling and they need your
help. They are looking for someone to step up and lead/orchestrate.
General Assembly (GA) is a global educational company. Focusing on the most relevant and in demand skills across data, design, business and technology.
General Assembly is empowering a global community to pursue work they love through best-in-class instruction and access to opportunities.
Experimentation at
General Assembly
A Little About Me
● I manage our Product Design Team, whose focus is on designing an
ecosystem to support online and in-person learning experiences.
● I also work quite extensively with our Marketing and Website CRO team to
improve the positioning of our Products.
● One of my many goals is to foster a fun and impactful culture of
experimentation deeply rooted in the design process.
Why Experiment?● Progress over perfection. Experimentation allows our team to ideate multiple
solutions to a problem and not paralyze the process in favor of “the one”.
● Expand potential. Anyone with a hypothesis can partake. Inclusion has become
an ally in bringing down silos among our product teams.
● Gain insights. To develop a better understanding of our data, and how we can
affect the levers of improvement.
● Build consensus. Save time and money by testing your assumptions.
Experimentation at General Assembly
● Started to hit the limits of our current solutions.
● Our business was scaling quickly and along with it, our growth targets.
● We needed to start questioning things that have been “established”.
● Our testing program has been running for about a year.
● Prior to that we had no repeatable testing process in place.
How to Energize Your Team
Get Creative
Really Creative...
Make It Known
Highlight and Acknowledge Success
Our Approach
Getting Started: Finding a ProcessWhere we all start
Where some give up
The goal, a smoother repeatable process
Our Initial Process
Our Current Process
Finding Great Ideas● All ideas must contain:
○ Problem definition○ Hypothesis○ Audience Type (Persona)○ Page url(s) affected
● All ideas are added to a backlog list○ Tickets are created or updated once ideas are vetted for impact, time to significance, and
resource allocation.
● Hold weekly A/B testing meeting to: ○ Discuss how a specific test would be implemented ○ Review cost / benefits for different possible solutions
○ Amount of traffic needed○ Length of the experiment○ Next steps post test
Prioritizing Test Ideas
Website Tests are prioritized by our Marketing Team & Director of SEO/CRO
● We define quarterly Objectives & Key Results (OKRs) and roll up each test result toward one of those objectives
● Test key routes (purchase flow, wayfinding, and value prop improvement)● As of today, all tests aim to improve conversion and engagement at key steps of our funnel
Stay Organized
Share Results
Clear communication, acknowledgement, and collaboration.
● #CRO Slack channel for asking questions, and posting results.
● Announces test results and shoutouts.● Big wins are also sent to an email group.● Slack is effective. Everyone is engaged in it,
from engineering to execs.
Program GoalsOur current benchmark is to have at least one live test on each of the most critical pages in our conversion funnel (to keep momentum):
○ New user signup modal○ Homepage○ Navigation○ Product pages○ Wayfinding systems○ Discovery & catalog pages
Some Things We Tested
25% Off Class & Workshop Offer
Modal Incentive Copy Test
Original Variation
Drive more urgency to the next course start dateCourse Start Date
Original Variation
Copy and styling tests in Navigation menu Copy and Style in Navigation
Original Variation
Show the customer potential ROI of our courses
No original variation
Adding Sections
Original Variation
Looking Forward
● As we scale our experimentation program we’ll likely bring in an agency to help build, design, & deploy tests in Optimizely.
● With their support, we’ll be able to move more quickly on our backlog of 80+ test ideas.
● Continue testing to inform design direction for our new site.● Penetration into our Product Apps and onboarding flows.
Key Takeaways● Become the champion of your experimentation program● Make your testing program known, give it a physical presence● Start backlogging and organizing ideas● Establish a process to vet, prioritize and execute those ideas● Set up a staging environment to act as your laboratory● Hold yourselves accountable for stagnation● Beware of vanity metrics. ● Demonstrate how testing can be inclusive, impactful and FUN!
Questions?
Thank You!Feel free to reach out to continue the conversation:
@bryanberger @ga