ab test platform - 우종호
TRANSCRIPT
![Page 1: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/1.jpg)
1/31
A/B Test PlatformPLab (Planet A/B Test)
SK Planet
System Software개발팀우종호
![Page 2: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/2.jpg)
2/31
Content
• 1. What? – A/B Test
– A/B Test –개념
– A/B Test –사용사례
• 2. So! PLab – Planet A/B Test Platform
– PLab 소개및특징
– PLab Tech Architecture & Data Flow
• 3. PLab 적용사례 & demo
![Page 3: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/3.jpg)
3/31
What is A/B Test?
![Page 4: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/4.jpg)
4/31
Why?
• 일반적인프로세스
문제인식
개선안도출
개선안적용
개선안적용이실적에끼치는영향을알수없음
![Page 5: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/5.jpg)
5/31
A/B Test 적용프로세스
A/B Test문제인식
개선안도출
개선안적용
A/B Test – Data-driven approach to decision and service
개선안적용문제인식 개선안도출
정량적인수치로검증필요
객관적인데이터기반결정필요
![Page 6: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/6.jpg)
6/31
What is A/B Test?
• 전체디자인에서
• 한가지요소(UI 레이아웃 / 이미지 / 검색결과 / feature / ..)에대해
• 두가지이상의버전(variation A, B, ..)를
• 실험하여(Experiment)
• 더나은것(Winner)를판별하는기법
Variation A/B
Experiment
Winner
![Page 7: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/7.jpg)
7/31
• 기존버전(A, control, 대조군) / 새로운버전(B, variant, 실험군)을가지고,
• 랜덤하게방문(random sampling)하는사용자별로다른버전을보여준후,
• 어떤버전이더나은지(winner),
• 데이터기반의정량적인수치(지표, 그래프)로검증한다.
What is A/B Test?
A, Control
B, Variant
Random Sampling
![Page 8: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/8.jpg)
8/31
• “결과”의 “원인”이우리가했던그개입때문이맞는지를판단.
• 통계적유의미성(statistical significance) – 집단에대한가설이가지는통계적의미
– P-value
What is A/B Test – P-value
P-value
![Page 9: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/9.jpg)
9/31
• 국내최대웹사이트가있다. 3개월간의전면적인디자인개편!.
고객은성별에따라관심있는상품이다를것이다! [전체/남/여]를선택하는탭을추가하자!.
새디자인적용후매출이 20%증가!
• But, “매출증가”가 “디자인개편” 덕분일까??? 생각하지도못한외부요인이너무많다.
정량적인수치로분석이힘들다.
• So, A/B Test !
A에게는기존디자인을보여주고, B에게는새디자인을보여준다!
동일한시간의흐름에따라발생하는다른외부요인들을통제할수있다.
데이터를기반으로정량적인수치를가지고분석할수있다.
Example with or without A/B Test
![Page 10: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/10.jpg)
10/31
So experiment your idea
![Page 11: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/11.jpg)
11/31
A/B Test – Use Case & Solution
![Page 12: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/12.jpg)
12/31
오바마선거적용사례 - Optimization• http://kylerush.net/blog/optimization-at-the-obama-campaign-ab-testing/
- A : 오바마에게포커싱이맞춰진이미지
- B : 오바마와영부인, 게스트들이사진에포함. 디너참석으로기대할수있는장면을사용
- B winner CR +19%
![Page 13: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/13.jpg)
13/31
Google Optimize
• Google의 AB Test 플랫폼으로 Google Analytics Suite를구성하는하위솔루션임
• 2015.10 런칭되었고 2016.03 현재와같은구성으로변경
• PC Web만지원하며현재까지 Android, iOS 미지원
![Page 14: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/14.jpg)
14/31
So! PLab – Planet A/B Test Platform
![Page 15: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/15.jpg)
15/31
사용자 실험 참가 방식
Random sampling
Audience Targeting
지원 플렛폼
Android iOS PC Web
표준화된결과지표
CR RBU AOV
P-value
A/B Test 전체 프로세스기능개발
실험생성
실험검증
실험진행
실험결과확인
실험수정
실험종료
Filtering
CTR
Average Rank
2 depth analysis
AB테스트전체프로세스기능
SK Planet A/B Test Platform - PLab
![Page 16: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/16.jpg)
16/31
A/B Test 전체프로세스
기능개발
실험생성
실험검증
실험진행
실험결과확인
실험수정
실험종료
• 01. (draft) 실험생성실험이름, 실험영역, variation정보, meta데이터
• 02. (running) 진행 & result page 실시간지표확인
• 03. (winner) winner설정수정버전배포전임시로winner가바로반영
• 04. (stop) 실험종료
AB테스트전체프로세스기능
SK Planet A/B Test Platform - PLab
![Page 17: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/17.jpg)
17/31
• Visitor(순방문자)• impression(노출수)• QC(검색쿼리카운트수)• Conversion / CR / CTR• Price 구매액, 매출 – GMV, AOV, RBU• P-value
CR, AOV, ..등에대한 p-value제공
CR RBU AOV
P-value
CTR
Average Rank
2 depth analysis
표준화된결과지표
SK Planet A/B Test Platform - PLab
![Page 18: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/18.jpg)
18/31
• Velocity: Fast
– 5분내에최초분석결과를제공후 1분단위 Update• Optimizely: 1시간이후, GA: 12시간이후
• Statistics: Statistical Hypothesis Test
– Price (AOV, RBU)에대한 P-Value 제공• Optimizely: Conversion만지원, GA: 수작업계산
• Volume: Small
– AB 통계 Metrics을누적하여실험결과Metrics 생성
SK Planet A/B Test Platform - PLab
![Page 19: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/19.jpg)
19/31
Can I use PLab ?
![Page 20: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/20.jpg)
20/31
It is useful to experiment with your idea
You Can’t use in this case
Can’t – 실험종류후추후분석, 사용자데이터를이용하여후분석?!
Can’t – PLab을쓰면앱개발을안해도된다!?
Useful – 빠른의사결정을위한근거가필요!
Data-driven approach to decision and service
![Page 21: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/21.jpg)
21/31
So! PLab – Planet A/B Test PlatformExperiment Process & Architecture
![Page 22: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/22.jpg)
22/31
1. A/B Test 실험설계 2. A/B Test 플랫폼에서 실험생성
4. 사용자가 A 또는 B에참가하며실험진행5. 다양한지표와그래프로실험결과
확인
6. Winner 설정으로실험종료
3. 개발및배포후실험검증
SK Planet A/B Test Platform - PLab
![Page 23: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/23.jpg)
23/31
SK Planet A/B Test Platform – PLab
![Page 24: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/24.jpg)
24/31
PLab 적용사례 & demo
![Page 25: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/25.jpg)
25/31
A/B Test 11번가적용현황
• PLab 도입전 (2016년도)– 실험개수: --개– 테스트를설정하는데어려움: 구성원의 AB 테스팅에대한이해도낮고위너를설정
하는데필요한지표(P-value)가제공되지않았음
• PLab 도입후 (2017.03 ~ - 약3개월)– 실험개수: --개
– 현재 --개 Running
This is
PLab
![Page 26: AB Test Platform - 우종호](https://reader033.vdocuments.mx/reader033/viewer/2022052116/5a6766a27f8b9a656a8b4de1/html5/thumbnails/26.jpg)
26/31
Sample
테스트기간: --일
A vs B
장바구니담기: --% 상승
구매횟수: --% 하락
위너:
A B