airflow scheduler as a service
TRANSCRIPT
!"#$%&'$()*+)*)!$(,-"$)
!"#$%&%'()#*)('+,-./#$('(#0#"1
!"#"$%&'"(#)*+*,,#%"&-./,$%./0".)1"/ 2%3
© 2020 Electronic Arts, Inc.
!"#"$%&'"(#)*+),#-.*/&012"1//*&33
14"(#)*5/.6()7
© 2020 Electronic Arts, Inc.
!"#"$%&'#'(!"#"$%&'#'(
!"#$!%&'!"#$!%&'!"#$!%&'()&'*+%,-'()&'*+%,-'()&'*+%,-'()&'*+%,-'()&'*+%,-'(+'$,./
5
!"#$%&'()"*+,-./(-#$0)'#$1.,2/&3
!"#$!%$&'%
!"#$%&%'()#*)('+,-. $('(#/#"0 *)(12- 32'4,-56-(.24,-5
782#928'-()#$('(#*)('+,-.#:2(.
;<==,-' ><)'%=)2?(.2#;'<@%,A!"#"$%&'(&))*+$,$-.()&#(+#+
$('(A2'#%8#*2'(B1'2ACDA#,+#:2-(B1'2A#,+#
@('(#&282-('2@#2E2-1#@(1
:F,<A(8@A#,+#!:G#H,BA
IJ#&(.2#A'<@%,A#A=-2(@#(K-,AA#'F2#
4,-)@
()*+,$*-$%&
!"#"$"#$%&
!"#$%&'()*+,%-*'.%/#0$))$)%12"34,'%5$"#,
6(3'*73$%)"'"%7*7$3*2$,%89+#%" :$2'9"3%)"'"%-"9$.+(,$;
<===,%+8%/5>%?+0,%"2)%@A=B%")C.+:%D($9*$,%$E$94)"4%(,*2F%+(9%73"'8+9#
G*F.34%,$2,*'*E$%)"'",$', H"9*$)%&>1%9$D(*9$#$2',
!"#"$"#$%&
'()*+,-*.$"#$%&
!"#$%"&'(%#)$* +(,-&'(%#)$* ./,/%"&$#&'(%#)$*
4)5",67*)/68(*"9#$ 7,/#*.:&;"*<:)= >3:#"9:,&;"*<:)=&$(%,?3:,*
73$#)@&7)/#"/3)3$&A/#,B*.#")/&C7AD 7)@@)/&+:3B"/$ 8,:<E8,*F,&4*(%,$#*.#")/&8G$#,@
A/EH)3$,&I,9:)G@,/#&J)): 8#./?.*?&>)/"#)*"/B A@9*)F,?&=)*K:).?&@./.B,@,/#
L.$"(&@)/"#)*"/B 7A67I
L.$"(&;3#%&>)?,: 884&"/#,B*.#,?
8#./?.:)/,&8G$#,@$ >3:#"E#,/./(G
'()*+,-*./"0/"/'*.12(*$345*(#21*0
6(3'*C'$2"2:4%-*'.%IJ1K%L&&M%-*'.%K(,'+#%1('.%6*))3$-"9$N
/",$%+8%?+0%)$73+4#$2'%L!*'3"0%KOPKQN
/",$%+8%*2'$98":*2F%-*'.%'.$%)"'",$'%LK(,'+#%R3(F*2,%-*'.%
1K>N
6+2*'+9*2F%"2)%13$9',%",%7"9'%+8%,$9E*:$%
© 2020 Electronic Arts, Inc.
8*//#%"&9.1/$%.1+),#-.*/&012"1//*&333
:2.1/$%.15/.6()7
Multi-Tenancy and RBAC
14EADP Data & AI
6,-#2/7*8"8(9$
:;&< "0$"$'*.12(*
RBAC as a Service
Job Deployment
18EADP Data & AI
&,#=$!*1/3>0$"0$"$'*.12(*
Auto Deployment – S3 Sync Sidecar
• Need a way to sync DAGs across multiple pods whenever there is new or updated DAG
• Sidecar runs alongside Web, Workers and Scheduler Pods
• DAG files and Plugins synced from S3 location periodically
'"8+4=B$"0$"$'*.12(*
© 2020 Electronic Arts, Inc.
;".)<"1 =%>+),#-.*/&012"1//*&333
?@%>5/.6()7
Custom PluginsEase of accessing the datasets
23EADP Data & AI
C-,D28$E24.".9$"0$"$'*.12(*
R3(F*2,%"9$%#"2"F$)%'.9+(F.%"%,*2F3$%9$7+,*'+94
I$D(*9$,%#$9F$%9$D($,'%"2)%"779+E"3,%'+%:+2'9*0('$
R3(F*2,%"9$%)$73+4$)%'.9+(F.%:+2'*2(+(,%)$73+4#$2'
O2C.+(,$%R3(F*2,%'+%*2'$98":$%-*'.%R9$,'+%
R3"'8+9#
<,0#=?$C-,D280$F2#)$&<E
R3(F*2%"::$,,%)$8*2$) 7$9%'$"# *2%
S&MT
!*'3"0%KO%5$,',%'+%$28+9:$%"::$,,%:+2'9+3
T$-%"779+E$)%73(F*2,%"))$)%'+%:+28*F
Monitoring and Alerts
26EADP Data & AI
&-*.#28D$"0$"$'*.12(*
!"#$%&$'$()*%+,-.$()/)#/($)0$,$0)+1)1/"0')'+0$%/&2$
3+.)0$,$0)4$'%-2()/&.)/0$%'()/,/-0/#0$)5-'6)3%+4$'6$"(
74/-0)#/($.)&+'-1-2/'-+&(
8-%10+5)9$%,-2$)0$,$0)4$'%-2()/&.)/0$%'(
:+#;<8=)0$,$0)4+&-'+%-&>
7?@)/0$%'()/&.)4+&-'+%-&>
6=82#=.28D$"0$"$'*.12(*
6=82#=.28D$"0$"$'*.12(*
Architecture and Integration
30EADP Data & AI
&2.G-=F$H8G."0#.,(#,.*
!"#$%&"'(%)(*+,"-)"."/
0"$"-&(12"3+.%-(4 5"'6/(,-%7"-
8%/.9-":;<(,=37")'
8-%2&(>=/"'(?+.@").63=.6%)
A+$.64.")=).(/&/."B
0").-=$6C"'(#$+96)(/&/."B .% 6)."-D=3"(E6.@(B+$.6#$"(#-"/.% 3$+/."- "3%/&/."BF
!"#"$