airflow scheduler as a service

!"#$%&'$()*+)*)!$(,-"$)

!"#$%&%'()#*)('+,-./#$('(#0#"1

!"#"$%&'"(#)*+*,,#%"&-./,$%./0".)1"/ 2%3

© 2020 Electronic Arts, Inc.

!"#"$%&'"(#)*+),#-.*/&012"1//*&33

14"(#)*5/.6()7

© 2020 Electronic Arts, Inc.

!"#"$%&'#'(!"#"$%&'#'(

!"#$!%&'!"#$!%&'!"#$!%&'()&'*+%,-'()&'*+%,-'()&'*+%,-'()&'*+%,-'()&'*+%,-'(+'$,./

5

!"#$%&'()"*+,-./(-#$0)'#$1.,2/&3

!"#$!%$&'%

!"#$%&%'()#*)('+,-. $('(#/#"0 *)(12- 32'4,-56-(.24,-5

782#928'-()#$('(#*)('+,-.#:2(.

;<==,-' ><)'%=)2?(.2#;'<@%,A!"#"$%&'(&))*+$,$-.()&#(+#+

$('(A2'#%8#*2'(B1'2ACDA#,+#:2-(B1'2A#,+#

@('(#&282-('2@#2E2-1#@(1

:F,<A(8@A#,+#!:G#H,BA

IJ#&(.2#A'<@%,A#A=-2(@#(K-,AA#'F2#

4,-)@

()*+,$*-$%&

!"#"$"#$%&

!"#$%&'()*+,%-*'.%/#0$))$)%12"34,'%5$"#,

6(3'*73$%)"'"%7*7$3*2$,%89+#%" :$2'9"3%)"'"%-"9$.+(,$;

<===,%+8%/5>%?+0,%"2)%@A=B%")C.+:%D($9*$,%$E$94)"4%(,*2F%+(9%73"'8+9#

G*F.34%,$2,*'*E$%)"'",$', H"9*$)%&>1%9$D(*9$#$2',

!"#"$"#$%&

'()*+,-*.$"#$%&

!"#$%"&'(%#)$* +(,-&'(%#)$* ./,/%"&$#&'(%#)$*

4)5",67*)/68(*"9#$ 7,/#*.:&;"*<:)= >3:#"9:,&;"*<:)=&$(%,?3:,*

73$#)@&7)/#"/3)3$&A/#,B*.#")/&C7AD 7)@@)/&+:3B"/$ 8,:<E8,*F,&4*(%,$#*.#")/&8G$#,@

A/EH)3$,&I,9:)G@,/#&J)): 8#./?.*?&>)/"#)*"/B A@9*)F,?&=)*K:).?&@./.B,@,/#

L.$"(&@)/"#)*"/B 7A67I

L.$"(&;3#%&>)?,: 884&"/#,B*.#,?

8#./?.:)/,&8G$#,@$ >3:#"E#,/./(G

'()*+,-*./"0/"/'*.12(*$345*(#21*0

6(3'*C'$2"2:4%-*'.%IJ1K%L&&M%-*'.%K(,'+#%1('.%6*))3$-"9$N

/",$%+8%?+0%)$73+4#$2'%L!*'3"0%KOPKQN

/",$%+8%*2'$98":*2F%-*'.%'.$%)"'",$'%LK(,'+#%R3(F*2,%-*'.%

1K>N

6+2*'+9*2F%"2)%13$9',%",%7"9'%+8%,$9E*:$%

© 2020 Electronic Arts, Inc.

8*//#%"&9.1/$%.1+),#-.*/&012"1//*&333

:2.1/$%.15/.6()7

Multi-Tenancy and RBAC

14EADP Data & AI

6,-#2/7*8"8(9$

:;&< "0$"$'*.12(*

RBAC as a Service

Job Deployment

18EADP Data & AI

&,#=$!*1/3>0$"0$"$'*.12(*

Auto Deployment – S3 Sync Sidecar

• Need a way to sync DAGs across multiple pods whenever there is new or updated DAG

• Sidecar runs alongside Web, Workers and Scheduler Pods

• DAG files and Plugins synced from S3 location periodically

'"8+4=B$"0$"$'*.12(*

© 2020 Electronic Arts, Inc.

;".)<"1 =%>+),#-.*/&012"1//*&333

?@%>5/.6()7

Custom PluginsEase of accessing the datasets

23EADP Data & AI

C-,D28$E24.".9$"0$"$'*.12(*

R3(F*2,%"9$%#"2"F$)%'.9+(F.%"%,*2F3$%9$7+,*'+94

I$D(*9$,%#$9F$%9$D($,'%"2)%"779+E"3,%'+%:+2'9*0('$

R3(F*2,%"9$%)$73+4$)%'.9+(F.%:+2'*2(+(,%)$73+4#$2'

O2C.+(,$%R3(F*2,%'+%*2'$98":$%-*'.%R9$,'+%

R3"'8+9#

<,0#=?$C-,D280$F2#)$&<E

R3(F*2%"::$,,%)$8*2$) 7$9%'$"# *2%

S&MT

!*'3"0%KO%5$,',%'+%$28+9:$%"::$,,%:+2'9+3

T$-%"779+E$)%73(F*2,%"))$)%'+%:+28*F

Monitoring and Alerts

26EADP Data & AI

&-*.#28D$"0$"$'*.12(*

!"#$%&$'$()*%+,-.$()/)#/($)0$,$0)+1)1/"0')'+0$%/&2$

3+.)0$,$0)4$'%-2()/&.)/0$%'()/,/-0/#0$)5-'6)3%+4$'6$"(

74/-0)#/($.)&+'-1-2/'-+&(

8-%10+5)9$%,-2$)0$,$0)4$'%-2()/&.)/0$%'(

:+#;<8=)0$,$0)4+&-'+%-&>

7?@)/0$%'()/&.)4+&-'+%-&>

6=82#=.28D$"0$"$'*.12(*

6=82#=.28D$"0$"$'*.12(*

Architecture and Integration

30EADP Data & AI

&2.G-=F$H8G."0#.,(#,.*

!"#$%&"'(%)(*+,"-)"."/

0"$"-&(12"3+.%-(4 5"'6/(,-%7"-

8%/.9-":;<(,=37")'

8-%2&(>=/"'(?+.@").63=.6%)

A+$.64.")=).(/&/."B

0").-=$6C"'(#$+96)(/&/."B .% 6)."-D=3"(E6.@(B+$.6#$"(#-"/.% 3$+/."- "3%/&/."BF

!"#"$