orchestration ownage - rsac 2017
TRANSCRIPT
SESSIONID:SESSIONID:
#RSAC
BryceKunz
OrchestrationOwnage:ExploitingContainer-CentricDatacenterPlatforms
CSV-R03
SeniorThreatSpecialistAdobe
MikeMellorDirector,InformationSecurityAdobe
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Intro
2
MikeMellor
Director,InformationSecurity@Adobe
BryceKunz
SeniorThreatSpecialist@Adobe
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Containers- TheFutureisNow!
3
2016Surveys:
15-16%ofallorganizationsarealreadyusingcontainersinproduction
35%organizationshavedoneaproof-of-concept
TheFutureisNow!
Containersareinproductionnow
Containersarecontinuingtogrowinpopularity
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Containersappearmoresecure
4
Thebiggestdrivers:
39%toincreasedeveloperefficiencyand
36%tosupportmicroservices
Organizationswanttoavoidcloudplatformlock-in
2016Surveys:
Many(42%)organizationsgainvalueinthe“secure/isolated”capabilitiesthatcontainersprovide
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ButmanagingContainersfeelscomplex
5
2016Survey:
Themoreexposureanorganizationhastocontainers,
Themorecomplexitiesareexposed.
Respondentssaidtheyfoundcontainers…
“toocomplextointegrateintoexistingenvironments,”
andrequire“toomanyskilledresources tomanage.”
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Andareverychallengingtomanageatscale
6
2016Survey:
The#1challengeofcontainers,accordingtothe53%ofrespondentswhoareeitherusingorevaluatingcontainers,is…
“ContainerManagement.”
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ProbableSecurityNightmare
7
“TooComplex”+ChallengingtoManage
=
ProbableSecurityNightmare
“ComplexitytheWorstEnemyofSecurity”- BruceSchneier
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ContainerandClusterManagementOptions
8
Technology Design Pros Cons
PublicCloudContainerServices
Container Centric Easy, Scalable Vendor Lock-in;Proprietary
DockerSwarm Docker Centric NativeClustering LimitedbyAPI
Kubernetes ClustersofContainerized Apps
Worksw/Docker;Mounts persistentvolumes
Custom overlayrequiresmorespecialization
Mesos &DC/OS ClusterManagement
Works w/Docker,Kubernetes, &NativeApps;VeryFlexible
Additionallayersaddsmorecomplexities
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
ClusterManagement
9
CoreOSLinuxOS
Manyserversin…DataCenterAWSAzureetc…
Howdoweeffectivelyusealloftheseresources?Datacenter,Azure,AWS,
GCE,etc…
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Mesos Master&Agents
10
Mesos Master5050/TCPbydefaultDistributesTasks
Mesos Agent5051/TCPbydefaultExecutesTasks
CoreOSLinuxOS
Datacenter,Azure,AWS,GCE,etc…
Master
AgentAgent
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Mesos istheKernelofDC/OS
11
Mesos isthekernelofthedistributedoperatingsystemknownasDC/OS
Datacenter,Azure,AWS,GCE,etc…
Kernel:Master
AgentAgent
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Frameworks
12
Frameworks
providethelogic
Init Jobs— Marathon
Cron JobsChronosMetronomeDatacenter,Azure,AWS,
GCE,etc…
Kernel:
Frameworks:
Master
AgentAgent
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Supporting:ConfigurationStores
13
ConfigurationStores
keepeveryoneonthesamepage
ZooKeeper
Etcd
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Supporting:
Master
AgentAgent
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Supporting:Discovery
14
Discovery
Enablesthefindingofotherserviceswithinthecluster
Mesos DNS
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Supporting:
Master
AgentAgent
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
DC/OSDesign
15
Containersw/Apps
DockerContainers
WebAppsetc…
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
InternetAccessibleContainers
16
Containersw/Apps
PublicInternetAccessible
PrivateInternal
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Scenario
17
InitialAccess(RCE)
Viaavulnerablewebapplication
Intoacontainer
Aslimiteduser(e.g.www-data)
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Scenario:RCEviawebappwithinacontainer
18
e.g.JBoss,Tomcat,OSGi Console,Axis2,etc…
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
ReconviaMesos DNS
19
Queryviapivot:
Mesos DNS53/UDP&TCP— DNSservice
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
.mesos TLD
20
Theeasywaytofindserviceswithinthecluster
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
ReconviaMesos DNS
21
Queryviapivot:
Mesos DNS
8123/TCPbydefault— DNSviaRESTAPI
ServiceDiscover— withintheCluster
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
Undocumented?
22
/v1/enumerate->allmesos dns information
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateMesos DNSusingRESTAPI
23
/v1/enumerate->allmesos dns information
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
FindIP&RHPTCPportsofallservices
24
/v1/enumerate->allmesos dns information
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure:DisableRiskyMesos DNSFeatures
25
Disablethe…
“AXFR”
“Enumerate”
APICalls
• Harderforattackertodiscoverallservices
• Applicationsshouldn’tcommonlybeusingtheseAPIcallsDatacenter,Azure,AWS,
GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
ReconviaMesos Master
26
Queryviapivot:
Mesos Master5050/TCPbydefaultDistributesTasks
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateMesos Master
27
RequestviatheRESTAPI
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateMesos Master
28
Response:json w/allMesos Agent’sIPaddresseswithinthecluster
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
ReconviaMesos DNS
29
Queryviapivot:
Mesos Agent5051/TCPbydefaultExecutesTasks
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateMesos Agent
30
RequestviatheRESTAPI
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateMesos Agent
31
Response:json w/whatcontainersarecurrentlyrunningontheserver(i.e.basic0012)
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure:LogicalInternalNetworkSegmentation
32
Separatesoutthenetworkintozones:
Appsw/Data
Management
CommonlywithCalico,Canal,orFlannelDatacenter,Azure,AWS,
GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
SecretsviaConfigurationStore
33
EtcdRHP/TCPbydefault— 2379/TCP
client/server— 2380/TCPpeersConfigurationStore— CoreOSFleetsUnits— Applications
ZooKeeper2181/TCPbydefault— BinaryProtocol
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateEtcd
34
RequestviatheRESTAPIrecursively
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
EnumerateEtcd
35
Response:json frequentlycontainingsecretsincludingcredentials
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure:SeparateConfigurationStores
36
Separateouttheconfigurationstoresintozones:
Appsw/Data
Management
Enforceseparationvia…
AuthenticationCredentialsand
LogicalNetworkSegmentationDatacenter,Azure,AWS,
GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Frameworks
37
MarathonLongRunningServices— e.g.Containers
Ensuresalwaysrunning
ChronosCron fortheClusterBatchJobs
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
RCEviaMarathonJobs
38
RequestviatheRESTAPI
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
RCEviaMarathonJobs
39
MarathonLongRunningServices— e.g.Containers
Ensuresalwaysrunning
ChronosCron fortheClusterBatchJobs
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
RCEviaMarathonJobs
40
Response:json withthemaliciousjobstatus
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
RCEviaChronos Jobs
41
MarathonLongRunningServices— e.g.Containers
Ensuresalwaysrunning
ChronosCron fortheClusterBatchJobs
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure:EnforceAuthentication
42
Applicationsmust…
supportandbeconfiguredtouseauthenticationaswell
securelystoreandusecredentials
bedeployedsecurelyand/orretrievecredentialssecurely
Alertonbruteforceattempts
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
CredsviaMitM withARPSpoofing
43
AnotherContainerhastheCredsforMarathon
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
CredsviaMitM withARPSpoofing
44
AttackerusesARPspoofingtoredirectthatcontainerstraffictothecompromisedcontainer
Attackercollectthecredentials
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE ARP
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
CredsviaMitM withARPSpoofing
45
AttackercannowcreatemaliciousMarathonjobsNegatingauthenticationsecuritycontrols
Datacenter,Azure,AWS,GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE ARP
RCE
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
ServerServer Server
Secure:TLSforInternalCommunications
46
EnableTLSw/validcertificatesforstrongHTTPScommunications
AnythingusingcredentialsneedsTLS!
ValidateCertificates
Failclosedonbadcertificates
AlertoncertificateserrorsDatacenter,Azure,AWS,
GCE,etc…
Kernel:
Frameworks:
Apps:
Supporting:
Master
AgentAgent
CntrApp
CntrApp
CntrApp
CntrApp
Internet
RCE ARP
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
StrategicActions
47
Nextweek:AssesswhichservicesyoucanenableAuthentication&TLSon— w/obreakingyourexistingapplicationswithinthecluster
Threemonthsfromnow:ImplementAuthentication&TLSonsafeservicesandframeworks— Focusingonservicesresponsiblefororchestrationwithinthecluster
DeployseparateserviceswherepossibleforAppsthatdonotsupportTLS&Auth
Sixmonthsfromnow:RetrofitallApplicationswithintheclustertouseTLS&AuthenticationEnforcetheuseofTLS&Authenticationinternaleverywhere(disableclear-text)
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
BigPicture
48
ContainerAdoptionIsMaturing,especiallyinEnterprises
Enterprisesareusingcontainersinproduction.
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
BigPicture
49
Pivoting fromacompromisedservicewithintheclusterNocontainerbreakout/0day/exploitneededJ
Mayenableanattackertocompletelycompromisethecluster
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
BigPicture
50
LookingBeyondtheBorder
withaDefenseinDepthstrategy
SecurestheFuture&thecluster
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
FutureResearch
52
TestingMitM fromcompromisedcontainerNCCGroup’sreportstatesthisispossibleforco-hostedcontainers
TestdowngradeHTTPScommunicationsCanwedowngradefromHTTPStoHTTPandcapturecredsfromanothercontainer?
TestCerts(e.g.cancertpinningbeenabled?)toRESTAPIsCanweMitM andimpersonatetheAPIservice?
TestAuthenticationBruteforceattacksFairlycertaintherearenolockouts,canweenablebetterauthenticationsecurity?— Writemoduletobrute-forceandguesscreds
TestLogicalNetworkSegmentationToolsCalico,Canal,Flannel— Note:theseshouldworkasadvertisedbutprobablyweshouldindependentlyverify
Presenter’s Company Logo – replace or
delete on master slide
#RSAC
References
53
https://www.cloudfoundry.org/wp-content/uploads/2016/06/Cloud-Foundry-2016-Container-Report.pdf
https://clusterhq.com/assets/pdfs/state-of-container-usage-june-2016.pdf
http://www.rightscale.com/blog/cloud-industry-insights/new-devops-trends-2016-state-cloud-survey