lascon 2016 - its 10pm do you know where your access keys are?

@cktricky

Download this presentation:

<insert link>

@cktricky

It's 10pm, Do You Know

Where Your Access Keys Are?

Ken Johnson

@cktricky@cktricky

Things to Mention

• DoubleTree by Hilton at 8901 Business

Park Drive in Austin, TX is great at:

– Selling your room for you without telling you

– Fire alarms

– Murdering puppies and kittens

– Created cancer?

– Created cancer in puppies and kittens?

@cktricky@cktricky

Things to Mention

• Ask questions throughout presentation

• There will be no dedicated Q&A – so stick

around after and find me if you want to

chat

• This presentation will cover a lot. Slides

will be available so don’t worry about

minutia.

@cktricky@cktricky

Background/About

• Ken Johnson, CTO and Partner at nVisium

• Veteran, US Navy

• I speak about:

– DevOps (In)Security

– Exploiting Web Applications

– Coding and Coding + Security

– Node, Elixir, Python, Ruby, Go

– AWS Security (clearly)

@cktricky@cktricky

Background/About

This talk came about because…

– I’m the CTO of a security company and we use

AWS… and it is a challenge

– For some, this is a new challenge, and this is my

opportunity to share

@cktricky@cktricky

Background/About

50k foot view of our security plan

– Prevent bad stuff

– Alert to bad stuff

– Recover from bad stuff

@cktricky@cktricky

Our Plan

Our “practical plan”

– Harden – Make it difficult to reach our AWS

environment

– Monitor – If our AWS environment is breached, we

need to know and alert ourselves

– Restore – Have the ability to reconstruct data/configs

after a “hack”

@cktricky@cktricky

AWS’s Plan

The AWS Security Fundamentals Course provides

the framework for your plan:

– You are responsible for leveraging the tools AWS

provides to secure your environment (financially)

– Your configuration… that is on you

– https://aws.amazon.com/training/course-

descriptions/security-fundamentals/

https://aws.amazon.com/training/course-descriptions/security-fundamentals/

@cktricky@cktricky

Most Security Checklists

Most AWS security talks and documentation

discuss:

– S3 bucket policies

– Security Group configurations

– SSH Key Management

– Encrypting Data (Volumes, S3 buckets)

@cktricky@cktricky

Most Security Checklists

What we’ll mention on the subject:

1. Trusted Advisor – Use it, because it catches a lot of

“low hanging fruit” style issues

2. There are checklists, use them:

– https://media.amazonwebservices.com/AWS_Operational_Che

cklists.pdf

– http://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing

_Security_Checklist.pdf

3. Again let’s reiterate that AWS provides a security

fundamentals course for free (CBT)

https://media.amazonwebservices.com/AWS_Operational_Checklists.pdf

http://d0.awsstatic.com/whitepapers/compliance/AWS_Auditing_Security_Checklist.pdf

@cktricky@cktricky

About / Background Recap

Recap:

– We’re not going to cover basic security fundamentals

of unencrypted volumes, security groups, etc.

– We are going to focus on:

• Hardening

• Monitoring

• Recovery

@cktricky

@cktricky

AWS IAM Hardening Basics

Making it difficult (for attackers) to reach our environment

@cktricky@cktricky

IAM Hardening Checklist

1. Don’t Use The Root Account!

2. Remove Access Keys for Root Account

3. Audit IAM user policies

4. Multi-Factor Authentication

5. API + MFA

6. Strong Password Policy

@cktricky

AWS Root Account

@cktricky@cktricky

Don’t Use The Root Account

Every AWS environment has a root account

– Root account is the king/god/all-powerful

– Use only when you absolutely must

– When those circumstances arise, notify your team

first

– This is because we will be configuring alerts to notify

our team when the root account is used

@cktricky@cktricky

Remove Access Keys for Root Account

Simple steps:

– Disable or delete access keys if they exist:

– Implement verbal/written policy that states “we don’t

create access keys for the root account”

@cktricky

Auditing IAM Permissions

@cktricky@cktricky

Audit IAM User Policies

IAM user policy management:

– A single IAM user can have…

• Multiple Managed Policies

• Multiple Inline Policies

• Belong to multiple IAM Groups which…

– Have multiple managed policies

– Have multiple inline policies

@cktricky@cktricky


Explanation

– Managed Policies: Policies that can be

attached to multiple users, groups, or roles

– Inline Policies: Directly attached to a single

user, group, or role

@cktricky@cktricky


Tool to inspect each user’s permissions:

– https://gist.github.com/cktricky/257990df2f36aa3a01a

8809777d49f5d

– Will create a CSV file

– Provides you with

• Usernames

• Inline Policies

• Managed Policies

• Groups

https://gist.github.com/cktricky/257990df2f36aa3a01a8809777d49f5d

@cktricky@cktricky


Tool output

@cktricky@cktricky


Closer look:

@cktricky@cktricky


@cktricky@cktricky


Why this is important

– If you house sensitive data, you need to know who

has access

– Permissions should be a need-to-have/know situation

in order to limit damage should creds get stolen

– AWS is a flexible environment that changes – your

permission model might need to change with it

(inventory it)

@cktricky

Multi-Factor Authentication (MFA)

@cktricky@cktricky

MFA

• MFA == 2-Factor Authentication

• If credentials are stolen or guessed, we want a second

layer of protection

• You can use apps or hardware to do this

– Google Authenticator (Apps)

– Gemalto (Hardware)

• Find the full list of MFA devices here:

https://aws.amazon.com/iam/details/mfa/

https://aws.amazon.com/iam/details/mfa/

@cktricky@cktricky

MFA

Let’s demonstrate enabling MFA using a virtual

device (app) on an IAM account

@cktricky@cktricky

MFA

Navigate to Identity & Access Management

@cktricky@cktricky

MFA

Next, manage the MFA device

@cktricky@cktricky

MFA

Choose a virtual device

@cktricky@cktricky

MFA

Lastly, use Google Authenticator to take a snapshot of the

QR code

@cktricky@cktricky

MFA

• At this point, its worth mentioning that non-

administrators or those without IAM privileges

cannot enable MFA on their own account

• Why is this a problem? Well, they need to be

able to enable MFA on their own device… not

the administrator’s

• Fortunately, we have a solution!

@cktricky@cktricky

MFA

@cktricky@cktricky

MFA

• Okay so that wasn’t the easiest to read, so

here is the link:

http://docs.aws.amazon.com/IAM/latest/Us

erGuide/id_credentials_delegate-

permissions_examples.html#creds-

policies-mfa-console

• Basically this IAM policy allows a user to

manage their *OWN* MFA device

http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_delegate-permissions_examples.html#creds-policies-mfa-console

@cktricky@cktricky

MFA (for Root Account)

• Need a shared MFA for root? TOTP!

• Recommend using something like

1password for teams, can share the TOTP

code: https://support.1password.com/guides/mac/totp.html

https://www.youtube.com/watch?v=eZyb-ArMK9g

https://support.1password.com/guides/mac/totp.html

https://www.youtube.com/watch?v=eZyb-ArMK9g

@cktricky

API & MFA

@cktricky@cktricky

API + MFA

API 101

– This is the alternative to interacting with the AWS

environment via the web console

– Typically used for automated tasks

– Automated tasks means “code”. Luckily, developers

never store keys in source, amiright?

– Hypothetically, what would happen if keys were

leaked?

@cktricky@cktricky

API + MFA

@cktricky@cktricky

API + MFA

So that’s the “worst case scenario”, more likely:

– Costs unexpectedly and dramatically increase

– We’ll show examples later but remember, you are

financially responsible for your AWS environment’s

configuration

– Let’s talk about prevention

@cktricky@cktricky

API + MFA

• You have the ability to place a restriction where

resources can only be interacted with if the user

has authenticated with MFA

• This helps prevent (ab)use should someone

steal access keys or credentials

@cktricky@cktricky

API + MFA

1. At a minimum, apply to administrator & power user

group policies… really any group that can do anything

of importance

@cktricky@cktricky

API + MFA

This entry requires MFA for Web/API

@cktricky@cktricky

API + MFA

• Truth be told, doing this can be painful at

first

• Things that used to work, might not (via

the API)

• Fortunately, we have some answers for

you

• Firstly, let’s discuss STS or SecurityToken

Service

@cktricky@cktricky

API + MFA

• Leverage STS in order to interact with the

AWS API should this MFA restriction be

placed on resources (and it should )

• Example of using STS:

https://gist.github.com/cktricky/127be4e431563a986f0f

https://gist.github.com/cktricky/127be4e431563a986f0f

@cktricky@cktricky

API + MFA

Use this script to retrieve creds (from gist)

@cktricky@cktricky

API + MFA

Output of script

@cktricky@cktricky

API + MFA

Use the creds to leverage tools like ec2-api-

tools(-O <access key id>–W <secret> and –T <session token>)

@cktricky@cktricky

API + MFA

And in case you don’t like Ruby…

https://github.com/jimbrowne/aws-sts-

helpers

https://github.com/jimbrowne/aws-sts-helpers

@cktricky@cktricky

API + MFA

• ElasticBeanstalk does not work with STS. Le

Terrible.

• However, there is a workaround, use

CodePipeline.

• Very simple process to setup but only works

with:

– GitHub

– AWS CodeCommit

– Amazon S3

@cktricky@cktricky

API + MFA

Remember MFA only protects against the web and

NOT the API… unless you change your policies

and use STS

@cktricky

Password Policy

@cktricky@cktricky

Password Policy

• Password policies are important because

historically people do not choose complex

passwords

• MFA should help, but we’re talking about a

layered approach

• Again, making our AWS environment

harder to reach

@cktricky@cktricky

Example Password Policy

@cktricky@cktricky

Hardening Recap

• Make credentials hard to guess

• Make credentials hard to use if stolen with

MFA

• Audit your accounts and their access

• Root account is King, protect your King

@cktricky

AWS Monitoring

Detecting malicious activity

@cktricky@cktricky

AWS Monitoring

• Assuming hardening (prevention) has failed,

how would we know?

• Luckily, AWS provides several services which

alert to anomalies

• We will walk through examples of using these

services, but ultimately decide what is right for

you

• Fair warning, some of these services will provide

a lot of noise

@cktricky@cktricky

AWS Monitoring

4 important services:

1. CloudTrail – Logs

2. SNS – Notifications

3. Config – Alerts for modifications &

noncompliance

4. CloudWatch – Alerts for specific types of

behavior

@cktricky@cktricky

AWS Monitoring

@cktricky@cktricky

AWS Monitoring

CloudTrail

Config

CloudWatch

SNS

@cktricky@cktricky

AWS CloudTrail

@cktricky@cktricky

AWS Monitoring (CloudTrail)

• CloudTrail is primarily used for log collection

• Other services like CloudWatch, for example,

use those logs to filter relevant data

@cktricky@cktricky


Pretty easy, first turn it on..

@cktricky@cktricky


Configure the log group

@cktricky@cktricky


Allow the creation of an IAM role by CloudTrail

@cktricky@cktricky


• At this point you have cloudtrail enabled

• Next step, BEFORE moving to CloudWatch or

Config, is configuring SNS topics

@cktricky@cktricky

AWS SNS

@cktricky@cktricky

AWS Monitoring (SNS)

Fantastic offering, <3 it

– Examples of ways to be notified by SNS

• SMS

• Email

• JSON Post to your Application’s API endpoint

• Lambda

@cktricky@cktricky


• Receive SMS/Email/Slack notifications for

important events

• ^ This is so you get immediate notifications

• You can have multiple subscribers, I’d suggest

you use that functionality

• Basic gist? Receive immediate updates for

things you want to see… immediately ☺

@cktricky@cktricky


Create a topic

@cktricky@cktricky


Create Subscription

@cktricky@cktricky


Create SMS (or whatever, but in this case, SMS)

@cktricky@cktricky


Example of creating email subscription… bottomline you

can have multiple ways of notifying multiple people

@cktricky@cktricky

AWS Config

@cktricky@cktricky

AWS Monitoring (Config)

Config:

– AWS resource inventory, configuration history, and

configuration change notifications

– Can either design custom Config rules or use

managed (pre-packaged) AWS Config rules

– Discovery

– Change Management

– Compliance

– Incident Response

@cktricky@cktricky


Pre-packaged “Managed” AWS Rules

– CLOUD_TRAIL_ENABLED

– EIP_ATTACHED

– ENCRYPTED_VOLUMES

– INCOMING_SSH_DISABLED

– INSTANCES_IN_VPC

– REQUIRED_TAGS

– RESTRICTED_INCOMING_TRAFFIC

– ROOT_MFA_ACCOUNT_ENABLED

– RDS_STORAGE_ENCRYPTED

@cktricky@cktricky


…And there are more and this list grows

https://docs.aws.amazon.com/config/latest/d

eveloperguide/evaluate-config_use-

managed-rules.html

https://docs.aws.amazon.com/config/latest/developerguide/evaluate-config_use-managed-rules.html

@cktricky@cktricky


Examples of things you can have alerts set for:

– Change in Firewall (Security Group) ports

– Changes in VPC

– Any change… at all

@cktricky@cktricky


Go to the Config service and choose resources to track

@cktricky@cktricky


Or choose to track everything

@cktricky@cktricky


Create a bucket, create an SNS topic (…we’ll discuss next)

@cktricky@cktricky


Allow the role to be created and you’re all set!

@cktricky@cktricky

AWS CloudWatch

@cktricky@cktricky

AWS Monitoring (CloudWatch)

• We can be very particular here about what it is we want

to see

• Some very interesting things you can monitor

• Some examples:

– Billing Alerts (Important for detection of abuse or

mistakes)

– Track Root Account Usage

– Failed login attempts

– Unauthorized Activity

@cktricky

Billing Alarm

@cktricky@cktricky

AWS Monitoring (CloudWatch - Billing)

• Used to prevent abuse or mistakes from costing your

organization money

• Analyze and approximate your monthly spend

• Configure via CloudWatch

• Use SNS for instantaneous alerting

@cktricky@cktricky


Navigate to billing & cost management; enable

billing alerts

@cktricky@cktricky


Create an SNS topic

@cktricky@cktricky


Subscribe to Topic

@cktricky@cktricky


Navigate to CloudWatch -> Metrics -> Billing

@cktricky@cktricky


Choose USD/EstimateCharges -> Create Alarm

@cktricky@cktricky


Set price point, SNS topic, and create alarm

@cktricky@cktricky


Exact steps to enable can be found here:

http://docs.aws.amazon.com/awsaccountbilli

ng/latest/aboutv2/free-tier-alarms.html

http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/free-tier-alarms.html

@cktricky

Root Login

@cktricky@cktricky

AWS Monitoring (CloudWatch – Root Login)

• Remember how I said don’t use the Root

account routinely?

• BUT… if this account is used, you should

know about it

• This is the reason you’ll want to notify

others (who receive SNS alerts) of the fact

you are about to use the account

@cktricky@cktricky


Choose log group, create metric

@cktricky@cktricky


Define Logs Metric Filter

@cktricky@cktricky


Assign/Create Filter

@cktricky@cktricky


Click “Create Alarm”

@cktricky@cktricky


Define Alarm and you’re good…

@cktricky@cktricky


Exact steps (with pics) exist here:

https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8

374D/How-to-Receive-Notifications-When-Your-AWS-

Account-s-Root-Access-Keys-Are-Used

https://blogs.aws.amazon.com/security/post/Tx3PSPQSN8374D/How-to-Receive-Notifications-When-Your-AWS-Account-s-Root-Access-Keys-Are-Used

@cktricky

Failed Logins

@cktricky@cktricky

AWS Monitoring (CloudWatch – Failed Logins)

• In the event someone is trying to break in,

let’s alert ourselves to this!

• Failed logins typically suggest either

someone forgot their password or…

someone is trying to guess yours

@cktricky@cktricky


The steps are pretty much the same as the

root login alarm

However, the Filter pattern is different

@cktricky@cktricky


Enter the relevant filter pattern, click create

@cktricky@cktricky


• Exact steps exist here:

http://docs.aws.amazon.com/awscloudtrail/la

test/userguide/cloudwatch-alarms-for-

cloudtrail.html#cloudwatch-alarms-for-

cloudtrail-signin

http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html#cloudwatch-alarms-for-cloudtrail-signin

@cktricky

IAM Unauthorized Activity

@cktricky@cktricky


• Aws-interrogate tool

• This alarm is the antidote

• Alerts us when someone is trying to

access something in AWS, and does not

have permissions

@cktricky@cktricky


• Steps are same as root login, failed logins,

etc.

• Filter pattern is different

@cktricky@cktricky


Enter relevant filter pattern

@cktricky@cktricky

AWS Monitoring (Unauthorized Activity)

What happens when we run interrogate

@cktricky@cktricky

AWS Monitoring (Unauthorized Activity)

The result of doing that is a nice nifty email to the

engineering & security team

@cktricky@cktricky

AWS Monitoring (CloudWatch) – Filter Patterns

Create your own custom filter patterns, here is a

resource for that:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/De

veloperGuide/FilterAndPatternSyntax.html

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/FilterAndPatternSyntax.html

@cktricky@cktricky

Lambda+ Slack

@cktricky@cktricky

Lambda & Slack

Select slack-cloudwatch blueprint when creating

function

@cktricky@cktricky

Lambda & Slack

Configure trigger

@cktricky@cktricky

Lambda & Slack

The function is pre-populated (coded)

@cktricky@cktricky

Lambda & Slack

Start configuring incoming webhook

@cktricky@cktricky

Lambda & Slack

Add configuration inside of slack

@cktricky@cktricky

Lambda & Slack

Choose the channel (can also choose pic, name,

etc.)

@cktricky@cktricky

Lambda & Slack

Grab the webhook URL

@cktricky@cktricky

Lambda & Slack

Create KMS key, later used to decrypt

@cktricky@cktricky

Lambda & Slack

Name the key, follow steps 1 - 4

@cktricky@cktricky

Lambda & Slack

Use the AWS KMS encrypt function to

encrypt the webhook URL

@cktricky@cktricky

Lambda & Slack

Put the Base 64 encoded + KMS encrypted URL

into the code:

@cktricky@cktricky

Lambda & Slack

The result:

@cktricky@cktricky

AWS + Splunk

@cktricky@cktricky

AWS + Splunk

Splunk is a pretty great resource for monitoring

activity

• Two separate plugins:

– Splunk App for AWS

• https://splunkbase.splunk.com/app/1274/

– Splunk Add-On

• https://splunkbase.splunk.com/app/1876/

https://splunkbase.splunk.com/app/1876/

https://splunkbase.splunk.com/app/1876/

@cktricky@cktricky

AWS + Splunk

• Examples of things you can view:

– Billing

– Topology

– Usage

– IAM Activity

– SSH Key Pair Activity

– User Activity

– Network ACL(s)

– VPC Activity

and a lot more…

@cktricky@cktricky

AWS + Splunk

• Pretty Screenshot 1

@cktricky@cktricky

AWS + Splunk


@cktricky@cktricky

AWS + Splunk

• Splunk will need an AWS account in order

to retrieve data

• Create account(s) for Splunk, grab the

necessary permission policy from here:

http://docs.splunk.com/Documentation/AddOns/r

eleased/AWS/ConfigureAWSpermissions

http://docs.splunk.com/Documentation/AddOns/released/AWS/ConfigureAWSpermissions

@cktricky@cktricky

AWS + Splunk

• Configure AWS App for Splunk, add account(s),

configure each input accordingly:

@cktricky@cktricky

AWS + Splunk

• To view things like IAM Activity…

– Subscribe to a cloudtrail log via SNS

– Utilize SQS and subscribe SQS to an SNS

Topic

@cktricky@cktricky

AWS Monitoring Recap

• Alert yourself when things change

• This will get noisy, find a way to filter that which is

important

– If it’s a high risk event, send an SMS/Slack/Email

blast

• At a minimum, alert yourself when odd things occur…

like:

– Billing increases past your normal spend

– When somebody authenticates as Root

– When someone has a login failure

– Unauthorized IAM Activity

@cktricky@cktricky

AWS Monitoring Recap

• Interesting Quora thread:

– https://www.quora.com/My-AWS-account-was-hacked-and-I-

have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay

• Highlights from the article:

– AWS has “a review board of sorts” to determine if you should be

refunded

– Bots are scouring GitHub searching for exposed access keys

– One of the more AWS-seasoned responders mentioned doing

part of what we discussed here today to avoid it

– A decent number of the people posting on this thread said “Yes,

happened to me too”

https://www.quora.com/My-AWS-account-was-hacked-and-I-have-a-50-000-bill-how-can-I-reduce-the-amount-I-need-to-pay

@cktricky

AWS Restoration & Recovery

Plan to fail, just don’t fail to plan

@cktricky@cktricky

AWS Restoration & Recovery – Basic Incident

Response (IR)

• Understand who to contact if things go bad

• Understand how to communicate (ex:

“speak only over the phone”)

• Understand what information to parse

• Understand where your backups are

located and how they are secured

@cktricky@cktricky

AWS Restoration & Recovery – Basic IR

• Do not USE AWS TO BACKUP YOUR

AWS

• Offsite backups (meaning, off AWS site)

• Common things to back-up:

– Databases/ Snapshots

– S3 Buckets

– EBS Volumes

– CloudFormation Templates

@cktricky@cktricky

AWS Restoration & Recovery – Basic IR

• Resources:

– http://stackoverflow.com/questions/17087542/

backup-solutions-for-aws-ec2-instances

– https://github.com/Scalr/installer-ng

– http://www.n2ws.com/blog/3-ways-ec2-

windows-backup-and-recovery.html

http://stackoverflow.com/questions/17087542/backup-solutions-for-aws-ec2-instances

https://github.com/Scalr/installer-ng

http://www.n2ws.com/blog/3-ways-ec2-windows-backup-and-recovery.html

@cktricky

AWS Incident Response

Resources

@cktricky@cktricky

AWS Incident Response

• Scout 2 -- https://github.com/nccgroup/Scout2

• Andrew Krug & Alex McCormack – Hardening AWS

Environments and Automating Incident Response

– https://www.youtube.com/watch?v=cmEUxxYFjK8

https://github.com/nccgroup/Scout2

https://www.youtube.com/watch?v=cmEUxxYFjK8

@cktricky

Presentation Recap

Summary

@cktricky@cktricky

Recap

• DoubleTree by Hilton at 8901 Business

Park Drive in Austin, TX

– Sells your room

– Loves fire alarms at 5am

– Behind 9/11?

– Can go f**k itself

@cktricky@cktricky

Recap

• Makes your environment harder to reach… for

the bad guys

– Limit what stolen or “otherwise obtained”

access keys or credentials could be used to

do

– Prevent them being stolen in the first place

• Alert yourself to anomalies

• Have a plan for if things go bad

• Stay safe out there!

@cktricky

Contact Info

• My Info

• Twitter: @cktricky

• Email: [email protected]

mailto:[email protected]

lascon 2016 - its 10pm do you know where your access keys are?

Software