customer distributed denial of service (ddos) experiences - networkshop44

Denial of Services (DDoS)

Steve Knibbs

@ulcc www.ulcc.ac.uk

Denial of Services (DDoS)

Steve Knibbs

Head of Infrastructure Services

University of London Computer Centre (ULCC)

@ulcc www.ulcc.ac.uk

DDoS Experiences

What happened and what we learned

@ulcc www.ulcc.ac.uk

Never assume it won’t happen

• 50 years in business without any serious attacks

• First serious attack

in 2015

• Plan now because it will happen to you

@ulcc www.ulcc.ac.uk

Things we got wrong

Powers of Persuasion

@ulcc www.ulcc.ac.uk

Things we got wrong

• Initially attributed to a Firewall

software update

• Our ability to have visibility of

external traffic, relied on firewall

logs/interface which we could

not access

@ulcc www.ulcc.ac.uk

Things we got right

• Communication (Web site, Twitter, email)

• Roles and responsibilities

• Protection of technical staff

• Quick engagement with 3rd party support

@ulcc www.ulcc.ac.uk

What we’ve done since

• External Network Monitoring –

SNORT

• Improved our OOB admin access to

equipment

• Improved our processes (Steps to take,

informing Police, etc)

• Implemented ‘BGP blackhole’

• Implemented further anti-DoS policies

on the firewall

@ulcc www.ulcc.ac.uk

…more improvements

• New Firewall with additional security

features

• Massive improvements to our core

infrastructure

• Improved governance and senior

management awareness

• Some customers moved to a Cloud based

‘washing’ service

• DDoS mitigation services have been

considered but ruled out for the time being

@ulcc www.ulcc.ac.uk

What we’re doing next

• Dedicated security team

@ulcc www.ulcc.ac.uk

Jail Sentence

…sometimes we do catch the bad

guys!

Sentenced to four years and 10

months for carrying out cyber-

attacks and holding a cache of

weapons

@ulcc www.ulcc.ac.uk

Questions

• Thank you for your attention

• Questions

jisc.ac.uk

Steve Knibbs

Head of infrastructure services, University of London

Steve.knibbs@lon.ac.uk

Distributed Denial of Service Attacks (DDoS)

Mike Turpin

Mike Turpin

Head of Network Services, UCL

Distributed Denial of Service Attacks (DDoS)

m.turpin@ucl.ac.uk

Timeline

Thursday 12th November 15.50-16.50

Target: Mail & Web

– Blocked ~3000 IPs at MAN router

– JANET blocked UDP 1900 inbound to server

Thursday 12th November 20.00-21.00

Target: Web

– Reflected DNS and UDP fragments sourced from open DNS resolvers

– JANET rate limited those ports to 5Gb/s

Friday 13th November 14.00-15.00

Target: Shibboleth & DNS

– DNS Amplification

– JANET blocked

Friday 13th November 18.00-19.00

Target: CS DNS server

– DNS and UDP fragments

– JANET added ns1.cs to rate limit

Bandwidth

Thursday

Bandwidth

Friday

Attacks sometimes used to hide other bad things!

Cost

• Reputation

• Lost work

• Lost revenue

• Estimate £250k!

Lessons Learnt & Mitigation

• CSIRT & Network Operations teams were invaluable

• Outsourcing websites isn’t a solution (They just get disconnected!)

• Changed Firewall monitoring (Logging added to load)

• Firewall redesign with DC work (separates campus and DC traffic)

• Assessing DDoS mitigation services ( Procurement)

DDoS Mitigation Services

Commercial providers reassuringly expensive?

£8K/Month (+VAT)! ?

Cheaper Options?

*#$%

Ideal Solution

• Lower cost

• Protection on demand (maybe not always enabled)

• Option to exclude selected traffic

• Automated (out of hours?)

• Alerts

Questions?

CSIRT for tirelessly monitoring our traffic for attacks

John Seymour and his team for implementing blocks/rate limits rapidly

And for providing instant updates on the situation

Without the above help we would have had to sit it out!

jisc.ac.uk

Mike Turpin

Head of network services, UCL

m.turpin@ucl.ac.uk