Honeypots as a Tool to Improve Incident Response Readiness at USP

Alberto CamilliIsabel Chagas

Centro de Computação Eletrônica Universidade de São Paulo

Educause Security Professionals Conference 2007Denver

12 April 2007

Agenda

1) University of São Paulo, numbers and IT organization

2) USP and the national Honeynet project3) USP honeypot-based Early Notification procedure

and results 4) Q&A

Presentation Objectives

• Show how honeypots take part in the incident notification process, how they are configured and managed at USP.

• Main incident statistics for USP.

• Show what changes in the management of honeypots at USP lead to a change in the campus profile of incidents, with reduction in quantity and in the solution times.

University of São PauloResearch Extensive

State University (São Paulo)

185 in ISI rank:

31.548 indexed papers

185.110 citations

2.270 PhD, 3.218 MSc

48.530 Undergraduate students

4.884 faculty

14.952 staff

32.059 graduate students

8 main campuses, 60% based in São Paulo (city)

85 “federation-like” Units (Teaching, Services and Extension)

Annual budget near US$ 700 M

Centro de Computação Eletrônica da USP

• Main USP Computer Center• NOC and CSIRT 24x7 operation• 400m2 data center• HPC main facilities for the University (e.g. computer

363o. in Top500 list)

USP Network Connections

USPBauru

USPPirassununga

USPPiracicaba

USPSão Carlos

USPRibeirão Preto

Hospitals

MedicalSchools

Telefonica

RegistroBR

RJMG

SC

RS

SP

Internet USP

Enclave

USP( Cidade Universitária )

KyateraTest Bed

GIGATest Bed

METRO IX( Non Commercial )

POP Clara( Global Crossing )

GEANT( EUROPE / SPAIN )

TV

POP RNP( USP )

USPnet

MetroSP

ANSP

InternetCommodity

USA and Brazil

Internet 2USA

• USPnet600Mbps I/O Internet commodity traffic

1,000 Buildings; 50,000 network points

25.000 (80.000) e-mail accounts (@usp.br)

850,000 e-mail/day (20% valid)

InternalExternal

USP IT organization

Dean

CIO

BusinessIT

ID_#inc

CSIRTUSP

LocalCSIRT1

LocalCSIRT2

Administration

ID_#inc

ID_#inc

CSIRTsNotifications

Internal-ExternalCampus Notifications

InternalCampus Notifications

Internal Units

USP business rules

CampusComputerCenters

IT services andinfrastructure campus 1

campus nExternal

143.107.xxx.xxx /16200.136.xxx.xxx /20200.144.xxx.xxx /20 Adm.

USP Cidade Universitária

USP Ribeirão Preto

USP Piracicaba

USP São Carlos

Coordination by CERT.br: http:///www.honeypots-alliance.org.br/

USP and the Brazilian Honeypot Alliance

The Project Brazilian Honeypots AllianceDistributed Honeypots Project

• Coordination: CERT.br and CenPRA Research Center• Use of low interation honeypots• Based on voluntary work of research partners

– 37 research partner instituitions • Industry, telcos, academic (USP and others), government and military

networks

– Each partner provides• Hardware and network• Honeypot(s) maintenance

USP Motivation

• To increase USP’s capacity of:– Incidents identification and knowledge– Incident detection – Event correlation with other Entities– Trend analysis– Which Units are more vulnerable?

• Very Useful for Incident Response• Sensors distributed in several campuses

USP Honeypots Location

Brazilian Honeypot Alliance Architecture

Netblock range from /28 to /24

Data Usage• Incident response (CERT.br):

– Identify well known malicious/abuse activities• Worms, bots, scans, spams and malwares in general

– Notify the Brazilian networks´contacts• Including recovery tips

• Partners:– Observe trends and scans for new vulnerabilities– Detect promptly:

• Outbreaks of new worms/bots• Compromised servers• Network configuration errors

• What about USP usage?

Honeypots project: how good it was (july06) after 3 years?

Evolução dos incidentes out/2002 - dez/2006incidentes por ip

0

50

100

150

200

250

300

350

out/02

jan/03

abr/03

jul/03

out/03

jan/04

abr/04

jul/04

out/04

jan/05

abr/05

jul/05

out/05

jan/06

abr/06

jul/06

out/06

Defacement Dos/Ddos Harvesting

Invasão Mau Uso Open-proxy

Open-Relay Scam/Phishing Scan

Spam Worm/Trojan Honeypot ( USP )

Misconfig. força bruta (set/2006)

But ... how better can it become?

3 years

Gigabit backbone

External Internal

90% 10%

July 06

Honeypot (CERT.br)

A closer look at our honeypot data...

Different External IPsPort 80

Protect applications! Protect backbone routers!

1. Threats from the outside

A closer look at our honeypot data ...

same subnettype of attack, e.g.: 135(tcp), 445(tcp)

Protect Windows desktop! Protect subnet!

2. Where and how the internal network is being attacked

Worm Propagation Times300 min = 5 hs

~ 10 hs

[1] Zou C., Gao L., Gong W. Monitoring and Early Warning for Internet Worms. CCS’03

Worm propagation Model [1]

Typical log for a honeypot at USP

then ... early notification?

Timeline logic in Early Notification

NotificationCert.br

NotificationCSIRT-USP

End ofIncident

Time

Tne Tca

•Contamination (int)•Improper Action (ext)

Tp1 Tni

CorrectionAction

Propagation

CERT.brCSIRT

HandlingCSIRT

HandlingLocal CSIRT

NotificationCERT.brORNotificationCSIRT-USP

Tn = max(Tne,Tni)Tp2 Tca

Tp2 = Tp1 – some measured averageTca ??

Time

Early Notification (EN) procedure• Hypothesis

– Unnoticed attacks should now begin to be identified.– CSIRT-USP is able to notify attacks in advance.– Units will be able to react accordingly to block these attacks.

• What we did?– Notifiy the victims as soon as an internal attack is being observed– No further considerations about the nature of the attack.

• Why?– We want better incident scores and honeypot logs are at our disposal.

• How and when?– A daily script generates a summary of the attacks. Each attacked Unit receives

the summary notification from CSIRT-USP, as a new security incident ticket.

Internal notifications message format

• CSIRT USP messages (daily summary):– Subject: [Honeypot] Máquina(s) suspeita(s) (20070309)– Content:

143.107.zzz.yyy : 139(247) 143.107.nnn.mmm : 135(202) 137(573) 139(3041) 1433(183) 445(1302) 80(568)

• Cert.br messages (on IP basis):– Subject:143.107.zzz.yyy: host(s) infectado(s) com Agobot/Phatbot Content:

Apr 27 13:30:15.918576 143.107.zzz.yyy.3683 > xxx.xxx.xxx.22.139: S [tcp sum ok] (src OS: Windows XP SP1, Windows 2000 SP4)

904637194:904637194(0) win 65535 <mss 1460,nop,nop,sackOK> (DF) (ttl 123, id 20447, len 48)

number of attempts

Internal notifications results after 6 months of EN adoption

Same data, Different analysis criteria,Different Interpretations

Honeypot USP Notification Share

78 195

9863

82 30

1740

2 2

329

15 12

83 43

0 1 7 4

109 41 131 124

0%

20%

40%

60%

80%

100%

Both

Cert.BR

sensorUSP

Antecipated 6hsby CSIRT-USP

Overview of EN resultsEvolução dos incidentes out/2002 - dez/2006

incidentes por ip

0

50

100

150

200

250

300

350

out/02

jan/0

3

abr/0

3

jul/03

out/03

jan/0

4

abr/0

4

jul/04

out/04

jan/0

5

abr/0

5

jul/05

out/05

jan/0

6

abr/0

6

jul/06

out/06

Defacement Dos/Ddos Harvesting

Invasão Mau Uso Open-proxy

Open-Relay Scam/Phishing Scan

Spam Worm/Trojan Honeypot ( USP )

Misconfig. força bruta (set/2006)

CSIRT USP/Cert.BR internal notifications2002-july 2006: internal notifications only from Cert.br

NotificationsIncrease

External Internal

Before EN 90% 10%

After EN 50% 50%

Classification of Incidents at USP

Definition Notification Expression or

Symptoms

Characteristics Possible Causes or

Aggravants

Internal CSIRT from USP alerting local Units worm/trojan in USP’s local nw

•CSIRT USP•Cert.br

•Port Scans•Brute Force

•Correlated events•Self propagating•Difficult to correct•Difficult to isolate•Exploring address space

•Non protected machines•Unpatched machines (mostly Windows)•Infected machines•Open Services

External External entities complaining with USP about a problem in their nw

•External entities•Cert.br•External CSIRTs

•P2P •Defacement•Spam•Open Proxies•Open/Relays•Phishing/Scam

•Isolated Events •Not self propagating•Easy id of causes•Easy correction

•Inadequate user behaviour•Bad configuration•Long term contaminated machines

Top 10 Units solution time(before and after EN)

Tca(days)

Incident SolutionTime (avg)

ΔT%Internal Incident Solution Time (avg)

ΔTi%External Incident

Solution Time (avg)

ΔTe%

01-06 07-12(EN)

01-06 07-12(EN)

01-06 07-12(EN)

ID_173 10 9,4 -6% 6,5 10,8 66% 10,3 7,2 -30%

ID_146 16,8 11,4 -32% 13,4 11,3 -16% 16,9 11,6 -31%

ID_ 112 20,4 15,5 -24% 11,2 15,2 36% 22,2 12,6 -43%

ID_ 86 12,4 15,9 +28% 7,0 8,2 17% 12,5 14,7 +18%

ID_70 18,5 16,1 -13% 4,6 16,2 152% 20,3 19,4 -4%

ID_55 7,6 6,5 -14% 10,3 5,8 -44% 6,8 7,2 +6%

ID_ 39 15,9 8,1 -49% 17,9 7,9 -56% 15,2 9,4 -38%

ID_39 17,2 7,8 -55% 15,3 8,21 -46% 17,6 7,2 -59%

ID_38 14,4 4,9 -65% 10,0 5,7 -43% 14,6 1,8 -88%

ID_36 8,9 6,9 -22% 9,7 3,2 -67% 8,8 8,7 -1%

2006 PeriodUnit ID

Incidents(2006)

All Incidents ΔI % Internal Incidents

ΔIi % External Incidents

ΔIe%

01-06 07-12(EN)

01-06 07-12(EN)

01-06 07-12(EN)

ID_173 48 115 140% 3 81 2600% 45 34 -24%

ID_146 69 77 12 % 5 50 2500% 64 27 -58%

ID_112 36 77 114 % 5 66 1220% 31 11 -65%

ID_ 86 23 63 174 % 1 25 2400% 22 38 73%

ID_70 15 55 267 % 0 40 3900% 15 15 0

ID_55 13 42 223% 4 29 625% 9 13 44%

ID_ 39 11 28 155% 1 22 2100% 10 6 -40%

ID_39 20 19 -5% 1 12 1100% 19 7 -63%

ID_38 21 17 -19% 2 12 500% 19 5 -74%

ID_36 17 19 12% 1 8 700% 16 11 -31%

2006 Period

Unit ID

Top 10 Units incidents (before and after EN)

now, a closer look to the profiles of incidents ...

Incident Profile ID_146 Incident Solution Rate (jan_06-jul_06)

0%

20%

40%

60%

80%

100%

120%

Days

Inci

dent

s S

olve

d

Incident Solution Rate (jul_06-dec_06)

0%

20%

40%

60%

80%

100%

120%

Days

Inci

dent

s So

lved

Before EN (69):

After EN (77): Tca ~ 20+ day

Tca ~12 day

Internal; 5,00

Illegal-Use; 8,00

Open-Proxy; 11,00

Other; 11,00

Spam; 34,00

Internal; 50,00

Illegal-Use; 19,00

Spam; 1,00

Open-Proxy; 2,00

Other; 5,00

49%

16%

16% 12%

25%65%

F(t)=1-eλt

Incident profile ID_112

Internal; 5

Illegal-Use; 0

Spam; 10

Open-Proxy; 7

Other; 14

Internal; 66

Illegal-Use; 6

Spam; 0

Open-Proxy; 0

Other; 5

Before EN (36):January-July 2006

After EN (77):July-December 2006

External (Spam, Open-Proxy, Other) “vanished”

86%

28%

39%

14%

19%

8%6%

EN limitations

internalexternalinternal

internal

external

externalinternalinternal

Notification rate

internal

Incident solutionrate

Spare Capacity (λ) OverloadedCapacity (λ´)

•Local Responses are limited by local Capacities•Capacity (skills, technology, staff/bw, staff/computers, ...)

•Local Capacities are related to the Local Incident Profiles (symptom)

CSIRT USP

Local team

Feedback to CSIRT

F(t)=1-eλt

Other (very) interesting profiles ...

Linux and good firewall management– Minimum contamination by worms – Little interaction to CSIRT-USP, no influence from notification process.

2005-2006 External Internal Obs.ID_56 54 2 LinuxID_44 41 3 FWID_3 3 0 FWID_20 14 6 FW

Similar Units profile (BW utilization, Staff, Technology, ....)

none from CSIRT USP !

Incident Response Readiness at USP• Early notification is essentially a CSIRT procedure that relys on:

– Honeypots, for the localization and identification of the problem– Available local internal capacities, for problem solving

– Long term Incident reduction and better responses can be achieved with:• Education

– Specializing local CSIRT managers– Training of local teams, to improve correction actions– General User Education (especially on Windows): diversified public: students,

professors, administration

• Preventive actions, to keep volume of internal notifications under manageble limits– Anti-virus distribution – Bandwith control– Network access control – Institutional network scanning – Other Specific tools

under study

on-going

Summary

Action taken by CSIRT-USP Possible Benefits Achieved results

1. Notification by CSIRT USP (jul/06)

2. Train CSIRT staff3. ORCA-like monitoration

with selectable features4. Deployment of more

honeypots (jan/07)

1. Antecipate treatment of local incident2. Improve awareness and local treatment3. Identify local profiles and capacities4. Attack identification and analysys5. Improve CSIRT relationships

1. Broader coverage of incidents notified (not obvious)

2. Reduction in incident lifetime (not obvious)

3. Reduction of external incidents (not obvious)

4. Visual “Real-time” incident monitoring (feb/07)

Conclusions• End-user’s freedom is normally obtained with some degree of

computers contamination.– Honeypot is an effective way to detect early stages of contamination and to

support the development of actions against later stages of the worm’s cycles. – Honeypot monitoration is centralized and demands minumum infrastructure

support – Honeypots permit suggest local actions according to Unit’s profiles

• Gobal worm mitigation doesn’t necessarily mean local worm mitigation.

• Honeypot-based Early Notifications by CSIRT-USP changed the profile of security incidents at USP– Incidents are closed in shorter times– External incidents has been reduced

Special Thanks

CISRT USP TEAM– Marta Bazzo Cilento – Hamilton Jun Higashizono – André Gerhard– Rogério Herrera Mendonça – Luis Ferreira– Bruno Darigo– Fernando Fugita– Solange Vieira– Olavo Rodrigues

References• Brazilian Honeypots Alliance – Distributed Honeypots Project

http://www.honeypots-alliance.org.br• CCE

http://www.usp.br/cce• CERT.br

http://www.cert.br• Honeyd

http://www.honeyd.org/Several papers about the projecthttp://www.honeynet.org.br/papers/

• USPhttp://www.usp.br

• OtherZou C., Gao L., Gong W.;Monitoring and Early Warning for Internet Worms. CCS’03Dagon D., Zou C., Lee W.; Modeling BotNet Propagation Using Time Zones. NDSS’06

[1]

Q&A

Thank you!