015 - remediating violated customers - 2012-10-21-a

7/29/2019 015 - Remediating Violated Customers - 2012-10-21-A

1/52

1

Remediating ViolatedCustomers

111


2/52

2

Time for Remediation Action

The cyber-civic society will be expecting allparties to do their part to protect against cyber-threats.

This includes Service Providers.

This module is based on the work in the IETF RFC6561 Recommendations for the Remediationof Bots in ISP Networks(http://tools.ietf.org/html/rfc6561)

We will review the US FCC CSRIC III Anti-Botnet Code of Practice

We will highlight the ENISA Anti-BOTNETRecommendations.
http://tools.ietf.org/html/rfc6561http://tools.ietf.org/html/rfc6561http://tools.ietf.org/html/rfc6561


3/52

3

Your Customers are Not theProblem!

There was a time where users and customerswere blamed for doing dumb things to get theirsystems infected.

When users who have up to date hardware,

operating systems, software, anti-virus, anti-malware, and is mindfully doing the right thinkstill getting infected, then we have to considerthat the real problem is beyond the user!


4/52

4

This is your Network!

See http://norton.com/cybercrimereport.
http://norton.com/cybercrimereporthttp://norton.com/cybercrimereport


5/52

5

Victimization Cost

See http://norton.com/cybercrimereport.
http://norton.com/cybercrimereporthttp://norton.com/cybercrimereport


6/52

6

Normal Malware Cycle

Creation

Activation

Replication

Victimization

Discovery

Remediation


7/527

Remediation Shortens theCycle

Minimizing Replication and Assimilation

Is the key to damage control

Replication

Victimization

Proactive Detection

Discovers Infection

In Early stage

Quarantine Contains

infection

Creation

Activation

Discovery

Remediation


8/528

Principles of Remediation

No one party can remediate a violatedcustomer.

It takes a team that involves the entire

eco-system ofoperating system vendors,application providers, on-line content,anti-virus vendors, service providers,professional computer repair

organizations, and the user of thedevice.


9/529

Expectations of Remedation

No way guarantee the remediation of all bots. Bot removal is potentially a task requiring specialized

knowledge, skills and tools, and may be beyond the abilityof average users.

Attempts at bot removal may frequently be unsuccessful, or

only partially successful, leaving the user's system in anunstable and unsatisfactory state or even in a state whereit is still infected.

Attempts at bot removal can result in side effects rangingfrom a loss of data to partial or complete loss of system

usability.

When a when a customers computer gets infected, we ask

them to go buy a new PC. Were in Hong Kong. New PCs

are cheaper than trying to clean up our customers

computer. (anonymous CTO in an SP)


10/5210

Detecting BOTNET & Malware

Service Providers have a range that gives theminsight into which of their customers are infected.

Reports (free and subscription) from external parties.

Service Provider Telemetry.

Partnership with Anti-Virus Vendors Helpdesk calls


11/5211

Where to Start

We currently have a multitude of organizations who willprovide detailed and traceable (i.e. through account logsand NATs) reports.

Arbor - Atlas, see http://atlas.arbor.net/

Internet Systems Consortium - Secure Information Exchange (SIE),see https://sie.isc.org/

Microsoft - Smart Network Data Services (SNDS), seehttps://postmaster.live.com/snds/

SANS Institute / Internet Storm Center - DShield Distributed IntrusionDetection System, see http://www.dshield.org/about.html

ShadowServer Foundation, see http://www.shadowserver.org/ Spamhaus - Policy Block List (PBL), see

http://www.spamhaus.org/pbl/

Spamhaus - Exploits Block List (XBL), seehttp://www.spamhaus.org/xbl/

Team Cymru - Community Services, see http://www.team-cymru.org/
http://atlas.arbor.net/https://sie.isc.org/https://postmaster.live.com/snds/http://www.dshield.org/about.htmlhttp://www.shadowserver.org/http://www.spamhaus.org/pbl/http://www.spamhaus.org/xbl/http://www.team-cymru.org/http://www.team-cymru.org/http://www.team-cymru.org/http://www.team-cymru.org/http://www.spamhaus.org/xbl/http://www.spamhaus.org/pbl/http://www.shadowserver.org/http://www.dshield.org/about.htmlhttps://postmaster.live.com/snds/https://sie.isc.org/http://atlas.arbor.net/


12/5212

Alerting Violated Customers

Communicating with customers is core tomodern customer experience.

Customer persistence and stickiness is

core to reducing churn. Any rational SP strategy to reduce churn

will have customer communications toolsthat include:

Email

Phone

Walled Garden

IM

Web Alert

Home Page Alert

SMS

TV Screen Alerts


13/5213


If you know that a customer has been violated,then there are civic society expectations to letthem know they are being victimized.

SPs doing this today find that it is a tool to

increase customer loyalty and decrease churn. Tracking violated customers means that the

Service Provider must update their customertracking & support system to know which are

identified as victimized and which have beennotified.


14/5214


Email Notification E-mail with customers sometimeswork but with all the SPAM, how do they know it is fromyou? Email notification with another approach to validatethe source works best.

Telephone Call Notification A simple phone call does

wonders. But also needs a secondary source to validate(fake support phone calls do happen).

Postal Mail Notification People do look at mail fromtheir service provider. The notification letter can have allthe information needed to help the violated customers start

their remediation work.


15/5215


Walled Garden Notification Violated customers whoare not paying attention or may be other devices in theresidence/business may need to be put into a walledgarden to notify. Careful attention is needed to insurecollateral impact to other devices in the residence/business

are not impacted (i.e. medial monitoring or emergencyservices).

Instant Message Notification Many people live onchat. A chat pop-up can be a way to get the attention of aviolated customer.

Short Message Service (SMS) Notification Mobilephone operators can send free SMS asking the violatedcustomer to go to a site and run a security check.

Web Browser Notification - In

Social Media -


16/5216


Web Browser Notification If the browser is where thecustomer lives, then explore tools that help interact at thebrowser level (i.e. plugins or toolbars).

Social Media A large majority of customers live in socialmedia. The same tools can be used to get the word out to

violated customers.


17/52

17

Notification Factor

Notification to Public Access Points. Alerting violatedcustomers that are tracked from a public WIFI point may ormay not be the best time to notify. A coffee shop would notbe a good place to try to recover your system from amalware infection.

Shared IP addresses. Many residence and businesses arebehind NAT with no logging (or they will have not clueabout NAT logging). Tools to help them figure out whichcomputer, device, or appliance is infected will be needed. Q. How do you remediate a violated Internet connected refrigerator?

Q. How do you remediate a violated diabetic monitoring device? Law Enforcement Lessons on how to help a Victim of

Crime are useful. The SPs support team can draw onlesson used in the LE community to help peopleproductively cope.


18/52

18

Ive checked everything!

Customer: Ive checked all my computers, my kidscomputers, my phones, my tables, my X-box, my Tivo, myprinters, my furnace, my light controls, my home securitysystem, my health monitoring system, my electric vehiclecharging station, my soar panel monitoring system .

Everything is patched and fixed why are you still sayingIm infected with malware!?

Support Team Have you checked to see if your neighborsare using your wireless?

Customer: How do I do that?


19/52

19

Walled Garden Systems doWork

Several major providers nowhave a decade ofexperience with productionwalled garden/quarantinesystems.

These systems work, theyhave not turn off customers,and have been updated towork with E.911 and medicaldevices.

Vulnerabilty

Checker

Computer

CPE

SBCIS IP NetworkPPPoX

IPTunn

el

RAS

Tunnel

Router

Service Network

DC

POP

Internet

Variable Access

Types: Ethernet,

Leased Line, ATM,Frame-Relay

Peering

Internet

Quarantine

Normal traffic

Web Portal

Controlled access to patch

sites

Anomaly

detection

Patch

server

Scan / test SW

server

Isolated network with limited /

controlled access to the outside.


20/52

20

Walled Gardens are EverydayEncounters

We, as an industry, knowhow to set up our AAA totrigger a interactive userresponse.

This is now an every dayactivity. There no longer asurprise factor with end-users.


21/52

21

Remediation Guidelines

Three approaches: Self Help Point customers to a self-help site or create

your own security landing page.

Professional Help Ask for the user to use aprofessional service to clean up the malware. Theprofessional service might offer help with the otherconsequence of the violation (i.e. identity theft or someother crime).

Get a new computer or device Unfortunately, wecould see malware evolving to the point where the

hardware is violated and the only remediation is to get anew device (ask the industry for consumer capable re-imaging).


22/52

22

Consequences of In-Action

We as an industry are at a stage whereService Providers need to play their part inthe remediation eco-system.

Cyber-Civic society will drive for actionthrough:

Government Guidelines, Regulation, and Laws

Through market forces (customer churn)

Through civic legal action Through insurance underwriters demanding

actions that reduce the over all risk to asystem.


23/52

23

Homework

Read through the IETF draft IETF RFC 6561Recommendations for the Remediation ofBots in ISP Networks(http://tools.ietf.org/html/rfc6561)

Talk to your peers at operations meeting likeNANOG, RIPE, APRICOT, etc to find out what theyare doing.

Join the SP Security effort that will document,

build, and teach remediation techniques thatwork.

E-mail [email protected] for more information or go tohttp://confluence.senki.org and select SP Security.
http://tools.ietf.org/html/rfc6561http://tools.ietf.org/html/rfc6561


24/52

24

US FCCs Anti-Botnet Code ofConduct


25/52

25

What is CSRIC?

The Communications Security, Reliability andInteroperability Council's (CSRIC) mission isto provide recommendations to the FCC toensure, among other things, optimal security and

reliability of communications systems, includingtelecommunications, media, and public safety.

Were currently in the middle of CSRIC III (seehttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iii)
http://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iiihttp://www.fcc.gov/encyclopedia/communications-security-reliability-and-interoperability-council-iii


26/52

26

CSRIC III

CSRIC III is covering these areas: WG 1: NG 9-1-1

WG 2: Next Generation Alerting

WG 3: E9-1-1 Location Accuracy

WG 4: Network Security Best Practices WG 5: DNSSEC Implementation Practices for ISPs

WG 6: Secure BGP Deployment

WG 7: Botnet Remediation

WG 8: E9-1-1 Best Practices

WG 9: Alerting Issues Associated With CAP Migration

WG 10: 9-1-1 Prioritization

CSRIC III BOTNET


27/52

27

CSRIC III BOTNETRemediation

This Working Group will review the efforts undertaken within theinternational community, such as the Australian Internet Industry Code ofPractice, and among domestic stakeholder groups, such as IETF and theMessaging Anti-Abuse Working Group, for applicability to U.S. ISPs.Building on the work of CSRIC II Working Group 8 ISP Network ProtectionPractices, the Botnet Remediation Working Group shall propose a set ofagreed-upon voluntary practices that would constitute the framework for

an opt-in implementation model for ISPs. The Working Group will proposea method for ISPs to express their intent to op-into the frameworkproposed by the Working Group.

The Working Group will also identify potential ISP implementationobstacles to the newly drafted Botnet Remediation business practices and

identify steps the FCC can take that may help overcome these obstacles.

Finally, the Working Group shall identify performance metrics to evaluatethe effectiveness of the ISP Botnet Remediation Business Practices atcurbing the spread of botnet infections.


28/52

28

Anti-Botnet Code of Practice

A voluntary code ofpractice was adoptedto insure nounrealistic cost are

imposed on theindustry.

Each SP is now askedto public state if they

will comply with thecode of practice.


29/52

29

What is the ABC?

Encourage ISPs to Educate end-users of the threat posed by bots and of

actions end-users can take to help prevent botinfections;

Detect bot activities or obtain information, including

from credible third parties, on bot infections among theirend-user base;

Notify end-users of suspected bot infections or helpenable end-users to determine if they are potentiallyinfected by bots; and

Provide information and resources, directly or byreference to other sources, to end-users to assist themin remediating bot infections.


30/52

30

ABCs Implementation

Implementation of the Code will be guided by thefollowing principles:

1. Voluntary participation is voluntary and encouragestypes of actions to be taken by ISPs, however thisCode does not require any particular activity.

2. Technology neutral this Code does not prescribeany particular means or methods.

3. Approach neutrality this Code does not prescribeany particular approach to implement any part of thisCode.

4. Respect for privacy ISPs must address privacyissues in an appropriate manner consistent withapplicable laws.


31/52

31

ABCs Implementation (cont)

5. Legal compliance activities must comply withapplicable law.

6. Shared responsibility ISPs, acting alone, cannotfully address the threat posed by bots. Other Internetecosystem participants must also do their part.

7. Sustainability ISPs should seek activities that arecost-effective and sustainable within the context oftheir business models.

8. Information sharing ISPs should indicate howthey are participating in the Code and share lessons-

learned from their activities with other appropriatestakeholders. All information sharing between ISPs andother involved parties must be performed inaccordance with applicable laws including, but notlimited to, antitrust and privacy laws.


32/52

32

ABC Implementation (cont)

9. Effectiveness ISPs should be encouraged to engagein activities that have been demonstrated to beappropriate and effective.

10.Effective Communication Communication withcustomers should take into account various issues

such as language and make sure that information isprovided in a manner that is reasonably expected to beunderstood and accessible by the recipients.


33/52

33

Participation Requirements

To participate in this Code, an ISP will engage in at least one

activity (i.e., take meaningful action) in each of the followinggeneral areas:

Education- an activity intended to help increase end-usereducation and awareness of botnet issues and how to help preventbot infections;

Detection- an activity intended to identify botnet activity in theISPs network, obtain information on botnet activity in the ISPsnetwork, or enable end-users to self-determine potential botinfections on their end-user devices;

Notification- an activity intended to notify customers ofsuspected bot infections or enable customers to determine if theymay be infected by a bot;

Remediation- an activity intended to provide information to end-users about how they can remediate bot infections, or to assistend-users in remediating bot infections.

Collaboration- an activity to share with other ISPs feedback and

experience learned from the participating ISPs Code activities.


34/52

34

Education

End-users are ultimately responsible forprotection of their devices and for remediating aninfected device. ISPs, like many other Internetparticipants and government actors, can assist in

helping to educate end-users about the threatspresented by bots and the steps end-users cantake to protect their devices and remediateinfections.

Education about bot prevention

Support of end-user bot remediation efforts


35/52

35

Education

Guidelines: In addressing the above requirements,ISPs should consider these guidelines:

Offer educational information and resources directly orthrough referral to third party services.

Keep educational content concise and focused on the mostimportant things users need to know.

Ensure that instructions can be followed by an audience ofnon-technical users.

Use multiple media, e.g., images, videos, text, captions, etc.,

and, where helpful, multiple languages to maximize customerunderstanding and accessibility.

Help end-users determine if they have a bot infection byproviding information or pointing to resources that describeanomalous behaviors of bot infected devices and the

availability and use of bot detection software tools or services.


36/52

36

Detection

ISPs can find out about malicious activity and botcompromised end-user devices in a variety ofways:

Receiving notifications from external entities,particularly those designed to aid with the overallunderstanding and real-time dissemination of botrelated data. A list of resources is listed in Appendix 2.

Deploying capabilities within their networks that aid inidentifying potential bot infections.

Directing customers to tools, a web portal, or otherresources that enable customers to self-identify apotential bot infection.


37/52

37

Notification

Recommended Action: Provide communication ofa suspected bot infection to the customer or helpenable customers to determine if they arepotentially infected by bots. Many notification

methods are outlined in references in Appendix2; however, other methods may be used.

The problem: Appendix 2 did not reference other

methods.


38/52

38

Remediation

Recommended Action:1. Bots are designed to be stealthy and difficult to

remove. As part of the notification, ISPs shouldoffer guidance, as described above. This may

include links to a variety of publically availableonline and third party sources of information,software, and tools. It might also include links toprofessional services. These need not be offeredby the ISP itself but may be offered by thirdparties.


39/52

39

Remediation (cont)

2. An ISP may provide remediation tools to the end-user, either during or after the notification process.However, the ISP should not mandate that theend-user run remediation tools. If the ISP provides

tools to the end-user, the end-user should beallowed to exit the process without running anysuggested tools or procedures.

3. As part of the notification process, ISPs may wishto include guidance (depending on the nature ofthe bot in question) that settings on customerowned network equipment such as home gatewaysand routers may have been altered and should berestored to a secure state, depending on the

nature of the bot infection.


40/52

40

Collaboration

Recommended Action: Code participation requires collaborationwithin ISP, industry, or broader fora through collaborativeactivities, of which the following are examples:

Sharing detection, notification, or mitigation methodsplanned for or deployed in ISP networks, and where practical

an evaluation of their effectiveness. Sharing of intelligence or operational attack data that may be

useful in bot prevention, defense, or remediation.

Identification of key data or technical resources that areneeded from systems or actors beyond the ISP network.

Participation in definition, development, or operation ofintegrated defense strategies or systems which extendbeyond the boundaries of the ISP network.

Other collaboration activities involving the sharing ofinformation with parties outside the ISP or data with systems

outside of the ISP network.


41/52

41

Impact to the Business (No ABC)

CAPEX

Capital expense on equipment Violated customer require moreresources from over the topcyber-criminals.

OPSEC The over all Operational cost of

certification, deployment, testing,

integration, and maintenance.

Help desk calls, excess bandwidthconsumption, and abuse processall increase OPSEC

CPGA

Cost per gross subscriber add

(primarily subsidies & provisioning)

No impact.

ARPU Average revenue per user month Basic services no extra securityservices

CCPU

Cash cost per user per month, ex-marketing (backhaul, customer support,

maintenance, & overhead)

BOTNET violated customer takeon more resources on the overallsystem.

Churn - % number of subscribers

disconnecting each month

Perception of slow internetservices churn the customer.


42/52

42

Impact to the Business (w/ ABC)

CAPEX

Capital expense on equipment Additional CAPEX to deploy theABCs

OPSEC The over all Operational cost of

certification, deployment, testing,

integration, and maintenance.

Automated notification systemsfacilitate call deflection.

CPGA

Cost per gross subscriber add(primarily subsidies & provisioning)

Security add on features havenew cost with new revenue.

ARPU Average revenue per user month Security features increase ARPU.

CCPU Cash cost per user per month, ex-

marketing (backhaul, customer support,

maintenance, & overhead)

Clean customs with new securitycapabilities have over all savingson the system.

Churn - % number of subscribers

disconnecting each month

Big SPs who deploy somethinglike the ABCs report lower churn.


43/52

43

Shoestring ABC Compliance

Education Create a /security page team upwith a non-profit industry organization to provideeducation.

Detection Subscribe to the free feeds from

Shadowserver, Team CYMRU, and Microsoft.Notification Deploy a E-mail notification systemand a billing notification system.

Remediation Same as education.

Collaboration - Deploy Passive DNS on your DNSResolvers. Deploy a Dragon Research, ArborAtlas, and Shadowserver.org box in your SinkHole (dark IP monitoring). Join groups likeMAAWG, OPSEC Trust, NSP-SEC and others.


44/52

44

Home Work

Sitting around waiting for your customersviolated by malware to adversely impact yourbusiness is not a wise business decision.

Recommend action given that there are cost

effective means to take action now.


45/52

45

ENISAs Anti-BOTNET WorkshopFindings

(see http://www.enisa.europa.eu/activities/Resilience-and-CIIP/critical-applications/botnets)

www.enisa.europa.eu


46/52

46

Key Recommendations for Countermeasures


47/52

47

Mitigate Existing Botnets

X X XX

X XXXXX


48/52

48

Prevent New Infections


49/52

49

Minimize Profitability

Governments


50/52

50

Responsibilities

Governments

Define clear andconsistent laws

Prosecute criminals

Define capabilities

Define centralcontact points

Data protection

End-users

Keep machinesclean

Civic responsibility

Corporate socialresponsibility

ISPIdentify, notifycustomers

Help users cleanmachines

Filter malicious traffic

Detection,measurement

Data protection

Victims

Resist extortion

Pursue perpetrators.

Cybercriminals

Software/OSDevelopers

Write securesoftware

Fix vulnerabilities(quickly)

Detect attacks andinform users

Researchers/AV Vendors

Detection

Disinfection

Responsible disclosure

Malware analysis


51/52

51

Beneficiaries

Cybercriminals

Innocentbystander/unwilling

participant

ISP, End-user

Victims

eCommerce, Banks,Web 2.0, Advertisers,

Governments

Current incentives

Rebalancing the incentives


52/52

Rebalancing the incentives

Incentives

Government

Public privatepartnerships

End-usersBot-Frei

Awareness raising

Raise sense of socialresponsibility

Software Vendors

Secure devprogrammes

SSE initiatives

Criminals

Prosecute and arrest

Attack all parts ofvalue-chainesp thosewith no backup e.g.money-laundering.

Attack revenuestreams

ISPs

Financial incentives forend user cleaninginitiatives

Clarify and harmoniseDP laws

Victims

Mutual assistance e.g. in legal and otherresources (DigitalAddio-Pizzo)

AV/Researchers

Fast legal procedures

Reward for reporting

Clarify capabilities

Information sharing

015 - remediating violated customers - 2012-10-21-a

Documents