effective open-source spam filtering for enterprisefor ......filtering/bounces &...
TRANSCRIPT
![Page 1: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/1.jpg)
Effective Open-Source Spam Filtering For EnterpriseFor Enterprise
Chris LewisThomas ChoiThomas ChoiOctober 2008
VB2008, Ottawa
![Page 2: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/2.jpg)
Agenda
• Introduction
BackgroundBackground
Something New - Rationale
The Open-Source Project
Basic RequirementsComponentsI t tiIntegrationTest/Performance
Ad d T h iVB2008, Ottawa
• Advanced Techniques
![Page 3: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/3.jpg)
Introduction/AuthorsIntroduction/Authors
Chris Lewis
Senior Security Analyst/Anti-Spam, Nortel
Senior Technical Advisor, MAAWG
Member, Canadian Federal Anti-Spam Task Force
Thomas Choi
Nortel
Ph D Student Carleton University
VB2008, Ottawa
Ph.D Student, Carleton University
![Page 4: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/4.jpg)
BackgroundBackground
Spam became a problem in 1994/1995Initially in UsenetInitially in UsenetClearly would transition to EmailCommenced Email Anti-Spam program in 1997Extremely customized Lyris MailshieldExtremely customized Lyris Mailshield implementationVB2004 “Corporate Spam Fighting: 5 years ofVB2004 Corporate Spam Fighting: 5 years of success and lessons Learned”: by Chris Lewis and John Morris – don't forget those lessons!
VB2008, Ottawa
John Morris don t forget those lessons!
![Page 5: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/5.jpg)
Something New - RationaleSomething New Rationale
L i M il hi ld h t d i d t dLyris Mailshield has stood us in good steadBut, getting a little elderly, higher volumes, difficult to extend with newer techniquesReview of many other vendor offerings:Review of many other vendor offerings:
All missing one or more of critical featuresIntegrated poorly with existing infrastructureNot, or poorly extensible/configurableNot, or poorly extensible/configurableNot as effective as current solution
VB2008, Ottawa
![Page 6: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/6.jpg)
Rationale ... ContinuedRationale ... Continued
Needed open architecture/modular/easy extensionLow capital/license cost (free obviously best!)p ( y )Use standard components to minimize development costsdevelopment costsUse existing basic low-medium size server class hardwarehardwareFocus on 3rd party/popular filtering methodologies, simple ad-hoc filtering capabilities, plus with our own “secret sauce”.
VB2008, Ottawa
Avoid training (software OR people) requirements
![Page 7: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/7.jpg)
The Open Source ProjectThe Open Source Project
Basic Requirements – Functional SpecificationComponent SelectionComponent SelectionIntegrationBack endTestingTesting
VB2008, Ottawa
![Page 8: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/8.jpg)
Basic Requirements - FilterBasic Requirements Filter
Support multiple recipient domainsConfigurable per-domain handlingPer-domain filter enableConfigurable archiving/quarantine/disposition g g q p(pass,filter, trap) Output routingp g
Full loggingNEVER b il t bl kh l ( t t )NEVER bounce or silent blackhole (except trap)Plugin architecture – each technique an
VB2008, Ottawa
g qindependent module
![Page 9: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/9.jpg)
Basic Filter Requirements ... ContinuedBasic Filter Requirements ... Continued
Fault tolerant (eg: failover)Support 3rd party facilities, eg:Support 3 party facilities, eg:
DNSBL (IP blacklists) SURBL/URIBL (URI blacklists)SURBL/URIBL (URI blacklists)“informational” lookups (eg: ASN)
Content Scoring filterAnti-virusArbitrary ad-hoc string filters anywhere/on anything
VB2008, Ottawa
Direct/real-time feedback to filtering
![Page 10: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/10.jpg)
Basic “Not filter” RequirementsBasic Not filter Requirements
Full end-user quarantine view/forwardEnd-user (recipient) notification (if desired)End user (recipient) notification (if desired)Full logs in database/arbitrary queries(Almost) fully automated false positive handling (forward, filter tune, notification/explanation) Operational and Management metricsPostfacto analysis and automated filter tuningPostfacto analysis and automated filter tuning
VB2008, Ottawa
![Page 11: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/11.jpg)
Components, Filter, Open-SourceComponents, Filter, Open Source
Core SMTP listening engine/agent: Qpsmtpd (Hansen, Sergeant et. al.). 100% Perl implementation (really!)
Async (event driven) modeAsync (event driven) modeVery high performance – 20M+/day small serversEntirely flexible by plugin interfacey y p gActively supported & robustHas many sample plugins
SpamAssassin (popular scoring addon filter). (Perl)
ClamAV (*ix-based) anti-virus signature-based engine( ) g g
Nearly two dozen ad-hoc filtering plugins, few more than a dozen lines.
VB2008, Ottawa
The libraries and utilities to make the above work (eg: ParaDNS)
![Page 12: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/12.jpg)
Components, Filter, GlueComponents, Filter, Glue
A spam filter is more than just a filter, needs:Start/stop/reboot/monitoringStart/stop/reboot/monitoringLog & quarantine handling and transferExtended filtering heuristic processes (for things that take too long for real-time) Install/deployment and filtering control
VB2008, Ottawa
![Page 13: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/13.jpg)
Components, BackendComponents, Backend
PostgreSQL databaseApache (admin and user interface)Apache (admin and user interface)Interface to corporate user databases (push to filters)filters) Admin (research, false positive, configuration, d l t) i t f CGIdeployment) interface CGIsUser interfaces (configuration and quarantine)( g q )Quarantine managementR l ti fi ti ft
VB2008, Ottawa
Rsync – log, quarantine, configuration, software transfer
![Page 14: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/14.jpg)
IntegrationSPAM PostgreSQL
SPAMDatabase
I t t DMZ
N S
ApacheInternet DMZ
QPSMTPD
Plugins
Non-Spam
Mail serversPluginsSpamAssassin
ClamAV
Mail servers
Config
Users
Rejection ConfigRejectionNotices
F l P iti
DNSBL3rd Party BL
False PositiveReports
CORWAN
VB2008, Ottawa
![Page 15: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/15.jpg)
Test/PerformanceTest/Performance
Spamtrap operating 9 monthsPerformance heavily depends on “early pruning”Performance heavily depends on early pruning
“Cheap” tests firstPrune filtering subsequent to block decisionPrune filtering subsequent to block decision“Expensive” (body scans, SpamAssassin, ClamAV) tests lastClamAV) tests last
Volumes: typical 7m/server (50-100/sec), mostly spamtrapspamtrapDeployment in progress
VB2008, Ottawa
![Page 16: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/16.jpg)
Advanced TechniquesAdvanced Techniques
State of AffairsHide!Hide!Banner delaysBot fingerprintingDNSBLs (local and/or otherwise)DNSBLs (local and/or otherwise)DNSBL infrastructureBounces & BATVOnes we've omitted and why
VB2008, Ottawa
Ones we ve omitted and why
![Page 17: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/17.jpg)
State of AffairsState of Affairs
Underground economy (spam, phish, spyware, CC, mules) increasingSome LE believe larger than International Drug tradeSome LE believe larger than International Drug tradeBOTS responsible for 80%+ of all spam.Most getting good at stopping BOTs (<1% deliverability)=> BOTs shifting to reputation theft (relay through legit MTA )MTAs)State of Anti-Virus: disaster. (new BOT caught by AV 23% of the time by battery of 35 AV tools only increases to 50%of the time by battery of 35 AV tools, only increases to 50% by 30 days)Inadequate AV => can’t find BOT, let alone remediate
VB2008, Ottawa
Inadequate AV can t find BOT, let alone remediate proven infections.
![Page 18: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/18.jpg)
Hide!Hide!
Make it difficult for BOTs to email you.BOTs not full MTAs, high volume/throughput g g prequirements.Primary MX – “refuse connections” (Google for y ( g“nolisting”)Tertiary MX – “always retry”y y yDumb bots try once (primary or tertiary), get refusal or retry, and give up. Real MTAs do right thing.y, g p g gAs much as 50% of BOT spam simply vanishes.L f t i
VB2008, Ottawa
Loss of metrics.
![Page 19: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/19.jpg)
Banner DelaysBanner Delays
Most BOTs impatient, and won’t retry20-40 second banner delays =>20 40 second banner delays BOTs give up in disgustSome legit MTAs equally impatient, may need to whitelist some.
VB2008, Ottawa
![Page 20: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/20.jpg)
BOT FingerprintingBOT Fingerprinting
Most BOTs have fingerprints in the headers and SMTP protocol that can be caught by pattern
t himatching.Some mutate, some don’t.Srizbi > 50% of all spam.F d IP f d t ti b k i t l lFeed source IP of detections back into local DNSBL.
VB2008, Ottawa
![Page 21: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/21.jpg)
DNSBL (DNS Blacklist)DNSBL (DNS Blacklist)
Hundreds of 3rd party DNSBLs (IP based, domain based, URIBL filtering etc)A handful are both reliable and effective. There are DNSBLs effective to 70 80%+ of allThere are DNSBLs effective to 70-80%+ of all spam & virus propagation attempts.
VB2008, Ottawa
![Page 22: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/22.jpg)
DNSBL MergeDNSBL Merge
High volume receivers may impose undue loading on 3rd
party DNSBL infrastructure.
Occasional erratic delays (including DDOS on DNSBL)
=> Host them locally
We use rbldnsd – very high performance DNS server designed for high-performance serving of DNSBL zones.
We combine multiple 3rd party zones (plus ones we create ourselves) into a single zone.
Each DNSBL source distinguishable by return code, multiple DNSBL results “scored”. But most hits at threshold.
VB2008, Ottawa
![Page 23: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/23.jpg)
Filtering/Bounces & BATVFiltering/Bounces & BATV
Accepting then bouncing email with forged from => bounce storms (aka backscatter/blowback) => evil
Simple blackholing also evil
Aim is inline reject, with remediation information.
Support costs of receiving end of blowback often exceed spam
BATV (Bounce Address Tag Validation) see http://mipassoc.org/batv/When sending email, encode bounce address (MAIL FROM)
VB2008, Ottawa
When receiving bounce, reject email not encoded)
![Page 24: Effective Open-Source Spam Filtering For EnterpriseFor ......Filtering/Bounces & BATVFiltering/Bounces & BATV zAccepting then bouncing email with forged from => bounce storms (aka](https://reader035.vdocuments.mx/reader035/viewer/2022062403/60fa1ef463aa1d37d7660bb1/html5/thumbnails/24.jpg)
Omitted Techniques & WhyOmitted Techniques & Why
Greylisting – (force retry of “new senders”).Increasing reports of BOTs doing retry.Doesn’t prevent spam-by-reputation-hijacking
Bayesian – needs training, in many cases defeatedy g, yChecksumming (Razor/DCC et. al.) –
Detects bulk not spam per seDetects bulk, not spam per-seProblemmatic when outsourcing user-contact (eg: HR)Needs whitelistingNeeds whitelistingBOT hash busting getting better
VB2008, Ottawa