nullcon 2011 - fuzzing with complexities

Fuzzing with complexities

Vishwas Sharma

http://null.co.in/ http://nullcon.net/

Introduction• We all have been a witness to major threats in the past years and I guess

no one could forget names like ‘Conficker’ (1), ‘Stuxnet’ (2) and ‘Aurora Project’ (3). All these malware had a unique delivery system which was based on exploiting the host operating system and further talking control of the OS.

• These threats are always there and only thing we expect to achieve is that, we find vulnerability before a bad guy do and do something about it.

• Software companies spend a lot of their time and money in making their product more stable, more reliable and more secure.

• Vista Microsoft has made sure that functions like strcpy, sprintf etc. are eliminated at the Software development lifecycle (SDL)


Introduction• In fact all major vendors have realized the importance of having a secure

SDL and importance of testing in their product.

• Google and Firefox have a policy of rewarding any researcher who comes up with a bug or a resulting exploit.


Figure 1: Microsoft Simplified SDL (4)

Software Testing

• Software testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results.

• Unlike most physical systems, most of the defects in software are design errors, not manufacturing defects


Code Coverage

• Code coverage is one of the most important metrics used to decide on the completeness of the test cases. This metrics gives us the relationship between the test conducted and instructions executed with the application.


Code CoverageOf course this metrics can be further broken down into more detailed metrics• Function coverage - Has each function (or subroutine) in the program

been called?• Statement coverage - Has each node in the program been executed?• Decision coverage - Has every edge in the program been executed? For

instance, have the requirements of each branch of each control structure (such as in IF and CASE statements) been met as well as not met?

• Condition coverage - Has each Boolean sub-expression evaluated both to true and false?

• Condition coverage - Both decision and condition coverage should be satisfied.


Code Coverage


An example of Code coverage

Code Coverage


Test needed to find bugs

Tests needed for coverage

Shows that even on a good coverage some bugs would still be left alone

BlackBox TestingNo knowledge of the inner working of the software, neither of the protocol or kind of input expected, this situation is rightly named as Black-box Testing


Black box testing

Expected Input

Boundary values

Illegal values

Whitebox TestingInformation on internal data structure and algorithms is completely shared between the product development team and the tester’s team

Information can be used to test API’s, Code Coverage, fault injection, Mutation of testing and many more.


White box testing

Statement Testing

Decision Testing

Conditional Testing

Fuzzing

• The first person credit of working and formulating this technique is Barton Miller and his students from University of Wisconsin-Madison in 1989

• In simple words it is the technique in which repeated invalid or mutated or malformed input is supplied to application with only intention to find bugs the application

• It is observed that fuzzing is most effective against application developed in C/C++, these languages make the programmer responsible for memory management whereas managed code i.e. developed in C#, Java etc. would yield bugs of a very different class


FuzzingTerm DefinitionDumb fuzzing Corruption of data packets randomly without awareness of data

structure.Smart fuzzing Corruption of data packets with awareness of the data structure, such

as encodings (for example, base-64 encoding) and relations (checksums, bits indicating the presence of some fields, fields indicating offsets or lengths of other fields).

Black-box fuzzing Sending of malformed data without actual verification of which code paths were hit and which were not.

White-box fuzzing Sending of malformed data with verification that all target code paths were hit—modifying software configuration and the fuzzed data to traverse all data validations in the tested code.

Generation Generation of fuzzed data automatically—not basing on any previous input.

Mutation Corruption of valid data according to defect patterns, to produce fuzzed data.

Mutation template Well-formed buffer that represents an equivalence class of the input. The fuzzer takes the mutation template as an input—producing a fuzzed buffer to be sent to the tested software.

Code coverage Technology (such as that which is bundled in Microsoft Visual Studio 2005) that allows inspection of which code paths were executed during testing. This is useful for verification of test effectiveness and improvement of test coverage.


Fuzzing


• Important distinction between Fuzzing and other testing activity. This distinction is the intent.

• A testing team knows a lot about the program and basically test that whether a program is behaving as it is supposed to behave where as a security researcher only care that his fuzzer crashes your tested application.

Fuzzer


I would like to make note of two python based fuzzing framework available in the open source community that I use most extensively.• PeachFuzzer - Peach is a SmartFuzzer that is capable of

performing both generation and mutation based fuzzing (10).• Sulley - Sulley is a fuzzer development and fuzz testing

framework consisting of multiple extensible components. Sulley (IMHO) exceeds the capabilities of most previously published fuzzing technologies, commercial and public domain

Fuzzer


I would like to make note of two python based fuzzing framework available in the open source community that I use most extensively.• PeachFuzzer - Peach is a SmartFuzzer that is capable of performing both

generation and mutation based fuzzing .

• Sulley - Sulley is a fuzzer development and fuzz testing framework consisting of multiple extensible components. Sulley exceeds the capabilities of most previously published fuzzing technologies, commercial and public domain

Peach Fuzzing Platform

Fuzzer

• Peach is been improved day in and day out and it is the only other open source fuzzer that is maintained apart from Metasploit fuzzer. Peach is written as primary data fuzzer, but as it open source it can be extended to secondary and even nth-class fuzzer. Peach fuzzer is also used by adobe in its testing of Adobe reader

• Sulley is not maintained but is as good as you can get when it comes to generation based fuzzing

• Collection of fuzzers http://packetstormsecurity.org/fuzzer/


http://packetstormsecurity.org/fuzzer/

Complexity

• “Software bugs will almost always exist in any software module with moderate size: not because programmers are careless or irresponsible, but because the complexity of software is generally intractable -- and humans have only limited ability to manage complexity. It is also true that for any complex systems, design defects can never be completely ruled out” - Jiantao Pan, Carnegie Mellon University

• In many of the fuzzers it is observed that test cases produced fails to achieve the basic packet sanitation test of the target application if the fuzzer is has improper understanding of the input type and structure


Complexity

• A study done by Microsoft on a 450 lines of code and then testing it with various fuzz combinations to see the effective results that was produced is shown below :


Technique Effort Code coverage Defects found

Black box + Dumb 10 min 50% 25%

White box + Dumb 30 min 80% 50%

Black box + Smart 2 hr 80% 50%

White box + Smart 2.5 hr 99% 100%

Analysis based on Effort in producing fuzzer and defects found correlated with kind of fuzzer

Packets

• An example of ASCII based packet (irc)

• There are few other examples quite popularly known eg.– HTML– CSS– FTP– And many more


WHO #UnrealIRCd

JOIN #Support

PONG irc.fyremoon.net irc2.dynam.ac

STATS u

Binary based PacketsZIP local file headerOffset Bytes Description[5]

0 4 Local file header signature = 0x04034b50 4 2 Version needed to extract (minimum) 6 2 General purpose bit flag 8 2 Compression method10 2 File last modification time12 2 File last modification date14 4 CRC-3218 4 Compressed size22 4 Uncompressed size26 2 File name length (n)28 2 Extra field length (m)30 n File name30+n m Extra field


But what happens when the formats no longer sticks to one data format? What happens when our data switches from one set of data format like ASCII to binary and then binary to ASCII again and to add a cherry on top sections are encoded differently even the ascii portion can be encoded and even imported from other binary or ASCII based formats

http://en.wikipedia.org/wiki/ZIP_(file_format)#cite_note-appnote-4

Example of one such format


%PDF-1.5%µµµµ1 0 obj<</Type/Catalog/Pages 2 0 R/Lang(en-IN) /StructTreeRoot 54 0 R/MarkInfo<</Marked true>>>>endobj2 0 obj<</Type/Pages/Count 7/Kids[ 3 0 R 23 0 R 29 0 R 38 0 R 47 0 R 49 0 R 51 0 R] >>endobj3 0 obj<</Type/Page/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 10 0 R/F3 12 0 R/F4 14 0 R/F5 16 0 R/F6 21 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI] >>/MediaBox[ 0 0 595.32 841.92] /Contents 4 0 R/Group<</Type/Group/S/Transparency/CS/DeviceRGB>>/Tabs/S/StructParents 0>>endobj4 0 obj<</Filter/FlateDecode/Length 5463>>streamxœ\[Û¸’~ÿ Gû ̂ Ý€AI&9ÈbgÏì&»ó0³jµÜÖÄq÷Hêtz~ýÖ…”HY”ÅÁœ´e²X$ëúUÉÑë_£Ÿ~zýË»?GÉõuôöçwÑÛÏ/_¼þ "™Å� � � � �‰Ž>ï^¾QÿQ.ãDê(×i,Šèó×—/’8)%±øÿ*K£înññýóå‹ßJ’¤¼ÎáŸ2¿¾*ñßwþ¿ÅÛë+ú>¹¾*ðsÉŸeÆ•æ¹—ñ¿ÅþÞŽ+Òë+‘à� �

Example of one such complex formats ie. PDF

We see these being used in every day applications like office documents, Adobe PDF, SMB protocols and more. One cannot try to randomly fuzz these files as they have pretty good input validation modules which prevent any dumb attempt to fuzz them

What we know so far


• What we have gathered until here is summarized here as we move ahead you will find answers to these problemsComplications

Code coverage is cannot be used as a metrics while testing test type of input streams

Data format consistency i.e. Either ASCII or Binary is not maintained

Multiple encoded levels exists which can be different within the same file

Multiple data format either binary or ASCII are embedded within these input stream

Some answers


• Code Coverage fails for these applicationsProtocol awareness can be used as once we have all the information of a protocol that we could have, we can intuitively say that the packet which contains the most number of tags or objects would require more code to be covered with that module. Now this could be said that we cannot guarantee the code coverage still because if we do not find a packet that contains all the tags or objectTesting all cases in one go was never the idea but multiple tests covering every tag is what will be fruitful.

• Data format inconsistencyOne can easy write a fuzzer of either and ASCII based packet or for binary based packet. But when these formats get together in a packet, it becomes unnaturally difficult to write one.

The solution lays in visualizing and breaking problem in parts which we most comfortable in. We can use the separate out the data generation capability from both ASCII and Binary format. Remember here I have trying to separate out these capabilities not necessarily for fuzzing.

Some answers


• Multiple Files Embedded in a single packetsWith separating of types we can further separate to a secondary level data production module ie. A different level of generating data. What this means is that if a PDF file if we have a font and image embedded inside the file we can actually write a different fuzzer for font and for an image and combines each of these result with the PDF files in the manner similar to multiple encoding level problem.

• Multiple Encoding levelsAs we have separated ASCII with Binary in the same format one can further add custom encoding in each packet as one like. They will all fall back together when we combine them later. See the case study for more clarification.

For example in a PDF file if we have a multiple font embedded inside the file we can make use of different encoders for each such font as each is generated separately

Strategy


• Now is the right time to talk about the strategy that I have used when fuzzing one such format, PDF. You will find different definition of these terms, but this is what I understand out of them. This process is typically described in the terms of system under test and called for directed area with the system, where as in my study I have taken it out of box and placed these conditions on Data packet itself.

Attack point selection

Not to fuzz To Fuzz

Not to Fuzz Not to Fuzz

To Fuzz


Attack Point Selection

The attach point selection is a simple process in which I have tried to specify a specific point within the packet which needs to be tested. Now selection of these points depends a lot upon some gathered intelligence of the system, including pervious vulnerabilities. As this eliminates a few attack point as they have already been attacked before. For example if working on a simple PDF file which contain a U3D file which is known to previously cause a vulnerability in Adobe reader one can say this format is previously been tested primarily (after looking at the vulnerability) so a lot more efforts would be required in finding a vulnerability next time. One can focus his time and energy in finding other routes into the application which has still not been tested by security researchers.

Directed Fuzzing

Not to fuzzNot to fuzz

Not to Fuzz Not to Fuzz


To Fuzz

Adobe Flash Player Multiple Tag JPEG Parsing Remote Code Execution Vulnerability

-- Vulnerability Details:

This vulnerability allows remote attackers to execute arbitrary code on vulnerable installations of Adobe Flash Player. User interaction is required in that a target must visit a malicious website. The specific flaw exists within the code for parsing embedded image data within SWF files. The DefineBits tag and several of its variations are prone to a parsing issue while handling JPEG data. Specifically, the vulnerability is due to decompression routines that do not validate image dimensions sufficiently before performing operations on heap memory. An attacker can exploit this vulnerability to execute arbitrary code under the context of the user running the browser.

Whenever a vulnerability is released it is released with a very few information. One such disclosure example would be.


DemoCVE 2010-2862

Integer overflow in CoolType.dll in Adobe Reader 8.2.3 and 9.3.3, and Acrobat 9.3.3, allows remote attackers to execute arbitrary code via a TrueType font with a large maxCompositePoints value in a Maximum Profile (maxp) table.

nullcon 2011 - fuzzing with complexities

Technology

condition coverage

decision coverage

code coveragecode coverage

good coverage

statement coverage

fuzzing framework available

mutation of testing

netan example of code