the state of the science of digital evidence examinationall.net/talks/2011-01-31-ifip.pdfjan 31,...

California Sciences Institute is a 501(c)3 non-profit educational and research institution. We do not discriminatein our hiring, admissions, offerings, or in any other way except by ability to do the work and learn the material.

California Sciences Institute

Fred Cohen & Associates

The State of the Science of Digital Evidence Examination

IFIP Forensics Conference – Jan 31- Feb 2, 2011

Dr. Fred CohenPresident - California Sciences Institute

CEO – Fred Cohen & Associates




Outline

● Introduction

● Consensus in Science and Forensics

● The State of the Literature

● Summary / Conclusions / Discussion




Your authors● Fred Cohen

– B.S. EE (C-MU '77), M.S. Info Sci (Pitt '81), Ph.D. EE (USC '86)

– CEO - Fred Cohen & Associates / President CalSci

● Julie Lowrie– J.D., M.S., B.A., M.S. Advanced Investigation (CalSci -2010)

– Ph.D. student at CalSci / Fraud investigator

● Charles Preston– B.A., M.S. Advanced Investigation (CalSci -2010)

– Long time criminal and civil investigator

● CalSci

– 501(c)3 research and educational institution

– M.S. Advanced Investigation / Ph.D. Digital Forensics




Background● Calls for “science” in forensics

– NRC 2009, SWGDE, many other recent authors

● US Federal Rules of Evidence

– Reliable methods properly applied

– Presented by experts (knowledge, experience, training, education, skill - beyond lay person)

● Daubert, Frye, and others

– Accepted methods that reliably and accurately reflect the data they rely on

– Findings generally accepted within relevant field

– Tools/methods test/calibrate, known reliability




More background● Foundations of science

– General theories that change rarely

– Testability with refutation king

– Generally accepted language and methods

– Agreed systems of measurement

● State of science (in other fields)

– Physics (very well established / widely agreed)

– Global climate change caused by humans● >86% agreement < 5% disagreement● Climatologists 2008

– Other fields (social sciences) also studied




Our research questions

● Is there a consensus around basic notions in digital forensics?

– Level of agreement to simple statements

– Based on (non-scientific) polling

● Are publications reflecting the elements of a mature science?

– Presence of common scientific elements

– In peer reviewed “scientific” publications

● What reviewers commented on/questioned

– How peer review reflects on other results




Outline

● Introduction







Consensus● Consensus doesn't make it right or wrong!

– This survey / study is NOT about what is true or false or right or wrong.

● “The World is flat” had consensus at one time.

– It is about whether and to what extent there is a consensus around certain things in the field.

● Consensus can be for or against!!

– Not all statements are / have to be true

– It does not matter if or to what extent the statements are believed to be true by those who wrote them

● It is about whether, as a field, we agree




Methodology & Limitations● Started with a poll at a conference

– Hands up and down – initial guestimate

– Largely “computer science research” crowd

● Added Digital Forensics Certification Board

– Experienced founding practitioners

– Testifying experts in real cases for years

● Added IFIP prior attendees

– Just like (some of) you in the audience today

● Added HTCIA Bay Area chapter

– Another independent group in the field




Methodology & Limitations● Self-selected participants – not random sample

– Selected from existing (loose-knit) groups

– Groups differ in characteristics

– No adjustment for demographics, etc.

● Instrument (questions) not tested / vetted

– Just a limited list of items

– Not a known / validated / etc. instrument

● Not commonly used (e.g., Likert) scale

– {I disagree / I don't know / I agree}

– v. 5-level Likert scale more commonly used

– Overload of “don't know” with “not sure”, etc.




Instructions

● This is a simple survey designed to identify, to a first approximation, whether or not there is a consensus in the scientific community with regard to the basic principles of the examination of digital forensic evidence. This survey is NOT about the physical realization of that evidence and NOT about the media in which it is stored, processed, or transported. It is ONLY about the bits.




More instructions● Please read carefully before answering.

● Don't look anything up. Only go from what you already know.

● If you haven't heard of the principle/concept, don't agree with it!

● These are not necessarily all true or false. Only go with what you know.

● This is ONLY about digital evidence - not its physical realization.

● Agreement means that it is normally the case when dealing with digital evidence, not a universal truth.

● EXCEPTIONS: Items marked (Physics) are about the normal physics of time and space.




Still more instructions

● Replaced “If you haven't heard of the principle/concept, don't agree with it!”

● With: “Agreement means:

– “I agree.” means it is normally the case when dealing with digital evidence, not a universal truth.

– “I disagree.” means it is normally not the case when dealing with digital evidence, not that it can never be true.

– “I don't know.” means you haven't heard of it or don't agree or disagree with it.




Specific questions

● Control questions: [and “correct” answers]

– A: F=ma (Physics) [I agree]

– B: The Johnston-Markus equation dictates motion around fluctuating gravity fields.(Physics) [I don't know]

– C: Matter cannot be accelerated past the speed of light. (Physics) [I agree]

● “Correct” here means the prevailing theory of physics today as taught worldwide

● Item B is a made up statement → not true, but impossible to know it → don't know




Consensus questions● Between control A and B

– 1: Digital Evidence consists only of sequences of bits. [definitional]

– 2: The physics of digital information is different from that of the physical world.

– 3: Digital evidence is finite in granularity in both space and time.

– 4: It is possible to observe digital information without altering it.

– 5: It is possible to duplicate digital information without removing it.




Consensus questions● After Control B (C was after 8)

– 6: Digital evidence is trace evidence. [definitional]

– 7: Digital evidence is not transfer evidence. [definitional and counter-historical]

– 8: Digital evidence is latent in nature. [definitional and historical]

– 9: Computational complexity limits digital forensic analysis. [computer science oriented]

– 10: Theories of digital evidence form a physics.

– 11: The fundamental theorem of digital forensics is "What is inconsistent is not true".




The eye chartS# NH NA % A/n DH DA % A/n Id Ia % d/n a/n Hd Ha % d/n a/n ∑a ∑d a/N d/N

A 22 22 100 n/a 8 6 75 .50 2 17 89 .08 .73 0 0 0 0 0 37 2 .68 .07

1 7 7 100 .50 9 6 66 .50 13 10 76 .56 .43 2 0 0 1.0 0 23 15 .42 .53

2 5 1 20 .07 3 2 66 .17 9 12 57 .39 .52 0 1 50 0 .50 16 9 .29 .32

3 6 4 66 .28 2 1 50 .08 6 16 72 .26 .69 1 1 50 .50 .50 22 7 .40 .25

4 12 9 75 .64 10 10 100 .83 6 17 73 .26 .73 1 1 50 .50 .50 37 7 .68 .25

5 12 9 75 .64 12 11 92 .92 3 20 86 .13 .86 1 1 50 .50 .50 41 4 .75 .14

B 1 0 0 0 0 0 0 0 1+ 2+ 0+ 0+ 0+ 0 0 0 0 0 na na na na

6 14 5 35 .35 8 4 50 .33 6 14 70 .26 .60 1 1 50 .50 .50 24 7 .44 .25

7 0 0 0 0 5 2 40 .17 5 6 54 .21 .26 1 1 50 .50 .50 9 6 .16 .21

8 2 1 50 .07 5 3 60 .25 5 13 72 .21 .56 1 1 50 .50 .50 18 6 .33 .21

C 20 18 90 n/a 10 4 40 .33 2 14 87 .08 .60 1 0 0 .50 0 32 3 .59 .10

9 12 12 100 .85 4 3 75 .24 3 18 85 .13 .78 0 2 100 0 1.0 35 3 .64 .10

10 2 1 50 .07 1 1 100 .08 9 7 43 .39 .30 1 0 0 .50 0 9 10 .16 .35

11 3 2 66 .14 0 0 0 0 13 7 35 .43 .30 1 1 50 .50 .50 10 14 .18 .50




Results

● NSF/ACM (n=14)

– Consensus > random● Only #9 (A/n=.85)● #4,#5 (A/n=.64) but

margin of error=23%

● DFCB (n=12)

– Consensus > random● #4 (A/n=.83)● #5 (A/n=.92)● Margin of error = 28%

● IFIP (n=23)

– Consensus > random● #5 (a/n=.86, d/n=.13)● #3 (a/n=.69, d/n=.26)● #4 (a/n=.73, d/n=.26)● #9 (a/n=.78, d/n=.13)● Margin of error = 19%

● HTCIA (n=2)

– Too small a sample for meaningful interpretation




Overall results (summed)● Total sample sizes:

– (N=54) agreement – all surveys: MoE=9%

– (N=28) disagreement – subset: MoE=13%

● No agreement at 86% confidence level

– No consensus at level of global climate change

● Only these in excess of random:

– A (a/N=.68, d/N=.07) / C (d/N=.10) controls

– #4 (a/N=.68) – BUT - d/N > .14 observe w/out altering

– #5 (a/N=.75, d/N=.14) duplicate without removal

– #9 (a/N=.64, d/N=.10) computational complexity limits




Outline

● Introduction







Methodology

● Read lots of relevant peer reviewed articles

● Do standard reviews of them describing

– Primary and secondary classifications

– Evidence of use of science

– Methodology identified, created, applied

– Physics identified, stated, applied

– Testability identified, stated, applied

– Validation identified, stated, applied

– Language identified, defined, applied

● Analyze for evidence of normal science




Areas of classification● One primary area – any number of 2ndaries




Limitations

● Purely the opinions of the reviewers

– Redundant reviews used for error control

– 125 reviews / 95 unique articles: 31% redundant

● Not ideally defined terms

– Science is messy stuff to define precisely

– When reading papers, it's a judgment

– Reviewers had common knowledge base

● Not random selection of papers

– Tried to cover major peer reviewed items

– Tried to cover relevant time frames




The sample we used● Papers were:

– 34% conferences, 25% journals, 18% workshops, 8% book chapters, 10% others

● Sources were:

– IFIP (4), IEEE (16), ACM (6), HTCIA (3) Digital Investigation (30), dissertations (2), and the rest were books and other sources

● Out of total corpus

– Estimate 500 total relevant peer reviewed papers

– 19% of total corpus → 95% confidence level w/ 9% margin of error (sort of...)

● Full reviews available at calsci.org → FDB




Basic results● Evidence of science seems to be lacking

– 88% no evidence of common language

– 82% no identified scientific conceptual basis

– 76% no identified testability or testing

– 75% no identified validation

– BUT - 59% do identify a methodology

● Internal (in)consistency of these results

– 20 redundant reviews (40 rev. 2/ea of 20 papers)

– Inconsistencies: Science (3/20 = 15%), Physics (0), Testability (4/20 = 20%), Validation (1/20 = 5%), Language (1/20 = 5%) → 9% error rate over all




More basic results● Internal consistency of categories tested by

comparing “primary” in redundant reviews– 13 had different “primary” areas

– Only 2 had identical area and sub-area

● Classifications were:– Primary (1/review): 26% legal methodology, 20% evidence

analysis, 8% tool methodology, 8% evidence interpretation, 7% evidence collection, 31% other (ea < 4%)

– Secondary: 28% evidence analysis, 20% legal methodology, 19% tool methodology, 15% evidence collection, 12% evidence interpretation, 10% tool reliability, evidence preservation, tool testing, 9% tool calibration, app. of defined methodology, <7% other




Some analysis of results● Entirely subjective and imperfect – but suggests

– Immature field

– Definitions not uniformly accepted

– Terms often not well-defined

– Testability, validation and scientific foundations are not as well addressed as in other areas

– Underlying scientific methodologies not regularly or rigorously applied

– Lack of consensus surrounding basic issues

– Heavy focus on identifying methodologies

– Many researchers “defining their own”




Longitudinal analysis● <2001:

– Methodology a major issue

● 2001-2005:

– Analysis, interpretation, attribution focus

● 2005-2009:

– Methodology again a focus

● 2009-present:

– Analysis focus returned

● Speculative results

– incomplete samples, “eyeballing it”, government publications, legal rulings, sampling error, etc.




Outline

● Introduction







Peer reviews● Climate change as a comparison basis?

– Jury pools understand this (presumably)

– Few other areas studied as well recently

– If that is controversial, digital forensics is too?

● Use of physics questions?

– (Issue 1) Control questions are commonly used

– Is digital forensics about physics or archeology?● Can you do archeology without a physics?

– Why are these issues important ones?● If we can't make copies without alteration...● If we don't know if it's latent, do we need tools? …




Peer reviews● Knowing these things helpful in doing forensics?

– You can “do forensics” without knowing how your tools work or why... but...

– Are you an expert if you don't know them?

– How do you explain why they are as they are?

– How do you know if your tools are working right?

● Questions unclear, terminology not widely used?

– The reviewers seem to agree... we lack a common terminology that is widely used

● The null hypothesis is confirmed – no consensus

– Paper accepted with these comments →● Consensus issue is important and problematic




Null hypothesis confirmed● Results suggest:

– Scientific consensus in the area of digital forensic evidence examination is lacking in the broad sense, but that different groups within that overall community may have limited consensus around areas in which they have special expertise.

– Current peer-reviewed publication process is not acting to bring about the sorts of elements typically found in the advancement of a science toward such a consensus.

● Further study is clearly indicated




Thank You

http://calsci.org/ - calsci at calsci.orghttp://all.net/ - fc at all.net

the state of the science of digital evidence examinationall.net/talks/2011-01-31-ifip.pdfjan 31,...

Documents