file format identification and archival processing

File Format IdentificationFile Format Identificationandand

Archival ProcessingArchival ProcessingWilliam Underwood NARA Briefing

GTRI Washington, DCAtlanta, Georgia February 6, 2009

OverviewOverview

BackgroundFile Command- Magic ExpressionsDROID-File Format Signature ExpressionsComparison-File Command/Magic &

DROID/FFSignaturesSummary

Background – Projects

Presidential Electronic Records PilOt System (PERPOS) (2001-2006)

Advanced Decision Support for Archival Processing of Presidential Electronic Records (2007-2009)

Backgound: Electronic Records atBackgound: Electronic Records atGeorge H.W. Bush Pres. LibraryGeorge H.W. Bush Pres. Library

One of the first presidential libraries to have electronic presidential records, particularly from hard drives◦ Word Processing Files◦ Databases ◦ Spreadsheets◦ Presentations◦ Email◦ Computer Programs◦ Scanned Paper Records

Background: Where We BeganBackground: Where We Began

The archival functions needed to process paper records are well understood.

We had few tools to identify, view or review electronic records in response to PRA/FOIA requests

Tools Initially Needed:◦File Format Identification Tool◦Viewers for Records in Legacy File Formats◦Tool for Filtering OS and Office Applications

Software from User-created Files◦Tools for Converting Legacy to Current Formats◦Tools to Support Redaction of E-records

Background: Background: Evolutionary PrototypingEvolutionary Prototyping

Result: Integrated set of tools called PERPOS

Background: Archival Activities Background: Archival Activities Supported by PERPOSSupported by PERPOS

Contents of PC Hard Disk

File Format Names

Filter Contents of a Hard Drive

OS and Software Application Files Blocked by Filter

File Types of Passed Files

Properties of Filtered Files

OS/App Hash Code Filter

National Software Reference Library

NSRL Reference Data Set

Viewers, Archive Extractors, Password Viewers, Archive Extractors, Password Recovery, Decrypters, Converters, RepairersRecovery, Decrypters, Converters, Repairers

Magic File – Man Page

Magic File – Man Page

Magic File – Man Page

Extensions of File Command and Magic File

Magic for individual file formats Output of file command/magic file is File Format

IDRewriting file command code for identifying

Characteristics of Text files and Document TypesDefined approx. 750 file format signaturesCollected examples of approx. 500 of the file

format typesCreated File Signature DatabaseVerified that magic file correctly identifies

approx. 500 File Types

GUI for File Type Identifier

File signatures for about 200 File Formats that are currently defined in DROID File Signature file only by file name extensions◦ Examples: Microsoft Outlook Personal folders (97-2002), AIFF

(Compressed), AutoCAD Design Web Format, Adobe Framemaker Document, Applixware Spreadsheet, Chiwriter 3 Document

File signatures for about 300 file formats that probably should be included in Pronom Registry and DROID Signature File.◦ Examples: MHTML Web Page Archive, Outlook Express E-mail

Folder, Autodesk Revit Project, CATIA Model File V4, CATIA Drawing V5, ClarisWorks 3 Document, MacWrite 4.x Document, PDF/X1a

DROID – File Signature Expressions

In PRONOM, an internal signature is composed of one or more byte sequences, each comprising a continuous sequence of hexadecimal byte values and, optionally, regular expressions. A signature byte sequence is modelled by describing its starting position within a bitstream and its value.

The starting position can be one of two basic types:•Absolute: the byte sequence starts at a fixed position within the

bitstream. This position is described as an offset from either the beginning or the end of the bitstream.

Variable: the byte sequence can start at any offset within the bitstream. The byte sequence can be located by examining the entire bitstream.

The value of the byte sequence is defined as a sequence of hexadecimal values, optionally incorporating any of the following regular expressions:

??: wildcard matching any pair of hexadecimal values (i.e. a single byte). *: wildcard matching any number of bytes (0 or more). {n}: wildcard matching n bytes, where n is an integer. {m-n}: wildcard matching between m-n bytes inclusive, where m and n are integers or

‘*’. (a|b): wildcard matching one from a list of values (e.g. a or b), where each value is a

hexadecimal byte sequence of arbitrary length containing no wildcards. [a:b]: wildcard matching any sequence of bytes which lies lexicographically between a

and b, inclusive (where both a and b are byte sequences of the same length, containing no wildcards, and where a is less than b). The endian-ness of a and b are the same as the endian-ness of the signature as a whole.

[!a]: wildcard matching any sequence of bytes other than a itself (where a is a byte sequence containing no wildcards).

[!a:b]: wildcard matching any sequence of bytes which does not lie lexicographically between a and b, inclusive (where a and b are both byte sequences of the same length, containing no wildcards, and where a is less than b).

DROID Applied to Sample Files

Comparison of DROID and GTRI file Type Identifier Technologies

DROID

Matches sequences of hex values at offsets

Regular expressions on hex values

Efficient substring search

Identifies all possible signatures and then selects the one of highest priority

Includes offsets from EOF

GTRI File Type Identifier Matches a variety of data

types at offsets Regular expressions on

strings in lines Less efficient substring

search, but more indirect offsets increase efficiency

Preorders signatures and stops search when pattern matches.

Lacks offsets from EOF

Summary

PERPOS File Format Resources◦ File Format Signatures◦ File Format Specifications/Reverse Engineering Documents◦ Software

Viewers/players Archive Extractors Converters Password Recovery & Decryption Repairers

◦ Sample FilesResearch Issues

◦ File Signature Representation Languages◦ Metadata Extraction Languages◦ File Format Description Languages