City of New Orleans - NOLA. of New Orleans Email Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd

Download City of New Orleans - NOLA.    of New Orleans Email Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd

Post on 26-May-2018

212 views

Category:

Documents

0 download

TRANSCRIPT

City of New Orleans Email Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd 1300 Perdido St. New Orleans, LA 70112 Submitted by: SunBlock Systems, Inc. David Sun 1616 Anderson Rd. #350 McLean, VA 22102 SunBlockSystems EmailRecordsRetention Page2of271 ExecutiveSummary..................................................................................................... 31.1 EmailSearch ......................................................................................................... 31.2 LTCReport ............................................................................................................ 31.3 ITOperationalIssues ............................................................................................ 42 EmailSearch................................................................................................................ 62.1 EmailSourcesIdentified....................................................................................... 62.1.1 Serverbasedsystems.................................................................................... 62.1.2 DesktopandLaptopcomputers.................................................................... 72.1.3 BlackBerryhandhelddevices ....................................................................... 72.1.4 Backuptapes................................................................................................. 82.2 DataPreservationMethodology ........................................................................ 132.2.1 Serverbasedsystems.................................................................................. 132.2.2 DesktopandLaptopcomputers.................................................................. 132.2.3 BlackBerryhandhelddevices ..................................................................... 142.2.4 Backuptapes............................................................................................... 142.3 SearchResults .................................................................................................... 142.3.1 BlackBerryDevices...................................................................................... 142.3.2 DesktopandLaptopComputers ................................................................. 152.3.3 Servers......................................................................................................... 152.3.4 AlternativeSearchMethodology................................................................ 162.4 EmailSearchFindings......................................................................................... 173 ReviewofLTCReport................................................................................................ 184 OperationalIssues .................................................................................................... 245 Appendix ................................................................................................................... 27SunBlockSystems EmailRecordsRetention Page3of271 Executive Summary SunBlockSystemswasretainedbytheCityofNewOrleanstosearchtheinformationtechnology(IT)infrastructureoftheMayorofNewOrleansinordertoidentifyandproduceemailssenttoandfromMayorC.RayNaginbetweenJuly20,2008andDecember1,2008,inclusive.Inaddition,SunBlockwasdirectedtoreviewtheEmailRecoveryProjectReportissuedbytheLouisianaTechnologyCouncilonJuly6th2009forirregularities.Lastly,SunBlockwasdirectedtodocumentanyIToperationalissuesencounteredaspartofourefforts.Thisreportdocumentsourfindings.1.1 Email Search SunBlockconductedathoroughreviewoftheMayorofNewOrleanssITinfrastructureandnumeroussystemswereidentifiedaspotentiallystoringMayorNaginsemaildata.Numerousservers,computersfromkeypersonnel,andBlackBerryhandheldswereforensicallyprocessedusingBitFlare,EnCase,andotherstandardcomputerforensicstools.Areviewofthesedevicesandsearchesforbackuptapesproducedvariousemailsbutdidnotprovideacomprehensivesourceofallmessages.Analternativesearchmethodologywasemployedtolocatemessages.AreciprocalmethodologywasutilizedthatreviewedemaildatafromallaccountsontheMayorsOfficeserversformessagesfrom/toMayorNagin.Usingthereciprocalmethodology,overtwomillionmessagesweresearched.UsingareviseddaterangeofJuly12008toDecember12008,thousandsofmessageswereidentifiedanddelivered.Basedonthesystemsinplace,itisunlikelythatanysignificantnumberofadditionalmessageswillbefound.1.2 LTC Report SunBlockwastaskedtoreviewthefindingsandreportissuedbytheLouisianaTechnologyCouncil(LTC)onJuly6th,entitledEmailRecoveryProjectReport.Issue3.1:LTCdidnotidentifyalternativesourcesLTConlyidentifiedtwoserversandNaginsdesktopaspotentialsourcesofemail.BlackBerryhandhelds,computersforkeystaffandfileserverscontaininghundredsofthousandsofmessageswerenotexaminedpriortoreleasingtheirfindings.Inaddition,alternativesearchmethodologiesthatwouldhaveyieldedresultswerenotemployed.Issue3.2:StandardforensicprotocolswerenotutilizedAlthoughtheuseofdatarecoverysoftwarewasdocumented,properforensictechniquespreventingthecontaminationoftheforensicdatawerenotemployed.OurreviewoftheharddrivesprovidedbyLTCindicatethatitwasnotuntilJune212009,approximatelysevenweeksafterworkbegan,thatLTCoroneofitsaffiliatesutilizedarudimentarytooltocreateaforensiccopyoftheMNOMail01server.Industrystandardpracticeincludestakingprecautionstopreventmodificationofcomputerevidenceduringananalysis.Ifaforensiccopyofacomputerdriveistobemade,thecopyshouldSunBlockSystems EmailRecordsRetention Page4of27bemadefirstandthentheanalysisconducted,nottoconductananalysisfirstandthenmakeaforensiccopy.Failuretofollowtheproperprocedurescancausethelossordestructionofpotentiallyrelevantinformationandcontaminationofanyforensicexaminations.Issue3.3:ResultsfromRecoverDatautilitywerenotproperlyanalyzedLTCprovidedtheresultsfromtheRecoverDatautilityinsupportoftheirconclusionthat22GBofdatawasdeletedfromthemailserverpriortotheirarrival.SunBlockstestingoftheRecoverDatautilityhasfoundanerrorintheprogramwhichcanincorrectlystatefilesizes.TheEDBfileinquestionwasextractedusingalternativetoolssuchasBitFlareandWindows.Acomparisonofthedigitalfingerprintsfortheseextractedfilesprovidedanexactmatch.ThiscorroboratesthefindingsoftheITdepartmentthatthatRecoverDatadidnotaccuratelyseeanyadditionaldatabeyondthe66GBavailable.1.3 IT Operational Issues Alongwiththepreviouslydescribedtasks,SunBlockwasalsoaskedtodocumentanyoperationalissuesencounteredthatwouldbeofinterest.ThegoalwastoassistinupdatingpoliciesandproceduresforthemanagementandoperationsoftheemailserverplatformaswellasprovidetheCitywithbestpracticesrecommendations.Issue4.1:NouniformemailretentionpolicyexistsDuringinterviews,OfficeofTechnologypersonnelstatedthatnoemailpolicyorAcceptableUsePolicyexisted.FurtherresearchbySunBlocklocatedanemailpolicydatedMay13,2008ontheCitysiteat:http://www.cityofno.com/Portals/Portal98/Resources/EmailPolicies.pdfHowever,thispolicyisnotwidelyknown.Inorderforapolicytobeimplemented,itmustbeuniformlydisseminatedsothatemployeesunderstanditexists.Issue4.2:ConfusionregardingimplementationofpoliciesThereissignificantconfusionamongCitypersonnelregardingtheimplementationofpoliciesandproceduresprovidedforemailretention.Sinceemailserverspaceandbackupcapacityislimited,anITdirectiveexistswhichencouragesalluserstoroutinelycleartheirmailbox.Usersareoftensentreminderstoreducethestorageutilizationoftheirinbox.Thisdirectivefailstoprovideproperinstructionsonhowtosatisfyrequirementsoftheemailretentionpolicyreferencedabove,potentiallyleadingtoconfusionamongusers.Issue4.3:BackupproceduresdrivenbytapebudgetsWithoutaformalpolicy,theOfficeofTechnologyhasimplementedbackuprequirementsandproceduresonanadhocbasis.Budgetconsiderationsandprocurementissueshaveunderminedbackuppractices.Initially,backupdatawasretainedfora30dayperiod.Assystemsgrewanddatadoubledinsize,insteadofpurchasingadditionalbackupcapability,backupretentiontimeswerereducedbyhalftotwoweeks.SunBlockSystems EmailRecordsRetention Page5of27Issue4.4:UtilizingsystembackupsforemailarchivingEmaildatacanbetransientinnature.Sincetraditionalsystembackupsonlyoperateonadailybasis,theyarenotabletokeepupwiththetransientnatureofemail.Withnightlysystembackups,amessagethatisreceived,readanddeletedonthesamedaywillnotbeproperlybackedup.Atrueemailarchivingsolutionswasnotutilized.Manysuchsolutionsexistinthecommercialmarketplaceandcanaddresstheneedsofpublicrecordsemailretentionbycapturingallincomingandoutgoingemailstoanoffsitearchive.Theyallowforsecurepreservationofemailsandauserisunabletodeletetheirmessagesfromthearchive.Thesesolutionsarerelativelyinexpensiveandshouldbeutilized.SunBlockSystems EmailRecordsRetention Page6of272 Email Search InconductingtheemailsearchforemailstoandfromMayorC.RayNagin,SunBlockbeganwithitsstandardeDiscoveryprotocolsuccessfullyusedinsimilarmatters.Thisprotocolbeginswithanoverallreviewofallsystemsandoperationalproceduresinplacewithoutinitiallyfocusingonanyspecificsourcesofemail.Thiscomprehensiveapproachprovidesabroadunderstandingofallpotentialsourcesofdata,allowingformorereliableidentificationofallpossiblesourcesofresponsiveemails.2.1 Email Sources Identified TheSunBlockeDiscoveryprotocolbeganbyprovidingCitytechnicianswithamultipagesurveyconsistingof24questionsregardingsystemsandoperationalproceduresinplaceattheMayorsOfficeofTechnology.ThesurveywasfollowedupwithphoneinterviewsandfacetofacemeetingsattheCitysOfficeofTechnology,theCityAttorneysOffice,andMayorNagin.Basedonthedataprovided,itbecameclearthattheOfficeofTechnologyadministerstwodifferentemailsystems:oneforallCityemployees,andonededicatedtotheMayorsofficeandpersonnel.ItwasdeterminedthatSunBlockwouldconfineitssearchtosystemsdedicatedtotheMayorsofficeandpersonnel.WithexclusivefocusonthesystemsfromtheMayorsoffice,numeroussourcesofemaildatawereidentified.Thesourcesincludedserverbasedsystems,DesktopandLaptopcomputers,BlackBerryhandhelddevices,andbackuptapes.2.1.1 Server based systems Basedonthereview,numerouscomputerserverswereidentifiedthatcouldpotentiallyhavetracesofemaildata.Thesesystemsstoreand/ortransferemailaspartoftheiroperationalrolesorhavehademaildataplacedontheminthepastasaresultofaspecificproject.Theseserversincludethefollowingmachines:ServerFunction StorageSize(GB)ServerAliasInitialMayoremailserver 136 MNOMail01NewerMayoremailserver 686 Mail2ProductionBlackBerryServer60 BlackberryvRetiredBlackBerryServer 68 BlackberrypFileserver 514 File1DomainControllers 164 DC1,DC2,DC02pITLabServer 137 ITLabRetiredSPAMServer 34 SPAMBackupServer 68 BackupRetiredFileserver 68 OldFileTotal 1,931 SunBlockSystems EmailRecordsRetention Page7of27 Note:Duetopossiblepublicreleaseofthisdocumentandrelatedsecurityrisks,actualmachinenameshavebeenreplacedwithaliasesfordevicesnotpreviouslydisclosed.2.1.2 Desktop and Laptop computers Aspartofthesearch,desktopandlaptopcomputerswhichmayhavestoredemailsofinterestwereidentified.ThesedevicesareofinterestsincetheymaystoreemailtoorfromtheMayoreitherinvisiblefilesorpreviouslydeleteddataavailablethroughaforensicreview.BasedonourinterviewswithITpersonnel,theMayor,andhisstaff,itwasdeterminedthatonlytheMayorandhisassistantPatriciaSmithhadaccesstoemailintheMayorsaccount.ThissecuritysettingwasverifiedbySunBlockduringthereview.Inaddition,itwasdeterminedthatMichaelLaFrancefromtheITdepartmentstoredcopiesoftheMayorandotheruseremailsonhiscomputers.Thisisduetohisoperationalresponsibilitiessupportingbackupsandprocessingofpublicrecordsrequests.Inall,thefollowingmachineswereidentifiedandsecured: MayorNagindesktop MayorNaginlaptop PatriciaSmithdesktop PatriciaSmitholddesktop MichaelLaFrancedesktop MichaelLaFrancelaptop MichaelLaFranceolddesktopInadditiontotheabove,asearchforoldmachinesfromexistingemployeesorthoseofdepartedemployeeswhomayhaveaccessedtheMayorsemailwasconducted.Thefollowingmachineswerenotavailableastheyarebelievedtohavebeenrepurposedordestroyed. MichaelBevinsdesktop WayneGatlindesktop MayorNaginolddesktop2.1.3 BlackBerry hand held devices TheMayorisaheavyuserofhisBlackBerryhandheld.Assuch,hiscurrentandpreviousBlackBerrydevicesareexpectedtopotentiallycontainemailsofinterest.Duringthereview,threedifferentdevicesreportedlyusedbyNaginwerelocatedandsecured.DataonthesedevicesarereportedtogoasfarbackasNovember20,2003.ThefollowingBlackBerryhandheldswerelocated:SunBlockSystems EmailRecordsRetention Page8of27BlackBerry8830 BlackBerry9530 BlackBerry90002.1.4 Backup tapes Areviewofthebackupproceduresproducedalackofpertinenttapes.TheoperationalproceduresinplacebytheOfficeofTechnologyonlyretaineddataforatwoweekperiod.Anybackuptapesolderthantwoweekswereoverwritten.AssuchtheoldestsetofbackuptapesretainedwasmadeinFebruary2009.Inaddition,noformalemailarchivingsystemwasinplace.Suchashortretentionperiodisunusual.Thisappearstobeduetoprocurementissuesandhumanerror.HistoricaldocumentationprovidedbyCibernotedthetwoweekretentionperiodasfarbackasJune2007.InJanuary2008,problemswithpurchasingmoretapesthreatenedtoreducetheretentionperiodtooneweek.NewtapesarrivedinMay2008.Inadditiontoprocurementissues,asdisclosedduringaninterviewwithITpersonnel,acitycontractorinchargeofmaintainingconsistenttapebackupsforvarioussystemsincludingtheemailserverfailedtoproperlyperformthisduty.ThecontractorisnolongeremployedbytheCity.OffsitestorageiscurrentlybeingprovidedbyIronMountainbutthisprocedurewasimplementedinMarch2009andnobackupsetsfromtheperiodofinterestwereeverstoredthere.AtSunBlocksrequest,aprocurementrecordsreviewwasconductedbytheOfficeofTechnologysContractingManagertoverifythatnootheroffsitestoragevendorsareutilizedbytheITdepartmentfortheCity.Whilethereviewdidnotidentifyanyknownbackupsetsfromthedesiredtimeframe,SunBlocksstandardprotocolincludesaphysicalinspectionofthedatacenterandrelatedareas.ThisinspectionisdesignedtoassistITpersonnelinpotentiallyidentifyingdatasourceswhichmayhavebeenoverlookedduringprevioussearches.GiventheSunBlockSystems EmailRecordsRetention Page9of27physicallayoutoftheCitysdatacenter,theinspectionencompassedmanyofficesuitesaswellasaninspectionunderthedatacenterflooring.DataCenterInspectionSamplePhotograph1DataCenterInspectionSamplePhotograph2SunBlockSystems EmailRecordsRetention Page10of27DataCenterInspectionSamplePhotograph3SunBlockSystems EmailRecordsRetention Page11of27DataCenterInspectionSamplePhotograph4DataCenterInspectionSamplePhotograph5SunBlockSystems EmailRecordsRetention Page12of27Aspartoftheinspection,additionalbackuptapeswereidentified.Threedistinctsetsoftapeswerefound.Whilethecontentsofthesetapeswerenotcataloged,theyarenotbelievedtocontainemaildatafromthespecifiedperiodofinterestduetothetypesoftapemediafound.However,basedontheoldertapemediausedandphysicallabeling,oneormoresetsmaycontainemaildatafromaperiodpriortothetimeframeoftherequested.AttherequestofSunBlock,44membersoftheCityHallMISstaffparticipatedinasearchfortapesencompassingtheirworkspace,homeandotherpossiblestoragelocationsundertheircontrolwithnofurtherresults.Basedonthis,nootherbackuptapesarebelievedtobeavailable.BackupTapesFoundPhotograph1SunBlockSystems EmailRecordsRetention Page13of27BackupTapesFoundPhotograph22.2 Data Preservation Methodology Oncedatasourceswereidentified,thenextphaseofthereviewconsistedofproperlypreservingallpotentialevidence.Whereverpossible,aforensicpreservationofthepotentialemailsourceswasperformed.Doingthisattheearliestpossibleopportunityisafundamentalcomponentofanyforensiceffort.Itiswellknownthatelectronicevidenceishighlyfragileinnatureandthatforensicdataislostovertimethoughthenormaloperationofacomputersystem.Inaddition,itisalsoknownthatunnecessarymodificationsofthecomputerpriortopreservationshouldbeavoidedasitmayspoilpotentialevidence.2.2.1 Server based systems SincesomeoftheserversfortheMayorofNewOrleanssystemwerephysicaldeviceswhileotherswerevirtualmachines,differenttechniqueswereemployedforserverbasedsystems.Forvirtualservers,theVMDKfileswhichcomprisethevirtualserverwerecopiedanddigitalsignatureswerecalculatedtoverifydataintegrity.ThephysicalserverslackedUSB2.0portsandwerethereforeforensicallycapturedusingEnCaseForensic,acommonforensictool.2.2.2 Desktop and Laptop computers ThedesktopandlaptopcomputerswereforensicallypreservedusingBitFlare.Duetothedesiretoremainascosteffectiveaspossibleandgiventimeconstraints,BitFlarewasutilizedfortheforensicpreservationandexamination.CityITstaffusedBitFlaretocreatesecuredforensiccopiesofharddrivesaswellasperforminsitusearchesandextractionofforensicdata.ThiswasfacilitatedbyBitFlaresautomatedChainofSunBlockSystems EmailRecordsRetention Page14of27Custodyandindependentvalidationfunctionality.Inaddition,byspecifyingtheappropriateEvidenceDiscoveryPack,preliminaryresultswereextractedwithouttheneedtowaitforalaboratoryanalysis.Fragmentsofresponsiveemailsfromthedesktopandlaptopcomputerswereimmediatelyextractedandavailableforreview.2.2.3 BlackBerry hand held devices ThethreeBlackBerrydeviceswerebackedupusingtheRIMDesktopManager.ThesedatafileswerethensavedforsubsequentanalysisusingAmberBlackBerryConverter.2.2.4 Backup tapes Preservationofthetapesconsistedofdocumentingthetapesandsecuringtheirlocationincasefutureanalysisiswarranted.TheFebruary2009MNOMail01backuptapewassecuredbySunBlock.TheMail2tapewasverifiedtobeattheIronMountainstoragefacilitywhereanysubsequenttransferstoCityofficialswouldbedocumented.Thebackuptapesfoundduringtheinspectionweresecuredinatamperevidentbox,preservingChainofCustodypendingfurtherexaminationbySunBlock.Furtherexaminationhoweverwashamperedwhenaconcurrentlawenforcementinvestigationremovedthiscontaineroftapes.2.3 Search Results Basedonthereviewofallsystemsandoperationalproceduresinplace,nosinglecompleteandreliablesourceofemailsfortheMayorinthetimeperiodofinterestwasidentified.Thisisdueinlargeparttothelackofaformalemailarchivalsysteminplaceandtheshorttermretentionofbackupdata.However,usingacombinationofnewtechniquesemployedbySunBlock,manypartialsourcesofemaildatawereidentifiedandexamined.Giventhelargenumberofsourcesidentified,limitedresourcesavailable,andpreviousworkperformedthebytheITdepartmentandotherentities,SunBlockfocuseditseffortsonnewareasmostlikelytogenerateadditionalresults.Adetaileddiscussionofthosesourcesandtheresultingmessagesfollows.2.3.1 BlackBerry Devices AreviewofthethreeBlackBerryhandhelddevicesreportedlyusedbyMayorNaginonlyprovidedonedevicewithrelevantemails.Twoofthedevicesdidnotcontainrelevantemails.Table1belowisatabledescribingthedataavailableforeachdevice.BlackBerryDevice EmailsBlackBerry8830 0BlackBerry9530 0BlackBerry9000 422(11/20/20038/5/2009)Table1.DataAvailablefromBlackBerryHandhelds.SunBlockSystems EmailRecordsRetention Page15of27OnesignificantfindingfromthereviewoftheBlackBerrydevicesisthatoldermessageswereavailableonthehandheldandnotallmessagesweredeleted.ThesemessagesdidnotmatchNaginsmessagesontheemailserver.ThiswouldcontradictatheoryprovidedbytheOfficeofTechnologyintheirresponsetotheLTCpreliminaryreport.ThattheorystatedthatmessagesweresystematicallyremovedfromMayorNaginsinboxbecausehedeletedmessagesfromhisBlackBerryandithadasettingofHandheldwins.Ifthistheorywascorrect,therewouldbeahigherdegreeofcorrelationbetweenthemessagesontheBlackBerryandthoseontheserver.2.3.2 Desktop and Laptop Computers Inthissearch,thedesktopandlaptopcomputerswerereviewedforemaildata.OutlookdatafilesfromPatriciaSmithandMayorNaginscomputerswerereviewed.MayorNaginsOutlookdatafilescontainedonlyafewemails.Inaninterview,theMayorstatedthatheoperatedunderthebeliefthathisemailmessageswerebeingpreservedindependentlyofhisactions.Assuchhewouldroutinelydeletehisemailsafterreadingthemtokeephisinboxclear.Confusionregardingtheemailretentionpolicymayhavecontributedtothis.Tofurthersearchforemaildata,aforensicexaminationwasalsoconductedofallrelevantdesktopandlaptopcomputers.EmailfragmentswereextractedbyBitFlareaspartofaninsituforensicanalysisandsubsequentlyreviewed.Whilenumerousemailfragmentswererecovered,thedifficultlyinreconstructingthesefragmentstocreateentiremessagesforthetimeperiodofinterestrequireddirectingsearcheffortstoalternativesources.Theabilitytoforensicallysearchthesedevicesremainsshouldsufficientresourcesbededicatedtothistask.2.3.3 Servers Apreliminaryforensicreviewoftheemailserverswasconducted.However,extensivechangesweremadetotheforensicdataontheserverduringtheLTCanalysisandasubsequentreviewwouldhaverequiredconsiderableefforttodistinguishbetweenactivitiesfromtheprioranalysisandtrueoperationalactivity.This,combinedwiththehigherlikelihoodofsuccessusingalternativesearchmethodologiestosearchunmodifiedsources,ledtoahaltofforensicallyexaminingtheemailservers.Sincetheemailserverswereforensicallypreserved,suchanexaminationmayeasilybeconductedinthefutureifnecessary.Otherserverssuchasfileserversandmachinesusedtoprocesspublicrecordsrequestswereexamined.WhileworkingwithCityITpersonnel,variousPSTfileswereidentifiedandanalyzed.ThesePSTfilesallowedformultiplesearchmethodstobeemployed.SunBlockSystems EmailRecordsRetention Page16of272.3.4 Alternative Search Methodology WithoutasinglecomprehensiverepositoryofemailsforMayorNagin,analternativesearchmethodologywasdeveloped.InsteadofsimplysearchingNaginsinboxformessages,areciprocalsearchwasimplementedinwhicheachpotentialrecipientsmailboxwassearchedformessagesfrom/toNagin.Suchanapproachhaslimitationsasitishighlyunlikelytocovertheentireuniverseofemailssent/receivedbyNaginandissubjecttothediscretionofeachrecipientsdeletion/retentionpreferences.However,itdidgenerateemailmessagesthatcanbereasonablylinkedtoNaginsemailaccount.Usingthisreciprocalsearchmethodology,areviewofemaildatafromalluseraccountsontheMayorsOfficeemailsystemwasconducted.EmailmessagesweregatheredfromtheMayorsOfficeemailserversaswellastheFileServerssincenumerousOutlookPSTfileswereidentifiedonthenetworkdriveforusers.ThesePSTsareemaildatabasefilescreatedbyuserstofacilitatestorageofmessageswithoututilizingemailserverstorage.Thisisoftendonebyauserinresponsetomailboxstoragelimits,archivalofmessages,personalpreferencesandotheroperationalneeds.Inall,153UserPSTfileswereidentifiedfromthefileserverinadditiontothemessagesstoredontheemailservers.Aspartofthisreciprocalsearch,amailboxforSelethaNaginwaslocated.ThisaccountwasofparticularinterestasnumerouspersonalmessagestoorfromMayorNaginmightbeexpectedinthisaccount.Whilenumerousmessagesdidexistinthisaccount,SunBlocksinspectionidentifiedacontinuitygapintheaccount.Theaccountwasuseddailyandhasnumerousemailmessageswhichspanamultiyearperiod.However,fromMay7,2008toOctober8,2008theaccountlacksanyemailmessages.Numerousmessagesappearbeforeandafterthistimeperiod.Inaddition,areviewofCalendar,ContactsandNotesfortheaccountshowsasimilarlapseinentries.Noexplanationisavailableatthistimeforthis.GiventhatMrs.NaginonlyaccessedheremailaccountusingaBlackBerry,itisunlikelyheractionswouldhavecausedthisgap.Thisactivitygapispossiblyduetoasystemlevelchangewiththeaccount.Whileitisdifficulttoknowtheexactchange,possibleexamplesofchangesthatmayexplainthisgapinclude:1) removalandsubsequentpartialrestorationoftheaccountand2) temporaryabandonmentoftheaccountwithmessagesbeingredirectedelsewhere.Notwithstandingthegapsinmessages,thereciprocalsearchexaminedovertwomillionmessagesfromtheMayorsOfficeemailserverweresearched.Thosemessages,combinedwithdatafromNaginsCityemailaccountprovidedbyITpersonnel,resultedinover20,000messagesto/fromNaginwithinareviseddaterangeofJuly1,2008toDecember1,2008.Duplicatesandcalendarentrieswereremovedandultimately4,946uniquemessageswereproducedfortheperiodofinterest.SunBlockSystems EmailRecordsRetention Page17of272.4 Email Search Findings Basedontheemailsearchconductedthefollowingconclusionscanbemade:Finding#1:Alargenumberofsystemspotentiallycontainingemaildatawereexaminedandthousandsofemailswereproduced.Usingvarioustechniquesincludinginterviewsofkeypersonnel,physicalinspectionsofCityHallITfacilities,andforensictechniques,numeroussourcesofemaildatawereidentified.Sourcesomittedbypreviouseffortssuchasfileserversandcomputerswerereviewed.Additionaltapesandpotentialsourcesofdatawerediscovered.Sourcesdeemedmostlikelytobefruitfulwereexaminedandthousandsofmessageswereidentified.Finding#2:AnarchiveofMayorNaginsemailswasnotcreatedormaintained.NosinglesourceofMayorNaginemailswasidentified.Backuptapesfromtherelevanttimeperiodwerenotretained.Noformalemailarchivesystemwasimplemented.Finding#3:Itisunlikelythatanyfurthersearcheswillyieldsubstantiveresults.Basedontheinformationgatheredandtheamountofdataexamined,itisunlikelythatfurthereffortswillyieldsubstantiveresults.Whileaforensicreviewoftheharddrivesmayproduceadditionalmessagefragments,searchingforthesefragmentsbasedsolelyondatesentisimpractical.Sincefragmentationofdeletedemaildataoftenresultsinthemessagebodybeingseparatedfromtheenvelopeinformationcontainingthedatesent,reconstructionofthesemessageswouldbecostprohibitiveSunBlockSystems EmailRecordsRetention Page18of273 Review of LTC Report Inadditiontotheemailsearch,SunBlockwasretainedtoreviewthereportandfindingsfromtheJuly6thLouisianaTechnologyCouncil(LTC)entitledEmailRecoveryProjectReportforirregularities.Severalissueswereidentifiedwithmethodologyandfindingsdetailedinthereport.Theissuesweredeterminedbasedonareviewofthereport,apreliminaryexaminationoftheharddrivesprovidedbyLTCfromtheiranalysis,analysisoftoolsemployedbyLTC,discussionswithpersonnelattheOfficeofTechnology,andareviewoftheOfficeofTechnologyResponsetoStateIrregularitiesdocument.Duetopendinglitigation,discussionswithpersonnelfromLTCwerenotpossible.SunBlockreservestherighttomodifythesefindingspendingadditionalinformationordiscussionswithLTCpersonnel.Issue3.1:LTCdidnotidentifyalternativeemailsourcesorsearchtechniquesFromtheLTCreport,onlytwoemailserverswereidentifiedtobeofinterest.Taperestorationsofthoseemailservers(primarilyMNOMail01)wereperformedbutasstatedinthereport,LTCsworkwasconfinedtotheCitystwomailservers.LTCalsostatesthatthetwomostlikelyplacesforrecoveryofalltheemailmessageswereontheMayorsdesktopcomputerintheformofOSTfiles.WhiletheMayorsdesktopcomputerisanadditionalsourceofemailmessages,itisclearthatthereexistnumerousothersourcesofemaildatanotidentifiedbyLTC.AswasdemonstratedbytheEmailSearch,othersourcesofemaildataincludedtheBlackBerryhandhelds,computersforkeystaffsuchashisassistant,MayorFileandEmailserversaswellaspossiblytheadditionalbackuptapesthatwerefoundthroughaphysicalreviewofthesite.ManyoftheseadditionalsourceswouldhavebeendeterminedbyconductingathoroughreviewoftheITsystemaswellasaskingpersonnelaboutadditionalbackuptapesandinspectingthefacilities.Inaddition,alternativesearchmethodologiesthatwouldhaveyieldedresultswerenotemployed.Areciprocalsearchmethodologyproducedthousandsofrelevantemails.Issue3.2:StandardforensicprotocolswerenotappropriatelyemployedBasedonthereportprovidedbyLTC,forensicprotocolswerenotappropriatelyemployed.Whiletheuseofdatarecoverysoftwarewasdocumented,properforensictechniquespreventingthecontaminationofforensicdatawerenotemployed.ConsiderableforensicdatawasmodifiedaspartoftheLTCanalysis.Useraccountswerecreated,programswereinstalledanddatawasrestoredontheserversthemselves.Alloftheseactivitiescausedpotentiallypreciousforensicdatatobedestroyed.Whiletracesoftheinformationstillexist,themodificationunderminesitsusefulnessandincreasestheexpenserequiredtoexaminethedata.SunBlockSystems EmailRecordsRetention Page19of27Itshouldbenotedthat,whilenotdocumentedintheLTCreportandbasedonforensicdataexamined,LTCoroneofitsaffiliatesfinallydidutilizetherudimentaryDDutilitytocreateaforensiccopyoftheMNOMail01serveronJune21,2009at4:50PM.DDisasystemadministrationtoolwhichcanalsoserveasacopyingprogramtomakeforensiccopiesofacomputerharddrive.AforensiccopyoftheMNOMail01datawasfoundondrivesprovidedbyLTCfromtheiranalysis.ThisforensiccopywasdatedJune22,2009,approximatelysevenweeksafterworkbeganandonlydaysbeforeLTChaltedtheirwork.Bythistime,LTChadalreadymodifieddataontheMNOMail01serverasuseraccountsforLTCpersonnelandsomeofthevariousdatarecoveryprogramsutilizedbyLTCwerecontainedwithinthatforensicimage.Industrystandardpracticeincludesprecautionstopreventmodificationoforiginalcomputerevidence.Forensiccopiesofacomputerdrive,ifmade,shouldbecreatedbeforetheanalysisisconducted,notafterwards.Followingtheproperprocedurescanpreventthelossordestructionofpotentiallyrelevantinformationandcontaminationoffutureforensicexaminations.Issue3.3:ResultsfromRecoverDatautilitywerenotproperlyanalyzedIntheLTCreport,muchisdiscussedregardingtheCityHallinformationstoreandpurporteddeletionof22GBworthofdata.Screenshotsprovidedintheirreport,repeatedbelow,demonstratethebasisforthisassertion.ThereportstatesthatthatthefirstscreenshotfromwithinMicrosoftWindowsdepictthedatabaseonMnoMail01now66Gb[sic]andthatthesecondscreenshotfromtheRecoverDatatoolindicateswhatcanbeseenasavailableforrecovery(meaningthatthesefileswereDeletedandmaybeabletoberecovered).Thesizeofthefileshownaspotentiallyrecoverableis88GB.Fromthis,LTCstatesthat22GBwasdeletedfromthePriv1.edbdatabase.Themostobviouserrorinthisstatementisinreferencetothe88GB(Gigabyte)value.AnexaminationofthescreenshotfromtheLTCreportclearlyshowsthesizevaluelistedbyRecoverDatais88875008(88,875,008)whichequalapproximately88Megabytes.AMegabyteis1/1024thofaGigabyte.RecoverDatadoesnotspecifyanyunitsbutthescreenshotsbelowcomparingthesizeforotherfilescorroboratethatRecoverDatadisplaysfilesizesintheunitofbytes.SunBlockSystems EmailRecordsRetention Page20of27Asaresultofthis,thereisnoevidencetosupportstatementsregarding22GBofmissingdata.Infact,ifoneweretofollowthesamelogicappliedintheLTCreport,usingaUnitisbytesSunBlockSystems EmailRecordsRetention Page21of27correctreadingofthevaluesprovidedbyRecoverData,onewouldcometotheconclusionthattheservercontains87.9GBofadditionaldata.TofurtherexaminetheaccuracyofRecoverData,thesizeofthePrive1.edbfileindicatedwascomparedwiththeoutputfromotherforensictools.Belowisascreenshotshowingsomeofthecomparisons.RecoverDatadisplaysafilesizeforthefileswhichareinconsistentwiththosedisplayedbyotherforensicandstandardtools.ThefollowingtablesummarizesthecomparisonofthefilesizesbetweenRecoverDataandothertools.Tool Priv1.edb Priv1.stm ResultWindows(KB)Convertedtobytes67,195,656KB68,808,351,7445,228,552KB5,354,037,248 EnCase(bytes) 68,808,351,744 5,354,037,248BitFlare(bytes) 68,808,351,744 5,354,037,248RecoverData(bytes) 88,875,008 1,059,069,952SunBlockSystems EmailRecordsRetention Page22of27LTCalsostatedintheReportthatthesefiles[priv1.edbandpriv1.stm]wereDeletedandmaybeabletoberecovered.Ascanbeseeninthepriorscreenshot,RecoverDatadisplaysthePriv1.edbandPriv1.stmfilesinblack.DocumentationforRecoverDatashownbelowclearlyindicatesthatdeletedfileswouldbelistedbyinred.ThefollowingscreenshotcreatedbySunBlockwithRecoverDataexaminingtheMNOMail01serververifiesthatthisred/blacklabelingisinuse.Thedeletedfileindicatedisoneofmanylogfilesthatamailserverwouldautomaticallygenerateanddeleteaspartofitsnormaloperation.SunBlockSystems EmailRecordsRetention Page23of27Giventhisred/blacklabeling,thereisnoindicationbyRecoverDatathatthepriv1.edbandpriv1.stmfilesareinadeletedstate.TheOfficeofTechnology,intheirresponsetotheLTCpreliminaryreport,usedRecoverDatatoextractthePriv1.edbfile.Theystatedthat,whenextractedbyRecoverData,theresultingPriv1.edbfilewas67,195,656KBinsize.Thismatchesthefilesizelistedbyothertoolsforthisfile.TheirextractionofthePriv1.stmfileproducedafile5,228,552KBinsize,whichalsomatcheswiththesizereportedbyothertools.TheseresultswouldindicatethatRecoverDataisreferringtotheexactsamesetofdataasalloftheotherforensictoolsemployedwhilelistingadifferent,likelyincorrect,filesizevalue.LTCappearstohaveutilizedtheincorrectresultsfromRecoverDataand/oramisinterpretationoftheunitofsizeastheirbasisthat22GBofdatawasdeletedfromthemailserver.SunBlockSystems EmailRecordsRetention Page24of274 Operational Issues Tofullyleveragetheinformationgatheredaspartofthisreview,SunBlockwasalsotaskedwithdocumentingoperationalissuesidentifiedaspartoftheemailsearchefforts.Atthistime,SunBlockhasidentifiednumerousoperationalissuesrelatingtotheOfficeofTechnologyandthehandingofelectronicpublicrecords.Ingeneral,thereisalackofpolicymanagementpertainingtothehandlingofelectronicpublicrecords,specificallyemails.Withoutproperpolicymanagement,policiesarenotcreated,implemented,orenforcedinanecessaryfashion.Inaddition,poolpolicymanagementcanalsoleadtoimproperimplementationandmaintenanceoftechnologysolutions.Specificissuesidentifiedinclude:Issue4.1:NouniformemailretentionpolicyexistsWheninterviewed,OfficeofTechnologypersonnelstatedthatnocomputer,emailpolicyorAcceptableUsepolicyexisted.UponfurtherresearchbySunBlock,anemailpolicydatedMay13,2008wasfoundontheCitywebsiteat:http://www.cityofno.com/Portals/Portal98/Resources/EmailPolicies.pdf.However,whendiscussedwithvariouspersonnelintheOfficeofTechnologyaswellastheCityAttorneysOffice,arangeofresponseswereprovidedregardingthispolicy.Theseresponsesrangedfromnotknowingthatsuchadocumentexistedtoindicatingthatthedocumentwaspublishedbutwasnotformallyimplemented.Inorderforapolicytobeimplemented,itmustbefullydisseminatedsothatemployeesunderstanditexists.Procedureswhichinformemployeeshowtocomplywiththepolicyshouldalsobeestablished.Lastly,policiesmustbeenforcedtobeeffective.Thecurrentstateofawarenesswouldindicatethatforallpracticalpurposes,noformalpolicyforemailmanagementexists.Withoutsuchapolicy,itisimpossibletoproperlyimplementanemailmanagementsolution.ItisrecommendedthattheCityAttorneysOfficeworkwiththeOfficeofTechnologyandothernecessaryresourcestofullydefineapublicdocumentsretentionpolicy.Issue4.2:ConfusionregardingimplementationofpoliciesWithoutproperpolicymanagement,thereissignificantconfusionamongCitypersonnelregardingtheproperimplementationofpoliciesandproceduresforemailretention.AfternumerousinterviewswithCitypersonnelintheITandLawdepartments,therewasobviousconfusionregardingtheexistenceofanemailretentionpolicy.Establishingauniformpolicythatisclearlydisseminatedtoallusersalongwithproceduresonhowtocomplywiththepolicyisparamounttocompliance.Forexample,emailserverspaceandbackupcapacityislimitedsoanITdirectiveexistswhichencouragesalluserstoroutinelycleartheirmailbox.Usersareoftensentreminderstoreducethestorageutilizationoftheirinbox.ThisdirectivefailstoprovideSunBlockSystems EmailRecordsRetention Page25of27properinstructionsonhowtosatisfyrequirementsoftheemailretentionpolicyreferencedabove.Specifically,thepolicystates:ThisemailretentionpolicyissecondarytoCityofNewOrleanspolicyonFreedomofInformationandBusinessRecordKeeping.AnyemailthatcontainsinformationinthescopeoftheBusinessRecordKeepingpolicyshouldbetreatedinthatmanner.AllCityofNewOrleansemailinformationiscategorizedintofourmainclassificationswithretentionguidelines:AdministrativeCorrespondence(4years)FiscalCorrespondence(4years)GeneralCorrespondence(1year)EphemeralCorrespondence(Retainuntilread,destroy)WhileAdministrativeandFiscalcorrespondenceretentionismanagedbyIT,thepolicyalsostates:TheindividualemployeeisresponsibleforemailretentionofGeneralCorrespondence.Thislackofuniformpolicymanagementcancreateconfusionfromtheusers.OneexamplecouldbeMayorNaginsbeliefthathisemailmessageswerebeingpreservedindependentlyallowinghimtodeletehisemailsafterreadingthem.Issue4.3:BackupproceduresdrivenbytapebudgetsLackingauniformlydisseminatedandenforcedretentionpolicy,theOfficeofTechnologyhasdevelopedbackuprequirementsandproceduresonanadhocbasis.Thishasallowedbudgetconsiderationsandprocurementissuestoundermineindustrybestpracticesforbackupanddocumentretention.Specifically,overaperiodoftime,thebackupretentionwindowwasreducedduetothelackoftapesandbudgetnecessarytopurchasemore.PriortoJuly2006,systembackupswereavailablefora30dayperiod.FromJuly2006toJune2007,systemswereexpandedanddatastorageutilizationapproximatelydoubledinsize.Whilethepurchaseofadditionalbackupcapabilitywasrequested,noactionwastaken.Asaresult,backupretentiontimeswerereducedbyhalftotwoweeks.Ingeneral,industryacceptedpracticesinvolvenightlybackupsofrelevantdata.Thesebackupsshouldberetainedforatwotofourweekperiod.Inaddition,fullmonthlybackupsshouldbemadeandretainedoffsiteforatwelvemonthperiod.Lastly,annualbackupsshouldberetainedoffsiteforthedurationspecifiedinadocumentretentionpolicy.ThesebackuptapesareanimportantcomponenttothedisasterrecoveryproceduresofanyITsystem.GiventhesizeandcomplexityoftheCitysITsystem,othertechnologiessuchasdisasterrecoverysitesandSANreplicationmaybeincorporatedbuttheabilitytorestoretheentiresystemtovarious,predefinedpointsintimemustbemaintained.SunBlockSystems EmailRecordsRetention Page26of27Issue4.4:UtilizingsystembackupstoperformemailarchivingEmaildatacanbefairlytransientinnature.Inaddition,SPAMandothermessagescanclutterausersinboxorBlackBerry,makingthemunwieldy.Traditionalsystembackupsonlyoperateinadailybasis.Assuchtheyarenotabletokeepupwiththetransientnatureofemailbecauseonlythedatawhichexistsatthetimeofbackupispreserved.Ifamessageisreceived,readanddeletedonthesameday,itwillnotbepreservedduringthedailybackup.ExceptionstothiscanbemadebyutilizingtheExchangedumpstersettingsbutultimatelythisisnotapracticalwaytoperformemailarchiving.GiventheincreasedoversightduetoSarbanesOxleyactof2002intheprivatesector,manyreliableemailarchivingsolutionshavebeencreated.Thesesolutionsaredirectlyapplicabletoaddressingthepublicemailrecordsretentionproblem.Thesesolutionscaptureallincomingandoutgoingemailsandcanstorethematanoffsitearchive.Theyallowforsecurepreservationofemailsandauserisunabletodeletetheirmessagesfromthearchive.Thesolutionscanalsoallowforsearchandretrievalofmessagesresponsivetoanypublicrecordsrequest.Pricesvarybetweensolutionsbutonevendoroffersasolutionfor$25/userfor1yearretentionand$45/userfor10yearsretention.Volumeandgovernmentdiscountsmaybeavailable.SunBlockSystems EmailRecordsRetention Page27of275 Appendix Thefollowingsoftwaretoolswereutilizedduringthecourseofthisreview.AmberBlackBerryConverter(ABCAmber)www.processtext.com/abcblackberry.htmlBitFlare(SunBlockSystems)www.bitflare.comBlackBerryDesktopManage(ResearchinMotion)www.blackberry.comDiscoveryAttender(SherpaSoftware)www.sherpasoftware.comEnCaseForensic(GuidanceSoftware)www.guidancesoftware.comExmerge(Microsoft)www.microsoft.comFsum(SlavaSoft)www.slavasoft.comRecoverData(NTFS)TechnicianEdition(RecoverData)www.recoverdatatools.com

Recommended

View more >