City of New Orleans - NOLA. of New Orleans Email Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd

Download City of New Orleans - NOLA.    of New Orleans Email Records Retention Final Report April 15, 2010 Prepared for: City of New Orleans Office of Technology M. Harrison Boyd

Post on 26-May-2018

212 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

<ul><li><p>City of New Orleans Email Records Retention Final Report April 15, 2010 </p><p>Prepared for: </p><p>City of New Orleans Office of Technology M. Harrison Boyd 1300 Perdido St. New Orleans, LA 70112 Submitted by: </p><p>SunBlock Systems, Inc. David Sun 1616 Anderson Rd. #350 McLean, VA 22102 </p></li><li><p>SunBlockSystems EmailRecordsRetention Page2of27</p><p>1 ExecutiveSummary..................................................................................................... 31.1 EmailSearch ......................................................................................................... 31.2 LTCReport ............................................................................................................ 31.3 ITOperationalIssues ............................................................................................ 4</p><p>2 EmailSearch................................................................................................................ 62.1 EmailSourcesIdentified....................................................................................... 62.1.1 Serverbasedsystems.................................................................................... 62.1.2 DesktopandLaptopcomputers.................................................................... 72.1.3 BlackBerryhandhelddevices ....................................................................... 72.1.4 Backuptapes................................................................................................. 8</p><p>2.2 DataPreservationMethodology ........................................................................ 132.2.1 Serverbasedsystems.................................................................................. 132.2.2 DesktopandLaptopcomputers.................................................................. 132.2.3 BlackBerryhandhelddevices ..................................................................... 142.2.4 Backuptapes............................................................................................... 14</p><p>2.3 SearchResults .................................................................................................... 142.3.1 BlackBerryDevices...................................................................................... 142.3.2 DesktopandLaptopComputers ................................................................. 152.3.3 Servers......................................................................................................... 152.3.4 AlternativeSearchMethodology................................................................ 16</p><p>2.4 EmailSearchFindings......................................................................................... 173 ReviewofLTCReport................................................................................................ 184 OperationalIssues .................................................................................................... 245 Appendix ................................................................................................................... 27</p></li><li><p>SunBlockSystems EmailRecordsRetention Page3of27</p><p>1 Executive Summary SunBlockSystemswasretainedbytheCityofNewOrleanstosearchtheinformationtechnology(IT)infrastructureoftheMayorofNewOrleansinordertoidentifyandproduceemailssenttoandfromMayorC.RayNaginbetweenJuly20,2008andDecember1,2008,inclusive.Inaddition,SunBlockwasdirectedtoreviewtheEmailRecoveryProjectReportissuedbytheLouisianaTechnologyCouncilonJuly6th2009forirregularities.Lastly,SunBlockwasdirectedtodocumentanyIToperationalissuesencounteredaspartofourefforts.Thisreportdocumentsourfindings.</p><p>1.1 Email Search SunBlockconductedathoroughreviewoftheMayorofNewOrleanssITinfrastructureandnumeroussystemswereidentifiedaspotentiallystoringMayorNaginsemaildata.Numerousservers,computersfromkeypersonnel,andBlackBerryhandheldswereforensicallyprocessedusingBitFlare,EnCase,andotherstandardcomputerforensicstools.Areviewofthesedevicesandsearchesforbackuptapesproducedvariousemailsbutdidnotprovideacomprehensivesourceofallmessages.Analternativesearchmethodologywasemployedtolocatemessages.AreciprocalmethodologywasutilizedthatreviewedemaildatafromallaccountsontheMayorsOfficeserversformessagesfrom/toMayorNagin.Usingthereciprocalmethodology,overtwomillionmessagesweresearched.UsingareviseddaterangeofJuly12008toDecember12008,thousandsofmessageswereidentifiedanddelivered.Basedonthesystemsinplace,itisunlikelythatanysignificantnumberofadditionalmessageswillbefound.</p><p>1.2 LTC Report SunBlockwastaskedtoreviewthefindingsandreportissuedbytheLouisianaTechnologyCouncil(LTC)onJuly6th,entitledEmailRecoveryProjectReport.Issue3.1:LTCdidnotidentifyalternativesourcesLTConlyidentifiedtwoserversandNaginsdesktopaspotentialsourcesofemail.BlackBerryhandhelds,computersforkeystaffandfileserverscontaininghundredsofthousandsofmessageswerenotexaminedpriortoreleasingtheirfindings.Inaddition,alternativesearchmethodologiesthatwouldhaveyieldedresultswerenotemployed.Issue3.2:StandardforensicprotocolswerenotutilizedAlthoughtheuseofdatarecoverysoftwarewasdocumented,properforensictechniquespreventingthecontaminationoftheforensicdatawerenotemployed.OurreviewoftheharddrivesprovidedbyLTCindicatethatitwasnotuntilJune212009,approximatelysevenweeksafterworkbegan,thatLTCoroneofitsaffiliatesutilizedarudimentarytooltocreateaforensiccopyoftheMNOMail01server.Industrystandardpracticeincludestakingprecautionstopreventmodificationofcomputerevidenceduringananalysis.Ifaforensiccopyofacomputerdriveistobemade,thecopyshould</p></li><li><p>SunBlockSystems EmailRecordsRetention Page4of27</p><p>bemadefirstandthentheanalysisconducted,nottoconductananalysisfirstandthenmakeaforensiccopy.Failuretofollowtheproperprocedurescancausethelossordestructionofpotentiallyrelevantinformationandcontaminationofanyforensicexaminations.Issue3.3:ResultsfromRecoverDatautilitywerenotproperlyanalyzedLTCprovidedtheresultsfromtheRecoverDatautilityinsupportoftheirconclusionthat22GBofdatawasdeletedfromthemailserverpriortotheirarrival.SunBlockstestingoftheRecoverDatautilityhasfoundanerrorintheprogramwhichcanincorrectlystatefilesizes.TheEDBfileinquestionwasextractedusingalternativetoolssuchasBitFlareandWindows.Acomparisonofthedigitalfingerprintsfortheseextractedfilesprovidedanexactmatch.ThiscorroboratesthefindingsoftheITdepartmentthatthatRecoverDatadidnotaccuratelyseeanyadditionaldatabeyondthe66GBavailable.</p><p>1.3 IT Operational Issues Alongwiththepreviouslydescribedtasks,SunBlockwasalsoaskedtodocumentanyoperationalissuesencounteredthatwouldbeofinterest.ThegoalwastoassistinupdatingpoliciesandproceduresforthemanagementandoperationsoftheemailserverplatformaswellasprovidetheCitywithbestpracticesrecommendations.Issue4.1:NouniformemailretentionpolicyexistsDuringinterviews,OfficeofTechnologypersonnelstatedthatnoemailpolicyorAcceptableUsePolicyexisted.FurtherresearchbySunBlocklocatedanemailpolicydatedMay13,2008ontheCitysiteat:http://www.cityofno.com/Portals/Portal98/Resources/EmailPolicies.pdfHowever,thispolicyisnotwidelyknown.Inorderforapolicytobeimplemented,itmustbeuniformlydisseminatedsothatemployeesunderstanditexists.Issue4.2:ConfusionregardingimplementationofpoliciesThereissignificantconfusionamongCitypersonnelregardingtheimplementationofpoliciesandproceduresprovidedforemailretention.Sinceemailserverspaceandbackupcapacityislimited,anITdirectiveexistswhichencouragesalluserstoroutinelycleartheirmailbox.Usersareoftensentreminderstoreducethestorageutilizationoftheirinbox.Thisdirectivefailstoprovideproperinstructionsonhowtosatisfyrequirementsoftheemailretentionpolicyreferencedabove,potentiallyleadingtoconfusionamongusers.Issue4.3:BackupproceduresdrivenbytapebudgetsWithoutaformalpolicy,theOfficeofTechnologyhasimplementedbackuprequirementsandproceduresonanadhocbasis.Budgetconsiderationsandprocurementissueshaveunderminedbackuppractices.Initially,backupdatawasretainedfora30dayperiod.Assystemsgrewanddatadoubledinsize,insteadofpurchasingadditionalbackupcapability,backupretentiontimeswerereducedbyhalftotwoweeks.</p></li><li><p>SunBlockSystems EmailRecordsRetention Page5of27</p><p>Issue4.4:UtilizingsystembackupsforemailarchivingEmaildatacanbetransientinnature.Sincetraditionalsystembackupsonlyoperateonadailybasis,theyarenotabletokeepupwiththetransientnatureofemail.Withnightlysystembackups,amessagethatisreceived,readanddeletedonthesamedaywillnotbeproperlybackedup.Atrueemailarchivingsolutionswasnotutilized.Manysuchsolutionsexistinthecommercialmarketplaceandcanaddresstheneedsofpublicrecordsemailretentionbycapturingallincomingandoutgoingemailstoanoffsitearchive.Theyallowforsecurepreservationofemailsandauserisunabletodeletetheirmessagesfromthearchive.Thesesolutionsarerelativelyinexpensiveandshouldbeutilized.</p></li><li><p>SunBlockSystems EmailRecordsRetention Page6of27</p><p>2 Email Search InconductingtheemailsearchforemailstoandfromMayorC.RayNagin,SunBlockbeganwithitsstandardeDiscoveryprotocolsuccessfullyusedinsimilarmatters.Thisprotocolbeginswithanoverallreviewofallsystemsandoperationalproceduresinplacewithoutinitiallyfocusingonanyspecificsourcesofemail.Thiscomprehensiveapproachprovidesabroadunderstandingofallpotentialsourcesofdata,allowingformorereliableidentificationofallpossiblesourcesofresponsiveemails.</p><p>2.1 Email Sources Identified TheSunBlockeDiscoveryprotocolbeganbyprovidingCitytechnicianswithamultipagesurveyconsistingof24questionsregardingsystemsandoperationalproceduresinplaceattheMayorsOfficeofTechnology.ThesurveywasfollowedupwithphoneinterviewsandfacetofacemeetingsattheCitysOfficeofTechnology,theCityAttorneysOffice,andMayorNagin.Basedonthedataprovided,itbecameclearthattheOfficeofTechnologyadministerstwodifferentemailsystems:oneforallCityemployees,andonededicatedtotheMayorsofficeandpersonnel.ItwasdeterminedthatSunBlockwouldconfineitssearchtosystemsdedicatedtotheMayorsofficeandpersonnel.WithexclusivefocusonthesystemsfromtheMayorsoffice,numeroussourcesofemaildatawereidentified.Thesourcesincludedserverbasedsystems,DesktopandLaptopcomputers,BlackBerryhandhelddevices,andbackuptapes.</p><p>2.1.1 Server based systems Basedonthereview,numerouscomputerserverswereidentifiedthatcouldpotentiallyhavetracesofemaildata.Thesesystemsstoreand/ortransferemailaspartoftheiroperationalrolesorhavehademaildataplacedontheminthepastasaresultofaspecificproject.Theseserversincludethefollowingmachines:</p><p>ServerFunction StorageSize(GB)</p><p>ServerAlias</p><p>InitialMayoremailserver 136 MNOMail01NewerMayoremailserver 686 Mail2ProductionBlackBerryServer</p><p>60 Blackberryv</p><p>RetiredBlackBerryServer 68 BlackberrypFileserver 514 File1DomainControllers 164 DC1,DC2,DC02pITLabServer 137 ITLabRetiredSPAMServer 34 SPAMBackupServer 68 BackupRetiredFileserver 68 OldFileTotal 1,931 </p></li><li><p>SunBlockSystems EmailRecordsRetention Page7of27</p><p> Note:Duetopossiblepublicreleaseofthisdocumentandrelatedsecurityrisks,actualmachinenameshavebeenreplacedwithaliasesfordevicesnotpreviouslydisclosed.</p><p>2.1.2 Desktop and Laptop computers Aspartofthesearch,desktopandlaptopcomputerswhichmayhavestoredemailsofinterestwereidentified.ThesedevicesareofinterestsincetheymaystoreemailtoorfromtheMayoreitherinvisiblefilesorpreviouslydeleteddataavailablethroughaforensicreview.BasedonourinterviewswithITpersonnel,theMayor,andhisstaff,itwasdeterminedthatonlytheMayorandhisassistantPatriciaSmithhadaccesstoemailintheMayorsaccount.ThissecuritysettingwasverifiedbySunBlockduringthereview.Inaddition,itwasdeterminedthatMichaelLaFrancefromtheITdepartmentstoredcopiesoftheMayorandotheruseremailsonhiscomputers.Thisisduetohisoperationalresponsibilitiessupportingbackupsandprocessingofpublicrecordsrequests.Inall,thefollowingmachineswereidentifiedandsecured:</p><p> MayorNagindesktop MayorNaginlaptop PatriciaSmithdesktop PatriciaSmitholddesktop MichaelLaFrancedesktop MichaelLaFrancelaptop MichaelLaFranceolddesktop</p><p>Inadditiontotheabove,asearchforoldmachinesfromexistingemployeesorthoseofdepartedemployeeswhomayhaveaccessedtheMayorsemailwasconducted.Thefollowingmachineswerenotavailableastheyarebelievedtohavebeenrepurposedordestroyed.</p><p> MichaelBevinsdesktop WayneGatlindesktop MayorNaginolddesktop</p><p>2.1.3 BlackBerry hand held devices TheMayorisaheavyuserofhisBlackBerryhandheld.Assuch,hiscurrentandpreviousBlackBerrydevicesareexpectedtopotentiallycontainemailsofinterest.Duringthereview,threedifferentdevicesreportedlyusedbyNaginwerelocatedandsecured.DataonthesedevicesarereportedtogoasfarbackasNovember20,2003.ThefollowingBlackBerryhandheldswerelocated:</p></li><li><p>SunBlockSystems EmailRecordsRetention Page8of27</p><p>BlackBerry8830 BlackBerry9530 BlackBerry9000</p><p>2.1.4 Backup tapes Areviewofthebackupproceduresproducedalackofpertinenttapes.TheoperationalproceduresinplacebytheOfficeofTechnologyonlyretaineddataforatwoweekperiod.Anybackuptapesolderthantwoweekswereoverwritten.AssuchtheoldestsetofbackuptapesretainedwasmadeinFebruary2009.Inaddition,noformalemailarchivingsystemwasinplace.Suchashortretentionperiodisunusual.Thisappearstobeduetoprocurementissuesandhumanerror.HistoricaldocumentationprovidedbyCibernotedthetwoweekretentionperiodasfarbackasJune2007.InJanuary2008,problemswithpurchasingmoretapesthreatenedtoreducetheretentionperiodtooneweek.NewtapesarrivedinMay2008.Inadditiontoprocurementissues,asdisclosedduringaninterviewwithITpersonnel,acitycontractorinchargeofmaintainingconsistenttapebackupsforvarioussystemsincludingtheemailserverfailedtoproperlyperformthisduty.ThecontractorisnolongeremployedbytheCity.OffsitestorageiscurrentlybeingprovidedbyIronMountainbutthisprocedurewasimplementedinMarch2009andnobackupsetsfromtheperiodofinterestwereeverstoredthere.AtSunBlocksrequest,aprocurementrecordsreviewwasconductedbytheOfficeofTechnologysContractingManagertoverifythatnootheroffsitestoragevendorsareutilizedbytheITdepartmentfortheCity.Whilethereviewdidnotidentifyanyknownbackupsetsfromthedesiredtimeframe,SunBlocksstandardprotocolincludesaphysicalinspectionofthedatacenterandrelatedareas.ThisinspectionisdesignedtoassistITpersonnelinpotentiallyidentifyingdatasourceswhichmayhavebeenoverlookedduringprevioussearches.Giventhe</p></li><li><p>SunBlockSystems EmailRecordsRetention Page9of27</p><p>physicallayoutoftheCitysdatacenter,theinspectionencompassedmanyofficesuitesaswellasaninspectionunderthedatacenterflooring.</p><p>DataCenterInspectionSamplePhotograph1</p><p>DataCenterInspectionSamplePhotograph2</p></li><li><p>SunBlockSystems EmailRecordsRetention Page10of27</p><p>DataCenterInspectionSamplePhotograph3</p></li><li><p>SunBlockSystems EmailRecordsRetention Page11of27</p><p>DataCenterInspectionSamplePhotograph4</p><p>DataCenterInspectionSamplePhotograph5</p></li><li><p>SunBlockSystems EmailRecordsRetention Page12of27</p><p>Aspartoftheinspection,additionalbackuptapeswereidentified.Threedistinctsetsoftapeswerefound.Whilethecontentsofthesetapeswerenotcataloged,theyarenotbelievedtocontainemaildatafromthespecifiedperiodofinterestduetothetypesoftapemediafound.However,basedontheoldertapemediausedandphysicallabeling,oneormoresetsmaycontainemaildatafromaperiodpriortothetimeframeoftherequested.AttherequestofSunBlock,44membersoftheCityHallMISstaffparticipatedinasearchfortapesencompassingtheirworkspace,homeandotherpossiblestoragelocationsundertheircontrolwithnofurth...</p></li></ul>