masking algorithms guide€¦ · the vendor can then identify accounts that need attention without...

MaskingAlgorithmsGuideNovember2015

DelphixDataMaskingOverview

Revision:13October2016

Youcanfindthemostup-to-datetechnicaldocumentationat:

http://www.delphix.com/support

TheDelphixWebsitealsoprovidesthelatestproductupdates.

Ifyouhavecommentsaboutthisdocumentation,submityourfeedbackto:

[email protected].

TableofContents

MANAGINGALGORITHMSETTINGS...........................................................................................5ALGORITHMSETTINGSTAB.................................................................................................................5

ADDINGNEWMASKINGENGINEALGORITHMS.........................................................................6PROCEDUREFORADDINGANALGORITHM.............................................................................................7

CHOOSINGANALGORITHMTYPE..............................................................................................8SECURELOOKUPALGORITHM.............................................................................................................8SEGMENTEDMAPPING......................................................................................................................8

IgnoringorPreservingSpecificValues.....................................................................................8MAPPINGALGORITHM......................................................................................................................9BINARYLOOKUPALGORITHM..............................................................................................................9TOKENIZATIONALGORITHM................................................................................................................9MINMAXALGORITHM....................................................................................................................10DATACLEANSINGALGORITHM..........................................................................................................10FREETEXTREDACTIONALGORITHM...................................................................................................10

ADDINGASECURELOOKUPALGORITHM................................................................................11SEGMENTEDMAPPINGALGORITHM.......................................................................................13

SEGMENTEDMAPPINGEXAMPLE.......................................................................................................13Todefinesegments:..............................................................................................................14SegmentedMappingProcedure............................................................................................15

MAPPINGALGORITHM...........................................................................................................18

BINARYLOOKUPALGORITHM.................................................................................................20TOKENIZATIONALGORITHM...................................................................................................20

CREATINGATOKENIZATIONALGORITHM.............................................................................................21CREATEADOMAIN.........................................................................................................................22CREATEATOKENIZATIONENVIRONMENT............................................................................................23CREATEANDEXECUTEATOKENIZATIONJOB........................................................................................24STEPSTORE-IDENTIFYMASKEDINFORMATION....................................................................................26

MINMAXALGORITHM............................................................................................................27PROCEDURE...................................................................................................................................27

DATACLEANSINGALGORITHM...............................................................................................28PROCEDURE...................................................................................................................................28

FREETEXTALGORITHM...........................................................................................................29FREETEXTREDACTIONEXAMPLE.......................................................................................................30

Thisdocumentprovidesthestepsrequiredtosetupagilemaskingalgorithms.

ManagingAlgorithmSettings

Anintegralpartofthedatamaskingprocessistousealgorithmstomaskeachdataelement.Youspecifywhichalgorithmtouseoneachindividualdataelement(domain)ontheMaskingEngine’stab.There,youdefineauniquedomainforeachelementandthenassociatetheclassificationandalgorithmyouwanttouseforeachdomain.UsetheAlgorithmsettingstabtocreateordeletealgorithms.

AlgorithmSettingsTab

WithintheSettingspage,theAlgorithmtabdisplaystheName,Type,andaDescriptionofeachalgorithmcurrentlyavailabletoyou.Onthistab,youwillseethedefaultalgorithmsandanyadditionalalgorithmsyouhavedefined.Thisisalsowhereyoucancreatenewalgorithms.

Note: Allalgorithmvaluesarestoredencrypted.Thesevaluesareonlydecryptedduringthemaskingprocess.

Figure 1 Algorithm Settings Tab

AddingNewMaskingEngineAlgorithms

Ifnoneofthedefaultalgorithmsmeetyourneeds,youcancreateanewalgorithmdirectlyontheAlgorithmtaboftheSettingspage.Then,youcanimmediatelypropagateit.AnyoneinyourorganizationwhohastheMaskingEnginecanthenaccesstheinformation.

Note:User-definedalgorithmscanbeaccessedbyallusersandupdatedbytheuserwhocreatedthealgorithm.System-definedalgorithmscanonlybeupdatedbyadministrators.

ProcedureforAddinganAlgorithm

1. Intheupperright-handcorneroftheAlgorithmsettingstab,clickAddAlgorithm.

Figure2SelectAlgorithmTypePopup

2. Chooseoneofthefollowingalgorithmtypes.Foruseexamplesofwhenyoumightwanttouseeachofthesealgorithmtypes,seethesectionChoosinganAlgorithmTypebelow.

• SecureLookupAlgorithm

• SegmentedMappingAlgorithm

• MappingAlgorithm

• BinaryLookupAlgorithm

• TokenizationAlgorithm

• MinMaxAlgorithm

• DataCleansingAlgorithm

• FreeTextRedactionAlgorithm

3. Completetheformtotherighttonameanddescribeyournewalgorithm.

4. ClickSave.

ChoosinganAlgorithmType

TheDelphixMaskingEngineoffers35individualalgorithmsfromwhichtochoose,soyoucanmaskdataaccordingtoyourspecificneeds.Eachalgorithmisbuiltusingoneofeightframeworks,oralgorithmtypes.Thedescriptionsbelowwillhelpyouselectwhichalgorithmtypeisappropriateforthewaythatyouwanttomaskdata.Theyappearinorderoftheirpopularity.

SecureLookupAlgorithm

Securelookupisthemostcommonlyusedtypeofalgorithm.Itiseasytogenerateandworkswithdifferentlanguages.Whenthisalgorithmreplacesreal,sensitivedatawithfictionaldata,itispossiblethatitwillcreaterepeatingdatapatterns,knownas“collisions.”Forexample,thenames“Tom”and“Peter”couldbothbemaskedas“Matt.”Becausenamesandaddressesnaturallyrecurinrealdata,thismimicsanactualdataset.However,ifyouwantthemaskingenginetomaskalldataintouniqueoutputs,youshouldusesegmentedmapping,describedbelow.

SegmentedMapping

Segmentedmappingproducesnooverlapsorrepetitionsinthemaskeddata.Youcanmaskuptoamaximumof36valuesusingsegmentedmapping.Youmightusethismethodifyouneedcolumnswithuniquevalues,suchasSocialSecurityNumbers,primarykeycolumns,orforeignkeycolumns.Youcansetthealgorithmtoproducealphanumericresults(lettersandnumbers)oronlynumbers.

IgnoringorPreservingSpecificValuesinSegmentedMapping

Withsegmentedmapping,youcansetthealgorithmtoignorespecificcharacters.Forexample,youcanchoosetoignoredashes[-]sothatthesameSocialSecurityNumberwillbeidentifiednomatterhowitisformatted.

Youcanalsopreservecertainvalues.Forexample,toincreasetherandomnessofmaskedvalues,youcanpreserveasinglenumbersuchas5whereveritoccurs.Or,ifyouwanttoleavesomeinformationunmasked,suchasthelastfourdigitsofSocialSecuritynumbers,youcanpreservethatinformation.

MappingAlgorithm

Amappingalgorithmallowsyoutostatewhatvalueswillreplacetheoriginaldata.Therewillbenocollisionsinthemaskeddata,becauseitalwaysmatchesthesameinputtothesameoutput.Forexample“David”willalwaysbecome“Ragu,”and“Melissa”willalwaysbecome“Jasmine.”Thealgorithmcheckswhetheraninputhasalreadybeenmapped;ifso,thealgorithmchangesthedatatoitsdesignatedoutput.

Youcanuseamappingalgorithmonanysetofvalues,ofanylength,butyoumustknowhowmanyvaluesyouplantomask.

NOTE:Whenyouuseamappingalgorithm,youcannotmaskmorethanonetableatatime.Youmustmasktablesserially.

BinaryLookupAlgorithm

Abinarylookupalgorithmreplacesobjectsthatappearinobjectcolumns.Forexample,ifabankhasanobjectcolumnthatstoresimagesofchecks,youcanuseabinarylookupalgorithmtomaskthoseimages.TheDelphixEnginecannotchangedatawithinimagesthemselves,suchasthenamesonX-raysordriver’slicenses.However,youcanreplaceallsuchimageswithanew,fictionalimage.Thisfictionalimageisprovidedbytheowneroftheoriginaldata.

TokenizationAlgorithm

Atokenizationalgorithmistheonlytypeofalgorithmthatallowsyoutoreverseitsmasking.Forexample,youcanuseatokenizationalgorithmtomaskdatabeforeyousendittoanexternalvendorforanalysis.Thevendorcanthenidentifyaccountsthatneedattentionwithouthavinganyaccesstotheoriginal,sensitivedata.Onceyouhavethevendor’sfeedback,youcanreversethemaskingandtakeactionontheappropriateaccounts.

Likemapping,atokenizationalgorithmcreatesauniquetokenforeachinputsuchas“David”or“Melissa.”TheDelphixEnginestoresboththetokenandtheoriginalsothatyoucanreversemaskinglater.

MinMaxAlgorithm

Valuesthatareextremelyhighorlowincertaincategoriesallowviewerstoinfersomeone’sidentity,eveniftheirnamehasbeenmasked.Forexample,asalaryof$1suggestsacompany’sCEO,andsomeagerangessuggesthigherinsurancerisk.Youcanuseaminmaxalgorithmtomoveallvaluesofthiskindintothemidrange.

DataCleansingAlgorithm

Adatacleansingalgorithmdoesnotperformanymasking.Instead,itstandardizesvariedspellings,misspellings,andabbreviationsforthesamename.Forexample,“Ariz,”“Az,”and“Arizona”canallbecleansedto“AZ.”

FreeTextRedactionAlgorithm

Afreetextredactionalgorithmhelpsyouremovesensitivedatathatappearsinfree-textcolumnssuchas“Notes.”Thistypeofalgorithmrequiressomeexpertisetouse,becauseyoumustsetittorecognizesensitivedatawithinablockoftext.

Onechallengeisthatindividualwordsmightnotbesensitiveontheirown,buttogethertheycanbe.Thealgorithmusesprofilersetstodeterminewhatinformationitneedstomask.Youcandecidewhichexpressionsthealgorithmusestosearchformaterialsuchasaddresses.Forexample,youcansetthealgorithmtolookfor“St,”“Cir,”“Blvd,”andotherwordsthatsuggestanaddress.Youcanalsousepatternmatchingtoidentifypotentiallysensitiveinformation.Forexample,anumberthattakestheform123-45-6789islikelytobeaSocialSecurityNumber.

Youcanuseafreetextredactionalgorithmtoshoworhideinformationbydisplayingeithera“blacklist”ora“whitelist.”

Blacklist–Designatedmaterialwillberedacted(removed).Forexample,youcansetablacklisttohidepatientnamesandaddresses.Theblacklistfeaturewillmatchthedatainthelookupfiletotheinputfile.

Whitelist–ONLYdesignatedmaterialwillbevisible.Forexample,ifadrugcompanywantstoassesshowoftenaparticulardrugisbeingprescribed,youcanuseawhitelistsothatonlythenameofthedrugwillappearinthenotes.Thewhitelistfeatureenablesyoutomaskdatausingboththelookupfileandaprofileset.

AddingaSecureLookupAlgorithm1. Intheupperright-handcorneroftheAlgorithmtab,clickAddAlgorithm.

2. ChooseSecureLookupAlgorithm.

TheCreateSLRulepaneappears.

Figure3CreateSecureLookupRulePane

3. EnteraRuleName.Thisnamemustbeunique.

4. EnteraDescription.

5. SelectaLookupFile.

Thisfileisasinglelistofvalues.Itdoesnotrequireaheader.Ensurethattherearenospacesorreturnsattheendofthelastlineinthefile.

Thefollowingissamplefilecontent:

SmallvilleClarkvilleFarmvilleTownvilleCitynameCitytownTowneaster

TheDelphixMaskingEngineonlysupportslookupfilessavedinASCIIorUTF-8format.Ifthelookupfilecontainsforeignalphabetcharacters,youmustsavethefileinUTF-8formatfortheMaskingEnginetoreadtheUnicodetextcorrectly.

Whenyouarefinished,clickSave.

Beforeyoucanusethealgorithmbyspecifyingitinaprofilingormaskingjob,youmustaddittoadomain.

AddingaNewDomain

1. AtthetopoftheDomainstab,clickAddDomain. Anewdomainwillbecreatedin-line.

2. EnterthenewDomainName.ThedomainnameyouspecifywillappearasamenuoptionontheInventoryscreenelsewhere.Domainnamesmustbeunique.

3. SelecttheClassification–forexample,customer-facingdata,employeedata,orcompanydata.

4. SelectadefaultMaskingAlgorithmforthenewdomain.

5. Forinformationaboutalgorithmsettings,seeManagingAlgorithmSettings.

6. ClickSave.

SegmentedMappingAlgorithm

Segmentedmappingalgorithmsletyoucreateuniquemaskedvaluesbydividingatargetvalueintoseparatesegmentsandmaskingeachsegmentindividually.Optionally,youcanpreservethesemanticallyrichpartofavaluewhileprovidingauniquevaluefortheremainder.Thisisespeciallyusefulforprimarykeysorcolumnsthatneedtobeuniquebecausetheyarepartofauniqueindex.

NOTE:Whenusingsegmentedmappingalgorithmsforprimaryandforeignkeys,youmustusethesamesegmentedmappingalgorithmforeachkeytomakesuretheymatch.

SegmentedMappingExample

Whenmaskinganaccountnumber,youcanseparateitintosegments,preservingsomesegmentsandreplacingothers.Forexample,withtheaccountnumberNM831026-04:

NMisaplancodenumberthatyouwanttopreserve,alwaysatwo-characteralphanumericcode.

831026istheuniquelyidentifiableaccountnumber.Toensurethatyoudonotinadvertentlycreateactualaccountnumbers,youcanreplacethefirsttwodigitswithasequencethatneverappearsinyouraccountnumbersinthatlocation.Forexample,youcanreplacethefirsttwodigitswith98because98isneverusedasthefirsttwodigitsofanaccountnumber.Todothat,youwanttosplitthesesixdigitsintotwosegments.

-04isalocationcode.Youwanttopreservethehyphenandyoucanreplacethetwodigitswithanumberwithinarange–inthiscase,arangeof1to77.

Todefinesegments:

1. ForNumberofSegments,select3.Remember,youdoNOTcountthesegment(s)youwanttopreserve.

2. Preservethefirsttwocharacters(“NM”inthesamplevalue).UnderPreserveOriginalValues:

a. ForStartingposition,enter1.

b. Forlength,enter2.

3. Definethenexttwo-digitsegment(“83”inthesamplevalue)toalwaysbe98or99:

a. ForSegment1,selectType>Numeric.

b. SelectLength>2.

c. ForMaskValuesRange#,enter98,99.

4. Definethenextfour-digitsegment(“1026”insamplevalue):


b. SelectLength>4.

c. Leaverangefieldsempty.

d. ClickAddtotherightofPreserveOriginalValues.

5. Preservethehyphen:

a. ForStartingposition,enter9.

b. Forlength,enter1.

6. Definethelasttwo-digitsegment(“04”insamplevalue):


b. SelectLength>2.

c. ForMaskValuesMin#,enter1.

d. ForMaskValuesMax#,enter77.

Usingthisalgorithm,thesamplevalueNM831026-04mightbemaskedtoNM981291-77.

SegmentedMappingProcedure

Toaddasegmentedmappingalgorithm:

1. Intheupperright-handcorneroftheAlgorithmtab,clickAddAlgorithm.

2. SelectSegmentedMappingAlgorithm.

TheCreateSegmentMappingpaneappears.

Figure4CreateSegmentMappingPane

3. EnteraRuleName.


5. FromtheNumberofSegmentsdropdown,selecthowmanysegmentsyouwanttomask.DoNOTcountthevaluesyouwanttopreserve.Theminimumnumberofsegmentsis2;themaximumis9.

Aboxappearsforeachsegment.

6. Foreachsegment,selecttheTypeofsegmentfromthedropdown:NumericorAlpha-Numeric.

IMPORTANT:“Numericsegments”aremaskedaswholesegments.“Alphanumericsegments”aremaskedbyindividualcharacter.

7. Foreachsegment,choosetheLengthofthesegment(numberofcharacters)fromthedropdown(maximumis4).

8. Optionally,foreachsegment,specifyrangevalues.(Youmightneedtospecifyrangevaluestosatisfyparticularapplicationrequirements,forexample.)

YoucanspecifyrangesforRealValuesandMaskValues.WithRealValuesranges,youcanspecifyallthepossiblerealvaluestomaptotherangesofmaskedvalues.AnyvaluesnotlistedintheRealValuesrangeswouldthenmasktothemselves.

Note: Specifyingrangevaluesisoptional.Ifyouneeduniquevalues(forexamplemaskingauniquekeycolumn)youmustleavetherangevaluesblank.Ifyouplantocertifyyourdata,youmustspecifyrangevalues.

• Numericsegmenttype:

- Min#—Anumber;thefirstvalueintherange.(Valuecanbe1digitoruptothelengthofthesegment.Forexample,fora3-digitsegment,youcanspecify1,2,or3digits.Acceptablecharacters:0-9.)

- Max#—Anumber;thelastvalueintherange.(Valueshouldbethesamelengthasthesegment.Forexample,fora3-digitsegment,youshouldspecify3digits.Acceptablecharacters:0-9.)

- Range#—Arangeofnumbers;separatevaluesinthisfieldwithacomma(,).(Valueshouldbethesamelengthasthesegment.Forexample,fora3-digitsegment,youshouldspecify3digits.Acceptablecharacters:0-9.)

Ifyoudonotspecifyarange,theMaskingEngineusesthefullrange.Forexample,fora4-digitsegment,theMaskingEngineuses0-9999.

• Alpha-Numericsegmenttype:

- Min#—Anumberfrom0to9;thefirstvalueintherange.

- Max#—Anumberfrom0to9;thelastvalueintherange.

- MinChar—AletterfromAtoZ;thefirstvalueintherange.

- MaxChar—AletterfromAtoZ;thelastvalueintherange.

- Range#—Arangeofalphanumericcharacters;separatevaluesinthisfieldwithacomma(,).Individualvaluescanbeanumberfrom0to9oranuppercaseletterfromAtoZ.(Forexample,B,C,J,K,Y,ZorAB,DE.)

Ifyoudonotspecifyarange,theMaskingEngineusesthefullrange(A-Z,0-9).Ifyoudonotknowtheformatoftheinput,leavetherangefieldsempty.Ifyouknowtheformatoftheinput(forexample,alwaysalphanumericfollowedbynumeric),youcanenterrangevaluessuchasA2andS9.

Note: Whendetermininganumericoralphanumericrange,rememberthatanarrowrangewilllikelygenerateduplicatevalues,whichwillcauseyourjobtofail.

10. Toignorespecificcharacters,enteroneormorecharactersintheIgnoreCharacterListbox.Separatevalueswithacomma.

11. Toignorethecommacharacter(,),selecttheIgnorecomma(,)checkbox.

12. Toignorecontrolcharacters,selectAddControlCharacters.

TheAddControlCharacterswindowappears.

Figure5AddControlCharactersWindow

13. Selecttheindividualcontrolcharactersthatyouwouldliketoignore,orclickSelectAllorSelectNone.

14. Whenyouarefinished,clickSave.YouarereturnedtotheSegmentedMappingpane.

13. PreserveOriginalValuesbyenteringStartingpositionandlengthvalues.(Positionstartsat1.)

Forexample,topreservethesecond,third,andfourthvalues,enterStartingposition2andlength3.

Ifyouneedadditionalvaluefields,clickAdd.

14. Whenyouarefinished,clickSave.

15. Beforeyoucanusethealgorithmbyspecifyingitinaprofilingormaskingjob,youmustaddittoadomain.IfyouarenotusingtheProfilertocreateyourinventory,youdonotneedtoassociatethealgorithmwithadomain.SeeAddingNewDomainselsewhereinDelphixdocumentation.

MappingAlgorithmAmappingalgorithmsequentiallymapsoriginaldatavaluestomaskedvaluesthatarepre-populatedtoalookuptablethroughtheMaskingEngineuserinterface.Withthemap-pingalgorithm,youmustsupplyatminimum,thesamenumberofvaluesasthenumberofuniquevaluesyouaremasking,moreisacceptable.Forexampleifthereare10000uniquevaluesinthecolumnyouaremaskingyoumustgivethemappingalgorithmatleast10000values.

Toaddamappingalgorithm:

1. Intheupperright-handcorneroftheAlgorithmtab,clickAddAlgorithm.

2. ChooseMappingAlgorithm.

TheCreateMappingRulepaneappears.

Figure6CreateMappingRulePane

3. EnteraRuleName.ThisnameMUSTbeunique.


5. SpecifyaLookupFile(*.txt).

ThevaluefilemusthaveNOheader.Makesuretherearenospacesorreturnsattheendofthelastlineinthefile.

Thefollowingissamplefilecontent(noticethere’snoheaderandonlyalistofvalues):

SmallvilleClarkvilleFarmvilleTownvilleCitynameCitytownTowneaster

6. Toignorespecificcharacters,enteroneormorecharactersintheIgnoreCharacterListbox.Separatevalueswithacomma.

Toignorethecommacharacter(,),selecttheIgnorecomma(,)checkbox.

7. Whenyouarefinished,clickSave.

8. Beforeyoucanusethealgorithmbyspecifyingitinaprofilingormaskingjob,youmustaddittoadomain.IfyouarenotusingtheProfilertocreateyourinventory,youdonotneedtoassociatethealgorithmwithadomain.SeeAddingNewDomainselsewhereinDelphixdocumentation.

BinaryLookupAlgorithmABinaryLookupAlgorithmismuchliketheSecureLookupAlgorithm,butisusedwhenen-tirefilesarestoredinaspecificcolumn.Thisisusefulformaskingbinarycolumns(e.g.blob,image,varbinary,etc).

Toaddabinarylookupalgorithm:

1. ClickAddAlgorithmatthetoprightoftheAlgorithmtab.2. ChooseBinaryLookupAlgorithm.

TheCreateBinarySLRulepaneappears.

3. EnteraRuleName.4. EnteraDescription.5. SelectaBinaryLookupFileonyourfilesystem.6. ClickSave.

TokenizationAlgorithmTokenizationusesreversiblealgorithmssothatthedatacanbereturnedtoitsoriginalstate.Actualdata,suchasnamesandaddresses,areconvertedintotokensthathavesimilarpropertiestotheoriginaldata(text,length,etc)butnolongerconveyanymeaning.

HereisasnapshotofthedatabeforeandafterTokenizationtogiveyouanideaofwhatitwilllooklike.

BeforeTokenization

AfterTokenization

CreatingaTokenizationAlgorithm

1. FromtheHomepage,clickSettings.

2. ClickAddAlgorithm.Youwillseethepopupbelow:

3. SelectTokenizationAlgorithm.4. Enteranameanddescription.5. ClickSave.

CreateaDomainAfteryouhavecreatedanalgorithm,youmustassociateitwithadomain.

1. FromtheHomepage,clickSettings.2. SelectDomains.3. ClickAddDomain.Youwillseethepopupbelow:

4. Enteradomainnameandassociateitwithyouralgorithm.

CreateaTokenizationEnvironment1. FromtheHomepage,clicktheEnvironmentstab.2. ClickAddEnvironment.Youwillseethepopupbelow:

3. SelectTokenize/Re-Identifyasthepurpose.

4. ClickSave.Note:Thisenvironmentwillbeusedtore-identifyyourdatawhenre-quired.

Atthispoint,youcanproceedinthesamefashionasifyouwereusingDelphixtoperformnormalmasking.YouhavemadeallthechangesneededtouseTokenization(reversible)algorithmsinsteadofMasking(irreversible)algorithms.Noteitispossibletocreatetwodif-ferentenvironmentsforthesameapplication–oneformaskingandonefortokenization.

CreateandExecuteaTokenizationJob1. FromtheHomepage,clickEnvironments.2. ClickTokenize. 3. SetupaTokenizejobusingtokenizationmethod.Executethejob.

4. Youwillbepromptedforthefollowinginformation:

a. JobName—Afree-formnameforthejobyouarecreating.b. TokenizationMethod—SelectTokenizationMethod.c. MultiTenant—Checkboxifthejobisforamulti-tenantdatabase.d. RuleSet—Selectarulesetthatthisjobwillexecuteagainst.e. Generatorf. No.ofStreams—Thenumberofparallelstreamstousewhenrunningthe

jobs.Forexample,youcanselecttwostreamstoruntwotablesintherulesetconcurrentlyinthejobinsteadofonetableatatime.(ThisoptiononlyappearsifyouselectDMsuiteastheGenerator.)

g. RemoteServer—(optional)Theremoteserverthatwillexecutethejobs.Thisoptionletsyouchoosetoexecutejobsonaremoteserver,ratherthanonthelocalDelphixinstance.Note:Thisisanadd-onfeatureforDelphixStandardEdition.(ThisoptiononlyappearsifyouselectDMsuiteastheGenerator.)

h. MinMemory(MB)—(optional)Minimumamountofmemorytoallocateforthejob,inmegabytes.(ThisoptiononlyappearsifyouselectDMsuiteastheGenerator.)

i. Maxmemory(MB)—(optional)Maximumamountofmemorytoallocateforthejob,inmegabytes.(ThisoptiononlyappearsifyouselectDMsuiteastheGenerator.)

j. CommitSize—(optional)Thenumberofrowstoprocessbeforeissuingacommittothedatabase.

k. FeedbackSize—(optional)Thenumberofrowstoprocessbeforewritingamessagetothelogs.Setthisparametertotheappropriatelevelofdetailrequiredformonitoringyourjob.Forexample,ifyousetthisnumbersig-nificantlyhigherthantheactualnumberofrowsinajob,theprogressforthatjobwillonlyshow0or100%.

l. DisableConstraint—(optional)Whethertoautomaticallydisabledatabaseconstraints.Thedefaultisforthischeckboxtobeclearandthereforenotperformautomaticdisablingofconstraints.Formoreinformationaboutdatabaseconstraints,seeEnablingandDisablingDatabaseConstraints.

m. BatchUpdate—(optional)Enableordisableuseofabatchforupdates.Ajob'sstatementscaneitherbeexecutedindividually,orcanbeputinabatchfileandexecutedatonce,whichisfaster.

n. Truncate—(optional)Whethertotruncatetargettablesbeforeloadingthemwithdata.Ifthisboxisselected,thetableswillbe"cleared"beforetheoperation.Ifthisboxisclear,dataisappendedtotables,whichpoten-tiallycancauseprimarykeyviolations.Thisboxisclearbydefault.

o. DisableTrigger—(optional)Whethertoautomaticallydisabledatabasetriggers.Thedefaultisforthischeckboxtobeclearandthereforenotper-formautomaticdisablingoftriggers.

p. DropIndex—(optional)Whethertoautomaticallydropindexesoncol-umnswhicharebeingmaskedandautomaticallyre-createtheindexwhenthemaskingjobiscompleted.Thedefaultisforthischeckboxtobeclearandthereforenotperformautomaticdroppingofindexes.

q. Prescript—(optional)SpecifythefullpathnameofafilecontainingSQLstatementstoberunbeforethejobstarts,orclickBrowsetospecifyafile.Ifyouareeditingthejobandaprescriptfileisalreadyspecified,youcanclicktheDeletebuttontoremovethefile.(TheDeletebuttononlyappearsifaprescriptfilewasalreadyspecified.)Forinformationaboutcreatingyourownprescriptfiles,seeCreatingSQLStatementstoRunBeforeandAfterJobs.

r. Postscript—(optional)SpecifythefullpathnameofafilecontainingSQLstatementstoberunafterthejobfinishes,orclickBrowsetospecifyafile.Ifyouareeditingthejobandapostscriptfileisalreadyspecified,youcanclicktheDeletebuttontoremovethefile.(TheDeletebuttononlyappearsifapostscriptfilewasalreadyspecified.)Forinformationaboutcreatingyourownpostscriptfiles,seeCreatingSQLStatementstoRunBeforeandAfterJobs.

s. Comments—(optional)Addcommentsrelatedtothisprovisioningjob.t. Email—(optional)Adde-mailaddress(es)towhichtosendstatusmessag-

es.5. Whenyouarefinished,clickSave.

StepstoRe-IdentifyMaskedInformationUsetheTokenize/Re-Identifyenvironment.

1. FromtheHomepage,clickEnvironments.2. ClickRe-Identify. 3. Createare-Identifyjobandexecute.

MinMaxAlgorithm

Thisalgorithmallowsyoutomakesureallthevaluesinthedatabasearewithinaspecifiedrange.

Procedure1. EnteranAlgorithmName.2. EnteraDescription.3. EnterMinvalueandMaxvalue.

Forexample,ifyouwantallagestobemaskedtosomething18yearsoldoryounger,enterMinValue0andMaxValue18.

4. ClickOutofrangeReplacementValue.If“OutofrangeReplacementvalue”ischecked,themaskingenginewilluseadefaultvaluewhenincannotevaluatetheinput.

5. ClickSave.

DataCleansingAlgorithmIfthetargetdataneedstobeputinastandardformatpriortomasking,youcanusethesealgorithms.

Procedure1. EnterAlgorithmName.2. EnteraDescription.3. SelectLookupfilelocation.4. Bydefault,delimiterseparatingvaluesisanequalssign(=).Ifyouprefer,youcan

changethistoanothersymbol,suchasanasterisk(*).5. ClickSave.

Thefollowingissamplefilecontent.Itdoesnotrequireaheader.Makesuretherearenospacesorreturnsattheendofthelastlineinthefile.

NYC=NYNYCity=NYNewYork=NYManhattan=NY

FreeTextAlgorithmThissectionprovidesanoverviewofhowtocreatefree-textredactionalgorithms.Formorein-depthinformation,seetheseparateguideonthistopic.

Youwillbepromptedforthefollowinginformation:

1. EnteranAlgorithmName.2. EnteraDescription.3. SelecttheBlackListorWhiteListradiobutton.4. Selectlookupfileandenterredactionvalue

OR/AND5. SelectProfilerSetfromdropdownandenterredactionvalue6. ClickSave

FreeTextRedactionExample1. Createinputfile.

a. Createinputfileusingnotepad.Enterthefollowingtext:“ThecustomerBobJonesissatisfiedwiththetermsofthesalesagreement.Pleasecalltoconfirmat718-223-7896.”

b. Savefileastxt.

2. Createalookupfile.

a. UsenotepadtocreateatxtfileandsavethefileasaTXT.Besuretohitre-turnaftereachfield.Thelookupflatfilecontainsthefollowingdata:

BobJones Agreement

3. Createthealgorithm.Youwillbepromptedforthefollowinginformation:

a. Enterthealgorithmname:Blacklist_Test1.

b. Enteradescription–BlacklistTest

c. SelecttheBlackListradiobutton

d. Selectthelookupfileandenterredactionvalue“XXXX.”

e. ClickSave.

4. CreateaRuleset:

a. FromtheEnvironmentspage,clicktheRuleSettab.

b. ClickCreateRuleSet.Youwillbepromptedforthefollowinginformation:

i. EnterRuleSetName:Free_Text_RS

ii. FromtheConnectorModedrop-downmenu,selectFreeText.

iii. SelecttheInputFilebyclickingtheboxnexttoyourinputfile.

iv. ClickSave.

5. Createthemaskingjob.Whenyoudo:

a. UsetheFree_TextRuleSetthatyoucreatedinstep4.OntheEnvironmentspage,clickMask.

b. ExecuteMaskingjob.

i. OntheEnvironmentspage,clickthenameofthejob.

ii. ClickAction.

Theresultsofthemaskingjobwillshowthefollowing:

RedactedInputFile:Thecustomerxxxxxxxxissatisfiedwiththetermsofthesalesxxxx.Pleasecalltoconfirmat718-223-7896.

Thewords“Bob,”“Jones,”and“agreement”areredacted.

masking algorithms guide€¦ · the vendor can then identify accounts that need attention without...

Documents