taking your customers to the cleaners: historical patron data cleanup and routine purge preparation

Taking Your Customers to the Cleaners:

Historical Patron Data Cleanup and Routine Purge Preparation

Roy Zimmer

Western Michigan University

About 5 or 6 years ago…

No more SSN switch to using WIN

WIN is our Western Identification Number



Banner




Banner

New campus ID cards


A few less years ago…

Rewrote the patron update process to use Banner

A few less years ago…

Rewrote the patron update process to use Banner

Started thinking about not being SSN-based

2007-2008

The WIN had become available in the data feeds for our patron update.

Needed to change Institution ID

interim step: arbitrary 14-digits -> WIN

final step: WIN -> Bronco NetID

Patron update was switched from being SSN-based to WIN-based.

BroncoNetID is our single signon ID

Summer 2008 – What we started with

Have data for about 74,000 patrons.

About 183,000 barcodes (less than half are active!).

Summer 2008 – What we started with

Have data for about 74,000 patrons.

About 183,000 barcodes (less than half are active!).

Several thousand duplicate records,

one with SSN, one with WIN (in the SSAN field)

The older duplicate record typically had charges, amounts owed, etc.

2008: August – October

Most of my time was spent on the cleanup…

Dali

Patron duplicate detector – LB4020

foreign students

various errors

Sample follows…

August

(WINs & SSNs above are not real)

Sample output used one day

Our first run came up with 3489 duplicate patron records.

We created a program that used the LB4020 report as input to identify patron records that we wanted to alter – call it LB4020fix.

These records needed to be extracted from Voyager for modification and re-import.

Modify me with LB4020fix

Voyager has a patron extract utility, but it doesn’t extract all relevant data for a patron. We’d started using our own – patronsif.pl - years ago.


Voyager extract

(Pptrnextr)

Up to 3 patron-barcode + group combinations

Similarly limited number of addresses

WMU extract

(patronsif.pl)

Unlimited patron-barcode + group combinations

Unlimited number of addresses+

- +- → +


For the patron cleanup we incorporated patronsif.pl into LB4020fix.

Patron notes field problem:

CR+LF stored if user pressed the RETURN key

creates unwanted extra lines within a record

drop_crlf utility replaces “CR+LF” with “space+space”

LB4020fix reads the duplicate report (LB4020) and extracts patron sif format data for the duplicate records.

SIF-A

new WIN-based records

BroncoNetID in InstitutionID

change expiredate to 1981.01.01

SIF-B

old SSN-based records

change InstitutionID to current BroncoNetID

SIF-C


have the current update, expire, and purge dates and BroncoNetID

The heart of the cleanup process

SIF-A




SIF-B



SIF-C



update, key on SSN

purge on expiredate 1982.01.01

[remove new records]

1



SIF-A




SIF-B



SIF-C



update, key on SSN



update, key on SSN

[prep old records to be “new”]

1 2



SIF-A




SIF-B



SIF-C



update, key on SSN



update, key on SSN


update, key on InstID

[unify old records with new data]

1 2 3



SIF-A


have current BroncoNetID


SIF-B



SIF-C



update, key on SSN



update, key on SSN


update, key on InstID

[unify old records with new data]

1 2 3



This clean-up process, with variations, was repeated many times.

Details omitted here for the sake of brevity (and sanity).

Several things went awry along the way.

Not all records could be matched up with a WIN or SSN (as reported by LB4020), so those had to be handled by

assigning temporary SSNs, WINs, and/or Institution IDs.

Several things went awry along the way.

Not all records could be matched up with a WIN or SSN (as reported by LB4020), so those had to be handled by

assigning temporary SSNs, WINs, and/or Institution IDs.

At another point, the interim records used in the process weren’t deleted during a purge. Those had to be detected, reassigned an older expiration date (1971.01.01), and carefully purged before proceeding.

We now had 1081 duplicate patron records.

We added the expiration date to the duplicate detector, LB4020.

Now we could see that all the SSN-based records were expired, or about to be.



At this time we discovered that new WIN-based records were coming in as duplicates to SSN-based records that were typically set to expire 2008.09.08.




This had to change!




This had to change!

And the semester was about to start…

Yes, we did avert disaster. But we had more problems.

Early September…


The duplicate detection report, which had grown to 60 pages, was now down to 1.

The next day it had grown to 3 pages.

Early September…


The duplicate detection report, which had grown to 60 pages, was now down to 1.

The next day it had grown to 3 pages.

Some records not having all fields populated on the LB4020 duplicate detector caused problems.

Also had to fix duplicate records where the SSAN field was null.

Early September…

We removed several hundred obsolete records that had neither WIN nor SSN.

Discovered records that had no Institution ID – yet another problem.

Mid September…

We removed several hundred obsolete records that had neither WIN nor SSN.

Discovered records that had no Institution ID – yet another problem.

We are now down to 1 SSN-based record.

Mid September…

This person had our assigned WIN being the same as the SSN. Not supposed to happen!

Identified 15 more such instances and submitted them to I.T. for correction.

Found some more SSN-based records – don’t know why they still existed – and converted them to being WIN-based.

October…

Flipped the “switch” so that we no longer get SSNs for our patron update.

Still had records from our NOTIS era – pre Summer 1998

Purged them if they:

did not have life-time borrowing privileges

did not have an SSN recorded

did have an Institution ID

Legacy data

Trouble ahead…

3M SelfCheck

Trouble ahead…

Multiple Active Barcodes

will NOT work with SelfCheck!

3M SelfCheck

3M SelfCheck requires 1 active barcode per patron.

We had 11058 patrons with multiple active barcodes.



Wrote a program to whittle that down.

Got them reduced to 300, but the next day, it was up to 1777!





Under control now, with patrononeactive.pl, running Monday – Friday. This keeps only the most current active barcode for a patron.





Under control now, with patrononeactive.pl, running Monday – Friday. This keeps only the most current active barcode for a patron.

Forgot about those patron records without an Institution ID.Had 882 of them. Fixed them.

We looked at records created before 2008, those that had no SSN but did have an Institution ID.

Extracted these records, modified them:

expiredate = createdate

purgedate = expiredate + 4 years

Reimported these records. They should disappear with future annual patron purges.

An eye towards the future…

We still had 11,696 records with no SSN (nor WIN).

We expect most of these to be routinely purged in the future, leaving us with 456.

What we ended with

We still had 11,696 records with no SSN (nor WIN).

We expect most of these to be routinely purged in the future, leaving us with 456.

When we started, we had about 250,000 patron records.

We now have about 68,000.

Duplicate records are routinely dealt with.

We filter out all but the single most current active barcode for a patron.

We will have annual patron purges.

What we ended with

Know what you’re starting with.

Keep your goal in mind.

Figure out a good solution.

Be flexible.

Be ready for mistakes.

Watch out for new/current data undoing your changes.

Know when you’re done.

Worthwhile points…

patronsif.pl

drop_crlf

lb4020.pl

lb4020fix.pl

patrononeactive.pl

patrononactive.ksh

Contact me if you would like to get any of the above.

Resources

patronsif.pl as listed, gets patron data and puts it in patron SIFformat. institution ID based. gets all patron+barcodegroupings. (not site-specific)

drop_crlf shell script that contains this line:

perl -pi -e's/\r\n/ /g' $1

replaces CR+LF combination with two spaces.

(this is useful anytime you use patronsif.pl)

Some details on the resources…

lb4020.pl detects duplicate patron records.shows: name, expired (Y/N), SSAN, expire date,modify date, institution IDWMU-specific: indicates whether SSN or WIN in SSAN.modification required for your institution.

lb4020fix.pl control structure around patronsif.pl code that useslb4020.pl output as starting point for the fixing process.creates one or more patron SIF files for fixing data. use drop_crlf if necessary.


patrononeactive.plqueries Voyager, checking patrons’ active barcodes. if more than one is found, changes all but the most recentactive barcodes to other. check the code carefully as itmay need modification for your use.(incorporates patronsif.pl code)

patrononeactive.kshcombines patrononeactive.pl and drop_crlf in a scriptsuitable for cron use


taking your customers to the cleaners: historical patron data cleanup and routine purge preparation

Technology