1 pug challenge americas 2013 click to edit master title style pug challenge americas 2013 –...

1 PUG Challenge Americas 2013

Click to edit Master title style

PUG Challenge Americas

2013 – Westford, MA

Tales from the Audit Trails

Presented by: Mike Furgal & John Harlow


Introductions

• Mike Furgal– Progress employee from 1989-2012– Director of the Database Group at BravePoint since

2012– Progress OpenEdge Database Expert– [email protected]

• John Harlow– Progress developer since 1984– Primarily focused on database and virtualization – President of BravePoint– [email protected]


About BravePoint

• Largest OpenEdge consulting organization• ABL developers• QAD implementation and support• Application design and modernization• Business intelligence and discovery• Pro2 replication products• Managed services

– OE Database– QAD – Entire IT organizations


Disclaimer

• The techniques covered in this presentation are directed at the users and administrators of OE applications.

• Out of scope topics include:– Establishing Trusted User Identity – Adding Auditing Events to an application


Why Auditing?

• Regulatory requirements…– SEC (Sarbanes Oxley)– FDA (CFR 21 Part 11)– HIPAA– Immigration (I-9s)– And more

• Security• Peace of mind


Auditing Options

• “Roll your own” ABL based solutions• OpenEdge Auditing

• Let’s do a Poll:– Who has an audit system?– Who’s audit system is written in the ABL?– Who’s already using OE Auditing?– Who wishes they had auditing on their OE

application?


“Roll your own” Solutions

– Typically use Replication or Database Triggers– Pros

• Program in the ABL• Very fine control

– Cons• May require access to application source code• Defeatable/insecure• High overhead• Complications (SQL access for example.)• Triggers are client based


OpenEdge Auditing

• Integrated in the product since 10.1A– No additional $ cost

• Built into the Database Engine– Low overhead– Supports both ABL and SQL transactions

• Secure and tamper proof• Tools for archiving and reporting• Unimpacted by ABL code constructs

– ie: DISABLE TRIGGERS


OpenEdge Auditing

• Comprehensive Auditing Solution– Table based Auditing– Field Based Auditing– Allows capture of Before and After versions of the

data• Auditing includes Security so there is no tampering

with data• Some reporting capabilities built into the product


What does OE Auditing track?

• Changes to...– The Database Schema– The Application Data– Application Defined Events– The Security (New users, deleted users)– Database Encryption– The Audit Policies


Where is the data stored?

• Auditing adds tables to your production DB– Audit trail data is stored in:

• _aud-audit-data• _aud-audit-data-value

– Audit policy and controls are stored in:• _aud-audit-policy• _aud-event• _aud-event-policy• _aud-field-policy• _aud-file-policy


Cust-Num^F4^F71^F^GName^F1^Fpocket billiards co.^F^GAddress^F1^F44 Saunders Ave.^F^GAddress2^F1^F^F^GCity^F1^FPhelan^F^GState^F1^Fca^F^GCountry^F1^FUSA^F^GPhone^F1^F(818) 666-4063^F^GContact^F1^FLeon Aida^F^GSales-Rep^F1^FKIK^F^GComments^F1^F^F^GCredit-Limit^F5^F5000^F^GBalance^F5^F0^F^GTerms^F1^FNet30^F^GDiscount^F4^F50^F^GPostal-Code^F1^F92371^F"

What is in the _Event-detail field?

• It can be a little…

• Or a lot….

"Credit-Limit^F5^F12345^F22222"


Enable Audit on the Database

• Enable auditing (with indexes inactive!)

• Shutdown and backup database• Add storage areas for audit data and indexes


Define Audit Policies

• Use Audit Policy Maintenance Tool in GUI OpenEdge


Determining Auditing Requirements

• What type of things do you want to audit?• How long do you keep your audit data?• What performance impact can you tolerate?• How does auditing fit into your disaster recovery

plan?• Who can access or manage your audit rules and

data?• What type of reporting/inquiry is required?


Roles and Responsibilities

• The auditing system requires an administrator– Best practice for this position has it as separate

person from the DBA• The administrator designates who can:

– Administer the audit rules– Report on audit data– Archive or delete audit data

• Requires use of OE security


The Audit Data

• The audit tables have 27 indexes defined– Best practice is to have most turned off in production– We’ll see why in a moment

• Effective reporting requires that these indexes be active.

• Having the audit data in an archive database allows this without negatively impacting production

• OpenEdge provides tools to securely move data into the archive database.


Overhead of Auditing

• Customer example: QAD System– Real world audit policies– Inventory Load (36.15.1&.2)

• Tested 5 scenarios:– Baseline without auditing– Auditing w/indexes inactive– Auditing w/indexes active– Auditing w/indexes inactive using value table– Auditing w/indexes active using value table


BI Logging

QAD No Auditing QAD Audit No-Index QAD Audit w/Index QAD Audit/Value no Index

QAD Audit/Value w/Index

0

10

20

30

40

50

60

BI Mb Logged

BI Logged


Record Creates & Locks

QAD No Auditing QAD Audit No-Index QAD Audit w/Index QAD Audit/Value no Index

QAD Audit/Value w/Index0

50000

100000

150000

200000

250000

300000

350000

400000

450000

Creates and Record Locks

CreatesLocks


Overhead of Auditing on DB Size

• Keeping audit history in production may not seem significant

5gb

15gb

5 GB database


Overhead of Auditing on space

25gb

75gb

25 GB Database

25gb

2

25 GB of Data

• As the database grows Audit data takes up an immense amount of space

• Archiving out audit data keeps the database at an appropriate size

- This will generate a lot of BI/AI activity


Archive Commands

• To periodically archive data out of production ready to load into the archive database

• To load the exported audit data into the archive database


Reporting from Audit Data

• Things to consider:– Do you report off production?– Do you report off archive– Do you report from both?

• Timeliness of the incident being analyzed– Can alerts be set up?– This is a “roll your own” activity


OE Reporting Options


DEMO

• Start with a Sports database• Enable Auditing• Use Audit Policy Maintenance Tool

– Audit all tables – Audit changes to customer max-credit

• Create a workload on db• Polling process watching for changes to max-credit

in _aud-audit-data• Examine the generated audit log


DEMO


Caveats

• Make sure you use deactivateidx on the production database audit tables– Spelling counts!!!! (ask Mike)– If you rebuild all indexes all 27 audit indexes become

active!!!• Index rebuild causes worse performance

• You need to retune both BI and AI– Think about OE Replication and AI size impacts

• Don’t forget your audit policies during a dump/load!• Managing 2 databases (production and archive)

– Backups– Space management– Recovery Plans


Conclusions

• OE Auditing is a powerful tool• Knowing what needs to be audited may not be

obvious• Use an iterative approach

– Audit less than you need to start– Ramp as needed

• Make sure you have a solid data management plan

• Report and alert as needed


Thank You!

Questions?

1 pug challenge americas 2013 click to edit master title style pug challenge americas 2013 –...

Documents

data auditing

application slide

auditing field based

policy slide

auditing events

auditing options

triggers slide

basic audit data