1

IVT EU CSV 2010

Automate Your CSV ProcessAutomate Your CSV ProcessTemplates, Tools & TricksTemplates, Tools & TricksTemplates, Tools & TricksTemplates, Tools & Tricks

Tyson M. Mew, PresidentOfni Systems Inc.

About the Speaker

Ty MewTy Mew is President of Ofni Systems, Inc., a is President of Ofni Systems, Inc., a software provider and Part 11 consulting firm Hesoftware provider and Part 11 consulting firm Hesoftware provider and Part 11 consulting firm. He software provider and Part 11 consulting firm. He has a strong background in software development has a strong background in software development and computer system validation, and has spoken at and computer system validation, and has spoken at many conferences and workshops on Part 11. Ty many conferences and workshops on Part 11. Ty regularly conducts training sessions for regularly conducts training sessions for organizations to educate employees about the organizations to educate employees about the specific requirements of the rule and how tospecific requirements of the rule and how to

October 1, 2010 2

specific requirements of the rule and how to specific requirements of the rule and how to incorporate it into daily practices. He is an active incorporate it into daily practices. He is an active member of ISPE, PDA, SQA and the Triangle member of ISPE, PDA, SQA and the Triangle PEERS group.PEERS group.

2

Getting Started

Why CSV is well-suited to iautomation

Formalize your strategyPreparing templates for documents and certain types of systemsyp yDetermine and execute your testing strategy

3

Quick Poll

Are you primarily validating:• Commercial Off-The Shelf (COTS) without access

to code (black box testing)• Open systems with access to system code (white

box testing)Do you execute protocols electronically?y p y

Automating Computer System Validation

Software is Well-Situated to Automation

Use automated software tools to • Analyze Software• Detect Objects (Screens, Reports, Code)• Identify Missing Objects• Create test scripts

4

Formalizing Validation Strategy

Create a General Strategy for All Projects:• Identify Validation Tactics/Testing Approaches for

Common Elements– Data Inputs– Formulas, Calculations– Reports, Charts

fi i f ifi f• Define Strategies for Specific Software TypesValidation Strategy Documented in Validation Master Plan

Formalizing Validation Strategy

Unifies Validation ApproachEnsures Internal Agreement with Validation StrategyImproves Training of Validation Professionals Facilitates Review/Approval ProcessProvides chance to implement best practices for lean project management

5

Validation TemplatesTemplates Improve document consistency and

efficiency of document creationefficiency of document creationTemplates Include:• SOP identified Sections (Objectives, Scope, etc.)• Instructions for completing specific sections• Template text with variable names• Accepted Formatting

Hint: Fewer templates are better than more templates. Use variables to create constant text.

Automated Document Generation

Extension of TemplatesIncludes specific verbiage with variables.

Variables are replaced with specific data.Requirement lists automatically inserted into appropriate sections of validation documentsThi h k ll ith i ilThis approach works well with similar

programs with similar requirements (Groups of spreadsheets, etc.)

6

Example Variable NamesSystem NameProgram Version NumberProgram Version NumberFile Name, Data File Name, Security File Name, External Code Library NameFile LocationCompany Name, Company AddressOwning Department System OwnerOwning Department, System OwnerDocument Names, ID#s, Abbreviations, TitlesProtocol Names, Abbreviation

Determining Testing Strategy

D t i h t i t d thDetermine what you are going to do, then execute your planValidation Professionals test according to planQuality review/approve documents based on existing standards

7

Automate the Generation of CSV Documents

Breaking down systems into discrete lid t bl bj tvalidatable objects

Collecting testable requirements Integrating Risk Assessments into the processGenerating Design Specifications directly from

dsource codeAutomatic generation, tracking and updating of the Requirements Traceability Matrix

Breaking Down Systems intoDiscrete Validatable Objects

fi i i f hDefine Distinct Parts of the Program•Forms, Reports•Worksheets, Charts•Workflows

Automatically Create List of Validatable ObjectsNote: We use software to analyze programs, record logical order for forms and reports. Software also identifies missing objects

8

Collecting RequirementsDefine requirements during project walk-through

Additional requirements can be gathered from key end users, developers, system owners.

Note: GAMP 5 specifically allows combining/separating of specification documents to facilitate validationfacilitate validation.

Hint: Use software which allows you to record requirements as the validatable objects are recorded

Integrated Risk AssessmentValidation Resources are limited; a Risk-Based Validation approach is necessary to focus limitedValidation approach is necessary to focus limited resourcesRisk Assessment is performed at the requirement level• Functional Specification• Design Specification

A k D l /K S t U /S t OAsk Developers/Key System Users/System Owners where system risk is. They know and you usually get a better ROI than trying to analyze every requirement.

9

Define Testing for Each Object

Forms: Input, Processing, Output, SecurityReports: Data, Calculations, FormattingSpreadsheets: Data Entry, Calculations, Data Outputs, LabelsCharts: Data Ranges, Calculations, L b l /F ttiLabels/FormattingWorkflows: Process Flow

Generating Design Specifications

Once requirements are defined, use software t l t l bj t d d it bltools to analyze objects and produce suitable outputOnly applicable to open source code.

Hi t U ft i t t lHint: Use software scripts to analyze common objects and produce suitable content for the Design Specification.

10

Example of Generating Design Specifications: Data Input Form

Examples of Generating Design Specifications: Spreadsheet Calculations

11

Automatic Requirement Traceability

Several Programs exist which will t ti ll t th R i tautomatically generate the Requirement

Traceability MatrixUpdates if Requirements are Moved/EditedDisplays if Functional Requirements exists without corresponding Design specificationwithout corresponding Design specificationDisplays if Requirements exist which do not have a defined Test Case

12

Test Cases and Test Plans

Testing based on Risk AssessmentsInput Testing vs. Challenge Testing, Workflow TestingAutomating with micro-test casesAuto-creation of test scripts from your requirementsrequirementsCreate specific test cases for use with change control

How Much Is Enough?Automating Testing allows more time for Testing, Less Time spent on DocumentationLess Time spent on Documentation.Defining specific requirements based on Risk allows for use of different templates based on specific functional needs.Automated creation of Test Cases/Protocols allows validation resources to focus on high risk sections of

f i liprogram functionalityRequired business functionality should receive special attention.

13

Input vs. Challenge TestingHigh Risk Fields Should have Challenge Tests

Status Fields • Do not accept <NULL>• If Drop-Down List Used, Status not accept other values• Default Value should not be Pass, Success, etc.

Dates• Do not accept clearly incorrect dates• Do not accept unordered dates (Date of Test before Date

Received, etc.)

Workflow Testing

Extremely useful technique for systems with a d fi d kfl (S ldefined process or workflow (Sample Management, Manufacturing Process, etc.)• Verify that entered data can move through entire

workflow• Verify that fail-points prevent data from moving

forward• Verify that data cannot be accessed from

inappropriate parts of the workflow

14

Using Micro-Test CasesMicro-Test Cases are test cases for common requirements have pre written verbiage (withrequirements have pre-written verbiage (with variables) which is inserted into appropriate sections of the document.Parallel sections inserted into Requirement Documents, Design Documents and Testing DocumentsDocumentsSmaller test cases combined into larger test cases

Using Micro-Test CasesMicro-Test Cases most useful for common, repeated test stepstest steps.• Adding/Closing Forms or Reports• Data Entry into Fields• Screen Navigation• Printing, Generating Charts, Etc.• Confirmation of Formulas/CalculationsConfirmation of Formulas/Calculations• Any place in Test Protocol where similar operations are

repeated with similar resultsContent of Micro-Test case modified with variables

15

Enter Required Variables

Lines of Test Protocol Generated

16

Automatic Generation of Test Protocols

In certain systems, software can be used to automate the generation of entire test protocols based on defined inputsData Entry ScreensIndividual Spreadsheet WorksheetspSecurity Testing

Automatic Generation of Test Protocols

17

Test Cases for Change Control

Certain Test Cases Are Particularly Useful h if i f ti lit ft Chwhen verifying functionality after Change

Control• Regression Testing• Functionality Identified as High RiskWherever possible Test Cases should beWherever possible, Test Cases should be written mindful of reuse during Change Control

Protocol Execution, Documentation and Managing Deviations

Paper vs. Electronic executionCapturing actual results and screen shots for maximum effectivenessGenerating, processing and closing test and script deviations

18

Paper vs. Electronic Execution

Electronic Protocol ExecutionR l t diti l d t h i• Replaces traditional pen and paper techniques

• Just as word processors improve writing, electronic protocol execution improves paper executions

Documents can be exported and reviewed by traditional means (“Type-writer Rule”)

N El i P l S f bNote: Electronic Protocol Software must be compliant with 21 CFR 11, with audit trails, security, electronic signatures, etc.

Example of Electronic Protocol Execution

19

Documenting Protocol Execution with Screen Shots

As each step is completed, collect screen shotInsert Screen shots automatically into finished protocoly pConfigure Screen shots to include user name, date/time, etc. which further document testing

Hint: We use screen shot software, integrated with Electronic Protocol Execution software. This significantly reduces time organizing and numbering screen shots.

Warning!Screen shots do not “prove” a test case was successful, only

provides evidence that a test was performed.

Documenting Protocol Execution with Screen Shots

20

Automating Deviation Reporting

When a test step fails, deviation is t ti ll t dautomatically created

User identifies deviation type, based on this determination, some fields are automatically filled.Deviations can be viewed in Real-TimeDeviations can be viewed in Real-TimeDeviations can be resolved according to each companies procedures

21

Validation Case Study ReviewsEnsure compliance with 21 CFR 11• Technological• Procedural

Risk AssessmentGather Requirements• Screen Specific• Workflow Specific

October 1, 2010 41

• SecurityWrite Requirement DocumentsTesting Documents

Case StudiesSpreadsheets for CRO

A li d t l t d h t t id• Applied tools to spreadsheets to provide technological controls, including audit trails, user level security and electronic signatures

• Spreadsheet requirements generated from micro templates.

• Requirements verified through protocol executionq g pResult: Entire spreadsheet library quickly and

efficiently validated

22

Case StudiesEngineering system for Pharmaceutical Manufacture

R i t th d f k d d l• Requirements gathered from key users and developers• Software emphasized moving through exact workflows, so

testing plan modified to focus on workflows over data entry screens with associated reports

• Requirements verified through protocol execution

Result: Emphasized workflows over input/output model when appropriate, which improved testing and overall validation effort.

Case StudiesElectronic Document Capture System• Worked with developers to demonstrate compliance with 21 W p p

CFR 11• Program used Client/Server model, with digital pen inputs• Extensive use of virtual machines to model each section of

program – different OS, different sections• Tested specific form events• Requirements verified through protocol execution

R lt V lid ti ff t ll d i k d ffi i tResults: Validation effort allowed quick and efficient compliance with 21 CFR Part 11 regulations. After the validation process was complete, the validation results were reviewed and approved by a third party auditor

23

Case Studies

Purchasing System for Pharmaceutical M f tManufacturer• Requirements gathered from company SOP and

instructions• Worked with 3rd party vendor to add security• Met with both End users and developersp• Requirements verified through protocol execution

Result: Third party software made compliant with 21 CFR 11, with implemented Security requirements

Case Studies

Clinical Research Database from CRO• Clear requirements, no requirement drift• Used existing templates, validation document

generation and electronic protocol execution• Requirements verified through protocol execution

Result: Validation completed in 12 business days.

24

Live Examples

Demonstration of the concepts described in this presentation

Thank You!

Questions? Tyson M. Mew(919) 844-2494Ofni Systems Inc.Ty.Mew@OfniSystems.com

October 1, 2010 48

www.OfniSystems.comReferences and additional materials will be available at

www.OfniSystems.com at the conclusion of the conference.

25

AdditionalAdditional Reference M t i lMaterial

Solutions for Common Software

Web Based Applications Custom Databases (MS Access, FoxPro, SQL Server, Oracle, etc.)Spreadsheets (MS Excel, Quattro, Lotus, etc.)Connections to LIMS

In addition to system specific issues, all systems must demonstrate control of data, and meet any applicable business requirements.

26

Web Based Applications

Define System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/ComputationsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security RequirementsMake sure all functions are testedMake sure all functions are tested

Custom DatabasesAccess, FoxPro, SQL Server, Oracle, etc.Access, FoxPro, SQL Server, Oracle, etc.

Define System InterfacesDefine System InterfacesDefine System InterfacesDefine System Interfaces•• Input ScreensInput Screens•• Processing ScreensProcessing Screens•• Report OutputsReport OutputsDefine Key System Functions/ComputationsDefine Key System Functions/Computations

October 1, 2010 52

Define Key Business WorkflowsDefine Key Business WorkflowsDefine Security RequirementsDefine Security Requirements

27

Spreadsheets

Excel, Quattro, Lotus, etc.Define Data Input Cells/RangesDefine Spreadsheet Processing• Calculation Cells• Internal FunctionsDefine System Outputs/ChartsDefine Security Requirements

Connections to LIMS

Define Data Type and RequirementsDefine Data Workflow• Define Data Input to LIMS • Define Output from LIMS• Calculations are performed inside p

LIMS and exported• Calculations are performed outside

LIMS and re-imported

28

Gathering Requirements

Interviews with:• Key System End-Users• System Owner• Developer

Good communication between the validation teamGood communication between the validation team and all end users improves all aspects of the project and adds more value to the project.

Gathering RequirementsScreen Specific

I t t t• Input tests– Make sure that all fields accept appropriate

data entry– Use challenge tests where appropriate

• Processing– Make sure all functions (buttons, etc.) operate as

expected– Make sure all calculations produce correct results

29

Gathering RequirementsWorkflow Specific

F ll ifi d t t th h t th kfl• Follow specific data entry throughout the workflow.• Follow data transitions in workflows

Security• Define specific groups

fi ifi kfl il bl• Define specific workflows available• Define specific functionality• Test what users can and cannot do

Risk Assessment

Determine what parts of the software create the t i k d f t ti thmost risk and focus testing there.

Risk Assessment should be multi-disciplinary, and should include validation professionals, key end users, quality review, developers and system owner.system owner.Focus on known areas first