1. The audit survival heuristics of an FDA regulated exploratory testing team CAST August 8th, 2011 1Griffin Jones Congruent Compliance LLC 2011

2. Preliminaries Who is in the room? My goal: Stimulate your interest to study the subject more Leave with a heuristic to help you organize and present with confidence your ET results to regulatory auditors Have a conversation and try to meet your needs Quick Preview The context The heuristic and how to apply it Some of the traps about ET in a regulated industry CAST August 8th, 2011 2Griffin Jones Congruent Compliance LLC 2011 3. Assumptions and Terms This is a living presentation Based on my experiences of auditing and being audited More reference information here than I will present Follow the for the key points Much of this can be adapted to other contexts i.e., not FDA regulated, Exploratory Testing Schools of Testing by Bret Pettichord Analytic , Standard, Quality, Context-Driven, Agile Exploratory Testing: Simultaneous learning, test design and test execution CAST August 8th, 2011 3Griffin Jones Congruent Compliance LLC 2011 4. Terms Congruence Being balanced between inner feelings & outer actions Smells Symptom that possibly indicates a deeper problem 5 Whys Questions-asking method to investigate root causes Mary had a little lamb heuristic Emphasize each of the individual words in a statement Checking: confirming existing beliefs; versus: Testing - finding new information (Michael Bolton) CAST August 8th, 2011 4Griffin Jones Congruent Compliance LLC 2011 5. The Problem Lets assume that you are FDA regulated and trying to do compliant context driven, Exploratory Testing You likely have these concerns about passing an audit: Evidence is not sufficient Documentation is not sufficient Process control is not sufficient Cant clearly explain what you do and why Auditors value different things than you, and speak a different language CAST August 8th, 2011 5Griffin Jones Congruent Compliance LLC 2011 6. Fast Takeaway The regulator is not your business partner The regulator has police powers Let the Wookie win Auditors are likely of the Quality (gatekeepers) or Routine (traceability matrix) testing school model You are Context Driven testing school. Deal with it. Auditors think testing is demonstration and checking Dont try and convert them. Deal with it. CAST August 8th, 2011 6Griffin Jones Congruent Compliance LLC 2011 7. Spoiler The regulations are not the problem How you are coping with the regulations is the problem Give the Auditors what they want: Clear traceable requirements and description of risks Description and demonstration of control Clear objective evidence The ability to understand their concerns, speak their language, and explain how you are compliant Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 7Griffin Jones Congruent Compliance LLC 2011 8. Not going to talk about The Fear, Uncertainly, and Doubt swirling in the field Vendor/Experts: You should be scared, but I have Silver Bullets and Big Magic so trust me and just buy my wares. By the way, .. Persistent Myths IMO the regulators frown on ET ( I dont sell it). The Typical Regulatory Affairs Presentation CAST August 8th, 2011 8Griffin Jones Congruent Compliance LLC 2011 9. Regulatory Overview Regulations For the public good - because people died Regulators FDA regulates >25% of the Gross Domestic Product Regulatory Auditors Police Powers Industry Auditors Assessors and valued advisors to management Audits CAST August 8th, 2011 9 Details Griffin Jones Congruent Compliance LLC 2011 10. Audit Survival Heuristics CHCMWCE Chocolate Mousse Congruent Honest Competent Model (Appropriate) Willing Control Evidence CAST August 8th, 2011 15 Model Competent Honest Evidence Control Willing Congruent Griffin Jones Congruent Compliance LLC 2011 11. Lets take a journey CAST August 8th, 2011 16 Practice Congruent Theory Less Stressful Audits Griffin Jones Congruent Compliance LLC 2011 12. The Congruence Triad Congruence is when you are balanced between inner feelings and outer actions The Congruence Triad Self, Other, Context Being congruent is a process A way of communicating with yourself and others Incongruence is when part of the triad is missing Placating, Blaming, Super-rational, or Irrelevant? What is missing and fill it in: Self, Others, Context CAST August 8th, 2011 17 Other Context Self Details Griffin Jones Congruent Compliance LLC 2011 13. The Theory Mountains Dishonest Incompetent Inadequate CAST August 8th, 2011 21 Honest Competent Appropriate Model Self-Incriminating Experts and Heroes Over-Constrained Griffin Jones Congruent Compliance LLC 2011 14. Honest Integrity, Truthful, Trust, Sincerity in: You and your organization Words, actions, and documents Smells Dishonest Self-incrimination Dont create even the appearance of a problem Tests How do you and the organization react to criticism? Are you a learning organization? (5 Why) CAST August 8th, 2011 22Griffin Jones Congruent Compliance LLC 2011 15. Competent Are you and your organization: Capable, credible, understands context, speaks the language; trained in the industry, technology, and regulatory obligations Smells Incompetent Experts and heroes Tests Do you believe you are capable of doing good work? (5 Why) CAST August 8th, 2011 23Griffin Jones Congruent Compliance LLC 2011 16. Appropriate Model Is the process model: Complete, reasonable, practical, logical, explainable Smells Inadequate model Over-constrained model Test: What problem is this model solving? How will it Fail? What is required in this model? Missing? Do you believe this model is sufficient? (5 Why) CAST August 8th, 2011 24Griffin Jones Congruent Compliance LLC 2011 17. The Practice Mountains Unwilling Out-of-Control No Evidence CAST August 8th, 2011 25 Excessive or Wasteful Micro-Management Obsessive-Compulsive Willing Under Control Evidence Griffin Jones Congruent Compliance LLC 2011 18. Willing Motivated, focused, prioritized, committed, resourced, staffed, supported, given attention, nurtured Smells Unwilling Excessive or Wasteful Test Do people care? (5 Why) Is there sufficient resources for the work and expectations? (5 Why) CAST August 8th, 2011 26Griffin Jones Congruent Compliance LLC 2011 19. Under Control Explain what you are doing and why. Are you living it? Coherently explain your: configuration control and authorization traceability and accountable organization, preparation, planning, independent review, prevention, correction, checking and testing Smells Out-of control Micro-managed Tests Is the type and level of controls appropriate? (5 Why) CAST August 8th, 2011 27Griffin Jones Congruent Compliance LLC 2011 20. Evidence Auditable evidence: Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions. Smells No-evidence Obsessive-compulsive evidence Tests Explain why the specific evidence meets the criteria. (5 Why) CAST August 8th, 2011 28Griffin Jones Congruent Compliance LLC 2011 21. How do you apply this? Application is as simple as: CAST August 8th, 2011 29 Remembering to ask the questions. Follow the energy of the answers. Fix the base, first. Griffin Jones Congruent Compliance LLC 2011 22. During an Audit Choosing a regulatory posture Manageable issues (within reason) Evidence Controls Willingness (resources and priority) Unmanageable issues Broken process model Lack of competence Broken trust Incongruence CAST August 8th, 2011 30Griffin Jones Congruent Compliance LLC 2011 23. More Fast Takeaways The FDA is open to agile processes and realizes that the current approach to software validation is not working At the same time, companies are more concerned about: the business risk that the FDA would not accept the agile process, than the product or project risk that is associated with waterfall type development Find the middle option for your context CAST August 8th, 2011 31Griffin Jones Congruent Compliance LLC 2011 24. Natural Evidence Periodically , take the observer point-of-view and ask: Is what I see and hear, about the theory and practice of what we do: acceptable from both a product qualification and regulatory compliance point of view? If yes, what is the most natural, efficient, and strongest evidence we could collect? Why not a video/audio recordings w/ paper summary? Is it being collected? If no, why not? (5 Why) organizational problem? CAST August 8th, 2011 32Griffin Jones Congruent Compliance LLC 2011 25. Organizational Smells Going Tilt Traps CAST August 8th, 2011 33Griffin Jones Congruent Compliance LLC 2011 26. Smells that lead to Paint the Village Visitors are coming. How shall we work today? The Best Practice Cargo Cult We dont really understand the details of what we do, why we do it, or how what we do works. But have faith. Testing Death Spiral Regulator does not care about testing and management might only care about regulatory compliance. Spiral. The Titanic The gigantic engineered process is perfect people are the source of problems, not solutions CAST August 8th, 2011 34Griffin Jones Congruent Compliance LLC 2011 27. Organizational Disasters Pathetic Compliance Following a regulatory compliant procedure in a way that does not solve the testing problem for which it was designed. Utopian Shelf-ware Procedures No one reads them. They are not reality. Close Enough I dont have to do it exactly. I know better. No one will notice or care. Read My Mind Because that is the only place where the evidence is. CAST August 8th, 2011 35Griffin Jones Congruent Compliance LLC 2011 28. Is the Auditor on Tilt? CAST August 8th, 2011 36 Maybe it is something we said or did, or are doing? History That you are unaware of, and it might be complicated Notches on the gun May be making a name for themselves Making an example of you May be constructing an example to deter others Griffin Jones Congruent Compliance LLC 2011 29. Classic ET Traps Implementation details identified as requirements Tighten and simplify your requirements Documentation lacks detail to support traceability Require less mind reading. Control is vague or assumed Summarize and document what control is for you CAST August 8th, 2011 37Griffin Jones Congruent Compliance LLC 2011 30. The BIG Trap Weak Evidence Clear, objective, retrieval, human readable, attributable, contemporary evidence that a third party can review or reconstruct (with minimal outside help); and quickly reach the same results and conclusions. Check it via Mary had a little lamb Collect it naturally Weak evidence is likely a symptom of other deeper issues Abundant, quality evidence mitigates your other problems CAST August 8th, 2011 38Griffin Jones Congruent Compliance LLC 2011 31. Audits can be Useful Candor can result in free consulting and insight Should you take the risk? Provides motivation management cares Provides actionable data The jiggle that is needed by the organization A counter-measure to low expectations & poor practices CAST August 8th, 2011 39 If you cant be a good example, you are going to be a stern warning. Griffin Jones Congruent Compliance LLC 2011 32. Recap of the Spoiler The regulations are not the problem. How you are coping with the regulations is the problem. Give the Auditors what they want: Clear traceable requirements and description of risks Description and demonstration of control Clear objective evidence The ability to understand their concerns, speak their language, and explain how you are compliant Abundant and quality evidence mitigates your other problems. CAST August 8th, 2011 40Griffin Jones Congruent Compliance LLC 2011 33. The Big Take Away Understand your regulatory context Work on your congruence Work each level of the model, ask the questions Document how you are under control Improve your evidence, collect it naturally Avoid the smells, disasters, and traps Summarize your regulatory story, practice explaining it Apply what you learn during the audit CAST August 8th, 2011 41 1 2 3 Griffin Jones Congruent Compliance LLC 2011 34. Questions? CAST August 8th, 2011 42 Model Competent Honest Evidence Control Willing Congruent Griffin Jones Congruent Compliance LLC 2011 35. Further Study - A FDA presentations and resources: Webinar with FDA's John Murray on Software Validation in the Field of Medical Devices Presentation: Preparing for an FDA Medical Device Sponsor Inspection Quality System Inspection Technique Inspection Guide General Principles of Software Validation; Final Guidance for Industry and FDA Staff CAST August 8th, 2011 43Griffin Jones Congruent Compliance LLC 2011 36. Further Study - B Regulatory Compliance The Art of Compliance: Turning Compliance into Sustainable Business Advantage by Robert Rhoades of Quintiles FDA inspections: How to Host an FDA Inspection by SGS Life Science Services Preparation for FDA Inspection by NEMA/ADVAMED/PHILIPS FDA Sponsor Inspections: How to Prepare and Survive by Medtronic, Inc CAST August 8th, 2011 44Griffin Jones Congruent Compliance LLC 2011 37. Further Study - C Audits The ASQ Auditing Handbook by J. P. Russell Congruence Beyond Blaming by Jean McLendon and Gerald M. Weinberg The Satir Model: Family Therapy and Beyond by Virginia M. Satir More Secrets of Consulting: The Consultant's Tool Kit by Gerald M. Weinberg CAST August 8th, 2011 45Griffin Jones Congruent Compliance LLC 2011 38. Further Study - D Agile and the FDA Business Risk (from the FDA) versus Product Risk http://blogs.construx.com/forums/t/432.aspx What is Exploratory Testing? And How it Differs from Scripted Testing by James Bach Coping With Complexity: Lessons From a Medical Device Project by Yaron Kottler Testers and Auditors Testers are like auditors by James Christie Evidence 21 CFR Part 11 Electronic Records by the FDA CAST August 8th, 2011 46Griffin Jones Congruent Compliance LLC 2011 39. Griffin Jones Congruent Compliance Griffin.Jones@CongruentCompliance.com Thank You! CAST August 8th, 2011 47Griffin Jones Congruent Compliance LLC 2011