oklahoma department of human services … pdf library/mosaicqaplanv04_epmo_09302010.pdfqa/qc lead,...

QUALITY ASSURANCE PLAN

OKLAHOMA DEPARTMENT OF HUMAN SERVICES

ENTERPRISE SYSTEM (MOSAIC PROJECT)

MOSAIC Quality Assurance Plan v04.02

QUALITY ASSURANCE PLAN APPROVALS Prepared by: QA/QC Lead Approved by: Program Quality Manager

Quality Team Lead Program Manager MOSAIC Project Sponsor

MOSAIC Quality Assurance Plan v04.02 1

TABLE OF CONTENTS 1.0 OVERVIEW 4

1.1 General Information 4 1.2 Purpose 4 1.3 Scope 4 1.4 QA Team Roles and Responsibilities 5 1.5 Resources for QA Team 9

2.0 QUALITY ASSURANCE TASKS 10 2.1 QA Team Tasks and Matrix 10 2.2 Conducting QA Team Tasks 11 2.3 QA Schedule 11 2.4 Evaluate Planning Oversight 11 2.5 Evaluate Project Management 12 2.6 Evaluate Quality Management 16 2.7 Evaluate Training 17 2.8 Evaluate Requirements Management 18 2.9 Evaluate Operating Environment 21 2.10 Evaluate Development Environment 22 2.11 Evaluate Software Development 23 2.12 Evaluate System and Acceptance Testing 26 2.13 Evaluate Data Management 30 2.14 Evaluate Operations and Business Oversight 31 2.15 Evaluate Software Products Review Process 31 2.16 Evaluate Component Deliverable (Release) Process 32 2.17 Evaluate Media Certification, Storage and Handling Process 32 2.18 Evaluate Non-Deliverable Software Certification 33 2.19 Evaluate Performance Standards 33

3.0 PROJECT DELIVERABLES 34 3.1 QA Activities for Deliverables 34 3.2 Walkthrough for Deliverables 34

4.0 REVIEWS AND AUDITS 34 4.1 Verify Document and Artifact Deliverable Review 34 4.2 Verify Project Management Compliance Reviews 35 4.3 Conduct Process Audits and Reviews 36 4.4 Evaluation 37

5.0 TESTING PROCESS AND ENVIRONMENTS 38 5.1 System and User Acceptance Testing Processes 38 5.2 System Environments 39 5.3 Quality Control Testing Process 41

6.0 VALIDATION AND ACCEPTANCE PROCESS 41 6.1 Deliverable Acceptance 41 6.2 Contractor's System Certification 42 6.3 Acceptance Plans and Releases 42 6.4 Federal Acceptance and Approval Preparation 42 6.5 Food and Nutrition Service (FNS) Acceptance Requirements 42 6.6 Health and Human Services (HHS) Acceptance Requirements 43

2 MOSAIC Quality Assurance Plan v04.02

7.0 ISSUE REPORTING AND CORRECTIVE ACTION 44 7.1 Escalation Procedure for Resolution Disputes 44 7.2 Corrective Action Process 45 7.3 Recording Defects (Incidents) in Software Code or Documentation 46

8.0 QUALITY METRICS 47 8.1 Measurements 47 8.2 Monitor and Control 47 8.3 Trend Analysis 48 8.4 Process Improvement Analysis 48

APPENDIX A. QA TEAM TRAINING 50 APPENDIX B. QA TASKS MATRIXES 51 APPENDIX C. PROCESS AUDIT AND REVIEW SCHEDULE 57 APPENDIX D. COMPLIANCY REPORT 60 APPENDIX E. SOFTWARE TOOL EVALUATION CHECKLIST 61 APPENDIX F. PERFORMANCE STANDARDS EVALUATION CHECKLIST 62 APPENDIX G. PILOT EVALUATION CHECKLIST 63 APPENDIX H. IMPLEMENTATION EVALUATION CHECKLIST 64


1.0 OVERVIEW 1.1 General Information. This Quality Assurance Plan (QA Plan) presents a

framework for managing quality assurance activities, which when followed, will verify delivery of quality products and services for the Oklahoma Department of Human Services (OKDHS) Enterprise System (MOSAIC Project). The QA Plan will: 1.1.1 Provide Project standards and procedures to be used as the basis for

Quality Assurance (QA) Team reviews and audits. 1.1.2 Document how to plan, implement, and assess the effectiveness of QA

and Quality Control (QC) operations. 1.1.3 Outline and reference QA and QC procedures used to evaluate overall

Project and product performance, business and technical processes, Deliverables software, and traceability within all MOSAIC Project documentation.

1.1.4 Be implemented by the Project QA Team. Contractor will assist OKDHS staff throughout the lifecycle of the Project.

1.2 Purpose. Quality Assurance activities performed will verify that Project management and Project Deliverables are of high quality and meet quality standards as determined by the MOSAIC Project stakeholders. The purpose of the QA Plan is to: 1.2.1 Define Project QA organization. 1.2.2 Identify QA tasks and responsibilities. 1.2.3 Implement quality standards and objectives to provide defect-free

systems and business and technical processes that are delivered on time and within budget, and are maintainable.

1.2.4 Provide reference documents and guidelines to perform QA and QC activities.

1.2.5 Define standards, practices, and conventions used in carrying out QA and QC activities.

1.2.6 Provide tools, techniques, and methodologies to support QA and QC activities and reporting.

1.3 Scope. The QA Plan will establish QA and QC functions and processes for the lifecycle of the Project that conform with OKDHS approved QA standards and procedures and comply with state and federal laws and regulations. 1.3.1 Quality Management Scope will implement a Quality Management

methodology and process that will verify all activities necessary to design, develop, implement, and utilize a product or service are effective and efficient with respect to the Enterprise System and its performance.


1.3.2 Quality Assurance Scope will implement the Quality Assurance methodology and processes to verify that all the processes, methodologies, and technologies used for the Project and used to build and utilize the product perform as specified.

1.3.3 Quality Control Scope will verify and validate that Deliverables comply with quality standards and Project requirements, and are complete and correct. 1.3.3.1 QA and QC tools will be utilized to verify and track the

Project lifecycle activities, create and track associated documents for those activities, and verify completeness of product and that it met the requirements through QC testing.

1.3.4 The QA Plan will ensure: 1.3.4.1 Quality Control system development, evaluation and

acceptance standards are developed, documented, and followed throughout the Project lifecycle.

1.3.4.2 QA Team records QC measurement results for use in re-evaluating and analyzing quality standards and processes.

1.3.4.3 Results from software quality reviews and audits performed by the QA Team are submitted to QA/QC Lead to track conformance to predetermined software requirements and standards.

1.3.4.4 Test results adhere to predetermined acceptance standards. 1.3.4.5 Confirm Deliverable and milestone acceptance criteria are

met. 1.4 QA Team Roles and Responsibilities. The QA Team has the responsibility

to report directly to the OKDHS Program Manager if the quality of the Project or product is being jeopardized or compromised. The QA Team will follow the escalation process as defined in this plan. While in practice this rarely occurs, as almost all defects are correctly addressed at the project level, the fact that the QA Team can escalate above the project level gives it the ability to keep the majority of these defects at the project level. Figure 1 reflects the reporting structure for resolving QA issues. 1.4.1 Program Quality Subject Matter Experts (SMEs) report directly to the

QA/QC Lead, but they may interact directly with Project Teams, including Contractors. They have the authority to delegate responsibilities of interacting functions.

1.4.2 Program Quality Manager and Quality Team Lead will review the QA Team work and have the final approval. QA Team will identify compliance areas as either conforming or non-conforming with the QA standards, procedures, and guidelines set forth in the QA Plan with the goal of ensuring compliance with QA requirements.


Figure 1. Roles of QA Team Members Relative to the Project.

1.4.3 The roles of the QA Team members that influence and control Project quality include. 1.4.3.1 OKDHS Program Manager, responsible for:

Establishing a quality program by committing the Project to implement quality standards and methodologies.

Reviewing and approving the QA Plan.

Gaining approval from the Decision Team for implementation of the QA Plan.

Resolv ing and following-up on any quality issues escalated to this level by the QA Team.

Facilitating identification of a person or group independent from the Project to audit and report on the Project’s QA Team function.

Facilitating identification of the quality factors to be implemented in the system and software.


Facilitating identification of, developing, and maintaining planning documents, such as the Project Management Plan.

1.4.3.2 Program Quality Manager and Quality Team Lead, responsible for:

Implementing the quality program in accordance with the approved QA Plan which defines QA standards, procedures, and guidelines.

Reviewing and approving the QA activities to be performed by QA/QC Lead.

Reviewing and approving the QA Team’s work and recommendations during all phases of the program.

Resolv ing and following-up on any quality issues escalated to this level by the QA Team.

Identifying, developing and maintaining Quality Management planning documents such as the Risk Management Plan, Change Management Plan, and QA Plan.

1.4.3.3 Quality Assurance/Quality Control (QA/QC) Lead, responsible for:

Implementation and adherence to the quality program in accordance with the approved QA Plan which defines QA standards, procedures, and guidelines.

Identifying the QA activities to be performed by QA Team.

Reviewing and approving the QA Team’s work and recommendations during all phases of the program.

Resolving and following-up on any quality issues raised by QA Team.

Identifying, developing, and maintaining planning documents, such as Test Plans, Standards and Process Manuals (Quality Control Testing and Migration Approval), and this QA Plan.

Implementing practices, processes, and procedures as defined in OKDHS Enterprise System RFP, Project Plan, Business or Technical Requirements and Design Documents, SOW, and QA Plan.

1.4.3.4 QA Team, responsible for:

Reviewing and commenting on the QA Plan.


Implementing the quality program in accordance with this QA Plan.

Resolving and following-up on any quality issues raised by QA Team related to software development and hardware implementation activities.

Identifying and evaluating the quality factors to be implemented in the system (software and hardware).

Implementing practices, processes, and procedures as defined in OKDHS Enterprise System RFP, Project Plan, Business or Technical Requirements and Design Documents, SOW, and QA Plan.

1.4.3.5 Contractor Project Manager and Contractors, responsible for:


Implementing the quality program in accordance with this QA Plan.

Resolving and following-up on any quality issues raised by QA Team related to software design, application development, and hardware implementation.

Identifying and evaluating the quality factors to be implemented in the software and hardware.

Implementing the software design/development practices, processes, and procedures as defined in the Project Software Development Methodology and Development Requirements Document, Functional and Technical Design Documents, and QA Plan.

1.4.3.6 Analysis, Design, Development, & Test (ADDT) Teams, responsible for:


Implementing the quality program in accordance with QA Plan.

Identifying and evaluating the quality factors to be implemented in the software and hardware.

Implementing the software design/development practices, processes, and procedures as defined in the Project Software Development Methodology and


1.4.3.7 Program Quality SME(s) and Quality Control Testing (QCT) Supervisor, responsible for:


Implementing the quality program in accordance with QA Plan.

Resolving and following-up on any quality issues raised by QA Team related to Change Management and QCT Standards and Process Manual.

Ensuring the quality factors are implemented in QCT of software as related to Change Management.

Implementing the practices, processes, and procedures as defined in Project Change Management Plan, Standards and Process Manuals (Quality Control Testing and Migration Approval), and QA Plan.

1.4.3.8 Quality Control Testers, responsible for:


Implementation and adherence to the quality program in accordance with this QA Plan.

Resolving and following-up on any quality issues raised by QA Team related to software testing.

Verifying the quality factors are implemented in the system, specifically software.

Implementing the software test practices, processes, and procedures as defined in Quality Control Standards and Process Manual, and Software Development Methodology, and QA Plan.

1.5 Resources for QA Team 1.5.1 Facilities and Equipment. The QA Team will have the same access to

facilities and equipment as the MOSAIC Project Team. QA Team will have access to computer resources to perform QA functions such as process and product evaluations, audits, reviews, and quality control testing.

1.5.2 Testing Sites. Testing sites that will be established will have all the necessary tools and documentation available to perform QA and QC activities.

1.5.3 QA Team Members. The QA Team selected and assigned to the Project has demonstrated experience in quality assurance as it relates


to software engineering and software development and how it directly impacts the business units and their functions. The QA Team has a marked and explicit understanding of quality assurance and QA implications to the Project, Project Team, and how it directly relates and results in quality results for overall Project success.

1.5.4 Independent Verification & Validation (IV&V) Contractor. The IV&V Contractor will assist OKDHS by determining if development products conform to requirements and whether software satisfies the intended use and user needs. The determination includes assessment, analysis, evaluation, review, inspection, and testing of software products and processes. The IV&V Contractor will work with the QA Team and will follow procedures established or referenced in the QA Plan that will be used as the basis for managing the quality assurance of Project Deliverables.

1.5.5 Quality Control Testing Team. OKDHS will determine how and when QA/QC staff will be brought into the Project to perform quality control testing. This will also be defined for the Staffing Plan.

1.5.6 QA Team Training. To establish a QA Team that is experienced in quality assurance activities and to supplement their formal training and experience, they will be trained on OKDHS specific activities to achieve an effective Quality Program. As required, other Project Team members will be provided with training to support the QA activities. Any QA training requirements will be coordinated with the Project Team. Training may be conducted in several formats as specified in the Training Plan. The training schedule will be compatible with the Project schedule. Appendix A provides a matrix that identifies the required skills to perform QA tasks to implement the QA Plan.

2.0 QUALITY ASSURANCE TASKSThe scheduling of QA tasks is driven by the Project Plan. Each QA task is performed in relationship to the Project activities that are taking place. One or more QA tasks may be performed concurrently. A task is considered completed when the required report, such as QA Report or Compliancy Report, is satisfactorily completed or action items have been closed. The tasks identified in this section requiring coordination and cooperation with the Project Team will be performed by the QA Team as defined in the Enterprise System RFP, QA Plan, and Project Standards and Guidelines reference materials. 2.1 QA Team Tasks and Matrix. The QA/QC Lead will ensure tasks are

performed to verify the quality of the Deliverables on the Project. The QA Team will assist the QA/QC Lead in ensuring the QA procedures are executed in accordance with the QA Plan. 2.1.1 The QA/QC Lead will monitor Project staff activities and review

products for compliance to applicable standards and procedures by utilizing this QA Plan methodology. The results of QA Team


monitoring and analysis along with QA Team's recommendations for corrective action will be reported to Project Program Quality Manager and Quality Team Lead, and as required, to OKDHS Program Manager.

2.1.2 All documents and software approved by the OKDHS Program Manager for release to the business units must have been reviewed and approved by QA Team. The QA task matrix is presented in Appendix B.

2.2 Conducting QA Team Tasks. To conduct QA/QC activities identified in Section 2, checklists will be designed and used as a tool for conducting task audit/review/evaluation to verify required steps have been performed. The results of these evaluated tasks will be documented using the Compliancy Report reflected in Appendix D, or sample evaluation checklists in Appendices E, F, G, and H. All checklists will be reviewed with appropriate team members, and submitted to the QA/QC Lead for review. 2.2.1 Adding QA Team Tasks. Any QA/QC tasks not identified in the

planning stages of the Project can be added to the Quality Team's scheduled activities by contacting the Project Program Quality Manager or Quality Team Lead for approval.

2.2.2 In the tables describing QA Team Tasks, Q indicators in the Task # column are activities that will be preformed in addition to the IV&V RFP tasks required.

2.3 QA Schedule. QA schedules are closely coordinated with the Project Deliverables and product development. Process compliance audits will be performed at the beginning of each new phase of development to verify that the appropriate processes are correctly implemented as defined in the planning documents. Unscheduled audits will be made during each phase of development to verify that the processes and procedures are being followed. At the completion of a software development phase, QA Team will review and report whether all steps required to transition to the next phase have been accomplished.

2.4 Evaluate Planning Oversight. As stated in the OKDHS Enterprise System RFP, key staff within each business area serves on the Project Decision Team. OKDHS Program Manager will report to the Executive Sponsors, who report directly to the OKDHS Chief Executive Officer (CEO). The Project Decision Team will perform overall oversight of the Project and will receive input from an independent Project Manager and the Independent Verification and Validation (IV&V) Team.


TABLE 1 - PLANNING OVERSIGHT – QA TEAM TASKS

TASK ITEM TASK # TASK DESCRIPTION Contract Verification

PO-1 Evaluate and verify that the obligations of Contractor, Subcontractors, and external staff, including terms, conditions, statement of work, requirements, technical standards, performance standards, development milestones, acceptance criteria, and delivery dates, are clearly defined. QA Team will verify that performance metrics are included that allow tracking of Project performance and progress against the criteria set by OKDHS.

Feasibility Study

PO-2 Perform ongoing evaluations and reviews of OKDHS methodologies used for the Feasibility Study, verifying the Feasibility Study is objective, reasonable, measurable, repeatable, consistent, accurate, and verifiable.

PO-3 Review and evaluate PAPD(U)s and IAPD(U)s. PO-4 Review and evaluate the Cost Benefit Analysis.

2.5 Evaluate Project Management. The overall guidance and direction to the Project is provided by the OKDHS Program Manager. The facilitation of Project tasks and Deliverables will be organized in a Project work plan with recommended sequence and schedule. The QA Team will review the joint Project management efforts as needed and initiate corrective actions where appropriate to verify Project delays and defects are minimal.

TABLE 2 – PROJECT MANAGEMENT - QA TEAM TASKS

TASK ITEM TASK # TASK DESCRIPTION

Project Sponsorship

PM-1 Evaluate and recommend improvements to verify continuous executive agreement, participation, support, and commitment, and that open pathways of communication exist among stakeholders.

PM-2 Verify that OKDHS management has signed off on all changes that impact Project objectives, cost, or schedule.

PM-3-Q Review and propose Project schedule, scope, and expenditure controls.

Management Assessment

PM-4 Evaluate Project management and organization, to verify that lines of reporting and responsibility provide adequate technical and managerial oversight of Project.

PM-5 Evaluate and report findings on Project progress, resources, budget, schedules, workflow, and reporting.

PM-6 Evaluate and review coordination, communication, and management to verify stakeholders work collaboratively and follow Project Communication Plan.




Project Management

PM-7 Evaluate Project Management Plans and procedures to verify they are developed, communicated, implemented, monitored, and complete.

PM-8 Evaluate Project Management Plans and Project reports to verify Project status is accurately tracked using defined Project metrics.

PM-9-Q Evaluate and verify that Project planning complies with Project Management Plan requirements.

PM-10-Q Evaluate and verify that milestones and completion dates determined by OKDHS are planned, monitored, and met.

PM-11-Q Evaluate and verify Deliverables are entered and tracked by the approved Deliverables Management Tool and comply with Deliverables requirements.

PM-12 Evaluate and verify that the Tracking Tool for Project issues and defects documents issues and defects as they arise; enabling communication of issues and defects to appropriate stakeholders; documents a mitigation strategy as appropriate; and tracks issues and defects to resolution. Tracking issues and defects will include, but not be limited to, technical and development efforts.

PM-13-Q Evaluate and verify that Defect (Incident) Tracking Reports are provided for any defects that could place in jeopardy any Project component presented in the Project Management Plan,and complies with Defect Tracking Report requirements.

PM-14-Q Review, analyze, and propose the Deliverable content, review, and submission process.

PM-15-Q Review, critique, and propose issue resolution and escalation procedures.

PM-16 Evaluate the lifecycle development methodology(s), such as waterfall, evolutionary spiral, rapid prototyping, and incremental, to verify the methodology(s) is appropriate for the system being developed.

Business Process Reengineering

PM-17 Evaluate and verify Project capabilities and plans to redesign business systems to achieve improvements in critical measures of performance, such as cost, quality, service, and speed.

PM-18 Evaluate and verify the Enterprise Architecture Methodology has the strategy, management backing, resources, skills, and incentives required for effective change.

Risk Management

PM-19 Evaluate and verify that the Project Risk Management Plan is created and followed.




PM-20 Evaluate Risk Management Plan and procedures to verify that risks are identified and quantified and that Mitigation Plans are developed, communicated, implemented, monitored, and complete.

PM-21-Q Review and be familiar with Risk Management Plan compliance standards and procedures.

Change Management

PM-22 Evaluate and verify that Project Change Management Plan is created and followed.

PM-23-Q Evaluate and verify that requirements identified as having potential defects are reviewed with the Change Control Committee to determine the impact or necessity of the change.

PM-24-Q Evaluate and verify that Change Control Management activities comply with Change Control Management requirements.

Organizational Readiness

PM-25 Evaluate Change Management Plan and Organizational Readiness Plan and procedures to verify they are developed, communicated, implemented, monitored, and complete and that resistance to change is anticipated and prepared for.

Communication Management

PM-26 Verify that Project Communication Plan is created and followed. QA Team will verify Organizational Management is addressed in the Project Communication Plan.

PM-27 Evaluate Communication Plan and strategies to verify they support communications and work product sharing between all Project stakeholders, and verify Communication Plan and strategies are effective, implemented, monitored, and complete.

Configuration Management

PM-28 Evaluate and verify Change Management Plan and procedures associated with the development process.

PM-29 Evaluate and verify that all critical development documents, including, but not limited to, requirements, design, and code, are maintained under the required level of control.

PM-30 Evaluate and verify that processes and tools are in place to identify code versions and rebuild system configurations from source code.

PM-31 Evaluate and verify that all source and object libraries are maintained for training, testing, and production, and that formal sign-off procedures are in place for approving Deliverables.

PM-32 Evaluate and verify that processes and tools are in place to manage system changes, including formal logging of change requests and review, prioritization, and timely scheduling of maintenance actions.




PM-33 Evaluate and verify that mechanisms are in place to prevent unauthorized changes to the system and prevent authorized changes from being made to the wrong version.

PM-34-Q Evaluate and verify that the configuration management process is in place to control and manage the baseline configuration for all hardware, communications equipment, network equipment, application development security principles, security controls, application deployment, and operational configurations.

Project Estimating and Scheduling

PM-35 Evaluate and make recommendations on Project estimation and scheduling process to verify that Project budget and resources are adequate for the Work Breakdown Structure and schedule.

PM-36 Review and evaluate schedules to verify that adequate time and resources are assigned for planning, development, review, testing, and rework.

Project Staff PM-37 Evaluate the job assignments, skills, training, and experience of the staff involved in program development to verify that they are adequate for the development task.

Project Organization

PM-38 Verify that lines of reporting and responsibility provide adequate technical and managerial oversight of the Project.

PM-39 Verify that Project organizational structure supports training, process definition, independent quality assurance, Configuration Management, product evaluation, and any other functions critical to Project success.

Sub-Contractors and External Staff, if any

PM-40 Evaluate the use, in Project development, of Subcontractors or other external sources of Project staff, such as IT staff from another State of Oklahoma organization.

PM-41 Evaluate and verify that the obligations of Subcontractors and external staff, including terms, conditions, statements of work, requirements, standards, development milestones, acceptance criteria, and delivery dates, are clearly defined.

PM-42 Evaluate and verify that Subcontractor's software development methodology and product standards are compatible with OKDHS standards and environment.

PM-43 Verify that Subcontractor has and maintains the required skills, staff, plans, resources, procedures, and standards to meet Subcontractor's commitment. QA Team will evaluate and verify the feasibility of any off-site support of the Project.

PM-44 Evaluate and verify that any proprietary tools used by Subcontractors do not restrict the future maintainability,




portability, and reusability of the Enterprise System. OKDHS Oversight

PM-45 Verify that OKDHS oversight is provided in the form of periodic status reviews and technical interchanges.

PM-46 Verify that OKDHS has defined the technical and managerial inputs the Subcontractor requires, including reviews, approvals, requirements, and interface clarifications, and has the resources to supply them on schedule.

PM-47 Evaluate Project oversight to verify that OKDHS, not Contractor, exercises ultimate responsibility for monitoring Project cost and schedule.

2.6 Evaluate Quality Management. QA Team will verify Contractor assists in updating the QA Plan, which will document how to plan, implement, and assess the effectiveness of Quality Assurance and Quality Control operations. The QA Plan will fully define the entire Quality system, including, but not limited to, organizational structure, roles and responsibilities, processes, and resources required to implement and manage the QA Plan.

TABLE 3 – QUALITY MANAGEMENT – QA TEAM TASKS


Quality Assurance

QA-1 Evaluate and make recommendations about Project Quality Assurance Plan, procedures, and organization.

QA-2 Verify the fidelity of all defined processes in all phases of the Project is monitored.

QA-3 Verify that the quality of all products produced by the Project is monitored by formal reviews and sign-offs.

QA-4 Verify that Project self-evaluations are performed and that measures are continually taken to improve the process.

QA-5 Monitor the performance of QA processes and designated staff by reviewing QA processes and reports, and by performing spot checks of system documentation. Contractor will evaluate findings and performance of the processes and reports.

QA-6 Verify that Project Management supports the appropriate levels of independence for QA activities.

QA-7-Q Review and be familiar with QA Plan compliance standards and procedures.


TABLE 3 – QUALITY MANAGEMENT – QA TEAM TASKS


Process Definition and Product Standards

QA-8 Evaluate and make recommendations about all defined processes and product standards associated with system development.

QA-9 Evaluate and verify that all major development processes are defined and that the defined and approved processes and standards are followed in development.

QA-10 Evaluate and verify that processes and standards are compatible with each other and with system development methodology.

QA-11 Evaluate and verify that all process definitions and standards are complete, clear, up-to-date, consistent in format, and easily available to Project staff.

QA-12-Q Evaluate and verify that Progress/Status Reports comply with Communications Plan requirements.

QA-13-Q Evaluate and verify that all QA Reports comply with the Reporting requirements as stated in the QA Plan.

QA-14-Q Evaluate and verify that Requirements Discovery and Documentation processes are followed as defined in the Project Requirements Management Plan.

QA-15-Q Evaluate and verify that Functional Requirements Discovery and Documentation processes are followed as defined in the Project Requirements Management Plan.

2.7 Evaluate Training. The QA Team will review Training Plan and the training systems along with any physical training space needs to verify they all meet the requirements.


TABLE 4 – TRAINING – QA TEAM TASKS


User Training and Documentation

TR-1 Verify that Training Plan and processes provide adequate training to system users.

TR-2 Verify that user-friendly training materials and Help Desk services are easily available to all users.

TR-3 Verify that all required standards and processes, and related documentation, are easily available to users.

TR-4 Verify that all training is provided on time, and is evaluated and monitored for effectiveness, with additional training provided as needed.

Developer Training and Documentation

TR-5 Verify that the training plan and processes provide adequate training to system developers.

TR-6 Evaluate and verify that developer training is technically adequate, appropriate for the Development Phase, and available at appropriate times.

TR-7 Verify that all required policy, process, and standards documentation is easily available to developers.

TR-8 Verify the required knowledge transfer occurs for maintenance and operation of the new system.

TR-9 Evaluate and verify that all training is provided on time and is evaluated and monitored for effectiveness, with additional training provided as needed.

TR-10-Q Evaluate and validate training results. TR-11-Q Evaluate and verify that Project staff involved in any processes

is trained in accordance with Training Plan in the required procedures and standards applicable to area of responsibility to do the job correctly.

2.8 Evaluate Requirements Management. Requirements analysis establishes a common understanding of the business unit's requirements between that business unit customer and the software Project Team. QA Team will verify that the requirements process reviews are conducted in accordance with the standards and procedures established by the Project Team. The Contractor will assist OKDHS in ensuring that all Enterprise System RFP requirements are not only met, but requirements are also incorporated and traceable within all documents, models, Deliverables, etc.; and accomplished efficiently and effectively.


TABLE 5 – REQUIREMENTS MANAGEMENT – QA TEAM TASKS


Requirements Management

RM-1 Evaluate and make recommendations about the Requirements Management Plan processes and procedures for managing system requirements.

RM-2 Evaluate and verify that requirements are well-defined, understood, documented, and traceable throughout all phases of the Project.

RM-3 Evaluate and verify the allocation of system resources per hardware and software requirements.

RM-4 Evaluate and verify that software requirements can be traced through design, code, and test phases to verify that the Enterprise System performs as intended and contains no unnecessary software elements.

RM-5 Evaluate and verify that requirements are under formal configuration control.

RM-6 Evaluate and verify Project security for managing requirements and performing the risk analysis on each requirement.

RM-7-Q Evaluate and verify that requirements are reviewed to determine if they are clearly stated and consistent.

RM-8-Q Evaluate and verify that changes to requirements, work products, and activities are identified, reviewed, and tracked to closure.

RM-9-Q Evaluate and verify that the prescribed processes for defining, documenting, and allocating requirements are followed and documented.

RM-10-Q Evaluate and verify that requirements are managed, controlled, and traced by the approved Requirements Change Management Tool.

RM-11-Q Evaluate and verify that Requirements Reviews comply with Requirements standards and guidelines.

RM-12 Evaluate and verify that processes and equipment are in place to back-up Project information and files and archive them safely at appropriate intervals.

RM-13 Evaluate and make recommendations on Project standards and procedures for ensuring the system is secure and privacy of client information is protected.

Security Requirements

RM-14 Evaluate and verify Project restrictions to verify system and information access control.


TABLE 5 – REQUIREMENTS MANAGEMENT – QA TEAM TASKS


RM-15-Q Evaluate and verify security testing and functional tests are performed.

RM-16 Verify that the Requirements Analysis of OKDHS and federal and state requirements and objectives has been performed to verify that Enterprise System requirements are understood, well-defined, and meet federal and state regulations.

Requirements Analysis

RM-17 Verify that all stakeholders have been consulted about the desired functionality of the Enterprise System, and users have been involved in prototyping the user interface.

RM-18 Verify that performance requirements, such as timing, response time, and throughput, meet user requirements.

RM-19 Evaluate and verify that user maintenance requirements for the Enterprise System are completely specified.

RM-20-Q Evaluate and verify that the requirements analysis process and associated requirements reviews are conducted in accordance with the standards and procedures established by the Project and as described in Project RFP, SOW, and Requirements Management Document.

RM-21-Q Evaluate and verify that action items resulting from reviews of the requirements analysis are resolved in accordance with these standards and procedures.

Interface Requirements

RM-22 Evaluate and verify that all system interfaces are exactly described, by medium and by function, including input/output control codes, data format, polarity, range, units, and frequency.

RM-23 Evaluate and verify approved interface documents are available and that appropriate relationships, such as interface working groups, are in place with all agencies and organizations supporting the interfaces.

Reverse Engineering

RM-24 Evaluate that requirements specifications have been developed for all hardware and software subsystems in a level of detail to verify successful implementation and business continuity.

RM-25 Evaluate and verify that a well-defined Transition Plan and process for reengineering the system is in place and is followed, if a legacy system or a transfer system is or will be used in development. The process, depending on the goals of the reuse/transfer, may include reverse engineering, code translation, re-documentation, restructuring, normalization, and re-targeting.


2.9 Evaluate Operating Environment. The QA Team will evaluate software and hardware compatibility in all operating environments to verify that system requirements have been met, are maintainable and upgradeable.

E 6 – OPERATING ENVIRONMENT – QA TEAM TASKS TABL


System Hardware

OE-1 Evaluate current and projected system hardware configurations to verify that performance meets system requirements.

OE-2 Evaluate and verify that hardware is compatible with existing OKDHS processing environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, CPUs and other processors, memory, network connections, bandwidth, communication controllers, terminals, printers, telecommunications systems (LAN/WAN), and storage devices.

OE-3 Evaluate and verify current and projected outside Contractor support of the hardware and OKDHS hardware Configuration Management Plan and procedures.

System Software

OE-4 Evaluate current and projected system software to verify that capabilities meet system requirements.

OE-5 Evaluate and verify that software is compatible with existing OKDHS hardware and software environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, operating systems, middleware, and network software, including communications and file-sharing protocols.

OE-6-Q Evaluate and verify that system requirements are traceable throughout all phases of the Project.

OE-7-Q Evaluate and verify that relevant documents are updated and based on approved requirements changes.

OE-8-Q Evaluate and verify that the agreed upon requirements follow the Systems Standards and Guidelines.

OE-9-Q Verify that design walkthroughs evaluate compliance of the design of requirements, identify defects in the design, and evaluation and report design alternatives.

OE-10-Q Participate in walkthroughs and verify all walkthroughs are conducted.

OE-11-Q Identify defects, verify resolution for previous identified defects, and verify change control integrity.

OE-12-Q Selectively review and audit the content of system documents.


TABLE 6 – OPERATING ENVIRONMENT – QA TEAM TASKS


OE-13-Q Evaluate and review demonstration prototypes for compliance with requirements and standards, verify that the demonstration conforms to standards and procedures, identify lack of compliance with standards, and determine corrective actions.

Database Software

OE-14 Evaluate current and projected database products to verify that capabilities meet system requirements.

OE-15 Evaluate and verify the data format of the database is easily convertible to other formats, supports the addition of new data items, is scalable, easily refreshable, and compatible with existing OKDHS hardware and software, including any online transaction processing (OLTP) environment.

OE-16-Q Review and evaluate the software tools are available to provide database back-up, recovery, performance analysis, and data creation control.

System Capacity

OE-17 Evaluate the existing processing capacity of the system and verify that it is adequate for current statewide requirements for both batch and online processing.

OE-18 Evaluate the historic availability and reliability of the system, including the frequency and criticality of system failure.

OE-19 Evaluate the results of any volume testing or stress testing. OE-20 Evaluate any existing measurement and capacity planning

program and will evaluate the system's capacity to support future growth.

OE-21 Make recommendations about changes in processing hardware, storage, network systems, operating systems, Enterprise System software, and software design to meet future growth and improve system performance.

OE-22-Q Evaluate and verify the system's response time meets requirements stated in the RFP.

OE-23-Q Evaluate and verify the system accessibility complies with RFP requirements.

OE-24-Q Evaluate and verify that Operational Support complies with the Operational Support requirements.

OE-25-Q Evaluate and validate system tests and procedures.

2.10 Evaluate Development Environment. The QA Team will evaluate software and hardware for compatibility in all development environments to verify that development requirements have been met and documentation complies with Project Software Development Methodologies and documentation standards.


TABLE 7 - DEVELOPMENT ENVIRONMENT - QA TEAM TASKS


Development Hardware

DE-1 Evaluate current and projected development hardware configurations to verify performance meets the requirements of system development.

DE-2 Evaluate and verify that hardware is compatible with existing OKDHS development and processing environment, maintainable, and easily upgradeable. Evaluation and verification will include, but is not limited to, CPUs and other processors, memory, network connections, bandwidth, communication controllers, terminals, printers, telecommunications systems (LAN/WAN), and storage devices.

DE-3 Evaluate and verify current and projected outside Contractor support of the hardware.

Development Software

DE-4 Evaluate current and projected development software to verify capabilities meet system development requirements.

DE-5 Evaluate and verify that software is compatible with the existing OKDHS hardware and software environment, maintainable, and easily upgradeable.

DE-6 Evaluate the environment as a whole to verify it shows integration. Evaluation will include, but is not limited to, operating systems, network software, Computer-Aided Software Engineering (CASE) Tools, Project management software, configuration management software, compilers, cross-compilers, linkers, loaders, debuggers, editors, and reporting software.

DE-7 Evaluate and verify current and projected outside Contractor support of the software.

DE-8-Q Evaluate and verify that documentation complies with Project Software Development Methodology and Documentation standards.

DE-9-Q Evaluate and verify that development requirements comply with Development standards and guidelines.

2.11 Evaluate Software Development. The system design process is to develop decisions about the system's behavioral design and other decisions affecting the selection and design of system components. Project Design Documents Standards & Guidelines template will define the organization and content of the Functional Design Document and Technical Design Document.


TABLE 8 – SOFTWARE DEVELOPMENT – QA TEAM TASKS


High-Level Design

SD-1 Evaluate and make recommendations on existing high-level design products to verify that the design is workable, efficient, and meets all system and system interface requirements.

SD-2 Evaluate and verify that high-level design products conform to design methodology and standards.

SD-3 Evaluate and make recommendations on high-level process, standards, methodologies, and CASE tools used.

SD-4 Evaluate and verify that design requirements can be traced back to system requirements.

SD-5 Evaluate and verify that all design products are under configuration control and formally approved before detailed design begins.

SD-6-Q Evaluate and verify that the software requirements analysis process and associated requirements reviews are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process.

SD-7-Q Evaluate and verify that action items resulting from reviews of the software requirements analysis are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process.

SD-8-Q Evaluate and verify that Design Reviews are conducted in accordance with the standards and procedures established by the Project and as described in the Project RFP, SOW, and Requirements Management Process.

Detailed Design

SD-9 Evaluate and make recommendations on existing detailed design products to verify that the design is workable, efficient, and meets all high-level design requirements.

SD-10 Evaluate and verify that detailed design products conform to design methodology and standards.

SD-11 Evaluate and make recommendations on the detailed design and analysis process, standards, methodologies, and CASE Tools used.

SD-12 Evaluate and verify that design requirements can be traced back to system requirements and high-level design.

SD-13 Evaluate and verify that all design products are under configuration control and formally approved before coding begins.




SD-14-Q Evaluate and verify that the method, such as the appropriate Software Development Repository, used for tracking and documenting the development of a software unit is implemented and is kept current.

Job Control SD-15 Evaluate and make recommendations on the existing job control and the process for designing job control.

SD-16 Evaluate the system's division between batch and online processing to verify system performance and data integrity.

SD-17 Evaluate batch jobs to verify appropriate scheduling, timing, and internal and external dependencies.

SD-18 Evaluate and verify that job control language scripts are under configuration control.

Code SD-19 Evaluate and make recommendations on the standards and process for code development currently in place.

SD-20 Evaluate the existing code base to verify portability and maintainability, taking into account software metrics, including, but not limited to, modularity, complexity, and source and object size.

SD-21 Evaluate code documentation to verify quality, completeness – including maintenance history, and accessibility.

SD-22 Evaluate the coding standards and guidelines to verify compliance with these standards and guidelines. Evaluation will include, but is not limited to, structure, documentation, modularity, naming conventions, and format.

SD-23 Evaluate and verify that developed code is under configuration control and is easily accessible by developers.

SD-24 Evaluate and verify use of software metrics in management and quality assurance.

Unit Test SD-25 Evaluate and make recommendations on the plans, requirements, environment, tools, and procedures used for unit testing system modules.

SD-26 Evaluate the level of test automation, interactive testing, and interactive debugging available in the test environment.

SD-27 Evaluate and verify the test process, that test results are verified, that the correct code configuration has been tested, and that the test is appropriately documented.

SD-28-Q Evaluate and verify that the software development methodology, associated code reviews, and software unit testing are conducted in conformance with the standards and procedures established by the Project.




SD-29-Q Evaluate and verify that action items resulting from reviews of the code are resolved in accordance with these standards and procedures established by the Project.

SD-30-Q Evaluate and verify that the system or module presented for acceptance must account for all required functionality, training, conversion, documentation, and any other related requirements of the Contract and federal and State of Oklahoma rules, regulations, and policies for the components comprising the release.

Software Development Lifecycle

SD-31-Q Evaluate and verify that Contractor in conjunction with OKDHS will provide the Iterative Application Software Development Methodology.

SD-32-Q Evaluate and verify that documentation and computer program materials are approved and placed under library control.

SD-33-Q Evaluate and verify the establishment of formal release procedures for approved documentation and software versions.

SD-34-Q Evaluate and verify that library controls prevent unauthorized changes to the controlled software and verify the incorporation of all approved changes.

2.12 Evaluate System and Acceptance Testing. QA Team activities will verify software integration and test activities combine individually developed components together in the testing environment to verify that they work together to complete the software and system functionality. A Pilot will verify the functional and technical usability of the Enterprise System in a limited production environment.


TABLE 9 – SYSTEM AND ACCEPTANCE TESTING - QA TEAM TASKS


System Integration Test

ST-1 Evaluate plans, requirements, environment, tools, and procedures used for integration testing of system modules.

ST-2-Q Evaluate and verify that software test activities are identified, test environments have been defined, and guidelines for testing have been designed. QA Team will verify the software integration process, software integration testing activities and all software performance testing activities are being performed in accordance with QA Plan processes and procedures, and the Software Development Methodology.

ST-3-Q Evaluate and verify any transfer of control of code to staff performing software integration testing or software performance testing is being accomplished in accordance with QA Plan processes and procedures as well as Software Development Methodology.

ST-4-Q Evaluate and review the Test Plan and Software Test Descriptions for compliance with requirements and standards.

ST-5-Q Evaluate and monitor test activities, witness tests, and certify test results.

ST-6-Q Evaluate and verify that requirements have been established for the certification or calibration of all support software or hardware used during tests.

ST-7 Evaluate the level of automation and the availability of the system test environment.

ST-8 Verify that an appropriate level of test coverage is achieved by the test process, test results are verified, the correct code configuration has been tested, and tests are appropriately documented, including formal logging of errors found in testing.

ST-9 Verify that the test organization has a level of independence from the development organization.

Pilot ST-10 Evaluate the plans, requirements, environment, tools, and procedures used to pilot the Enterprise System.

ST-11 Verify that test scenarios are used to verify comprehensive but manageable testing, and that tests are run in a realistic, real-time environment.

ST-12 Verify that test scripts are complete, with step-by-step procedures, required pre-existing events or triggers, and expected results.

ST-13-Q Verify that pilot results meet criteria, that the correct code configuration has been used, and that the test runs are




appropriately documented, including formal logging of errors found in testing as defined in the Implementation Plan.

ST-14-Q Evaluate and validate pilot effectiveness. ST-15-Q Evaluate and verify all functional aspects of the system. ST-16-Q Evaluate and verify operability and stability of software. ST-17-Q Evaluate and verify accuracy of conversion of legacy data

and manual data. ST-18-Q Evaluate and verify impact of missing and erroneous data. ST-19-Q Evaluate and verify completeness and accuracy of system

documentation. ST-20-Q Evaluate and verify effectiveness of training methods and

materials. ST-21-Q Evaluate and verify Pilot impact on workflow and staff

productivity. ST-22-Q Evaluate and verify response time and overall system and

network performance. ST-23-Q Evaluate and verify system hardware, software and

telecommunications performance. ST-24-Q Evaluate and verify appropriateness of system, data and

application security. ST-25-Q Evaluate and verify accuracy and performance of system

interfaces. Implementation ST-26-Q Evaluate and verify Project Managers create a work plan that

includes an approach to the implementation of components with recommended sequence and schedule.

ST-27-Q Verify Contractor, with OKDHS staff, will lead the effort to develop the Implementation Plan for implementing new and reengineered processes.

ST-28-Q Evaluate and verify the tool selected for the Data Integration will provide the ability for data profiling, data quality, data integration, data enrichment, and data monitoring.

ST-29-Q Evaluate and verify Contractor's proposed Data Conversion Plan presents a comprehensive strategy for automated and manual conversion efforts and incorporate the OKDHS schedule for Pilot testing and statewide Implementation.

Benchmark Tests

ST-30-Q Evaluate and verify Contractor tests performance of the software during a Pilot and of the application after Implementation conducting Benchmark Tests and reporting Benchmark Results to OKDHS.

ST-31-Q Evaluate and verify Contractor meets expected capacity




simulation results, tuning specifications, and performance benchmarks for all installations and deployments.

Interface Testing

ST-32 Evaluate interface testing plans and procedures for compliance with requirements and verify that plans meet acceptance criteria.

ST-33-Q Verify interface testing is in accordance with Interface Control Document.

Acceptance and Turnover

ST-34 Verify that acceptance procedures and acceptance criteria for each product are defined, reviewed, and approved by program stakeholders prior to test, and that results of the test are documented. QA Team will verify that acceptance procedures address the process by which any software product that does not pass acceptance testing will be corrected.

ST-35-Q Evaluate and verify that as many of the software integration tests as necessary and all software performance tests are witnessed to verify that the approved test procedures are being followed, that accurate records of test results are being kept, that all discrepancies discovered during the tests are being properly reported, that test results are being analyzed, and the associated test reports are completed.

ST-36-Q Evaluate and verify that discrepancies discovered during software integration and performance tests are identified, analyzed, documented, and corrected; software unit tests, and software integration tests are re-executed as necessary to validate corrections made to the code; and the software unit's design, code, and test is updated based on the results of software integration testing, software performance testing, and corrective action process.

ST-37-Q Evaluate and verify the software performance tests produce results that will permit determination of performance parameters of the software.

ST-38-Q Verify that the responsibility for testing and reporting on results has been assigned to a specific program owner for sign-off.

ST-39 Verify that acceptance testing based on the defined acceptance criteria is performed satisfactorily before acceptance of products.

ST-40 Verify that the acceptance test organization has a level of independence from the Sub-Contractor or Contractor contracted with OKDHS to develop the Enterprise System.




ST-41 Verify that training of OKDHS staff in using Contractor-supplied software is ongoing throughout the development process.

ST-42 Review and evaluate Implementation Plan. ST-43-Q Evaluate and validate implementation success. ST-44-Q Evaluate and validate post-implementation activities. ST-45-Q Assist in activities to secure federal acceptance and approval.ST-46-Q Verify that warranty and maintenance periods do not start

until OKDHS acceptance. Turnover Documentation

ST-47-Q Evaluate and verify Contractor provides the Data Conversion Test Results Document.

2.13 Evaluate Data Management. QA Team will verify that the Enterprise Database Repository being developed can perform quickly and efficiently by meeting Enterprise System RFP performance goals. QA Team will develop processes to verify that the converted data is not duplicated to the degree practicable and that the data is accurate.

TABLE 10 – DATA MANAGEMENT – QA TEAM TASKS


Data Conversion

DM-1 Review and evaluate existing and proposed plans, procedures, and software for data conversion.

DM-2 Verify that procedures are in place and are being followed to review the completed data for completeness and accuracy and to perform data clean-up as required.

DM-3 Evaluate conversion error rates to verify the error rates are manageable and meet acceptance criteria.

DM-4 Make recommendations to make conversion process more efficient and maintain integrity of data during conversion.

Database Design

DM-5 Evaluate new and existing database design documents to verify they meet existing and proposed system requirements.

DM-6 Verify that appropriate processes are used in database design to improve data integrity and system performance.

DM-7 Verify appropriate processes are used to verify the database design has maintainability, scalability, concurrence, normalization – where appropriate, and any other factors affecting performance and data integrity.

DM-8 Review and evaluate the process for administering the database, including back-up, recovery, performance analysis, and control of data item creation.


2.14 Evaluate Operations and Business Oversight. This QA task ensures that Operation Process Improvement will be an ongoing and continuous process.

TABLE 11 – OPERATIONS OVERSIGHT - QA TEAM TASKS


Operational & Business Change Tracking

OB-1 Review and evaluate the OKDHS statewide systems change request and defect tracking processes.

OB-2 Evaluate implementation of the Operational & Business process activities and request data to verify processes are effective and are being followed.

User Operational/ Business Satisfaction

OB-3 Review and evaluate user satisfaction with system and make recommendations for improvement.

Operational & Business Goals

OB-4 Evaluate impact of the system on program goals and performance standards.

Operational Documentation

OB-5 Evaluate operational/business plans and processes.

Operational Processes and Activities

OB-6 Evaluate implementation of operational/business processes and activities, including back-up, disaster recovery testing, and day-to-day operations, to verify the processes are being followed.

2.15 Evaluate Software Products Review Process. This QA task ensures that quality review processes are in place for all software products, which may include representations of information other than traditional hard-copy documents, and that these products have undergone software product evaluation, testing, and corrective action as required by the standard. QA Team will review all software tools and verify that OKDHS becomes the owner of all software tools at the conclusion of the Project.

TABLE 12 – SOFTWARE PRODUCTS REVIEW – QA TEAM TASKS


Software Product Review

SP-1-Q Evaluate and verify that software products that are ready for review are reviewed and results are reported.

SP-2-Q Evaluate and verify issues or defects reported are resolved in accordance with QA standards and procedures.

Software Tool Evaluation

SP-3-Q Conduct evaluations of tools, both existing and planned, used for software development and support.

SP-4-Q Evaluate and verify tools are adequate by assessing whether they perform the functions for which the tools are intended.


TABLE 12 – SOFTWARE PRODUCTS REVIEW – QA TEAM TASKS

SP-5-Q Evaluate and verify tools for applicability by assessing whether the tool capabilities are needed for the software development or support.

SP-6-Q Evaluate and verify planned tools are feasible by assessing whether they can be developed with the techniques and computer resources available or by procurement.

SP-7-Q Research and evaluate software tools available to automate QA functions.

SP-8-Q Evaluate the tools to verify the techniques used include review of the use of standards, software inspections, requirements tracing, requirements and design verification, reliability measurements and assessments, and rigorous or formal logic analysis.

2.16 Evaluate Component Deliverable (Release) Process. QA Team will evaluate the activities in preparation for component delivery to verify that program or Project requirements for functional and physical audits of the end-item product are being satisfied. In some cases, QA Team may prohibit delivery of certain items, such as documentation, code, or a system, if Project fails to meet contractual requirements or standards.

TABLE 13 – COMPONENT DELIVERABLE PROCESS – QA TEAM TASKS


Component Deliverable

CD-1-Q Evaluate and verify the Project Migration Approval Process guide defines the migration of products being released or migrated from one environment to another.

CD-2-Q Evaluate and verify that all required interfaces are properly connected and integrated.

2.17 Evaluate Media Certification, Storage and Handling Process. The Project will have various media and storage needs throughout. QA Team will put processes in place to verify media, products, versioning and storage are current, compatible and in sync.


TABLE 14 – MEDIA CERTIFICATION PROCESS – QA TEAM TASKS


Media Certification

MC-1-Q Evaluate and verify the media containing the source code and the media containing the object code that are delivered to the Project correspond to one another.

MC-2-Q Evaluate and verify that all products delivered will be in a format consistent with Standards as described in the Project RFP, SOW, and Requirements documents.

MC-3-Q Evaluate and verify that the software version represented by this media matches that on which software performance testing was performed, or correctly represents an authorized update of the code, as applicable.

Storage and Handling

MC-4-Q Evaluate and verify that a plan, methodology, or set of procedures for storage and handling of media is in place.

MC-5-Q Evaluate and verify the storage of the software product and documentation to verify that storage areas for paper products or media are free from adverse environmental effects, such as high humidity, magnetic forces, and dust.

MC-6-Q Verify that electronic copies of source code are in the custody of an independent escrow agent.

2.18 Evaluate Non-Deliverable Software Certification. The Project may use non-Deliverable software in the development of Deliverable software, provided the operation and support of the Deliverable software after delivery to Project do not depend on the non-Deliverable software, or provision is made to verify that OKDHS has or can obtain the same software.

TABLE 15 – NON-DELIVERABLE SOFTWARE CERTIFICATION – QA TEAM TASKS


Non-Deliverable Software

SC-1-Q Evaluate and verify that the use of non-Deliverable software meets the above criteria, that is, Deliverable software is not dependent on non-Deliverable software to execute, or verify that OKDHS can obtain the same software.

SC-2-Q Evaluate and verify that all non-Deliverable software used on the Project performs its intended functions.

2.19 Evaluate Performance Standards. The minimum performance standard expectations and requirements are described in detail in the Enterprise System RFP and will provide the state and federal documentation the Contractor will follow regarding both business and systems requirements.


TABLE 17 – PERFORMANCE STANDARDS – QA TEAM TASKS


Performance Evaluation

PE-1-Q Evaluate and verify Contractor adheres to the performance requirements established in the Project RFP.

PE-2-Q Review and measure response time standards are met in the production environment for the life of the Project.

PE-3-Q Review and evaluate that response time standards are met during the testing of the system.

PE-4-Q Review and evaluate performance metrics.

3.0 PROJECT DELIVERABLES 3.1 QA Activities for Deliverables. During the Project, QA activities will be

performed to verify Contractor complies with OKDHS minimum requirements for content, submission, review, testing, and acceptance of all Project Deliverables. In the event reviewing or testing identifies a requirement is not met in whole or in part, OKDHS may return the Deliverable to Contractor immediately for rework or continue reviewing or testing until reviewing or testing is complete, then return the Deliverable to Contractor for rework.

3.2 Walkthrough for Deliverables. Contractor will facilitate a walkthrough with OKDHS for each Project Deliverable prior to delivery. Following the walkthrough, OKDHS will receive the Deliverable unless a deficiency is discovered in the walkthrough. In the event OKDHS identifies a Deliverable that requires more than the minimum period for review, the period will be scheduled in the Project Plan. OKDHS will receive Project Deliverables based upon the Project Plan. Deliverable Acceptance is further explained in the Project RFP and Section 6.1 of this QA Plan.

4.0 REVIEWS AND AUDITS In order for QA Team to evaluate compliance with the QA Plan and Project RFP, QA Team will review and approve Deliverables throughout the Project lifecycle. These reviews will specify that the evidence of work generated is adequate to verify compliance with Project scope, contract, and quality requirements. Audits performed by the QA Team or state and federal auditors will include examination of both internal and external Project and product Deliverables. 4.1 Verify Document and Artifact Deliverable Review. All documentation and

artifacts generated throughout the Project lifecycle will be subject to a QA review. In addition to Project Management Plans, other identified documentation for review will be that which control the processes, development, verification and validation, use and maintenance of the software and hardware. Appropriate stakeholders and Project Team members will be requested as needed by the QA Team to help in the document reviews. QA checklists will be developed to verify all document components meet Project and contract requirements.


4.1.1 Disciplines for Documentation Standard Practices. Project documentation templates will be used as the basis for all document Deliverables of the Project RFP. A waiver exempting Project documentation Deliverables from standard template usage will be required. 4.1.1.1 The Project Team will use industry best practices

documentation standards and application-related documentation modeling tools, and adhere to sound modeling principles to verify standardization and traceability of all system application documentation.

4.1.1.2 QA reviews will also focus on Business Requirement Documents, Technical Requirement Documents, and Design Documents which will be the basis for developing Test Plans and performing quality control testing.

4.1.2 Document and Artifact Review Timeframes. All document Deliverables will be reviewed in a group walkthrough conducted by Contractor with OKDHS. No more than seven documents will be scheduled for review by OKDHS during the same time frame. As stated in the Project RFP, for each program document Deliverable, the schedule must allow: 4.1.2.1 10 business days for review of documents 50 pages or less. 4.1.2.2 15 business days for documents 51 – 100 pages. 4.1.2.3 20 business days for documents 101 pages or more.

4.2 Verify Project Management Compliance Reviews. QA Team periodic Project management compliance review of Project status, progress, defects, and risks will provide an independent assessment of Project activities. Compliance reviews and reports will be conducted and reported, including: 4.2.1 Scheduled Compliance Reviews. QA Team will generate and maintain

compliance review schedule. Reviews will occur based upon milestones or triggers as indicated in the Project Plan. The results of audits will be discussed with the Program Quality Manager and the Project Team Lead or other person(s) responsible for the production of the Deliverable. Results will be submitted by the QA Team to the OKDHS Program Manager(s) in scheduled status reports.

4.2.2 Unscheduled Compliance Reviews. QA Team will perform random and unannounced compliance reviews to verify the corrective actions agreed to during the scheduled reviews are being followed. The results of the reviews will be discussed with the Program Quality Manager and Project Team Lead or other person(s) responsible for the production of the Deliverable. Results will be submitted by the QA Team to the OKDHS Program Manager in scheduled status reports.

4.2.3 Compliance Review Reports. Compliance review reports and recommended corrective actions generated by QA Team will be


brought to the attention of the persons or Team Lead responsible for producing the Deliverable, using the Project Deliverable process. Corrective action will be recommended and reviewed with the persons and Program Quality Manager or Team Lead. Compliance, defect areas, and risks will be followed-up and tracked to closure. The results of reviews of the QA function will be tracked and maintained by the QA Team. QA Team will provide the following information to Project Team: 4.2.3.1 Compliance – Identification of the level of compliance of the

Project with established organizational and Project processes.

4.2.3.2 Defect areas – Identification of potential or actual Project defect areas based on analysis of technical review results.

4.2.3.3 Risks – Identification of risks based on participation and evaluation of Project progress and trouble areas.

4.3 Conduct Process Audits and Reviews. Project processes are audited according to the tasks specified in this QA Plan and performed in accordance with the Project Plan and schedule. The forms utilized by QA Team for audit reporting include, but are not limited to, the Compliancy Report. The Compliancy Report can be customized and used on any specific compliance audit or review. 4.3.1 Process Audit and Review Schedule. Project Teams will provide

current status to the Executive Sponsor, and team members through the reviews, compliance audits, reports, and working interchanges established for the program. Additional audits, reviews, inspections, walkthroughs, tactics, and measurements will be further refined in later phases of the Project. Appendix C reflects the Process Audit and Review Schedule.

4.3.2 Compliancy Report. QA Team will report the results of the process audit and provide recommendations, if necessary, using the Compliancy Report. The Compliancy Report is used to record that the process is being followed correctly and working effectively, being followed but is not working effectively, or not being followed. Appendix D reflects the Compliancy Report form.

4.3.3 Submittal and Disposition of Compliancy Report. The Compliancy Report will be directed to: 4.3.3.1 Decision Team – The results of process audits is used in

conjunction with other Project status information to guide the Decision Team's attention to identify and mitigate Project risks at the organizational level.

4.3.3.2 Program Quality Manager – The Program Quality Manager utilizes the report to:


Determine whether there is compliance with the development process and its effectiveness in meeting Project goals. Where necessary and appropriate, Program Quality Manager may initiate enforcement activities or initiate change to the established processes using the approved procedures.

Provide information to Team Lead regarding compliance with the development process and its effectiveness in meeting Project goals.

Indicate agreement, disagreement, or deferral of recommendations cited in the Compliancy Report. If the OKDHS Program Manager indicates disagreement with the recommendations recorded on the Compliancy Report, the final disposition of report recommendations will be made by Decision Team.

4.4 Evaluation. Project processes are evaluated according to the tasks specified in this QA Plan and performed in accordance with the Project Plan and schedule. The forms utilized by QA Team for evaluation reporting include, but are not limited to, the Software Tool Evaluation checklist, Performance Standards Evaluation checklist, Pilot Evaluation checklist, and the Implementation Evaluation checklist. The Evaluation checklists can be customized and used on any specific review. 4.4.1 Software Tool Evaluation. Quality review processes are in place for all

software products and these products will be evaluated, tested, and corrective action performed as required by the standard. Appendix E reflects the Software Tool Evaluation checklist.

4.4.2 Performance Standards Evaluation. Quality review processes are in place for evaluation of performance standards. The QA Team will work with Contractor and the OKDHS Performance Analysis Team to analyze capacity and performance information and resolve any capacity and performance issues prior to or in conjunction with Implementation. Appendix F reflects the Performance Standards Evaluation checklist.

4.4.3 Pilot Evaluation. Pilot criteria will be evaluated by the QA Team for Pilot success. As stated in the Project RFP, the Pilot Plan that Contractor will provide will detail the methodologies and best practices used, and will include, but is not limited to, Pilot test criteria that is subject to evaluation of how the criteria will be performed, captured, or measured. Appendix G provides the Pilot Test Evaluation checklist.

4.4.4 Implementation Evaluation. Implementation criteria will be evaluated by the QA Team for Implementation success. The Implementation Plan will detail the methodologies and best practices used, and include, but are not limited to, Implementation test criteria that are


Production TrainingUser AcceptanceQA/QC Development

End

Start

System / IntegratedTesting

TestingRegression

TestingPerformance

/ Load

u F nctional Testing

Unit TestingTesting

Acceptance User

TestingPerformance

/ Load

Training

Production

.by italicsDSD Environments are indicated th Approval is required to move objects from one environment to e next nts . Demarcation for environme

subject to evaluation of how the criteria will be performed, captured, or measured. Appendix H provides the Implementation Evaluation checklist.

5.0 TESTING PROCESS AND ENVIRONMENTS Six types of testing and five OKDHS DSD environments have been defined and will encompass Project testing activity for all the environments. The QCT Standards and Processes document provides the test process flow. QA Team will audit the QCT activities as defined in the QCT Standards and Processes document and QA Plan, and verify that the software and test documentation is subject to change management control. QA Team will witness the tests and verify that test results are evaluated and documented. 5.1 System and User Acceptance Testing Processes. State system and User

Acceptance testing will not begin until after Contractor has completed thorough internal testing, all programming is completed and approval of all documents has been received. All internal test documents will be forwarded to OKDHS for review to verify Contractor has performed testing of each component. Failure to adequately test or provide components that have not been tested or failed testing will result in the application of liquidated damages as outlined in the Project RFP. Figure 2 reflects the component testing processes in DSD environments.

Figure 2. Component Testing Process

5.1.1 Unit testing is usually performed on a smaller scale and is focused specifically on functionality of the component. Each data field will be tested for its limits, as described below. All paths through the component will be tested. Test scenarios will be defined for unit


testing based on the functionality required and followed as part of the unit test.

5.1.2 Integration testing is performed either with multiple new components or a mixture of one or more new components and production code. All functionality (new and old) will be tested during this testing phase. All paths through the component will be tested with multiple values of test data.

5.1.3 Functional testing focuses upon both new functionality requested and on previously existing functionality. Functional testing compares new functionality to requirements documents and verifies that requirements have been met successfully.

5.1.4 Regression testing is a subset of functional testing. Testing is performed either with new components and existing components to determine if all work well together. All functionality (new and old) will be tested during this testing phase. All paths through components will be tested with multiple values of test data.

5.1.5 Load/Performance testing is a subset of functional testing. Performance best practices and guidelines are met concerning, but not limited to, resource utilization, response time, baselines, and processing time. Success criteria will be based upon defined metrics.

5.1.6 User acceptance testing is focused primarily upon new functionality requested, or if a completely new component, in its entirety.

5.2 System Environments. The six OKDHS DSD environments are Research, Development, QA/QC, User Acceptance, Training, and Production. Only five of these environments will be utilized in the development of the Enterprise System. Figure 3 illustrates a typical migration path for all production bound modifications and enhancements. QA Team will verify that the Project Team will develop and provide acceptance criteria for the hardware, software, and Enterprise database for each environment.

Figure 3. Typical Migration Path Through the Six OKDHS DSD Environments.

Dotted line represents check-out of production objects. Dashed line represents that the training environment is optional.


5.2.1 Research Environment is primarily for allowing the research analysts to install, deploy and test new technology and products in an isolated environment. Research activities that change the nature of the environment enough to negatively impact the development environment occurs here. Basic feasibility studies and preliminary proofs of concept can be implemented here. It will include elements of the production environment as needed, but will rarely, if ever, mimic the complete production environment. This environment will be a stand alone environment and will not be used in the regular migration path.

5.2.2 Application Development Environment is primarily for allowing the analysts, designers and developers to create source objects (programs, databases etc.) for initial development in a protected environment. It closely resembles the production environment. Unit testing and maintenance occurs here. Unit testing will generally be performed by the requestor of the object.

5.2.3 QA/QC Environment will be primarily used to perform quality assurance and quality control testing for all applications before implementing in production. The objects created in the development environment will be migrated to this environment after unit testing by the analysts, designers and developers is complete. Application, integration, performance and regression testing occurs here. Changes to the software, hardware or process will never be made directly to this environment. Each change will need to follow the migration process and start with the Application Development environment.

5.2.4 User Acceptance Environment will be used to perform user acceptance testing of new or changed applications in a protected environment. Performance testing occurs here. The new applications and their supporting elements and processes are the only variances from the production environment. End-users will use this environment to verify that requirements have been translated to the end product. This is where the users review, reject, or approve applications. Changes to the software, hardware or process will never be made directly to this environment. Each change will need to follow the migration process and start with the Development environment.

5.2.5 Training Environment will be primarily used for training for all OKDHS production applications. Changes to the software, hardware or process will never be made directly to this environment. Each change will need to follow the migration process and start with the Development environment.

5.2.6 Production Environment is the final destination of all finished products. All OKDHS production applications and databases reside in this environment. Applications can only be introduced here after they have been through the migration process and software configuration management procedures. Changes to the software, hardware or process will never be made directly


to this environment. Each change will need to follow the migration process and start with the Development environment.

5.3 Quality Control Testing Process. A thorough and consistent quality control process will be implemented by the QA Team. The QA Team will form a testing team to perform activities defined or referenced in this plan. Each major Deliverable will be reviewed by the Project QA/QC Lead, QA Team, and development Contractor against the quality control procedures to verify that no requirements are overlooked. 5.3.1 Quality Control Testing (QCT) and Migration Manual. The QCT

processes to be followed by the testing team are outlined in the Project QCT Standards and Processes document. This document will establish and flowchart the QCT processes to follow and tools to be utilized during testing of the various framework modules. The QCT document shows how the QCT Deliverables will be met for each of the planned modules.

5.3.2 Project Migration Approval Process Manual. The Project migration approval processes will be followed to identify the processes to follow for migrating modules or sub-modules into the OKDHS DSD environments. An approval process with QC checklists will be utilized that gives OKDHS the final approval for any migration packages and to verify that all impacted customers are notified promptly.

5.3.3 Test Plans. As stated in the Project RFP, Contractor will assist OKDHS in developing the Test Plans and test cases, scenarios, scripts, and data sufficient to fully prove the Enterprise System meets all business and technical requirements. The template requirements for the Test Plans are documented in the Project QCT Standards and Processes document. The QCT document describes how the system will be tested, how the Test Plans will be developed, and who will perform each required test. The QA Team will verify the Test Plan will be traceable to the requirements and design documents to validate that all requirements have been addressed.

6.0 VALIDATION AND ACCEPTANCE PROCESS QA Team will verify all validation and acceptances are completed in accordance with the Project RFP. QA Team will document completed Deliverables that have been accepted or not accepted along with reasons for non-acceptance. A Deliverable Tool will be maintained by the QA Team which will track each Deliverable, review and approval dates, and issues. Contractor, in conjunction with the OKDHS Quality Team, will facilitate the User Acceptance process. 6.1 Deliverable Acceptance. Contractor will provide to OKDHS a copy of the

Deliverable Acceptance Letter with the invoice for each Deliverable. OKDHS will notify Contractor in writing of the acceptance status, by the end of the review/testing period for the Deliverable, whether Deliverable is: 6.1.1 Accepted without condition.


6.1.2 Accepted with cosmetic deficiency. 6.1.3 Not accepted and returned to Contractor for rework. In the event

OKDHS returns a Deliverable to Contractor for rework, the review period begins again on Contractor redelivery. At that time, Contractor will estimate the redelivery date. Deliverables returned to Contractor for rework will be considered delayed and appropriate liquidated damages may be applied.

6.2 Contractor's System Certification. Contractor and OKDHS must certify in writing, via the Final Implementation Acceptance Letter, that the statewide Implementation of the module successfully met the requirements of the Implementation Plan. Final Implementation Acceptance Letter is required for final payment of the implemented module. OKDHS will certify the acceptance of systems when all requirements are met and federal or State of Oklahoma approval, or both, if applicable, is received.

6.3 Acceptance Plans and Releases. As stated in the Project RFP, upon successful implementation of each incremental release, Contractor must present the systems/modules to OKDHS for acceptance. The system presented for acceptance must account for all required functionality, training, conversion, documentation, and any other related requirements of the Project RFP and the federal regulations, OKDHS, and policies for the components comprising the release. Contractor will turn over the system component(s) release to OKDHS for final acceptance upon successful implementation of each Project module in all OKDHS offices.

6.4 Federal Acceptance and Approval Preparation. Contractor will assist OKDHS in activities to secure federal acceptance and approval of the Enterprise System. OKDHS will submit the Enterprise System Certification Review Plan for federal approval. Contractor will provide support to OKDHS and documentation, as required, in all reviews in preparation for or during the federal approval process, to attain federal approval of the Enterprise System. Contractor will provide all Required System Documentation and support to obtain federal approval.

6.5 Food and Nutrition Service (FNS) Acceptance Requirements. FNS policies and procedures that OKDHS must follow in order to receive federal funding to develop, acquire, and implement the Enterprise System are described in FNS Handbook 901.6.5.1 Status Reports. The results of Project monitoring will be reported in

status reports. FNS requires status reporting that will be defined in the Communication Plan. The FNS Handbook 901 refers to Appendix D-23 for Status Report Checklist and Appendix E for a sample status report. QA Team will verify the required information is included in the status reports.

6.5.2 Planning Document Reviews and Closures. Reviewing planning documents such as PAPD, IAPD, APDU, and APD includes confirming


the Project objectives have been met and determining the actual costs incurred. The required information is in the FNS Handbook 901, section 2.7.

6.5.3 FNS Post-Implementation Review. FNS may conduct a post-implementation review of the system once it is fully operational statewide (approximately 6 months after system deployment statewide and to accommodate the initial user learning curve). FNS may conduct an onsite post-implementation review to verify OKDHS accomplished the goals stated in its APD. This review encompasses the program, technical, security, and financial aspects of the system. The required information is in the FNS Handbook 901, section 2.7.1. The Food Stamp Program Post-Implementation Review printable checklist can be found at http://www.fns.usda.2ov/apd/FSP PIR/Full Checklist. PDF.

6.5.4 Systems Functional Requirements Review. FNS may elect to conduct a System Functional Requirements Review before and/or during the initial pilot training and before the deployment of software. The required information is in the FNS Handbook 901, section 2.11.3.1.

6.5.5 Cost Reviews and Audits. OKDHS will provide access to all cost records relating to system development and operations. FNS may use data mining software during these reviews. This will require OKDHS to provide FNS staff with Project expenditures in an electronic format. Failure to cooperate with Federal requests for information in support of a review or audit may result in suspension or termination of FNS funding for the system and its operations. The required information is in the FNS Handbook 901, section 7.4.

6.5.6 Regional Office Expenditure Review. FNS RO will compare reported expenditures for IS development from the Form SF-269 (http://www.whitehouse.gov/omb/grants/sf269.pdf), or other expenditure reports, with the expenditures reported in the annual APD. Any differences will be examined and will need to be reconciled. There should be no significant differences between expenditures reported on the Form SF-269 and those reported on the annual APDU.

6.6 Health and Human Services (HHS) Acceptance Requirements.Automated Systems For Child Support Enforcement: A Guide For States was developed by the U.S. Department of Health and Human Services (DHHS) Administration for Children and Families (ACF). The guide addresses the requirements associated with federal certification of comprehensive, automated, statewide Child Support Enforcement systems. The guide and other related certification material may be found at http://www.acf.hhs.gov/programs/cse/stsys/dsts_cert_guide.html. 6.6.1 Authority. The origin of the programs overseen and financed by

DHHS/ACF is the Social Security Act. Included under Administration for Children and Families scope of review authority is Title IV-D, Child


http://www/

http://www/

http://www/

Support Enforcement. All other authority is referenced in Chapter I, Section B, of the Automated Systems Guide.

6.6.2 General Requirements. OKDHS staff and Contractors working on systems subject to certification will use this guide throughout the lifecycle of the system development effort. Chapter II of the Automated Systems Guide encompasses the general requirements to adhere to.

6.6.3 CSE System Requirements. Nine functional areas of child support enforcement with their related system requirements are identified in Chapter III of the Automated Systems Guide. Several hyperlinks to related websites are referenced throughout Chapter III of the guide.

7.0 ISSUE REPORTING AND CORRECTIVE ACTION This section describes the issue reporting and control process used by QA Team to record and analyze issues, discrepancies, and defects; and to monitor the implementation of corrective action. Corrective action will involve actions taken as a result of a QA/QC measurement, audit, or review that indicates that the development process exceeds established parameters. Jointly with OKDHS, Contractor will review, critique, and propose issue and defect resolution and escalation procedures. 7.1 Escalation Procedure for Resolution Disputes. When affected Project

staff dispute the findings and recommendations of a Compliancy Report, Defect (Incident) Report, or evaluation checklist, QA Team will first communicate with OKDHS Program Manager to resolve the dispute. If the OKDHS Program Manager also disputes the findings or recommendations, Program Quality Manager will direct final disposition of recommendations and may implement, defer, or cancel the implementation of corrective actions. This direction is recorded and dated by OKDHS Program Manager to be added to the QA evaluation records of the Project. QA Team retains the original record of findings and subsequent resolution data in its audit files. The Authority/Decision Chart as reflected in the Project Management Plan and the QA Plan will be the Escalation Levels followed by the QA Team.


PERSON RESPONSIBLE AUTHORITY RESPONSIBLE TO

Commission and Legislative Liaison

1 OKDHS Commission

Sponsor 2 Commission and Legislative Liaison

Decision Team 3 Sponsor

Program Manager 3 Decision Team

Project Manager or Team Lead

4 Program Manager

Team Member 4 Project Manager or Team Lead

DEGREE OF AUTHORITY 1 = Final authority to decide or act on any and all matters.

2 = Final authority to decide or act on any matters within budgets and existing OKDHS staff.

3 = Authority to decide or act on any matters within Project budget and existing Project staff.

4 = Authority to decide or act; participates with, or act after first consulting with the person to whom they are responsible.

7.2 Corrective Action Process. Defects identified for resolution may be subject to a corrective action process and may include, but not limited to, schedule and plan non-conformance, documentation errors, software errors, and noncompliance with standards and procedures. 7.2.1 The steps of the corrective action process include:

7.2.1.1 Defect identification and correction occurring during software development to verify early detection of actual or potential defects.

7.2.1.2 Reporting of the defect to the proper authority. 7.2.1.3 Analysis of the defect to propose corrective measures. 7.2.1.4 Timely and complete corrective action. 7.2.1.5 Recording and follow-up on each defect's status.

7.2.2 QA activities for the corrective action process include: 7.2.2.1 Periodically review the corrective action process and their

results against the Change Management Plan to assess the effectiveness of the corrective action process.

7.2.2.2 Perform periodic analysis of all reported defects to identify trends that may disclose generic defect areas. These


analyses will include the study of the defect causes, magnitude of impact, frequency of occurrence, and preventive measures.

7.3 Recording Defects (Incidents) in Software Code or Documentation.Defects (Incidents) found in the software code or documentation that is being developed must be recorded by means explained below regardless of how or by whom the defect was discovered. QA Team will analyze defects for trends in an effort to prevent recurring discrepancies. QA Team will report the results of trend analyses along with suggestions for defect resolution and prevention. As stated in the Project RFP, Contractor will track questions, issues, and defects, issue and defect resolution, and approved changes to design and make necessary changes to documentation within 30 calendar days after the approved change, using an agreed-upon issue and defect tracking system. 7.3.1 Defect (Incident) Tracking Tool. If a defect is found during OKDHS

QCT Process, the defect will be documented using the approved Defect (Incident) Tracking Tool and submitted to Contractor for review and statement of corrective measures. The following process will be followed for the documented Defect Tracking Tool: 7.3.1.1 Contractor will document corrective actions in the original

Defect Tracking Tool.7.3.1.2 OKDHS will review the correction and re-test the correction

and all related automation components that the correction could affect.

7.3.1.3 If the re-test was successful, OKDHS will document the re-testing effort within the Defect Tracking Tool, close out the incident, and notify Contractor.

7.3.1.4 If the re-test was unsuccessful, OKDHS will notify Contractor that the defect still exists and further corrective action must be taken. Contractor will continue to use the Defect Tracking Tool and document all actions.

7.3.2 Resolve Defects (Incidents). As stated in the Project RFP, Contractor will correct any defect during system testing by OKDHS within two business days from the time the defect is entered in the Defect Tracking Tool. Contractor must use the approved Change Control process to make corrections. Contractor will immediately report any corrections anticipated to require more than two days to the OKDHS Program Manager for assessment and determination of consequences.

7.3.3 Defect Repair Review. Defect repair review is an action taken by the QA Team to verify that product defects are repaired or replaced and brought into compliance with requirements or specifications. Project Team will make every reasonable effort to minimize the errors that


cause the need for defect repair. A defect log will be used to collect the set of recommended repairs and will be implemented in an automated defect tracking system.

8.0 QUALITY METRICS To verify the delivery of a fully conforming, high-quality product, every person assigned to the Project will participate in quality assurance. This section describes the procedures used by QA Team to verify that the quality assurance provisions of this QA Plan will measure the value of Deliverables and activities. The metrics will relate to business, project, process, and technical operations or performance. Performance metrics must be tracked within the application for an identified set of transactions and be reportable in order to provide information concerning the performance requirements. 8.1 Measurements. Measurements will be made by making a numerical

assignment to specific attributes of each above identified Project component selected for measurement and/or any other Project component that management believes is a candidate for measurement and reporting. The objective of measuring these components is to help management better understand the Project and the relationship of Project components to each other as well as take corrective action to verify the success of the Project, if needed. 8.1.1 Contractor will work with OKDHS in selecting the tools and techniques

for the Quality Management plans and establishing proper and sufficient measurements and metrics to assess the Project. OKDHS and Contractor will also identify, define, and create the Quality Metrics Definitions and Report, which will measure the value of tasks and Deliverables related to the Project.

8.1.2 Records and reports that provide a history of product quality throughout the Project lifecycle will be used to document QA activities. Measurement data collected will be reviewed for trends and process improvement as explained in the next section.

8.1.3 All QA Team records will be collected and maintained for the life of the Project in the established data repositories. Decision Team will indicate whether every activity must start on time or only finish on time and whether individual activities will be measured, or only certain Deliverables and, if so, which ones.

8.2 Monitor and Control. QA Team will verify the monitor and control process is performed to monitor Project processes associated with initiating, planning, executing, and closing. Corrective or preventive actions are taken to control Project performance. Monitoring includes collecting, measuring, disseminating performance information, and assessing measurements and trends to effect process improvements. Continuous monitoring gives the Decision Team insight into the health of the Project, and identifies any areas that may require special attention. The Monitor and Control process includes:


8.2.1 Comparing actual Project performance against the Project Plan. 8.2.2 Assessing performance to determine whether any corrective or

preventive actions are indicated, and then recommending the actions required.

8.2.3 Analyzing, tracking, and monitoring Project risks to make sure the risks are identified, their status is reported, and that appropriate risk action plans are being executed.

8.2.4 Maintaining an accurate, timely information base concerning Project product(s) and associated documentation through Project completion.

8.2.5 Providing information to support status reporting, progress measurement, and forecasting.

8.2.6 Providing forecasts to update current cost and current schedule information.

8.2.7 Monitoring implementation of approved changes when and as they occur.

8.3 Trend Analysis. Contractor will work with OKDHS QA Team to analyze Project performance over time in an effort to prevent recurring discrepancies. QA Team will perform periodic analysis of all reported defects to identify trends that may disclose generic defect areas. These analyses will include the study of the causes, magnitude of impact, frequency of occurrence, and preventive measures. QA Team will report the results of trend analyses along with suggestions for defect resolution and prevention. QA Team, with assistance from Contractor, will provide QA Trend Analysis Report, which will identify trends that will assist with the planning, development, and implementation of Deliverables.

8.4 Process Improvement Analysis. Contractor will work with OKDHS QA Team to create and consolidate the process improvement methodologies, which should detail the steps for analyzing processes that will facilitate the identification of valued and non-valued activities. The methodologies will also provide guides for process improvement activities, such as internal surveys, and process analysis. 8.4.1 Process Improvement Plan details the steps for analyzing processes

that will facilitate the identification of waste and non-value added activity, thus increasing Deliverable value, such as: 8.4.1.1 Process boundaries, which describe the purpose, start, and

end of processes, their inputs and outputs, data required, if any, and the owner and stakeholders of processes.

8.4.1.2 Process configuration, which is a flowchart of processes to facilitate analysis with interfaces identified.

8.4.1.3 Process metrics, which maintain control over the status of processes.


8.4.1.4 Targets for improved performance, which guide process improvement activities.

8.4.2 Lessons Learned Documentation. The causes of variances, the reasoning behind the corrective action chosen, and other types of lessons learned from quality assurance reviews and audits will be documented so that they become part of the historical data for the Project and OKDHS. Lessons learned will be documented throughout the Project lifecycle, and finalized during Project closure.


APPENDIX A - QA TEAM TRAINING

QA TEAM TRAINING

TASK SKILL REQUIREMENTS TYPE OF TRAINING

SOURCE OF TRAINING

Application Software Reviews

Project knowledge, Product knowledge, Source Language knowledge

Peer Review, OJT

TBD - from Training Plan/ Coordinator

Documentation Reviews

Project knowledge, Product knowledge, Subject matter expertise

Peer Review, Workshops, OJT


Process Audits Project knowledge, Product knowledge, Process Improvement knowledge, Project Methodology knowledge

Process Improvement, Six Sigma/Lean


OCT Testing QCT Methodology knowledge, Project knowledge, Product knowledge, Process Improvement knowledge, Project Methodology knowledge

QCT Methodology, Workshops, OJT


QA Management

Project Management, Process Improvement knowledge, Project knowledge, Product knowledge, Project Methodology knowledge

Project Management, Process Improvement, Six Sigma/Lean, OJT

OU Lean Institute, Character First

Metrics Collection

Data Collection skills, Data Analysis skills, Metrics Definition skills, Data Measurement skills, Process Improvement knowledge

Tools Training (EXCEL, Minitab), Process Improvement, Six Sigma/Lean, OJT

OU Lean Institute

Defect Reporting

QCT Methodology knowledge, Project knowledge, Product knowledge, Defect Tracking Methodology knowledge

QCT Methodology Workshops, OJT

TBD - from Training Plan/ Coordinator Corrective

Action

Tool Utilization Product knowledge Training with Product Contractors


Risk Management

Risk Tools knowledge, Risk Identification skills, Data Collection skills, Data Analysis skills, Metrics Definition skills, Data Measurement skills, Process Improvement knowledge

Risk Management Training, Tools Training (EXCEL), Process Improvement, OJT



APPENDIX B - QA TASKS MATRIXES This appendix contains the responsibility matrixes for the tasks identified in Section 2.0 of the QA Plan. Codes that are used include: P = Participant A = Accountable R = Review Required I = Input Required S = Sign-off Required

Evaluate Planning Oversight Sec 2.4

Prog Qual. Mgr/ Lead

QA/ QC Lead

QA Compl.Spec.

Chg Mgt Lead/ Config. Spec.

Risk Mgt Lead/ Risk Spec.

Prog Mgrs

Team Lead

Contract Verification I R, A R, I P P I, S P, A

Feasibility Study I R, A R, I P P I, S P, A

Evaluate Project ManagementSec 2.5


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Project Sponsorship I R, A R, I I, P, A,

S S

Management Assessment I R, A R, I I, P, A,

S I, S

Project Management I R, A R, I I, P, A,

S S

Business Process Reengineer

I R, A R, I I, P, A, S

I, P, A, S

Risk Management I R R, I I, P, A I, P, A,

S I, P, S

Change Management I R R, I I, P, A I, P, S I, A, S

Communica-tion Management

I R R, I I, P, S I, A, S

Configuration Management I R R, I I, P, A I, P, S I, A, S


Evaluate Project ManagementSec 2.5


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Project Estimating/ Scheduling

I R R, I I, P, A, S I, S

Project Staff I R R, I I, P, S I, A, S Project Organization I R R, I I, P, S I, A, S

Sub-Contractors and External Staff, if any

I R R, I I, P, A, S I, S

OKDHS Oversight I R R, I I, P, A,

S I, S

Evaluate Quality Management Sec. 2.6


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Quality Assurance I, A I, P, A R, I I, P I, P I, P, S I, P, S

Process Definition and Product Standards

I, A I, P, A R, I I, P I, P I, P, S I, P, S

Evaluate Training Sec 2.7


QA/ QC Lead

QA Compl. Spec.

TrainingSpec.


ProgrMgrs

Team Lead

User Training and Documentation

I, A I, P, A R, I, S I, P, A I, P I, P I, P

Developer Training and Documentation

I, A I, P, A R, I, S I, P, A I,P I, P I, P


Evaluate Reqmts Mgmt Sec. 2.8


QA/ QC Lead

QA ComplSpec.

Bus. Engr.



Prog Mgrs

Team Lead

Reqmts Mgmt

I, A I, P, A

R, I, S I, P, A I, P I, P I, P, S I, A, S

Security Reqmts

I, A I, P, A

R, I, S I, P, A I, P I, P I, P, S I, A, S

Reqmts Analysis

I, A I, P, A

R, I, S I, P, A I, P I, P I, P I, A

Interface Requmts

I, A I, P, A


Reverse Engring

I, A I, P, A


Evaluate Operating Environ-ment Sec. 2.9


QA/ QC Lead

QA Compl. Spec.

Tech. Spec.

Chg Mgt Lead/ Config.Spec.

Risk Mgmt Lead/ Risk Spec.

Prog Mgrs

Team Lead

System Hardware I, A I, P,

A R, I I, P, A, S I, P I, P I, P I, A

System Software I, A I, P,


Database Software I, A I, P,


System Capacity I, A I, P,


Evaluate Dev'mt Environ-ment Sec. 2.10


QA/ QC Lead

QA Compl.Spec.

Infra-struc-ture Spec.



Prog Mgrs

Team Lead

Dev'mt Hardware I, A I, P,


Dev'mt Software I, A I, P,



Evaluate Software Dev'mt Sec. 2.11


QA/ QC Lead

QA Compl.Spec.

Qual. Control Testing Supv.


QualityControl Testers

Prog Mgrs

Team Lead

High-Level Design I, A I, P,

A R, I I, P, A I, P I, P I, P, S I, A

Detailed Design I, A I, P,


Job Control I, A I, P,


Code I, A I, P, A R, I I, P, A I, P I, P I, P, S I, A

Unit Test I, A I, P, A R, I I, P, A I, P I, P I, P, S I, A

Evaluate System and Acceptance Testing Sec. 2.12


QA/ QC Lead

QA Compl.Spec.

QualityControlTesting Supv.


QualityControl Testers

ProgMgrs

Team Lead

System Integration Test

I, A I, P, A R, I I, P, A I, P I, P I, P I, A

Implementa-tion I, A I, P,

A R, I I, P, A I, P I, P I, A I, A

Benchmark Tests I, A I, P,

A R, I I, P, A I, P I, P I, A I, A

Interface Testing I, A I, P,

A R, I I, P, A I, P I, P I, P I, A

Acceptance and Turnover


Pilot I, A I, P, A R, I I, P, A I, P I, A I, A I, A

Turnover Documenta-tion



Evaluate Data Mgmt Sec. 2.13


QA/ QC Lead

QA Compl. Spec.

Tech. Spec.



Prog Mgrs

Team Lead

Data Conversion I, A I, P, A R, I, S I, P, A,

S I, P I, P I, P I, A

Database Design I, A I, P, A R, I, S I, P, A,

S I, P I, P I, P I, A

Evaluate Operations Oversight Sec. 2.14


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Operational & Business Change Tracking

I, A I, P, A R, I I, P, S I, P I, P I, A

User Operational/ Business Satisfaction


Operational & Business Goals I, A I, P, A R, I I, P, S I, P I, P I, A

Operational & Business Documentation


Operational & Business Processes and Activity


Evaluate Software Products Review Process Sec. 2.15


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Software Product Review I, A I, P, A R, I, S I, P, S I, P I, P I, A

Software Product Eval. I, A I, P, A R, I, S I, P, S I, P I, P I, A


Evaluate Component Deliverable Process Sec. 2.16


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Component Deliverables I, A I, P, A R, I, S I, P, S I, P I, P I, A

Evaluate Media Certif. Storage/ Handling Process Sec. 2.17


QA/ QC Lead

QA Compl. Spec.



Prog Mgrs

Team Lead

Media Certification I, A I, P, A R, I, S I, P, S I, P I, P I, A Storage and Handling

I, A I, P, A R, I, S I, P, S I, P I, P I, A

Evaluate Non-Deliv. Software Certif. Sec. 2.18


QA/ QC Lead

QA Compl. Spec.

Bus. Engineer



ProgMgrs

Team Lead

Non-Deliv. Software I, A I, P,

A R, I, S I, P, S I, P I, P I, A I, A

Evaluate Performance Standards Sec. 2.19


QA/ QC Lead

QA Compl. Spec.

Infra-struc-ture Spec.



Prog Mgrs

Team Lead

Performance Evaluation I, A I, P,

A R, I, S I, P, S I, P I, A I, A I, A


APPENDIX C - PROCESS AUDIT AND REVIEW SCHEDULE PROJECT QUALITY

AUDIT/REVIEW

PLANNED FREQUENCY OR PHASE

QUALITY REVIEW

AUDITOR COMMENTS

DSD Project Compliance Review

Monthly DSD Business Quality Unit

DSD QA Unit will complete Project Quality Audit to assess Project quality. Report and results are submitted to Program Manager and Sponsors.

Decision Team Reviews

Periodically as needed

Decision Team Decision Team will conduct or delegate periodic reviews of funding, resource leveling, and Project performance.

IV&V Audit Every 6 months

IV&V Contractor IV&V Contractor as required by the federal partners will perform audits. Audit items are outlined in IV&V RFP.

Project Quality Reviews

Weekly Quality Assurance Team

Project Management audits and reviews will be performed as determined by Project Team.

Scope and Project Reviews

Periodically as needed

Program Manager Project Manager will monitor all new tasks added to schedule in order to assess scope change.

Procurement Audit As scheduled OIG Auditors Procurement audits may occur at any time during life of the Project.

Cost & Variance Review

As scheduled Program Manager and/or Quality Assurance Team

PM or QA Team will assess noted variances reviewed to determine Project impact.

State Audit As scheduled State Auditor & Inspector

State Auditor may audit books and financial records for compliance with state laws, accounting controls, and government auditing standards.

Earned Value Audit As scheduled Program Manager and/or Quality Assurance Team

PM or QA Team will assess physical work completed and the authorized budget.


PROJECT QUALITY

AUDIT/REVIEW


QUALITY REVIEW

AUDITOR COMMENTS

Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) Review for Certification

As scheduled Federal Auditors ACF may assess whether data provided by computer-based system is reliable and accurate.

Statewide Automated Child Welfare Information System (SACWIS) Review for Certification

As scheduled Federal Auditors Division of State Systems may assess system's functionality requirements.

Treasury Offset Program (TOP) Review for Certification

As scheduled Treasury Department Auditors

Treasury Department may assess compliance with federal tax offset program.

Adoption & Foster Care Analysis and Reporting System (AFCARS) Review for Certification

As scheduled Federal Auditors ACF may assess the ability of system to accurately gather, extract, and submit correct data.

National Child Abuse & Neglect Data System (NCANDS) Review for Certification

As scheduled Federal Auditors Voluntary national data collection and analysis system.

Risk Management Review

As scheduled Quality Assurance Team

QA Team will assess management of risks; documenting, monitoring, and reporting processes are followed.

Change Management Review

As scheduled Quality Assurance Team

QA Team will assess that procedures are followed to identify, report, track, and approve all changes.

Project Management Reviews

TBD Team Leads To review areas of mgmt responsibility in coordination with Quality Assurance Team as needed.

Children and Family Services Review

January 25-29, 2010

Federal Auditors To examine state's capacity and performance improving outcomes for families in child welfare services.


PROJECT QUALITY

AUDIT/REVIEW


QUALITY REVIEW

AUDITOR COMMENTS

Title IV-E Foster Care Eligibility Review

January 25-29, 2010

Federal Auditors ACF may assess accuracy of state claims made on children placed in foster care services.

National Youth in Transition Database (NYTD) Review

Implemented by October 1, 2010

Federal Auditors ACF may assess states' performance in operating independent living programs.

FNS SNAP Post-Implementation Review Section 6.5.3

As scheduled Federal Auditors FNS may conduct review of the systems once it is fully operational.

FNS Systems Functional Requirements Review Section 6.5.4

As scheduled Federal Auditors FNS may conduct a review before and/or during initial Pilot training and before the deployment of software.

FNS Cost Reviews and Audits Section 6.5.5

As scheduled Federal Auditors FNS may conduct review of all cost records relating to system development and operations.

FNS Regional Office Expenditure Review Section 6.5.6

As scheduled Federal Auditors FNS RO will compare reported expenditures for IS development from Form SF-269.

DHHS/ACF CSE System Certification Audit Section 6.6

As scheduled Federal Auditors ACF certification of comprehensive, automated, statewide Child Support Enforcement systems.


APPENDIX D - COMPLIANCY REPORT

Compliancy Report

Task Identifier: _______________ Lead Auditor: ______________________ Date of Report: ____________________ Audit Team: __________________________________________________________ Project Name: _____________________ Date of Audit: _____________________ Process/Procedure Audited: _____________________________________________ Audit Checklist Used: (Attach) ___________________________________________ Audit Findings: (Check one) _____ Process/Procedure Acceptable _____ Process/Procedure Conditionally Acceptable (Subject to satisfactory completion of action items listed below) Conditions noted: _____ Process/Procedure Unacceptable (Subject to satisfactory completion of action items listed below) Conditions Noted:

Action Item # Title:

Assigned to: Due Date: Completed Date:

Corrective Action:

Disposition: Approve Cancel Defer

Signoff: Program Manager: ________________________________ Date: ____________

Audit Closure: Yes No

QA Team Sign-off: _________________________________ Date: ___________


APPENDIX E - SOFTWARE TOOL EVALUATION CHECKLIST Software Tool Evaluation Checklist

QA Team Member: _________________________

Date of Evaluation: ___________

Software Tool Evaluated:

Methods or criteria used in the evaluation:

Evaluation Results:

Recommended Corrective Actions:

Corrective Action Taken:

QA Team Signoff: _______________________________Date:__________________


APPENDIX F - PERFORMANCE STANDARDS EVALUATION CHECKLIST Performance Standards Evaluation Checklist

Quality Assurance Team Member: _______________________________________ Date of Evaluation: __________________ Time of Evaluation: _______________ Location of Evaluation: _______________________________________________ Performance Standard Evaluated: ________ Search Results of new and existing records ________ Refresh and Error Notification ________ Print Results ________ Navigation Results ________ Functionality Report ________ Data Entry/Transaction Results ________ Capacity Results ________ Batch Turnaround Results ________ Other: _______________________


Evaluation Results:



QA Team Sign-Off: ______________________________ Date: _________________


APPENDIX G - PILOT EVALUATION CHECKLIST

Pilot Evaluation Checklist

Quality Assurance Team Member: _____________________________________ Date of Evaluation: ______________________ Time of Evaluation: __________ Location of Evaluation: ______________________________________________ Pilot Criteria Evaluated: ________ Staffing requirements ________ User satisfaction ________ Verification of business processes and workflow ________ Verification of system functionality ________ Operability and stability of software ________ Accuracy of conversion of legacy data and manual data ________ Impact of missing and erroneous data ________ Completeness and accuracy of documentation ________ Effectiveness of training methods and materials ________ Impact on workflow and staff productivity ________ Response time and overall system and network performance ________ System infrastructure performance ________ Appropriateness of system, data, and application security ________ Accuracy and performance of system interfaces and integrated

processes ________ Periodic and Final Pilot communication


Evaluation Results:





APPENDIX H - IMPLEMENTATION EVALUATION CHECKLIST Implementation Evaluation Checklist

Quality Assurance Team Member: ______________________________________ Date of Evaluation: ___________________ Time of Evaluation: _____________ Location of Evaluation: ______________________________________________ Implementation Criteria Evaluated: ________ User satisfaction ________ Verification of system functionality ________ Operability and stability of software and hardware ________ Accuracy of conversion of legacy and manual data ________ Impact of missing or erroneous data ________ Completeness and accuracy of documentation ________ Complete product turnover documents ________ Response time, overall system and network performance ________ System infrastructure performance ________ Security of system, data and application in place ________ Accuracy and performance of system interface ________ Integrated process (accuracy and performance) ________ Training complete ________ Workflow and staff impact


Evaluation Results:



