world challenges for software quality -...

International Conference on Software Quality - ICSQ 07

World Challenges for Software Quality

Taz DaughtreyExecutive Director, WCSQ

Harrisonburg, Virginia

[email protected]


PanelistsJoe Jarzombek

Mark Paulk

Shinji Fukui

Watts Humphrey


Organizations struggle to produce software products

… that meet and exceed expectations

… on predictable schedules

… for acceptable costs.


Seeking Consensus onSoftware Quality Principles

peoplediscussion process

draft outputsexperiencesinformed by their


What and Why• Starting last year, Taz Daughtrey initiated an effort to define the critical challenges in software.

– He contacted many people for suggestions.– There was widespread interest and participation.

• We decided to first define a set of software quality principles. – The principles of software quality have not been clearly stated.– These principles are neither well understood or widely followed.– This is a principal reason for the consistently poor quality

performance of the software industry.

• Our objective is to provide a clear statement of quality principles for– Software developers and their management– Software users and acquirers– The academic community


The ParticipantsUnder the guidance of Taz Daughtrey, organizer of the World Congress for Software Quality, those who have participated in this effort include:

Gary GackTom GilbWatts HumphreyJoe JarzombekCapers JonesStephen KanHerb KrasnerGary McGrawPatricia McQuaidMark PaulkColin TullyJerry Weinberg


Software Quality PrinciplesThere are five overall principles, which each have further elaboration.

• Properly managed quality programs reduce total costs, increasequality and business value, and shorten development times.

• To get quality work, the customer must demand it.

• The developers must feel personally responsible for the quality of the products they produce.

• For the proper management of software development projects, thedevelopment teams themselves must plan, measure, and control thework.

• Software management must recognize and reward quality work.


#1: Quality PaysProperly managed quality programs reduce total costs, increase quality and business value, and shorten development times.

• If development cost or time increases, the quality program has not been properly implemented.

• Product size, including all changes, must be estimated and tracked.

• Schedules, budgets, and quality commitments must be based on sound methods and data.

• The development strategy must be consistent with requirements stability.


#2: Customer Quality DemandsTo get quality work, the customer must demand it.

• The product’s quality attributes must be stated in measurable terms.

• The customers and developers must agree on the quality specifications

• Any deviations are to be considered defects.

• The development contract must quantitatively specify the required product quality and how it is to be tracked.


#3: Developer ResponsibilityThe developers must feel personally responsible for the quality of the products they produce.

• Development teams must plan their own work.

• Development teams must negotiate their commitments with management and the customer.

• Software managers must provide appropriate technical and plan-management training for the developers.

• A developer is anyone who produces part of the product: designer, documenter, coder, systems designer.


#4: Self-Directed TeamsFor the proper management of software development projects, the development teams themselves must plan, measure, and control thework.

• Project teams must have suitable domain and technology knowledge and skill.

• Removal yield at each step must be measured and tracked.

• All development effort must be measured and tracked.

• All defects must be recorded and the defect data analyzed.

• Measurements must be recorded by those performing the work – both managers and developers.


#5: Quality RecognitionSoftware management must recognize and reward quality work.

• In crisis-driven organizations, quality work is invisible.

• Projects must use multiple appraisal methods to verify agreed defect levels.

• Managers must use measures to assess and improve process quality.

• Managers must respect personal data.


Software Quality Body of Knowledge


The SQuBOK®

October 17th, 2007

JUSE SQiP [Software Quality Profession] Committee


What’s SQuBOK® Guide ? (1)• SQuBOK Guide is…

– “Guide to the Software Quality Body of Knowledge”• Objective of SQuBOK

1. To provide training materials for software quality engineers.2. To make the Japanese “Tacit knowledge” regarding

Software Quality to “Explicit knowledge”.3. To summarize & to provide access to the Software Quality

Body of Knowledge.4. To promote a consistent view of software quality engineering.5. To support organizations which are trying to establish their

own software management processes.


What’s SQuBOK® Guide ? (2)• Position of SQuBOK

– The SQuBOK doesn’t replace the other BOKs such as SWEBOK and PMBOK, but does abstract the knowledge of Software Quality.

– The SQuBOK does not offer the Body of Knowledge itself, but provide access to the available knowledge in the published literature.

• SQuBOK project– Established under 2 parties as follows

• JUSE (Union of Japanese Scientists and Engineers)• JSQC (Japanese Society for Quality Control)

– Project Approach• “Fast Track” Approach, release an Alpha as quick as

possible, and take a review by Japanese Software Quality Leaders.


The Structure of SQuBOK®

Guide to the Software QualityBody of Knowledge

Beta Version

Software QualityManagement

Software QualityMethodologies

Fundamental Conceptsof Software Quality

Topic　(220)Knowledge sub- area (60)

Knowledge Area (26)

Category (3)


The Tree Structure of SQuBOK®

Note: (*1) means " To BeDescribed in the Version 2"

Guide to the Software Quality Body of KnowledgeBeta Version

1. Fundamental Conceptsof Software Quality

1.1 Concept of Quality

1.2 Management of Quality

Organizational Level SoftwareQuality Management

2.1 Quality Management SystemEstablishment / Maintenance

2.2 Life Cycle Management2.3 SPA/ SPI Management

2.4 V&V Management

2.5 Audit Management

2.6 Education / Training Management

2.7 Legal Responsibly Management

2.8 Decision Management

2.9 Acquisit ion Management

2.10 Configuration Management

2.11 Risk Management

2.12 Project Management

Requirements Analysis Management (*1)

Implementation Management(*1)

2.14 Review Management

2.15 Testing Management

2.17 Quality Analysis / Evaluation Management2.16 Quality Analysis & Evaluation Management

3.1 Metrics

　Category

Sub-Category

2.Software Quality Management 3. Software QualityMethodologies

Project Level Software QualityManagement (Phase Common)

Project Level Software QualityManagement (Phase Unique)

3.3 Requirements Analysis Methods

Architecture & Design Methods(*1)

3.4 Review Methods

Implementation Methods (*1)

3.5 Testing Methods

3.6 Quality Analysis & Evaluation Methods

3.7 Operations & Maintenance Methods

KnowledgeArea

Architecture & Design Management (*1)

2.13 Project Management

2.17 Operations & Maintenance Management

3.2 Quality Planning Methods


The Current Status

• The first Beta version was released for Public Comment.

• The first version of SQuBOK Guide is planed to publish by the end of this year.

• English translation is planed, but schedule is under planning.


World

Challenges for

Software

Quality


The Future of Quality Initiatives

• Best practice frameworks• Improvement philosophies• Navigating the quagmire• Keeping an eye on the reason we care


Evolving Quality Philosophies

1900 1925 1950 1975 2000

Taylor’s Scientific Management

Industrial engineering

World War II quality initiatives

Shewhart’s statistical process control

Deming, Juran, … in Japan

Quality rediscovered as TQM in USA


Quality and Process Frameworks

1960 1970 1980 1990 2000

DOD, NATO, NASA, … quality standardsDeming Prize

Baldrige AwardISO 9001 (Quality management systems)

ISO 15504 (Process assessment)ISO 12207 (Software life cycle processes)

Capability Maturity Model for Software

CMM Integration

eSourcing Capability Model for Service Providers


Taxonomy of Improvement Frameworks

• Principle-based improvement – Shewhart, Deming, Juran, …

• “Best practices” frameworks– binary sets of requirements (ISO 9001)– points-based ratings (Baldrige)– staged improvement of organizational capability (Software

CMM)– continual representation for process capability (ISO 15504)

• ⇒ continual, measurable improvement


The Quagmire…J.W. Moore, “An Integrated Collection of Software

Engineering Standards,” IEEE Software, November/December 1999.

• 315 standards, models, guidelines developed by 46 different organizations

• Baldrige• CBA IPI• CMMI• CobiT• DOD-STD-2167A• DOD-STD-2168• DOD-STD-7935A• EIA 632 • EIA 731• eSCM• FAA iCMM• IEEE 1220 • IEEE/EIA 12207• IPD-CMM• ISO 15939• ISO 9000

• ISO/IEC 12207• ISO/IEC 15288• ISO/IEC TR 15504• J-STD-016• MIL-STD-498• MIL-STD-499B• People CMM• PSM• PSP• Q9000• QS 9000• RTCA DO-178B• SA-CMM• SCAMPI• SCE• SDC/CR• SDCE• SECAM• SE-CMM• SW-CMM• TL 9000• TSP• …


Drivers for the Quagmire

• Customer dissatisfaction with the “software crisis”

• Standish Group (2000) – 23% of software projects failures – 49% of software projects challenged

• Standish Group (1994)– 31% failures– 53% challenged


Drivers: Credentials, Statistical Thinking

• Best practice frameworks for certification and to provide a foundation for high quality

• Cross-certification to minimize cost and maximize value– systematic inclusion of various

assessment, evaluation, and audit results

• Measurable improvement + benchmarks against industry norms


DHS Software Assurance Program Overview• Program based upon the National Strategy to Secure

Cyberspace - Action/Recommendation 2-14: “DHS will facilitate a national public-private effort to promulgate best practices and methodologies that promote integrity, security, and reliability in software code development, including processes and procedures that diminish the possibilities of erroneous code, malicious code, or trap doors that could be introduced during development.”

• DHS Program goals promote the security of software across the development, acquisition and implementation life cycle

• DHS Software Assurance (SwA) program is scoped to address:– Trustworthiness - No exploitable vulnerabilities exist, either maliciously or

unintentionally inserted– Predictable Execution - Justifiable confidence that software, when

executed, functions as intended– Conformance - Planned and systematic set of multi-disciplinary activities

that ensure software processes and products conform to requirements, standards/ procedures

CNSS Instruction No. 4009, "National Information Assurance Glossary," Revised 2006, defines Software Assurance as: "the level of confidence that software is free from vulnerabilities, either intentionally designed into the software or accidentally inserted at anytime during its lifecycle, and that the software functions in the intended manner".

Also See Wikipedia.org for “Software Assurance”

Defects

IntentionalVulnerabilities

UnintentionalVulnerabilities

Note: Chart is not to scale – notional representation -- for discussions

Software Assurance Addresses Exploitable Software:Outcomes of non-secure practices and/or malicious intent

EXPLOITABLE SOFTWARE

Exploitation potential of vulnerability is independent of “intent”

*Intentional vulnerabilities: spyware & malicious logic deliberately imbedded (might not be considered defects)

Malware

Disciplines Contributing to Software Assurance*

In Education and Training, Software Assurance could be addressed as:• A “knowledge area” extension within each of the contributing disciplines;• A stand-alone CBK drawing upon contributing disciplines;• A set of functional roles, drawing upon a common body of knowledge; allowing more in-depth coverage dependent upon the specific roles.

Intent is to provide framework for curriculum development and evolution of contributing BOKs

Safety & Security

Project Mgt

Software Acquisition

Software Engineering

Software Assurance

Systems Engineering

Information Assurance

* All require Measurement; see ‘Notes Page’ view for contributing BOK URLs and relevant links

*Info Systems Security Eng

*Test & Evaluation

The intent is not to create a new profession of Software Assurance; rather, to provide a common body of knowledge: (1) from which to provide input for developing curriculum in related fields of study and (2) for evolving the contributing disciplines to better address the needs of software security, safety, dependability, reliability and integrity.

Most Important Attributes

0

11

27

32

55

70

95

0 20 40 60 80 100

Other

Rich Feature Set

Convenience & Ease of Use

Software conforms to Requirements& Industry Standards

Ease of Integration & Configuration

Software free from securityvulnerabilities and malicious code

Reliabile software that functions aspromised

Percent

Summary from Survey• Reliable software & vulnerability-free software are the top priorities;• CIOs have low-medium confidence in software’s ability to be free of

flaws, security vulnerabilities and malicious code flaws;• Eighty-six percent (86%) of CIOs rate the fundamental security of

software as vulnerable or extremely vulnerable;• The majority have had to redeploy staff, incur increased IT costs and

suffer reduced productivity due to software flaws;• Internal testing, contracts/SLAs and reputation among peers are the

most preferable means for CIOs to determine if software is free of flaws;

• The majority would like vendors to certify software meets a designated security target and to scan for flaws and security vulnerabilities using qualified tools

Industry Lacks Labels for Software


Quality without Security: Vulnerable Software Enables Exploitation

• Rather than attempt to break network or system security, hackers are opting to target application software to circumvent security controls rather than attempting to defeat them.

– This is the case with most exploitable software vulnerabilities related to insecure coding practices.

• Functional Correctness must be exhibited even when software is subjected to abnormal and hostile conditions; therefore, “In an era riddled with asymmetric cyber attacks, claims about system reliability, integrity and safety must also include provisions for built-in security of the enabling software.”

Software applications with exploitable vulnerabilities

Software applications with exploitable vulnerabilities

SECURITY


Why Software Assurance is Critical

• Software is the core constituent of modern products and services – it enables functionality and business operations

• Dramatic increase in mission risk due to increasing:– Software dependence and system interdependence (weakest link syndrome)– Software Size & Complexity (obscures intent and precludes exhaustive test)– Outsourcing and use of un-vetted software supply chain (COTS & custom)– Attack sophistication (easing exploitation)– Reuse (unintended consequences increasing number of vulnerable targets)– Number of vulnerabilities & incidents with threats targeting software– Risk of Asymmetric Attack and Threats

• Increasing awareness and concern

Software and the processes for acquiring and developing softwarerepresent a material weakness


SW Assurance related to Engineering Disciplines

*Adopted from Jim Moore, IEEE CS S2ESC Liaison to ISO SC7Predictable Execution = Resilient characteristic

For a safety/security analysis to be valid …

The execution of the system must be predictable.

This requires …

– Correct implementation of requirements, expectations and regulations.

– Exclusion of unwanted function even in the face of attempted exploitation.

Traditional concern

Growing concern

System and SWEngineering and

Information Systems Security Engineering

InformationAssurance

System Safety

Predictable Execution

Cyber Security


Software Assurance Forum and Working Groups *

PeoplePeople

Developers and users education & training

ProcessesProcesses

Sound practices, standards, & practical guidelines for secure software development

TechnologyTechnology

Security test criteria, diagnostic tools, common enumerations, SwA R&D, and SwAmeasurement

AcquisitionAcquisition

Software security improvements through due-diligence questions, specs and guidelines for acquisitions/ outsourcing

… encourage the production, evaluation and acquisition of better quality and more secure software through targeting

Products and ContributionsProducts and ContributionsBuild Security In - https://buildsecurityin.us-cert.govand SwA community portal – http://us-cert.gov/SwA

SwA Common Body of Knowledge (CBK) & Glossary SwA Developers' Guide on Security-Enhancing SDLC

Software Security Assurance State of the Art Report

Systems Assurance Guide (via DoD and NDIA)

SwA-related standards – ISO/IEC JTC1 SC7/27/22, IEEE CS, OMG, TOG, & CMM-based Assurance

Practical Measurement Guidance for SwA/InfoSec

SwA Metrics & Tool Evaluation (with NIST) SwA Ecosystem w/ DoD, NSA, NIST, OMG & TOG NIST Special Pub 500 Series on SwA Tools

Common Weakness Enumeration (CWE) dictionary Common Attack Pattern Enumeration (CAPEC) Malware Attribution & Enumeration (with ASC)

SwA in Acquisition: Mitigating Risks to EnterpriseSoftware Project Management for SwA SOAR

* SwA Forum is part of Cross-Sector Cyber Security Working Group (CSCSWG) established under auspices of the Critical Infrastructure Partnership Advisory Council (CIPAC) that provides legal framework for participation.

Software Assurance Resources• DHS National Cyber Security Division serves as a focal point for software assurance,

facilitating national public-private efforts to promulgate best practices and methodologies that promote integrity, security, and reliability in software development and acquisition.

• Collaborative efforts of the Software Assurance (SwA) community have produced several publicly available resources:

– SwA Common Body of Knowledge with Guiding Security Principles (curriculum development guide, update available Oct 2007);

– Securing the Software Lifecycle: Making Application Development Processes - and Software Produced by Them - More Secure, v2.0 (developer’ guide available Dec 07);

– State-of-the-Art Report on Software Security Assurance at http://iac.dtic.mil/iatac/download/security.pdf;

– Practical Measurement Guidance for SwA and InfoSec, v1.0 (Measurement Guide to support information needs; draft update available Dec 2007);

– SwA in Acquisition: Reducing Risks to the Enterprise, v1.0 (procurement guide -https://buildsecurityin.us-cert.gov/daisy/bsi/resources/dhs/908.html)

– State-of-the-Art Report on Software Project Management for Software Assurancehttps://buildsecurityin.us-cert.gov/daisy/bsi/resources/dhs/906.html

– Common Attack Pattern Enumeration and Classification (CAPEC -http://capec.mitre.org), and

– Common Weakness Enumeration (CWE - http://cwe.mitre.org) with links to the National Vulnerability Database - http://nvd.nist.gov/nvd.cfm.

• For more information, see Build Security In web site https://buildsecurityin.us-cert.gov/ and the Software Assurance (SwA) Community of Practice portal http://www.us-cert.gov/swa

Software Security Assurance: A State of the Art Report, 31 July 2007

IATAC, an Information Analysis Center in Defense Technical Information Center

• Publicly available resource provides a comprehensive look at efforts to improve the state of Software Security Assurance:

– describes the threats and common vulnerabilities to which software is subject;

– presents the many ways in which the S/W Security Assurance problem is being framed and understood across government, industry, and academia;

– describes numerous methodologies, best practices, technologies, and tools currently being used to specify, design, and implement software that will be less vulnerable to attack, and to verify its attack-resistance, attack-tolerance, and attack-resilience;

– offers a large number of available print and online resources from which readers can learn more about the principles and practices that constitute Software Security Assurance;

– provides observations about potentials for success, remaining shortcomings, and emerging trends across the S/W Security Assurance landscape.

• Free via http://iac.dtic.mil/iatac/download/security.pdf

•The SOAR reflects output of efforts in the DoD-DHS Software Assurance Forum and Working Groups that provide collaborative venues for stakeholders to share and advance techniques and technologies relevant to software security.

Launch http://us-cert.gov/SwA for Software Assurance Community of Practice (Oct 07)

Build Security In •Will continue as a related website (on same server)•Will continue to serve as a detailed reference source for developers•Will continue to be a part of the SwA Processes & Practices WG

SwA WORKING GROUPS•SwA WGs are created to give focus to specific areas within the effort. •More description would be provided for the specific efforts. -- A comprehensive description would provide information to the user

to determine what is the purpose of WGs and what they are like. -- It could also reference results of the working group activity here in

this area as an example. -- It will outline the different levels of participation: active & observer.

Matrix provides linkage among SwA WORKING GROUPS and SwA FOCUS AREAS -- Enables more effective navigation and access to relevant material

Serving broader stakeholder community

http://us-cert.gov/SwA

Process Agnostic Lifecycle

Architecture & DesignArchitectural risk analysisThreat modelingPrinciplesGuidelinesHistorical risksModeling toolsResources

CodeCode analysisAssembly, integration & evolutionCoding practicesCoding rulesCode analysisResources

TestSecurity testingWhite box testingAttack patternsHistorical risksResources

SystemPenetration testingIncident managementDeployment & operations Black box testingResources

RequirementsRequirements engineeringAttack patternsResources

FundamentalsRisk managementProject managementTraining & awarenessMeasurementSDLC processBusiness relevanceResources

KeyBest (sound) practicesFoundational knowledgeToolsResources

https://buildsecurityin.us-cert.gov

Touch Points & Artifacts

Launched 3 Oct 2005

https://buildsecurityin.us-cert.gov/portal/index.html

Role of Assurance CaseLife Cycle Processes

Requirements Analysis

Risk Management

Measurement

Project Assessment and Control

Assurance Case

Assurance Issues

Assurance Risks, Threats,

Hazards, etc

Assurance Measurements

Assurance Requirements

Project Planning

Assurance Plan

Transition

Operation

Maintenance

Changes in Operational Characteristics

Maintenance Updates

Operational Constraints

Life Cycle Processes

Requirements Analysis

Risk Management

Measurement

Project Assessment and Control

Assurance Case

Assurance Issues

Assurance Risks, Threats,

Hazards, etc

Assurance Measurements

Assurance Requirements

Project Planning

Assurance Plan

Transition

Operation

Maintenance

Changes in Operational Characteristics

Maintenance Updates

Operational Constraints

ISO/IEC 15026 System & Software Assurance

Evidence

Arguments

Claimssupports

justify belief inQuality / Assurance Case

Make the case for adequate quality/ assurance of the

System, Software, or Work Product

Quality / AssuranceFactor

Quality / AssuranceSubfactor

is developed for

What constitutes sufficient Evidence to support Arguments that justify Claims?

How might “scaling”be structured to enable and encourage more suppliers and acquirers to make use of assurance cases?

Assurance Case --ISO/IEC 15026 System & Software Assurance

Adopted from US TAG ISO/IEC 15026 proposal May 2007 and CMU SEI QUASAR tutorial by Donald Firesmith, March 2007

Bi-Monthly Working Groups & Semi-Annual SwA Forum:Next WG sessions held 4-6 Dec 2007 – Next SwA Forum 12-13 Mar 2008

Typical Format Tuesday Wed Thursday

Session 1:Technology, Tools & Product Evaluation

Working Group

Session 6:Processes & Practices

Working Group on “Standards & Models”

Session 2:Business CaseWorking Group

Joint Session 8:Measurement WG with

another SwA WG

Session 1:Technology, Tools & Product Evaluation

Working Group

Session 4:Malware

Working Group

Session 6:Processes & Practices

Working Group on “Standards & Models”

Session 3:Workforce Education & Training Working Group

Session 5:Acquisition

Working Group

Session 7:Measurement Working

Group

Afternoon1pm - 5pm

Plenary SessionMorning9:00am -11:30am

Presentations from previous SwA WGs and Forums are on US-CERT Portal (https://us-cert.esportals.net/) under the appropriate Working Group in the Library folder. Access to WG folder is restricted to those who have participated in the WG. Contact DHS NCSD if you do not yet have access to the appropriate folders.

DHS Software Assurance (SwA) Outreach• Co-sponsor bi-monthly SwA WG sessions (next 4-

6 Dec) and semi-annual Software Assurance Forum for government, academia, and industry to facilitate ongoing collaboration -- 2-3 Oct 2007 with next 12-13 Mar 2008

• Co-sponsor SwA issues of CROSSTALK (since Oct 05); provide SwA articles in other journals to “spread the word” to relevant stakeholders

– March 2007 issue on “Software Security”– May 2007 issue on “Software Acquisition”– Sep 2007 issue on “Service Oriented Architecture”

• Provide free SwA resources via “BuildSecurityIn”portal to promote relevant methodologies (since Oct 05)

• Launch http://us-cert.gov/SwA for SoftwareAssurance Community of Practice (Oct 07)

• Provide DHS Speakers Bureau speakers• Support efforts of consortiums and

professional societies in promoting SwA


Software Quality Challenges

“Software quality hasn’t been tried and found wanting …. It has been found hard and not tried.”


Software Quality Challenges

Devise and implement incentives for applying quality principles.

Professionalize software quality, especially organizationally.

Develop a true engineering discipline for defect prevention.

Move beyond testing as a primary quality management method.

Conduct and report rigorous quantitative research.


www.asq.org/conferences/wcsq

4th World Congress for Software Quality

September 15-18, 2008Hyatt Regency Bethesda

Bethesda, Maryland


World Challenges for Software Quality

Taz DaughtreyWorld Congress for Software Quality 434 237 2723 voice/fax

James Madison University 540 568 2778 voice

[email protected]

world challenges for software quality -...

Documents