conformance lynne s. rosenthal mary brady national institute of standards and technology...
TRANSCRIPT
ConformanceConformance
Lynne S. Rosenthal
Mary Brady
National Institute of Standards and [email protected]
OutlineOutline
Terminology Testing Methodologies Benefits of Conformance Testing Test Suites for XML Technologies
– XML, DOM, XSLT,– XSL FO, XML Schema Conformance Testing
Certification
Certificates Branding
Conformanceclause
Test suites Test tools
Validation
Procedures, reporting
Unscrambling the TerminologyUnscrambling the Terminology
Conformance
Specification Conformance Clause
Conformance Testing test suite, test tools
Validation process, reporting
Certification certificate, brand
Terminology - ConformanceTerminology - Conformance
CONFORMANCE - the fulfillment of a product, process or service of specified requirements (ISO Guide 2)
– These requirements are specified in a standard or specification as part of a conformance clause or in the body of the specification
CONFORMANCE CLAUSE - a section of a specification that states all the requirements or criteria that must be satisfied to claim conformance
Terminology - Conformance TestingTerminology - Conformance Testing
CONFORMANCE TESTING – a way to verify
implementations of a specification to determine
whether or not deviations from the specifications exist (through the use of test suites)– tests capture the technical requirements within the specification– testing is bound in scope by the specification – internal structure of product-under-test not accessible (black box
testing)
Necessary, but not sufficient, for interoperability
Note: referred to as conformity assessment in ISO Guide 2
Testing MethodologyTesting Methodology
Falsification Testing– find errors by means of experimentation– show presence of errors not their absence– prove non-conformance; can never prove conformance
Tests Based on:– required functionality
– logical errors by misunderstanding requirements
– errors from boundary conditions and divergence
– common programming errors
OutputInput
Conformance Tests
Requirements
Test programs
Specification
Implementation
Terminology - Testing ProcessTerminology - Testing Process
VALIDATION - process necessary to perform conformance testing in accordance with a prescribed procedure and official test suite– ensures that testing can be repeatable and reproducible– ensures that conclusions are consistent with facts presented
in the evaluation
CERTIFICATION - acknowledgement that a validation was completed and the criteria established by CIO for issuing certificates (brands) was met.
Testing IngredientsTesting Ingredients
Standard Conformance clause
Conformance TestingTest assertions
Test suite (test software, test scripts, test criteria)
Validation Process - policy and procedures for testing
Certificationqualified bodies to do the testing and certification
control board - advisory and arbiter
Benefits of Conformance TestingBenefits of Conformance Testing
Increased probability that products are implemented correctly– Contains required functionality– Behaves as expected– Performs functions in a known manner
Increased likelihood of portability and interoperability – Portability - move software or applications among different
systems – Interoperability – ability of 2+ systems to exchange and use
information
Benefits of Conformance Testing Benefits of Conformance Testing cont.cont.
For Developers of Specifications– Feedback loop
For Software Developers– Help to improve quality - identify areas in which
they conform or not conform For Buyers
– Increases buyer’s confidence in a product and its ability to meet their needs
– Substantiate seller’s claims– Not locked into purchasing from a single vendor
Components of a Test SuiteComponents of a Test Suite
Test Harness
Test Cases
Test Scenario
Test Assertions, Axioms, Semantic Requirements
Specification
OASIS XML Conformance OASIS XML Conformance
XML Conformance Committee– Develop tests to improve the quality of XML
processors– Provide a forum to discuss issues related to
interoperability– Provide input to W3C regarding errors/ambiguities
Deliverables– Test Suite containing over 2000 tests, Second Edition– Associated test report– Built using XML/XSLT
XML Test SuiteXML Test Suite
Types of XML Tests
– Binary
• Validating processors must check validity constraints.
• Non-validating processors do not have to read external entities. Accordingly, they may not report errors which would have been detected had those entities been read.
– Output Tests
• Requirements on XML processors to ensure that they pass the necessary information on to applications.
XML Test MatrixXML Test Matrix
Non-validating
ValidatingExternal Entities Ignored
External Entities Read
Valid Documents Accept Accept Accept
Invalid Documents Accept Accept Accept
Not-wf Documents Reject Reject Reject
WF Errors tied to External Entity
Accept (varies) Reject Reject
Documents with Optional Errors
(not specified) (not specified) (not specified)
XML Test SuiteXML Test Suite
Conformance.xml
XML Test Description FilesTest Case DTD
Testcases.dtd• Test Profile
• Test URI
• Unique ID
• Spec Section
• Type
• Entities
• Description
• NIST / OASIS
• SUN Microsystems
• James Clark
• FUJI Xerox
• IBM
Test Report•XSL Stylesheets•IE5, HTML renditions
<TESTCASES PROFILE=“SUN Microsystems XML Tests”><TEST URI=“sun/invalid/e101.xml” ID=“e101” SECTIONS=“3 [39]” TYPE=“invalid” ENTITIES=“none”>Tests the Element Valid VC by including an undeclared child element.</TEST></TESTCASES>
XML.COM• Java application• Tests XML processors• Dave Brownell
Outputs
XML Test Suite Results, v1XML Test Suite Results, v1
XML.COM Conformance Update – Dave Brownell Validating Parsers
– Tested JAXP 1.0, IBM Xerces/Java 1.0.3, Layered Validator (distributed with Aelfred 2), MSXML3 Technology Preview, Oracle V2.0.2.7
– Pass Rate 82% - 99% Non-validating Parsers
– Tested all of the above plus XP 0.5– Pass rate 86% - 99%
Microsoft Update – July 2000 MSXML 3.0 Beta – Chris Lovett– Non-validating: DOM: 95%, SAX: 97%– Validating: 97%
NIST DOM Test SuiteNIST DOM Test Suite
Test Harness – automatically runs available tests and presents user witha color-coded table of results; gives access to spec, assertions, and source code
Test ScenarioLoad appropriate XML or HTML file, exercise DOM calls
Test Assertions – Fundamental, Extended, HTML Interfaces
DOM L1 Specification
800 ECMAScript fundamental, extended, HTML
200 Java – fundamental, extended
Test Cases
DOM Test SuiteDOM Test Suite
Test Results:– Run tests against IE5, Mozilla (some fundamental
interfaces), Xerces, JAXP, and Oracle; Pass rate: 82 – 98%– Implementations differ greatly in the way that parsers handle
white-space and entity expansion issues – differences in DOM tree
DOM Level 2– Working with W3C to identify resources – Modified Java tests to be consistent with SUN contribution– Will update NIST tests to L2 status
XSLT Test SuiteXSLT Test Suite
OASIS Technical Committee, chaired by G. Ken Holman
Member contributions include Lotus, NIST, Sun
Expands on XML Test Description file Dave Marston, XSLT/XPATH Conformance
Wed, 4pm, Delaware Room Public Information Meeting, Wed, 8pm, Suite
8228
XSL FO Test SuiteXSL FO Test Suite
W3C WG to test Candidate Recommendation Develop test assertions for all basic formatting
objects and properties Develop test DTD and associated harness Work with vendors to categorize available tests Research translating available CSS and XSLT
tests Develop tests for additional spec components Expected March 2001
XML Schema Test SuiteXML Schema Test Suite
Test Control
• Test Types• # schemas• # instances
Test Control
• Test Types• # schemas• # instances
Constraints
• Structure• Limits• Special info
Constraints
• Structure• Limits• Special info
Schema Tests
Test GeneratorTest Generator
SummarySummary
Interesting URL’s– http://www.nist.gov/xml/– http://www.w3.org/– http://www.oasis-open.org/
Questions???