patrick jcp
DESCRIPTION
TRANSCRIPT
![Page 1: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/1.jpg)
1http://jcp.org
Conformance testing and standards
How do you know it worksif you don't know what it's supposed to do?
Patrick CurranJCP [email protected]
![Page 2: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/2.jpg)
2http://jcp.org
Standards make the world go round
![Page 3: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/3.jpg)
3http://jcp.org
Language
![Page 4: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/4.jpg)
4http://jcp.org
Writing
![Page 5: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/5.jpg)
5http://jcp.org
Number systems
![Page 6: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/6.jpg)
6http://jcp.org
Currency
![Page 7: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/7.jpg)
7http://jcp.org
Time...
![Page 8: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/8.jpg)
8http://jcp.org
And space
![Page 9: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/9.jpg)
9http://jcp.org
Weights and measures
![Page 10: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/10.jpg)
10http://jcp.org
Machine tools
![Page 11: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/11.jpg)
11http://jcp.org
Screws and threads
![Page 12: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/12.jpg)
12http://jcp.org
Interchangeable parts
![Page 13: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/13.jpg)
13http://jcp.org
Railways
![Page 14: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/14.jpg)
14http://jcp.org
Mass production
![Page 15: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/15.jpg)
15http://jcp.org
Shipping
![Page 16: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/16.jpg)
16http://jcp.org
Traffic
![Page 17: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/17.jpg)
17http://jcp.org
Beer
![Page 18: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/18.jpg)
18http://jcp.org
Chocolate
WHO/FAO: Codex Alimentarius Official Standard for Chocolate
![Page 19: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/19.jpg)
19http://jcp.org
Music
ISO 16:1975 Acoustics -- Standard tuning frequency(Standard musical pitch)
![Page 20: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/20.jpg)
20http://jcp.org
Color
http://www.color.org
![Page 21: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/21.jpg)
21http://jcp.org
Sport
http://www.lords.org/laws-and-spirit/laws-of-cricket/laws/
![Page 22: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/22.jpg)
22http://jcp.org
Medicine
Chronic rheumatic heart diseases I05: Rheumatic mitral valve diseases
Includes:conditions classifiable to 105.0 and 105.2-105.9, whether specified as rheumatic or not Excludes:when specified as nonrheumaticI05.0: Mitral stenosis Mitral (valve) obstruction (rheumatic) I05.1: Rheumatic mitral insufficiency Rheumatic mitral● Incompetence●RegurgitationI05.2: Mitral stenosis with insufficiency Mitral stenosis with incompetence or regurgitation I05.8:Other mitral valve diseases Mitral (valve) failure I05.9: Mitral valve disease, unspecified Mitral (valve) disorder (chronic) NOS
From the World Health OrganizationInternational Classification of Diseases
![Page 23: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/23.jpg)
23http://jcp.org
Shopping
![Page 24: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/24.jpg)
24http://jcp.org
Home entertainment
![Page 25: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/25.jpg)
25http://jcp.org
The Holy Grail
•Components, systems, and processes that are:–Reproducible–Predictable–Reusable– Interoperable– Interchangeable
![Page 26: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/26.jpg)
26http://jcp.org
Then
![Page 27: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/27.jpg)
27http://jcp.org
Now
• We are no longer willing to buy all of our hardware and software from a single supplier.
• We want the freedom to chose and the option to switch.– All systems are heterogeneous.
• This requires standards.– For interfaces, so we can mix and match components.– For protocols, so systems can talk to each other.
![Page 28: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/28.jpg)
28http://jcp.org
Interfaces
![Page 29: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/29.jpg)
29http://jcp.org
Protocols
![Page 30: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/30.jpg)
30http://jcp.org
No vendor lock-in
![Page 31: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/31.jpg)
31http://jcp.org
Specifications...
![Page 32: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/32.jpg)
32http://jcp.org
Are not enough...
• Interoperability and interchangeability are harmed by: • Poor-quality specs.– Imprecise or ambiguous, language.
• Poor-quality implementations.– Specified requirements are not met.– Specified requirements are implemented incorrectly.
![Page 33: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/33.jpg)
33http://jcp.org
We also need testing
![Page 34: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/34.jpg)
34http://jcp.org
Conformance testing
• The process of verifying that implementations of a technology conform to the specification.
• Tests only what is normatively required in the specification.– Quality, robustness, performance, usability, and other
desirable attributes of software must not be tested (unless explicitly specified.)
• Can make no assumptions about the internals of the implementation (black-box testing.)
• Improves the quality of specifications:– by identifying ambiguities, contradictions, omissions,
• And of implementations:– by identifying failures to conform to the spec.
![Page 35: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/35.jpg)
35http://jcp.org
What makes a good spec?
• Specify.– Unspecified or implementation-specific behaviour can't
be tested.• Require.– In clear, unambiguous language (see RFC 2119)– We like “must,” “shall,” “must not”...– We don't like “may,” “it's obvious,” “it's up to you”...
• Beware optional functionality.– Can be tested, but doesn't promote interoperability or
application portability.• Developers won't know what they can depend on.
– If you must, clearly define optionality with Profiles.
![Page 36: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/36.jpg)
36http://jcp.org
Java: what?
Open, standards-based technologies enabling the development and deployment of secure, portable, reliable, and scalable applications on hardware platforms from cellphones to high-end servers.
![Page 37: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/37.jpg)
37http://jcp.org
Java: how?
![Page 38: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/38.jpg)
38http://jcp.org
Deliverables
![Page 39: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/39.jpg)
39http://jcp.org
Deliverables
![Page 40: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/40.jpg)
40http://jcp.org
Java conformance testing
![Page 41: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/41.jpg)
41http://jcp.org
Why conformance-test?
• To promote the compatibility and interoperability of Java technology implementations.
• To ensure that the technologies are well specified and that implementations conform to the specifications.
• Results:– Multiple compatible implementations are available.– Developers know how implementations will behave.
![Page 42: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/42.jpg)
42http://jcp.org
Compatibility
• Compatibility is a contractual obligation.– Shipping incompatible products is prohibited.
• Compatible products can use the Java name and display the Java Compatible logo.
• Compatibility is binary.– You can't be “almost compatible”
or “a little bit incompatible.”– You must pass all the tests and meet all of the
compatibility requirements.
![Page 43: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/43.jpg)
43http://jcp.org
Planning and building a high-quality TCK
![Page 44: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/44.jpg)
44http://jcp.org
A TCK is not just a collection of tests
• It should also include:– A test harness to automate execution.– Documentation explaining• How to run the test suite.• How to interpret test results.• Compatibility Requirements (The Rules.)• The test appeals process.
• It must be portable.– Unlike most other software, a TCK must be capable of
running on systems that don't yet exist.• You can't test it on the platforms where it will be run!
• The Spec Lead must commit to ongoing maintenance.– Fix bugs, expand coverage.
![Page 45: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/45.jpg)
45http://jcp.org
A good test is...
• Mappable to the specification.– You must know what portion of the specification it tests.
• Atomic.– Tests a single feature rather than multiple features.
• Self-documenting.– Explains what it is testing and what output it expects.
• Focused on the technology under test rather than on ancillary technologies.
• Useful.– Likely to catch real-world problems.
• Correct, efficient, portable, and maintainable.
![Page 46: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/46.jpg)
46http://jcp.org
Specification markup
• Identify normative requirements (test assertions) within the spec.
• Provide feedback to the authors where the spec is ambiguous, contradictory, incomplete, or untestable.
• Publish an assertion list.– Ask the spec authors to review and approve it.
• This process significantly improves spec quality.
![Page 47: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/47.jpg)
47http://jcp.org
Test assertions
• A specific statement of functionality or behavior derived from a specification.– java.lang.Integer.toString(int i, int radix)• "If the radix is smaller than Character.MIN_RADIX
or larger than Character.MAX_RADIX, then the radix 10 is used instead."
– “During preparation of a class or interface C, the Java virtual machine also imposes loading constraints (§5.3.4). Let L1 be the defining loader of C. For each method m declared in C that overrides a method declared in a superclass or superinterface, the Java virtual machine imposes the following loading constraints: Let T0 be the name of the type returned by m, and let T1, ..., Tn be the names of the argument types of m. Then TiL1=TiL2 for i = 0 to n (§5.3.4).”
![Page 48: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/48.jpg)
48http://jcp.org
How many tests are enough?
• There is no simple answer to this question.– It depends on your goals and on the available resources.
• Aim to get the best possible coverage with the resources you have available.
• You cannot do this unless you set explicit goals, and measure or estimate test coverage.
![Page 49: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/49.jpg)
49http://jcp.org
Calculating coverage
![Page 50: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/50.jpg)
50http://jcp.org
Coverage analysis
• Partition the spec.– By feature, APIs, language elements, testable assertions,
logical sections, or even pages or paragraphs.• Estimate or measure the extent of coverage in each area• Breadth coverage (relatively simple)–What percentage of spec elements are covered by at
least one test?• Depth coverage (more subjective)– On average, what percentage of the tests that would be
required to completely test each element have actually been written? • (How thoroughly is each element tested?)
![Page 51: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/51.jpg)
51http://jcp.org
Test development strategy
• Define coverage goals.– Where should resources be focused?– How extensively should each area be tested?
• Start with breadth (test everything minimally.)• Drill down (increase depth coverage) in selected areas.• Publish a test-coverage report.– Minimally, map tests to areas of the spec.– Ideally, provide counts and averages of the number of
tests in each area.• This helps users to understand the strengths and
weaknesses of the test suite.• It will also help you to improve the next version.
![Page 52: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/52.jpg)
52http://jcp.org
What to test and what not to test?
• “Full coverage” for the majority of real-world specs is impossible (impractical.)
• Don't just test what's easiest.• Focus on areas where:– The consequences of non-conformance are greatest.• Eg, breaking interoperability or jeopardizing security.
– Implementations are more likely to be non-conformant because:• Implementation presents technical difficulties.• The specification is ambiguous.• Implementers are less likely to discover problems.• Implementers have an incentive to cheat (eg, to increase
performance.)
![Page 53: Patrick jcp](https://reader033.vdocuments.mx/reader033/viewer/2022042518/5481d775b47959e20c8b4603/html5/thumbnails/53.jpg)
53http://jcp.org
Spec + RI + TCK =