Erik P. DeBenedictis

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

Testing Integration and Validation Group

Agenda

• Hardware Inventory– Oak Ridge– Sandia– Berkeley– Argonne

• Intellectual Property– Process

• Documentation Standards– TEX– HTML– Embedded

• Bug tracking– Bugzilla

• Test Framework– Deja Gnu– QMTest

Hardware

• Berkeley– Hardware

• 32 nodes P II 400

– Software• Redhat 6.2

• Argonne– Hardware

• On demand 256 nodes• P III 500s myrinet

– Software• Redhat 7.1

• Oak Ridge (posted in notebook)– Hardware

• 4 dual 450 with 1 gig ram• X Torque 64 nodes 2 gig

– Fast ethernet + gig ethernet

– Software• Oscar install 1.2.1

• Sandia– Hardware

• 8 Alpha + admin node

– Software• Cplant

Servers

• Ames– CVS Directory

• E-mail Brett

• Argonne– Bitkeeper

• E-mail Narayan

– Bugzilla• E-mail Narayan

• Sandia– CVS

• E-mail Erik

Intellectual Property

• Process– Fred (DOE) requests Open Source

– Labs “Assert Copyright” on Open Source License• What license• Needs to be same

– Contribute Source

– Publish Source

Documentation Standards

• Objectives– Set standards for English

Language contributions to group’s product

• HTML– (Doesn’t paginate well --

Hargrove)

• Rusty’s TeX Macros– (Not browsable? --

Hargrove)– (Paginates well

Jackson)

• Microsoft Word– (Violent objection to this

one)

• ASCII

• What types of documentation?– Design documents– User guide– Man pages– XML Schema specs– Installation guide– Administration guide– Reference guide (optional)

• Format requirements– Source format

• Html, tex

– Distribution format• Pdf, html

Bugtracking

• Bugzilla– Argonne has bugzilla installed and can create a

branch for SciDAC

Test Harness

• The T-I-V-X group investigated test harness products– Selected Deja GNU (but did not review QMTest)

• Sandia investigated test harness products for Cplant– Selected QMTest

• The SciDAC SSS group finds QMTest interesting and support for Deja GNU has withered

What is QMTest?

• Scriptable (Python) test driver• Test metadata (results) defined with XML• Web-based interface• Extensible webserver built in (Zope)• HTML contents customizable• New test types through inheritance• More: http://www.codesourcery.com/qm/qmtest

Why QMTest?

• Opensource• Scriptable (you can extend it)• Object-oriented• Declarative results in XML• Web-based UI supports decentralized dev.• Active and growing development commun.• Good “separation of concerns” design

How Does Cplant Use It?

• Mostly as intended • Subclass ExecTest• Declare test results using UI right now• Drive cases sequentially. PBS integration?• We’re coming up the learning curve…

How QMTest Looks

Example Test Report

Step 1: Define a New Test

Step 2: Describe It

Who’s Doing It

• Lead: Sue Goudy spg@sandia.gov• Grunt: Mike Carifio carifio@usys.com• Carifio is ½ time…

When Did We Do This?

• 12/01 Carifio invest. test frameworks• 1/02 Goudy, Carifio select QMTest• Q1/02 Goudy,Carifio learn Python, QMT.• Q2/02 Carifio gathers test “packages”,• implements “build framework”• Q3/02 Drive packages with QMTest• Q4/02 UI mods? PBS integration?

Conclusions

• QMTest is scriptable driver• Tests are defined in XML• UI is web-based, content generated• New test kinds derived from base classes• Best price• Active development• Seems to be “best in class” today