Open Standard Voting Localization with CAM

- Localization Mechanisms

- Publishing Localizations

- Leveraging Open Standards

- XSD ingesting David RR WebberChair OASIS CAM TC(Content Assembly Mechanism)E-mail: drrwebber@acm.orghttp://wiki.oasis-open.org/cam

slide 2

Overview

Goal is to promote the use of open public standards for voting systems

Develop in-country localizations from standard international base set

Provide common open public lexicon for elections management and verification

Leverage XML and open standards approach, including NIEM*

Provide developers with tools to aid delivery, documenting and testing of solutions *NIEM – National Information Exchange Model

slide 3

Approach Voting Localizations built from OASIS EML

EML has 29 pre-built components (schema) for voting processes

Use desktop visual editor to pick components Trim and select only what you need from each standard schema Desktop toolkit makes templates for the XML required

Generate localization details Automatically build XML components to use Build your own dictionary, compare to EML standard Create readable documentation of details

Test, Package and Publish The CAM desktop toolkit environment supports NIEM IEPD approach

Available as Open Source on SourceForge.net Implementation of the public OASIS CAM standard for schema

templates (camprocessor)

Localization Mechanisms

Creating a Country Voting Localization Package

slide 5

Localization Key Components

XML Schema

2

1

3

4

DocumentationReadable information use details (HTML), EML cross-reference Spreadsheet

XML artifacts The localized schema and picklist.xml

Examples & Test CasesRealistic test XML instances for conformance and interoperability testing

Key components(aka IEPD*)

EML Schema XSD aligns voting information model, processes and domain dictionary

*IEPD – Information Exchange Package Documentation

slide 6

Localization Approach StepsEML

XMLXSDXMLXSD

VotingVoting

ExchangeExchange

ManageManage

developdevelop

11

ingestingest

22BaseCAM

Template

BaseCAM

Templatelocalizelocalize

33 RevisedCAM

Template

RevisedCAM

Template

GenerateGenerate

44 Local XMLXSD

Local XMLXSD

CompareCompare

55

EMLDictionary

EMLDictionary

Steps:1. OASIS develops standard international XSD schema

Implements the voting process, management and information exchange needs

2. Ingest the standard XSD into base CAM template 3. Apply localizations to CAM template

Exclude unwanted optional components Align local exchange needs to standard Extend for local needs

4. Generate new localized XSD schema that conform to EML standard5. Build dictionary cross-reference of localization usage6. Publish documentation and examples

PublishPublish

66Documentation

+ Examples

Documentation+

Examples

slide 7

Creating a Package for Localization

Step 1 - Ingest EML XSD – rules, structure, annotations Step 2 - Select & mark out your use model in visual

editor - Save your “pick list” selections - Compress template to desired structure

Step 3 - Generate your localized use XSD schema subset

Step 4 - Generate and test rich concrete data examples (complete with content hints / pass / fail / random options)

- Run rules engine - Verify operation and outcomes

Step 5 - Build localization documentation of structure elements, usage, rules and cross-reference map

Step 6 – Package, Publish and Share (IEPD)

slide 8

Challenge: XSD schema has limitations! The schema contains the superset of voting exchange

component variations for every country

XSD syntax does not have direct context mechanisms

Hence people make everything in schema optional

Dependencies are not clear

It is difficult to understand the constructs and to

document the rules clearly for election authorities to

verify

It is hard to create test cases and instances (the “pick list”

tough to visualize)

These are all limitations inherent in W3C Schema language

itself

slide 9

Completing Localization Picture:

Base EML XSD schema structure model

CAM template + rules / refinements

Documentation of use patterns (aka “pick list” +

subset XSD)

Test cases and examples

Content hinting (localization use data)

Validation engine for unit and regression testing

Open standard, open platform and open source

allows consistent agreements between participants

slide 10

Localization Package Use Scenario

Testing,Agreement /Alignment.

Validate

Templates

Report

DeveloperUses &Creates

Results

Test

Pass / Fail

XMLhtml

32

4

Test Cases

XML

Package

StructureRulesContextExamplesDocumentation

Publish

1

Verify Result Outcomes

XML

LocalizationContent

Hints

slide 11

Tutorial

Base CAM templates – (ingesting XSD schema ) step by step example

Documenting the Localization Patterns Creating “pick list” selections

Compressing template structure

Subset XSD generation

Publishing Localization package Cross-reference spreadsheet

Documentation reporting options

Generate use dictionary

Testing and Conformance Creating Test Case examples

Content Hinting

Running Test Cases

Advanced Techniques

Base CAM Templates

(ingesting XSD Schema)

Using jCAM editor Wizard

(http://www.jcam.org.uk)

slide 13

EML V6.0 provides base CAM templates Templates come pre-packaged in the EML V6

download

However, for completeness we show how to generate one…

slide 14

Ingesting XSD to CAM template format

EML xmlEML xml

EML xsdEML xsd

LocalizationLocalization

Commonxsd

Namespace

XML Schema

ComponentsComponents CIQ xsdCIQ xsd

INGESTINGEST

Structure

Rules

Documentation

XSLT toolsprocess

entire EML XSDcollection

XSLT toolsprocess

entire EML XSDcollection

CAM = Content Assembly Mechanism

OASIS EML XSD Collection

slide 15

Step 1 & 2 – Pick the XSD schema to ingest

1

Choose File / New Template

First location is the rootfolder for the XSD collection.

Typically this is the same as the location for the XSD schema you want to ingest.

(Complex XSD can have this in separate folder; hence need for option)

2

Pick XSD

Specify XSD Location root directory

slide 16

Step 3 – Choose the XSD parent element

From the dropdown list pick the correct root element you wish to use (EML).

XSD’s may have more than one collection in them (as shown here) or may not have the parent node element cleanly defined.

The Wizard shows you the list of all possible ones it finds – so you can select and confirm the right one.

3Choose XSD parent element

Confirm and start the XSD ingesting

slide 17

Step 4 – Ingesting complete – Save Results

4

Completed template is loaded and save dialogue appears

Processing usually takes a few seconds, some larger EML XSD can take a few minutes to process.

Tip: jCAM runs the ingesting as a background task – so you can continue to use the computer while such long ingesting is proceeding.

4

slide 18

Step 5 – Review ingested structure

5

Completed template is shown in the structure navigator panel for review and editing

Template contains all the default content model and structure rules ingested from the XSD.

All annotations and documentation from XSD also ingested (show as “paperclip” symbol).

Code lists and typical content values inserted for easy visual reference.

Documenting the Localization Patterns

“Pick lists”, documentation and XSD subset generation

slide 20

Building a pick list

MARK WHATIS NOT

NEEDED

MARK WHATIS NOT

NEEDED

Structure

Rules

Documentation

Make pick list

DESIRED RESULTS

excludeditems

EML structure is

all inclusive!!!

slide 21

Marking Items for exclude - pick list

Can exclude at all levels within the structure excludeTree() excludeElement() excludeAttribute()

Use XPath operators to control scope: Specific node Group of nodes Anywhere occurs in structure Contextually based on condition

Can add new domain elements with own namespace to EML template

slide 22

Using Editor to mark Exclude items

Structure Editor Tools

Select focus on item

Invoke action menu (right mouse click)

Pick action

Tip: exclude rule display is context sensitive and only available on optional items

Tip: use “Add New Rule” mode to specify different XPath for exclude (quick mode assumes “current path”)

slide 23

Export and Save completed pick list

File Menu Option

Select Export / Pick List and then specify filename of destination.

Excluded items are designated with red “dot” in structure editor and italics font with no bold highlight

slide 24

Pick List Details

EXCLUDE FLAGVALUE

Exported Example

Pick lists provide a handy way to catalogue the localization model and can be re-used later by importing into other templates

slide 25

Compress Operation

File Menu Option

Select option and specify filename for new copy of your template.

Compress process removes all rules and structure items marked with an exclude statement.

Note: ignores excludes that have a conditional context expression (advanced section technique).

Note: You can retain both original and new compressed templates

slide 26

Extending the base EML schema In addition to excluding existing optional

components you may need to add new pieces

New pieces should be assigned their own extension namespace prefix – to identify them from the base EML components

Typically the OASIS EML committee will review new pieces as candidates for adding to future versions of the standard

• Send comments and suggestions to election-comment@lists.oasis-open.org

slide 27

Adding New Domain Elements

Select root element node in structure; right mouse click for context menu; select “Add Namespace”; then enter prefix and URI

1

Select element node in structure; right mouse click for context menu; select “Add Child Attribute / Element”; then enter prefix and name

2

slide 28

Generating your EML use schema

Once your localization details are completed you are ready to complete the remainder of the process

This starts with generating your own new subset schema

Then once schema details are OK – creating test cases, cross-reference spreadsheet and documentation

slide 29

Generating subset use schema

Select Export CAM as XSD menu option

1

File Menu Option

Confirm the location and filename, and namespace mode.

Select ‘default’ for Mode to write a regular import set of XSD schema for each namespace

2

slide 30

Schema subset generated

Set of XSD files with filename and namespace suffix

Each namespace file is import for those specific type definitions

Reviewing XSD results in a schema visual editor tool

slide 31

Information Restriction Considerations

The CAM template allows full use of XPath conditional expressions and a rich set of over 30 functions including: setNumberRange(), setLength(), setLimit(),

setDateMask(), setNumberMask(), restrictValues(), lookup()

Those that are compatible with XSD constraints will cause constraint schema assertions to be written out when exporting to schema

In the advanced topics section we will look at cross field validations using XPath conditional rules

Generating Testing and

Conformance Examples

Generating base examplesCustomizing content with HintsSelecting valid and invalid modesRunning rules validation check

slide 33

Test Case Generation Quick Start

File Menu OptionDefault directory to write examples into

Name to be used for the examples

How many examples to create

Repeating elements count

for Quick Test – just click “OK” to use default settings

slide 34

Test Case Results

Active links to view the generated examples

slide 35

Advanced Generation Options

How to handle optional items: all | random | none

If you want deliberate errors for fail testing; (will give variety of data and structure errors)

Use namespaces or not; if ‘false’ is

selected – then XML instances are

created with minimized

namespace usage.

Optional content hints (explained next )

Use slider to pick a specific seed value – or leave blank for random seed

Optional schema file validation link; use this to have example validate with EML schema or local subset schemaUse content type

or item name (name is useful for checking backend transform processing)

slide 36

Test Case Generator Feature Summary

Make both Pass / Fail testing examples

Content hinting so examples use real not fake data

Test optional item logic with: all / random / none

Uses exclude() assertions so does not include those items – makes realistic examples of your use pattern

Can pass in seed value – use when adding and testing hints (each test case is labelled with its seed value)

Make hundreds of test cases without manual editing

Can link test case to XSD schema for structure tests

You can modify XSLT tool for own testing needs

slide 37

Run CAM Rules Check on Examples

Pick Test Case Example to VALIDATE; click Finish to run validation rules

Review validation results

Run Menu Option

1

3

2

slide 38

Content Hinting Mechanisms Designed to create realistic data

examples Hints can be provided in two ways Firstly - using ‘Value’ notes in

annotations on specific items in the structure editor

Second – create your own Hints XML file and add matching rules to globally apply across your template(s) – e.g. FirstName, LastName, Address, BirthDate, etc.

Can export from one template, import into another

slide 39

First Approach: annotation Value Hints

Select focus on structure item

Invoke action menu (right mouse click)

Choose “Edit Annotations”

slide 40

Then add Value annotation item

Click on “Add New”, then enter “Value” as Type and confirm “OK”

Select “Value” tab

Enter values terminated with “|” character

1

2

3

4

5 Re-run Example export to see results

slide 41

Second: Hints File Mechanism (XML file)

like / with partial name matching

key matching on tag name

key / parent path matching

use when same name occurs within different parents – e.g. Country and Person / NameElement with different content and context

use for exact match of items

3

2

1use for component match on items – e.g. first with name matches <nxnl:first_name>

Note: matching is case sensitive but ignores namespaces

TIP: can use Export Hints to create initial XML file for editing

slide 42

A- Using Examples Generator with Hints

Select XML hints file to be used here

slide 43

B- Import Hints into Annotations (merge)

set and select as needed

Option to auto-reload new anno file into current template

Documentation

Cross-Reference SpreadsheetDefault reporting options

slide 45

Generating Cross-Reference Spreadsheet

Use Compare to Dictionary tool Compare to default EML-V6-dictionary

Drag and Drop XML file into Microsoft Excel

Review and save spreadsheet workbook

slide 46

Dictionary Compare Tool

Select compare tool menu option

Browse and choose EML dictionary

slide 47

Drag and Drop to Excel Spreadsheet

ClickOK

slide 48

Documentation Layouts

Five options Source XML Component details (XML) Tabular format (HTML) Interactive web page (wiki) Code list

Tabular format page layout for data analyst use and designed to make rules and use patterns clear

Each documentation layout XSLT script can be customized as desired

slide 49

Tabular DocumentationDocumentation Menu Option

Select Format, resulting HTML opens in browser viewer

slide 50

Tabular HTML Content Details

EnhancedData typeLogic

Extended Code listhandling

Clear use pattern

XPath referencesandfunctions

Annotations

Summary

Ingesting XSD Creating use pattern (aka pick list) Generating test examples Hints system Generate XSD schema subset Running tests

slide 52

Localization Checklist Review

XML Schema

2

1

3

4

DocumentationReadable information use details (HTML), EML cross-reference Spreadsheet

XML artifacts The localized schema and picklist.xml

Examples & Test CasesRealistic test XML instances for conformance and interoperability testing

EML Schema XSD aligns voting information model, processes and domain dictionary

slide 53

Summary Capabilities covered

Ingesting XSD to Template Creating use pattern (aka pick list) Generating test examples Hints system Generate local use XSD schema subset Running tests Generating documentation and spreadsheets

Allows rapid localization development Enables consistent implementations Documentation provides clear details

on structures, content, rules and definitions

Cross-reference to EML standard

Advanced Techniques

Extending rules for actual use cases Using XPath selector wizard Handling ingestion recursion issues SourceForge XSLT repository

slide 55

Importing pick list operation

File Menu Option

Select Import and specify filename of your existing pick list xml.

Import process matches the path expressions in your pick list to the XPath expressions in the template.

Exclude statements generated for matching items.

Makes it easy to re-apply a pick list on new versions of schemas, or on similar schemas with same blocks of content – address, company, person, etc.

slide 56

Illustrative EML requirements

When Ballot type > 0 - AND Language code > 1

Require Ballot State = CA, NV, NM

When Registered Voters Total < 1000 - AND Ballots > 1000

Audit alerting

Resource Matching

Region Restriction

slide 57

Template rules syntax

<as:BusinessUseContext> <as:Rules> <as:default> <as:context>

<as:constraint condition="//CastBallots /Total > ‘1000’ and( // RegisteredVoters /Total < ‘1000’)"

action="restrictValues(// CastBallots/Total, ‘Alert – possible audit required’) "/>

<as:constraint action="restrictValues(//eml:AdministrativeArea /eml:NameElement , 'CA|NV|NM')"/>

</as:context> </as:default> </as:Rules> </as:BusinessUseContext>

slide 58

XPath selector wizard

Check boxes allow user to select scope of rule action

Conditional context rules can be optionally used

Documentation notes can be added here

XPath syntax is automatically generated for you

slide 59

“CAM Kit” of XSLT tools used

XSD 2 CAM extracts structure and rules and builds template from

schema

XML 2 pick list builds pick list from any XML instance (uses Level Depth

setting)

CAM 2 Examples Generates a collection of XML instance test cases

Import/Export Hints Manage and apply content hinting across CAM templates

CAM 2 XSD export Creates a subset XSD from CAM template

slide 60

SourceForge XSLT svn repository

Using any svn client the XSLT scripts can be retrieved from: https://camprocessor.svn.sourceforge.net/svnroot/

camprocessor/camed/uk.org.jcam.camed/trunk/xsl/

Resources

Selection of useful links and additional technical details

slide 62

Simple EML 440 Example

Example using the EML 440 cast vote schema

Simplified to minimum requirements Example can be downloaded here:

http://markmail.org/message/qyrpfq75mtctblyq

slide 63

www.oasis-open.org/committees/election

docs.oasis-open.org/election

www.oasis-open.org/committees/cam

wiki.oasis-open.org/cam

docs.oasis-open.org/cam

www.jcam.org.uk

Resources: