DRIVEN PROFILING OF OPEN STANDARDS FOR OFFICE APPLICATIONS

Bjoern Kirchhoff

FRAUNHOFER INSTITUTE FOR OPEN COMMUNICATION SYSTEMS, BERLIN The exchange of electronic documents between organizations, administrations and citizens is of growing relevance. Connected to this, the interoperability of office application documents is of essential interest. Even though the introduction of open standards in the field of office application fundamentally increased the degree of compatibility, interoperability issues still exist. As a result of this, the idea of the utilization of interoperability profiles emerged in relating OASIS and ISO committees. This paper proposes a feature driven but widely format independent approach for the profiling of office document standards that can magnificently simplify the definition of domain specific profiles.

Introduction lready in the 1980ies the awareness increased, that the absence of open standards in the field office applications could have a widely negative impact on communication inside organizations (Krönert, 1989). As a result of this the Open Document Architecture and Interchange Format (ODA/ODIF) was released in

1989 as an ISO Standard. ISO 8613-1:1989 never found a broad acceptance and market relevance (Eckert, 2009). Instead a number of companies introduced corporate IT standards within the scope of IT strategies

(Tiemeyer, 2006) that are regulating the usage of concrete software products and versions, in order to decrease interoperability issues inside the organization. This method is suitable in widely homogenous environments, but problems can especially appear at the periphery, e.g. when communicating with external partners. Besides this the introduction of new product versions can be critical, as producers may have changed internally used proprietary file formats in an incompatible way.

Today the exchange of electronic documents between companies, administrations and citizens are of growing relevance. The introduction of legally binding electronic communication channels aggravates this development. Accordingly the compatibility of applications reading and generating exchanged documents is of a crucial interest.

A main intention of this paper is therefore the outlining of an approach to improve the interoperability of office documents and office suites that can easily be adapted to existing technologies.

In order to make such a methodology capable for the standardization of office application standards, a technical description will be made.

Presently there are two open standards in the field of the storage and exchange of presentation-, spreadsheet- and word processing documents. The introduction of the Open Document Format for Office Applications (ISO/IEC 29500) and Office Open XML (ISO/IEC 26300) as open standards seemed to imply a high degree of interoperability, since different software implementations would be based on identical specifications. At a first view even the existence of two competing standards for office application documents appeared to be uncritical concerning interoperability, because converters could be created, which would allow a translation of a given document from one format into another. On a closer examination the standardization of complex storage formats - like ISO/IEC 26300 and ISO/IEC 29500 - is not a per se guarantee for compatibility or interoperability. A varying understanding or interpretation of unclear normative rules within specifications can lead to divergent application behaviors. In addition the standards sometimes are leaving a great degree of freedom for implementations due to a lack of completeness.

“ODF 1.1 is widely implemented, but there are areas where the specification is not clear or not complete. As a result of this ambiguity, interoperability issues arise when exchanging documents between different implementations.” (OASIS, 2010)

A

Consequently inter format interoperability cannot be guaranteed either. As Eckert, Ziesing and Ishionwu showed, furthermore architectural and conceptual differences as well as incompletely convergent feature sets aggravate or even prohibit in many cases a 1:1 translation from ODF to OOXML or vice versa.

“It may be concluded that many of the functionalities, especially those found in simpler documents, can be translated between the standards, while the translation of other functionalities can prove complex or even impossible.” (Eckert, 2009) Besides this, an increasingly relevant functionality like digital signatures characteristically cannot endure a

converting process (transformation is modification). Hence format converters are inherently limited. Occasionally even the software architecture of a format consuming applications can prohibit a

comprehensive compatibility to more than one standard. Some office suites read files in a specific file format by converting them into those that are natively supported in a first step. Thus a distinction between document reading support and import has to be made.

It is a common approach reducing interoperability issues is utilizing interoperability profiling. Interoperability profiles have been published e.g. for the Security Assertion Markup Language V2.0 metadata, Web Single-Sign On and for many more applications.

This paper uses the term “profile” for a subset of a standard that increases the interoperability by reducing its complexity. In case of ODF as well as in the case of OOXML efforts emerge in the relating OASIS and ISO committees, to precisely define semantic and profile conformity in order to make them automatically verifiable. With the aim to improve the compatibility of different ODF 1.1 implementations, the Organization for the Advancement of Structured Information Standards published a textual committee draft of the “ODF 1.1 Interoperability Profile v1.0” in 2010.

“This document tries to reduce the number of possible interpretations of the standard by increasing the number of strict conformance clauses” (OASIS, 2010). From a technical perspective the creation of interoperability profiles for ODF and OOXML can be very

complex. Because both standards are different in their underlying concepts, architectures and vocabulary (e.g. data pilot and pivot table) this paper presents a widely format independent approach for the formalization of interoperability profiles. It can be used for automatized profile conformance validation. The presented approach is a way to simplify the creation of subsets of (primarily XML based) standards, by defining a common vocabulary for equivalent concepts.

Basic Concept In order to automatize the profile validation of documents, a simplified understanding of standards for

office application formats can be helpful1: As the digital representation of an ODF or OOXML document can be seen as string of symbols of an

alphabet , this simplification can be made by defining the file formats/languages ISO/IEC 26300 ( ) and ISO/IEC 29500 ( ) as the sets of all documents that are conformant according to the standard.

The question whether an office application document really belongs to a standard or not is a real world problem. OpenDocument as well as Office Open XML use so called schema languages to describe normative grammatical rules that have to be fulfilled by all instance documents. In order to describe these normative grammatical rules ISO/IEC 26300 uses the schema language Relax NG whereas ISO/IEC 29500 utilizes XML Schema. Based on these schema files, several software products like Office-O-Tron, ODF Toolkit Validator etc. have been developed, that can verify the schema conformance of office documents. Programs that are able to analyse if a document is conformant to an office standard are called validators. Due to a limited expressiveness, schema validity is not equivalent to standard conformance. Thus most validators perform additional conformance checks, which cannot be covered by the schema validation2.

Mathematically seen a validator (supporting ODF and OOXML) can be defined as function :0,1 that decides if a given document is standard conformant3 ( ) or not ( ).

1 To make this part more readable for non-technicians, the terms are introduced by using very basic elements of formal

language theory (Sisper, 2006) as well as set theory. 2 It should be underlined that no currently available validator is able to verify standard conformance in all aspects. 3 ODF and OOXML validators in practice also offer - when indicated - fine grained information why a document is not

conformant

The validation of documents is very close to the profile conformance testing, since profiling only minimizes the number of conformant documents to interoperable/translatable documents. Consequently profiling conformance checking is another validation task.

An interoperability profile for ODF and or OOXML, as pointed out before, is a subset of interoperable or translatable documents . Thus a profile validator is a function : 0,1 that decides whether a given document is in or not. All Documents are interoperable or translatable from one format to the other.

Figure 1: The Set and the translatable/interoperable documents, is in P and therefore translatable

A feature is a specific attribute of a document that can be formalized and automatically be detected. For two reasons it was decided to specify interoperability profiles indirectly by naming the features that profile conformant documents are allowed to use.

1. The definition of a specific profile can be very complex. The specification of Office Open XML e.g. has more than 6000 pages.

2. Secondly the translatability of ODF and OOXML has so far been analyzed based on scenarios (Eckert, 2009) and document properties (Shah & Kesany, 2008). These approaches are strongly feature oriented and therefore the definition of profiles through feature lists is close.

In order to allow checking the profile conformance of documents based on feature sets, it has to be possible to detect the feature usage, which may not always be trivial. In this text a function that can detect the occurrence of a specific document attribute is called detection function.

A feature list generator is a function : , where is the set of feature names4. It returns the set of the feature names that are contained in the document (e.g. plain text, footnotes and headers). In simplification yields the set of the feature names that are used by all documents contained in the document set .

To ensure that a given document is conformant to a profile, it has to fulfil two constraints: 1. The document has to be conformant to one standard. 2. It has to contain only those features that where defined in the profile. Therefore the following profile conformance rule can be noted: Using a feature list generator the formalization of an example profile ( ’) can then be simplified the

following way: 1 plain text, footnotes, headers Interoperability profiles for single standards can be created with the same approach, by defining one

standard as an empty set.

4 ODF and OOXML are sometimes using a different vocabulary of concepts.

Figure 2: Profiling for a single standard (e.g. ISO/IEC 26300)

The standards ISO/IEC 26300 and ISO/IEC 29500 use precisely defined verbal forms for the expression of provisions in normative clauses, as they are described in Annex H of the ISO/IEC Directives (Part 2).

• may (need not) • shall (shall not) • should (should not) • can (cannot) When defining interoperability profiles, this keyword concept should be considered. For this purpose the

profiling model can be extended, by slightly modifying the profile conformance rule. The following table illustrates the extension of a profile to a -tupels with : , , that supports a distinction between features that may, shall and shall not be used inside documents.

may use the feature(s) 1 shall use the feature(s) 1 shall not use the feature(s) 1

Table 1: Features that shall, shall not or may be used by documents in P

So far it was assumed that the validator functions can verify if a given document is contained in . It has to be said, that in reality this is not always the case5. If there is only one validator for each standard

available, can easily be constructed by using one validator function for each standard ( and :

This is true, because and are disjunctive sets. This means that 1 0 and 1 0.

A Prototypic Software Implementation A prototypic software implementation of the feature driven profiling approach was realized strictly

utilizing standardized XML technologies (XML, ISO Schematron, XProc and XSLT). The testing of profile conformance was fragmented into three phases:

1. Since only standard conformant documents can be contained in an interoperability profile, the first phase is the validation of a given document with an already existing ODF or OOXML validator.

2. The next phase is the generation of the list of features that were used inside the document. 3. Finally (in the profile validation phase) the feature list is compared to the profile and the output will be

a validation result.

A major benefit from separating the generation of feature lists from the profile validation is that phase three is completely decoupled from any specific standard. As a profile is only described through a collection of

5 Office-o-Tron, developed by Alex Brown is the only public available validator that can check ODF as well as OOXML

conformity. I also developed a similar tool (metavalidator) supporting both standards at Fraunhofer FOKUS that was not published.

feature names, the definition of interoperable subsets of ODF or OOXML is very unassuming. Profiles for different use cases and applications can easily be created or adapted. This may be needed since “[…] the capabilities of ODF applications extend beyond the common desktop editors, and include other product categories such as web-based editors, mobile device editors, document convertors, content repositories, search engines, and other document-aware applications, interoperability will mean different things to users of these different applications” (Weir & Hanssens, 2010). Profiles for domain specific document categories can be created without a deeper knowledge of standard’s internals.

Assuming there were feature generators for other document markup language and binary formats available, one could also define interoperability profiles for further constellations (e.g. latex and ODF or latex and OOXML). The profiling of unidirectional translation processes, for instance ODF to PDF/A, is even less complex, since here feature detection functions only have to be provided for only one format. To a certain degree interoperability profiles for ODF to PDF/A and OOXML to PDF/A could be similar.

Another advantage is that a separated feature list generation makes it possible to examine the diffusion of specific features in large amounts of documents, which previously could only be done manually.

Profiling Coordination

ODF and OOXML are both file formats that use ZIP-archives as containers. These so called packages are containing mostly XML files. Therefore ODF and OOXML – like many other6 - belong to the class of XML-In-Zip file formats. In order to check the profile conformance of a document, the content of the relating containers has to be analysed. Common XML-technologies were developed to operate on single XML instance documents. Consequently a validation of full packages makes it necessary to validate multiple XML documents contained in the package. In the OOXML terminology these XML entries are also called parts.

In order to create an overall cumulated validation result, sequences of validation- and transformation steps have to be performed. The XML pipeline language XProc, which is W3C recommendation since May 2010, allows the composition of such processes. Even though an in-place-validation of ODF and OOXML is possible with custom URI Resolvers e.g. by supporting the not standardized JAR-URL-Syntax (Oracle), this pipeline language is used to transform packages into a simple “envelope format”, very much like a suggestion made by Rick Jelliffe on the ISO/IEC JTC 1/SC 34 mailing list:

„I think that all that is needed is a simple vocabulary with zip:archive, zip:folder and zip:entry. Non-XML files could have an empty file with their name, to allow validation that a link points to the appropriate media etc.“ (Jelliffe, A simple method for integrating XML-in-ZIP formats into DSDL, 2010)

This flat representation of a package as single XML instance document has the advantage, that common XML transformation and validation technologies can be adapted. Besides this it simplifies the expression of part crossing constraints. Feature List Generator

A single XML document also allows generating lists of features that are used by applying a suitable XSLT stylesheet. Since the creation of such a stylesheet is not trivial, a simplified XML language for feature definitions is introduced.

6 InDesign Markup Language, SCORM, Widgets etc.

Figure 3: Simplified langu

At first a feature has a uniformat dependent detection functionexamples of features and relating dfootnotes inside ODF and OOXML

Feature Standard

Image ISO/IEC 26300

ISO/IEC 29500

Table ISO/IEC 26300

ISO/IEC 29500

Footnote ISO/IEC 26300

ISO/IEC 29500

Table 2: Examples of featu

The introduction of a denecessitates the creation of a compilist is transformed by an XSLT-proctransformation) can be used to calcu

uage for the definition of document features

ique name (e.g. “footnote”, “header” etc). This name ns. A detection function is expressed in XPath. The fodetection functions that can be used to verify the usdocuments.

Detection Func//odf-draw:image

//ooxml-a:blip

//ooxml-w:tbl

//odf-table:table

//odf-text:note[@odf-text:note-class='footnote']

//ooxml-w:footnoteReference

ure and relating detection functions

escription language for document features and reliler or converter. In the presented software implementcessing scenario. Then the resulting XSLT stylesheet ulate the list of features detected within the flattened do

is associated to number of ollowing table shows some sage of images, tables and

ction

lating detection functions tation a feature description (the output of the previous ocument.

Figure 4: An example architecture for a Feature List Generator

The component for the creation of feature lists cannot only be helpful in the context of document profiling. Usually the analysing of document round-trips (a common method for interoperability tests) implies a lot of manual work. A feature list generator can automatize some essential test regarding the persistence of specific characteristics of a document (e.g. is a table of contents still contained in a document after saving it with another application).

Profile Validator

The architecture of the profile validation component is similar to the feature list generator previously presented. Instead of directly creating a validation schema for each profile, a simple XML based language is introduced, which can express the characterizing features in a compact way. It also allows the distinction between degrees of obligation (may, shall, should etc.). Consequently in this implementation a profile is a list of allowed and possibly disallowed features names. This decouples the definition of profiles form the standard dependent definition of feature detection functions.

Feature Level of ObligationImage Shall Header May Footer May Table Shall Footnote May

… …

Table 3: Examples of a simple profile definition

Whereas the feature definitions were translated to an XSLT stylesheet, profiles are mapped to ISO Schematron files. ISO Schematron is a rule based validation language that can be integrated into XProc pipelines.

Running this validation schema against a feature list produces a validation result. Violations of rules of the obligation degrees “should” and “should not” are reported as warnings to the user.

Figure 5: An example architecture for a profile validator

It has to be accented, that this way of profiling – even though it may not be obvious - is strongly exclusion oriented. Features that are not explicitly mentioned in a profile have to be inherently forbidden. This coherency has to be taken into account when translating a profile to a Schematron file. Therefore the profile to Schematron translator needs to access the feature definition list in order to create a “shall not” rule for unmentioned entries.

Feature Detection In Examples The following paragraphs will demonstrated some capabilities of this feature driven approach for the

detection of specific document properties. Since the profile validation based on identified features is straight forward, this will not be part of a closer examination here.

Figure 6 shows the distribution of thirty exemplarily chosen document properties in mixed ODF and OOXML documents. The test candidates consist of 2326 word processing documents that where retrieved from the internet, making use of Google web services.

As you can see, a lot of uncritical properties like the usage of plain text, bold, italic are commonly used. On the other hand there are a number of documents using so far not translatable functions like blinking text, embedded fonts, text borders or change tracking.

Figure 6: Spreading of 30 features over a set of 2326 sample word processing documents

The feature list generator also allows the analysis of the embedding of specific file formats (e.g. images)

inside office documents packages.

Figure 7: Proportion of image formats embedded into example word processing documents

Figure 7 shows the proportion of specific image file formats used by the test candidates. Since all file formats should be implemented by ODF and OOXML consuming office suites, no specific critical documents can be identified in this case.

0 500 1000 1500 2000 2500

blinking textembedded fonts

equotionsend notes

text bordersinitials

commentscharts

change trackingline numbers

footnotestable of contentright alignment

headersjustification

footersimagestables

hyperlinksfirst line indent

numberingsbullets

line spacingsborders

underlinecenter alignment

italicleft alignment

boldplain text

png 57%

jpg27%

wmf8%

gif4%

emf3%

tiff1%

eps0%

In addition feature driven document analysis allows examining more complex interrelationships, like nested features, that can also be relevant regarding interoperability concerns. The following figure shows exemplarily the number of test candidates that have images embedded into headers, footers or tables.

Figure 8: Number of selected nested feature inside the test group of 2326 documents

Even though images can be contained into tables, footers and headers in both file formats, other features

like sub tables that are seamlessly embedded in parent table cells are only supported by ODF (Eckert, 2009). A counter-example of table properties that are only available in OOXML are background patterns.

Conclusion This paper outlined an approach for the profiling of open standards for office applications based on the

definition of a common vocabulary for document properties or features. It takes into account the way interoperability of ODF and OOXML has so far been analysed. In the majority of publications specific features that are critical regarding interoperability and translatability have been identified. Basic concept of feature driven profiling as presented in this paper, is the splitting of document feature descriptions and profile definitions. Document profiles can therefore be described format independently by sets of feature names in connection with keywords that express the degree of obligation as defined in the ISO/IEC Directives Part 2 Annex H. The feature definitions in contrast are tightly coupled to the manifestation in each standard.

One major benefit of this method is that profile definitions – due to the reusability of feature definitions – can be created in a very compact way and therefore easily be adapted to multiple different use cases. Feature driven profiling can be used for standard-to-standard as well as for application-to-application scenarios. Another advantage of this approach is that it offers an automatized way for the analysis of documents on a logical level7 that can reduce time-consuming manual work in interoperability research.

The prototypic implementation showed that feature driven profiling for ODF and OOXML can be implemented in software by strictly utilizing standardized and well established XML technologies. Examples showed that the feature list generation component can be used to identify complex document properties like nested features.

A disadvantage of the current methodology is that the number of document features definitions may strongly increase when the context of occurrence (like nested features) has to be recognized. Therefore an extension of the presented approach regarding a feature context may be reasonable. Since the approach is strongly exclusion oriented, at least the majority of critical features have to be implemented into feature detection functions in order to provide a profiling solution that is suitable for practical applications.

Even though the outlined concept seems to be promising, more research work has to be done in order to show the effectiveness. Especially the set of currently only 45 available features has to be increased with a focus on critical document properties. It is also planned to utilize the developed technologies to analyse documents of the German public administration, in order to identify the characteristics of domain specific document categories.

7 At the OpenDocument-Plugfests 2011 Miloš Šrámek from the Comenius University in Bratislava presented a metric for the

automatized comparison of documents on a presentation level.

0 50 100 150 200 250

images in footers

images in tables

References DIN NA 043‐01‐34. „OpenDocument Format / Office Open XML Translation.“ Berlin, 2009. Eckert, Ziesing, Ishionwu. Document Interoperability. Berlin: fokus Basic, 2009. ISO/IEC. OpenDocument Format ODF. Genf, 30. November 2006. Jelliffe, Rick. 2010. http://lists.dsdl.org/dsdl-discuss/2010-03/0008.html. —. 2008. http://news.oreilly.com/2008/08/the-challenge-of-validating-xm.html. Krönert, Günther. „PIK - Praxis der Informationsverarbeitung und Kommunikation.“ (Carl Hanser Verlag) 12, Nr.

13 (1989). OASIS. „ODF 1.1 Interoperability Profile, Committee Draft 03.“ Juni 2010. http://docs.oasis-

open.org/oic/odf1.1i/v1.0/CD03/ODF1.1-InteropProfile-v1.0-cd03.pdf. OASIS. OpenDocument v1.0 Specification. http://www.oasis-

open.org/committees/download.php/12572/OpenDocument-v1.0-os.pdf, May 2005. OASIS. OpenDocument v1.1 Specification. http://docs.oasis-open.org/office/v1.1/OS/OpenDocument-v1.1.pdf,

February 2007. OASIS. OpenDocument v1.2 Specification. http://docs.oasis-open.org/office/v1.2/part1/cd04/OpenDocument-v1.2-

part1-cd04.odt, Januar 2010. —. „The State of Interoperability v1.0, Committee Draft 04.“ Februar 2010. http://www.oasis-

open.org/committees/download.php/36669/StateOfInterop-v1.0-cd04.odt. Oracle, Sun Microsystems /. „JavaTM 2 Platform, Standard Edition, v 1.4.2.“ kein Datum.

http://download.oracle.com/javase/1.4.2/docs/api/java/net/JarURLConnection.html. Shah, Rajiv, and Jay P. Kesany. Lost in Translation: Interoperability Issues for Open Standards – ODF and

OOXML as Examples. University of Illinois College of Law, 2008. Sisper, Michael. Introduction to the Theory of Computation, Second Edition. Thomson, 2006. Tiemeyer, Ernst. Handbuch IT-Management. Konzepte, Methoden, Lösungen und Arbeitshilfen für die Praxis.

Hanser Fachbuchverlag, 2006. Weir, Rob, und Bart Hanssens. The State of ODF Interoperability Version 1.0 - Version 1.0. OASIS Open

Document Format Interoperability and Conformance (OIC) TC, 2010.