achieving a better assessment development process

Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

2013 Users Conference Baltimore | March 3rd – 6th

Achieving a Better Assessment-Development Process

Using well established architecture principles to improve the assessment building workflow.

2013 Users Conference Baltimore Copyright © 1995-2013 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark

is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Where do you fit into this presentation? 74% stated they did not have a roadmap to show where the

team was supposed to be going 69% reported their assessment team received no professional

training 25% responded they did not allocate any funds specifically to

improving their development process Architecture interests: 75% interested in architecture road map 60% interested in customized reusable components for repository 59% interested in documenting their development process

Results of 2012 Conference Poll

Slide 2



TIBCO Software…who we are TIBCO’s Knowledge Assessment program What is Architecture? The Value Proposition How we “Architected” our Knowledge Assessment

development process Integrated Knowledge Assessment System (IKAS)

Discovering, Designing & Building Reusable Components in the Questionmark Perception Repository

Q&A & Open Discussion

Topics

Slide 3



An Introduction to TIBCO Software, Inc.

Slide 4

TIBCO Software Inc. (NASDAQ: TIBX) is a provider of infrastructure software for companies to use on-premise or as part of cloud computing environments.

www.tibco.com

http://www.tibco.com/



TIBCO Certification Program

Slide 5

TCP

TCA

TTP TTP – TIBCO Trained Professional Actually attended TIBCO training

Passed on-line Knowledge Assessment

Available for most TIBCO courses

TCP – TIBCO Certified Professional Attended TIBCO training (or has equivalent knowledge &

experience)

Passed formal certification exam

Available for core technologies

TCA – TIBCO Certified Architect Attended formal TIBCO training

Passed knowledge assessments and certification exams

Passed 5-day performance-based test, followed by evaluation by TIBCO Global Architects



Our program supports a battery of tests for a rapidly changing high-tech environment. The subject matter is exceptionally volatile. While we have

only 30-40 assessments at any single time, the lifespan of an assessment is measured in weeks.

A single technical product enhancement can have a significant impact on an existing item bank.

The subject domain ranges from general software development concepts through computer science architecture principles.

The stakes for our participants are not life-threatening but they can be career-busting.

TIBCO’s Knowledge Assessment Program

Slide 6



What is Architecture?

Slide 7

It is planning the development process from conception through maintenance using proven Architecture Best Practices and experienced architects.

Formal or semi-formal information gathering sessions

Applying standards, specifications, and best practices

Modeling a good infrastructure architecture (the foundation)

Modeling a good development process architecture (the building)


is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 8

The business drivers: To improve the Knowledge Assessment Development

development process To improve the information provided to SMEs To document our development process

To avoid creating a spaghetti architecture in the

repository This is a technical term to describe systems built without

architecture planning.

Why Did We Architect our Knowledge Assessment System?


is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Slide 9

The Value Proposition

Higher item quality Reduced time-to-market Improved development process Improved information provided to SMEs Enhanced and automated item-level audit trail



How We Apply Architecture

Slide 10

We Use Architecture Best Practices to: 1. Gather Information in a formal setting 2. Establish an Architecture Vision 3. Architect the Infrastructure to Support that Vision 4. Architect the development processes

For more about modeling see: Appendix A: Architects and Architecture



The Architecture Vision2011-2013

Slide 11



The Architecture Vision as Implemented 2013

Slide 12

Automation

Enhanced Quality Assurance



Definition: Reusable components are preconfigured frequently-used

well-defined functional units within a repository.

It is important to formalize their development To maximize and ensure an attractive ROI: apply standards,

specifications, and conventions

Defining Reusable Components in the Context of the Questionmark Repository (1/3)

Slide 13




Slide 14

Major Characteristics of Reusable Components: 1. Significant ROI 2. Highly stable. 3. Highly reusable 4. A crisp boundary.




Slide 15

Will my reusable components be the same as yours? Some, like templates, yes. Others, probably. Some will be unique to the way your business is conducted

It is the concept of actively discovering candidate components and then selectively implementing them as reusable components in a planned manner that is important.



Types of Reusable Components for the Questionmark Repository

Slide 16

Types of reusable components: Complete Assessment templates (common) Participant screen templates (common) Outcomes (sometimes overlooked) Demographics (can be powerful) Dummy items (placeholders) Mini item banks (scored or un-scored items used across

multiple assessments) Others



Discovering Reusable Components in Our Questionmark Repository (1/3)

Slide 17

1. First, it is not simply looking for common pieces of

repository stuff and reusing them.

2. Second, it is important to select and implement only those candidate components that will ensure an attractive ROI.

3. Third, think outside the box: Example: During the discovery phase, it became logical that a

quality assessment template would be of considerable value. At the time this was unexpected; in hindsight it is now obvious.




Slide 18

Make a list of the “big” things in your repository that have the most value to the way you do business. Topics Assessments Templates Global Resources Global Content

The above statement seems obvious, even trivial. But is it?




Slide 19

Discovering candidate reusable components The most important “big” things in our repository for the way

we at TIBCO do business are: Topics Assessments Templates Global Resources Global Content

Now we know the most productive places to look for discovering reusable components.



Overview of TIBCO’s Reusable Components (1/2)

Slide 20

Assessments Templates

Intro Screens

Quality Assurance Demographics

Topics



Overview of TIBCO’s Reusable Components (2/2)

Slide 21

Pass and Fail Emails

Embedded in templates Pass email has Certificate

embedded in the email sent by the QM system



IKAS: Integrated Knowledge Assessment System

Slide 22



Top Three Benefits of IKAS[1]

Slide 23

1. Knowledge Assessment Status diagram Initially used MS Project / …overkill & too time consuming A quick glance provides complete status of KA projects Easy detection of bottlenecks in workflow

2. Improved information provided to SMEs Personalized standardized professional documentation sent to

SMEs for their assignments 3. Reduced time-to-market

2011: one to three new or updated assessments per quarter 2012: fourteen in one quarter

[1] TIBCO’s Integrated Knowledge Assessment System



Improved Information Sent to SMEs

Slide 24



Reduced Time-to-Market

Slide 25

One example: Standardized Automated Item Management

When SMEs began receiving standardized documentation , item quality skyrocketed. Much of the quality improvement was directly attributable to the use of MS Word tables.

QA improvement = fewer rejects, less rework = reduced time to market



Knowledge Assessment Status Diagram

Slide 26

A GREAT project

management tool



Lessons Learned

Slide 27

1. The MS Visio Knowledge Assessment workflow diagram provided the highest ROI. Much of which was due to detecting and resolving bottlenecks.

2. The covert enforcement of item-writing specifications implemented as MS Word tables provided the greater portion of item-quality improvements.

3. I implemented 30-50% more item-management macros than necessary to support the matured workflow.



Thank You!

Slide 28

achieving a better assessment development process

Technology

tibco courses tcp tibco

formal tibco training

tibco softwarewho

cloud computing environments

conference poll slide

assessment team

tibco global architects

knowledge assessments