WHITE PAPER BLACKBOX Multimodal Display Platform

Multimodal Interface Design as a Solution for Data Display, Data Fusion and Data Sharing Author Kurt Kratchman kurt@blackbox.tv Abstract – Today data comes at us from different directions in different formats, structured and unstructured. Viewing so much data and creating data narratives is challenging due to the various types of sources, supporting operating systems, hardware, software, platforms, devices, channels, formats, codec, streaming data, and proprietary data architectures, just to name a few. Fusing data is an enormous challenge and opportunity we are focused on solving. The display layer - the human computer interface - is the frontier this paper addresses. We attempt to provide context, frame the problem statement and offer a strategic paradigm and solution that is pragmatic, tactical and cost effective. The functional role of a multimodal display platform is to assist the end user in organizing and presenting multimedia content. At the lowest level, the display platform organizes disparate data types and organizes relevant incoming data. At the highest level, the system presents actionable intelligence to the user. In this paper, we use the BLACKBOX model as a reference for displaying data driven content, which utilizes multimedia fusion and multimodal characteristics to enhance user performance. The design allows for the user to passively monitor and actively interact with the display system with controls for interface customization, template creation, interactivity with web services, high end design and usability features, and a client / cloud based architecture. We present a new display paradigm that utilizes a multimodal interface to (1) take advantage of the human’s ability to fuse multisensory information, (2) facilitate situational awareness, (3) enhance the users cognitive task analysis and (4) share intuitively. Keywords: adaptive interface, cognitive task analysis, command control, cross platform, common operating picture, display bridge, display platform, force multiplier, fusion, information sharing, multimodal interface design, multimedia, situational awareness, synthetic vision, user interface, visual technology.

www.blackbox.tv 2

1 Introduction

“We are drowning in data…” The integration of senses by the human is a natural form of data fusion. Extensions to human processing include computer displays to guide attention, facilitate efficient task analysis, and provide cues for situation awareness. A computer is good at processing large amounts of object data while a human is good at making inferences over the data. Thus, the key question to ask is “What framework the human has to inference over the data and what capabilities allow for active reasoning?” To develop the user refinement capabilities to include in a design, four issues are paramount (1) situational awareness, (2) cognitive task analysis, (3) multimodal interface and (4) easy information sharing. The focus of this paper is multimodal interface and describes a highly scalable display model called BLACKBOX. BLACKBOX provides a multimedia interactive display platform that transforms the way information is communicated. BLACKBOX is specifically designed for fusion/command center analysts who need to view and analyze data from secure and public sources in context. With BLACKBOX dynamic data can be displayed by simultaneously force multiplying your existing screens with video feeds, web services, analyst reports, digital content, charts and graphs all on a single screen. BLACKBOX makes having your own mission control affordable and easy to set up. BLACKBOX gives IT and security organizations important enhancements that not only help reduce support and remediation costs, but can also provide a foundation for future solutions as security and communications needs change to support increasingly virtualized or “multi-tenant” shared data center resources. BLACKBOX is architected to be available as an externally or internally delivered application that has built-in application and content syncing and caching. Users can integrate a wide variety of content types and existing applications to be delivered within BLACKBOX.   2 Multimodal Interface Design Passive computer presentations might accurately present the fused data, however if the mental model of the human operator is not consistent with the display, the system usability might not improve performance. One way to improve operator performance is to design an interactive interface that allows for the capability to interact and refine the display. As an example, we might be able to refine the data collection so as to get more image perspectives for a 3D target presentation or multiple looks on the same target. When we interact with the system, the end user is transformed from a passive operator to an active user. Another way to improve human operator performance is to design a multimodal interface. Multimodal interface design (MID) is a concept that leverages our ability to filter incoming data. MID design is a key feature in military and

www.blackbox.tv 3

intelligence applications, security, aviation, simulators, weather analysis platforms, and video games. MIDs augment display information with control inputs, audio warnings, and some proprioceptive information. Such examples are audio warnings for threat assessment and control interactions affording the analyst the ability to cue in on data or drill down to areas of interest. Additionally, we might think of classifying image fusion selections as an extension to the MID design. For instance, allowing the user to select between electrical-optical (EO), synthetic-aperture radar (SAR), and infrared (IR) for day, high-altitude, and night missions, respectively, would enhance the multimodal visual capabilities. A key component of the MID is the ability to facilitate the analyst’s cognitive ability to fuse multiple image presentations for task completion. Most fusion systems are highly specialized, designed as narrow point solutions, and are expensive to build and operate, like the examples above. Some software packages are so complex they are sold with their own analysts and engineers. The typical Law Enforcement/intelligence fusion/command center cannot afford such luxuries yet the operators are being asked to evolve their skills to handle more time sensitive demands. What if the end user was empowered to select their own data sets, live feeds, PowerPoint files, web services, maps and other content for a fused display presentation with out support from software engineers and consultants? What if they could set up templates of existing and approved feeds and share them? This would assist in removing the need for time-constrained resources and let the experts manage what they know best. More times than not, the least dramatic fusions of data can have the greatest impact on day-to-day productivity. 3 Content is Data, Rich Media, and Multimedia As previously mentioned, the key to a multimodal interface system design is the ability of the system to convey information desired by the user. With the growth of rich media in all forms of communication and analysis, information technology systems have more to deliver for users who handle and analyze multimedia. Multimedia is media and content that uses a combination of different content forms. The term can be used as a noun (a medium with multiple content forms) or as an adjective describing a medium as having multiple content forms. The term is used in contrast to media, which use only rudimentary computer display such as text only, or traditional forms of printed or hand-produced material. Multimedia includes a combination of text, audio, still images, animation, video, or interactivity content forms. The key to consuming multimedia on display screens is the interface, referred to as a GUI (graphical user interface). Multimedia terminology has evolved over the years and conceptual terms such as rich media, or layered media, have entered the professional lexicon. More synthesized forms of content include the multimedia presentation of the assets organized into PowerPoint, web pages and other curated forms of digital content, often relied on during mission briefs and presentations. Digital content can be stored locally or on remote servers such as cloud based data. The data

www.blackbox.tv 4

fragments fused can be web services and conventional application files such as PowerPoint, web browsers, MP4 video assets and their players. (this seems repetitive of the first sentence) Compatibility between operating systems, file standards, proprietary software, and platforms make it difficult to fuse data into a common view that can be presented on a display screen. With our reliance of computer displays, interface design has become a critical area of focus to increase performance. The objective of designers and developers of interactive software systems is to align the mental model of the human with the display of the data. One way UX (user experience) designers improve operator performance is to design an interactive interface enabling the human operator the capability to interact and refine the display.

 Fig. 1 NASA Mission Control – Johnson Space Center

www.blackbox.tv 5

Fig. 2 NASA BLACKBOX

4 Force Multiply Command centers and fusion centers are designed to provide situational awareness and a common operating picture. Command centers such as NASA’s are proliferating around the world. Sensors, surveillance and intelligence solutions rely more and more on video imagery. Video displays are sharing the real estate with computer screens. As prices go down, the demand for large displays increases, driving new requirements and benefits. But most displays are not utilized to their full potential. Historically, splitting the screens and running multiple applications on a single view require additional hardware and software components. Consumer oriented Smart TVs are leveraging web services and web applications running through browser like application. While this makes the viewing experience more dynamic, it is quite narrow and limited. In mission critical command centers, rarely is budget spent on enhancing the usability of the content that these architectures provide. Most system integrators and IT professionals are not trained in user experience or interface design and such resources have not been made a priority in the designers of NOCs or fusion centers. Yet it is the human computer interface that is paramount to understanding what the data means in context. Data presentation that creates actionable intelligence saves lives and protects our security. Synthetic vision supports our air force pilots. Where is the synthetic vision support for the analysts who are piloting our security data centers?

www.blackbox.tv 6

5 The Market for New Ways to Communicate The market for new ways to communicate continues to grow and evolve dramatically. Organizations expect and are willing to pay for, seamless integration between their access to data, media and new technologies. BLACKBOX consolidates data driven content through its unified digital display platform. Few organizations have the resources or expertise to manage large internal teams of strategic, technical and creative personnel necessary to execute comprehensive integrated digital media solutions in-house. BLACKBOX empowers users to easily display multiple types of data and media tailored to varying audiences for maximum impact. BLACKBOX’s award-winning interface assembles new and existing technologies, enabling business and technical users to combine information maintained in numerous databases across various platforms. CIO’s and CFO’s are able to maximize their return on investment with dozens of features that enables end users to access data that might not otherwise be available or difficult to find. 6 Usage Models MAJOR MILITARY DEFENSE CONTRACTOR. A leading military defense and aerospace pioneer focused on intelligence, surveillance and reconnaissance solutions for the US military worked with BLACKBOX to develop a customized version of the BLACKBOX software as a multimodal interface paradigm for their wide ranging software R&D and drone programs. The problem for most of these companies is that they spend so much money on responding to RFPs, building prototypes too slowly, developing intense hardware, software engineering and system integration solutions that take months if not years of planning, design and development, that by the time the end user operator gets to interface with the system, they have run out of time or money to consider the adapting needs of the end user and the context they need to see the data in. More often than not, the needs of the ultimate end user are never studied or designed. Great ideas and solutions die because of a failure to develop an interface that makes the system usable. Hence interface design and usability together the next generation in advancement for all software solutions and BLACKBOX is helping to set the agenda. NASA. Imagine the need to obtain approval for $30 million in funding; knowing that the program you are trying to sell is extremely complicated and difficult for even management to understand. This is the task that faces the engineers on the Multi-Mission Space Exploration Vehicle Simulation program at NASA. The team at NASA knows that in order for their project to survive they need be able to demonstrate what they are attempting to accomplish with real data. Standard presentation tools were not even an option. We proposed our solution to NASA

www.blackbox.tv 7

highlighting the depth and power of BLACKBOX including the ability to translate a multitude of data points into graphs and charts (HDF5 conversion). To a common viewer the data is meaningless, but to an engineer trying to analyze real time and simulation data this is a valuable tool. The team at NASA recognized the value in being able to show live satellite feeds, adjacent to sensor data views in context with their plans and their mission – all as a means to get funding. Project approved. MAJOR EDUCATIONAL INSTITUTION. Private and public educators are forced to adapt and augment curriculums, pedagogy, and content distribution to remain relevant in a world of digital technology. A leading private content creator and publisher has a news program broadcast via satellite to 8,000 middle and high schools across the United States. The company even partners with the largest broadcasting company in the world to help produce their content. Traditionally, all of this content was broadcast on a single TV video screen in the classroom. Recognizing the need to keep up with new technologies the company contacted BLACKBOX to create an interactive learning platform. Desiring to engage students and empower teachers even more, the educational leader will run a pilot program utilizing BLACKBOX as a tool to augment content and provide a deeper educational experience. BLACKBOX will be used to merge content and active learning applications providing a more interactive approach to teaching and learning. BLACKBOX offers schools a powerful tool to improve teacher effectiveness. With the integration of interactive whiteboards, students truly become active participants in the daily news broadcast, interacting with digital content in real-time. GLOBAL AUTOMOTIVE BRAND. This company is known throughout the world for their innovations at every level in the global automotive industry. Unfortunately marketing communication tools and messaging has not kept pace with this rapidly changing market. A tailor-made version of BLACKBOX was created for the marketing teams to be used as a sales tool to access information about assets available to dealerships, as well as marketing resources that they might not otherwise know existed. Management discovered that BLACKBOX is also an effective tool for recruiting and employee on boarding. 7 How BLACKBOX Works BLACKBOX provides the ability to display a multitude of media in a single presentation screen: streaming video, web browsers, audio, RSS Feeds, Google Maps, PDF, HDF5 data, graphs…the list of application driven “pods” goes on and will continue to grow. There can be up to sixteen (16) pods on a screen. There can be an unlimited number of screens, or “acts” that are the rotating sides of a box. BLACKBOX allows this same content to be rearranged, overlaid or enlarged on demand. The BLACKBOX platform allows multiple types of content to be shared and distributed at the same time through an adaptive interface that allows users to simultaneously control the display and organization of any type of digital content.

www.blackbox.tv 8

8 BLACKBOX Architecture The BLACKBOX application runs as an externally or internally delivered application that includes built in application syncing and caching. It is built using Action Script 3 for the Adobe Air platform as a key technology. Adobe AIR is a cross-platform runtime environment developed by Adobe Systems for building Rich Internet Applications (RIA) that can be run as desktop applications or on mobile devices. The Adobe AIR Runtime is the core of the application delivery. Not unlike a JVM or other virtual machine, it provides a base layer for the application. BLACKBOX customers are provided with either an external or internal (behind the firewall) installation of the system. Installations can be a single machine running locally only or can consist of several machines with various synchronization patterns as best suits the needs of the customer. The BLACKBOX application is built on top of Adobe Air and provides core features including playback, editing, content and application synchronization and caching. The application is made up of several major components including:

• Authentication and Authorization. The application validates a single installation on a single machine and ensures appropriate authorization to access the installer.

• Update Manager. This is used to keep application versions consistent within an installation. Update manager also can push content as part of a regular update.

• Playback. The application is able to execute playback of a series of acts (pages) and pods (individual windows). Pods can consist of a wide variety of different media types and applications and can be expanded.

• Content Editor. This allows users update the content that will be played via each instance of the application. Through the update manager this will be pushed across the installations.

www.blackbox.tv 9

The BLACKBOX application development follows best practices established by Adobe Technical Services and partners. We use Cairngorm as the foundation on how we deliver successful enterprise Flex and AIR projects. Most large-scale applications within Adobe Technical Services are built by applying patterns and practices of Cairngorm 3 using Inversion-of-Control (IoC) containers. Design patterns advocated with Cairngorm MVC, such as encapsulating operations within command objects, declaring models and services externally, and separating presentation and domain concerns, can be applied with Inversion-of-Control containers to simplify development and testability. The following diagram illustrates the BLACKBOX software framework.

Fig. 3 BLACKBOX Framework Diagram

www.blackbox.tv 10

A Cairngorm application consists of:

• Architectural Layers that separate classes with different kinds of responsibility

• Functional Areas that group classes related to the same area of functionality

• Design Patterns that coordinate objects in consistent ways Architectural Layers Classes with different kinds of responsibility belong to different layers than can be changed separately from one another. So if the visual design needs to be tweaked, most changes are confined to the classes in a presentation layer, and not other parts of the code. Cairngorm recommends the layers described by Eric Evans in Domain-Driven Design:

1. Presentation - presents data to the user and gathers input. i.e. the fancy UI

2. Application - performs the operations of the application. i.e. saving a form 3. Domain - models the business concerns of the application. i.e. the form

data 4. Infrastructure - coordinates objects and integrates with other systems. i.e.

talking to the server Extensions The BLACKBOX architecture uses an extend relationship to specify that one use case (extension) extends the behavior of another use case (base), increasing product flexibility and adaptability. This type of relationship reveals details about a system or application that are typically hidden in a use case. While the base use case is defined independently and is meaningful by itself, the extension use case is not meaningful on its own. The extension use case consists of one or several behavior sequences (segments) that describe additional behavior that can incrementally augment the behavior of the base use case. Each segment can be inserted into the base use case at a different point, called an extension point. The extension use case can access and modify the attributes of the base use case; however, the base use case is not aware of the extension use case and, therefore, cannot access or modify the attributes and operations of the extension use case. The diagram below illustrates the BLACKBOX extensions.

www.blackbox.tv 11

Fig. 4 BLACKBOX Extension Diagram

Component Model BLACKBOX utilizes the Spark component architecture to support a seamless way to customize the visuals and behaviors of components in a much more direct and effortless manner. Spark is the name of the new component skinning architecture and component set that is part of the Adobe Flex 4 open source software development framework. The main tenet of the Spark architecture is a skinning model that provides a neat separation of a component’s visual elements from its logic. This separation gives designers and developers more freedom, because the visual elements of a Flex component can be designed in an unfettered manner, independent of the implementation of the logic powering the component. The following diagram illustrates the BLACKBOX Spark Component Model.

www.blackbox.tv 12

Fig. 5 BLACKBOX Component Model

Class Diagram BLACKBOX class diagram in the Unified Modeling Language (UML) describes the structure of the system by showing the system's classes, their attributes, operations (or methods), and the relationships among the classes. The class diagram is the main building block of object-oriented modeling. It is used both for general conceptual modeling of the systematics of the application, and for detailed modeling translating the models into programming code. The classes in the class diagram below represent both the main objects and or interactions in the application and the objects to be programmed. In the BLACKBOX class diagram below these classes are represented with boxes which contain two parts:

• The upper part holds the name of the class • The bottom part gives the methods or operations the class can take or

undertake

www.blackbox.tv 13

Fig. 6 BLACKBOX Class Diagram

9 API The BLACKBOX API provides a way to extend the core capabilities of BLACKBOX in a seamless way. The Open Pod provides a container that is similar to modern web browsers. HTML, CSS and JavaScript are used to present content and access BLACKBOX functionality through JSON. The API

www.blackbox.tv 14

includes the ability to access the raw HTML DOM. Taken together this provides rich hooks to content objects and functionality giving a simple mechanism that is widely used by software engineers as a means to build interesting extensions BLACKBOX. 10 Security When discussing security for BLACKBOX applications, it's first important to understand that this is a controlled delivery environment that is quite different from the security issues that exist when delivering Flash or Adobe AIR applications across public web sites. Most installations use the controlled environment as the first layer of protection that eliminates entry of malicious code. Coupling that with code signing from Adobe AIR provides a high degree of protection. Adobe AIR requires developers to digitally sign their applications in order to maintain the integrity of the software and the identity of its publisher. BLACKBOX has a certificate from one of the most respected Certification Authorities (CA) - Thawte. Digitally signing AIR files provides significant assurance to users that the application they are installing has not been altered. That said, BLACKBOX is being built to work in public installations that will have broader security concerns. Thus, we've architected it to leverage several of the core security features of the Adobe AIR framework: Sandbox The Adobe AIR runtime like other kinds of JVMs prevents code from running directly on the machine. Further, the Adobe AIR sandbox provides additional security that limits application access according to given security specifications. BLACKBOX leverages the sandbox to restrict access of the application. Memory Restrictions BLACKBOX also leverages the built in memory restrictions of Adobe AIR that prevent buffer overflow, memory corruption or other cross application memory issues. Data Encryption For added security we have initiated the Data Encryption to store local data in an AES-CBC128-bit encryption. This ensures that local data cannot be attacked or corrupted by malicious attacks. Further we restrict local data to non-script data, i.e., that is not executed. Rather it is only content oriented information that would cause playback issues but is not an entry for execution. HTML Security BLACKBOX is capable of rendering HTML that includes untrusted JavaScript. This is greatly restricted by the sandbox and the AIR runtime. References

www.blackbox.tv 15

Further reading around Adobe AIR security can be found in the following references:

• http://help.adobe.com/en_US/air/security/air_security.pdf • https://docs.google.com/open?id=15SBR17kOVz8kVADOBgIGqAacKe

jhNSCKSEMNf0J0igbtV-Bc3yJJy_DbGCYy  • http://help.adobe.com/en_US/air/security/index.html

11 BLACKBOX System Requirements The system requirements for BLACKBOX playback are as follows: Windows

• Intel® Pentium® III 1GHz or faster processor; Intel Pentium 4 2GHz or faster

• Microsoft® Windows® XP Home, Professional, or Tablet PC Edition with Service Pack 2 or 3 (including 64-bit editions); Windows Server® 2003 or 2008; Windows Vista® Home Premium, Business, Ultimate, or Enterprise (including 64-bit editions) with Service Pack 2; or Windows 7 (including 64-bit editions)

• 512MB of RAM (1GB recommended) MAC OS

• Intel Core™ Duo 1.83GHz or faster processor • Mac OS X v10.5 or v10.6 • 512MB of RAM (1GB recommended)

12 SUMMARY Most enterprise software tools execute only on functional requirements and system architectures designed in isolation of other point solutions, communication systems and end user contexts. The demands for integration and sharing across channels are more dynamic than ever. Today’s businesses and organizations need this kind of enhanced productivity to help optimize critical customer, employee, and financial data, and preserve systems infrastructure. This grows ever more crucial as companies adopt more virtualized, shared, and multi-tenant infrastructure models. With BLACKBOX enabled solutions you can:

• Leverage existing infrastructure and force multiply the views of data to help make informed decisions

• Improve productivity and operator effectiveness by fusing disparate data feeds into a multimodal interface

• Improve communications across the enterprise, with customers and constituents

www.blackbox.tv 16

• Give (consistency in tense) control to the analyst to customize their data views without needing support from designers, front-end coders, and IT for rapid data fusion - reduce strain on IT resources and limited budgets

• Share complex data views at multiple levels on multiple platforms • Address the increasing and evolving security threats across your physical

and virtual infrastructure through display enhancements • Facilitate compliance with government and industry regulations and data

protection standards • Reduce precious real estate costs by using less hardware • Reduce hardware costs by reducing the need for more displays and

dedicated computers - doing more with less • Reduce user fatigue by removing all the visual noise from the useful

content the operator needs to focus on “Few organizations have the resources or expertise to manage large internal teams of strategic, technical and creative personnel necessary to execute comprehensive integrated digital media solutions in-house.” Overall, BLACKBOX is enabling a significant opportunity for organizations to “future proof” their infrastructures. Using BLACKBOX-enabled solutions, they can get ahead of the curve of emerging technologies. IT organizations can gain important data display solutions for their growing virtualized environments to allow them to better control the flow of confidential, privileged, or sensitive workloads or data by restricting these to more thoroughly evaluated or trusted platforms. While at the same time, enable their customers to rapidly exploit new data feeds with minimum strain on existing infrastructure. While the near-term model will be the creation of up-sell and cross sell strategies amid their legacy system vendors, increasingly, BLACKBOX platform trust will grow on the ability to take advantage of the data to create unfair advantage for the “good guys.” Having the BLACKBOX platform enables a baseline level of assurance as systems are refreshed - essentially “raising the bar” for fusion and data center optimization, profitability and security over time. 13 Suggested Reading

1. EDWARD TUFTE, Visual Explanations: Images and Quantities, Evidence and Narrative, Graphics Press (February 1997), ISBN-10: 0961392126.

www.blackbox.tv 17

2. EDWARD TUFTE, The Visual Display of Quantitative Information, Graphics Press; 2nd edition (May 2001), ISBN-10: 0961392142.

3. EDWARD TUFTE, Beautiful Evidence, Graphics Pr; 1St Edition (July 2006), ISBN-10: 0961392177.

4. STEPHEN FEW, Information Dashboard Design: The Effective Visual Communication of Data, O'Reilly Media; 1 edition (January 1, 2006), ISBN-10: 0596100167.

5. JERALD HUGHES, Baruch College, CUNY and KARL LANG, Baruch College, CUNY, White Paper: “Transmutability: Digital Decontextualization, Manipulation, and Recontextualization as a New Source of Value in the Production and Consumption of Culture Products,” Proceedings of the 39th Hawaii International Conference on System Sciences, 2006.

6. DIMITROS TZOVARAS, Multimodal User Interfaces: From Signals to Interaction (Signals and Communication Technology), Springer; (November 25, 2010), ISBN-10: 3642097030.

7. OLIVIERO STOCK (Editor), MASSIMO ZANCANARO (Editor), Multimodal Intelligent Information Presentation (Text, Speech and Language Technology), Springer; 1st edition (April 29, 2005), ISBN-10: 1402030495.

8. ANDREW LOSOWSKY, Visual Storytelling: Inspiring A New Visual Language, Gestalten, Berlin, 2011, ISBN 978-3-89955-375-8.