development of a flexible architecture for interactive media façade applications

Development of Interactive Media Façade Applications with a Flexible Hard- and Software Architecture 5

Development of Interactive Media Façade Applications with a Flexible Hard- and

Software Architecture

Stephan Bergemann, Julien Letellier, Jens Reinhardt, Jürgen Sieck

Research Group INKA HTW Berlin

{stephan.bergemann, julien.letellier, jens.reinhardt, juergen.sieck}@htw-berlin.de

Abstract This paper starts with a short overview of the history of media façades, gives an introduction to several visualization and interaction tech-niques for buildings and architecture as well as discusses advantages and disadvantages of different examples. It then explains some of the technical background and finally it proposes an architecture that combines different hardware and software interfaces. This architecture enables students, us and third-party developers to build applications for our media façade, by using common standards.

6 Culture and Computer Science – Visual Worlds and Interactive Spaces

Figure 1 Schematic overview of the media façade at the FKI in Berlin

1 Introduction

1.1 Media façadesMedia façades create new connections between digital and urban space. Never before was there an interface between the physical and the digital world that appeals not only to individual users, as in the case of a personal computer, but also to a collective. [Tsch13] Existing media façades are often temporary and mostly pure passive only playing back videos, pictures, and other pre-rendered content. Interactive use cases for media façades are currently focused by many research institutions.Several installations have been described in the past allowing users to interact with media façades. In the Tower of Winds in Japan environment parameters like wind speed, direction and noise level are sampled and used to illuminate the tower accordingly [Haeu09]. Given the 20th anniversary of the “Chaos Computer Club” foundation, the Haus des Lehrers building in Berlin has been transformed into a monochrome dot-matrix display that could be remotely controlled by cell phones either sending text messages or calling a provided number and then pressing number keys for interaction [Blin13]. Also using


text messages but in combination with a slingshot device, the Spread.Gun [FZH10] and the SMSlingshot [FH12] projects were realised to virtually shoot messages at a projection façade. The ARS Electronica Center in Linz created an application to control their installed neon tube based media façade [Bori11]. Using a smartphone and touch gestures users were able to draw on the building. This was realised using camera tracking in the smartphone and mapping the touch events on the building. All these projects used different interaction methods and protocols for data interaction. These examples show the usage of a wide range of input data and methods. There exist message based systems such as Spread.Gun and SMSlingshot as well as button based input at the Haus des Lehrers and finally live video streaming and touch gesture based applications like the drawing application for the Ars Electronica Center.

1.2 Research Center for Culture and Computer Science (FKI)The façade of the FKI located at Wilhelminenhof campus of HTW Berlin (University of Applied Sciences) not only contains a photovoltaic system, but also a media façade. This media façade will not be temporary and is not designed to display only a single application, but is a permanent installa-tion being open for students and staff members at the university, interaction designers and video artists to realize their applications within an interactive media development environment. One special feature of this media façade is that it does not consist of one single screen area, but combines different display technologies to gain a bigger area. Parts of the façade have LEDs, rear projection with video projectors and ambient lights, see figure 1. About 30,000 RGB-LEDs cover an area of approximately 7 by 3 meters of the north façade. The windows are projected by 12 video projectors on the north and another 12 projectors on the south façade are planed with a full HD resolu-tion. The remaining space between the windows of the first and second floor and besides and the LED-Grid is covered by 36 LED-Spots that generate an ambient light. The usage of different visualization technologies raises a few key questions regarding the resolution, display of continuous content on the heterogeneous sub systems, and finally capabilities to interact with either one application for the whole façade or different synchronized applications distri-buted to a subset of output devices serving parts of the façade.


2 Visualization and interaction technologies

As of the characteristics of buildings and the diversity of display technologies for buildings, it is not always possible to cover a whole façade with a single projection area by one output device. In general, multiple output devices have to be combined in order to cover the whole façade where each one provides a part of the whole image to be displayed (segmentation). For images and even more videos and interactive applications, synchronization of the visualizations is a crucial requirement where its segments have to be played back at the same time.

2.1 Display technologiesLight-emitting diodes (LED) are used on a variety of equipment and installa-tions since they are small, efficient and can cover a wide spectrum of colors when using red, green and blue LEDs. Depending on the amount and the arrengement of the LEDs a high resolution can be achieved. One drawback are the high initial costs for large screens with many diodes. With video projectors it is possible to project moving images onto a projec-tion screen. The images can be projected either onto the front façade of the building from the outside (front projection) or from the inside, typically using the windows (rear projection). Depending on the projector, the resolution is very high, but the brightness compared to LEDs is considerably low (or makes the projectors very expensive, usually bigger and also noisy). So when using standard projectors for front or rear projection, the ambient light level has to be as low as possible.

2.2 Visualization technologiesMost façades will not only display prerendered content, but also live and interactive media. When using multiple screens and display technolo-gies, synchronization, scaling, and transition challenges must be solved. ClusterGL21 is a library accompanied with some utility programs that tries to solve some of these challanges by intercepting the rendering commands form a given application and distributing them to other computers via network. One drawback is that developers must have extensive programming and basic mathematical skills, because ClusterGL2 can only work with OpenGL applications using the overall implemented subset of OpenGL-Commands.

1 http://clustergl.org/


If all OpenGL-Commands defined by the Standard(s) would be covered by ClusterGL2, also existing Frameworks like OpenFrameworks2 or existing game engines like Unity3 might become useful for implementing applications.

2.3 Interaction technologiesThere are many hardware devices and sensors that can be used by visitors to interact with a media façade: mobile devices, like smartphones or tablets; individually developed hardware or do-it yourself (DIY) hardware, like custo-mized Arduino4 devices or passive hardware, like microphones, cameras or the Kinect5 are just a few. Mobile devices are the most promising because they have a wide distribution and a lot of people carry smartphones or tablets with them.Some of these technologies have to be brought to the façade and others might also be provided by the façade. In the case of the FKI we decided to install a camera into the façade and add a wifi hotspot for outdoor network.

3 Requirements engineering

After describing this technical background, ideas are presented that will focus mostly on the question why people should interact with buildings, architec-ture, or urban screens and how interaction methods can be enabled in form of hardware and software interfaces. Therefor we prepared and conducted a survey to evaluate the necessity of different sensors, supported programming languages, primarily used software tools and more as we plan to establish cooperations with other research institutes. Focusing on the local FKI media façade, types of supported applications and interaction forms are described as well as participation methods for developers, designers, media artists, and visitors.A major group of people developing for and interacting with our media façade will be students. Several groups of students at the university of applied sciences as well as our staff members have already developed prototypical applications in the past few semesters, exploiting the different options for interacting and visualizing their content. These applications were imple-2 http://www.openframeworks.cc/3 http://unity3d.com/4 http://www.arduino.cc5 http://www.kinect.org


mented lacking the possibility to deploy it on the real façade as it was still under construction but gave us an impression of what students expect in terms of interaction interfaces. These use cases formed the foundation for the final system and where extended by further application concepts to fulfill the needs of all defined target groups.Some of the interactive applications where using web and/or smartphone based applications to interact with the media façade. This lead to the conclusion, that we would need outdoor wireless network or have to rely on UMTS beeing available in the area around the building. Other applications used camera and thereby gesture based interac-tion input methods to control their applications.

Figure 2: Schematic overview of technical infrastructure


Therefore we installed a camera that currently is able to monitor the area in front of the building when needed for the application to calculate on its images. As of the different use cases for our façade (at night the whole façade can be used, including the rear projection areas, whereas during day time only the LED based areas are usable) there was also always the decision to make for the students, whether to implement their application on the low resolution but usable during daytime LED panels, or to implement their appli-cation for the whole façade. This also implied the decision, to either think about the proper distribution of their application (e.g. using ClusterGL2), or being able to use whatever Software they liked but only display their applica-tion on a very small part of the façade.Already implemented and further possible applications are:

• Classical games, like Pong, Space Invaders and Tetris• Multi-user and collaborative games (e.g. Paint It [BGHK13], Line-

Runner [GSVT13])• Visualization of environmental information (e.g. sonic, weather, data

from the photovoltaic façade), conference and event information

4 Proposed architecture

Tracking down the complexity of the described ideas in terms of their interac-tive needs, an application and library architecture was implemented to flexibly solve the above described occurring challenges of combining multiple output devices, providing access to diverse sensor input data (the architecture allows the integration of indoor (e.g. ambient light, sonic, temperature), outdoor (e.g. camera, weather, sonic), and mobile sensors and devices (e.g. do-it-yourself (DIY), smartphones including their numerous sensors)) and finally deploying applications on the facade.One central element of the architecture is an application launcher, that is capable of starting and stopping applications by either randomly switching between non-interactive applications (like different visualizations) or selec-tively starting interactive applications all in their appropriate environment. This launcher also keeps track of the fact, that each application may only run a certain maximum time.As described in [BS13] there have been some prototypical applications implemented showing the general capabilities of OSC and RTP for inter-


actively controllable applications. OSC is an easily adoptable and therefor usable message format. RTP describes a protocol mainly used in conferen-cing systems. It is therefore capable of transporting various multimedia data formats with a clean focus on realtime processing. Combining OSC and RTP enables application developers to implement a vast amount of different multi-media application scenarios.The architecture also includes a component that handles the interactive data from the client applications. The communicator was implemented using Node.js, a system that uses JavaScript as the programming language and implements the concept of non-blocking IO, so each file operation or network request is not handled in separate threads but stored in a form of a queue that is processed sequentially. This allows for a large number of connections and is an ideal solution for media façades. It is possible to connect to the Node.js application via different protocols (e.g. OSC or RTP) and from various platforms. This makes it easy to implement web applications for example. [Lete13] describes this part of the architecture in detail.

5 Conclusion and future work

Media façades provide a way to transport art, information and multimedia content. Even interactive installations can be integrated right into the façade using plenty of different input devices and provide enhanced possibilities to turn a regular building into a memorisable one. However, current research only has investigated in different single application approaches that are usually implemented only for one specific purpose, e.g. festival or trade fair, and that are usually temporary installations. For the media façade of the FKI we developed an architecture that works with a wide range of display tech-nologies. The architecture combines several standards and enables external developers to quickly build interactive applications for the façade. Our future work besides implementing a wide range of different applications exploiting this new architecture for our façade will include the development of a web-based platform to upload applications, specify their settings and capabilities to provide this information to users that want to use the applica-tions on the façade with their own controllers.


6 Acknowledgement

This paper describes the work undertaken in the context of the project SIGNAL hosted by the research group Information and Communication Systems INKA that is generously funded by the European Regional Develop-ment Fund (ERDF).

Literature

[BGHK13] Buff, Andreas; Gerlach, Thomas; Helbig, René; Kampe, Toralf; Kasimova, Gyuzel; Müller, Arend; Paeschke, Markus; Sieck, Jürgen: Interactive Media Facade Web Application with ClusterGL, 2013 (unpublished).

[Blin13] blinkenlights.net. Project Blinkenlights. url: http://blinkenlights.net, Last visited: April, 6th, 2013.

[Bori11] Boring, Sebastian et.al. Multi-user interaction on media façades through live video on mobile devices. In: Proceedings of the 2011 annual conference on Human factors in computing systems. CHI ’11. Vancouver, BC, Canada: ACM, 2011.

[BS13] Bergemann, Stephan; Sieck, Jürgen: Protocol- and Systemdesign for Interactive Media Façades. In: Informatik, Kultur, Technik 2013. Odessa, Ukraine.

[FH12] Fischer, Patrick Tobias; Hornecker, Eva: Urban HCI: spatial aspects in the design of shared encounters for media façades. In: Procee-dings of the 2012 ACM annual conference on Human Factors in Computing Systems. CHI ’12. Austin, Texas, USA, 2012.

[FZH10] Fischer, Patrick Tobias; Zöllner, Christian; Hornecker, Eva. VR/Urban: Spread.gun - design process and challenges in developing a shared encounter for media façades. In: Proceedings of the 24th BCS Interaction Specialist Group Conference. BCS ’10. Dundee, United Kingdom: British Computer Society, 2010, pp. 289–298. isbn: 978-1-78017-130-2.

[GSVT13] Gropengießer, Uwe; Sellmann, Marcus; Vermeulen, Fokke; Thiele-Maas, Michael; Sieck, Jürgen: Gaming with media facade, 2013 (unpublished).

[Haeu09] M. Hank. Häusler. Media façades: history, technology, content. avedition, [Ludwigsburg, Germany], 2009. isbn: 9783899861075 3899861078.

[Lete13] Letellier, Julien: Development of an Architecture for Interactive Media Façades. In: Modern Information Technology 2013. Odessa, Ukraine.


[Tsch13] Tscherteu, Gernot: Media Facades: Fundamental terms and concepts, http://www.mediaarchitecture.org/mediafacades2008/exhibition/basics-eng/, Last visited: April, 6th 2013.

development of a flexible architecture for interactive media façade applications

Documents