Bridging the Gap BetweenObject-Based and

Narrative-Based Storytelling

Axel Bremer11023325

Bachelor thesisCredits: 18 EC

Bachelor Opleiding Kunstmatige Intelligentie

University of AmsterdamFaculty of ScienceScience Park 904

1098 XH Amsterdam

Supervisordr. ir. J. Kamps

Capaciteitsgroep Media & CultuurFaculty of Humanities

University of AmsterdamTurfdraagsterpad 9

1012 XT Amsterdam

June 29th, 2018

1

ABSTRACT

Object-based and narrative-based storytelling both have strengths and weak-nesses. Narrative-based storytelling misses the interaction with the objects,and object-based storytelling makes it hard to portray the story behind the ob-jects due to visitor’s museum fatigue. This thesis proposes a solution to bridgingthe gap between these methods by creating an application that integrates theobject in the narrative-based method, in this case, a museum catalog. This so-lution allows creators of exhibitions to enhance the catalogs for their exhibitionin an easy way. It even provides a method for retrospective enhancing of alreadyexisting catalogs.

2

Acknowledgement

I would like to thank Jaap Kamps for helping me come up with the ideas forthe applications made for this thesis. I would also like to thank Wim Hupperetzand the Allard Pierson museum for letting me use the Crossroads exhibitionand taking the time to look at the application demos.

3

Contents

1 Introduction 5

2 Related work 8

3 Approach 103.1 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 First Two Demos 124.1 Virtual Museum Application . . . . . . . . . . . . . . . . . . . . 124.2 Augmented Reality Application Version 1 . . . . . . . . . . . . . 15

5 Augmented Reality Application Version Further Development 205.1 Development of Version 2 . . . . . . . . . . . . . . . . . . . . . . 205.2 Development of Version 3 . . . . . . . . . . . . . . . . . . . . . . 23

6 Discussion and Conclusions 296.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Appendices 32

A Code 32A.1 Narrative XML file . . . . . . . . . . . . . . . . . . . . . . . . 32A.2 AnnotationScript.cs . . . . . . . . . . . . . . . . . . . . . . . . 32A.3 PinchZoom.cs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32A.4 TouchRotate.cs . . . . . . . . . . . . . . . . . . . . . . . . . . . 32A.5 Process.py . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33A.6 VuforiaHandler.cs . . . . . . . . . . . . . . . . . . . . . . . . . 33A.7 AromaTrackableEventHandler.cs . . . . . . . . . . . . . . . 33

B Figures 34B.1 Contrast image versions . . . . . . . . . . . . . . . . . . . . . 34

4

1 Introduction

The primary role of a museum is telling a story about events from the past.Remnants of cultures past, objects of cultural importance and many other thingsare found around the globe and museums often are the institutions that takethe task upon them to research these objects, preserve their stories and portraythose stories to the people that want to know more about the history of theworld. The way this is done can be divided into two categories; object-basedstorytelling and narrative-based storytelling, which will be exemplified in thenext paragraphs. Both of these categories have their strengths and weaknesseswhich will be discussed below.

Firstly, object-based storytelling will be discussed. An example of object-basedstorytelling could be an exhibition in a museum gallery. A group of objects thatshare a specific connection together is displayed in a room, and each object hasa small bit of information given about it.

Figure 1: Example of object-based storytelling

One significant advantage of this method is that visitors can examine the objectin great detail because the actual object is right in front of them. However, itdoes have a significant disadvantage concerning storytelling. Museum visitorsdo not have time to read a long story about every object in the room, becausethey can only spend a certain amount of time in the museum. So the onlyinformation given is a little information about the object itself and sometimesa small explanation about how these objects are connected to each other. Dueto the museum fatigue[6] of museum visitors, the exhibition becomes about theobjects more than the narrative behind the objects. Hence the name, object-based storytelling.

The other category is narrative-based storytelling. Its strengths and weaknessesare opposite to those of object-based storytelling. An example of narrative-based

5

storytelling is the Crossroads book which pertains to the Crossroads exhibition[1], two pages of which can be seen in figure 2

Figure 2: Crossroads, an example of narrative-based storytelling

This method of storytelling, as the name would suggest, focuses much more onthe narrative behind the objects and uses the objects to enhance the narrativerather than use the information to enhance the objects. The advantage of thismethod is that the reader can take their time to read about this narrativebecause a book like this can be read at home at the readers own leisure. Thedisadvantage of this method is the lack of interaction with the object. Readerscan only examine the object as a 2D photograph instead of seeing the actualobject and being able to see it from all angles.

The problem concerning these two methods is that they are seen as mutuallyexclusive. But creating a method which mixes these two methods should bepossible. Which is why, in this thesis, we will try to answer the followingquestion:

How can we bridge the gap between object-based andnarrative-based storytelling?

This main question is divided into three sub-questions:

• What are current approaches to object-based and narrative-based story-telling?

6

• How can we bring the narrative to the object, or bring the object to thenarrative?

• Is augmented reality with 3D models effective to augment the narrativesin a traditional museum catalog?

We will start in chapter 2 by looking at the work that is already being doneregarding object-based and narrative-based storytelling. After that, chapter 3will cover the software and data used to create the applications whose demoswill be covered in chapter 4. Chapter 5 examines the further development ofthe best of the two demos.

7

2 Related work

The question that will be looked at in this chapter is:

What are current approaches to object-based and narrative-based storytelling?

Firstly we will look at what is being done by the Allard Pierson Museum re-garding object-based and narrative-based storytelling by looking at the Cross-roads exhibition. This exhibition shows the different ways a museum can tell astory.

Object-based Crossroads is a traveling exhibition focusing on connectivityand cultural exchange during the Early Middle Ages in Europe.[5] A part of theexhibition consists of the standard ‘objects in a room’, which is the object-basedmethod mentioned earlier.

Narrative-based Crossroads can also be enjoyed in book-form. As stated inthe preface of the book:

The international Crossroads project connects European culturalheritage as it emerged between AD 300 and 1000. In this project, dif-ferent narrative contexts are explored in terms of continuity, changeand entanglement, taking into consideration the effects of the con-verging pagan and Christian influences as the transition was madeinto a predominantly Christian society which transformed the EarlyMiddle Ages. The exhibition is presented through specially selectedmuseum objects, displayed thematically, and different media such asan interactive interpretative mapping tool, the cross-culture time-line.

This book is more than just a catalog because it shifts its focus from the objectsto the stories behind the objects.

Mixing of the methods The Crossroads exhibition already has some wayswith which it tries to mix the narrative-based and object-based methods. Therewas a digital application developed specifically for this exhibition. The applica-tion is called the Cross Culture Timeline. Which shows, using large projectionsof maps on the wall, the connections between different objects. It is made toshow the diversity of the objects, as well as the links that tie them all together.[9]Another way they tried to enhance the exhibition was by creating holographicanimations which bring four key objects from the exhibition to life. Anima-tions were also made to show how some of these objects were used in the past.[2]

Concerning Augmented Reality Augmented Reality is an up and comingphenomenon in museums. There are a number of museums which use Aug-

8

mented Reality in their exhibitions. Fossils are being brought back to life, visi-tors can look for and catch plants and animals in a museum, and holograms ofastronauts can be seen doing a spacewalk all using Augmented Reality.[4]

These are all examples of exhibitions being enhanced using technology. Theseenhancements often require quite a bit of work and are made especially for aspecific museum, collection, or exhibition. In this project, a solution is offeredthat requires little configuration when changing it for use in a different museum,collection, or exhibition.

In this chapter, we studied the question: What are current approaches to object-based and narrative-based storytelling?. Our main findings are the following.First, the Crossroads exhibition at the Allard Pierson museum shows a few waysof telling a story using both methods and also a mix of these methods. Second,there is a fair amount of Augmented Reality being used in museums at themoment. However, most of these are specifically made for certain exhibitions.In the next chapters, we will introduce an application which is easily configuredfor other collections.

9

3 Approach

In the next chapter, we will investigate our second research question by creatingtwo applications that have different approaches to mix Object-based and Story-based narrative telling. On the halfway point of the project, the two applicationswere evaluated, and one application was picked to be further developed. Inthis chapter, we will look at which software and data are used to create theseapplications. The first application is a type of virtual museum showing 3Dmodels and their corresponding stories. The second application is an AugmentedReality application that uses already existing catalogs of exhibitions.

3.1 Software

To create these systems the Unity 3D engine is used. Unity is a cross-platformengine used primarily to develop three-dimensional and two-dimensional gamesfor a multitude of devices.[14] Unity was picked for its ease of scripting (usingthe C# language) and for its cross-platform capabilities. This cross-platformproperty is useful for making these systems as this helps with releasing thesystems on both major mobile operating systems (iOS and Android).

For the Augmented Reality part of the system, the Vuforia Augmented RealitySDK is used. This is a Software Development Kit for Augmented Reality onmobile devices. Vuforia supports multiple types of targets which can bind the3D models to the real world. 2D image targets, 3D targets of multiple images,basic 3D object targets, and targets based on 3D models.[10] In this system,only the 2D image target is used, but the application could easily be extendedto use other types of targets.

3.2 Data

The data used in this project consists of a collection of 3D models of the Cross-roads exhibition set up by the CEMEC project.[5] (Connecting Early MedievalEuropean Collections) The 3D models are high-quality scans of the objectsthat were in the Crossroads exhibition at the Allard Pierson Museum and thatare now exhibited at the Byzantine and Christian Museum in Athens (April-September 2018). Some of these models are annotated with data about certainaspects of the object. The models are uploaded on Sketchfab, an online 3Dmodel sharing website that has an online viewer and editor as well as an APIto integrate those features in other applications.[11]

10

Figure 3: Incense burner viewed in Sketchfab viewer with annotation.

The Sketchfab API would allow us to dynamically download the models andtheir annotations from Sketchfab during the application runtime. The downsideof this is that if this were to happen, each end user would have to log in usinga Sketchfab account. To keep the applications as easy to use as possible ithas been decided not to make us of the Sketchfab API at this moment and todownload the models and add them directly to the applications. Contact isbeing made with Sketchfab about the ability to use the API without needingend users to log in so there is a possibility that at the time of the actual launchof the application the Sketchfab API will be used.

The narratives that belong with the Crossroads exhibition is recorded in thebook of the same name. This book is more than just a catalog of objects. Itis most of all a book filled with stories which are enhanced with the use of 2Dphotographs of objects. That is what makes it a great example of the narrative-based storytelling.

Evaluation of the applications was done by letting a number of potential users(the amount ranging from 5 to 10 depending on the version) play around withthe system. The potential users were observed while they tested the softwareand afterwards, they could give feedback concerning the usage of the applica-tion.

11

4 First Two Demos

This approach will be used to answer the following question in this chap-ter:

How can we bring the narrative to the object, or bring the object to thenarrative?

Two basic ideas for applications were proposed during an early brainstorm ses-sion for this project. One to better incorporate the narrative in the object-basedmethod, A Virtual Museum application, and one to integrate the object in thenarrative-based method, an Augmented Reality application. Two very basicdemos were to be made of these ideas to decide which one would be developedfurther. When meeting with the director of the Allard Pierson Museum, WimHupperetz, he showed far greater interest in the Augmented Reality applicationthan in the Virtual Museum application. After this meeting, it was decidedthat the focus should lie more on the Augmented Reality application, whichwas therefore developed further.

The interest shown in the Augmented Reality application was significant enoughthat it was decided to enter this application in an App Challenge. This Diggingfor Data App Challenge is organized by the province Zuid-Holland to piquepeoples interest about cultural heritage and archaeological findings. On the22nd of June, a 2,5-minute pitch and demo were given to get picked to wina D20.000,- development budget. Unfortunately, this application was not thewinner that day.

In this chapter, both the applications will be shown. First will be the VirtualMuseum application and its corresponding Toolkit. After that, the multipleversions of the Augmented Reality application will be shown in chronologicalorder.

4.1 Virtual Museum Application

One of the two starting ideas was the virtual museum. The thought behind thiswas that people who did not have access to a certain collection could enjoy thisnarrative from the comfort of their own home. However, most of all it was to bea proof of concept for the toolkit that lets storytellers easily create a narrativeand have it stored in an easy to use digital form. At the moment the way objectsare saved in a digital form has much information about the object itself but nota lot about the story in which this object is interwoven. [3]

The Allard Pierson Museum does have an online collection of stories in whichobjects are interwoven in a digital form. Take for example the Near East story[8]. It tells a story about the Near East and contains links to object wherevernecessary. However, this links takes the reader to a 2d photograph of the objectand some information about the object itself and nothing more.

12

The objective of the virtual museum application was to give the user the sameexperience of reading the story but give them an even more interactive methodto discover the objects. By using the high-detail 3D models of the objectsin question the user can see a lot more of the object than from a 2d photo-graph.

Figure 4: Proof-of-concept demo Virtual Museum Application

The left and right arrow buttons load the next object in the narrative, the zoombuttons make the object bigger and smaller, and the up and down arrows rotatethe object along the horizontal axis. Rotation along the vertical axis is beingdone automatically. This choice was made because more buttons would clutterup the user interface and take the focus away from the object. At this point,the change to touch input has not yet been made.

The main point of this proof of concept demo was to test a new way of digiti-zation of stories. This is realized by creating certain Narratives in XML files.Such a Narrative would consist of a collection of objects each with its own storyas to how it fits in the Narrative. Each of these objects can also be linkedto another object with a specific type of link. For example, the objects couldhave been used for the same purpose, used by the same subculture or foundin approximately the same location. When observing an object, the user canchoose to see the next object in the narrative or to see one of the objects thatare linked to the current object by pressing the buttons to those objects as canbe seen in figure 4.

13

Figure 5: Flowchart of the Virtual Museum Application

These objects, their stories and their links are saved in an XML file. This is anexample of how such a file would be formatted:

14

<Narrat ive>

<IntroText>This i s the i n t r o text to the na r r a t i v e</ IntroText>

<Period>Period in which the Narrat ive happens</Period>

<!−− O b j e c t s u s e d t o t e l l t h e n a r r a t i v e −−>

<Objects>

<Object id=”0”> <!−− The i d o f t h e o b j e c t −−>

<Name>Name of the ob j e c t</Name>

<FileName>Name of 3D model and tex tu r e s</FileName>

<Summary>Short summary about ob j e c t</Summary>

<Text>Longer Text about how the ob j e c t connects to the na r ra t i v e .</Text>

<!−− How t h e o b j e c t i s l i n k e d t o o t h e r o b j e c t s i n t h e n a r r a t i v e . −−>

<Links>

<Link>

<ObjectId>Id o f l i nked ob j e c t</ObjectId>

<ObjectName>Name of l i nked ob j e c t</ObjectName>

<LinkType>how i t i s l i nked to that ob j e c t</LinkType>

</Link>

</Links>

</Object>

<Object id=”1”>

. . .

</Object>

<Object id=”2”>

. . .

</Object>

</Objects>

</Narrat ive>

Figure 6: Example of Narrative XML formatting

For a more concrete example see appendix A.1, this XML file is based on a partof the Near East story from [8].

4.2 Augmented Reality Application Version 1

The second demo to be made was that for an Augmented Reality application.The goal of this application is to mix the two storytelling methods by incorpo-rating the 3D models into narrative sources that are already written down usingAugmented Reality.

Version 1 was made using the two downloadable 3D models from the CEMECsketchfab account, an incense burner and a St. Menas flask. It was designed touse the Google Cardboard Augmented Reality Viewer[7] to inspect the 3D mod-els in Augmented Reality. It was made to test if viewing 3D models of MuseumObjects in Augmented Reality is on par with viewing them in a museum.

Vuforia offers a collection of sample image targets which are made for properfeature point detection. Using the developer portal one can check what featurepoints Vuforia has found in the image targets. These feature points are usedfor target detection and target following. On these targets, the models can beaugmented and shown to the user.

15

Figure 7: Drone sample image target and its features

Although the feature point detection algorithm Vuforia uses is not specifiedin their documentation, the documentation does specify to use image targetsthat are rich in detail, have good contrast and do not have repetitive patterns.Shown in figure 7 is one of the sample image targets offered by Vuforia. As canbe seen in the picture, most of the features are found in the sharp edges of theimage.

16

Figure 8: Unity Scene view with image targets and models

Figure 9: Google Cardboard Version 1 Incense burner

The image targets are added to the Unity scene, and the models are added aschildren to the image targets as can be seen in figure 8. When the programdetects one of the image targets from the database, it finds all the child objectsand enables their renderers to make them visible in the world space. Thisallows the user to move around the image target and see the object from all

17

angles.

The blue and black spheres that can be seen on the incense burner model arethe annotations which tell more about that particular part of the object. Usingthe script found in appendix A.2 the closest annotation is found and turnedblack to show the user that it is the closest one. If the user taps the screen,the text belonging to that annotation is displayed on the screen (figure 10) andwith another tap the screen is cleared.

Figure 10: Google Cardboard Version 1 Incense burner with annotation

Evaluation After letting some users play around with the system using theGoogle Cardboard viewer the consensus was that viewing museum objects throughAugmented Reality may not be on par with viewing them in a museum, butit is a good alternative for when one is at home. The other opinion that a bigpart of the users had was that the Google Cardboard element had no addedvalue. It was not that easy to use and therefore distracted the user from ac-tually viewing the object. The resolution of most smartphones is also not highenough for the right amount of immersion when viewing the object throughGoogle Cardboard.

After meeting with the director of the Allard Pierson Museum where the demosof both the applications were shown it was decided that the Virtual Museumapplication would not be developed any further and all further attention shouldbe focused on the Augmented Reality application. This decision was madebecause the Augmented Reality application showed more promise and generallygot a more enthusiastic reaction from users.

In this chapter we tried to answer the question: How can we bring the narrativeto the object, or bring the object to the narrative? Two application demos weremade for this purpose. The demo of the Augmented Reality, which brings the

18

object to the narrative, was picked to be developed further. This developmentshall be covered in the next chapter.

19

5 Augmented Reality Application Version Fur-

ther Development

In this chapter, we will examine the further development of the AugmentedReality application, which was picked from the first two demos to be developed.It will be an application that enhances a regular catalog of museum objects.In this particular case, it enhances the already existing book of the Crossroadsexhibition, also named Crossroads.[1]

When a user is reading the book and wishes to examine one of the objects closerhe or she would just have to take out their phone launch the app and aim it atthe book to see the 3d model and examine it more closely. With this in mind,the question we are answering in this chapter is:

Is augmented reality with 3D models effective to augment the narratives in atraditional museum catalog?

This is being done by improving the demo application with new features andletting them get evaluated by potential users.

5.1 Development of Version 2

One significant advantage of this application is that because of the usage of theVuforia platform any image can be used as an image target. This means thatthe application could be made retrospectively for any catalog one would wantas long as there are 3D models of the objects in the catalog.

The next step was to pick which part of the book would be the image target.Figure 11 shows that an entire page has a lot of usable features. However,this does not guarantee a good image target. Apart from the features Vuforia’sdeveloper portal also gives a rating of how augmentable the image is. The full-page got a rating of 0 out of 5. This rating and the fact that using the wholepage would make the application language dependent was a reason for not usingthe entire page as the target.

20

Figure 11: A page as seen in the book and it’s usable augmentation features

The image targets used in this version are the photos of the objects as theyappear in the book. They are extracted from a pdf version of the book usingthe Microsoft snipping tool and are uploaded to Vuforia. Figure 12 displaysthe number of features found in the Menas Flask picture and shows that thereare not a lot of usable features. However, when comparing the performanceof the single picture target versus the entire page target, it is found that theindividual picture is more easily recognized than the entire page and has a morestable performance overall. It does not disappear as often as the full page target,and it does not cause jitter as much.

Figure 12: The photo as seen in the book and it’s usable augmentation features

21

Evaluation Despite providing a more stable performance than its full-pagecounterpart, some images, for example, the Menas flask, still not had a perfor-mance as stable as one would need to enjoy viewing the object. Testers reportedthat the jittering of the object was too distracting to examine the object in anormal way. In figure 13 the incense burner image target can be seen. This is atarget that gave a stable performance and allowed users to examine the objectto their leisure.

Figure 13: The photo as seen in the book and it’s usable augmentation features

In version 2 there was not yet a way of controlling the object when viewing it.To see all sides of the object one would have to walk around the book or movethe book itself physically. When potential users were testing the application, itwas found that most of them immediately tried to rotate the object and zoomin on it using touch gestures.

Figure 14: Version 2 of the app using the Menas flask image as target

22

5.2 Development of Version 3

In version 3 of the application, three updates were made; touch controls wereadded; image targets were improved; target and model loading was improved.

Touch controls The object being viewed can now be rotated and zoomedinto using touch input. Using the Touch scripting API[13] the user can zoomin and out by pinching on the screen, and the user can rotate the object bydragging their finger across the screen. See appendix A.3 and A.4 for the scriptused to control this.

Improved image targets The low amount of features found in some images(figure 12) caused the application to be near unusable for those models. Vuforiadocumentation claims that image targets with higher contrast make for betterimage targets. Using the GIMP image manipulation program[12] the contrastfor the image targets was raised from 0 to 55 for a high contrast version of theimages and from 0 to 100 for a full contrast version. The differences betweenall versions can be seen in figures 12, 15 and 16.

Figure 15: The photo with higher contrast and it’s usable augmentation features

23

Figure 16: The photo with full contrast and it’s usable augmentation features

The high contrast version shows a lot more feature points than the normalcontrast version, it also still shows a good resemblance to the actual picture inthe book. The full contrast version shows an abundance of feature points andgets a full 5-star rating on augmentability. However, the image is visually verydifferent from the picture in the book which may impede recognition duringruntime.

To test which level of contrast performed best all three the versions were addedto the image target database simultaneously. The application is started withthe camera pointed away from the target image. The camera is then pointedtowards the target image, and the contrast versions are written down in theorder they triggered. Normal, high and full contrast are written down as N, Hand F respectively. This was done for the incense burner, oil lamp, and Menasflask. Every version of these image targets can be seen in appendix B.1. Theresult is shown in table 1.

24

Incense Lamp Flask

1 H H N H

2 H N H N H

3 H N H F N H F

4 H F N H F N H

5 H N H F N H F

6 H N H H

7 H N H F N H

8 H N H H

9 H H H

10 H N H N H

11 H H N H

12 H N H H

13 H F N N H

14 H N H F H

15 H H H

Table 1: Order of detection of normal (N), high (H) and full contrast (F)

All of these detections occurred within 1 second of moving the camera to facethe image target. In every case but one the high contrast version was detectedand in each case where it was detected it was the first one to be detected. Inthe case of the St. Menas flask, it is the only one detected 13 out of 15 times.These results show us that the high contrast version of the image targets is thebest choice to implement in the application.

Target and model loading The goal is to make this application as easy touse and configure as possible. One way this is achieved is through the way onecan configure it for a new collection. To do that only a few things need to bedone:

• Put the models, textures, and images from the books in their respectivefolders.

• Run Process.py to preprocess the data.

• Upload the processed target images to Vuforia.

• Download the Vuforia database to Unity.

25

Inside the Unity editor, no changes have to be made apart from choosing theright image target database. All the image targets are automatically loadedat runtime using the VuforiaHandler script found in appendix A.6. When animage target is detected the corresponding model is loaded into the scene andthe right texture and scripts are added. This is done by adding the followingcode to the default trackable script provided by Vuforia. The entire script canbe examined in appendix A.7.

// Get the object name

string modelName = this.name;

// Instantiate the model using the name of the trackable

g = GameObject.Instantiate( Resources.Load("MuseumModels/" + modelName) ) as GameObject;

// Set as child of the trackable

g.transform.parent = this.transform;

// Load in the textures and add them to the object

Texture tex = (Texture)Resources.Load("MuseumTextures/" + modelName + "-diffuse");

Texture norm = (Texture)Resources.Load("MuseumTextures/" + modelName + "-normal");

g.GetComponentInChildren<Renderer>().material.shaderKeywords =

new string[1]{"_NORMALMAP"};

g.GetComponentInChildren<Renderer>().material.SetTexture("_MainTex", tex);

g.GetComponentInChildren<Renderer>().material.SetTexture("_BumpMap", norm);

// Set the right scale and position

g.transform.localScale = new Vector3(.05f,.05f,.05f);

g.transform.localPosition = new Vector3(0,.05f,0);

// Add the zoom and rotation scripts

g.AddComponent<PinchZoom>();

g.AddComponent<TouchRotate>();

Figure 17: Code added to the DefaultTrackableEventHandler.cs script

This code only works when the image target, model, and textures have thesame filename (the textures have an added ”-diffuse” or ”-normal” depending

26

on the type of texture). To make sure all these files have the same name theProcess.py (appendix A.5) script was created. This script warns the user ifthere are images, models or textures with missing counterparts. An example ofsuch a warning is shown in figure 18. It also changes the contrast of each bookimage to a higher contrast that has better tracking performance.

E:\ Dropbox\Thesis > python .\ Process.py

All objects should have a model , a book image ,

a normal texture and a diffuse texture.

The following objects are not complete and need to be fixed:

apm -reused -capital is missing: a diffuse texture and a book image and a model

lvr -helmet is missing: a book image and a model

apm -horse -and -rider is missing: a normal texture and a book image

E:\ Dropbox\Thesis >

Figure 18: Warning about incomplete objects after running Process.py

27

In this chapter the question to be answered was: Is augmented reality with 3Dmodels effective to augment the narratives in a traditional museum catalog? Byaddressing the concerns brought up the people who tested version 1 we tried tocreate an application that can enhance the traditional museum catalog. It wasfound that we can use the already existing images in the book as targets foraugmentation while giving a performance stable enough for viewing the objects.We also made the process of configuring this application for other collectionseasy to perform. The users testing this application responded enthusiasticallyand liked using the application. With this information, it can be said that usingaugmented reality to enhance the traditional museum catalog is effective.

28

6 Discussion and Conclusions

In this thesis the main question was:

How can we bridge the gap between object-based and narrative-basedstorytelling?

This question was divided into three sub-questions which we’ve tried to answerin the previous chapters.

What are current approaches to object-based and narrative-based story-telling? Our main findings are the following. First, the Crossroads exhibitionat the Allard Pierson museum shows a few ways of telling a story using bothmethods and also a mix of these methods. Second, there is a fair amount ofAugmented Reality being used in museums at the moment. However, most ofthese are specifically made for certain exhibitions. In the next chapters, we willintroduce an application which is easily configured for other collections.

How can we bring the narrative to the object, or bring the object to thenarrative? Two application demos were made for this purpose. The demoof the Augmented Reality application, which brings the object to the narrative,and the demo of the Virtual Museum application, which brings the narrative tothe object. After testing and evaluating both demos, the Augmented Realityapplication was chosen to be developed further.

Is augmented reality with 3D models effective to augment the nar-ratives in a traditional museum catalog? By addressing the concernsbrought up the people who tested version 1 of the application we tried to cre-ate an application that can enhance the traditional museum catalog. It wasfound that we can use the already existing images in the book as targets foraugmentation while giving a performance stable enough for viewing the objects.We also made the process of configuring this application for other collectionseasy to perform. The users testing this application responded enthusiasticallyand liked using the application. With this information, it can be said that usingaugmented reality to enhance the traditional museum catalog is effective.

The application created in this project bridges the gap between object-basedand narrative-based storytelling by integrating the objects into already existingnarrative sources in an easily configurable way. The application was tested bya small group of users who reacted enthusiastically to the demo. This way, wehave proposed one solution to mixing these two storytelling methods. There areundoubtedly a myriad of ways this can be done, but this method is a step inthe right direction, and it will be developed further.

29

6.1 Future Work

During this project, a lot of people were enthusiastic about the idea of theAugmented Reality application, and at the end of the project, I was asked tofurther develop the application in association with the 4D Research Lab at theUniversity of Amsterdam.

The following are some things that can be done to get this application to aready-to-release state:

Testing Testing during this project was done on a small scale, about 5-10users for each version. Before this application can be released, it needs to betested on a bigger scale, with more users and more 3D models.

Sketchfab API Should a more substantial amount of 3D models cause anyproblems during testing, the Sketchfab API could be used to download modelsdynamically and decrease the size of the application considerably. This integra-tion will also allow for the downloading of annotations, which at this momenthave to be positioned on the object manually. This integration with Sketchfabcan only work if it is possible to circumvent the log-in required by the API,which is going to be discussed with Sketchfab.

Other targets There is a multitude of targets that can be used with theVuforia toolkit. The application could be extended to not only work on acatalog but could, for example, be used on an actual museum object to showwhat it would have looked like in its original state. These options are to beconsidered in further development.

30

References

[1] David Abulafia et al. Crossroads: Travelling through the middle ages. Al-

lard Pierson Museum Amsterdam and partners in CEMEC project, 2017.

[2] Cemec admin. Audiovisual contents in the Crossroads exhibition. 2017.

url: https://cemec-eu.net/cms/?p=324.

[3] Library of the University of Amsterdam. Inventory Database. 2018. url:

https://www.uvaerfgoed.nl/beeldbank/en/allardpiersonmuseum.

[4] Jennifer Billock. Five Augmented Reality Experiences That Bring Museum

Exhibits to Life. 2017. url: https://www.smithsonianmag.com/travel/

expanding-exhibits-augmented-reality-180963810/.

[5] CEMEC-EU. Connecting Early Medieval European Collections. 2016. url:

https://www.cemec-eu.net/.

[6] Benjamin Ives Gilman. “Museum fatigue”. In: The Scientific Monthly 2.1

(1916), pp. 62–74.

[7] Google. Google Cardboard - Google VR. 2018. url: https://vr.google.

com/cardboard/.

[8] Allard Pierson Museum. Beeldbank Stories: Near East. 2018. url: https:

/ / www . uvaerfgoed . nl / beeldbank / en / story / allard - pierson -

museum/near-east.

[9] Inge-Kalle den Oudsten. The Cross Culture Timeline. 2016. url: https:

//cemec-eu.net/cms/?p=227.

[10] PTC. Vuforia Engine. 2018. url: https://www.vuforia.com/engine.

html.

[11] SketchFab. Cemec Collection. 2018. url: https : / / sketchfab . com /

moobels/collections/cemec.

[12] The GIMP team. GIMP - GNU Image Manipulation. 2018. url: https:

//www.gimp.org/.

[13] Unity Technologies. Unity - Scripting API: Touch. 2018. url: https:

//docs.unity3d.com/ScriptReference/Touch.html.

[14] Unity Technologies. Unity 3D. 2005. url: https://unity3d.com/.

31

Appendices

A Code

Every script mentioned in this thesis can be found on this GitHub page:

https://github.com/AxelBremer/BachelorProject

To actually create an application one would need to install the Unity Editor andthe Vuforia toolkit. Because of the big number of small files unity creates, up-loading the entire project folder to GitHub is very inconvenient. The Readme fileon github gives a more detailed description of setting up the application.

A.1 Narrative XML file

https://github.com/AxelBremer/BachelorProject/blob/master/XML/NearEast.

xml

An example of a Narrative XML file based on the Near East story from [8]

A.2 AnnotationScript.cs

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

AnnotationScript.cs

This script checks which of the annotations currently on the screen is closestand turns on its halo. It turns off the other halos.

A.3 PinchZoom.cs

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

PinchZoom.cs

This script enables the user to use pinch gestures to zoom in and out of themodel.

A.4 TouchRotate.cs

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

TouchRotate.cs

This script enables the user to rotate the object using touch.

32

A.5 Process.py

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

Process.py

This script processes the models, textures, and images for use in the application.It gives the user a warning if any objects are incomplete.

A.6 VuforiaHandler.cs

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

VuforiaHandler.cs

This script loads in the imagetargets for every object in the Vuforia database.

A.7 AromaTrackableEventHandler.cs

https://github.com/AxelBremer/BachelorProject/blob/master/scripts/

AromaTrackableEventHandler.cs

This is an edited version of the default trackable event handler provided by Vu-foria. When a trackable is detected the matching model and textures are loadedin and the right scripts are added to the object.

33

B Figures

B.1 Contrast image versions

Normal Flask

High Contrast Flask

Full Contrast Flask

34

Normal Burner

High Contrast Burner

Full Contrast Burner

35

Normal Lamp

High Contrast Lamp

Full Contrast Lamp

36