Masters Thesis January 2012 Staffordshire University European Interactive Multimedia

The Integration of Augmented Reality in an Art Gallery

Case Study at the Hamilton Gallery

Thomas Lavelle

Table of Contents
Abstract.....................................................................................................................................................1 Acknowledgements....................................................................................................................................4 Introduction................................................................................................................................................5 1 About The Hamilton Gallery .................................................................................................................6 1.1 Motivation.............................................................................................................................6 1.2 Goals and Objectives.............................................................................................................7 1.3 Research Questions...............................................................................................................8 1.4 Methodology.........................................................................................................................8 1.5.Research Contribution ..........................................................................................................9 1.6 Terms, Abbreviations and Acronyms...................................................................................10 1.7 Dissertation Outline.............................................................................................................11 2 Background ....................................................................................................................................12 2.2 The AR-ization of Art Galleries.......................................................................................12 2.4 The Potential Down Side of AR inside Art Galleries........................................................14 2.5 Overview of AR Technologies used for AR enabled galleries..........................................14 3. What is Augmented Reality? ...............................................................................................................15 3.1 The Basics..........................................................................................................................15 3.2 Related Technologies..........................................................................................................15 3.3 Augmentation.....................................................................................................................17 3.4 POI displays.......................................................................................................................17 3.5 Image (Natural Feature) Tracking Displays.......................................................................18 3.6 Marker-based Tracking ..................................................................................................19 3.7 Bar Codes ..........................................................................................................................19 4 Augmented Reality Technology for Art Galleries: A Literature Review.............................................20 4.1 Keyword Choices ...............................................................................................................20 4.2 Search Plan..........................................................................................................................21 4.3 Top Conferences and Influential Journals..........................................................................23 4.4 Latest Research....................................................................................................................25 4.5 Current Best Examples of AR use in Art Galleries............................................................25 4.5.1 Samek Art Gallery at Bucknell University ...............................................................25 4.5.2 Kudan Art Gallery Experience .................................................................................26 4.8.3 Green Box Project .....................................................................................................27 4.5.4 Lab Art, a Los Angeles gallery ...............................................................................27 4.5.5 Dark Arts in Burlington, VT. .....................................................................................28 4.6 Cutting Edge and Future Projects........................................................................................28 5 The Hamilton Gallery AR Project: A Case Study.............................................................................32 5.1 The Gallery Environment ...................................................................................................33 5.2 Options and Issues With Popular AR Development Platforms.........................................33 5.2.1 Layar..........................................................................................................................34

5.2.2 Wikitude......................................................................................................................34 5.2.3 Aurasma.....................................................................................................................35 5.2.4 Total Immersion.........................................................................................................36 5.2.5 Junaio..........................................................................................................................36 5.3 AR Development Process...................................................................................................36 5.4 Learning the technology.....................................................................................................37 5.4.1 Getting Support ...........................................................................................................38 5.5 Phase One : Creating a Point of Interest ............................................................................39 5.5.1 Images Assets .........................................................................................................39 5.5.2 Audio Video and Text Assets.................................................................................40 5. 6 Discussion...........................................................................................................................40 5.7 T esting .............................................................................................................................42 5.7.1 Testing POI outside pop up in Live View and from a distannce..............................42 5.7.2 Route.........................................................................................................................43 5.7.3 Email ........................................................................................................................44 5.7.3 Call ...........................................................................................................................44 5.7.4 Video Intro .............................................................................................................44 5.7.5 The website...............................................................................................................45 5.8 Phase Two: AR in Print Advertising Brochure ....................................................................45 5.8.1 Brochure (video augment)......................................................................................45 5.9 Technical Specifications.......................................................................................................49 5.9.1 Tracking File ....................................................................................................................50 5.10 Phase Three: Inside the Gallery.........................................................................................52 5.11 Overlaying Extra Information about Artwork ...................................................................53 5.12 Another Look at the Research Questions ........................................................................53 6 Gallery user evaluations of Augmented Reality ...............................................................................54 6.1 Experiment Design ..................................................................................................................54 6.2 Testers ................................................................................................................................55 6.3 Evaluation Methods ...........................................................................................................55 6.3.1 Observation (Objective)...........................................................................................56 6.3.2 Observation (Subjective).........................................................................................56 6.3.3 Survey......................................................................................................................57 6.4 User Testing Preparation....................................................................................................57 6.5 Pre-Tests...........................................................................................................................58 6.6 Environment Preparation.................................................................................................59 6.7 Analysis of Data.............................................................................................................60 7 Conclusion.............................................................................................................................................63 7.1 Discussion and Summary..................................................................................................63 7.2 Future Research and Experiments.................................................................................................65 References and Citations..........................................................................................................................70 Appendex A: Forms for User Testing......................................................................................................71 1. Points of Interest ...........................................................................................................................76 2. Augmented Brochure ..................................................................................................................77 3. Artwork inside the gallery launches artist's portfolio.....................................................................77

Abstract

This is the follow up and continuation of the design and development project, Hamilton Gallery: Augmented Reality Applications for the iPhone/Android The goal of this project was to take advantage of Augmented Reality technology for the iPhone and Android-based smartphones and tablets that support the promotional aims of the Hamilton Gallery, an artist collective in Baltimore Maryland, USA. The aims include using the smartphone or tablet to enhance the user enjoyment and awareness of the many unique facets at the gallery at every stage of the user experience: from quickly locating , contacting and getting information about

travelling to the gallery without having to search the internet obtaining enhanced gallery information in the form of multimedia

that is triggered from print images and overlaid on the print material of various kinds once inside the gallery, being able to navigate to points of interest

inside the gallery from the geolocation- based overlays that appear on phone screen using the camera/tablet to gain instant access to enhanced,

overlaid information about artists and their work in the gallery as the user browses the artwork. The capability to superimpose additional information on a physical object is one of the most exciting aspects of AR and the obvious

creative potential for using Augmented Reality in a Gallery or Museum setting is the impetus for this project .AR has been around for many years but it has only been in the recent two years that Augmented Reality experiences did not need to depend on special bulky and expensive equipment. Now we only need a smartphone or tablet to enjoy some very interesting enhanced experiences for visitors.

This is a culmination of the Design and Development plan created previously which laid the groundwork for the results described in this project. The success of this project is evidenced by the fact that the technology works and works well. Confronted with many technical hurdles on the way to completion, I was able to find a development platform , Junaio, that allowed me to reach each goal that was set out in the Design and Development Plan.

User testing and experience and reaction to the AR technology in the Hamilton Gallery was generally positive with only a few negative comments. Augmented Reality technology changes rapidly and new ideas and ways to implement it are appearing each week. Given just a little longer, even the technology mishaps that occurred will be able to be overcome.

Keywords Hamilton Gallery, augmented art gallery, augmented reality, Junaio, augmented artwork

Acknowledgements
This project would not have been possible without support and trailblazing efforts of many people. I was introduced to the idea of Augmented Reality in 2009 and have been fascinated with it's possibilities ever since.Only two years ago, Augmented Reality and the programming knowledge needed to implement it ,was still very difficult, if not impossible for average designers to grasp. Furthermore, it was awkward to use computer webcams and hold up ugly tracking codes that sometimes registered properly and sometimes not. However, the people at the German company Junaio (Metaio) have been working very hard to create a developer experience that makes it possible, with just minimal coding background knowledge , to create new and exciting Augmented Reality projects that are useful in advertising and marketing. They have made it possible to do away with the ugly black and white patterns and instead use images as triggers to many exciting overlays, called channels .So thanks first must go to Junaio team for not only providing the smooth path to creating Augmented Reality but also supporting us and quickly answering, to the best of their ability, all the questions that come up during development. Next, I would like to sincerely thank my supervisor on this project, Professor Colin Chambers, whose invaluable advice and constant support as well as his top notch project management skills helped push me along and resulted in a project of quality higher than my own abilities alone could produce. Any failings or gaps in the project results are simply the failure of not following his advice more closely. I also need to thank my work colleague at ITT Technical Institute, Suresh Nariyanisami, for his help to sort out the 3D issues that came up while working on this. He was able to help me see more clearly how 3D models are used with Junaio

and how textures are baked onto 3D objects. He helped me understand various 3D formats. He also was the videographer who helped me create the videos for demonstrating the work from the viewpoint of a user with the phone. Finally I would like to thank the forward -thinking and wonderful people who run the Hamilton Gallery, Ron, Cara Sue and Theresa, for their support in giving me anything I needed and allowing me to have the run of the place and experiment in any way I wanted. Their comments and encouragement provided the necessary fuel to keep the project going when it looked like it was running out of gas. I would also like to thank artist Karl Franz for his generous access to sketches and other items that allowed me to show his artistic processes using Augmented Reality.

Introduction
This dissertation is the second part of an ongoing project that revolves around the use of Augmented Reality as a tool to enhance users experience of connecting with an art gallery from anywhere and viewing art at the Hamilton Art Gallery in Baltimore, MD, USA. In this dissertation, I focus on the advances that have been made within the last three years in Augmented Reality(AR) with the primary goal of applying this technology to an indoor art gallery setting. Moreover, this work demonstrates how a relatively user-friendly, free AR development platform can be used to help user get access to a wide range of multimedia- enhanced content that connects with the gallery in general and to specific pieces of artwork in the gallery. In order to specifically detail what this can be, in the second part of this dissertation,I have implemented an AR case study of the Hamilton Gallery featuring the works of local Baltimore artists. During the period of preparing this dissertation and case study, my knowledge of the technology has been improving quickly and I have had the pleasure of running a workshop called, The Augmented Reality- enhanced Art Gallery to the gallery owners and patrons to demonstrate some of the results I have been achieving. In terms of the development of this project,this has helped generate much useful feedback from the Hamilton Gallery owners as well as some of the artists themselves. Moreover, as an instructor of Visual Communications at ITT Technical Institute, I have had the pleasure of demonstrating my results of this emerging technology to faculty and students.

largely on the premise that visitors will come to them based on traditional modes of promotion ; marketing and advertising in journals, magazines or write-ups in newspapers. However, as AR technology becomes easier to implement and more more ubiquitous, this technology affords a great opportunity for opening up new user engagement.

1 About The Hamilton Gallery

The Hamilton Gallery is small non-profit artists collective that features local Baltimore artists. The gallery is membership -operated with monthly shows featuring the works of members as well as a featured monthly guest artist. Membership is open to all interested artists who want to showcase and sell their work as well as people who want to be involved in the growth and development of the gallery. 1.1 Motivation Art Galleries have used digital technology for several years , allowing users to listen to introductions and tours on hand held devices that they carried around with them. Within the last 4 years or so,the new emerging technology of Augmented Reality has progressed to the point that it has enabled art galleries to push the visual boundaries of what can be appreciated and learned from a piece of artwork. Artists can now overlay multimedia presentations on top of their artwork with the goal of either giving more in-depth explanations to a piece of work or actually overlaying new visual qualities to the artwork itself. As a promotional device, AR can help galleries with marketing by creating AR-responsive print material that the AR enabled smartphone or tablet can use to see multimedia content on a business card or any other print medium. The gallery can also be placed as geo-located point of interest (POI) which can be found easily by those who have the Junaio Browser (or other popular AR browser) on their phone or tablet. The motivation behind this project was not to push

the technology as far as possible in an art gallery setting, but to find out if AR could be integrated successful into an art gallery and ,more importantly, whether AR technology actually added any value to the user experience with the gallery. Based on what I have researched in other gallery situations, I believed that AR could increase the enjoyment of the gallery experience for visitors and I wanted to test this belief with my own case study. 1.2 Goals and Objectives

Figure 1-1. My area of concentration in red among the range of uses for digital technology in an art gallery setting .

The objective of this thesis is show how an Augmented Reality technology can be integrated into the Hamilton Gallery's hierarchy of existing framework of digital technology which is commonly available in many art galleries. It was very important to only use AR technology that could be implemented with a minimum amount of effort from the user,without special or bulky equipment . Also the development of an AR

application must be relatively easy to implement on the part of the developer. The hardware for this prototype is the smartphone, either the iPhone or iPad 2 or an Android-based smartphone or tablet The development platform was Junaio, a branch of Metaio. The hierarchy in Figure 1-1. above was derived from on the Hawkeye [1] classification used in museums. My area of interest can be seen by following the red arrows in Figure 1-1. 1.3 Research Questions 1.What is the current state of the art in Augmented Reality systems for an art gallery? 2. How can an Augmented Reality application for the Hamilton Gallery exhibit be designed and developed using open source development platforms? 3.Would the gallery visitors be receptive to interacting with Augmented Reality using smartphones or tablets? The third question came about based on the implementation of AR in case study of the Hamilton Gallery and is detailed in Section 6. 1.4 Methodology The first step was to define and frame the Augmented Reality challenge in a clear context and then decide which aspects of AR would be the most important to focus on for the Gallery. Much of this was fleshed out in discussions with the Gallery owners and artists. These initial steps were accomplished in this dissertation by a combined methodology of a Literature Review of Augmented Reality as it pertains to Art Galleries and a case study, which is the Hamilton Gallery and AR implementation that follows. The artifacts that came from this project are evaluated in terms of answering the final research question,Would gallery visitors be receptive to interacting with the Gallery via smartphones or tablets? Here I am experimenting with a particular case and the results are not meant to be generalizable. Saying that the results are not generalizable is not an

admission of a weak result but as Shramm and Roberts argue,[2], The essence of a case study, the central tendency among all types of case studies, is that it tries to illuminate a decision or set of decisions: why they were taken, how they were implemented, and with what result" However Yin disagrees [3]slightly and argues , that ...case studies, like experiments, are generalizable to theoretical propositions and not populations and universes. In this sense , the case study,like the experiment, does not represent a sample and in doing so, your goal will be to expand and generalize theories(analytic generalization) and not to enumerate frequencies(statistical generalization).

At any rate, the information gained from implementing AR in an art gallery setting and subsequent gathering of information from actual users of the technology has provided useful information about the value to a gallery and their users of integrating AR technology. 1.5 Research Contribution In this dissertation,I have made an effort to systematically review the area of Augmented Reality, including its current trends and its future prospects. I have presented the research that has been done in using AR in art galleries (section 4.8) and based on that work; I have implemented a case study (section 5) and presented the visual results and the users' experience with the available technologies both hardware and a popular

development platform. In my project I have considered an actual current art showing at Hamilton Gallery, the restrictions that were encountered using the AR technology and the steps I took taken to overcome them. As a part of my work I have developed a working prototype of an AR application to clearly show the choices made within Hamilton Gallery Case study. By necessity, I have made videos of the results so that the various ways that AR have been implemented could be seen from the user's point of view. I have also offered user reaction and evaluation of our AR technology in section 6.7 1.6 Terms, Abbreviations and Acronyms

AR

Augmented Reality is the real time overlaying of digital content such as audio video, 3D, and images onto real objects that users can access extra information or interact with in some way. Point of Interest. This is when the location of a place appears on the phone or tablet screen based on the person's geolocation point. It is triggered by the device's compass and contains information such as how far away a place is, the phone number, driving directions and link to a website. Quick Response Code. These are usually black and white markers with pixel like squares on them that react to the smartphone camera to lead the user to a website or other information. Live view is used in the Junaio AR development system and refers to what types of content appear on the devices's screen when the camera is activated. AR content may appear but the surrounding real objects are still in view on the screen. This is used in the Junaio AR development system and refers to being able to access the AR content without the use of the devices camera. It more like navigation links on a website and takes the user to specific places. Image tracking refers to a digital object being glued or tracked on to a real image. When the device's phone is pointed at a tracking image, multimedia content can appear on top of the object. This the coding language that was used to develop the various applications that make the AR work in this project.

POI

QR Codes

Live View

List View

Image Tracking

php
Extensible Markup Language was what the output of the php code was and refers to the results of what the user of AR sees on their device. There are other languages that can achieve the same results with Junaio. This is the language that I am familiar with

xml

10

Natural feature tracking

Relies on the recognition of a whole object or distinct features in an object or image in the unmodified environment Since there could be unlimited possibilities of features in the natural environment, these techniques are usually more computationally extensive and the tracking algorithms require more extensive training.[4] Longitude Latitude Altitude Markers. This refers to black and white square markers which contain exact information about the location of point of interest. These are especially useful inside buildings where it is often difficult to get the geolocation features on smart devices to register accurately

LLA markers

1.7 Dissertation Outline This section gives an overview of this dissertation work by showing briefly the contents. Chapter 1 (Introduction) established the focus of this dissertation work by discussing the context of this AR research work, its goals and the way the work has been implemented as well as its contributions. Chapter 2 (AR in an Art Gallery Environment) presents the short review of the digitization of art galleries along with the review of AR technologies that are being experimented with. Chapter 3 (Augmented Reality: What is it?) tries to define and clarify what Augmented Reality is and discusses the associated technologies in AR. Chapter 4 (AR Literature Review) sheds light on the steps taken to carry on the systematic literature review. Next, it presents the important researchers in AR today as well as the current and future trends. This chapter also reviews the leading projects that have used AR in Art galleries around the world. Chapter 5 (The Hamilton Gallery: A Case Study) discusses the workflow taken in producing the prototype AR application, the real-world constraints, the development environment considerations and the device considerations. Chapter 6 (Evaluation of Prototypes) presents the particulars of user testing

11

conducted to evaluate the interaction with the Hamilton Gallery AR application. It also illustrates the analysis of the results. Chapter 7 (Conclusion) finishes the dissertation with discussion and ideas for future work.

Background
2.2 The AR-ization of Art Galleries. Using AR technology, the first benefit to any art gallery would be the increased ability to add customer connection. This is easily done with Points of Interest (POI) overlaid on the device screen when called up by the user and can give convenient information such as gallery location, website, phone number and other details. These points of interest can be made to appear automatically on the device whenever someone is in the vicinity of the art gallery. Next, because of the ability of AR to use either Quick Response markers (QR) or regular images, advertisements can be places in magazines, books, brochures, posters etc.. and can overlay multimedia presentations that are glued to the image or code. Other uses are to lead users to a website or full screen video presentation. This is a new and exciting way to promote an art gallery via traditional print advertising. Finally, inside the gallery, each piece of artwork can be made AR ready via QR codes or ,nowadays image recognition(natural feature tracking). By pointing the devices camera at the the artwork or QR code marker , any combination of images, videos, text , 3D or 3D animation can be superimposed on an object. In addition, these overlaid digital objects can be made interactive and can lead to a website or more multimedia presentations.

12

Figure 2-1

Animated AR prototype presented at the Abstraction-Now Exhibition

In the near future,Augmented Reality can actually BE the artwork as in this example from the Abstract-Now Exhibition [5] . The claim of the world's first Augmented Reality Art gallery has launched in London recently [6]

Figure 2-2

A gallery dedicated to Augmented Reality Art in London

2.3 The Potential Down Side of AR inside Art Galleries Schmalstieg and Wagner (2008) [7] discuss how camera quality imaging capabilities of camera sensors typically deployed in smartphones are poor under bad lighting conditions. Images are blurry and colors start to suffer from signicant aberration. Low-level access to the camera sensor hardware is usually prohibited. APIs only provide a high level of access to the camera sensor, rendering control of exposure, aperture, or focal length impossible. Small CCD sensors are the cause of increased amounts of noise in the camera feed, hurting the performance of subsequent CV

13

algorithms signicantly. Quality lost during image acquisition can hardly be compensated by further processing steps.

2.4 Overview of AR Technologies used for AR enabled galleries This AR technology affords a great opportunity for opening up new modes of engagement.Galleries have a great potential before them, for example to bring represented artists or their collections closer to the mind of the public if they initiate fresh ways of exposing them. This could be done through:

Collaborating with the artist to produce and publish a short video as a studio insight

tour, or an artist interview that can be launched from the artwork itself or from a picture in a magazine of brochure.

Organizing a virtual tour of a current or past exhibition to help with providing the art

work an exhibition context, or some of the footage captured from its opening night and available on the smartphone or tablet via AR without the need to navigate to a website.

Conducting and producing an interview with a gallery director or one of the gallerys

collectors. Here is an educational tool that could broaden our understanding of industry practices why people collect, what it means to represent and support creative initiatives in the new century.. AR technology allows release from the linear work flow of a video or website because it does not have to be linearly product it is instead fluctuating constantly between appearing and not appearing at the user's request. Augmented Reality has allowed us to break the linear art viewing experience. A user can call up new fresh information and immediately release that extra information. Basically all that is needed is a smartphone or tablet device. With these common devices, users can allow content to appear on demand and for free.

14

3. What is Augmented Reality?

3.1 The Basics The basic idea of augmented reality is to superimpose graphics, audio and other sensory enhancements over a real-world environment in real time. In an art gallery, there are real objects, such as framed paintings, photographic art and sculptures. Using AR technology and a smart device digital material such as a video or 3D animation can be made to render and appear on top of a real object and even seem to come from out of that object. To summarize, an art gallery user has access to real and virtual objects at the same time. 3.2 Related Technologies Scene Generator The scene generator is the device or software responsible for rendering the scene or object . Rendering is not currently one of the major problems. In the Hamilton Gallery case study, the software was the Junaio browser which connects Junaio back end which in turn connects to my Internet Hosting Service in AR, because only a few virtual objects need to be drawn, and they often do not necessarily have to be realistically rendered in order to serve the purposes of the application [4].

Tracking System Tracking and Registration Apart from the display technology, the single most important technological challenge to general mobile AR is tracking and registration.[ 7] AR requires extremely accurate position and orientation tracking to align, or register, virtual information with the physical objects that are to be annotated. It is difficult to

15

convince people that computer-generated virtual objects actually live in the same physical space as the real world objects around us. In controlled environments of constrained size indoor computing laboratories, researchers have succeeded in creating environments in which a persons head and hands can be motion-tracked with sufficiently high spatial accuracy and resolution, low latency, and high update rates, to create fairly realistic interactive computer graphics environments that seemingly coexist with the physical environment. Doing the same in a general mobile setting is much more challenging. In the general mobile case, one cannot expect to rely on any kind of tracking infrastructure in the environment. Tracking equipment needs to be light enough to wear, fairly resistant to shock and abuse, and functional across a wide spectrum of environmental conditions, including lighting, temperature, and weather. Under these circumstances, there does not currently exist a perfect nor is one expected to be found in the near future. Display The technology for AR is still in development and solutions depend on design decisions. Displays devices for AR are available as HMD (Head Mounted Display), but now other solutions can be found. When combining the real and virtual world two basic choices are available: optical and video technology. There are some compromises that need to be made depending on factors like resolution, exibility, eld-of-view, registration strategies, among others [7]. Display technology continues to limit the development of AR systems.

16

There are still no see-through displays that have enough brightness, resolution, eld of view, and contrast to seamlessly blend a wide range of real and virtual imagery still do not exist to a high degree.[8] However companies like Junaio, Layar and Wikitude are working to bring AR display quality on small, lightweight, and low-cost devices like the smartphone as I discuss next.

3.3 Augmentation According to Azuma [ 9 ] in addition to adding objects to a real environment, Augmented Reality also has the ability to remove them. Recent work has focused on adding virtual objects to a real environment. However, graphic overlays can be also be used to remove or hide parts of the real world from a user. For example, to remove a desk in the real environment, draw a representation of the real walls and floors behind the desk and "paint" that over the real desk, effectively removing it from the user's sight. This has been done in feature films. Doing this interactively in an AR system will be much harder, but this removal may not need to be photorealistic to be effective. 3.4 POI displays Point of Interest displays are a place mark or button-type of objects depending on the development platform that appear on a smart device screen based on the location of the device user's position in the world. The developer creates the code within a development environment that allows a special browser such as Wikitude or Junaio to connect to the device.

17

Figure 3-1 Example of a POI display from the Junaio development platform

3.5 Image (Natural Feature) Tracking Displays Image Tracking is part of AR technology that allows an image to act as trigger for launching multimedia content that is either opened in full screen or glued onto that image on the device screen and move with it as the user moves the device.

Figure 3-2 Image Tracking example from Celebrity High Magazine . 3D Object overlaid on image courtesy of Junaio

3.6 Marker-based Tracking Marker-based tracking refers to the commonly seen black and white Quick Response (QR) codes. A smart device pointed at one of these causes a launch of a website or other content to appear on the device screen.

18

Figure 3-3 QR code example. From the smart device camera, this launches a website or other multimedia content.

3.7 Bar Codes

Figure 3-4

standard bar code

AR technology has now been developed around the ubiquitous bar code. Developers have begun to create sophisticated AR applications which consumers can compare prices of the same item at other stores or get more detailed information about the products by scanning a bar code from their smartphone.[9]. Microsoft tags like the QR markers and bar codes are quick ways to launch information about a product or service from the smartphone. But unlike the bar code, these tags can be updated quickly with new information.The tags also can provide valuable information about how many times the tag was used as well as set an expiration date for how long the tag remains live. In addition, whereas QR codes are all similar in appearance with black and white squares patterns, Microsoft tags to some degree are customizable and can be made slightly more attractive. [10].

19

Figure 3-5 Sample of a Microsoft's Augmented Reality tag

4 Augmented Reality Technology for Art Galleries: A Literature Review

In this section, I present a literature review of Augmented Reality as it pertains to Art Galleries. I have made a list keyword selection, search plan and article selection. The purpose of this review is to seek out and include the prominent innovators in this field , current state of the Augmented Reality and art as well as and the future potential of AR in the art gallery world. I have attempted to narrow the research to works that involved art galleries and present them according to unique and innovative elements in those studies. The overall objective is to profit from the previous research work so as to better frame the specific details of the Hamilton Gallery AR prototype problem and the best options to initiate the case study in chapter 5. 4.1 Keyword Choices
Table 4.1.Beginning the Search process with these key words

Overall Search Terms Augmented Reality and Art, Artists Augmented Reality in Art Galleries Digtal Uses in Art Galleries

20

At first I was attempting to tightly narrow the focus to pull up only items that had the words Augmented Reality as it would apply to Art Galleries but later decided to also include the full range of digital technology that art galleries use in order to better describe how AR integrates with other digital technologies. This yielded a few of the same specific articles about AR and Art. With the field of AR within an Art Gallery context being so new, I felt I needed to add more variety to the search query so I began adding the following terms
Table 4.2 Updated Search Terms

Secondary Search Terms Augmented Reality History Of Augmented Reality in Art Galleries Augmented Reality Exhibitions Interactive Art Galleries And Augmented Reality Augmented Reality Trends and Art Mixed Reality and Art Current Galleries that Use Augmented Reality Negative Reactions Augmented Reality and Galleries Augmented Reality in Art USA Europe Asia Augmented Reality and Art

4.2 Search Plan I began the search by using the keywords presented in Table 4.1 and finished the first phase of search. From those articles I first sorted those that I could not directly use but I was able to find very good prospects in referenced work inside by these articles. From the other articles of value, I have retained the articles that related to AR and Art and worked on highlighting ideas that related to art galleries of any type. Later, in a follow up plan, I used the keywords in Table 4.2 to check for any missed articles related to AR and art galleries. I used Harzins Perish and Publish [11] to count most cited authors and their g-index [12] and h-index [13] and to check their impact in writing about AR.

21

Figure 4.3 Overview of Search Strategy

22

Author Name Steve Ditlea R.Azuma R Silva J. C. Oliveira Rietmyr

Number of Research Articles 46 69 202 73

Citations Count 112 51181 618 1432

h-index g-index 10 96 10 20 5 155 20 37

Table 4.4 List of Leading AR Authors work searched between 2000 and 2011

4.3 Top Conferences and Influential Journals The conferences related to Augmented Reality began in 1998. Prior to that AR related work has been presented in conferences relevant to the enabling technology or applied technology discussed in that work. ISMAR is the leading conference on AR according to the work presented by prominent authors (section 4.3). The articles gathered in this research were presented in the following conferences: 1st International Workshop on Augmented Reality (IWAR'98), San Francisco, United States, November 1998. 2nd International Workshop on Augmented Reality (IWAR'99), San

Francisco, United States, October 1999. 1st International Symposium on Mixed Reality (ISMR'99), Yokohama,

Japan, March 1999. 2nd International Symposium on Mixed Reality (ISMR'01), Yokohama,

Japan, March 2001. 1st International Symposium on Augmented Reality (ISAR 2000), Munich,

23

Germany, October 2000. 2nd International Symposium on Augmented Reality (ISAR 2001), New York, New York,

United States, October 2001. 1st International Symposium on Mixed and Augmented Reality (ISMAR 2002),

Darmstadt, Germany, October 2002.[ 2nd International Symposium on Mixed and Augmented Reality (ISMAR 2003), Tokyo,

Japan, October 2003. 3rd International Symposium on Mixed and Augmented Reality (ISMAR 2004),

Arlington, Virginia, United States, November 2004. 4th International Symposium on Mixed and Augmented Reality (ISMAR 2005), Vienna,

Austria, October 2005. 5th International Symposium on Mixed and Augmented Reality (ISMAR 2006), Santa

Barbara, United States, October 2006.[ 6th International Symposium on Mixed and Augmented Reality (ISMAR 2007), Nara,

Japan, November 2007. 7th International Symposium on Mixed and Augmented Reality (ISMAR 2008),

Cambridge, United Kingdom, September 2008. 8th International Symposium on Mixed and Augmented Reality (ISMAR 2009), Orlando,

Florida, United States, October 2009. 9th International Symposium on Mixed and Augmented Reality (ISMAR 2010), Seoul,

Korea, October 2010. 10th International Symposium on Mixed and Augmented Reality (ISMAR 2011), Basel,

Switzerland October 2011. At the time of this writing, I wasn't able to find a single journal that specializes in Augmented Reality or Augmented Reality in Art . A journal that covers the Augmented Reality field among others is The International Journal of Virtual Reality. They cover a diverse range of

24

emerging technologies including Augmented Reality-Virtuality 4.4 Latest Research New research at MIT Media Lab [14] is making inroads into holographic projections. Michael Boves Object-Based Media Group presented a new system that can capture visual information using offthe-shelf electronics, send it over the Internet to a holographic display, and update the image at rates approaching those of feature films. If new research from the MIT Media Lab is any indication, holographic TVs could becoming soon. At the Society of Photo-Optical Instrumentation Engineers (SPIE) Practical Holography conference in San Francisco the weekend of Jan. 23, members of Michael Boves Object-Based Media Group presented a new system that can capture visual information using off-the-shelf electronics, send it over the Internet to a holographic display, and update the image at rates approaching those of feature films. 4.5 Current Best Examples of AR use in Art Galleries In this section I show five current examples of how Augmented Reality is being implemented in other art galleries around the world.

4.5.1 Samek Art Gallery at Bucknell University The MANIFEST.AR artworks [15] can only be seen in Venice or Lewisburg in the display of a smartphone using the Layar augmented reality browser. The artworks are displayed only with a smart device.

25

Figure 4.5 Kudan Art Gallery- viewing artwork can only be done with a smart phone or tablet

4.5.2 Kudan Art Gallery Experience Kudan Art Gallery[16] featured an exhibition that contained no art-until you held up the smart device to the wall-mounted artwork- then the artwork appeared.
Figure 4.6 The Green Box Project sponsored by Beck's Beer

26

4.8.3 Green Box Project [16] Sponsored by Beck's Beer, this is being presented as the 'the world's first global networked augmented reality gallery'. Taking place over three years, the Green Box Project [17] showcases 1,000 art projects by "individuals with unique creative vision." How this is initiated is through 1,000 "green boxes" placed all over London, New York, Los Angeles, Miami, Rome and Milan. When you see one you can view the gallery inside on your smart phone (using the Becks Augmented Reality App, now available on iTunes). There is one placed at the top of the Statue of Liberty, where you'll be able to view a 200ft digital sculpture by Belgian artist Arne Quinze 4.5.4 Lab Art, a Los Angeles gallery Presenting Augmented Reality in a street at exhibition. [18] .This is an example of one of the goals of the Hamilton Gallery AR Case study. That is, overlaying digital material onto real artwork.

Figure 4-7 Lab Arts Gallery in LA using a mobile phone to view an augmented reality version of a wall painting

27

4.5.5 Dark Arts in Burlington, VT. Dark Arts [19] featured a Halloween party where the guests mask would be augmented and placed on the face when the smart device was focused on the marker.

Figure 4-8 Dark Arts Gallery hold a Halloween party with Augmented Reality

4.6 Cutting Edge and Future Projects These sample screen grabs from Mistry TED Talk [20] demos the SixthSense research work in progress at MIT and the possible future of augmented reality. The application to artists' work is fairly obvious in the following examples. In the future, the need for a display element may be unnecessary . Any surface will be able to be the display.

28

Figure 4-10 No

need for a phone. Any surface will work!

Figure 4-11 Reading and displaying the price of an object

Figure 4-12

The time projected onto to your arm as needed

29

Figure 4-13 .A multimedia

show inside a printed newspaper.

30

Figure 4-11 The current weather overlaid on the map of the US

In terms of how this relates to Visual Arts and Art Galleries, it can be seen from the examples above that the display of artwork may no longer need to be just on walls. An artist's work may have the ability to be displayed anywhere there is a surface.Prof. Blair MacIntyre, [21]director of Augmented Environments Lab, Georgia Institute of Technology describes the transition between where we are now and where AR might be headed:

We're 80 per cent of the way there with AR but the last 20 is going to be hard.Not a lot is going to happen until we have head worn displays. Itll be more compelling when its there. The day I have a pair of funky sunglasses and walk around with non-obnoxious advertising, news, social networking, totally immersed in the world, then AR will really have arrived..But before that, there are so many huge questions to answer: can we get all of the technology we need onto a mobile device; can we make it usable and not invasive, not a dystopian existence; and can we use it to build a positive future?

31

The Hamilton Gallery AR Project: A Case Study

After careful study of the literature on Augmented Reality and its application to an Art Gallery environment next I will show how I have begun and implemented AR in the case study of the Hamilton Gallery. At the first meeting with the gallery owners I laid out the options as I understood them for how I could implement three types of AR solutions within gallery and away from the gallery. First I suggested implementing the Point of Interest markers as a way for people anywhere in the area could find automatically get basic information about the Gallery. These would include navigation that would

lead the user to: la route map telephone website video introduction to the gallery email

Figure 5-1 Interface that leads to getting information or contacting the gallery

32

5.1 The Gallery Environment The Hamilton Gallery is located in Baltimore City in a well populated residential area so it is convenient with nearby cellphone towers . This is an important consideration due to the use of smartphones in this case study. It is a small, one story building with two viewing rooms.

Figure 5-2 The view inside the Hamilton Gallery

5.2 Options and Issues With Popular AR Development Platforms In the past 2 years. several companies have come into existence that have enabled the process of developing AR applications more accessible to more people. All of the platforms described below use similar hardware to display AR technology: the smartphone's camera, GPS, compass, accelerometer and internet connection . There are differing opinions regarding the question as to whether the location-based approach of Augmented Reality pioneered by the companies below can be considered as true augmented reality . Mok [20] observes that the researchers and pioneers of AR would say that this is not Augmented Reality in the sense that AR is not simply overlaying virtual multimedia objects on a real object. The problem is that for that to be true, totally accurate registration must

33

take place and that is currently, technically not happening yet. However, for many purposes such as this art gallery case study, perfection in registration is not a necessity. 'Good enough' can suffice for now. Here I will review the most popular development platforms. 5.2.1 Layar Layar is a pioneering AR company in Amsterdam, founded in 2009 by Raimo van der Klein, Claire Boonstra and Maarten Lens-FitzGerald. They were one of the first to offer relatively easy AR development. For users,they provide a free mobile browser called Layar. The browser allows users to find various items, or points of interest (POI) based augmented reality technology. Data in the browser comes in the form of layers. Layers are web services that serve geo-located points of interest in the vicinity of the user. These layers are developed and maintained by anyone using their free API. Layar as a company is responsible for their validation in the publication process. I have experimented with Layar in 2010 but found the instructions and tutorials difficult to follow. In addition, the user needed an newer version of the iPhone to use it.

5.2.2 Wikitude Wikitude is similar to Layar mobile augmented reality software which was developed by the Austrian developer company Wikitude GmbH (formerly Mobilizy GmbH) and offers also a free downloadable browser. As with Layar It displays information about the users' surroundings in a camera view of the smart phone or tablet. Wikitude was the first publicly available application that used a location-based approach to Augmented Reality. In 2011 according to the website augmentedreality.Connected [20] .Wikitude came in first place as the Best Augmented Reality Development platform in 2011as voted by readers. Wikitude has also been awarded the best augmented reality browser for two years in succession [20].

34

5.2.3 Aurasma Appearing on the scene in 2011, Aurasma, a British AR development company ,added some new innovations to the AR technology, called Cylindrical mapping. The Aurasma app can recognize the cylindrical shape of a bottle and overlay digital multimedia content on it. A wonderful example of how this has been exploited can be seen here in this advertisement for Lucozade sports drink. Aurasma is unique in that it allows any image in the real world to trigger digital content.

Figure 5-3 Example of Aurasma Augmented Reality for Lucozade

[21]

I have experimented with this app but it is only available for iPhone version 4 and Android version 2 user. This was considered a drawback when deciding which development platform to use for the Hamilton Gallery case study. A quick review of the free downloadable Aurasma Lite from the Apple App store shows that many people report finding the app Aurasma Lite crashing, freezing or not working at all. This was my experience as well. There is now a 3D content option.

5.2.4 Total Immersion DFusion Mobile is an Augmented Reality development product from the Canadian

35

company Total Immersion for smartphones (iPhone and Android Phones) or tablets (iPad). They offer a 3D rendering engine and location based services are available to develop AR mobile applications. 5.2.5 Junaio Junaio, from the German company Metaio, and the development platform of choice for the Hamilton Gallery case study, has a robust Point of Interest (POI) AR capability as well as markerless tracking options with image recognition. Junaio was chosen for this case study project because of the numerous tutorials and resources offered by the company to help AR developers. They have clear development examples to try and one only has to modify the code they provide with original content to make the AR processes work. There is also very good developer support via a forum. They also have a Junaio Development Program to encourage and help developers hone and market their skills. 5.3 AR Development Process The development plan for the integration of AR into the Hamilton Gallery was divided into 3 phases,

g and

POI location and information, Print Advertisin Promotion Augmented within the

Artwork Gallery.

36

Figure 5-4 Three development phases for the AR implementation of this case study

5.4 Learning the technology Learning how to use the Junaio development platform was the first step in this process. It was challenging but became easier with practice. Junaio allows developers set up 'channels'. Channels are individual space that a developer creates to set up one AR project. After signing up as a developer, one can simply create a new channel. Channels begin as testing sandboxes until the developer has successfully validated and tested the channel on a smart device. The channel is then submitted for approval to Junaio. If approved, the channel is now set to active and can be seen on any smartphone or tablet. If there are issues, the user will receive feedback from Junaio and must correct the errors and resubmit the channel. I began to take the tutorials as in Figure 5-4 below that can lead a developer step-by-step from creating simple Point Of Interest pop up icons then gradually proceeded to more complex AR options.

37

Figure 5-5 Sample Tutorials for creating channels in Junaio from the Junaio website

They also offer a channel creator which create channels automatically and can launch 3D or videos over a tracking image. No coding is necessary. I decided to take time to go through each tutorial and be sure that I could make the demos work before beginning the Hamilton Gallery project. This took approximately one month.

5.4.1 Getting Support Junaio has a Google Groups forum which is set up for developers. Answers to questions are quick and help is available from both developers and Junaio staff. I encountered several php/xml code related problems but they were resolved quickly through this forum. 5.4.3 Overall Workflow for creating AR Channels in Junaio

38

Figure 5-6 Workflow of the Junaio channel creation process

5.5 Phase One : Creating a Point of Interest This goal of this phase is to create a way for user with an smart device and the Junaio browser can access information about the gallery. 5.5.1 Images Assets Images were needed when creating the POI for the gallery. All that was necessary was to upload the Hamilton Gallery thumbnail so I recreated the Hamilton Gallery logo in Photoshop.

5.5.2 Audio Video and Text Assets I created a video that shows one of the owners of the gallery introducing the gallery. One of the navigation buttons will open up a full screen video. For the videos used to augment images ,artwork and as part of the POI buttons, I used only my iPhone video capabilities. I immediately uploaded them to my youTube channel directly from the iPhone and downloaded the .flv file. I then needed to prepare the video for a smart device. This was done using a free piece of software called Handbreak. This allows me

39

to compress the video down to a size of 10 mb or less. 5.6 Discussion Using the processes above, I was able to successfully implement Step One of the AR project, creating a POI that appears on the smartphone or device when the use has the Junaio Browser installed on the phone. (See figure 5.8 below). This was the easiest, most trouble-free part of the development process. To see how a user would experience this on a iPhone, I created a video from the user's point of view and placed it on youTube.

40

Figure 5-7 The Basic Point of Interest Interface using the Junaio Browser

5.7 Testing Testing involved using an iPhone Version 3 and a Samsung smartphone with Android 2 operating system on a 3G network everywhere possible in the Baltimore area. I avoided connecting to any WiFi so that I could test it under the worst conditions. It mostly worked well. However deep inside buildings there were problems accessing the 3G network. Connection speed overall was similar to accessing any website.

41

5.7.1 Testing POI outside pop up in Live View and from a distance Live View in Junaio means using the the device's GPS, camera and accelerometer to display points interest in the nearby area on the screen while the surrounding areas can also be viewed by the camera. The opening screen in Live View looks like this:

Figure 5-8 Live view Icon

Clicking Live View takes the user to this screen:

Figure 5-9 The introduction and POI interface

42

5.7.2 Route The route also offers step by step driving directions
Figure 5-10 The route to the gallery generated is Google maps

5.7.3 Email An email button in the Live View interface allows a user to email the gallery

Figure 5-11

Accessing the email function.

43

5.7.4 Call

Figure 5-12 Making a call from the Live View interface

5.7.5 Video Intro The video button in Live View brings up a full screen video.The current version has an introduction to the Hamilton Gallery by one of the owners.

Figure 5-13 Watching a video

44

5.7.6 The website An additional button can be programmed to access the Hamilton Gallery website can also added for convenience.

5.8 Phase Two: AR in Print Advertising Brochure Phase Two involved creating a promotional brochure that could be placed anywhere. The goal was to use Junaio and the image recognition ability of the smart device camera to overlay video material on top of the image when the phone camera was pointed towards it and scanned . 5.8.1 Brochure (video augment) The first step was to create a brochure. I first visited the gallery and took pictures of the current artwork hanging on the wall and received permission from the artists to use their images in the brochure. Next I received some text copy from the Gallery owners that would be the text content of the brochure. I designed the brochure in Adobe InDesign as an 17X 11 document . Figure 5.14 and 5.15 shows the printer's comp of the design before it was folded. For users to be able to discern which images could launch and overlay digital content, I placed a Junaio icon next the images that accessed extra content. the sign that that particular object was ready to be scanned for extra content. This was

Figure 5-14 The graphic that allows users to know that this item contain augmented content

The following is the brochure that was created with Augmented Reality hot spots. These are the areas marked in red. As the camera scans this area of the brochure, videos

45

appear on top of the image that gives users more information related to that image.

Figure 5-15 Hamilton Gallery front cover and Augmented Reality enabled area marked in red

46

Figure 5-16 Hamilton Gallery inside cover and Augmented Reality enabled area marked in red

A second video was made of how the user might experience the overlay of virtual to real objects on a brochure. (See Appendix B-2)

Figure 5-17 The work flow for preparing videos for use on the mobile phone.

47

5.9 Technical Specifications The videos used in all the projects were made with the video function in the iPhone 3 . They were uploaded to youTube and then downloaded as an flv. From there they were brought into a program called Handbrake which allows videos to be compressed for use in mobile applications. Junaio requires that all videos that will be glued on to markers, such as the images in the brochure and videos must be converted to3g2 format. Therefore, another converter software was needed. I used a converter called WinFX video converter for this. As with all AR development in Junaio, channel needs to be created . Because the videos when triggered by the camera were to be glued onto a specific part of the brochure, I create a Glue channel. The main obstacle here was finding the correct size and registration point to get the videos as closely aligned to the original picture as possible. As can be seen from the video in appendix B2, the alignment is not 100% accurate. I attempted to adjust the parameters that control size and alignment in the php code. These parameters are show in figure 5.17
Figure 5-17 I simply adjusted these numbers until I came as close as I could to an exact overlay.

48

5.9.1.Tracking File.

Figure 5-18 How to begin a scan of an image in Junaio. Three images within the brochure (shown in red in Figure 5-18) are able to activate and overlay video textures. The user opens the Junaio Browser, selects the Scan option and

points the camera at the image. Using an image as a tracking object to trigger another
multimedia item to be placed on top requires the use of tracking file consisting of special movieplane in .md2_enc format as well as a video texture in .3g2 format. The movieplane was easily downloaded from the Junaio development area and there was no need to create this.This .md2_enc file was then referenced in the code and produces the virtual object to overlay the video, which attaches to the movieplane, which in turn attachs to the onto the original image.

Figure 5-19 Using a tracking file to lay a video onto an image

49

Figure 5-20 This is is the structure of an AR Tracking video in Junaio

5.10 Phase Three: Inside the Gallery The final task was to integrate AR inside the Hamilton Gallery. The goal was to be able have a user point the smart device camera at a piece of art hanging on the wall and launch multimedia content onto the device screen on top of the real object. For this task, I chose two artists who I had access to Theresa Reuter and illustrator John Thompson. For Theresa Reuter's art piece called Rosalina, I implemented a video AR overlay that shows Theresa's introduction to how the artwork came to be. The video was taken with a iPhone 3 and processed according to the workflow in section 5.5. John Thompson's illustration made use of a QR code to call up a channel that featured more of his work.

50

5.11 Overlaying Extra Information about Artwork The work flow for Theresa Reuter's art piece was exactly the same as the work done on the AR enable sections of the brochure. I only needed to make a new channel and change the code name and absolute path of the video. Pointing the smartphone camera at her work hanging on the wall and scanning the artwork triggers a video which overlays over the artwork on the camera screen. In this video Theresa introduces how this artwork was conceived and painted. A video shows the results as a user would experience them.( see link to video in Appendix B-3) John Thompson's work used a QR code marker to generate his channel. Once the camera recognized the QR code marker and found the channel, the Junaio browser output overlaid an introduction and a portfolio of seven other pieces of art. The user can click on any of the images to be taken to the website. This piece features code different from others, in that it arranged the artwork in a 360 degree circle. The user only needed to rotate the smart device in a circle to take a tour through the portfolio.(see link to video in Appendix B-4)

5.12 Another Look at the Research Questions In Section 4 I have addressed the first research question which produced a comprehensive Literature Review. The step by step description of the Hamilton Gallery case study conducted has been presented in Section 5 to address the second research question. The question that remained was whether or not gallery patrons would be attracted enough to the technology to enjoy it and want to take steps use it? To find out, I had to evaluate it in the real environment of the Hamilton Gallery. The design of the evaluation to address this third question is shown in the next section.

51

6 Gallery user evaluations of Augmented Reality

The case study involved ways for users to interact with the gallery. First, a POI helps smartphone users find the gallery and get basic information automatically when they are in the vicinity of the gallery. Next, a Hamilton Gallery promotional brochure contained images of artwork that , when scanned with the smartphone, produces extra content in the form of videos and overlaid it on those images. Finally, some artwork within the gallery had been augmented so that user could have access to digital content overlaid on the artwork. I wanted to find out how users react to these interactions and which of three were seen as having value to them. 6.1 Experiment Design Designing an evaluation for new technology such as Augmented Reality is difficult because there are no solid guidelines that have stood the test of time. For this evaluation I have used elements of Gabbard and Swan(2001) [23]who have surveyed user-based testing models. I was able to pull out several questions which would help me gauge user reaction of their experience with gallery. The factors that I considered for each type of interaction in this evaluation include: registration weaknesses lighting and how affected gallery the experience ease of keeping Live view objects in the camera view. response speed for overlays. enjoyment- was it fun to use? ease of learning- was the experience intuitive?

52

interest sustained after initial use? user priorities- was there anything that was missing or expected that would improve the experience? The discussion and results presented at the end of this section were based on these inquiries. 6.2 Testers I was able to find six Hamilton gallery members that were willing to help test the AR prototypes. Each member owned a smartphone.Three users had an iPhone 3 , one had an iPhone 4, one had an HTC Sensation 4G and one had a Motorola Droid 4G. They were given instruction on downloading the Junaio browser. Then I ask them to do a series of tasks for each of the 3 phases of the case study: Interacting with POI access points, interacting with the brochure and interacting with the artwork within the gallery. 6.3 Evaluation Methods Having set up the questions used the following evaluation types: Objective Observation, Subjective Observations and Survey. 6.3.1 Observation (Objective) Through objective observation I wanted to explore and document the following: How often does the user have trouble accessing the Junaio browser and

user interface? (count) How many times does the user have trouble getting the augmented content to show or overlay in Live View? (count) How does the quality of lighting either outside or inside lighting affect the augmented the display of the augmented content?

53

detection? (descriptive) Other miscellaneous observations

6.3 .2 Observation (Subjective) In the subjective observation section I asked user's to respond descriptively to specific AR tasks that I assigned. These tasks were assigned to users: Find out how to get directions to the gallery from Live View in Junaio. Find which images in this Hamilton Gallery brochure can launch multimedia content.(descriptive reaction) Find the best distance to hold the camera away from augmented image. (descriptive reaction) Find the artwork in the gallery that has augmented content and launch that content with your smartphone.(descriptive reaction) Other descriptive reactions 6.3.3 Survey Finally I wanted the users to fill out a survey that expresses their various experiences with all three types of interactions. Which of the three interaction types was most interesting to you? Why? (comment) Would you use these AR interaction types again in the future? Why? Or why not?(comment) Were the 3 interactions easy to figure out?(comment) Did the digital content respond quickly and well to the phone's camera?(scale of 1 to 5, 5 being best) Did the extra digital content add anything of value to your experience with the

54

Hamilton Gallery, such as enjoyment, educational value or convenience? Were there any of the three interaction types that were tiring or frustrating. For example, did holding the camera up to an image and viewing content overlays get tiring?(comment) Other comments Ideas and suggestions 6.4 User Testing Preparation I set up the experiment both inside the gallery and at short distance away from the gallery. Before beginning the user testing I gave each subject a five minute introduction about the testing situation and what exact the focus of my research work was. I prepared printed forms for the observations,. (See Appendix C for evaluation forms) Since there were six participants, I decided to run the user testing based on Baxter's research [25]in two groups to avoid any issues with participants' learning bias. I created 2 groups of three participants at random. In the first walkthrough, I decided to use Outside -the- Gallery tasks first and Inside -the- Gallery tasks later and in a second walkthrough with the second group of three, vice versa. In each walkthrough the users carried out the following three phases. Phase 1: Observe user interaction with objective observation of user without any intervention in their actions or results of the tasks Phase 2: Ask the user to perform tasks mentioned in section 6.3.2 and wrote down the users subjective reactions and observations. Phase 3: Invited testers to fill out a survey. In this sections I have introduces my user testing plan and how I designed the testing parameter before the test begins. Next I will describe the details of the user test.

55

6.5 Pre-Tests I thought it was important to carry out two pre-tests with two non tester's to get some initial ideas of what reactions I might expect during user testing. Some important feedback was offered about the observation methods and the design of the survey. I gave an brief introduction to the technology and how to download the Junaio Browser and access it to look for the Hamilton Gallery. During these pretests, the users had thought that Live View icon would automatically appear on the phone screen. They didn't realize that the phone needed to be adjusted in the air to find the sweet spot for the geolocation overlay to appear. As for actually answering questions for the subjective observations, it turned out to be difficult to answer due to different reactions from the different iPhone versions of each user. One said he would have answered differently if he would have had the same reaction as Pre Tester number 2 had. One of the main difficulties the pre-testers experienced was getting the AR content to register properly on the object on display screen of the phone. They were interrupted five times holding the phone this way and that way, trying to get the right angle for the content to display properly on the object and on the screen. One positive result was that the video content that was called from image recognition and from the main interface, loaded quickly , within five seconds ,for both pre-testers. This meant that the treatment I gave the videos to prepare them for a mobile environment was well done. 6.6 Environment Preparation In order to carry out the user testing, three areas were prepared. For the testing the Junaio initial POI user interface, we walked down the street one-half a

56

mile to a coffee shop. The idea was to access the POI of the gallery from a random distance from the gallery and successfully carry out the tasks mentioned in Section 5. Next, a Hamilton Gallery brochure was placed on the table for the users to try to access AR content on the images in the brochure. They were to locate the Junaio icon within the brochure, point the camera at those images to launch relevant content that overlays over the image. Finally the testers walked back to the gallery to experiment with AR within the gallery environment. Next to one of the pieces of the artwork was a QR code marker which allowed users to access more portfolio pieces from this artist in a 360 degree circle within Live View with some further text information as well as clickable image that takes the users to a website. In a second task within the gallery, users were to access a video from that introduced more artwork from this artist. 6.7 Analysis of Data Initially I was concerned that the small number of testers, six, was too small to draw any conclusions from the testing. However, according to usability expert Jakob Nielson,[27] elaborate user testing is unnecessary and is waste of time and money. He recommends no more than 5 testers and suggests running more small tests. The graph in Figure 6-1 shows that after 5 testers , the results quickly begin to show dimishing returns.

57

Figure 6-1 After 5 testers , a usability test shows dininishing returns

In this section I will go over the results of the observations and the survey covered in section 6.1.
Registration weaknesses and limitations Overall most testers had some difficulty getting the AR content to behave properly, meaning, to center the content properly on the display screen. There was a constant need to rotate the camera until the right position was found. For the video overlays on the brochure and inside the gallery, the content never quite overlaid exactly on the object but it did not interfere the users implementation of the AR content. Occasionally, the video content appeared and abruptly disappeared , depending on the position and angle that the user held the camera.

Response Testers generally were satisfied with the response of AR objects that were launched from the phone camera. Users results made clear that in general the iPhone performed slightly better and faster that Android-based smartphones, meaning that content seemed to come on faster and stay stable better on the iPhone. Lighting

58

Lighting had no effect on the ability of users to see the content clearly either outside of the gallery or inside the gallery. Enjoyment All testers enjoyed the novelty of the AR smartphone experience. They were less impressed with the POI test because it was so similar to a website experience. However, when the videos were overlaid over the images on the brochure, the users were very impressed. While they enjoyed the ability to get AR content from the artwork, there was a fatigue that sent in after awhile. Constantly holding

the phone up in the air became tiring after awhile. All six users gave a score of 5 (best) when asked if they enjoyed the experience. 5 out of 6 would try this again on a future visit to the gallery. All of the six subjects replied with Yes when asked, whether they would recommend these three kinds of AR experience to others. When question why , one of commented that it was cool seeing those multimedia objects lay on top of real objects. Another commented, It was kind of fresh and new.' A third comment was its different . One tester also commented that it was very convenient. Ease of learning Users reported that the user interface of the Junaio browser was quite intuitive. The only learning problem users had was trying to keep AR content centered in the display. There was a lot trial and error to get the phone positioned in space so that content would stay stable and easy to view.

Interest sustained Users did not report losing interest as the testing continued. Once they were

59

able to activate and overlay digital content based on image recognition, users continued to enjoy trying out the smartphone on different objects. There was an interest in what content would come up from an image or art piece. User priorities When asked whether they would like the Augmented Reality they experienced to be part of the gallery experience one out of six subjects responded with No. One of these users commented that, If the content can come in smoother on the phones I think it will be good. Another reported that it seems easy but I wish there was more things to do other than view videos, more chances to interact the AR content, another wrote, Yes, but with more information, yet another said, Young people will really like this. One of the subjects also commented that, it will help users connect more with the artists, When questioned about suggestions, one user from the 40 and above age group wrote, I think it was fun but it seems a little gimmicky , not sure if it will stick in the long run'. One of the users wrote, Definitely a nice way to get people more involved with the gallery and the artwork.

7 Conclusion
7.1 Discussion and Summary The goal of this masters dissertation was to explore the technology of Augmented Reality and apply it in an art gallery environment using freely available development platform. With my literature review in the first section of the study, I have discussed how AR is currently being used in art galleries . A summary of these art gallery applications showed that most of the Augmented Reality applied in an art gallery setting involve the smartphone or tablet and the overlaying of digital objects onto real artwork by way of markers, QR codes and image-based tracking.

60

Using these applications as a starting point, I wanted to be able to focus on the latest image recognition abilities of the Junaio development platform to give users the ability to point the camera at any image and instantly launch relevant new content and overlay it on the image or artwork and display that content on the display screen of the phone. This is major step forward for art galleries to be able use the image recognition power of Junaio to allow their patrons to increase their knowledge and enjoyment of the gallery in a convenient way. This work also showed how a gallery itself can use Augmented Reality technology to successfully increase outreach to the community through the use of Points of Interest . The most interesting aspects of the outreach, (the promotion and marketing aspects of the gallery,) were seen in the way that print material such as brochures, posters, business cards and images in print material can launch digital multimedia presentations over the print material for enhanced enjoyment, information and education. The testers in our user testing phase were mostly delighted by the their first experience using AR, as I presented in the discussion of the evaluation results (section 6.7). Most of the testers were interested in the instant overlaying of digital content on real objects more than they were interested in the Point of Interest Icons and navigation interface. The results of this user testing suggests that the image recognition abilities of AR browsers like Junaio show great potential to attract and delight users compared with simple QR codes, eventhough a little more physical effort is involved to keep the camera positioned for a minute or more and positioning it to find the right spot for best viewing. While the testers enjoyed this experience, felt it had value to offer an art gallery viewer and said they would like to use it again, some wondered if this was more of

61

a gimmick and whether or not it would hold attention after the excitement of the initial trial wore off. I learned from the testers that just, like a website, if the AR content is not continually kept fresh and interesting , users are not likely to get in the habit of using it. Users also expressed the desire for more variety of multimedia content. Many said that while the functional navigation videos, images and sounds were nice, they hoped their could be more variety, such as short films, documentaries and 3D animations. 7.2 Future Research and Experiments The case study I have demonstrated using the smartphone and Junaio AR browser can be enhanced further in a many different avenues of application. One of the main ideas could be to make this prototype more interactive by creating AR games, quizzes and treasure hunt scenarios within the gallery to increase and improve interaction. For more in depth AR content , research can extract better ways to optimize video and 3D for use on smartphones and tablets of AR models. Research into how such a processor intensive animation could be optimized for smartphone and tablet would be useful. Furthermore, more usability studies of this AR prototype could be carried out in the future to observe patterns in how users of different age groups relate to and interact with the artwork in the gallery as well as the POI's and AR-enhanced print material. This could help refine what AR features are best in an art gallery .

62

References and Citations

[1] R. Hawkey, 2004. "Learning with Digital Technologies in Museums, Science Centres and Galleries retreived from http://hal.archives-ouvertes.fr/docs/00/19/04/96/PDF/hawkey-r-2004-r9.pdf [2] W. Schramm and D.F. Roberts, The Process and Effects of Mass Communication, University of Illinois Press, 1971. [3] R. Yin (2009) Case study research: design and methods Sage

Publications
[4] J. Vallino Introduction to Augmented Reality April 1998 retrieved from http://www.se.rit.edu/~jrv/research/ar/introduction.html| [5] Abstract-Now Exhibition http://bodytag.org/augment2/ [6] The Worlds First Augmented Reality Gallery retrieved from http://www.3news.co.nz/Augmented-reality-art-exhibition-inLondon/tabid/312/articleID/233000/Default.aspx [7] Schmalstieg and Wagner (2008) Mobile Phones as a Platform for Augmented Reality retrieved from

http://www.icg.tugraz.at/publications/SCHMALSTIEG_SEARIS0 8.pdf/view [8] L Madden (2011) Professional Augmented Reality Browsers for Smartphones: Programming for Junaio, Layar and Wikitude. John Wiley and Sons [9] Mechanical Engineering Blog retrieved from http://www.mechanicalengineeringblog.com/1948-augmented-reality/

63

[10] R. Azuma , R. Silva, J. C. Oliveira, G. A. Giraldi (2003) A Survey of Augmented Reality.retrieved from http://wenku.baidu.com/view/d52958f9aef8941ea76e052a.html? from=related [11] L Madden(2011). Professional Augmented Reality Browsers for Smartphones: Programming for junaio, Layar and Wikitude. John Wiley and Sons. [12]Publish or Perish - Anne-Wil Harzing." [Online]. Available: http://www.harzing.com/pop.htm [Accessed: Dec 11, 2011]. [13]L. EGGHE (2006) Theory and Practice of the g-index Scientometrics, Vol. 69, No. 1 (2006) 131152 retreived from http://sci2s.ugr.es/hindex/pdf/Egghe2006.pdf [14]L Bornmann and W Marx (August 2011) The h -index as a research performance indicator European Science Editing; 37(3) retrieved from http://www.ease.org.uk/artman2/uploads/1/viewpointsBornmann77-801.pdf [15] L Hardesty (Jan 24, 2011) 3-D TV? How about holographic TV?

MITnews retrieved from http://web.mit.edu/newsoffice/2011/videoholography-0124.html [16] Not Here :An augmented reality project by MANIFEST.AR ( June 6 -

November 27, 2011) Samek Art Gallery Bucknell University retrieved from

http://www.bucknell.edu/x70622.xml [17] Kudan Art Gallery retrieved at http://www.youtube.com/watch?

64

v=ALxuE01TZlc [18] Green Box Project retrieved from http://vimeo.com/greenboxproject [19] G Dewald (Oct 2009) Augmented Reality Mask: A project for Dark Arts Burlington VT retrieved from http://thoughtfaucet.com/making-things/examples/augmentedreality-mask-a-project-for-dark-arts-burlington-vt/ [20]Mistry (2009) TED Talk retrieved from http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.html [21] D Sung(Mar. 4, 2011)The future of augmented reality. What the experts say is just round the corner. Pocket-lint online retrieved at http://www.pocketlint.com/news/38902/future-of-augmented-reality-experts [22] Mok Oh (2009-08-23). "Is That *Really* Augmented

Reality?". allthingsv.com. Retrieved from http://allthingsv.com/2009/08/23/is-thatreally-augmented-reality/ [23] Sundin Jan 15 2012 augmentedreality.Collected Retrieved from http://augmentedreality.collected.info/ [24] Lucozade Aurasma - Plan B + Tinie Tempah retrieved from http://vimeo.com/32157715 [25]Gabbard and Swan(2001) Usability Engineering for Augmented Reality: Employing User-based Studies to Inform Design Retrieved at http://people.cs.vt.edu/jgabbard/publications/tvcg07.pdf [26]. Baxter A. (2000) Model of Inductive Bias Learning Journal of Articial Intelligence Research 12 149198 retrieved from http://people.ee.duke.edu/~lcarin/Baxter2000.pdf

65

[27] J. Nielson (2001) Why You Only Need to Test with 5 Users. retrieved from http://www.useit.com/alertbox/20000319.html

66

67

Appendex A: Forms for User Testing

68

69

70

71

Appendix B: Videos of the Hamilton Gallery Augmented Reality Experience

1. Points of Interest This video shows how a gallery user experiences a Point Of Interest for the Hamilton gallery with the Junaio browser. http://www.youtube.com/watch?v=Iwyf1-7TxoA

72

2. Augmented Brochure

The following videos show the results of the Hamilton Gallery brochure that has been modified with image recognition capability. When the phone camera is pointed at a picture , video content is overlaid on the image to entertain the reader of the brochure. a) http://www.youtube.com/watch? v=t88K2yTOCwE&feature=context&context=C3766631ADOEgsToPDskJ2a7gOVzR3 OF7wOT5XRriI

73

b) http://www.youtube.com/watch?v=YpRRi_Eccw&feature=context&context=C3766631ADOEgsToPDskJ2a7gOVzR3OF7wOT 5XRriI

c)http://www.youtube.com/watch?v=tyrqu7pQWbg&feature=related

3. Artwork inside the gallery launches artist's portfolio with QR marker

This video features a gallery user scanning a QR marker next to a piece of artwork inside the gallery. The QR marker launches the artists portfolio on the camera screen. http://www.youtube.com/watch? v=oaS2CJprjII&context=C332004cADOEgsToPDskLGB5jqPr8encNQmZxIUY3D

74

75