Promoting Virtual Social Interaction With Older People

InternationalWorkshoponCognitiveProsthesesand AssistedCommunication(CPAC2006)
In conjunction with 2006 International Conference on Intelligent User Interfaces (IUI2006) Sydney,Australia January29,2006 Organizers:
NormanAlm,UniversityofDundee(UnitedKingdom) ShinjiAbe,ATRIRCLabs.(Japan) NoriakiKuwahara,ATRIRCLabs.(Japan)
ProgramCommittee:
SueBalandin,SydneyUniversity(Australia) JosephF.Coughlin,MITAgeLab(U.S.A.) JoyGoodman,UniversityofCambridge(UnitedKingdom) MotoichiroKato,KeioUniversity(Japan) HenryKautz,UniversityofWashington(U.S.A.) KazuhiroKuwabara,ATRIRCLabs.(Japan) MichaelJ.Lyons,ATRIRCLabs.(Japan) AndrewMonk,UniversityofYork(UnitedKingdom) SiriwanSuebnukarn,AsianInstituteofTechnology(Thailand) ToshiyoTamura,ChibaUniversity(Japan) YasuyukiYanagida,MeijoUniversity(Japan) KiyoshiYasuda,ChibaRosaiHospital(Japan)
Preface
Various prostheses have been developed to help people with physical impairments, by replacing lost or underdeveloped abilities, and these have played an important part in facilitating the persons social participation. Providing prostheses for lost or underdeveloped cognitive abilities, however, presents a challenge of a different order. Someworkonmemoryaidsanddailyschedulemanagementsystemshasshownpromise, but progress in this field will depend upon keeping up to date with our advancing knowledge about cognition generally as well as emerging technologies which could be platforms for various kinds of cognitive support. An interdisciplinary approach to this programmeofworkwillbeessentialinordertoproduce: Intelligent systems for augmenting or replacing impaired or absent cognitive functions. User interfaces that make it possible for cognitively impaired people to make effectiveuseofthesupportoffered. Contextawarenessthatrecognizesthesituationofthecognitivelyimpairedperson anddetectsappropriatemomentsforofferingsupport. Thisworkshopwilloffertheopportunityforresearchersinthefieldsofassistivetechnology, cognitive psychology, user interface design and contextawarenessto present the state of theartineachfieldandtodiscussanapproachandaresearchagendaforrealizingeffective cognitiveprostheses. Whiletheintendedaudienceforthisworkshopincludesthosewithexperienceorinterest inAssistiveTechnologyespeciallyfordementiapeopleandtheircare,itisalsocrucialto have participation by those with expertise in other areas such as Artificial Intelligence, IntelligentUserInterfaceDesign,Contextawaresystems,andCognitivePsychology.
TableofContents

PersonalizedAssistance(9:0010:30)
UsingSemanticWebTechnologiesforCognitiveProsthesesinNetworkedInteraction Therapy ...................................................................................................................................... 1 KazuhiroKuwabara(ATR),NoriakiKuwahara(ATR),ShinjiAbe(ATR),andKiyoshiYasuda (ChibaRosaiHospital) Towards a Wearable Cognitive Prosthesis to Support What and Who Type MemoryActivities .................................................................................................................. 6 TatsuyukiKawamura,YasuyukiKono,andMasatsuguKidode(NAIST) Visionbased Behavior Detection for Monitoring and Assisting MemoryImpaired People ..................................................................................................................................... 10 Akira Utsumi (ATR), Daisuke Kanbara (Ritsumeikan University), Shinjiro Kawato (ATR), ShinjiAbe(ATR),andHironoriYamauchi(RitsumeikanUniversity)
InterfaceDesign(11:0012:30)
Developinganengagingmultimediaactivitysystemforpeoplewithdementia ......... 16 ArleneAstell(UniversityofSt.Andrews),NormanAlm(UniversityofDundee),GaryGowans (DuncanofJordanstoneCollegeofArt),MaggieEllis(UniversityofSt.Andrews),RichardDye (UniversityofDundee),andPhilVaughan(DuncanofJordanstoneCollegeofArt) Networked Reminiscence Content Authoring and Delivery for Elderly People with Dementia.................................................................................................................................. 20 NoriakiKuwahara,ShinjiAbe,andKazuhiroKuwabara(ATR) AneasytouseInterfaceforEvidentialReasoner.............................................................. 26 QingXiangWu,DavidBell,andRashidHafeezKhokhar(QueensUniversityBelfast)
CommunicationandCognitivePsychology(14:0016:00)
Acognitiveprosthesistosupportcommunicationbypeoplewithdementia ............... 30 Norman Alm (University of Dundee), Richard Dye (University of Dundee), Arlene Astell (University of St. Andrews), Maggie Ellis (University of St. Andrews), Gary Gowans (DuncanofJordanstoneCollegeofArt),andJimCampbell(UniversityofDundee) PromotingVirtualSocialInteractionwithOlderPeople .................................................. 37 PeterBagnall,GuyDewsbury,VictorOnditi,andIanSommerville(LancasterUniversity) Talking with Individuals with Dementia on Video Phone: A Preliminary Study for NetworkedInteractionTherapy ........................................................................................... 43 KiyoshiYasuda(ChibaRosaiHospital),KazuhiroKuwabara(ATR),NoriakiKuwahara(ATR), ShinjiAbe(ATR),andNobujiTetsutani(TokyoDenkiUniversity) TheRelationshipsamongCognition,ParentalSpeechPatternsandForeignLanguage Ability,andMultipleIntelligenceTheorybasedForeignLanguageTeaching .............. 47 KeikoHayashi(HiroshimaJogakuinUniversity)
Using Semantic Web Technologies for Cognitive Prostheses in Networked Interaction Therapy
Kazuhiro Kuwabara
ATR 2-2-2 Hikaridai, Seika-cho Soraku-gun, Kyoto 619-0288 Japan
Noriaki Kuwahara
kuwabara@atr.jp Shinji Abe

kuwahara@atr.jp Kiyoshi Yasuda

Chiba Rosai Hospital 2-16 Tatsumidai-Higashi Ichiharashi, Chiba 290-0003 Japan
sabe@atr.jp ABSTRACT
This paper explores the concept of a semantic desktop as a platform for Networked Interaction Therapy, which is a framework for providing support for people (clients) with dementia or memoryimpairment. Since personalized services are essential in networked interaction therapy, there is a need for both clients and caregivers to handle personal data. The semantic desktop can be used as a platform for handling these personal data. Application scenarios include supporting the creation of multimedia content to be used in therapy, such as reminiscence videos, and storing and analyzing care sessions, as well as providing a collaboration infrastructure intended for remote care sessions. The networked interaction therapy system is composed of a number of applications, and the metadata attached to the application data is utilized for sharing the data among dierent applications.
fwkk5911@mb.infoweb.ne.jp
memory-impairment is becoming an important problem. In addition, a brain injury or disease sometimes results in cognitive impairment. Networked Interaction Therapy is intended to support caregivers and handicapped people (clients) through networked communication [5]. The communication, here, includes presenting proper content to keep the clients attention focused. For example, a reminiscence video, which is a slide show video made from old photo albums of the clients, is used to stimulate past memories. Its clinical eectiveness has been reported [14]. We created an authoring tool that utilizes a Semantic Web framework to support the creation of reminiscence videos [7], where annotations attached to the photos are used to determine what kind of visual eect is to be added to make the reminiscence video more attractive. Since the reminiscence video involves personal content, such as the clients own photos from the past, it can be viewed from the aspect of personal information management. Similarly, networked interaction therapy involves other kinds of personal data, such as the clients schedule (for arranging care sessions), a history of care sessions (to prepare for better care sessions in the future). Thus, a unied approach for personal information management is quite useful in networked interaction therapy. This paper explores the potential of the semantic desktop [11] as a basis for the networked interaction system. The next section briey introduces the networked interaction therapy framework and the concept of the semantic desktop. The subsequent section describes application scenarios that are related to the semantic desktop scheme.
Categories and Subject Descriptors

D.2.2 [Design Tools and Techniques]: User interfaces
General Terms
Design, Human Factors
Keywords
Cognitive Prostheses, Assisted Communication, Networked Interaction Therapy
1. INTRODUCTION
As the number of elderly people among the population increases, caring for them, especially those who suer from dementia and/or
2.
NETWORKED INTERACTION THERAPY
Basically, a system for networked interaction therapy is intended to keep the attention of people (clients) with dementia occupied. Figure 1 illustrates an overview of networked interaction therapy. It is assumed that a client sits in front of a video monitor, which is used to present interesting content, such as a reminiscence video, as well as a video phone terminal. The client is connected to the network, and companions (often volunteers) can talk with the client over the network using a video phone so that the care session can be conducted remotely. When a human companion is not available, a screen character may be used instead. Sensing devices, such as
Network
Friends Volunteers Screen Characters
Client
Alternative Interaction Companions
Figure 1: Overview of Networked Interaction Therapy
a camera, are also employed to monitor the status of the client. In addition to visual and audio stimulus (that is, video content), we may use the olfactory or haptic sense to provide a more engaging stimulus. When we consider the use of personal information management, the overall architecture of the networked interaction therapy system can be as shown in Figure 2. The system is designed around a personal data repository that uses the Resource Description Framework (RDF) [13], which is one of the basic semantic web technologies for handling metadata. The personal data repository holds personal data, such as old photos (for reminiscence videos) and a list of acquaintances. It also contains the clients schedule to arrange for care sessions. Additionally, the history of care sessions may be stored for the future analysis. These kinds of data are inherently personal, and are used in a variety of applications that comprise the networked interaction therapy system. On top of the personal data repository, various applications such as an annotation tool, an authoring tool (for reminiscence videos), and a scheduler are employed. The application may include a video phone to communicate with companions (caregivers) over the network. The personal data repository allows the data to be shared with other personal data repositories. This is useful, for example, for arranging remote care sessions with potential companions. This data centric view of the network interaction therapy system requires personal information management for both the client and the caregivers. Since the metadata plays an important role, the semantic desktop [11] concept can be applied. The core idea of the semantic desktop is to bring semantic web technologies to desktop computers [10]. Various kinds of applications concerning personal information management are integrated by making use of metadata described using ontologies in the semantic desktop. Its personalized aspect, i.e., the fact that the personal model of information handling is respected, and its social aspect, i.e., the fact that information sharing among people is inherently facilitated though common ontologies, make the semantic desktop well suited as a platform for networked interaction therapy. In the next section, several use case scenarios of networked interaction therapy are presented.
of this. The reminiscence video is intended to help elderly clients to remember old times and enjoy peaceful moments. It is a slide show video made from old photos. We developed an authoring tool for a reminiscence video, which utilizes annotations attached to photos (Figure 3) [7]. The process of creating a reminiscence video is as follows. First, the photos to be used are annotated with information provided by the clients family members. We used the following vocabulary to describe annotations: Dublin Core, FOAF, and Image Regions. Dublin Core is used to describe properties of the photo itself, such as the title and the date the photo was taken. FOAF is used to describe people in the photo. Since a reminiscence video often involves people whom the client knows in the present and past, this vocabulary plays a central role in the photo annotations. Image Regions are used to describe certain regions in the photo. This is used to represent the portion of the photo that depicts a certain person. The depicted person is then described by using FOAF. In addition, since the relationship between people in the photo and the client (viewer) needs to be described in detail, we used the RELATIONSHIP vocabulary to describe the kinship relationship, such as parent, son, and daughter. Figure 4 shows an example of photo annotations using this vocabulary. The video creator then selects the photos to be used in the reminiscence video from the annotated photo archive. The Jena toolkit is used to store the annotations in RDF format, and RDQL is used to query the photo archive for suitable photos. The photos are then sorted according to certain criteria, such as the date that the photo was taken. In order to make a reminiscence video more eective, visual eects like zooming and panning are added. The visual eects to be applied are automatically selected based on the annotation attached to the photo. The visual eect templates determine what kinds of visual eects should be added. The reminiscence video is a typical example of tools that can be of help to caregivers and clients. Other examples include the CIRCA project, which utilizes multimedia content to aid conversations between the caregivers and the clients [3]. In addition, the concept of a video respite is proposed in dementia care and utilized in the care setting [8]. The respite video is intended to give rest to the caregivers (often family members) while the client is watching the video. As these examples indicate, multimedia content can play an important role in dementia care. Since eective content is dierent from
3. APPLICATION SCENARIOS 3.1 Content Creation

3.1.1 Reminiscence videos
Video and audio are inherently powerful and eective stimuli for networked interaction therapy clients. The reminiscence video, which is mentioned in the introduction section, is a typical example
client (handicapped)
caregiver
desktop user interface authoring tool annotation tool scheduler video phone
communication
reminiscence videos photos
care session records acquaintances
schedule data
data sharing
RDF-based personal data repository
Figure 2: Personal Data Repository for Networked Interaction Therapy
Image Regions
Dublin Core
FOAF
Figure 3: Annotation tool for creating reminiscence videos
person to person, personalized content should be employed. If personal content could be compiled semi-automatically from a set of data in the personal data repository, it would be benecial both to the client and the caregivers. In the case of the reminiscence video, dierent videos can be created from the same set of photos taking into consideration the viewers preferences, which may be recorded in his/her personal data repository.
One of the possible ways to create such content is to use a slide show in a similar way to the reminiscence video described above. In this case, a photo will be taken at an important step of the target task by a caregiver or volunteer, and stored in the personal data repository. Annotations that describe the step that the photo depicts are then added. The navigational slide show video can be created from these photos using an authoring tool similar to the reminiscence video authoring tool by making use of the annotations attached to the photos.
3.1.2 Instructional content

For patients with memory-impairment, it is often dicult to plan their everyday activities. In order to cope with this problem, it is important to provide a cue to the client in a timely fashion. Examples include navigational content that guides the client through a certain task, such as cooking. Another example is a navigation system for taking public transportation [4]. Since everyday life activities are dierent from person to person and the content and timing of the cues need to be provided to satisfy the clients personal needs, they are often better created with the help of family members who live with the client.
3.2
3.2.1
Collaboration infrastructure
Scheduling remote care sessions
In networked interaction therapy, a care session is conducted over the network using a network communication tool, such as a video phone. In this case, scheduling a remote session is one of the essential functions of the networked interaction therapy system. In order to facilitate scheduling of the remote care session, the personal data repository of the client can exchange the schedule information with that of potential companions. The iCalendar format [2] can be used
at Disney at Disney Animal Kingdom dc:title dc:description They are very happy.
20040716 foaf:Image dc:date rdf:type sample.jpg
(x1,y1) h1 w1 (x2,y2)
imgReg : hasRegion
imgReg : hasRegion rdf:type
rdf:type
imgReg : Rectangle
imgReg:coords
foaf:Person rdf:type
rdf:type rel:grandchildOf foaf:gender
x1,y1,x2,y2 imgReg : imgReg : boundingBox regionDepict x1,y1,w1,h1
foaf:name
Literal Resource
Viewer
female
Haruka
Figure 4: Example of photo annotation
to represent such kind of data, and its RDF based representation has also been proposed [1]. The RDF representation is suitable for the personal data repository. In addition, when the companion talks with the client over the network, the background information on the client, such as his/her interests and hometown, might be helpful. If this kind of private information could be stored in the personal data repository and shared with the companion, it would help the companion to conduct a care session more smoothly.
be modied for the client, for example, by incorporating a touchpanel [3]. At the same time, since the degree of impairment is dierent from client to client, how the GUI should be customized is better described in a declarative way, and can be stored in the personal data repository.
3.3.2
Capturing the interaction and its analysis
3.2.2 Remote collaborative annotation

The remote care session described above can also be used for adding annotations to the photos. That is, the companion and the client can talk over the network using a video phone. While the companion asks questions about a photo to the client, the companion inputs photo annotations to the system from the answers given by the client. This process assumes the functionality that the same photos can be displayed in dierent locations in a synchronized way. Overhead involved in the annotation process can be lessened, since the process of adding annotations is incorporated in the care sessions [6]. Presenting the content in a synchronized way remotely would be useful not only for the photo annotation process, but for other types of remote collaboration. This functionality would be essential when the semantic desktop is applied as an infrastructure for remote collaboration.
Since the system can potentially include a video camera to take a video of the client, it is relatively easy to obtain the responses of the client to the content presented. When the response is recorded, what kind of content is being presented can also be recorded at the same time. This information is added as an annotation to the recorded video and can be used later to analyze the response of the client to the content. The resultant video is useful for evaluating the eectiveness of the content presented. For example, a caregiver can analyze whether the client is interested in the video [12]. By incorporating an image understanding technique, the system may automatically mark the point at which the response of the client changes. These marks will help a human user (in this case, the caregiver or medical sta) to look back at the clients responses and interpret the consequences [9]. The results of this analysis can also be added to the video as an annotation. Moreover, by combining the results of the viewer with the personal schedule of the client, it makes it easier to further analyze the effects of the content, which may be dependent on the condition of the client. By gathering much evidence, it might be possible to extract knowledge or know-how on the content to be presented to the client using a data-mining technique.
3.3 Content presentation and analysis of the care sessions

3.3.1 Playback environment
The semantic desktop can possibly be used as an interface to provide a slide show video to the client. Of course, the GUI needs to
4.
CONCLUSION AND FUTURE WORK
This paper discussed the potential use of a semantic desktop concept in networked interaction therapy. Since personal data with metadata plays a crucial role, the semantic desktop can be a basis for the networked interaction therapy system.
There are a number of issues to be resolved in order to develop the system according to the application scenarios described here. For example, an access control mechanism needs to be implemented since very personal data is handled in networked interaction therapy. In addition, a vocabulary (or ontology) needs to be developed to describe data, such as the responses of the client to the video content. The use of metadata is a key for connecting the various applications that comprise networked interaction therapy. Furthermore, we need to provide two dierent kinds of user interface: one for the client (the handicapped person) and the other for the caregiver. The same data might be represented dierently according to the type of user, and dierent modes of interaction need to be designed. Currently, the system presented here is being developed. Various kinds of applications are possible in addition to those described here, which make use of the personal data repository. Through the application to the networked interaction therapy system, we plan to further explore the potential of the semantic desktop.
[9] M. J. Lyons, M. Funk, and K. Kuwabara. Segment and browse: A strategy for supporting human monitoring of facial expression behaviour. In M. F. Costabile and F. Paterno, editors, Interact 2005: Tenth IFIP IC13 International Conference on Human-Computer Interaction (LNCS 3585), pages 11201123, 2005. [10] L. Sauermann, A. Bernardi, and A. Dengel. Overview and outlook on the semantic desktop. In S. Decker, J. Park, D. Quan, and L. Sauermann, editors, Proceedings of the 1st Workshop on The Semantic Desktop. 4th International Semantic Web Conference (Galway, Ireland), 2005. [11] Semantic desktop. http://www.semanticdesktop.org/. [12] A. Utsumi, S. Kawato, K. Susami, N. Kuwahara, and K. Kuwabara. Face-orientation detection and monitoring for networked interaction therapy. In Proceedings of Joint 2nd International Conference on Soft Computing and Intelligent Systems and 5th International Symposium on Advanced Intelligent Systems, 2004. [13] W3C. Resource description framework. http://www.w3.org/RDF/. [14] K. Yasuda, A. Iwamoto, T. Nakamura, and K. Kuwabara. Reminiscence video for higher brain dysfunctions. In Proceedings of General Conference of Japan Society for Higher Brain Dysfunctions, page 109, 2004.
5. ACKNOWLEDGMENTS
The authoring tool of the reminiscence video was designed and implemented by Koji Saito. This research was supported by the National Institute of Information and Communications Technology.
6. REFERENCES
[1] Rdf calendar workspace. http://www.w3.org/2002/12/cal/. [2] Internet calendaring and scheduling core object specication (iCalendar). http://www.ietf.org/rfc/rfc2445.txt, 1998. [3] G. Gowans, J. Campbell, N. Alm, R. Dye, A. Astell, and M. Ellis. Designing a multimedia conversation aid for reminiscence therapy in dementia care environments. In CHI 04: CHI 04 extended abstracts on human factors in computing systems, pages 825836, New York, NY, USA, 2004. ACM Press. [4] J. James F. Sullivan and G. Fischer. Mobile architectures and prototypes to assist persons with cognitive disabilities using public transportation. In 26th International Conference on Technology and Rehabilitation, 2003. [5] K. Kuwabara, N. Tetsutani, N. Kuwahara, S. Kawato, M. J. Lyons, A. Utsumi, Y. Yanagida, and K. Yasuda. Networked interaction therapy: Supporting people with mild dementia and their family members with internet. In 20th International Conference of Alzheimers Disease International Kyoto 2004, Abstracts, page 377, 2004. [6] N. Kuwahara, S. Abe, and K. Kuwabara. Networked reminiscence content authoring and delivery for elderly people with dementia. In Workshop on Cognitive Prostheses and Assisted Communication (this proceedings). 2006. [7] N. Kuwahara, K. Kuwabara, N. Tetsutani, and K. Yasuda. Using photo annotations to produce a reminiscence video for dementia patients. In 3rd International Semantic Web Conference (ISWC2004) Demo Papers, 2004. [8] D. Lund, R. Hill, M. Caserta, and S. Wright. Video respite: an innovative resource for family, professional caregivers, and persons with dementia. The Gerontologist, 35(5):683687, 1995.
Towards a Wearable Cognitive Prosthesis to Support What and Who Type Memory Activities
Tatsuyuki Kawamura
Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, Nara 630-0192, JAPAN
Yasuyuki Kono
Masatsugu Kidode
kawamura@is.naist.jp ABSTRACT
kono@is.naist.jp
kidode@is.naist.jp
In this paper, we introduce a wearable cognitive prosthesis1 we have implemented previously [6, 5, 9, 12, 7, 8]. The technology for recording and sharing experiences such as those used in augmented memory has been extensively studied in recent years [1]. The aim of these studies is to enable users to receive the following benets; 1) less time wasted in everyday life, 2) a recycling of experiences, and 3) a facilitation of human-to-human interactions. Previously designed systems, including ours were designed for people with ordinary cognitive processing abilities. These systems are not suitable for people with dementia because the memory process of people with dementia is dierent from the memory process of a person with ordinary cognitive processing. In this paper, we discuss the design of a system for people with dementia. Our existing systems must be partially or totally redesigned in order to realize a cognitive prosthesis that can perform for the suerers of dementia.
people with dementia and people with amnesia are also studied as target users in memory aid research [2]. Unfortunately, however, suerers of dementia or amnesia have not been studied in term of computer-aided memory prostheses as much as ordinary, unremarkable users. At least two reasons exist for this: One is the diculty of design and evaluation. Generally, almost all system developers, researchers, and their colleagues are cognitively ordinary people. Therefore, designing a system useful for the suerer2 is dicult. Moreover an evaluation of the designed system is even more dicult than designing the system itself because few opportunities for experiments using test subjects with dementia or amnesia under the controlled conditions. Another reason for less design and evaluation for suerers of dementia is the gap in market size. The population of people with ordinary cognitive processing is larger than the population of people with dementia or amnesia. Market size obviously determines what research will be done and what will not. Facilitating ordinary people with people with dementia/amnesia interaction is necessary to both group living together because both must interact in many situations together daily. One of possibility of making this interaction smoother and more comfortable is a universal design of a cognitive prosthesis from ordinary people through people with dementia or amnesia. The universal design would expand market size. In order to achieve universal cognitive prostheses, researchers and developers must employ an ordinary person and a person with dementia or amnesia as users to design services and interfaces. In this paper, we rst introduce designs of wearable cognitive prostheses for people with ordinary cognitive processing and then discuss these designs and how they may be used for suerers of dementia or amnesia. We have designed and implemented four types of systems. The systems are respectively called Residual Memory [6], Ubiquitous Memories [5], Im Here! [9, 12], and Nice2CU [7, 8]. We discuss especially the Im Here! system and the Nice2CU system. The Im Here! system supports an object nding task (a what-type memory activity). The Nice2CU system enables a user to remember a persons name, birthday, hobby, etc. The system supports a recollection of an event that is triggered by a person who is in front of the user (a who-type memory activity). Both memory activities are important for not
2 Note that we term simply a person with dementia or amnesia a suerer in this paper.
1. INTRODUCTION
Wearable computing technology has been investigated extensively in recent years resulting in smaller and smaller computers with larger and larger storage capacities. An advantage of a wearable system is that a user can get various services anytime and anywhere. This advantage is good for memory aid because human memory activities take place anytime and anywhere. Interesting research in recording a users experiences has been discussed in such examples as life-log [3]. J. Gemmell et al. have attempted trying to digitize their lives as much as possible in the MyLifeBits project [4]. The main point of view of the life-log project is the method of recording and analyzing experiences. In addition to people with ordinary cognitive processing,
1 We usually call our memory aid system augmented memory. In this study, we term this kind of system cognitive prostheses.
(i )
( ii )
(i )
( ii)
(i ii )
( iv )
Figure 1: Registration mode of the Im Here!. only ordinary people but also for people with dementia or amnesia.
(ii i)
Figure 2: Retrieval mode of the Im Here!.
2. OVERVIEW OF OUR SYSTEMS 2.1 Im Here!

This system retrieves the last recorded video of a users viewpoint including a handled object from a video database. The Im Here! system is composed of a wearable camera [11], which is specially provided for detecting an object handled by the user, a sleeve-on display, and a sleeve-on switch. The Im Here! system includes three operation modes as follows: Registration: A user rst selects this mode (Figure 1 (i) and (ii)) (STEP-I). The user then picks up an object that the user wants to register (STEP-II). The user moves and rotates the object in front of the wearable camera (Figure 1 (iii)) (STEP-III). The system starts capturing appearance data of the object as images (Figure 1 (iv)) (STEP-IV). The system then extracts features for recognizing the registered object (STEPV). Finally, the system records both the feature data and a thumbnail including the object (STEP-VI). Observation: The system has been recording a video while a registered object has been in the view of the wearable camera (STEP-I). The system then updates the database just after the object disappears from the captured image (STEP-II). Retrieval: The user selects this mode (STEP-I). The user checks thumbnails (Figure 2 (i) and (ii)) (STEP-II). The user selects one thumbnail including the object that the user wants to nd (STEP-III). The system replays the last recorded video including the handled object (STEP-IV). Finally, the user nds the object (Figure 2 (iii)) (STEP-V).
Figure 3: Observation and retrieval mode of the Nice2CU (Radio Frequency ID) tag reader. The Nice2CU system includes the following two operation modes: Registration: A person comes to meet a user. The person gives the user a business card attached to an RFID tag (STEP-I). The user reads the information recorded in the RFID tag by using the wrist-on RFID tag reader (STEP-II). The system registers the persons information data (name, birthday, hobby, and etc.) and feature data that are for recognizing the person using the wearable camera (STEP-III). Observation and Retrieval: The user meets the person again. The system recognizes the person (STEP-I). The system has been recording a video while the registered person has been in the view of the wearable camera (STEP-II). The system then updates the database just after the person disappears from the captured image (STEP-III). The system displays the persons information data, and replays the video recorded when
2.2 Nice2CU
This system retrieves the last recorded video of a users viewpoint including the person who is in front of the user. The Nice2CU system is composed of a wearable camera for detecting the person in front of the user, and a wrist-on RFID
the user last met the person (Figure 3) (STEP-IV).
3. DISCUSSION OF SYSTEM DESIGNS 3.1 Distinguishing an Operator from a User

A system design model distinguishing an operator from a user is important in achieving a wearable cognitive prosthesis for suerers. Both the Im Here! system and the Nice2CU system have registration, observation, and retrieval mode. It is especially hard for the suerer to understand how he or she can operate the system in the registration mode and the retrieval mode. A caregiver including the family must control the system as an operator in these two modes. The subject person in this case is the suerer who records his or her own experiences. The suerer then implicitly requires service as a user.
to complete an object nding task. A person doing the object nding task instead of the suerer would be a caregiver including the family. The number and variation of objects that must be managed are fewer than in the case of people with ordinary cognitive processing. Objects, which should manage carefully, are valuables for the suerer. These objects are sometimes troubled objects between the suerer and the caregiver. The priority registration of troubled objects would lift the burden of nding objects that the suerer wants from the caregiver. We assume that the number of troubled objects for everyday life is 10 at most. In the case of the appearance-based object recognition method, a user uses the system to capture many appearances of an object so that there are many appearances when the object posture is changed. Also, the user must associate the name and property of the object with the features calculated from the set of the appearances of the object. In comparison with the appearance-based method, the tag-based object recognition method allows the user to register only the name and property of the object with an RFID tag. Most objects would be represented by attached or implanted RFID tags, respectively only if a tag-based ubiquitous society was widely extended from what it is today. Also, the tag would include the name and property of the object with the tag in advance. However, a user with the appearance-based object recognition method never omits an operation for an association of the name and property of the object and the feature computed from the set of the appearances of the object. The tag-based object recognition method is more suited to performing as a cognitive prosthesis for suerers than the appearance-based object recognition method. In the case of the tag-based method, there are some objects to which RFID tags cannot be attached. Nonetheless, RFID tags can be attached to all the suerers objects if a caregiver attentively selects objects to buy for the suerer.
Figure 4: A system design model distinguishing an operator from a user Figure 4 illustrates the system design model which distinguishes an operator from a user. An person with ordinary cognitive processing behaves as a user in conjunction with an operator. The system supports the user directly. On the other hand, a suerer is only a user. The suerers system establishes communication with a caregivers system. The caregivers system supports the caregiver to be able to care for the suerer by using the information from the suerers system. A system developer can design both systems with a system conguration. Performing rules for input and output depending on the user is important if the core design is universalized.
3.3
Controlling Inter-Personal Communication
3.2 Arranging Articles for Everyday Use

In terms of a wearable system, methods exist for recognition of objects: 1) an appearance-based object recognition method and 2) a tag-based object recognition method. In the case of appearance-based object recognition, a wearable system extracts an object image from an image captured by using a wearable camera. On the other hand, a wearable system can employ a visual tag or an RFID tag to detect an object. In this paper, we select an RFID tag system for recognizing an object. A person with ordinary cognitive processing arranges physical objects by him or her self. Objects can be classied by the frequency type of their use. High-use objects are articles for everyday use. Low-use objects represent objects stored in a warehouse, cupboard, etc. for a long time. The total number of registered objects should be over 100 even though a user limits objects to those he or she wants to nd quickly. In the case of a suerer, the suerer would nd it dicult
A suerer might feel a higher calmness of mind by interacting with ordinary people. The person with ordinary processing emphasizes exchanging correct information to keep communication channels open with other people. The sufferer, in contrast, nds it dicult to establish communication with an another person, such as an acquaintance. In order to establish a connection, both the suerer and the acquaintance must be aggressive. Facilitation of a communication channel and keeping it open is an important issue in allowing the person with ordinary processing to perform action for the connection. The Nice2CU system we have designed for ordinary people activates thier communication channels by allowing the exchange of information between communicators. Four types of information about a person are essential to this system:
A PROFILE includes current information about the person (e.g. persons face, name, sex, birthday, blood type, birthplace, current address and aliation.) An EXPERIENCE contains an event memorized by a certain person, and includes the context of what hap-
pens in an event that was experienced together by a recipient and a target person. A MESSAGE includes a temporary memorandum similar to a post-it for delivering a message to the target person. For instance, the recipient suddenly remembers that he or she wants to repay money to the target person when the recipient accidentally runs into this target person. HUMAN RELATIONS consists of the above elements: PROFILES, EXPERIENCES, and MESSAGE. For example, the target person was the recipients former boss when he was in college. The system selects and displays appropriate information for facilitating communication by calculating the latest relationship among the user, the target person, and other people. The ordinary person can interact with others by considering displayed information correctly. On the other hand, the suerer nds it dicult to establish communication with others by using only the displayed information. Knowing how to get information from a suerer is necessary for keeping communication channels open with the suerer. To get such information, the system must give appropriate information to activate communication to a suerer, an acquaintance, and a caregiver respectively. If a suerer as a user meets a person he or she has met before, the suerer cannot establish communication with the person because the suerer will nd it dicult to recognize who the person is. The system must give the latest past-information in all information that the suerer can remember, e.g. events in the childhood. For instance, a system, which can recognize when they have met, similar to the Nice2CU system, displays photos or videos recorded in the childhood. Also, the system must let the acquaintance understand that the past-information is useful in making conversation with the suerer. The recorded video of when the suerer met the acquaintance is also available for the caregiver. The caregiver can talk about the event with the acquaintance of the suerer. Additionally, recorded video would be more useful once the caregiver edits the video and annotate semantic data on the video [10].
6.
[2]
REFERENCES
http://www.media.mit.edu/wearables/mithril/memoryglasses.html.
[1] http://www.ii.ist.i.kyoto-u.ac.jp/ sumi/pervasive04/.
[3] K. Aizawa, T. Hori, S. Kawasaki, and T. Ishikawa. Capture and ecient retrieval of life log. In Pervasive 2004 Workshop on Memory and Sharing Experiences, pages 1520, 2004. [4] J. Gemmell, R. Lueder, and G. Bell. Living with a lifetime store. In ATR Workshop on Ubiquitous Experience Media, pages 6976, 2002. [5] T. Kawamura, T. Fukuhara, H. Takeda, Y. Kono, and M. Kidode. Ubiquitous memories: Wearable interface for computational augmentation of human memory based on real world object. In 4th International Conference on Cognitive Science (ICCS2003), pages 273278, 2003. [6] T. Kawamura, Y. Kono, and M. Kidode. A novel video retrieval method to support a users recollection of past events for wearable information playing. In 2nd IEEE Pacic-Rim Conference on Multimedia (PCM2001), pages 2431, 2001. [7] T. Kawamura, Y. Kono, and M. Kidode. Nice2cu: Managing a persons augmented memory. In 7th IEEE International Symposium on Wearable Computers (ISWC2003), pages 242243, 2003. [8] T. Kawamura, C. C. Lin, Y. Kono, and M. Kidode. Rating human relations for recommendation of an augmented video memory using human prole data and meeting logs. Technical Report NAIST-IS-TR2003017, 2003. [9] T. Kawamura, Y. Miyake, T. Shibata, K. Suzuki, K. Nishino, N. Saiwaki, Y. Kono, and M. Kidode. A casual wear with an object-nding support system. In Extended Abstract of the 9th IEEE International Symposium on Wearable Computers (ISWC2005), 2005. [10] N. Kuwahara, K. Kuwabara, N. Tetsutani, and K. Yasuda. Using photo annotations to produce a reminiscence video for dementia patients. In 3rd International Semantic Web Conference (ISWC2004), 2004. [11] T. Ueoka, T. Kawamura, S. Baba, S. Yoshimura, Y. Kono, and M. Kidode. Wearable camera device for supporting object-triggered memory augmentation. In 3rd CREST/ISWC Workshop on Advanced Computing and Communicating Techniques for Wearable Information Playing, pages 4653, 2004. [12] T. Ueoka, T. Kawamura, Y. Kono, and M. Kidode. Im here!: a wearable object remembrance support system. In 5th International Symposium on Human Computer Interaction with Mobile Devices and Services (MobileHCI2003), pages 422427, 2003.
4. CONCLUSIONS
In this paper, we have introduced wearable cognitive prostheses that we have implemented in the past. Especially, we explain the Im Here! system for supporting an objectnding task and the Nice2CU system for remembering an acquaintances information. We also discussed the designs of these systems in regard to people with disease associated with dementia or amnesia so as to improve or redesign these systems for people with dementia or amnesia. We have proposed a system design model distinguishing an operator from a user. It is important for developers to achieve a cognitive prostheses for both people with ordinary cognitive faculties and people with dementia or amnesia.
5. ACKNOWLEDGMENTS
This research is supported by Core Research for Evolutional Science and Technology (CREST) Program Advanced Media Technology for Everyday Living of Japan Science and Technology Agency (JST).
Vision-based Behavior Detection for Monitoring and Assisting Memory-Impaired People

Akira Utsumi
ATR Intelligent Robotics and Communication Laboratories 2-2-2 Hikaridai Seikacho Sorakugun Kyoto 619-0288, Japan
Daisuke Kanbara
Ritsumeikan University 1-1-1 Nojihigashi, Kusatsu City Shiga 525-8577, Japan
Shinjiro Kawato
kanbara@atr.jp
utsumi@atr.jp Shinji Abe

skawato@atr.jp Hironori Yamauchi

Ritsumeikan University 1-1-1 Nojihigashi, Kusatsu City Shiga 525-8577, Japan
yamauchi@se.ritsumei.ac.jp
In this paper, we describe our methods to detect human behavior for monitoring and assisting memory-impaired people. To assist human activities in daily life, we must be able to understand a users situation/state in the current task: what information is useful for the user now. Our posturedetection system using IR cameras and IR pattern projectors detects the users state as the 3-D appearance (posture) of him or her. In the system, human behavior is modeled as a distribution of 3-D appearances, and the results of this behavior detection are used to determine instructions to be given to the user. We show an sample implementation of the system for a toilet task. In addition, we also address a method to display video content to people with dementia. In the method, user behavior, including facial movements and body motions, is detected with vision-based methods. User attention to the displayed content is then estimated based on the facial orientation and body motions. The estimated attention level is used for content control to attract and maintain the attention of the viewer to the display system. Experimental results suggest that the content-switching mechanism does indeed eectively attract user interest. SENTATION (e.g., HCI)]: User InterfacesInteraction styles
General Terms
Vision-based HCI
Keywords
Behavior Detection, Action Assistance, Content Switching, Computer Vision
1. INTRODUCTION
For a computer system to eciently support human activities, it has to recognize users behaviors and understand their intentions. Therefore, the ability to recognize human behavior by using sensors embedded in living environments is becoming an increasingly important research endeavor. The same situation prevails in our networked interaction therapy system, which entertains and eectively attracts the attention of memory-impaired people and lightens the burden of helpers or family members [1]. The main goal of networked interaction therapy is to support the daily activities of memory-impaired people by enhancing their quality of life and reducing the burden on their family. For this purpose, we give networked interaction therapy the capability to automatically detect the intentions of a memory-impaired person. This provides the necessary information/entertainment for guiding the individual to a more comfortable condition on behalf of his or her family before the occurrence of such behavioral problems as wandering at night, incontinence, ts of temper, and so on. Various sensory devices can be used to detect user behaviors. Vision-based detection of human behavior ts our needs since it does not require any special attachments [2, 3, 4, 5]. The system can remotely detect human motions, and users do not need to be aware of the sensory systems.

H.5.2 [INFORMATION INTERFACES AND PRE-
10
In this paper, we describe a method to detect human behavior for monitoring and assisting memory-impaired people. To assist human activities, we have to accurately know the users situation/state, and to do this we use an IR-based range nder for estimation. In the system, human behavior is modeled as a distribution of 3-D appearances, and the results of this behavior detection are used to determine instructions to be given to the user. In addition, we estimate user attention based on facial orientation and body motions related to visual/aural stimuli in displayed video/audio contents detected with a vision-based system. In the proposed method, if user attention strays from the target medium, the system changes the contents of the displayed information or prompts the user to lure him or her back. In the next section, we describe our action assistance method using 3-D posture observation. Section 3 outlines the framework of our attention estimation, and in Section 4, we give our conclusions.
Figure 2: Behavior Detection by IR Pattern Projection tect 3-D positions on the target object. To avoid interference among projected spot patterns in the camera observations, we calculate the geometrical relations between every two spot patterns and then form groups of spot patterns so that a separate spot observation is guaranteed within each group. By projecting the group of spot patterns, we can avoid confusion among multiple spot patterns and thus simplify the spot detection process.
2.
ACTION ASSISTANCE USING HUMAN POSTURE OBSERVATIONS
In the daily activities of memory-impaired people, it becomes a serious problem that caregivers must always watch and assist each action made such as using the toilet, shower, etc. Some of these actions are routine tasks that are sometimes dicult and often boring for caregivers due to their repetitive nature.
2.1 Behavior Detection using IR Detection system

In our method, we can model the typical behavior of users and detect the current state by using matching based on the 3-D model. Regarding behavior detection, our system detects 3-D shape patterns of the scene around the detected human location, and compares them with the pre-stored data (Fig. 3).
IR pattern Recognition
Projector
Observe
Behavior Database
Camera
Figure 1: IR range nder Figure 3: Behavior Detection To achieve action assistance, we have to rst conrm the state of the user (i.e., a memory-impaired person). Here, we consider a scenario where the system gives voice instructions to a user for him/her to achieve a typical daily task (using the toilet, changing clothes, etc. (Figure 2)). In these situations, normal vision systems are not suitable due to privacy considerations. In addition, a method that is robust against changes in clothing color/illumination conditions is required. Therefore, we have employed an IR-pattern-based system to detect the 3-D geometries of multiple sample points. By using the system, we can robustly detect 3-D positions of multiple points on the users body without detecting texture (normal camera image) information. This capability is suitable for our purpose. Figure 1 shows an overview of the implemented system. Here, we used DLP projectors (resolution 800 600 pixels) to project IR patterns. CCD cameras are used for image observations. Both projectors and cameras have IR (infrared) lters on their lens units. The 3-D measurement method using projected patterns is popularly used for detecting 3-D object shapes in computer vision [6, 7]. Here, we use multiple IR spot patterns to deHere, let Pi be the i-th 3-D point in the model P and Qj be the j-th point of the current 3-D patterns. Pi = xp i Qj = xq j yp i yq j zp i , zq j . (1) (2)
Accordingly, the distance between the two sets of 3-D points P and Q can be dened as follows. Distance(P, Q) =
N X i=1 j{1,...,M }
min
(Pi Qj ) (Pi Qj )/N. (3)
After comparison of the input 3-D shape with all stored models, the model that best ts the current input is selected as the current users state.
2.2 Example: Toilet Task Instruction using Behavior Detection

We applied the above system to a real scene: a toilet. As we mentioned, there is diculty with installing image
11
Table 1: Measurement Errors (cm) X Y Z Ave. 6.37 -1.42 3.89 S.D. 3.56 0.99 2.42
sensors to private spaces like toilets due to invasions of privacy. Our method, however, only observes 3-D positions of discrete spot patterns and ts them to the situation. Figure 4 shows the conguration of the implemented system. Here, an IR range nder is placed above the toilet and a small LCD display is placed on the side wall to display a CG character that gives voice instructions to users (Figure 5). We designed the system to instruct on toilet usage to users in a stepwise manner. We divided the toilet task into six small subtasks (i.e., open the top cover, turn around, take pants down and sit down, ush the toilet, stand up and pull pants up, and close the top cover). When a user has completed one subtask, the system then provides instructions for the next subtask. If the user does not undertake a subtask within a certain time, system provides the same instruction again. If a user takes a dierent action and deviates from the task scenario, the system will alert the caregiver(s). The behavior in each subtask is modeled as a series of 3-D posture transitions (the number of 3-D postures for one subtask is one to three here). In the next section, we conrm the systems feasibility with a larger number of behavior sets.
Figure 5: CG Character Display
Absence
Stand Open Cover Rotate
Sit
target
Figure 6: 3-D shape reconstruction (top: scene, bottom: reconstruction results)
with a larger number of categories. Figure 7 shows representative appearances for eight dierent pose categories: stand frontal (STF), stand leftward (STL), stand rightward (STR), sit frontal (SIF), sit leftward (SIL), sit rightward (SIR), dressing (DR), and reaching action (RE).
2.3 Experiments
To conrm the eectiveness of the proposed system, we conducted the following experiments. First, we measured an arbitrary 3-D plane with our system and evaluated the systems accuracy. Table 1 shows the results of the evaluation. It contains the average and standard deviations of errors for each axis, revealing that our system can measure 3-D locations with errors of less than 10 cm. Next, we reconstructed the 3-D shapes of real objects, with Fig. 6 showing an example of 3-D reconstruction. Here, one absence scene and four typical human postures in the toilet task are measured by our system. Fig. 6 bottom shows the result of 3-D reconstruction. As can be seen, the 3-D shape of a person is accurately reconstructed. We also conducted experiments on behavior recognition
STF
STL
STR
SIF
SIL
SIR
DR
RE
Figure 7: Eight Categories Table 2 shows the recognition results for 48 samples. Finally, used our system to continuously observe a persons behavior. Figure 8 shows the results, where the horizontal axis denotes the time and the vertical axis corresponds to the category selected for each frame. As can be seen, our system properly recognizes the users behaviors.
IR Range Finder
3. CONTENT SWITCHING FOR ATTRACTING AND MAINTAINING THE ATTENTION OF VIDEO WATCHERS
In this section, we address our content-switching method for attracting and maintaining the attention of video-watching people. Further details, including a pilot observation on actual behavior of memory-impaired patients watching videos, is described in [8]. Since the human vision system plays an important perceptual and communicative role in our daily life, gaze can be a strong cue for estimating user attention. In our attention monitoring, we rst focus on the orientation of the face. Strictly speaking, facial orientation is dierent from human gaze due to the lack of eye information. How-
CG Character Display
Figure 4: System Conguration (Toilet)
12
STF STL STR SIF SIL SIR DR RE
STF 50.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
Table 2: Recognition Rate (%) STL STR SIF SIL SIR DR 0.0 0.0 0.0 0.0 0.0 50.0 83.4 16.6 0.0 0.0 0.0 0.0 16.6 83.4 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0
RE 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0
STATE Sit Rotate Open Cover Stand Absence TIME Recognition Result
Actual Motion
systems have achieved both robustness and high-speed processing. In an eort to solve this problem, we have developed a real-time and stable face tracking system via SVM-based face detection coupled with preltering with SSR lters. Figure 9 (left) shows an example of face tracking. Graphics are overlayed on the facial image, indicating detected eyes and the tip of the nose. The detected facial orientation is indicated by the bold line in the narrow region along the top of the image. Figure 9 (right) shows the results of face orientation estimation (horizontal). Here 1 in the vertical axis denotes the limit of detection, and for this subject it corresponds to about 45 degrees.
Figure 8: Recognized Sequential Behaviors
Detecting Body Motion

To detect body motion, we currently employ an inter-frame subtraction scheme. Here, the magnitude of motion is measured as the number of moving pixels N . Nt = j si,t = X
i
ever, in most cases a loss of visual attention is accompanied by head movement. Therefore, we still consider facial orientation information useful for attention estimation. Another means of estimating human attention is body motions, especially synchronous motions in visual/aural signals that appear in displayed content, which should represent the users level of absorption in the content. In this paper, we deal with the extraction of synchronous beat actions from music programs. Detected attention can be used for controlling audio/video contents displayed to users. For example, such positive synchronous reactions as looking at the TV and/or keeping time with ones body can be used as the basis to make a system that shows the contents for longer. In contrast, such negative behavior as a user turning his/her head away from the TV can be used as a trigger for the system to switch contents.
si,t ,
(4)
0 ((ci,t ci,t1 ) (ci,t ci,t1 ) < threshold) 1 (otherwise)
(5) Here, ci,t denotes a color vector of the i-th pixel in an observed image at time t. This method is simple but still valid for extracting temporal patterns of motion magnitude. For instance, a keeping the beat action can be detected as frozen points where the motion suddenly stops in a motion-magnitude sequence. j beat =
t1 1 ( dt < 0 and 0 (otherwise).
dN
3.1 Vision-Based Behavior Extraction Face Orientation Tracking

Vision-based face tracking is a popular research eld that many researchers have investigated [9, 10, 11]. However, few
1 0.5 0 -0.5 -1 0 50 100 [frame] 150 200
dNt dt
0)
(6)
Figure 10 shows an example of an extracted beat action. Here, the person was expressing the beat by hand.
3.2 Contents Control

Our system displays video contents to users whose behavior (facial orientation and body motions) is observed by cameras and other sensory systems, with displayed content characteristics simultaneously extracted by the system. Currently, only the beat structures of audio signals are extracted in this process. After beat structure extraction, the temporal correlation between the beat structure and user motion is evaluated. User attention to the displayed contents is then estimated from the correlation and facial orientations.
Figure 9: Example of facial orientation estimation
13
100
200 [frame]
300
400
500
Figure 12: Two states: attention on (looking at TV set) and attention o (looking at other objects)
Figure 10: Example: Extracted Beat Actions
Sound Signal 0-250Hz
250-500Hz
If the system recognizes a loss of attention to the presented medium, it switches contents to regain the users attention.
500Hz-1KHz
1-2KHz
3.3 Experiments
We conducted the following experiments to conrm the feasibility of our proposed method. All experiments described in this section were performed using healthy adults. First, we applied our facial orientation detection to a sequence of head rotation images. Figure 11 denotes the results of attention estimation. Here, the subject mainly watched our TV system and sometimes looked down at a magazine on his lap. Figure 12 shows typical sample images of the two states: attention on (looking at the TV set) and attention o (looking in other directions). The results indicate that in this case, our system accurately detected the loss of attention away from the TV system. Next we performed detection of synchronous beat behavior related to displayed musical signals. Beat structures in musical signals are extracted based on frequency analysis [12, 13]. In this work, we also formalized beat-structure detection based on a set of signal power banks consisting of several dierent frequencies. Input signals are rst fed into a frequency analyzer (FFT) and separated into discrete lter banks. In our current implementation, guard frequencies are 0-250 Hz, 250-500 Hz, 500-1 kHz, 1-2 kHz, and 2-4kHz. For each bank, the power transition is examined, an envelope signal is produced, and the beat structures are extracted as a rising point using dierential operations. Figure 13 shows both the detected beat from the displayed musical signal and the human beat motions. Here, several popular songs were used for the experiment. At the bottom of Fig. 13, we nd the extracted synchronous motion (beat action). Furthermore, Fig. 14 shows the result of synchronous/nonsynchronous behavior detection. Here on the left side, a
2-4KHz Body Motion
Extracted Synchronous Beat

1280 1300 1320 1340 [frame] 1360 1380
Figure 13: Synchronous beat behavior extraction
subject is listening to music and is instructed to keep the beat. On the right side, he moves his hand randomly, indicating that our system can clearly distinguish between the two states. Finally, we examined the eect of content control based on attention monitoring (here, only face orientation is used.) We placed two TV sets in front of a subject. One employed the content-switching mechanism based on facial orientation, whereas the other had no attention monitoring; video contents were just switched at regular intervals in a predened order. Table 3 and Fig. 15 compare the total viewing time of the two TV sets for six subjects. As we can see, subjects watched the controlled TV signicantly longer, suggesting that content switching can eectively attract user interest.
4. CONCLUSIONS
We have described methods to track human behavior us75 Number of Sync. Beats Per Minute
Estimated Attention
70 65 60 55 50 45 40 35 30 beat action random
ON
OFF
30
35
40
45
50
55
60
65
70
75
time [sec]
Figure 11: Results of Attention Estimation
Figure 14: Distinguishing Beat Action and NonBeat Action
14
Table 3: Watching time and Subject Watching Time (sec) with control without control false-negative detection error false-positive
number of missed detections a b c d e f 528 526 439 422 444 294 289 314 391 288 336 426 0 3 6 0 3 1 2 0 0 4 0 0
Avg. 442* 340* 2.2 1
*p=0.06
550 Total Viewing Time [sec] 500 450 400 350 300 250 With Control W/O Control
[5]
[6]
Figure 15: TV Watching Time with/without Content Control
[7]
ing vision-based system to monitor and assist human activities in daily life. In the action assistance, human behavior is modeled as the change of 3-D appearances. we detect the 3-D appearance by using IR pattern projections, it achieves widespread use and impose low visual impact on users, taking into account the human need for privacy. We also described a method to estimate human attention based on face orientation and body actions, employing an SVM to model human faces, and determined facial orientation by using the geometrical relation between the eyes and the tip of the nose. Body motions (beat actions) were also extracted and examined in terms of synchronization with the beat elements of displayed musical signals. In experiments, we controlled the timing of content switching to attract user attention to the presented video contents. Experimental results demonstrated the eectiveness of human tracking and behavior monitoring by the proposed methods.
[8]
[9]
[10]
[11]
[12]
5.
ACKNOWLEDGMENTS
[13]
This research was supported in part by the National Institute of Information and Communications Technology.
International Conference on Pattern Recognition, pages 6872, 1996. Jakub Segen and Sarma Pingali. A camera-based system for tracking people in real time. In Proceedings of 13th International Conference on Pattern Recognition, pages 6367, 1996. Kosuke Sato and Seiji Inokuchi. Range-imaging system utilizing nematic liquid crystal mask. In Proc. of 1st International Conference on Computer Vision, pages 657661, 1987. K. Hattori and Y. Sato. Handy rangender for active robot vision. In Proc. of Int. Conf. Robotics and Automation, pages 14231428, 1995. A. Utsumi, S. Kawato, and S. Abe. Attention monitoring based on temporal signal-behavior stuructures. In LNCS3766, Proc. of ICCV2005 Workshop on HCI, pages 100109, 2005. J. Yang, R. Stiefelhagen, U. Meier, and A. Waibel. A real-time face tracker. In Proc. 3rd IEEE Workshop on Application of Computer Vision, pages 142147, 1996. J. C. Terrillon, M. David, and S. Akamatsu. Automatic detection of human faces in natural scene image by use of skin color model and invariant moment. In Proc. of IEEE Int. Conf. on Automatic Face and Gesture Recognition, pages 112117, 1998. J. Heinzmann and A. Zelinsky. 3-d facial pose and gaze point estimation using a robust real time tracking paradigm. In Proc. of IEEE Int. Conf. on Automatic Face and Gesture Recognition, pages 142147, 1998. E. D. Scheirer. Tempo and beat analysis of acoustic musical signals. Journal of Accoustical Society of America, 103(1):588601, 1998. M. Goto. An audio-based real-time beat tracking system for music with or without drum-sounds. Journal of New Music Research, 30(2):159171, 2001.
6.
REFERENCES
[1] N. Kuwahara, K. Kuwabara, A. Utsumi, K. Yasuda, and N. Tetsutani. Networked interaction therapy: Relieving stress in memory-impaired people and their family members. In Proc. of IEEE Engineering in Medicine and Biology Society, 2004. to be appeared. [2] Akira Utsumi and Jun Ohya. Multiple-camera-based human tracking using non-synchronous observations. In Proceedings of Fourth Asian Conference on Computer Vision, pages 10341039, 2000. [3] D. M. Gavrila and L. S. Davis. 3-d model-based tracking of humans in action: a multi-view approach. In Proc. of Computer Vision and Pattern Recognition, pages 7380, 1996. [4] Q. Cai and J. K. Aggarwal. Tracking human motion using multiple cameras. In Proceedings of 13th
15
Developing an engaging multimedia activity system for people with dementia

Arlene J. Astell1 Maggie Ellis1
1
Norman Alm2 Richard Dye2

University of Dundee, Division of Applied Computing Faculty of Engineering and Physical Sciences Scotland, DD 4HN, +44 1382 345596 nalm@computing.dundee.ac.uk rdye@computing.dundee.ac.uk
2
Gary Gowans3 Phil Vaughan3

Duncan of Jordanstone College of Art Division of Graphic Design, Design School, Dundee Dundee, Scotland, DD1 4HN +44 1382 348273 g.m.gowans@dundee.ac.uk P.B.Vaughan@dundee.ac.uk
3
University of St. Andrews, School of Psychology South Street, St. Andrews Fife, Scotland, KY16 9JU +44 1334 462056 aja3@standrews.ac.uk mpe2@st-andrews.ac.uk
Abstract
Finding ways to engage people with dementia in stimulating but safe activities would be beneficial both to them and to their caregivers. We have previously demonstrated that a simple but carefully designed computer-based touch screen interface can capture and hold the attention of people with dementia who are normally difficult to engage [2]. In the previous project, people with dementia used the system together with a caregiver to provide a shared activity. We are now developing a similar hypermedia system to be used alone by a person with dementia without caregiver assistance. The system is intended to capitalise on the living in the moment state of people with dementia and make the most of their preserved skills. At the same time, the system will compensate for diminished abilities, such as short-term memory. We have piloted three navigable virtual environments a garden, a museum and a pub with both people with dementia and professionals in the field of dementia care. We report the results of this pilot work and consider the further questions to be addressed in developing an engaging, multimedia activity for people with dementia to use independently.
characterised in the early stages by memory problems and failure to learn new information. Over time, all aspects of cognitive function are affected and daily living skills erode. People with dementia gradually lose the ability to keep safe, keep busy and care for themselves. Left to their own devices, people with mild to moderate dementia may become lost and confused if they go out of the house, and at home may start activities, such as cooking or running a bath, which they fail to complete. These situations present considerable risks and reducing these places significant demands on caregivers, who are typically family members. As dementia progresses, people are commonly confused and disorientated. Many attempt to leave the unfamiliar nursing or residential settings full of apparently unknown people. Their agitation and distress present a challenge to caregivers and highlight the difficulties of keeping people occupied. Finding safe, stimulating activities that engage people at all stages of dementia would therefore be beneficial not only to them but also to their caregivers [3]. Participation in activities is important for stimulating and maintaining cognitive and social skills [8]. In addition, previous studies have shown positive effects of activity on reducing agitation and engendering a sense of well-being in people with dementia [4]. Engaging activity for people with dementia also provides caregivers with well-earned breaks from constant caring. In our previous work (The CIRCA Project; 1,2). We demonstrated that people with dementia are comfortable in front of a touch screen computer and will use the system when prompted. We have shown that a carefully designed system can tap into well-rehearsed cognitive and social scripts and use them to engage people with dementia [1,2]. In CIRCA we developed such a system to prompt and support social interactions between people with dementia and caregivers using a hypermedia structure. Hypermedia is ideal for tackling the short-term memory problem of people with dementia, as there is no penalty for losing the place [5,6]. Users are encouraged to explore and get lost in the system to make the most of the material. Thus wherever the user is within the interface is the right

J.3 [Computer Applications] : Life and medical sciences Health ; Social and Behavioural Sciences - Psychology General terms : Design, Experimentation, Human Factors -
General Terms
Design; Experimentation
Keywords
Dementia; activity; engagement.
1. Introduction
Dementia is a progressive, irreversible disorder and age is the greatest risk factor for developing it. The most common cause of dementia is Alzheimers disease, which is
16
place to be. This is excellent for people with dementia whose short-term memory is extremely impaired, and who are unable to remember where they have just been. Using multimedia technology it is possible to include text, photographs, graphics, sound recordings and film recordings. These contents can be linked in various dynamic and flexible ways to promote the users engagement. The user is invited to interact with the material presented in a more lively way than by just looking at text and pictures on a page. In addition hypermedia provides abundant opportunities for users to make decisions as they explore the system. The way that hypermedia operates fits well with the Multiple Trace Theory of how memories are stored and accessed [7]. In this theory older memories are accompanied by multiple traces within the brain, which facilitate retrieval by providing more ways to access the memories. However, people with dementia have extreme difficulty accessing memories at will. Hypermedia provides
multiple prompts from across a range of stimuli to tap into these multiple access routes and facilitate retrieval of stored memories. Hypermedia offers further possibilities for creative interventions for people with dementia to access selfgenerated activities, which are safe and enjoyable. The main challenges in developing such a system are impaired short-term memory and problems learning new skills and information. Our solution with CIRCA was to take the here-and-now state of people with dementia and design the system around this. We are repeating this approach in the new project, living in the moment. We have thus developed three virtual environments for users to navigate. We wanted to assess the usability of these for people with dementia. We also wanted to investigate ways to prompt people with dementia to touch the screen. Finally, we wanted feedback from professionals in the field of dementia care on the suitability of the contents.
Figure 1: Virtual botanic garden
17
2. Pilot study (i) - Dementia professionals Method

2.1 Participants
Respondents were recruited at the UK Alzheimers Society Annual Conference. Thirteen people completed a questionnaire after trying out the virtual environments.
experience and the possibility for giving back some control to people with dementia. These included greater variety and therefore more possibilities for everyone; it brings fun and a sense of achievement - you can make things happen; and you can control what you are looking at. One respondent additionally comment that the activity provided Confidence building opportunities. Other additional comments referred to the ease of navigation, the size of the control buttons and the ion screen instruction. Some respondents made suggestions for additional VR environments, which included: shopping; dancing; roller skating; fairground rides; animals, i.e. a zoo; and seasonal scenes.
2.2 Materials
Three navigable virtual environments were created using Macromedia Director 8.5 as the authoring software. The following additional software were used to provide the content: Adobe Photoshop 6.0; Adobe Illustrator 9.0.1; Adobe Premier 6.0; QTVR Authoring Studio 1.0; SoundEdit 16 version 2; Infini-D 4.5. The environments were run on an Apple G4 laptop and were presented through a 20-inch touch-screen monitor. SONY SRS-T77 speakers were used to output stored music. The VR environments were viewed at a resolution of 1280 x 1024 pixels. Written prompts were included to indicate places to touch the screen (see Figure 1). A brief questionnaire about the virtual environments was compiled comprising the following questions: Did you try out the virtual reality environments?; Which, if any, did you prefer and why? Do you think they add to the experience?; Why?; Any additional comments; Suggestions for other VR environments.
4. Pilot study (ii) People with dementia

4.1. Participants
Five people with dementia (2 men) were invited to look at the virtual environments in a day care setting.
4.2 Materials
The laptop and touch screen as described above (section 2.2) were taken to a Day Centre for people with a diagnosis of dementia.
4.3 Procedure
Each participant was invited to look at the VR environments with a member of the research team. In addition, each participant was encouraged to use the touch screen individually and explore the environments. When they had finished each participant was interviewed.
2.3 Procedure
Participants were invited to use the computer by themselves for as long as they liked. When they had finished each user was asked to complete the brief questionnaire about the virtual environments.
3. Results
Most respondents gave general comments about the VR environments. One respondent commented, I like to see things that are related to the person with dementias life. Another commented on the nature of the VR contents, stating that it was Good to see moving pictures. Of the thirteen respondents, eight expressed a preference for one of the three VR environments. Of these eight, five preferred the botanical gardens. The reasons for this included reference to the contents such as beautiful to look at; excellent music; and lovely scenery. Finally, one respondent commented, the range of possibilities is greater than the other environments. Eleven respondents commented on the additional value of having VR environments. Responses were addressed to three areas. First was the experience itself where respondents commented that it was very realistic and the garden more real looking and multidimensional. Provides a choice of where to go and what to look at more closely, etc.. A second group of comments referred to the engaging properties of the VR environments, such as an escapism into a relaxing, enjoyable and engaging environment. Further comments in this vein were holds the attention and stimulates memories, gives enjoyment through several senses colour, movement and sound. The third group of comments referred to the appeal of the VR
5. Results
All of the people with a diagnosis of dementia appeared to enjoy looking at the VR environments and 3 out of the 5 tried using it independently. All participants spontaneously commented about the experience both when using it and afterwards. It was apparent that the botanic garden was the most popular environment with this group, just as it had been with the dementia care professionals. The participants with dementia commented on the contents, for example lovely garden, beautiful flowers, and looks like a good pint (in the pub VR environment). One person went further in relating to the experience with the comment Ill sit here (bench in garden) for a while. Another person commented on the totality of the experience, saying the music is in time with it all (in the VR botanic garden). A final group of comments reflected the users reaction to the experience including amazing, wonderful, clever, and interesting.
6. Discussion
Evaluation of our three initial VR environments by a group of dementia care professionals and a group of people with a diagnosis of dementia was positive and provided useful feedback for the development of the Living in the Moment project. Of particular interest were the comments relating to the engaging properties of the stimuli. The dementia care professionals directly commented on the way the system
18
captured attention and provided an engaging experience. These observations were further supported by the comments of the users with a diagnosis of dementia, who were able to imagine themselves in the environments, for example sitting on a bench in the botanic garden or having a pint of beer in the pub environment. A number of useful comments were also made about the ease with which one could navigate around the environments. These included comments about the size of the arrows and on screen prompts for where to touch. We are currently carrying out detailed investigation of how best to prompt people with dementia to initiate and maintain interaction with the touch screen when they are using it alone. It is intended that the finished Living in the Moment system will provide a way for people with dementia to independently engage in activities that they can no longer participate in outside of the care environment, for example shopping and cooking. VR is one way to provide people with a diagnosis of dementia with opportunities to reexperience activities they previously enjoyed in a safe and risk-free situation. The evidence from this pilot work suggests that people with dementia respond positively to the environments and with adequate prompting can explore them independently. As such we plan to develop an extensive network of interconnected VR environments each containing activities to occupy and stimulate people with dementia. It is further hoped that Living in the Moment will be an enjoyable experience, which provides opportunities for people with dementia to exercise autonomy over their activities. This would assist families and others who care for people with mild to moderate dementia by providing safe and positive ways to occupy their relatives, creating a well-earned respite from the heavy demands of keeping people with dementia active and stimulated. In addition, it will help to address the problems of safety for people with dementia with as little infringement on their rights and autonomy as possible.
Acknowledgements
We are grateful to everyone at the Alzheimers Society Annual Conference in York (UK) and at the Alzheimer Scotland Dundee Project who took part in this pilot work. The Living in the Moment project is funded by grant GR/T09279/01 to the first author and GR/T09262/01 to the second (Alm) and third authors (Gowans) from the EPSRC.
References
[1] Alm, N., Astell, A., Ellis, M., Dye, R., Gowans, G., & Campbell, J. (2004). A cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation, 14 (1-2), 117-134. [2] Astell, A. J., Ellis, M. P., Alm, N., Dye, R., Campbell, J., & Gowans, G. (2004). Facilitating communication in dementia with multimedia technology. Brain and Language. 91(1), 80-81. [3] Gori, Pientini, & Vespa. (2001). The selection of meaningful activities as a treatment for day-care in dementia. Arch Geron Ger, supp &. 207-212. [4] Lou, A. (2001). The use of music therapy to decrease agitated behaviour of the demented elderly: the state of the science. Scandinavian Journal of Caring Sciences, 15(2), 165-173. [5] Maddy, D. Brouwer-Janse (1996) User centred design, the contour of multimedia, theoretical and empirical developments, Academia Research Monograph 19. Ch12. Luton: John Libbey Media. [6] McKerlie, D and Preece, J. (1992) The hypermedia effect: more than just the sum of its parts, in Proceedings of the St.Petersburg HCI Conference, pp 115-127. [7] Nadel, L. and Moscovitch, M. Consolidation, retrograde amnesia and the hippocampal formation. Current Opinion in Neurobiology, 7: 217-227, 1997. [8] Tzrinski & Higgins. (2001). Therapeutic play activities: building cohesion and socialization among nursing home residents. Activities, Adaptation and Ageing, 25(3-4), 121-135.
19
Networked Reminiscence Content Authoring and Delivery for Elderly People with Dementia
Noriaki Kuwahara kuwahara@atr.jp Shinji Abe sabe@atr.jp Kazuhiro Kuwabara kuwabara@atr.jp
Department of Assisted Communication ATR Intelligent Robotics and Communication Laboratories Keihanna Science City, Kyoto, +81-0774-95-1401 ABSTRACT
Providing good care at home to people with dementia is becoming an important issue as the elderly population increases. One main problem in providing such care is that it must be constantly provided without interruption, which puts a great burden on caregivers who are often family members. Networked Interaction Therapy is what we call our methods designed to relieve the stress of people suffering from dementia as well as their family members. This therapy aims to provide a system that interacts with people with dementia by utilizing various engaging stimuli. One such stimulus is reminiscence videos created from old photo albums, a promising way to capture a persons attention for a long time. In this paper, we first present reminiscence videos and an authoring tool to assist in their production. Next, we explain the Networked Reminiscence Content Authoring and Delivery service that uses our authoring tool as a key component and show how this system is used by both caregivers and dementia sufferers. dementia involved and keep their attention. Figure 1 illustrates an example of the home terminal for Networked Interaction Therapy. Currently, we plan to provide our service by using a large screen TV and a set-top box that controls network communication along with a camera and microphone for monitoring user status. One of our services provides an audio-visual stimulus because it is the most effective way to hold user attention. In addition to audio and visual stimuli, we are also considering an air cannon and a vibrator in the sofa to activate olfactory [2] and haptic senses [3].
Air Cannon for Olfactory Sense Camera and Microphone

H.4.3 [Communications Applications]: Computer conferencing, teleconferencing, and videoconferencing
Vibrator in the Sofa for Haptic Sense
TV Set for Displaying Images and Sounds
General Terms
Design
Figure 1. The illustrative Example of the Terminal Equipment of Networked Interaction Therapy In this paper, we first present a reminiscence video and an authoring tool to assist its production. Next, we explain the Networked Reminiscence Content Authoring and Delivery service, whose key component is our authoring tool, and show how this system is used by both caregivers and dementia sufferers.
Keywords
Interactive Reminiscence Contents, Content Sharing, Photo Annotation, Semantic Web, Dementia
2. REMINISCENCE VIDEO 1. INTRODUCTION

Providing good care to people with dementia is becoming an important issue, especially as the aging of society inevitably increases their numbers. Care for the elderly must be constantly provided, placing a great burden on caregivers, who are often family members. The survey we conducted with four support groups indicates that caregivers urgently need a system that holds the attention of their loved ones so that they can enjoy short breaks [1]. Networked Interaction Therapy is the name for our proposed methods to relieve the stress of both people with dementia and their family members. These methods use a system that interacts with users to hold their attention. Human caregivers can thus take a break while the user is engaged with the system. A Networked Interaction Therapy system needs to get people with Much research has presented audio-visual stimuli to people with dementia. For example, the video respite concept [4] was created to capture and maintain the attention of people with dementia while giving caregivers time to rest. Respite videos often feature a character who talks to the person with dementia in a way that attracts his or her attention. Furthermore, Project CIRCA utilized multimedia technology to stimulate the long-term memory of people with dementia [5]. People with dementia sometimes have problems with short-term memory even while retaining long-term memory. This project, in a sense, augments reminiscence therapy with multimedia technologies. From the viewpoint of stimulating the long-term memory of people with dementia, we selected a so-called reminiscence video as an audio-visual stimulus. This is a slide show video
20
produced from the users old photo albums. Its effectiveness was experimentally shown [6]. In addition, once created, it is relatively easy for caregivers to use the video.
2.1 Requirements for Reminiscence Videos

First, we have already conducted several experiments [7] that clarified the following as the most important requirements for reminiscence video to satisfy caregiver needs: 1. 2. 3. Adding narration to produce an attractive reminiscence video Zooming in on familiar faces in a photo Narration and zooming synchronization. When the narration is about a specific person in the photo, it should be dictated while that region is being zoomed in on and that corresponds to this person.
Our tool achieves these features that make it possible to automatically transform one type of content (photo) to another kind of content (slide show video more specifically, reminiscence video) by annotation data. Furthermore, as mentioned above, content distribution and sharing on the Web will become much more common and help make reminiscence videos more interesting for elderly people.
2.2 Overview of Authoring Tool

Figure 3 depicts the overall process of slide show generation. The architecture is based on the RDF database. Our photo annotation tool can annotate photos and store the annotations in an RDF format.
Use Cases Annotating Photos Querying Photos Photo Annotation Tool (J2SE 1.4.2_04) Slide-Show Generating Tool (J2SE 1.4.2_04) Refer Selecting BGM and Narrations Rendering Video Generating Movie File Reminiscence Video
Creating such a video, however, is not easy because it involves many steps: first, photo albums are collected from the users family. Next, photos are selected from the photo albums that should ideally invoke distant memories of the user. Then the photos are shot with a video camera, often using panning and zooming effects (the so-called Ken Burns effect [8]). Finally, narration is added to increase user involvement in the video. These processes are usually performed by skillful volunteers, since they are not the kinds of tasks family caregivers can do while looking after people with dementia. Moreover, for some users, showing the same reminiscence video again and again may erode the videos original effects. Thus, a simple method for making different reminiscence videos from the same set of photos is needed. To solve this problem, we developed an authoring tool that uses photo annotations [9, 10]. Annotation attached to a photo includes a theme, such as travel or school, and the time it was taken. In addition, region in the photo can be specified by annotation. Each region will represent, for example, an area of a persons face or some interesting objects in the photo. These regions will be used when panning and zooming effects are added. These are popular when making an attractive slide show video from a set of still photos. Figure 2 shows the overall reminiscence video production flow with consideration given to the abovementioned features of our system.
START
RDF Database (Jena2.1 +Postgresql 7.4)

Image Regions
Narrations
FOAF
RELATION SHIP
BGM Playlist
Dublin Core`
Photos
Rendering Templates
Annotations
Figure 3. Overview of System Architecture

at Disney Animal Kingdom dc:title 20040716 dc:date foaf:Image rdf:type dc:description They are very happy. sample.jpg imgReg: hasRegion imgReg: hasRegion w1 (x2,y2) (x1,y1) h1
rdf:type
Annotating Photos Continue to edit Reminiscence Video
imgReg: Rectangle
rdf:type imgReg: regionDepict
imgReg:coords imgReg: boundingBox x1,y1,x2,y2
Querying Photos for the Reminiscence Video and Choosing proper set of the photos
Rendering Video by adding Ken Burns Effect, BGM and Narrations with Semantic Synchronization NO Finish editing? YES Generating Movie File of the Reminiscence Video FINISH
foaf:Person rdf:type
rdf:type rel:grandchildOf foaf:gender foaf:name Haruka
x1,y1,w1,h1
Literal Class Resource Instance Resource
Selecting BGM and Narrations for the Reminiscence Video NO Preview Video? YES
Viewer
female
Figure 4. Annotation of Photo Data For photo annotation, annotations basically focus on the people in the photo because photos containing relatives and/or friends are often included in creating a reminiscence video. We employed de facto standard vocabularies as much as possible to describe annotations such as Dublin Core [11], FOAF [12], and Image Regions [13] because using these standards is a great advantage when the creator wants to use materials from the
Figure 2. Reminiscence Video Production Flow
21
Internet or to publish their own materials. We show an illustrative example of photo annotation in Figure 4, where Dublin Core is utilized to describe various properties of the photo itself. The date, a title, and a description of the photo are represented with this vocabulary. FOAF is mainly used to describe a person included in the photo. The persons name, gender, age, and contact information are represented by FOAF. Image Regions are used for storing each photos region data that correspond to the person in the photo. We also employ a vocabulary called RELATIONSHIP [14], which is a refined vocabulary of the knows attribute of FOAF. It represents the relationship between the persons in the regions of the photo specified by Image Regions. In addition to photo annotation, we annotated the narration stored in the database. Figure 5 shows an illustrative example of the annotation of narration data. First, we defined our original vocabulary for annotating narration. In Figure 5, na represents the name space of this vocabulary. The narration text of Narration1.wav is specified by na:text, and Narration1.wav includes several instances that represent the keyword specified by na:keyword. In the figure, the left instance represents the person described by the narration text and indicates an instance of foaf:Person by using na:referTo. On the other hand, the left instance indicates the persons appearance, while dc:description points to the Literal that represents the word used for generating the narration text.
na:text Your daughter is smiling. Narration1.wav na:keyword urn:atr:person:daughter daughter URI
2.
Caregivers can get feedback from dementia sufferers on the content, i.e., the date, the place, and names of each person in the photo, and so on. The terminals of dementia sufferers require no installation of special hardware or software. Remote caregivers and dementia sufferers should feel as if they are enjoying reminiscence contents and talking about them while sitting beside each other.
3. 4.
One motivation for defining these goals acknowledges the results of interviews with family caregivers who participated in our experiments. They recognized the efficiency of the reminiscence videos used in experiments to hold their dementia family members attention, and they also found that reminiscence videos greatly increased dementia sufferers willingness to talk. This encourages family caregivers to talk more with their dementia family members about memories by using their old photos. However, the number of elderly people living alone is rapidly increasing in developed countries, reducing opportunities for elderly people to communicate with others and increasing the development of various physical and mental disorders. Finally, these disorders are more likely to cause dementia. If technologies can offer elderly people living alone more chances for communication with others about their memories by using old photos, it might be possible to prevent such disorders. These technologies also provide opportunities to communicate with remote relatives and friends of dementia sufferers, greatly improving their quality of life. Another motivation was cost reduction of photo annotation for producing reminiscence videos because collecting information about every photo requires a lot of deliberate interviews with dementia suffers and their relatives. Otherwise, inappropriate photo narration destroys the warm atmosphere of a reminiscence video. If a remote caregiver can talk with the dementia sufferer and family members by sharing old photos on the Internet during such a session, the caregiver will be able to gather information on such photos to annotate them.
na:keyword urn:atr:appearance:smile
dc:description na:referTo rdf:type foaf:Person foaf:name rel:childOf Viewer Anonymous foaf:gender female
dc:description Smile
rdf:type
Literal Class Resource Instance Resource
3.2 Overview of Prototype System

To achieve our goals, first we made the photo database function of our reminiscence video authoring tool accessible from HTTP clients so that both the caregivers and the dementia sufferers terminals can retrieve photos without special software. Secondly, we developed a mechanism that makes it possible to publish and subscribe to messages between web browsers by using HTTP (Pub/Sub over HTTP) that works as a wrapper of simple web contents, such as showing photos in the photo database. The web browser on the dementia sufferers terminal is fully controlled by the web browser on the caregivers terminal to load contents for showing photos and to specify the URLs of photos to be displayed. Furthermore, we plan to use a touch panel screen for the terminal of the dementia sufferer, and the caregiver encourages the dementia sufferer to point to persons or objects of interest in the photo. Then, using the above Pub/Sub over HTTP, the dementia sufferers web browser sends a message that includes the data of the pointing position on the photo, which the caregivers terminal can display by an icon that shows this position on the photo.
Figure 5. Annotation of Narration Data
3. NETWORKED REMINISCENCE CONTENT DELIVERY

We are now developing the Networked Reminiscence Content Authoring and Delivery service by using our authoring tool as the key component. In this section, we will present this services design goals and an overview of our prototype system.
3.1 Design Goals of the Platform

When designing our service, we defined the following goals: 1. Dementia sufferers do not need to operate terminals to start communication or to share reminiscence contents with caregivers.
22
Thirdly, we use an IP TV phone that supports H.323 protocol for communication between the dementia sufferers and caregivers. Combined with this communication device and photo sharing functions, we create an atmosphere where they can enjoy reminiscence contents and talk about them under the illusion of sitting beside each other. Figure 6 shows an illustrative example of our prototype system. The dementia sufferer and caregiver are remotely sharing and talking about the same photo.
Communication by using IP TV Phone Pointing the interest Person by Finger Reminiscence Photo Database Feedback from Dementia Sufferer
The caregiver starts the reminiscence video authoring tool and connects to the photo database to retrieve photos related to the dementia sufferer with whom the caregiver is now opening communication channels. The caregiver calls the dementia sufferers terminal to start communication by using an IP TV phone, and the dementia sufferers terminal automatically responds.
3.3.2 Sharing Photos between the Dementia Sufferers and the Caregivers Terminals
The caregiver selects a photo from the photo database and drops and drags a thumbnail of it into the main panel The reminiscence video authoring tool sends a command message to the dementia sufferers web browser so that it loads the web page for photo sharing and displays the corresponding photo on the browser by using this page. The dementia sufferers web browser automatically executes the command from the caregivers terminal and displays the photo on the web browser. Figure 8 shows an example of the terminal GUI just after sharing the photo between the two terminals.
Dementia Sufferer
Internet
Remote Caregiver
The Main Panel of the Authoring Tool
Caregivers Terminal
Figure 6. Overview of Prototype System
3.3 Use Cases

We show GUI examples of our implementation along with use cases where the caregiver opens a communication channel with the dementia sufferer, shares the photo, gets feedback from the dementia sufferer, and annotates the photo.
Drag & Drop Caregivers Image
3.3.1 Opening Communication Channel between the Dementia Sufferers and the Caregivers Terminals
Caregivers Terminal
Dementia Sufferers Image Thumbnail Images of Photos in the Photo Database
Caregivers Image
Dementia Sufferers Terminal
Figure 8. Sharing a Photo
3.3.3 Getting Feedback from the Dementia Sufferers Terminal

Photo Database
The caregiver talks with the dementia sufferer through the verbal and visual communication provided by the TV phone and encourages the dementia sufferer to point to a person of interest in the photo. The dementia sufferers terminal has an eight-inch touch panel screen, and the position data on the screen where the dementia sufferer points are sent to the caregivers terminal. Then an icon showing the position on the photo where the dementia sufferer has pointed is displayed on the photo shown on the caregivers terminal. Figure 9 shows examples of receiving feedback from the dementia sufferer.
Figure 7. Opening Communication Channels The dementia sufferers terminal is set up to start the web browser and access the URL for subscribing to the message from the caregivers terminal.
23

Dementia sufferer points to a person of interest.
Pub/Sub over HTTP for exchanging command messages between the dementia sufferers and the caregivers terminal is available. Figure 11 shows the system architecture of Pub/Sub over HTTP.
Web Browser Web Content Web Content Wrapper XML RPC Asynchronous HTTP Request Reminiscence Video Authoring Tool Java Application XML RPC Asynchronous HTTP Request HTTP Protocol Subscribe Command XML RPC Publish Command
Caregiver gets feedback on a person of interest by an icon on the photo.
Notify Command
Pub / Sub Manager
Caregivers Terminal
Web Server
Figure 11. Pub/Sub over HTTP Figure 9. Getting Feedback from Dementia Sufferer
3.3.4 Photo Annotation

After getting feedback from the dementia sufferer, the caregiver annotates the photo. To collect information on the person in whom the dementia sufferers shows interest, the area corresponding to that person in the photo is zoomed up on to promote conversation, as shown in Figure 10.
Caregivers Terminal
Caregiver selects this area to be annotated Annotation Form
Annotated area is zoomed up automatically to promote conversation.
Figure 10. Photo Annotation
3.4 Implementation
To implement our prototype system, we chose NTT Flets Phone [15], a commercial NTT product that works as an IP TV phone by H.323 protocol, as the dementia sufferers terminal because its eight-inch touch panel screen design is as easy to use as home electric appliances. Also, it supports Internet Explore 6.0, and our
The dementia sufferers terminal has been implemented as Web Content that displays the photo specified by URL and can pan and zoom up in the rectangular region on the photo given as X, Y coordinate. A Web Content Wrapper can invoke these Web Content functions according to the command notified from the Reminiscence Video Authoring Tool. But first, the Web Content Wrapper needs to request the Pub/Sub Manager to subscribe to commands from the authoring tool. The Pub/Sub Manager, implemented as a servlet in the Web Server, stops the request process until the authoring tool publishes the command. The authoring tool publishes the command to the Pub/Sub Manager, which restarts the subscribe request process from the Web Content Wrapper; consequently, the command is notified. Every command is represented by an XML format, and we use XML RPC for encoding and decoding commands. After receiving the command, the Web Content Wrapper immediately sends a subscribe request and simultaneously invokes a Web Content function corresponding to the command. Web Content Wrapper uses an Asynchronous HTTP Request to send the subscribe request, and then Web Content can continue to perform such functions as photo display, panning to a specific area of the photo, and zooming without waiting for responses from the Web Server. This Pub/Sub mechanism is symmetric between the Web Browser and the Reminiscence Video Authoring Tool. To obtain the point of interest position of the dementia sufferer in the photo from the Web Browser, the authoring tool sends the request for subscribing commands from the Web Browser, which publishes the command that includes an X, Y coordinate of the pointing position of the dementia sufferer in the photo when the dementia sufferer pushes the terminals touch panel screen. By using this Pub/Sub mechanism, we achieved our three design goals in section 4.1. The dementia sufferers terminal does not need special software to be remotely operated from the caregivers terminal. There is no need to set up a broadband router for IP port forwarding. The dementia sufferer and his/her family members simply prepare an IP TV phone and access our web contents for reminiscence photo sharing from the web
24
browser. Then remote caregivers can fully control his/her terminal and talk about the reminiscence photos.
4. CONCLUSION AND FUTURE WORK

In this paper, we presented our Reminiscence Video Authoring Tool and Networked Reminiscence Content Authoring and Delivery service that uses an authoring tool as a key component. We defined our design goals for developing the Networked Reminiscence Content Authoring and Delivery service and showed that we achieved our three design goals from section 4.1 by using Pub/Sub over an HTTP mechanism. Currently, our prototype system can only handle reminiscence photos. However, our content sharing mechanism is quite versatile, and we are enhancing it to handle reminiscence videos, music, and other types of interactive reminiscence content, such as CIRCA [5]. Now we are conducting preliminary experiments on the possibility of communication between caregivers and dementia sufferers by TV phones and the effectiveness of reminiscence contents during such communication. From these experiments, we will evaluate whether our system satisfies our design goals. Also, we are planning large scale experiments with the cooperation of a senior citizens care facility where terminals for dementia sufferers will be set up in each private room and remote caregivers will take care of dementia sufferers by TV phone. Through this experiment, we will examine whether this type of service satisfies all participants, including the dementia sufferers, their family members, caregivers in the facility, and remote caregivers.
[2] Yanagida, Y., Kawato, S., Noma, H., Tetsutani, N., and Tomono, A.: A Nose-Tracked Personal Olfactory Display, SIGGRAPH 2003 Sketches & Applications (2003) [3] Kakita, M., Yanagida, Y., and Tetsutani, N.: Presentation of A Character Using Distributed Vibro-Tactile Display, IEICE Society Conference A-16-23 (2003) 318 (in Japanese). [4] Lund, D.A., et al.: Video Respite: an innovative resource for families, professional caregivers, and persons with dementia, The Gerontologist, Vol. 35, Issue 5 (1995) 683-687. [5] Gowans, G., Campbell, J., Alm, N., Dye, R., Astell, A., and Ellis, M.: Designing a multimedia conversation aid for reminiscence therapy in dementia care environments, Extended abstracts of the 2004 conference on Human factors and computing systems (2004) 825 - 836. [6] Yasuda, K., et al.: Reminiscence Video for Higher Brain Dysfunctions, Proceedings of General Conference of Japan Society for Higher Brain Dysfunctions, (2004) (in Japanese). [7] Kuwahara, N., Kuwabara, K., Tetsutani, N., and Yasuda, K.: Reminiscence Video - Helping At-Home Caregivers of People with Dementia, Home-Oriented Informatics and Telematics, Proceedings of the IFIP WG 9.3 HOIT 2005 Conference (2005) 145-154. [8] http://en.wikipedia.org/wiki/Ken_Burns [9] Kuwahara, N., Kuwabara, K., Tetsutani, N., and Yasuda, K.: Using Photo Annotations to Produce a Reminiscence Video for Dementia Patients, 3rd International Semantic Web Conference (ISWC2004) Demo Papers (2004). [10] Kuwahara, N., Kuwabara, K., Shinji Abe, Yasuda, K., and Tetsutani, N.: Semantic Synchronization: Producing Effective Reminiscence Videos, 4th International Semantic Web Conference (ISWC2005) Demo Papers (2005). [11] http://dublincore.org/ [12] http://www.foaf-project.org/ [13] http://www.bnowack.de/w3photo/pages/image_vocabs [14] http://vocab.org/relationship/ [15] http://www.fletsphone.com/index_f.html
5. ACKNOWLEDGMENTS
We would like to thank the participants of the reminiscence video experiments. We also would like to thank Koji Saito for the design and implementation of the authoring tool. This research was supported by the National Institute of Information and Communications Technology.
6. REFERENCES
[1] Kuwahara, N., et al.: Networked Interaction Therapy: Relieving Stress in Memory-Impaired People and Their Family Members, Proceedings of the 26th Annual International Conference IEEE EMBS (2004).
25
An easy-to-use Interface for Evidential Reasoner

QingXiang Wu
School of Computer Science Queens University Belfast Belfast, BT7 1NN, N.I. UK Tel:(44)2890974783
David Bell
Rashid Hafeez Khokhar

Q.wu@qub.ac.uk
Da.bell@qub.ac.uk
R.khokhar@qub.ac.uk
ABSTRACT
In this paper, we propose the system for cognitive prostheses by using AI technology through a simplified interface. Evidence theory is often used as an uncertainty handling mechanism when applied to complex reasoning systems. On the other hand, this theory can be applied to compensate human brains limitations when making decisions in daily life, business, or political arguments. Here the very simple interface is designed for the reasoner based on evidence theory so that it can be simply used to help human to make decisions in different situations with only a few keyboard clicks. A couple of examples for the applications are demonstrated in this paper.
software systems for such applications. The purpose of this paper is to study the simplified interface for applications of evidence theory and provide a simple reasoner for such applications. In Section 2, Demspters evidence combination rule is reviewed. In Section 3, a basic reasoner is designed with simple interfaces. A user guide of the reasoner is provided to demonstrate the operations in Section 4. In Section 5 some examples are presented, and finally conclusions are given in Section 6.
2. THE DEMSPTERS RULES

In order to handle uncertain and imprecise information, Bayesian belief networks, Rough set theory, evidence theory and fuzzy logic are among the techniques that have been proposed in the artificial intelligence domain. Evidence theory which can be regarded as uncertain reasoning using beliefs. Based on a mass function and uncertain rules, evidence theory (Dempster-Shafer, 1976; Barnett, 1991; Bell, 1993; Fixsen et al, 1997) has applied to solve a variety of problems in uncertain reasoning. Let represent a set of hypotheses in a hypothesis space. A function m: 2 [0,1] is defined as a mass function if it satisfies M1. m ( ) = 0 M2.
Keywords
Cognitive prostheses, evidential reseaner, decision making, uncertainty reasoning.
1. INTRODUCTION
There are limitations (Wang, 2004) of Bayesianism and shortcomings of other techniques for handling vagueness (Shi and Bell, 1996). An evidence-based system is proposed for carry out uncertain reasoning. The evidence theory can be embedded in complex systems, and it can be applied to daily life, business, or political arguments. In other words, evidential reasoning can be used to make decisions for cognitive prostheses [3] where AI systems are regarded as a system [4,5] that interacts with humans and extends humans capability, rather than as an isolated problem solving system. For example, a businessman has to make dozens of important decisions every day, politician need to make decisions to approval for huge projects according to experts opinions. Evidence theory is a good way to support such decisionmaking. The research results show that the theory has very high potential applications in widely varying decision-making domains (Bell, 1993; Fixsen et al, 1997). However, there are few simple
X m( X ) = 1
Here represents an empty set. A function bel: 2 [0,1] is defined as a belief function if it satisfies B1. bel ( ) = 0 . B2. bel () = 1 . , B3. For any collection A1, A2, , An(n 1) of subsets of
bel ( A1 A1 ... An )
I {1,..., n}
(1)| I |+1bel (iI Ai ).
Based on these functions, a set of evidential reasoning operations is defined. The fundamental operation of evidential reasoning is the orthogonal sum of evidential function, which is known as Dempsters rule for combining evidence. Bayesian method take account of an event in two situations, true or false. Evidence theory takes account of a piece of evidence in three situations, true, false and unknown. Therefore evidence theory is more powerful.
26
For example, a shop manager has done a series of survey for product A to see if the product satisfied customers. The answer from one area C1 is 50% satisfied, 30% unsatisfied, 20% not sure. Another answer from area C2 is 40% satisfied, 50% unsatisfied, 10% not sure. In this case, hypothesis space can be regarded as = {S ,U } , S for satisfied and U for unsatisfied. Two sets of opinions are regarded as two pieces of evidence C1 and C2 respectively. It is very often required to combine the two sets of opinions. According to evidence theory, the combination of two pieces of evidence can be represented in Table 1. Table 1. Combination of two customer surveys
interface. The user can collect evidence, conduct reasoning and attain the results. Knowledge acquisition interface
Core of the reasoner
C1C2
C2{S}0.4 C2{U}0.5 C2{S,U}0.1
C1{S} 0.5 0.4x0.5=0.2 0.1x0.5=0.05
C1{U}0.3 0.5x0.3 =0.15 0.1x0.3=0.03
C1{S,U}0.2 0.4x0.2=0.08 0.5x0.2=0.10 0.1x0.2=0.02 Two stages are indicated in use of the reasoner. At first stage, knowledge or experts opinions must be entered into the reasoner. Then it is said that the reasoner has knowledge. At second stage, the reasoner can be used to make decisions. Reasoning operation interface Figure 1. Architecture of evidence reasoner
In the combination, we have applied the Dempster rule as follows.

N=
X Y
m1 ( X ) m2 (Y ) = 1
X Y =0 m1( X )m2 (Y )
=0.2+0.08+0.15+0.10+0.05+0.03+0.02 =0.63
3.1 Knowledge acquisition interface

The reasoner is designed as one executable file in Microsoft windows system. The reasoner can be started with a double click the file name. The Evidence tab can be seen on the menu bar. Click Evidence and Knowledge items. A knowledge input window is opened. The interface is shown in figure 2
(C1 C2 )({h}) = (1/ N )
X Y ={h} m1( X )m2 (Y )
(C1 C2 )({S }) = (1/ 0.53)(0.2 + 0.05 + 0.08) = 0.524 (C1 C2 )({U }) = (1/ 0.53)(0.15 + 0.10 + 0.03) = 0.444
(C1 C2 )({S ,U }) = (1/ 0.53)(0.2) = 0.032
The combination result is 52.4% satisfied, 44.4% unsatisfied, and 3.2% unknown. For a few of such surveys, the calculation can be done manually. For a large number of customers, a large matrix needs to be processed. Therefore, a light reasoner with simple interfaces is designed for different spot applications in daily life, business, and political arguments. In order to avoid processing a dynamic matrix, we have proposed an incremental combination algorithm that is implemented in the reasoner. Figure 2. Knowledge acquisition interface
3. EVIDENTIAL REASONER
The architecture of the reasoner is shown in Figure 1. In order to use the reasoner easily, it is constructed with three modules: knowledge acquisition module, reasoning module and the core of reasoner. Two simple interfaces are designed to interact with the users. We can input arguments or experts opinions through the knowledge acquisition interface. After the reasoner has acquired the arguments or experts opinions, a user can go to the reasoning Experts opinions can be easily inputted to the reasoner through the interface in Figure 1. Each piece of evidence can support multiple hypotheses, and each hypothesis can be supported by multiple pieces of evidence. Each pair of evidence and hypothesis forms an uncertain argument. The argument can be added to the reasoner and display in the argument list box. A list of buttons beneath the argument list box can be used to edit the arguments. For example, a user can select an argument in the list box and then click the EditSlect button. The argument goes to argument
27
input box. The user can edit the argument and click EndEdit button to confirm the edited argument. If the user clicks the button AddasNew, the edited argument will be regarded as a new argument and add in the argument list. The user can delete an argument or all the arguments in the argument list.
Table 2. Opinion represented by number Earning Director 1 Director 2 Director 3 Director 4 Director 5 Director 6 Director 7 Director 8 Director 9 Director 10 0.9 0.6 0.2 0.5 0.2 0.7 0 0 0.4 0.4 Losing 0 0.2 0.7 0.4 0.5 0.1 0.8 0.7 0.6 0.3 Unknown 0.1 0.2 0.1 0.1 0.3 0.2 0.2 0.3 0.1 0.3
3.2 Reasoning Interface

Reasoning interface is designed to make decisions based on the knowledge input through the knowledge acquisition interfaces. Two list boxes at top show the evidence names and hypothesis names that refer to the input knowledge. Clicking the display button, all the names can be listed. Double clicking on the evidence or hypothesis, the evidence or hypothesis will be selected to the list box at middle row. Click the reasoning button; the final results from the selected evidence to the selected hypothesis can be obtained. See the example in Figure 3.
From Table 2, we can see that the opinions are very different. Directorate 1 said 90% earning, no losing, only 10% not sure. However, Director 7 said that no earning money, 80% losing, and 20% not sure. Using humans brain, it is not easy to judge whether the project will earn money or lose money. If these data are entered into the reasoner, the answer is obtained by several clicks. The results are shown in Figure 4. The result is 46% for earning money, and 54% for losing.
Figure 3. Reasoning interface Based on this interface, a user can calculate different evidence combinations or different hypothesis. If the reasoner has knowledge, reasoner always can give a result despite how many evidences are obtained. Figure 4. Result obtained from the reasoner
4. APPLICATIONS OF THE REASONER 4.1 Decision in a directorate committee

Traditionally, the number of directorates is an odd number so that the rule the minority obeys the majority can be applied to some argument issues. If each directorate can represent his/her opinion with numbers, their opinions can be easily combined with the reasoner and the result will show supports or against despite there are odd number of directorates or not. For example, there are 10 directors in a committee. They have very different opinions for an investment Project B. Their opinions are represented by number shown in Table 2. There is a chance that someone suggestion may not be significant, and his/her opinion can be taken away and get new results cpompanies. For example, click Directorate 2 in the left list box in the middle row; then click remove button. Directorate 2s opinion is taken away. Click the reasoning button. The new result are shown in Figure 5. The result is 30% earning, and 70% losing.
28
Figure 6. Results for house evaluation
5. CONCLUSION
In this paper, we proposed to design simple human-machine interface to apply an AI technology for cognitive prostheses. A simplified, easy to use, evidential reasoner has been designed in a straightforward way and can be applied to make decisions for daily life and business. In fact, the reasoner can be applied to different situations to help humans decision making. A very important issue is how to make the meaningful input data. Then result can be explained using the same set of inteface. The examples in Section 5 show how to assign mean up to the percentage numbers and how to get the percentage numbers so that the results have very clear mean. This paper only focuses on evidence combination. The other operations from evidence theory are easy to integrate into the reasoner.
Figure 5. New result after taking away Directorate 2s opinion
4.2 Decision in daily life

Suppose that you are going to buy a house at location C. You can ask your friends to evaluate whether your decision is good or not by using evidential reasoner. You can say to your friend, Please evaluate the house at location C with percentages: which percentage for advantages, which percentage for disadvantages, and which percentage not sure. Then you will get a list of numbers from your friends. Suppose the numbers are shown in Table 3. Inputting the data into our reasoner, the combination results can again be obtained by several clicks. The final results are shown in Figure 6. The results are 90% for advantages and 10% for disadvantages. Table 3. Evaluation of a house Advantages Friend 1 Friend 2 Friend 3 Friend 4 0.8 0.7 0.3 0.4 Disadvantage s 0.1 0.2 0.5 0.4 Unknown 0.1 0.1 0.2 0.2
6. ACKNOWLEDGMENTS
The authors would like to thank Northern Ireland Investment Ltd. to fund this project.
7. REFERENCES
[1] Pei Wang, The Limitation of Bayesianism, Artificial Intelligence, 158,1( 2004), 97-106. [2] Shengli Shi and D.A.Bell, An approach to system control based on combination-update of belief, Journal of Control and Cybernetics, 25, 2(1996). [3] Ford, Kenneth M. and Patrick J. Hayes. Cognitive Prostheses. Invited talk FLAIRS-99, Orlando,1999. [4] P. Gregor and A. F. Newell, "Information technology for rehabilitation and as cognitive prostheses", Journal of the International Neuropsychological Society, 6, 4 (2000)419420. [5] Eric Neufeld, David Callele, David Mould, Sonje Kristtorn, Raymond Rabu, A Contribution to the Theory and Practice of Cognitive Prostheses, LNCS, Springer-Verlag GmbH, Volume 2733 (2003) 241 250. [6] Bell, D.A., From Data Properties to Evidence, IEEE Transactions on Knowledge and Data Engineering, 5, 6 (December 1993) 965-969. [7] Fixsen, D., Mahler, R.P.S., The Modified Dempster-Shafer Approach to Classification, IEEE SMC-A, 27, 1( January 1997) 96-104. [8] Shafer, Glenn, A Mathematical Theory of Evidence. Princeton University Press, 1976.
29
A cognitive prosthesis to support communication by people with dementia

Norman Alm
University of Dundee Division of Applied Computing University of Dundee, Dundee, UK Tel: +44 (0) 1382 385596
Richard Dye
Arlene Astell
nalm@computing.dundee.ac.uk
University of Dundee University of St Andrews Division of Applied Computing School of Psychology University of Dundee, Dundee, UK University of St Andrews, St Andrews, Tell: +44 (0) 1382 384150 UK Tel: +44 (0) 1334 462056 rdye@computing.dundee.ac.uk
aja3@st-andrewsac.uk
Maggie Ellis
Gary Gowans
University of Dundee University of St Andrews Department of Computer-Aided Design School of Psychology University of St Andrews, St Andrews, University of Dundee, Dundee, UK Tel: +44 (0) 1382 345571 UK Tel: +44 (0) 1382 385119 g.m.gowans@dundee.ac.uk
Jim Campbell
University of Dundee Department of Computer-Aided Design
University of Dundee, Dundee, UK Tel: +44 (0) 1382 385471
mellis@computing.dundee.ac.uk
ABSTRACT
Given the devastating impact of dementia, developing cognitive prostheses for elderly people with this condition offers the possibility of making significant improvements in their lives. We report on a project that has developed a hypermedia based communication support system for older people with dementia, based on reminiscence material. The system has been designed to be usable by people with short-term memory deficits. The system has proved remarkably effective in supporting the communication of people with dementia.
1.
DEMENTIA AND WORKING (SHORT-TERM) MEMORY
Dementia consists of a group of symptoms involving progressive impairment of cognitive abilities. The two major causes of dementia are Alzheimer's disease, which is a progressive loss of nerve cells, and vascular dementia, which is loss of brain function due to a series of small strokes. The gradual loss of working (short term) memory is often the first symptom of the disease to be noticed. Caring for people with dementia will be a major challenge in all countries of the world in the 21st century. The number of people with dementia is projected to increase due to demographic changes which are taking place worldwide, with the number of older people proportional to the younger population growing steeply. The percentage of the population with dementia increases from the age of 65 to reach 25% of people over 85 [1]. The loss of working memory in elderly people as a result of dementia is a very serious problem for the person and for their family and carers. Many social activities and interactions become increasingly difficult as the condition progresses, since they depend on short term memory for effective participation. As a result people with dementia can become socially isolated and deprived of the range and variety of social interactions that characterise everyday life for unimpaired people. This can have a profound effect on their mental wellbeing, and put severe strains on family and carers.

J.3 [Computer Applications] : Life and medical sciences - Health ; Social and Behavioural Sciences - Psychology General terms : Design, Experimentation, Human Factors
General Terms
Design, Experimentation, Human Factors
Keywords
Dementia, Alzheimer's disease, working memory, short-term memory, multimedia, hypermedia
30
The potential of using computers as cognitive prostheses would seem to be well worth exploring. If a way could be found for a person with dementia to be able to interact with a helpful computer system, this may have a significant impact on the quality of their lives, and that of their carers. Systems might then be developed to assist the person with dementia in carrying out conversations with others, and to provide them with stimulation, entertainment, and other types of computer-based support. None of these interventions would be expected to offer an improvement in the dementia itself. But they would enable the person to maximise their positive skills, empower them, and restore to a degree their participation in social life. As a starting point in creating a range of systems which are usable by people with short-term memory deficits, we have developed a hypermedia conversation support system based on reminiscence that can be used by a person with dementia along with a carer. Our plans are to follow this work by devising ways in which the person with dementia could themselves take direct control of navigating through such a system.
relatives and carers. Also the process of conducting these sessions entails a great deal of effort. Reminiscence sessions can be pleasurable and empowering for the person with dementia, but the experience for the carer is far from a relaxed natural interaction. One hypothesis of this research is that suitably presented hypermedia can be helpful in supporting people with dementia in taking part in conversations. An obvious advantage of a computer based system over using traditional materials is the bringing together of the various media into one easily accessible system. But we also wanted to explore ways in which the hypermedia could in some way mimic the way memory is used in conversations. It is inherently plausible that the associative links in memory [9,10] and in free-flowing conversation [11] could be roughly modelled with hypermedia links. We have developed a hypermedia-based system that takes this approach.
3. A SUPPORT FOR COMMUNICATION
2. THE DIFFICULTY OF COMMUNICATION WITH DEMENTIA

Everyday conversation is fundamental to human social interaction. Through conversation we learn to talk, we are socialised, we make friends and form lifelong partnerships, and we establish ourselves in the social hierarchy. We in fact create our sense of identity and sense of self worth primarily through our conversational interactions with others [2]. Holding a conversation is very difficult without a correctly functioning working memory. People with dementia will often repeat what they have only just recently said, and can give the appearance of not listening to what others are saying. This can be disempowering for the person and distressing for relatives and friends. Ways of structuring conversations have been investigated, for instance with the use of specially prepared memory books that contain autobiographical material, daily schedule information, and problem resolution information. When such material is individually prepared, and when staff are trained in using it, communication improves between the carers and people with dementia [3]. Reminiscence sessions are also a useful way to structure interaction so as to maximise the positive contribution that can be made by a person with dementia. Although their working memory is impaired, the long term memory of a person with dementia is often still functioning even at the latter stages of the disease. By guiding and supporting the person to take advantage of long term memory they can be helped to take a more active part in conversations [4,5,6,7,8]. While very valuable, reminiscence activities involve prior planning and gathering of material, which is time consuming for busy
BASED ON HYPERMEDIA AND REMINISCENCE MATERIAL

It is unrealistic to expect people with dementia to acquire new skills, as their working memory problems normally preclude this. For the current generation of older people, therefore, using a mouse to interact with a system ( or a keyboard, if the person has not been a typist) would not be suitable. A touchscreen, however, has proved to be a suitable input device for an application such as this. It is intuitive to use and does not require fine muscle control to operate. Previous work, by ourselves and others, has established that people with dementia can usually manage to make use of touchscreens. The direct sense of manipulation they provide seems to offer enough affordance that, with encouragement, they work very well even with people who have fairly severe dementia [12,13]. Because of potential vision problems with elderly people, and since the display was to be shared during a reminiscence session it was decided to use the largest available (21) flat panel touchscreen. This size of screen was also chosen to create a more engaging experience than that offered by standard TV screens, but still staying on the table top, with a screen that suggested active use, and not just passive viewing of a large public screen on the wall. Sound output was provided using a standard computer sound card together with speakers. An iterative design approach was used to develop the first prototype. Forty people with dementia were identified who expressed an interest in helping us to design and evaluate the system. Thirty relatives and carers also agreed to take part as advisors and evaluators for the project. As a first step, we asked the group to comment on suitable content for the system. Ideas for themes the system might include (such as national events, local industries, street life, celebrations) were suggested by means of high quality photographs. Using the photographs as prompts, discussions were held with people with dementia and
31
their carers to help decide on the best choices for content for the system. The system was developed using Macromedia Director. Director was chosen because it is a cross-platform development package that allows rapid development of complex multimedia systems. It also has its own programming language which allows it to be interactive and to be connected to a database. The completed system was given the name CIRCA, standing for Computer Interactive Reminiscence and Conversation Aid.
3.1 Description of the interface

Consultation with experts on dementia had indicated that the interface would need to be as simple as possible while still appearing attractive and encouraging interaction. One problem that people with dementia have is an inability to cope with too
many items that compel attention. In this state the user often will focus on one item and stay with it, not being able to scan easily over the other possibilities. To cope with this, we designed the interface so that the background and navigation features are in muted colours. In most cases only one item of interest is shown at a time. In this way, even a black and white photograph stands out clearly as the point at which the user's attention should be directed. In order to keep the navigation process as simple as possible It was decided to have just three themes to choose from, and for each theme to have there media types that could be chosen. Each theme is associated with a different colour scheme. When a theme is selected the background colour and colour of all the buttons are changed to reflect the selected theme. When the carer or the person with dementia selects one of the three themes, they are then offered a choice of photographs, video or music.
Figure 1.
Screen shot of the Photographs section in the Dundee Life theme. A title and brief description are given as a help to the carer, who might not recognise what is shown. The menu bars of the touchscreen are at the bottom for easier reaching for older users. Visible are the three themes, the three media choices, and simple navigation buttons.
32
3.2 Media content of the system

Based on feedback from our user population, the three themes chosen for the prototype were; Recreation, Entertainment and Dundee Life. The Recreation theme focused on UK material with an emphasis on Dundee and Scotland. The Entertainment theme contained both US (movies and music) and UK material (TV and music). The Dundee Life theme focused on local material. In total the system contained 80 photographs, 10 videos, and 23 music items. Photographs are often used in reminiscence sessions as they are relatively easy to source. There are many photograph archives and library available, covering many themes and time periods. It was therefore relatively easy to add a large number of photographs to the prototype. As it is usual to view photographs in short sequences, the photographs within
each theme are grouped into categories. Two navigating buttons, Next picture and Previous picture are provided for stepping through the photograph sequences (see Figure 1). The system allows the users to select and play video clips related to the theme selected. People with dementia may not be able to follow a long video clip because of their working memory problems. The video clips have a short duration for this reason and also because they are intended to act as conversation prompts and not be too immersive. The system allows the users to select and play music related to the theme selected. Rather than simply having a static screen while the music plays, a display is presented of the type of device that particular music would normally be coming from. A record player, radio or tape recorder is displayed depending on the theme selected, as shown in Figure 2. This representation of the music producing device can also act as a conversation prompt in itself.
Figure 2. Music is accessed through representations of the type of player that might have produced it in times past. The record player and the reel-to-reel tape recorder are animated to give the impression of working while the song is played.
33
4.
EVALUATING
THE
PROTOTYPE
Range MMSE Mean Standard Deviation Range
71-89 16.00 6.71 5-23
An evaluation was performed to determine if people with dementia were able to make sense of the system, and understand its use well enough to direct a carer in operating it, or to operate it themselves. In the evaluation sessions, the system was compared with traditional reminiscence sessions. In a traditional session, the carer takes the responsibility for guiding the session, and at all points compensating for the short-term memory problems of the person with dementia. We designed the CIRCA system so that it hopefully would be able to take over this role of supporting the cognitive abilities of the person with dementia, freeing the carer to take part in the session more naturally. Eighteen people with dementia and their carers took part. Each person with dementia and their carer undertook a 20 minute reminiscence session: nine using the prototype, and nine using traditional reminiscence methods. As expected, the carers did most of the direct operation of the touch screen, but they were often prompted and directed by the person with dementia. With encouragement a number of people with dementia also made direct use of the touch screen to make selections. The carers were asked to complete a questionnaire at the end of each session. All the sessions were video recorded, and all the items accessed using the prototype were logged. A protocol was developed for observing and recording behaviour during the sessions, to identify the degree of participation, engagement and enjoyment of the experience. Eighteen people with dementia took part, 13 women and 5 men. Nine participated in CIRCA sessions and nine participated in traditional reminiscence sessions). The age range was 65-95 yrs. The Mini Mental State Examination (MMSE) is a rough measure of degree of severity of dementia ranging down from 30 (no measured cognitive impairment) to 0. The MMSE range was from 2 to 23. Profile of participants : Carers (N = 12) CIRCA sessions Age Mean Standard Deviation Range MMSE Mean Standard Deviation Range Traditional sessions Age Mean Standard Deviation 82.00 6.17 14.88 6.14 2-22 81.88 7.93 65-95
A between subjects protocol was used (Mann Whitney test). All the people with dementia were able to make sense of the material the system was presenting, and showed an understanding of how it worked. Some spontaneously commented several times that they were enjoying using CIRCA. One person said, "It takes you back and refreshes your memory". Other spontaneous comments were : "This covers everything", "Good thing, this", "It's good to remember things", and "That's entertainment!'" One person said she certainly would like to use CIRCA again. She said she thoroughly enjoyed it and found it very interesting and "something new". Another person commented that she enjoyed using the system herself. In comparing CIRCA with using traditional reminiscence aids, we found that the person with dementia was offered a choice of reminiscence subject/materials more often when using CIRCA (U = 1.50, p<0.001). We also found that the person with dementia chose reminiscence subjects/materials more often when prompted when using CIRCA (U = 3.60, p<0.001). The traditional sessions were characteristically a series of one question from the carer followed by one response from the person with dementia. The CIRCA sessions were more of a conversation, with each person contributing an equal amount, and control of the direction of the conversation being shared. The carers asked more direct questions when using traditional reminiscence methods (U = 5.00, p=0.01).
5.XXCONCLUSIONS
CIRCA proved to be usable by people with dementia. Their problems with short-term memory did not stop them from understanding the material presented to them, and working out how to navigate to other material, either by directing the carer or using the touchscreen themselves. CIRCA facilitated a more equal participation in communication for the person with dementia, as compared with traditional reminiscence. In offering a richness of choice, the carer could hand over more of the control of the session to the person with dementia. By choosing reminiscence items, the person with dementia guided the session. In this way, people with dementia could set the agenda and pace of the session. This was enhanced by the fact that people at earlier stages of dementia could use the touchscreen themselves. By providing an engaging focus of joint attention, CIRCA allowed the carer and the person with dementia to explore a wide range of topics provided in various media. This created subjects of conversation for both parties which could be changed easily and quickly. Moreover, periods of silence which
34
might create awkwardness in traditional reminiscence sessions were filled continuously by CIRCA. What was supplied was a third element in the interaction that at the same time relieved the carer of the sole responsibility for directing the session and provided stimulating possibilities for conversation. The person with dementia could use CIRCA in different ways. They could tell a story or simply enjoy the experience of exploring the presentations with another person. By filling in the gaps in a typical conversation, CIRCA encouraged a failure-free social interaction. CIRCA encouraged the person with dementia to take a more active part in creating the reminiscence session, and carers as a result asked fewer direct questions in CIRCA sessions, compared with traditional reminiscence work. Several carers mentioned that they felt less pressure to maintain the conversation when using CIRCA. Moreover, in contrast with traditional reminiscence sessions, time spent using CIRCA did not need to be planned by the carer. Thus, the experience of reminiscence could be more relaxed for both parties. This project has been a multidisciplinary effort, with participants from software engineering, psychology and design. We feel this is an essential aspect of this work. One way this is evidenced is in the attractive and interesting look that the system has, which has been produced by an experienced team of designers. The look and feel of the system is polished, professional and inviting. We are convinced that the look of the system invites engagement in a way that a worthy but less aesthetically pleasing interface might not. This relates to what some designers have called the 'aesthetic-usability effect'. Aesthetic designs are perceived as easier to use than lessaesthetic designs [14,15,16]. And since the aim of this system is to engaged the continued attention and involvement of people for whom this has proved difficult, this is not just a matter of a desirable extra feature.
each time the system is used. This was felt to be particularly important given the fact that the equal control over the interaction was dependent on both conversation partners sharing a unique experience, so that the carer did not need to feel in charge of the conversation. Plans are also underway to extend what we have learned about interfaces that are usable by people with dementia to develop an interactive entertainment system which they can use on their own, without a carer to assist them. Whereas the CIRCA system was developed as a conversation support, and not as a provider of entertainment, we would like to develop a range of systems that can engage and hold the involvement of people with short-term memory deficits in a way which is stimulating and entertaining. CIRCA has shown that technology can help restore a degree of lost social functioning. What we would now like to do is to develop ways of helping people with dementia to enjoy themselves without the need for the constant presence of a carer or relative. This could be significant help, since one of the principle pressures on families and care faculties is the inability of people with dementia to occupy themselves in satisfying activities without the continual involvement and guidance of another person.
7.XXACKNOWLEDGEMENTS
We would like to acknowledge the essential participation in this work of the people with dementia, their relatives, and carers. Alzheimer Scotland and Dundee City Council Social Work Department are active and supportive partners in the project.
REFERENCES
[1] Jorm, A.F., Korten, A.E., Henderson, A.S. (1987) The prevalence of dementia : a quantitative integration of the literature. Acta Psychiatrica Scandinavica. 76, pp 465-479. [2] Beattie, G. (1983) Talk: An Analysis of Speech and Nonverbal Communication. Milton Keynes : Open University Press. [3] Bourgeois, M., Dijkstra, K., Burgio, L., Allen-Burge, R. (2001) Memory aids as an augmentative and alternative communication strategy for nursing home residents with dementia. Augmentative and Alternative Communication. Vol 17, pp 196-210. [4] Baines, S., Saxby, P., Ehlert, K. (1987) Reality orientation and reminiscence therapy: a controlled cross over study of elderly confused people. British Journal of Psychiatry, 151, pp 222-231. [5] Feil, Naomi (1993) The Validation Breakthrough. Maryland: Health Professions Press. [6] Finnema, E., Dres, R-M, Ribbe, M., Van Tilburg, W. (1999) The effects of emotion-oriented approaches in the care for persons suffering from dementia: a review of the literature. International Journal of Geriatric Psychiatry, Vol 15, pp 141-161.
6. FURTHER WORK
Following the lessons learned from the evaluation of the system, improvements and additions are being made to the system as follows : (1) The system will be made modular with a separation of form and content, to allow the creation of new material easily. This will allow for localisation of the material, and also for the potential to include personal material, where this is suitable. (2) The system will be extended to allow for a larger number of themes and individual items to be available, with a random choice of three of these being made by the system for each session, in order to keep the continued interest of relatives and carers. Similarly, the number of potential media items will be increased, with a random selection being made among these
35
[7] Woods, R. T. (1994) Management of memory impairment in older people with dementia. International Review of Psychiatry, Vol 6, pp 153-161. [8] Woods, R. T (1999) Psychological therapies in dementia. In R. T. Woods (Ed.) Psychological Problems of Ageing. Chichester: Wiley. [9] Conway, M. A., & Pleydell-Pearce, C. W. (2000). The construction of autobiographical memories in the selfmemory system. Psychological Review, 107 (2), pp 261-288. [10] Brown, N. R., & Schopflocher, D. (1998). Cueing, event clusters and the temporal distribution of autobiographical memories. App.Cog. Psych., 12 (4), pp 305-319.
[12] Alm, N., Dobinson, L., Massie, P., Hewines, I. (2001) Computers as cognitive assistants for elderly people. In Proceedings of Human-Computer Interaction INTERACT01. (Tokyo, July 2001). Michitaka Hirose, Ed. Amsterdam: IOS Press. pp 692-693 [13] Alm, N., Ellis, M., Astell, A., Dye, R., Gowans, G., Campbell, J. (2004) A cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation. Vol 14 (1/2), pp 117-134. [14] Lidwell, W., Holden, K., Butler, J. (2003) Universal Principles of Design. Gloucester, Massachusetts : Rockport Publishers. [15] Kurosu, M., Kashimura, K. (1995) Apparent usability vs. inherent usability : Experimental analysis on the determinants of the apparent usability. Proceedings of CHI95. pp 292-293. [16] Norman, D. A. (2002). Emotion and design: Attractive things work better. Interactions Magazine, ix (4), pp 36-42.
[11] Jefferson, G. (1984) On stepwise transition from talk about trouble to inappropriately next-positioned matters, in Atkins, J. and J. Heritage, (Eds.) Structures of Social Action -- Studies In Conversation Analysis. London: Cambridge University Press. pp 191-222.
36
Promoting Virtual Social Interaction with Older People

Peter Bagnall, Guy Dewsbury, Victor Onditi, Ian Sommerville,
Computing Department, InfoLab21 South Drive, Lancaster University Lancaster, LA1 4WA +44 1524 510352
bagnall | dewsbury | onditi | is @comp.lancs.ac.uk ABSTRACT

This paper charts the development of a software platform and applications designed specifically for older people to promote communication and sociability. Through this we attempt to help alleviate isolation or loneliness by providing appropriate informal communication and virtual interaction. The software has been developed using a person-centred design approach that specifically favours simplicity and flexibility of use over functionality and power. Early participation from the older people was in the form of adapted cultural probes [8]. As the design progressed participation took the form of iterative criticism of the design, and informal user tests with older people. The system allows older people to easily discover who else is currently using the system, and therefore potentially available to chat with, interact with or play games with. The games also allow voice chat so players can socialise while playing. The application of this software has the possible use of encouraging intergenerational communication. investigation, our original conceptions of designing some form of smart or ubiquitous home was found to be misguided by the participants whose concerns were mundane, relating to undertaking everyday tasks and avoiding isolation and loneliness. Our investigations found that, even in housing specifically designed for elders with available social facilities, older people felt lonely and isolated. They used the artefacts such as the television as a means of avoiding the consequence of their isolated experience, the television taking the place of the lost partner, in a silent static form. As a response to the isolation that many of the participants were experiencing, we started to investigate how we could make use of computer-mediated communications as a way to facilitate nonintrusive, informal communications between older people. Critical to this was the development of a system that was both usable by people suffering from many of the normal infirmities of old age and which did not look like a conventional computer with all of the negative preconceptions that this entailed.
General Terms
Person-Centred design, Experimentation, Human Factors.
2. PROBLEM IDENTIFICATION
Unfortunately, our common metaphors for computer interaction lead us away from the invisible tool, and towards making the tool the center of attention. [12] Lancaster Universitys Computing Department is renowned for having a multidisciplinary staff membership consisting of computer experts and social scientists. This multidisciplinary mix has proven extremely successful in a number of software developments and has recently been focused on developing assistive technology and smart homes for senior and disabled people as well as CATCH - a Compendium of Assistive Technology Checklists for the Home, for professionals to determine technology requirements for supporting independence in the home (www.smartthinking.ukideas.com/CATCH.html). The research team undertook ethnographic field studies with seniors in their own homes over a period of two years. A number of new methods were piloted and adapted to achieve this task including adapting cultural probes [10] [5], technology tours [2] [11], observations, qualitative interviews and open questionnaires [7]. The researchers have worked with a many seniors throughout the UK and allowed them to determine their own technology requirements. As a result of this fieldwork, it was revealed that the seniors were less concerned with the introduction and use of high-end technology, such as smart homes, and more concerned with everyday mundane routines and activities such as cleaning their homes, meeting other people and being isolated for much of the time. Participants explained this isolation was due to family
Keywords
Person-Centred Design, Older People, Cultural Probes
1. INTRODUCTION
An increasing concern with modern society is that the rise in technology has empowered and supported those people who have been able to take advantage of the available technologies, whilst leaving the remainder technologically alienated and isolated [1]. One group for whom this is certainly a possibility is older adults who are increasing in proportion as well as becoming more isolated through family breakdowns [4] [10]. This paper reports on the work being undertaken at Computing Department at Lancaster University in the UK where some software is being developed to enable seniors to communicate informally with each other for example their friends and families through a computer without the complexity associated with traditional computing platforms. Our research originated as part of the DIRC (www.dirc.org.uk) initiative, a multi-institutional investigation funded by the EPSRC on dependability in computing systems and continues through Microsoft Research funding. On this project, we worked with a number of older people in the community to design a technology system that would be suitable for them. In the process of this
37
and friends not visiting or having passed away. Many of the older adults found that at weekends they could be alone all weekend. On further investigation this time period expanded considerably such that some older people could spend up to a week without seeing another person socially. Many participants had circumnavigated this by employing people to come in to their homes and clean or do other jobs, which provided some level of company at a high price, but many participants were still, at times, chronically lonely.
daunting and inappropriate. Whats more, most of the functionality offered by such systems is not relevant to seniors.
4. FROM REQUIREMENTS TO DESIGN

The hardware and software needed to be simplified. For the hardware a Tablet PC form factor was chosen, specifically an NEC T400, a slate style tablet, which has no keyboard (although a keyboard can be connected via USB). This particular machine is relatively light, at 1kg (2.2lb) and has a reasonable sized screen, which is important to be able to present information at a useful size. As with all Tablets, it offers stylus-based input. The styluses (figure 1) that are provided with the tablet come in two sizes. One is about the size of a normal biro, the other much smaller. Most older people were much more comfortable using the larger stylus, but so far the selection of available styluses is less than ideal for older people.
3. DERIVING OUR SOLUTION

There was no obvious technology solution to preventing or alleviating isolation without bringing more people into the equation. Loneliness and isolation cannot be solved without contact with other people. Yet for many older people in our study, the contact with other people was a source of concern and worry and hence the cycle was extended. Over many months of work with the older participants, we determined that an informal communication system was what the older people wanted. A system that would indicate when other friends or family were available to be contacted was what the older people required. The system needed to show who was on the network but this should not place any obligation on the user to make contact if they did not want to. Contact with others was therefore voluntary. Our participants informed us that often knowing that other people are there and available is enough to console the isolated person and make them feel less alone. Similarly we found in the fieldwork that many seniors were reluctant to make overt contact even with neighbours as they did not wish to cause bother to them and did not wish for visitors themselves as this meant a lot of preparation beforehand. Social communications, therefore, needed to be informal, and not require the users to enter into more disclosure than they were happy to. For this reason cameras were ruled out, as the seniors did not want to be seen by others although they would like to see the other people themselves. Avatars were considered as a possibility to replace real pictures but this becomes over complex and possibly of little use to the older people themselves, where as a still picture of the person they are communicating with does help them feel they are talking to a real person. Our initial intention was to build software package that could be used on a standard PC but we soon discovered that there are numerous problems with traditional desktop or laptop computers for seniors. Firstly and most important, the older people in our study were by and large computer illiterate and feared anything that resembled a computer. In practical terms of hardware, desktop computers require significant amounts of space, often at a premium. Input via a keyboard assumes either some keyboard skills or great patience, and mouse input requires learning a coordination skill, and is awkward for many older people even once they have learnt. Interaction idioms such as double clicking are mechanically difficult for many older people. Laptops suffer from most of these problems, but exacerbate them by having a smaller keyboard, and even more difficult pointing devices. Modern operating systems are highly complex, and require substantial training or experience to use effectively. For seniors who have limited energy to devote to learning new skills this is
Figure 1 The actual design was informed in large part by observations of older people using computers and prototypes at various stages. From this a number of simple design guidelines were laid down. Having seen that they were disinclined to explore an interface in the way experienced computer users do, one principle was that functionality should be clearly visible, not hidden inside GUI widgets such as dropdowns or in dialog boxes. To keep the complexity to a minimum only a small set of widgets are being used. Buttons, which can be tapped with the stylus, have been tested with older people, and cause few problems. Initially many older people tried, very logically, to tap the button with their finger. Having been told this didnt work they quickly became used to using the stylus instead. Ideally the hardware would support using a finger for tapping buttons, but pragmatically the problem appears to be solvable by documentation and training since it was very readily learnt. Writing with a stylus is also relatively familiar, although making it obvious where they on the screen they are allowed to write is an important design consideration. Using a consistent visual style for regions that can be written on is important, but it appears that older people are less adept at noticing these relatively subtle cues in the interface. Again, training and documentation may help, but the design needs to assume that these subtle visual cues are not understood, so redundant mechanisms may be required. The last interaction style that is allowed is dragging. Mechanically this is identical to drawing a line. While we dont have as much experience with seniors using a drag interaction there have, as yet, been no cases of seniors having trouble with this idiom. However, the objects to be dragged must be very clearly indicated. Another guideline related to the labelling of user interface elements. Fiske et al suggest [8] that since icons are rarely well understood, textual labels are better. Observations of older users confirm this finding. This lead to a design guideline that all buttons must have a textual label. Iconic labels are optional.
38
Figure 2 The Chooser User Interface This simple set of guidelines is quite restrictive, but this has proved to be an asset rather than a hindrance, since it serves to concentrate the design on those functions that are of greatest importance. they found that they had reached a point where they required more functionality than the platform could provide then it would be time to start using the full functionality that is provided by Microsoft Windows. When seniors use the chooser to select an activity, in most cases they will be required to select another user to participate in the activity. To begin an activity the user is required to make a minimum of taps on the screen. Tapping an activity selects it, tapping another user selects them. Once selected tapping start activates and initiates the activity. Tapping a selected user or activity deselects it, which allows for a straightforward way to correct errors.
4.1 Platform Design

As one of the concerns being addressed by the designs was the reluctance to intrude, a presence awareness interface, similar in principle to a Microsoft Messenger Buddy List was developed for the system. The Chooser (figure 2), by which we refer to the main screen, displays other users, indicating whether they are online and allows each user of the system to write a short phrase to convey their emotional state or whatever information they wish other users to know. This means users can see at a glance who is available to connect with. Alongside the buddy list is a list of potential activities. The exact set of activities depends on which programs have been installed on the device. In accordance with the established guidelines, the buttons for the activities are large, with large text labels, to make them easy to read and easy to tap with the stylus. This means that there is a limit to the number of activities that can be accommodated on the interface, but again this in fact is a useful limitation, since it helps provide an incentive to keep the number of activities (and hence the level of complexity) down. It was also determined by the seniors that if
4.2 Chat Design

The design for Chat (Figure 3), although inspired to some extent by instant messenger (IM) applications, deviates from this is a few important ways. Firstly there is an invitation step, on entering the chat activity the user writes a short invitation to the person they with to chat to or determine if the user is ready for communication. This extra step is to allow a way for older users to politely decline a conversation through not accepting the invitation. This step is essential as a major barrier to communication is a sense of not wanting to intrude on another person. If the chat activity simply launched directly into chat,
39
Figure 3 The Chat User Interface without giving the other person a way of declining it is more likely that this would be considered intrusive, and therefore the same barriers might reappear, in a similar way that a telephone call can be annoying when it is from someone you do not wish to speak with. Once an invitation has been written and sent, the receiver sees the invitation on screen and can either accept it, in which case the chat begins, or decline, in which case they can if they wish write a brief message saying why, again to allow polite behaviour. Invitations deliberately do not time-out. It was considered better to let the user decide when to stop waiting rather than impose some arbitrary time limit. This is especially true for older people where it may take a few minutes for them to respond if they are away from the tablet when an invitation arrives. The actual chat is essentially a shared white-boarding system. This has some advantages over conventional IM. It avoids the threading problem within a conversation, of utterances getting out of order, because the position of writing indicates clearly what relates to what. It is also more powerful in that users can draw, scribble, and even play simple games. Of course screen space is limited, and this solved by using a paging system. One user can suggest that a new page is created. When the other user agrees both move to the new page together. This rather involved interaction ensures that both parties are aware of new pages being created, so it avoids one person writing on one page, while the other writes on a different page. The users can look back over old pages at any time by using paging buttons to move back and forth. Scrolling was avoided because it is mechanically awkward, conceptually somewhat confusing (novices often find it upsidedown), and it is open to the failure of having the two participants looking at entirely different parts of the system, and therefore not communicating. It would also introduce another interaction style, whereas the paging model can be achieved with button, a widget that is already in use in many other parts of the system.
5. GAMES AS A COMMUNICATIONS FACILITATOR

There are many other applications that we are developing for the platform including email / Internet access and photograph sharing options, but we turn to consider the development of a card game application - FreeCard. As an icebreaker and a common shared experience games can support a means of obviating isolation. It is not required that one needs to know or like a person to play a virtual game with them and get enormous pleasure from doing so. Chris Crawford, remarked in his book "The Art of Computer Game Design" that:
40
"Games are frequently used (especially by adults) as social lubricants. The game itself is of minor importance to the players; its real significance is its function as a focus around which an evening of socializing will be built. Card games and some light board games serve this function. [6] Traditional games provide a social environment where conversations can arise. The game provides a reason for meeting, and an excuse for conversation. Between friends, conversation around a game table will generally not relate to the game itself, although from time to time undoubtedly the game will be referred to. That social experience of gathering with friends is what the game environment aims to replicate. When strangers meet to play a game, conversation may initially revolve more around the game, but it is less certain that this design is so well suited towards strangers playing meeting and playing games since. The game environment has been designed, for now, to support card games, as these provide a wide range of games with a wide audience. However, when exploring which games older people played it was found that, amongst the participants, there was a great variation in the games played, and for those games that were known to many, there were significant variations in the rules. The variation of rules and the general desire to keep the design simple lead to the concept of a games environment that is agnostic towards rules. In effect the game environment, ideally, should behave as little more than a virtual table. As with real cards, the rules are agreed between the players, and enforced by the players. This is a departure from typical computer-mediated games, where the computer often acts a referee. The advantages of this are considerable. It allows players to play a wider range of games, since they decide on the rules. It simplifies the game interface, since the system doesnt have to enforce rules and it means the selection of games is simpler, since there is no need to select which rules are being used. It also makes it possible for players to cheat, which in the context of a grandparent playing a game with
played. But since the purpose of the system is to increase sociability, increasing the need for conversation is actually a benefit, not an inconvenience. The ideal medium for this communications channel is clearly voice. It is rapid, simple, and doesnt clutter the virtual game-table interface as a visual communication medium (such as the chat activity) would. The tablet hardware has both speakers and a microphone, so it is possible to use voice communication without additional hardware. It is possible that older people will find this somewhat unnatural, since they are not generally familiar with hands-free telephony style systems. For this reason, and to gain improved audio quality it may be better to provide a headset, although this then becomes more cumbersome. There is also the issue of compatibility with hearing aids, so it is very likely that a range of solutions are required to cater for a range of needs and preferences. Currently we are exploring the use of Skype (www.skype.com) for supporting a multiple-player audio conference. Skypes audio conferencing facility is currently limited to a maximum of five participants. As a consequence, FreeCard can be played by a maximum of five players even though theoretically, more players could be supported. Our practical experience using the facility is that the sound quality degenerates dramatically with an increase in participants. The quality is very impressive for two players. The game platform automatically instigates the creation of a Skype session without any additional work on the part of the users. From the users perspective audio conferencing is simply part of the games activity. The user interface design of FreeCard (figure 4) aims to be reminiscent of a game table. One of the aspects of this is the relative positions of the various players. With a real card game the players hand is held between the player and the table, and so with the interface the natural place for the players hand is at the bottom of the screen, closest to the player when the tablet is resting in the lap. Since in many games where players sit relative to one another
Figure 4 - FreeCard Game Prototype
a grandchild may actually be an advantage too! Since the system will not referee the game, the need for players to communicate increases. They need to prompt people when its their turn, to enforce rules, and to decide which game will be
is significant it is important that the interface makes it possible for people to sit where they choose. For example, it often defines the order of play. To keep the interface as simple as possible however, it was decided to not allow players to move seats once a session had started. The problem is that if players are allowed to
41
move seat midway through a session, and a player always appears at the bottom of the screen, moving from one seat to another would mean the table would need to be rotated. It is likely that this would be somewhat confusing since the player would appear to return to where they were originally. Not allowing this is the simplest way to avoid the issue, although other solutions may be tried in the future. The participants also expressed views on how their hand should be laid out. They preferred that cards should be laid out with their full face showing, rather than allowing a card to partly obscure another. It is possible that this reflects a difficulty with reading the smaller numbers in the corner of the cards. Future card designs will aim to address this issue with alternative card designs, as well as layouts.
Microsoft Research as part of their Create, Play and Learn collaborative research programme.
8. REFERENCES
[1] Richard Adler http://www.seniornet.org/php/default.php?PageID=6694 [2] Baillie, L. and Petersen, M. G. (2001) Emerging Themes in Designing Household Environments. In Proceedings of the OIKOS2001 Workshop, Aarhus University Press, pp. 44-46. [3] Bagnall P, Dewsbury G and Sommerville I, (2004) Easy For Everyone: Using components to offer specialised interfaces for software in Baxter G and Dewsbury G (Eds), (2004) HEAT 2004: Proceedings of the First HEAT: The Home and Electronic Assistive Technology Workshop, Computing Department, Lancaster University. [4] Batty D (2004) Older people face a lonely future, thinktank warns, Guardian Politics, 30June, 2004, http://politics.guardian.co.uk/thinktanks/story/0,10538,12502 81,00.html [5] Cheverst K, Clarke K, Dewsbury G, Hemmings T, Hughes J and Rouncefield M, (2003) 'Design With Care: Technology, Disability and the Home', in Harper R (Ed) Inside The Smart Home, Chapter 9, pp163-180, Springer-Verlag, London Ltd. [6] Crawford C, (1982) The Art of Computer Game Design, http://www.vancouver.wsu.edu/fac/peabody/gamebook/Coverpage.html [7] Dewsbury G, Clarke K, Rouncefield M, and Sommerville I (2004) Depending on Digital Design: Extending Inclusivity, Housing Studies Vol 19, No 5, September 2004, ISSN 02673037 [8] Fiske, A., Rogers, A., Charness, N., Czaja, S., & Sharit, J. (2004). Designing for older adults: Principles and creative human factors approaches. Boca Raton, Florida, CRC Press. [9] Gaver, W.H., Dunne, A. and Pacenti, E. Design: cultural probes, Interactions, vol. 6 (1), 1999, 21-29. [10] Help The Aged, (2005) Social Isolation and Loneliness, Help the Aged Policy Statement 2005, available from http://policy.helptheaged.org.uk/NR/rdonlyres/e7nauzeva2yb a5cjgr7nerhygd64mfr3yjf2gokljerpxglx4t2zixflnmlqbdhx2jk xrtux6g3oo3pqg2mzqkllnkb/index_social_isolation_loneline ss_01_05.pdf [11] Petersen, M. G. and Baillie, L. (2001) Methodologies for Designing Future Household Technologies. In Proceedings of the OIKOS2001 Workshop, Aarhus University Press, pp. 47-49.Petersen, M. G., Madsen, K. H. and Kjr, A. (2002) Usability of Everyday Technology Emerging and Fading Opportunities. In ACM Transactions. on Computer-Human Interaction, Vol. 9, No. 2, June 2002, pp. 74-105 [12] Weiser M (1993) The world is not a desktop computer, http://www.ubiq.com/hypertext/weiser/ACMInteractions2.ht ml
6. CONCLUSION
It is difficult to draw firm conclusions at this time, although we know the system works and that older people in the UK are enjoying using it. We are currently putting Broadband connections in to participants homes in order to undertake longer and more rigorous studies of its efficacy and usage. A clear indication at this stage is that the systems will have much wider applications that its initial design specification. For this reason we will be developing an installation package CD ROM to be given to friends and relatives of the users, so that they can use the system. This will take considerable work as this will require allowing keyboard and mouse support and usage on a standard PC. Moreover, the implementation of the platform and the activities that it supports separate the user interface from most of the underlying engineering [3]. The user interface that is being deployed includes some configurability for colour schemes but future plans include allowing UIs that are more suited to desktop or laptop computers, without stylus capability. At the moment the software can be used with Wacom style tablets without modification. We also consider that this disconnection of the UI is advantageous as we are able to make special UIs for people with specific eye conditions, or people who speak other languages. It is also envisaged that part of the system could be implemented on mobile devices, such as mobile phones or PDAs, which would allow much greater contact between young and old generations. One possibility here includes allowing younger users to take photographs with a mobile phone and upload it directly to a senior relative. At this stage we can see many ways in which this devise can be further developed to support communication between older people, but we also consider the software has considerable applicability for use with children, people with learning difficulties and disabled people who have limited functionality.
7. ACKNOWLEDGMENTS
The authors would like to thank Age Concern Carnforth and MHA Care Group for their assistance and support in this project. This work was partially funded in the UK by the Engineering and Physical Sciences Research Council as part of the DIRC Interdisciplinary Research Collaboration and partially funded by
42
Talking with Individuals with Dementia on a Video Phone: A Preliminary Study for Networked Interaction Therapy
Kiyoshi Yasuda
Chiba Rosai Hospital Chiba, Japan
Kazuhiro Kuwabara
ATR Kyoto, Japan
Noriaki Kuwahara
ATR Kyoto, Japan
fwkk5911@mb.infoweb.ne.jp Shinji Abe

ATR Kyoto, Japan
kuwabara@atr.jp
kuwahara@atr.jp Nobuji Tetsutani

Tokyo Denki University Saitama, Japan
The purpose of networked interaction therapy is to provide individuals with dementia various types of intervention using a video phone, appropriate video contents, and scheduling prompts. As part of the preliminary study to investigate the effectiveness of a video phone, we have conducted an experiment to compare how long individuals with dementia can talk with a partner on a video phone and in a face-to-face session. In the experiment, nine individuals are requested to reply to 20 questions asked by a partner. The results reveal that there is little difference in the total response time to these questions between the two sessions. This suggests that the video phone conversation can potentially have the same effects as face-to-face conversion from the viewpoint of dementia care. In addition, we discuss the use of several kinds of contents that can augment a video phone session.
tetsutani@ia.dendai.ac.jp
social services are available to relieve the stress of home caregivers. Various therapeutic approaches have also been undertaken such as memory training, music therapy, behavioral therapy, and reminiscence therapy [3]. Nowadays the use of computer-based intervention is also being investigated [1, 5]. We are currently proposing a networked interaction therapy intended to provide various types of communicative intervention such as talking on video phones, applying appropriate contents, and conducting daily-life management through a TV monitor [5]. This system uses a set-top box with a camera and microphone for monitoring the users status and controls various types of intervention. Additionally, an air cannon is also considered for stimulating the olfactory sense. If these forms of intervention can increase individuals quality of life (QOL) and reduce behavioral disturbances, caregivers will be able to gains some respite while individuals with dementia are engaged in networked interaction therapy. Communication, especially verbal conversation, is an easy, common, and enjoyable activity for most of us. Dementia sufferers, however, tend to be alone and poorly informed, with few chances to converse. One of the most important forms of intervention in networked interaction therapy is to provide them with talking partners on the Internet, such as friends, volunteers, and family members living in remote locations. A simple interface would thus enable individuals with dementia to talk with partners on the video phone whenever they want. However, there are no studies on how long individuals with dementia can talk with a partner on the video phone, in comparison with the face to face settings. Since the concept of networked interaction therapy is based on the use of video phonebased intervention, it is important to verify that the dementia sufferers can converse sufficiently for a certain duration on the video phone. In this study, individuals with dementia were requested to reply to 20 prepared questions from a partner in the video phone session and in a face-to-face session. Total response times to these questions are compared between the two sessions. Furthermore, we plan to deliver several contents during the video phone session in order to augment talking on the video phone. In the latter part of this paper, we introduce several such contents that we consider promising for video phone sessions.

J.3 [LIFE AND MEDICAL SCIENCES]: Health
General Terms
Human Factors
Keywords
Dementia, intervention, video phone, talking, conversation
1. INTRODUCTION
The onset of dementia usually begins with mild forgetfulness and often develops into a variety of behavioral disturbances such as wandering, agitation, illusion, and incontinence. Because of these behavioral disturbances, caring for dementia sufferers is a difficult and frustrating task, especially for home caregivers [2]. Since caregivers bear a heavy burden, they are in need of some regular respite time [7]. In addition to pharmacotherapy, several
43
2. VIDEO PHONE EXPERIMENT 2.1 Subjects

Nine individuals with mild and moderate dementia, whose ability to converse still remains were selected as subjects. They were diagnosed to be suffering dementia by their physician (neurologist) in charge. Table 1. Profiles of five subjects Subjects OI NK YM KM SS IS SN FS KM Gender M F M M F F F F F Age 81 71 76 87 83 76 65 73 79 MMSE 10 23 23 15 17 25 23 20 17 Etiology Alz, Infar Levy, Park Alz, Infar Alz, Infar Alz Alz Alz Alz Alz, Infar
utterances such as Oh, I see. If the subject uttered nothing for five seconds and no additional comments were forthcoming, the partner proceeded to the next question. When the subject tried to utter further comments, the partner waited for a maximum of ten more seconds. Two minutes were allocated to each question as the maximum responding time; accordingly, the total responding time was 40 minutes at maximum longest. When a subject took more than two minutes on a question, the partner interrupted the subject from talking and proceeded to the next question. Their faces and gestures as well as utterances were recorded by a digital video camera. Before the experiment, the partner explained its aim and procedure to each subject and their caregiver, and obtained their consent. After the partner chatted with the subjects on common opening topics such as the weather and their health condition about for three minutes, he started to ask the prepared questions, beginning with Now, please let me hear about your past. We compared the total response time between the video phone session and the face-to-face session.
2.5 Results
The experimental hypothesis was that there is no difference in the total response time between the video phone session and the faceto-face session. The total response time of each subject for the two sessions was calculated, with the results presented graphically in Fig. 1. 40 35 30 25 20 15 10 5 0
Note: MMSE: score of the Mini Mental State Examination, Alz: Alzheimer's disease, Infar: infarction, Levy: Levy-type dementia, Park: Parkinsons disease.
2.2 Experimental Setup

We used commercially available TV conferencing software that runs on a personal computer (PC) 1 . We prepared PCs, 19-inch monitor, USB camera, microphones and speakers for the subjects use and for their partner. The face image approximately the same size as the real face of each person was displayed on the monitor. PCs were connected by a LAN at a speed of 100 Mbps.
Video Phone Session Face to Face Session
2.3 Questions
Twenty questions were prepared for this experiment. All questions were easy, common, and concerned with each subjects younger days, such as their hometown, family members, favorite foods, and primary school activities. The talking partner in this experiment was a speech therapist who was in charge of the subjects evaluation and training. He asked them the questions in a slow, relaxed, and gentle tone. One subject (OI) with hearing difficulty wore earphones.
OI NK YM KM SS
IS
SN KM FS
Figure 1. Total response time (minutes) of each subject in the videophone and face-to-face sessions The average response time was 16.5 minutes in the video phone sessions but 14.1 minutes in the face-to-face sessions. These results demonstrate there was no significant difference in the total response time between the two sessions.
2.4 Procedures
The experiment was carried out in speech therapy rooms of a hospital that the nine subjects visited. The AB or BA design was applied, where A was the video phone session, and B was the face-to-face session. The AB design was applied to five subjects, while the BA design was applied to the other four. The intervals between A and B, or B and A, were varied from a week to two months. The partner asked each subject the 20 prepared questions. After asking each question, he only listened to the subjects reply passively, asking no additional questions, just responding with some subtle, customized gesture (e.g. nodding, smiling) or vague
1
3. AUGMENTING WITH VIDEO CONTENTS

Some individuals with dementia cannot comprehend words and sentences, nor can they recall old memories. These impairments often disturb continuing conversations. To cope with these impairments, then, presentation of various visual, auditory, verbal supportive contents are a great help. The following contents can be delivered as augmentative contents for video phone conversations. We plan to evaluate their effectiveness in the near future.
http://www.polycom.com/common/pw_cmp_updateDocKeywor ds/0,1687,4550,00.pdf
3.1 Personalized Reminiscence Photo Video

Reminiscence intervention is one type of intervention suggested as being effective for decreasing behavioral disturbances [4]. In
44
spite of its effectiveness, regular reminiscence intervention is difficult to perform at home. Therefore, we have created a personalized reminiscence photo video that consists of an individuals old photographs with background music (BGM), narration and pan/zoom visual effects as a convenient reminiscence intervention [6]. The photo video helped home caregivers to perform the reminiscence intervention in the home. It could also be performed repeatedly even without the caregivers attendance [9].
4. DISCUSSION AND CONCLUSION

If we can provide some types of intervention that can attract and hold the attention of individuals with dementia, it can potentially reduce the behavioral disturbances and increase caregivers free time [7]. In this study, we investigated how long individuals with dementia can talk with a partner on a video phone. The results revealed that individuals with dementia can talk on the video phone as long as they engage in face-to-face conversation. Therefore, video phone conversation can be used as an effective form of intervention to support them. Furthermore, the experiment result also indicates that the concept of networked intervention is a promising one. To increase the opportunities for individuals with dementia to participate in a conversation, we consider the video phone to be a useful tool. As an individuals dementia worsens, normal conversation becomes difficult to perform. At this stage, it would be better to provide a variety of contents as augmentative contents to promote conversation. Presentation of the contents that we introduced in the previous chapter will enable them to more easily comprehend verbal questions, stimulate old memories, and activate conversation. Although several commercially available on-line services are used in practice for health care, these services are limited to basic physical/behavioral checks or to conducting short conversations. Networked interaction therapy, on the other hand, will extend networked supportive intervention for individuals with dementia more widely and systematically by providing an interface that is simple for them to use.
3.2 Talk Video

Several videotape series have been on the market for the generic use of individuals with dementia. In a series of videos titled Video Respite, an actor or actress talks to the viewer about favorite activities such as gardening, Christmas, and family gatherings, in a calm and friendly tone [7]. They also sing old songs in the video. We created two kinds of a similar generic video named Talk Video, in which a woman talks on various topics about her hometown and the changing of the seasons, and sings old songs. This video can be delivered when the partner on the network is too busy to participate in a video phone session.
3.3 Vocabulary Data File

To facilitate conversation for people with severe or moderately severe communication deficits, a vocabulary data file called Rakuraku Jiyu Kaiwa (Easy, Natural Conversation) has been compiled [8]. This file contains more than 50,000 words, mainly proper names (for example, all the titles of the Beatles songs, titles of old movies etc.). These words function as key words in conversations (for example, What is your favorite Beatles song?). The words are classified into ten categories, which have a four-level tree structure, and by accessing a website (http://rousai.tripod.co.jp/), users can easily search for words related to current topics. Furthermore, by using these words, related pictures or photographs can also be collected easily from the Internet.
5. ACKNOWLEDGMENTS
This study was supported in part by the National Institute of Information and Communications Technology (NICT).
6. REFERENCES
[1] Alm, N., Astell, A., Ellis, M., Dye, R., Gowans, G., and Campbell, J. A. Cognitive prosthesis and communication support for people with dementia. Neuropsychological Rehabilitation, 14, 117-134. 2004. [2] Davis, R. N., Massman, P. J., and Doody, R. S. Cognitive intervention in Alzheimer disease: A randomized placebocontrolled study. Alzheimer Disease and Associated Disorders, 15, 1-9. 2001. [3] De Vreese, L. P., Neri, M., Fioravanti, M., Belloi, L., and Zanetti, O. Memory rehabilitation in Alzheimer's disease; A review of progress. International Journal of Geriatric Psychiatry, 16, 794-809. 2001. [4] Finnema, E., Droes, RM., Ribbe, M., and Van Tilburg, W. The effects of emotion-oriented approaches in the care for persons suffering from dementia: A review of the literature. International Journal of Geriatric Psychiatry, 15, 141-161. 2000. [5] Kuwabara, K., Tetsutani, N., Kuwahara, N., Kawato, S., Lyons, M. J., Utsumi, A., Yanagida, Y., and Yasuda, K. Networked Interaction Therapy: Supporting People with Mild Dementia and their Family Members with Internet. 20th International Conference of Alzheimers Disease International Kyoto 2004, Abstracts, 377. 2004.
3.4 Reminiscent Short Movies

The DVD movie series Omoide no Tamatebako (Reminiscent Treasure Box), published in Japan, contains about 20 short movies depicting things such as old foods, customs, childrens play, and traditional events in Japan. Each movie is about three minutes long and features soothing music.
3.5 Historical Social News Movies

A collection movies featuring Japanese social news originally made after World War II were edited and published in a DVD series named Syowa Nippon (Japan 1925-1988). The series consists of about 20 DVDs that are arranged chronologically. Each DVD is about 60 minutes long and contains about 50 short movie clips.
3.6 Categorized Photograph Files

Many generic photographs are sold on the market as royalty-free photographs. These photographs are often categorized according to themes such as Japanese people, landscapes, and animals. Such photographs may be used to elicit memories from subjects when the partner asks about their hobbies.
45
[6] Kuwahara, N., Kuwabara, K., Utsumi, A., Yasuda, K., and Tetsutani, N. Networked Interaction Therapy: Relieving Stress in Memory-Impaired People and Their Family Members. Proc. 26th Annual International Conference of the Engineering in Medicine and Biology Society (EMBS 2004), 3140-3143. 2004. [7] Lund, D. A., Hill, R. D., Caserta, M. S., and Wright, S. D. Video RespiteTM; An innovative resource for family, professional caregivers, and persons with dementia. The Gerontologist, 35, 683-687. 1995.
[8] Yasuda, K., Nemoto, T., Takenaka, K., Mitati, M., and Kuwabara, K. The effectiveness of vocabulary data file, electronic, encyclopedia, and Internet homepages as a conversation support system for people with moderate-severe aphasia. Aphasiology, in submission. [9] Yasuda, K., Kuwabara, K., Kuwahara, N., Abe, S., and Tetutani, N. Personalized Reminiscence photo video for individuals with dementia. Neuropsychological Rehabilitation, in submission.
46
The Relationships among Cognition, Parental Speech Patterns and Foreign Language Ability, and Multiple Intelligence Theory based Foreign Language Teaching
Keiko Hayashi
Hiroshima Jogakuin University (home telephone: 81-072-833-6828 e-mail:aabch005@kcat.zaq.ne.jp)
ABSTRACT
This paper reports (1) the study on the relationships among intelligence, cognition, and foreign language ability, and discusses (2) the Multiple Intelligence (MI) Theory based teaching method for developing communicative ability.
II. THE RELATIONSHIP BETWEEN INTELLIGENCE AND COGNTION

It is often said that intelligence affects language skill and aptitude, but it is not clear if intelligence is related to language or cognition. For example, the TOEFL score in the Netherlands is the highest in the world. According to some researchers in the Netherlands, this is because Dutch people are more intelligent than other people (cf. Kee de Bot, 1999). However, the result of PISA2003 indicates that there is little relationship between intelligence and language ability. Although Finland shows that all three subjects are the highest in the world, the Netherlands and Japan show that the mathematics is higher than that of other countries, but literacy is lower. The correlation between IQ and language aptitude is 0.43 (Gardner and Lambert, 1972) and 0.44 (Skehan,1998).The relationship between general cognitive abilities and the second language ability is 0.65Skehan,1998.1 It is difficult to define precisely what intelligence means. IQ score is often referred to intelligence, but it seems that cognitive abilities influence on foreign language (FL) learning due to the necessity of inference of rules and meaning from context. According to H. Gardner, each person possesses all eight intelligences. He proposes a pluralistic view of the mind, recognizing many different and discrete facets of cognition and acknowledging that people have different cognitive strengths (Gardner 1993). It is necessary for us to study on how intelligences and cognitive abilities develop and how MI theory integrates into FL learning. 1 See Hayashi, K. (2004:22)

A.1 [Introductory and Survey]: The influence of the parental speech on cognition and foreign language (FL) learning, and the introduction of the Multiple Intelligence (MI) into FL teaching.
General Terms
Languages, Theory, Performance, and Experimentation
Keywords
Cognition, Parental Speech, Foreign language learning, the Multiple Intelligence (MI), Cooperative Learning
I.
INTRODUCTION
The study of parental speech patterns indicates that the linguistic input is an essential factor for developing FL ability, and childrens success or failure in FL learning depends on the way of parental input. Interpersonal and intrapersonal intelligences developed in childhood affect EF learning. Using the MI theory developed by H. Gardner, this workshop shows the effect of the debate task through cooperative learning in FL classrooms.
47
III. THE INFLUENCE OF ENVIRONMENT ON COGITIVE DEVELOPMENT

Every parent in the world hopes that their own children have success in academic pursuit, and want to know how to bring up their children to be successful. There may be genesis for developing childs intelligence, but there is also a large possibility that a child is influenced by the environment. Children are influenced by the parental attitude. From the psychological point of view, it is said that every person does not like to listen their own reproachful words and criticism, and likes to listen praise words, applaud their achievements, victories, etc. Parental positive attitude towards their childrens curiosity and ideas about making something must be most important for child development. As for general cognitive development, Piaget classified cognitive development into six stages from sensory-motor intelligence level to formal operations. The formal operations begin towards eleven to twelve years. Eleven to twelve years old children have the possibility of reasoning by means of a hypothesis, and they will develop the deductive operation until about the age of fifteen. However, individual differences in cognitive abilities are large. Children possess the mechanisms of cognition as well as language, but whether the mechanism functions to bring about a rapidlearner or a slow learner depends on the environments. The environments involve the speech patterns and attitude of parents, teachers, and peers. Their speech patterns have positive effects and negative effects on children's cognitive development.
Table 1 shows that there are significant correlation coefficients between FL proficiency and parental speech patterns. The democratic interaction speech pattern has a highly significant correlation with FL proficiency. The imperative/authoritarian, negative/reproachful, and non-feedback speech patterns have negative correlations with FL proficiency. Advanced level learners have a high rate of democratic interaction with both their fathers and mothers and a low rate of negative/reproachful interaction with both their fathers and mothers. Intermediate level learners have the same frequency of the mother's democratic pattern as that of advanced level learners but the frequency of the father's democratic pattern is lower than that of the advanced level. The next significant point of parental speech patterns is the negative/reproachful pattern. Beginning level learners reported the highest frequency of both father and mother's negative patterns. It is also noteworthy that the proportion of the non-feedback pattern for intermediate and beginning level learners is higher than that of advanced level learners. The speech pattern indicates that children need input data, and at the same time, the way of parental input is crucial to facilitate their childrens cognitive development such as guessing ability, high mentality, concentration, and reasoning. The students with democratic interaction use contextualization strategies guessing the meaning of words and sentences from context in FL learning. They use skimming and scanning strategies for understanding the passage. Students with the imperative/authoritarian interaction learn by rote memory/ imitation. They are unwilling to understand abstract meaning. The students with negative/reproachful interaction have high affective filter and empathy, and confuse and frustrate about what to do in FL learning. Finally, they fear or blame others for an incomprehensible explanation. The nonfeedback/non-interference speech pattern is important in consideration of the childrens independent study but children cannot develop cognitive abilities with a let-alone policy. The children who are left alone do not know how to speak due to lack of a phonetic coding ability and are unwilling to understand meanings and grammatical rules. The learners success or failure in FL learning is related to their parental speech. Some children are interactive and cooperative with others, while others do not. Children do not develop only with parents interaction, but with social interaction. Hence, MI theories suggest collaboration with others and synthesize mind to get some ideas about how to learn and teach foreign language in the classroom.
IV. PARENTAL SPEECH AND FOREIGN LANGUAGE (ENGLISH) ABILITY

Referring to the Bernsteins theories (elaborated cord and restricted cord developed in 1967), I have examined the influence of parental speech on cognitive development and FL proficiency using 153 Japanese university students.
Table 1 Correlations between Parental Speech Patterns and FL Proficiecy I. Father's Speech 1. 2. 3. I. Father's Speech 1. Dem 1 -0.18* -0.41** -0.58** 0.47** -0.25** -0.20** -0.29** 0.46** 2. Imp. -0.18* 1 0.03 -0.55** -0.09 3. Neg -0.41** 0.03 II. Mother's Speech 5. Dem 0.47** -0.09 -0.27** -0.24** 1 -0.60** -0.43** -0.38** 0.31** 6. Imp -0.25** 0.31** 0.16* -0.05 -0.60** 1 0.12 -0.26** -0.13 7. Neg. -0.20** 0.02 0.42** -0.03 -0.44** 0.12 1 -0.15 -0.22** 8. Nof -0.29** -0.13 0.06 0.39** -0.38** -0.26** -0.15 III. L2 Proficiency 9. skills 0.46** -0.06 -0.42** 0.18* 0.31** -0.13 -0.22** -0.17* 1 Dem.: Democratic interaction; Imp.: Imperative/Authoritarian; Neg.: Negative/Reproachful; Nof.: Non-feedback Skills: including reading,listening,vocabulary, and grammar. *p<0.05 **p<0.01 (Hayashi 1997). 1 -0.17* 0.31** 0.02 -0.13 -0.06 1 -0.15 -0.27** 0.16* 0.42** -0.06 -0.42** II. Mother's Speech 4. 5. 6. 7. III. L2 Proficiency 8. 9.
4. Nof -0.59** -0.54**-0.15 1 -0.23** -0.06 -0.03 0.39** -0.18*
V. INTEGRATIONG MI THEORIES INTO FL TEACHING

It is often said that many Japanese university students keep their accustomed silence in the classroom. They never ask any questions or give any comments on what teachers say in the lesson. Thus, no communicative ability develops. This is because teacher-centered instruction is more popular than learner-centered instruction from elementary school. In the case of teachercentered instruction students get knowledge about the rules and words, but they have little opportunity to interact with others in the classroom. Learner-centered instruction develops students creativity and understanding through interaction in the classroom, but some students do not like to do group work. MI theories offer
48
EFL teachers a way to examine their best teaching techniques and strategies in light of human differences (ref. Christison,1998; Kagan & Kagan, 1998). According to H. Gardner, humans possess at least eight intelligences: linguistic, logical-mathematical, musical, bodily/kinesthetic, spatial, naturalist, interpersonal, and intrapersonal intelligences. Each intelligence is useful for FL learning and instructional practice in the classroom, but everyone has weakness and strength of the intelligence. In what ways should teachers apply MI Theory in FL classrooms? There are many ways to build each intelligence strength in the tasks. The purpose of using of MI theory in FL teaching is to enhance FL and communicative abilities. We can use all intelligences in the classroom. Table 2 shows 1) what is each intelligence, 2) how can each intelligence nurture, and how teachers can use the intelligence in the activities. Table 2 What is each intelligence? How can you nurture? The activities to enhance FL and communicative abilities? (1) Linguistic - the ability to use language to express meanings, understand others, tell a simple story, and learn new vocabulary or a FL that is used naturally. Nurture: make sure your program is rich with language opportunities and that childrens linguistic expression is listened to and appreciated. Activity: tell jokes, riddles, or puns. Read, write, or tell stories. Play word games, crossword puzzles, and word searches. Create poems and stories. Keep diary. Have students interest in debates. (2) Logical-mathematical - the ability to understand the basic properties of numbers, appreciate principles of cause and effect, one-to-one correspondence, and recognizes and solves problems using reasoning skills; the ability to predict, as in which objects will float, sink, etc. Nurture: work with numbers, figure things out, and analyze situations; use simple machines to help children think about cause and effect. Activity: work with numbers, figure things out, and analyze situations. Work in situations in which there are clear black-and-while solutions. Construct Venn diagrams. Use games of strategy. Have students demonstrate understanding using concrete objects. Establish time lines and draw maps. Logic puzzles and games. (3) Musical - the ability to produce and recognize simple songs; sensitive to nonverbal sounds in the environment, and aware of patterns in rhythm, and pitch. Nurture: incorporate music daily, use a tape recorder for listening, singing, and recording songs. Activity: play with melodies, varying speed and rhythm. Sing, hum, and move to music. Rewrite song lyrics to teach a concept. Teach history through music of the period. (4) Bodily/kinesthetic - the ability to use the body or parts of the body (hands, feet, etc.) to communicate and solve problems, as in playing a ballgame, dancing, or making things with the hands. Nurture: provide opportunities for physical challenges throughout the day, not just outdoors. Activity: Play sports and be physically active. Use body language. Provide tactile and movement activities. Offer role playing and acting opportunities. Allow the students to move. (5) Visual-Spatial - the ability to be able to form a mental image of large and local spatial layouts; think three-dimensionally; find ones way around a new building. Nurture: provide many opportunities for mapping of the classroom and beyond; explore new spaces and encourage children to vary the arrangements of
materials in the space. Activity: doodle, paint, or create threedimensional representations. Draw charts, maps and mazes. Take things apart and put them back together. Photographs, videos, slides, movies. (6) Naturalist - the ability to recognize species of plants or animals in ones environment. Nurture: play games in which children recognize fine distinctions among members of a plant or animal group; explore the outdoors regularly and bring the outdoors in; provide ample books, visuals, and props related to the natural world. Activity: use the outdoors as a classroom. Create a nature area on the playground. (7) Interpersonal - the ability to understand other people and work effectively with them and to notice who plays with whom at school, and why. Nurture: give children lots of opportunities to talk about one another and their social interactions, and to problem-solve conflicts together; play games in which one has to figure out the knowledge or intentions of other players. Activity: Use cooperative learning. Assign group projects. Give students opportunities for peer teaching. Use debate task in group work, and brainstorm solutions to problems. Create situations in which students are given feedback from others. (8) Intrapersonal - the ability to understand things about oneself, how one is similar to, different from others; know own strengths and weaknesses and how to soothe oneself when sad. Nurture: let children express their own unique emotions, preferences, and strategies; help them understand their own wishes and fears and how best to deal with them. Activity: Assign individual, selfdirected projects. Provide opportunities for students to get feedback from one another. Involve the students in journal writing and other forms of reflection (ref. Christison,1998) Table 2 shows that each intelligence develops through interaction, and plays the important role for developing FL ability. However, individual difference in each intelligence, particularly the interpersonal and intrapersonal intelligences, is large. Some students like to do group work, while others do not. The activities in cooperative learning attempt to better understand concepts and strategies, and enhance interpersonal and intrapersonal intelligences needed for developing FL ability, rather than simply reading and writing sitting at tables. For cooperative learning, elementary and junior high school children should be given a lot of opportunities to express their own ideas in ways that respect the individual interest and strengths while talking among group members. For senior high school and university students, debate task seems to be most appropriate for better understanding the concepts of the topics and strategies for communication.
VI. COOPERATING LEARNING 1. INVESTIGATION OF DEBATE TASK

For developing students communicative abilities debate tasks have been administered to my university students majoring in linguistics, education, and sociology. In 2001, I have examined the effect of the collaborative learning involved in the debate task using 119 Japanese sophomore students majoring in linguistics. The result shows that for some students the debate activities are
49
effective to improve FL proficiency, while for others there was little improvement. In order to develop reading and grammatical abilities extensive reading task is more effective than debate task. However, the survey on the students reactions to the debate task shows that only 2 students out of 119 had a negative impression. That is, 98% of the students showed a feeling of satisfaction with debate task. They wrote as follows: I learned how to find materials, and could reflect on other points of view. I learned how to make others understand my opinion. I learned to think about the topic deeply, to write my opinions in English, and to cooperate with others. I learned logical thinking, how to discuss and write, and the importance of group work. I took a lot of time to prepare, but I found it very useful for learning English. I heard others views too. I came to understand my topic in depth. Even though there were many things to be prepared, I really enjoyed it. We got together and worked hard. Debating in English is very difficult, but its a good way to study language. We have to tell our opinions to others. Its one of the most interesting classes and tasks that I have ever done with group work. Also, 98% of the students majoring in education showed the same positive and satisfactory reaction to the debate task. On the other hand, nearly 50% of the freshmen majoring in sociology showed the negative reaction to the debate task. They are not autonomous learners. They are usually expected to receive teacher-instruction, neither tell any opinions against others in the debate activity. In order to become independent learners, the students should be accustomed to interact and communicate with people from elementary school days.
building, problem solving and understanding alternative point of view. In traditional FL classes, teacher talk dominates, but cooperative learning is learner-centered. In group work, the students take responsibilities to collect reading materials suitable for the topic by themselves, talking with group members, thinking through ideas, and listening to others opinion. The students learn a lot of vocabulary and ideas through interaction with group members. They often misunderstand ideas when reading by themselves, but they become aware of the appropriate interpretation through interaction. Cooperative learning increases the quantity of speech. When teachers select the group members carefully, reticent students also have the chance to speak out. Cooperative learning from childhood helps students to be more positive in their learning attitudes and interaction.
VII. CONCLUSION
Intelligences related with cognitive abilities develop from childhood through interaction. Parental speech patterns indicate that the roles of parent and teacher feedback are crucial to developing cognitive abilities and intelligences. As Gardner (1993) suggests, everyone has the capacity to develop all eight intelligences to a reasonably high level of performance with appropriate encouragement, enrichment, and instruction. Each intelligence has its own developmental time. Although it is said that linguistic intelligence can peak very late, the interpersonal and intrapersonal intelligences will develop through interaction from childhood. The two intelligences are important for developing communicative and linguistic abilities. Giving children opportunities for peer teaching to talk about one another and to get feedback from one another in cooperative learning help children express their own ideas, emotions, preferences, and strategies. For senior high school and university students cooperative learning such as debate task is effective to enhance understanding ability of concepts, logical thinking, strategies, and communication ability in FL learning.
2. THE EFFECTS OF COOPERATIVE LEARNING ON FL TEACHING

Thinking about how to communicate effectively in written and spoken form for understanding goals, the debate task was given in the FL classroom. It is a partial activity of the cooperative learning for Teaching for Understanding concept orientation of MI theory. From unit title the students engage with the current events topics. The debate task builds a supportive classroom environment in the following. As listeners and readers, students analyze experiences, ideas, information, and issues presented by others. As speakers and writers, they use oral and written language from a variety of perspectives to help them understand their opinions and judgments on these performances, ideas, etc. While speaking and writing, the students interpret and apply the concepts of words or phrases to transmit information. To develop students understanding of the various aspects of listening and reading, students will collect data, facts, and ideas, and discover relationships, concepts, and generalization. To become an independent learner, students learn reading strategies. They understand how discussion with group members helps to improve their own skills by asking questions and answers to the questions. They learn how to speak in public and how to argue against specific points of view. The debate task develops knowledge
ACKNOWLEDGEMENTS
This study was funded by a Kaken-Houga Research Grant No. 17652063 from the Japan Society for the Promotion of Science (jsps). I would also like to thank Professor Kuwahara Noriaki and other anonymous reviewers of this International Workshop on Cognitive Prostheses and Assisted Communication 2006 for their helpful comments on this paper.
REFERENCES
[1] Budden, J. Multiple Intelligences. British Council, Barcelona. Teaching English. 2005. [2] Christison, M.A. Multiple Intelligences and Learning Styles in Second Language Teaching and Learning. TESOL Academy. Seattle, Washington, August 14-16, 1998. [3] Gardner, H. Multiple Intelligences: The Theory and Practice. New York: Basic Books. 1993. [4] Haley, M.H. Learner-Centered Instruction and the Theory of Multiple Intelligences With Second Language Learners.
50
Teachers College Record Volume 106, No.1, Jan.2004. pp.163-180. [5] Harvard Graduate School of Education. The Project Zero Classroom 2005: Views on Understanding. July 2005. Hayashi, K., The Influence of Cognitive Development in EFL/ESL(L2) Learning and Environmental Variables. Communication, Cognition, and Second Language Acquisition Journal., Belgium. 1997. [6] Hayashi, K., The Effectiveness of Collaborative Learning. The Japanese Learner. England: University of Oxford. 2001.
[7] Hayashi, K. Gaikokugogakusyu ni Eikyou o oyobosu Oya to Ko no Komyunikeisyon. Tokyo: Kazamasyobo.2004. [8] Hayashi, K. Oranda no Gaikokugokyouiku no Doukou. Paper presented at Symposium, Nihon Gengo Seisaku Gakkai. 2005. [9] Kagan, S.& Miguel K. Multiple Intelligences. 1998. [10] Kee de Bot. Foreign Language Teaching and Learning in the Netherlands. http://www.nlconference. org/docs.2004.
51

Promoting Virtual Social Interaction With Older People

Caricato da

Informazioni sul documento

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Promoting Virtual Social Interaction With Older People

Caricato da

Copyright:

Formati disponibili

InternationalWorkshoponCognitiveProsthesesand AssistedCommunication(CPAC2006)

kuwabara@atr.jp Shinji Abe

kuwahara@atr.jp Kiyoshi Yasuda

Categories and Subject Descriptors

NETWORKED INTERACTION THERAPY

Alternative Interaction Companions

Figure 1: Overview of Networked Interaction Therapy

3. APPLICATION SCENARIOS 3.1 Content Creation

reminiscence videos photos

care session records acquaintances

RDF-based personal data repository

Figure 2: Personal Data Repository for Networked Interaction Therapy

Figure 3: Annotation tool for creating reminiscence videos

3.1.2 Instructional content

20040716 foaf:Image dc:date rdf:type sample.jpg

imgReg : hasRegion rdf:type

rdf:type rel:grandchildOf foaf:gender

x1,y1,x2,y2 imgReg : imgReg : boundingBox regionDepict x1,y1,w1,h1

Figure 4: Example of photo annotation

Capturing the interaction and its analysis

3.2.2 Remote collaborative annotation

3.3 Content presentation and analysis of the care sessions

CONCLUSION AND FUTURE WORK

2. OVERVIEW OF OUR SYSTEMS 2.1 Im Here!

the user last met the person (Figure 3) (STEP-IV).

3. DISCUSSION OF SYSTEM DESIGNS 3.1 Distinguishing an Operator from a User

Controlling Inter-Personal Communication

3.2 Arranging Articles for Everyday Use

[1] http://www.ii.ist.i.kyoto-u.ac.jp/ sumi/pervasive04/.

Vision-based Behavior Detection for Monitoring and Assisting Memory-Impaired People

utsumi@atr.jp Shinji Abe

skawato@atr.jp Hironori Yamauchi

Categories and Subject Descriptors

ACTION ASSISTANCE USING HUMAN POSTURE OBSERVATIONS

2.1 Behavior Detection using IR Detection system

(Pi Qj ) (Pi Qj )/N. (3)

2.2 Example: Toilet Task Instruction using Behavior Detection

Figure 5: CG Character Display

Stand Open Cover Rotate

Figure 6: 3-D shape reconstruction (top: scene, bottom: reconstruction results)

Figure 4: System Conguration (Toilet)

STF STL STR SIF SIL SIR DR RE

STF 50.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

RE 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0

Figure 8: Recognized Sequential Behaviors

Detecting Body Motion

0 ((ci,t ci,t1 ) (ci,t ci,t1 ) < threshold) 1 (otherwise)

3.1 Vision-Based Behavior Extraction Face Orientation Tracking

3.2 Contents Control

Figure 9: Example of facial orientation estimation

Figure 10: Example: Extracted Beat Actions

Sound Signal 0-250Hz

Extracted Synchronous Beat

Figure 13: Synchronous beat behavior extraction

70 65 60 55 50 45 40 35 30 beat action random

Figure 11: Results of Attention Estimation

Figure 14: Distinguishing Beat Action and NonBeat Action

Avg. 442* 340* 2.2 1

Figure 15: TV Watching Time with/without Content Control

Developing an engaging multimedia activity system for people with dementia

Norman Alm2 Richard Dye2

Gary Gowans3 Phil Vaughan3

Categories and Subject Descriptors

Figure 1: Virtual botanic garden