Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Computers in Industry
journal homepage: www.elsevier.com/locate/compind
Department of Industrial Engineering, Chosun University, 375 Seosuk-dong, Dong-gu, Gwangju 501-759, South Korea
Department of Industrial Engineering, Chonnam National University, 300 Yongbong-dong, Buk-gu, Gwangju 500-757, South Korea
A R T I C L E I N F O
A B S T R A C T
Article history:
Received 3 December 2007
Received in revised form 4 July 2008
Accepted 6 September 2008
Available online 17 December 2008
Proposed in this paper is a novel approach to virtual prototyping of digital handheld products using
augmented reality (AR)-based tangible interaction and functional behavior simulation. For tangible user
interaction in an AR environment, we use two types of tangible objects: one is for a product, and the other
is for a pointer. The user can create input events by touching specied regions of the product-type
tangible object with the pointer-type tangible object. Rapid prototyping and paper-based modeling are
adopted to fabricate the AR-based tangible objects which play an important role in improving the
accuracy and tangibility of user interaction. For functional behavior simulation, we adopt a state
transition methodology to capture the functional behavior of the product into an information model, and
build a nite state machine (FSM) to control the transition between states of the product based on the
information model. The FSM is combined with the AR-based tangible objects whose operations in the AR
environment facilitate the tangible interaction, realistic visualization and functional simulation of a
digital handheld product. Based on the proposed approach, a prototyping system has been developed and
applied for the design evaluation of various digital handheld products with encouraging feedback from
users.
2008 Elsevier B.V. All rights reserved.
Keywords:
Virtual prototyping
Tangible objects
Augmented reality
User interaction
Functional behavior simulation
1. Introduction
For most digital handheld products such as a mobile phone and
an MP3 player, their functional behavior is very complicated and
nearly all expressed as humanmachine interaction (HMI) tasks,
each of which may trigger the transition between the states of the
products. For successful entry of a new product into the
competitive world market, it is imperative to reduce time to
market as much as possible while precisely converting its demands
into actual product forms, features, and functions [1,2]. An
essential activity required is the efcient and extensive use of
prototypes during the product development process [1,2].
With recent advances in computer technology, virtual prototyping (VP) has been considered as a new and powerful
prototyping solution to overcome the shortcomings of conventional prototyping methods. The concept of VP has been widely
employed and implemented in many industrial elds including
automotive and airplane industries [7,8], but most works have
been based on using virtual reality (VR) techniques [310], and
they have been focused on visualization [2,9], assembly and
* Corresponding author. Tel.: +82 62 230 7039; fax: +82 62 230 7128.
E-mail address: hzpark@chosun.ac.kr (H. Park).
0166-3615/$ see front matter 2008 Elsevier B.V. All rights reserved.
doi:10.1016/j.compind.2008.09.001
environment can improve the accuracy and tangibility of interaction with the products. Rapid prototyping is a manufacturing
technology to generate physical objects so-called RP models
directly from geometric data without traditional tools easily and
rapidly [2,14]. An RP model usually serves the purpose of
communicating information and demonstrating ideas. It can also
support various kinds of tangibility for experiments and interactions which gives rapid and critical feedback to the product
development and evaluation.
The tangible objects, composed of paper and RP models without
any hardwired connection using electronic components, are easily
available at low cost. This makes the AR environment more
accessible to developers, stakeholders, and even consumers.
Moreover, the proposed approach suggests how to combine the
forms, functions, and interactions of digital handheld products
physically and virtually at the same time.
The rest of the paper is organized as follows: Section 2
summarizes previous work related to virtual prototyping. In
Section 3, the proposed approach is described with its key
components. Section 4 explains the operations of the virtual
product model in a tangible AR environment. Section 5 addresses
the implementation and application of the product design
evaluation system based on the proposed approach. Section 6
describes a preliminary user study to show the usefulness of the
approach. Section 7 closes the paper with some concluding
remarks and future work to be done.
2. Previous work
Early attempts at supporting VP were based on CAD and VR
systems. Powerful tools including stereoscopic display systems,
head mounted displays (HMD), data gloves and haptic devices have
been introduced [9] and combined to construct VP systems that
provide realistic display of products in a simulated environment
and offer various interaction and evaluation means. Bochenek et al.
compared the performance of four different VR displays in a design
review setting and mentioned that the best approach for design
review activities could be a combined technology approach [24].
Park et al. suggested virtual prototyping of consumer electronic
products by embedding HMI functional simulation into VR
techniques for design evaluation [15,16]. As it is not easy to build
a virtual environment of ne quality and to acquire tangible
interaction with VR-based systems, many alternative solutions
have been proposed.
Greenberg and Fitchett presented toolkits called Phigets that
allow designers to explore a tangible user interface (TUI) for
interactive product design [25]. Hartmann et al. presented similar
toolkits called d.tools for visually prototyping physical user
interfaces [26]. In TUI, physical objects and ambient spaces are
used to interact with digital information [27]. Hardwired connection
is often employed using electronic components. Tangible interfaces
are quite useful because the physical objects used in them have
properties and physical constraints that restrict how they can be
manipulated. However, it is difcult to change and evaluate an
objects physical properties dynamically. The human computer
interfaces and interaction metaphors originating from AR research
have proven advantageous for a variety of applications [17,18]. AR
techniques can naturally complement physical objects by providing
an intuitive interface to a three-dimensional information space
embedded within physical reality. However, although an AR
interface provides a natural environment for viewing spatial data,
it is often challenging to interact with and change the virtual content.
To overcome the limitations of the AR and TUI approaches while
retaining their benets, tangible AR has been suggested [19,20].
Verlinden et al. suggested the concept of augmented prototyping
115
116
117
Fig. 5. AR-based tangible object for a game phone: (a) the RP model with an AR marker and (b) the augmented image of the game phone.
118
Fig. 6. AR-based tangible object for a pointer: (a) an unfolded sheet for a paper model; (b) the paper model with four AR markers; (c) the augmented image of the pointer.
The distance from the tip to the button or slider is the shortest
among the distances from the tip to the other buttons and sliders.
The distance is kept smaller than a tolerance during a specied
time period.
119
Fig. 8. Distance computation in the AR environment: (a) coordinate transformation between a camera and an AR marker; (b) distance between two points using the camera
coordinate frame.
120
we carried out a preliminary user study of the MP3 player and the
game phone with a subject group consisting of 10 university
students. Of the 10, 8 learned the basics of 3D geometric modeling
from CAD/CAM courses. Simple task performance measures and
questionnaires were used to evaluate the approaches using four
different virtual prototypes: traditional 2D screen prototypes
(2DSCR), 3D stereoscopic prototypes (3DSTR), 3D augmented
prototypes (3DAR), and 3D tangible augmented prototypes
(3DTAR). As we have built the virtual prototypes from commercial
products based on reverse engineering, we could include the use of
real products (REAL) in the tests as the target reference of the
prototyping approaches. Obviously, using a real product is the best
but most costly approach to its design evaluation.
As shown in Fig. 12, 2D screen prototypes present front and/or
side views of products, but the image of the views usually has a
limited resolution. 2D screen prototypes were built by combining
the images of views with multimedia contents data, functional
models, and FSMs. These 2D screen prototypes allow users to
perform the image-based functional simulation by clicking buttons
on the views. 3DTAR corresponds to the approach proposed in this
paper and 3DSTR corresponds to the VR-based prototyping
approach proposed by Park et al. [15,16]. Similar to the 3DTAR
approach, each virtual model in 3DSTR also consists of a product
model, multimedia contents data, a functional behavior model, and
an FSM.
Fig. 10. MP3 player in four different states: (a) MP3 Play; (b) Mode Select; (c) FM Radio; (d) Hold; and (e) its tangible augmented prototyping.
Fig. 11. Game phone in four different states: (a) On; (b) Calling; (c) Multimedia menu; (d) Movie; and (e) its tangible augmented prototyping.
Fig. 12. 2D screen prototypes for (a) the MP3 player and (b) the game phone.
121
122
Fig. 13. VR-based prototyping of (a) the MP3 player and (b) the game phone.
tasks. Before performing each task, the subject could have access to
a simple graphical manual describing the steps required to
complete the task. The results of the performance measures are
summarized in Table 1 and plotted in Fig. 15.
After completing all the tasks, each subject was asked to ll
questionnaires in order to capture qualitative aspects (i.e. understandability of functions, ease-of-use, tangibility, sense of realism,
legibility) of his or her experience of four prototyping approaches
(2DSCR, 3DSTR, 3DAR, 3DTAR). Verlinden et al. and Park et al.
performed similar qualitative comparison of their virtual prototyping approaches [16,33]. The questions asked are summarized in
Table 2. All responses were scored on a ve-point scale and each
question included a eld to add some comments. Fig. 16 shows the
analysis results of the questionnaires collected from all the
subjects.
From the results of task performance, we found that the
overall ranking of task performance was as follows:
REAL > 3DSTR > 2DSCR > 3DTAR > 3DAR. We found that the ease
of button clicks is the most dominant factor affecting the task
performance. Button clicks are done with some keystrokes and the
mouse in 2DSCR and 3DSTR, with simple AR markers in 3DAR, and
with AR-based tangible objects in 3DTAR. Using the keystrokes and
the mouse was faster and easier than using the AR-based markers
or tangible objects. Especially, all the subjects felt severe
inconvenience in clicking buttons with traditional AR markers.
When they tried to click buttons with simple AR markers, they
often encountered confusing situations in which 3D virtual objects
overlapped each other (i.e. the pointer object penetrated the
product object). This can explain why the 3DAR approach required
much more time to complete the tasks than the others. On the
other hand, the subjects mentioned that they did not experience
any object overlap and penetration and felt the sense of touch
when using tangible objects in the 3DTAR approach.
Fig. 14. AR-based prototyping: (a) simple AR markers; (b) manipulation of the MP3 player; and (c) manipulation of the game phone.
123
Table 1
Task descriptions and performance measures.
Product type
Task
Task steps
MP3 Player
1.
2.
3.
4.
5.
1.
2.
3.
4.
5.
1.
2.
3.
1.
2.
3.
4.
1.
2.
3.
4.
Game Phone
3D AR
REAL
6.5
5.9
3D STR
3D TAR
9.2
42.4
5.3
8.8
8.1
13.0
45.8
7.5
10.0
7.7
13.3
47.8
6.4
9.6
7.3
10.3
42.4
6.7
9.0
6.6
8.8
35.2
5.6
some buttons are small and compact. Moreover, the visibility of the
2D screen prototype in either case is not good. Some subjects
commented that they felt inconvenience and made mistakes when
clicking buttons with the mouse pointer in the 2DSCR approach. In
the 3DSTR approach, subjects can go closely to the prototypes to
see them in ner detail. This helps to make the subjects click
buttons more easily and accurately. As Kuutti et al. pointed out
[32], it might be unnatural to go more closely to an object than
where the eye can be focused. Nonetheless, it must be one of
advantages of using 3D virtual prototypes to manipulate (move,
scale, and rotate) freely them in a VR environment. In the AR-based
approaches, subjects can also have a close look at the prototypes as
long as the AR markers associated with the prototypes are captured
and identied. The visibility of prototypes in the AR-based
approaches is not better than the one in the 3DSTR approach
since the resolution of PC camera is lower than that of HMD. On the
other hand, some subjects without experience of using HMD
commented that in the 3DSTR approach they had some inconvenience in visibility of 3D prototypes and felt some unnatural
weight gain in their heads while wearing the HMD.
From the analysis results of the questionnaires, we found the
advantages of 3D-based approaches over the 2D-based approach
in most aspects, the signicant advantages of 3DTAR over 3DAR,
and some trade offs between 3DSTR and 3DTAR. As the same HMI
functional models were integrated into all the four kinds of
prototypes, there were no signicant differences between the
four approaches in understandability of the product functions.
The 3D-based (3DSTR, 3DAR, 3DTAR) approaches gave the
subjects better sense of realism than the 2DSCR approach.
Table 2
Questionnaires contents (translated from Korean).
Fig. 15. Graphical plots of task performance measures of (a) AR-based approaches
and (b) four approaches.
Q1
Q2
Q3
Q4
Q5
Q6
Q7
Q8
Q9
Can you understand the product functions by using the virtual prototype?
Is it easy to click buttons when using the virtual prototype?
Does the virtual prototype make you feel as if you push real buttons?
Does the virtual prototype look like the real product?
Can you gure out the size of the product with the virtual prototype?
Can you feel a three-dimensional effect when using the virtual prototype?
Is the liquid crystal display (LCD) of the product legible?
Do you think the use of virtual prototype interesting?
Does the virtual prototype offer enough information for a decision to buy the
product?
124
125
[22] T.J. Nam, Sketch-based rapid prototyping platform for hardwaresoftware integrated interactive products, in: Proceedings of conference on human factors in
computing systems (CHI), 2005, pp. 16891692.
[23] W. Lee, J. Park, Augmented foam: touchable and graspable augmented reality for
product design simulation, Bulletin of Japanese Society for the Design Science 52
(2006) 1726.
[24] G.M. Bochenek, J.M. Ragusa, L.C. Malone, Integrating virtual 3-D display systems
into product design reviews: some insights from empirical testing, International
Journal Technology Management 21 (2001) 340352.
[25] S. Greenberg, C. Fitchett, Phidgets: easy development of physical interfaces
through physical widgets, in: Proceedings of the ACM UIST, 2001, pp. 209218.
[26] B. Hartmann, S.R. Klemmer, M. Bernstein, N. Mehta, d.tools: visually prototyping
physical UIs through statecharts, in: Proceedings of ACM Symposium on User
Interface Software and Technology (UIST), 2005.
[27] H. Ishii, B. Ullmer, Tangible bits: towards seamless interfaces between people, bits
and atoms, in: Proceedings of Conference of Human Factors in Computing
Systems, 1997, pp. 234241.
[28] T. Varady, R. Martin, J. Cox, Reverse engineering of geometric modelsan introduction, Computer-Aided Design 29 (1997) 255268.
[29] J. Rumbaugh, State trees as structured nite state machines for user interfaces, in:
Proceedings of ACM SIGGRAPH Symposium on User Interface Software, 1988, pp.
1529.
[30] D. Harel, Statecharts: a visual formalism for complex systems, Science of Computer Programming 8 (1987) 231274.
[31] H. Feng, Dcharts, a formalism for modeling and simulation-based design of
reactive software systems, Master Thesis, McGill University, 2004.
[32] K. Kuutti, K. Battarbee, S. Sade, T. Mattelmaki, T. Keinonen, T. Teirikko, A. Tornberg,
Virtual prototypes in usability testing, in: Proceedings of the 34th Hawaii International Conference on System Sciences, 2001, pp. 17.
[33] J. Verlinden, W. van den Esker, L. Wind, I. Horvath, Qualitative comparison of
virtual and augmented prototyping of handheld products, in: Proceedings of
International Design Conference, 2004, pp. 533538.
Hyungjun Park is an associate professor at the
Department of Industrial Engineering, Chosun University, Korea. He received his BS, MS, and PhD degrees in
Industrial Engineering from Pohang University of
Science and Technology (POSTECH), Korea, in 1991,
1993, and 1996, respectively. From 1996 to 2001, he
worked as a senior researcher at Samsung Electronics,
Korea. He involved in developing commercial CAD/CAM
software and in-house software for modeling and
manufacturing aspheric lenses used in various optical
products. Since 2001, he has been a faculty member of
Chosun University. His current research interests
include geometric modeling, virtual prototyping of
engineered products, 3D shape reconstruction using reverse engineering, biomedical engineering, and CAD/CAM/CG applications.
Hee-Cheol Moon received his BS and MS degrees in
Industrial Engineering from Chosun University, Korea,
in 2005 and 2007, respectively. He is currently a PhD
student at Chosun University. His main research topic is
virtual prototyping of portable electronic products
using augmented reality and CAD/CAM techniques.