Sei sulla pagina 1di 10

Autonomous Robots 11, 2938, 2001

c 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.




Tele-Existence Techniques of Heavy Work Vehicles


JUSSI SUOMELA AND AARNE HALME
Automation Technology Laboratory, Helsinki University of Technology, FIN-02015 HUT, Finland
jussi.suomela@hut.fi

Abstract. The problem of finding a feasible and technically simple tele-existence system for teleoperation of
heavy work vehicles is considered with the aid of two experimental test series. Several test persons of different ages
and experience were used in experiments in which a basic audio/video tele-existence system with head-tracking
was modified into different versions and tested. Conclusions indicate that details of the tasks and work environment
have an important role in finding the optimal solution.
Keywords: tele-existence, teleoperation, heavy work vehicles

1.

Introduction

Partially teleoperated semi-autonomous working


machines are no doubt the next step in the development
of automation in various worksites. Simple tasks can
be automated easily, whereas more sophisticated ones
need the hand of an experienced operator. When most
of the work cycle of working machines is automated,
one operator can easily handle from 2 to 5 machines,
assuming that all machines can be teleoperated from
one place. Teleoperation is also the most natural technical solution in cases where robotic machines are used
for individual and unique operations (Sheridan, 1995)
like surveillance and rescue work in disaster areas or
planet exploration.
When tasks are too difficult to be automated they are
probably very demanding to teleoperate, especially in
cases where large, heavy machines have to be driven
in narrow spaces or unknown areas.
A teleoperation interface can be made more natural
and easier for the operator by creating a feeling of presence, for example, by the use of stereo video cameras
that follow the movements of the users head, to give
a more natural visual perception of the environment.
Adding microphones and audio deepens still further
the feeling of presence, especially in cases where the
sounds are related to the work of the machine. Another

dimension by which the operation interface can be


further expanded is virtual reality. Objects related to
the environment or to the work being done can be represented in a virtual world overlaying the view of the
operator. Tele-existence is then realized by the aid of
augmented reality.
Augmented reality can be set up and utilized in teleoperation in a variety of ways, and the objects which
are used to augment the reality can be used for several
purposes. For example, artificial walls can be created to
warn or guide the operator in bad visibility; or the work
done with the machine may be conducted with the aid
of simulated guide marks, etc. In Halme et al. (1997)
the authors describe and demonstrate a system which
can assist the operator to create interactively a simplified augmented reality for an unknown environment.
The system utilizes cameras and a remotely aimable
laser range finder. Using the same basic idea with
the hardware and software developed further, we have
made a system by which complex human work, like
diagnosis or maintenance of complex machinery, can
be supported remotely by an expert sitting in his/her of A-system

fice. The ETAL


utilizes Internet technology
in communication. Part of the communication pathway can go via satellites enabling global location of the
distance unit. More details can be found in our web-site
http://www.automation.hut.fi

30

Suomela and Halme

2.

Tele-Existence Systems

The term tele-existence refers to the situation where


the main senses of an operator, like sight and hearing, are transferred to the remote place by means of
telecommunication so that he/she has the feeling of
presence. This feeling is supposed to help when executing teleoperation, because human senses are evolved
for local use rather than for supporting operations at a
distance. In addition to sight and hearing, other senses,
like feeling and balance, may be used. One of the pioneers in this field is professor Tachi of Tokyo University, who has constructed several notable systems
and published many interesting papers (see web-site
http://www.star.t.u-tokyo.ac.jp/).
A full tele-existence system with all trimmings is
technically demanding. The system easily becomes
complex and expensive. In the case of heavy working
vehicles, the price matters and so does the robustness
of the system. Depending on the work and the environment in which it is done, a simplified system is often
sufficient, and in some cases is even better than a more
complex one. It is not very clear, however, what the
main factors affecting are. In what follows, we try to
illustrate some related problems through a number of
field tests.

compressing. Especially in applications where wireless


communication is the only choice, the cost of the data
transmission system for long distances can be very
high. Complex servo systems for cameras should have
up to 7 DOF if all head and eye movements are to
be tracked and repeated. As the number of DOFs in the
system doesnt only increase the cost but also the fault
probability, the minimum number of DOFs (=2) which
still might be acceptable from the operators point of
view is preferred in practical applications. Deterioration of the visual information due to delays and too
small a bandwidth in communication is obvious. Augmenting the system with overlaid technique requires
graphical software and computer power.
With the above facts in mind, we made two series of
experimental tests to discover a feasible tele-existence
teleoperation system for the application in question. We
chose initially a system with full video/audio capability
and minimum delays in the control loops, but with 2
DOFs only in the servo head. The small number of
DOFs was chosen because the task was to teleoperate a
vehicle on more or less flat terrain. Several alternatives
to simplify this complete system were then made and
the effect of quality deterioration of task performance
was assessed systematically by taking into account the
capability of the additional tools and senses installed.

3.

4.

Problems of Tele-Existence Systems

In heavy work vehicle applications sight and hearing


are without doubt the most important senses for the
teleoperator. In certain applications, in construction or
forestry for example, force feeling might be also useful. Hearing, however, replaces in many cases direct
force feeling because an experienced operator can interpret machine noise as sense of power or force. As to
sight, full immersion in the work environment usually
requires stereovision and a head-tracker that turns the
camera head at the remote site in accordance with the
head motions of the operator. The vision may be displayed for the operator in several alternative ways using
an HMD (helmet) or just glasses and normal display
which generate stereo impressions.
The main two problems when applying augmented
tele-existence systems in practice are technical complexity and the need of broad bandwidth for data
transmission. Ergonomic problems are not minor and
the human interface aspects of such problems need
careful consideration. The basic requirementa realtime stereo imageneeds 2 120 Mbit/s without

Typical Task and Work Environment

The test site was an underground tunnel construction


site illustrated in Fig. 1. Actually the test site and task
were both characteristic of a typical mining environment. The task was a loading task with a relatively
long transportation part. The task consisted of driving
about 500-m to the loading place, loading the bucket,
driving back and emptying the bucket. The loading end
and the emptying end of the route were connected with
a vertical cleft which closed the loop. The 3D sketch
and the map of the test route are shown in Fig. 1. Most
of the route was located in a 5-m wide and 5-m high
rock tunnel.
The tasks performed by the operator during a single
mission were threefold: 1) loading the bucket, 2) transporting the bucket along the transportation route, 3)
emptying the bucket. The nature of the tasks was quite
different in each case. Loading required the operator
to load the bucket as full as possible to get maximum
payload for transportation. Although the vehicle speed
was almost zero, loading nevertheless needed accurate
simultaneous control of the bucket actuators and the

Tele-Existence Techniques of Heavy Work Vehicles

Figure 1.

Underground test route: 3D sketch and freehand map.

vehicle. Transportation required the speed to be kept


as high as possible to reduce the transportation time
while performing the difficult task of driving safely
along narrow and winding parts of the route. Emptying
the bucket involved the difficult task of parking as close
as possible the dumping shaft.

Figure 2.

31

Test vehicle for first series of experiments.

5.

Experiments with Test-Bed Vehicle

Simplification of the tele-existence system was first


tested with the aid of experiments done with a smallscale outdoor vehicle, shown in Fig. 2. The vehicle
was a standard ATV which was fitted with instruments

32

Suomela and Halme

the operator can get more precise information about


the geometry of the environment. A 3D virtual world
software environment was used to create an interactive
overlaid augmented interface for the operator. The laser
pointer was used at this interface.
The tele-existence hardware was configured to five
different systems with different respective enhancement levels in vision and camera control:
A: stereovision, sound, and 2DOF head-tracking
B: monovision, sound, and 2DOF head-tracking
C: image on screen, sound, and 2DOF head-tracking
D: image on screen, sound, manual camera control
E: image on screen, sound, fixed camera
Sound was present in each alternative because omitting it was observed to deteriorate the system radically.
Using the laser pointer, the user was able to measure
distances up to 23 m with the optical axis of the
cameras. This feature was used to provide independent depth information for the driver. The pointer could
also be used as a tool when virtual reality objects were
defined to the augmented system.
The experiments included tasks, which simulated
real world driving tasks in material handling and transportation. The test tasks were the following:
Figure 3.

Operator station for first series of experiments.

and computerized. The test route was established in


an open field. More about the vehicle can be found in
Schonberg et al. (1994). The experiments were planned
with more generic tasks in mind than just the mining
one. The teleoperation system was implemented with
a steering wheel and pedal combination providing the
feeling of normal car driving. The operator station is
shown in Fig. 3. Communication between control station and ATV was made with one pair of half-duplex
radio modems. The tele-existence equipment included
a stereo-HMD, a head-tracker, two monitors, two cameras, a laser pointer, a 2 DOF servo-head, two pairs of
short range video links, a half duplex radio (for noise
transmission), and pair of radio modems for transmitting and receiving the head-tracking data from tracker
to servo-head.
The system used for driving tests was a simplified
version of the full tele-existence system developed previously in the laboratory (Halme et al., 1997). The laser
range finder, which was installed between the cameras,
was used as a pointer and a measurement stick by which

Corridor driving: Driving on defined routes replicating driving on the road, ore transportation in mine
tunnels, etc.
The test was conducted by following winding
path, which included narrow gates. Driving over
the path border and colliding with obstacles were
counted as errors.
Unknown terrain driving: Driving on an unknown area
where there were many obstacles and no specific
route, a task typical for forestry machines.
The test was conducted after driving to an unknown forest area where the operator had to follow
a natural path after perceiving it.
Loading tasks: The vehicle had to take a load into its
manipulator and move during the loading, a task typical for different kinds of loaders. The test involved
pushing boxes from one line to another one with the
aid of a beak assembled on the front of the vehicle.
Maneuvering tasks: Maneuvering in confined places,
a task typical for forestry and loading machines.
The test was conducted on a slalom track, where
the driver was required to dodge piles, stop on a line
and park in a given slot.

Tele-Existence Techniques of Heavy Work Vehicles

Fast driving: Driving at speeds of more than 5 m /s,


this is typical for transporting tasks.
The test required the operator to drive fast and to
stop the vehicle in a given position in an open field.
Off-road driving: Driving in areas where both obstacles and surface conditions could stop the vehicle.
The test required the operator to drive over uneven ground and cross obstacles such as ditches and
stones.
Five different men aged 2640 were used as test
operators. Two of them were classified as experienced,
and the others as amateurs. In the case of each system
the following properties were evaluated:

Easy to drive with continuous motion


Easy to drive accurately
Easy to navigate
Perception of obstacles and unexpected objects in
the environment
Possible ergonomic drawbacks
The results of each test were evaluated by measuring
overall execution time and the number of errors during
the test. Verbal assessments concerning the properties
of the system were obtained from the operators. When
repeating a certain test several times the effect of learning could be clearly seen. This effect was independent
of the system used. In order to eliminate this effect,
only the best results obtained after a training period
were taken into account in further evaluation.
The test results are reported in detail in Halme et al.
(1997). After making about 80 test drives with 5 different operators the overall conclusion was that no unique
answer can be given to the question of optimal choice
among the systems AE. It depends on the details of
the task, and the fact that learning has very important
role in tasks, which are repeatable in nature. Our overall
experiences can be summarized as follows:
There were not many differences between the systems after a repeatable task was learned well.
The head-tracking camera helped many tasks including new and changing situations.
For each system, training improved results.
The most difficult of all test tasks was loading, which
demands accurate estimation of distances and fine
control of actuators.
An important conclusion was that head-tracking helps
and is a good solution especially if the task includes

33

unknown and changing elements. Whether it should


be implemented with a stereo-HMD or just a single
mono-monitor is a more difficult question. It seems that
ergonomy is better in the latter case, but tasks which
demand ultra high accuracy and distance estimation can
be done better with the stereo system. A laser range
finder installed on the camera system can help solve
this problem.
6.

Tests with Full-Scale Loader

Experiments were continued using a full-scale loader


in cooperation with the manufacturing company. The
companys test site, containing the tunnel construction
illustrated in Chapter 4, provided the test environment.
The equipment used in the test consisted of a commercial front-end loader and tele-existence equipment.
The overall system is shown in Fig. 4.
6.1.

Test Vehicle

The loader used in the tests weighed 40 tons and had a


loading capacity of about 5 m3 . The vehicle was diesel
powered, and had mechanical power transmission with
hydrodynamic torque converter. The steering was the
articulated type, i.e., with a frame divided into two
parts of about same size connected about the steering
joint. The bucket had two degrees of freedom: lift and
tilt.
6.2.

Loader Control Equipment

The loader was equipped for full teleoperation. All driving actions required could be performed remotely from
a remote control station.
Remote Control Station. The actual user interface in
the remote control station was the control chair with a
steering wheel and pedals shown in Fig. 5. The steering
could be controlled with either the steering wheel or the
joystick on the left arm of the chair, while the throttle
and brake were controlled respectively with one of two
pedals. Engine start, gears, camera, etc. were controlled
with the buttons, switches and another joystick fitted
to the other chair arm. The feedback data was shown
in the control PCs monitor. The video image from the
vehicle was shown in a separate monitor. Also sound
from the vehicle was available. The control PC read the
data from control devices (pedals, joysticks, buttons,
etc.) and sent it to the vehicle.

34

Suomela and Halme

Figure 4.

Test equipment for a full-scale loader.

Data Transmission. The control data was transmitted by means of a pair of radio modems between the
vehicle and the control station. A leaky feeder cable system was used in order to cover the whole test
area.
Image and Sound Transmission. Two video channels and one sound channel were required for the tests.
The leaky feeder system used didnt support video frequencies so two analog video-links with 2,4 GHz frequency were used. To ensure the connection in the
whole test route, 3 pairs of video receivers were located along the route. The right receiver was chosen
manually during an initial run.

6.3.

Tele-Existence Equipment

The tele-existence equipment was basically the same


as that used in the previous tests. Only the servoed
camera head was redesigned to obtain better tracking
properties. Details of the equipment can be found in
Suomela et al. (1999).
6.4.

Test Persons

The following persons were chosen for the tests:


A (age 40): A professional loader driver of 20 years
experience, and with few years experience in loader

Tele-Existence Techniques of Heavy Work Vehicles

Figure 5.

Control chair, steering wheel and pedals.

teleoperation. He had driven several thousands of


buckets with a system similar to the test system with
the fixed camera configuration.
B (age 43): A professional loader driver with several
years experience, and experience of other types of
workmachines. He hadnt any teleoperation experience before the tests.
C (age 35): A professional loader driver, with no teleoperation experience.
D (age 30): A research engineer. He had no experience
of loaders, although he had teleoperated the ATV
(vehicle used in the first experiments) in all teleexistence configurations. He was an amateur pilot
with extensive experience of video and computer
games.
6.5.

35

Tele-Existence Configurations

The described tele-existence system was configured to


four different enhancement levels:
Fixed cameras and monitor: The image came from
the fixed cameras pointing straightforward and backward. Cameras were located on the centerline of the

vehicle. The front camera was fixed to the front part


and the back camera to the back part of the vehicle. The image was shown on a monitor ahead of
the driver, and was automatically switched between
the front and the back camera depending on gear
position. It was also possible to look in the opposite direction by pressing a button on the arm of the
operating chair.
Servo camera and monitor: The image came from the
right camera of the servo head located on the top of
the cabin about 1 m from the vehicle centerline. The
operator controlled the camera movements with the
ADL head-tracker, while the image was shown in a
monitor as in the previous configuration. The main
pointing direction of the servo head could be rotated
180 by a button in chair arm.
Servo camera and mono HMD: As in the previous configuration, but with the HMD replacing the monitor.
The image from the right camera is divided to the
both displays in the HMD.
Servo camera and stereo HMD: As in the previous configuration, but both cameras were used. The images
from the cameras were transferred to both displays
of the HMD to create a stereo image.

36

Suomela and Halme

Figure 6. Tele-existence aided driving. Driver (front) is driving by using servo camera and normal monitor. Virtual passenger (back) is looking
at the stereo image from servo cameras with HMD.

The sound feedback was in all configurations. The


operator station is illustrated in Fig. 6.

6.6.

Test and Evaluation Methods

The test was conducted in two phases. In the first,


the drivers drove the test route three times using each
of the three steering configurations, but only the first
tele-existence configuration. In the second, the drivers
used the steering configuration they liked most, and all
three tele-existence configurations. The evaluation was
based on the method suggested by Schloerb (1995) for
tele-existence systems. In this method, objective evaluation is made by concluding how well the defined task
is performed, while subjective evaluation is based on
the operators feeling how good the presence is.
In the objective evaluation, performance was evaluated on the basis of the recorded performance data. In
the subjective evaluation, the quality of the presence

was evaluated on the basis of comments of the drivers


both during driving, and following each run, when each
driver was interviewed.
In the objective evaluation, the following data were
detected: 1) Time of runs: The time of each run was
measured, and the e-stop interrupts subtracted. The
problem remained that the amount of the test runs didnt
allow a proper statistical analysis. 2) Errors: All the errors, which were mostly hits to the walls or emergency
braking by the safetyman, were recorded 3) Logging
the operator driving data: The steering and throttle
movements made by the operator were logged with
100 Hz frequency. The data recorded shows in the time
phase and, especially, in the frequency phase, very well
how nervous the driving was. 4) Logging the vehicle
positional data: The test vehicle also had navigation
equipment. Vehicles 2D position was calculated from
optical gyro and velocity. Speed, motor speed, angle of
the middle joint, time and angular speed were measured
and logged.

Tele-Existence Techniques of Heavy Work Vehicles

6.7.

Summary of Results

The test results have been reported in detail in Suomela


et al. (1999). The results were partly a surprise, especially when compared to those of the preliminary tests.
The servo cameras controlled by head movements were
considered very bad by all drivers. The objective evaluation from the driving data also supported this. The only
advantage in the servo camera configurations was the
stereovision. It created real tele-existence and helped
the driver to estimate distances. This helped in loading
especially. The servo cameras were located away from
the centerline of the vehicle in the position where the
driver sits when driving on board. This was probably
a mistake. When the view was not along the centerline the teleoperator had difficulties in driving along
tunnels, and in repositioning the camera back to the
driving position again after a glance elsewhere. However, even if the control had been perfect, the results
wouldnt have changed much. The drivers reported that
it would have been useful to have been able to control
the cameras in some cases, but that most of the time they
should be fixed. Also, control with the head was judged
unergonomic. This result was the opposite of what was
obtained in the preliminary tests. The only rational explanation lies in the different sensing environment. The
test route of the preliminary tests was made in an open
field where the operator followed a path rather than a
tunnel. In a tunnel the walls cause an optical flow that
possibly helps driving with minimal head movements.
The other problem was the HMD, which was
considered uncomforted and caused nausea in the
drivers. This was the same result as in the preliminary
tests. The results clearly indicate that permanent working with a HMD is not recommended. For the operator
it is better to be in real life most of the time and
receive visual information from a monitor. The HMD
can be used for short times when its really needed. The
stereo effect can also be created on a monitor.
7.

37

that when driving on an unfamiliar route without the


tunneling effect the servo-controlled cameras have
advantages. The vertical walls in the underground site
make a video game-like driving environment which,
contrary to outdoor applications, doesnt favor headtracking. In addition to this, the working environment
in mining is relative static and most of the time was very
familiar to the drivers. Learning has a strong effect and
should be taking into account when developing practical applications of tele-existence for such work environments. Both these facts disagree against the level
of advancement needed in the field of tele-existence.
This doesnt mean, however, that in other applications
the case would be the same. The preliminary tests conducted in a slightly different environment support this
assumption. Although not tested experimentally yet,
we could say that in forestry, for example, or in construction site application, the case might be different.
References
Halme, A. and Rintala, N. 1997. An interactive tele-existence system augmented with VR-models of the working environment.
ISMCR97 Topical Workshop on Virtual Reality and Advanced
Man-Machine Interfaces, Tampere 45 June, 1997.
Halme, A., Rintala, N., Savela, M., and Suomela, J. 1997. Applying
telepresence and augmented reality techniques in teleoperation of
mobile field robots. FSR97, Canberra, Australia, 8. 10.12.1997
(see also Robotics and Autonomous Systems Nro 26, pp. 117125,
1999).
Schloerb, D.W. 1995. A quantitative measure of telepresence.
Presence, 4(1):6480.
Sheridan, T.B. 1995. Teleoperation, telerobotics and telepresence: A
progress report. Control Engineering Practice, 3(2):205214.
Schonberg, T., Ojala, M., Koskimaki, E., Suomela, J., and Halme,
A. 1994. A small scaled autonomous test vehicle for developing
autonomous off-road applications. International Conference on
Machine Automation, Tampere, February 1994.
Suomela, J., Savela, M., and Halme, A. 1999. Tele-existence technics
of different enhancement degrees in front end loader teleoperation.
FSR99, Pittsburgh, USA, 2931, August 1999.
Tachi, S., Arai, H., and Maeda, T. 1989. Development of anthromorphic tele-existence slave robot. In Proceedings of the International
Conference on Advanced Mechatronics, Tokyo, 1989.

Conclusions

The results from the two completed and one partly


completed sets of experiments indicate that the optimal
system for tele-existence in teleoperation of workmachines depends very much on the tasks done with the
machines and the details of the environment where the
machine is working. It is quite clear that a fixed camera
is enough for teleoperation of the loader in most cases.
In open field tests (Halme et al., 1997) it was noticed

Jussi Suomela received his MSc in Electrical engineering from


Helsinki University of Technology (HUT) 1992. From 1992 he has
worked as research scientist and project manager in several mobile

38

Suomela and Halme

robotics projects in Automation Laboratory of HUT. At the moment


he is doing his doctoral thesis concerning teleoperation of semiautonomous work machines.

Aarne Halme received his Ph. D. of Technology with honors from


the Helsinki University of Technology in 1972. He subsequently

worked as associate professor of control engineering (19721977)


at the Tampere University of Technology and then as professor of
control and systems engineering (19771985) at the University of
Oulu. Since 1985 he has been Professor of Automation Technology in the Department of Automation and Systems Engineering at
the Helsinki University of Technology. He has published extensively
(more than 230 publications) in the areas of control, computer aided
design, and robotics and his current research interests include service and field robotics, robot societies, sensor fusion and control of
biotechnological processes.