Robotics for Artificial Intelligence – KIMROB03
ALICE - A Domestic Service Robot
Group n. 04 — Parth Tiwary and Danny Rogaar
Abstract— A domestic service robot is tasked with retrieving
known objects from fixed locations. We implement the overall
behaviour, using a navigation algorithm, object recognition and
grasping methods. Experiments show the robots capability in
navigation and object recognition. Due to a flaw in object
recognition, we were unable to test the robots grasping ability
on the platform, although simulations have shown the robot to
be able to grasp targeted objects.
I. INTRODUCTION
A domestic service robot is primarily used for performing
household chores for assisted living, domestic cleaning,
and surveillance. It is a multidisciplinary field and requires
a sophisticated integration of all the involved disciplines,
whether computer vision, navigation or grasping. With re-
cent advances in recognition and grasping of an array of
household objects and the increased accuracy in such tasks
[Hodson, 2018], there is a rise in demand for service robots.
We work with an example of one such domestic service robot
in this project.
In this paper, we elaborate on our implementation of domes-
tic service robot ALICE, which is three folds: navigation,
object recognition and grasping. We discuss each of these
topics and our implementation in-depth in the following
sections. In navigation we implement Dijkstra path planning
algorithm ([Dijkstra, 1959]) for finding the path between two
waypoints, our implementation uses heapq data structure for
improving the runtime of the algorithm and we use push/pop
mechanisms using heappush and heappop. In object recog-
nition, we use a convolutional neural network for classifying
objects into five pre-defined classes, we use two convolutions
each followed by a max pool layer and two fully connected
layers sandwiched in between. For grasping we implement
an action server, details on which will be discussed in the
following sections.
In navigation, we implement a hierarchical structure for
navigation. The implementation is two folds and addresses
once core issue faced during the implementation of final
demo behaviour. First, we pre-define a set of waypoints for
the final demo as shown in figure 1. Then, whenever a start
and goal state is given to the robot, we use Dijkstra path
planning algorithm to obtain an optimal set of waypoints.
Once we have obtained the optimal set of waypoints, we
provide these waypoints to our navigation sub-behaviour
which further invokes Dijkstra path planning algorithm to
plan a path between each of these waypoints in order to
The authors are with Faculty of Science and
gineering, University of Groningen, The Netherlands.
{p.p.tiwary,p.d.rogaar}@student.rug.nl
travel from start to goal state. Hierarchical navigation is a
crucial and more sophisticated alternative for hardcoding sets
of waypoints for each possible route in the final demo.
In the following sections, we further elaborate on imple-
mentation of all the three components of the service robot:
navigation, object recognition and grasping, and in the ex-
periments section we put up a discussion on why some
components didn’t work as expected during the final demo
and a reasonable explanation.
A. Neural object recognition
Object recognition has been an incredibly successful area
of research since Krizhevsky showed the performance us-
ing neural networks [Krizhevsky et al., 2012]. For image
processing using such architectures, typically convolutional
neural networks are applied.
Basic fully connected networks are centered around the
artificial neuron. The artificial neuron is basically a weighted
sum of inputs minus a certain threshold. The output can then
be given to an activation function that manipulates the linear
output of the neuron. For example, linear activation means
using the output of the weighted sum directly. Rectified
Linear Unit output is pretty much a linear output. However,
a fully linear network has no better potential than a shallow
version of it. ReLU avoids being fully linear by truncating
the output below zero input [Glorot et al., 2011].
Again, images are better processed using convolutional neu-
ral networks, as images are only locally connected which is
exploited by the technique. 2D convolutions apply a kernel
which forms its weights, onto the input image at all possible
locations. Since the kernel weights are multiplied with the
current piece of the image, the operation can be seen as
applying a fully connected network to a small part of the
image, at all possible parts, which is much more efficient
than having a single giant fully connected layer.
The output is a feature map that represents different patterns
at a scale depending on the number of convolutions before.
That is, early feature maps are only able to learn simple
patterns such as edges using only one convolution. Later
feature maps may, however, be able to learn complete classes.
Given that deeper layers are able to learn more complex
patterns, the depth of a network is related to the complexity
of the given problem. I.e. with few classes, only basic
features may be good enough for the solution
Max-pooling can be used to subsample feature maps/images
by choosing only the highest value in a small part of the
En-
image. This introduces some invariance to location which is
useful for classification but less so for localisation.
Gradient descent for parameter optimisation means adjusting
the model parameters in a way that makes a predefined
loss function to go down, by using the network function
derivative.
II. METHOD
A. Navigation
For navigating through the environment, ALICE uses
a custom implementation of Dijkstra algorithm for path
planning, making sure navigation from one point to another
is accomplished through the shortest path. For the robot, the
shortest path often means barely scraping any obstacles in
the way. The robot then slows down not to carefully navi-
gate around such encounters. Luckily, intermediate locations
dubbed waypoints can be set so that path planning happens
between the waypoints and not the immediate robot and
its target. This can steer the path of the robot around any
obstacles such as tables and closets, in turn preventing slow
movement for having to be more careful. The input for the
navigation is thus set to be a list of waypoints which are
followed in order.
Implementation
As for the specifics of the implementation, a flattened 1D
ros array is provided as the costmap, which we use as it
is. Every index in the 1D costmap array corresponds to a
node in the typical graph based approach to the algorithm.
For the extraction of neighbours of a given node in the
1D array, appropriate offsets were used to index actual
neighbours in 2D costmap onto 1D. This approach for the
extraction of neighbours for a given node is implemented
in find neighbours function. Distances are euclidean though
only 8 neighbouring nodes are considered (adjacent and di-
agonally adjacent positions). An infinite distance is returned
whenever the costmap is any larger than 0, implying the
robot is near an obstacle. The former makes sure paths are
considered only when they do not encounter obstacles.
Run-time for Dijkstra implementation is optimised using
python heapq implementation of priority queue. Heaps are
binary trees in which every parent has value less than or
equal to the child node. In our implementation, heappush
mechanism is used for pushing all the neighbours of the
current node with their respective distances onto the heap and
heappop mechanism is used for extracting next node with the
shortest distance. heappop has a runtime of O(log(n)) for the
extraction of minimum instead of usual O(n) which signifi-
cantly improves the runtime of the Dijkstra implementation.
Hierarchical navigation
When navigating a new environment with a range of
different tasks, for example, the final demo, it would be in-
feasible to define a new list of waypoints for each possible
start position of the robot and each possible goal. We
implement it by doing a recursive path planning, once for the
waypoints, in order to extract the optimal set of waypoints
to use in order to reach from start position of the robot to the
goal state and then we use the same path planning algorithm
to plan a path between the waypoints.
To be more specific, instead of defining waypoint paths for
each possible route, we define a coarse grid of waypoints in
the map (See fig. 1 for the actual map and its grid). Whenever
the robot is tasked to move, path planning happens between
the waypoints by converting them to a cost-map with cost
depending on the existence of connections and a euclidean
distance. The result is a set of waypoints that can be used for
navigating between the robot and its goal while successfully
navigating around obstacles. We also set the orientation of
waypoint vectors on the fly since the robots direction changes
depending on its location and target. The orientations are
set along the longest axis of movement towards the relevant
waypoint. For the start and waypoints next to the tables,
orientations are fixed which is also seen in the image.
Fig. 1: Waypoint graph for intermediate navigation tar-
gets. Tables appear as two dots on the map due to
the laser scanner, thus red squares denote their ap-
proximate footprint on the map. For waypoints with-
out arrows, a direction is determined on the fly.
Legend: A:Start, B:Table1, C:Table2 left, D:Table2 right,
E:mid right, F:mid left, G:mid center, H:drop-off.
Augmentation
Data is the key to all state-of-the-art deep learning
implementations. However, it is not usually the case that
we have enough data and hence, data augmentations can
come in handy. But why would data augmentations help
with training the deep learning models at all?
We can think of data augmentations as added noise provided
to the network during the training process with each
variation of augmentation, which consequently makes sure
that our model learns general features and doesn’t over-fit
on the given dataset.
The provided dataset includes 5 classes one for each box,
the classes include BaseTech, TomatoSoup, Eraserbox,
UsbHub and Evergreen, and we have around 200 images
in each class. In order to improve the usefulness of our
data, we augment images such that the network learns
more invariance to them. These augmentations include
brightness changes, horizontal and vertical flips and random
crops in the image. Specifically, we used the following
augmentations for our dataset:
• Flip Image Horizontal: In horizontal flip, we flip the
image around the central vertical axis in the image,
which provides the invariance against the orientation
of the box around the center axis in the image.
• Change Brightness: For changing the brightness of
the image and providing data to the model, so that
it learns features which are invariant to variations in
brightness of the image. We change the RGB image to
HSV and then we randomly choose a value for HSV
in the range of [50, 200] and assign this value to the
v component(value) of HSV and then we convert this
image back to BGR format as defined in opencv which
is returned by the function.
• Random Crop: The ROIs obtained from ALICE are
not always accurate and as a result of that on a lot of
occasions we obtain images in which the objects are
partially hidden or edges are cut from either top-bottom
or sides. For our model to be invariant to such peculiar
cases, we include random crop in order for our model to
learn invariance to such crops. An array of augmented
images can be seen in figure 2
Fig. 2: A subset of the training set, including from left to
right: original image, brightness, crop and flip augmentation.
Object Recognition
For object recognition, we used a convolutional neural
network architecture with two fully connected layers, two
convolutions followed by a max pool, finally connecting
to the softmax layer. A description of the architecture in
provided in figure 3.
We used a gradient descent optimiser with a learning rate
of 0.002. Input to the network are 64x64x3 images and
output is a softmax array, which we further pass through
the argmax to obtain the class of the object present in the
image. This network was trained for 200 epochs and the
accuracy on the validation set plateaued around 99%, which
seems exceptionally high and hints towards over-fitting. We
tried to run multiples training cycles with smaller epochs
however, the validation accuracy very quickly reaches the
peak, and leaves less room for stopping the training process
earlier. We wanted to stop the training process earlier in
order to avoid a probable over-fitting, unfortunately this was
not possible.
Recognition results on ROIs images obtained from ALICE:
We further tested our network on the ROIs obtained from
ALICE during the final demo. Results obtained on the ROIs
can be seen in the figure 4. Out of total 18 ROI images
obtained from ALICE, our network mis-classified 5 as can
be seen in the above referenced figure. We also observe
that these mis-classifications are for the images in which
the object is kept upside down, and such instances were
not provided to our network during the training process.
Hence, it is reasonable to assume that our network bound
to mis-classify these instances, this can be explained given
the training process and augmentations provided during the
training.
B. Grasping
In the main behaviour, there are 4 components, one
of which is the grasping action server. For grasping, we
define grasps inside of moveit.py, where we further de-
fine grasp pose, pre grasp posture, pre grasp approach and
post grasp retreat. These are four essentials for creating a
grasp and are defined inside create grasps function.
To briefly define the grasping pipeline: as ALICE starts
approaching the table while trying to detect any objects on
the table from the point cloud data. It returns ROIs for the
objects that might be present on the table. Depending on the
ROIs obtained the object recognition server extracts a ROI,
resizes it to an 64x64 image and tries to recognise the object
present in each of these ROIs. If the object is classified as an
object in the given order, the hardcoded dimensions of the
object is used to create a collision model for the object in
the scene, after which the grasping process is initiated. This
further goes onto show that object recognition is an essential
part of the grasping pipeline. Also, since object detection is
performed after every grasp, this action will clear the table
of all objects in the order.
For specifics of the grasps created, for grasp pose we define
 Fig. 3: Object recognition neural network. Convolutions use 5x5 and Max-pooling uses 2x2 kernels.
an object top for each object and this object top is defined
as follows:
ob ject top = 0.5 ∗ size[2] (1)
Where size[2] is z component of the object size, that is
height of the object. For grasp pose, the arm is just directly
positioned above the object using the object top and this
generalises to any object depending on the height of the
object (fig. 5). We define array of parameters for each of
these predefined states(pre-grasp posture, pre-grasp approach
and post-grasp retreat) for creating a grasp. Grasps are
attempted until either the table did not contain any objects
in the order, or a certain number of attempts were reached.
Despite sometimes no valid grasps are found, the robot resets
and then re-calibrates the box location and orientation and a
followup grasp may in fact work.
C. Behaviour
The main behaviour controls the execution of 4 main
processes. These processes are the navigation sub-behaviour,
and the action servers for grasping, approaching a table with
objects, and moving back from a table (called reproach).
Keeping track of these processes is done by keeping a local
state, whose values and transitions are shown in fig. 7. The
robot starts by waiting for its order. Such an order contains a
list of tables to go to and objects to collect. Given an order,
the robot is tasked to go to all tables in said order and remove
all objects matching the order as well. The instructions imply
that the robot will be able to go to a specific table and grab
a given object, as well as being able to clear a table by
adding all objects to the order. To complete the task, the
robot enters (in a sequential and ordered way) a navigation
state, an approach state, a grasping and backwards movement
state. All states can be entered three times at max, since there
are three potential locations for the objects. If the robot has
visited a table and no other table is in the order, it also resets
the state to waiting for order. After visiting every table, the
robot moves to the drop-off point. All objects are mentioned
and the state is reset such that the robot waits for an order
again.
The navigation behaviour uses a few states itself (fig. 6), to
keep track of each waypoint that is visited. The subbehaviour
receives a set of waypoints to follow before the starting
state. It sets its state to waiting and starts moving towards
the first waypoint. Successful navigation changes the state
to end waypoint which sets the next waypoint as the nav-
igation target. However, failed navigation changes the state
to end navigation. Both the end navigation state as well as
successful navigation along all of the given waypoints results
in a finished state for the subbehaviour.
As for the other processes, grasping is described in its own
section and the reproach state precedes a simple command
to the alicecontroller action server to move backwards 80cm.
Alice approach is a more complex action. Roughly, the action
uses the camera to search for any objects in front, and turns
the robot so that driving will place the robot in front of them,
before aligning the robots rotation to look straight.
III. EXPERIMENTS
Various parts of the robot should function and we observe
their performance through our experiments.
A. Navigation
The navigation including navigation using a grid of way-
points works very well in simulation. Moving along way-
points made the movements smooth as the robot does not
have to drive slow to avoid collision. Moreover, a starting
waypoint had been implemented to simulate the position of
the robot at the start of the experiments. The start can be set
to anywhere and is always followed up by the mid center
waypoint. After changing our behaviour to be continuous

Fig. 4: Predictions on ROIs obtained from ALICE. Out
of 18 images obtained from the ROIs, we have 5 miss-
classifications, specifically in the cases where we have object
put upside down and our network is not able to base it’s
prediction on any of the features it has learn during the
training process. Which seems reasonable.
(i.e. resetting the state to waiting for order after object drop-
off), we found the navigation to be very robust as well.
Specifically, wherever the robot was located, it would always
move to start which was typically quite successful and
ensures proper execution of further navigation behaviour.
Testing the navigation in the real world showed similar
results to the simulation. Although waypoints had to be set
more carefully and the map was more noisy (See e.g. fig.
1), navigation worked similar to the simulation. We did find
some drawbacks in our approach. That is, failing to navigate,
when sporadically the robot got stuck in its costmap, meant
Fig. 5: A grasp pose for an object as defined in grasping
section
Fig. 6: Navigation subbehaviour states. Navigation happens
to a waypoint happens in the waiting state. end waypoint
denotes a success and starts the cycle with the next waypoint
until none are left (finished). Failed navigation results in the
end waiting state.
manual recovery had to happen as the robot could no longer
move itself away. Instead, the behaviour was finished and the
robot started to approach; an action that requires shutdown
when no tables with objects are seen. Assuming that the
costmap is a fair representation of obstacles to the robot, it
will be best to just warn the operators and halt behaviours
temporarily. We also found smoothness of navigation to
depend on the map and waypoints, with a map used in the
final demo resulting in the robot often trying to rotate to
align itself with waypoints. With enough space in the map,
alignment rotations more often happened automatically when
moving to the waypoint. Although we also found our derived
waypoint orientations to be off from the desired orientation
by up to 90 degrees, this did not influence the overall
navigation negatively in first experiments and is therefore
not attributed to the problems we found with orientation.
B. Object Recognition
The object recognition network was trained until 98.98%
accuracy and 97% accuracy on validation. For a much better
test, the performance needed to be tested on new images
created by the robot. While integrating the network that
was meant for real objects (different to the network during
simulation), differences between our simulation and actual
recognition model and assumptions on its output lead to
an extraneous application of an argmax function. Without
Fig. 7: The local state of our main behaviour. Colours denote which behaviours are used for the state. From light to dark
respectively: Navigation subbehaviour, alice approach action server, grasping action server and the alicecontroller action
server. The states ’finished’ and ’waiting for order’ are handled in the main behaviour.
the moment to test recognition, the issue led to a constant
prediction from the network and the object recognition was
faulty. The constant prediction of the network was on a box
not included in the final experiment, leading to a constantly
wrong prediction. In order to get the results on object
recognition, then, we have tested the performance afterwards
on the ROIs collected during the final demo. These ROIs
appeared with problems in their colours channels, seemingly
a problem with cv2 mixed RGB channels in imwrite. How-
ever, switching these channels does not fix the issue. We
have shown the results on the given images in fig. 4, since
the problem possibly also occurs on the robot. Nonetheless,
only 5 out of 18 images were misclassified.
C. Grasping
The method of grasping as explained in sec. II-B, has
shown to be effective in simulation. As mentioned, grasps
were attempted until either the table did not contain any
objects in the order, or a certain number of attempts were
reached. The former implementation meant that despite
sometimes no valid grasps are found, the robot reset its
hypothesis of the box location and orientation and a followup
grasp may in fact work. Overall, our robot was able to
pick up many objects in simulation, however the grasping
is very dependent on the object recognition. We had not
attempted grasping before the final demo, and having our
object recognition bugged (see sec. III-B) meant that we only
saw the robot grasp one time, and no adequate evaluation can
be performed.
IV. DISCUSSION / CONCLUSION
We complete the behaviour of a domestic service robot
when tasked to retrieve objects from a fixed set of locations.
Navigation is implemented using Dijkstras algorithm be-
tween target and goal in waypoints, and between these given
waypoints in the map. The navigation proved quite suc-
cessful, allowing successful and robust navigation between
targets. Edge cases such as failed navigation handling could
be improved, also, the final demo showed too much robot
rotations although we did not observe them in a better map.
Aligning waypoint orientations with the robot seemed to
not have a major impact on performance, as failing to do
so perfectly did not result in bad rotations of the robot, or
inabilities in navigating.
Object recognition was based on a convolutional neural
network (fig. 3) which we found (after the experiments) to
perform adequately for the task with 13 out of 18 objects
observed during the experiments classified correctly.
Grasping was shown to be successful in simulation, grabbing
objects and managing to place them on the robot. Unfortu-
nately, we were unable to evaluate the performance of object
grasping on the robot since our object recognition was flawed
at that time.
R EFERENCES
[Dijkstra, 1959] Dijkstra, E. W. (1959). A note on two problems in
connexion with graphs. Numer. Math., 1(1):269–271.
[Glorot et al., 2011] Glorot, X., Bordes, A., and Bengio, Y. (2011). Deep
sparse rectifier neural networks. Journal of Machine Learning Research,
15:315–323.
[Hodson, 2018] Hodson, R. (2018). How robots are grasping the art of
gripping. Nature: international journal of science, 557:s23–s25.
[Krizhevsky et al., 2012] Krizhevsky, A., Sutskever, I., and Hinton, G.
(2012). Imagenet classification with deep convolutional neural networks.
In Advances in Neural Information Processing Systems 25 (NIPS 2012).