DEVICE AWARE 3D LIGHT PAINTING

by Nazerke Safina, BSc Computer Science

April 2013

Supervisors: Prof Jan Kautz

This report is submitted as part requirement for the BSc

Degree in Computer Science at UCL. It is substantially the
result of my own work except where explicitly indicated in the
text.

The report may be freely copied and distributed provided the

source is explicitly acknowledged.

Device aware 3D Light painting

Nazerke Safina, Bsc Computer science

Supervised by Jan Kautz

Abstract
The project was to develop an android application for light painting photography. The
application extrudes 3D models, while taking a picture in a long exposure. This will result in a
picture of 3D objects hovering in the air creating futuristic effect. The first part was recreating few
similar existing applications. The second part of the project is research based, exploring possibility
of using phone's sensors for sensing device motion and incorporating this knowledge to the
application. It particularly focuses on two Android sensors: rotation vector and linear acceleration.
The research part describes many choices made during the development of the object including
rotation representation: Euler angles vs Quaternions, accelerometer choice: acceleration sensor vs
linear acceleration sensor, etc. It also summarises some of the relevant analysis made in this field
like accelerometer noise and filtering. The research part of the project was challenging as phone
accelerometer sensor was never used before for finding position of the device.
The project achieved developed light painting application with rotation sensing capabilities
and also made substantial research on the double integration of the acceleration obtained from the
MEMS sensors incorporated in the smart phones today.

Table of contents
1. Introduction
1.1. Background information
1.2. Problem statement
1.3.Project's aim and goals
1.4. Organization of the report
2. Related work
2.1. Outline of the sources of information and related work
2.2 Technologies Used
3. Design and implementation
3.1.Loading a model into the application and fitting it into the screen
3.2. Light painting iteration 1. Slicing 3D geometry
3.3. Camera settings for light painting
3.4. ImageAverager or if you are too poor for a DSLR
3.5. Light painting iteration 2. Device-aware rotation of the geometry
3.6. Light painting iteration 3. Device-aware extrusion of the
geometry
3.7. GraphHelper
4. Digital Signal Processing or What I would do if I had more time
5. Conclusion and evaluation
5.1. Future work
5.2. Evaluation

Chapter 1. Introduction
1.1 Background information
Light painting is an old photography technique of taking photo of a light
traces in dark location using long exposure and a light source such as a torch
as a real-world paint brush. The technique was developed for research
purposes as a part of a national program of psychic and physical regeneration
and military preparedness and was used to study movements of human body,
especially gymnasts and soldiers.[1] The first known such light painting
photography is called Pathological walk from in front and was taken by
attaching incandescent bulbs to the joints of a person. As many other military
inventions, light painting has been adapted for more peaceful purposes by
ordinary photographers as an artistic endeavor.
With emergence of smart phones and mass development of applications the
idea migrated to the app markets of Android phones, IPhones and IPads. Most
of them use the device as a simple torch, i.e. a circle in different colors and
sizes or to draw holographic 2D or 3D texts.
The most advanced of such applications is Holographium - currently
available in app store. It has many features such as importing logos, icons and
photos in PNG, BMP, JPG, GIF formats and converting them to 3D objects. It
also has functions of user definable extrusion time and depth. However the
idea of my application came from Making Future Magic of Dentsu London
studio.
1.2.Problem statement:
In order to take good light painting photo it is important to balance between
the exposure time, the speed of the extrusion and screen angle. Thats why it
normally takes many attempts to get it right, mostly photos come out
squashed or blurred and with intrusive haziness. While exposure time depends
on the settings of a camera, the extrusion speed and the screen angle depends
on the person dragging the phone. It has to be possible to adjust the 3D model
being extruded according to the movement and rotation of the device and
make extrusion process independent of the human factor.

1.3.Project's aim and goals

My projects aim is to take advantage of computer graphics techniques and
newly emerging area of Android programming - sensor programming and take
the good old light painting to the next level. The latter, use of phone sensors,
is what makes my application different from the available applications Ive
described and what makes it both challenging and interesting project.
In order to carry out the project, first I had to learn many technologies such
as android programming, especially its Opengl ES API(Opengl for Embedded
Systems) for displaying high-end, animated graphics and Android sensor
programming which requires understanding what are the physical values the
sensors measure, what are the physical intuition behind them and how to
interpret them. As well as these programming interfaces, it was also
necessary to inquire theoretical knowledge and computer graphics concepts
such as camera position, bounding box, projection, transformation matrices,
rotation mathematics such as Euler angles and Quaternions. Of course, I didnt
learn everything, but in order to gain the necessary knowledge I had to cover
much wider range than it was necessary.
Ive carried out my project in incremental and iterative fashion, developing
one goal at each iteration, with each iteration developing on top of existing
functionalities and as such enhancing the application capabilities.
These goals were set in the beginning:
1. Parse a model object from an .obj file and load it into the screen
2.

Make it possible to load any object so that the objects centre is in the
centre of the phone screen and the object fits into the screen

3. Move small virtual window along the model in order to get slices or
cross-sections of a model, similar to CAT scan used in medical imaging.
These slices are rendered in the application so that later, while dragging
the phone in front of the camera and taking photo in a long exposure,
the slices are extruded and the composition of these 2D slices form 3D
image in the photo. This is very similar to stop motion, where hundreds
of photos of still objects in different positions form illusion of a
movement.
4. Adding some functionalities to the user interface, like buttons to increase

and decrease near and far planes of the window, i.e. specify the
thicknessof the 3D painting.
To make it independent of the human factor it is possible to sense the
device in the environment and adjust the 3D model accordingly.
Therefore following goals were set as well:
5. Find the appropriate sensor to sense how much was the phone rotated,
i.e. what is the angle of rotation of the phone and use this information to
keep the 3D model stable and stay in its original orientation,
independent from the rotation of a phone.
6. Use the appropriate sensor to extrude slices as quick as the phone is
dragged.
Goals number 5 and 6 are where the capabilities of Android sensors were
to be explored and used to find the rotational angle and speed of the device
movement. This is the research part of the project, which aims to see if it is
possible, are the sensors accurate enough and will they fit into purpose.
The report is organised in sequential fashion, building up the application as it
advances.
1.4 Organization of the report
Chapter 2. Related work
In this chapter I will outline and reference the sources of information: research
papers, source codes and books which assisted my project. I will also list
software tools I used during the development.
Chapter 3. Design and implementation
This chapter is the core of the report where I'll describe the development
process of each iteration and show deliverables of each iteration. I'll also go
through little helper applications which I've built to assist my research.
Chapter 4. Digital Signal Processing or What I would do if I had more time
This chapter talks about types of noise in the accelerometer sensor that
constrains realisation of some ideas. And also describes different filtering
techniques that might be used to eliminate these noise.
Chapter 5. Conclusion and evaluation.
This is capstone chapter where I will sum up the project, its derivations and

lessons learned from it and will provide evaluation of achievements of goals,as

well as future work ideas.
Chapter 2. Related work
2.1. Outline of the sources of information and related work
Androids developer site is the primary place where I learned
programming Android applications
OpenGL ES 2.0 Programming guide by Aaftab Munshi, Dan Ginsburg
and Dave Shreiner is where I learned basics of OpenGL ES.

Android Sensor Programming by Greg Milette and Adam Stroud was of

great help. Part 2 of this book, Inferring information from physical
sensors, gave me extensive base necessary to deal with sensor
programming of my project.

Implementing Positioning Algorithms Using Accelerometers - report

paper by Kurt Seifert and Oscar Camacho describes a positioning
algorithm using the MMA7260QT 3-Axis accelerometer with adjustable
sensitivity from 1.5 g to 6 g. Although this industrial accelerometer
sensor is of better quality in terms of noise and sensitivity compared to
micro-machined electromechanical systems (MEMS) accelerometer
sensors(usually with maximum range 2g) used in current cell phones, it
gives ground to build on.

An introduction to inertial navigation - a technical report by Oliver

J.Woodman, researches the error characteristics of inertial systems. It
focuses on strapdown systems based on MEMS devices and particularly
useful - describes MEMS accelerometers error characteristics such as
constant bias, velocity random walk, bias stability, temperature effects
and calibration errors.

2.2 Technologies Used

This is a brief summary of technologies used in the project, how and where
they were used.
Android SDK(Software Development Kit) provides the API libraries and
developer tools necessary to build, test, and debug apps for Android
including Eclipse IDE and Android Development Tools.
Eclipse IDE is a software development environment with extensible plugin system for customising its base workspace.
ADT (Android Development Tools) is a plugin for Eclipse that provides a
suite of Android specific tools for project creation, building, packaging,
installation, debugging as well as Java programming language and XML
editors and importantly, integrated documentation for Android framework
APIs.
OpenGL ES is an API for programming advanced 3D graphics in
embedded devices such as smartphones, consoles, vehicles, etc. It is an
OpenGL API modified to meet the needs and constraints of the
embedded devices such as limited processing capabilities and memory
availability, low memory bandwidth, power consumption factor, and lack
of floating-point hardware.
Rajawali is an OpenGL ES 2.0 Based 3D Framework for Android built on
top of the OpenGL ES 2.0 API. From its many functionalities I have
used .obj file importer, frustrum culling and quaternion based rotations.
MeshLab is a system for the processing and editing of unstructured 3D
triangular meshes. The system helps to process big, unstructured models
by providing a set of tools for editing, cleaning, healing, inspecting,
rendering and converting this kind of meshes. I downloaded many open
source .obj files from different web sites to load them into my
application. Preprocessing those files in MeshLab before using in my
application helped to remove duplicated, unreferenced vertices, null
faces and small isolated components as well as provided coherent normal
unification, flipping, erasing of non manifold faces and automatic filling of
holes.
Octave is a high-level programming language, primarily intended for
numerical computations. It has adapted gnuplot,a graphing utility for

data visualisation. Octave was used in this project to plot graphs.

Chapter 3. Design and implementation

3.1.Loading a model into the application and fitting it into the screen
In my application Ive used models defined in geometry definition file - .obj.
This file format is a simple data-format that represents essential information
about 3D geometry: position of each vertex, the UV position of each texture
coordinate vertex, the faces and the normals. These information takes
hundreds of lines and looks similar to this:
1. v 3.268430 -28.849100 -135.483398 0.752941 0.752941 0.752941
2. vn 1.234372 -4.395962 -4.233196
3. v 4.963750 -28.260300 -135.839600 0.752941 0.752941 0.752941
4. vn 1.893831 -3.827220 -4.498524
...
1178. f 3971//3971 3877//3877 3788//3788
1179. f 2489//2489 2608//2608 2447//2447
1180. f 3686//3686 3472//3472 3473//3473
...

Models are loaded into the application from this file by parsing each line and
saving the information into the Arraylists or a similar data structure of vertices,
normals, texture coordinates, colours and indices. This information after being
sorted into lists, is now ready to be drawn using typical Opengl ES program.
For this purpose Ive used Rajawali, an open source 3D framework for Android,
by Dennis Ippel. It has a file parsing functionality which can import geometry
definition file formats such as .obj, .md2, .3ds and .fbx files and shader
rendering class using Opengl ES, which is precisely what is needed.
Models come in different sizes and are located in different coordinates in
camera space. When loaded it would be nice if a model is automatically placed
in the centre of the screen and fit it, instead of manually changing camera
position or far and near planes or other settings for every model.

To position a model object to the centre of the screen, it has to be placed to

the centre of the bounding box. To deal with bounding box Ive used Rajawalis
BoundingBox class. The centre of the box can be found by finding midpoint of
maximum and minimum points of the bounding box. Once this point is found,
the models position can be simply reset.

Geometry3D anObject = mObjectGroup.getGeometry();

BoundingBox aBox = anObject.getBoundingBox();
Number3D min = aBox.getMin();
Number3D max = aBox.getMax();
float averageX = (min.x+max.x)/2;
float averageY = (min.y+max.y)/2;
float averageZ = (min.z+max.z)/2;
mObjectGroup.setPosition(-averageX, -averageY, -averageZ);

To fit the model into the screen it has to be scaled to fit into unit bounding
box. For that, the scale factor, i.e. how much the model should be scaled,
needs to be calculated. The amount, by which unit bounding box of the model
is bigger than the bounding box of the model, is the scaling factor. The depth,
width and height of the unit box are all equal to 2(|-1|+1).

float bbWidthx = max.x min.x;

float bbHeighty = max.y - min.y;
float bbDepthz = max.z - min.z;
float scaleFactorx = 2/bbWidthx;
float scaleFactory = 2/bbHeighty;
float scaleFactorz = 2/bbDepthz;
float maxScale = Math.min(Math.min(scaleFactory,
scaleFactorz),scaleFactorx);
mObjectGroup.setScale(maxScale,maxScale,maxScale);

3.2. Light painting iteration 1. Slicing 3D geometry

Once we are able to load a model into the application and fit it into the screen,
simple light painting solution can be developed. The moving window has to
be developed which would move along the model and therefore create 2D
slices.
These sequence of images demonstrate the process of movement of such a
window.

Figure 1. Sliding window[2]

Achieving this effect is nothing more than moving Znear and Zfar clipping
planes along the Z axis. By specifying distance between Znear and Zfar, slice
thickness can be defined. The starting point of the window is initial near. The
window stops when reaches some maximum far limit. Figure 3 illustrates this.

float initialFar = 4.2f;

float
float
float
float
float

far = initialFar;
initialNear = 3.1f;
near = initialNear;
increment =0.05f / 3;
farLimit = 6.7f;

public void onDrawFrame(GL10 glUnused) {

if(far<farLimit)
{far+=increment;
near += increment;
}
else{far = initialFar; near = initialNear;}
if(near<far){
mCamera.setNearPlane(near);
mCamera.setFarPlane(far);
}
mCamera.setProjectionMatrix(mViewportWidth, mViewportHeight);
super.onDrawFrame(glUnused);
}
Figure 2. UI with buttons for adjusting essential parameters

The increment describes how fast should the window move, large increment
matches quick movement and small increment is for slow movement. The
increment and other specifiable variables are incorporated into the UI as in
Figure 2.
Opengls onDrawFrame() draws current frame. Frame rate(FPS-frames
per second) can be specified, which also allows to control how quick should the
window move, the larger is the frame rate, the quicker will the window
move.
Figure 3. Illustration of initialFar and initialNear and sliding window

This is enough for the first working light painting iteration - photo of a 3D
models hovering in the air can be taken, but it doesnt sense devices rotation
or its speed, therefore requires the holder to drag the phone in a very stable
manner and in a constant speed. But if the phone is rotated, the resulting
image will be distorted, because the model being extruded doesnt change its
orientation with the rotation of the phone. This can be seen in the Fig. 4
However, dragging phone in a stable manner results in the kind of images as
in Fig. 5.

Figure 4. Distorted image as a result of the device rotation to the right

Figure 5. Light painting while holding the phone straight and stable, without any rotation

3.3 Camera settings for light painting

To take light painting picture a camera with these settings is needed: ISO
range 100-600, shutter speed from 4 seconds to 8 seconds, manual focus and
timer if the release cable isnt at hand. The camera have to be absolutely
stable in the whole process because while its taking a picture during 4 seconds
or whatever the shutter speed was set to, it is extruding the light and needs to
be jitter-free. The tripod is essential for this reason. The darker the room, the
better will be the quality of the resultant picture.

3.4. ImageAverager or if you are too poor for a DSLR

At first I didnt have a proper camera, so I used my laptops webcam to
record a video while I was dragging the phone. The video is then saved as
sequence of images to be averaged later in the application I have written for
this purpose.

The code in Code listing 1 takes sequence of images stored in a list as an input
and averages red, green and blue components of all pixels separately and
creates new raster using these new averaged samples, then sets this new
raster data to newly created BufferedImage. The buffered image is then saved
as a jpeg file.

Figure 6. Averaged images

3.5. Light painting iteration 2. Device-aware rotation of the geometry

Now we can add rotation capabilities to the 3D model. That is, rotating the
object in the opposite direction to the rotation of the device. That is, if the
phone is turned left, turn the object to the right by the same amount of angle,
if the device is turned right, turn the object to the left, etc, so that the model s
orientation doesnt change. Device can rotate around X, Y and Z axes.
Therefore the object has to be able to rotate in 3 axes accordingly. Its also
good idea to add touch event, so that model starts rotating and window starts
moving only after the touch, giving a holder and the camera opportunity to act
in sync.
To find the amount of angle the phone is rotated by we can use Androids
inertial sensors: accelerometer, linear accelerometer, magnetometer, gravity
sensor in combination. The rotation vector sensor can also be used for this
purpose. It is a synthetic sensor that combines the raw data of multiple raw
sensors, possibly accelerometer, magnetometer and may be gyroscope if

available, and modifies some of those raw sensor data to output values
corresponding to the device coordinate system. The output is angular
quantities around axes. Rotation is described about any given axis in terms of
the number of degrees of difference between the device's coordinate frame
and the Earth coordinate frame. They can be measured in degrees or
quaternions, therefore the output can be in 3-vector, rotation matrix or
quaternion forms.
Figure 7. Rotations around different axis.[3]

I have used rotation vector sensor, as all those options give

approximately the same readings, except it is easier to extract data from
rotation vector sensor and process them straightaway. Synthetic sensors are
usually easier to work with, as they were developed with the aim to make
consumption of the data easier and more obvious. They sometimes modify raw
data before outputting.
Another reason I have chosen rotation vector sensor is because its output is
in a form similar to a quaternion, which is an alternative representation of a
rotation. To get a true normalized quaternion from the output of this sensor it

is recommended to use SensorManager.getQuaternionFromVector().[4] It also

provides SensorManager.getRotationMatrixFromVector() for simple Euler
representation of a rotation. The problem with Euler representation of a
rotation is - it causes gimbal lock. Gimbal lock occurs in physical sensors
gimbals, when two gimbals in a three gimbal system driven into parallel
configuration, i.e when two axes in a three axes system align, causing one of
the degrees of freedom being lost and locking the system into rotation in a
deteriorated 2D system. It is still possible to rotate around all 3 axes, but
because of the parallel orientation of two of the gimbal axes there is no gimbal
available to accommodate rotation along one axis. This takes effect when the
phone is upright and the pitch is plus or minus 90 degrees. The sudden
inaccurate changes occur when the phone is rotated from the position when it
is laying down on the surface to the upright position.
Once the rotation amount of the phone is found and outputted as
quaternions, it is possible to use them straight away to rotate the object.
Rajawali library has useful class Quaternion for quaternion processing. Inverseing the quaternion will give the quaternion of the opposite direction:
case Sensor.TYPE_ROTATION_VECTOR:
SensorManager.getQuaternionFromVector(Q,event.values);
quatFinal = new Quaternion(Q[0],Q[1],Q[2],Q[3]);
quatFinal = quatFinal.inverse();
mRenderer.setQuat(quatFinal);
Figure 8. inverse-ing results on orientation equal in amount to the inverted angle but in the opposition
direction

When the models are loaded

they might be in inconvenient
positions and one would want to
rotate them first before starting to
use the applications primary
functionality. For example the
model might be loaded as in Figure 9 initially. It would be nice if we could
rotate it, programmatically or even by incorporating this functionality into UI,
before starting light extrusion, so that it, for example, is facing us as in Figure

2. And it would be regarded as offset orientation.

Figure 9. Inconveniently facing monkey

Programmatically it is enough to include this line in the renderer's

constructor:
mObjectGroup.rotateAround(Number3D.getAxisVector(Axis.X),90);

Different combination of parameters of axis and angle is possible. But the last
line of the previous code

mRenderer.setQuat(quatFinal);

model to the orientation defined by

quatFinal

resets the orientation of the

quaternion, which is derived from

rotation vector sensor readings. So different approach is needed to address

this problem, rather than quaternion inverse.
When the phone is tapped the rotation of the object is enabled and the
readings of the sensor at that moment is saved as initial orientation. The next
time when sensor senses a tiny rotation, it gives readings of new orientation,
then the difference between old orientation (initial orientation) and new
orientation is found, the new orientation is now saved as old orientation. As
such each consecutive time when the sensor delivers new readings, the
delivered readings are saved as new orientation, while the current new
orientation is save d as old orientation. The negative difference between

quaternions is then added to the offset quaternion. The resulting quaternion is

the final orientation. The offset quaternion is a quaternion of offset orientation.
Figure 10. offsetQuaternion (old orientation new orientation) = final orientation. offsetQuaternion is
the result of mObjectGroup.rotateAround(Number3D.getAxisVector(Axis.X),90);

public boolean onTouchEvent(MotionEvent event) {

startSensing = true;
firstTimeAfterTap=1;
quatInit = quatFinal;
return super.onTouchEvent(event);
}

case Sensor.TYPE_ROTATION_VECTOR:
SensorManager.getQuaternionFromVector(Q, event.values);
Quaternion forLaterUse = new Quaternion(Q[0],Q[1],Q[2],Q[3]);
quatFinal = new Quaternion(Q[0],Q[1],Q[2],Q[3]);
if(startSensing ==true)
{
if(afterTap == 1)
{
offsetQuaternion = mRenderer.getInitialOrientation();}

else if(afterTap>1)

{
Quaternion differenceOfRotation =
findDifferenceOfRotation(quatInit,quatFinal);
differenceOfRotation.inverseSelf();
offsetQuaternion.multiply(differenceOfRotation);
}
quatInit = forLaterUse;
afterTap++;
mRenderer.setQuat(offsetQuaternion);
}
}
}
private Quaternion findDifferenceOfRotation(Quaternion initialOrientation,Quaternion
finalOrientation)
{
initialOrientation.inverseSelf();
finalOrientation.multiply(initialOrientation);
return finalOrientation;
}

3.6. Light painting iteration 3. Device-aware extrusion of the geometry

In the light painting iteration 1 the sliding window was developed, where Znear
and Zfar is moved by incrementing them by some constant amount. This
amount is constant during extrusion, therefore to get full, undistorted, realistic
image of a 3D model it requires the holder to drag the phone in constant
speed. The image below shows what happens if that is not the case, i.e. if the
phones speed varied during light painting.
Figure 11. Phone dragged too fast or too slow

It is possible to take good picture with just first iteration of light painting,
but it requires many trials to get it right. The Holographium, the light
painting application in the market, has features to help the user to control his
speed to get undistorted result. For example, extrusion distance shows the
distance for which user must drag the device to get an undistorted result. Or
the duration of extrusion can be set before the rendering. One-second
interval sounds and 50%-done sounds are also included to warn the user to
slow down or quicken the dragging of the phone.
In my project, I tried to explore the idea of sensing the devices position
relative to its starting point to eliminate the need for controlling stability of
phones speed in the process. So instead of moving near and far planes by the
same amount during extrusion, a solution where this amount is dependent on
devices disposition can be proposed. By disposition I mean the difference
between devices new position and old position. As the phone changes its
position, its new and old positions will be updated accordingly. If the device is
dragged slowly the increment will be small, therefore the window will slide
slower,if it is dragged quicker, the window speed will increase accordingly.
So, instead of float increment =0.05f / 3; it is improved to be float increment = distance;
The window can be imagined to be sliding as the device is being dragged. If
the device stops moving, sliding window will also stop, because increment =
old position new position = 0. But this is in the case if the distance the phone
travels can be accurately found. Otherwise, some alpha can be introduced to
compensate inaccuracy, such that float increment =alpha*distance; where
alpha value has to be chosen through testings among different alpha values,
until the increments are close to the distances the phone travels and 3D image
is clear.
Now, mathematically, distance can be found by integrating velocity and
velocity can be found by integrating acceleration. There are two sensors
available in most phones, that can give accelerometer values.
Accelerometer sensor gives raw accelerometer data. When the device is

still on a table and not accelerating, the accelerometer sensor reads

approximately a magnitude of g = 9.81 m/s^2. Similarly, when the device is in
free-fall and the accelerating towards ground at 9.81 m/s^2, its accelerometer
reads a magnitude of 0 m/s^2. Consequently, in order to measure the real
acceleration of the device, the contribution of the force of gravity must be
eliminated by applying a high-pass filter to the output of accelerometer sensor.
A high-pass filter attenuates the slowly varying background and emphasizes
the higher-frequency components.
Another possible sensor that can be used to fulfill our need is linear
accelerometer. It is recommended to use this sensor if all we need is to filter
out the constant downward gravity component of the accelerometer data and
keep the higher-frequency changes. Greg Millet et al explains it by saying:
The Linear Acceleration sensor is synthetic sensor that consists of high-pass
filtered accelerometer data, and has already been optimized for the particular
hardware sensors in each device that will run your app.[4]
Theoretically it is realistic, but innate errors present in the accelerometer
sensor such as zero offset and drift will make simple integration give poor
results and over time it will quickly explode to give large physically unrealistic
numbers even with no actual movement, because offset and drift are
accumulated under the integral, each time the integration is calculated.
According to Greg Milette et al, The accumulated offset and drift is one reason
why you cant measure distance by double integrating[5]
Yet one more reason, they point out at the same book, is that since constant
non-zero velocity and zero-velocity is the same acceleration, they wont
contribute to the double integration, and as a result the calculated distance is
meaningless. Even if its assumed that in this application phone has nonconstant, non-zero velocity, the drift and offset requires heavy processing to
get rid of.
The positioning algorithm I have used in my application can be described with
this flow chart:
Figure 12. Positioning algorithm

When the device is flat on a table, accelerometer shows non-zero, nonconstant readings. If the output signal is not zero when the measured property
is zero, it is said, that the sensor has an offset or bias. The graph in Figure 13
showing the raw accelerometer data obtained when phone was dragged along
z axis in the device coordinate space for 15 sm as in Figure 12 and held still for
a while in this final position.
The graph shows how the device was motionless until 10th second and
started its movement at this time, then it was dragged for about 5 seconds and
stopped. At the moment when it starts its movement, it accelerates by some
amount, but when it starts stopping and until the moment it stops deceleration
is not equal in magnitude to the acceleration. It will affect the integration error
later in the process.
Figure 12 Dragging the phone along its z axes

Figure 13. Raw accelerometer data

The offset cant be completely eliminated, as its value is not constant and
predictable. However it is possible to suppress its influence by performing
calibration. In my application, Ive decided to store first one hundred readings
of the accelerometer sensor, when it is not undergoing any acceleration, in a
list and average them in order to get calibrated value. However, observations
showed that first twenty five or so readings arent even closely related to
physical reality, therefore they were ignored and readings from 26 to 126 were
used for calibration. The long term average is commonly used to calibrate
readings, the more output is averaged first, the better calibration can be
achieved. Once the calibrated value is known it was subtracted from each
subsequent reading.
As it was said before, it is only possible to weaken the influence of the
offset, because of its fluctuating nature. As such, any residual bias introduced
causes an error in position which grows quadratically with time.

While the application is reading first 125 sensor outputs to use for
calibration, the phone needs to be still for several seconds. The countdown can
be implemented in the UI to keep the user informed about calibration process.
Drift in the accelerometer is caused by a small DC bias in the signal. To
resolve the problem of a drift, a high-pass filter was applied. There are number
of possible filters that could be used, they are discussed in the Chapter 4 of
this report.
Even after the filtering, a no movement condition wont show zero
acceleration. Minor errors in acceleration could be interpreted as a non-zero
velocity due to the fact that samples not equal to zero are being summed. This
velocity indicates a continuous movement condition and hence unstable

position of the device. To discriminate between valid data and invalid data
in the no movement condition, a filtering window was implemented.
This method high-pass filters the calibrated data by using weighted
smoothing and ignores anything between [-0.02;0.02] by setting them to zero.
private float[] highPass(float x, float y, float z)
{
float[] filteredValues = new float[3];
lowFrequency[0] = ALPHA * lowFrequency[0] + (1 - ALPHA) * x;
lowFrequency[1] = ALPHA * lowFrequency[1] + (1 - ALPHA) * y;
lowFrequency[2] = ALPHA * lowFrequency[2] + (1 - ALPHA) * z;
filteredValues[0] = x - lowFrequency[0];
filteredValues[1] = y - lowFrequency[1];
filteredValues[2] = z - lowFrequency[2];
//window filter
if(filteredValues[0]<=0.02&&filteredValues[0]>=-0.02)
{filteredValues[0]=0.0f;}
if(filteredValues[1]<=0.02&&filteredValues[1]>=-0.02)
{filteredValues[1]=0.0f;}
if(filteredValues[2]<=0.02&&filteredValues[2]>=-0.02)
{filteredValues[2]=0.0f;}
return filteredValues;
}

After the high pass filter and filtering window the accelerometer data looks like this:

Now, when the accelerometer data is more or less filtered, it can be used
to find velocity, further, to find disposition. First integration of accelerometer is
v = v0 + at, where t is time interval. Androids SensorEvent data structure
contains timestamp - time in nanoseconds at which the event happened, along
with other information passed to an app when a hardware sensor has
information to report. Using this timestamp information, it is possible to know
the time interval at which accelerometer sensor brings new data, i.e. the time
interval between sensor events.
float currentTimestamp = event.timestamp ;//nanoseconds
float timeInterval= (currentTimestamp timestampHolder)/1000000000;//seconds
timeValues.add(tempHolder);
if(timestampHolder!=0.0f){
tempHolder+=timeInterval;}
timestampHolder = currentTimestamp;

Velocity is found by calculating its x,y,z components separately and taking

square root of the sum of squared components.
v = sqrt(vx^2+vy^2+vz^2)
//find v = v0 + at in 3 dimensions
float velocityX = findXVelocity(timeInterval,accelVals[0]);
float velocityZ = findZVelocity(timeInterval,accelVals[2]);
float velocityY = findYVelocity(timeInterval,accelVals[1]);
finalSpeed = velocityX*velocityX +velocityY*velocityY+velocityZ*velocityZ;
finalSpeed = (float) Math.sqrt(finalSpeed);
float initialVelocityX = 0.0f;
private float findXVelocity(float timeInterval, float f) {
float finalVelocity = initialVelocityX+ f*timeInterval;
initialVelocityX = finalVelocity;
return finalVelocity;
}

The graph shows obtained result.

As it can be seen from the graph, when the device stops its motion velocity
doesnt go back to 0. This is due to the fact that accelerometer, even if filtered
wont be equal to zero, and therefore v0 will give accumulated error.
The second integration, d = d0 + v0*t, will give disposition. It was also
calculated by finding its three components separately:
d = sqrt(dx^2+dy^2+dz^2).
//find d = d0 + v0*t
float positionX = findXPosition(timeInterval,velocityX,accelVals[0]);
float positionY = findYPosition(timeInterval,velocityY,accelVals[1]);
float positionZ = findZPosition(timeInterval,velocityZ,accelVals[2]);
float finalPosition = positionX*positionX+positionY*positionY+positionZ*positionZ;
finalPosition = (float)Math.sqrt(finalPosition);
float initPosition = 0.0f;
private float findXPosition(float timeInterval, float velocityX, float acceleration) {
float position = initPosition+velocityX*timeInterval;
initPosition = position;
return position;
}

The resulting graph is below:

The fact that velocity is never zero when phone comes to rest after the
movement, causes even more error in the second integration as it is shown in
the graph. There are other factors which cause the position to wander off. It

was shown that the accumulated error in position due to constant bias is 4.2.1
oliver woodman. where t is the time of integration.[6]
These graphs described the process when phone is dragged along only one
axes - Z, the case where the phone is dragged making angles with all three
axes is described in Appendix 1.
There are many sources of noise that makes accelerometer sensors very
unreliable for double integration. In the technical report Intro to inertial
navigation, Woodman analyzes these intrinsic noises and gives them numeric
characteristics. [6]
The accelerometer noise comes from electronic noise from the circuitry
that converts motion into a voltage signal and the mechanical noise from the
sensor itself. MEMS accelerometers consist of small moving parts which are
averse to the mechanical noise that results from molecular agitation,
generating thermo-mechanical or white noise.[7] Integrating accelerometer
output containing white noise results in velocity random walk. Woodman
analyzes what effect this white noise has on the calculated position, by double
integrating the white noise and finding the variance. The analysis shows that
accelerometer white noise creates a second order random walk in position,
with a standard deviation standard deviation which grows proportionally to
t^(3/2).
There are many sources of accelerometers electronic noise: shot noise,
Johnson noise, flicker noise and so forth. The flicker noise causes the bias to
stray over time. Woodmans analysis shows that flicker noise creates a second
order random walk in velocity whose uncertainty grows proportionally to
t^(3/2), and a third order random walk in position which grows proportionally
to t^(5/2).
Furthermore, the analysis reveals that gyros white noise also influences
the noisiness of the acceleration. Woodman says, Errors in the angular
velocity signals also cause drift in the calculated position, since the rotation
matrix obtained from the attitude algorithm is used to project the acceleration
signals into global coordinates. An error in orientation causes an incorrect

projection of the acceleration signals onto the global axes. This fact will result
in the accelerations of the device being integrated in the wrong direction.
Moreover, acceleration due to gravity can no longer be correctly removed.
3.7. GraphHelper
Helper class GraphHelper was developed by myself to help to collect
necessary data and write it to a file in the external storage of a phone.
Once the time interval, velocities and positions are calculated this method is
called
saveDataToPlotGraph(timeInterval,raw,offset,highPassed,velocityX,velocity
Y,velocityZ,finalSpeed,positionX,positionY,positionZ,finalPosition);whic

h organizes and saves data in value-timeInterval pairs, to make it easier for

Octave, to read from.
.....
// all we need to plot graph of acceleration_X
float[] time_ax = {timeInterval,raw[0]};
rawAx.add(time_ax);
float[] time_calibrated_ax = {timeInterval,raw[0]-offset[0]};
calibratedAx.add(time_calibrated_ax);
float[] highpassed_ax = {timeInterval,highPassed[0]};
highpassedAx.add(highpassed_ax);

When the application pauses, all the accumulated data are now written into a
file
protected void onPause()
{
super.onPause();
mSensorManager.unregisterListener(this,mRotVectSensor);
mSensorManager.unregisterListener(this,mLinearAccelSensor);
writeDataToFile();
}

GraphHelper graphHelper;
private void writeDataToFile() {
graphHelper = new GraphHelper(getApplication());
graphHelper.writeToFile("rawAx.txt",rawAx);
graphHelper.writeToFile("calibratedAx.txt",calibratedAx);
graphHelper.writeToFile(highpassedAx.txt",highpassedAx);

...}

In the GraphHelper class, writeToFile() method will do actual writing to a file

in external storage:
public void writeToFile(String filename,ArrayList <float[]>data)
{
if(isExternalStorageAvailable()&&!isExternalStorageReadOnly())
{
myExternalFile = new File(directory, filename);
try {
outputStream = new FileOutputStream(myExternalFile);
writer = new PrintWriter(outputStream);
for(int i=0;i< data.size();i++)
{
toWrite = data.get(i);
writer.print(toWrite);
}
writer.close();
outputStream.close();
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();}}
}

Figure 14. The screenshot from the folder holding all the saved files and the content(time interval acceleration) of one of those files.

Chapter 4. Digital Signal Processing

or
What I would do if I had more time
There are many advanced filtering techniques that could be tried to get
accurate position data by double integrating the acceleration. The most
recommended of those is Kalman filtering. It is said to provide excellent signal
processing results, but is known for being complicated. With a set of prior
knowledge about the signal, a Kalman filter algorithm can provide an accurate
estimate of somethings true value. Moreover Kalman filters can be applied to
any signal from any sensor.
D Slifka in his thesis explores and experiments with different filtering
techniques that could be applied to acceleration data, such as Finite Impulse
Response(FIR), Infinite Impulse Response and Fast Fourier Transform(FFT). [8]
The IIR and FFT cant be used in real time, therefore is not an option for this
project. The FIR can be used in real time, as it is recursive method, but has a
long delay time, which is also a big drawback for the project application.
Most of the filtering methods require accumulating the data beforehand and
processing it only after enough data was inquired, i.e. they are not real-time
filters. That left me with the impression that if I had more time I would learn
Kalman filtering algorithm and try to apply it to my application. It requires
background knowledge in probability, especially in Bayes law. Then the
scenario for fitting those into the Kalman filter needs to be implemented.
Chapter 5. Conclusion and evaluation
5.1. Future work
The future work will be exploring the filtering mechanisms or the sensors that
coupled with accelerometer would result in more accurate position results. The
best place to start would be Kalman filters, which known to be effective and
resulting good estimates of the noisy data. One way or another, the goal of the
future work is to implement a better positioning algorithm, i.e. way to calculate
accurate position of the phone in the space with respect to its starting point.

5.2. Evaluation
The first iteration of the project developed an application which can transform
any .obj model to fit into the screen after it has been loaded. It also
accomplished the task of sliding window for getting cross sections of the
model object. It provided UI for adjusting parameters to make it easy to
experiment with speed, thickness and starting and finishing z points of sliding
window. It is extremely helpful to be able to try different settings. Android
application development essentials, Opengl ES, and basic computer graphics
theory like camera positioning and bounding box were learned during the first
iteration of the project.
The second iteration of the project enhanced the first iteration by integrating
device-aware rotational abilities to the model object. Much knowledge about
androids rotation vector sensor and different representations of rotation such
as Euler angles and Quaternions were learned during the development.
The first and the second iteration of the project were successfully
accomplished and makes good working application, which can be used for light
painting with a constraint that holder has to drag the phone in constant speed.
While the result of the first iteration, extruding slices of 3D image, resembles
existing light painting applications, the second iteration, where rotation sensor
is used to sense the devices rotation and incorporate the output of the sensor
to the 3D model, is a novelty that improved the technology. Now, users can
rotate the device and dont be afraid of distorted results because of devices
rotation.
The third iteration of the project, although didnt reach the goal - getting
phones accurate disposition by double integrating accelerometer data,
attempted the theoretical base of double integration and has showed that it is
impractical to use acceleration alone for the positioning or the inertial
navigation systems generally. It has to be highly processed by advanced
filtering technologies. The results of each step of double integration were
plotted to visualise the obtained data. The existing noise measurements of
accelerometer sensor and ideas about filtering technologies were researched.
Considerable amount of experiments were held in order to reach accurate
representation of the accelerometer, due to its nature, which senses tiniest
environmental vibrations and natural shaking of the holding hand. Great deal
of knowledge about androids inertial sensors were gained during the

development of the third iteration of the project.

Overall, the considerable amount of effort was put to the project in the given
time constraint and much knowledge and skills were gained during the
development and the research.

Appendix 1.System manual

The Android SDK provides you the API libraries and developer tools
necessary to build, test, and debug apps for Android. With a single download,
the ADT Bundle includes everything you need to begin developing apps:
Eclipse + ADT plugin
Android SDK Tools
Android Platform-tools
The latest Android platform
The latest Android system image for the emulator
Once you download the Android SDK, import the code to the workspace. For
that go to File-> Import->Existing Android Project as shown in the
screenshots. It will open the window where you'll need to show where the
project is. Once you have chosen Finish.

Once it's created, the Rajawali external library needs to be downloaded from
https://github.com/MasDennis/Rajawali and imported the same way as described above. Once the
Rajawali is in the workspace as well, it should be included as external library to the LightPainter3D
project. Right click the project->Properties->Android will bring the following window, where in the
Library section the new library Rajawali needs to be added

Now it's ready to be run in the android device. In the phone's settings
USB debugging must be enabled (usually Settings->Developers-> USB
debugging, but might differ from phone to phone) and then phone should be
connected to the computer via USB. Once connected, run the project as
Android application.

The Android device chooser window must appear

If it doesn't then in Eclipse:

goto run menu -> run configuration.
right click on android application on the right side and click new.
fill the corresponding details like project name under the android tab.
then under the target tab.
select 'launch on all compatible devices and then select active devices from
the drop down list'.
save the configuration and run it by either clicking run on the 'run' button
on the bottom right side of the window or close the window and run again
Appendix 2. User manual
For this light painting application you need a camera with adjustable
shutter speed and ISO range, as well as manual focus. A release cable or
camera's timer is useful to get prepared both you and the application get
prepared. The shutter speed can be around 6 10 seconds, and ISO 200-400.
You might want to experiment with different settings, this will create different
effects. The photo needs to be taken in the dark location and the camera made
sure to not to shake during the process. A tripod will be helpful for this
purpose.
The application allows you to define the thickness of the slice and near
and far planes, as well as frames per second, which is related to how quick the
extrusion be. In my experience the lower fps and the thinner slices create
photos of better quality. The photos I've taken were extruded from the slices of
thickness = 0.1 and 20fps. These are only suggestions, light painting is
creative process!
Different models have different sizes, therefore back or front of the
object might be out of the frustrum. Znear and Zfar regulators were provided
to widen or to narrow the frustrum.

When the camera is set and you are ready, the photo can be taken.
While it is being taken you'll need to drag the phone in front of the camera in a
constant speed. You are not constrained by the occasional rotations of the
device that can happen while you are dragging the phone in the air, the
application senses the rotation and rotates the object accordingly to keep it in
the place, therefore you have more freedom than with other applications of
this kind.
Appendix 3. More results in graphs
Here are the results obtained from dragging phone in the air for 30 sm,
now, making angles with all three axes like so:

Here is the highpass filtered accelerometer data:

Here is the first integration of the acceleration:

Here is the second integration of the acceleration:

Appendix 4. Code listing. Image averaging method

public static BufferedImage average(ArrayList<BufferedImage> images) {
int n = images.size();
// Assuming that all images have the same dimensions
int w = images.get(1).getWidth();
int h = images.get(1).getHeight();
BufferedImage average =
new BufferedImage(w, h, BufferedImage.TYPE_3BYTE_BGR);
WritableRaster raster =
average.getRaster().createCompatibleWritableRaster();
for (int y=0; y < h; ++y){
for (int x=0; x < w; ++x) {
double sumRed = 0.0f;
double sumGreen = 0.0f;
double sumBlue = 0.0f;
for (int i=1; i<n; ++i){
sumRed = sumRed + images.get(i).getRaster().getSample(x, y, 0);
sumGreen = sumGreen + images.get(i).getRaster().getSample(x,
y, 1);
sumBlue = sumBlue + images.get(i).getRaster().getSample(x, y,
2);
}
raster.setSample(x, y, 0, Math.round(sumRed/n));
raster.setSample(x, y, 1, Math.round(sumGreen/n));
raster.setSample(x, y, 2, Math.round(sumBlue/n));
}}
average.setData(raster);
return average; }

BIBLIOGRAPHY

[1] M. Mulcahy, Chronophotohraphy, Building Better

Humans,http://cinemathequefroncaise.com/Chapter1-1/Figure_B01_01_Station.html
[2] Sliding window screenshots
http://thequietvoid.com/client/objloader/examples/OBJLoader_FaceLocation_MATT
D/applet/index.html
[3] Device orientation screenshots
https://developer.mozilla.org/enUS/docs/DOM/Orientation_and_motion_data_explained
[4] Milette G, Stroud A, Professional Android Sensor Programming, Chapter 2
[5]Milette G, Stroud A, Professional Android Sensor Programming, Chapter 6
[6] Woodman, O (2007), An introduction to inertial navigation
[7] Miller S, et al, Noise measurement
[8] D Slifka (2004), An accelerometer based approach to measuring displacement of
a vehicle body
Munshi A et al, OpenGL ES 2.0 Programming Guide
Seifert Kurt et al (2007), Implementing Positioning Algorithms Using
Accelerometers
Ribeiro J.G.T. et al, New improvements in the digital double integration
filtering method to measure displacements using accelerometers, p 538-542
Ribeiro J.G.T. Et al, Using the FFT-DDI method to measure displacements
with piezoelectric, resistive and ICP accelerometers

Android: computing speed and distance using accelerometer,

http://maephv.blogspot.co.uk/2011/10/android-computing-speed-anddistance.html

Slater M et al (2002), Computer graphics and virtual environments