Android SDK Augmented Reality CAMERA

All Topics Find tutorials, courses, and more...
Code Categories Learning Guides
ANDROID SDK
Android SDK Augmented

Reality: Camera & Sensor
Setup
by Shane Conder & Lauren Darcey 23 Sep 2011 0 Comments
7 4
Augmented reality (AR) is a hot topic in mobile apps today. Smartphones and
tablets have the power and the hardware capable of enabling developers to write
interesting new applications that incorporate live camera and video feeds, hyper-
open in browser PRO version Are you a developer? Try out the HTML to PDF API pdfcrowd.com
accurate sensor data, and other real-time user data in interesting ways. Today well
begin exploring the world of augmented reality, and what the Android platform has
to offer for developers looking to build AR applications and provide deep, rich
experiences to their users.
Also available in this series:

1. Android SDK Augmented Reality: Camera & Sensor Setup
2. Android SDK Augmented Reality: Location & Distance
Step 0: Prerequisites and Getting Started

This tutorial moves at a pretty fast pace. As such, we expect readers to be familiar
with all the basics regarding creating and running Android projects. We also expect
that readers will have an Android device that is sufficiently powerful for running AR
apps (such as the Nexus S). Most testing of this application will need to be done on
an actual Android device, as the app relies heavily upon the camera, sensors, and
location data that is not readily available in the emulator.
We have provided a sample application in conjunction with this tutorial for

download. This project will be enhanced over the course of this premium AR tutorial
series, so go download the Android project and follow along!
Part 1: What Kind of AR App are We
Developing?
First, we need to define (in broad terms) what this application will do. We want to
stick with the typical features of an AR application, and improve it over time.
So what kind of application will we build? Were going to build a location-based AR

app. These types of AR apps generally show where things are in the real-world by
indicating where the app thinks they are over the camera view when the user holds
the phone up and moves it about.
A different type of AR app is the type that recognizes the object using computer
vision algorithms and can either draw information over the camera to show what the
object is, how its oriented, or perhaps add or show other objects in the view. While
we wont be building this sort of AR app, many of the techniques we teach here can
be applied to either type of AR application.
The basic architecture of a location-based application involves a camera view, the

users location, the phones orientation compared to the real world, and a list of
locations for items that we want to show on the display the data with which we
augment the cameras vision of the world.
This first part of our series will talk about the camera and the sensors. In the next
part, we'll talk about location data and the math involved in building the app.
App Requirements: What Device Features Does

our App Need?
Your typical AR application requires access to the following device features:
Access to the camera view

Access to the users location
Access to other device sensors, especially a compass (accelerometer and
gyroscope can also be useful)
The application may also require other services and permissions, such as network
access, but these are not central to the AR aspects of the application.
A Note on Target Devices: The Nexus S
Not all Android devices (or mobile devices in general) have the hardware specs to
be able to handle AR application requirements. For the purposes of this tutorial, we
are targeting the Nexus S running Android 2.3.4, which meets our specific hardware
requirements nicely. Therefore, our application will target a very specific version of
the Android SDK. This will keep the app simple. However, the SDK requirements
can be more flexible once the Android manifest correctly identifies any required
hardware features, or you implement alternative functionality for less powerful
devices.
While the Nexus S is more than capable of performing reasonable AR tasks, many
other devices are also capable, including some older devices. If youre building a
production app that uses AR technology, youll want to test carefully on your target
devices for response, usability, and correct functionality.
Part 2: Project Setup
Now were ready to start developing! Lets get an AR project set up properly first.
This application is configured much like any other application project in Eclipse.
Step 1: Create a New Eclipse Project

Begin by creating a new Android project in Eclipse. Here are the project details we
used for the sample project (available for download):
Project Name: ARTUT

API Level Min/Target: 10
Launch Activity Class: ArtutActivity
Step 2: Configure the Manifest File
Next, youll want to configure the Android Manifest File. Youll want to specify the
following settings:
On the Manifest Tab, add settings for the following:
android.hardware.camera (Required=true)
android.hardware.camera.autofocus (Required=false)
android.hardware.location.gps (Required=true)
android.hardware.sensor.compass (Required=true)
android.hardware.sensor.gyroscope (Required=true)
android.hardware.sensor.accelerometer (Required=true)
On the Application Tab:
Debuggable should be set to true
On the Permissions Tab, add settings for:
android.permission.CAMERA
android.permission.FINE_LOCATION
Step 3: Connect the Target Device
Finally, if you havent already done so, youll need to setup an Android device for
testing and plug it into your development machine via USB. If youve not done this
before, you may need to go download and install drivers prior to use.
Theres little point in creating an Android Virtual Device (AVD) for use with this
project, as the emulator does not have sufficient features for testing live-camera or
sensor based applications with any degree of effectiveness. We will do all of our
development and testing against real hardware. Besides providing a better testing
environment when using sensors, the camera, and other hardware, we also find that
testing on real hardware is much faster due to the performance and feature
limitations of the emulator.
Part 2: Working with the Camera
Our first real step in developing an AR app involves getting camera content
displayed on the screen. We do this by implementing a custom SurfaceView that is
displayed within the activitys layout.
Step 1: Defining the App Screen Layout
First, we need to define the layout resource that will be used for the main AR
screen. A FrameLayout is most appropriate for our screen, as it allows the layering
of views within the display area. Therefore, our layout resource,
/res/layout/main.xml, can be quite simple:
1 <?xml version="1.0" encoding="utf-8"?>
2 <FrameLayout xmlns:android="http://schemas.android.com/apk/res/android
3 android:id="@+id/ar_view_pane"
4 android:layout_width="fill_parent"
5 android:layout_height="fill_parent">
6 </FrameLayout>
Step 2: Defining a Custom SurfaceView to Hold

Camera Content
Next, you need to define a custom view control for use within your FrameLayout.
The SurfaceView class can be used for this purpose. Therefore, you can use this
class to encapsulate your camera drawing.
01 public class ArDisplayView extends SurfaceView

02 {
03 public static final String DEBUG_TAG = "ArDisplayView Log"
;
04 Camera mCamera;
05 SurfaceHolder mHolder;
06 Activity mActivity;
07
08 public ArDisplayView(Context context, Activity activity) {
09 super(context);
10
10
11 mActivity = activity;
12 mHolder = getHolder();
13 mHolder.setType(SurfaceHolder.SURFACE_TYPE_PUSH_BUFFERS);
14 mHolder.addCallback(
this);
15
16 }
The ArDisplayView class begins with a constructor. Weve added the Activity as a
configuration parameter (more on this in an upcoming step). The constructor must
initialize the underlying SurfaceHolder, which controls the drawing on the screen.
(One note here is that although the setType() method is deprecated and the Android
SDK documentation says it is no longer necessary, we have found that the Camera
code will not function without it, it simply fails to start up, even on the newest
devices.)
Weve added the appropriate member variables for working with a Camera, and we
also keep the Activity around so that we can react to orientation changes and any
other convenient access to the Context or Activity that we may need.
Step 3: Implementing the SurfaceHolder

Callbacks
Next, youll need to implement the SurfaceHolder callbacks within the class. Update
your ArDisplayView class so that it implements the SurfaceHolder.Callback
interface, like this:
1 public class ArDisplayView extends SurfaceView implements SurfaceHolder.Callback

2 {
3 //
4 }
This will necessitate overriding and implementing three new methods:

surfaceCreated(), surfaceChanged(), and surfaceDestroyed().
When working with the Camera, theres a set of steps needed to startup and
shutdown. These steps are clearly documented in the Android SDK docs for the
Camera class. We basically drop these steps into the appropriate SurfaceHolder
callbacks.
Step 4: Implementing the surfaceCreated()

Callback
We start with the surfaceCreated() callback method. Here we request the Camera,
set the cameras display orientation based on the current orientation of the device
(so that the camera preview always displays right-side up), and call the
setPreviewDisplay() method to tie the Camera preview to our SurfaceHolder.
01 public void surfaceCreated(SurfaceHolder holder) {

02 mCamera = Camera.open();
03
04 CameraInfo info = new CameraInfo();
05 Camera.getCameraInfo(CameraInfo.CAMERA_FACING_BACK, info);
06 int rotation = mActivity.getWindowManager().getDefaultDisplay().getRotation();
07 int degrees = 0;
08 switch (rotation) {
09 case Surface.ROTATION_0: degrees =0; break;
13 }
14 cam.setDisplayOrientation((info.orientation - degrees 360)
+ % 360
15
16 try {
17 mCamera.setPreviewDisplay(mHolder);
18 } catch (IOException e) {
19 Log.e(DEBUG_TAG, "surfaceCreated exception: ," e);
20 }
21 }
Step 5: Implementing the surfaceChanged()
Callback
Much of the interesting code for the Camera preview happens in the
surfaceChanged() callback. Here we request the camera parameters and check
what the supported preview sizes are. We need to find a preview size that can be
accommodated by the surface, so we look for the closest match and use it as our
preview size. Next, we set the camera preview format, commit the camera
parameter changes, and finally call the startPreview() method to begin displaying
the camera preview within our surface.
01 public void surfaceChanged(SurfaceHolder holder,int format, int width,

02 {
03 Camera.Parameters params = mCamera.getParameters();
04 List<Size> prevSizes = params.getSupportedPreviewSizes();
05 for (Size s : prevSizes)
06 {
07 if((s.height <= height) && (s.width <= width))
08 {
09 params.setPreviewSize(s.width, s.height);
10 break;
11 }
12 }
13
13
14 mCamera.setParameters(params);
15 mCamera.startPreview();
16 }
Step 6: Implementing the surfaceDestroyed()

Callback
We end with the surfaceDestroyed() callback method. Here we shut down the
camera preview and release the Camera resources.
1 public void surfaceDestroyed(SurfaceHolder holder) {

2 cam.stopPreview();
3 cam.release();
4 }
Step 7: The Custom Camera View and the Main

Activity
Now you need to add your custom views to the FrameLayout control. To do this,
update the onCreate() method of the ArtutActivity class, like this:
1 public void onCreate(Bundle savedInstanceState) {

2 super.onCreate(savedInstanceState);
3 setContentView(R.layout.main);
4
5 FrameLayout arViewPane = (FrameLayout) findViewById(R.id.ar_view_pane);
6
7 ArDisplayView arDisplay =new ArDisplayView(this);
8 arViewPane.addView(arDisplay);
9 }
This adds the camera preview view class, called ArDisplayView to the
FrameLayout. If you run the application now, the activity starts up and you should
see a camera preview on the screen.
A note on figures: The default DDMS screen capture does not capture the camera
preview. Run the app yourself to see the true functionality.
Part 3: Working with the Device Sensors
Our next step in developing an AR app involves accessing the device sensor data.
We also need a way to overlay content on top of the camera preview. We do this by
implementing a custom view that can be layered on top of the ArDisplayView
currently displayed in the FrameLayout control of the activity.
Step 1: Displaying Overlay Content
First, you need to define a custom view control to draw overlay content in. Create a
new class called OverlayView that is derived from the View class. This class starts
out simple for this part of the tutorial. We simply want to overlay some sensor data
on top of the camera view (no fancy math yet). To do this, we create member
variables to store the last known sensor data for each sensor we are interested in.
We then implement the onDraw() method of our view and draw some text on the
canvas.
01 public class OverlayView extends View {

02
03 public static final String DEBUG_TAG = "OverlayView Log";
04 String accelData = "Accelerometer Data";
05 String compassData ="Compass Data";
06 String gyroData = "Gyro Data";
07
08 public OverlayView(Context context) {
09 super(context);
09 super(context);
10 }
11
12 @Override
13 protected void onDraw(Canvas canvas) {
14 super.onDraw(canvas);
15
16 Paint contentPaint =new Paint(Paint.ANTI_ALIAS_FLAG);
17 contentPaint.setTextAlign(Align.CENTER);
18 contentPaint.setTextSize(
20);
19 contentPaint.setColor(Color.RED);
20 canvas.drawText(accelData, canvas.getWidth()/
2, canvas.getHeight()/
21 canvas.drawText(compassData, canvas.getWidth()/
2, canvas.getHeight()/
22 canvas.drawText(gyroData, canvas.getWidth()/
2, (canvas.getHeight()*
23 }
24 }
25
The onDraw() method does all the work here. We configure a Paint with the
appropriate text attributes and then draw some text on the canvas using the
drawText() method.
Why use a custom view instead of regular controls? We'll want precision later on for
placing AR objects within the camera view that are representative of locations of
real world places. An alternate to this method is to use 3D rendering to better get a
handle of depth and distance. In a location AR app, that's usually less important as
the distances are often pretty vast (kilometers, not meters).
Step 2: Implementing the SurfaceHolder
Callbacks
Next, youll need to implement the SensorEventListener interface within the class.
Update your OverlayView class so that it implements the SensorEventListener
interface, like this:
1 public class OverlayView extends View implements SensorEventListener {
This will necessitate overriding and implementing two new methods:

onAccuracyChanged() and onSensorChanged(). For the purposes of this tutorial,
we are really only interested in the onSensorChanged() method.
Step 3: Registering for Sensor Data Updates
Before we implement the SensorEventListener callbacks, we want to register for the

appropriate sensor data updates. To do this, update the onCreate() method of the
OverlayView class.
Begin by requesting the SensorManager and retrieving the sensors you want the
view to watch. In this case, well watch the accelerometer, the compass, and the
gyroscope. Next, register to listen for events on each of these sensors.
1 SensorManager sensors = (SensorManager) context.getSystemService(Context.SENSOR_SER

2 Sensor accelSensor = sensors.getDefaultSensor(Sensor.TYPE_ACCELEROMETER);
3 Sensor compassSensor = sensors.getDefaultSensor(Sensor.TYPE_MAGNETIC_FIELD);
4 Sensor gyroSensor = sensors.getDefaultSensor(Sensor.TYPE_GYROSCOPE);
5
6 boolean isAccelAvailable = sensors.registerListener(
this, accelSensor, SensorManager.S
7 boolean isCompassAvailable = sensors.registerListener(
this, compassSensor, SensorManag
8 boolean isGyroAvailable = sensors.registerListener(
this, gyroSensor, SensorManager.SEN
Now youre ready to implement the sensor callback, onSensorChanged(), in order to

react to these sensor updates.
Step 4: Implementing the onSensorChanged()

Callback
Much of the interesting code for the sensor updates happens in the
onSensorChanged() callback. Here we need to have the view react to sensor
changes. Weve registered for several different types of changes, so this method
may be called for any sensor change. For now, we simply want to save off the
sensor data that changed and then force the view to update its content. We can do
this by simply invalidating the view, and forcing it to be redrawn.
01 public void onSensorChanged(SensorEvent event) {

02 StringBuilder msg =new StringBuilder(event.sensor.getName()).append(
03 for(float value: event.values)
04 {
05 msg.append("[").append(value).append(
"]");
06 }
07
08 switch(event.sensor.getType())
09 {
10 case Sensor.TYPE_ACCELEROMETER:
11 accelData = msg.toString();
12 break;
13 case Sensor.TYPE_GYROSCOPE:
14 gyroData = msg.toString();
15 break;
16 case Sensor.TYPE_MAGNETIC_FIELD:
17 compassData = msg.toString();
18 break;
19 }
20
21 this.invalidate();
22 }
Advertisement
Step 5: Your Custom Overlay and the Main

Activity
Now you need to add your custom view to the FrameLayout control. To do this,
update the onCreate() method of the ArtutActivity class. Add the following code just
below the addView() call that adds the ArDisplayView:
1 OverlayView arContent =new OverlayView(getApplicationContext());
2 arViewPane.addView(arContent);
This stacks the two views: first the camera preview view class, called
ArDisplayView, and then overlays the OverlayView which displays the sensor data
on top of it. If you run the application now, the activity starts up and you should see
a camera preview on the screen with sensor data displayed on top of it (recall you
cannot take screenshots of the camera view, so the following figure only shows the
overlay content).
If you are running the app at this point, you've probably noticed that the sensor data
updates very quickly, despite asking for a rate slow enough to work with orientation
changes. These device sensors have a very high degree of sensitivity. Ultimately,
this will lead to shakiness in the UI. We'll eventually need to implement some
smoothing algorithms to keep the jitters away when we begin to use the sensor data
within our application.
You might also be wondering if we'll be using all of these sensors and what their
units are. We won't necessarily be using all of the sensors, but we thought we'd
show you what they look like, and how to access them, in case your particular AR
interests lay with different sensors.
The accelerometer numbers are in SI units (m/s^2, i.e. meters per second
squared, where Earths gravity is 9.81 m/s^2).
The magnetic sensor units are in micro-Teslas. As they come in x, y, and z
forms we can measure the vector and compare to gravity to determine where
(magnetic) north is with respect to the device.
The gyroscope measures rotation around each axis in radians per second.
This can be used to calculate the relative device orientation, too.
Conclusion
This concludes the first part of our AR tutorial. Augmented reality is an exciting
genre for applications on the Android platform. The Android SDK provides
everything necessary for developing sleek and interesting AR apps, but not all
devices can meet the hardware requirements that those apps demand. Luckily, the
newest generation of Android devices is the most powerful yet, and many of the
Android devices out there are perfectly capable of running advanced AR
applications.
In the next part of this tutorial series, we'll move on to collecting location data and
placing AR objects on the screen in a specific location. This will involve various
mathematical algorithms, but the end result will beaugmentation!
Advertisement
Difficulty: Suggested Tuts+ Course

Advanced
Length:
Medium
Categories:
Android SDK
Translations Available:
Tuts+ tutorials are translated by our community

members. If you'd like to translate this post into
Building Android Apps In C# With Start
another language, let us know! Xamarin
Download Attachment
Related Tutorials
Create a YouTube Client on Android

Code
About Shane Conder & Lauren Darcey

Mobile developers Lauren Darcey and
Shane Conder have coauthored numerous
Sharing Files With NFC on Android
books on Android development. Our latest
Code
books include Sams Teach Yourself Android
Application Development in 24 Hours (3rd Edition),
Introduction to Android Application Development:
Android Essentials (4th Edition), and Advanced
Streaming Video in Android Apps
+ Expand Bio
Code
Jobs
Web Developer / Site Management

at Zanes Law in Phoenix, AZ, USA
PHP Coder with Magento Knowledge

at Yoginet Web Solutions in New Delhi,
Delhi, India
Advertisement
Envato Market Item
0 Comments Tuts+ Premium
Sort by Best
Start the discussion
Be the first to comment.
Subscribe d Add Disqus to your site Privacy
Advertisement
18,890 Tutorials 461 Video Courses

Teaching skills to millions worldwide.
Follow Us Email Newsletters
Get Tuts+ updates, news, surveys &

offers.
Email Address
Help and Support Email Address
FAQ
Subscribe
Terms of Use
Contact Support Privacy Policy
About Tuts+
Advertise
Teach at Tuts+
Custom digital services like logo design, WordPress installation, video

production and more.
Check out Envato Studio
Build anything from social networks to file upload systems. Build faster
with pre-coded PHP scripts.
Browse PHP on CodeCanyon
2014 Envato Pty Ltd. Trademarks and brands are the property of their respective
owners.

Android SDK Augmented Reality CAMERA

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Android SDK Augmented Reality CAMERA

Caricato da

Copyright:

Formati disponibili

All Topics Find tutorials, courses, and more...

Code Categories Learning Guides

Android SDK Augmented

Also available in this series:

Step 0: Prerequisites and Getting Started

We have provided a sample application in conjunction with this tutorial for

So what kind of application will we build? Were going to build a location-based AR

The basic architecture of a location-based application involves a camera view, the

App Requirements: What Device Features Does

Access to the camera view

A Note on Target Devices: The Nexus S

Part 2: Project Setup

Step 1: Create a New Eclipse Project

Project Name: ARTUT

Step 2: Configure the Manifest File

On the Manifest Tab, add settings for the following:

On the Application Tab:

Debuggable should be set to true

On the Permissions Tab, add settings for:

Step 3: Connect the Target Device

Part 2: Working with the Camera

Step 1: Defining the App Screen Layout

Step 2: Defining a Custom SurfaceView to Hold

01 public class ArDisplayView extends SurfaceView

Step 3: Implementing the SurfaceHolder

1 public class ArDisplayView extends SurfaceView implements SurfaceHolder.Callback

This will necessitate overriding and implementing three new methods:

Step 4: Implementing the surfaceCreated()

01 public void surfaceCreated(SurfaceHolder holder) {

01 public void surfaceChanged(SurfaceHolder holder,int format, int width,

Step 6: Implementing the surfaceDestroyed()

1 public void surfaceDestroyed(SurfaceHolder holder) {

Step 7: The Custom Camera View and the Main

1 public void onCreate(Bundle savedInstanceState) {

Part 3: Working with the Device Sensors

Step 1: Displaying Overlay Content

01 public class OverlayView extends View {

1 public class OverlayView extends View implements SensorEventListener {

This will necessitate overriding and implementing two new methods:

Step 3: Registering for Sensor Data Updates

Before we implement the SensorEventListener callbacks, we want to register for the

1 SensorManager sensors = (SensorManager) context.getSystemService(Context.SENSOR_SER

Now youre ready to implement the sensor callback, onSensorChanged(), in order to

Step 4: Implementing the onSensorChanged()

01 public void onSensorChanged(SensorEvent event) {

Step 5: Your Custom Overlay and the Main

Difficulty: Suggested Tuts+ Course

Tuts+ tutorials are translated by our community

Create a YouTube Client on Android

About Shane Conder & Lauren Darcey

Web Developer / Site Management

PHP Coder with Magento Knowledge

Envato Market Item

Start the discussion

Be the first to comment.

Subscribe d Add Disqus to your site Privacy

18,890 Tutorials 461 Video Courses

Follow Us Email Newsletters

Get Tuts+ updates, news, surveys &

Custom digital services like logo design, WordPress installation, video

Potrebbero piacerti anche