Android OpenCV Interface

Pi19404
March 30, 2013

Contents

Contents
Android OpenCV Interface
0.1 0.2 0.3 0.4 0.5 Introduction . . . . . . . . . . . Packages used . . . . . . . . . . Processing Raw Camera Data Implementation Details . . . . Compilation Details . . . . . . 0.5.1 Native Code . . . . . . . 0.5.2 Java Code . . . . . . . . 0.5.3 Project Setup . . . . . . 0.5.4 Compilation Setup . . . 0.5.5 Compilation . . . . . . . . 0.6 Launch Application . . . . . . . 0.7 Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3 3 3 6 7 7 7 8 9 10 11 11

2 | 11

Android OpenCV Interface

Android OpenCV Interface

0.1 Introduction
In this article will look at simple application to demonstrate Android Opencv Interface on android platform using OpenCV and javaCV libraries on Ubuntu Linux Platform.

0.2 Packages used

1. OpenCV 2. JavaCV 3. Android

0.3 Processing Raw Camera Data

The android application provides the raw camera image in YUV format as byte image in the registered preview Callback function. The first step is to convert this to a format suitable for interface with Java or Open CV. Thus the aim is to obtain a bitmap image of type ARGB_8888. Different methods can be used to perform this operation.One method is to use this data to create YuvImage object which can be converted to bitmap object. The data represents the byte array which is input to the preview. The functions framework() and getFrameHeight() return the width and height of the image as set in camera parameters.
1 2 3 4

YuvImage yuv_image = new YuvImage ( data , ImageFormat . NV21 , getFrameWidth () , getFrameHeight () , null ) ; Rect rect = new Rect (0 , 0 , getFrameWidth () , getFrameHeight () ) ; ByteArrayOutputStream output_stream = new ByteArrayOutputStream () ; yuv_image . compressToJpeg ( rect , 100 , output_stream ) ;

3 | 11

Android OpenCV Interface

5

mBitmap = BitmapFactory . decodeByteArray ( output_stream . toByteArray () , 0 , output_stream . size () ) ;

The next method is to decode the YUV byte array and construct the BitMap image. The function to decode YUV byte array is given below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

public void decodeYUV4202RGBA ( int width , int height , byte [] data , Bitmap bitmap ) { int [] rgba = new int [ data . length ];; int frameSize = width * height ; for ( int i = 0; i < height ; i ++) for ( int j = 0; j < width ; j ++) { int index = i * width + j ; int supply_index = frameSize + ( i >> 1) * width + ( j & ~1) ; int y = (0 xff & (( int ) data [ index ]) ) ; int u = (0 xff & (( int ) data [ supply_index + 0]) ) ; int v = (0 xff & (( int ) data [ supply_index + 1]) ) ; y = y < 16 ? 16 : y ; float y_conv = 1.164 f * ( y - 16) ; int r = Math . round ( y_conv + 1.596 f * ( v - 128) ) ; int g = Math . round ( y_conv - 0.813 f * ( v - 128) - 0.391 f * ( u 128) ) ; int b = Math . round ( y_conv + 2.018 f * ( u - 128) ) ; r = r < 0 ? 0 : ( r > 255 ? 255 : r ) ; g = g < 0 ? 0 : ( g > 255 ? 255 : g ) ; b = b < 0 ? 0 : ( b > 255 ? 255 : b ) ; rgba [ i * width + j ] = 0 xff000000 + ( b << 16) + ( g << 8) + r ; } mBitmap . setPixels ( rgba , 0 , width , 0 , 0 , width , height ) ; rgba = null ; }

The second method is slower compared to the first method . The third method is the fast method i have found for decoding the camera data. is to use JNI interface.The decoding method is written in C and the call is made from Java to C method using Java native interface. In the below code yuv is represents the camera input data which is a byte array and bgra is the output data which is integer array. The bitmap can be created from the output data.The ARGB bitmap can now be converted to IplImage object provided by Java which is equivalent to Open CV IplImage or Mat data structure.

4 | 11

Android OpenCV Interface This object can also be passed to the native opencv calls and type-casted as Mat data type process_image functions performs the desired computation and passes as output the result of the computation which is returned to the android application. The _yuv variable represents the camera byte array. The _bgra represents the integer array after decoding and processing to be passed back to the camera application. Open CV cvtColor has bultin decoding routine which will be used to decode the byte array to a BGRA Mat image. After BGRA to GRAY processing is performed on the image and then results is sent back to the android application.
1 2 3 4 5 6 7 8

cvtColor ( myuv , mbgra , CV_YUV420sp2BGR , 4) ; process_image ( mbgra ) ; // creating bitmap from output data fBitmap . setPixels ( rgba , 0 ,w , 0 , 0 , w , h ) ; // decoding the bitmap data to IplImage IplImage orgImg = IplImage . create ( cvSize ( fBitmap . getWidth () , fBitmap . getHeight () ) , 8 , 4) ; fBitmap . copyPixelsToBuffer ( orgImg . getByteBuffer () ) ;

The task is to compile this native code into a shared library(so) that can be used by the android application. The easiest method is to place all the code in a single header file.But is a bad designing practice Hence the file can be prepared manually from different C/C++ files as part of pre processing step.The advantage is that there is not need to explicitly write the native interface files as it is auto generated. The header file used for the present application will be called opencv_androidh Now we have a interface between camera data and OpenCV data type and further image processing task can be performed. A simple color conversion example is used to demonstrate the application The entire processing is performed in the native code and re-

5 | 11

Android OpenCV Interface sult is passed to the android application after the BGR2HSV color conversion has been completed.

0.4 Implementation Details

The AndroidActivityMain is the main file responsible for launching the application. In the present android framework the camera frame can be accessed as long as we display is allocated for displaying the camera preview frame. Hence the processed image needs to be written over the camera preview frame. The Class DrawOnTop is responsible of generating a view which is drawn on top of the camera preview . The Preview class is responsible for setting the camera parameter and preview callback routines which provides access to the camera data and opening and closing the camera etc. The Preview class contains an instance of DrawOnTop class. One the camera data is available the native routine is called to process the camera frame and result is store in an array which is used by DrawOnTop class to convert the integer array into a bitmap and display on top of the camera preview.

6 | 11

Android OpenCV Interface

0.5 Compilation Details

0.5.1 Native Code
The first step of compilation using the opencv,javacv method is to write the header files.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

namespace OpenCV { class OpenCVProcess { OpenCVProcess () { } void run ( int width , int height , unsigned char * _yuv , int * _rgba ) { // allocating the array used by opencv Mat myuv ( height + height /2 , width , CV_8UC1 , ( unsigned char *) _yuv ) ; Mat mbgra ( height , width , CV_8UC4 , ( unsigned char *) _bgra ) ; // decoding the camera byte array cvtColor ( myuv , mbgra , CV_YUV420sp2BGR , 4) ; // perfroming BGR to GRAY conversion and writing the result back to the // original array Mat image ; cvtColor ( mbgra , image , CV_BGRA2GRAY ) ; cvtColor ( image , mbgra , CV_GRAY2BGRA ) ;

} }; }

0.5.2 Java Code

The next step is to write the Java Files to native interface with the header files
1 2 3 4 5

@Namespace ( " OpenCV " ) public class ProcessImage { // specifying the path where the header and library is to be found public static final String androidIncludepath = " ../ include /"; public static final String androidLinkpath = " ../ libs / armeabi " ;

7 | 11

Android OpenCV Interface

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

public static final / include / " ; public static final / lib / " public static final CV_MAJOR_VERSION CV_MINOR_VERSION CV_SUBMINOR_VERSION

String genericIncludepath String genericLinkpath int = 2, = 4, = 2;

= " / opt / local = " / usr / local

public static final String CV_VERSION = CV_MAJOR_VERSION + " . " + CV_MINOR_VERSION + " . " + CV_SUBMINOR_VERSION ; // loading the opencv libraries at instantiation static { if ( load () != null ) { String platformName = getPlatformName () ; } } // declaring the interface of native OpenCV class public static class OpenCVProcess extends Pointer { static { Loader . load () ; } public OpenCVProcess () { allocate () ; } private native void allocate () ; public native void run ( int width , int height , byte yuv [] , int [] rgba ) ; } }

0.5.3 Project Setup

The next step is to copy the libraries and set up project dependencies. The the javacv and javacpp files found in the javacv package to the libs directory of the android project and also add the jar files to the project. Copy all the opencv libraries to libs/armeabi directory of the android project The javacv package also provides javacv-android-arm.jar file.Unzip contents of the files which are .so library files and copy them to the libs/armeabi directory of the android project.

8 | 11

Android OpenCV Interface

0.5.4 Compilation Setup

The next step is code compilation.First step is to compile the native code. Go to the libs/armeabi directory in the file system.Open the javacpp.jar file and navigate to the path /com/googlecode/javacpp/ properties and open the file android-arm.properties. make changes in the files as per the local android ndk path and set the appropriate flags as per the arm architecture of the desired target devise. below is the sample of the file that is used in the present project
1 2 3 4 5 6 7 8

9 10

11 12

13 14 15 16 17 18 19 20 21 22

platform . name = android - arm platform . root =/ opt / android - ndk - r7 / path . separator =: compiler . path = toolchains / arm - linux - androideabi -4.4.3/ prebuilt / linux - x86 / bin / arm - linux - androideabi - g ++ compiler . sysroot . prefix = - - sysroot = compiler . sysroot = platforms / android -14/ arch - arm / compiler . includepath . prefix = - I compiler . includepath = sources / cxx - stl / gnu - libstdc ++/ include /: sources / cxx - stl / gnu - libstdc ++/ libs / armeabi - v7a / include /::/ usr / local / include :/ usr / local / include / opencv :/ usr / local / include / opencv2 compiler . options = - march = armv7 - a - mfloat - abi = softfp - mfpu = vfp ffast - math -Wl , - - fix - cortex - a8 compiler . output . prefix = - Wl , - rpath , lib / - DANDROID - ffunction sections - funwind - tables - fstack - protector - funswitch - loops finline - limit =300 - Wall - O3 - nodefaultlibs - fPIC - shared -Wl , - no - allow - shlib - undefined -s -o \ u0020 compiler . options . fastfpu = - march = armv7 - a - mfloat - abi = softfp - mfpu = vfp - ffast - math -Wl , - - fix - cortex - a8 compiler . output . prefix = - Wl , - rpath , lib / - DANDROID - ffunction sections - funwind - tables - fstack - protector - funswitch - loops finline - limit =300 - Wall - O3 - nodefaultlibs - fPIC - shared -Wl , - no - allow - shlib - undefined -s -o \ u0020 compiler . linkpath . prefix = - L compiler . linkpath = sources / cxx - stl / gnu - libstdc ++/ libs / armeabi - v7a /: libs / armeabi2 compiler . link . prefix = - l compiler . link . suffix = compiler . link = log : gnustl_static : gcc : dl : z : m : c compiler . framework . prefix = - F compiler . framework . suffix = compiler . framework = source . suffix =. cpp library . prefix = lib

9 | 11

Android OpenCV Interface

23

library . suffix =. so

0.5.5 Compilation
Place the commands required for compilation in the script file and execute from the workspace/main directory of the android project. First step is to copy the native header file OpenCVProcessImage.hpp to the ndk package include directory.The base directory is as per compiler.sysroot path configured in the above file(/opt/ android-ndk-r7/platforms/android-14/arch-arm/). Next set the java home and other variable required by Java.This Java SDK should be same as one used by the Java IDE used to compilation of android application. Next step is to compile the Java file which provides the native interface passing the javacv and Java jar files as class path arguments. The final step is to run the Java command passing the jar files argument as javacpp.jar ,properties file name as android-arm and platform.root argument as the local android ndk path and compiled native Java class files as input and output is the shared library.

2 3 4 5 6 7 8 9 10

cp / home / pi19404 / OpenCVProcessImage . hpp / opt / android - ndk - r7 / platforms / android -14/ arch - arm / usr / include / opencv_support_class . h export home1 = ` pwd ` cd $home1 export JAVA_HOME =/ opt / softwares - linux / jdk1 .7.0 _04 / JAR = $home1 / libs / javacpp . jar JAR1 = $home1 / libs / javacv . jar SRC = $home1 / src / com / android / opencv / ProcessImage . java LIBS = $home1 / libs / armeabi $JAVA_HOME / bin / javac - cp $JAR - classpath build / classes : $JAR1 : $JAR $SRC $JAVA_HOME / bin / java - Dplatform . root =/ opt / android - ndk - r7 - jar $JAR - properties android - arm - classpath $home1 / src : $JAR1 : $JAR com . android . opencv . ProcessImage -d libs / armeabi

This will generate the libjniProcessImage.so file in the libs/armeabi directory.

10 | 11

Android OpenCV Interface

Next compile the android java application .

0.6 Launch Application

Transfer the apk file generated to device and test the application.

(a) Screenshot Image

Image

0.7 Code
The code can be found in code repository https://github.com/ pi19404/m19404/tree/master/Android/AndroidOpenCV1 or https://code. google.com/p/m19404/source/browse/Android/AndroidOpenCV1. The header files is located in jni directory.The library files are not placed in the repository download them from appropriate packages on send a mail separately for download link.

11 | 11