3D Res (2014) 5:5
DOI 10.1007/s13319-013-0005-0
3DR EXPRESS
Employing Light Field Cameras in Surveillance:
An Analysis of Light Field Cameras in a Surveillance
Scenario
Rogério Seiji Higa • Yuzo Iano •
Ricardo Barroso Leite • Roger Fredy
Larico Chavez • Rangel Arthur
Received: 12 July 2013 / Revised: 10 November 2013 / Accepted: 8 December 2013
Ó 3D Research Center, Kwangwoon University and Springer-Verlag Berlin Heidelberg 2014
Abstract The light field cameras are becoming a
new trend in photography. They have characteristics
that could overcome some problems present in
surveillance videos, like refocusing and depth esti-
mation. The main advantage is that the light field
camera allows focus reconstruction, increasing the
depth of field. However, these new cameras require
different processing power and storage than conven-
tional cameras. In this paper, it is shown how these
cameras work and how they could be employed in
surveillance. Images of a standard surveillance cam-
era versus the plenoptic camera are compared and
some issues about the design of the light field camera
are discussed. Finally, we compare the performance of
video compression standards applied to plenoptic
image sequences. The power signal noise ratio results
show that MPEG4 is the best choice.
R. S. Higa (&)  Y. Iano  R. B. Leite 
R. F. L. Chavez  R. Arthur
State University of Campinas, Campinas, SP, Brazil
e-mail: rhiga@decom.fee.unicamp.br
Y. Iano
e-mail: yuzo@decom.fee.unicamp.br
R. B. Leite
e-mail: rleite@decom.fee.unicamp.br
R. F. L. Chavez
e-mail: rlarico@decom.fee.unicamp.br
R. Arthur
e-mail: rangel@ft.unicamp.br
Keywords Light field  Surveillance 
Plenoptic  Depth of field  Depth map 
Framework
1 Introduction
The image resolution of surveillance cameras has
increased over the years but this does not guarantee a
good quality image to recognize the face of a person of
interest [1]. A big resolution sensor is not enough for
good image quality. The sensor/lens set is what
determines the image quality.
One of the difficulties of the normal surveillance
cameras is that the depth of field is limited, if two
objects at different depths are in the scene, one of them
will be in focus and the other one will be out of focus.
Sometimes the surveillance videos seen on the news
are focused on the front of the shop, the interest point,
but the subject of interest is on the back of the scene
where the image is blurred. The depth of field depends
on the size of the aperture; it increases as the size of the
aperture diminishes. However, an aperture with a
smaller size blocks more light and the image captured
has more noise, also the shutter speed will have to be
slower, which in video applications is limited by the
frame rate.
The light field cameras are becoming a new trend in
photography, with the capacity to adjust the focus
post-capture as the most acclaimed feature. These
123
5 Page 2 of 11
features are ideal to overcome some problems present
in surveillance videos.
Light field cameras can increase the depth field by
capturing the angle information from the light rays.
There is a tradeoff between the angle information and
the image resolution, so the image generated by the
light field camera has a lower effective image
resolution compared to the sensor resolution. There
are many types of cameras that try to sample the
plenoptic function. The coded aperture cameras [2, 3]
use a mask on the aperture plane of a lens, which
encodes the light that reaches the sensor. The convo-
lution of the encoder mask and decoder mask results in
a Dirac delta function at the origin. Another approach
puts a mask near the sensor and uses the principles of
modulation to gather more information in the captured
image [4]. In [5] is proposed a design of a camera that
uses a lattice-focal lens, using pieces of lens with
different focal lengths. Each piece of lens has a
different transfer function for each depth.
The camera used in our experiments was the
Raytrix R5 color GigE model that has a microlens
array in front of the sensor and is based on integral
photography [6, 7]. The digital light field camera was
proposed by Ng [8] and consists of microlenses that
are positioned at a distance f from the sensor and
focused on infinity, where f is the focal length. By the
proposal of Lumsdaine and Georgiev [9], the micro-
lenses are at 4/3f, focusing on the image produced by
the main lens. This design allows the maximum
exploitation of the captured information. Since then,
various companies started researching and the first
commercial light field camera was made by Raytrix
[10] and then followed by Lytro [11]. Although the
drawbacks of the images captured by the light field
cameras are the loss in the sensor resolution, the
quality is not dependent on the resolution alone. For
example, an out of focus high resolution image can be
worse than in a focus low resolution image.
2 Depth of Field
The depth of field is the range where the object appears
sharper in the image. If the object is not on the focus
plane, its image gets blurred. A way to diminish the
blurriness is to reduce the size of the aperture stop; this
will block the rays and make the circle of confusion
smaller. Figure 1 shows two situations with the same
123
3D Res (2014) 5:5
circle of confusion size, which means that the objects’
sharpness are almost the same.
However, looking from the position they are from
the camera, the distance between the objects is bigger
in the second situation. A smaller aperture allows the
objects to be farther from the focal plane and
maintains the same sharpness, as shown in Fig. 1b.
If the objects in Fig. 1a were put at the same distance
that in Fig. 1b, the circle of confusion would be bigger,
decreasing the sharpness.
Decreasing the size of the aperture is a standard
way to increase the depth of field, but this lowers the
quantity of light that enters the camera. A surveillance
camera has to work day and night, so the lower amount
of light is not good for night situations.
3 Plenoptic Function
The plenoptic function describes the light distribution
over space. It has been described as a 7-dimensional
function (5 spatial dimensions, 1 color dimension, and
1 temporal dimension), as defined in Eq. 1.
P ¼ Pðh; u; k; Vx; Vy; Vz; tÞ; ð1Þ
where ðVx ; Vy ; Vz Þ is the observer position, h and u are
the azimuth and elevation angles, k is the wavelength
of the light and t is the time. In RGB colorspace, k
defolds into three more dimensions.
A two plane parametrization of the plenoptic
function was proposed in [13]. This parametrization
has 4 spatial dimensions and enables a better under-
standing of the plenoptic function. Figure 2 shows how
this parametrization is made. The plenoptic function
analysis is made using only two dimensions, as it is
hard to analyze things in four dimensions. Figure 3
shows three objects at different depths, the x-axis
represents the x–y plane and the u-axis represents the
u–v plane. Most of the analysis made in this context can
be extrapolated to the full 4D version, since the
behavior is similar in the others axes. The inclination of
the light field pattern indicates the depth of the object.
The angle of inclination of the red object light field is
larger than the green object although they are at the
same distance from the blue object. This is because the
parameterization is not linear in relation to the depth.
In an ideal pinhole camera, there is only one point
sampling per pixel, which corresponds to the center of
the squares in the right side of Fig. 3. A real pinhole
3D Res (2014) 5:5 Page 3 of 11 5

Fig. 1 Depth of field with

two different apertures [12].
Diminishing the size of the
aperture increases the depth
of field. a Large aperture and
b small aperture

Fig. 3 Plenoptic function sampling of a pinhole camera

Fig. 2 Plenoptic function parameterization using two planes

camera has the sampling more like the squares shown
in the figure. The vertical length represents the size of
the pinhole, while the horizontal length represents the
size of the pixels.
In a standard lens camera, the sampling is made by
integrating all the incident rays into a pixel. In
comparison to the pinhole camera, it samples more
light from the scene, the lens concentrates the light
coming from the focal plane. If the lens is focused on
the green object, the plenoptic representation of the
sampling corresponds to the area inside a parallelo-
gram, as shown in Fig. 4a. The inclination of the
parallelogram represents the focal plane of the lens.
In all previous situations, the sampling occurs along
the u-axis. The light field information along the x-axis

123
5 Page 4 of 11
Fig. 4 Plenoptic function sampling of standard lens camera.
a Focus on green object, b focus on blue object and c focus on
red object
Fig. 5 Plenoptic function sampling of a light field camera
is lost. Figure 5 shows how a plenoptic camera
samples the light field along both axes.
Imagine a sensor in the place of the microlens,
where the size of a pixel is the size of a microlens, as
123
3D Res (2014) 5:5
Fig. 6 The microlens recovers the angle of the incidence. a All
rays focusing on the sensor are integrated, the angle of incidence
is lost. b Putting a microlens in the place of the sensor allows the
recovery of the angle of incidence using a higher resolution
sensor. a Sensor and b microlens
shown in Fig. 6a. All rays falling on that pixel are put
out as a single value and the information about the
angle of incidence is lost. Nonetheless, if a microlens
is put in the place of the sensor, the rays are separated
again and a sensor with a higher resolution can be used
to store the intensity of these rays, as shown in Fig. 6b.
4 Light Field Camera
Section 3 has shown how a light field camera samples
the plenoptic function and captures more information.
In this section it is explained how to render the image
captured by a light field camera into a conventional
image. Also presented is how to extract depth
information and render a view image. The camera
used in this work is the Raytrix GigE R5 model, one of
the first commercial light field cameras.
4.1 Rendering
The resolution of the image produced by the rendering
algorithm described in [8] was limited by the number
of the micro lenses, as it is rendered one pixel per
microlens. In the full resolution rendering described in
[10], a patch from each of the microimages is taken
from the plenoptic image, and then all of them are tiled
together to form a rendered view. This results in more
resolution, rendering more pixels per microlens.
Figure 7 illustrates the process. The size of the patch
is what determines the focal plane (Fig. 8).
3D Res (2014) 5:5
Fig. 7 Process of rendering the focused image
Fig. 8 Example of rendering. a Focus on near object and
b focus on far object
4.2 Calibration
The calibration method used here is suggested by the
Raytrix camera manual. A white image is captured
Page 5 of 11 5
Fig. 9 Rendering with wrong parameters
with an aperture smaller than the ideal size, so the
white circles would be well-defined. The image
rendering process needs to know the centers of the
microimages, as long as, if the centers were not
perfectly matched, the rendered image will present
artifacts. Figure 9 shows these artifacts.
4.2.1 Hexagonal Grid
On a square grid, the position of a microlens can be
found by simple horizontal and vertical translations, but
on a hexagonal grid it is not that simple. The horizontal
position depends on which line of the grid it is on.
Equation 2 shows a method for calculating the micro-
image centers based on the distance between the centers.
 
cx ¼ ox pitch þ 0:5 oy % 2 pitch;
rffiffiffi
3 ð2Þ
cy ¼ o y pitch;
2
where ðcx ; cy Þ are the center coordinates of the
microimages, pitch is the distance between two
microimages, and ox and oy are the horizontal and
vertical indices, respectively. Notice that it is pre-
sumed that the central microlens is at coordinate (0,0)
and the grid is perfectly aligned with the horizontal
axis. Translation and rotation of the coordinates are
applied to fit the calibration image.
4.2.2 White Image
The localization of the centers of the microimages is
made by capturing a white image as shown in Fig. 10.
123
5 Page 6 of 11
Fig. 10 White image used for calibration
Fig. 11 Center found after processing
The MATLAB function regionprops is used to find the
centers of the white circles. Figure 11 shows the results
of this step. Also, from Fig. 11 it can be seen that the
shape of the microimage is a pentagon. This is because
the main lens had five aperture blades. The shape of the
microimage is controlled by the shape of the aperture.
Once the center coordinates are found, the next step
is to find the central microimage coordinate and
compensate any rotation in the image. Then, the pitch
parameters of Eq. 2 are adjusted to match the points
discovered in Fig. 11.
Using a simple white image permits the calibration
can be performed anywhere. Any change in the main
lens adjustments affects the calibration parameters.
Surveillance cameras are normally installed in hard-
to-access places, so a simple calibration method is
well suited.
123
3D Res (2014) 5:5
Fig. 12 Light field image
Fig. 13 Depth estimation
4.3 Depth Estimation and All-In-Focus Rendering
Depth estimation is usually made by finding the disparity
between two images (Fig. 12). The same technique can
be applied to plenoptic images, finding the disparity
between two microimages [14]. The algorithm is simple
when working on a rectangular grid, but a little more
complex on a hexagonal grid. The method for estimating
the depth, described in [15], is more suitable for
hexagonal grids. In analyzing the edges of the micro-
images, if it gets smoother, then probably the patch is at
the correct size. Figure 13 shows a depth estimation
using this algorithm. The depth information can be used
to help people detection algorithms [16].
3D Res (2014) 5:5
Fig. 14 All-in-focus rendering
The depth estimation is not perfect because it is
based on the assumption that the image has textures,
otherwise the estimation goes wrong. See Fig. 14 and,
considering regions that there is no texture, the depth
estimation has an unexpected behavior.
The all-in-focus rendering is made based on the
depth estimation, if the estimation fails, then the
rendering is compromised. A few artifacts near the
wire can be seen in Fig. 14.
5 Framework
The framework proposed here uses the light field
camera to increase the depth of field of surveillance
cameras. The Raytrix R5 color GigE light field camera
is employed as a surveillance camera and the raw feed
is saved into a computer for later processing. This is
not the best scenario in relation to the bandwidth used,
as the raw images require a high bitrate stream.
A second approach possible is to adapt existing
cameras to produce plenoptic images. Olympus
received a patent that adapt Four Thirds standard
cameras into a plenoptic one [17]. The same kind of
adapter can be made for surveillance cameras. The
advantage of this approach would be a fast conversion
of the existing surveillance cameras into light field
cameras. The disadvantage is that the effective
resolution would be lower, and a high power computer
would be needed to view all the streams.
A third approach would be to build a light field
surveillance camera from the start, using the light field
processing inside the camera and send two streams:
Page 7 of 11 5
Fig. 15 Experimental setup using conventional and plenoptic
cameras
one with an all focus image, and the other with a depth
estimation image. Toshiba is developing a small
sensor specialized for light field capture [18], which
can be used for surveillance cameras. The advantages
of this approach would be the better image quality
using a higher resolution sensor, real time processing,
and less storage space. The disadvantage is that the
camera will need a high processing power or special-
ized hardware; also the rendering of an all-in-focus
image could be affected by rendering artifacts.
In our approach, the light field camera is used as a
surveillance camera like the second approach, but the
plenoptic image stream is sent without processing. The
advantage is that the information is not degraded by
any processing and the use of a higher resolution
sensor. The disadvantage is that the storage space
needed will be much higher, and there is also needed a
high processing power for viewing the surveillance
video stream in real-time.
The experimental setup is shown in Fig. 15, where
the Pelco Sarix 2.1 MP surveillance camera is below
the Raytrix R5 color GigE plenoptic camera.
5.1 Surveillance Image Resolution
In the context of surveillance, which is better? A high
resolution image with no special attention to focus
settings or a low resolution but sharp image? Figure 16
shows a comparison between these two situations. The
out of focus image was taken with a surveillance camera
and the low resolution was rendered by the image taken
with the light field camera. The surveillance camera
resolution is 1,920 9 1,080 px and the rendered image
123
5 Page 8 of 11
Fig. 16 Using the light
field camera for surveillance
in an indoor environment.
Surveillance out of focus.
a Surveillance, b plenoptic
image and c rendered from
plenoptic image
is 640 9 480 px. The face of the person on the right in
Fig. 16a is unrecognizable, but in Fig. 16b the face can
be recognized even with the low resolution.
This comparison was made because light field
cameras have an effective resolution lower than the
sensor resolution. As stated earlier, the resolution
alone does not guarantee a good quality image. A low
resolution image is still useful for surveillance as long
as it is in focus.
5.2 Light Field Surveillance
The result of using a light field camera for surveillance is
shown in Fig. 16. Figure 16b is the original plenoptic
image captured, and Fig. 16c is the rendered image from
the original plenoptic image with the focus on the
subject of interest. Considering that the indoor environ-
ment has less light available, Fig. 16c is darker than the
standard surveillance cameras. The Raytrix light field
camera uses a fixed f/8 aperture, while the surveillance
camera has an automatic f/1.9 aperture. The surveillance
camera has the advantage of letting more light in, so the
images have less noise and are brighter.
Observing the images in Fig. 16a, c, the compar-
ison of depth of field of these cameras must take in
consideration that the Raytrix uses a smaller aperture,
123
3D Res (2014) 5:5
which gives a larger depth of field. So the increase of
depth of field is not only due to the light field camera
characteristics, but even with the increase in depth of
field, an out of focus object could still be blurred. So
the advantage of refocusing after capture is still true,
this is useful when the automatic focus sets on the
wrong object of interest like in Fig. 16a, where it is
focused on the poster.
5.3 Performance of Existing Codecs on Plenoptic
Images
Compression is an important topic that affects how
much footage the surveillance camera can store. The
creation of a new standard is not easy and takes a
long time, so knowing the performance of existing
compression standards over plenoptic images is
useful. The comparison is made not only on the
raw plenoptic images, but also on the processed
rendered views. The question is whether the rendered
views will have stronger degradation than the raw
image or not.
The plenoptic image sequences shown in Fig. 17
were processed using three different codecs
(MJPEG2000, MJPEG and MPEG4). The quality
comparison is made in both plenoptic and renderized
3D Res (2014) 5:5 Page 9 of 11 5

Fig. 17 Plenoptic image sequences used in the compression tests. a Sequence 1, b Sequence 2 and c Sequence 3

Fig. 18 Performance of existing compression standards applied on plenoptic images. a Sequence 1, b Sequence 2 and c Sequence 3

images using the PSNR. The qualities of the codecs The size depends not only on the quality parameter,
were adjusted so the compressed file would have but also on the content of the sequence, and, because of
roughly the same size for each sequence. The objec- this, the exact same size for all compressed sequences
tive is to use the same bits per pixel for each codec. is not possible.

123
5 Page 10 of 11
Figure 18 shows that the compression standards [19]
give good results, even if it is not taking advantage of the
plenoptic image geometry. The full shapes markers
present the performance of the compression over the
plenoptic images. The hollow shapes markers present
the image rendered using the plenoptic image without
compression, and a rendered image using a plenoptic
image extracted from the compressed video sequence.
The MPEG-4 standard showed the best performance
overall, followed by MJPEG2000 and MJPEG.
The objective of this section is to show that, if a
normal surveillance camera were adapted using a
microlens array or using existing codecs on light field
cameras, the captured plenoptic images through the
compressed camera stream would have an acceptable
performance.
6 Conclusion
This work has analyzed how the light field camera can
be used in a surveillance application. The optics of the
plenoptic camera allows the focus reconstruction and
increases the depth of field.
Light field cameras can improve people detection
algorithms and the depth information could be used to
better segment the scene, but the algorithms of depth
estimation are not perfect and may fail when there are
no edges on the image.
The size of aperture controls how much light enters
the camera, and on a light field camera it also controls
the size of the microimage. A standard surveillance
camera has a typical maximum aperture of f/1.9, which
is ideal for low light situations. The light field cameras
have to match the microlens and main lens aperture,
otherwise the microimages would get bigger and
interfere with each other. The Raytrix camera used has
a fixed aperture of approximately f/8, which is four
times less light than a surveillance camera. So the
camera used is not suited for surveillance applications,
mainly because of the small fixed aperture.
The desirable characteristics of a light field sur-
veillance camera would include a large aperture
opening, low f-number, and a main lens compatible
with the surveillance camera field of view. A careful
microlens design would allow more light inside the
camera and make the light field camera suitable for
surveillance. The Lytro camera, for example, has a
fixed f/2 aperture, so such design is possible.
123
3D Res (2014) 5:5
Another concern in surveillance is the video
storage, which can be used as evidence. The size of
the image captured by the Raytrix camera is
2,560 9 1,920 px, which is five times more informa-
tion than a HD surveillance camera. The storage has to
be increased five times or the system will record five
times less footage. The experiments showed that
existing compression standards could be used with the
light field camera, so a fast solution for storing
plenoptic images would not be an issue, but dedicated
compression could increase the efficiency and reduce
the space needed by the plenoptic image.
The future of light field cameras is very promising,
with new results appearing very frequently. The
application of this technology on surveillance cameras
seems a logical next step to improve security systems,
but, as mentioned, the camera must be designed for
this application in mind.
Acknowledgments The authors would like to thank the
following institutions for making this work possible:
Coordenação de Aperfeiçoamento de Pessoal de Nı́vel
Superior (CAPES), Programa de Formação de Recursos
Humanos em TV Digital (CAPES-RHTVD), Centro de
Pesquisa e Desenvolvimento em Tecnologias Digitais para
Informação e Comunicação—Rede Nacional de Ensino (CTIC-
RNP), Conselho Nacional de Desenvolvimento Cientı́fico e
Tecnológico (CNPq), Fundação de Amparo à Pesquisa do Estado
de São Paulo (Fapesp), Fundo de Apoio ao Ensino, Pesquisa e à
Extensão—State University of Campinas (Faepex-Unicamp).
References
1. Kovesi, P. (2009). Video surveillance: Legally blind? In
Digital image computing: techniques and applications,
DICTA 2009, 1–3 December, 2009, Melbourne. IEEE
Computer Society. doi:10.1109/DICTA.2009.41.
2. Zhou, C., & Nayar, S. K. (2009). What are good apertures
for defocus deblurring? In IEEE international conference
on computational photography.
3. Marcia, R. F., Harmany, Z. T., & Willett, R. M. (2009).
Compressive coded aperture imaging. In Proc. SPIE 7246,
Computational Imaging VII. doi:10.1117/12.803795.
4. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., &
Tumblin, J. (2007). Dappled photography: Mask enhanced
cameras for heterodyned light fields and coded aperture
refocusing. ACM Transactions on Graphics, 26(3), 1–12.
doi:10.1145/1276377.1276463.
5. Levin, A., Hasinoff, S. W., Green, P., Durand, F., & Freeman,
W. T. (2009). 4D frequency analysis of computational cam-
eras for depth of field extension. ACM Transactions on
Graphics, 28(3), 97:1–97:14. doi:10.1145/1531326.1531403.
6. Ives, F. (1903). Parallax stereogram and the process of
making same. US Patent 725,567.
3D Res (2014) 5:5
7. Lippmann, G. (1908). La Photographie Integrale. Comptes
Rendus Academie des Sciences, 146(9), 446–451.
8. Ng, R. (2006). Digital light field photography. Dissertation,
Stanford University.
9. Lumsdaine, A., & Georgiev, T. (2008). Full resolution
lightfield rendering. Adobe Tech Report, January 2008.
10. Perwass, C., & Wietzke, L. (2012). Single lens 3D-camera
with extended depth-of-field. In Proc. SPIE 8291, Human
vision and electronic imaging XVII, 829108. doi:10.1117/
12.909882.
11. Lytro. (2013). Accessed November 11, 2013, from https://
www.lytro.com/.
12. Higa, R. (2013). What is depth of field? Accessed Novem-
ber 11, 2013, from http://www.photographycompendium.
com/what-is-depth-of-field/.
13. Levoy, M., & Hanrahan, P. (1996). Light field rendering. In
Proceedings of the 23rd annual conference on computer
graphics and interactive techniques, SIGGRAPH’96 (pp.
31–42). New York: ACM. doi:10.1145/237170.237199.
14. Georgiev, T., & Lumsdaine, A. (2010). Reducing plenoptic
camera artifacts. Computer Graphics Forum, 29(6),
1955–1968. doi:10.1111/j.1467-8659.2010.01662.x.
Page 11 of 11 5
15. Wanner, S., Fehr, J., & Jahne, B. (2011). Generating EPI
representations of 4D light fields with a single lens focused
plenoptic camera. Advances in Visual Computing, Lecture
Notes in Computer Science, 6938, 90–101.
16. Salas, J. & Tomasi, C. (2011). People detection using color
and depth images. In Proceedings of the third Mexican
conference on pattern recognition, MCPR’11 (pp. 127–135).
Berlin: Springer. http://dl.acm.org/citation.cfm?id=2026143.
2026160.
17. (2013). Olympus patent: Light field adapter for micro four-
thirds cameras. Accessed June, 2013, from http://lightfield-
forum.com/2013/06/olympus-patent-light-field-adapter-for-
micro-four-thirds-cameras/.
18. Alabaster, J. (2013). Toshiba demos ‘Lytro chip,’ converts
phones to light-field cameras. Accessed June 2013, from
http://www.techhive.com/article/2029395/toshiba-demos-
lytro-chip-converts-phones-to-lightfield-cameras.html.
19. Higa, R. S., Chavez, R. F. L., Leite, R. B., Arthur, R., &
Iano, Y. (2013). Plenoptic image compression comparison
between JPEG, JPEG2000 and SPITH. Cyber Journals:
JSAT, 3(6).
123