2015 3rd IAPR Asian Conference on Pattern Recognition
Stereoscopic Image Warping for Enhancing Composition Aesthetics
Md Baharul Islam, Wong Lai-Kuan, and Wong Chee-Onn
Multimedia University, Malaysia
bahar mag@yahoo.com, {lkwong, cowong}@mmu.edu.my
Abstract
The increased popularity of stereo photography due
to the availability of stereoscopic lens and cameras has
aroused research interest in stereo image editing. In this
paper, we present an automatic, aesthetic-based warping
approach to recompose both the left and right stereo image
pair simultaneously using a global optimization algorithm.
To maximize image aesthetics, we minimize a set of aes-
thetics errors formulated based on selected photographic
composition rules during the warping process. In addition,
our algorithm attempts to preserve the stereoscopic prop-
erties by minimizing disparity change and vertical drift in
the resulting image. Experimental results shows that our
approach successfully relocate salient objects according to
the selected photographic rules to enhance compositional
aesthetics and maintain disparity consistency to create a
comfortable 3D viewing experience.
1. Introduction
Due to the availability of stereoscopic lens and cameras
as well as stereoscopic displays, stereo photography is be-
coming increasingly popular. As in 2D photography, the
aesthetics of a stereo image can be significantly enhanced
by adopting some photographic composition rules such as
rule-of-thirds and visual balance. However, amateur stereo
photographers are often unaware of these rules and post-
processing techniques to enhance their images can often be
laborious as it involves modifying not just one image, but a
stereo image pair.
In the recent years, researchers have proposed some
promising interactive stereo image editing tools [2, 7, 10].
These tools simplify the stereo image editing tasks by al-
lowing objects in a stereo image pair to be easily selected
and modified, while ensuring the stereoscopic properties
of the stereo image pair are preserved during the process.
Experienced photographers who are aware of the aesthet-
ics composition rules would find these interactive stereo
editing tools very useful. However, amateur photographers
may be at a lost on how to make their stereo photographs
978-1-4799-6100-9/15/$31.00 ©2015 IEEE
Kok-Lim Low
National University of Singapore
lowkl@comp.nus.edu.sg
more aesthetically pleasing. Thus, an automatic, aesthetics-
driven image recomposition tool would be more desirable.
To our best knowledge, stereo cropping [8] is the only auto-
matic recomposition tool available. This approach utilizes
the cropping method to recompose a stereo image, where
salient objects are recomposed based on selected photo-
graphic composition rules. It works very well in recompos-
ing stereo images with sufficient uninteresting background
that can be cropped away. However, this cropping method
suffers from loss of content and may not work well when
a single object occupies a significant portion of the image
frame or multiple important objects are spread all over the
image. Furthermore, the stereoscopic depth perception may
be reduced if the stereo image is aggressively cropped.
To overcome the limitations of stereo cropping, we pro-
pose an aesthetics-driven, recomposition approach based on
image warping technique. Our approach aims to modify the
composition of an image based on selected photographic
rules while preserving the stereoscopic properties of the
stereo image pair. We first formulate a set of aesthetics qual-
ity errors based on two popular photographic rules; rule of
thirds and visual balance. Next, we formulate the stereo-
scopic error that is aimed to minimize changes in disparity
between the source and recomposed stereo image. Our al-
gorithm then recomposes both the left and right images of
a stereo image pair simultaneously using a triangular-mesh
based warping method that minimizes the stereoscopic and
aesthetics quality errors. In addition, our approach can be
extended to perform aesthetics-driven stereo image retarget-
ing. Experimental results show that our approach success-
Figure 1. Comparison of aesthetics-driven recomposition results.
(left) Left image of stereo input pair, (middle) results of stereo
cropping [8], and (right) results of our approach.
645
fully preserves the stereoscopic properties and modify the
composition of stereo images according to the selected pho-
tographic rules. Figure 1 illustrates that our approach can
preserve the global image context better than stereo crop-
ping and thus, reduces the loss of image content.
2. Related Works
Stereo image retargeting is the process of resizing a
stereo image pair to fit a targeted display. The retarget-
ing process involves modifying the composition of an im-
age such that the important elements in the image is pre-
served. In the recent years, researchers has proposed a va-
riety of methods such as cropping [8], seam-carving [1],
warping [2, 6] and shift-map [9] to perform stereo image
retargeting. Each of these approaches has its own strength
and produces good retargeting results for certain category of
images. However, all these approaches, except for [8], do
not consider the enhancement of image aesthetics during the
retargeting process. Niu et al. [8] proposed an aesthetics-
driven, cropping-based approach to modify the composi-
tion of an image based on a set of selected photographic
rules during the retargeting process. This approach pro-
duces promising results in images with sufficient uninter-
esting background but fails in images with multiple subjects
as well as images with photo subject that fills the frame.
In the recent years, interactive stereo image editing tools
is also starting to gain attention. Content-aware stereo-
scopic image editing [2] method allows users to interac-
tively modify the composition of a stereo image by adjust-
ing the location, and depths of selected objects, giving users
aesthetic control for depth perception. The 3D Copy &
Paste [7] method enables foreground objects from a source
stereo image to be copied to a target stereo image while en-
suring the resulting image produces comfortable 3D view-
ing. The StereoPasting [10] approach went a step further
to allow objects from a 2D image to be copied and blended
on a stereo image. These interactive tools are useful for
experienced photographers who are aware of compositional
aesthetics but amateur photographers without knowledge of
image aesthetics may find these tools undesirable.
3. Proposed Method
We propose an aesthetic-driven warping method that
aims to enhance compositional aesthetics of a stereo image,
while ensuring the stereoscopic properties are preserved.
In order to maximize image aesthetics, our approach min-
imises a set of aesthetics errors formulated based on two
common photographic composition rules: the rule of thirds
and visual balance. According to the rule of thirds, an im-
age frame is divided into 9 equal parts by two horizontal and
vertical lines and the four intersection points that are formed
by these lines are called power points. Photographers are
646
Figure 2. Example of images that adhere to (left) rule of thirds
and (right) visual balance.
encouraged to keep the center of mass of the photo subjects
on these power points as illustrated in the left image of Fig-
ure 2. Visual balance rule is illustrated by the right image in
Figure 2 where two objects of varying size counterbalance
each other within the frame.
Figure 3 illustrates an overview of our approach. Given
a stereo image pair, our algorithm first performs pre-
processing to obtain the disparity maps, triangular meshes
and importance maps for the stereo image pair. Then, both
the left and right image are optimized simultaneously to re-
compose the stereo image. The optimization algorithm min-
imisies a set of errors that consists of warping errors, stereo-
scopic error and aesthetics quality errors. This approach can
also be extended to support stereo image retargeting by pro-
viding the target image scale.
3.1. Pre-processing
We first compute the disparity map from a stereo im-
age pair using Sum of Absolute Difference (SAD). Next,
we construct a triangular mesh over the left stereo image.
The correspoinding triangular mesh is then propagated to
the right image using the disparity information. We then
generate the importance maps for the stereo image pair us-
ing the graph based visual saliency algorithm [4]. Triangles
in the triangular meshes that correspond to areas of high
saliency in the importance maps are marked as salient tri-
angles. In order to avoid distortion of salient objects in the
stereo image, these salient triangles will be kept as rigid as
possible during the optimization process.
3.2. Warping errors
Let L and R denote the left and right image respectively.
In the warping process, the source triangular meshes,
ML and MR are mapped to the target meshesML and
MR . The set of triangles and objects in ML MR are
T and O respectively. The warping quality errors is the
formulated as the weighted sum of scale transformation
error, smoothness error, and stereoscopic error.
Scale transformation error. For each triangle tεT , we per-
form non-uniform scaling [5]. Then the scale transforma-
tion error Ew is defined as,
Figure 3. Overview of our proposed aesthetics-driven, stereo recomposition approach. Sx and Sy indicate the target horizontal and vertical
scale respectively.

 2
Ew = At Jt − Gt F
tεT
2 should be kept as boundary vertices in the outputstereo im-
where, At is the area of triangle t, .F is the Frobenius
norm, and Jt is a 2 × 2 Jacobian matrix that maps a triangle
to its corressponding triangles in the output mesh.
Smoothness error. The smoothness error is formulated to
avoid the discontinuity between the neighboring triangles,
 2
Es = Ast Gt − Gs F (2)
s,tεT
where Ast = (As + At )/2 and s, t are adjacent triangles.
Stereoscopic quality error. To preserve the stereoscopic
properties, we minimize the change of two stereoscopic
properties [6]; (1) disparity, and (2) vertical drift between
the left and the right output meshes. Let, (pL
i , pi ) and
R
where α, β, γ are the corresponding weights.
(1)
Image boundary constraint. Original boundary vertices
age. For each boundary vertex v of mesh ML MR , we
apply the positional constraints vx = 0, vx = W, vy =
0, andvy = H to the left, right, top and bottom border ver-
tices respectively, where W and H are the width and height
of the output stereo image.
3.3. Aesthetic quality errors
Disparity needs to be considered in the aesthetic rules
implementation. A disparity value can be positive or
negative, indicating whether the particular point in the
stereo image is located behind or in front of the screen.
In order to relocate the object to a more accurate power
point or power line location, we compensate the disparity
variation by substracting or adding half of the centroid

(qi , qi ) denote 
L R
the set of corresponding pointsof the in- disparity value from the object centroid accordingly.
put meshes ML MR and output meshes, ML  MR  re-
sepectively. The stereoscopic quality error is defined as The rule-of-thirds. The rule of thirds error is formulated
 as,
Et = Ed (pL
i , pi ) + Ev (pi , pi )
R L R

(3) D ci
(qiL ,qiR )εT EP = (At Dp (Oi ) − ), Dci >= 0 (7)
2
iεO
where Ed indicates the disparity consistency, Ev ensures
 D ci
zero vertical drift. EP = (At Dp (Oi ) + ), Dci < 0
2 (8)
iεO
2 where Dp (Oi ) is the distance between object i and the near-
Ed (pL
i , pi ) = ((qi (x) − qi (x)) − (pi (x) − pi (x)))
R R L R L

(4) est power point, At is the normalized area of object i, and

Dci is the centroid disparity of object i.
2
Ev (pL
i , pi ) = (qi (y) − qi (y))
R R L
(5) Figure 4 shows our results with and without disparity
where (x) and (y) refers the x and y coordinate of the compensation for rule of thirds. The disparity value of the
particular point. object centroid is 16 pixels. In our results, the object cen-
troid is moved 8 pixels nearer to the power point compared
Total warping error. The total warping error, Ewarp is, to the results without disparity compensation.
To create a more pleasant aesthetics experience, pro-
Ewarp = αEw + βEs + γEt (6) fessional photographers normally leave more space in the

647

Figure 4. Disparity consideration for rule of thirds formulation.
(left) Original stereo image; black eclipse, red rectangle and yel-
low dot indicates the positive disparity, negative disparity and ob-
ject centroid respectively, (middle) our result without disparity
compensation, and (right) with disparity compensation.
facing direction of the salient object. Therefore, if an
optional face direction is specified, our algorithm restricts
the salient object to be relocated to the power points that
leave sufficient space in the facing direction.
Visual balance. The visual balance error, EV B is defined
as,
 D ci
EV B = (At C(I) − C(Oi ) ± ) (9)
2
iεO
4. Results and Discussion
Our results in this paper are generated on a desktop with
Intel i7 CPU 3.40GHz. The computation time depends on
the number of triangles used to generate the triangular mesh
for the stereo image pair, which can be kept relatively con-
stant for images of different resolution. It took about 4 sec-
onds to produce a recomposed stereo image with resolution
1024 × 768 and 1360 triangles.
In Figure 5, we compare our results generated with and
without minimization of the stereoscopic error. It is obvious
that the disparity is not preserved, and depth perception is
lost if stereoscopic properties is not considered during the
optimization. On the other hand, by including the stereo-
scopic errors in the optimizaton, the disparity of the output
image is well preserved.
Next, we examine the adherence of our results to the se-
lected photographic composition rules. Our recomposition
results in Figure 6 illustrates that our approach successfully
re-position the photo subject nearer to one of the power
points or power lines depicted by the rule of thirds. In Fig-

where C(I) is the image center, C(Oi ) is weighted centroid

of object i, and Dci is the centroid disparity of object i.
The visual balance error is EV B = 0 for image with only
one object.

Total aesthetic quality errors. The total aesthetic error,

EA is defined as,

EA = δEP + ηEV B (10)

where δ and η are the weights of each aesthetic quality error.

3.4. Total errors Figure 5. (left) Input stereo image, (middle) our results without
disparity preservation, and (right) our results with disparity preser-
To perform automatic stereo recomposition, we mini- vation.
mize the total errors,

Etotal = Ewarp + EA (11)

where Ewarp is the total warping errors and EA is the total
aesthetic errors.

3.5. Implementation details

The minimization of the total errors function in Equa-
tion 11 is a convex quadratic function. The warping error
weights α, β and γ are set to 1, 0.5 and 0.1 respectively in
our experiment. The aesthetics quality error weights δ is set
to 0.8 and η is set to 0.5 for better composition aesthetics.
We can allow users to modify the weights; δ and η in order
to adjust the composition to suit their preferences. These
weights depend on the size and position of foreground ob- Figure 6. Results that adhere to rule of thirds. (column 1, 2) Input
jects. We utilize the CVX Matlab toolbox [3] to find the left, and stereo (red-cyan) image, and our results; (column 3) left
solution to the quadratic function. image, and (column 4) stereo image (red-cyan).

648

Figure 7. (left) Left image of stereo input pair, (middle) result with-
out face direction consideration, and (right) result with face direc-
tion consideration.
Figure 8. Results that adhere to visual balance rule. (column 1, 3)
Input left stereo image, and (column 2, 4) our results.
ure 7, our results show that if the face direction is specified,
our algorithm successfully move the object to the power
point that leaves more space in front of the object, giving
it a more pleasant aesthetics feel. Figure 8 shows that our
method is able to achieve better visual balance for images
with more than one salient objects.
Our approach can also be utilized to perform aesthetics-
driven stereo image retargeting by specifying the target im-
age scale. In Figure 9, we compare our retargeting results
to the state-of-the-art stereo retargeting approaches [1, 5].
We can observe that the photo subject is squashed and dis-
torted in the results of uniform scaling and seam carving
respectively but is well-preserved in non-aesthetics driven
warping [5] and our approach. In terms of aesthetics com-
parison, only our approach is able to relocate the photo sub-
ject to the left power line according to rule of thirds, thus
enhancing the image aesthetics.
5. Conclusion
In this paper, we present an automatic, aesthetic-based
warping approach to recompose a stereoscopic image using
a global optimization algorithm. To maximise image aes-
thetics, we minimize a set of aesthetics errors formulated
Figure 9. Comparison of our retargeting results with state-of-the-
art methods. (left to right) Input stereo image, uniform scal-
ing, seam carving [1], non-aesthetics driven warping [5] and our
aesthetics-driven warping approach.
649
based on selected photographic composition rules during
the warping process. In addition, our algorithm attempts
to preserve the stereoscopic properties by minimizing dis-
parity change and vertical drift in the resulting image. Ex-
perimental results shows that our approach successfully re-
locate salient objects according to the selected photographic
rules and maintain disparity consistency to create a comfort-
able 3D viewing experience. Besides, compared to stereo
cropping, our method can better preserve the global image
context, leading to reduced content loss. Drawback of our
approach includes unavaoidable feature disstortion in stereo
images with complex background. In this work, our algo-
rithm preserves the vertical drift and thus, limit the reloca-
tion of objects to its current depth location. In our future
work, we would explore changing the depth of the objects
to enhance 3D depth perception.
Acknowledgement
This work is supported by FRGS Grant No.
EP20130326018 and MMU Grant No. IP20131108001.
References
[1] T. Basha, Y. Moses, and S. Avidan. Geometrically consistent
stereo seam carving. In ICCV, pages 1816–1823, 2011. 2, 5
[2] C. H. Chang, C. K. Liang, and Y. Y. Chuang. Content-aware
display adaptation and interactive editing for stereoscopic
images. IEEE TMM, 13(4):589–601, 2011. 1, 2
[3] M. Grant, S. Boyd, and Y. Ye. Cvx: Matlab software for
disciplined convex programming, 2008. 4
[4] L. Itti, C. Koch, and E. Niebur. A Model of Saliency-Based
Visual Attention for Rapic Scene Analysis. IEEE TPAMI,
20(11):1254–1259, 1998. 2
[5] Y. Jin, L. Liu, and Q. Wu. Nonhomogeneous scaling opti-
mization for realtime image resizing. The Visual Computer,
26(6-8):769–778, 2010. 2, 5
[6] K.-Y. Lee, C.-D. Chung, and Y.-Y. Chuang. Scene warping:
Layer-based stereoscopic image resizing. In IEEE CVPR,
pages 49–56, 2012. 2, 3
[7] W.-Y. Lo, J. van Baar, C. Knaus, M. Zwicker, and M. Gross.
Stereoscopic 3d copy & paste. In ACM TG, volume 29, page
147, 2010. 1, 2
[8] Y. Niu, F. Liu, W. C. Feng, and H. Jin. Aesthetics-
based stereoscopic photo cropping for heterogeneous dis-
plays. IEEE TMM, 14(3):783–796, 2012. 1, 2
[9] S. Qi and J. Ho. Shift-map based stereo image retargeting
with disparity adjustment. In ACCV, pages 457–469. 2013.
2
[10] R.-F. Tong, Y. Zhang, and K.-L. Cheng. StereoPasting: in-
teractive composition in stereoscopic images. IEEE TVCG,
19(8):1375–85, 2013. 1, 2