IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO.

11, JUNE 1, 2013 2887

On Projection Matrix Optimization for

Compressive Sensing Systems
Gang Li, Member, IEEE, Zhihui Zhu, Dehui Yang, Liping Chang, and Huang Bai

Abstract—This paper considers the problem of designing the
projection matrix for a compressive sensing (CS) system in
which the dictionary is assumed to be given. The optimal
projection matrix design is formulated in terms of finding those
such that the Frobenius norm of the difference between the Gram
matrix of the equivalent dictionary and the identity matrix
is minimized. A class of the solutions is derived in a closed-form,
which is a generalization of the existing results. More interestingly,
it is revealed that this solution set is characterized by an arbitrary
orthonormal matrix. This freedom is then used to further enhance
the performance of the CS system by minimizing the coherence
between the atoms of the equivalent dictionary. An alternating
minimization-based algorithm is proposed for solving the cor-
responding minimization problem. Experiments are carried out
and simulations show that the projection matrix obtained by the
proposed approach significantly improves the signal recovery
accuracy of the CS system and outperforms those by existing
algorithms.
Index Terms—Averaged mutual coherence, compressed sensing,
optimization techniques, sparse representation.
The projection matrix , alternatively called measurement ma-
trix or sensing matrix, acts as a multitude of probes on the in-
formation contained in the original signal .
As , recovering the original signal from the mea-
surement given by (1) is a undetermined problem, which has
an infinite number of solutions. Therefore, extra constraints on
the solutions have to be added in order to make it unique.
Sparsity of is actually such a constraint in CS theory, which
ensures that the mapping between and is one-to-one.
Let with being its th element. The -norm of
vector is defined as
with denoting the number of non-zero elements in
(though it is not a norm in a strict sense). In a CS system, the
original signal is assumed to be in the following form

I. INTRODUCTION (2)

C OMPRESSED sensing (CS), also known as “compres-

sive sensing”, is an emerging research area in signal pro-
cessing community. It has attracted a lot of attention since its
where is called the dictio-
nary of the CS system under discussion. A vector given by (2)
is said -sparse in if .
introduction at the early of this century [1]–[3]. Excellent re- By substituting in (1) with (2), can be rewritten as
views on the basics of CS and new directions of research on CS
can be found from [4], [5]. (3)
By nature, CS is a mathematical framework that deals with
accurate recovery of a signal vector from a measure- where the matrix is sometimes referred to as equiva-
ment vector with , where the measurement lent dictionary of the CS system. Spark of a matrix , denoted
paradigm consists of linear projections of the signal vector via as , is defined as the smallest number of columns in
a carefully chosen projection matrix : that are linearly dependent. Clearly, for one has
. It was shown in [6] that any
(1) -sparse signal can be exactly recovered from the observa-
tion/measurement given by via , where is
Manuscript received September 20, 2012; revised December 20, 2012 and
the solution of the following constrained problem:
February 26, 2013; accepted February 26, 2013. Date of publication March
21, 2013; date of current version May 10, 2013. The associate editor coordi- (4)
nating the review of this manuscript and approving it for publication was Prof.
Xiang-Gen Xia. This work was supported by the Grants of NSFCs 61273195
and 61101231, CPSF 2012M511386, and ZJNSF Y1100629.
which can be solved using the orthogonal matching pursuit
G. Li, Z. H. Zhu, L. P. Chang, and H. Bai are with the Zhejiang Provincial (OMP) technique [7] as long as . Furthermore,
Key Laboratory for Signal Processing, College of Information Engineering, it can be shown that the solution to the above problem is the
Zhejiang University of Technology, Hangzhou, 310023, Zhejiang, China
(e-mail: ieligang@zjut.edu.cn; clping@zjut.edu.cn; cezhuzhihui@126.com;
same as the one to the -based minimization below
bh667770@163.com).
D. H. Yang is with the Department of Electrical Engineering and Com- (5)
puter Science, Colorado School of Mines, Golden, CO 80401 USA (e-mail:
yang_de_hui@126.com). while the latter can be solved efficiently using algorithms such
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org. as basis pursuit (BP) [4] and the -based optimization
Digital Object Identifier 10.1109/TSP.2013.2253776 techniques [8].

1053-587X/$31.00 © 2013 IEEE

2888
As seen, the larger the spark of , the bigger the signal space
among which the CS systems can guarantee an exact recovery.
For a given dictionary , the spark of the equivalent dictionary
is determined by the sensing matrix . It would be of great
interest to design such that is maximized. This is
not an easy task as computing the spark of a matrix has com-
binatorial complexity. It is therefore preferable to use alterna-
tive properties of that can be easily manipulated to provide
recovery guarantees. One of such properties is the mutual co-
herence of a matrix. See [6], [9], [10].
The mutual coherence, denoted as , represents the worst-
case coherence between any two columns (atoms) of and
is one of the most fundamental quantities associated with CS
theory. As shown in [6], any -sparse signal can be exactly
recovered from the observation/measurement via (4)
as long as
(6)
The first work reported on optimal design of sensing matrix
was probably the one by Elad in [10]. As (6) can guarantee
exact signal recovery, one would design such that is
minimized. It has been observed, however, that the CS system
with such a sensing matrix usually yields a lower recovery ac-
curacy than that with a randomly generated sensing matrix. In
fact, (6) is just a worst-case bound and can not reflect the average
signal recovery performance. It is due to this reason that, instead
of , an averaged mutual coherence, denoted as , thonormal and the diagonal elements of
was dealt with in [10]. A numerical procedure was given with
the intention of reducing and simulation results showed
that the optimized sensing matrix outperforms the one generated
randomly in terms of signal recovery accuracy.
The most challenging issue in projection/sensing matrix de-
sign, as pointed out in [10], is to derive a measure that is di-
rectly related to a CS performance (say signal recovery accu-
racy). Unfortunately, such a measure is not available yet. In CS
theory, it is desired to have such a sensing matrix that the cor-
relation between any two columns of the equivalent dictionary
is as small as possible. This suggests that the Gram matrix, de-
fined as , 1 of the ideal equivalent dictionary
is the identity matrix. Based on this observation, an alternative
approach to learning the sensing matrix was formulated in [11]
by Duarte-Carvajalino and Sapiro for a given dictionary as
(7)
where denotes the Frobenius norm2,
is a singular value decomposition
(SVD) of the dictionary , and . A non-iterative
algorithm was derived for attacking the above minimization
problem in [11]. Compared with Elad’s approach, such a
numerical procedure, though not globally optimal, was reported
to be faster and for some situations, the obtained sensing matrix
led to a more accurate signal recovery than the former.
1Here, denotes the transpose operator, and this will be assumed in the
sequel.
2Throughout this paper, denotes a zero matrix of proper dimension.
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013
A drawback of the approach in [11] is that the measure used
in (7) does not have a clear physical meaning as it has lost the
original intension of making the Gram matrix as close to the
identity matrix as possible due to several approximation proce-
dures involved in. Recently, the following optimal sensing ma-
trix design problem has been investigated [12], [13]
(8)
where denotes the identity matrix of dimension . Compared
with (7), (8) has a much clearer physical meaning, which can be
justified by noting
As seen, the 1st term is the averaged coherence factor (up-to a
factor ), while the 2nd term can be interpreted as a set of
constraints on the norms of equivalent atoms (or frames)
to be one, where is the th column vector of the equivalent
dictionary .
Efforts have been made to solve (8). A gradient-based algo-
rithm was proposed to attack (8) in [12], while a closed-form
solution to (8) was attained in [13] for the case where has
full row rank, i.e., . A similar problem was for-
mulated in [14] for improving the average MSE performance,
which is actually a constrained version of (8) with being or-
being all equal to
one. It was shown in [14] that the solution set to such a con-
strained problem consists of those sensing matrices that make
the frames unit-norm tight frames. One of the main ob-
jectives in this paper is to extend these results to more general
cases.
The main contribution of this paper is three-fold.
• Analytical solutions to (8) and the one with normalization
constraint are derived and more importantly, our results are
applicable to a wider range of dictionaries than those in
[13] and [14];
• It is revealed that there exists a class of sensing matrices,
which, though all being a solution of (8), may yield dif-
ferent Gram matrices. This makes us think of selecting
from this class of optimal sensing matrices a proper one
to enhance the performance of the CS systems further.
• Based on the characterization of the optimal sensing ma-
trices, an iterative algorithm is derived to find the best
sensing matrix in terms of reducing the coherence of the
equivalent dictionary. As to be seen, such a sensing matrix
improves significantly the signal recovery accuracy and
outperforms those obtained using existing approaches.
The outline of this paper is given as follows. In Section II,
some preliminaries on the sensing matrix optimization problem
are provided and two technical lemmas are also given, which
will be used in our own approaches. Our main results are
presented in Sections III and IV. The optimal sensing matrix
problem defined in (8) is investigated in Section III, where
the solutions to the optimal equivalent dictionary design are
characterized and a class of the solutions to the optimal sensing
matrix design problem are derived. As there exist some degrees
LI et al.: ON PROJECTION MATRIX OPTIMIZATION 2889

of freedom in this class of solutions, the best sensing matrix defined as a sequence of column vectors of matrix
issue is discussed in Section IV in terms of further enhancing that satisfies a generalized Parseval condition:
the recovery accuracy of the CS systems. An iterative algorithm
is also derived in this section to find an optimal sensing matrix (12)
that minimizes the coherence of the equivalent dictionary. In
order to demonstrate the performance of the proposed algo- for all , where and are two positive constants.
rithm, computer simulations are carried out in Section V, which Such a frame is said to be -tight if in (5). Clearly, a
show the effectiveness of our proposed method in improving frame is tight if and only if the corresponding matrix
signal reconstruction accuracy. Some concluding remarks are has an SVD of form
given in Section VI to end this paper.

II. PRELIMINARIES This condition is equivalent to the statement that non-zero

Let . The mutual coher- eigen-values of the Gram matrix are all equal to .
ence of this matrix is defined as A unit-norm frame (i.e., ) is said to be
equiangular if
(9)

Denote as the element of the Gram matrix where is some positive constant. It can be shown [15] that
of and the matrix with achieves if
and only if is equiangular tight frame (ETF), and that
can only hold if . For conve-
nience, such an Gram is called ETF Gram, denoted with .
The Gram matrix of , denoted as ,
Clearly, the orthonormal bases form a special class of equian-
is normalized such that . Obviously,
gular tight frames.
.
An ETF has a very nice averaged mutual coherence behavior
Roughly speaking, measures the maximum linear de-
and has been used in optimal dictionary design [17], [18] and
pendency possibly achieved by any two columns of matrix .
such an idea was recently extended by Abolghasemi, Ferdowsi
It can be shown [15] that for a matrix of dimension ,
and Sanei [19] to the optimal projection matrix design problem
is bounded with
formulated:

(13)
with the low bound given by
where the dictionary is assumed to be given and
is the targeted Gram which belongs to
(10)
the space

Note that when , the low bound is approximately equal

to . (14)
Simulations have shown that the recovery accuracy of a CS
system is more related to the average behavior of the off-diag- in which is a constant to control the searching space.
onal elements of the normalized Gram matrix than the one When , the ideal ETF Grams of dimension (with all pos-
whose absolute value is . The first measure used for de- sible ranks) are confined in [17]. The problem defined by (13)
sign of optimal sensing matrix is actually an averaged mutual is usually attacked with alternating minimization methods [18],
coherence, defined as [10] [19], where the dictionary/projection matrix is updated using a
gradient-based algorithm.
The design strategy (13) was partially motivated by the fact
(11)
that it is more realistic to make be an ETF Gram than the
identity matrix as the latter can not be achieved. This is par-
where with a ticularly true for over-complete dictionary design. As far as the
given number and is the number of elements in the index projection matrix design problem is concerned, it is not always
set . It should be pointed out that was used as an possible to find a projection matrix such that the Gram of the
indicator of convergence for a proposed iteration procedure but corresponding equivalent dictionary is an ETF Gram because
not minimized directly in [10]. of, as pointed out in [19], 1) dimensionality constraint and 2)
being constrained by the given dictionary , which greatly re-
A. Tight Frames and Equiangular Tight Frames duces the number of degrees of freedom in . Noting this fact,
The concept of frames [16] plays a very important role in it was recommended in [19] to use an enlarged searching space
signal analysis. A finite frame for the Hilbert space is by taking .
2890 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013

B. Some Technical Results Having been well equipped, we are now going to investigate
the problem (8) in the next sections.
In this subsection, we set up several results that will be used
in the next sections to solve (8). In order to characterize the
solutions to (8), we need the following technical lemmas. III. SENSING MATRIX OPTIMIZATION
Lemma 1: Let be a differentiable function of
In this section, we investigate the optimal sensing matrix de-
a matrix and , where
sign problem defined by (8). As to be seen, a class of the solu-
and are matrices of a proper dimension with all in-
tions to such a problem can be characterized by an orthonormal
dependent of . Define . Then
matrix which can be chosen almost arbitrarily. This motivates
us to find in Section IV the ‘best’ one that can further improve
(15) the performance of a CS system.

A. Optimal Equivalent Dictionary Design

Proof: Denote as the elementary (column) vector
whose elements are all zero except the th one which is 1. It First of all, we consider the following problem
follows from with
that3 (17)

which is clearly a special case of (8), where . It can

also be viewed as the optimal (equivalent) dictionary design
problem. As to be seen, the results established in this subsec-
tion provide the foundation for solving the optimal sensing ma-
trix design problem (8).
One of our key results in this paper is presented in the fol-
Noting that , one has lowing theorem.
Theorem 1: Let with as well as
, and be its Gram matrix. Then, (i)
The solutions of the problem defined by (17) are characterized
by

and hence (15) follows immediately. (18)

Applying the above technical lemma, one has the following
result. where both and are arbitrary orthonormal matrices of
Lemma 2: Let , where proper dimension, and the minimum is ; (ii) The solution
, where are independent of . Then set of (17) under the normalization constraints

(16) (19)

where is structured by

(20)

where is an arbitrary orthonormal matrix of proper dimension

Proof: First of all, it follows from and is any orthonormal matrix such that (19) holds, and the
that . Denote . The ex- minimum is .
Proof: Define . It follows from Lemma 2
pression for given above in the lemma follows directly as
that
a consequence of Lemma 1. Based on this result, (16) can be
shown with the same procedure as the one used in the proof of
Lemma 1.
In fact, it follows from and
that Any solution to (17) should satisfy

Let be an SVD of . Then it can be

which is equivalent to (16). shown with some manipulations that
3In the expression below, the dimension of and is generally different
from that of and .
LI et al.: ON PROJECTION MATRIX OPTIMIZATION 2891

which leads to due to the fact and hence (18). It is interesting to note that in the 7th footnote of [11],
It is easy to verify that all given by (18) lead to one and Duarte-Carvajalino and Sapiro pointed out a closed-form
the same , the achievable minimum. solution to (7), given in the form . This
As to the 2nd part of the theorem, denote , that actually yields a subclass of the solutions obtained using
is our approach, corresponding to the case .
In the next subsection, we will consider the problem of op-
timal sensing matrix design, where the problem we confront
with is to find those such that for a given dictionary , the
equivalent minimizes .
It can be shown [20] that can be made
to satisfy (19) by a proper choice of if and only if
. The minimization B. Characterization of the Optimal Sensing Matrices
of with the constraints (19) can be solved with the following
Lagrange approach: Theorem 1 states that is minimized if and
only if (see (18)). An intuitive question
one may ask is whether there exists a triple such that
. The answer, as to be seen in Theorem
It turns out from Lemma 2 and that 2 below, is positive and hence such an is an optimal sensing
matrix in the sense that it is a solution of (8).
Theorem 2: Let with and
with an SVD of , where
It then follows from and with being assumed.
that , leading to . Since Define
, and hence ,
yielding (20). It is easy to see that such an yields (21)
.
Comment 3.1:
• One notes that (18) is actually the set of the 1-tight frames, where both and are arbitrary or-
while (20) forms the set of unit-norm -tight frames. thonormal matrices of proper dimension. Then yields a
• A simplified version of the second part of Theorem 1 is class of solutions to the optimal sensing matrix problem defined
given in [14]. Our result (20) yields a clearer characteriza- by (8).
tion of the solution set and hence provides an easier way Proof: Denote and
to construct a unit-norm tight frame than the one in [14].
More importantly, our characterization allows us to deal , where is orthonormal with dimension .
with the optimal sensing matrix design for more general
dictionaries than the orthonormal ones required in [14]. According to Theorem 1, achieves its minimum
• The problem defined in (7), initially formulated and studied if and only if the Gram matrix of the equivalent dic-
in [11], can be attacked using the same procedure. Denote tionary is equal to a . In what to follow, we will
. It then follows from Lemma 2 show that the solutions of subject to
that and hence all possible
solutions should be found from

are characterized by (21).

First of all, it can be shown with some manipulations that
Letting be an SVD of , the above
yields

where . Note that

Let be the orthonormal matrix such that
is diagonal in such a way that the diagonal
elements of the sub-diagonal matrix (of dimension )
are given by the top diagonal elements of . Then it is
easy to see that and hence
The latter simply implies that with
. Clearly, has an SVD of form
2892 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013

. Therefore, if and only if can Taking , it is always possible, as mentioned

before, to find a such that has its diagonal elements
be rewritten into all equal to one. The same result was obtained in [14] under
the condition that the dictionaries are orthonormal, which
is stricter than ours.
• It is impossible in general to find a proper pair
such that has its diagonal elements all equal to one.
This is mainly due to the fact that the degrees of freedom
which indicates , where both and in the matrix are greatly constrained by (i.e., ).
Equation (21) provides a set of solutions to the optimal
are orthonormal with a proper dimension. With such a
sensing matrix design problem (8) in a closed-form and more
obtained above, the expression for in (21) follows directly.
importantly, it reveals that there exist some degrees of freedom
The proof is then completed.
in this solution set. An interesting question is how to exploit
Comment 3.2:
this freedom to find the ‘best’ optimal sensing matrix to further
• The solution set of the optimal sensing matrices
enhance the CS system performance. This is the topic for the
specified by (21) is not empty. In fact, (i.e.,
next section.
and ) yields a subset of the solutions.
• In [12], a gradient-based algorithm was proposed to solve
the minimization of . As seen, there
IV. AN ALGORITHM FOR THE BEST SENSING MATRIX
exists an infinite number of stationary points such that
, the gradient-based algorithms may easily fall It is interesting to note that though any solution by (21) makes
into a local minimum if the initial is not properly chosen, -tight and hence minimized, different
while with (21) a solution can be obtained directly. solutions yield different Gram matrices:
• In [13], the same problem was considered and a closed-
form solution was derived under the condition that has
full row rank, i.e., : (24)

In fact, there exist two types of freedom in it. The first one
which is a particular solution from our solutions given by is —an arbitrary orthonormal matrix of dimension , the
, corresponding to both and equal to the identity second one is . As is an eigen ma-
matrix. As seen, our result (21) is applicable to situations, trix of , so is , and for a given
where . In that sense, it is more , say one has two different Gram matrices, equiva-
general. lently, two different optimal sensing matrices in the sense speci-
Similarly, the constrained optimal sensing matrix design fied in (8). In this section, we focus on how to improve the signal
problem can be formulated as recovery by further minimizing the coherence of the equivalent
dictionary with respect to .
To do so, we propose an algorithm to find an orthonormal
(22)
such that the Gram of the corresponding given by (21) is
close to a Gram matrix in the space defined in (14):
It can be shown in a similar way that if there exist orthonormal
matrices and such that the
matrix given below

where is given by (24).

Comment 4.1: The problem has the same essence as (13) and
can be attacked using the alternating minimization strategy. As
has its diagonal elements all equal to one, then to be seen, the matrix is updated in the same way as the
one used in [18], [19], while the projection matrix (equivalently,
the matrix) is attained analytically rather the gradient-based
(23)
iterative algorithms.
Denote as the normalized Gram of a given Gram
provides a solution of (22), which yields the minimum of of form (24). The corresponding matrix, denoted as ,
for any orthonormal matrix . is obtained by applying the following shrinking operation [18]:
Comment 3.3:
• Consider the case and has full row rank, i.e.,
, for which

(25)
otherwise.
LI et al.: ON PROJECTION MATRIX OPTIMIZATION 2893

The best is then found as a solution of the following Since is symmetric and has the same eigenvalues as
minimization does, one concludes that is bounded by the
sum of the first largest eigenvalues of . This ends the
proof.
(26) Let , where is such an eigenvector matrix
of that and
Now, let us consider (26). It follows from (24) that with . Then the bound of with
can be expressed into , i.e., , is achieved with

which then yields one of solutions to (27), equivalently, to (26).

with with . Based on the discussion above, an iterative algorithm for
finding the best sensing matrix is proposed below.
Clearly, (26) is equivalent to
Initialization: With a given initial (say ) and
a prescribed iteration number , compute the corresponding
(27)
Gram, denoted as .
• Step I: While , find and hence via
The solutions to this problem are given by Theorem 3 below. (25).
Theorem 3: Let be the eigenvalue of the sym- • Step II: With obtained above, find a solution of
metric matrix in the minimization problem (27) with (26) using Theorem 3.
. The value of is bounded from below by • Step III: With , compute the corresponding
Gram matrix using (24). If
(28) , then and go to Step I
with . Otherwise, and stop the
and this low bound is achieved by such a that the sum of the algorithm.
first diagonal entries of is equal to . • Step IV: End while if .
Proof: Denote as the element of the matrix Comment 4.2:
. Then • In a discrete-to-discrete CS system, implementation of the
projection matrix is an important issue, which is espe-
cially true to real-time applications. The matrix of di-
mension , generated either optimally or randomly,
is in general fully parametrized and hence, computing a
measurement vector via such a requires multi-
plications plus additions. Such an amount of
computation can slow down the processing, especially for
large-scale scenarios. It is desired to design the projection
matrix with both recovery performance and implementa-
As understood, tion complexity taken into account. Note that for any
given, according to the -decomposition [21], there ex-
ists an orthonormal matrix such that , where
. Since and
have the same Gram, the CS system implemented with
Let be Hermitian. Suppose and be as the projection matrix not only yields the same recovery
the set of diagonal entries and that of the eigenvalues of , both performance as that with but also improves the imple-
arranged in decreasing order mentation efficiency by reducing multipliers.
• Our proposed method for optimal projection matrix design
contains two steps. The first is intended to find a class of
According to [21] (see pp. 193), the following holds projection matrices that minimize the mutual coherence of
the equivalent dictionaries in the sense specified by (8).
The procedure involved in this step is simple as the so-
lutions are given in a closed-form. The second step is to
identify the best projection matrix from this class of op-
timal matrices in the sense that the corresponding Gram
It then follows from that is close to some ETF Gram. The combination of the two
steps is then expected to yield a high performance projec-
tion matrix, which is supported by simulations to be seen
in Section V. As the update of projection matrix is also an-
2894 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013

TABLE I
PERFORMANCE EVALUATED WITH DIFFERENT MEASURES FOR EACH OF
THE FIVE CS SYSTEMS

Fig. 2. Histogram of the absolute off-diagonal entries of the corresponding nor-

malized Gram matrix to each of the five sensing matrices for
.

TABLE II
PERFORMANCE EVALUATED WITH DIFFERENT MEASURES FOR EACH OF
THE FIVE CS SYSTEMS

Fig. 1. Histogram of the absolute off-diagonal entries of the corresponding nor-

malized Gram matrix to each of the five sensing matrices for
.

alytical, the gradient-based iterative procedure is avoided.

Consequently, our proposed algorithm is more efficient in The statistics of performance evaluation for the five CS sys-
terms of computational complexity than the traditionally tems are given in Table I for the case of
used alternating minimization-based algorithms. .5
The histogram of the absolute off-diagonal entries of the cor-
V. EXPERIMENTAL RESULTS responding normalized Gram matrix to each sensing matrix is
depicted in Fig. 1.
In this section, we present some experiments to illustrate the Comment 5.1:
performance of the CS system, denoted as , that uti- • The results are self-explanatory. As expected, the values of
lizes the optimal sensing matrix obtained using our proposed ap- for and are the same and
proach, and compare it with those CS systems utilizing a random much smaller than the others.
sensing matrix, one particular solution given by (21) with • It is interesting to note that tries to minimize only
equal to the identity matrix, and the ones obtained using the al- those very large off-diagonal elements in the Gram matrix
gorithms given in [10]4 and [11]. For convenience, these CS sys- and that focuses on the total off-diagonal elements
tems are denoted with , , , and , without emphasizing very large ones which may greatly
respectively. Note that the difference between and affect the performance of a CS system, while
lies in the fact that the former uses the optimized . takes advantages of the two algorithms, which explains
why it achieves the lowest as well as .
A. Theoretical Results • It is seen from the histograms in Fig. 1 that the effect of
the optimization in our proposed algorithm is to reshape the
We generate a dictionary with normally distributed histogram such that the number of very correlated columns
entries and a random matrix as the initial condition of the equivalent dictionary is much reduced in com-
(also used in ), then compute the sensing matrix for parison with that by for which is simply set to
each of , , and (with ). the identity matrix. It is then expected that the former yield
Taking as the initial condition, we run the algorithm (using a better recovery accuracy than the latter.
) proposed in [10] and obtain a sensing matrix (used The same observations apply to the case of
in ) after iterations. . See Table II and Fig. 2.
4This algorithm has two parameters and . In our simulations, is 5In Table I and Table II, is the averaged mutual coherence defined in
fixed. (11), in which is set to given by (10).
LI et al.: ON PROJECTION MATRIX OPTIMIZATION 2895

Fig. 3. Evolution of the averaged mutual coherence by and

for the case of .

Both and are iterative. Fig. 3 demon-

strates the evolution of the two algorithms in terms of the av- Fig. 4. (a) Relative error figure versus signal sparsity ; (b) Relative error
eraged mutual coherence . rate versus signal sparsity , for , in which a
vanishing graph implies .
B. Evaluation of Recovery Performance
Now, let us consider the signal reconstruction performance
issue via experiments.
For a given dictionary , a set of -sparse
vectors is generated, where each
non-zero element of is randomly positioned with an Gaussian
distribution of i.i.d. zero-mean and unit variance. The set of the
test signal vectors is produced with . The
measurements are obtained using , where
is an matrix taken from the five sensing matrices ob-
tained in the previous subsection, whose performance has been
seen in Table I and Table II.
The recovered signal is obtained using , where
is the solution of the following minimization:

solved using the OMP algorithm.

We use the relative error figure and the relative error rate, de- Fig. 5. (a) Relative error figure versus measurement dimension
noted as and , respectively, to measure the signal recovery ; (b) Relative error rate versus measurement dimension , for
performance of a CS system: , in which a vanishing graph implies .

matrices for signal sparsity varying from 1 to 7 with

, while Fig. 5 yields the behavior of the five
with the total number of testing realizations ( in different CS systems for the measurement dimension
our experiments) and with the number of elements varying from 12 to 36 with .
in the set defined as Case II:
Similar simulations to Figs. 4 and 5 are shown in Figs. 6
and 7.
Case III: Effects of We have also examined the effect
where is a parameter, indicating that the reconstruc- of the number of atoms on the performance of the CS
tion of is successful if the relative error is smaller than . systems. See Fig. 8.
In that sense, is actually the probability of failing in signal Case IV: Effects of in (25) Like in [19], we have also
recovery. As in [19], we take in our experiments. observed the effect of the parameter in (25) on the signal
Case I: , Fig. 4 shows the signal recon- recovery accuracy. The influence of this parameter to the
struction accuracy of the CS system using different sensing performance of is shown in Fig. 9.
2896 IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013

Fig. 6. (a) Relative error figure versus signal sparsity ; (b) Relative error
rate versus signal sparsity , for , in which a Fig. 8. (a) Relative error figure versus the number of atoms ; (b) Relative
vanishing graph implies . error rate versus the number of atoms , for .

Fig. 7. (a) Relative error figure versus measurement dimension

Fig. 9. (a) Relative error rate and versus ; (b) Relative error rate
; (b) Relative error rate versus measurement dimension , for
and versus , both with .
, in which a vanishing graph implies .

relative error rate than , which supports the argument

Comment 5.2:
of using , instead of the latter, as a performance
• As seen from the simulations, our proposed algorithm out-
indicator [10].
performs the others in terms of signal reconstruction accu-
• It is observed that the overall performance by is
racy. This coincides with the theoretical results obtained in
just 2nd to that by our proposed algorithm for most of
the previous subsection.
the simulations. The fact that both projection matrices are
• As shown in Fig. 9, the best value for the parameter used
the solutions of the optimal sensing matrix design problem
in (25) in our proposed algorithm is around 0.15, very close
(8) indicates that such a design criterion does make much
to the bound . For this example, the optimal
sense compared with other existing ones.
is smaller than . It is believed that the conver-
gence speed of the algorithm can be improved by using a
time varying , starting with , say and VI. CONCLUSIONS
ending with . In this paper, we have investigated the problem of designing
• It is also observed from Fig. 9 that as a function of , the the optimal sensing matrix. The contribution is two-fold. The
behavior of is more coincident with that of the first one is to have characterized the expressions in closed-form
LI et al.: ON PROJECTION MATRIX OPTIMIZATION
for the solutions of the optimal sensing matrix design prob-
lems defined and studied in [11]–[14]. These results are gen-
eralizations of those obtained in [13] and [14], and more inter-
estingly, reveal that there are some degrees of freedom in the
solution set. The second one is to have explored the possibility
of finding those sensing matrices that minimize the coherence
between the atoms of the equivalent dictionary among the so-
lution set. An algorithm has been derived for solving the cor-
responding optimization problem. Experiments have been con-
ducted, which show that the sensing matrix obtained using the
proposed method significantly improves the signal recovery ac-
curacy and outperforms those achieved with the existing ap-
proaches.
As seen, there are still some degrees of freedom in the optimal
sensing matrix set specified in (21) that are not exploited.
For example, as mentioned at the beginning of Section IV, is
not unique and the behavior of the Gram matrix obtained using
(24) may be very different for one from that for another .
How these unused degrees of freedom affect the performance
of CS systems in terms of recovery accuracy and implementa-
tion complexity is under investigation. Besides, further effort is
needed for tight frame-based approaches to sensing matrix de-
sign [22], [23] and more efficient algorithms are also needed for
the alternating minimization-based design strategy. It is noticed
that the performance of a CS system can be enhanced if extra
features of signals are taken into account in the design [11], [13]
and that the criterion (cost function) used for optimal projection
matrix design should be directly related to the performance of
the system in a specific application [14]. All these yield possible
directions for optimal sensing matrix design.
ACKNOWLEDGMENT
The authors would like to thank the reviewers’ constructive
comments and suggestions that help improve the quality of this
paper a lot.
REFERENCES
[1] E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles:
Exact signal reconstruction from highly incomplete frequency infor-
mation,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp, pp. 489–509, Feb.
2006.
[2] D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory, vol.
52, no. 4, pp. 1289–1306, Sep. 2006.
[3] E. J. Candès and T. Tao, “Near optimal signal recovery from random
projection: Universal encoding strategies?,” IEEE Trans. Inf. Theory,
vol. 52, no. 12, pp. 5406–5425, Dec. 2006.
[4] E. J. Candès and M. B. Wakin, “An intoduction to compressive sam-
pling,” IEEE Signal Process. Mag., vol. 25, no. 2, pp. 21–30, Mar.
2008.
[5] M. F. Duarte and Y. C. Eldar, “Structured compressed sensing: from
theory to applications,” IEEE Trans. Signal Process., vol. 59, no. 9, pp.
4053–4085, Sep. 2011.
[6] D. L. Donoho and M. Elad, “Optimally sparse representation in general
(nonorthonormal) dictionaries via minimization,” Proc. Nat. Acad.
Sci., vol. 100, no. 5, pp. 2197–2202, Mar. 2003.
[7] J. Tropp, “Greed is good: Algorithmic results for sparse approxima-
tion,” IEEE Trans. Inf. Theory, vol. 50, no. 10, pp. 2231–2242, Oct.
2004.
2897
[8] M. Zibulevsky and M. Elad, “ optimization in signal and image
processing,” IEEE Signal Process. Mag., pp. 76–88, May 2010.
[9] R. Gribonval and M. Nielsen, “Sparse representations in unions of
bases,” IEEE Trans. Inf. Theory, vol. 49, no. 12, pp. 3320–3325, 2003.
[10] M. Elad, “Optimized projections for compressed sensing,” IEEE Trans.
Signal Process., vol. 55, no. 12, pp. 5695–5702, 2007.
[11] J. M. Duarte-Carvajalino and G. Sapiro, “Learning to sense sparse
signals: simultaneous sensing matrix and sparsifying dictionary opti-
mization,” IEEE Trans. Image Process., vol. 18, no. 7, pp. 1395–1408,
2009.
[12] V. Abolghasemi, S. Ferdowsi, B. Makkiabadi, and S. Sanei, “On opti-
mization of the measurement matrix for compressive sensing,” in Proc.
Eur. Signal Process. Conf., Aalborg, Denmark, Aug. 2010.
[13] L. Zelnik-Manor, K. Rosenblum, and Y. C. Eldar, “Sensing matrix op-
timization for block-sparse decoding,” IEEE Trans. Signal Process.,
vol. 59, no. 9, pp. 4300–4312, 2011.
[14] W. Chen, M. R. D. Rodrigues, and I. J. Wassell, “On the use of unit-
norm tight frames to improve the average MSE performance in com-
pressive sensing applications,” IEEE Signal Process. Lett., vol. 19, no.
1, pp. 8–11, Jan. 2012.
[15] T. Strohmer and R. W. Heath, “Grassmannian frames with applications
to coding and communication,” Appl. Comp. Harmon. Anal., vol. 14,
no. 3, pp. 257–275, May 2003.
[16] O. Christensen, An Introduction to Frames and Riesz Bases. Boston,
MA, USA: Birkhäuser, 2003.
[17] J. Tropp, I. S. Dhillon, R. W. Heath, Jr., and T. Strohmer, “Designing
structured tight frame via alternating projection,” IEEE Trans. Inf.
Theory, vol. 51, no. 1, pp. 188–209, 2005.
[18] M. Yaghoobi, L. Daudet, and M. E. Davies, “Parametric dictionary de-
sign for sparse coding,” IEEE Trans. Signal Process., vol. 57, no. 12,
pp. 4800–4810, 2009.
[19] V. Abolghasemi, S. Ferdowsi, and S. Sanei, “A gradient-based alter-
nating minimization approach for optimization of the measurement ma-
trix in compressive sensing,” Signal Process., vol. 92, pp. 999–1009,
2012.
[20] M. Gevers and G. Li, Parametrizations in Control, Estimation and Fil-
tering Problems: Accuracy Aspects, Communication and Control En-
gineering Series. London, U.K.: Springer-Verlag, 1993.
[21] R. A. Horn and C. R. Johnson, Matrix Analysis. Cambridge, U.K.:
Cambridge Univ. Press, 1985.
[22] W. Chen, M. R. D. Rodrigues, and I. J. Wassell, “On the design of
optimized projections for sensing sparse signals in overcomplete dic-
tionaries,” in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.,
Mar. 2012, pp. 3457–3460.
[23] W. Chen, M. R. D. Rodrigues, and I. J. Wassell, “Projection design for
statistical compressive sensing: A tight frame based approach,” IEEE
Trans. Signal Process., vol. 61, no. 8, pp. 2016–2029, Apr. 2013.
Gang Li (M’92) received the B.Eng. degree in
electrical engineering from Beijing Institute of Tech-
nology, Beijing, China, in 1982, and the M.Eng. and
the Ph.D. degrees, both from Louvain University,
Belgium, in 1988 and 1990, respectively.
He was a Postdoctoral Fellow from November
1990 to April 1992 with the Control Group of Lou-
vain University. From May 1992 to January 2007,
he was a Research Fellow, Lecturer (Assistant Pro-
fessor), and Associate Professor, respectively, with
the School of Electrical and Electronic Engineering,
Nanyang Technological University, Republic of Singapore. Since February
2007, he has been with the College of Information Engineering, Zhejiang
University of Technology, P.R. China, where he holds a position of Qianjiang
Chair Professor. His research interests include signal transforms, system
theory, and digital filter design and implementation. His current researches
focus on compressive sensing, system structure theory, and their applications.
He is a coauthor, with Michel Gevers, of the book Parametrizations in Control,
Estimation, and Adaptive Filtering Problems: Accuracy Aspects (London,
U.K.: Springer-Verlag, 1993) in Communications and Controls Engineering
Series.
2898
Zhihui Zhu received the B.Eng. degree in commu-
nications engineering from Zhejiang University of
Technology, Zhejiang, China, in 2012.
He is now working towards the M.Eng. degree
in communications and information systems at the
Zhejiang University of Technology. His research
interests are in the areas of compressive sensing and
system control.
Mr. Zhu was the Meritorious winner of MCM/ICM
in 2011.
tions to speech analysis, image compression, and optics.
Dehui Yang received the B. Eng. degree in com-
munications engineering from Zhejiang University
of Technology, Hangzhou, China, in 2011. He is
currently working towards the Ph.D. degree at the
Colorado School of Mines, Golden.
His research interests include signal processing
using sparse, low-dimensional signal models and
compressive sensing.
IEEE TRANSACTIONS ON SIGNAL PROCESSING, VOL. 61, NO. 11, JUNE 1, 2013
Liping Chang received the B.Sc. degree in mea-
sure-control technology and instruments from
Zhengzhou University, Zhengzhou, China, in
2003, and the Ph.D. degree in optical engineering
from the Institute of Optical and Fine Mechanics,
Shanghai, in 2008.
She has been with the Zhejiang University of
Technology since 2008, where she currently serves
as an Associate Professor in the College of Informa-
tion Engineering. Her major interests are in signal
processing, compressed sensing and their applica-
Huang Bai received the B.Eng. degree in commu-
nications engineering from Zhejiang University of
Technology, Hangzhou, China, in 2010.
He is now working towards the Ph.D. degree in
control science and engineering at the Zhejiang Uni-
versity of Technology, China. His research interests
are in the areas of digital signal processing and com-
pressive sensing.