Least Squares Pursuit for Sparse Signal Recovery
Nicolae Cleju
Faculty of Electronics, Telecommunications and Information Technology
Gheorghe Asachi Technical University of Iasi, Romania
and
Institute of Computer Science, Romanian Academy, Iasi, Romania
Email: {nikcleju}@etti.tuiasi.ro
AbstractThis paper presents a novel algorithm for sparse
signal recovery, named Least Squares Pursuit, based on least-
squares minimization followed by choosing the atom with the
largest resulting coefficient, in a greedy one-by-one fashion. It
follows a similar approach to that of Orthogonal Matching
Pursuit, but with different prioritization of the constraints. We
propose an efficient implementation for Least Squares Pursuit,
derive theoretical guarantees for signal recovery, and finally
present promising simulation results.
I. I NTRODUCTION
Sparse signal recovery and its applications has been an
active research field in signal processing for more than a
decade. It is now well known [1] that a signal x which has
a sparse decomposition ? in some basis or overcomplete
dictionary
x = D ? , with k? k0 = k,
can be recovered from a reduced set of linear measurements
forming an acquisition matrix M , possibly contaminated with
noise z of energy  = kzk22 :
y = M x + z.
The recovery requires solving the NP-complete optimization
problem
x = D arg min kk0 with ky M Dk22 < . (1)
This is known as compressed sensing [2], and it has been
extensively researched in the last 10 - 15 years.
In the noiseless case, which we consider in this paper,  = 0
and the problem reduces to that of finding the sparsest soution
(in the `0 sense) of an undetermined equation system. We
therefore consider our canonical problem defined as:
arg min kk0 subject to y = D. (2)
where, for brevity we renamed the product M D from (eq. 1)
as simply D. This paper presents a novel greedy algorithm for
solving this problem.
The optimization problem (eq. 2) is known to be NP-hard,
therefore one must settle with suboptimal solutions. One of the
most common algorithms for (approximately) solving (eq. 2)
is Orthogonal Matching Pursuit [3][4] (OMP). OMP iteratively
estimates the support of the solution by progressively adding,
at every iteration, the most likely atom, i.e. the one most
similar to the current approximation residual. Our proposed
algorithm in this paper is based on a similar greedy approach,
but with different prioritization of the internal constraints
compared to OMP.
This paper is organized as follows. Section II presents
the Least Squares Pursuit algorithm and its relation with
Orthogonal Matching Pursuit, and Section III presents an
efficient implementation procedure. Guarantees for successful
recovery are derived in Section IV, and simulation results
are presented in Section V. Finally, conclusions are drawn in
Section VI.
II. T HE L EAST S QUARES P URSUIT ALGORITHM
A. Background
Directly solving (eq. 2) is not feasible because the `0 norm
is making the problem NP-hard. The class of greedy pursuit
algorithms, featuring well-known algorithms like Matching
Pursuit and Orthogonal Matching Pursuit, try to estimate
the support in a one-by-one fashion. Specifically, Orthogonal
Matching Pursuit (OMP) selects a new atom at step k as the
atom number i most correlated with the current residual:
i = arg max kDT r(k) | = kDT (y D (k) )k,
and then updates the current solution estimate (k) by pro-
jecting y onto the set of all selected atoms until this moment,
T (k) . In other words, y is approximated as best as possible
using only the atoms from T (k) , while the remaining atoms,
(k)
which make up the complement set Tc , are not used at all.
This can be expressed as follows:
(k) = arg min ky Dk22 subject to T (k) = 0, (3)
c
where T (k) indicates the restriction of to the indices in
c
(k)
Tc .
The idea of our algorithm is to swap the data-fidelity term
ky Dk22 with sparsity term T (k) = 0 in (eq. 3). Note
c
that neither of the two terms is a priori more important than
the other, since the true solution ? with the true support
T satisfies both terms perfectly. Their relative importance
is just a choice of algorithm design, and OMP chooses to
strictly enforce the sparsity term T (k) = 0, while accepting
c
approximate data-fidelity and working to improve it with each
new atom. We propose to take the opposite route: always
enforce perfect data-fidelity, ky Dk22 = 0, while accepting
non-exact sparsity T (k) , working to improve it with each new
c
Algorithm 1 Least Squares Pursuit The name Least Squares Pursuit (LSP) is justified by the fact
1: (0)
Initialize: T ,
(0)
{1, 2, ...N }
Tc that the algorithm follows the same one-by-one atom selection
2: repeat as OMP (hence the name Pursuit), but the orthogonality
3: Update current solution: induced by OMPs projection step is replaced with a least
squares problem.
(k) = arg min kIT (k) k22 subject to y = D
c
III. E FFICIENT IMPLEMENTATION
4: Update current support: A naive implementation would solve (eq. 4) anew for (k) at
i = arg max kT (k) k every iteration. In the following, we describe a more efficient
c
implementation that starts from the solution at the previous
(k+1) (k)
T =T {i} step (k 1) and updates it. This is analogue to the update
step in OMP.
Tc(k+1) = Tc(k) \ {i}
Consider the initial least-squares solution:
5: until stop criterion
(0) = arg min kk22 subject to y = D.
Since T (0) is empty, this is the full least-squares solution of
selected atom. Thus our update step, given the estimate support the system, and thus it can be represented compactly using the
T (k) at step k, is: Moore-Penrose pseudoinverse of D, denoted here as D :
(k) = arg min T (k) subject to y = D. (4) (0) = D y
c

This approach was inspired by the Greedy Analysis Pursuit .

algorithm [5] designed for the cosparse model (also known Suppose that at step k we have the current estimate of the
as analysis sparsity), which follows the similar approach of support T (k) . The new solution estimate (k) from (eq. 4)
favoring the data-fidelity term over the sparsity one. differs from (0) by the fact that the coefficients of the selected
atoms T (k) are not anymore restricted.
B. Least Squares Pursuit The estimate (k) can be split in two parts, corresponding
(k)
Our proposed algorithm consists of iterating the two usual to the estimate support T (k) and the cosupport Tc , leading
steps of pursuit methods: (i) update the solution estimate given to:
(k) (k)
the current support estimate, and (ii) update the support by y = DT (k) T (k) + DT (k) (k) . (5)
c Tc
choosing one new atom to add.
At step k, considering the current support estimate T (k) and Let us further consider the orthogonal projection operator
(k)
(k)
the remaining atoms in set Tc , the algorithm estimates the P (k) on the span of DT :
solution via a weighted least-squares decomposition: (k) (k)
P (k) = DT (DT ) ,
k
= arg min kIT (k) k22 subject to y = D,
c as well as the complementary orthogonal projection operator
Here, IT (k) is a unit matrix which has 1s only on the locations on its complement subspace:
c
(k) (k) (k)
corresponding to the indices in Tc . Using IT (k) is just a
c
Pc(k) = I P (k) = I DT (DT ) .
notation trick equivalent to simply stating T (k) from (eq. 4),
c
since its effect is to constrain the norm only of the coefficients With every new atom added to T , the rank of operator P (k)
from the set T (k) (the coefficients corresponding to the current is incremented, whereas the complementary subspace spanned
c (k)
support T (k) are left unconstrained). by Pc decreases. As such, the corresponding subspaces
(k)
Thus, we seek the decomposition of y which has minimum spanned by Pc are nested as follows:
norm for the coefficients of the non-selected atoms, with no
Pc(0) Pc(1) ... Pc(k) ...
restriction for the selected ones.
Once the solution estimate is found, the support is updated (k)
Applying Pc to (eq. 5) results in
by adding the atom i with the largest coefficient in T (k) :
c (k)
Pc(k) y = Pc(k) DT (k) (k) = Pc(k) D (k) , (6)
i = arg max kT (k) k, c Tc
c
(k) (k)
i.e. the atom most likely to belong in the support (in other since Pc is orthogonal to DT .
(k)
words, the atom most likely not to belong in the co-support). Since we want (k) of minimum `2 norm, it will be the
Tc
The algorithm is summarized as Algorithm 1. The stop least squares solution of (eq. 6), i.e. orthogonal to the null
criterion of the iterations can be, just like for OMP, a fixed space of this system. The null space of (eq. 6) consists of
number of steps (if sparsity level is known beforehand) or a the original null space of D plus the subspace spanned by
(k)
sufficiently small value of the objective kIT (k) k22 . D DT , as can be seen from the following two observations:
c
 (k)
1. This subspace is orthogonal to the system matrix Pc D: for some u, meaning that IT c (k) (k) lives in the space spanned
(k) (k) by D. Let N designate a basis for the null space of D (row-
Pc(k) DD DT = P c DT = 0. wise). Then, left multiplying with N yields 0:
(k)
2. The subspace spanned by D DT has dimension |T | = NT (k)
(k)
= 0.
(k)
k and thus precisely matches the null space dimension c Tc

of (eq. 6). Moreover, it is embedded in the row space of Considering that there were no wrong atom choices until the
the original dictionary D, since the columns of D and current iteration k, the original co-support Tc is completely
(k) (k)
the rows of D always span the same subspace. inside Tc . Therefore, we can split the set Tc into the
(k)
These considerations show that the subspace spanned by remaining support we still want to find, T? , and the whole
(k)
D DT is the additional null space brought by the decreasing true co-support Tc . Splitting the previous equation in this
(k) manner leads to:
range of Pc in addition to the original null space of D.
How does (k) differ from the previous estimate (k1) ? (k) (k)
NTc Tc = NT (k) (k) . (8)
They are both least-squares solutions of their respective ? T?
systems (eq. 6), and thus orthogonal to their null spaces The difference between the true sparse solution ? and
(k1) (k) (k) (k1)
D DT and D DT . But DT consists of DT together (k)
lives in the null space, since both solutions produce y.
with the last selected atom, henceforth denoted dk , and there- Therefore
fore we can obtain (k) by merely projecting and removing (k) ? = N T v,
from (k1) the part which lays in this additional null space
spanned by D dk : for some vector v. Restricting the above only to the rows in
the true co-support Tc , and considering that ? is zero for
(k) = (I D dk {D dk } ) (k1) . these atoms, leaves
Since dk is a vector, D dk is also a vector, and in this case (k)
Tc = NTTc v,
its pseudoinverse {D dk } is simply the transposed vector
(k)
divided by its squared norm. It follows that: therefore Tc lives in the span of NTTc . As such, it holds that:
1 T
(k) = (k1) T
D dk dTk D (k1) . (k)
Tc = NTc NTc Tc .
(k)
dTk D D dk
Replacing eq. 8 in the above leads to:
We can use the above equation for updating the estimate
(k) (k)
solution after every new atom is added. Tc = NTc NT (k) (k) .
? T?
Alternatively, we can consider the whole set DT (k) of atoms
selected until step k, and compute the solution at time k Thus, having
starting every time from the initial solution (0) = D y: kNTc NT (k) k, < 1
?

(k) = D y D DT (k) DT (k) T D T

D y, is a sufficient condition that
(k)
i.e. kTc k < kT (k) k ,
?

T thereby guaranteeing that atom selection via the largest coeffi-

(k) = D (In DT (k) DT (k) T D D )y.
IV. G UARANTEE FOR SUCCESSFUL RECOVERY
In this section, we derive a guarantee for successful signal
recovery with Least Squares Pursuit. The approach is inspired
by the similar guarantees of Greedy Analysis Pursuit and
Orthogonal Matching Pursuit.
We start from the unconstrained version of (eq. 4), i.e.:
k = arg min ky Dk22 + kIT c k2 .
where 0. The condition 0 ensures that the sparsity
term kIT c k2 is infinitely less important than the the data
fidelity term ky Dk22 , which means that the latter will be
essentially a strict constraint, thus leading to (eq. 4). Note that
leads to the OMP problem, conversely.
From the Lagrangian of this unconstrained formulation, we
get:
ITc (k) (k) = DT u,
cient in (k) never goes wrong and the chosen atom is always
(k)
from the remaining true support T? .
(k)
Since the set T? at step k can be in general as large as
the original support T (e.g. before the first atom selection), the
overall condition guaranteeing correct atom selection at every
iteration k is:
kNTc NT k, < 1.
V. S IMULATION RESULTS
(7)
To investigate the performance of Least Squares Pursuit
as a sparse recovery algorithm, we perform a recovery test
on synthetic sparse data, and represent the phase-transition
obtained.
Specifically, we generate a random dictionary of size n
N = 200 240, and generate exact-sparse data by randomly
combining atoms from the dictionary. The data is then acquired
with an acquisition matrix of varying number of rows m (no
noise is added), and then we perform reconstruction. A signal
is considered successfully recovered if the relative error of the
reconstruction is less than 106 .
We plot the percentage of successfully recovered signals as
a function of = m n (e.g. compression ratio of acquisition)
k
and = m (relative sparsity). The values of and go from
0.1 to 0.95 in steps of 0.05, and we average 100 recovered
signals for every pair (, ).
The graphical representation of Fig. 1 constitutes a phase-
transition plot [6] which indicates the area where Least
Squares Pursuit is capable of successful reconstruction. Here,
white indicates 100% recovered signals, black 0%, and levels
of gray indicate intermediate performance, correspondingly.
0.95
0.5
0.1
0.1 0.5 0.95
Fig. 1: Sparse recovery with LSP from noiseless measure-
ments: percent of sparse signals successfully reconstructed.
White indicates 100% recovered signals, black 0%.
The results show that Least Squares Pursuit is a viable
algorithm for exact-sparse signal recovery, with good recovery
capabilities in a wide area. The area of successful recovery
(white zone) is comparable to the one obtained with similar
algorithms like Orthogonal Matching Pursuit or Greedy Anal-
ysis Pursuit (the latter for analysis-based recovery), which are
however not included due to space reasons.
VI. C ONCLUSIONS AND FUTURE WORK
In this paper we proposed the Least Squares Pursuit algo-
rithm for sparse signal recovery, based on iteratively choosing
the largest atom from successive least-squares decomposition.
This approach is related to that of Orthogonal Matching
Pursuit, but with different prioritization of the data-fidelity
and sparsity constraints, and also with that of Greedy Analysis
Pursuit from the cosparse model. Although we only analyzed
the noiseless case, the results are promising and similar to
those of OMP.
Many avenues remain open for future work. Both OMP and
LSP can be considered as particular cases of an unconstrained
formulation (eq. 7) of the optimization problem, which opens
the way to having common theoretical analysis and guarantees
in the general unconstrained case. Also, the performance of
LSP with noisy measurements must be analyzed and compared
with that of OMP. Finally, the OMP recovery guarantees are
also valid for `1 -minimization based recovery [7], something
which remains to be investigated in the case of LSP.
ACKNOWLEDGEMENT
This work was supported by a grant of the Romanian
National Authority for Scientific Research and Innovation,
CNCS UEFISCDI, project number PN-II-RU-TE-2014-4-
0832 Medical signal processing methods based on com-
pressed sensing; applications and their implementation.
R EFERENCES
[1] E. J. Candes, J. K. Romberg, and T. Tao, Stable signal
recovery from incomplete and inaccurate measurements,
Communications on Pure and Applied Mathematics, vol.
59, pp. 12071223, 2006, ISSN: 0010-3640.
[2] D. L. Donoho, Compressed sensing, IEEE Transactions
on Information Theory, vol. 52, no. 4, pp. 12891306,
2006.
[3] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, Orthog-
onal matching pursuit: Recursive function approximation
with applications to wavelet decomposition, in Proc.
27th Annual Asilomar Conf. on Signals, Systems, and
Computers, 1993, pp. 4044.
[4] G. Davis, S. Mallat, and M. Avellaneda, Adaptive
greedy approximations, Constructive approximation,
pp. 5798, 1997.
[5] S. Nam, M. Davies, M. Elad, and R. Gribonval,
Cosparse analysis modeling - uniqueness and algo-
rithms, in Proc. ICASSP 2011, 2011, pp. 58045807.
DOI : 10.1109/ICASSP.2011.5947680.
[6] D. L. Donoho and J. Tanner, Precise undersampling
theorems, Proceedings of the IEEE, vol. 98, no. 6,
pp. 913924, 2010. DOI: 10.1109/JPROC.2010.2045630.
[7] J. Tropp, Greed is good: Algorithmic results for sparse
approximation, Information Theory, IEEE Transactions
on, vol. 50, no. 10, pp. 22312242, Oct. 2004, ISSN:
0018-9448. DOI: 10.1109/TIT.2004.834793.