Feature Article:
2933972
Deep Learning for Classification of Mini-UAVs Using
Micro-Doppler Spectrograms in Cognitive Radar
Albert Huizing, Matijs Heiligers, Bastiaan Dekker, Jacco de Wit, Radar
Technology Department, TNO, The Hague, The Netherlands
Lorenzo Cifola, Ronny Harmanny, Advanced Development Thales Nederland
B.V., Delft, The Netherlands
INTRODUCTION
Military radar can not only provide kinematic information
(position, speed, and course) of land, sea, and air targets
during day and night in all weather conditions, but also
provides information about the type of target using micro-
Doppler properties. The micro-Doppler properties of a tar-
get are determined by the moving parts on the target body.
The number, location, and type of motion of these parts
are specific for a given target type and consequently the
related micro-Doppler signature can be exploited for clas-
sification. Analysis of the micro-Doppler signature may
provide detailed properties of rotating parts, such as the
rotation rate, number of blades, and blade length. How-
ever, for operators and/or image analysts, the interpreta-
tion and understanding of radar micro-Doppler
spectrograms is much more difficult and time-consuming
than the analysis of optical images because of the different
nature of the radar imaging principle and target scattering
mechanisms. Consequently, there is a need for automatic
target recognition (ATR) in radar using micro-Doppler
spectrograms.
An ability to automatically recognize complex pat-
terns in radar signals is one of the key elements of the per-
ception–action cycle in cognitive radar as identified by
Haykin et al. [1]. A cognitive radar uses the feedback
from the receiver to adjust the transmitted waveforms to
Authors’ current addresses: Albert Huizing, Matijs
Heiligers, Bastiaan Dekker, Jacco de Wit, Radar
Technology Department, TNO, 2594 AK, The Hague,
The Netherlands, (E-mail: albert.huizing@tno.nl). Lor-
enzo Cifola, Ronny Harmanny, Advanced Develop-
ment, Thales Nederland B.V. 2628 XH, Delft, The
Netherlands.
Manuscript received March 27, 2019, revised July 12,
2019, and ready for publication August 6, 2019.
Review handled by P. Willett.
0885-8985/19/$26.00 ß 2019 IEEE
46 IEEE A&E SYSTEMS MAGAZINE
DOI. No. 10.1109/MAES.2019.
operate effectively and robustly in a dynamic environ-
ment. In addition to the feedback from the receiver, a cog-
nitive radar can also use prior knowledge about the
environment and targets of interest to optimize its perfor-
mance [2]. This prior knowledge can be provided by
human experts in the form of heuristics, terrain databases,
or computer models of the interaction between the radar
and the environment, including target signatures, such as
micro-Doppler spectrograms. Alternatively, prior knowl-
edge can also be derived from previous radar measure-
ments in the same or similar environments [3]. Examples
of this are a land clutter map and micro-Doppler spectro-
grams of targets of interest.
Figure 1 shows a high-level block diagram that incor-
porates two of the key elements of a cognitive radar: a per-
ception–action cycle and priori knowledge stored in long-
term memory, i.e., a knowledge base (KB). This diagram
has been adapted from the cognitive radar architecture
proposed by Guerci [2]. A knowledge-aided processor
(KAP) uses prior knowledge in the KB to control the radar
scheduler, the waveforms transmitted by the adaptive
transmitter, the radio frequency filter parameters and the
sampling rate in the adaptive receiver, and the parameters
and configuration of the radar signal processing. The sig-
nal processing typically includes functions, such as pulse
compression, Doppler filtering, detection, tracking, and
classification. The KAP may decide to update the KB
based on information that is extracted from the radar sig-
nal. For example, if the number of false alarms in an area
is too large due to a change in the clutter distribution, the
KAP may decide to update the clutter map.
In conventional radars, the recognition of targets, such
as mini-UAVs typically relies on the extraction of prede-
fined target features from the received radar signal. These
target features are defined and implemented by a human
expert in the radar signal processing, often after a long
and tedious analysis of large amounts of radar data.
Instead of this hand-crafted feature engineering approach,
a cognitive radar can use deep learning techniques to
NOVEMBER 2019
automatically extract the relevant features from a set of
radar measurements or radar simulations that are stored in
the KB. The extracted features can then be used by the
KAP to classify the target.
Deep learning has become the preferred method for
many pattern recognition applications, such as segmenta-
tion, speech recognition, and face recognition since the
quantum leap in image recognition performance realized
with the AlexNet convolutional neural network (CNN) in
the ImageNet 2012 challenge [4]. Despite the major
achievements of deep learning techniques, such as CNNs
in the past few years, it is not obvious that the success of
deep learning in the commercial domain can be replicated
in the military domain. Large labeled datasets, which are
the key to the success of most commercial applications of
deep learning, are often not available in military applica-
tions. In addition, the cost of decision errors in the military
domain is typically much higher than in the commercial
applications. This makes it mandatory to achieve a robust
performance with a low error rate. Furthermore, military
ATR algorithms should not only be accurate, but their
behavior should also be predictable to gain the trust of
military commanders.
One of the first applications of deep learning for ATR
using synthetic aperture radar imagery was presented by
Morgan [5]. Since then many papers on the application in
radar of CNNs and other types of deep neural networks
have been published, including the application of deep
learning for target classification, and human gait and ges-
ture recognition using micro-Doppler spectrograms [6]–[9].
This paper investigates the potential of deep learning
techniques for the classification of mini-UAVs using micro-
Doppler spectrograms in the context of cognitive radar. The
paper also presents preliminary results on the use of deep
learning in the preprocessing and training process of a classi-
fier. This paper is organized as follows. “CLASSIFICATION
OF MINI-UAVS” shows how deep neural networks can be
applied for the classification of mini-UAVs using measured
or simulated sets of micro-Doppler spectrograms that are
stored in the long-term memory (or knowledge base) of a
cognitive radar. “DETECTION OF UNKNOWN TARGET
NOVEMBER 2019 IEEE A&E SYSTEMS MAGAZINE
CLASSES” describes how the spectrograms of target classes
that are not represented in the training set, i.e., targets
unknown by the cognitive radar, can be detected with
deep neural networks. The detection of an unknown target
can provide a trigger for the radar scheduler to collect
micro-Doppler spectrograms of this target for inclusion in
the training set, i.e., the knowledge base. “DENOISING OF
SPECTROGRAMS” shows how a trained neural network
can be used to denoise spectrograms. This denoising process
may enable a cognitive radar to recognize targets at a longer
range. “ADVERSARIAL TRAINING FOR SPECTRO-
GRAM GENERATION” describes the generation of new
spectrograms for training a classifier using generative adver-
sarial networks. Finally, in “CONCLUSION,” conclusions
are drawn with respect to the application of deep learning for
classification of mini-UAVs with cognitive radar.
CLASSIFICATION OF MINI-UAVS
The commercial availability of compact electronics and
advanced open source software has led to a proliferation
of mini-UAVs that can be used for many different pur-
poses including criminal, terrorist, and military activities.
Owing to their relatively slow speed and small size, they
are hard to distinguish from natural targets, such as birds.
Recently, radar signal processing techniques using fea-
tures extracted from spectrograms and cepstrograms have
been developed to discriminate birds and mini-UAVs
[10], [11]. The recognition of different classes of mini-
UAVs is, however, a more difficult problem due to the
overlapping features and rapidly changing characteristics
of mini-UAVs. Therefore, an enhanced capability for
radars is needed to recognize the type of mini-UAV for
threat evaluation and assignment of countermeasures.
MINI-UAV MEASUREMENTS
To investigate the potential of deep learning techniques
for the recognition of mini-UAVs, radar measurements of
five different types of mini-UAVs have been acquired
47
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar

Figure 1.
High-level block diagram of a cognitive radar (adapted from Guerci [2]).

during a measurement campaign at the NLR RPAS Test
Centre in Marknesse, The Netherlands. The mini-UAVs
included two types of fixed wing aircraft (Robbe Air
Trainer 140 and Sky Walker X8), an Align T-REX 550
helicopter, a Mikrokopter QuadroXL quadcopter, and a
Mikrokopter OktoXL octocopter, see Figure 2.
Measurements of the mini-UAVs were conducted with
an experimental continuous wave radar that operates at a
frequency of 9.5 GHz and samples the radar signal at a
rate of 96 kHz. The complex radar signals are converted
to micro-Doppler spectrograms using a short time fourier
transform (STFT). The Doppler frequency of the mini-
UAV body is removed to center the Doppler spectrum
around 0 Hz. Examples of the measured micro-Doppler
spectrograms for all 5 mini-UAVs can be seen in Figure 2.
The coherent integration time of the spectrograms is
10.7 ms. The modulations of the radar signal due to the
moving parts can clearly be observed.
MICRO-DOPPLER SIGNAL SIMULATIONS
In addition to the measurements, micro-Doppler signals
have been simulated using computer models of mini-

Figure 2.
Photos (1st row), measured spectrograms (2nd row), computer models (3rd row), and simulated spectrograms (4th row) of five mini-UAVs.

48 IEEE A&E SYSTEMS MAGAZINE NOVEMBER 2019

 Huizing et al.

Figure 3.
Workflow for the classification of mini-UAVs with deep neural networks using micro-Doppler spectrograms.

UAVs to test the idea that a deep neural network can be CLASSIFICATION OF MINI-UAVS WITH DEEP NEURAL
trained with simulated micro-Doppler spectra. The trained
neural network can then be used to classify micro-Doppler NETWORKS
spectra measured with a real radar system during opera-
Figure 3 shows the workflow for the classification of mini-
tions. This concept would reduce the need for expensive UAVs with deep neural networks. The training phase con-
and time-consuming radar measurements to populate the sists of generating a training set with (a) simulated radar
knowledge base of a cognitive radar. It also could enable signals using target models or (b) real target signals mea-
the classification of mini-UAVs of which physical charac- sured with a radar. Micro-Doppler spectrograms are cre-
teristics are available, but no measurements can be made ated by applying an STFT successively to overlapping
due to security reasons. sequences of the time-domain signals. The preprocessing
The simulation of micro-Doppler spectra of targets, includes removal of the speed of the mini-UAV body, and
such as mini-UAVs is difficult because of the complex
normalization of the power spectrogram. In the test phase,
mechanical structure of these targets and the time varying the trained neural network is used to classify the measured
characteristic of the radar cross section (RCS) due to the radar signals.
moving parts. To limit the computation time for micro- Table 1 shows the number of micro-Doppler spectro-
Doppler signal simulations, an approach was chosen in grams that have been extracted from the measured
which the mini-UAVs are represented with so-called and simulated radar signals for the training and test sets
principal scatterers for which the analytic expressions of in the mini-UAV classification experiments described
their RCS exist [12]. Examples of these principal scatter- below.
ers are point scatterers, cylinders, ellipsoids, flat triangu-
lar plates, and thin wires. For each sample of the radar
signal, the position and attitude of each principal scat- Table 1.
terer is determined and the associated RCS for this scat-
terer is computed. Finally, the reflected radar signal for Micro-Doppler Spectrograms for the Five Mini-UAVs
the entire target is calculated by adding the contributions
of all principal scatterers. This simulation approach is rel- Training Test
atively fast, but neglects the occlusion of some scatterers Target
by other scatterers on the body, multiple reflections, dif- Simulated Measured Simulated Measured
fraction, and materials other than metal. Figure 2 shows
the computer models of the five mini-UAVs considered Air trainer 506 570 525 499

in this paper and examples of the associated simulated Skywalker 475 886 545 775
micro-Doppler spectrograms. Figure 2 clearly shows a
T-REX 550 965 1284 1080 1092
resemblance between the measured and the simulated
micro-Doppler spectrograms, but also some measured QuadroXL 477 841 546 869
characteristics that are not present in the simulated OktoXL 480 1370 555 1222
spectrograms.

NOVEMBER 2019 IEEE A&E SYSTEMS MAGAZINE 49

Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar

Table 2.

Deep Neural Network Test Cases for Classification of Mini-UAVs Using Micro-Doppler Spectrograms

Case Training Test data Neural Sampling Integration Accuracy

data network rate time

A Simulated Simulated CNN 96 kHz 10.7 ms 91.7%

B Simulated Measured CNN 96 kHz 10.7 ms 36.7%
C Measured Measured CNN 96 kHz 10.7 ms 82.8%
D Measured Measured CNN 24 kHz 10.7 ms 87.1%
E Measured Measured CNN 24 kHz 2.7 ms 97.7%
F Measured Measured LSTM-RNN 24 kHz 2.7 ms 89.0%

shadowing, multiple reflections, diffraction, and material

CLASSIFICATION EXPERIMENTS AND RESULTS
properties. More advanced RCS modeling techniques
To investigate the accuracy of different types of deep neu- using a hybrid finite element method and method of
ral networks for the classification of mini-UAVs, several moments approach may improve the fidelity of the micro-
test cases were defined, see Table 2. Test case A involves Doppler spectrograms at the expense of computational
the use of a CNN with a high sampling rate (96 kHz) of cost [13].
the radar signal and a medium coherent integration time A second conclusion is that the decimation of the sam-
(10.7 ms) in the Doppler filtering and simulated training pling rate by a factor of 4 improves the mean classification
and test data. Test case B gives an idea if the simulated accuracy for the measured spectrograms considerably.
spectrograms can be used to classify measured spectro- This is due to the fact that the part of the spectrogram that
grams from real mini-UAVs. does not contain any relevant information is removed by
Test case C uses measured spectrograms for training
and testing. Test case D also uses measured spectrograms
for training and testing, but employs a reduced sampling Table 3.
rate by decimating the radar signal with a factor 4. Test
case E uses a low sampling rate and a short integration CNN Configuration
time (2.7 ms). Finally, test case F uses a long short-term
memory (LSTM) recurrent neural network (RNN) instead Layer Type Size Kernel Filters
of a CNN as a classifier [14].
Table 3 shows the configuration of the CNN used in the 1 Input 1024
177 – –
first three test cases (A, B, and C). All convolutional and 2 Convolution 1024
177 5
5 32
fully connected layers (except for layer 13 which is linear)
3 Convolution 1024
177 5
5 32
use a rectified linear unit as a nonlinear activation function.
The convolutional and max-pool layers of the CNNs in test 4 Max-pool 512
88 2
2 32
cases D and E have smaller sizes than the configuration in 5 Convolution 512
88 5
5 64
Table 3 due to the reduced sampling rate and integration
6 Convolution 512
88 5
5 64
time. The CNN is trained with a categorical cross-entropy
loss function and the ADAM optimizer [5]. 7 Max-pool 256
44 2
2 64
The LSTM RNN in test case F does not use a spectro- 8 Convolution 256
44 5
5 64
gram as an input, but operates on an input sequence of
9 Convolution 256
44 5
5 64
Doppler spectra. The LSTM-RNN configuration has four
layers and a hidden state size of 256 for the LSTM cells. 10 Max-pool 128
22 2
2 64
A comparison of the mean classification accuracy in 11 Flatten 1
180224 – –
Table 2 shows that good results are achieved except for
the case where simulated spectrograms are used for train- 12 Fully con. 1
512 – –
ing a CNN and measured spectrograms are used during 13 Fully con. 1
5 – –
testing. This is perhaps not surprising due to the lack of
14 Soft-max 1
5 – –
fidelity in the target models, which do not include

50 IEEE A&E SYSTEMS MAGAZINE NOVEMBER 2019


Figure 4.
Adversarial training of an auto-encoder with the GANomaly method to detect spectrograms of unknown targets.
the decimation. This simple and effective preprocessing
step is apparently difficult to learn by the CNN from the
limited number of training samples in test case C.
Another conclusion is that the reduction in coherent
integration time in the Doppler filtering from 10.7 ms to
2.7 ms also gives a significant improvement in classifica-
tion accuracy. This improvement can be explained by the
fact that the modulations by the rotating propeller or rotor
blades are less smeared in the Doppler spectrum due to
the shorter integration time, and the changes in the Dopp-
ler spectrum become clearer.
Although the use of an LSTM-RNN proved to be less
accurate than the CNN in the overall classification, an
important advantage of an LSTM-RNN is that it can pro-
vide good classification results already after a few coher-
ent processing intervals, whereas the CNN only provides a
classification result after an entire spectrogram has been
processed [14]. Another advantage of the LSTM-RNN is
that it is capable to deal with transitions in target behavior,
e.g., a fixed wing mini-UAV that takes off vertically, and
then, proceeds to horizontal flight.
DETECTION OF UNKNOWN TARGET CLASSES
During military operations, cognitive radars will often be
employed in conditions where novel target classes are
observed of which there are no examples of micro-Dopp-
ler spectrograms in the training set of the deep neural net-
work. In this case, the cognitive radar should detect the
presence of this unknown target class and may decide to
schedule specific radar measurements to add spectrograms
of the unknown target class to the training set.
The detection of unknown target classes can be
achieved in several ways. The simplest approach is the
NOVEMBER 2019 IEEE A&E SYSTEMS MAGAZINE
Huizing et al.
application of a threshold to the output of the Soft-max
layer of a CNN [15]. If the maximum output of the Soft-
max layer does not exceed the threshold, an unknown tar-
get class is declared. This Soft-max approach is also
referred to as a reject option for a classifier [16]. By vary-
ing the value of the threshold, a receiver operation charac-
teristic (ROC) curve can be obtained, which shows the
probability of detecting an unknown target class (true pos-
itive rate) as a function of the probability of falsely declar-
ing an unknown target class (false positive rate).
An alternative to the Soft-max approach called GANo-
maly is based on an auto-encoder that is trained in an
adversarial manner [17]. This technique has been used to
investigate the potential of deep learning to screen avia-
tion luggage for anomalous items using X-ray screening.
Figure 4 shows the configuration of the GANomaly net-
work that consists of three subnetworks. The auto-encoder
acts as a generator network that learns a compact latent
space representation z of the input spectrograms x by try-
ing to reconstruct the input spectrogram. The encoder
compresses the reconstructed spectrogram to an estimate
of the latent representation using the same architecture as
the encoder in the auto-encoder. The discriminator deter-
mines, with an encoder architecture, if the input is a real
spectrogram or a fake spectrogram.
The detection of a spectrogram originating from an
unknown target assumes that the auto-encoder is not able
to reconstruct such a spectrogram accurately because the
network is trained only with spectrograms from known
target classes. A reconstructed spectrogram for an
unknown spectrogram will also lead to discrepancies
between the latent vector and its estimate.
This discrepancy is used by the GANomaly method
during the test phase to detect a spectrogram of an
unknown target class.
51
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
Figure 5.
ROC curve showing the true positive rate versus the false positive
rate for detecting an unknown Align T-REX 550 mini-UAV with
the Soft-Max and GANomaly methods.
The objective function for training the GANomaly net-
works consists of three loss functions. The adversarial loss
function Ladv computes the L2 distance between features
computed by the function f from an intermediate layer of
the discriminator network for the real and generated spec-
trograms. This feature matching approach reduces the
instability of GANs during training [18]. The context loss
function Lcon penalizes the auto-encoder for reconstruction
errors by measuring the L distance between the real and
generated spectrograms. Finally, the encoder loss Lenc
measures the L1 distance between the latent representations
of the input and reconstructed spectrograms.
Experiments with the Soft-max and GANomaly methods
for the detection of unknown target classes have been con-
ducted using the dataset with mini-UAV spectrograms, as
described in “CLASSIFICATION OF MINI-UAVS.” Each
of the mini-UAVs has been used alternatively as an unknown
target class when training the neural networks on the micro-
Doppler spectrograms from the remaining ‘“known” mini-
UAVs. Figure 5 shows an example of an ROC curve for a
test where the T-REX 550 helicopter was used as an
unknown target class. The GANomaly method performs bet-
ter at low and high false positive rates, whereas the Soft-max
method works better at intermediate false positive rates.
Figure 6 shows the area-under-curve (AUC) for the
detection of unknown target classes with the Soft-max and
GANomaly methods. Each experiment is performed three
times to capture the performance variations due to the random
initialization of the neural networks and due to the random
selection of the data samples in the test set and training set.
The above results are obtained using the measured
spectrograms of test case E. All spectrograms are truncated
after the first 64 time samples resulting in square input
data of 64
64 pixels. This input size enabled the use of
network configuration and training parameters described
in the paper by Akcay et al. [17] that are proven to be
successful for the GANomaly method. For Soft-max, the
52 IEEE A&E SYSTEMS MAGAZINE
Figure 6.
AUC results for the detection of unknown target classes with Soft-
max and GANomaly. The error bars represent variations due to the
use of random seeds for the initialization of the neural networks.
network and hyperparameters are as described in
“CLASSIFICATION OF MINI-UAVS.” The models for
GANomaly and Soft-max are trained for, respectively, 15
and 50 epochs.
Figure 6 shows that both the Soft-max and the GANo-
maly achieve AUC ranging from 0.5 to 0.8. Note that
comparing the performance of deep learning methods is
often tricky because the result can highly depend on the
effort that has been put in optimizing the networks, train-
ing parameters, and data preprocessing steps. Especially
adversarial training methods, such as GANomaly, are
notorious for instability issues. In the experiments, it was
observed that the final performance of the GANomaly
method was not yet fully satisfactory, and perhaps better
performances can be achieved with other GAN configura-
tions for the detection of unknown target classes.
DENOISING OF SPECTROGRAMS
The micro-Doppler spectrograms collected during the
measurement campaign described in “CLASSIFICATION
OF MINI-UAVS” have a relatively high signal-to-noise
ratio (SNR) due to the short range between the radar and
the mini-UAVs. However, the SNR will be lower at longer
ranges, and therefore, the performance of a target classifier
will degrade. To investigate if deep learning can be used
to improve the SNR of the spectrograms in the preprocess-
ing before classification, and hence, increase the range at
which a cognitive radar can recognize a mini-UAV, an
experiment was conducted with a denoising adversarial
auto-encoder. Adversarial trained auto-encoders can be
used to restore input data, such as images that are cor-
rupted by noise [19]. The architecture of the denoising
adversarial auto-encoder is like the architecture of the
NOVEMBER 2019

Figure 7.
Denoising adversarial auto-encoder applied to spectrograms corrupted with synthetic noise.
GANomaly network, as shown in Figure 4, except that the
separate encoder is omitted and the objective function for
training of the auto-encoder only consists of the mean
square error between noise corrupted versions of the spec-
trograms and the uncorrupted spectrograms.
Figure 7 shows two examples of measured micro-
Doppler spectrograms of a T-REX 550 helicopter with a
high SNR in the first column. The second column shows
the same spectrograms to which synthetic Gaussian noise
has been added to lower the SNR. The third column shows
the reconstructed spectrograms using a denoising adver-
sarial auto-encoder. Although some details of the micro-
Doppler modulations are distorted, it is still possible to
recognize the micro-Doppler contribution due to the tail
rotor and, up to a certain extent, the blade tip contribution.
In a second experiment, the denoising adversarial
auto-encoder has been tested with measured spectrograms
that are characterized by a lower SNR. As apparent from
the reconstructed spectrograms shown in Figure 8, a signal
to background ratio enhancement in the order of 20 dB is
achieved by the denoising auto-encoder. Furthermore,
information associated to tail rotor signature and blade tip
rotation is partially restored. Although the SNR of the
spectrograms seems to be significantly enhanced by the
denoising adversarial auto-encoder, the impact of this pre-
processing on the accuracy of a target classifier, such as a
CNN still has to be investigated.
ADVERSARIAL TRAINING FOR SPECTROGRAM
GENERATION
Training sets for target classifiers based on machine learn-
ing are often characterized by an imbalance in the number
of examples for different target classes. For example, the
NOVEMBER 2019 IEEE A&E SYSTEMS MAGAZINE
Huizing et al.
imbalance in the training set used in this paper
(see Table 1) is approximately a factor of two. This imbal-
ance can have a negative impact on the convergence of
the classifier during training and the overall accuracy [20].
There are several ways to mitigate the effect of an imbal-
anced training set. The first category concerns methods
that leave the training set intact and modify the training
procedure or the classifier to deal with the imbalance. The
second category involves changes to the training set itself
by generating data for classes that are underrepresented.
In this section, a generative network called InfoGAN has
been investigated for the generation of realistic training
data for underrepresented target classes [21]. When com-
pared with a standard GAN, an InfoGAN has the advan-
tage that it learns in an unsupervised way, interpretable
and disentangled representations of challenging datasets.
Figure 9 shows the architecture of an InfoGAN network
Figure 8.
Denoising adversarial auto-encoder applied to measured low SNR
spectrograms.
53
Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
Figure 9.
InfoGAN architecture.
that consists of a generator, a discriminator, and an auxil-
iary unit.
The generator input vector consists of the concatena-
tion of two parts: a noise vector (z) and a latent code (c).
The generator tries to fool the discriminator by generating
a realistic spectrogram, whereas the discriminator tries to
distinguish the synthetic and real spectrograms. In addi-
tion to the probability that an input is a real or a fake spec-
trogram, the discriminator also computes a distribution
Q(c j G(c,z)) that measures the mutual information bet-
ween the latent code c and the generated spectrogram.
Figure 10.
InfoGAN applied to micro-Doppler spectrograms with examples of synthesized spectrograms per category of the latent code.
54 IEEE A&E SYSTEMS MAGAZINE
Based on an information-based regularization term
within the overall training cost function, a structure is
introduced in the latent variable that enables the aggrega-
tion of spectrograms characterized by similar characteris-
tics. This is apparent in the generated spectrograms shown
in Figure 10 for nine different categories associated with
nine different states for the latent code (c). In addition,
since clustering is based on spectrogram appearance,
instances of the same class characterized by differing
spectrograms will be associated to different categories.
For example, both examples from category 2 and 3 belong
to the T-REX 550 helicopter class, although characteri-
zed by a different Doppler bandwidth. When the specified
categories are higher than the level of diversity among the
dataset examples, then overlap between categories can be
expected, for instance as observed for categories 5 and 9.
CONCLUSION
The results of several experiments in this paper have dem-
onstrated the potential of deep learning techniques for the
classification of mini-UAVs with a cognitive radar. Deep
neural networks, such as CNNs and RNNs can provide an
accurate classification performance after being trained on
a set of measured micro-Doppler spectrograms that are
stored in the long-term memory of a cognitive radar.
The analysis of the classification performance of dif-
ferent types of deep neural networks and different prepro-
cessing parameters showed that the choice of the coherent
NOVEMBER 2019
 Huizing et al.

integration time in the Doppler filtering had a higher Dutch Radar Center of Expertise (D-RACE), a strategic
impact on the classification performance than the type of alliance between Thales Nederland B.V. and TNO.
the deep neural network (CNN versus LSTM-RNN) in the
classifier. The use of simulated spectrograms to classify
measured spectrograms requires a higher fidelity modeling REFERENCES
of the micro-Doppler signals than the modeling approach
[1] S. Haykin, Y. Xue, and P. Setoodeh, “Cognitive radar: Step
based on principal scatterers used in this paper.
toward bridging the gap between neuroscience and engi-
When a cognitive radar measures a micro-Doppler
neering,” Proc. IEEE, vol. 100, no. 11, pp. 3102–3130,
spectrogram from a target class that is not represented in
Nov. 2012.
the training set, an anomaly detection algorithm could pro-
[2] J. R. Guerci, Cognitive Radar: The Knowledge-Aided
vide a trigger to the cognitive radar scheduler to collect
Fully Adaptive Approach, Norwood, MA, USA: Artech
micro-Doppler spectrograms from this unknown target.
House, 2010.
The performance of two deep learning techniques, Soft-
[3] M. S. Greco, F. Gini, P. Stinco, and K. Bell, “Cognitive
max and GANomaly, for the detection of spectrograms
radars: On the road to reality: Progress thus far and possibil-
from unknown target classes has been compared. Both
ities for the future,” IEEE Signal Process. Mag., vol. 35,
techniques had a similar performance with respect to the
no. 4, pp. 112–125, Jul. 2018.
probability of detecting unknown classes with the GANo-
[4] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,”
maly technique showing a larger performance variation
Nature, vol. 521, no. 7553, pp. 436–444, 2015.
due to the difficulty in training adversarial networks. Dif-
[5] D. A. E. Morgan, “Deep convolutional neural networks for
ferent GAN architectures for anomaly detection that are
ATR from SAR imagery,” Proc. SPIE, vol. 9475, 2015,
easier to train, such as a Wasserstein GAN may provide a
Art. no. 94750F.
better performance [22].
[6] Y. Kim and T. Moon, “Human detection and activity clas-
A denoising adversarial auto-encoder has been trained
sification based on micro-Doppler signatures using deep
to remove noise from spectrograms. An improvement of
convolutional neural networks,” IEEE Geosci. Remote
20 dB in the SNR of a measured spectrogram has been
Sens. Lett., vol. 13, no. 1, pp. 8–12, Jan. 2016.
observed. If this denoising technique does not alter the €
[7] M. S. Seyfio
glu, S. Z. G€urb€uz, A. M. Ozbayo
glu, and
micro-Doppler characteristics significantly, a considerable
M. Y€uksel, “Deep learning of micro-Doppler features for
increase in the range at which a cognitive radar can recog-
aided and unaided gait recognition,” in Proc. IEEE Radar
nize mini-UAVs could be achieved.
Conf., 2017, pp. 1125–1130.
To mitigate the effects of class imbalances in the num-
[8] R. P. Trommel, R. I. A. Harmanny, L. Cifola, and
ber of training examples on the performance of a classi-
J. N. Driessen, “Multi-target human gait classification
fier, an InfoGAN can be used to generate additional
using deep convolutional neural networks on micro-
spectrograms of underrepresented classes. A quantitative
Doppler spectrograms,” in Proc. Eur. Radar Conf., 2016,
study concerning the benefits of this approach will be con-
pp. 81–84.
ducted in the future research.
[9] B. Dekker, S. Jacobs, A. S. Kossen, M. C. Kruithof,
The overall conclusion of this paper is that deep learn-
A. G. Huizing, and M. Geurts, “Gesture recognition with a
ing techniques have a high potential for improving the
low power FMCW radar and a deep convolutional neural
classification of mini-UAVs using micro-Doppler spectro-
network,” in Proc. Eur. Radar Conf., 2017, pp. 163–166.
grams in cognitive radar. However, the selection and tun-
[10] P. Molchanov, K. Egiazarian, J. Astola, R. I. A. Harmanny,
ing of the appropriate neural network architectures for
and J. J. M. de Wit, “Classification of small UAVs and birds
different processing tasks, such as classification and
by micro-Doppler signatures,” in Proc. Eur. Radar Conf.,
denoising is tedious and time consuming. A better under-
2013, pp. 172–175.
standing of the theory behind deep neural networks, poten-
[11] R. Harmanny, J. J. M. de Wit, and G. Premel-Cabic,
tially enabled by the link with compressive sensing and
“Radar micro-Doppler mini-UAV classification using
sparse signal representations, would not only accelerate
spectrograms and cepstrograms,” Int. J. Microw. Wireless
the development process, but also improve the acceptance
Technol., vol. 7, no. Special Issue 3/4, pp. 469–477, 2015.
of these advanced techniques in military radar
[12] V. C. Chen, F. Li, S.-S. Ho, and H. Wechsler, “Micro-
applications.
Doppler effect in radar: Phenomenon, model, and simula-
tion study,” IEEE Trans. Aerosp. Electron. Syst., vol. 42,
no. 1, pp. 2–21, Jan. 2006.
ACKNOWLEDGMENT
[13] P. J. Speirs, A. Schr€oder, M. Renker, P. Wellig, and
TNO has received funds for this study from the Netherlands A. Murk, “Comparisons between simulated and measured
Ministry of Defense, under Grant V1512 and Grant V1908. X-band signatures of quad-, hexa- and octocopters,” in
The study has been conducted within the scope of the Proc. 15th Eur. Radar Conf., 2018, pp. 325–328.

NOVEMBER 2019 IEEE A&E SYSTEMS MAGAZINE 55

Deep Learning for Classification of Mini-UAVs Using Micro-Doppler Spectrograms in Cognitive Radar
[14] G. Klarenbeek, R. I. A. Harmanny, and L. Cifola, “Multi-
target human gait classification using LSTM recurrent neu-
ral networks applied to micro-Doppler,” in Proc. 14th Eur.
Radar Conf., 2017, pp. 167–170.
[15] A. Bendale and T.E. Boult, “Towards open set deep
networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Rec-
ognit., Jun. 2016, pp. 1563–1572.
[16] D. M. J. Tax and R. P. W. Duin, “Growing a multi-class
classifier with a reject option,” Pattern Recognit. Lett.,
vol. 29, no. 10, pp. 1565–1570, 2008.
[17] S. Akcay, A. Atapourabarghouei, and T. P. Breckon,
“GANomaly: Semi-supervised anomaly detection via
adversarial training,” in Proc. ACCV, Perth, Australia,
Dec. 2018.
[18] T. Salimans, I. Goodfellow, W. Zaremba, W. Cheung,
A. Radford, and X. Chen, “Improved techniques for train-
ing GANs,” in Proc. 30th Int. Conf. Neural Inf. Process.
Syst., 2016, pp. 2234:2242.
56 IEEE A&E SYSTEMS MAGAZINE
[19] A. Creswell and A. A. Bharath, “Denoising adversarial
autoencoders,” IEEE Trans. Neural Netw. Learn. Syst.,
vol. 30, no. 4, pp. 968–984, Apr. 2019.
[20] M. Buda, A. Makib, and M. A. Mazurowski, “A system-
atic study of the class imbalance problem in convolutional
neural networks,” Neural Netw., vol. 106, pp. 249–259,
2018.
[21] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever,
and P. Abbeel, “InfoGAN: Interpretable representation
learning by information maximizing generative adversarial
nets,” in Proc. 30th Int. Conf. Neural Inf. Process. Syst.,
2016, pp. 2172–2180.
[22] T. Schlegl, P. Seeböck, S.M Waldstein, G. Langs, and
U. Schmidt-Erfurth, “f-anogan: Fast unsupervised anomaly
detection with generative adversarial networks,” Medical
image analysis, vol. 54, pp. 30–44, May 2019.
NOVEMBER 2019