Journal of Neuroscience Methods 293 (2018) 359–374

Contents lists available at ScienceDirect

Journal of Neuroscience Methods

journal homepage: www.elsevier.com/locate/jneumeth

EEG-Annotate: Automated identification and labeling of events in

continuous signals with applications to EEG
Kyung-min Su a , W. David Hairston b , Kay Robbins a,∗
a
Department of Computer Science, University of Texas-San Antonio, San Antonio, TX 78249, USA
b
Human Research and Engineering Directorate, US Army Research Laboratory, Aberdeen Proving Ground, MD 21005, USA

h i g h l i g h t s g r a p h i c a l a b s t r a c t

• Proposes a framework for a new

“reverse inference” problem for EEG –
that tries to identify every place that
a particular response occurs.
• Shows that it is possible to perform
non-time-locked EEG classification
on unlabeled data using data from
other subjects.
• Provides a complete open-source
implementation of a working system
that includes visualization and anal-
ysis tools.
• Includes an open-source EEG data
collection and an associated Data
Note.

a r t i c l e i n f o a b s t r a c t

Article history: Background: In controlled laboratory EEG experiments, researchers carefully mark events and analyze
Received 8 August 2017 subject responses time-locked to these events. Unfortunately, such markers may not be available or may
Received in revised form 11 October 2017 come with poor timing resolution for experiments conducted in less-controlled naturalistic environ-
Accepted 13 October 2017
ments.
Available online 20 October 2017
New method: We present an integrated event-identification method for identifying particular responses
that occur in unlabeled continuously recorded EEG signals based on information from recordings of
Keywords:
other subjects potentially performing related tasks. We introduce the idea of timing slack and timing-
EEG
Single-trial ERP
tolerant performance measures to deal with jitter inherent in such non-time-locked systems. We have
Time-locked developed an implementation available as an open-source MATLAB toolbox (http://github.com/VisLab/
Event identification EEG-Annotate) and have made test data available in a separate data note.
Stimulus-response Results: We applied the method to identify visual presentation events (both target and non-target) in data
from an unlabeled subject using labeled data from other subjects with good sensitivity and specificity.

∗ Corresponding author.
E-mail addresses: kyungmin.su@utsa.edu (K.-m. Su), william.d.hairston4.civ@mail.mil (W.D. Hairston), kay.robbins@utsa.edu (K. Robbins).

https://doi.org/10.1016/j.jneumeth.2017.10.011
0165-0270/© 2017 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
360 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374

The method also identified actual visual presentation events in the data that were not previously marked
in the experiment.
Comparison with existing methods: Although the method uses traditional classifiers for initial stages, the
problem of identifying events based on the presence of stereotypical EEG responses is the converse of the
traditional stimulus-response paradigm and has not been addressed in its current form.
Conclusions: In addition to identifying potential events in unlabeled or incompletely labeled EEG, these
methods also allow researchers to investigate whether particular stereotypical neural responses are
present in other circumstances. Timing-tolerance has the added benefit of accommodating inter- and
intra- subject timing variations.
© 2017 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).

1. Introduction experimenter cannot be certain that a subject actually “saw”

A large body of traditional EEG neuroscience research uses
“stimulus-response” paradigms in which a stimulus is presented
to a subject and the subject’s neurological response, as measured
by EEG or other imaging, is recorded. Analysis begins by extract-
ing windows or epochs consisting of channels × time data-points
that are time-locked to particular known stimulus events. ERP
(event related potential) analysis averages the epochs associated
with each type of stimulus event and then examines differences
in the average patterns. In order for averaging to work, precise
signal time-locking to the events is critical and depends on hav-
ing an accurate representation of event onset times (Larson and
Carbine, 2017). More sophisticated time-locked methods based on
regression (Rivet et al., 2009; Souloumiac and Rivet, 2013; Congedo
et al., 2016) account not only for the temporal jitter due to vari-
ability of individual responses, but also for temporal overlap of
responses due to closely spaced stimuli. However, all of these
methods rely on precise knowledge of the time of stimulus with
a goal of modeling “stereotypical” responses to specified stim-
uli.
Single trial time-locked analyses (Jung et al., 2001) and BCI
(brain-computer interface) systems (Müller et al., 2008) also map
epochs time-locked to stimuli into feature vectors labeled by the
stimulus event type. These methods build classifiers (Blankertz
et al., 2011) or use other machine learning techniques (Lemm
et al., 2011) to map feature vectors into class labels for use in later
stimulus identification of time-locked unlabeled data. Single-trial
analysis is also used in regression and other types of statisti-
cal modeling (Kiebel and Friston, 2004a,b; Maris and Oostenveld,
2007; Rousselet et al., 2009). Frishkoff et al. (2007) have used
features extracted from ERPs with decision-tree classifiers to gen-
erate rules for categorizing responses to stimuli across studies.
They have developed NEMO (Neural ElectroMagnetic Ontologies)
a formal ontology for the ERP domain (Frishkoff et al., 2011)
and methods of matching ERPs across heterogeneous datasets
(Liu et al., 2012a). Time-locked classification, time-frequency, and
ERP analyses have many applications in laboratory and clinical
settings where scientists are trying to understand neurological
processes associated with particular stimuli (Roach and Mathalon,
2008; Dietrich and Kanso, 2010; Mumtaz et al., 2015; Lotte et al.,
2007).
As EEG experiments move to more naturalistic settings (Kellihan
et al., 2013; McDowell et al., 2013), neither time-locked analysis
nor expert identification of stereotypical patterns may be pos-
sible. In a typical laboratory event-related experiment involving
visual stimuli, the subject focuses on the screen, and researchers
assume that subjects “see” what is presented at the time of
presentation. For subjects moving in natural environments, a myr-
iad of stimuli compete for attention and the mere existence
of a visible line-of-sight does not guarantee perception. Even if
eye-tracking data indicates a fixation on the visual target, the
the stimulus and “paid attention”. Clearly, the analysis of neural
responses in EEG acquired under complex naturalistic experimen-
tal conditions requires alternative approaches for understanding
stimulus-response relationships.
This paper introduces a new class of methods for analysis of
EEG data, which we refer to as “automated event identification”.
Automated event identification matches responses observed in
continuous unlabeled EEG data to responses known to be asso-
ciated with particular stimuli in other datasets. While traditional
ERP analysis tries to identify the stereotypical responses result-
ing from a particular stimulus, event identification tries to find
stereotypical responses and hence locate when potential stimuli
might have occurred. We employ machine-learning approaches
to train multiple classifiers using data from experiments that fol-
low well-controlled time-locked stimulus-response paradigms and
then apply these classifiers to time windows extracted from con-
tinuous data in other settings to determine whether the same
types of neural responses occurred in these new settings. Ideally
such systems could allow researchers to tackle the inverse of the
“stimulus-response” problem and ask, “Which stimuli trigger a par-
ticular response?” rather than “What response is triggered by a
particular stimulus?”
In this paper, we apply our automated detection method to show
that not only does the expected neural response to visual stimu-
lus occur in predictable ways across subjects in a visually evoked
potential experiment, but also that similar responses can be evoked
by incidental visual stimuli during the course of an experiment.
Included in the supplemental material is a complete implementa-
tion of the method as a MATLAB toolbox (available at http://github.
com/VisLab/EEG-Annotate) along with the labeled data used in this
paper in order to allow researchers to label their own data using
these tools.
2. Materials and methods
2.1. Method overview
In traditional time-locked EEG analysis, we know the stimulus
times exactly and analyze responses by extracting fixed-length por-
tions of the signal, called windows or epochs, starting at known times
relative to the stimulus events as illustrated schematically by the
yellow window (x) of Fig. 1. We can transform response epochs
to feature vectors labeled by stimulus type and build classifiers to
identify stimulus based on response.
In the context of automated event identification, however, we
have no knowledge of where stimulus events are located or even
if a particular brain response was actually associated with a “stim-
ulus”. Our goal is to identify stereotypical neural responses and
then to determine whether these responses were associated with
identifiable stimuli.
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374 361

Fig. 1. Time-locked vs. non-time-locked decomposition of EEG signals into fixed sized windows or epochs. A time-locked epoch (x) used in traditional ERP or classification
analysis appears in yellow. Non-time-locked epochs (a through l) generated using fixed offsets (dotted lines) appear in blue and apricot. Apricot windows (d through k)
partially overlap the epoch time-locked to this stimulus event. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of
this article.)

Table 1 offset or sample size (125 ms) is smaller than the time scale over
Summary of the event annotation algorithm.
which the relevant event process occurs. Thus, the classifiers out-
Step Section Event annotation algorithm put hits for clusters of adjacent, overlapping epochs that actually
a) 2.2 Extract epochs from the training and test correspond to a single stimulus event. Another way to view this
datasets. redundancy is that the individual predictions align to within a spec-
b) 2.3 Convert epochs to feature vectors. ified timing error.
c) 2.4 Train multiple classifiers using training and A second difference is that in the automated identification of
possibly test features.
stimulus events, such as those that occur in naturalistic settings,
d) 2.5 Use each classifier to score each unlabeled
unlocked test feature. we are mainly interested in the most significant events and assume
e) 2.6 Use threshold adapted for imbalance to shift that events (positive labels) occur relatively infrequently. After
scores. making and combining the initial predictions, we compare scores
f) 2.7 Shift and rescale to bring window scores
across the dataset as a whole to determine which labels are the most
between 0 and 1.
g) 2.8 Apply weights of Table 2 to smooth window likely to correspond to events (true positives). Event identification
scores. in its second phase is similar to retrieval systems, which seek to
h) 2.9 Apply greedy zero out to obtain sub-window identify the most relevant matches first. Precision and recall, which
or sample scores. do not depend on true negatives (non-events), are the relevant
i) 2.10 Combine multiple classifiers scores by
measures.
averaging and smoothing.
j) 2.10 Apply greedy zero-out to obtain candidate The remainder of this section provides details about a par-
events. ticular implementation we provide with the annotation toolbox.
k) 2.11 Histogram resulting scores and apply adaptive Researchers can easily modify these choices to better suit their
threshold.
problems.
l) 2.11 Rerank if appropriate.

2.2. Extracting epochs or windows (Step a)

To approach this problem, we decompose the continuous chan-
nels × time EEG signal into windows (epochs), each offset by a fixed
time as illustrated by the blue and apricot windows in Fig. 1. We
refer to the portion of an epoch from its start to the start of the
next epoch as a sample. We choose the sample width (the window
offset) to evenly divide the window length. The choice of sample
offsets that integrally divide the length of windows (epochs) simpli-
fies overlap calculations. We transform these overlapping windows
into feature vectors and apply a federation of classifiers trained on
time-locked data from other labeled datasets. We then combine the
results, accounting for variations in timing, to identify the epochs
most likely to contain potential stimuli. Table 1 summarizes the
steps of the method.
On the surface, this approach seems similar to ordinary clas-
sification with multiple datasets for training and a new dataset
for testing, but there are crucial differences. Ordinary classification
treats the epochs as independent of each other. In contrast, adja-
cent non-time-locked epochs extracted from continuous data are
highly correlated, particularly for features that are relatively insen-
sitive to time locking. If we choose an epoch size of one second
with epoch offsets of 125 ms, 15 non-time-locked epochs overlap
to some degree with each time-locked epoch. Further, the epoch
As described in Fig. 1, we extract windows (epochs) starting at
the beginning of the dataset from the continuous channels × time
EEG signals of both training and test datasets. The windows are
one second in length, and each window is offset from the previous
window by an offset of 125 ms. The “positive” instances of a given
event class correspond to windows that contain an event of that
class in their first sub-window. The “negative” instances are those
windows that do not overlap at all with the positive instances.
2.3. Transforming epochs to feature vectors (Step b)
In order to perform classification, we must convert epochs in
the training and test datasets into feature vectors. We have selected
spectral power features, which capture spectral, temporal, and spa-
tial information without heavy dependence on exact timing. We
form a feature vector of length channels × 8 × 8 for an epoch by
computing for each channel the spectral power in each of eight
(8) frequency sub-bands on eight one-second time intervals offset
by multiples of 125 ms from the start of the epoch.
Specifically, after high-pass filtering the EEG signals at 1.0 Hz, we
band-pass filter using EEGLAB’s pop eegfiltnew function (Delorme
362 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
and Makeig, 2004a) to form eight frequency sub-bands of approxi-
mately 4 Hz in width in the range 1–32 Hz. These bands, which start
at 0.75, 4, 8, 12, 16, 20, 24, and 28 Hz, respectively, align to well-
known neurologically relevant frequency bands (Klimesch, 1999).
For example, brain-computer interface systems (BCIs) often use Mu
(8–12 Hz) signals detected over motor cortex areas (Arroyo et al.,
1993), while EEG drowsiness monitoring systems sometimes use
Theta (4–7 Hz) signals originating over the cerebral cortex (Lin et al.,
2008). Theta rhythms originating from the hippocampus are gen-
erally too deep for EEG surface recordings to detect (Buzsáki, 2002).
Alpha waves (also 8–12 Hz) found in the occipital lobe are associ-
ated with attention lapses (Palva and Palva, 2007), while Beta band
activity (16–32 Hz) is associated with motor and cognitive control
(Engel and Fries, 2010).
2.4. Training the classifiers (Step c)
Each of type of event generates a separate one-against-all clas-
sification problem. The question is whether an event of a particular
type occurs in an epoch, not which of a selected set of events
occurs in that epoch. We illustrate the approach using two classi-
fier algorithms – simple linear discriminant analysis (LDA) (Zhang
et al., 2013a) and a more sophisticated domain adaptation classi-
fier called ARRLS, which was originally introduced by Long et al.
(2014) and later modified to handle imbalanced data by Su et al.
(2016). LDA classifiers do not use information about the test set,
so an annotation system can train a single classifier for each event
type-training set combination and store the classifier along with
the training set data for a simple and fast implementation. Domain
adaptation classifiers use the distribution of the unlabeled test data
to improve performance, so we must train a different classifier for
each event type-training set-test set combination.
LDA classifiers find linear combinations of features that maxi-
mize the variance between classes while minimizing the variance
within classes for the training set. LDA assumes that the samples
within a class come from the same normal distribution and that
the distributions of different classes have different means but the
same covariance matrices. LDA also assumes that the test set has
the same class distributions as the training set. We use the MAT-
LAB fitcdiscr (fit discriminant analysis classifier) function with a
linear discriminant type and empirical prior probabilities to train the
LDA classifiers. After training an LDA classifier, we use the MATLAB
predict function in the Statistics Toolbox to estimate the posterior
probabilities to score the test epochs.
In contrast to LDA, domain adaptation adapts the training distri-
butions to the test distributions in order to achieve better accuracy
during learning transfer (Liu et al., 2012b; Wu et al., 2014). The
basic ARRLS algorithm proposed by Long adds several regulariza-
tion terms to standard regression-based classification. One term,
based on maximum mean discrepancy, tries to minimize the dif-
ference between the joint distributions of the training and test sets.
The joint distribution is a combination of the marginal distribu-
tion of the feature vectors and the conditional distributions of the
feature vectors for the different classes. ARRLS also uses manifold
regularization to assure that the local geometry of the marginal dis-
tributions is similar in both the training and test sets. ARRLS finds
an optimal solution by minimizing the least-squared classification
error with a standard ridge regression term for regularization. In
order to approximate the conditional distributions of the classes in
the test set, ARRLS uses logistic regression classifier trained with
the training data to calculate pseudo-labels for each element of
the test data. We always balance the training data for this initial
classification.
ARRLS assumes that training data and test data have the similar
class balances. The highly imbalanced nature of event annota-
tion (∼100:1) creates an intrinsic difficulty in estimation of the
conditional distributions, since the test set labels are completely
unknown and not likely to be the same as that of the training data.
We have developed a modification of ARRLS, called ARRLSimb to
handle highly imbalanced datasets (Su et al., 2016). To estimate
the structural risk and the joint distributions, ARRLSimb makes
an initial prediction of the class labels for the test set to gener-
ate pseudo-labels. ARRLSimb uses the pseudo-labels of the test set
along with the actual labels of the training set to weight the class
samples when generating the conditional distributions.
2.5. Classifying test epochs or windows (Step d)
We apply the one-against-all classifiers developed for each
event type-training dataset combination (LDA) or each event type-
training dataset-test dataset combination (ARRLSimb) to produce
initial prediction scores for the individual overlapping epochs in
each test dataset. LDA sets the class label using a threshold based
on posterior probabilities. The original ARRLS algorithm produces
two scores for each test data item, one corresponding to a positive
class label and the other corresponding to a negative class label.
ARRLS selects the positive class if the difference in scores (positive
– negative) is positive. Unfortunately, ARRLS performance is very
sensitive to class imbalance. ARRLSimb uses an adaptive thresh-
olding method described in the next subsection to compensate for
imbalance when determining the class label.
2.6. Adaptive thresholding to adjust labels for imbalance (Step e)
ARRLSimb fits the distribution of difference scores based on the
assumption that the test set is highly imbalanced, with far more
negative class instances. ARRLSimb assumes that the distribution
of difference scores is the sum of two Gaussian distributions, with
the Gaussian representing the negative class having far more points
than the Gaussian representing the positive class. Instead of trying
to fit directly two Gaussians, ARRLSimb fits a single Gaussian cor-
responding to the largest peak in the score histogram, removes an
approximation to this distribution from the histogram, and then
fits a second Gaussian to the remaining positive differences. We
use MATLAB scripts provided by Long et al. (2014) modified for
imbalance as described in (Su et al., 2016) for the ARRLSimb classi-
fications of this paper.
2.7. Scaling epoch or window scores (Step f)
The ranges of score differences for ARRLS differ widely across
classifiers, preventing straightforward combination of scores from
multiple classifiers. We have experimented with various types of
scaling such as z-scoring prior to combination. To avoid undue
influence by outliers, we subtract the adaptive threshold from the
score difference computed as described in the previous section and
set all negative scores to 0. We then divide by the 98th percentile
value of the non-zero scores and set all scores above 1 to 1. This
shift and scaling results in scores from the individual classifiers to
lie between 0 and 1.
2.8. Smoothing the window scores (Step g)
The algorithm combines scores from overlapping epochs (win-
dows) to produce a single score for each sample (sub-window) by
each classifier, accounting for the contribution of each epoch (win-
dow) to many samples (sub-windows) as illustrated schematically
in Fig. 2. Fig. 2A shows a plot of the raw window scores produced by
Step f plotted against the starting position of the window. A win-
dow consisting of N sub-windows (eight in our case) overlaps and
contributes to the score 2 × N − 1 sub-windows. If Xk is the window
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
Fig. 2. A schematic of the weighted averaging and zero-out procedure for producing
sub-window scores from overlapping window scores. A) The initial window scores
plotted against position of the first sub-window in a window. B) Sub-window scores
produced by calculating the weighted averages of overlapping window scores. C)
Events are identified in isolated sub-windows by applying greedy zero-out.
starting at sub-window k, we compute the score Yi of sub-window
i as:

N
 
Yi = Wj × Xi+j (1)
j=−N
Wj is the weight applied to the window offset by j sub-windows
from the center of sub-window i. Table 2 shows the particular
weight vector used in this work. (We omit division by the sum of
the weights for readability.)
Weighted averages account for the contributions of different
windows and smooth local variations in the scores as illustrated
schematically in Fig. 2B, which depicts the sub-window scores after
window weighting.
2.9. Scoring of samples or sub-windows (Step h)
A sub-window with a high score is likely to have neighbors with
a high score. Under the assumption that distinct events have some
temporal separation, high scoring sub-windows in a neighborhood
are likely designate the same event. To avoid false positives due
to double counting of immediate neighbors with a high score, we
apply a greedy masking procedure to remove the conflicts. The pro-
cedure is as follows: first find the largest sub-window score and
annotate the associated sub-window as a hit. Apply the zero-out
mask (Table 2) centered on the labeled sub-window and update
by multiplying the sub-window scores by the mask. This masking
procedure eliminates nearby high scores as shown schematically in
Fig. 2C. Repeat this procedure until there are no unmarked non-zero
scores.
363
In summary, the predictions of a target tend to cluster near
a target, but usually spread over adjacent sub-windows due to
timing variations and the contribution of each window to multi-
ple sub-windows. Applying a weighted window average followed
by thresholding gives better predictions than simply thresholding
each score. Furthermore, since sub-windows adjacent to an event
have high scores, we use a zero-out procedure to avoid double
counting.
2.10. Combining multiple classifiers to obtain candidate events
(Step i and Step j)
At this point, we have an array of sub-window scores for each
classifier. Due to the zero-out procedure, the array contains iso-
lated non-zero values. We average all of the sub-window score
arrays for each test set-event class combination. We then apply the
smoothing function of Table 2 and then the zero-out procedure. The
non-zero values are the scores for the candidate events.
2.11. Using adaptive thresholding to select the actual events (Step
k)
Finally, we use adaptive thresholding to fit the score distribu-
tion of the candidate events and separate high-scoring candidates
(likely events) from low-scoring candidates (likely noise). When
the histogram of this distribution separates well into low and high
scores, the annotation is likely to have been successful. However,
if there is no clear separation between high and low scores, one
should be conservative in accepting the annotation results. In some
cases, a re-ranking procedure, such as reclassifying the samples that
have non-zero scores in the annotation process, may improve the
separation and the effectiveness of the annotation.
2.12. Metrics and timing tolerance
We use balanced accuracy as an initial measure of classification
performance:
 TP TN

Accuracy = 0.5 × + (2)
TP + FN TN + FP
where TP, TN, FP, and FN are the true positives, true negatives, false
positives, and false negatives, respectively. However, in the con-
text of annotation, as in retrieval, we are primarily interested in
how well the algorithm annotates (retrieves) relevant items. In this
context, precision, recall, and related measures are more useful.
The annotation process determines how many items to annotate
(retrieve) based on quality scores. Precision is the ratio of the num-
ber of correct or relevant items annotated (retrieved) over the total
number annotated (retrieved). Recall is the ratio of the number of
correct or relevant items annotated (retrieved) over the total num-
ber relevant items. We also use the ranked average precision (RAP)
to measure annotation quality.
The ranked average precision (RAP) is the sum of P(ri ) for all
correctly retrieved items divided by the total number of correct
or relevant items. Here P(ri ) is the precision calculated by anno-
tating all of the items whose scores are at least that of the ith top
scoring correct item. RAP effectively assigns a zero precision to cor-
rect items that are not annotated (retrieved) and takes on a value
of 1 when all correct items and only correct items are annotated
(retrieved). RAP is similar to the average precision (AP) used in infor-
mation retrieval (Deselaers et al., 2007). However, since the term
“average precision” in classification usually refers to averaging val-
ues over different folds in cross-validation, we use the term RAP to
avoid confusion.
Timing presents a difficulty for evaluation because if a prediction
is off from the true location by even one sub-window, the metrics
364 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
count the item as a fail no matter how closely the prediction is
to a true event. To account for these timing variations, we modify
the algorithms for determining precision and recall to inspect the
neighboring samples of a prediction. The non-time-locked versions
of precision and recall count a sample as a positive if it is within a
specified number of sub-windows (the tolerance) of a true positive
event. Table 3 illustrates the process.
True events occur during sub-windows 3, 6 and 10. Depending
on the specified timing tolerance, the algorithm may or may not
count predictions for neighboring sub-windows as hits. We also
modify the RAP calculation to use timing precisions with a spec-
ified tolerance expressed as a number of sub-windows. Because
of the zero-out procedure, each prediction will not have another
prediction in its immediate neighborhood.
2.13. Calculations of statistical significance
We used an empirical bootstrapping technique to answer the
question of whether the performance of annotation in labeling m
samples from a subject’s n labeled samples was significantly better
than random. To generate a random base annotation, we assigned
a random score between 0 and 1 to each element of an n-element
vector. We selected the m annotated samples using greedy proce-
dure (remaining sample with the highest non-zero score selected
next). However, after each selection, we applied the zero-out pro-
cedure described in Section 2.9 to remove the neighbors of the
selected consideration from further consideration. We generated
10,000 random annotations and evaluated each performance met-
ric on the annotations with a specified timing tolerance to form a
base distribution. We applied the MATLAB ztest using the mean and
standard deviation of this distribution to determine the statistical
significance of the annotation for each subject. We also performed a
non-parametric test of significance using the empirical distribution
of the metrics based on random annotations.
2.14. Test data
To evaluate the proposed annotation system, we used prior
collected, anonymized data containing no personally identifiable
information from a standard Visual Evoked Potential (VEP) oddball
task acquired during a cross-headset comparison study compiled
by the U.S. Army Research Laboratory (Ries et al., 2014; Hairston
et al., 2014). The voluntary, fully informed consent of the persons
used in this research was obtained in written form. The document
used to obtain informed consent was approved by the U.S. Army
Research Laboratory’s Institutional Review Board (IRB) in accor-
dance with 32 CFR 219 and AR 70-25 and also in compliance with
the Declaration of Helsinki. The study was reviewed and approved
(approval #ARL 14-042) by the U.S. Army Research Laboratory’s IRB
before the study began. Table 4 summarizes the properties of this
dataset.
Eighteen subjects were presented with a sequence of two types
of images: a U.S soldier (Friend events) and an enemy combatant
(Foe events). The images were presented in random order at a fre-
quency of approximately 0.5+/−0.1 Hz. Subjects were instructed to
identify each image with a button press. The data used in this study
was acquired using a 64-channel Biosemi Active2 EEG headset.
The experiments recorded events for approximately 560 s. Longer
records were trimmed to include only the portion containing the
actual experiment with recorded events.
We applied the PREP preprocessing pipeline (Bigdely-Shamlo
et al., 2015) to remove line noise and interpolate bad channels. We
then removed the remaining noise and subject-generated artifacts
using the Multiple Artifact Rejection Algorithm (MARA) (Winkler
et al., 2011), which is an ICA-based EEG artifact removal tool. Both
PREP and MARA are completely automated pipelines.
Each dataset contains approximately 34 Foe events and 235
Friend events. Typical time-locked classification analyses extract
one-second epochs time-locked to these events and label epochs
as “friend” or “foe”. In these cases, a visual oddball-response to the
enemy combatants (foe) is expected.
In contrast to classification, annotation processes the two dif-
ferent types of events in two separate analyses, each based on
one-against-all classification. The “positive” samples are either the
Friend events or the Foe events. To form the “negative samples” we
select the samples remaining after eliminating the windows with
any overlap with the positive events. Approximately 1000 nega-
tive samples remain for the “friend” problem and 4000 negative
samples remain for the “foe” problem.
To predict labels for a test subject, we combine the scores from
the 17 other subjects and predict labels for the test subject based on
the combined score. Since the training and test datasets are distinct,
we use all of the samples of the training subjects for training and
all of samples of the test subject for testing.
Note that the classification problems described above (Friend
versus others and Foe versus others) are extremely imbalanced. In
each classification problem, we balance the training datasets by
oversampling the positive samples before training the classifiers.
3. Results
To test the EEG annotation system, we perform two separate
one-against-all classifications within the annotation process: the
first annotates the appearance of U.S. soldiers (Friend events) and
the second annotates enemy combatants (Foe events). Unlike the
typical classification analyses, we use non-time-locked samples for
both training and testing. A positive sample is a window whose first
sub-window contains the targeted event. The negative samples are
non-time-locked windows that do not overlap with the positive
samples. Therefore, the negative samples depend on the type of
positive samples. If positive samples are friends, then the negative
samples are non-time-locked samples that do not overlap with the
friend samples. Note that, in this case, the negative samples overlap
with the foe samples. However, in annotation, the question is not
whether the response between these two types of events is distinct,
but rather at what times such events were likely to have occurred.
3.1. Pairwise classification accuracy
As an initial point of reference, we compare the performance
of LDA (no domain adaptation), ARRLS (with domain adaptation)
and ARRLSimb (ARRLS modified for imbalanced data) for non-time-
locked classification using one subject as the training subject and
another as the test subject. No labeled data from the test subject
is included, but ARRLS and ARRLSimb accounts for the distribution
of the test data. Because there are 18 subjects, we tested 306 pairs,
consisting of 17 training subjects for each of 18 test subjects. Fig. 3
compares the balanced accuracies for the two inter-subject classi-
fication problems: Friend vs. Others and Foe vs. Others. In the plot,
each dot represents the accuracy of one training-test pair.
As indicated by Fig. 3, both ARRLS and ARRLSimb consistently
outperform LDA in the pairwise classification task. As expected, the
foe classification task is more difficult, in part because of the limited
training set available for foe classification. Based on 306 training-
test subject pairs, the average balanced classification accuracy for
LDA for Friend vs. Others was 0.60 (std 0.08) and for Foe vs. Others
was 0.52 (std 0.06). ARRLS had an average classification accuracy
of 0.76 (std 0.11) for Friend vs. Others and 0.66 (std 0.14) for Foe
vs. Others. ARRLSimb had an average classification accuracy of 0.75
(std 0.10) for Friend vs. Others and 0.78 (std 0.1) for Foe vs. Others.
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
Fig. 3. Comparison of LDA, ARRLS and ARRLSimb classification accuracies for pairwise classification with zero timing tolerance. The left plot shows Friend vs. Others and the
right plot shows Foe vs. Others.
A one-sided paired t-test shows ARRLS and ARRLSimb are sig-
nificantly better than LDA for Friend vs. Others (T(305) = 34.5,
p < 0.001) and (T(305) = 33.6, p < 0.001), respectively. For Foe vs. Oth-
ers, ARRLS and ARRLSimb are also significantly better (T(305) = 17.7,
p < 0.001) and (T(305) = 43.7, p < 0.001). ARRLS and ARRLSimb have
similar performance for Friend vs. Others, but ARRLSimb is signifi-
cantly better than ARRLS for Foe vs. Others (T(305) = 18.0, p < 0.001).
The 18 subjects fall into three distinct groups based on the aver-
age of the Friend and Foe balanced accuracies of ARRLSimb. Subjects
6, 12, 14, and 17 have average accuracy less than or equal to 0.66.
Subjects 1, 3, 7, 10, 13, 15, and 16 have average accuracies between
0.7 and 0.8, while subjects 2, 4, 5, 8, 9, 11, and 18 have average
accuracies of 0.80 or above.
The ARRLS classification results are similar to those reported in
Wu et al. (2015) on the same data collection using features consist-
ing of raw EEG on selected channels.
3.2. Raw window scores
The initial classification step calculates a score for each window.
Fig. 4 over plots the difference between the positive and nega-
tive raw class scores output from all ARRLSimb training-test set
pairs. Friend and foe classification problems are considered sep-
arately. The window scores are aligned so that the sub-windows
containing the actual event appear at position zero. The thick black
line corresponds to the average of the scores for the Friend and
Foe events, respectively. The overlay of the initial window scores
shows the expected elevation in score for the positive events in
each classification task, confirming the need for zero-out.
3.3. Predicted annotations and timing errors
Fig. 5 summarizes the success of prediction of the true events
(Friend and Foe separately) with respect to timing errors. In this
case, the tolerance refers to the distance of the sub-window con-
taining each actual event from the sub-window of the nearest
prediction of that event. ARRLS and ARRLSimb predict approxi-
mately 50% of all events with zero timing error, and 75% of the
365
events with a timing error of one sub-window or less (in this case,
125 ms). In either case, 85% of the events have a timing error of
four sub-windows or less. ARRLS and ARRLSimb perform similarly
for both friend and foe events. LDA, on the other hand, performs
much more poorly, predicting 35% of friend events and 10% of the
foe events with zero timing error.
The average timing error for Friend events is 1.52 (std 1.64)
sub-windows (equivalent to 190 ms) for LDA and 0.91 (std 1.38)
sub-windows (114 ms) for ARRLS and 0.95 (std 1.36) sub-windows
(119 ms) for ARRLSimb. Foe events have an average timing error of
2.54 (std 1.70) sub-windows (318 ms) for LDA and 0.68 (std 1.02)
sub-windows (85 ms) for ARRLS and 0.66 (std 0.93) sub-windows
(83 ms) for ARRLSimb.
Fig. 6 shows the annotation performance metrics by subject for
the individual subjects in the VEP datasets. In each case, the other
17 subjects formed the training data for the annotation. The left
column shows Friend vs. Others classification, and the right column
shows Foe vs. Others classification. The Foe classification problem
is very difficult and extremely imbalanced. Fig. 6 shows the sta-
tistical significance using the bootstrap ztest described in Section
2.13. Highly significant (p < 0.001) use black squares and results
that are not statistically significant (0.05 < p) use red crosses. The
remaining results have 0.001 < p < 0.05. Subjects, 6, 12, 14, and 17,
which are subjects with poor pairwise classification accuracies,
have relatively poor performance results, although many of them
are statistically significant. All results using the non-parametric test
were significant.
In general, performance is an increasing function of timing toler-
ance, while statistical significance is a decreasing function of timing
tolerance. Intuitively, the latter is true because it is easier to hit a
true positive randomly within say two sub-windows than it is to hit
within one sub-window. Subjects 1, 3, 7, 10, 15, and 16, which have
moderate pairwise classification accuracy, have much better per-
formance at timing tolerances of one than at zero. Performance for
timing tolerance of two is virtually the same as for one for all sub-
jects. We also calculated performance after eliminating Subject 17
from the training set and noticed very little change in performance.
Results here include Subject 17.
366 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374

Fig. 4. Difference between positive and negative ARRLSimb classifier window scores for friend and foe events aligned at sub-window 0.

Fig. 5. Cumulative fraction of positive events that have predictions within the specified timing error. The timing tolerances are in sub-windows, and the events from all 18
subjects are combined.

Table 5 summarizes the average performance in leave-one-
subject-out annotation for VEP for different timing tolerances.
Subjects are grouped by their average pair-wise classification accu-
racy.
The average performance for Friend training is quite high for the
Good group, especially for a timing tolerance of one sub-window,
while performance for the Poor group is low. As expected, Foe
events are more difficult to annotate, most likely because of the
very limited amount of training data available.
A closer examination of the results revealed that many false pos-
itives were due to hits on the other class. For this reason, we also
computed performance by scoring a hit if an event of either class
was present within the specified timing tolerance. Table 5 displays
these results in parentheses. As expected, precision improved. For
example, when the training class was Friend, the precision went
from 0.83 to 0.93 for the Good group and from 0.65 to 0.75 for
the Moderate group when the timing tolerance was one. When the
training class was Foe, the precision went from 0.25 to 0.83 for the
Good group and from 0.20 to 0.66 for the Moderate group when the
timing tolerance was one. The recall when the training class was
Friend did not change very much. As expected, recall was much
lower for combined classes when the training class was Foe. This
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
Fig. 6. Performance of annotation by subject for VEP datasets for different timing tolerances. Red × ’s mark values that are not statistically significant. Square boxes mark
values that are highly significant (p > 0.0001). All other values have 0.001 < p < 0.05.
is because the number of actual positive events increases by a fac-
tor of seven. The statistical significance of the results for combined
class hits is slightly lower due to an increased probability of hitting
a positive event at random.
3.4. Observation of annotation patterns
Misalignments in event response timing exist both within a sub-
ject and across subjects. The sliding window annotation process
introduces another level of temporal “jitter”. Fig. 7 illustrates some
sample misalignments of predicted labels using Subject 1 of the
VEP dataset as the test subject. As mentioned previously, we train
ARRLSimb classifiers for each of the 17 other subjects using either
Friend or Foe samples for the positive class and non-overlapping
other samples for the negative class. We then extract the predicted
labels for a given test-training subject pair by using the window
weighting and zero-out procedure.
367
Each short vertical black line in the upper 17 rows of each panel
in Fig. 7 marks a label predicted by one of the 17 training subjects.
The top panel corresponds to annotation of Friend events, and the
bottom panel corresponds to annotation of Foe events. The short
vertical red lines in the bottom row show the timing of actual Foe
samples. The blue vertical lines in the bottom row mark Friend sam-
ples. As expected, the predicted timings are not exactly aligned,
and we must consider these timing variations when combining the
predictions. Because of zero-out, predicted events are isolated. The
data drawn in Fig. 7 are actually the normalized scores estimated
by each classifier as shown in Fig. 3.
Several interesting features are apparent here. In the experi-
ment, subjects were asked to identify friend or foe images, with
the expectation that relatively rare Foe events would elicit an
oddball response. Images were presented roughly at 2-s intervals
and approximately one-seventh of the images were foes. Odd-
ball responses at the beginning of the experiments were present
368 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374

Table 2
Weights W and mask used to compute sub-window scores.

Timing (j) −7 −6 −5 −4 −3 −2 −1 0 1 2 3 4 5 6 7

Wj × 21 0.5 0.5 0.5 0.5 0.5 1 3 8 3 1 0.5 0.5 0.5 0.5 0.5
Mask 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0

Table 3
Evaluation of predictions for different timing error tolerances in units of sub-windows. Sub-windows with true events are marked with a T and those with predicted events
are marked with a P. Sub-windows for which no prediction is made are marked with 0.

Sub-window 1 2 3 4 5 6 7 8 9 10

True event 0 0 T 0 0 T 0 0 0 T
Predicted event P 0 0 0 0 P 0 0 P 0
Timing tolerance 0 Fail Hit Fail
Timing tolerance 1 Fail Hit Hit
Timing tolerance 2 Hit Hit Hit

Fig. 7. Misaligned predicted labels for a representative sample when Subject 1 is the test subject and the other 17 subjects are the training subjects for Friend versus others
and Foe versus others, respectively. The true Foe events appear as red vertical lines and Friend events appear as green vertical lines in the bottom row. The blue arrows at the
top indicate interesting or unexpected predictions. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

in several datasets. Anecdotal feedback from other experimenters
suggests that experimentalists sometimes exclude the beginning
events in an experimental record for this reason. Many of the sub-
jects show an initial response before the first event (first blue
vertical line in the bottom row in Fig. 7 at around 16 s). The experi-
menter indicated that an image cueing the start of the experiment
was displayed prior to the first visual stimulus presentation.
Fig. 7 shows clear oddball responses annotated at around 38 s
and 48 s (marked with the first two blue arrows at the top). Notice
that expected event presentations were missing from the exper-
imental record. Events corresponding to subject button presses
occurred after each of these “missing” events. Foe events, or at
least some events, were roughly “due” at that time. The annota-
tion system also detected “missing” events in other experimental
records.
Based on this observation, we re-examined the original raw data
and the log of experimental event codes. In doing so, we uncov-
ered a variable hardware error had caused a small number of trial
event codes to be omitted from the recording of the data stream,
although they corresponded to actual presentations to the subject.
The “missing” codes that were highly predicted corresponded to
missing log entries for an oddball event. This hardware error did not
affect standard ERP analysis, because epoching only occurs around
events explicitly marked in the data.
We examined the entire experimental record and also found
that many subjects showed an oddball response after each ses-
sion block ended. A closer examination of the experimental
protocol revealed that subjects were presented with a congrat-
ulatory “You’ve made it through the block” image at the end of
these sessions, which corresponded to these unanticipated odd-
ball responses. The fact that the method found very few oddball
responses that could not be explained supports the usefulness of
this approach for analyzing unlabeled data.
The annotation system identifies samples in the data that are
likely to be related to the positive class in a set of labeled samples.
After annotating, we can order the sub-windows by their overall
score. Fig. 8 shows the high scoring sub-windows of subjects 9 (top
row) and 14 (bottom row), respectively. Subject 9 had the best aver-
age pair-wise classification accuracy (Fig. 3), while subject 14 had
the worst. Each row of each panel corresponds to a candidate event
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374 369

Fig. 8. Predicted Friend (left panel) or Foe (right panel) events for subjects 9 (top row) and 14 (bottom row). The scores appear in decreasing order using a timing tolerance of
two sub-windows. The predicted events are aligned with respect to their sub-window 0, color-coded by their closest true event/subject response. The marked sub-windows
on either side of zero correspond to other true events that appear within +/− 2 s (16 sub-windows) on either side of the predicted events. All non-zero scores are displayed,
with dotted lines marking the retrieval threshold.

(step j of Table 1) for either Friend (left panel) or the Foe (right panel)
as the positive class. The sub-windows for the candidate events are
aligned at zero, with candidate events arranged vertically in order
of decreasing score.
Fig. 8 represents each candidate event by a short horizontal bar,
marking the sub-window during which it occurs by color-coding
based on the relationship of this event to the closest true event as
described in Table 6.
Predicted events displayed in red are within two sub-windows
of an actual foe event and have been recognized by the subject as
foe by a correct button press (CR, Correct Response). If the closest
true event is a Foe event that the subject incorrectly identified (IR,
Incorrect Response) as a Friend event, the predicted event appears
in orange. If the closest true event was a Foe event in which the
subject subsequently missed the button press (NR, No Response),
the sub-window appears in pink. Similarly, if the closest true event
is a friend, the response options appear in blue (CR), aqua (IR),
or purple (NR), respectively. If no true event is within two sub-
windows of this predicted event, the sub-window appears in black.
The “No event” condition marks a situation in which no Friend or Foe
event was marked in the data. The panels only display the scores
corresponding to the candidate events of step j of the annotation
algorithm of Table 1. The dotted line marks the score corresponding
to the adaptive threshold cutoff of step k of the algorithm.
370 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374

Table 4 Table 6
Characteristics of the VEP data collection and preprocessing. True event categories broken down by subject response.

Variable Values Code Category Description

Channels 64 CRa Foe w/CR Foe image with correct response

Sampling rate 512 Hz
Number of subjects 18
Dataset length 563 s (average length/subject)
Friend vs others samples 235 Friend vs 1070 others (average/subject)
Foe vs others samples 34 Foe versus 3991 others (average/subject)
Sample window size 8 sub-windows (1 s)
Sub-window length 0.125 s
Window offset One sub-window (0.125 s)
Sub-bands 8 bands
Feature size 64 channels × 8 sub-windows × 8 sub-bands
Bad channels and referencing PREP pipeline
Artifact removal MARA
In addition to the predicted events, which are aligned at sub-
window 0, Fig. 8 also displays all of the other true events within
+/− 2 s (16 sub-windows) of the predicted event. Since subjects
viewed images at roughly 2-s intervals, the display area of each
graph in Fig. 8 roughly corresponds to three true events. The true
events directly before and after the high-scoring predicted events
align well with the actual timing of the true events. In the case of
the Foe events, a number of the lower scoring predictions do not
correspond to true events, but the predictions seem to be directly
out of phase with actual events of any type. Very few predicted
events appear in areas where no event of either type occurs. Subject
9 only has 33 Foe events, and most Foe predictions are clustered at
the top of the scoring order. The challenge for annotation when the
events are truly unknown is to decide where to set the annotation
threshold with no prior knowledge of how many events actually
appear in the data.
Table 7 shows the results for Subject 9 in more detail. Many
of the incorrect predictions correspond to actual events from the
other class. The problem with the Foe events in Table 9 is that the
default classifier thresholds include more events than actually exist
in the data. Thresholds based on predicting the top 30 events give
much better Foe event accuracies.
Fig. 9 shows how the distribution of annotation scores can give
an indication of the success from the unlabeled data. The top left
graph, corresponding to Friend events for Subject 9, is typical of
annotations with high performance. The Gaussians representing
Negative and Positive annotations are well-separated. The second
row in the first column, representing Friend events for Subject 1 is
IRa Foe w/IR Foe image with incorrect response
NRa Foe w/NR Foe image with no response
CRb Friend w/CR Friend image with correct response
IRb Friend w/IR Friend image with incorrect response
NRb Friend w/NR Friend image with no response
No No event Neither friend or foe
typical of moderate performance. The two Gaussians representing
Negative and Positive scores are moderately well-separated, and
the leftover distribution for the Positive fit is reasonably Gaussian
in shape. The remaining cases have much lower performance. These
types of distributions indicate that one should be more conservative
in accepting annotations with scores near the cutoff.
3.5. The EEG-Annotate toolbox
We have released an open-source MATLAB toolbox called EEG-
Annotate on Github (http://github.com/VisLab/EEG-Annotate) and
have released as a data note the VEP dataset used for training and
testing. The toolbox supports four base classification methods: LDA,
ARRLS, ARRLSMod, and ARRLSimb. ARRLSMod and ARRLSimb are
our modifications of the original ARRLS algorithm to better gener-
ate initial pseudo labels and to handle highly imbalanced datasets.
The classifiers are designed for batch processing on a directory of
datasets and produce score data structures that can then be pro-
cessed by the annotator. The toolbox includes report-generating
functions to create the graphs and tables presented in this paper
and to evaluate the statistical precision using bootstrapping. The
toolbox also contains functions for preprocessing EEG and gen-
erating features. The annotation pipeline does not require that
training and testing datasets correspond to EEG. The only depen-
dence of the toolbox on EEG occurs in the preprocessing and feature
generation steps of the process. The toolbox depends on EEGLAB
(Delorme and Makeig, 2004b), PREP (Bigdely-Shamlo et al., 2015),
and MARA (Winkler et al., 2014). We have found that for EEG anno-
tation, artifact removal is essential to prevent the annotator from
being confounded by artifacts such as blinks. The pipeline is fully-
automated and requires no user intervention. We are making the
data used in this study available on NITRC (https://www.nitrc.org/)

Table 5
Average performance for leave-one-subject-out annotation of VEP with subjects grouped by pairwise classification accuracy: Good (2, 4, 5, 8, 9, 11, 18), Moderate (1, 3, 7, 10,
13, 15, 16), and Poor (6, 12, 14, 17). Numbers in parentheses are performance when hits are counted for both classes.

Training class Metric Tolerance Subjects grouped by pairwise accuracy

Good Moderate Poor All

Friend Precision 0 0.65 (0.72) 0.41 (0.45) 0.18 (0.20) 0.45 (0.50)
1 0.83 (0.94) 0.67 (0.77) 0.29 (0.34) 0.65 (0.74)
2 0.83 (0.95) 0.69 (0.80) 0.37 (0.43) 0.68 (0.78)
Recall 0 0.71 (0.69) 0.35 (0.34) 0.14 (0.13) 0.44 (0.43)
1 0.90 (0.90) 0.59 (0.59) 0.23 (0.24) 0.63 (0.63)
2 0.91 (0.91) 0.61 (0.62) 0.28 (0.29) 0.65 (0.65)
RAP 0 0.57 (0.58) 0.19 (0.20) 0.04 (0.04) 0.30 (0.31)
1 0.82 (0.89) 0.44 (0.50) 0.09 (0.11) 0.51 (0.57)
2 0.83 (0.90) 0.47 (0.54) 0.14 (0.17) 0.54 (0.59)
Foe Precision 0 0.19 (0.45) 0.12 (0.32) 0.05 (0.15) 0.13 (0.34)
1 0.25 (0.80) 0.20 (0.65) 0.09 (0.33) 0.20 (0.63)
2 0.25 (0.83) 0.22 (0.74) 0.12 (0.45) 0.21 (0.71)
Recall 0 0.72 (0.30) 0.45 (0.16) 0.21 (0.08) 0.50 (0.19)
1 0.92 (0.49) 0.79 (0.32) 0.42 (0.19) 0.76 (0.36)
2 0.93 (0.51) 0.84 (0.37) 0.51 (0.26) 0.80 (0.40)
RAP 0 0.38 (0.18) 0.16 (0.08) 0.02 (0.01) 0.22 (0.11)
1 0.59 (0.45) 0.42 (0.25) 0.07 (0.06) 0.41 (0.28)
2 0.60 (0.47) 0.45 (0.30) 0.09 (0.11) 0.43 (0.32)
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374 371

Table 7
Breakdown of predicted positives by category of true event with +/− two sub-windows for Subject 9. The event category codes are from Table 6. The fraction correctly
annotated appears in the last column. Subject 9 has 237 Friend events and 33 Foe events.

Class CRa IRa NRa CRb IRb NRb No Correct

Friend 29 4 0 226 5 0 4 0.86

Foe 29 4 0 202 5 0 6 0.13

Fig. 9. Distribution of non-zero annotation scores for Friend (left column) or Foe (right column) events for subjects 9 (top row), 1 (middle row) and 14 (bottom row). A
Gaussian is fit to the largest peak (medium gray line (denoted Negative) and removed. A second Gaussian is fit to remaining scores (denoted Positive). Amplitudes of fitting
Gaussians are scaled to maximum value of their respective distributions.

and providing a detailed description of the data in an accompanying
data note linked to the paper.
4. Discussion
Event identification and annotation presents several challenges:
one rarely has sufficient time-locked data for a single subject to
develop an accurate classifier to label continuous data not used
to develop the classifier. In addition, inter-subject classification is
notoriously difficult for EEG based on the variability of individual
subject responses. Further, it is difficult to establish the accuracy of
the annotations when only unlabeled data is available.
This work demonstrates the feasibility of automated annota-
tion by developing a specific practical implementation. Here we
demonstrate this by applying the approach to an 18-subject dataset
to annotate unlabeled data from one subject based on labels from
the other 17 subjects. We use the time-locked simulus-response
labels as “ground truth” for the verification. We demonstrate that
even a system based on simple features can provide useful and
detailed annotations for unlabeled EEG and present exemplar cases
where the system even found erroneously missing event codes in
a dataset. We emphasize that many choices could be made for the
required components and multiple approaches can be federated to
improve accuracy. Many implementations of the annotation sys-
372 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
tem outlined above are possible, and success depends on factors
such as the features selected, the classifiers used, the method of
balancing the training sets, and the similarity of the tasks and sub-
jects. However, annotation can label not only events in the data
for subsequent time-locked analysis, but can also identify times
when a subject’s brain response is “similar” to the pattern exhib-
ited by other subjects during other tasks. Thus, we might apply
this type of automated brain response labeling in much the same
way as researchers use automated genome annotation systems to
conditionally identify genes and their potential functions (Yandell
and Ence, 2012). In genomics, these provisional annotations have
led to many experiments designed to verify the annotations and
to characterize individual genomic variability (Boyle et al., 2012).
Similarly, brain response annotation may lead to new insights into
brain function, connectivity, and association of response with tasks,
once sufficient annotated data becomes available. At the very least,
these methods can be used to improve the overall robustness of
existing labels, and eventually whole databases, by leveraging their
inherent redundancy to provide means to identify cases where data
and metadata may be missing or aberrant.
In this work, we start the sub-windows at the beginning of
the dataset rather than time-locking the positive samples, treating
the labeled training and unlabeled test data in the same fashion.
Because of the lack of time-locking in the test data, we focus on
features that are not particularly sensitive to exact timing. Exam-
ples of time-insensitive features include total power, bag-of-words
(BOW), autoregressive (AR), and spectral features. We have also
experimented with both time-locked and non-time-locked positive
training windows and found very little difference in the annotation
results for these particular features. Other choices of features may
be more sensitive to positive sample time-locking. For simplicity,
in this work we use features on total power by channel, since the
datasets we consider have roughly the same channel configura-
tions.
Simple power features contain temporal and frequency infor-
mation but the spatial information is headset dependent. For
cross-headset leave-one-subject-out, bag-of-words (BOW) fea-
tures give higher classification accuracies on the VEP cross-headset
collection (Su and Robbins, 2015). We are exploring the possibil-
ity of using common dictionaries across collections to annotate a
variety of tasks and headsets using the same core classifiers.
Many choices of parameters can be manipulated beyond the
choice of features. Selection of classifier is one such choice. We
have used several classifiers: Linear Discriminant Analysis (LDA),
and Adaptation Regularization using Regularized Least Squares
(ARRLS), ARRLSMod (ARRLS modified for pseudo-labels), and ARRL-
Simb (ARRLS modified to handle imbalance). More recently we have
experimented with deep learning classification using convolution
networks and power features, but the results of pairwise classi-
fication particularly for the Foe vs others case are poor. Work on
artificially expanding the training set and using more sophisticated
network approaches is ongoing.
We use straightforward methods for computing and combining
sub-window scores after the initial classification. The predictions of
a target tend to cluster near a target, but usually spread over adja-
cent sub-windows due to timing variations and the contribution of
each sub-window to multiple windows. Applying a weighted win-
dow average and then thresholding gives better predictions than
simply thresholding each score. Furthermore, since sub-windows
adjacent to an event have high scores, we use a zero-out proce-
dure to avoid double counting. Other weight matrices are possible,
and the results appear to be insensitive to the exact choice of
weights. We have considered only symmetric weights, but asym-
metric weights are also possible and may be more appropriate in
some cases.
Classifier fusion is a classical problem in machine learning, and
there are many potential ways to combine results from multiple
classifiers (Ho et al., 1994; Kuncheva et al., 2001; Hou et al., 2014;
Woźniak et al., 2014; Britto et al., 2014; Cruz et al., 2015). We
have only considered the most straightforward approaches, but
more sophisticated methods may be useful. For example, one could
use many different annotation methods and then apply weight-of-
evidence approaches based on Dempster-Shafer theory (Liu et al.,
2014) or Bayesian networks (Zhang et al., 2013b). Recent work
in spectral ranking (Parisi et al., 2014) and variational methods
(Nguyen et al., 2016) also show potential for improving the output
from federations of classifiers beyond simple addition of scores or
votes. The goal of this work is not to provide an exhaustive exami-
nation of the possibilities, but rather to demonstrate the feasibility
of annotation of EEG data. Much work has also been done in the
image retrieval and web search communities on re-ranking algo-
rithms (Mei et al., 2014) and this work is potentially applicable to
EEG annotation.
While annotation generally places more emphasis on precision
(reducing the number of false positives), good recall results can
be useful input for downstream re-ranking or for use with addi-
tional voting schemes by combining results from using other types
of features or methods.
Automated annotation of EEG is a new direction in EEG analysis
that is oriented towards enabling discovery in non-ideal scenarios.
As EEG moves into more realistic cases or non-laboratory locations
(McDowell et al., 2013), researchers can no longer clearly mark and
time lock to events. Analyses will become increasingly challeng-
ing for multiple reasons. First, consistent, precise time locking is
extremely difficult, if not impossible to achieve in many situations
due to the complexity of real-world events, where event markers
may have to be added post-hoc during manual review (such as
identification of events from video feed taken in the field). Addi-
tionally, with added experimental or environmental complexity,
the probability that events are missed or overlooked due to faulty
equipment or human error during post-hoc encoding increases.
The approaches developed here can help with these challenges and
facilitate analyses through proper annotation.
Note also that a data scenario such as the VEP can provide “pro-
totypical” oddball training to search for oddball responses in other
types of data. More advanced annotation systems can use multi-
ple feature sets in conjunction with other types of classifiers and
combination methods as input to weight-of-evidence approaches.
One can envision the creation of EEG databases with provisional
annotations, in much the same as genomic databases have used
machine learning to identify potential genes. With such large-
scale databases at hand, one might use this information to identify
potentially similar subjects and prototypical brain responses. This
could also enable examining enrichment approaches − determin-
ing when features and responses tend to co-occur and beginning
to isolate the features that characterize these responses. Such
databases could also provide the core infrastructure for organizing
knowledge about brain responses and behavior.
This paper focused on a relatively clean dataset with well-
defined events known to have prototypical responses. Even with
extremely simple spectral power features, the system was able to
successfully identify most events without using any labeled data
and no knowledge of when the events occurred in the test dataset.
Our next step is to apply these algorithms on a larger scale to deter-
mine how often such a system is likely to identify such events in EEG
that are not associated with a stereotypical oddball experiment.
We believe that this system is the first step in applying large-scale
annotation to build EEG databases of brain responses. The EEG-
Annotate MATLAB toolbox is freely available at http://github.com/
 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374
VisLab/EEG-Annotate. The VEP dataset is available in a companion
data note.
Conflict of interest statement
All three authors ascertain that they have no conflicts of interest
in this work.
Acknowledgments
The authors would like to thank Scott Kerick and Amar Marathe
of the Army Research Laboratory for their comments and sug-
gestions. This research was sponsored by the Army Research
Laboratory and was accomplished under Cooperative Agreement
Number W911NF-10-2-0022. The views and conclusions contained
in this document are those of the authors and should not be inter-
preted as representing the official policies, either expressed or
implied, of the Army Research Laboratory or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute
reprints for Government purposes notwithstanding any copyright
notation herein.
References
Arroyo, S., Lesser, R.P., Gordon, B., Uematsu, S., Jackson, D., Webber, R., 1993.
Functional significance of the mu rhythm of human cortex: an
electrophysiologic study with subdural electrodes. Electroencephalogr. Clin.
Neurophysiol. 87 (September (3)), 76–87.
Bigdely-Shamlo, N., Mullen, T., Kothe, C., Su, K.-M., Robbins, K.A., 2015. The PREP
pipeline: standardized preprocessing for large-scale EEG analysis. Front.
Neuroinf. 9.
Blankertz, B., Lemm, S., Treder, M., Haufe, S., Müller, K.-R., 2011. Single-trial
analysis and classification of ERP components — a tutorial. Neuroimage 56
(May (2)), 814–825.
Boyle, A.P., et al., 2012. Annotation of functional variation in personal genomes
using RegulomeDB. Genome Res. 22 (September (9)), 1790–1797.
Britto Jr., A.S., Sabourin, R., Oliveira, L.E.S., 2014. Dynamic selection of classifiers—a
comprehensive review. Pattern Recognit. 47 (November (11)), 3665–3680.
Buzsáki, G., 2002. Theta oscillations in the hippocampus. Neuron 33 (January (3)),
325–340.
Congedo, M., Korczowski, L., Delorme, A., Lopes da silva, F., 2016. Spatio-temporal
common pattern: a companion method for ERP analysis in the time domain. J.
Neurosci. Methods 267 (July), 74–88.
Cruz, R.M.O., Sabourin, R., Cavalcanti, G.D.C., Ing Ren, T., 2015. META-DES: a
dynamic ensemble selection framework using meta-learning. Pattern
Recognit. 48 (May (5)), 1925–1935.
Delorme, A., Makeig, S., 2004a. EEGLAB: an open source toolbox for analysis of
single-trial EEG dynamics including independent component analysis. J.
Neurosci. Methods 134 (March (1)), 9–21.
Delorme, A., Makeig, S., 2004b. EEGLAB: an open source toolbox for analysis of
single-trial EEG dynamics including independent component analysis. J.
Neurosci. Methods 134 (March (1)), 9–21.
Deselaers, T., Keysers, D., Ney, H., 2007. Features for image retrieval: an
experimental comparison. Inf. Retr. 11 (December (2)), 77–107.
Dietrich, A., Kanso, R., 2010. A review of EEG, ERP, and neuroimaging studies of
creativity and insight. Psychol. Bull. 136 (September (5)), 822–848.
Engel, A.K., Fries, P., 2010. Beta-band oscillations — signalling the status quo? Curr.
Opin. Neurobiol. 20 (April (2)), 156–165.
Frishkoff, G.A., Frank, R.M., Rong, J., Dou, D., Dien, J., Halderman, L.K., 2007. A
framework to support automated classification and labeling of brain
electromagnetic patterns. Comput. Intell. Neurosci. 2007 (December), e14567.
Frishkoff, G., et al., 2011. Minimal Information for Neural Electromagnetic
Ontologies (MINEMO): a standards-compliant method for analysis and
integration of event-related potentials (ERP) data. Stand. Genomic Sci. 5
(November (2)), 211–223.
Hairston, W.D., et al., 2014. Usability of four commercially-oriented EEG systems. J.
Neural Eng. 11 (August (4)), 046018.
Ho, T.K., Hull, J.J., Srihari, S.N., 1994. Decision combination in multiple classifier
systems. IEEE Trans. Pattern Anal. Mach. Intell. 16 (January (1)), 66–75.
Hou, J., E, X., Xia, Q., Qi, N.-M., 2014. Evaluating classifier combination in object
classification. Pattern Anal. Appl. 18 (February (4)), 799–816.
Jung, T.-P., Makeig, S., Westerfield, M., Townsend, J., Courchesne, E., Sejnowski, T.J.,
2001. Analysis and visualization of single-trial event-related potentials. Hum.
Brain Mapp. 14 (November (3)), 166–185.
Kellihan, B., et al., 2013. A real-world neuroimaging system to evaluate stress. In:
Schmorrow, D.D., Fidopiastis, C.M. (Eds.), Foundations of Augmented
Cognition. Springer, Berlin Heidelberg, pp. 316–325.
Kiebel, S.J., Friston, K.J., 2004a. Statistical parametric mapping for event-related
potentials: I. Generic considerations. Neuroimage 22 (June (2)), 492–502.
373
Kiebel, S.J., Friston, K.J., 2004b. Statistical parametric mapping for event-related
potentials (II): a hierarchical temporal model. Neuroimage 22 (June (2)),
503–520.
Klimesch, W., 1999. EEG alpha and theta oscillations reflect cognitive and memory
performance: a review and analysis. Brain Res. Brain Res. Rev. 29 (April (2–3)),
169–195.
Kuncheva, L.I., Bezdek, J.C., Duin, R.P.W., 2001. Decision templates for multiple
classifier fusion: an experimental comparison. Pattern Recognit. 34 (February
(2)), 299–314.
Larson, M.J., Carbine, K.A., 2017. Sample size calculations in human
electrophysiology (EEG and ERP) studies: a systematic review and
recommendations for increased rigor. Int. J. Psychophysiol. 111 (January),
33–41.
Lemm, S., Blankertz, B., Dickhaus, T., Müller, K.-R., 2011. Introduction to machine
learning for brain imaging. Neuroimage 56 (May (2)), 387–399.
Lin, C.-T., Pal, N.R., Chuang, C.-Y., Jung, T.-P., Ko, L.-W., Liang, S.-F., 2008. An
EEG-based subject- and session-independent drowsiness detection. IEEE
International Joint Conference on Neural Networks, 2008. IJCNN 2008. (IEEE
World Congress on Computational Intelligence), 3448–3454.
Liu, H., Frishkoff, G., Frank, R., Dou, D., 2012a. Sharing and integration of cognitive
neuroscience data: metric and pattern matching across heterogeneous ERP
datasets. Neurocomputing 92 (September), 156–169.
Liu, Y., Chen, W., Yang, S., Huang, K., 2012b. Domain adaptation to automatic
classification of neonatal amplitude-integrated EEG. 2012 12th International
Conference on Intelligent Systems Design and Applications (ISDA), 131–136.
Liu, Y., Wang, X., Liu, K., 2014. Network anomaly detection system with optimized
DS evidence theory. Sci. World J. 2014 (August), e753659.
Long, M., Wang, J., Ding, G., Pan, S.J., Yu, P.S., 2014. Adaptation regularization: a
general framework for transfer learning. IEEE Trans. Knowl. Data Eng. 26 (May
(5)), 1076–1089.
Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., Arnaldi, B., 2007. A review of
classification algorithms for EEG-based brain–computer interfaces. J. Neural
Eng. 4 (2), R1.
Müller, K.-R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., Blankertz, B.,
2008. Machine learning for real-time single-trial EEG-analysis: from
brain–computer interfacing to mental state monitoring. J. Neurosci. Methods
167 (January (1)), 82–90.
Maris, E., Oostenveld, R., 2007. Nonparametric statistical testing of EEG- and
MEG-data. J. Neurosci. Methods 164 (August (1)), 177–190.
McDowell, K., et al., 2013. Real-world neuroimaging technologies. IEEE Access 1,
131–149.
Mei, T., Rui, Y., Li, S., Tian, Q., 2014. Multimedia search reranking: a literature
survey. ACM Comput. Surv. 46 (January (3)), 38:1–38:38.
Mumtaz, W., Malik, A.S., Yasin, M.A.M., Xia, L.K., 2015. Review on EEG and ERP
predictive biomarkers for major depressive disorder. Biomed. Signal Process.
Control 22 (September), 85–98.
Nguyen, T.T., Nguyen, T.T.T., Pham, X.C., Liew, A.W.-C., 2016. A novel combining
classifier method based on Variational Inference. Pattern Recognit. 49
(January), 198–212.
Palva, S., Palva, J.M., 2007. New vistas for ␣-frequency band oscillations. Trends
Neurosci. 30 (April (4)), 150–158.
Parisi, F., Strino, F., Nadler, B., Kluger, Y., 2014. Ranking and combining multiple
predictors without labeled data. Proc. Natl. Acad. Sci. 111 (January (4)),
1253–1258.
Ries, A.J., Touryan, J., Vettel, J., McDowell, K., Hairston, W.D., 2014. A comparison of
electroencephalography signals acquired from conventional and mobile
systems. J. Neurosci. Neuroeng. 3 (February (1)), 10–20.
Rivet, B., Souloumiac, A., Attina, V., Gibert, G., 2009. xDAWN algorithm to enhance
evoked potentials: application to brain-computer interface. IEEE Trans.
Biomed. Eng. 56 (August (8)), 2035–2043.
Roach, B.J., Mathalon, D.H., 2008. Event-related EEG time-frequency analysis: an
overview of measures and an analysis of early gamma band phase locking in
schizophrenia. Schizophr. Bull. 34 (September (5)), 907–926.
Rousselet, G.A., Husk, J.S., Pernet, C.R., Gaspar, C.M., Bennett, P.J., Sekuler, A.B.,
2009. Age-related delay in information accrual for faces: evidence from a
parametric, single-trial EEG approach. BMC Neurosci. 10 (1), 114.
Souloumiac, A., Rivet, B., 2013. Improved estimation of EEG evoked potentials by
jitter compensation and enhancing spatial filters. 2013 IEEE International
Conference on Acoustics, Speech and Signal Processing, 1222–1226.
Su, K.M. and Robbins, K.A., 2015. Space-time frequency bag of words models for
capturing EEG variability. A comprehensive study, UTSA-CS-TR-2015-001,
2015.
Su, K.M., Hairston, W.D., Robbins, K.A., 2016. Adaptive thresholding and
reweighting to improve domain transfer learning for unbalanced data with
applications to EEG imbalance. 2016 15th IEEE International Conference on
Machine Learning and Applications (ICMLA), 320–325.
Winkler, I., Haufe, S., Tangermann, M., 2011. Automatic classification of artifactual
ICA-components for artifact removal in EEG signals. Behav. Brain Funct. 7 (1),
1–12.
Winkler, I., Brandl, S., Horn, F., Waldburger, E., Allefeld, C., Tangermann, M., 2014.
Robust artifactual independent component classification for BCI practitioners.
J. Neural Eng. 11 (3), 035013.
Woźniak, M., Graña, M., Corchado, E., 2014. A survey of multiple classifier systems
as hybrid systems. Inf. Fusion 16 (March), 3–17.
Wu, D., Lance, B., Lawhern, V., 2014. Transfer learning and active transfer learning
for reducing calibration data in single-trial classification of visually-evoked
374 K.-m. Su et al. / Journal of Neuroscience Methods 293 (2018) 359–374

potentials. 2014 IEEE International Conference on Systems, Man and
Cybernetics (SMC), 2801–2807.
Wu, D., Lawhern, V.J., Lance, B.J., 2015. Reducing offline BCI calibration effort using
weighted adaptation regularization with source domain selection. 2015 IEEE
International Conference on Systems, Man, and Cybernetics (SMC), 3209–3216.
Yandell, M., Ence, D., 2012. A beginner’s guide to eukaryotic genome annotation.
Nat. Rev. Genet. 13 (May (5)), 329–342.
Zhang, R., Xu, P., Guo, L., Zhang, Y., Li, P., Yao, D., 2013a. Z-Score linear discriminant
analysis for EEG based brain-computer interfaces. PLoS One 8 (September (9)),
e74433.
Zhang, H., Yang, H., Guan, C., 2013b. Bayesian learning for spatial filtering in an
EEG-based brain–computer interface. IEEE Trans. Neural Netw. Learn. Syst. 24
(July (7)), 1049–1060.