Speeding Up Your Analysis With Distributed Computing

Speeding up your analysis with distributed computing
tutorial, qsub, peer, distcomp, matlab, MEG-language
Speeding up your analysis with distributed

computing
Introduction
Many times you are faced with the analysis of multiple subjects and experimental conditions, or with
the analysis of your data using multiple analysis parameters (e.g. frequency bands). Parallel
computing in MATLAB can help you to speed up these types of analysis.
Note that this is usually referred to as distributed computing if you are submitting multiple
independent computations to multiple computers. The term parallel computing is usually reserved
for multiple CPUs or computers working simultaneously at the same problem that requires constant
sharing of small snippets of data between the CPUs. Since the analyses of multiple subjects are done
independently of each other, we call it distributed computing.
This tutorial describes two approaches for distributing the analysis of multiple subjects and
conditions. The data used in this example is the same as in the tutorial scripts: 151-channel MEG
was recorded in 4 subjects, and in each dataset there are three experimental conditions (FC, FIC,
IC). Both approaches rely on the qsubcellfun function which applies a given function to each element
of a cell-array. The function execution is done in parallel on the Torque batch queue system.
After this tutorial you should be able to execute your multi-subject analysis in parallel and design
analysis scripts that allow for easy parallelization, either over subjects or over parameters used in
the analysis.
In this tutorial we use the qsub toolbox that is released along with FieldTrip. There are alternative
methods for distributed computing, such as the MATLAB parallel computing toolbox (e.g. using
parfor or dfeval) or with the peer-to-peer toolbox (also included with FieldTrip). More general
information about the different approaches for distributed processing in MATLAB can be found in
the frequently asked questions.
Background
The qsub distributed computing toolbox was implemented with FieldTrip (and SPM) in mind. At the
moment however, FieldTrip itself does not yet make use of distributed computing, i.e. FieldTrip
functions do not automatically distribute the workload. We are of course planning to make that
possible, i.e. that a single cfg.parallel='yes' option will automatically distribute the computational
load over all available nodes.
At the moment the only way of distributing the workload over multiple nodes requires that you adapt
your scripts. The easiest is to distribute the workload of the analysis of multiple subjects and
conditions over multiple nodes.
http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

Procedure
To distribute your processes and to speed up your analyses, we provide two examples. The first
example script will show you how to use basic FieldTrip functions for the distribution.
Using the basic FieldTrip functions in a memory efficient manner requires that you save the
intermediate data of each step to disk, and that you load it upon the next (parallel) step in the
analysis. If you prefer not to store all intermediate results, or if you want to have more control over
other aspects of the parallel execution, you can provide your own functions that are executed in
parallel. This is demonstrated in the second example script.
The distributed operations of FieldTrip functions in this example require the original MEG datasets
for the four subjects, which are available from
ftp://ftp.fieldtriptoolbox.org/pub/fieldtrip/tutorial/Subject01.zip
Or, when at the Donders Centre for Cognitive Neuroimaging, use
cd /home/common/matlab/fieldtrip/data
Example 1: using only FieldTrip functions in distributed

computing
This example script demonstrates how to run basic FieldTrip functions in parallel. The idea is
schematically depicted in the following figure.
subjectlist = {
'Subject01.ds'
'Subject02.ds'
'Subject03.ds'
'Subject04.ds'
};
conditionlist = {
'FC'
'FIC'
'IC'
};
triggercode = [
9
3
5
];
Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

% start with a new and empty configuration

cfg = {};
for subj=1:4
for <a
href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/cond.html">co
nd</a>=1:3
cfg{subj,<a
nd</a>} = [];
cfg{subj,<a
nd</a>}.dataset = subjectlist{subj};
cfg{subj,<a
nd</a>}.trialdef.prestim = 1;
cfg{subj,<a
nd</a>}.trialdef.poststim = 2;
cfg{subj,<a
nd</a>}.trialdef.eventtype = 'backpanel trigger';
cfg{subj,<a
nd</a>}.trialdef.eventvalue = triggercode(<a
nd</a>);
end
end
cfg = qsubcellfun(@ft_definetrial, cfg);
% this extends the previous configuration

for subj=1:4
for <a
nd</a>=1:3
cfg{subj,<a
nd</a>}.channel = {'MEG', '-MLP31', '-MLO12'};
cfg{subj,<a
nd</a>}.demean = 'yes';
cfg{subj,<a
nd</a>}.baselinewindow = [-0.2 0];
cfg{subj,<a
nd</a>}.lpfilter = 'yes';
cfg{subj,<a

nd</a>}.lpfreq = 35;
end
end
data = qsubcellfun(@ft_preprocessing, cfg);

cfg = {};
for subj=1:4
for <a
nd</a>=1:3
% timelockanalysis does not require any non-default settings
cfg{subj,<a
nd</a>} = [];
end
end
timelock = qsubcellfun(@ft_timelockanalysis, cfg, data);
% from here on we won't process the data in parallel any more

% average each condition over all subjects
cfg = [];
avgFC = ft_timelockgrandaverage(cfg, timelock{:,1});
avgFIC = ft_timelockgrandaverage(cfg, timelock{:,2});
avgIC = ft_timelockgrandaverage(cfg, timelock{:,3});
cfg = [];
cfg.layout = 'CTF151.lay';
ft_multiplotER(cfg, avgFC, avgFIC, avgIC);
cfg = [];
cfg.channel = {'MLC33', 'MLC43', 'MLP11', 'MLP12', 'MLP13', 'MLP33',
'MLP34', 'MLT14', 'MLT15', 'MLT25'}
ft_singleplotER(cfg, avgFC, avgFIC, avgIC);

In the code above all data is processed by the distributed computers and subsequently returned to
the workspace of your desktop computer. The data can take quite a lot of RAM, which you can check
like this.
>> whos
Name Size Bytes Class Attributes
conditionlist 3x1 350 cell

subjectlist 4x1 544 cell
triggercode 3x1 24 double
cfg 4x3 9732384 cell
data 4x3 1203237420 cell
timelock 4x3 65515200 cell
...
Instead of returning the 12 variables for the different subjects and conditions all to your workspace,
you can also use the cfg.inputfile and cfg.outputfile options to have the distributed computers
read/write the data to/from disk. For example the section on ft_preprocessing and
ft_timelockanalysis could be changed into
% ...
for subj=1:4
for <a
nd</a>=1:3
cfg{subj,<a
cfg{subj,<a
cfg{subj,<a

cfg{subj,<a
cfg{subj,<a
cfg{subj,<a
nd</a>}.outputfile = <a
href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/sprintf.html"
>sprintf</a>('subj%02d_cond%02d_raw.mat', subj, <a
nd</a>);
end
end
% note that here we don't specify an output parameter

qsubcellfun(@ft_preprocessing, cfg);
cfg = {};
for subj=1:4
for <a
nd</a>=1:3
cfg{subj,<a
nd</a>}.inputfile = <a
>sprintf</a>('subj%02d_cond%02d_raw.mat', subj, <a
nd</a>);
cfg{subj,<a
nd</a>}.outputfile = <a
>sprintf</a>('subj%02d_cond%02d_avg.mat', subj, <a
nd</a>);
end
end
% note that here we don't specify an output parameter

qsubcellfun(@ft_timelockanalysis, cfg);
% ...
Example 2: writing custom functions for distributed

computing
This example script demonstrates how you can efficiently design your custom code for distributed
computing.
subjectlist = {
'Subject01.ds'
'Subject02.ds'
'Subject03.ds'
'Subject04.ds'
};
conditionlist = {
'FC'
'FIC'
'IC'
};
triggercode = [
9
3
5
];

cfg1 = {};
cfg2 = {};
cfg3 = {};
cfg4 = {};
outputfile = {};
for subj=1:4
for <a
nd</a>=1:3
% this is for definetrial and preprocessing
cfg1{subj,<a
nd</a>} = [];
cfg1{subj,<a
nd</a>}.dataset = subjectlist{subj};
cfg1{subj,<a
nd</a>}.trialdef.prestim = 1;
cfg1{subj,<a
nd</a>}.trialdef.poststim = 2;
cfg1{subj,<a

nd</a>}.trialdef.eventtype = 'backpanel trigger';

cfg1{subj,<a
nd</a>}.trialdef.eventvalue = triggercode(<a
nd</a>);
cfg1{subj,<a
cfg1{subj,<a
cfg1{subj,<a
cfg1{subj,<a
cfg1{subj,<a
% this is for timelockanalysis

cfg2{subj,<a
nd</a>} = [];
% this is for megplanar

cfg3{subj,<a
nd</a>} = [];
% this is for combineplanar

cfg4{subj,<a
nd</a>} = [];
% this defines the file that will contain the output

outputfile{subj,<a
nd</a>} = <a
>sprintf</a>('subj%02d_cond%02d_combined.mat', subj, <a
nd</a>);
end
end
% note that the "preproc_timelock_planar" function is defined further down in

this tutorial
qsubcellfun(@preproc_timelock_planar, cfg1, cfg2, cfg3, cfg4,

outputfile);
% let's now load the individual subject data from the 12 *.mat files and
% average it for subsequent plotting
timelock = {};
for subj=1:4
for <a
nd</a>=1:3
tmp = <a
href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/load.html">lo
ad</a>(<a
>sprintf</a>('subj%02d_cond%02d_combined.mat', subj, <a
nd</a>));
timelock{subj,<a
nd</a>} = tmp.combined;
<a
href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/clear.html">c
lear</a> tmp
end
end
cfg = [];
avgFC = ft_timelockanalysis(cfg, timelock{:,1));
avgFIC = ft_timelockanalysis(cfg, timelock{:,2));
avgIC = ft_timelockanalysis(cfg, timelock{:,3));
cfg = [];
cfg.layout = 'CTF151.lay';
ft_multiplotER(cfg, avgFC, avgFIC, avgIC)
This way you can distribute your custom function (e.g. see below) along with the input and output
parameters.
function preproc_timelock_planar(cfg1, cfg2, cfg3, cfg4, outputfile)
cfg1 = ft_definetrial(cfg1);
data = ft_preprocessing(cfg1);
timelock = ft_timelockanalysis(cfg2, data);

<a
lear</a> data
planar = ft_megplanar(cfg3, timelock);

<a
lear</a> timelock

combined = ft_combineplanar(cfg4, planar);

<a
lear</a> planar
<a
href="http://www.mathworks.com/access/helpdesk/help/techdoc/ref/save.html">sa
ve</a>(outputfile, 'combined');
<a
lear</a> combined
Summary and suggested further reading
This tutorial covered how to distribute your computations/workload over multiple computers in a
cluster that uses the Torque or SGE batch queue system. In our example, we have performed a
relatively simple timelock analysis (ERF) on MEG data, but one can imagine that it does not need
many adjustments to distribute any other type of analysis. Using the configuration demonstrated in
Example 2, one can distribute any form of analysis.
FAQs related to issues in this tutorial:
Does a firewall affect the communication between peers?

How can I combine FieldTrip with peer distributed computing?
How can I debug a problematic distributed job?
How can I determine the number of threads that MATLAB uses?
How can I prevent a job from executing twice?
How can I read and write files if I use other people's peers?
How can I set up the peer distributed computing on a large Linux cluster?
How can I set up the peer distributed computing on a single multicore computer?
How can I set up the peer distributed computing on a small number of computers?
How can I stop the different threads created by peermaster and peerslave?
How can I use the command-line peerslave and optimize the MATLAB licenses?
How do I avoid having to allocate N copies of my data if I want to execute N jobs?
How does the peer smartshare algorithm work?
How does the smartmem algorithm work?
How should I call peercellfun when a function requires many inputs (e.g. key-value pairs)?
How to compile MATLAB coooode into stand-alone executables?
How to get started with distributed computing using qsub?
How to get started with peer distributed computing on my own desktop computer?
How to get started with the MATLAB distributed computing toolbox?
What are the different approaches I can take for distributed computing?
What happens if a job fails to execute properly?
What happens with a job that has an error on the slave?
Why are the peers using multicast to announce themselves?
Why does peercellfun resubmit jobs that take too long to get started?

From:
http://www.fieldtriptoolbox.org/ - FieldTrip
Permanent link:
http://www.fieldtriptoolbox.org/tutorial/distributedcomputing
Last update: 2014/01/22 09:26

Speeding Up Your Analysis With Distributed Computing

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

Speeding Up Your Analysis With Distributed Computing

Caricato da

Copyright:

Formati disponibili

Speeding up your analysis with distributed computing

tutorial, qsub, peer, distcomp, matlab, MEG-language

Speeding up your analysis with distributed

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

Or, when at the Donders Centre for Cognitive Neuroimaging, use

Example 1: using only FieldTrip functions in distributed

Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

% start with a new and empty configuration

cfg = qsubcellfun&#40;@ft_definetrial, cfg&#41;;

% this extends the previous configuration

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

data = qsubcellfun&#40;@ft_preprocessing, cfg&#41;;

% start with a new and empty configuration

timelock = qsubcellfun&#40;@ft_timelockanalysis, cfg, data&#41;;

% from here on we won't process the data in parallel any more

Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

conditionlist 3x1 350 cell

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

% note that here we don't specify an output parameter

% note that here we don't specify an output parameter

Example 2: writing custom functions for distributed

Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

% start with a new and empty configuration

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

nd</a>&#125;.trialdef.eventtype = 'backpanel trigger';

% this is for timelockanalysis

% this is for megplanar

% this is for combineplanar

% this defines the file that will contain the output

% note that the "preproc_timelock_planar" function is defined further down in

Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

function preproc_timelock_planar&#40;cfg1, cfg2, cfg3, cfg4, outputfile&#41;

timelock = ft_timelockanalysis&#40;cfg2, data&#41;;

planar = ft_megplanar&#40;cfg3, timelock&#41;;

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

combined = ft_combineplanar&#40;cfg4, planar&#41;;

Summary and suggested further reading

FAQs related to issues in this tutorial:

Does a firewall affect the communication between peers?

Printed on 2016/06/20 07:34 http://www.fieldtriptoolbox.org/

Last update: 2014/01/22 09:26

http://www.fieldtriptoolbox.org/ Printed on 2016/06/20 07:34

Potrebbero piacerti anche

cfg = qsubcellfun(@ft_definetrial, cfg);

data = qsubcellfun(@ft_preprocessing, cfg);

timelock = qsubcellfun(@ft_timelockanalysis, cfg, data);

nd</a>}.trialdef.eventtype = 'backpanel trigger';

function preproc_timelock_planar(cfg1, cfg2, cfg3, cfg4, outputfile)

timelock = ft_timelockanalysis(cfg2, data);

planar = ft_megplanar(cfg3, timelock);

combined = ft_combineplanar(cfg4, planar);