IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 3, NO.

4, OCTOBER-DECEMBER 2015 411

LsPS: A Job Size-Based Scheduler for Efficient

Task Assignments in Hadoop
Yi Yao, Jianzhe Tai, Bo Sheng, and Ningfang Mi, Member, IEEE

AbstractThe MapReduce paradigm and its open source implementation Hadoop are emerging as an important standard for
large-scale data-intensive processing in both industry and academia. A MapReduce cluster is typically shared among multiple users
with different types of workloads. When a flock of jobs are concurrently submitted to a MapReduce cluster, they compete for the
shared resources and the overall system performance in terms of job response times, might be seriously degraded. Therefore, one
challenging issue is the ability of efficient scheduling in such a shared MapReduce environment. However, we find that conventional
scheduling algorithms supported by Hadoop cannot always guarantee good average response times under different workloads. To
address this issue, we propose a new Hadoop scheduler, which leverages the knowledge of workload patterns to reduce average
job response times by dynamically tuning the resource shares among users and the scheduling algorithms for each user. Both
simulation and real experimental results from Amazon EC2 cluster show that our scheduler reduces the average MapReduce job
response time under a variety of system workloads compared to the existing FIFO and Fair schedulers.

Index TermsMapReduce, hadoop, schdeuling, heavy-tailed workloads, bursty workloads

1 INTRODUCTION

M APREDUCE [1] has become an important paradigm for

parallel data-intensive cluster programming due to
its simplicity and flexibility. Essentially, it is a software
framework that allows a cluster of computers to process a
large set of structured or unstructured data in parallel.
MapReduces users are quickly growing. Apache Hadoop
[2] is an open source implementation of MapReduce that
has been widely adopted in the industry area [3]. With the
rise of cloud computing, it becomes more convenient for IT
business to set a cluster of servers in the cloud and launch a
batch of MapReduce jobs. Therefore, there are a large vari-
ety of data-intensive applications using the MapReduce
framework.
In a classic Hadoop system, each MapReduce job is parti-
tioned into small tasks which are distributed and executed
across multiple machines. There are two kinds of tasks, i.e.,
map tasks and reduce tasks. Each map task applies the
same map function to process a block of the input data and
produces intermediate results in the format of key-value
pairs. The intermediate data will be partitioned by hash
functions and fetched by the corresponding reduce tasks as
their inputs. Once all the intermediate data has been
fetched, a reduce task starts to execute and produce the final
results. The Hadoop implementation closely resembles the

Y. Yao, J. Tai, and N. Mi are with the Department of Electrical and Com-
puter Engineering, Northeastern University, 409 Dana Research Center,
360 Huntington Avenue, Boston, MA 02115.
E-mail: {yyao, jtai, ningfang}@ece.neu.edu.

B. Sheng is with the Department of Computer Science, University of Mas-
sachusetts Boston, Boston, MA.
E-mail: shengbo@cs.umb.edu.
Manuscript received 18 Nov. 2013; revised 23 June 2014; accepted 25 June
2014. Date of publication 29 July 2014; date of current version 9 Dec. 2015.
Recommended for acceptance by R. Campbell.
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TCC.2014.2338291
2168-7161 2014 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
MapReduce framework. A single master node is adopted to
manage distributed slave nodes. The master node commu-
nicates with slave nodes with heartbeat messages which con-
sist of status information of slaves. Job scheduling is
performed by a centralized jobtracker routine in the master
node. The scheduler assigns tasks to slave nodes which
have free resources and response to the heartbeats as well.
The resources in each slave node are represented as map/
reduce slots. Each slave node has a fixed number of slots,
and each map (resp. reduce) slot only processes one map
(resp. reduce) task at one moment.
Typically, multiple users compete for the available slots
in a MapReduce cluster when these users concurrently sub-
mit jobs to the cluster. As a result, the average job response
times,1 an important performance concern in MapReduce
systems, might be seriously degraded. Recent studies [4],
[5], [6] found that MapReduce workloads often exhibit the
heavy-tailed (or long-tailed) characteristics, where a Map-
Reduce workflow consists of few but extremely large jobs
and many small jobs. In such a MapReduce system, an effi-
cient scheduling policy is a critical factor for improving sys-
tem performance in terms of average job response times.
However, we found that the existing policies supported by
Hadoop do not perform well under heavy-tailed and
diverse workloads. By default, Hadoop uses a first-in-first-
out (FIFO) scheduler which is originally designed to opti-
mize the completion length (i.e., makespan) of a batch of
jobs. However, it is not efficient in clusters which serve
diverse workloads since small jobs will experience
extremely long waiting time when submitted after a large
job. Fair scheduler [7], [8] was proposed to improve the
average job response times in shared Hadoop clusters by
1. The response time of a MapReduce job is measured from the time
when that job is submitted to the time when it is completed, i.e., the
summation of that jobs waiting time and execution time.
412 IEEE TRANSACTIONS ON CLOUD COMPUTING,
assigning to all jobs, on average, an equal share of resources
over time. However, we notice that the Fair scheduler makes
its scheduling decision without considering workload pat-
terns of different users.
Compared to the early stage, the workloads in Map-
Reduce system have been changing towards the follow-
ing three directions. First, a MapReduce cluster, once
established, is no longer dedicated to a particular job,
but to multiple jobs from different applications or users.
For example, Facebook [9] is one of Hadoops biggest
champions, which keeps more than 100 petabytes of
Hadoop data online, and allows multiple applications
and users to submit their ad hoc queries to the shared
Hive-Hadoop clusters. Second, as a data processing ser-
vice, MapReduce is becoming prevalent and open to
numerous clients from the Internet, like todays search
engines service. For example, a smartphone user may
send a job to a MapReduce cluster through an App ask-
ing for the most popular words in the tweets logged in
the past three days. Third, the characteristics of MapRe-
duce jobs vary a lot. It is essentially caused by the diver-
sity of user demands. Recent analysis on MapReduce
workloads of current enterprise clients [10], e.g.,
Facebook and Yahoo!, has revealed the diversity of Map-
Reduce job sizes which range from seconds to hours.
Overall, workload diversity is common in practice when
jobs are submitted by different users. For example, some
users run small interactive jobs while other users submit
large periodical jobs; some users run jobs for processing
files with similar sizes while the sizes of jobs from other
users are quite vary. We thus argue that a good Hadoop
scheduler should take into consideration the diversity in
workload characteristics with the goal of reducing Map-
Reduce job execution times.
In this paper, we propose a novel Hadoop scheduler,
called LsPS, which aims to improve the average job
response time of Hadoop systems by leveraging the job
size patterns to tune its scheduling schemes among users
and for each user as well. Specifically, we first develop a
lightweight information collector that tracks statistic
information of recently finished jobs from each user. A
self-tuning scheduling policy is then designed to sched-
uler Hadoop jobs at two levels: the resource shares across
multiple users are tuned based on the estimated job size of
each user; and the job scheduling for each individual user is
further adjusted to accommodate to that users job size
distribution. Experimental results in both the simulation
model and the Amazon EC2 Hadoop cluster environment
confirm the effectiveness and the robustness of our solu-
tion. We show that our scheduler improves the average
job response times under a variety of system workloads
under the comparisons with FIFO, Fair and Capacity
schedulers which have been widely adopted as the stan-
dard scheduling disciplines in MapReduce frameworks,
such as Apache Hadoop.
2 MOTIVATIONS
In order to investigate the pros and cons of the existing
Hadoop schedulers (i.e., FIFO and Fair), we conduct sev-
eral experiments in a Hadoop cluster at Amazon EC2. We
VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
lease 11 EC2 nodes, where one node serves as the master
and the remaining 10 nodes run as the slaves. In this
Hadoop cluster, each slave node contains two map slots
and two reduce slots. The WordCount application is run
to compute the occurrence frequency of each word in
input files with different sizes. The randomtextwriter
program is used to generate random files as inputs to the
WordCount applications.
2.1 How to Share Slots
Specifically, there are two tiers of scheduling in a Hadoop
system which is shared by multiple users: (1) Tier 1 is
responsible for assigning free slots to active users; and (2)
Tier 2 schedules jobs for each individual user. In this sec-
tion, we first investigate different Hadoop scheduling poli-
cies at Tier 1. When no minimum share of each user is
specified, Fair scheduler fairly allocates available slots
among users such that all users get an equal share of slots
over time. However, we argue that Fair unfortunately
becomes inefficient when job sizes of active users are not
uniform.
For example, we perform an experiment with two
users such that user 1 submits 30 WordCount jobs to scan
a 180 MB input file, while user 2 submits six WordCount
jobs to scan a 1.6 GB input file. All the jobs will be sub-
mitted at roughly the same time. We set the block size of
HDFS to be equal to 30 MB. Thus, the map task number
of jobs from user 2 is equal to 54 (1.6 GB/30 MB), while
each job from user 1 only has six (180 MB/30 MB) map
tasks. The reduce task number of each job is set to be
equal to its map task number. As the average task execu-
tion times of these two users are similar, we say that the
average job size (i.e., the average task number times the
average task execution time) of user 2 is about nine times
larger than that of user 1.
In the context of single-user job queues, it is well known
that giving preferential treatment to shorter jobs can reduce
the overall expected response time [11]. However, directly
using the shortest job first (SJF) policy has several draw-
backs. First, large jobs could be starved under SJF, and SJF
lacks flexibility when certain level of fairness or priority
between users is required, which is common in real use
cases. Moreover, precise job size prediction before execution
is also required for using SJF, which is not easy to achieve in
real world systems. In contrast, the sharing based schedul-
ing could easily solve the starvation problem and provide
the flexibility to integrate fairness between users by, for
example, setting up minimal shares for users. Allowing all
users to run their application concurrently also helps to
improve the job size prediction accuracy in Hadoop system
by getting information from finished tasks. Motivated by
this observation and the analysis of discriminatory proces-
sor sharing between multiple users in [12], we evaluate the
discriminatory share policies in Hadoop systems. It is
extremely hard and complex to find out an optimal share
policy under a dynamic environment where user workload
patterns may change frequently across time. Therefore, we
opted to heuristically assign slots that are reversely propor-
tional to the current average job sizes of users, and dynami-
cally tune the share over time according to the workload
pattern changes. This method introduces few overheads,
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP
TABLE 1
Average Job Response Times (in Seconds) for
Two Users with Different Job Sizes under Fair
and Two Variants
ShareRatio User 1 User 2 All
Fair 1:1 548.06 1189.33 656.61
Fair_V1 1:9 1132.33 983.16 1107.47
Fair_V2 9:1 375.56 1280.66 516.41
which is important to real implementation. We compare the
Fair policy and two variants, i.e., share slots proportional to
the average job sizes of users (Fair_V1), and reversely pro-
portional sharing policy (Fair_V2), under the two user
scenario.
Table 1 shows the average response times of jobs from
user 1, user 2, and both users under Fair and the two var-
iants, i.e., Fair_V1 and Fair_V2. We observe that Fair_V2
achieves a non-negligible improvement by assigning more
slots to user 1 who has small jobs. We therefore conclude
that when the job sizes of various users are not uniform, a good
Hadoop scheduler should adjust the slot shares among multiple
users based on their average job sizes, aiming to improve the over-
all performance in terms of job response times.
2.2 How to Schedule
Now, we look closely at the two Hadoop scheduling poli-
cies at Tier 2, i.e., allocating slots to the jobs from the same
user. As shown in [10], job execution times may vary from
seconds to hours in enterprise workloads. The average job
response times under FIFO scheduling policy thus becomes
unacceptable because small jobs are often stuck behind
large ones and thus experiencing long waiting times. On the
other hand, Fair scheduler solves this problem by equally
assigning slots to all jobs no matter what sizes those jobs
have and thus avoiding the long wait behind large jobs.
However, the average job response time of Fair scheduler
depends on the job size distribution, similar as Process Shar-
ing policy [13]: when job sizes have high variances, i.e., coef-
ficient of variation of job sizes CV > 1, Fair achieves better
performance (i.e., shorter average job response time) than
FIFO; but this performance benefit disappears when the job
sizes become close to each other, with CV  1.
To verify this observation, we conduct experiments in
our Hadoop cluster by running WordCount applications
under three different job size distributions: (1) input files
have the same size with CV 0; (2) input file sizes are
exponentially distributed with CV 1; and (3) input file
sizes are highly variable with CV 1:8. As shown in
Table 2, when input file sizes are exponentially
Fig. 1. Response times of each WordCount job under FIFO and Fair when the input file sizes have different CV .
413
TABLE 2
Average Job Response Times under FIFO and Fair
when Job Sizes Have Three Different Distributions
CV 0 CV 1 CV 1:8
FIFO 239.10 sec 208.78 sec 234.95sec
Fair 346.45 sec 220.11 sec 128.35sec
distributed, both FIFO and Fair obtain similar average job
response times; while Fair significantly reduce the aver-
age job response times under the case of high variance
but loses its superior when all files have similar sizes.
The response times of each job in the three experiments
with different job size distributions are also plotted in Fig. 1.
We observe that when the job sizes are similar, most of jobs
experience shorter response times under FIFO than under
Fair, see Fig. 1a. However, as the variation of job sizes
increases, i.e., CV > 1, the percentage of jobs which are fin-
ished more quickly under Fair increases as well, which thus
allows Fair to achieve better average job response time.
These results further confirm that the relative performance
between two scheduling policies depends on the job size
distribution. Clearly, the response time of each individual
job is mainly related to that particular jobs size under Fair
scheduling policy. On the other hand, under the FIFO pol-
icy, each jobs response time may be affected by other jobs
which were submitted earlier. FIFO allows most of the jobs
to experience faster response times when the job sizes are
similar; while most jobs are finished faster under Fair when
jobs have variable sizes. The CV value of job sizes could
then be a great threshold to determine which policy can
achieve shorter job response times under a certain work-
load. We thus argue that a good Hadoop scheduler should
dynamically adjust the scheduling algorithms at Tier 2 according
to the distribution of the job sizes.
3 ALGORITHM DESCRIPTION
Considering the dependency between map and reduce
tasks, the Hadoop scheduling can be formulated as a two-
stage multi-processor flow-shop problem. However, finding
the optimal solution with the minimum response times
(flow times) is NP-hard [14]. Therefore, in this section we
propose LsPS, an adaptive scheduling algorithm which lev-
erages the knowledge of workload characteristics to dynam-
ically adjust the scheduling schemes, aiming to improve
efficiency in terms of job response times, especially under
heavy-tailed workloads [6].
The details of our designed LsPS scheduler is pre-
sented in Algorithms 1-3. The architecture of LsPS is
414 IEEE TRANSACTIONS ON CLOUD COMPUTING,
Fig. 2. The architecture overview of LsPS.
shown in Fig. 2. Briefly, LsPS consists of the following
three components:

Workload information collection: monitor the execu-
tion of each job and each task, and gather the work-
load information.

Scheduling among multiple users: allocate (both
map and reduce) slots for users according to their
workload characteristics, i.e., scheduling at Tier 1.

Scheduling for each individual user: tune the sched-
uling schemes for jobs from each individual user
based on that users job size distribution, i.e., sched-
uling at Tier 2.
LsPS appropriately allocates slots for Hadoop users and
guides each user to select the right scheduling algorithm for
their own job queue, even under highly variable and heavy-
tailed workloads. In the remainder of this section, we
describe the detailed implementation of the above three
components. Table 3 lists some notations used in the rest of
this paper.
Algorithm 1. Overview of the LsPS
1. When a new job from user i is submitted

a. Estimate job size and avg. job size S i of
user i using Eq. (5);
b. Adjust slot shares among all active users, see
Algorithm 2;
c. Tune the job scheduling scheme for user i, see
Algorithm 3;
2. When a task of job j from user i is finished
a. Update the estimated average task execution

time ti;j ;
3. When the jth job from user i is finished
a. Measure avg. map/reduce task execution time
tm r m
i;j /ti;j and map/reduce task number ni;j /ni;j ;
r
b. Update history info. of user i, i.e., ti ; S i ; CVi , using
Eq. (1-4);
4. When a free slot is available
a. Sort users in decreasing order of deficits ASi  SUi ;
b. Assign the slot to the first user in the sorted list;
c. Increase num. of actual received slots ASi by 1;
d. Choose a job from user ui to get service based on
the current scheduling scheme.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
TABLE 3
Notations Used in the Algorithm
U / ui number of users / 0ith user, i 2 1; U.
Ji / jobi;j set of all user is jobs / jth job of user i.
tm r
i;j / ti;j average map/reduce task exe. time of jobi;j .
m r
ti / ti average map/reduce task exe. time of jobs from ui .
nm r
i;j / ni;j number of map/reduce tasks in jobi;j .
si;j size of jobi;j , i.e., total exe. time of tasks.

Si / Si average size of completed/current jobs from ui .
CVi / CVi CV of completed/current job sizes of ui .
SUi / SJi;j the slot share of ui / the slot share of jobi;j .
ASi the slot share that ui actually received.
TW time window for collecting historic informations.
W window size for collecting historic informations.
3.1 Workload Information Collection
When a Hadoop system is shared by multiple users, job sizes
and patterns of each user must be considered for designing
an efficient scheduling algorithm. Therefore, a light-
weighted history information collector is introduced in LsPS
for collecting the important historic information of jobs and
users upon each jobs completion time. Here we collect and
update the information of each jobs map and reduce tasks
separately, through the same functions. To avoid redundant
description, we use the general term task to represent both
types of tasks and the term size to represent size of either
map phase or reduce phase of each job as follows.
Algorithm 2. Tier 1: Allocate Slots for Each User
Input: historic information of each active user;
Output: slot share SUi of each active user;
for each user ui do
Update that users slot share SUi using Eq. (6);
for jth job of user i, i.e., jobi;j do
if currently scheduling based on submission times
then
if jobi;j has the earliest submission time in Ji
then
SJi;j SUi ;
else
SJi;j 0;
else
SJi;j SUi =jJi j.
Algorithm 3. Tier 2: Tune Job Scheduling for Each User
Input: historic information of each active user;
Output: UseFIFO vector;
for each user ui do
if user ui is active, i.e., jJi j > 1 then
calculate the CVi of current jobs;
if CVi < 1 and CVi < 1 then
schedule current jobs based on their submission
times;
if CVi > 1 k CVi > 1 then
equally allocate slots among current jobs;
clear history information and restart collection.
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP
In LsPS, the important history workload information that
needs to be collected for each user ui includes its average
m r
task execution time ti (and ti ), average size S i , and the coef-
ficient of variation of sizes CVi . We here adopt the Welfords
one-pass algorithm [15] to on-line update these statistics as
follows:
si;j tm m r r
i;j  ni;j ti;j  ni;j ; (1)
S i S i si;j  S i =j; (2)
vi vi si;j  S i 2  j  1=j; (3)
p 3.2 Scheduling Among Multiple Users
CVi vi =j=S i ; (4)
where si;j denotes the size of the jth completed job of user ui
(i.e., jobi;j ), tm r
i;j (resp. ti;j ) represents the measured average
map (resp. reduce) task execution time of jobi;j , nm i;j (resp.
nri;j ) means the measured map (resp. reduce) task number of
the jobi;j . A jobs size si;j is defined as the summation of the
execution times of all tasks of the job, which is independent
on the level of task concurrency during the execution, i.e.,
concurrently running multiple map (or reduce) tasks may
reduce the execution time of the job, but not the job size.
The estimation of a jobs size uses the historic task execution
times of the job based on a well accepted assumption that
the same type of tasks (either map or reduce tasks) of the
same job have similar execution times. Additionally, vi =j
denotes the variance of ui s job sizes. S i and vi are both ini-
tialized as 0 and updated each time when a new job is fin-
ished and its information is collected. The average map
m r
(resp. reduce) task execution time ti (resp. ti ) can be
updated as well with Equations (2)-(4) by replacing si;j with
tm r
i;j (resp. ti;j ).
Our solution uses the Welfords one-pass algorithm to
online update the statistics without keeping all data records.
Therefore, the data structure for each users information
only includes user ID, number of jobs that have been
recorded, average map/reduce task execution times, aver-
age and variance of job sizes, and the last update time for
detecting inactive users. The memory space needed for each
user is 48 bytes and our solution only requires a total mem-
ory space of 288 bytes when considering six users in our
experiments. This is a trivial overhead for regular MapRe-
duce clusters. To further reduce the space overhead, our
solution periodically evicts the out-of-date user records if a
user has not submitted any jobs in 30 minutes. In addition,
the average task execution time of each active job (i.e., cur-
rent running or waiting) is recorded in another data struc-
ture, e.g., JobInfo used by Fair. JobInfo is used for
storing the information such as number of running tasks for
each active job, which is created when a job is submitted
and deleted when that job is finished. Our solution extends
this data structure to keep the estimated average task execu-
tion time for each active job by adding additional 24 bytes to
JobInfo.
We use a moving window to collect and update the
workload information of each user. Let TW be a window for
415
monitoring the past scheduling history. In each monitoring
window, the system completes exactly W jobs; we set
W 100 in all the experiments presented in the paper. We
also assume that the scheduler is able to correctly measure
the information of each completed job, such as its map/
reduce execution times as well as the number of map/
reduce tasks. This assumption is reasonable for most
Hadoop systems. Upon each jobs completion, LsPS updates
the workload statistics for job owner using the above equa-
tions, i.e., Eqs. (1)-(4), see Algorithm 1 step 3. The statistic
information collected in the present monitoring window
will then be utilized by LsPS to tune the schemes for sched-
uling the following jobs.
In this section, we present our algorithm (i.e., Algorithm 2)
for scheduling among multiple users. Our goal is to decide
the deserved amount of slots and allocate appropriate num-
ber of slots for each active user to run their jobs. In a MapRe-
duce system, there are two types of slots, i.e., map slots and
reduce slots. Therefore, we have designed two algorithms,
one for allocating map slots and the other for allocating
reduce slots. However, they share the same design princi-
ples. For simplicity, we present a general form of the algo-
rithm in the rest of this section.
Basically, our solution is motivated by the observation
(see Section 2.1) that equally assigning slot shares cannot
achieve better performance than the policies that allocate
slots based on average job sizes when the job sizes of vari-
ous users are not uniform. Therefore, we propose to adap-
tively adjust the slot shares among all active users such
that the share ratio is inversely proportional to the ratio of
their job average sizes. For example, in a simple case of
two users, if their average job size ratio is equal to 1:2,
then the number of slots assigned to user 1 will be twice
that to user 2. Consequently, LsPS implicitly gives higher
priority to users with smaller jobs, resulting in shorter job
response times.
One critical issue that needs to be addressed is how to
correctly measure the execution times of map or reduce
phase of jobs that are currently running or waiting for the
service. In Hadoop systems, it is not possible to get the
exact execution times of jobs tasks before it is finished.
However, the job sizes are predictable in Hadoop system
as discussed before in this section. In this work, we esti-
mate the job sizes as task number times average task
execution time, through the following steps: (1) the num-
ber of tasks of jth job from user i (jobi;j ), i.e., ni;j , could be
obtained immediately when the job is submitted; (2) simi-
lar to [16], we assume that the execution times of tasks
from the same job are close to each other, and thus the

average task execution time, ti;j , of the finished tasks of
current running job jobi;j could be used to represent the
overall average task execution time ti;j of that job; and (3)
for the pending jobs and the running jobs that have no fin-
ished tasks, pre-configured job profiles could be used to
estimate task execution times [17]. If there are no available
job profiles, then our solution considers the historic infor-
mation of the past jobs from the same user, i.e., using the
average task execution times of recently finished jobs from
416 IEEE TRANSACTIONS ON CLOUD COMPUTING,
Fig. 3. Illustrating actual task execution times and on-line estimated
average task execution times of a Sort job with 300 map tasks and a
Grep job with 300 map tasks.
ui (ti ) to approximate the average task execution time ti;j of
jobi;j . Therefore, the average map phase size of jobs from
user ui is calculated as follows:
1 X
jJ j
i
 m
Si  nm  t ;
jJi j j1 i;j i;j
where Ji represents the set of jobs from user ui that are cur-
rently running or waiting for service. The average reduce
phase size of jobs from user ui could be calculated in the
same way. Fig. 3 illustrates an example, where the estimated
average map task execution times for both Sort and Grep
jobs quickly converge to a stable value which is consistent
with the actual one, i.e., the average task execution time of
all 300 map tasks.
As shown in Algorithm 2 step 1, once a new job arrives,
LsPS updates the average size of that jobs owner and then
adaptively adjusts the deserved map slot shares (SUi )
among all active users using Eq. (6)
 1

 of waiting and running jobs sizes is calculated based on the
S
SUi SUi  a  U  PU i 1
1a ;
i1 S 
i smaller than 1, the LsPS scheme schedules the current jobs in
8i; SUi > 0;
X
U X
U
SUi SUi ;
i1 i1
where SUi represents the estimated slot shares that should
be assigned to user ui , SUi represents the deserved slot
shares for user ui under the Fair scheme, i.e., equally dis-
patching the slots among all users, U indicates the number
of users that are currently active in the system, and a is a
tuning parameter within the range from 0 to 1. Parameter a
in Eq. (6) can be used to control how aggressively LsPS
biases towards the users with smaller jobs: when a is close
to 0, our scheduler increases the degree of fairness among
all users, performing similar as Fair; and when a is
increased to 1, LsPS gives the strong bias towards the users
with small jobs in order to improve the efficiency in terms
of job response times. In the remainder of the paper, we set
a as 1 if there is no explicit specification. When all users
have the same average job sizes, one can get SUi equal to
SUi , i.e., fairly allocating slots among users. Also, when
using Eq. (6) to calculate the SUi for each user, it is guaran-
teed that no active users gets starved for map/reduce slots,
VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
see Eq. (7), and all available slots in the system are fully dis-
tributed to active users, see Eq. (8).
The resulting deserved slot shares (i.e., SUi ) are not nec-
essarily equal to the actual assignments among users (i.e.,
ASi ). They will be used to determine which user can receive
the slot that just became available for redistribution, see
Algorithm 1 step 4. LsPS sorts all active users in a non-
increasing order of their deficits, i.e., the gap between the
expected assigned slots (SUi ) and the actual received slots
(ASi ), and then dispatchs that particular slot to the user
with the largest deficit. Additionally, it might happen in the
Hadoop system that some users have high deficits but their
actual demands on map/reduce slots are less than the
expected shares. In such a case, LsPS re-dispatches the extra
slots to those users who have lower deficits but need more
slots for serving their jobs.
3.3 Scheduling for a User
The second design principle used in LsPS is to dynamically
(5)
tune the scheduling scheme for jobs within an individual
user by leveraging the knowledge of job size distribution.
As observed in Section 2.2, the scheme of equally distribut-
ing shared resources outperforms by avoiding small jobs to
waiting behind large ones. However, when the jobs have
similar sizes, scheduling jobs based on their submission
times becomes superior to the former one.
Therefore, our algorithm (as described in Algorithm 3)
considers the CV of job sizes, i.e., map size plus reduce size,
of each user to determine which scheme should be used to
distribute the free slots to jobs from that user. To improve the
accuracy, we combine the history job size information and
the estimated size distribution of running and waiting jobs
in system. CVi of currently finished jobs sizes of user i is pro-
vided directly by the history information collector, and CVi
(6) estimated job sizes. When the two values of a user are both
that users sub queue in the order of their submission times,
(7)
otherwise the user level scheduler will fairly assign slots
among jobs. When the two values are conflicting, i.e.,
(8) CVi > 1 and CVi < 1 or vise versa, which means the users
workload pattern may change, the fair scheme will be
adopted at this time, the history information will be cleared
and a new collection window will start at this time, see the
pseudo-code in Algorithm 3.
4 MODEL DESCRIPTION
In this section, we introduce a queuing model that is devel-
oped to emulate a Hadoop system. The main purpose of
this model is to compare various Hadoop scheduling
schemes, and give the first proof of our new approach. This
model does not include all the details of the complex
Hadoop system, but provide a general guideline to users
which is useful for performance evaluation without running
experiment tests. In Section 5, besides experiments on a real
Hadoop clusters, we also conduct trace-driven simulations
based on this model to evaluate the performance improve-
ment of LsPS.
The model, as shown in Fig. 4 consists of two queues for
map tasks (Qm ) and reduce tasks (Qr ), respectively. Once a
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP
Fig. 4. Design of the Hadoop MapReduce cluster simulator.
job is submitted, its tasks will be inserted into Qm (resp. Qr )
through the map (resp. reduce) task dispatcher. Furthermore,
the model includes s servers to represent s available slots in
the system, such that sm servers are used to serve map tasks
while the remaining servers, i.e., sr s  sm , connect to the
reduce queue for executing reduce tasks. Note that the values
of fsm ; sr g are based on the actual Hadoop configuration.
An important feature of MapReduce jobs need to be con-
sidered in the model is the dependency between map and
reduce tasks. Typically, in a Hadoop cluster, there is a
parameter which decides when a job can start its reduce
tasks. By default, this parameter is set as 5 percent, which
indicates that the first reduce task can be launched when
5 percent of the map tasks are committed. Under this set-
ting, a jobs first wave of reduce tasks will overlap with its
map phase and could prefetch the output of map tasks in
the overlapping period. However, previous work [18] found
that this setting would lead to performance degradation
under the Fair scheduling policy and proposed to launch
reduce tasks gradually according to the progress of map
phase. We further found that delaying the launch time of
reduce tasks, i.e., setting the parameter to a large value such
as 100 percent, can improve the performance of the Fair and
the other slots sharing based schedulers. Therefore, in our
experiments, we set the parameter to 100 percent, i.e., run-
ning reduce tasks when all map tasks are completed, in all
the three policies (i.e., FIFO, Fair and our LsPS). However,
this is not a necessary assumption in the model. Our sched-
uler works in the same way under the other two cases, i.e.,
launching the first reduce task when 5 percent of the map
tasks are committed or launching the reduce tasks gradually
according to the progress of map phase.
Finally, compared to the complex architecture of a Map-
Reduce system, the simulator built based on this model is a
much-simplified tool. Our goal is to use a queuing model to
compare different scheduling policies quickly and effec-
tively. The key point of the simulation model is to capture
the impact of different scheduling policies on the job
response time under different situations. Therefore, the
model we designed mainly simulates this key feature, i.e.,
how to share slots, without capturing the low-level details
of a MapReduce system, such as communication costs,
locality of data, and fault-tolerant mechanism.
5 EXPERIMENTAL RESULTS
In this section, we turn to present the performance evalua-
tion of the proposed LsPS scheduler, which aims to improve
417
the efficiency of a Hadoop system, especially with highly
variable and/or bursty workloads from different users.
5.1 Simulation Evaluation
We first evaluate LsPS with our simulation model which is
developed to emulate a classic Hadoop system. On the top
of this model, we use trace-driven simulations to evaluate
the performance improvement of LsPS in terms of average
job response times. Later, we will verify the performance of
LsPS by implementing the proposed policy as a plug-in
scheduler in an EC2 Hadoop cluster.
In our simulations, we configure the number of both map
and reduce slots in cluster to be equal to 300 in the simula-
tion model. We also have U users fu1 ; . . . ; ui ; . . . ; uU g to
share the Hadoop cluster by submitting Ji jobs to the sys-
tem. The specification of each user includes its job inter-
arrival times and its job sizes, which are created based on
the specified distributions and methods. Recall that each
Hadoop job size is determined by the number of map (resp.
reduce) tasks from that job as well as the execution time of
each map (resp. reduce) task. In our model, we consider to
change the distributions of map/reduce task numbers for
investigating various job size patterns, while fixing the uni-
form distribution to draw the execution times of map/
reduce tasks.
In general, we consider the following four different dis-
tributions to generate job inter-arrival times and job map/
reduce task numbers.

Uniform distribution (u), which indicates similar job
sizes.

Exponential distribution (e), which implies medium
diversity of job sizes or inter-arrival times.

Hyperexponential distribution (h), which means
high variance of traces.

Bursty pattern (b), which indicates high variance and
high auto-correlation of traces.
Here, we use a hyperexponential distribution to approxi-
mate a heavy-tailed distribution [19], [20], which has been
often observed in MapReduce clusters [6].
We first consider two simple cases where the cluster is
shared by two users. We evaluate the impacts of different
job size patterns in case 1 and job arrival patterns in case 2.
We then validate the robustness of LsPS with a complex
case where the cluster is shared by multiple users with dif-
ferent job size and arrival patterns.
5.1.1 Simple Case 1-Two Users with Diverse Job Size
Patterns
Consider a simple case of two users, i.e., u1 and u2 , that
concurrently submit Hadoop jobs to the system. We first
focus on evaluating different Hadoop schedulers under
various job size patterns, i.e., we conduct experiments with
different job size distributions for u2 , but always keeping
the uniform distribution to generate job sizes for u1 . Specif-
ically, we consider u2 with (1) similar job sizes; (2) high
variability in job sizes; and (3) high variability and strong
temporal dependence in job sizes. We also set the job size
ratio between u1 and u2 as 1:1, i.e., two users having the
same average job sizes. Furthermore, both users have the
418
Fig. 5. Average job response times of (a) two users, (b) user 1, and (c) user 2 under three different scheduling policies and different job size distribu-
tion settings. The relative improvement with respect to Fair is also plotted on each bar of LsPS.
exponentially distributed job interarrival times with the
same mean of 300 seconds.
Fig. 5 shows the mean job response times of both
users under different policies and the relative improve-
ment with respect to Fair. Job response time is measured
from the moment when that particular job is submitted
to the moment that all the associated map and reduce
tasks are finished. We first observe that high variability
in job sizes dramatically degrades the performance
under FIFO as a large number of small jobs are stuck
behind the extremely large ones, see plot (a) in Fig. 5. In
contrast, both Fair and LsPS effectively mitigate such
negative performance effects by equally distributing
available slots between two users and within a single
user. Our policy further improves the overall perfor-
mance by shifting the scheduler to FIFO for the jobs
from u1 and thus significantly reducing its mean job
response time by 60 and 62 percent with respect to Fair
when the job sizes of user 2 are highly variable (i.e., u:
h) and temporally dependent (i.e., u:b), respectively,
see plot (b) in Fig. 5. On the other hand, Fair loses its
superiority when both users have similar job sizes, while
our new scheduler bases on the features of both users
job sizes to tune the scheduling at two tiers and thus
achieves the performance close to the best one.
To further investigate the tail of job response times,
we plot in Fig. 6 the complementary cumulative distribu-
tion functions (CCDFs) of job response times, i.e., the
probability that the response times experienced by indi-
vidual jobs are greater than the value on the horizontal
axis, for both users under the three scheduling policies.
Consistently, almost all jobs from the two users experi-
ence shorter response times under Fair and LsPS than
under FIFO when job sizes of u2 are highly variable. In
addition, compared to Fair, LsPS reduces the response
times for more that 60 percent of jobs, having shorter
tails in job response times.
Fig. 6. CCDFs of response times of all jobs under different schedulers, where user 1 has uniform job size distribution while user 2 have (a) similar job
sizes; (b) high variability in job sizes; and (c) high variability and strong temporal dependence in job sizes.
IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
In order to analyze the impacts of relative job sizes on
LsPS performance, we conduct another two sets of experi-
ments with various job size ratios between two users, i.e.,
we keep the same parameters as the previous experiments
but tune the job sizes of u2 such that we have the average
job size of u1 is 10 times less (resp. more) than that of u2 , see
the results shown in Fig. 7(I) (resp. Fig. 7(II)). In addition,
we tune the job arrival rates of u2 to keep the same loads in
the system. Recall that our LsPS scheduler always gives
higher priority, i.e., assigning more slots, to the user with
smaller average job size, see Section 3.2. As a result, LsPS
achieves non-negligible improvements of overall job
response times no matter which user has smaller job sizes,
see plots (a) in Fig. 7(I) and (II). Further confirmation of this
benefit comes from the plots in Figs. 8(I) and 8(II), which
show that most jobs experience the shortest response times
when the scheduling is LsPS. Indeed, the part of the work-
load whose job sizes are large receives increased response
times, but the number of penalized jobs is less than 5 per-
cent of the total.
Now, look closely at each users response times. We
observe that under the cases of two different job size ratios,
LsPS always achieves significant improvement in job
response times for the user which submits small jobs in
average by assigning more slots to that user, see plot (b) in
Fig. 7(I) and plot (c) in Fig. 7(II). Meanwhile, although LsPS
discriminately treats another user (i.e., having larger jobs)
with less resource, this policy does not always sacrifice that
users performance. For example, as shown in plot (c) of
Fig. 7(I), when job sizes of u2 are highly variable and/or
strongly dependent, shorter response times are achieved
under LsPS or Fair than under FIFO because small jobs now
have the reserved slots without waiting behind the large
ones. Another example can be found in plot (b) of Fig. 7(II),
where we observe that LsPS is superior to Fair on the perfor-
mance of user 1 by switching the tier 2 scheduling algorithm
to FIFO.
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP 419

Fig. 7. Average job response times of (a) two users, (b) user 1, and (c) user 2 under three different scheduling policies. The relative job size ratio
between two users is (I) 1:10, and (II) 10:1.

5.1.2 Simple Case 2-Two Users with Diverse Job
Arrival Patterns
We now turn to consider the changes in job arrival patterns.
We conduct experiments with varying arrival processes of
the second user, i.e., u2 , but always fixing the uniform job
size distributions for both users as well as the relative job
size ratio between them as 1:10. Therefore, the job interar-
rival times of u2 are drawn from three different arrival pat-
terns, i.e., exponential, hyper-exponential and bursty, while
user 1s job interarrival times are exponentially distributed
in all the experiments. We then depict the average job
response times of two users in Fig. 9 and the CCDFs of job
response times in Fig. 10.
Consistent to the previous experiments, our LsPS sched-
uler outperforms in terms of the overall job response times,
see plot (a) in Fig. 9. We observe that this benefit indeed
comes from the response time improvement of u1 , i.e., LsPS
assigns u1 with more slot shares due to its smaller average
job size and further schedules its jobs based on the FIFO dis-
cipline because its job sizes have low variability. However,
comparing to FIFO, this outcome unfortunately penalizes
user 2, especially when this users arrival process is hyper-
exponential or bursty, see plot (c) in Fig. 9. Meanwhile, due
to the uniform job size distribution, LsPS schedules the jobs
from u2 in the order of their submission times, which indeed
compensates for the less resources and thus reduces the
average response time when compared to Fair. The CCDFs

Fig. 8. CCDFs of response times of all jobs under three different scheduling policies. The relative job size ratio between two users is (I) 1:10, and (II)
10:1.
420 IEEE TRANSACTIONS ON CLOUD COMPUTING,
Fig. 9. Average job response times of (a) two users, (b) user 1, and (c) user 2 under different scheduling policies and different job interarrival time dis-
tributions. The relative job size ratio of two users is 1:10.
shown in Fig. 10 further confirm that a large portion of jobs
experiences shorter response times under LsPS than under
the other two policies.
5.1.3 Complex Case- Multiple Users with Diverse Job
Arrival/Size Patterns
To further verify the robustness of LsPS, we conduct experi-
ments under a more complex case of 6 users which have the
mixed workloads of varying job arrival and job size pat-
terns. Table 4 presents the detailed experimental settings.
Here, users with larger IDs have relatively larger job sizes
in average. We also adjust the average arrival rate of each
user such that all the users submit the same load to the sys-
tem. Table 5 and Fig. 11 present the average job response
times as well as the distributions of job response times of all
users under different scheduling policies. We compare our
proposed LsPS scheduler with the commonly adopted ones,
i.e., FIFO, Fair, and Capacity schedulers. For the Capacity
scheduler, we consider three different capacity sharing con-
figurations, i.e., equal capacity shares in Capacity_v1, capac-
ity shares proportional to each users average job sizes in
Capacity_v2, and capacity shares reversely proportional to
each users average job sizes in Capcaity_v3. Table 5 shows
the average job response times of each individual user and
of all six users as well. Furthermore, in order to analyze the
impact of parameter a in Eq. (6), Table 5 also shows the sim-
ulation results under LsPS with a equal to 0:3, 0.6 and 1.0.
We first observe that LsPS with different a significantly
improves the overall job response times compared to
FIFO, Fair and Capacity. Meanwhile, the average response
times of the first four users are improved as well under
LsPS because those users have relatively smaller job sizes
and thus receive more map/reduce slots for executing
their jobs. On the other hand, although the last two users
u5 and u6 are assigned with the least number of slots, their
average job response times have not been dramatically
increased. In contrast, their jobs even experience faster
response times compared to Fair for u5 , and FIFO for u6 .
Fig. 10. CCDFs of job response times under different schedulers, where job interarrival times of user 1 are exponentially distributed and user 2s
arrival process is (a) exponential; (b) hyper-exponential; and (c) bursty. The relative job size ratio of two users is 1:10.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
The main reason is because LsPS switches the scheduling
for each user between FIFO and Fair based on their job size
distributions and thus improves that particular users
response times. Additionally, LsPS completes the jobs
from the first four users within a short time period such
that the occupied slots will be released soon and then reas-
signed to the last two users, which further decreases the
job response times of these two users. We also observe that
different sharing configurations have significant impacts
on Capacitys performance. However, even under Capaci-
ty_v3, the average job response time is still two times
slower than LsPS when a is set to be 1.0.
Recall that parameter a in Eq. (6) is a tuning parameter to
control how aggressively LsPS discriminates large jobs from
small ones. The larger the a is, the stronger bias is given
towards the users with small jobs. Table 5 shows that LsPS
with a 1:0 introduces the strongest bias on user slot
shares and achieves the best response time improvement.
Therefore, we set a 1:0 in the remainder of this paper.
5.2 Case Studies in Amazon EC2
To further verify the effectiveness and robustness of our
new scheduler, we implement and evaluate LsPS in Ama-
zon EC2, a cloud platform that provides pools of computing
resources to developers for flexibly configuring and scaling
their computational capacity on demand.
5.2.1 Experimental Settings
In particular, we lease a m1.large instance as master node,
which provides two virtual cores with two EC2 Compute
Units each, 7.5 GB memory, and 850 GB storage to per-
form heartbeat and jobtracker routines for job scheduling.
We also use the same 11 m1.large instances to launch
slave nodes, each of which is configured with two map
slots and two reduce slots. Such a configuration ensures
that the system bottleneck is not our scheduler on master
node, while the overall job response times depend on the
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP 421

TABLE 4
Experimental Settings for Each User

User Job Size Pattern Arrival Pattern Avg. Size Ratio

1 Bursty Hyper-exp. 1
2 Exponential Exponential 5
3 Uniform Exponential 10
4 Hyper-exp. Exponential 20
5 Uniform Bursty 50
6 Hyper-exp. Hyper-exp. 100
scheduling algorithms as well as the processing capability
of each slave node.
As the Hadoop project [2] provides an API to support
pluggable schedulers, we implement our proposed LsPS
scheduling policy in Amazon Hadoop by extending the
TaskScheduler interface. In particular, we add a module in
our scheduler to periodically predict the job sizes of users
based on the execution times of finished tasks (which are
recorded for logging purpose in original Hadoop imple-
mentation). We also integrate another module to calculate
the slot share between users upon the submission of new
jobs and assign tasks of different users according to the defi-
ciency between their running tasks and deserved slot
assignments.
The benchmarks we consider for performance evaluation
includes the following four classic MapReduce applications.

WordCounttaking text files as inputs and comput-
ing the occurrence frequency of each existing word;

PiEstimatorestimating the value of p using
quasi-Monte Carlo method;

Grepextracting and counting the strings in input
files that match the given regular expression;

Sorttaking sequential files as inputs and frag-
menting and sorting the input data.
In addition, the randomtextwriter program is used to
generate a random file as the input to WordCount and Grep
applications. We also run the RandomWriter application to
write 10 G of random data, that is used as the input to Sort
applications. For PiEstimator applications, we set the sample
space at each map task as 100 million random points. We
have 20 map tasks for each PiEstimator job, thus the total
number of random points for one p estimation is 2 billion.
5.2.2 Workloads with Mixed Applications
We conduct experiments with the mixed MapReduce appli-
cations, aiming to evaluate LsPS in a diverse environment
Fig. 11. CCDFs of job response times of the complex case where six
users having the mixed workloads.
of both CPU-bound applications, such as PiEstimator, and
IO-bound applications, e.g., WordCount and Grep Applica-
tions. In this experiment, there are four users which submit
a set of jobs for one of the above four MapReduce applica-
tions, according to the specified job size and arrival pat-
terns, see Table 6.
The experimental results of overall and each users aver-
age job response times are shown in Table 7. We first
observe that LsPS reduces the overall job response times by
a factor of 3.5 and 1.8 over FIFO and Fair, respectively. We
interpret it as an outcome of setting suitable scheduling
algorithms for each user based on their corresponding
workload features. More importantly, LsPS significantly
reduces the average response times for the first three users
and even slightly improves the average response time of
user 4 which indeed is assigned with less resources due to
its large job size. The CCDFs of job response times are
depicted in Fig. 12. A large fraction of jobs experience faster
response times under LsPS than under FIFO and Fair, while
the number of penalized jobs which receive increased
response times is less than 3 percent of the total. Summariz-
ing, the real experimental results are consistent with the
results shown in our simulations (see Section 5.1), which
further confirm the effectiveness and robustness of our new
LsPS scheduler.
5.2.3 Non-Stationary Workloads
In the previous sections, we have confirmed that LsPS per-
forms effectively under stationary workloads, where all
users have the stable job size/arrival patterns. Now, we
turn to evaluate LsPS under non-stationary workloads, fur-
ther verifying its effectiveness when the workloads of some
users change over time.
In particular, we conduct experiments with two users
which submit a set of WordCount jobs to our Hadoop
cluster consisting of 18 map and 18 reduce slots on m1.

TABLE 5
Average Response Times (in Seconds) of All Users and Each User under Different Scheduling Policies

User FIFO Fair Capacity_v1 Capacity_v2 Capacity_v3 LsPS LsPS LsPS

a 0:3 a 0:6 a 1:0
1 7357.60 211.43 606.52 8688.95 193.64 163.00 150.05 142.51
2 11520.03 283.43 426.94 9648.50 242.47 234.64 222.00 220.20
3 10822.45 475.00 376.19 4992.06 357.96 276.18 258.88 254.41
4 10626.55 1182.14 2083.75 2532.49 2372.63 742.41 734.50 647.84
5 12017.48 40677.55 30441.12 22275.19 29416.86 22637.01 17557.73 13317.92
6 11346.46 3318.84 14598.88 3415.35 14997.93 3194.37 4760.66 5587.35
All 8488.00 939.24 1211.70 8513.58 886.27 583.05 505.94 441.66
422 IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 3, NO. 4, OCTOBER-DECEMBER 2015

TABLE 6
Case Study: Experimental Settings for Four Users in Amazon EC2

User Job Average Input Input Size Job Arrival Average Inter-arrival Submission
Type Size Pattern Pattern Time Number
1 WordCount 100 MB Exponential Bursty 20 sec 150
2 PiEstimator - - Uniform 30 sec 100
3 Grep 2000 MB Bursty Exponential 100 sec 30
4 Sort 10 GB Uniform Exponential 600 sec 5

large instances (i.e., slave nodes) in Amazon EC2. We
further generate a non-stationary workload by changing
the job size/arrival patterns of user 1 while keeping
fixed user 2s input file sizes and job interarrival times
both exponentially distributed with mean of 500 MB and
25 seconds, respectively. Table 8 illustrates the three
changes in user 1s workloads.
Table 9 shows the mean response times of two users as
well as of each user under the Fair and LsPS policies. The
average job response times measured during each period
are also shown in the table. We observe that LsPS success-
fully captures the changes in user 1s workloads and
dynamically tunes the two-level scheduling (i.e., between
two users and within each user) based on the measured job
size/arrival patterns. As a result, LsPS always achieves
noticeable response time improvement by 26, 24, and 40
percent during the three periods and by 34 percent in over-
all with respect to Fair.
To better understand LsPSs processing on non-station-
ary workloads, Fig. 13 illustrates how LsPS dynamically
adjusts its two-level scheduling algorithms in an on-line
fashion. Specifically, the transient distributions of 18 map
slots between two users are depicted in Fig. 13a, where red
areas indicate the slots assigned to user 1 while greed areas
represent those assigned to user 2. We also plot the changes
of the scheduling within user 1 as a function of time in
Fig. 13b.
As we observed, during the first period, LsPS assigns
more shares to user 1 than to user 2 because LsPS detects
that user 1 has smaller average job size, see Fig. 13a. Mean-
while, the jobs from user 1 are scheduled according to the
FIFO discipline, see Fig. 13b, which further reduces the
response times of user 1 and thus results in better overall
response times during this period. Once LsPS captures the
changes in user 1s job size distribution, i.e., from uniform
to hyperexponential, LsPS quickly switches the scheduling
within user 1 from FIFO to Fair and thus consistently
achieves shorter response times in the second period, see
Fig. 13b. Later, when user 1 starts to submit large jobs with
the uniform distribution, LsPS turns to dispatch more
resources to user 2, decreasing its job response times during
the last period. On the other hand, user 1 still experiences
shorter job response times than under Fair even though this
user now receives less resources. We interpret this by
observing the FIFO scheduling for this user in the third
period. Moreover, the long delay existing in the shifting
from period 2 to 3 indeed only affects few number of jobs
because the job interval time actually becomes quite long
during period 3.
Fig. 13c shows the number of jobs that are running or
waiting for service under Fair and LsPS policies, giving
an evidence that LsPS can consistently improve the aver-
age job response times through dynamically adapting the
scheduling to the workload changes. Therefore, we con-
clude that these results strongly demonstrate the effec-
tiveness and robustness under both stationary and non-
stationary workloads.
6 RELATED WORKS
Scheduling in Hadoop system has already received lots of
attention. An early work of Matei Zaharia et al. [7] proposed
the Fair scheduler which assigns resources equally to each
user and provides performance isolation between users.
However, its objective is not to optimize the system perfor-
mance. In [21], a delay scheduler was proposed to improve
the performance of Fair scheduler by increasing data local-
ity. It simply delays the task assignment for a while if the
tasks data is not local. This improvement is at task level,
and can be combined with our proposed job scheduling pol-
icy. The quincy scheduler [22] considered similar direction
and found a fair assignment while considering locality by
formulating and solving a minimum flow network problem.
However, it is limited due to its high computation complex-
ity. In [23], Sandholm and Lai considered the profit of the
service provider and proposed a scheduler that splits slots
to users according to the bids they pay instead of fair share.
The efficiency of scheduler is not considered in their work.

TABLE 7
Average Response Times (in Seconds) of All Users
and Each User in the Amazon Hadoop Cluster

User FIFO Fair LsPS

1 251.36 121.18 67.50
2 280.06 149.95 74.79
3 235.33 118.36 75.18
4 330.20 248.00 209.00
All 259.61 132.90 74.16 Fig. 12. CCDFs of job response times when four different MapReduce
applications are running on Amazon EC2 cluster.
YAO ET AL.: LSPS: A JOB SIZE-BASED SCHEDULER FOR EFFICIENT TASK ASSIGNMENTS IN HADOOP 423

TABLE 8
Experimental Settings for User 1s Non-Stationary Workload

Periods Avg. Input Size Pattern Interarrival Submission

Size Time Number
1 100 MB Uniform 5 sec 50
2 100 MB Hyper-Exp. 5 sec 50
3 2500 MB Uniform 125 sec 10

Wolf et al. proposed a slot allocation scheduler called FLEX

[24] that could optimize towards a given scheduling metric,
e.g., average response time, makespan, etc., by sorting jobs
with generic shemes before allocating slots. Their scheme
are used under the batch job situation, i.e., all jobs are sub-
mitted at the same time for scheduling, while our schedul-
ing policy works well under the on-line situation. Verma et
al. [25] and Tian et al. [5] focus on improving the makespan
of a batch of jobs that submitted at the same time instead of
average job response times in multiple user situation.
Another major direction of improving the Hadoop sched-
uling policy is considering the deadline or SLA of jobs. A
Fig. 13. Illustrating (a) the slot shares between users, where the grey
deadline based scheduler was proposed in [17], which uti- (resp. dark) areas indicate the slots assigned to user 1 (resp. user 2); (b)
lizes earliest deadline first policy to sort jobs and the the scheduling scheme of user 1 across time, where 1 indicates FIFO
lagrange optimization method to find out the jobs minimum and 0 indicates Fair; and (c) the transient number of jobs that are run-
map and reduce slots number requirements to meet their ning or waiting for service.
deadline. This solution required a detailed profile for each
unnecessary speculative executions. Dhok and Varma [28]
job to provide the execution time of map task and reduce
applied the Bayes pattern classification method to classify
task of that job. Polo et al. [16] estimated the task execution
and sort jobs and dynamically decided the slots numbers a
time of a job from the average execution time of already fin-
node can provide according to its load limitation instead of
ished tasks of that job, and calculate the slots number a job
assigning fixed number of slots to each node. Ghodsi
needs from its deadline and estimated task execution time.
et al. [29] considered the difference of resources require-
We partly adopt the method to help estimate the job size of
ments of Hadoop jobs and proposed a dominant resource
each user in our proposed LsPS scheduler.
fairness policy to assign resources fairly to jobs instead of
Some other works focus on theoretical understanding of
assign slots to jobs. Since homogeneous systems are more
scheduling problem in Hadoop system and try to give
commonly used in Hadoop implementation, we do not con-
approximate scheduling algorithms [14], [26]. Chang et al.
sider heterogeneous system in our work.
[26] proposed a 2-approximate scheduling algorithm. How-
ever, their assumption that there are no precedence relation-
ships between tasks belonging to a given job is not true in 7 CONCLUSION
Hadoop system. Moseley et al. [14] formalized job schedul- In this paper, we have proposed LsPS, an adaptive schedul-
ing problem in MapReduce system as a generalization of ing technique for improving the efficiency of Hadoop sys-
the two-stage flexible flow-shop problem and gave the off- tems that process heterogeneous MapReduce jobs.
line and on-line version of approximate algorithm. How- MapReduce workloads of contemporary enterprise clients
ever, their work assumes that the execution time of tasks have revealed the diversity of job sizes, ranging from sec-
are known. onds to hours and having varying distributions as well. Our
Furthermore, some prior work utilized the heteroge- new online policy captures the present job size patterns of
neousness of system and jobs resource requirements to each user and leverages this knowledge to dynamically
improve the performance of scheduler. Zaharia et al. [27] adjust the slot shares among all active users and to further
indicated that the Hadoop speculative execution will lead on-the-fly tune the scheme for scheduling jobs within a sin-
to poor performance in heterogeneous Hadoop environ- gle user. Using real experiments in Amazon EC2, we have
ments. They thus proposed a LATE scheduler which stops shown that LsPS consistently improves the performance in

TABLE 9
Average Job Response Times (in Seconds) of Fair and LsPS under Non-Stationary Workloads

User Period 1 Period 2 Period 3 All

Fair LsPS Fair LsPS Fair LsPS Fair LsPS
1 108.96 50.30 100.72 60.78 803.00 771.10 168.30 120.59
2 277.90 354.50 326.70 329.60 357.52 158.40 341.74 210.87
All 136.32 101.00 138.38 105.58 431.33 260.51 235.75 155.70
424 IEEE TRANSACTIONS ON CLOUD COMPUTING,
terms of job response times under a variety of system work-
loads. We have also shown that the effectiveness and
robustness of LsPS under the nonstationary workloads.
ACKNOWLEDGMENTS
This work was partially supported by the National Science
Foundation grant CNS-1251129, and the AWS in Education
Research Grant.
REFERENCES
[1] J. Dean, S. Ghemawat, and G. Inc., Mapreduce: Simplified data
processing on large clusters, in Proc. 6th Conf. Symp. Operating
Syst. Des. Implementation, 2004, p.10.
[2] Apache Hadoop. (2014). [Online]. Available: http://hadoop.
apache.org/
[3] Apache Hadoop Users. (2014). [Online]. Available: http://wiki.
apache.org/hadoop/PoweredBy
[4] Y. Chen, A. Ganapathi, R. Griffith, and R. Katz, The case for eval-
uating mapreduce performance using workload suites, in Proc.
IEEE 19th Int. Symp. Model., Anal. Simul. Comput. Telecommun.
Syst., 2011, pp. 390399.
[5] C. Tian, H. Zhou, Y. He, and L. Zha, A dynamic mapreduce
scheduler for heterogeneous workloads, in Proc. 8th Int. Conf.
Grid Cooperative Comput., 2009, pp. 218224.
[6] S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan, An analysis of
traces from a production mapreduce cluster, in Proc. 10th IEEE/
ACM Int. Conf. Cluster, Cloud Grid Comput., 2010, pp. 94103.
[7] M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker,
and I. Stoica, Job scheduling for multi-user mapreduce clusters,
University of California, Berkeley, CA, USA, Tech. Rep. UCB/
EECS-2009-55 , Apr. 2009.
[8] Fair scheduler [Online]. Available: http://hadoop.apache.org/
common/docs/r1.0.4/fair_scheduler.html, 2014.
[9] A. Thusoo, Z. Shao, S. Anthony, D. Borthakur, N. Jain, J. Sen
Sarma, R. Murthy, and H. Liu, Data warehousing and analytics
infrastructure at facebook, in Proc. ACM SIGMOD Int. Conf. Man-
age. Data, 2010, pp. 10131020.
[10] Y. Chen, A. S. Ganapathi, R. Griffith, and R. H. Katz, A method-
ology for understanding mapreduce performance under diverse
workloads, Univ. California, Berkeley, CA, USA, Tech. Rep.
UCB/EECS-2010-135 , 2010.
[11] L. E. Schrage and L. W. Miller, The queue m/g/1 with the short-
est remaining processing time discipline, Operations Research,
vol. 14, no. 4, pp. pp. 670684, 1966.
[12] K. Avrachenkov, U. Ayesta, P. Brown, and R. N unez-Queija,
Discriminatory processor sharing revisited, in Proc. IEEE
INFOCOM, Mar. 2005, pp. 784795.
[13] I. Mitrani, Probabilistic Modelling. Cambridge, U.K.: Cambridge
Univ. Press, 1998.
[14] B. Moseley, A. Dasgupta, R. Kumar, and T. Sarl os, On scheduling
in map-reduce and flow-shops, in Proc. 23rd Annu. ACM Symp.
Parallelism Algorithms Archit., 2011, pp. 289298.
[15] B. P. Welford, Note on a method for calculating corrected sums
of squares and products, Technometrics, vol. 4, pp. 419420, 1962.
[16] J. Polo, D. Carrera, Y. Becerra, J. Torres, E. Ayguade, M. Steinder,
and I. Whalley, Performance-driven task co-scheduling for map-
reduce environments, in Proc. IEEE Netw. Operations Manage.
Symp., 2010, pp. 373380.
[17] A. Verma, L. Cherkasova, and R. H. Campbell, Aria: Automatic
resource inference and allocation for mapreduce environments,
in Proc. 8th ACM Int. Conf. Autonomic Comput., 2011, pp. 235244.
[18] J. Tan, X. Meng, and L. Zhang, Performance analysis of coupling
scheduler for mapreduce/hadoop, In INFOCOM, 2012 Proceed-
ings IEEE, pp. 25862590, Mar. 2012.
[19] A. Feldmann and W. Whitt, Fitting mixtures of exponentials to
long-tail distributions to analyze network performance models,
in Proc. IEEE INFOCOM, 1997, vol. 3, pp. 10961104.
[20] A. Riska, V. Diev, and E. Smirni, Efficient fitting of long-tailed
data sets into hyperexponential distributions, in Proc. IEEE
GLOBECOM, 2002, pp. 25132517.
[21] M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker,
and I. Stoica, Delay scheduling: A simple technique for achieving
locality and fairness in cluster scheduling, in Proc. 5th Eur. Conf.
Comput. Syst., 2010, pp. 265278.
VOL. 3, NO. 4, OCTOBER-DECEMBER 2015
[22] M. Isard, Vijayan Prabhakaran, J. Currey, U. Wieder, K. Talwar,
and A. Goldberg, Quincy: Fair scheduling for distributed comput-
ing clusters, in Proc. 5th Eur. Conf. Comput. Syst., 2009, pp. 261276.
[23] Thomas Sandholm and K. Lai, Mapreduce optimization using
regulated dynamic prioritization, in Proc. SIGMETRICS: 11th Int.
Joint Conf. Meas. Model. Comput. Syst., 2009, pp. 299310.
[24] J. Wolf, D. Rajan, K. Hildrum, R. Khandekar, V. Kumar, S. Parekh,
K.-L. Wu, and A. Balmin, Flex: A slot allocation scheduling opti-
mizer for mapreduce workloads, in Proc. ACM/IFIP/USENIX
11th Int. Conf. Middleware, 2010, pp. 120.
[25] A. Verma, L. Cherkasova, and R. H. Campbell, Two sides of a
coin: Optimizing the schedule of mapreduce jobs to minimize
their makespan and improve cluster performance, in Proc. IEEE
20th Int. Symp. Model., Anal. Simul. Comput. Telecommun. Syst.,
2012, pp. 1118.
[26] H. Chang, M. Kodialam, R. R. Kompella, T. V. Lakshman, M. Lee,
and S. Mukherjee, Scheduling in mapreduce-like systems for fast
completion time, in Proc. IEEE INFOCOM, 2011, pp. 30743082.
[27] M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and
I. Stoica, Improving mapreduce performance in heterogeneous
environments, in Proc. 8th USENIX Conf. Operating Syst. Des.
Implementation, 2008, pp. 2942.
[28] J. Dhok and V. Varma, Using pattern classification for task
assignment in mapreduce, in Proc. ISEC, 2010.
[29] A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker,
and I. Stoica, Dominant resource fairness: Fair allocation of mul-
tiple resource types, in Proc. 8th USENIX Conf. Netw. Syst. Des.
Implementation, Mar. 2011, p. 24.
Yi Yao received the BS and MS degrees in com-
puter science from the Southeast University,
China, in 2007 and 2010. He is working toward
the PhD degree at the Department of Electrical
and Computer Engineering, Northeastern Univer-
sity, Boston, MA. His current research interests
include resource management, scheduling, and
cloud computing.
Jianzhe Tai received the BS and MS degrees in
electrical and information engineering from the
Dalian University of Technology, China, in 2009
and 2007. He is working toward the PhD degree
at the Department of Electrical and Computer
Engineering, Northeastern University, Boston,
MA. His current research interests include soft-
ware-defined storage systems and virtualization
in hybrid storage.
Bo Sheng received the PhD degree in computer
science from the College of William and Mary in
2010. He is an assistant professor at the Depart-
ment of Computer Science, University of Massa-
chusetts Boston. His research interests include
mobile computing, wireless networks, security,
and cloud computing.
Ningfang Mi received the BS degree in computer
science from Nanjing University, China, in 2000
and the MS degree in computer science from the
University of Texas at Dallas, TX in 2004. She
received the PhD degree in computer science
from the College of William and Mary, VA in
2009. She is an assistant professor at the Depart-
ment of Electrical and Computer Engineering,
Northeastern University, Boston. Her current
research interests include performance evalua-
tion, capacity planning, resource management,
simulation, data center, and cloud computing. She is a member of the
ACM and the IEEE.