Experimental Study of Parallel Downloading Schemes
for Internet Mirror Sites
Spiro Philopoulos
Dept. of Elect. and Comp. Eng.
University of Manitoba
Winnipeg, Canada
email: umphilop@cc.umanitoba.ca
ABSTRACT
A common method used to reduce document retrieval
times is the use of content replication i.e., mirror servers.
The mirror servers provide several alternate sites to down-
load a specific document and were traditionally used to in-
crease the availability of content. Recently, several stud-
ies focused on using multiple mirror sites to concurrently
download portions of a document from a set of mirror sites.
Following are some of the issues involved in using multiple
mirror sites concurrently: (a) selection of the “best” mir-
ror servers from the client, (b) coping with dynamic over-
loading of the network and servers, and (c) coping with
faults. This paper briefly examines two existing schemes
for concurrent downloading or parallel-access download-
ing, or paraloading as it is called. It proposes a third par-
aloading scheme called the Dynamic Parallel Access. The
performance of this scheme is experimentally evaluated.
Recommendations for further improvements are also dis-
cussed.
1. Introduction
As various applications are Internet-enabled, the number
of repositories in the Internet that hold valuable content are
increasing. For example Internet-based repositories are be-
ginning to hold content such as multi-gigabit movie files,
operating system and other large software distributions, and
large multimedia documents. This creates a need for the
clients to find faster downloading schemes for both online
and offline usage. The conventional way of downloading
files from an Internet-based server is to open one or more
connections between it and the client. While opening mul-
tiple connections might reduce download times compared
to a single connection, due to the following issues the per-
formance enhancement can be limited: (a) server load and
capacity, (b) bottleneck link bandwidth, (c) instantaneous
bandwidth, (d) multi-connection client overhead, and (e)
interconnection resource allocation.
Replicating content such that it can be accessed from
multiple locations is one way for decreasing the download
times. Caching, content delivery, and mirroring are some
of the techniques that replicate content using different poli-
Muthucumaru Maheswaran
Dept. of Computer Science
University of Manitoba
Winnipeg, Canada
email: maheswar@cs.umanitoba.ca
cies and primary purposes. Mirror servers were tradition-
ally used to improve the availability of the content. Re-
cently, however, several projects have examined the con-
cept of concurrently using multiple mirror servers to down-
load content for a single client. The concurrent or parallel
downloading is performed to reduce the download time for
a client.
One of the key problems with paraloading schemes is
the mirror site selection problem. This problem is compli-
cated because accurate “performance” information to select
the best set of mirror servers is not available to the clients.
Additionally, the problems mentioned above for the sin-
gle server case still exist. Further, network conditions can
change during the download that can lead to decreased per-
formance. This observation motivated us to examine adap-
tive parallel download techniques that use a set of mirror
servers. The membership of the mirror server set may dy-
namically change. In paraloading segments of the file are
downloaded from each server in the set and are then re-
assembled at the client. The parallel-access scheme was
first proposed by [4]. Following are some of the advantages
of paraloading:
 Depending on the topology, in the ideal case, the ag-
gregate bandwidth of all individual connections may
increase the overall throughput to the client.
 Because multiple connections are used, paraloading is
more resilient to link and general route failures.
 The inherent load balancing associated with parallel
loading due to the fact that connections are spread
out to many servers and not just one. Therefore, the
scheme can be immune to individual server load fluc-
tuations, bottleneck link bandwidth and traffic fluctu-
ations.
Section 2. examines three different paraloading
schemes (two existing and one new) all of which use
application-level negotiation to schedule the transmission
of different segments of a file. Section 3. describes the ex-
periments performed to evaluate the different schemes and
examines the results obtained from the experiments.
2. Parallel Downloading Schemes
2.1 History-Based Parallel Access
History-based parallel access [4] is a relatively simple
scheme in which information regarding the previous trans-
mission rates between the client and every mirror server is
recorded in a database and used to determine how large a
file segment is downloaded by each server. More specif-
ically, forM mirror servers, the file will be divided into
M unequally-sized disjoint segments each of which will be
assigned to a mirror server based on “expected” data rates
derived from historical data between the client and each
mirror, i.e., the faster a server is (based on the history) the
larger the file block it will be assigned to be downloaded
from that server. File blocks are downloaded from each in-
dividual server using the HTTP 1.1 byte range header fea-
ture resulting in a server-side transparent solution requiring
modifications only on the client side.
The problem with the history-based parallel access is
of course the validity of the recorded transmission rates i.e.,
how close those recorded rates are compared to the actual
transmission rates in future downloads. The larger the di-
vergence between historical and (future) actual data trans-
mission rates, the worse the performance is since slower
servers will be assigned file blocks larger than should have
been and vice versa for mirror servers faster than what his-
torical data “proclaims.” One issue that must be addressed
with such a scheme is how exactly is server transmission
rate data obtained and updated.
2.2 Semi-dynamic Parallel Access
The semi-dynamic parallel access downloading method
was first proposed by [4], where it is referred to as dynamic
parallel access downloading but in this paper is referred
to as semi-dynamic in order not to confuse with our own
parallel downloading method examined next. The semi-
dynamic scheme is conceptually simple: initially the re-
ceiver will obtain the size of the file it wishes to download
(e.g., by polling one of the mirror servers) and then the file
is segmented by the receiver into equal-size blocks. The
receiver will request at the beginning from all the mirror
servers to download one block. Once a server has com-
pleted sending the requested file block, the client will re-
quest a new (undelivered) block from the server. This
will continue until all file blocks have been downloaded,
at which point the receiver will reassemble the file from the
individual downloaded blocks. As in history-based paral-
lel access downloading, the HTTP 1.1 byte range feature
is used to download individual blocks of the file from each
mirror server.
In order to reduce overhead, persistent TCP connec-
tions are used between the client and each server. One en-
hancement used is that if there are less file blocks left than
servers, a file block that has already been requested but not
yet completely downloaded can be downloaded simultane-
ously in parallel by another server leading potentially to
a faster download for that block. In this scheme, faster
servers provide larger portions (more blocks) of the total
transmitted data.
Based on the paraloading scheme proposed in [4], [3]
proposed a modified paraloading scheme which is essen-
tially identical to that proposed by [4] but with the follow-
ing three enhancements:
 Minimizing the delay at startup by piggybacking onto
a data block a request for the file size and a request for
the list of mirror servers that posses the requested file.
 Minimizing the idle time between block downloads by
pipelining the block requests.
 Minimizing the idle time in downloading the last
block in one of three ways: a) use small block sizes b)
dynamically adjust file block size for that last blocks
c) send requests to the idle servers to download the
remaining portions of the last block.
2.3 Dynamic Parallel Access
In this section a new type of paraloading scheme developed
by the authors of this paper is examined, called dynamic
parallel access downloading. This new paraloading method
is based to a certain extent on the semi-dynamic paraload-
ing scheme first proposed by [4], as examined above, how-
ever it was designed with large size file transfers in mind. In
this new scheme, like those before, the client segments the
file into fixed-size blocks which are requested and down-
loaded from the individual mirror servers. More specifi-
cally, operation is as follows.
Initially connections are opened to all the mirror
servers. In the current implementation the list of available
mirror servers is obtained from a file containing the server
list, however in the future for actual practical use there must
be a way to dynamically obtain the list of available servers.
This could be done in various ways, such as retrieving the
server list from some type of a directory service, or by ex-
tending the DNS system to provide such information [2].
In order to reduce overhead, persistent TCP con-
nections are used between the client and each server so
that the TCP connection-setup three-way handshake de-
lay is avoided and also so that several slow-start phases
are avoided. This paraloading scheme, as opposed to the
history-based and semi-dynamic parallel access paraload-
ing methods examined before, is not based on the HTTP
protocol (using the HTTP byte range header to download
a block of a file), but uses a proprietary paraloading server
and client running on top of the TCP transport layer proto-
col.
The file size is obtained from the first server with
which the client comes into contact with and thus no time
is lost by performing separate server probing for the ex-
plicit purpose of obtaining the file size value. The file, as
mentioned above, is divided into fixed-size blocks and a re-
quest is made to each server to download a distinct block.
Once a given server has completed the assigned file block
download a new block for downloading is assigned to it.
Currently the block size used is set to 1 Mbyte. The selec-
tion of the block size is an important issue, in which three
facts should be taken into account:
 The file block size should be such, such that the num-
ber of blocks is larger than the number of servers. Oth-
erwise the faster servers will exhibit large idle times.
 Each block should be small enough so as to provide a
fine enough granularity, for the same reason as men-
tioned in i).
 On the other hand, each block should be large enough
so as to reduce the number of requests that need to be
made to servers, thus reducing the ratio of idle time to
download time.
The major point of difference between this paraload-
ing algorithm and semi-dynamic paraloading, is that the
number of mirror servers used in downloading does not re-
main static. After a connection is established to each server
and a block download request is made to every server,
the so-called server downscaling testing commences in the
case that 4 or more servers are currently being used. With
downscaling testing, the transmission rate of every server
is monitored during the parallel download. At given time
intervals, the slowest of the servers (based on the recorded
transmission rates) is selected to remain idle for a period of
time by not being given any new block download requests.
After the testing time has elapsed, the aggregate download
transmission rate (i.e. the sum of the individual transmis-
sion rates of each server) is compared to the aggregate data
rate of the time before the selected server was made idle.
If the new aggregate rate is lower than the old aggre-
gate rate by less than a certain threshold percentage (cur-
rently a value of 15% is used, but can be varied if desired)
or the new rate happens to be equal or even higher, then
the server is deemed to be unnecessary since it offers no
substantial increase in bandwidth and is deleted from the
list of active mirror servers for the current download and
is used no further in downloading, thus freeing up unnec-
essarily tied up server and network resources. In this case
server downscaling testing will continue by proceeding to
the next server (which will be the slowest among the cur-
rently active servers, as before). Otherwise, if the drop in
the aggregate download rate is above the threshold percent-
age, then the server is taken out of the idle state and used
again in downloading. In addition, server downscaling will
cease in this case since it is considered that we have now
reached the “ideal” number of servers needed i.e. the least
number of servers required to give the maximum possible
download rate. Server downscaling will also terminate at
any time if less than four servers are currently active par-
ticipating in the download process.
After server downscaling has terminated (i.e. no more
servers will be deactivated in the particular download),
server upscaling testing will commence. Namely, if a sig-
nificant decrease in the aggregate download rate exists for
a sustained period of time then an additional server will be
added for use in paraloading. More specifically, at periodic
time intervals (currently every 30 seconds, although this
value can be adjusted) testing will start by monitoring the
aggregate download rate twice at given intervals (10 sec-
onds after test commencement and 10 seconds after that,
again these values can be varied). If at both of those time
intervals there is a significant drop in the aggregate down-
load rate (of 15% or more, although as previous testing pa-
rameters this can be varied) then it is concluded that there
is a sustained drop in the download rate and to remedy this
a mirror server, if an available one exists of course, is added
for use in downloading to try to increase the rate.
The fact that paraloading starts using all available mir-
ror servers and then downscales as opposed to for example
starting from one server and then adding mirror servers un-
til there was no substantial increase in bandwidth is deliber-
ate. The reason is that the purpose is to decrease download
times and thus bandwidth underutilization (using too few
servers) is a much more significant factor than bandwidth
overutilization (using too many servers). While bandwidth
underutilization results in lost time, bandwidth overutiliza-
tion simply results in temporary use of unneeded resources
that will be released eventually.
One observation that should briefly be made here is
the duration of the testing intervals used in server down-
scaling testing as examined above. On the one hand it is
desirable that testing periods be as short as possible so as to
complete mirror server deletion as quickly as possible. On
the other hand though these testing intervals must be long
enough so as to obtain valid data (transmission rate read-
ings) that can be used to make a valid decision. In other
words when a server is made idle in order to measure the
effects on the aggregate download rate, sufficient time must
elapse to allow the network to enter into a steady-state so
to speak so as to measure the real effects on the aggregate
download rate. The value used currently, and believed to
satisfy both requirements stated above, is 10 seconds.
3. Experimental Results and Analysis
Experiments were conducted by measuring the download
times using various downloading methods. More specifi-
cally, the download times using dynamic paraloading were
compared to those of single server FTP file downloading
and single server multiple parallel connection downloading
using the dynamic paraloading client and server.
In all tests conducted clients from the same domain
were used along with a multitude of geographically dis-
persed servers in Canada and the United States. More
specifically, a total of eight servers were used: three at the
University of Victoria in British Colombia, two TRLabs
servers in Winnipeg and Regina, two servers at the Pur-
due University and an additional server at the University of
Illinois.
 1600
1400
1200
loading time/ (sec)
1000
800
600
400
200
0 hour of day (2:30AM - 2:30PM)
1 3 5 7 9 11 13 15 17
hour of day (2:30 AM - 2:30 PM)
Multiserver Paraloading Fastest FTP
Figure 1. Download times of multi-server dynamic par-
aloading and single server FTP.
The tests (dynamic paraloading, single-server multi-
ple connection paraloading and simple single-server FTP
downloading) were conducted over a 24-hour period with
results being obtained every hour in order to get results for
the performance of each of the three downloading schemes
used under relatively heavy and varying traffic conditions
(during the day) and under lighter and less varying traffic
conditions i.e during the early and late hours.
In testing dynamic parallel access downloading, all
eight remote hosts were used as mirror servers to download
a 45 Mbyte file. The same 45 Mbyte file was used for test-
ing the other two single-server downloading schemes also,
testing with many different servers. For the single-server
multiple connection paraloading scheme, a single mirror
server was used each time with eight connections between
client and server being used, i.e. as many as the number of
mirror servers used in multi-server dynamic paraloading.
Figure 1 compares the results obtained for multi-
server dynamic paraloading with those obtained for single-
server FTP downloading (fastest and slowest cases). From
the results it can be seen that while paraloading signif-
icantly outperformed the slowest FTP downloading case,
being approximately 10 times faster, the difference is sig-
nificantly smaller when compared to the fastest FTP down-
load, but still much faster proving the obvious benefit of
paraloading. The maximum theoretical performance of
multi-server dynamic paraloading is if the aggregate down-
load rate is equal to the sum of the individual server down-
load rates which is not the case here. This is due to the fact
that given a sufficiently large number of servers a satura-
tion point is reached beyond which the aggregate rate can
increase no more no matter how many servers are added
due to a bottleneck at the receiving client and/or along the
network path between client and mirror servers. This is
250
200
loading time/ (sec)
150
100
50
0
11
13
15
17
19
21
23
19 21 23
Multiserver Paraloading
Fastest Single Server Paraloading
Slowest FTP Slowest Single Server Paraloading
Figure 2. Download times of multi-server dynamic par-
aloading and single-server multiple connection paraload-
ing.
also the reason for which server downscaling, as explained
before, was added to the dynamic paraloading scheme.
Figure 2 compares single-server multiple connection
paraloading results to those obtained for multi-server dy-
namic paraloading. Here we notice that mutli-server par-
aloading is somewhat slower than the fastest case of single-
server multi-connection paraloading. This is not unex-
pected as its only natural that in multi-server paraloading
the slower servers will degrade the performance and will
very likely be somewhat slower than when using the same
number of connections to the fastest server(s). The advan-
tage of multi-server paraloading even in this case is the bet-
ter load balancing it achieves by spreading the load across
multiple servers and not just one server where with multiple
connections to one server the server will very soon become
congested. The case is the opposite though for the slowest
single-server multi-connection paraloading, where multi-
server paraloading significantly outperforms it due mainly
to one of the advantages of using multiple servers which
is higher performance mirror servers compensating for the
lower performance servers.
Figure 3 summarizes the results displayed in the pre-
vious two figures comparing multi-server dynamic par-
aloading performance to the best FTP and single-server
paraloading performances. It is worthy of mention here
that the download performances, particularly that of FTP,
degrade during the morning and afternoon hours and the
variation also of the download times increases during those
hours. This applies mostly to FTP, with the two paraloading
schemes being affected to a lesser extent, thus concluding
that the use of multiple connections also has a “smooth-
ing” effect on download performance isolating to a certain
extent the overall download performance from the perfor-
 by the authors of this paper was presented and examined.
Dynamic parallel access downloading, based on the exper-
250 iments performed, proved that it performed very well in
terms of reducing download time even under varying traf-
200
loading time/ (sec)

fic/network conditions. Additionally, dynamic parallel ac-

150 cess possesses some advantages over the other two down-
loading methods briefly examined, such as being able to
100
adjust the number of mirror servers that are active thus re-
50 leasing server and network resources that are unnecessarily
utilized, and also improved server load balancing compared
0 to the other downloading schemes. It is strongly believed
1

11

13

15

17

19

21

23
hour of day (2:30AM - 2:30PM)
Multiserver Paraloading
Fastest Single Server Paraloading
Fastest FTP
Figure 3. Download times of multi-server dynamic par-
aloading and best single-server multi-connection paraload-
ing and FTP performances.
mance of any individual server or other traffic/network con-
dition variations.
3.1 Comparison to Semi-Dynamic Paraload-
ing
From the experimental results, presented and analyzed
in the previous section, it is apparent that the dynamic
paraloading scheme performs very well increasing down-
load performance significantly when compared to slower
servers.
One aspect were dynamic paraloading has an advan-
tage over semi-dynamic, is the adjustable number of mir-
ror servers that are used at any given time. Instead of us-
ing all the mirror servers that are available, dynamic par-
aloading can reduce the number of active servers thus re-
leasing server and network resources that are unnecessarily
utilized.
Another advantage of dynamic paraloading is the bet-
ter server load balancing achieved compared to the other
downloading schemes. While all downloading schemes
achieve server load balancing to a certain extent, by dis-
tributing the connections among all mirror servers, dy-
namic paraloading with its server downscaling feature re-
leases servers that are unnecessarily utilized (i.e. that offer
very little to the aggregate download rate) and this would
include heavily loaded servers.
that performance can be further improved with additional
enhancements.
Some possible enhancements to dynamic parallel ac-
cess paraloading that should be examined as future work
are:
 The ability to be able to add additional connections
between the client and a given server. More specif-
ically, the ability to add a 2nd, 3rd etc. connection
between the client and the fastest servers rather than
just being able to add an additional mirror server.
 Use pipelining of block download requests in order to
minimize the number of idle periods between block
downloads.
 The development of a method to dynamically retrieve
the list of mirror servers, such as a directory service
for example.
 Determine the effects of paraloading in terms of net-
work congestion.
References
[1] J. Byers, M. Luby,and M. Mitzenmacher, “Accessing
multiple mirror sites in parallel: Using tornado codes
to speed up downloads,” IEEE INFOCOM, 1999.
[2] J. Kangasharju, K.W. Ross, and J. W. Roberts, “Lo-
cating copies of objects using the domain name
system,” 4th International Caching Workshop, Mar.
2000.
[3] A. Miu and E. Shih, Performance Analysis of a Dy-
namic Parallel Downloading Scheme from Mirror
Sites Throughout the Internet, Technical Report, Lab-
oratory of Computer Science, MIT, 2000.
[4] P. Rodriguez, A. Kirpal, and E. Biersack, “Parallel-
access for mirror sites in the Internet,” IEEE INFO-
COM, 2000.

4. Conclusions and Future Work

In this paper a new parallel access downloading scheme
referred to as Dynamic Parallel Access scheme developed