See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221615479

SPIDER: A system for scalable, parallel / distributed evaluation of large-

scale RDF data

Conference Paper · January 2009

Source: DBLP

CITATIONS READS

23 208

5 authors, including:

Hyunsik Choi Jihoon Son

Gruter Korea University
7 PUBLICATIONS   418 CITATIONS    9 PUBLICATIONS   86 CITATIONS   

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Jihoon Son on 04 June 2014.

The user has requested enhancement of the downloaded file.

 SPIDER : A System for Scalable, Parallel / Distributed
Evaluation of large-scale RDF Data

Hyunsik Choi, Jihoon Son, YongHyun Cho

∗
Min Kyoung Sung, Yon Dohn Chung
Department of Computer Science and Engineering
College of Information and Communication
Korea University, South Korea
{hyunsikchoi, jihoonson, yonghyun, mj999, ydchung}@korea.ac.kr

ABSTRACT In these applications, the size of dataset is usually very

RDF is a data model for representing labeled directed graphs,
and it is also an important building block of semantic web.
Due to its flexibility and applicability, RDF has been used in
broad applications, such as bioinformatics, social networks,
so on. In these applications, large-scale graph datasets are
very common. However, existing techniques are not effec-
tively managing them. In this paper, we present a query
processing system for RDF data, named SPIDER, based on
the well-known parallel/distributed computing framework,
Hadoop. SPIDER consists of two major modules (1) the
graph data loader and (2) the graph query processor. The
loader analyzes and dissects the RDF data and places parts
of data over multiple servers. The query processor parses the
user query and distributes subqueries to cluster nodes. Also,
the results of subqueries from multiple servers are gathered
(and re-evaluated if necessary) and delievered to the user.
Both modules utilize the MapReduce framework of Hadoop.
In addition, our system supports some features of SPARQL
query language. This prototype will be foundation to de-
velop real applications in many domains where large-scale
graph data are common.
1. INTRODUCTION
As a data model for representing labeled, directed graphs,
RDF is used as important building blocks of semantic web.
RDF can also be extended easily to ontologies, such as RDFS
and OWL, where these ontologies provide a means to define
vocabularies specified to some domain, schemas, and rela-
tions between elements of vocabulary. Recently, owing to its
flexibility and applicability, RDF has been popularly used
in a variety of applications, such as semantic web, bioinfor-
matics[3], and social networks[6].
∗the corresponding author
Permission to copy without fee all or part of this material is granted provided
that the copies are not made or distributed for direct commercial advantage,
the VLDB copyright notice and the title of the publication and its date appear,
and notice is given that copying is by permission of the Very Large Data
Base Endowment. To copy otherwise, or to republish, to post on servers
or to redistribute to lists, requires a fee and/or special permission from the
publisher, ACM.
VLDB ‘09, August 24-28, 2009, Lyon, France
Copyright 2009 VLDB Endowment, ACM 000-0-00000-000-0/00/00.
huge. For example, in semantic web large-scale graph datasets
are required for complex inferencing, in life science many
projects[13, 3, 4, 8] are accumulating DNA sequence and
protein interaction network datasets, and the social net-
works area needs massive sample data sets for complex an-
alyzing of relations between people (or groups). Volumes
of graph data generated from these applications reach tens
of petabytes. However, existing systems[7, 19, 18] are not
sufficient to deal with these large-scale RDF graph datasets
due to following reasons:
• Existing systems are based on single machine. So, they
have no capability to keep the dramatically increasing
RDF data. It is known that the storage technology
(e.g., SAS SCSI and SATA2 HDD interfaces[14]) has a
theoretically maximum transfer rate of 3Gb/s. For ex-
ample, 1 petabyte scanning on a recent HDD requires
at least 33 days.
• The subgraph query is the core RDF query type. How-
ever, existing systems cannot efficiently process sub-
graph queries for big graph datasets because it is NP-
Complete.
As a practical approach to overcome these problems above,
we use the server-cluster approach, where the large RDF
data is partitioned and distributed over multiple servers,
and the user subgraph query is dissected/delivered into mul-
tiple subqueries and processed over multiple servers in a dis-
tributed and parallel manner.
In this demonstration, we present a RDF subgraph pro-
cessing system, which enables Scalable, Parallel / Distributed
Evaluation of large-scale RDF data (SPIDER). It aims to
store and process petabyte-scale RDF graph datasets. In or-
der to store and process a huge amount of RDF graph data,
we use a distributed file system and employ the MapRe-
duce framework[10] provided by Hadoop[2]. Hadoop cluster
can be composed of from several to tens of thousands ma-
chines. Due to its scalability, it is regarded as a representa-
tive framework in cloud computing environments.
The main contributions are as follows:
• Our system can store and process large-scale RDF
graph datasets on commodity cluster of servers.
• When more storage and computation power are re-
quired, our system can be easily expanded by adding

Importing Graph Data
XML TURTLE N-TRIPLE
Input Formatter
Graph Deployment
Graph Data Partitioner
Graph Data Contractor
Local Contractor Global Contractor
Graph Data Allocator
Index Constructor
Tree Bitmap
Hash Table List
Graph Data Loader
Figure 1: A System Overview of SPIDER
more cluster nodes.
• Our system allows users to retrieve subgraphs matched
to user-defined graph patterns.
• Our system supports main features of SPARQL query
language.
The rest of this paper is organized as follows. In Sec-
tion 2, we give an overview of our proposed system, and
we description two main components, the graph loader and
the graph query processor. In Section 3, we describe our
demonstration system that we developed to store and pro-
cess large-scale graph data. In Section 4, we discuss the
applicability of our system and some directions for further
improvement.
2. SYSTEM OVERVIEW
This section gives an overview of SPIDER. Our system
SPIDER is developed on Hadoop[2] distributed system frame-
work. Using Hadoop, the SPIDER system stores the RDF
data on the distributed file system (DFS), and processes
subgraph queries in a parallel/distributed manner with the
MapReduce technique. For these, our system has two main
components, which are (1) the graph loader and (2) the sub-
graph query processor. We will disscuss them in the follow-
ing subsections. This design allows our system to deal with
a huge graph data whose volume is from several hundred
megabytes to several petabytes. Fig. 1 shows the overall
architecture of the SPIDER system. The left-hand side is
the Graph Data Loader module and the right-hand side is
the Graph Query Processor module. In this figure, dashed
lines indicate not yet implementated modules.
2.1 Underlying Architecture
In this section, we briefly describe the underlying architec-
ture on which SPIDER is based. As we mentioned, SPIDER
is developed under Hadoop[2], a distributed system frame-
work. Hadoop is designed to build parallel/distributed com-
puting environments composed of tens of thousands com-
modity servers, and it is known to be very scalable. Thus,
Hadoop
Distributed File System Pre-Processor
Query Parser
Query Input
Query Dissemination
Users
Query Processing
Map Process
Partial Matching
Web UI
Reduce Process
Full Matching
Output
Output Formatter
Applications
Graph Query Processor
it is popularly used as a representative framework in cloud
computing environments, such as Amazon EC[1] and Ya-
hoo!.
Hadoop consists of a distributed file system called Hadoop
Distributed File System (HDFS)[5] and MapReduce frame-
work[10]. HDFS is inspired by Google File System (GFS)[11].
To store huge amount of data, HDFS divides big data into
small data blocks, and then it disseminates them into all
cluster nodes uniformly as possible. Data blocks are repli-
cated for reliability.
MapReduce is a programming model and a framework
based on shared-nothing architecture[17]. In shared-nothing
architecture each node does not communicate one another
during computation. Therefore, it does not incur overheads
caused by managing distributed processing. Thus, the shared-
nothing architecture provides the ability to have linear scal-
ability. Based on shared-nothing, MapReduce provides a
means to divide a big task into many small tasks. The
MapReduce computation basically has two steps, Map and
Reduce. In each step, each cluster node is assigned to either
mapper or reducer. A mapper performs map function on
input key-value pairs extracted from raw input data, and
outputs a new pair consisting of a key and a set of values
corresponding to the key. Then, reducers aggregate input
pairs passed from mappers and output final key-value pairs.
2.2 RDF Data Model
Before we discuss main components, we need to discuss
the data model on which our system is based. RDF (Re-
source Description Framework)[15] is a graph data model
to represent labeled directed graph, and it has several in-
stance document types, such as XML, TURTLE, N3, and N
Triple. Although RDF is originally designed to represent se-
mantic web data, because of its flexibility and applicability
it has also been widely used in various areas, such as social
networks[6, 9] and life sciences[3].
In RDF data model[15], graphs are a set of triples, each
of which consists of a subject, a predicate, and an object.
They are usually represented as < S, P, O >. A subject can
be either a URI (Uniform Resource Identifier) or a blank
 Map
Process phase. In the full matching phase, each reducer is assigned
Node N1 Node N2 Node N3
Q1 Q1 Q1 to an individual query graph. They accumulate triples passed
Partial Partial Partial
Matching Q2 Matching Q2 Matching Q2 by mappers and joins them in terms of their identifications
Local Storage
Q3
Queries Local Storage
Q3
Local Storage
Q3 (i.e., URI). Eventually, joined triples are organized to con-
Queries Queries
nected graphs. After that, reducers retrieve subgraphs fully
matched to query graphs.
Fig. 2. shows an example of the query processing through
Reduce
MapReduce. In this example, there is a cluster consisted of
Process three nodes, N1 , N2 and N3 . When three queries Q1 , Q2
Node N1 Node N2 Node N3
Complete Complete Complete
and Q3 are put into the GQP, map operations start to per-
Matching Q1 Matching Q2 Matching Q3 form the partial matching phase. During the phase, mappers
Partial Partial Partial
Matched Matched Matched find partially matched triples to any queries and pass them
Graphs Graphs Graphs
to reducers. Nodes N1 , N2 and N3 also perform reduce op-
erations for queries Q1 , Q2 and Q3 respectively. Each of
Figure 2: Query Processing carried out with them connects partially matched triples and validates them
MapReduce with each query.
Under the shared-nothing architecture, our system has
the high scalability. In each phase of the query processing,
node1 . A predicate can be only URI. A object can be URI or
there is no communication between mappers and reducers.
literal string. In RDF, the role of URI is a unique identifier.
Therefore, the performance of our system is increased lin-
A subject (or an object) corresponds to a vertex in graph,
early with an increasing the number of clusters.
and a predicate corresponds to an edge. Then, a triple <
S, P, O > is a directed arc in the graph.
3. DEMONSTRATION SCENARIO
2.3 Graph Data Loader
SPIDER is developed under Hadoop, which is a distributed
The graph data loader (GDL) is to read RDF graph data system framework. Our hadoop cluster that our demonstra-
and then to store them into DFS. GDL can load RDF graph tion runs on is composed of 10 cluster nodes, and they are
data either from local file system or from DFS. When GDL connected to 100Mbps LAN.
loads RDF data from DFS, it can be carried out with dis- SPIDER provides web-based user interface (UI) that pro-
tributed and parallel manner by using MapReduce. vides two features, which are the graph loader UI and the
Initially, GDL disseminates information of splits data to graph query UI. It enables users to import a RDF dataset
cluster nodes. During loading RDF data, all cluster nodes from a local file system or DFS. Later, users will choose from
try to read RDF data on their local storage, and they save which data are loaded. The graph loader UI enables users
them into DFS. This loading process is performed in dis- to give an instance name to the loaded graph data. When
tributed manner, so it is appropriate for a huge amount of users describe graph query, they can choose which instance
RDF datasets. is queried. When a user specifies both import directory and
As storing triples to DFS, GDL immediately stores a triple instance name, the graph loader UI passes the parameters
into a fixed length record. We called this storing method to the graph data loader. According to whether the import
the triple record. This method is a simple and efficient way directory indicates local directory or distributed file system,
to manage lots of triples in DFS because it can well take the graph data loader decides to be performed with MapRe-
mechanisms that HDFS provides. We expect the graph data duce. The graph query UI allows users to submits subgraph
loader to read and place petabyte-scale RDF graph. patterns defined by SPARQL query language[16], W3C rec-
ommendation. When a user submit a query, the UI passes a
2.4 Graph Query Processor query to the graph query processor and then the graph query
The graph query processor(GQP) reads graph data and processor installs the given query across all cluster nodes.
searches subgraphs matched to query graphs given by the After then, query processing starts with map operation in
query pre-processor. The query processing is carried out each node. While map operation is executing, each mapper
with two phases, the partial matching phase and the full finds triples matched to any of the given query patterns.
matching phase. The retrieved triples are passed to a reducer. The reducer
tries to join triples passed from mappers, and then it checks
• Partial matching phase. Each node reads locally
stored graph data sequentially and tries to find triples
matched to any condition of given queries.

• Full matching phase. Each node aggregates partially

matched triples to connected graphs and makes final
results which are fully matched with given queries.

We implement GQP using MapReduce framework. The

partial matching phase is performed in the map step. Each
mapper finds the triples matched with any queries and passes
the results to reducers. Reducers perform the full matching
1
A blank node is an anonymous node that groups one or Figure 3: Web-based User Interface enables users to
more RDF statements. submit SPARQL and shows visualized results.
 joined triples (i.e., a subgraph matched to the given query).
If the subgraph is matched, the query processor outputs the
retrieved subgraph. Finally, the graph query UI can output
the result of the given query. At that time, the user can
choose to see the result as visualized graph and download
a text file. This UI is shown in Fig. 3. For representing
visualized graph, we employ the prefuse[12], the toolkit for
interactive information visualization.
As we mentioned above, our system supports some fea-
tures of SPARQL. SPARQL provides a means to define var-
ious subgraph patterns to be retrieved. Among various
graph patterns, our system provides Basic Graph Pattern
and Group Graph Pattern. The basic graph patterns are
filters to find a set of triples that must match patterns.
The group graph patterns are filters to find a set of graphs
matched to all patterns. For instance, the group graph pat-
terns are following:
select ?x
where {
?x type Book .
?x hasAuthor ‘‘Donald E. Knuth’’ .
?x year 1998 .
} F. Demiralp, N. Faruque, R. Gibson, G. Hoad,
Above example shows an example of SPARQL. In this ex-
ample, graph patterns are defined by ‘where’ clause. This
query statement is to find subjects that satisfy with all fol-
lowing conditions: a type of the subject is Book, the suject
has author ‘Donald E. Knuth’, and it is published in 1998
year. Two types of graph patterns are frequently used in
many applications.
4. DISCUSSION
SPIDER focuses the scalability for storing and querying
large-scale RDF graph datasets. Our system wins the goal
by making use of DFS and the MapReduce framework pro-
vided by Hadoop. Hence, our system can be easily extended
by adding more cluster nodes when the system needs more
storage and computation power. In addition, SPIDER takes
advantages of shared-nothing architecture. Thus, SPIDER
has linear scalability as cluster nodes increases. Being scal-
able, SPIDER can be widely used in many areas where large-
scale graph processing is required.
There still remain many research issues for improving SPI-
DER.
• Our system is still insufficient to provide on-line ser-
vices that face users. It is because Hadoop takes long
time to initialize tasks. In order to mitigate long re-
sponse time, we will need some cache techniques for
subgraph queries.
• Index structures for various queries, such as path queries
and subgraph queries are required. These index struc-
tures have to reflect distributed file systems and MapRe-
duce framework.
• A programmable graph query language (PGQL) for
analyzing graph datasets are required to provide the
abstraction of the distributed systems and expressive
power. In addition, a PGQL program composed of a
sequence of graph operations will give the system more
opportunities to optimize complex queries because the
system expects whole graph operations.
View publication stats
5. REFERENCES
[1] E. Amazon. Amazon elastic compute cloud. aws.
amazon. com/ec2.
[2] Apache Software Foundation. Hadoop, 2006.
http://hadoop.apache.org/core.
[3] A. Bairoch, R. Apweiler, C. Wu, W. Barker,
B. Boeckmann, S. Ferro, E. Gasteiger, H. Huang,
R. Lopez, M. Magrane, et al. The Universal Protein
Resource (UniProt). Nucleic Acids Research, 33:D154,
2005.
[4] H. M. Berman, J. D. Westbrook, Z. Feng,
G. Gilliland, T. N. Bhat, H. Weissig, I. N. Shindyalov,
and P. E. Bourne. The protein data bank. Nucleic
Acids Research, 28(1):235–242, 2000.
[5] D. Borthakur. The Hadoop Distributed File System:
Architecture and Design. Hadoop Wiki, 2008.
[6] D. Brickley and L. Miller. FOAF Vocabulary
Specification. Namespace Document, 3, 2005.
[7] E. I. Chong, S. Das, G. Eadon, and J. Srinivasan. An
efficient sql-based rdf querying scheme. In VLDB,
pages 1216–1227, 2005.
[8] G. Cochrane, R. Akhtar, J. Bonfield, L. Bower,
T. Hubbard, C. Hunter, et al. Petabyte-scale
innovations at the European Nucleotide Archive.
Nucleic Acids Research.
[9] M. P. Consens. Managing linked data on the web: The
linkedmdb showcase. In LA-WEB, pages 1–2, 2008.
[10] J. Dean and S. Ghemawat. Mapreduce: Simplified
data processing on large clusters. In OSDI, pages
137–150, 2004.
[11] S. Ghemawat, H. Gobioff, and S.-T. Leung. The
google file system. In SOSP, pages 29–43, 2003.
[12] J. Heer, S. Card, and J. Landay. Prefuse: a toolkit for
interactive information visualization. In Proceedings of
the SIGCHI conference on Human factors in
computing systems, pages 421–430. ACM New York,
NY, USA, 2005.
[13] M. Kanehisa and S. Goto. Kegg: Kyoto encyclopedia
of genes and genomes. Nucleic Acids Research,
28(1):27–30, 2000.
[14] M. Kawamoto. HDD Interface Technologies. Fujitsu
Scientific and Technical Journal, 42(1):78–92, 2006.
[15] O. Lassila, R. Swick, et al. Resource Description
Framework (RDF) Model and Syntax Specification.
1999.
[16] E. PrudHommeaux, A. Seaborne, et al. SPARQL
query language for RDF. W3C working draft, 4, 2006.
[17] M. Stonebraker. The case for shared nothing. IEEE
Database Eng. Bull., 9(1):4–9, 1986.
[18] Y. Tian, J. M. Patel, V. Nair, S. Martini, and
M. Kretzler. Periscope/gq: a graph querying toolkit.
PVLDB, 1(2):1404–1407, 2008.
[19] K. Wilkinson, C. Sayers, H. A. Kuno, and
D. Reynolds. Efficient rdf storage and retrieval in
jena2. In SWDB, pages 131–150, 2003.