2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering
Big Data System Development:
An Embedded Case Study with a Global Outsourcing Firm
Hong-Mei Chen Rick Kazman
Department of IT Management Department of IT Management
Shidler College of Business University of Hawaii at Manoa
University Of Hawaii at Manoa & SEI/CMU
Honolulu, USA Pittsburgh, USA
hmchen@hawaii.edu kazman@hawaii.edu
Abstract— Big data system development is dramatically different
from small (traditional, structured) data system development. At
the end of 2014, big data deployment is still scarce and failures
abound. Outsourcing has become a main strategy for many
enterprises. We therefore selected an outsourcing company who
has successfully deployed big data projects for our study. Our
research results from analyzing 10 outsourced big data projects
provide a glimpse into early adopters of big data, illuminates the
challenges for system development that stem from the 5Vs of big
data and crystallizes the importance of architecture design choices
and technology selection. We followed a collaborative practice
research (CPR) method to develop and validate a new method,
called BDD. BDD is the first attempt to systematically combine
architecture design with data modeling approaches to address big
data system development challenges. The use of reference
architectures and a technology catalog are advancements to
architecture design methods and are proving to be well-suited for
big data system architecture design and system development.
Index Terms— big data, system engineering, software
architecture, data system design methods, embedded case study
methodology, collaborative practice research
I. INTRODUCTION
The big data phenomena is unprecedented: data volume
doubling rates are accelerating, and 90% of all the data in the
world has been created in the past 2 years. Big data holds big
promise: data—structured, semi-structured and unstructured—
are being collected from everywhere by enterprises for real-
time decision-making, operational intelligence, customer
intelligence, business innovation and competitive advantage.
Big data is no doubt in a “hype” phase, and has been touted as
the new oil [17] and the new gold [15]. Most enterprises
believe that managing big data is important to improve
business value. However, how to develop systems for mining
the new gold appears to be a daunting task.
First and foremost, the 5V (volume, velocity, variety,
veracity, and value) characteristics of big data require
capabilities beyond traditional data systems and challenge
existing software engineering and system development
principles, methods and tools. Second, the rapid proliferation
and advancements of open source and proprietary technologies
to address the 5Vs has added a new dimension of complexity to
system development. Currently, big data technologies include
distributed database processing, MPP (Massively Parallel
Processing) databases, NoSQL databases, New SQL databases,
978-1-4673-7025-7/15 $31.00 © 2015 IEEE 44
DOI 10.1109/BIGDSE.2015.15
Serge Haziyev, Olha Hrytsay
SoftServe Inc.
Austin, TX, USA
{shaziyev@softserveinc.com,
ohrytsay@softserveinc.com}
in-memory databases, real time stream processing, advanced
analytics, big data analytic clouds, and big data appliances, just
to name a few. Each technology type has many vendors and
products. For instance, there are over 150 products in the
NoSQL space alone. The Hadoop ecosystem continues to
expand and change. For instance, Hadoop MapReduce is
falling out of favor while Spark and the new Hadoop 2.0 are
taking its place. The challenges in technology selection (that is,
the selection of each system component) are non-trivial and
affect the selection of architecture patterns, data models,
programming languages, query languages, access methods, and
these in turn affect system performance, latency, scalability,
availability, consistency, modifiability, etc. Third, there are
uncharted territories. For example, the death of traditional
datawarehouses has been predicted, as they are believed to be
inadequate to support big data exploration. Many concepts
such as data lake, enterprise data hub, data refinery, Lambda
architecture [13] and Polyglot persistence [14] have been
recently proposed. And the integration of these new systems
with existing ones is a challenge for companies who cannot
afford or do not want to start anew.
Big data system development in enterprises has a relatively
short history: Big data became a megatrend in 2011 when IBM
created the hashtag #bigdata [16]. “2013 is the year of big data
experimentation, 2014 too.” Gartner reports [10]. As of the
end of 2014, big data deployments are still scarce [10] and
failures abound. According to a CIO Survey in 2013, 55% of
big data projects were not completed. Technical roadblocks and
inappropriate scope were among the major reasons for failure.
To help enterprises navigate through these uncharted waters
and be better equipped for their big data endeavors, our
research aims at developing a methodological framework for
big data system development, which departs from “small data”
system development—traditionally based on structured
(relational) databases or data warehouses. Our research centers
on how big data system can be cost effectively developed.
Specifically, we ask:
1. How does big data system development (processes and
methods) differ from “small” (traditional, structured) data
system development? How should requirement analysis,
architecting, data modeling (conceptual, logical and physical)
and testing be done differently?
2. Given the importance of architecture design in
developing complex big data systems, (this point will be
elaborated in Section II), how can existing software
architecture approaches be extended or modified to address
new requirements for big data system design?
3. How can data modeling/design methods in traditional
structured database/datawarehouse development be extended
and integrated with architecture methods for effective big data
system design?
In answering these questions, we have uncovered
methodological voids and identified practical guidelines. We
employed an empirical research approach to answer our
research questions as system development, be it big or small
data, cannot be separated from its organizational and business
contexts. Furthermore, because outsourcing is an important and
common means to realize a big data strategy, the research
method we chose is an embedded case study [18] with a global
software outsourcing company.
Given the changing technology landscape, we looked for an
outsourcer who was open to innovation, quick to adapt,
methodologically sound, and who has successfully deployed a
number of big data projects that can be triangulated as multiple
case studies of early adopters. We eventually selected a
company, Softserve Inc. (hereafter called SSV), who met all
our selection criteria and allowed us to conduct collaborative
practice research (CPR) [12] on outsourcer-specific as well as
general big data system development issues. The findings from
this embedded case study have contributed toward the creation,
refinement and validation of a big data system design method,
called BDD (Big Data-system Design).
The remainder of this paper is organized as follows:
Section II describes our research foundations highlighting
architecture design importance for big data system
development. Section III details our embedded case study
research method. Section IV presents the analysis of the 10 big
data projects developed by SSV. Section V describes our CPR
action research process and the resulting BBD method. Future
research called for is discussed and Section VI concludes.
II. RESEARCH FOUNDATIONS
Traditional structured or “small” data system development
benefits from the ANSI standard 3 tier DBMS architecture
which clearly defines data/program independence for data
system development [9]. The data system development process
is well-established, consisting of 7 phases: requirement
analysis, conceptual design, selection of DBMS, logical design,
physical design, prototyping/testing, implementation and
performance evaluation [9]. Typically, a RAD or agile
development lifecycle is employed for iteratively refining
performance, accommodating new requirements, and managing
system evolution. In each development phase there are models
and methods to assist DB designers and developers.
Relational data modeling and its query language, SQL, have
dominated the data management world in practice—relational
DBMS technologies have over 95% market share. The
popularity of the relational DB approach owes partially to the
existence of a conceptual data modeling tool. Entity
Relationship (ER) modeling has been a useful tool for
conceptual design and can be transformed into relational
45
logical design in 3rd normal form [9], which simplified and
facilitated the design tasks. The “5Vs” of big data have,
however, challenged these established processes:
1. The sheer volume of big data requires distributed and
parallel processing that most traditional DBMSs were not
designed for. Hadoop MapReduce, for example, is a simple
parallel processing algorithm. Traditional DB designers must
therefore now pay attention to architectural issues such as
parallelism, scaling up and scaling out for big data systems.
The volume of big data also challenges how traditional
prototyping is done so that it has fidelity of scale for testing.
2. The variety of big data created challenges in
modeling and metadata management for access to data.
NoSQL was invented to handle unstructured data. There are
four popular NoSQL technologies: document-oriented,
column-oriented, key-value store, and graph databases for
different types of data. These data models differ in their
performance and the CAP theorem spells out the major tradeoff
[2]: consistency vs. availability vs. tolerance to network
partitions. This is an ACID vs. BASE (Basically Available
Soft-state services with Eventual-consistency) tradeoff. There
are no “one size fits all” solutions so far. This, therefore, gave
rise to polyglot persistence [14]—solving a complex problem
by breaking it into segments and applying different database
models. It is then necessary to aggregate the results into a
hybrid data storage and analysis solution. Furthermore, the
query languages are also different for each data model. In
addition, NoSQL databases are often “schemaless” or use a
“flexible schema” which creates difficulty for queries that
traditional databases can conveniently perform using SQL.
SQL-on-Hadoop, New SQL, and Spark SQL are all efforts to
help address query issues, bringing back the familiar SQL
syntax. However, without the schema, data modeling and
metadata management is difficult to perform. How can
conceptual modeling be performed and mapped into each of the
NoSQL data models? It’s hard to separate logical and physical
design in the NoSQL world. The separation of concerns in
traditional DB development is now undone, and DB designers
need to know each NoSQL system in great detail (e.g.
programming language, data modeling, metadata management)
to develop the system. Obviously, this will result in systems
that are harder to modify and evolve.
3. The velocity of big data requires real-time processing
(ingestion, collection, preparation) of raw “fast” data to
facilitate quick decision making. Existing complex event
processing capabilities are challenged by the variety of data.
The Lambda architecture [13] was proposed to address the real-
time stream processing of raw data combined with materialized
views of stored data residing in batch-processing oriented
storage such as Hadoop HDFS. This illustrates the importance
of architectural design choices in big data system development.
The architecture design for traditional data systems has been
relatively straightforward. The biggest decisions—reference
architecture and major patterns—in traditional DB design are
already made: the N-tier client-server architecture is the norm.
4. The veracity of big data brings challenges in data
validation, modeling data context and governance. Big data
comes from everywhere—IoT, social media, web log files,
etc.—and much of it could be “dirty” or illegal data whose
collection would violate data privacy laws. In traditional DB
system development, the data collection is mostly internal and
validation is relatively simple as data are structured. In big
data, unstructured data needs contextual information for
interpretation. This added complexity also highlights the need
for architecture design in the early stages for understanding the
data sources (and the cleanliness of each).
5. Extracting value from big data requires capabilities
beyond the traditional data warehouse. Every Fortune 500
company now has a datawarehouse. Data mining and analytics
utilizing structured data in datawarehouses have 20 years of
established practice. Relational datawarehouse design is
relatively straightforward even though the design of ETL
(extract, transform, load) processes is not trivial. The 5th V of
big data requires integration of existing master data, internal
operational data as well as external reference data. The prior
4Vs of big data mean complex integration for extracting value.
How to bridge the new and old also highlights the criticality of
architecture choices in big data system development.
Our research goal is to develop a big data system
development methodology to help enterprises address these
challenges which are further complicated by rapid technology
changes. Our research centers on the “how” questions. Our first
research question is how to address these big data system
development challenges cost-effectively to achieve business
goals. We examine how big data system design tasks would be
carried out in each design phase of the traditional DB
development life cycle. As argued above, architecture design is
critical to big data system development, much more so than for
traditional data systems. Our second research question
therefore focuses on how to extend existing architectural
methods for big data system design. Our third research question
is the combination of the first two: How can data modeling and
design methods in traditional DB development be extended and
integrated with architecture methods for effective big data
system development. Based on a design science approach, we
employed an empirical case study method to identify issues in
practice, and collaboratively explored, developed and validated
the results with practitioners. We now detail our research
method.
III. RESEARCH METHOD
An embedded case study is a case study containing more
than one sub-unit of analysis [18]. The identification of sub-
units allows for a more detailed level of inquiry. A case study
research method is an empirical inquiry aimed at revealing
aspects of a contemporary phenomenon inseparable from real-
life context, and thus, difficult to replicate in a laboratory
environment. Our motivation for using this method rests on the
following: 1) case studies are well suited for “how” research
questions, which are our main research foci; 2) exploratory
case studies are particularly appropriate for initial evaluation as
they allow the course of the study to be adjusted along the way
to account for what is learned, and 3) case studies are also the
46
most satisfactory approach when there are many variables of
interest and few data points, and where resources do not permit
enough replications to isolate individual variables.
We selected a company in the outsourcing industry for the
following reason. Finding talent, finding the right tools, and
understanding platforms have been cited as the top challenges
for big data system development [8]. These challenges,
combined with the desire for fast deployment, have forced
many firms to turn to outsourcing companies for their big data
solutions. Outsourcing helps organizations focus on their core
competencies while mitigating shortages of skills and expertise
in the areas where they want to grow. Furthermore, big data
professional services have been in great demand and have
grown rapidly, from $2.8B in 2011 to $10.1B in 2014. A
survey of the big data market [11] showed that professional
services made up the largest market segment in 2013. This is
not unexpected. Enterprise practitioners require significant help
from professional services organizations to identify use cases,
design and deploy systems, and integrate the technology and
output into business processes and workflows. This is predicted
to grow to $17.2B in 2017, exceeding all other categories
(hardware, software, cloud, etc.) in the big data market.
Our case company, SSV, provides global outsourcing
solutions including Cloud, Mobile, UI/UX, and Big
Data/Analytics. Since its founding in 1993, the company has
grown to over 3,500 employees and opened multiple offices in
the US and Europe. Since its inception, SSV has successfully
completed thousands of outsourced projects.
The selection of SSV was somewhat opportunistic. SSV
was suitable for our research objectives as it 1) has successfully
deployed many big data projects that can be embedded in this
case study as second-order multiple studies and 2) is open to
collaboration and accessible.
Our research method in this case study is a particular form
of action research called collaborative practice research (CPR)
[12]. Our research involved a collaboration between
researchers and practitioners to devise an effective big data
system development method. The authors are collectively:
creators of the architectural methods in use at SSV, big data
system architects, and system developers. SSV has been
employing Attribute Driven Design (ADD) [1] from the start
for their projects. In early 2014, SSV invited the researchers to
participate in the training of their architecture team. Thus, the
researcher and practitioner teams came together to evolve the
ADD method into a method appropriate for big data system
development called BDD. The researchers have first-hand
knowledge of the initiation, planning, and execution of ADD
and have worked with other practitioners to improve the
method. The practitioner team has managed all the big data
projects completed by SSV and had full access to all personnel
involved in the projects as well as documentation, including
contracts, meeting minutes, archived notes of workshops, and
program files associated with each project.
There are two levels of analysis in our embedded case study
research. The primary unit of analysis is the big data system
development methodology employed by SSV. The second level
of analysis is the set of challenges and solutions in each of
SSV’s big data projects. Data was collected for analysis from
semi-structured interviews, documentary sources, workshops,
meeting notes, e-mail communications, and dialogues with
practitioners. We followed the steps of the CPR methodology
to create the BDD method. An iteration in the CPR process
include 9 steps: 1) Appreciate problem situation 2) Study
literature 3) Develop framework 4) Evolve Method 5) Action
6) Evaluate experiences 7) Exit 8) Assess usefulness 9) Elicit
research results Six projects started prior to 2014 (see Table 1)
were used to understand the problem situation and six projects
started in 2014 or still on-going were used for developing the
research framework, evolving methods, action, evaluation,
assessment and eliciting research results. We have gone
through several iterations of the CPR cycle to create the BDD
method.
IV. CASE ANALYSIS & FINDINGS
SSV started their big data projects in 2010 before the big
data megatrend took off. In Table 1, 10 of SSV’s big data
projects are listed in chronological order. The order is
important to show the progression of technology available and
what was selected at the time of development (or
redevelopment). In analyzing these 10 projects, we found:
1. 50% of the client companies are large enterprises (>5000
employees and annual revenue >US$1 billion), 50% are
successful Internet companies with annual revenues in the
range of US$8-180 million. This is not surprising, given
that big data requires substantial upfront investment.
2. All of the client companies have increased their revenue
after the deployment of the big data projects. This
correlation invites two possible explanations: 1) earlier
adopters of big data technology are more risk-taking and
innovative, hence they will always take advantage of new
technology to increase their competitive advantage; 2) the
successful big data deployment indeed solves the client
companies’ problems and achieves their business goals,
resulting in increased revenue.
3. The 10 projects range from network security, network
administration, email security and spam prevention, social
marketing platform, e-coupon site, web analytics platform,
cloud app development platform and operational
intelligence for healthcare fraud detection. These
application areas are critical components of the clients’
enterprise systems. This indicates the changing role of
outsourcing companies, which have increasingly become
strategic partners of enterprises. It also showed the early
adopters of big data concentrates on operational intelligence
and customer intelligence.
4. Cloud based big data deployment is important to satisfy
elasticity requirements and hence was adopted by 7 out of
10 projects. The deployment in the cloud is straightforward
with US companies but not with European or multi-national
companies due to data governance laws. The public cloud,
multi-tenant environment is a constraint on performance.
5. The project scope and use cases are substantial in all 10
cases but have been well-defined in the discovery phase.
This is typical in outsourcing agreements and may imply a
47
new big data development lifecycle for enterprises. (See
Figure 1).
6. Initially, the data volume (and then velocity) were the
major concerns of the clients. As the technology
progressed, the design concerns have shifted the data
variety and veracity issues. To support data analysis with
good performance is a common design requirement for all
projects.
7. Key requirements center on cost and quality attributes
(scalability, availability, consistency, elasticity, low latency,
etc.). Initially, open source solutions were important to
reduce cost for SSV’s clients, but more expensive
components could be selected for performance reasons.
8. Design challenges throughout the projects center around
technology selection issues that impact on both cost and the
quality attribute impacts of the selected technologies.
9. SSV, in their early projects, learned that architecture
choices cannot be separated from technology selection.
This realization has motivated the collaborative
development of the BDD method. (See next section and
Figure 1).
10. Enterprise (traditional, legacy) datawarehouses are not
going away soon. The integration of the new and the old is
a major design challenge in the vast majority of the
projects.
11. A majority of the projects involved updated their
technology (which was, in some cases, less than one year
old), some from the same vendor, some from changing to a
different platform or vendor.
12. Data modeling to support real time analytics were most
challenging. Lambda architecture was employed but the
selection of technology for each component to instantiate
the reference architecture remains a challenging task.
13. Data modeling is intertwined with technology selection.
V. ACTION RESEARCH AND RESULTS
In the projects prior to 2014 (first iteration of the CPR
process), to address design challenges, SSV employed an
architecture method called ADD that was developed by the
Carnegie Mellon Software Engineering Institute (SEI). The
first version of ADD (ADD 1.0) was published in January 2000
and the second version (ADD 2.0) was published in November
2006 [1]. ADD is, to our knowledge, the most comprehensive
architecture design method available. SSV’s architects took
SEI training classes and are well-versed in ADD 2.0.
When ADD appeared, it was the first design method to
focus specifically on quality attributes and their achievement
through the selection of architectural structures and their
representation through views. Another important contribution
of ADD is that it includes architecture analysis and
documentation as an integral part of the design process. In
ADD, design activities may include refining the sketches that
were created during early design iterations to produce a more
detailed architecture, and also performing a more formal
evaluation of the design, perhaps using a method such as the
ATAM [3], a well-known architecture evaluation method
developed by one of the researchers.
 As discussed in Section II, architectural choices for big data discovery with big data before the design process. This is very
systems and the technologies chosen for realizing conceptual different from small data system development. In addition,
system components are critical design considerations. Based on according to a multi-case study of 23 large enterprises we
our literature review and experiences and as shown in SSV’s conducted, in the use case development stage, prototyping for
early big data projects, big data system development rests business value demonstration were often attempted. In the
largely on the “orchestration” of a set of technologies. While design phase, prototyping and scale-up testing with chosen
ADD 2.0 was useful for linking quality attributes to design technology may also be performed. However, due to the
choices, there were three shortcomings that needed to be scaling problem in proving prototype fidelity, we are
addressed for big data system development: recommending using architecture analysis techniques to
First, ADD 2.0 guides the architect to use and combine tactics replace or compliment the prototyping.
and patterns to achieve the satisfaction of quality attribute
scenarios. But patterns and tactics are abstractions. The method
did not explain how to map these abstractions to concrete
implementation technologies.
Second, ADD 2.0 was invented before agile methods became
popular and thus did not offer guidance for architecture design
in an agile setting.
Third, ADD 2.0 was meant to be general, and hence
technology agnostic. While this is good for generality, ADD
2.0 did not explicitly promote the (re)use of reference
architectures which are an ideal starting point for big data
system architects.
ADD 3.0—the latest version—was catalyzed by the
creation of ADD 2.5.1 One researcher published ADD 2.5 in
2013 [4] which advocated the use of frameworks such as JSF,
Spring, Hibernate, and Axis2 as first class design concepts.
This change was to address ADD 2.0’s shortcoming of being
too abstract to easily apply. ADD starts with architecturally
significant requirements—drivers and constraints—
systematically links them to design decisions and then links
Figure 1: BDD process model
those decisions to implementation options available via
frameworks. For agile development, ADD 3.0 promotes design
In BDD, reflecting the paradigm shifts and addressing 5V
iterations, with each iteration retaining the 6 design steps of
requirements, a new process integrates architecture analysis,
ADD 2.5. In addition, ADD 3.0 explicitly promoted (re)use of
data modeling techniques and technology selection
reference architectures and was paired with a “technology
systematically in the Requirement Analysis (RA) as well as the
catalog”. This catalog includes tactics, patterns, frameworks,
Design stage. As shown in Figure 1, business goals and
reference architectures, and technologies. Along with each
design constraints/concerns/drivers are captured in RA Steps 1-
technology is a rating of its quality attributes such as
2. Big data scenarios focusing on “futuring” [7] are generated
scalability, modifiability, availability, performance, etc.
along with quality attribute scenarios in RA Steps 3-4. There
In the second iteration of the CPR process, SSV adopted
are 6 elements in a quality attribute scenario: source, stimulus,
ADD 3.0 and used their own experiences, prototyping, and
environment, artifacts, response, and response measure. When
benchmarking to obtain ratings for each quality attribute of
the stimulus are related to data, a big data template is used.
each technology in the catalog. With this catalog, SSV’s design
The big data template includes 14 data architecture elements
knowledge is preserved and reused for other projects.
(including data source quality, data variety, data volume,
ADD 3.0 facilitates the technology selection and agile
velocity, read/write frequency, time to live, queries, OLTP or
orchestration for big data system development, however, the
OLAP, etc.) are captured as inputs for each data source. All
data modeling aspect (hence, the selection of data management
queries are considered in the design input. The big data
components) requires additional effort. As argued in Section II
template is used for recording these design inputs. Each design
and based on the development experiences in SSV, data
input has a direct implication on the subsequent architecture
modeling tasks are tightly coupled with architectural design. In
choices, data model selection, technology selection and data
the third iteration of CPR process, ADD 3.0 was thus
access patterns. In BDD, expert rules are developed and used
integrated with extended data modeling analysis techniques
in linking these design inputs to resulting system architecture
created to form the BDD method, as shown in Figure 1. Note
and data model selections, access pattern and query design for
that the design process of BDD starts with an enterprise’s vaule
meeting quality attributes requirements.
In BDD, Design (Steps 1-3) starts with choosing a
1
Note that this is our own coding notation; the 2.5 number is not used reference architecture, forming architecture landscape [7]
elsewhere. (encompassing all architecture choices from big data

48
scenarios), and sketching data flows in the architecture
landscape to form a DFD (Data Flow Diagram) context
diagram. In Steps 4-9, the entire system will be decomposed
and modeled in detail in a number of iterations. Each design
iteration will establish iteration goals (Step 4) and then select a
quality attribute or a driver or a concern or a system component
to focus on (Step 5). The selection of design concepts (such as
patterns and tactics) and selection of data models are performed
simultaneously in Step 6. In Step 7, the instantiation of
architectural elements includes selection of databases, analytics
platforms and other system components. BDD follows ADD’s
emphasis on documenting design decisions. In Step 8, BDD
added metadata as part of design documentation. All
architecture views are then sketched and metadata is modeled.
In each iteration, the design will be evaluated based on the
iteration goals in Step 9. Once the design iterations are
complete, an architecture analysis will be performed on the
entire system architecture to discover risks and identify design
tradeoffs before implementation in Step 10. In BDD, we
employ BITAM method [6], which extends ATAM, to include
business and IT alignment assessment.
Our collaborative development and validation of BDD
continues. The successful Phase 1 development of projects
(Cases 7-10) validated the benefits of the initial version of
BDD. Our method requires architects to have data modeling
experience or to work closely with database designers. The
continued development of the BDD method suggests several
future research directions that may help reduce the design
complexity:
1. Rule base for design concepts and data model selection;
2. Automation for technology catalog update;
3. New design patterns and tactics for big data systems;
4. Decision support systems for technology selection;
5. Conceptual modeling of NoSQL databases;
6. Metadata management tools for big data.
VI. CONCLUSION
The embedded case study method, which strikes a balance
between relevance and rigor [18], is most appropriate to
address our research questions for moving towards an effective
big data system development method even though the
generalization of the results beyond the cases implemented by
SSV may be limited and researchers’ biases are inevitably
embedded in the analyses. The research method we employed
allowed us to help SSV serve their clients while collaboratively
developing a new method and gathering insights for improving
big data system development knowledge and practice.
The output of our collaborative practice research is the
BDD method. Big data system development is dramatically
different from traditional database system development. Our
research both illuminates the challenges for big data system
development that stem from the 5Vs of big data and crystallizes
the importance of architecture design choices.
We collaboratively developed the BDD method with SSV,
extending the architecture design method ADD 3.0. The
projects successfully developed by SSV not only provided
49
insights and lessons learned for big data method development
but also an empirical validation. The BDD method is the first
attempt to systematically combine architecture design with data
modeling approaches to address big data system development
challenges. The use of reference architectures and a technology
catalog are advancements to architecture design methods and
are proving to be well-suited for big data system architecture
design and system development.
REFERENCES
[1] ADD (Attribute-Driven Design Method), SEI,
http://www.sei.cmu.edu/architecture/tools/define/add.cfm.
[2] Brewer, E. , "CAP Twelve Years Later: How the "Rules" Have
Changed", Computer, vol.45, no. 2, pp. 23-29, Feb. 2012.
[3] Bass, L. Clements, P. and Kazman, R. Software Architecture in
Practice (3rd ed.). Pearson, 2013.
[4] Cervantes, H., Velasco-Elizondo, P., Kazman, R. “A Principled
Way to Use Frameworks in Architecture Design Software,”
IEEE Software (Volume:30 , Issue: 2 ) 46 - 53 2013
[5] Cervantes, H., and Kazman R., Designing Software
Architectures: A practical approach, Addison-Wesley, 2015,
forthcoming.
[6] Chen, H-M., Kazman R, Garg, A. "Managing Misalignments
Between Business and IT Architectures: A BITAM Approach,"
Journal of Science of Computer Programming, Vol. 57/1, 5-26,
2005
[7] Chen, H-M and Kazman R. “Architecting for Ultra Large Scale
Green IS,” Proceedings of GREENS 2012 with the 34rd ICSE,
Zurich, Switzerland, June 2-9, 2012.
[8] “CIOs & Big Data,” http://visual.ly/cios-big-data, retrieved Sep.
2014
[9] Elmasri R. and Navathe S., Fundamentals of Database Systems,
6th edition, Addison Wesley, 2010.
[10] Gartner, "Survey Analysis: Big Data Investment Grows but
Deployments Remain Scarce in 2014."
http://www.gartner.com/document/2841519. September 2014
[11] Kelly, J., “Big Data Vendor Revenue and Market Forecast 2013-
2017”,
http://wikibon.org/wiki/v/Big_Data_Vendor_Revenue_and_Mar
ket_Forecast_2013-2017, Feb 12, 2014
[12] Mathiasen, L. "Collaborative practice research," Information
Technology & People (14:4), 321-345, 2012.
[13] Marz, N and Warren, J. Big Data: Principles and best practices
of scalable realtime data systems. Manning Publications, 2013.
(advanced edition)
[14] Sadalage P.J., Fowler M. NoSQL Distilled: A Brief Guide to the
Emerging World of Polyglot Persistence, Pearson Education In.
2013
[15] Singh, A. “Is Big Data the New Black Gold?” Wired,
http://www.wired.com/2013/02/is-big-data-the-new-black-gold/,
Feb.13. 2013
[16] Winshuttle.com, “Big Data and the History of Information
Storage.” http://www.winshuttle.com/big-data-timeline/
retrieved Nov. 2014
[17] Yonego, J.T. “Data Is the New Oil of the Digital Economy”,
Wired, http://www.wired.com/2014/07/data-new-oil-digital-
economy/, July 23.2014
[18] Yin, R. K. Case study research, design and methods, 5th ed.
Newbury Park: Sage Publications, 2009

Case #
1 analysts to improve intrusion
Network Security,
Intrusion Prevention
US MNC IT corp.
Employees > 320,000
2 developed set of anti-spam
Anti-Spam Network
Security System
US MNC Networking
equipment corp.
employees > 74,000
3 Platform for Conversion Funnel
Online Coupon Web
Analytics Platform
US MNC
“14 Revenue > US$200M
4 Platform for ROI measurement
Social Marketing
Analytical Platform
US MNC Internet
marketing (user reviews)
‘14 Revenue > US$ 168M
5 building custom mobile
Cloud-based Mobile
App Development
Platform
US private Internet Co.
Funding > US$100M
6 platform to improve sales and
Telecom E-tailing
platform
Russian mobile phone
retailer; 2013: 108B rubles
Table 1: 10 Outsourced Big Data Projects at Softserve
Business goals Start Big data
• Provide ability for security
detection techniques;
• Observe traffic behavior and
make infrastructure
adjustments:
• Adjust company security
policies
• Validation of the new
rules against the large training
set of known emails
• Detection of the best anti-spam
rules in terms of performance
and efficacy
• In-house Web Analytics
Analysis, marketing campaign
optimization, user behavior
analytics
• clickstream analytics, platform
feature usage analysis
• Build in-house Analytics
and performance analysis of
every product and feature by
the e-commerce platform;
• Provide analysis on end-users
interaction with service
content, products, & features
• Provide visual environment for
applications
• Charge customers by usage
• Analysis of platform feature
usage by end-users and
platform optimization
• Build an OMNI-Channel
operations
• analyze all enterprise data
from multiple sources for real-
time recommendation and
cross/up sales
Technologies Challenges
Late • Machine generated data -
2010, 8.5 7.5BLN event records per
month day collected from IPS
devices
• Near real-time reporting
• Reports which “touch”
billions of rows should
generate < 1 min
2012- • 20K Anti-spam rules
2013 • 5M email training set
• 100+ Nodes in Hadoop
Clusters
2012, • 500 million visits a year
Ongoing • 25TB+ HP Vertica Data
Warehouse
• 50TB+ Hadoop Cluster
• Near-Real time analytics
(15 minutes is supported
for clickstream data)
2012, •Volume - 45 TB
ongoing • Sources - JSON
• Throughput - > 20K/sec
• Latency (1 hour – for
static/pre-defined reports
/real-time for streaming
data)
2013, 8 • Data Volume > 10 TB
month • Sources: JSON
• Data Throughput > 10K/sec
• Analytics - self-service, pre-
defined reports, ad-hoc
• Data Latency – 2 min
End of • Analytics on 90+ TB (30+
2013, TB structured, 60+ TB
(only unstructured and semi-
discovery structured data)
• Elasticity: through SDE
)
principles
•ETL - Talend • High throughput, different device data
•Storage/DW – InfoBright EE, schemas (versions)
HP Vertica • keep system performance at required
•OLAP – Pentaho Mondrian level when supporting IP/geography
•BI – JasperServer Pro analysis: avoid large table join.
• Keep required performance for complex
querying over billions rows
• Vanilla Apache Hadoop • MapReduce was written on Python and
(HDFS,MapReduce,Oozie,Z Hadoop Streaming was used. The
ookeeper ) challenge was to optimize jobs
• Perl/Python performance.
• SpamAssassin • Optimal Hadoop cluster configuration to
• Perceptron maximize performance,and minimize
Map-Reduce processing time
• Data Lake - (Amazon EMR) • Minimize transformation time for semi-
/Hive/Hue/MapReduce/Flu structured data
me/Spark • Data quality and consistency
• complex data integration
• DW: HP Vertica, MySQL
• ETL/Data Integration – • fast growing data volumes,
custom using python • performance issues with Hadoop
• BI: R, Mahout, Tableau Map/Reduce (moving to Spark)
• Lambda architecture • Hadoop upgrade – CDH 4 to CDH 5
• Amazon AWS, S3 • Data integrity and data quality
• Apache Kafka, Storm • Very high data throughput caused a
• Hadoop - CDH 5, HDFS(raw challenge with data loss prevention
data), MapReduce), (introduced Apache Kafka as a solution)
Cloudera Manager, Oozie, • System performance for data discovery
Zookeper (introduced Redshift considering Spark)
• HBase (2 clusters: batch • Constraints - public cloud, multi-tenant
views, streaming data)
• Middleware: RabbitMQ, • schema extensibility
Amazon SQS, Celery • minimize TCO
• DB: Amazon Redshift, RDS, • achievimg high data compression
S3 without significant performance
• Jaspersoft degradation was quite challenging.
• Elastic Beanstalk • technology selection: performance
• Integration: Python benchmarks and price comparison of
Redshift vs HP Vertica vs Amazon RDS.
• Hadoop (HDFS, Hive, • Data Volume for real-time analytics
HBase) • Data Variety: data science over data in
• Cassandra different formats from multiple data
• HP Vertica/Teradata sources
• Microstrategy/Tableau • Elasticity: private cloud, Hadoop as a
service with auto-scale capabilities

7 platform that allows enterprise
Social Relationship
Marketing Platform
US private Internet Co.
Funding > US$100M
8 mobile, and social channels
Web Analytics &
Marketing Optimization
US MNC IT consulting co.
(Employees > 430,000)
9 availability, performance,
Network Monitoring &
Management Platform
US OSS vendor
Revenue > US$ 22M
• Build social relationship
brands and organizations to
manage, monitor, and measure
their social media programs
• Build an Analytics module to
analyze and measure results.
• Optimization of all web,
• Optimization of recomm-
endations for each visitor
• High return on online
marketing investments
•Build tool to monitor network
events and configuration.
• Integrate data storage and
collection processes with one
web-based user interface.
•IT as a service
2013 • > one billion social • Cassandra • MySQL 5.6/5.1 • Minimize data processing time (ETL)
ongoing connections across 84 • Elasticsearch • Implement incremental ETL, processing
(redesign countries • SaaS BI Platform - and uploading only the latest data.
2009 • 650 million pieces of social GoodData
content per day • Clover ETL, custom in Java,
system)
• MySQL (~ 11 Tb) Cassandra • PHP, Amazon S3,Amazon
(~ 6Tb), ETL (> 8Tb per day) SQS
• RabbitMQ
2014, • Data Volume > 1 PB • Vanilla Apache Hadoop • Hive performance for analytics queries.
Ongoing • 5-10 GB per customer/day (HDFS,MapReduce,Oozie,Z Difficult to support real-time scenario
(Redesign • Data sources – clickstream ookeeper ) for ad-hoc queries.
2006- data, webserver logs • Hadoop/HBase • Data consistency between two layers:
• Aster Data raw data storage in Hadoop and
2010
• Oracle aggregated data in relational DW
system)
• Java/Flex/JavaScript • Complex data transformation jobs
2014, •collect data in large • MySQL • High memory consumption of HBase
Ongoing datacenters (each: • RRDtool when deployed in a single server mode
(Redesign gigabytes to terabytes) • HBase
2006 •real-time data analysis and • Elasticsearch
monitoring (< 1 minute)
system)
• types of devices: hundreds

10
Healthcare Insurance
Operation Intelligence
US health plan provider
Employees> 4,500
Revenue> US$10B
• Operation cost optimization for
3.4 million members
• Track anomaly cases (e.g.
control schedule 1 and 2 drugs,
refill status control)
• Collaboration tool between
65,000 providers (delegation,
messaging, reassignment)
2014, • Velocity: 10K+ events per
Phase 1: second
8 month, • Complex Event Processing -
ongoing pattern detection,
enrichment, projection,
aggregation, join
• High scalability, High-
availability , fault-tolerance
• AWS VPC
• Apache Mesos, Apache
Marathon, Chronus
• Cassandra
• Apache Storm
• ELK (Elasticsearch,
Logstash, Kibana)
• Netflix Exhibitor •Chef
• Technology selection constraints by
HIPAA compliance: SQS(selected) vs
Kafka
• Chef Resource optimization:
extending/fixing open source
frameworks
• 90% utilization ratio
• Constrains: AWS, HIPAA

50