Big Data: Concept, Applications, & Challenges
Novan Zulkarnain
Bina Nusantara University
Indonesia
novan@binus.ac.id
Abstract—Any organization either public or private rely on
accurate data analytic to take decisions. The utilization of big
data is pivotal for driving organization extract value from a large
amount of data. Big data is a method and technique to retrieve,
collect, manage and analyze a very huge volume of both
structured and unstructured data that is difficult to process using
traditional database which entail new technologies and technique
to analyze them. This paper covers the big data with the
intentions to find out the concept, its applications and challenges
from literature analysis as well as discussing and reviewing
interpretations from these findings along with possible
recommendations.
Keywords—big data; big data analytics; big data application;
Social Networks
I. INTRODUCTION
Big data is a term for large and complex sets of data in
which traditional methods of processing data are insufficient.
This essay will cover about the big data as a whole, with the
intentions to find out the origins of the concept, its applications
and challenges from findings resulted from cited sources and
publications as well as discussing and reviewing interpretations
from these findings along with possible recommendations. Big
data need to be analyzed to gain its value either the trends,
patterns or behavior anything related to the people or
customers. Though data comes from the rapid growth of
volume, yet it does rapidly and efficiently processing those
data refers to velocity of data processing [20]. Big data analytic
leads to more precise analysis thus helps to bring more accurate
decision-making and better performance. Big data are collected
either through structured or unstructured data sources (online
or offline). Unstructured data can come from social media
(Facebook, Instagram, Twitter posts, etc) [25]. While,
structured data sources can come from internal database of
organization. In businesses, both sources are used to
understand the patterns of the customers. Indeed, an
organization nowadays relies the fact that any data could be
analyzed and used to reveal patterns of their customers. In
other words, big data will help the organization to understand
the behavior of their customers and use it to win a competition.
Even though business organizations are still in early stage
of perceiving big data as an asset, public agencies are still
struggling with the issue of open data, whereas science and
technology are exploring the potentials of big data and its
innovation, yet general public are keep producing a huge
amount of data in daily basis poses challenges for all
organizations. An organization faces the fact that the reality of
big data that can affect their competitiveness. The aim of this
978-1-5090-3352-2/16/$31.00 ©2016 IEEE 16-18 November 2016, Aston Tropicana Hotel, Bandung, Indonesia
2016 International Conference on Information Management and Technology (ICIMTech)
Page 307
Muhammad Anshari
Universiti Brunei Darussalam
Brunei Darussalam
anshari.ali@ubd.edu.bn
study is to examine fundamental concept, applications, and
challenges that are closely related to big data in organization.
This paper is organized as follows: the next section discusses in
more detail the literature analysis on big data, Section 3
explains research methodology, the discussion is in Section 4,
and Section 5 is the conclusion.
II. LITERATURE REVIEW
A. Definition
Historically, data is accumulated in organization by
entering data into computer system, however, as the Internet of
Things (IoT) evolve users can generate their own data. The
excessive use of Internet will increase by IoT due to the
integration of all objects via embedded systems. This leads to a
high distribution of network of devices communicating with
people as well as other devices [26]. From these network of
devices, huge portions of raw data can be generated and stored
as big data. This data comes from anything and anywhere such
as customers, smart mobile devises, sensors, purchase
transaction records, social media, digital images, CCTV, GPS,
audio that can be used to gather specific information. The
definition of big data is unclear [8], yet there are attempts to
define it. One definition about big data is that it refers to sets of
data where their size is too advanced, too complex for typical
data software tools to use their abilities for analysis or
management etc. [13]. However the definition of big data is
intentionally subjective and as it incorporates a non-fixed
definition depending factors such as advancing technology in
which would result in the size of datasets that would qualify as
big data to increase [13]considering how as of 2012, big data
ranges from a few dozen terabytes to many petabytes of data
[9]. Furthermore, Big Data can be differentiated with 3 key
differences i.e. the 3 V’s. They are Volume, in which
organizations collect data from different sources with different
sizes of data considering how in 2012, 2.5 Exabyte are created
each day [15]. Velocity, where many applications priorities the
speed of data creation over its volume which allows real time
information to be processed and Variety, where big data can
take in different forms such as messages, updates and even
pictures [15].
Big data is defined as a massive amount of data, very
quickly in processing from many different forms to support
decision making [9]. Therefore, it is widely known with the
volume, velocity and variety. Other definition is a massive
volume of structured and unstructured data that is gathered and
analyzed through new methods that can produce value for the
organization [6]. Social media is a part of big data, in which the
number of people using social media are high in number. It had
shown that 3.419 billion are internet users whereby 2.307
billion are active social media users [4]. Analyzing those
amount of data requires a new tools and techniques which is
beyond a traditional processing tools [1, 17] and exceeds the
processing capacity of conventional database systems. It is
based on the ability which able to harness data in different
ways to generate effective understanding and utilize values
[20]. Big data grows exponentially, accumulates quickly, and
combine multiple data types. Therefore, organization should
use advance data analytic to process them [8, 25].
In short, there are many authors defines big data but
majority of them has a term for big data and that term is
explosion of data. Meanwhile, big data as a non-sampled data
which characterized by the invention of databases from various
electronic primary sources other than statistical interpretation
[10]. Whereas, another author has believed that big data is
different because it can generate on a vast scale through social
media or not such as online interaction between people,
transactions between people and systems and sensor enabled
machinery [19].
B. Characteristics & Model
Big data have brought about by the increased data with a
3Vs model, i.e., the increase of Volume, Velocity, and Variety,
[11]. Gartner and many other enterprises, including IBM [27]
and Microsoft still used the “3Vs” model to describe big data
[16]. In the “3Vs” model, Volume means collection of massive
data scale becomes increasingly huge (Figure 1). Velocity
refers to the timeliness of big data, specifically, data collection
and processing analysis, must be rapidly and timely conducted
to gain maximum value. Variety means many types of data
including structured and unstructured data such as audio, video,
text, and traditional structured data (CRM, ERP, SCM). Both
unstructured and structured data can be analyzed using
Hadoop. Hadoop is able to analyze a large volumes of data as
well as mining it to reveal the pattern.
Fig. 1. Big Data Model
There are many benefits of big data. Firstly, an organization
come up to more accurate information as big data can discover
value, connections, trends, and pattern. Secondly, it improves
decision making process since it is richer in term of data
gathering. Thirdly, big data can reduce the maintenance costs.
A certain type of equipment is likely to wear out after years, so
it costs a lot to replace every technology, even the important
files or documents are left in them. Furthermore, it keeps the
data save by detecting the internal threats. Forth, it improves
978-1-5090-3352-2/16/$31.00 ©2016 IEEE 16-18 November 2016, Aston Tropicana Hotel, Bandung, Indonesia
2016 International Conference on Information Management and Technology (ICIMTech)
Page 308
customer service since an organization can understands
customer’s preferences. For instance, Netflix can detect traffic
details for various devices, spot problems in the area and add
systems that can help the future demand [14]. They are also
able to get more vision of their customer’s desire [Ibid.].
Lastly, value is an outcome of big data analytic so that is able
to make the right decision for his business.
TABLE I. DATA ANALYSIS IN PERIOD
Source [7]
Naming Period Key points
Decision 1970-1985 Analyzing some structured data to
Support support decision making.
Executive 1980-1990 Data analysis for senior executives to
Support take action.
Online 1990-2000 Application for analyzing
analytical multidimensional data tables.
processing
(OLAP)
Business 1989-2005 Applications to support data-driven
intelligence decisions, with emphasis on reporting.
Analytic 2005-2010 Statistical and mathematical modelling
analysis for decision.
Big Data 2010-Present Analysis of structured and unstructured
of very large data, fast-moving data, in
short period of time.
The table above shows the evolution of data analysis for
decision making over the last 45 years. It has transformed from
decision support, executive support, online analytical
processing, business intelligence, analytics and now to big data
(see Table 1).
C. Big Data Analytics
Big data analytic is a process of discovering patterns and
trends from a large amounts of data to extract its value and
correlations [21, 23]. Based on the report done [22], the stages
that are needed to be done in data retrieval for big data are as
followed: 1) Data acquisition: Data acquired from the medium
where data generation is growing at an exponentially rate.
Although, the data that is being produced continuously mostly
made up of unprocessed data that are useless and due to its
unstructured form, selecting and discarding unneeded data can
be quite challenging. 2) Data extraction: Majority of the
acquired raw data are not useful. Hence, deciding which data
are needed to be kept and which one should be discarded is a
difficult task to perform as well as there is an abundance of it.
3) Data collation: Most of the time, utilizing data from one
sample is inadequate to be used in an analysis or prediction. So,
data is retrieved from different sources and would be combined
or superimposed so that a bigger and more detailed picture is
formed. From this point, the data can be analyzed more
properly. 4) Data structuring: It is important for the analyzed
data to be organized in a structured form. This enables the
retrieval of information to be easier. 5) Data visualization:
Usually, case studies would concentrate on certain part of an
area or region. The data that is being pulled from these areas
are then analyzed and converted into a more specific and visual
format. 6) Data Interpretation: This is where valuable
information will be extracted. There are two types of
information that can be acquired: Retrospective Analysis and
Prospective Analysis. Retrospective involves gaining insights
from the past events and actions. Prospective Analysis is
distinguishing patterns and discovering trends for future based
off the data that was recorded.
III. METHODS
Referred journal publications were used to find all sources
for referencing, with publications ranging from 2010 to 2015
mostly in English languages. The methods employed of content
analysis of the research on big data in any organizations either
public or private. Thirty literature reviews were analyzed from
research of big data published in peer-reviewed journals. Then,
the results were clustered into a thematic of technical, value
creation, and possible adoption to an organization.
IV. APPLICATIONS
Big data gives tremendous impact on business, ranging
from consumers to supply chains operation and companies
[18]. While, big data applications are management and
processing of distributed data, and it can be a new tool for data
analysis and visualization [19]. Big data is useful for business
organizations to help those gaining deeper insights of
customers’ habits and behaviors. Because big data is a growing
concept, big data is present in variety of areas. There are many
applications of big data despite it being a relatively new thing
in the public. For example, by 2011, 235 terabytes was
collected by the US Library of Congress and around 30 billion
content is shared in Facebook every month. By 2016, it is
predicted that around $7.4m will be spend on data-related
initiatives and $13.8m in enterprises [13].
The usage of mobile service provides data that can be used
to improve public sector understanding of educational needs
and knowledge gaps, permits more targeted and timely
capability to circulate critical information. Data from e-
commerce can give a deep understanding into spending and
saving habits across sectors. Online transaction histories
provide a credit histories and allow the individuals to make
loans and other credit-based financial services. Healthcare
organizations are able to gather information regarding disease
trends and treatments for their patients. Big data can be used to
make a large datasets with treatments and comparison of the
outcomes so that it can be made efficiently and cost effective
manner. In addition, big data can be used to develop the auto
alert products and services. For instance, the usage of data
which is automatically obtained from the sensors embedded in
products can provide after-sales service offerings such as
proactive maintenance or alert to avoid failures in products,
then the alert is synchronize with the smart mobile device of
the user.
There are at least three components in order to fully operate
big data. First, tools and technologies are important to process
big data. Tools and applications will gather, store, analyze, and
display the analytics or results. Hardoop is one of the platform
for big data analytic. Many organizations use Hardoop for their
database for storage and data processing. Though, cloud
978-1-5090-3352-2/16/$31.00 ©2016 IEEE 16-18 November 2016, Aston Tropicana Hotel, Bandung, Indonesia
2016 International Conference on Information Management and Technology (ICIMTech)
Page 309
computing becomes trends for any organization either large
corporation or small companies to manage big data storage.
Value Creation
The era of big data is coming to bring new opportunities for
discovering new values. It can deliver the valuable aspect in
improving innovation, competitiveness and creation of
business value [24]. Big data creates a high demand for
organizations that can analyze and use them [25]. Big data
have an enormous potential to create a better customer services
in any industries like healthcare, education, agriculture, mining,
financial matters, security, communication, and even traffic
control. In healthcare, according to the Bernard Marr, at the
hospital, by collecting, and analyzing every heart rate of an
infant can figure out the symptoms or condition of the new
born. Not only that, it can save lives.
Improve Performance
Organizations deploy big data in order to enhance their
performance especially in quantify their sales. With the help of
big data analytic, organization has better view of their
customers based on their online behaviors such clicks, e-
commerce’s transactions, web visit histories, etc. Big data
analytic helps e-commerce Company to understand what their
customer’s profile, trends, interests, and personality. Then, the
company uses the result to predict what most likely the
customers’ preference their next purchase over product or
service so then the company might propagate to their web /
Apps visit an advertising or offering discounts of products that
would likely they will bound in transaction. For instance,
ticketing online will push customers with advertisements based
on their last visit. Customer who visit the ticketing web for
searching hotel in Bangkok. He can expect some
advertisements in his free email account or website he visit at
the next day with full advert of hotel in Bangkok. Ticketing
company tailors the advertisement based on website history for
each customer. Furthermore, business will then expand by
attracting more customers from reading the positive reviews by
these loyal customers. Businesses may increase their
performance by understanding better for their customers’ need
and preferences. Then, big data become ‘a must’ strategy for
business to consider in order to forecast what the needs of their
customers are. Furthermore, big data in business can allow
updating the details such as transactions’ history and gives an
accurate predictions and decisions how to utilizing it.
V. CHALLENGES
Even though applications and the potential of big data is
substantial, there are of course challenges that would occur
regarding big data. The main challenge is issue of privacy and
confidentiality. For examples, any online activities like posting
or tweeting on social media are read by the public or person
who handles the big data. Though, previous studies indicated
that people have very little understanding and concern about
how organizations are using big data [5].
Furthermore, that getting data into the big data platform can
be difficult as different scale and variety of data can overcome
a data specialist who is unprepared in this sort of area [12].
Furthermore, according to analyst firm. The United states alone [6]
will face a scarcity of around 140,000-190,000 people that
requires data analytical skills for 1.5 million managers and [7]
analysts that mastery in using big data analytics’ tool to support
[8]
decisions in the organizations [2]. There will be about 44
million IT jobs created globally to support big data by 2015 yet [9]
there is no guarantee of employees filling up these positions
[3].
Finally, big data analytic is given to data management using [10]
supporting tools to analyze, store, and present results.
[11]
Presenting a comprehensive results yet user friendly from
structured and unstructured data sources are the biggest [12]
challenge in big data analytic. It requires a new advancement of
tools and methods in order to gain an expected value. For
instance, it is a big challenge for combining CCTV record,
audio conversation, social networks activities, and CRM record [13]
to reveal pattern of a potential customer. It involves technique
to analyze and obtain intelligence from those big data sources
[14]
and analytics are closely connected to each other. We can
expect some advancement in tools and technologies for big
data analytic. [15]
VI. CONCLUSION [16]
Business organizations are in need to consider big data to make [17]
more accurate analysis, leading to a better decision making.
Big data can be characterized by three Vs; Volume, Variety,
and Velocity. Big data is defined as a large volume of data that [18]
needs to advanced techniques and tools to its data analytics’
complexity, data size and type. Big data analytic does process
multiple sources of data to present any patterns, trends, or [19]
customers’ behavior. By then. Big data can be used to detect
future transactions based on customers’ behavior or discover [20]
future business trends due to its capabilities to gather a huge
data at a speed and come up with the extracted value.
[21]
[22]
REFERENCES
[1] Beyer, Mark. "Gartner Says Solving'Big Data'Challenge Involves More
Than Just Managing Volumes of Data." Gartner. Archived from the [23]
original on 10 (2011).
[2] Brown, Brad, Michael Chui, and James Manyika. "Are you ready for the [24]
era of ‘big data’." McKinsey Quarterly 4, no. 2011 (2011): 24-35.
[3] Caballero, Ismael, Manuel Serrano, and Mario Piattini. "A Data Quality [25]
in Use Model for Big Data." In Advances in Conceptual Modeling, pp.
65-74. Springer International Publishing, 2014.
[4] Chaffey, D. (2016). Global social media research summary 2016. Smart [26]
Insights. Retrieved from http://www.smartinsights.com/social-media-
marketing/social-media-strategy/new-global-social-media-research/
[27]
[5] Clemons, Eric K., James Wilson, and Fujie Jin. "Investigations into
Consumers Preferences Concerning Privacy: An Initial Step towards the
Development of Modern and Consistent Privacy Protections around the
Globe." In System Sciences (HICSS), 2014 47th Hawaii International
Conference on, pp. 4083-4092. IEEE, 2014.
978-1-5090-3352-2/16/$31.00 ©2016 IEEE 16-18 November 2016, Aston Tropicana Hotel, Bandung, Indonesia
2016 International Conference on Information Management and Technology (ICIMTech)
Page 310
Davis, B. (2013). 10 actual uses of big data. Retrieved from
https://econsultancy.com/blog/63594-10-actual-uses-of-big-data/
Davenport, Thomas H. "Analytics 3.0." Harvard Business Review 91,
no. 12 (2013): 64-+
Dumbill, E. (2013). Making sense of big data. Big Data, 1(1), 1-2.
Hashem, Ibrahim Abaker Targio, Ibrar Yaqoob, Nor Badrul Anuar,
Salimah Mokhtar, Abdullah Gani, and Samee Ullah Khan (2015). "The
rise of “big data” on cloud computing: Review and open research
issues." Information Systems 47: 98-115
Horrigan, M.W. (2013), Big Data: A Perspective from the BLS, Amstat
News, January 2013, 25-27.
Laney, Douglas (2001). 3-d data management: Controlling data volume,
velocity and variety. META Group Research Note, February, 6, 2001.
Loshin, D.(2014) Adressing five emerging challenges of big data.
Retrieved from https://www.progress.com/docs/default-source/default-
document-library/Progress/Documents/Papers/Addressing-Five-
Emerging-Challenges-of-Big-Data.pdf
Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C.,
& Byers, A. H. (2011). Big data: The next frontier for innovation,
competition, and productivity.
Matteson, Scott (2013). "Big Data Basic Concepts and Benefits
Explained." Web blog, http://www. techrepublic. com/blog/big-data-
analytics/
McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J., & Barton,
D. (2012). Big data. The management revolution. Harvard Bus Rev,
90(10), 61-67.
Meijer, Erik. (2011). The world according to linq. Communications of
the ACM, 54(10):45–51, 2011.
Ohlhorst, F, J.(2013). Big data analytics. Turning big data into money.
Retrieved from http://www.scribd.com/doc/244559565/Big-Data-
Analytics-Frank-J
Pahm, P. (2015). The Impacts of Big Data That You May Not Have
Heard Of. Retrieved from
http://www.forbes.com/sites/peterpham/2015/08/28/the-impacts-of-big-
data-that-you-may-not-have-heard-of/#1d2c12fbc957
Rodriguez, R.N. (2012), Big Data and Better Data, Amstat News, June
2012, 3-4.
Schönberger, V. M., & Cukier, K. (2013). Big Data: A revolution that
will transform how we live, work, and think. New York, NY: Houghton
Mifflin Harcourt.
Sagiroglu, S., & Sinanc, D. (2013, May). Big data: A review. In
Collaboration Technologies and Systems (CTS), 2013 International
Conference on (pp. 42-47). IEEE.
Toshniwal, R., Dastidar, K. G., & Nath, A. (2015). Big data security
issues and challenges. International Journal of Innovative Research in
Advanced Engineering (IJIRAE), 2(2), 15-20.
Villars, R.L. et al. (2011) Big Data: What It is and Why You Should
Care. IDC
Williamson, J. (2014). Getting a big data job for dummies. Hoboken,
New Jersey: John Wiley & Sons, Inc.
Watson, H.J. (2014) Tutorial: Big Data Analytics: Concept, technology
and application. Communication for the association for information
systems, 34(65), 1247-1268. Retrieved from http://aisel.aisnet.org/cais/
Xia, F., Yang, L. T., Wang, L., & Vinel, A. (2012). Internet of things.
International Journal of Communication Systems, 25(9), 1101-1102.
Zikopoulos, P., Chris Eaton, et al. Understanding big data: Analytics for
enterprise class hadoop and streaming data. McGraw-Hill Osborne
Media. 2011.