White Paper

Harnessing big data for security

what are the key considerations and capabilities?

A White Paper by Bloor Research

Author : Fran Howarth
Publish date : February 2013

Harnessing big data is

complicated and security data
presents unique challenges. For
a big data security and analytics
system to be effective, all parts
of it need to be tightly integrated,
built from the ground up to work
together effectively by design.
Fran Howarth

Harnessing big data for security

what are the key considerations and capabilities?

Executive summary
Volumes of data are expanding rapidly, and
effectively harnessing the data volumes generated by organisations today brings both
significant gains as well as challenges. In particular, big data security sets bring specialised
challenges owing to the nature of the information that is produced, which must be stored in
a sequential, time-stamped manner and which
must be stored in its raw, unchanged format
that proves its integrity; in other words, it has
not been tampered with.
This requires the use of specialist tools. Traditional database and warehousing technologies
do a good job of handling operational data,
but were not designed with the specificities of
security data in mind. They are also based on
an architecture that is difficult to scale effectively to meet the challenges of huge data sets,
such as the huge volumes and constant flow
of security event information from systems
throughout the organisation.
Even where the back end technology has been
built with the needs of big data security event
information in mind, effective analysis of that
information requires a security intelligence
platform be integrated at the front end so that
information flows freely in an uninterrupted
manner between all parts of the system. Many
vendors offer partial solutionseither the
back end technology or the front end security
intelligence platform. This creates multiple integration and management challenges that are
not only a challenge in the upfront implementation, but also in the ongoing management
and extensibility of the system as new data
sources are added. Few organisations have
the resources or budget available to effectively
overcome the challenges of integrating disparate technologies. Far better is to look for
a system that was built from the ground up as
one integrated system, designed specifically
for the intricacies involved with security data.

Fast facts

Security event information analysis and

management is more complex than for
operational information and requires a
specialised approach using technology
designed from the ground up with security
needs in mind.

It requires a specialised architectural approach that can efficiently enable event

information from myriad sources to be
collected, stored, analysed and reported
on to unveil patterns that indicate security
threatseven advanced, previously unseen
threats.

Tight integration is required across all parts

of the system so that raw information can
be turned into actionable intelligence. It is
a much harder task to cobble together systems designed in isolation.
The bottom line
Harnessing big data is complicated and security
data presents unique challenges. Specialised
technology is required that was built from the
ground up for security purposes and in which
all parts of the puzzle are tightly integrated
to save the cost and complexity of integrating
and managing separate technologies built for
managing only parts of the overall equation.

This report discusses some of the challenges

of harnessing big data security and outlines
some of the key considerations and capabilities that organisations should consider when
selecting a system that can handle the whole
gamut of needs in a unified manner that is
simple to integrate and manage.

A Bloor White Paper

2013 Bloor Research

Harnessing big data for security

what are the key considerations and capabilities?

What constitutes big data?

The world is increasingly becoming digital.
This change was ushered in by the widespread
availability of personal computers, which
revolutionised both leisure and work activities. Since then, technology has moved on fast.
Today, we are used to the always-on, instant
communication and interactivity that internetconnected mobile devices provide. For many
people, their first activity of the day is to check
messages and social networking updates on
their ever-handy mobile devices. As of early
2013, there are more mobile devices in use that
there are people on this earth. The amount of
data generated on and stored by such devices
and through interactive web channels is contributing to the vastly expanding volumes of
information produced digitally.
Another phenomenon that is playing a role in
expanding not just the volumes but also the
types of digital information that are produced
is the increase in the amount of machine-tomachine data that is generated, as an everexpanding array of devices such as sensors
are connected to networks for the greater
operational efficiencies that this brings. Industrial automation systems, which traditionally
ran on proprietary networks, are now routinely
being connected to internet protocol networks,
with myriad pieces of equipment generating
data that is useful for operational purposes.
According to a recent report by IDC, the
amount of information produced digitally in
2012 amounted to 2.8 zettabytes (1021 bytes)
double the amount seen just two years prior to
that1. It estimates that, by 2020, the digital universe will comprise 40 zettabytes of information57 times the number of grains of sand on
all beaches in the worldas volumes of data
generated will continue to grow two-fold every
year to 2020. Machine-generated data alone is
expected to grow 15 times in volume by 2020,
showing the importance of new data sources.
To put this data growth in perspective, the UC
Berkeley School of Information estimated that
the sum of all human-produced information
was just 12 terabytes (1012 bytes) at the end of
19992.

2013 Bloor Research

Organisations upping their investments in big

data
The term big data has only come into the common vernacular recently. Professional services firm Deloitte estimates that there were
only a handful of big data projects as recently
as 2009, amounting to total revenues for this
sector of less than US$100 million3. However,
it is a market that is growing fast. By the end
of 2012, Deloitte estimated that more than 90%
of Fortune 500 organisations had at least some
big data initiatives underwayeven if they are
just dipping their toes in the water.
According to the McKinsey Global Institute,
the ability to harness and analyse big data will
become a key basis of competition and growth
for organisations4. It states that big data has
the potential to add value to organisations in
the following ways:

Creating transparency by making relevant

data more accessible.

Enabling

experimentation to discover
needs, expose variability and improve performanceuse data to analyse variability in performance and understand the root
causes.

Segmenting populations to customise actions and tailor products and services to

meet specific needs.

Replacing/supporting

human decisionmaking with automated algorithms in order

to minimise risk.

Innovating new business models, products

and services.
It states that harnessing internal and external big data sources will give an organisation
a much broader view of what is happening
in its operations, providing a wealth of opportunity for improving business processes,
new product development, customer service,
brand awareness, product revision cycles and
partner networksall of this by mining information sources that are readily available.

A Bloor White Paper

Harnessing big data for security

what are the key considerations and capabilities?

The value of big data for improving security

40% of enterprises will actively

analyse patterns using data
sets of at least 10 terabytes
in order to flag potentially
dangerous activity by 2016.

Gartner Group

As harnessing big data has become a strategic priority for many organisations, many have focused primarily on operational data in an effort
to improve business performance and efficiency. However, the value
of big data extends way beyond this and has enormous application for
improving security within organisations. Whilst the use of big data for
security currently lags behind its use for operational purposes, the
Gartner Group estimates that 40% of enterprisesled by the banking, insurance, pharmaceutical and defence industrieswill actively
analyse patterns using data sets of at least 10 terabytes in order to flag
potentially dangerous activity by 20165.
The ability to harness big data sets from throughout the organisation
provides valuable insight into the security intelligence buried within very
large volumes of complex data from disparate data sources that will
boost the organisations ability to understand and manage the risks and
threats that it faces (See Figure 1). By analysing and correlating this
information to provide greater context, organisations will be afforded a
much higher level of understanding as to which of their assets are most
vulnerable to attack and where those attacks are most likely to come
from. This will allow them to take more focused and directed action,
prioritising risks against the severity of vulnerabilities faced in order to
take better decisions as to what risks to fix first.

Figure 1: Using big data to protect and manage data

Through advanced analytics and correlation across multiple, voluminous data sets, organisations will also be better able to ward off the
increasingly sophisticated, complex and targeted attacks that are the
reality today. Many of those threats are specifically designed to avoid
defences such as anti malware and intrusion detection systems that
are based on fingerprinting specific patterns associated with known
exploits. Big data analytics will boost an organisations ability to develop
models of what constitutes expected behaviour, which can then be used
as a baseline against which the analysis will be able to discern meaningful patterns and pinpoint behaviour that is abnormal or suspicious
across multiple parts of the network to identify behaviour likely to be
associated with an attack. In this way, organisations will be able to take
a more proactive stance on security; identifying trends, attack profiles
and possible threats, rather than just reacting to each incident as it occurs. They will be in a better position to not only prevent attacks, but
also to gain a better understanding regarding the threats that they face
in order to take more focused and directed action against those threats
and sources of breaches.

A Bloor White Paper

2013 Bloor Research

Harnessing big data for security

what are the key considerations and capabilities?

The value of big data for improving security

Nirvana would be data analytics

predicting the future and
enabling the prevention of all
incidents.
Information
Security Forum

2013 Bloor Research

One further benefit of using big data for enhancing security intelligence
is that huge swathes of business data can be stored in a secure and
tamper-proof manner that is useful for historical analysis in order to
learn from trends, for supporting forensic and legal investigations,
and for proving compliance with corporate governance and regulatory
requirements. This is extremely useful in looking for patterns across
large data sets that could indicate threats such as fraud being committed in a covert manner over long periods of time, or insider threats
from disgruntled or other employees whose behaviour is exposing the
organisation to risk. With such forensic capabilities, organisations will
be able to look for the root cause of security incidents; identifying, for
example, how a piece of malware moved through systems and which
systems were accessed, changed or potentially compromised over a
certain time period. This will allow the organisation to identify causes of
attacks in order that it can learn from such events and prevent similar
incidents from occurring in the future.
Big data analytics forensics capabilities are also useful in protecting
against the damage caused by so-called advanced persistent threats
(APTs) that aim not only to infiltrate a network, but to maintain a
presence on those networks over long periods of time, using techniques specifically aimed at avoiding detection. According to the SANS
Institute, more than half of Fortune 500 organisations have been targeted by such APTs7. The need for effective forensic analysis can be
seen in research conducted recently by Verizon Business, which found
that 85% of respondents have suffered data breaches that took weeks
or more to discover8.

A Bloor White Paper

Harnessing big data for security

what are the key considerations and capabilities?

A specialised approach is required for big data security

The use of big data for security intelligence
purposes is fast gaining ground and many
organisations are now seeing the value of a
centralised data strategy for their security information, as they have done with operational
information such as inventory records and
customer information. Such operational data
is generally captured as single records, which
may be frequently updated and changed, such
as when a transaction related to an order is
fulfilled. The majority of such data is also held
in a structured format, making it easier to capture and analyse.
However, security information is more complicated in nature than operational records, as
it is data records showing events as they happened and the sequence is important. Event
data is sometimes referred to as an audit trail.
As such data forms a system of record, it is essential that the data is written once and never
changed as audit trails must never be altered.
It must be stored in chronologically sequenced
order and all data must be time stamped so
that the data can be played back to show how a
sequence of events occurred.
Big data records related to security are also
massive in terms of volume as event and log
data must be taken from a huge variety of
extremely diverse sources. The following can
be considered to be characteristics of security information that is required for big data
analytics:

It comprises huge volumes of event and log

data that accumulates quickly from a wide
variety of data sources and it must be stored
for long periods of time. Collection must be
in real time and ongoing as new event data is
constantly being generated.

Although some data is in easy-to-digest

structured form, the majority is unstructured, or semi-structured at best.

The system used to collect the data must be

Event data must be captured once as soon

as it is generated, retained in its original
form as the single source of the truth, must
be time stamped for security in order to
find threats and related patterns, and must
never be changed in order for its integrity
to be maintained and for it to be admissible
as evidence. Event data must be reported
on based on time, which introduces storage
and querying challenges that relational databases do not easily support. Queries need
to be performed quickly in real time across
massive data volumes spanning long time
periodsfor example, to comply with regulatory requests.
The differences in security data over operational data mean that there are specific requirements for its collection, storage, analysis and
retrieval. It must be collected from multiple
sources so that events can be correlated and
stored for long periods of time. The system
used for managing it must support deep, broad
and frequent analysis across all relevant systems of record to provide sufficient detail about
what has happened, and to look for incidents in
real time and for long-term trends. Capabilities must include complex and ad hoc analysis
for forensic purposes, real time reporting and
compliance reporting.
For handling more homogenous, structured
data, such as that related to transactions and
operations, traditional data warehousing technologies are often sufficient. However, such
technologies can fall down not only in the face
of big data challenges, but because they were
also designed primarily with structured data in
mind. Many traditional data warehouses cannot support the huge volumes of data generated in big data environments and most are
only capable of collecting information from a
limited range of data formats and endpoints.
Event data also needs to be collected and
stored in its raw format, not normalised as is
the case with most databases and warehouses
based on relational technology.

capable of taking in data from a very diverse

range of data sources and types, as well
as from a diverse range of endpoints. This
requires that the management system uses
a common taxonomy, dictionary of terms
and event profile schema based on industry
standards so that such data sources can be
compared directly.

A Bloor White Paper

2013 Bloor Research

Harnessing big data for security

what are the key considerations and capabilities?

Essential considerations for a big data security vault

Because of its various characteristics, traditional data warehouses are unsuited for the
collection and storage of big security data,
including considerations such as performance
and cost. This requires that more specialised
technology be implemented. Rather than a
general-purpose data warehouse that has been
designed primarily for analysis of structured
data, data regarding security incidents needs
to be collected as events, time stamped so that
sequences of events can be reconstructed.
Traditional data warehouses are built for the
type of information that needs to be regularly
updated, such as information regarding progress with a customer order. Therefore, they
provide capabilities for moving, changing or
otherwise overwriting data.
More suited to the task is an event data warehouse that is designed from the ground up
to collect, store and report on time-stamped
events to find threats and related patterns
to show how a sequence of events occurred.
This is something that is not handled well
by traditional SQL and relational databases
because standard SQL does not have the relevant time-based functionality and is more
suited to processing sets of data, rather than
events that occur one at a time. In contrast,
event data warehouses with big data analytical
capabilities are specifically designed to support queries of and analytics against timestamped data.
An event data warehouse should be built using
a columnar architecture in which data is stored
by column, rather than row as in traditional
relational databases. This means that data can
be queried faster since data in each column is
stored directly together, meaning that queries
only need to reference data selected as each
column is essentially an index. This makes
them especially suited for search-based environments as indexes dont need to be created
as data does not need to be read from other
columns in the row. This has implications for
performance since queries can be run faster,
allowing for reports to be generated in real
time, and also for storage, since compression
algorithms can be applied to each column to
vastly reduce the size of the overall data set.
This also means that exponentially larger volumes of data can be collected, stored and analysed for much lower cost that with traditional
database architectures. This is essential since
big data security analysis requires that data
be collected from myriad systems generating
event logs from both machine-generated and

2013 Bloor Research

human-generated information, in both structured and unstructured form.

Among the benefits offered by event data
warehouses are:

The ability to collect, store and analyse massive amounts of log and event data from
multiple, disparate sources.

Keep that data highly compressed for costeffective storage and future analysis for
forensic purposes.

Query and analyse huge volumes of compressed data, often amounting to terabytes
of records, quickly and efficiently, in almost
real time.

Provide support for queries of time-stamped

data within specific time ranges to playback
events as they occurred, which is a key requirement for security investigations.

Provide a rich set of analytical and reporting capabilities that converts information
regarding security events that have occurred into actionable intelligence for faster
remediation, improving the overall security
posture of the organisation.
For the system to work effectively, it must
also provide a front end analytics console that
enables security intelligence to be applied to
the data resulting from queries and analysis of
records from the event data warehouse. This
console provides the means through which
usersand not just those from IT, but general
business users who need access to complex
and ad hoc information reports for real time
status information and forensic analysis
interact with the system. This must provide the
means for users to directly access the back
end data for search and query purposes in an
easy manner, negating the need for users to
request reports be generated for them by IT
staff, which can be a serious bottleneck.
The console should also provide management
and administrative tools, including those for
constantly monitoring the data flows in real
time to look for abnormalities that could indicate suspicious behaviour. When such behaviour is uncovered, an alert should automatically
be generated and sent to relevant resources
to ensure speedy remediation can be taken.
Those alerts should also take into account
information such as that from vulnerability

A Bloor White Paper

Harnessing big data for security

what are the key considerations and capabilities?

Essential considerations for a big data security vault

assessments, so that the most urgent actions
that are of the highest risk can be prioritised.
The system should also generate reports of all
activity, which forms the audit trail required
for compliance and governance purposes, and
for demonstrating that security controls are
effective. To make the information as useful as
possible for compliance purposes, predefined
reports should be provided for the most common regulations and industry standards.
In order to ensure that new threats are taken
into account as they are seen, the console
should incorporate threat and reputation feeds
from worldwide sources that include input
from multiple users and intelligence communities. Intelligence capabilities can also be
boosted if the management console interfaces
with business intelligence tools from third
party specialists to provide even greater context for the information.
In order to make such intelligence actionable,
it should be provided to users and administrators in an intuitive and visual manner through
a customisable dashboard that can provide an
at-a-glance view of all events that have occurred, alerts that have been generated and
remediation actions that have been taken. This
will allow management to focus on the most
important events affecting its most critical
data, showing the results of all queries and
providing a timeline view of all data collected
and incidents that have occurred.

A Bloor White Paper

The benefits of including a front end security

intelligence platform include:

The ability to run queries that can monitor and correlate events in real time so
that members of management and security teams can ensure that risks to the most
critical assets are minimised.

Providing the ability for all users, not just

those in IT and security teams, to access
raw data quickly and easily to analyse both
real time and historic data for investigative
purposes.

Timely alerts that can drastically reduce

the time taken for incident response and
resolution.

Correlate security events with the identity of

users and devices to provide an enterprisewide view of what happened where and
when, and who the perpetrator was.

Providing the ability to take a more proactive stance on security through inclusion of
up-to-date threat information, rather than
just relying on reactive, signature-based
systems.

Complements and extends the capabilities of existing security controls, providing

greater context to events.

2013 Bloor Research

Harnessing big data for security

what are the key considerations and capabilities?

Why an integrated system is required

For a big data security and

analytics system to be effective,
all parts of it need to be tightly
integrated, built from the
ground up to work together
effectively by design, rather
than separate technology
systems that require a high
degree of integration.

For a big data security and analytics system to be effective, all parts
of it need to be tightly integrated, built from the ground up to work together effectively by design, rather than separate technology systems
that require a high degree of integration. This requires that the system
be a combination of an event data warehouse, empowered by sophisticated analysis and security intelligence tools capable of analysing
voluminous and varied data sets; it needs to have been designed with
the needs of security in mind, supporting complex and ad hoc analysis
of time-stamped events to uncover threats and related patterns in real
time; and to support forensic investigations and compliance reporting
requirements.
Owing to the volumes of data that must be processed and analysed,
it must be hugely scalable and capable of handling an ever-growing
number and variety of data sets so that all event data generated by the
organisation can be included so that there are no gaps left in protection.
However, for performance purposes, it must be capable of compressing
data so that storage requirements are reduced, allowing the system to
scale more effectively.
There are a variety of technology solutions available, some of which
only incorporate some of the capabilities required for big data analysis
and most of which were not designed with integration in mind. This is
something that is problematic for most organisations as many lack the
resources in-house and many of the tools available have a wide variety
of configuration options for fine-tuning systems before they can be truly
useful. It is well recorded that many of the back end systems available
are complex to deploy, making integration an even harder task.
The integration challenges of systems that were designed in isolation
will become even more pressing as numerous surveys are reporting
that there is a shortage of resources with the skillsets required for big
data security. In a recent survey undertaken by AIIM, more than 50%
of respondents reported that big data analytics would be very useful,
but we dont have the skillsets.9 The survey also found that big data
skillsets are among the most sought after for respondents.
Given the lack of skilled resources, the expense of hiring consultants
and the integration challenges posed by cobbling together systems that
were designed for specific purposes, rather than as one homogenous
system, a far better choice is one unified system with common, centralised management tools that were not only designed for big data
analytics, but also for handling the complex nature of big data security
event information. A more general-purpose system built around big
data analytics, but not the specifics of big data security analytics, will
not only increase the implementation challenges, but will also require
heavy lifting by the security team, which negates the benefits of having
the data analysis available to all members of the organisation.

2013 Bloor Research

A Bloor White Paper

Harnessing big data for security

what are the key considerations and capabilities?

Summary
Many organisations are starting to realise
the benefits that harnessing big data brings.
In terms of big data analysis for operational
performance improvements, there are many
challenges to overcome. But big data security
analytics is even more challenging owing to
the complex nature of security information
and the specifics that are required for collecting, storing and processing security event
information so that its integrity is maintained.
This requires the use of an event-driven data
warehouse, specifically designed for compliance and discovery purposes, as opposed to a
traditional warehouse, which is better suited
for more basic analytics than search-based
environments. An event data warehouse
specifically designed for queries against
time-stamped data provides the only reliable
method of showing threat events and patterns
related to security.
In order to be able to effectively turn the information provided by the event data warehouse
into actionable intelligence that will guide the
organisation in improving its overall security
posture, tight integration needs to be natively
provided with advanced security intelligence
capabilities at the front end, provided through
a single console with advanced, centralised
management tools, and available to all users
in the organisation, not just security and IT
specialists.
Only through one integrated system, designed
so that all parts work together out of the box,
can organisations achieve the promised benefits of reduced cost and higher performance.
It must also be able to scale across multiple,
disparate and voluminous event data sets
and to be able to store massive data sets in
an efficient, cost-effective manner. Analytical
capabilities need to be integrated across all
parts of the system, with data collection and
analytics being uniform and intuitive for users,
without the need for proprietary tools that do
not effectively support integration across all
parts of the system.

A Bloor White Paper

References
1. http://www.emc.com/collateral/analystreports/idc-the-digital-universe-in-2020.
pdf
2. http://www2.sims.berkeley.edu/research/
projects/how-much-info-2003/
3. http://www.deloitte.com/view/en_GX/
global/industries/technology-mediatelecommunications/tmt-predictions-2012/
technology/70763e14447a4310VgnVCM100
0001a56f00aRCRD.htm
4. http://www.mckinsey.com/insights/mgi/
research/technology_and_innovation/
big_data_the_next_frontier_for_innovation
5. http://searchsecurity.techtarget.com/
news/2240157901/Gartner-Big-datasecurity-will-be-a-struggle-but-necessary
6. http://blog.varonis.com/big-data-security/
7. http://computer-forensics.sans.org/
blog/2012/06/02/the-apt-is-already-inyour-network-time-to-go-hunting-learnhow-in-new-training-course-sans-for508
8. http://www.verizonbusiness.com/
resources/reports/rp_data-breachinvestigations-report-2012_en_xg.pdf
9. http://www.aiim.org/Research-andPublications/Research/AIIM-White-Papers/
Career-Development
Further Information
Further information is available from
http://www.BloorResearch.com/update/2156

2013 Bloor Research

Bloor Research overview

About the author

Bloor Research is one of Europes leading IT

research, analysis and consultancy organisations. We explain how to bring greater Agility
to corporate IT systems through the effective
governance, management and leverage of
Information. We have built a reputation for
telling the right story with independent, intelligent, well-articulated communications content and publications on all aspects of the ICT
industry. We believe the objective of telling the
right story is to:

Fran Howarth
Senior Analyst - Security

Describe the technology in context to its

business value and the other systems and
processes it interacts with.
Understand how new and innovative technologies fit in with existing ICT investments.
Look at the whole market and explain all
the solutions available and how they can be
more effectively evaluated.
Filter noise and make it easier to find the
additional information or news that supports both investment and implementation.
Ensure all our content is available through
the most appropriate channel.
Founded in 1989, we have spent over two decades distributing research and analysis to IT
user and vendor organisations throughout the
world via online subscriptions, tailored research services, events and consultancy projects. We are committed to turning our knowledge into business value for you.

Fran Howarth specialises in the field of security, primarily information security, but with a keen interest
in physical security and how the two are converging.
Frans other main areas of interest are new delivery models, such as cloud computing, information
governance, web, network and application security,
identity and access management, and encryption.
Fran focuses on the business needs for security technologies, looking at
the benefits they gain from their use and how organisations can defend
themselves against the threats that they face in an ever-changing landscape.
For more than 20 years, Fran has worked in an advisory capacity as an
analyst, consultant and writer. She writes regularly for a number of publications, including Silicon, Computer Weekly, Computer Reseller News,
IT-Analysis and Computing Magazine. Fran is also a regular contributor to
Security Management Practices of the Faulkner Information Services division of InfoToday.

Copyright & disclaimer

This document is copyright 2013 Bloor Research. No part of this publication may be reproduced by any method whatsoever without the prior
consent of Bloor Research.
Due to the nature of this material, numerous hardware and software
products have been mentioned by name. In the majority, if not all, of the
cases, these product names are claimed as trademarks by the companies that manufacture the products. It is not Bloor Researchs intent to
claim these names or trademarks as our own. Likewise, company logos,
graphics or screen shots have been reproduced with the consent of the
owner and are subject to that owners copyright.
Whilst every care has been taken in the preparation of this document
to ensure that the information is correct, the publishers cannot accept
responsibility for any errors or omissions.

2nd Floor,
145157 St John Street
LONDON,
EC1V 4PY, United Kingdom
Tel: +44 (0)207 043 9750
Fax: +44 (0)207 043 9748
Web: www.BloorResearch.com
email: info@BloorResearch.com