1 I N N O V A T I O N S T H A T M A T T E R

2 B I G D A T A
CONTENTS
T H E N E XT F R O N T I E R O F
P U B L I C S E C T O R I N N O V A T I O N : B I G D A T A 3
C i t i e s a s C e n t e r s f o r
I n n o v a t i o n : L o u i s v i l l e , K y 6
Building the Data-Driven Culture: LouieStat
Data Analysis Improves Emergency Health Services
T h e N e w N o r ma l :
H o w D a t a D r i v e s Yo u r Mi s s i o n F o r wa r d 10
T h e U . S . A r my s B i g D a t a P r o b l e m:
A S e a o f D a t a b a s e s 12
The EMDS Solution: Rapid, Holistic, and Visual
EMDS as an Evolving Process
A B i g D a t a I mp e r a t i v e :
H a v i n g a F l e xi b l e D a t a I n f r a s t r u c t u r E 16
MA K I N G B I G D A T A S T I C K A T YO U R A G E N C Y 18
Understanding Governments Big Data Challenges
Our 8 Best Practices for Big Data Analysis
H o w I n - Me mo r y C o mp u t i n g i s C h a n g i n g
H o w We T h i n k A b o u t B i g D a t a 24
G o v L o o p s B i g D a t a C h e a t S h e e t 25
A b o u t G o v L o o p 27
A c k n o wl e d g e me n t s 27
3 I N N O V A T I O N S T H A T M A T T E R
THE NEXT FRONTI ER OF PUBLI C SECTOR I NNOVATI ON //
Although we are living in historic times,
todays digital age holds many parallels to the
industrial revolution. During that era, there were
countless breakthroughs in manufacturing, cre-
ations of new products, workforce changes and
movements to create unions to protect workers
rights. These advancements were all shaped and
sculpted by the technological advancements of the
18th and 19th centuries.
We also witnessed the development of new gov-
ernment regulations and the emergence of new
markets, propelling america to the status of global
economic leader. Now we are in the midst of
another technological era, which is reshaping the
cultural and socioeconomic fabric of society. As we
sit on the cusp of remarkable innovations, we must
remember that this time, modern innovations are
powered by data.
Today, governments are using data to improve
our standard of living. This is certainly not only an
american phenomenon. Governments worldwide
have recognized that by leveraging data, they can
improve services to citizens and gain a competitive
advantage in a attening world.
This report is part of govloops innovations that
matter series, in which we explore top innovations
that truly matter and how to make them stick in
your agency. Well explore how organizations are
using big data technologies and solutions to trans-
form their organizations. Weve talked with thought
leaders and experts in government and industry in
order to understand the best practices to leverage
big data as a mission-centric tool, driving innova-
tions for the public sector.
Govloop reports are designed to empower you and
give you the tools to excel with big data. Heres
what youll nd in this report:
A local government spotlight showing how
the city of louisville, ky., Uses big data to
improve performance management.
A federal government case study
highlighting the armys enterprise
management decision support program.
Industry insights on the current big data
landscape.
8 Strategies and best practices for smart big
data adoption and analysis.
Govloops big data cheat sheet.
Understanding big data will turn data into insights
and transform your agencys operations. Now is
the time to take charge of your data and learn the
best practices and methods to modernize your
agency. We understand that innovation is an intimi-
dating word. But were here to help. Lets start with
learning about the effect that big data analysis is
having across government.
B I G D A T A
4 B I G D A T A
Big data analysis offers public-
sector institutions an unprec-
edented opportunity to transform
themselves through big data
analysis. But rst they have to
understand and effectively in-
terpret big data. Many agencies
are struggling with that because
each has dened and invested
in big data analysis differently.
Across government, agencies
need software that can quickly
and securely unlock insights from
their information.
We can start by dening big
data as data that arrives in such
enormous quantities, at such a
fast rate and in such a diverse
array of mediums that it is impos-
sible to understand or leverage
through traditional methods.
Similarly, a big data problem
describes a situation in which
you are not able to use your data
to accomplish a goal or deliver
a service because one of those
characteristics (volume, velocity
or variety) prevents you from cap-
turing its value within a predeter-
mined window of opportunity.
This is just a starting point to
dene big data. The real issue at
hand is understanding what data
analysis can do for your agency.
For the public sector, the key is
to appreciate how big data has
the power to drive changes to or-
ganizational workows, improve
productivity and efciency, and
reduce costs and overhead.
A recent GovLoop survey of 256
public-sector employees ex-
plored the benets of analytics
and big data in government. The
survey found that:
To fully leverage big data, leaders
face multiple challenges. Respon-
dents noted some of the chal-
lenges in our survey:
These challenges are explored
throughout this guide, along with
strategies and best practices to
avoid roadblocks to adopting big
data programs. Although chal-
lenges exist, many agencies have
moved to adopt big data pro-
grams. Our survey highlighted
four ways:
Additional benets were noted,
such as improved morale and
communication between groups.
One respondent said big data
analytics shows the best return
on investment, benets all stake-
holders and drives innovation.
The message is clear across gov-
ernment: In order to understand
the power of big data, agencies
need to invest the time to dene,
learn and capitalize on informa-
tion. Our report is the rst step
to encourage the sharing of best
practices, use cases and informa-
tion sharing across government.
To start things off, we learned
some remarkable insights about
performance measurements from
the City of Louisville, KY.
Understandi ng Bi g Data s EffecT
on the Publi c Sector //
cited improved efciency
& productivity
said big data drives
improved decision-making
cited transparency &
accountability
said they use big data
to manage resources,
budgets & controls costs
said big data helps control
waste, fraud & abuse
Performance
Analytics
Predictive
Analytics
Business
Intelligence
Financial
Analytics
Lack of organizational support
Unclear mission and goals
Lack of clarity on metrics
Data governance
Data locked in legacy systems
5 I N N O V A T I O N S T H A T M A T T E R
6 B I G D A T A
C I T I E S A S C E N T E R S F O R
I N N O V A T I O N :
L O U I S V I L L E , K Y
Cities are already complex ecosystems
to manage. Today local government leaders
are faced with dwindling budgets, deteriorating
infrastructure and the need to attract and retain
business for economic growth. We can condently
say that local government administrators are chal-
lenged like never before. Yet, many governments
are moving quickly to transform their cities into
centers of innovation and reigniting service deliv-
ery by capitalizing on their most important com-
modity: data.
Faced with unprecedented challenges, cities no
longer have the option to do business as usual. In
Louisville, business is far from usual. Led by Mayor
Greg Fischer and the chief of the Ofce of Perfor-
mance Improvement (OPI), Theresa Reno-Weber,
Louisville has positioned itself to use data in trans-
formative ways. Through their vision and leader-
ship, Louisville has become rooted in data and
offers a framework for data innovation for other
municipalities to follow.
City government has so much data across so many
different data systems we are missing opportuni-
ties, Reno-Weber said. Were missing synergies
between some of our departments and missing
clues into information that can predict issues that
we will need to address.
That challenge has led the city to not only improve
data sharing across agencies, but to create open-
data strategies. In doing so, the city has created a
platform for civic innovation for citizens, business
and developers.
We are excited about the potential for both innova-
tion and economic growth as it relates to making
data more open and available to the public, Reno-
Weber said. Weve already seen some of that with
our open-data push.
Louisville leaders have dened data as an impera-
tive for innovation and as the engine for economic
growth. As a pledge to the citys commitment,
Fischer signed into law an executive order mandat-
ing that data is open by default. The policy covers
many of the Open Data Policy guidelines from the
Sunlight Foundation, a nonprot that is working
toward greater government openness and trans-
parency. When the executive order was announced,
Louisville became one of the rst municipalities to
codify the open by default provision.
In this section of our report, we highlight two in-
novations that matter from Louisville: highlighting
the LouieStat program and how they have city has
leveraged data to improve medical response times.
7 I N N O V A T I O N S T H A T M A T T E R
Bui ldi ng the
Data-Dri ven
Culture:
Loui eStat //
One of the premiere data
initiatives has been the
LouieStat program, which
launched in 2012. Administered
by Reno-Weber, it was designed
to help citizens understand the
effectiveness and efciency of
government programs by provid-
ing answers to three fundamen-
tal questions that every citizen
should know and have easy
access to:
What are the key services
Metro Government
performs?
How well is Louisville Metro
Government performing?
How can Louisville Metro
Government perform
better?
The program brings Louisville city
leaders together to discuss con-
sistent metrics and benchmark
success within their department.
These insights take in various
data streams to guide success
and have built a culture of contin-
uous improvement. There have
been a lot of opportunities that
we have identied to eliminate
waste and improve the tactical
way we deliver services for citi-
zens in the city, said Reno-Weber.
One example is the work that has
been done in regard to Louis-
villes emergency management
services. (See below.) Another
comes from the Public Works and
Assets Department. Their mission
is: To deliver superior customer
service, efciently manage solid
waste and recycling, proactively
maintain and enhance city facili-
ties, assets, infrastructure and
Metro eet, initiate and support
progressive environmental and
energy conservation programs
and champion innovative busi-
ness practices.
On LouieStat, the data that is ag-
gregated is:
Dollars spent on overtime
Overtime hours paid
Hours lost due to work
related illness and injury
OSHA recordable injury
rate
Employees with high sick
leave consumption
Hours not worked
LouieStat also pulls data from
the Ofce of Management and
Budget, Louisville Zoo, Human
Resources and Public Health and
Wellness. This resource provides
a snapshot of the citys perfor-
mance, and links together data
from various sources to improve
accountability and transparency.
OPI staff meets with department
leadership teams to regularly de-
ne and assess key performance
indicators. This focus allows
decision-makers to know what
results they are trying to achieve
and to measure success. Next,
OPI works to develop bench-
marks and spot areas of improve-
ment. Key metrics are routinely
evaluated to track if it has already
met its goal, approaching goal
or off goal. Additionally, every six
to eight weeks department leads
provide a report to the mayor that
highlights performance.
This initiative was inspired by the
work of Baltimores CitiStat and
Marylands StateStat. Reno-Weber
shared one specic example of
success from LouieStat, highlight-
ing work done to improve emer-
gency health services below.
8 B I G D A T A
Data Analysi s
I mproves
Emergency
Health Services //
Every municipality is faced
with the challenge of
improving emergency
health services.
Local governments invest
signicant time understanding
how to build more efcient
emergency relief systems
without increasing budgets.
Reno-Webers team decided to
explore how to reduce the time
emergency medical services
(EMS) staff spent at the hospital
after dropping off a patient.
Any reduction in time would
increase efciency and allow
staff to deliver emergency health
services more effectively.
One of the things we were look-
ing at was: How do we get more
efcient use out of the ambulanc-
es that we already have?, said
Reno-Weber. With large expenses
tied to increasing ambulance
eets and stafng, her team was
tasked with nding an innovative
solution to increase ambulance
pick-ups.
This was no easy task, since
there are a series of protocols
that EMS staff must follow when
they arrive at the hospital. They
must ll out paperwork, clean
and disinfect the ambulance, and
prepare it for the next departure.
In a metro area as large as Louis-
ville, every second counts to get
EMS back on the road and as-
sisting citizens in need of health
services.
To discover new efciencies, the
city decided to observe the drop-
off process at one of the busiest
hospitals in Louisville. The team
took notes and collected data,
breaking down each process.
We put someone in the emer-
gency room tracking the time of
the crews and guring out what
the different process steps were,
said Reno-Weber.
Within the rst three months
of observing and analyzing the
process, EMS drop-off times de-
creased by 5 minutes. We said,
Theres denitely an opportunity
to improve, Reno-Weber said.
After analyzing the data, the
team set a new data standard
in place. The city now required
that ambulances must leave the
hospital within 30 minutes of
dropping off a patient. If that time
was to be extended, crews must
notify a supervisor.
Reno-Weber conducted the study
between September and Decem-
ber 2013. Today, crews are leav-
ing the hospital in 30 minutes or
less 90 percent of the time. That
is basically the equivalent of plac-
ing two additional ambulances on
the street, she said.
Compared to the same time
frame in 2012, the city of Louis-
ville was able to deliver 18,000
more patients to the hospital.
This is an example of how a com-
mitment to understanding how
data can improve services within
a city government.
The Louisville case study shows
the power of understanding data
and turning data into insights.
Powered by data, Louisville has
improved transparency, citizen
engagement and quality of the
services it provides.
You can follow
Theresa Reno-Weber at
@RenoWeber on Twitter.
9 I N N O V A T I O N S T H A T M A T T E R
Learn more at www.cloudera.com
10 B I G D A T A
The New Normal:
How Data Dri ves Your
Mi ssi on Forward //
Today, government agencies are looking
at new ways to transform their agencies
through big data analysis.
As Joey Echeverria, Chief Architect, Cloudera
Government Solutions, said, Big data presents a
lot of opportunities to change the way government
agencies and organizations manage their business.
For government, big data is forcing leaders to think
about data analysis as a way to move their mission
forward, rather than as a by-product of their service
delivery efforts.
This is a new way of thinking for government, as
Echeverria noted, Government never really looked
at data as something as core to what it is trying to
accomplish, that is the real thing that big data is
changing. Its giving governments an opportunity
to see that they can integrate data and information
into their everyday processes. The integration of
data into process is essential for agencies, facilitat-
ing new insights, ndings and better decision mak-
ing for organizations.
To help agencies integrate data and information,
Cloudera powers the enterprise data hub. An enter-
prise data hub is one place to store all data, for as
long as desired or required, in its original delity; in-
tegrated with existing infrastructure and tools; with
the exibility to run a variety of enterprise work-
loads together with the robust security, governance,
data protection and management that organizations
require. The advantage of an enterprise data hub
is that you can bring together any kind of data and
access it in ways that makes sense for your agency.
It purposely tries not to be prescriptive about how
you are going to end up accessing your data. This
means you end up with the right techniques and
framework for your specic needs, without being
locked in to a pre-dened solution. Another key
benet is the ability to centralize your data manage-
ment strategies.
Having data in a central location provides many
benets, especially in terms of accessing informa-
tion and collaborating across your agency. Echever-
ria provides additional insights on the benets, as
he said, Having one place to store all your data is
going be far more inexpensive rather than duplicat-
ing your data and storing over numerous tools. And
thats really where our platform and offerings differ
from whats been available in the past.
Not only does centrally storing data allow organiza-
tions to be fully integrated, but also information-
driven. By capturing data from web portals, web-
sites or from various ways government provides
services, agencies are creating a framework to
understand citizen needs. Capturing that data is just
the rst step, the second step is using that informa-
tion to improve service delivery.
If you are providing services to your citizens
through portals or websites, you should be har-
vesting the logs and the interactions that those
websites generate. This information should ow
back into feedback on how you model and run your
agency, said Echeverria.
Like many technology adoption projects, the IT
deployment is only one stage. To have success
with big data, organizations need to think about
the changes to culture. Cloudera provides pro-
fessional consulting services to help meet these
objectives.We work with our customers and clients
so that they learn our best practices, not just for
how to use technology, but how that technology
can be applied to their specic problem, said Ech-
everria.
Big data takes time and commitment. A best prac-
tice is dont go into a big data project with the idea
of revolutionizing your agency on day one, said
Echeverria. Government leaders must build consen-
sus and support, and cast a vision of how big data
can impact their agency.
11 I N N O V A T I O N S T H A T M A T T E R
12 B I G D A T A
As mentioned in the introduction, big data
problems usually involve more than just
juggling petabytes of data.
In its most essential form, a big data problem is the
inability to fully capture the value of data within a
predetermined window of opportunity. This is most
often due to the size, speed or format of the data
collected.
The U.S. Army had one such problem. Its current
inventory of information technology systems is
more than 3,600, many with unique access require-
ments. These systems are used to track 10,000 op-
erating units, which are comprised of more than 1.1
million soldiers. They also track millions of pieces
of equipment -- from tanks to batteries -- and thou-
sands of training events. The Armys data infra-
structure is large, complex and dispersed among
stovepiped and legacy systems, and it operates
under various data owners and governance rules.
As a result, the process of gathering data to inform
decision-making about deployment and assess unit
readiness was cumbersome and time-consuming. It
also meant that leaders were basing their decisions
primarily on information within their particular spe-
cialty area, often missing the rich layer of supple-
mental information available on other systems.
The response to this problem was the Armys
Enterprise Management Decision Support (EMDS)
system. This initiative is designed to serve as a de-
cision support tool for senior leaders. Its goal is to
provide a single access portal for culling pertinent
data from certain key databases.
What we do is reach across those systems, nd
the most relevant information for our customers
and bring that into one place, former EMDS Divi-
sion Chief Lt. Col. Bobby Saxon said. So it is one
username, one password, one location to go to, so
they can see critical information about personnel,
on readiness, on equipment, on installations and
on training.
In this way, EMDS served as both an aggregator
and curator of information.
T h e U . S . A r my s
B i g D a t a P r o b l e m:
A S e a o f D a t a b a s e s
13 I N N O V A T I O N S T H A T M A T T E R
Understanding unit readiness is paramount for the U.S. Armys operational success
The EMDS
Soluti on:
Rapi d, Holi sti c, &
Vi sual //
Now that the EMDS system per-
forms the front-end work of piec-
ing together data from disparate
database systems, the process
of leveraging the data is more
streamlined and timely. However,
speed was not the primary mo-
tivation for EMDS. Saxon noted
that making decisions under very
aggressive time-frames is simply
par for the course in the Army.
Instead, EMDS provides a crucial
additional layer of understanding
to this existing capability.
It takes not only the information
about the personnel community,
but it lays on top of that the infor-
mation for the equipment or the
training, he said. Then, we array
across the Armys Force Genera-
tion model [the Armys equivalent
of a supply-chain management
system], so they see much more
of the picture.
In other words, decision-makers
have a much broader, compre-
hensive understanding of a given
situation, which allows them to
see the wider context in which
they are making their decisions.
In a sense, EMDS provides a
holistic view of the situation,
while also providing access to
the granular details in relevant
specialty areas when necessary.
Given these dual, somewhat op-
posing objectives, visualization
was an integral design compo-
nent. Prior to EMDS, information
came in a variety of forms, such
as spreadsheets, charts and
graphs. When you multiply this
output by the multitude of dif-
ferent systems accessed, users
were left with the formidable task
of managing this unwieldy collec-
tion of information.
To contrast, EMDS has a number
of dashboards specically de-
signed to illustrate the situation
in a friendlier, easy-to-understand
format. Perhaps more important-
ly, this format is now standard-
ized across the enterprise, which
provides users in different areas
access to the same information in
a common format.
Finally, EMDS provides users with
14 B I G D A T A
the opportunity to discover new
or previously unknown pieces of
information, thereby adding an
additional layer of comprehen-
sion. The goal is to ensure that
users are able to focus on their
primary objectives, rather than
piecing together ad hoc reports
by themselves.
EMDS as an
Evolvi ng
Process //
When the EMDS program was
launched in 2008, the Army
encountered a few challenges.
For one, the surrounding big data
community, both proprietary and
open source, was nowhere near
as advanced as it is today. More
important, however, were the
cultural barriers to sharing infor-
mation across the enterprise.
There was not a clear under-
standing from stakeholders about
how valuable all this information
in one place could be, Saxon
said. We still deal with some of
these challenges today. Informa-
tion is king, and if you are some-
one who has information that
other people may not have, it
may give you a bit of a leg up.
The Army is still working through
these cultural barriers, but the
progress thus far has been trans-
formative. Senior leaders are
continuing to push that all data is
the Armys data and that it should
be there for all of us to use to
make decisions, Saxon said.
EMDS also evolved from solely
providing a near-real-time snap-
shot of the present into a product
that provides historical and pro-
jected pictures in addition to hav-
ing the foundation for a planned
predictive analytical component.
The mind-set now is the belief
that we have access to nearly ev-
erything. Now people are starting
to think, Well where is the data?
How do I get my hands on it?
Previously the thinking was that
there was no way we even know
what that information is, Saxon
said.
Thus, EMDS has been a revolu-
tion in IT business process ows
but also in organizational think-
ing, which is the hallmark of a
successful big data solution.
Former EMDS Division Chief Lt. Col. Bobby Saxon
For over a decade, MarkLogic has delivered a powerful, agile and trusted
Enterprise NoSQL database that enables government agencies to turn all data
into valuable and actionable information. Key features include ACID transactions,
horizontal scaling, real-time indexing, high availability and disaster recovery,
government-grade security, and more. Organizations around the world rely
on MarkLogics enterprise-grade technology to power a new generation of
information application.
For more information, please visit www.marklogic.com.
MarkLogic is the proven platform
for government Big Data
applications designed to drive
real-time analysis, situational
awareness, and service delivery.
Say hello to the new generation
Hello, were your
Enterprise NoSQL
database
solution.
16 B I G D A T A
A Bi g Data I mperati ve:
Havi ng a Flexi ble Data
I nfrastructure //
Discussions about big data in government
are generally dominated by the volume,
velocity and variety of the data being
produced by our agencies today.
Yet, what is often missing is a matching discussion
about the changing requirements for the agencys
data infrastructure. This infrastructure is just
as essential to improving the way services are
delivered to constituents as the data itself.
Kevin Shelly, VP of MarkLogic Public Sector, spoke
of the changing data landscape, The traditional
way to store operational data is in relational da-
tabase systems, which have been around for 30
years. While they are very good at what they do,
their specialty is transactional processing on struc-
tured data, and they havent really changed with
the times.
Today, government agencies understand the op-
portunity that big data presents. Many agencies are
looking to capitalize on the various data streams to
which they have access, whether it is social media,
video, photos, documents, mobile, or other forms
of structured and unstructured data. It is this mix of
unstructured and structured data that presents a
challenge for organizations. How can we glean new
insights from this data?
Governments also recognize that it is no longer an
either-or proposition for structured and unstruc-
tured data. They need to capitalize on both forms of
data to drive innovation and meet mission need in
new ways. Shelly notes, Agencies need a database
that can link structured and unstructured data so
you can leverage 100% of the data that is out there
to accomplish your mission or get closer to your
constituents.
To nd value from structured and unstructured data,
agencies must invest in new generation databases.
For many agencies, this means looking at NoSQL
(Not Only SQL) databases. Relational databases
were developed primarily for use with structured
data, while NoSQL allows much more exibility and
agility to collect, process and analyze a wide variety
of data sources. Also, unlike relational database
systems, No SQL databases allow you to ingest
the data without having to spend time and money
up-front developing a data model, hence you can
develop and deploy systems much quicker. The
reduced time to value is just one of the advantages
of the NoSQL approach.
I think there will be a very large shift towards
NoSQL databases, said Shelly. Ultimately what
MarkLogic can provide is a richer experience,
leading to improved mission capability and abil-
ity to provide enhanced services to constituents.
MarkLogic can operate on premise or in the Cloud,
ingest all data types, provide extensive search ca-
pabilities, and deliver the data, to various devices,
whether its a mobility platform, laptop or desktop
computer, said Shelly.
For many government agencies, now is the time to
start investing in big data, and thinking about how
data can drive a more efcient and effective agen-
cy. Data will continue to grow at a rapid pace, and
as Shelly observes, If the haystack keeps getting
bigger, it just takes more work to nd the needle.
Therefore, now is the time for smart investments
such as an enterprise-grade NoSQL database with
integrated search and application services that can
handle all of the data, not just some of it.
Yet, navigating the big data frontier does not come
without challenges. Agency leaders are challenged
to clearly navigate the cultural changes to tech-
nology. As Shelly noted, Most of the challenges
are cultural; the MarkLogic technology is trusted,
proven, and innovative. Big data represents a cul-
tural change, where people need to look at things
differently and do things differently than their pre-
decessors. This can mean everything from a new
business process, a change to workows, and even
changing the way government delivers services.
Now is the time to invest, and re-imagine how
services are delivered and missions accomplished.
This process can start by assessing your data infra-
structure, and ensuring you have the right database
to support your agencys needs.
17 I N N O V A T I O N S T H A T M A T T E R
18 B I G D A T A
Understandi ng
Government s Bi g
Data Challenges //
Before we get to the best practices for big data
adoption, its important to set a foundation. Weve
identied four challenges from our research and
important trends to be aware of as you begin to
deploy big data challenges.
1. Large Capital Investments
The primary big data challenge for governments
is that most agencies have spent an enormous
amount of capital on existing IT infrastructure ,
most of which was implemented long before big
data was even on the radar. This often means that
governments have the ability to capture large
quantities of data, but they dont have a feasible
way to process it in a timely manner. In other cases,
these systems were not built for the variety of data
sources currently available for capture.
MA K I N G B I G D A T A
S T I C K A T Y O U R
A G E N C Y
Former EMDS Division Chief Lt. Col. Bobby Saxon
19 I N N O V A T I O N S T H A T M A T T E R
2. Data Rests in Silos
Perhaps most crucially, many
systems exist in their own data
silos, without a clear method for
extraction and integration. This
challenge is coupled with the
fact that investing in big data
solutions can mean adopting
technologies that are potentially
expensive, untested, less secure
and developed by a third party
outside the government agency.
3. Finding the Needle in the
Haystack
Governments collect so much
data -- and have such a variety of
sources and mediums to choose
from -- that it is difcult to pin-
point value. Even in the private
sector, it is common for corpora-
tions to nd the majority of value
in a minor percentage of the
total data they collect. Therefore,
there is a real risk of investing in
an analytics apparatus that col-
lects and processes redundant
sources of information, especially
since existing systems operate
in isolation. At the same time,
requirements and use cases may
evolve, with capabilities unlock-
ing value previously inconceiv-
able without new advancements.
This presents a challenge for gov-
ernments to gure out the smart-
est and most efcient way to get
the most out of their investments.
4. No One-Size-Fits-All
Solution
There really is no one-size-ts-all
big data solution, just as the chal-
lenges and opportunities for data
use vary for each agency. This
presents potential adopters with
the challenge of nding the right
solution with the right attributes
while dealing with limited capital
and space to experiment.
Our 8 Best
Practi ces
foR Bi g Data
Analysi s //
Given these challenges, we
have identied eight best
practices to help capture
the most value while
expending the minimal
amount of resources.
1. Executive Leadership
The shift to big data from tradi-
tional data analytics may require
more of a revolution in thought
and organization than in the
technical solution itself. For one,
big data operations cannot ex-
ist within a silo. They must be
integrated across the enterprise,
with a common design layout
and operating procedures, as the
EMDS example illustrates. This
requires leadership and vision
that go beyond individual use
cases into a platform for future
integration at all entry points. Ad-
ditionally, there may be friction
among internal units that must
be addressed for proper insti-
tutional cohesion. The Armys
senior leaders are still transform-
ing the culture to ensure all data
is shared across the enterprise,
which is vital to the success of
EMDS.
Finally, there will be costs -- -
nancial, time, labor -- so any
successful implementation will
require a leader to ask difcult
questions to ensure the invest-
ment is worth the cost and to see
the process through during peri-
ods of complication or difculty.
2. Business before
Treasure: Use Cases Come
Before Technical Solutions
The key here is that the technical
solutions should be matched to
business requirements and not
the other way around. It is very
easy to become seduced by the
technology before clearly de-
ned use cases have been found.
This places the organization at
risk of wasting valuable time and
resources implementing a broad
program with only marginal
value added. Therefore, it is vital
that data owners consult with
stakeholders, users and execu-
tive leaders to come up with very
specic business requirements or
problems.
20 B I G D A T A
Technology is important, but the
people are critical, Saxon said.
It is also important that program
managers integrate their plans
into the larger strategic objec-
tives for the organization.
A best practice and lesson
learned from Louisville is that [big
data] needs to be tied to your
strategic vision or your strategic
goal or whatever it is you are
trying to accomplish in your area,
Reno-Weber said.
3. Know Thyself: Dene
Initial Use Cases
Now that the emphasis is on
potential use cases, a clear iden-
tication of the way big data is
specically going to address your
data problems must be estab-
lished. Think of the three Vs we
mentioned earlier in this guide:
volume, velocity and variety.
There are different solutions for
each of these problems, so the
best practice is to nd your de-
nition of what a big data solution
means to your agency.
Ours isnt as much a volume
problem as it is a difcult to
understand problem, Saxon
said. It is the ability to answer
questions that were previously
beyond reach.
Therefore, the Army peeled away
at the broader, general deni-
tions of a big data solution and
crafted its own denition based
on its specic needs.
But how is this done? First,
clearly establish your realization
criteria (such as reduced trans-
action times), business require-
ments (such as security require-
ments) and performance metrics.
It is better to start small and
specic, building on your initial
deployment than to try to capture
everything at once.
The questions you ask mat-
ter. There is a lot of data in city
government and in government
in general. You could run yourself
ragged trying to analyze all of
it and make it all good quality,
Reno-Weber said. The key is that
you should identify a clear proj-
ect or goal to solve, such as the
example of reducing ambulance
time at the hospital.
Although every agencys require-
ments are different, it is possible
to learn from their experiences to
help identify where big data solu-
tions may be applicable to your
organization.
4. Leverage Existing
Resources & then Augment
In most cases, good big data so-
lutions are built on an already ro-
bust IT infrastructure. This allows
organizations to initially make
relatively narrow and precise
upgrades to address big data
problems without remaking their
entire technological enterprise.
It is helpful to think of this pro-
cess like upgrading components
on your personal computer and
analyzing them for efcacy with
each phase. This is in contrast
to replacing your computer each
time a new innovation in process-
ing or storage is released.
Dont be limited by thinking
big data requires a huge infra-
structure and a lot of special-
ized tools, Saxon said. Start a
database. Start accumulating
some of the data. You could even
use Microsoft Excel to start do-
ing data analytics to move down
that path. You will nd nuggets
of information that will help you
sell your story and help you keep
moving forward.
21 I N N O V A T I O N S T H A T M A T T E R
Similarly, any new augmentation
should be able to do two things:
address the immediate data
problems and be easily scalable
to include future requirements for
later phases.
5. Integrate Legacy
Systems
The vast majority of existing
systems currently in operation in
government were not calibrated
to handle big data. As a result, it
may be difcult to quickly pro-
cess or extract pertinent data.
Also, these systems may not be
equipped to handle unstructured
data such as video feeds or social
media. This makes augmenta-
tions and upgrades a near ne-
cessity. However, integration of
older systems is essential to the
enterprise-wide analytical power
big data provides. Extra time and
consideration must be built into
program planning to allow for
proper eld matching techniques,
data integration and processing
output times.
Reno-Weber said Louisville has a
number of systems for re, health,
311, nancial systems and human
resources, and her teams under-
stand the importance of sharing
and connecting these various
data streams.
You see the power of the data
when it is put in the hands of the
users and the operational teams,
she said. We ask, How can we
get our departments to be able
to access their own data and use
insights to best allocate resourc-
es? You cant do that if only one
or two people can get to informa-
tion.
6. Carefully Select Your
Vendors/Architecture
The explosion of advancements
in execution strategies and tech-
nological solutions makes this a
unique time in the relatively short
history of big data. In contrast to
just a few years ago, data owners
now have a wealth of options to
build out their big data infrastruc-
ture, from competing vendors to
a range of architectural options.
Therefore, it is vital that you care-
fully select a solution -- and a
partner -- that best matches your
needs. In this case, it is best to
slow down to speed up. Taking
22 B I G D A T A
your time at this critical juncture
will ensure future phases do not
require large-scale redesigns.
7. Focus on Governance
and Data Quality
Reno-Weber reminds us of the
importance of data quality and
governance strategies: Data is
not always going to be perfect,
but that should not stop you from
analyzing and working it, be-
cause the minute you start doing
that is the moment data quality
starts to get better.
She also notes the importance
of having consistency as to how
data is entered. The challenges
that I think a lot of us see is that
bad information in equals bad in-
formation out, she said. A lot of
the work that we have done over
the last two years has been to
get the quality of the input data
as good as possible, so what we
are analyzing on the back end is
valid as possible.
8. Have a Deep Bench:
Train Employees to Serve in
Support Roles
Data is changing the roles and
responsibilities of the future
workforce. Having only a few
employees who know how to
derive insights from data will not
work. Instead, the capability must
extend across the agency.
You need to have a deeper
bench as it relates to people
who can access and analyze big
data in your organization, Reno-
Weber said. Agencies must place
an emphasis on training and
educating staff at all levels.
Leaping into the world of big
data, however you choose to
dene it, is not a casual or easy
process. Following the best
practices presented here will
help you take the right rst steps
toward capturing value from your
data, which is vital for success.
Each step builds on the last, and
a careful plan and execution on
the front end can mean all the
difference on the other side.
Big data is changing the way
our government operates and is
transforming communities. For
public-sector leaders, now is the
time to understand how to lever-
age data in new and transforma-
tive ways. Theres no denying
the power of data. Our report
has given you access to the case
studies, lessons learned and best
practices from big data leaders.
Now, its up to you to take this
information and bring your big
data knowledge into your agency.
GovLoop is always here to help.
Send us your questions, tell us
your case studies and lets work
in partnership to transform the
public sector.
Louisvilles Chief of the Office of
Performance Improvement,
Theresa Reno-Weber
23 I N N O V A T I O N S T H A T M A T T E R
24 B I G D A T A
How I n-Memory Computi ng
i s Changi ng How We Thi nk
About Bi g Data //
Government has been tasked to
modernize in a dynamic and quickly
changing landscape.
While most agencies have recognized the need
to leverage their data in transformative ways,
they still face challenges in building a sustainable
infrastructure for the future.
With shrinking budgets, agencies are trying to nd
ways to modernize, but also be more cost effec-
tive. One critical way to accomplish this goal is for
more agencies to realize that there are technology
developments that will enable them to scale-up and
out while simultaneously leveraging their existing
infrastructure, said Michael Ho, VP for Software AG
Government Solutions.
One such IT development is the use of in-memory
computing. In-memory computing is a transforma-
tive technology that is already becoming a core
component of high performing applications within
government and with great results, said Ho.
In-memory computing is about achieving massive
scale within your existing infrastructure all while
supercharging the speed of your enterprise applica-
tions so you can tackle big data quickly. Fundamen-
tally, the challenge for any database environment
today is that with even the best built applications
hitting it, the amount of data that each application
user carries is growing exponentially. This is tax-
ing for your database because any time users do
anything, your app must do a round-trip to the data-
base, ultimately slowing it down and leaving users
stuck waiting. So what happens? Organizations
feel forced to continually buy more servers and
database licenses which are costly. In-memory com-
puting moves agency data sets to the applications
memory which means less work for your database
and more speed for your users. Ho said, What it
allows you to do is take any size of data, whether
its massive amounts or even smaller amounts, and
move that data closer to where the user ultimately
consumes it. This eliminates the need to wait for
the transaction gaps you get with normal data-
bases. What took hours or several minutes can take
microseconds with in-memory computing.
Ho believes that in-memory computing is the next
evolution of data storage and processing technol-
ogy. In-memory allows us to take all the systems
of records that you have in place, whether that be
your Hadoop cluster, data warehouse or existing
databases and applications, and speed them up to
help you meet the massive scale of data and de-
mand that you are going to receive from your end
users and enterprise, said Ho.
Modernizing does not need to be a scary proposi-
tion for agencies. Often, modernizing is as much
about learning how to leverage existing technology
as it is about investing in new technology. Ho noted
the benets of in-memory computing to capitalize
on data. Thats where in-memory computing is re-
ally great, it allows you to complement your exist-
ing systems and blend the speed and scale to your
unique infrastructure, said Ho.
For public sector organizations, in-memory comput-
ing means that you do not have to start building
from the ground up. You dont need to re-design
your entire strategy from square one. Its about
taking your existing investments and making them
more valuable by adding the proper pieces to
them, said Ho.
Ho believes that in-memory computing is changing
how organizations think about big data. Big data
as a notion is going to be more than just having
terabytes of data. Its going to be about helping
folks make the decisions they want to make, by
enabling them to see and analyze the data they
have, in the time window of relevance thats opti-
mal. Thats where these technologies are focused,
said Ho.
To help organizations modernize, new technologies
will emerge that will help organizations to capital-
ize on their data. The future of big data is not about
size of data; its about speed and processing to
maximize value from information collected.
25 I N N O V A T I O N S T H A T M A T T E R
What is Big Data?
Big data is characterized as data that arrives in
such enormous quantities, at such a fast rate and
in such a diverse array of mediums that it is impos-
sible to understand or leverage through traditional
methods. Similarly, a big data problem describes a
situation in which you are not able to use your data
to accomplish a goal or deliver a service because
one of those characteristics (volume, velocity,
and variety) prevents you from capturing its value
within a predetermined window of opportunity.
Why Does Big Data Matter?
Data is at the epicenter of the next technological
revolution. The organizations best poised to thrive
are those that treat their data as a valuable com-
modity a raw resource, similar to petroleum or a
precious metal. Big data solutions provide unique
opportunities to turn these raw resources into key
insights and ultimately transform the way your
agency does business.
G o v L o o p s
B i g D a t a
C h e a t
S h e e t
L o o k i n g t o g e t
s m a r t f a s t o n
b i g d a t a ? L o o k n o
f u r t h e r //
26 B I G D A T A
These statistics illustrate how
important big data is:
The Obama administration is
investing $200 million in big
data research projects. (Source)
Research firm IDC estimates
that by 2020, business
transactions on the Internet
will reach 450 billion per day.
(Source)
Ventana Research found that
94 percent of Hadoop users
perform analytics on large
volumes of data not possible
before, 88 percent analyze data
in greater detail and 82 percent
can now retain more of their
data. (Source)
More than 5 billion people are
calling, texting, tweeting and
browsing on mobile phones
worldwide, according to a white
paper from SAS, a business
analytics software and services
firm. (Source)
Poor data across businesses
and the government costs the
U.S. economy $3.1 trillion a
year. (Source)
By 2018, the United States
could have a shortage of
140,000 to 190,000 people
with deep analytical skills to
fill the demand of big data
jobs, according to research
firm McKinsey Global Institute.
(Source)
See more data by visiting:
http://wikibon.org/blog/big-
data-statistics/
How Can I Start My Big Data Journey?
There are a number of signicant challenges to
implementing big data solutions in the public
sector, such as prior investments and the fact
that any big data solution requires customizing
to t each agencys specic needs. Here are
eight best practices to overcome them.
Executive leadership: Effective program
stewardship begins with strong executive
leadership and support.
Business before treasure: Identify
use cases and associated business
requirements before building your
technical solution.
Know thyself: Define a specific big data
problem and solve it before ramping up
throughout your enterprise.
Strong foundation: It is essential to
have a robust existing IT infrastructure.
Leverage existing resources and then
augment them, being careful to allow
room for scalability.
Integrate with legacy systems: Big data
solutions cannot exist in an innovation
silo.
Carefully select your vendors/
architecture: Choose a vendor that best
meets your needs and provides desired
functionality.
Focus on governance and data quality:
The quality of the data going in indicates
the quality of the data going out. Be sure
that you have a standard approach and
are enforcing your governance policies
to ensure high-quality data.
Have a deep bench: Your agency
must invest the proper time in training
employees across your agency. Having a
deep bench ensures that the entire team
can step in and assist as needed.
27 I N N O V A T I O N S T H A T M A T T E R
About GovLoop //
GovLoops mission is to
connect government to improve
government. We aim to inspire
public-sector professionals
by serving as the knowledge
network for government.
GovLoop connects more than
100,000 members, fostering
cross-government collaboration,
solving common problems and
advancing government careers.
GovLoop is headquartered in
Washington, D.C., with a team of
dedicated professionals who share
a commitment to connect and
improve government.
Acknowledgements //
Thank you to Cloudera, MarkLogic
and Software AG Government
Solutions for their support of this
valuable resource for public-sector
professionals.
Authors: Patrick Fiorenza, senior
research analyst, GovLoop, and
Adrian Pavia, GovLoop research
fellow
Designers: Jeff Ribeira, Senior
Interactive Designer, GovLoop, and
Tommy Bowen, Junior Designer,
GovLoop
Editor: Steve Ressler, president
and founder, GovLoop
28 B I G D A T A
29 B I G D A T A
1 1 0 1 1 5 t h S t NW, S u i t e 9 0 0
Wa s h i n g t o n , DC 2 0 0 0 5
P h o n e : ( 2 0 2 ) 4 0 7 - 7 4 2 1
F a x : ( 2 0 2 ) 4 0 7 - 7 5 0 1