Data Mining
Automated Mechanism,
Set alert
Data Mining
Colleen McCue, in Data Mining and Predictive
Analysis, 2007
Is Data Mining Evil?
Further confounding the question of whether to
acquire data mining technology is the heated
debate regarding not only its value in the public
safety community but also whether data mining
reflects an ethical, or even legal, approach to the
analysis of crime and intelligence data. The
discipline of data mining came under fire in the
Data Mining Moratorium Act of 2003.
Unfortunately, much of the debate that followed
has been based on misinformation and a lack of
knowledge regarding these very important tools.
Like many of the devices used in public safety,
data mining and predictive analytics can confer
great benefit and enhanced public safety through
their judicious deployment and use. Similarly,
these same assets also can be misused or
employed for unethical or illegal purposes.
One of the harshest criticisms has addressed
important privacy issues. It has been suggested
that data mining tools threaten to invade the
privacy of unknowing citizens and unfairly target
them for invasive investigative procedures that
are associated with a high risk of false allegations
and unethical labeling of certain groups. The
concern regarding an individual's right to privacy
versus the need to enhance public safety
represents a long-standing tension within the law
enforcement and intelligence communities that is
not unique to data mining. In fact, this concern is
misplaced in many ways because data mining in
and of itself has a limited ability, if any, to
compromise privacy. Privacy is maintained
through restricting access to data and
information. Data mining and predictive analytics
merely analyze the data that is made available;
they may be extremely powerful tools, but they
are tools nonetheless. With data mining,
ensuring privacy should be no different than with
any other technique or analytical approach.
Unfortunately, many of these fears were based on
a misunderstanding of the Total Information
Awareness system (TIA, later changed to the
Terrorism Information Awareness system), which
promised to combine and integrate wide-ranging
data and information systems from both the
public and private sectors in an effort to identify
possible terrorists. Originally developed by the
Defense Advanced Research Projects Agency
(DARPA), this program was ultimately dismantled,
due at least in part to the public outcry and
concern regarding potential abuses of private
information. Subsequent review of the program,
however, determined that its main shortcoming
was related the failure to conduct a privacy impact
study in an effort to ensure the maintenance of
individual privacy; this is something that
organizations considering these approaches
should include in their deployment strategies and
use of data-mining tools.
On the other hand, some have suggested that
incorporation of data mining and predictive
analytics might result in a waste of resources.
This underscores a lack of information regarding
these analytical tools. Blindly deploying resources
based on gut feelings, public pressure, historical
precedent, or some other vague notion of crime
prevention represents a true waste of resources.
One of the greatest potential strengths of data
mining is that it gives public safety organizations
the ability to allocate increasingly scarce law
enforcement and intelligence resources in a more
efficient manner while accommodating a
concomitant explosion in the available
information—the so-called “volume challenge”
that has been cited repeatedly during
investigations into law enforcement and
intelligence failures associated with 9/11. Data
mining and predictive analytics give law
enforcement and intelligence professionals the
ability to put more evidence-based input into
operational decisions and the deployment of
scarce resources, thereby limiting the potential
waste of resources in a way not available
previously.
Regarding the suggestion that data mining has
been associated with false leads and law
enforcement mistakes, it is important to note that
these errors happen already, without data mining.
This is why there are so many checks and
balances in the system—to protect the innocent.
We do not need data mining or technology to
make errors; we have been able to do that without
the assistance of technology for many years.
There is no reason to believe that these same
checks and balances would not continue to
protect the innocent were data mining to be used
extensively. On the other hand, basing our
activities on real evidence can only increase the
likelihood that we will correctly identify the bad
guys while helping to protect the innocent by
casting a more targeted net. Like the difference
between a shotgun and a laser-sited 9mm, there
is always the possibility of an error, but there is
much less collateral damage with the more
accurate weapon.
Again, the real issue in the debate comes back to
privacy concerns. People do not like law
enforcement knowing their business, which is a
very reasonable concern, particularly when viewed
in light of past abuses. Unfortunately, this
attitude confuses process with input issues and
places the blame on the tool rather than on the
data resources tapped. Data mining can only be
used on the data that are made available to it.
Data mining is not a vast repository designed to
maintain extensive files containing both public
and private records on each and every American,
as has been suggested by some. It is an analytical
tool. If people are concerned about privacy issues,
then they should focus on the availability of and
access to sensitive data resources, not the
analytical tools. Banning an analytical tool
because of fear that it will be misused is similar to
banning pocket calculators because some people
use them to cheat on their taxes.
As with any powerful weapon used in the war on
terrorism, the war on drugs, or the war on crime,
safety starts with informed public safety
consumers and well-trained personnel. As is
emphasized throughout this text, domain
expertise frequently is the most important
component of a well-informed, professional
program of data mining and predictive analytics.
As such, it should be seen as an essential
responsibility of each agency to ensure active
participation on the part of those in the know;
those professionals from within each organization
that know where the data came from and how it
will be used. To relinquish the responsibility for
analysis to outside organizations or consultants
should be viewed in the same way as a suggestion
to entirely contract patrol services to a private
security corporation: an unacceptable abdication
of an essential responsibility.
Unfortunately, serious misinformation regarding
this very important tool might limit or somehow
curtail its future use when we most need it in our
fight against terrorism. As such, it is incumbent
upon each organization to ensure absolute
integrity and an informed decision-making
process regarding the use of these tools and their
output in an effort to ensure their ongoing
availability and access for public safety
applications.
View chapter Purchase book
Multivariate Analysis: Overview
I. Olkin, A.R. Sampson, in International Encyclopedia
of the Social & Behavioral Sciences, 2001
6.7 Data Mining
Data mining refers to a set of approaches and
techniques that permit ‘nuggets’ of valuable
information to be extracted from vast and loosely
structured multiple data bases. For example, a
consumer products manufacturer might use data
mining to better understand the relationship of a
specific product's sales to promotional strategies,
selling store's characteristics, and regional
demographics. Techniques from a variety of different
disciplines are used in data mining. For instance,
computer science and information science provide
methods for handling the problems inherent in
focusing and merging the requisite data from
multiple and differently structured data bases.
Engineering and economics can provide methods for
pattern recognition and predictive modeling.
Multivariate statistical techniques, in particular,
clearly play a major role in data mining.
Multivariate notions developed to study relationships
provide approaches to identify variables or sets of
variables that are possibly connected. Regression
techniques are useful for prediction. Classification
and discrimination methods provide a tool to identify
functions of the data that discriminate among
categorizations of an individual that might be of
interest. Another very useful technique for data
mining is cluster analysis that groups experimental
units which respond similarly. Structural methods
such as principal components, factor analysis, and
path analysis are methodologies that can allow
simplification of the data structure into fewer
important variables. Multivariate graphical methods
can be employed to both explore databases and then
as a means for presentation of the data mining
results.
A reference to broad issues in data mining is given by
Fayyad et al. (1996). Also see Exploratory Data Analysis:
Multivariate Approaches (Nonparametric Regression).
View chapter Purchase book
Introduction
Jiawei Han, ... Jian Pei, in Data Mining (Third
Edition), 2012
1.7.5 Data Mining and Society
How does data mining impact society? What steps
can data mining take to preserve the privacy of
individuals? Do we use data mining in our daily lives
without even knowing that we do? These questions
raise the following issues:
■ Social impacts of data mining: With data mining
penetrating our everyday lives, it is important to
study the impact of data mining on society. How
can we use data mining technology to benefit
society? How can we guard against its misuse?
The improper disclosure or use of data and the
potential violation of individual privacy and data
protection rights are areas of concern that need to
be addressed.
■ Privacy-preserving data mining: Data mining will
help scientific discovery, business management,
economy recovery, and security protection (e.g.,
the real-time discovery of intruders and
cyberattacks). However, it poses the risk of
disclosing an individual's personal information.
Studies on privacy-preserving data publishing and
data mining are ongoing. The philosophy is to
observe data sensitivity and preserve people's
privacy while performing successful data mining.
■ Invisible data mining: We cannot expect everyone
in society to learn and master data mining
techniques. More and more systems should have
data mining functions built within so that people
can perform data mining or use data mining
results simply by mouse clicking, without any
knowledge of data mining algorithms. Intelligent
search engines and Internet-based stores perform
such invisible data mining by incorporating data
mining into their components to improve their
functionality and performance. This is done often
unbeknownst to the user. For example, when
purchasing items online, users may be unaware
that the store is likely collecting data on the
buying patterns of its customers, which may be
used to recommend other items for purchase in
the future.
These issues and many additional ones relating to the
research, development, and application of data
mining are discussed throughout the book.
View chapter Purchase book
Process Models for Data Mining
and Predictive Analysis
Colleen McCue, in Data Mining and Predictive
Analysis (Second Edition), 2015
Abstract
Data mining and predictive analytics can best be
understood as a process, rather than specific
technology, tool, or tradecraft. Chapter 4 includes an
overview of four complementary approaches to
analysis: the Central Intelligence Agency (CIA)
Intelligence Process, the CRoss Industry Standard
Process for Data Mining (CRISP-DM), SEMMA, and
the Actionable Mining and Predictive Analysis process
developed specifically for the operational public safety
and security environment. The Actionable Mining
and Predictive Analysis process addresses unique
requirements and constraints associated with the
applied setting, including data access and availability,
public safety-specific evaluation, and the requirement
for operationally relevant and actionable output. Data
privacy and security also are addressed.
View chapter Purchase book
About ScienceDirect Remote access
Terms and conditions Privacy policy
We use cookies to help provide and enhance our service and tailor content and ads. By continuing you agree to the use of
cookies.
Copyright © 2020 Elsevier B.V. or its licensors or contributors. ScienceDirect ® is a registered trademark of Elsevier B.V.
Related terms:
Design Documentation,
Security Officer, System
Configuration, Systems
Design
View all Topics
About this page
Domain 3: Security Engineering
(Engineering and Management of
Security)
Eric Conrad, ... Joshua Feldman, in CISSP Study
Guide (Third Edition), 2016
Data Mining
Data mining searches large amounts of data to
determine patterns that would otherwise get “lost in
the noise.” Credit card issuers have become experts in
data mining, searching millions of credit card
transactions stored in their databases to discover
signs of fraud. Simple data mining rules, such as “X
or more purchases, in Y time, in Z places” can be
used to discover credit cards that have been stolen
and used fraudulently.
Data mining raises privacy concerns: imagine if life
insurance companies used data mining to track
purchases such as cigarettes and alcohol, and denied
claims based on those purchases.
View chapter Purchase book
Introduction
Vijay Kotu, Bala Deshpande PhD, in Predictive
Analytics and Data Mining, 2015
1.1 What Data Mining Is
Data mining, in simple terms, is finding useful
patterns in the data. Being a buzzword, there are a
wide variety of definitions and criteria for data
mining. Data mining is also referred to as knowledge
discovery, machine learning, and predictive analytics.
However, each term has a slightly different
connotation depending upon the context. In this
chapter, we attempt to provide a general overview of
data mining and point out its important features,
purpose, taxonomy, and common methods.
Data mining starts with data, which can range from a
simple array of a few numeric observations to a
complex matrix of millions of observations with
thousands of variables. The act of data mining uses
some specialized computational methods to discover
meaningful and useful structures in the data. These
computational methods have been derived from the
fields of statistics, machine learning, and artificial
intelligence. The discipline of data mining coexists
and is closely associated with a number of related
areas such as database systems, data cleansing,
visualization, exploratory data analysis, and
performance evaluation. We can further define data
mining by investigating some its key features and
motivation.
1.1.1 Extracting Meaningful Patterns
Knowledge discovery in databases is the nontrivial
process of identifying valid, novel, potentially useful,
and ultimately understandable patterns or
relationships in the data to make important decisions
(Fayyad et al., 1996) The term “nontrivial process”
distinguishes data mining from straightforward
statistical computations such as calculating the mean
or standard deviation. Data mining involves inference
and iteration of many different hypotheses. One of
the key aspects of data mining is the process of
generalization of patterns from the data set. The
generalization should be valid not just for the data set
used to observe the pattern, but also for the new
unknown data. Data mining is also a process with
defined steps, each with a set of tasks. The term
“novel” indicates that data mining is usually involved
in finding previously unknown patterns in the data.
The ultimate objective of data mining is to find
potentially useful conclusions that can be acted upon
by the users of the analysis.
1.1.2 Building Representative Models
In statistics, a model is the representation of a
relationship between variables in the data. It
describes how one or more variables in the data are
related to other variables. Modeling is a process in
which a representative abstraction is built from the
observed data set. For example, we can develop a
model based on credit score, income level, and
requested loan amount, to determine the interest
rate of the loan. For this task, we need previously
known observational data with the credit score,
income level, loan amount, and interest rate. Figure
1.1 shows the inputs and output of the model. Once
the representative model is created, we can use it to
predict the value of the interest rate, based on all the
input values (credit score, income level, and loan
amount).
Sign in to download full-size image
Figure 1.1. Representative model for Predictive Analytics.
In the context of predictive analytics, data mining is
the process of building the representative model that
fits the observational data. This model serves two
purposes: on the one hand it predicts the output
(interest rate) based on the input variables (credit
score, income level, and loan amount), and on the
other hand we can use it to understand the
relationship between the output variable and all the
input variables. For example, does income level really
matter in determining the loan interest rate? Does
income level matter more than credit score? What
happens when income levels double or if credit score
drops by 10 points? Model building in the context of
data mining can be used in both predictive and
explanatory applications.
1.1.3 Combination of Statistics, Machine Learning,
and Computing
In the pursuit of extracting useful and relevant
information from large data sets, data mining derives
computational techniques from the disciplines of
statistics, artificial intelligence, machine learning,
database theories, and pattern recognition.
Algorithms used in data mining originated from
these disciplines, but have since evolved to adopt
more diverse techniques such as parallel computing,
evolutionary computing, linguistics, and behavioral
studies. One of the key ingredients of successful data
mining is substantial prior knowledge about the data
and the business processes that generate the data,
known as subject matter expertise. Like many
quantitative frameworks, data mining is an iterative
process in which the practitioner gains more
information about the patterns and relationships
from data in each cycle. The art of data mining
combines the knowledge of statistics, subject matter
expertise, database technologies, and machine
learning techniques to extract meaningful and useful
information from the data. Data mining also typically
operates on large data sets that need to be stored,
processed, and computed. This is where database
techniques along with parallel and distributed
computing techniques play an important role in data
mining.
1.1.4 Algorithms
We can also define data mining as a process of
discovering previously unknown patterns in the data
using automatic iterative methods. Algorithms are
iterative step-by-step procedure to transform inputs
to output. The application of sophisticated algorithms
for extracting useful patterns from the data
differentiates data mining from traditional data
analysis techniques. Most of these algorithms were
developed in recent decades and have been borrowed
from the fields of machine learning and artificial
intelligence. However, some of the algorithms are
based on the foundations of Bayesian probabilistic
theories and regression analysis, originated hundreds
of years ago. These iterative algorithms automate the
process of searching for an optimal solution for a
given data problem. Based on the data problem, data
mining is classified into tasks such as classification,
association analysis, clustering, and regression. Each
data mining task uses specific algorithms like
decision trees, neural networks, k-nearest neighbors,
k-means clustering, among others. With increased
research on data mining, the number of such
algorithms is increasing, but a few classic algorithms
remain foundational to many data mining
applications.
View chapter Purchase book
Data Mining Trends and
Research Frontiers
Jiawei Han, ... Jian Pei, in Data Mining (Third
Edition), 2012
Mining Multimedia Data
Multimedia data mining is the discovery of
interesting patterns from multimedia databases that
store and manage large collections of multimedia
objects, including image data, video data, audio data,
as well as sequence data and hypertext data
containing text, text markups, and linkages.
Multimedia data mining is an interdisciplinary field
that integrates image processing and understanding,
computer vision, data mining, and pattern
recognition. Issues in multimedia data mining
include content-based retrieval and similarity search,
and generalization and multidimensional analysis.
Multimedia data cubes contain additional dimensions
and measures for multimedia information. Other
topics in multimedia mining include classification and
prediction analysis, mining associations, and video and
audio data mining (Section 13.2.3).
View chapter Purchase book
Theoretical Considerations for
Data Mining
Robert Nisbet, ... Gary Miner, in Handbook of
Statistical Analysis and Data Mining Applications,
2009
What Is Data Mining?
Data mining can be defined in several ways, which
differ primarily in their focus on different aspects of
data mining. One of the earliest definitions is
The non-trivial extraction of implicit, previously
unknown, and potentially useful information from
data (Frawley et al., 1991).
As data mining developed as a professional activity, it
was necessary to distinguish it from the previous
activity of statistical modeling and the broader activity
of knowledge discovery. For the purposes of this
handbook, we will use the following working
definitions:
• Statistical modeling: The use of parametric
statistical algorithms to group or predict an
outcome or event, based on predictor variables.
• Data mining: The use of machine learning
algorithms to find faint patterns of relationship
between data elements in large, noisy, and messy
data sets, which can lead to actions to increase
benefit in some form (diagnosis, profit, detection,
etc.).
• Knowledge discovery: The entire process of data
access, data exploration, data preparation,
modeling, model deployment, and model
monitoring. This broad process includes data
mining activities, as shown in Figure 2.1.
Sign in to download full-size image
Figure 2.1. The relationship between data mining and knowledge discovery.
As the practice of data mining developed further, the
focus of the definitions shifted to specific aspects of
the information and its sources. In 1996, Fayyad et al.
proposed the following:
Knowledge discovery in databases is the non-trivial
process of identifying valid, novel, potential useful,
and ultimately understandable patterns in data.
The second definition focuses on the patterns in the
data rather than just information in a generic sense.
These patterns are faint and hard to distinguish, and
they can only be sensed by analysis algorithms that
can evaluate nonlinear relationships between
predictor variables and their targets and themselves.
This form of the definition of data mining developed
along with the rise of machine learning tools for use
in data mining. Tools like decision trees and neural
nets permit the analysis of nonlinear patterns in data
easier than is possible in parametric statistical
algorithms. The reason is that machine learning
algorithms learn the way humans do—by example,
not by calculation of metrics based on averages and
data distributions.
The definition of data mining was confined originally
to just the process of model building. But as the
practice matured, data mining tool packages (e.g.,
SPSS-Clementine) included other necessary tools to
facilitate the building of models and for evaluating
and displaying models. Soon, the definition of data
mining expanded to include those operations in
Figure 2.1 (and some include model visualization
also).
The modern Knowledge Discovery in Databases
(KDD) process combines the mathematics used to
discover interesting patterns in data with the entire
process of extracting data and using resulting models
to apply to other data sets to leverage the information
for some purpose. This process blends business
systems engineering, elegant statistical methods, and
industrial-strength computing power to find structure
(connections, patterns, associations, and basis
functions) rather than statistical parameters (means,
weights, thresholds, knots). In Chapter 3, we will
expand this rather linear organization of data mining
processes to describe the iterative, closed-loop
system with feedbacks that comprise the modern
approach to the practice of data mining.
View chapter Purchase book
Shopping cart Advertise Contact and support