2010 International Conference on Advances in Computer Engineering
Clustering Based URL Normalization Technique for
Web Mining
Naresh Kumar Nagwani,
Asst Prof., Department Of CSE,
NIT Raipur
nknagwani.cs@nitrr.ac.in
Abstract – URL (Uniform Resource Locator) normalization is an
important activity in web mining. Web data can be retrieved in
smoother way using effective URL normalization technique. URL
normalization also reduces lot of calculations in web mining
activities. A web mining technique for URL normalization is
proposed in this paper. The proposed technique is based on
content, structure and semantic similarity and web page
redirection and forwarding similarity of the given set of URLs.
Web page redirection and forward graphs can be used to
measure the similarities between the URL’s and can also be used
for URL clusters. The URL clusters can be used for URL
normalization. A data structure is also suggested to store the
forward and redirect URL information.
Keywords – URL Normalization, Clustering, Web Page Forward
and Redirect Similarity Tree.
I. INTRODUCTION
URL normalization is performed by crawlers to determine if
two syntactically different URLs are equivalent. The ultimate
aim of the URL normalization is to reduce redundant Web
crawling by having a set of URLs which point to a unique set
of Web pages and to improve search engines for better and
unique results. URL normalization is deployed by search
engines to determine the importance of Web pages as well as
to avoid indexing same Web pages. URL normalization is also
refers as the process of identifying the similar and equivalent
URL’s. The equivalent URL’s points to the same required
resource which is in web user’s interest.
A. Standard URL Normalization Methods
There is several type of normalization that may be performed
for URL normalization, some of the common techniques are
Converting the scheme and host to lower case, Removing
directory index, Removing the fragment, Removing “www”
as the first domain label, Removing arbitrary query string
variables and Normalization based on URL lists. Standard
URL Normalization are classified in the following three major
categories:
1. Syntax based normalization techniques
These methods are based on the syntax of the URL. Some of
the examples are case normalization which converts all letters
at scheme and authority components to lower case. Percent-
978-0-7695-4058-0/10 $26.00 © 2010 IEEE
DOI 10.1109/ACE.2010.47
encoded normalization is another technique which decode any
percent-encoded octet that corresponds to an unreserved
character, such as %2D for hyphen and %5F for underscore
etc. Another common technique is path segment
normalization which removes dot segments from the path
components.
2. Scheme based normalization techniques
From the scheme of the web application and its propertied
URL can be normalized these technique comes under the
scheme based normalization technique. Some of the examples
of this technique are adding trailing ‘/’ after the authority
component of URL, removing default port number, such as 80
for http scheme and truncating the fragment of URL, e.g.
http://www.example.com/name.html#ali is truncated to
http://www.example.com/name.html etc.
3. Protocol based normalization techniques
Using communication protocol for the web applications some
techniques are invented, which comes under the category of
protocol based normalization techniques. Some of the
examples are only appropriate when the results of accessing
the resources are equivalent for example;
http://example.com/data is directed to
http://example.com/data/ by http origin server.
B. Web Similarity Types
Web Page can have the following three types of similarities:
1. Web Content Similarity: Web content similarity refers to
the similarity among the web pages whose actual web contents
like html data, images, tables etc. are similar. Web content
similarity is a important measure in URL normalization.
Whenever the two URL’s web page content will be similar
then both of the URL can be assumed as equivalent.
2. Web Structure Similarity: URL points to a particular
specific web page, which not only contains the useful
information but is also the source of another web page. A user
can number of useful links on reaching to a particular web
page, and may select another page from that page. In this way
these sequences of web pages forms a web structure. For the
similar kind of operations on web generally the structure is
also similar. For example doing sing-up or registration process
in a web application consist of some typical set of operations
and a typical traversal of some web pages like first filling the
349
required information, then email verification then continuation done using the content, structure and semantic similarity
etc. Whenever similarity of the sequence of web page traversal between the different URL’s.
comes into the picture for any analysis it is technically known
III. PROPOSED TECHNIQUE
as the web structure similarity.
3. Web Semantic Similarity – Web semantic similarity is one
This method is based on the structure in which the URLs are
of the advance parameter to check the similarity between two
traversed. One the structure is available; the clustering
URL’s. It focuses on the meaning of the web pages rather than algorithm is applied to discover the different URL cluster for
their contents and structures. If two URL’s are meaningful URL normalization. The structure also includes the
similar then they are said to be as semantic similar web pages. forwarding or redirecting a URL from another URL.
Figure-1 depicts the proposed algorithm for URL
C. Clustering normalization. A set of URL’s are given input to the proposed
Clustering is a technique of creating groups of similar objects. model. Out of these URL’s set Forward-Redirect tree is
Each group, called cluster, consists of objects that are similar generated and also similarity information are captured, the
between themselves and dissimilar to objects of other groups. clustering algorithm is applied to create the URL clusters.
There exists large number of clustering algorithms. The Figure 2 depicts the possible outcome of the algorithm. In this
clustering algorithm are categorized as partitioning based,
hierarchical, density based, grid based and model based way the URL normalization can be achieved.
clustering algorithms. K-Mean is a popular partitioning based
clustering algorithm. In this paper clustering is applied over a
set of given URL’s and group of similar URL’s are created for
URL normalization.
This paper is organized in five sections. Section two discusses
about the related and previous work done in the similar area.
Section three consists of proposed technique for URL
normalization with pseudo code of the algorithm and section
four discussed about the conclusion and future scope of the
proposed technique.

II. RELATED WORK DONE

Number of techniques exists and suggested by researchers for

URL normalization. Some of them are studied and mentioned
in this section. Kim, Jeong, and Lee [3] given a method of
URL normalization in which URL’s strings are transformed
into canonical form and duplicate URL’s are eliminated. Lee,
S.H., Kim, S.J. and Hong [4] have suggested three additional
URL normalization steps and two parameters redundancy rate
and coverage loss rate for effective URL normalization. The
redundancy rate shows how many web pages are duplicated
Fig.1. Clustering based URL normalization technique.
due to equivalent URLs.coverage loss rate shows how much
valid web pages are lost due to false positives.
Soon and Lee [1, 5] has proposed to enhance the standard
URL normalization by incorporating the semantically
meaningful metadata of the Web pages. The metadata taken
into account are the body texts of the Web pages, which can
be extracted during HTML parsing. Given a URL which has
undergone the standard normalization mechanism, URL
signature is constructed by hashing or fingerprinting the body
text of the associated Web page using Message-Digest
algorithm. URLs which share identical signatures are
considered to be equivalent in our scheme. The problem of
different URLs with similar text have been studied by Yossef,,
Keidar. Schonfeld [2], this problem is named as DUST.
DUST-BUSTER algorithm is proposed to solve this problem.
A new URL clustering based URL normalization
approach is proposed in this paper. The URL clustering is Fig.2. URL cluster tree for a given URL set.

350
Forward and redirect tree is a data structure for representing
the web page forwarding and redirecting operations. The
URL’s are the parent nodes and the children nodes represent
the sequence of page traversal. Web page forwarding refers to
the process by which URL can forward the web request to
some other URL for further operations. Web page redirecting
is process by which the request URL is automatically changed
to some other URL for further process, since the location is
got changed or there may be some other reason.
A. Forward and Redirect Tree
Forward-and-Redirect tree is the basic data structure for the
proposed model. This tree is created by mapping the forward
URL’s information and redirect URL’s information for a
given URL. Transition from node N1 to N2 is referred to as
forward or redirects transition for a given URL present at node
N1. The root node in the Forward-and-Redirect tree is the first
input URL.
B. Algorithm for Proposed Model
The algorithm has four main steps that are shortly described in
the following section.
1. The first step is to define the initial cluster centroids.
These centroids are initialized randomly.
2. The second step is to compute distances from all cluster
centroids for all URL’s and assign each URL to the
nearest cluster. The distance measure used is the function
of web content similarity, web structure similarity and
web semantic similarity. It also includes the forward and
redirect URL from a URL.
3. The third step is to calculate new cluster centroids.
4. The final step is to calculate the value of a cost function
that is based on a similarity distance measure.
C. Pseudo-code of Algorithm
The pseudo-code of the above mentioned algorithm is
mentioned in this section. The comments are also given for the
explanation.
/**** Input is a Set U = {u1, u2,…,uN} constituting of N
number of URL’s
Output is a Set of K URL cluster’s {{u11, u12,…}, {u21,
u22,…},…, {uK1, uK2,…} } ****/
/**** Function Dist(i,t) returns the distance measure of URL i
and cluster t. This function is needed later in the clustering
algorithm .The distance function is a function of four
parameters–
1. Forward & Redirect cost from a URL
2. Content Similarity between a pair of URL’s
3. Structure similarity between a pair of URL and
4. Semantic Similarity between a pair of URL. ****/
Dist (i,t)
dist = 0
351
dist = Φ(FFwd,Redirect(i,t) , Fcontent-similarity(i,t) , Fstructure-
similarity(i,t) , Fsemantic-similarity(i,t) )
return dist
/*****
Set the initial cluster centres to be random numbers. FRDisttj
denotes the average distance value of the forward and redirect
from a url for the cluster t, CSimtj , StrSimtj , SemSimtj denotes
the average content similarity measure, structural similarity
measure and semantic similarity measures . ****/
for t = 1 to k /*Initializing the cluster centers. */
FRDisttj = random number
CSimtj = random number
StrSimtj = random number
SemSimtj = random number
endfor
repeat
for i = 1 to k /* For K Clusters */
Calculate the new similarities mean.
for j = 1 to N /* N Number of URL’s*/
for each URL j calculate the Dist(j,i)
if(Dist(j,i) < ε) /* ε is the minimum
threshold value for the distance */
assign URL j to the cluster i
endfor
endfor
until there is no new assignment
IV. CONCLUSION AND FUTURE WORK
In this paper a clustering based web mining technique for
URL normalization is proposed. The similar type of URL’s
can be clustered together for normalizing the URL’s. A new
data structure named forward-and-redirect tree is also
proposed for web mining. The future scope related to the
proposed algorithm could be studying and simplifying the
various similarity measurement algorithms for the web mining
and implementing the proposed technique over a standard
URL repository.
REFERENCES
[1] Lay-Ki Soon Sang Ho Lee, “Enhancing URL Normalization using
Metadata of Web Pages”, Computer and Electrical Engineering,
International Conference on, International Conference on Computer and
Electrical Engineering, pp. 331-335, 2008.
[2] Bar-Yossef, Z., Keidar, I., Schonfeld, U., “Do Not Crawl in the DUST:
Different URLs with Similar Text”, in the Proceedings of the
International World Wide Web Conference (WWW 2007), pp. 111 –
120, May 2007.
[3] Sung Jin Kim, Hyo Sook Jeong, and Sang Ho Lee: Reliable Evaluations
of URL Normalization, ICCSA 2006, LNCS 3984, pp. 609 – 617, 2006.
[4] Lee, S.H., Kim, S.J. and Hong, S.: On URL Normalization, Springer-
Verlag Lecture Notes in Computer Science, Vol. 3481. pp. 1076-1085,
2005.
[5] Lay-Ki Soon, Sang Ho Lee, “Identifying Equivalent URLs using URL
Signatures”, IEEE International Conference on Signal Image
Technology and Internet Based Systems, 2008.