An Approach To Topic-Specific Web Resource Discovery

K Naresh Babu, S.Ramanjaneyulu Assistant Professor, Department of CSE, Geethanjali college of Engineering & Technology, A.P., India
nareshbabu6513@gmail.com

Assistant Professor, Department of CSE, Geethanjali college of Engineering & Technology, A.P., India
ramanji.csit@gmail.com

Abstract Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it.The focus has now been moved to invisible Web or hidden Web, which consists of large warehouse of useful data such as images, sounds, presentations and many other types of media. To utilize such data, there is a need for specialized program to locate those sites as we do with search engines.Web crawlers are one of the most crucial components in search engines and their optimization would have a great effect on improving the searching efficiency. Topic-specific web crawler collects relevant web pages of interested topics from the Internet, there are many relevant researches focusing on topic-specific crawling. However few works detail the topic-specific crawling with the user interests. In this paper, we present a new user interests model to optimize the performance of the topic-specific crawler. The crawler can learn from the previous experience to improve the proportion of the number of relevant pages and the number of the whole pages by using the user information, which is collected by data mining approach.
Index Terms-Information retrieval,Web mining,

large part of the Web is available behind search interfaces and is reachable only when users fill up those forms with set of keywords or queries [2] [4].These pages are often referred to as the Hidden Web [3] or Deep Web [2]. The data in digital libraries,various government organizations, companies is available through search forms. Formally, a deep Web site is a Web server that provides information maintained in one or more back-end Web databases,each of which is searchable through one or more HTML forms as its query interfaces [4]. The hidden Web is qualitatively different from surface Web in the sense that it contains real data on different subjects.According to many research studies, the size of the Hidden Web rapidly increases as more and more of organizations put their valuable content online through an easy-to-use Web interface [2] and is estimated as more than 550 times larger than the surface Web in 2001[1]. As the volume of information grows, there is a need for tools and techniques to utilize the information. It has become very important to automatically discover hidden Web sites through an easy to use interface. The traditional Web crawler automatically traverse, retrieve pages and build a large repository of Web pages. Retrieving data from hidden Web sites has two tasks: resource discovery and content extraction [5].The first task deals with automatically finding the relevant Web sites containing the hidden information.The second task deals with obtaining the information from those sites by filling out forms with relevant keywords. This paper deals with locating relevant forms that serve as the entry points to the hidden Web data using a multi-agent based Web mining crawler. Finding searchable forms is useful in the following fields [6]:

data mining, information process, knowledge discovery.

1 INTRODUCTION Given the large volume of Web pages, users increasingly use search engines to find specific information. These search engines index only static Web pages. Recent literature shows that

Entry points for deep Web Derive source descriptions in the databases Form matching to find correspondences among attributes

1.1. Web Crawlers Crawler is one of the most critical elements in a search engine. It traverses the web by following the hyperlinks and storing downloaded documents in a large database that will later be indexed by search engine for efficient responses to users' queries.Crawlers are designed for different purposes and can be divided into two major categories. Highperformance crawlers form the first category. As the name implies, their goal is to increase the performance of crawling by downloading as many documents as possible in a certain time. They use simplest algorithms such as Breadth First Search (BFS) to reduce running overhead. In contrast, the latter category doesnt address the issue of performance at all but tries to maximize the benefit obtained per downloaded page. Crawlers in this category are generally known as focused Crawlers. Their goal is to find many pages of interest using the lowest possible bandwidth. They attempt to focus on a certain subject for example pages in a specific topic such as scientific articles, pages in a particular language, mp3 files, images etc. 2 Related Work Conventionally, agent based Web mining systems can be classified under three categories [14]: intelligent search agents, information filtering / categorization and personalized Web agents. Several intelligent Web agents like Harvest[15], ShopBot, iJADE Web miner, have been developed to search for relevant information using domain characteristics and user profiles to organize and interpret the discovered information. An adaptive Web search system based on reactive architecture has been presented in . The system is based on ant searching behaviour and proved to be robust against environment alternations and adaptive to users need. An adaptive meta search engine was developed based on neural network based agent which improves search results by computing users relevance feedback. 2.1 Related Works to Focused Crawlers

Focused Crawler was introduced by Chakrabarti in 1999. Filippo Menczer et al evaluated a Topic-Driven web crawler and compared their crawler with different crawling strategies in their paper. Jason Rennie and Andrew McCallum used Reinforcement Learning. They measured the probability that a hyperlink in a page is likely to be linked to a relevant page, according to the texts in the hyperlink neighborhood and as a result their crawler follows the link with the highest probability value. Their job is a part of CORA, a domain-specific search engine containing computer science research papers. For each topic they first retrieve a specific number of top weighted retrieved pages from Google for that topic and then they extract some other keywords from them. For example most pages returned by Google for a search on Information Retrieval contain the word indexer. So, indexer is defined as a sub-topic of the topic Information Retrieval and the crawler will also look for pages containing the word indexer. As a result, their crawler returns a greater number of pages and with greater precision. 3. TSCU (TOPIC-SPECIFIC CRAWLER WITH USER INTERESTS) A. System Architecture ( figure 1) The architecture of our system consists of three main parts: 1) Metasearch downloader We collect the pages by Google and other benchmark search engine APIs, we use this interface to download the rough web pages, and then we use top pages( 10 pages per each) to crawl the web pages and alter the order of the pages according to the users interests which are acquired by the datamining approaches. 2) Classifier Because of the topic-specific and the classification of users,we need a classification, It can specify the users into two categories, the login user and the guest, we use the classifier to define the initial urls, according to users interests graph, the better urls can be found to be the initial urls. The classifier is supported by the Bayesian algorithm. 3) Users model(The users interest tree and initial seedsextension graph ) As we know, we can use many methods to datamine usersinterest, but one cannot only have a preference, For each preference, it has a set of basic urls, using this methods,the users initial seeds extension graph can be created automatically.We create the users interest tree in order to prepare for

collaborative filtering (social filtering),this can estimate the users interest by the user who has the same preference of the new user, we can put the pages in the order of the old users, especially the new user is a guest about whom we do not have any information from the histories.

(1) The algorithm uses the vector space model like other algorithm, each page can be seen to be n-dimensionvector X={x1,x2, ,xn} can be considered to the ndimension attributes (A1,A2,A3,,An) . (2) Theory basis: assuming there are m different categorizations, C1, C2Cm. Given an unknown categorization data sample X (web page).

Figure 1. System architecture B. Relevant Techniques 1) Nave Bayesian classifiers Assuming X is a sample that we do not which categorization it belongs to.H is an assumption, if sample X belongs to a particular category, so the categorization problem is to determine the value p(H|X).p(H|X) is the probability of H on the condition of X.for example, Assuming the data samples as the fruits, the attributes that describe the fruit are color and shape.X is considered to be red and round,H is the assumption that X is an apple, so the p(H|X)describes the probability that X is an apple if X is red and round.Similarly P(X|H) is the X probability based on H, in the example described above, if X is an apple, the probability that it is red and round can be described to be P(X|H). In the categorization of web pages, the P(X),P(H) and P(X|H) can be acquired from the data set, The Bayesian rule describes how to get p(H|X) according to the P(X),P(H)and P(X|H),the related formula is given below.

The Bayesian Algorithm using in the process of web pages categorization

The Bayesian categorizer has the least error probability compared to other categorizer. 2) The user interest tree and initial seeds extension graph implementation. We create the user interests tree in order to recommend the pages to the user who has the same or similar to the given user. The users who have the same interest will be put into the same group. Especially the user is the new or the guest, we do not have any relevant information of him or her. The interests tree has several hierarchies (figure 2).The initial seeds context extension graph has several layers too, it is expanded by the algorithm Hits. The basic seeds is in the most inner layer, we usually call them hub seeds, which are fetched by the metasearch downloader, each seeds can be gotten by entering the query, we get the top pages with a limit number, for example, 10 pages per a

search engine. By the web pages extension, Our crawler TSCU can return the pages ,then we make use of the user model to modify the order of the web pages or delete the pages which user have no interest in. 3) Data-mining Approaches For Acquiring The UsersInterests It is important to know which web pages are the userspreferences, the users experience can be gotten by the browsing histories or other information, if the user is a guest,i.e, the tree has no information about him, as we described above, we can recommend the pages of a user which is most likely similar to him. Actually, the users interests are determined by the topics they are interested in. a) The time that user stay When a person is interested in a topic of categorization, he will browse the pages longer than others, however, The time that users spend on pages is also determined by the length of the pages, so the feedback of rt is direct ratio to the read time but inverse ratio to the length of web pages, then our user model records the topic and the time the user spent on it. b) The frequencies of the click In the process of the browsing, users must click the urls link or anchor texts so that they can get to the target web pages. The more the user click on pages, the more interesting this pages the users are interested in, otherwise, user will leave immediately, so our user model, user interest tree, will record the click frequencies cf and the topic which the web pages belong to. c) Save or print the pages or not However, in some case, the users have other things to do,they have no time to browse, so they dont click the pages, but they save the pages or prints the pages on the paper in order to read them later when they are idle, also our model record the action to the sv,saving is 1,otherwise not saving is 0,sometimes the user only partly need the pages, they copy the part of the content of the pages, we record it to be cp,copying is 1,not copying is 0. d) Search something at the present literatures When someone is interested in the pages, they may explore some things in the pages using the search engine the web stations have, our user model record the action as fe and the topic keywords, searching is 1,not searching is 0. Given the five factors of the user action, we define the implicit feedback f (i) below.

The users interest is changed from time to time, we can get the changes by the feedback, assuming that wij is the jth character item weight of the web page i,wqj is the jth character item weight of query q.

C. Creating The Interests Tree According to described above, we make use of the W to construct the trees The structure of the trees: the root node of the tree is the whole category (entity set), the others are the interesting nodes.Every node is composed of the two items group (topic,keyword, weight) .for example, one interest tree is described below.

Figure 2. A users interest tree If the user enter the keyword java, it means two aspect: one is the java program language, the another is one kind of coffee,according to the user tree, our crawler TSCU will put the coffee results on the top and then the computer program language.

4. EXPERIMENT In order to examine the performance of our proposed crawling architecture TSCU, we conduct two user evaluation experiments to evaluate and compare our crawler to other existing web stations. In the first experiment ,we asked domain experts to judge and compare the precision of the search results from TSCU to two other commercial search engines: Google and Citeseer to gain further insights into how the user model, user interest trees could help a focused crawler improve the collection quality, we conducted a second user evaluation experiment in which we built a collection by using traditional focused crawler and compared the results from this collection to those from the collection built by the crawling of TSCU. A. User Evaluation Experiment 1 1) experiment details In our first experiment, we let experts judge and compare the search results from TSCU to those from the two benchmark systems: Google and Citeseer,because Google is currently known as the best general search engine and Citeseer is the one of topic-specific search engine which focuses on search of research paper. We select 23 queries, and the top 20results for each query, the two experts were asked to judge whether or not they have an interest in the result pages .Then the major measure used to compare the three systems was defined below. So we can conclude that TSCU system achieved a significantly higher interest than both Google and Citeseer,while Google achieved a higher interest than Citeseer in the experiment. B. User Evaluation Experiment 2 1) experiment details To gain further insights into how the user interest tree could help the crawlers improve the quality of the collection, we conducted a second user evaluation experiment to directly compare the user interest tree enhanced focused crawler to a traditional focused crawler. We disabled the user interest tree component in our crawler, we use the 23queries and the top 20 results for each query, the same as the first experiment. Also two experts which we built the user interest tree before were selected and the crawler which disabled user interest tree can not use the user information. Then the experts were asked to give each of the result pages an interest assessment score in the range 1 to 5, where 5 meant most interesting. The we compute the average interest scores of the results from the collections were compared. 2) Experiment results Then results from the collection built by the user interest tree enhanced focused crawler achieved an average interest score of 3.58 Which is significantly higher than the score of 2.89 obtained by the results from the collection built by the crawler without user interest tree. So we can conclude that the user interest tree helped improve the quality of the collection in terms of the user interest. 5. CONCLUSION In this paper, we have introduced a new perspective architecture improving the performance of the

2) Experimental results The results on interest are summarized in Table 1. TSCUsystem had an interest of 51.09%, compared with 45.65% and 39.13% obtained by Google and Citeseer.The results on interest are summarized in Table 1. TSCUsystem had an interest of 51.09%, compared with 45.65% and 39.13% obtained by Google and Citeseer. Table I. Results Of The First User Evaluation Experiment

topic-specific crawler, we use the data collected from the action of the user on the result page of the topic-specific search engine. Because the user is the aim of anything, we want to build better user model in the future,the another line for the future of the TSCU is put into distributed environment so that it can be used in business environment ,a lot of significant work needs to be done before we can show the effectiveness and feasibility of the proposed method. 6. REFERENCES [1] BrightPlanet. Com, The deep Web: Surfacing hidden value. Accessible at http://brightplanet.com, July 2000. [2] M.K. Bergman, The deep Web: Surfacing the hidden value, http://www.press.mich.edu/jep/0701/bergman.html [3] D. Florescu, A.Y. Levy and A. O. Mendelzon, Database techniques for world wide Web: A Survey,SIGMOD record, 27(3), 59-74, 1998. [4] Kevin Chen-Chuan Chang, Bin He, Chengkai Li, Mitesh Patel, Zhen Zhang, Structured databases on the Web: Observations and Implications, Technical Report, UIUC. [5] Sriram Raghavan, Hector GarciaMolina,Crawling the Hidden Web, Proc. Of the 27th VLDB Conference, 2001. [6] Luciano Barbosa, Juliana Freire, Searching for hidden-Web databases, Eighth Intl. workshop on the Web and Databases, 2005. [7] S. Chakrabarti, M. van den Berg, B. Dom,Focused crawling: A New Approach to Topic specific Web Resource Discovery, Computer Networks,31(11-16), 1623-1640, 1999. [8] J. Akilandeswari, N.P. Gopalan, A Web Mining System using Reinforcement Learning for Scalable Web Search with Distributed, Faulttolerant Multiagents,WSEAS transactions on Computers, Issue 11,Vol 4, p1633-1639, November 2005. [9] M. Diligenti, F. Coetzee, S. Lawrence, C.L. Giles and M. Gori, Focussed Crawling using Context Graphs, In Proc of the 26th Intl conf. on Very Large Databases, p527-534, 2000. [10] R.C. Miller and K.Bharat, Sphinx: A framework for creating personal, site specific Web crawlers, Proc for the 7th Intl WWW conf, 1998.

[11] Luciano Barbosa, Juliana Freire, An Adaptive Crawler for Locating Hidden-Web Entry Points, Proc of Intl WWW conf, p441-450, 2007. [12] Jason Rennie, A.K. McCallum, Using Reinforcement Learning to Spider the Web Effeciently, Proc of 16th Intl. conf on Machine Learning, 1999. [13] Leslie Pack Kaelbling, Michael L. Littman,Andrew W. Moore, Reinforcement learning: A survey, Journal of Artificial Intelligence Research, pp 237-285, 1995. [14] Jaideep Srivatsava, B. Mobasher, R. Cooley, Web mining: Information and pattern discovery on the world wide Web, International conference on tools with Artificial Intelligence, pp558-567, Newport beach, 1997. [15] C.M. Bowman, P.B. Danzig, U. Manber, M.F.Schwartz, Scalable internet resource discovery: Research problems and approaches, communications of the ACM, 37(8), 98-107, 1994.