基于用户浏览行为的深度网络挖掘

发布时间：2018-02-16 01:08

本文关键词： Deep Web 深度网络挖掘用户浏览行为浏览路径 Deep Web数据源发现 Deep Web数据源采集　出处：《中国科学技术大学》2012年博士论文　论文类型：学位论文

【摘要】：近年来,随着互联网的快速发展,网络中蕴含了海量的信息,并且仍在以惊人的速度增长。一般来说,互联网中信息的主要发布形式为静态网页,每个静态网页都含有一定数量的静态超链接,指向其他的静态网页。传统的搜索引擎正是利用这些静态网页中的超链接来收集、索引和显示用户所感兴趣的网页和信息。然而,除此之外,互联网中还有很大一部分信息是以动态数据源的形式存在的。这些信息并不存在于静态网页中,而是存储在网站背后的在线数据库中,并且根据用户的关键词实时地、动态地生成网页来呈现给用户。由于缺乏足够的静态超链接指向这些动态网页,传统的搜索引擎很难发现和索引这些网页,因此这部分信息相对于用户是“隐藏”的。这些“隐藏”信息的集合被称为深度网络(即Deep Web,又名Invisible Web或Hidden Web)。与此相对应,那些静态网页集合被称为Surface Web。现在,Deep Web的信息量远远超过了Surface Web,尤其是Deep Web中的高质量数据,更是高达Surface Web的2000多倍。但是,有效而充分地利用DeepWeb中的高质量数据在目前仍然是一个巨大的挑战,其中最重要的问题就是Deep Web数据源的发现和Deep Web数据源的采集。一当前的关于Deep Web数据源的发现与采集的研究工作各有一些不足,比如,有些需要人工参与,有些依赖于特定的领域,所以它们都很难大规模应用。因此,本文围绕着Deep Web挖掘的研究,重点关注Deep Web数据源的发现和采集这两个问题,以方便用户利用Deep Web中的信息,进一步推动Deep Web的发展。本文通过仔细分析用户在Deep Web中特有的浏览行为,归纳出了用户在Deep Web中特有的浏览路径,并基于此浏览路径提出了全自动的、不依赖特定领域的、高效的Deep Web数据源发现和采集的方法,使得大规模的Deep Web挖掘成为可能。本文的创新之处主要有三点： 1.深入分析了网络用户在Deep Web中的浏览行为首先分析了用户在Surface Web和Deep Web中的浏览记录与浏览行为,并将它们转换为更为直观的图形表示(浏览图),然后仔细对比了它们的不同之处；再结合Deep Web中网页的功能与布局特点和链接规则,最后提出了用户在DeepWeb中的模型化的浏览路径：表单页面→列表页面→目标页面。这条浏览路径很好地描述了用户在Deep Web中的浏览行为的独特性。据我们所知,本文是第一次提出类似的概念。 2.提出了一种高效的发现Deep Web数据源的方法基于用户在Deep Web中的独特的浏览路径,提出了一种高效的从浏览记录中发现Deep Web数据源的方法。该方法利用Deep Web中的链接特点,首先通过链接聚类把用户浏览过的表单页面、列表页面、目标页面聚类到一块,然后根据用户在浏览过程中的转移关系重建用户的浏览图；接着,该方法从建好的浏览图中检测浏览路径,来发现Deep Web数据源。由于该方法使用链接聚类取代了页面聚类,因此大大提高了Deep Web数据源发现的效率,而且也不依赖于特定的主题。此外,从用户浏览记录中寻找Deep Web数据源,进一步降低了代价,而且提高了发现Deep Web数据源的准确率和发现高质量Deep Web数据源的概率,降低了发现低质量Deep Web数据源的风险。 3.提出了一种高效的采集Deep Web数据源的方法基于用户在Deep Web中的独特的浏览路径,提出了一种高效的采集DeepWeb数据源的方法。由于用户的浏览过程就是访问大量目标页面的过程,因此我们尝试模拟用户的浏览行为,沿着用户在Deep Web中的浏览路径来获取大量的目标页面。该方法从表单页面出发,首先收集一定数目的列表页面；然后,该方法利用DOM树对齐技术和目标链接的布局特点在列表页面上检测目标链接；之后,在列表页面和目标页面上,该方法利用翻页链接的特点来检测翻页链接。当收集到足够的链接后,该方法会学习这些链接的链接规则,并使用学到的这些链接规则去采集目标Deep Web数据源,以提高采集效率。
[Abstract]:In recent years, with the rapid development of Internet, the network contains a vast amount of information, and is still growing at an alarming rate. In general, Internet information release form is mainly static pages, static static pages each contains a certain number of hyperlinks, refers to static page other traditional search engines are. Is the use of hyperlinks to collect these static pages in the index, and that users are interested in web pages and information. However, in addition, the Internet and a large part of the information is in the form of dynamic data source. This information does not exist in a static page, but stored online database on the site behind and, according to the key words in real time users, dynamically generated web pages to show to the users. Due to the lack of sufficient static hyperlinks to these dynamic pages, the traditional search engine It is difficult to find and index these pages, so this part of the information relative to the user is "hidden". These "hidden" information set is called the depth network (i.e. Deep Web, also known as Invisible Web or Hidden Web). Correspondingly, the static page set is called the Surface Web.
Now, the amount of information Deep Web much more than Surface Web, especially the high quality data of Deep Web, Surface Web is as high as 2000 times. However, effective and full use of high quality data in DeepWeb is still a huge challenge, the most important problem of Web data source is the Deep Web data source discovery and Deep collection. The current research work on the Deep Web data source discovery and acquisition have some shortcomings, for example, some artificial participation, some rely on specific areas, so they are difficult to large scale application. Therefore, this research around Deep Web mining the focus on the Deep Web data source discovery and acquisition of these two issues, for the convenience of users using Deep Web information, promote the further development of Deep Web. Through careful analysis for the user browsing unique in Deep Web To sum up the unique browsing path of users in Deep Web, and based on this browsing path, we propose a fully automatic and efficient way to discover and collect Deep Web data sources, which makes large-scale Deep Web mining possible.
There are three main points in the innovation of this paper.
1. in-depth analysis of the browsing behavior of network users in Deep Web
First analyzes the users in the Surface Web and Deep Web in the browsing and browsing behavior, and convert them into a more intuitive graphical representation (see chart), and then carefully compare their difference; combined with the "Deep in the Web function and layout characteristics and link rules, finally put forward the browsing path the user model in DeepWeb: the form page list page, to the target page. This path has a good description of user browsing behavior in Deep Web's uniqueness. To our knowledge, this is the first time put forward a similar concept.
2. a efficient method for finding Deep Web data sources is proposed.
Users in the Deep Web in the unique browsing path based on the proposed an efficient browsing method found in Deep from the Web data source. This method uses the link characteristics of Deep Web, the first through the link clustering users browse the form page, list page, page clustering to a target, and then according to the user browsing in the process of transfer of the reconstruction of the relationship between the user's browsing map; then, the method built from the browse path detection map, to find the Deep Web data source. Because the method uses link clustering instead of page clustering, thus greatly improving the efficiency of the Deep Web data source discovery, and does not depend on the specific the theme. In addition, from the user browse for Deep Web data source records, to further reduce the cost, but also improve the accuracy of Deep found in the Web data source and found that high quality Deep Web data sources are Rate, reducing the risk of finding a low quality Deep Web data source.
3. a efficient method for collecting Deep Web data sources is proposed.
Users in the Deep Web in the unique browsing path based on the proposed an efficient method of data acquisition source. Because of the DeepWeb user browsing process is to visit a large number of target pages, so we try to simulate the user's browsing behavior, the user browsing paths in Deep Web in order to obtain large amounts of the target page. From the form page of the first collection of a certain number of pages; then, using the layout features of DOM tree alignment technology and target link detection target link in the list on the page; after that, in the list page and the target page, the next page link to detect features page links. When collecting enough after the link, the method will learn these links rules, and use these links to learn rules acquisition target Deep Web data sources, in order to improve the collection efficiency.

【学位授予单位】：中国科学技术大学
【学位级别】：博士
【学位授予年份】：2012
【分类号】：TP393.09

【相似文献】