基于主题的数据API检索平台关键技术的研究与实现

发布时间：2018-03-17 16:41

本文选题：互联网文档数据　切入点：数据API　出处：《浙江大学》2017年硕士论文　论文类型：学位论文

【摘要】：在当前的互联网时代,存在着数据增长迅速,然而信息庞杂,用户想要从海量数据中搜索有用的信息费时费力的问题。用户需要的是更详细、细分的信息,针对这种情况,本文提出了一种基于主题的数据API检索平台。本文设计的分布式、可扩展的基于主题的数据API检索平台,首先将海量的互联网信息采集到子系统中,然后对数据进行分类,对每一类的数据通过检索的方式提供给用户,用户通过选取自己感兴趣的信息来消费平台中的数据。为了提供这样一个数据API检索平台,首先是能够将大量的网页抓取下来,并且需要此功能支持水平扩展和稳定性。此外为了减少人工成本,又能有效的提取网页中的信息,本文提出了半自动化的基于模板的网页信息提取方法。面对海量的互联网文档数据,需要提供一种合理的方式对数据进行分类,以便用户有针对性的选择对自己有用的数据。为此,本文设计了基于主题的数据分类和检索系统。通过基于LDA主题模型来推断文档中的主题分布,然后根据主题分布建立相应的API主题和相应的API-Key。考虑到API主题及对应的API-Key数据量很大,所以需要对这些API主题提供检索方式,通过基于相似度的方式来对检索出来的API主题进行排序,最后将API主题及其对应的结果数据集返回给用户。本平台提供多方面的数据,通过简单的API接口提供给用户,不论是科研还是商业用户,都可以消费该平台中的数据。最后通过实验分析,证明本平台具有一定的科研和商业价值。
[Abstract]:In the current era of the Internet, there is a rapid growth of data, but the quantity of information, the user wants to search data from the mass of useful information and time-consuming problem. Users need more detailed information, segmentation, in view of this situation, this paper proposes a retrieval platform theme data based on API. This paper designed the distributed, scalable data retrieval platform based on the API theme, the massive Internet information collection subsystem, and then classify the data provided to the user for each type of data through the retrieval, user consumption platform using the data in the selected information of interest to them. In order to provide such a API data retrieval platform, the first is to a large number of web crawling down, and this supports horizontal expansion and stability. In order to reduce labor costs, but also can effectively. Take the information in the web page, proposed web information extraction method based on template semi automation. The face of the Internet document data, the need to provide a reasonable way to classify the data, so that users targeted selection of useful data on their own. Therefore, this paper designs the data classification and retrieval based on based on the LDA system. Through the topic model to infer the topic distribution of the document, and then establish the corresponding distribution according to the theme of API themes and corresponding API-Key. considering the amount of data and the corresponding API-Key API theme is very big, so need to provide the retrieval of these API theme, through the similarity based method to sort the retrieved API theme finally, the API theme and the corresponding results are returned to the user. The data set provides many data of the platform, available to the user through a simple API interface, Whether it is scientific or business users, the platform can consume data. Finally, through experimental analysis, it proved that this platform has a certain scientific and commercial value.

【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】