专业信息定向采集与多维搜索系统

发布时间：2018-04-21 20:27

本文选题：分布式爬虫 + 信息抽取　；参考：《浙江大学》2017年硕士论文

【摘要】：决策依靠经验、直觉和逻辑驱动,但大数据时代以来,特别对专业领域来讲,决策越来越依靠数据驱动。互联网上存在大量领域相关文本信息,这些信息多以新闻报道、博客、社区门户等形式存在。多维搜索是一种基于排序与分类的搜索技术,如何将这些技术应用到互联网专业领域数据是一个非常复杂的流程,涉及内容从如何从互联网采集信息、保证HTML文本质量到自然语言处理与多维搜索技术等。本文结合分布式定向爬虫、数据清洗、文本分析、多维搜索等技术,构建了一个面向专业领域的互联网定向数据采集与多维搜索系统。本文基于一个实际的应用例子,聚焦于实现一个简单易用且可扩展的系统用于专业领域的互联网文本分析。本文主要工作包括:1)基于专业领域互联网数据采集需求,构建了一个分布式定向爬虫。与全文抓取不同,分布式定向爬虫要求从互联网快速抓取深层次、精确化的结构化数据;2)构建信息抽取通道,用于清洗加工分布式爬虫采集下来的数据。步骤主要包括去除网页噪音、进行内容查重、丢弃无关专业领域数据、文本分词与实体抽取;3)为使系统更具备交互性,系统提供对采集数据的多维检索服务。在数据加工的基础上,根据专业领域概念体系,提供多维搜索以辅助全文检索服务;
[Abstract]:Decision-making is driven by experience, intuition and logic, but since big data's time, especially in the professional field, decision-making has been more and more data-driven. There are a large number of related text information on the Internet, mostly in the form of news reports, blogs, community portals and so on. Multidimensional search is a kind of search technology based on sorting and classification. How to apply these technologies to the data of Internet professional domain is a very complicated process, which involves how to collect information from the Internet. Ensure HTML text quality to natural language processing and multidimensional search technology. Based on distributed directional crawler, data cleaning, text analysis, multidimensional search and other technologies, an Internet oriented data acquisition and multidimensional search system for professional field is constructed in this paper. Based on a practical application example, this paper focuses on the implementation of a simple and extensible system for Internet text analysis in specialized fields. The main work of this paper includes: (1) A distributed oriented crawler is constructed based on the requirement of Internet data acquisition in professional domain. Different from full-text crawling, distributed directional crawlers need to quickly capture deep and accurate structured data from the Internet and construct information extraction channels for cleaning and processing the data collected by distributed crawlers. The steps mainly include removing the noise of the web page, checking the content, discarding the irrelevant domain data, extracting the text word and entity 3) in order to make the system more interactive, the system provides the multi-dimensional retrieval service to the collected data. On the basis of data processing, according to the concept system of professional domain, multi-dimensional search is provided to assist the full-text retrieval service.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】