分布式电商主题搜索引擎研究

发布时间：2018-07-02 09:11

本文选题：电商 + 主题　；参考：《华南理工大学》2016年硕士论文

【摘要】：当前互联网信息规模正在不断的增长,互联网已经进入了大数据时代。人们在互联网如此海量的数据中查找到自己所需的信息变得越来越困难。人们越来越趋向于使用搜索引擎来搜索信息,而且多数情况下,人们的搜索目的往往趋向于某个特定的领域。当前互联网上的搜索引擎多数都是通用的搜索引擎,搜索的内容主题不明确,内容广泛,对于用户特定领域信息的搜索需求已经显得力不从心。随着互联网信息数据的迅速增长,主题化、智能化、个性化已经慢慢的成为搜索引擎发展的方向,特别是基于某种特定主题的搜索引擎是其中的研究热点。当前消费者获取电商产品信息依赖于电商门户网站,而无法通过搜索引擎来进行查询搜索,而且以电商为主题的搜索引擎为消费者提供相关商品信息的研究还比较少。本文针对通用搜索引擎搜索无法为用户提供深入商品信息的问题,设计开发基于电商主题的搜索引擎系统,方便用户快速准确地查询到相关的商品信息。本文以电商搜索为主题,结合实际应用需求,研究基于Hadoop的分布式主题搜索引擎以及其相关的技术原理。首先深入分析搜索引擎相关的技术原理,如网络爬虫,数据索引,中文分词,网页分类,搜索评分等主题搜索引擎必须的技术,为本文的研究打下结实的基础;然后深入剖析分布式计算框架MapReduce和分布式文件系统HDFS,引入基于MapReduce编程模型的设计方案。完成了在Hadoop平台上的系统架构设计,系统功能划分,网页爬虫流程、索引流程以及搜索流程的分析和设计,完成了三个子系统的实现与改进。在系统实现过程中引用Nutch作为网页爬虫的基本框架,引用Solr作为搜索框架,同时引入中文分词器(IK-Analyzer)来处理中文内容。在基于Nutch的网络爬虫中针对当前电商网络爬虫存在的问题做了分析并提出了解决方案。最后部署实现了4个节点的分布式搜索引擎系统,并以京东、天猫为例,对本系统进行搜索查询测试与评价。
[Abstract]:At present, the scale of Internet information is constantly growing, the Internet has entered the big data era. It is becoming more and more difficult for people to find the information they need in such a huge amount of data on the Internet. More and more people tend to use search engines to search for information, and in most cases, people tend to search for a specific field. At present, most search engines on the Internet are general search engines, the content of search is not clear, the content of a wide range of search requirements for users in specific areas of information has become inadequate. With the rapid growth of Internet information data, theming, intelligence and personalization have gradually become the direction of search engine development, especially the search engine based on a specific topic is one of the research hotspots. At present, consumers rely on e-commerce portal to obtain e-commerce product information, but can not search through search engines. Moreover, the research on how to provide consumers with relevant commodity information by e-commerce search engines is relatively rare. Aiming at the problem that the general search engine can not provide the users with in-depth commodity information, this paper designs and develops a search engine system based on the topic of e-commerce, which is convenient for users to query the relevant commodity information quickly and accurately. In this paper, we study the distributed subject search engine based on Hadoop and its related technical principles. Firstly, the necessary technology of search engine, such as web crawler, data index, Chinese word segmentation, web page classification, search score and so on, is deeply analyzed, which lays a solid foundation for the research of this paper. Then, the distributed computing framework MapReduce and the distributed file system HDFSare deeply analyzed, and the design scheme based on MapReduce programming model is introduced. The system architecture design, system function partition, web crawler flow, index flow and search flow are analyzed and designed on Hadoop platform. The implementation and improvement of three subsystems are completed. Nutch is used as the basic frame of web crawler, Solr is used as the search frame, and IK-Analyzer is introduced to deal with Chinese content. In this paper, the problems of current ecommerce network crawlers are analyzed in Nutch-based network crawlers and solutions are proposed. Finally, the distributed search engine system with four nodes is implemented, and the JingDong and Tmall are taken as examples to test and evaluate the system.
【学位授予单位】：华南理工大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.3

【相似文献】