基于Hadoop云平台推荐系统的研究与设计

发布时间：2018-09-12 05:52

【摘要】：在信息技术高速发展的时代,信息过载现象越发严重,如何能在大量的资源中快速挖掘出用户感兴趣的信息,已成为亟待解决的问题,在这种时代背景下,推荐系统应运而生。然而在实际应用中,稀疏矩阵问题是致使推荐系统推荐准确率下降的一个重要原因。另外,用户的行为数据呈爆炸式增长,这种现象导致单台服务器已经很难满足推荐系统海量数据的运算需要。综上所述,基于Hadoop云平台的推荐系统的研究具有理论和实际的双重价值。协同过滤推荐系统是被使用最广泛的推荐系统,因此本文以协同过滤推荐系统为主要研究目标,旨在解决推荐系统的稀疏矩阵、处理海量数据计算瓶颈等问题。基于以上两种关键问题,本文从算法与系统两个层面进行优化,研究并设计了一种基于Hadoop云平台的推荐系统。本文工作主要包括以下几点内容:1)阅读了大量有关推荐系统协同过滤算法的文献,总结前人的研究成果和现在国内外相关研究状况。2)为了有效防止传统协同过滤方法存在的项目维度过高、数据稀疏性、主观因子干扰等问题,本文提出了一种基于用户兴趣模型以及惩罚主观因子的协同过滤算法(Interests Model Weaken S.ubjective Collaborative Filtering,IMWS-CF)。该方法引入兴趣因子,用户兴趣评分因子、惩罚主观因子等概念,通过采用高效可行的方法来降低数据集的稀疏性与提高算法的精度,进而解决推荐系统的稀疏矩阵问题。3)在研究了推荐系统技术细节的基础上,利用之前的优化算法(IMWS-CF),设计一种基于Hadoop云平台的推荐系统。运用模块化的思想对系统进行优化设计,在考虑高并发、稳定性、易扩展性等因素的同时,还提出并设计了环境分析引擎,基于不同的推荐环境,采用不同的推荐策略,从系统架构层面上优化了推荐系统的精确性。4)从稀疏矩阵与并行计算能力两方面进行实验设计,验证本文设计与实现的基于Hadoop云平台推荐系统,其在缓解稀疏矩阵问题与海量计算瓶颈问题上,都起到了优化的作用。
[Abstract]:In the era of rapid development of information technology, the phenomenon of information overload is becoming more and more serious. How to quickly excavate the information interested by users in a large number of resources has become a problem to be solved urgently. Under this background, recommendation system emerges as the times require. However, in practical application, sparse matrix problem is an important reason for the decrease of recommendation accuracy. In addition, the behavior data of users is increasing explosively, which makes it difficult for a single server to meet the need of computing massive data in recommendation system. To sum up, the research of recommendation system based on Hadoop cloud platform has both theoretical and practical value. Collaborative filtering recommendation system is the most widely used recommendation system, so this paper focuses on collaborative filtering recommendation system to solve the sparse matrix of recommendation system and deal with the bottleneck of mass data computing. Based on the above two key problems, this paper studies and designs a recommendation system based on Hadoop cloud platform by optimizing the algorithm and system. This paper mainly includes the following contents: 1) read a lot of literature about collaborative filtering algorithm of recommendation system. In order to effectively prevent the traditional collaborative filtering methods, such problems as high project dimension, data sparsity, subjective factor interference and so on, are summarized. This paper presents a collaborative filtering algorithm (Interests Model Weaken S.ubjective Collaborative Filtering,IMWS-CF based on user interest model and penalty subjective factor. In this method, the concepts of interest factor, user interest score factor and penalty subjective factor are introduced to reduce the sparsity of data sets and improve the accuracy of the algorithm by using efficient and feasible methods. On the basis of studying the technical details of the recommendation system, a recommendation system based on the Hadoop cloud platform is designed by using the previous optimization algorithm (IMWS-CF). Using modularization to optimize the design of the system, considering the factors of high concurrency, stability, expansibility and so on, the environment analysis engine is proposed and designed. Based on the different recommendation environment, different recommendation strategies are adopted. The accuracy of recommendation system is optimized from the system architecture level. 4) the sparse matrix and parallel computing ability are designed experimentally to verify the design and implementation of the recommendation system based on Hadoop cloud platform. It plays an important role in alleviating the sparse matrix problem and the bottleneck problem of mass computing.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.3;TP393.09

【参考文献】