基于Storm的实时推荐系统研究与设计

发布时间：2018-10-20 06:44

【摘要】：伴随着移动互联网,电子商务以及物联网的爆发式发展,产生了海量的数据,产生了信息超载(information overload)问题。当前,此类问题的解决方法之一是使用推荐系统。传统的推荐系统大多基于非实时的Hadoop处理框架,以“批”方式进行处理,无法满足数据的实时处理要求。而Storm对数据以“流”方式进行处理,基于内存进行运算,可以保证数据信息被快速处理,且实时更新并写入数据库。本文基于实时的、分布式的流数据处理框架Storm进行推荐系统相关研究,进行了以下工作:第一:对Storm框架进行学习。了解消息可靠处理机制(ACK);掌握了核心组件Spout、Bolt的功能以及实现方法;能够设计Topology结构,基于Storm Trident的高级抽象进行开发。第二:基于Storm的推荐系统常规采用基于SVD的k-means协同过滤算法。SVD先将评分矩阵中的缺失值补全,然后将其分解成3个低阶矩阵,最后用计算得到的用户预测评分来更新评分矩阵的缺失项,在矩阵项达到数百万的情况下,上述过程时间消耗代价巨大。为了进一步提升计算效率,本文提出了基于改进SVD的k-means协同过滤算法,改进SVD引入损失函数的概念,建立均方差最小化函数,通过梯度下降法迭代对其进行优化求解,简化了矩阵分解运算。在MovieLens-10m数据集上进行实验,与传统SVD算法相比,本文提出算法在推荐时间效率方面得到了提升。第三:实现了基于Storm的实时推荐系统,首先,增加了Flume日志采集过滤子系统,该系统用于接收多种类型的用户消息数据并进行过滤;其次,设计了Kafka数据缓冲队列模块,防止数据丢失,确保系统可靠;最后,对Storm集群进行部署,设计了算法拓扑逻辑结构,经过测试已达到设计要求,并实际上线成功应用于电影推荐中。
[Abstract]:With the explosive development of mobile internet, electronic commerce and Internet of things, massive data and information overload (information overload) are produced. Currently, one of the solutions to such problems is to use recommendation systems. Most of the traditional recommendation systems are based on the non-real-time Hadoop processing framework, which can not meet the requirements of real-time data processing because of the "batch" processing. Storm processes the data in a "stream" mode and computes the data based on memory, which ensures that the data information can be processed quickly, and the data can be updated and written to the database in real time. In this paper, based on the real-time, distributed streaming data processing framework (Storm), the recommendation system is studied. The following work is done: first, the Storm framework is studied. (ACK); has grasped the function and the implementation method of the core component Spout,Bolt, and can design the Topology structure and develop it based on the advanced abstraction of Storm Trident. Second, the recommendation system based on Storm adopts k-means collaborative filtering algorithm based on SVD. SVD first complements the missing value in the score matrix and then decomposes it into three low-order matrices. Finally, the missing items of the scoring matrix are updated with the calculated user prediction scores. When the matrix items reach millions, the cost of the process time is enormous. In order to further improve the computational efficiency, this paper proposes a k-means collaborative filtering algorithm based on improved SVD. The concept of loss function is introduced into the improved SVD, and the mean-variance minimization function is established. The matrix decomposition operation is simplified. Compared with the traditional SVD algorithm, the proposed algorithm is more efficient in recommending time than the traditional SVD algorithm. Third, a real-time recommendation system based on Storm is implemented. Firstly, the Flume log collection and filtering subsystem is added, which is used to receive and filter various types of user message data. Secondly, the Kafka data buffer queue module is designed. Finally, the Storm cluster is deployed and the algorithm topology logic structure is designed. The algorithm has been tested to meet the design requirements, and in fact, the line has been successfully applied in the movie recommendation.
【学位授予单位】：中北大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】