基于Spark Streaming的动态社区发现及其在个性化推荐应用中的研究

发布时间：2018-06-20 22:08

  本文选题：电影推荐模型 + 动态社区发现　； 参考：《江苏大学》2017年硕士论文

【摘要】：社区发现作为一种重要的网络分析技术,能够挖掘出网络中具有某些共性的节点集合。研究网络中的社区对理解整个网络的结构和功能起到至关重要的作用,它不仅可帮助我们分析及预测整个网络各元素间的交互关系,而且可以分析用户行为以及为用户提供更加个性化的搜索结果。现实中,社区发现已经在多个领域发挥着重要作用。本文在深入研究社会网络中社区发现算法、Spark相关技术以及个性化推荐技术的基础上,提出了基于叶子社区与节点比较策略的Louvain算法,并将改进的Louvain算法融入Spark Streaming流处理框架,使其能够动态调整社区结构,捕捉社区信息。最后将社区发现思想应用到个性化推荐领域中,用于解决大量的向量运算等问题。论文的主要工作如下:1.提出了改进的Louvain算法,包括叶子社区策略以及节点度数比较策略。其中叶子社区是指含有叶子节点且节点总度数为2n-1(n为节点数目)的社区,叶子社区策略是指直接将叶子社区中的节点划分到与之相邻并且度数小于或者等于2的节点所在的社区。节点度数比较策略则直接比较相邻节点的∑tot的值来找出maxΔQ的邻居节点。改进算法大量减少ΔQ值计算,提高了执行效率。2.针对Louvain社区发现算法仅适应于静态社会网络的问题,提出了基于Spark Streaming的动态社区发现框架(SDCDF),SDCDF中采用的动态社区发现策略减少了对整个网络进行社区划分的次数,提高社区动态发现效率。3.针对传统的电影推荐模型随用户与电影数量的增长导致向量运算过于复杂以及矩阵对系统内存开销大的问题,提出基于Louvain改进算法的电影推荐模型(LFRM)。LFRM依据Louvain改进算法的社区划分结果,将用户-电影矩阵转变成社区-电影矩阵,再使用ALS训练模型,并进行偏好值预测与电影推荐,该模型通过对用户-电影矩阵降维来避免大量的向量运算,一定程度上提高了个性化推荐的效率。
[Abstract]:Community discovery, as an important network analysis technology, can excavate some common node sets in the network. The study of communities in the network is crucial to understanding the structure and functions of the network as a whole. It not only helps us analyze and predict the interactions between the elements of the network, It can also analyze user behavior and provide users with more personalized search results. In reality, community discovery has played an important role in many fields. On the basis of deeply studying the community discovery algorithm (Spark) and the personalized recommendation technology in social network, this paper proposes a Louvain algorithm based on the comparison strategy between the leaf community and the node, and integrates the improved Louvain algorithm into the Spark streaming stream processing framework. So that it can dynamically adjust the community structure, capture community information. Finally, the community discovery idea is applied to the field of personalized recommendation, which is used to solve a large number of vector operations and other problems. The main work of the thesis is as follows: 1: 1. An improved Louvain algorithm is proposed, including leaf community strategy and node degree comparison strategy. The leaf community refers to the community with leaf nodes and the total degree of nodes is 2n-1(n as the number of nodes. Leaf community strategy is to divide the nodes in the leaf community directly into the community in which the nodes with a degree of less than or equal to 2 are located. The node degree comparison strategy directly compares the 鈭，

本文编号：2045819