移动App用户海量日志分析的优化策略与算法研究

发布时间：2018-12-14 05:12

【摘要】：随着“互联网+”的走热,各行各业都与互联网紧密的结合了起来,随之产生了海量的数据信息,尽管面对如此多的数据信息,但人们想真正找到自己感兴趣的内容也是件难事。同时互联网企业绞尽脑汁分析用户兴趣,为用户提供更好的产品、更好的服务,日志数据记录了用户的行为,基于此,如何分析挖掘互联网产品产生的海量日志数据变得非常重要。在目前数据挖掘的研究中,关联规则的挖掘是非常重要的研究方向,关联规则涉及到信息、医疗、交通、金融等多个领域的研究。在关联规则的挖掘算法中,由R.Agrawal和R.Srikant于1994年提出的Apriori算法最为经典和重要,算法的核心思想是通过迭代由k-1轮的频繁项集连接剪枝生成K轮的频繁项集。本课题主要对用户海量数据处理算法Apriori进行研究,其中主要包括以下几个研究内容:(1)关联规则算法在日志数据挖掘中的应用。该研究采集了视频APP用户的浏览、观看、检索日志数据,提出数据挖掘的流程框图,并基于Apriori算法得出视频作品之间的关联度,根据作品关联程度为用户推荐合适的视频作品。(2)Apriori算法改进的研究。Apriori算法在处理日志数据时由于频繁的扫库和生成大量的频繁项集而使时间性能较低,本研究在大量优化研究的基础上,提出对源数据、频繁项集进行二进制编码并用整数表示,通过位运算和海明距离生成频繁项集的优化策略。我们将优化策略编程实现,并分析了优化的效果。(3)优化的Apriori算法在MapReduce并行化编程模型下的扩展。面对海量日志数据,单机计算能力远不能满足大数据挖掘的需求了,本课题通过研究并行化编程模型MapReduce,将优化的Apriori算法在MapReduce下进行并行化扩展。
[Abstract]:With the "Internet" hot, various industries have been closely combined with the Internet, resulting in a huge amount of data information, despite the face of so much data information, But it is also difficult for people to really find what they are interested in. At the same time, Internet enterprises have racked their brains to analyze the interests of users, to provide better products and better services for users. How to analyze and mine the massive log data generated by Internet products becomes very important. In the current data mining research, association rules mining is a very important research direction, association rules related to information, medical, transportation, finance and other fields of research. Among the algorithms for mining association rules, the Apriori algorithm proposed by R.Agrawal and R.Srikant in 1994 is the most classical and important. The core idea of the algorithm is to generate K-wheel frequent itemsets by iterating the frequent itemsets of k-1 rounds. This paper mainly studies the user mass data processing algorithm (Apriori), which mainly includes the following research contents: (1) the application of association rule algorithm in log data mining. This study collects the browsing, viewing and retrieving log data of video APP users, puts forward the flow chart of data mining, and obtains the correlation degree between video works based on Apriori algorithm. (2) Research on the improvement of Apriori algorithm. (2) the performance of Apriori algorithm in processing log data is low due to frequent scanning and generating a large number of frequent itemsets. Based on a large number of optimization studies, this paper proposes an optimization strategy for generating frequent itemsets from source data, frequent itemsets and integer representation, and generating frequent itemsets by bit operation and hamming distance. We implement the optimization strategy programming and analyze the effect of optimization. (3) the extension of the optimized Apriori algorithm in the parallel programming model of MapReduce. In the face of massive log data, the single computer can not meet the needs of big data mining. This paper extends the optimized Apriori algorithm under MapReduce by studying the parallelization programming model MapReduce,.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】