针对大规模网络访问数据的相关性及用户行为预测分析

发布时间：2018-08-30 09:32

【摘要】：随着互联网的不断普及和电子商务的深入发展,人们的生活方式也受到了深刻的影响。互联网上的各种服务,如web点击浏览、搜索引擎、网上购物、社交网站等,在为互联网用户提供便捷的同时,也记录了越来越多的用户行为数据,而这些大规模的网络访问数据间往往存在着某种隐含的特殊关系,如何从这些数据中挖掘出有价值的信息已成为研究的热点。本文以运营商大规模的DPI数据为基础,并结合爬虫获取的分类标签数据,采用分布式的思想对数据实现分布式的统计分析、关联分析,并基于用户的历史行为数据,采用Markov预测模型中的PPM模型对用户将来的行为进行预测。本文主要是针对运营商大规模的网络访问数据进行处理分析,并结合基于Python实现的爬虫程序,爬取网站URL以及对应的分类标签数据,然后采用Hadoop提供的HDFS实现对数据的分布式存储,并基于MapReduce框架实现对访问数据可靠、高效的预处理以及针对汽车行业从页面访问量、独立用户数、平均访问时长、集中访问时间点分布四个角度进行统计分析,然后利用爬虫得到的汽车URL分类标签数据对DPI数据中的汽车行业用户进行识别,并提取汽车行业用户的静态行为特征和动态行为特征,基于用户的动态行为序列即浏览行为和搜索行为进行关联规则挖掘和行为预测。以分布式的思想实现FP-Growth算法,并应用于大规模的DPI数据中,然后对用户的动态行为序列进行关联规则挖掘,找出用户动态行为特征之间的关联性,最后基于短期内用户历史动态行为特征序列的积累,构建PPM预测模型对用户的行为进行预测,在构建预测模型前,对预测周期进行了更为精确地划分,选择以小时为单位,根据用户在前几个时刻的历史行为特征预测用户下一时刻的行为,然后将预测得到的结果与用户的实际访问行为进行比较,计算预测的准确率,并得出预测准确率随着用户访问行为序列长度的增加而明显提高的结论,最后,对所有的数据输出结果进行整理并以图表的形式可视化呈现以便于进行分析。
[Abstract]:With the continuous popularization of the Internet and the further development of electronic commerce, people's way of life has also been deeply affected. Various services on the Internet, such as web click browsing, search engine, online shopping, social networking sites and so on, not only provide convenience for Internet users, but also record more and more user behavior data. However, there is an implicit special relationship between these large-scale network access data. How to extract valuable information from these data has become a hot topic. Based on the large-scale DPI data of the operator and the classified label data obtained by the crawler, the distributed statistical analysis, association analysis and user's historical behavior data are realized by using the distributed idea in this paper. The PPM model of Markov prediction model is used to predict the future behavior of users. This paper is mainly for the operators of large-scale network access data processing and analysis, and based on the implementation of Python crawler program, crawling website URL and the corresponding classification label data, Then it uses HDFS provided by Hadoop to realize the distributed storage of data, and based on MapReduce framework to realize reliable access data, efficient preprocessing and the number of page visits, the number of independent users, and the average access time for the automobile industry. This paper makes statistical analysis from four angles of centralized access to the distribution of time points, and then identifies the automobile industry users in the DPI data by using the URL classification label data obtained by the crawler. The static and dynamic behavior characteristics of users in automobile industry are extracted, and association rules mining and behavior prediction are carried out based on the dynamic behavior sequence of users, namely browsing behavior and searching behavior. FP-Growth algorithm is implemented with distributed idea and applied to large-scale DPI data. Then association rule mining is carried out on user's dynamic behavior sequence to find out the correlation between user's dynamic behavior characteristics. Finally, based on the accumulation of historical dynamic behavior characteristics of users in a short period of time, a PPM prediction model is constructed to predict the behavior of users. Before the prediction model is constructed, the forecasting period is divided more accurately and the unit of hours is selected. According to the historical behavior characteristics of the user at the first few times, the behavior of the user at the next moment is predicted, and then the predicted results are compared with the actual user's access behavior, and the accuracy of the prediction is calculated. It is concluded that the prediction accuracy increases with the increase of the length of the user's access behavior sequence. Finally, all the data output results are sorted out and visualized in the form of charts to facilitate the analysis.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.0

【相似文献】