基于电信DPI数据的电商用户行为分析

发布时间：2018-06-24 15:23

本文选题：DPI + Hadoop　；参考：《北京邮电大学》2017年硕士论文

【摘要】：伴随着Internet的高速发展和Web技术的不断升级,人们的日常生活方式越来越多的搁置进Web中。随着而来的,是大数据时代的来临。大数据的来源是庞大的Web用户群体进行日常网络访问时产生的各式各样的行为,这些用户行为背后蕴藏着丰富的信息和潜在的价值,对企业运行经营方式产生了巨大影响。运营商通过DPI技术对网络流量进行采集分析,并对电商用户的行为进行分析挖掘,但海量的数据对传统的方法和模型提出了艰巨的挑战。利用并行计算和分布式存储技术,Hadoop平台为海量DPI数据的处理提供了解决方案。本文主要目标是利用DPI数据为基础数据,通过Wireshark工具分析用户访问电商的行为,提取电商用户行为的特征向量,结合利用爬虫程序采集的电商商品信息,在Hadoop平台实现构建用户画像MapReduce 的程序。本文的研究内容主要有以下几个方面:(1)给出了面向大数据的电商用户行为分析系统的整体设计方案。该系统主要有基础数据采集模块,DPI数据预处理模块以及用户行为分析模块组成。并给出了实现这三个模块的设计实现;(2)基于MapReduce思想的用户行为特征向量提取方法,根据用户地域特征,用户终端特征,访问不同网站的情况统计用户行为;(3)通过DPI和电商商品信息数据的聚合,实现了在Hadoop平台电商用户画像多种维度的构建;(4)根据电商用户画像,实现了基于ItemBased协同过滤的用户兴趣推荐模型。
[Abstract]:With the rapid development of Internet and the continuous upgrading of Web technology, more and more people's daily life style is put on the Web. What comes with it is the advent of the big data era. The source of big data is a variety of behaviors that occur when a large group of web users visit the network daily. Behind these user behaviors are abundant information and potential value, which has a great impact on the operation and management of enterprises. Operators collect and analyze network traffic through DPI technology, and analyze and mine the behavior of e-commerce users. However, the huge amount of data poses a formidable challenge to the traditional methods and models. Using the parallel computing and distributed storage technology Hadoop platform provides a solution for mass DPI data processing. The main goal of this paper is to use DPI data as the basic data, analyze the behavior of the user to access the electronic quotient by Wireshark tool, extract the characteristic vector of the user's behavior, and combine the information collected by the crawler program. In Hadoop platform to build a user portrait MapReduce program. The main contents of this paper are as follows: (1) the overall design scheme of big data oriented user behavior analysis system is presented. The system consists of basic data acquisition module, DPI data preprocessing module and user behavior analysis module. The design and implementation of the three modules are given. (2) the method of extracting user behavior feature vector based on MapReduce idea, according to the user regional characteristics, user terminal features, (3) through the aggregation of DPI and e-commerce commodity information data, we have realized the construction of various dimensions of e-commerce user portrait in Hadoop platform; (4) according to e-commerce user portrait, A user interest recommendation model based on Item-based collaborative filtering is implemented.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092;TP311.13

【相似文献】