基于web数据的特定行业用户网络行为分析

发布时间：2018-08-26 07:01

【摘要】：web数据含有用户网络访问模式的丰富信息,挖掘web数据对于用户的网络行为分析具有重大意义。本文研究的web数据来源于两部分,一是国内某运营商提供的基于DPI包检测技术获得的网络流数据,主要是用户web日志数据;二是利用网络爬虫程序抓取的网页数据。基于这两种web数据,分析了电商和汽车网站用户的网络行为。(1)电商行业的用户网络行为分析。主要以京东、天猫、国美、苏宁这四个电商网站的用户为研究对象,利用MapReduce基本统计分析方法,分析这四个电商网站的用户对具体商品的访问行为,主要包括对商品的浏览、搜索和加购物车等行为。采用BulkLoad工具将电商用户行为数据批量导入HBase表中,解决了数据逐条导入而频繁IO、GC操作造成的系统响应变慢、节点超时退出等问题,增加了集群系统的稳定性、提高了数据入库效率。最后通过编写HBase数据查询接口,对电商用户的行为数据进行定制化查询,即可根据指定条件对用户行为集中查询和分析。(2)分析汽车行业的用户网络行为。主要以长安福特汽车品牌下的锐界车系为研究样例,使用AprioriAll序列模式挖掘算法得到用户访问汽车网站的频繁序列集,分析对锐界车系感兴趣的用户倾向在哪些汽车网站(排名靠前的15个汽车网站)上获取锐界相关信息,以及对这些网站的访问先后次序又是怎样的;再利用MapReduce统计特性和RESTful API技术对用户的站点访问情况以及兴趣标签等信息进行可视化分析。其中,用到正则表达式提取汽车用户数据,通过RegexBuddy工具调试和优化正则表达式,并结合数据的Hash存储特性,使得数据提取程序的时间复杂度由O(N)降为O(1),提高了程序运行效率。本文的研究结果在一定意义上,能够给特定商品和特定汽车的网络经销商或者广告商在用户群体定位、广告精准投放或交叉推送等方面提供参考意见。
[Abstract]:Web data contains abundant information of user's network access pattern. Mining web data is of great significance to user's network behavior analysis. The web data in this paper come from two parts, one is the network stream data based on DPI packet detection technology provided by a domestic operator, mainly the user web log data, and the other is the web page data captured by the web crawler program. Based on these two kinds of web data, the network behavior of the users of e-commerce and automotive websites is analyzed. (1) the network behavior of users in e-commerce industry is analyzed. Mainly taking the users of four ecommerce websites JingDong, Tmall, Gome and Su Ning as research objects, and using the basic statistical analysis method of MapReduce, to analyze the visit behavior of the users of these four ecommerce websites to specific commodities, mainly including the browsing of commodities. Search and shopping cart, etc. The BulkLoad tool is used to import the user behavior data into the HBase table in batches, which solves the problems such as the slow response caused by the frequent IO,GC operation and the node time-out, which increases the stability of the cluster system. The efficiency of data storage is improved. Finally, through compiling HBase data query interface, the behavior data of e-commerce users can be customized queried and analyzed according to the specified conditions. (2) analyzing the behavior of users in automobile industry. Taking the sharp boundary vehicle system under the Changan Ford brand as the research example, the frequent sequence set of users visiting the automobile website is obtained by using AprioriAll sequence pattern mining algorithm. Analyze which automobile websites (the top 15 auto websites) that interested users tend to obtain relevant information, and what is the order of visits to these websites; Then, the user's site access and interest tags are analyzed visually by using MapReduce statistics and RESTful API technology. Among them, the regular expression is used to extract the automobile user data, the regular expression is debugged and optimized by RegexBuddy tool, and combined with the Hash storage characteristic of the data, the time complexity of the data extraction program is reduced from O (N) to O (1), and the running efficiency of the program is improved. To a certain extent, the results of this paper can provide reference to the network dealers or advertisers of specific commodities and vehicles in terms of user group positioning, accurate advertising delivery or cross-push, and so on.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP393.09

【相似文献】