基于Hadoop的IP用户访问行为动机分析研究

发布时间：2018-06-03 10:40

本文选题：Hadoop + IP用户　；参考：《北京信息科技大学》2017年硕士论文

【摘要】：随着互联网技术的快速发展,基于其信息流动自由、言论自由和使用自由的特点,使得互联网形成一个无国界的虚拟沟通网络,同时由于IP具有唯一性的特点,所以它更像是人们在互联网这个“社会”的“身份证”,它可以更鲜明的突出网络IP用户的个人特色。因此,从网络日志中挖掘IP用户的上网行为,以此发现用户的搜索意图、兴趣偏好以及上网动机具有十分重要的意义。不同于传统的基于网络日志的用户行为分析方法,本文重点关注于网络日志中蕴含的隐性信息,通过对隐性信息的深入挖掘分析,找出IP用户上网行为与用户心理之间的关系,以期为IP用户的上网行为分析研究提供新的思路与方向。具体的研究工作包括以下四个方面:(1)网络日志数据的采集处理及IP辅助知识库的构建构建了一套面向全球的IP地域类特征知识库。针对采集到的日志数据中存在IP定位不全、冲突及噪声数据等问题,实现了网络日志数据的清洗、整理和存储。针对全球IP地域类特征的提取,该知识库提升了对IP地理位置的查询效果,同时为IP地域类特征的精确提取提供了一种可行的解决方案也为后续的IP用户上网行为特征的提取提供了数据支撑。(2)网络用户上网行为分析及异常流量检测方法研究提出了一种基于滑动时间窗口的网络异常流量检测方法。通过分析网络日志,分别从IP用户的地域分布、活跃时间分布和访问内容分布三个层面对网络用户的上网行为进行分析,并采用滑动时间窗口技术,对特定时间段下的网络异常流量进行分析检测,实现对异常IP的监督与关注。通过设计的相关实验验证,结果表明,所提出的基于滑动时间窗技术的异常流量检测方法是有效和可行性的。(3)IP用户访问内容的主题发现及用户聚类方法研究提出了一种基于LDA(Latent Dirichlet Allocation)模型的用户访问内容的主题发现方法,该方法既弥补了原有LDA模型提取的主题分散之缺点,又解决了主题知识库词汇不完整的问题。同时,通过对IP用户所访问的URL解析,提取访问页面的关键字、标题及网页描述等内容信息,利用K-means聚类算法实现了对具有相似访问内容的IP用户的聚类。(4)网络用户上网行为动机分析方法研究网络用户所访问的内容可以反映其上网的动机,因此,我们可以将上网动机的发现转换成对其所访问内容的分类问题。提出了一种基于LIBSVM模型和LIBLINEAR模型相融合的加权分类模型LLA(Libsvm and Liblinear Algorithm)。该模型针对LIBSVM分类准确率高和LIBLINEAR适于大数据的特点,通过实验分析,对两个模型进行加权处理,以得到面向大数据处理的更优分类模型,两种模型的加权权重通过实验获得。实验结果表明,LLA模型在对在用户行为动机分类中,取得了较高的准确率。
[Abstract]:With the rapid development of Internet technology, based on the characteristics of freedom of information flow, freedom of speech and freedom of use, the Internet has formed a virtual communication network without national boundaries. At the same time, due to the unique characteristics of IP, So it is more like the Internet, the "social" identity card ", it can highlight the personal characteristics of network IP users. Therefore, it is of great significance to mine IP users' Internet behavior from web logs and to find out users' search intention, interest preference and Internet motivation. Different from the traditional method of user behavior analysis based on network log, this paper focuses on the hidden information contained in the log, through the in-depth mining and analysis of the hidden information, find out the relationship between IP users' online behavior and users' psychology. In order to provide a new way of thinking and research for IP users'Internet behavior analysis. The specific research work includes the following four aspects: 1) the collection and processing of network log data and the construction of a set of global IP geographical feature knowledge base. Aiming at the problems of incomplete IP location, collision and noise data in the collected log data, the cleaning, sorting and storing of the log data are realized. According to the feature extraction of global IP region, the knowledge base improves the query effect of IP geographical location. At the same time, it provides a feasible solution for the accurate extraction of IP regional features, and provides data support for the subsequent IP users' Internet behavior feature extraction. 2) Network users' Internet behavior analysis and abnormal traffic detection method. A method of network anomaly detection based on sliding time window is proposed. By analyzing the network log, this paper analyzes the Internet access behavior of the network users from the three aspects of IP users' geographical distribution, active time distribution and access content distribution, and adopts sliding time window technology. This paper analyzes and detects the abnormal network traffic in a specific time period, and monitors and pays attention to the abnormal IP. The experimental results show that, The proposed method of anomaly traffic detection based on sliding time window is effective and feasible. The topic discovery and user clustering method of IP user access content based on LDA(Latent Dirichlet allocation model is proposed. A user access content based on LDA(Latent Dirichlet allocation model is proposed. Method of topic discovery, This method not only makes up for the problem of topic dispersion in the original LDA model, but also solves the problem that the topic knowledge base vocabulary is incomplete. At the same time, by parsing the URL visited by IP users, the key words, title and description of the page are extracted. Using K-means clustering algorithm to realize clustering of IP users with similar access content. We can convert the discovery of Internet motivation into a classification of the content it accesses. A weighted classification model, LLA(Libsvm and Liblinear algorithm, is proposed based on the fusion of LIBSVM model and LIBLINEAR model. According to the characteristics of LIBSVM classification accuracy and LIBLINEAR suitable for big data, the two models are weighted by experimental analysis to obtain a better classification model for big data processing. The weighted weights of the two models are obtained by experiments. The experimental results show that the LLA model has a high accuracy in user behavior motivation classification.
【学位授予单位】：北京信息科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【参考文献】