基于HTTP协议的网络数据分析系统的设计与实现

发布时间：2018-04-11 23:11

本文选题：HTTP协议 + 数据分析　；参考：《哈尔滨工业大学》2016年硕士论文

【摘要】：随着经济社会的发展,网络深刻地改变和影响着人们各方面的生活习惯,包括社交、电子商务、阅读、知识学习等。科技进步也导致了网络终端接入设备的增加,个人用户能够更加容易接触网络。互联网新经济的崛起,导致人们的网络活动完全是现实生活的缩影。网络上的信息量呈现爆炸式增长,在此背景下,通过收集网络流量日志,分析某一个IP地址下的网络状况,从而刻画用户画像,以了解各个IP地址的信息以及其下可能存在的用户数据。本系统提出基于HTTP协议数据的分析系统,从HTTP协议的流量信息、报文信息、正文信息三个方面分析当前的网络状况,描述了从IP地址到终端设备再到用户个人的网络模型。统计了HTTP协议中的IP流量信息,设计了应用于IP地址依据应用服务划分的朴素贝叶斯分类模型,在IP地址层面描绘了网络状况。参考了HTTP协议中User_Agent字段的变动历史,提出了从User_Agent字段中提取浏览器信息、设备名称和操作系统的方法,实现了终端设备层面的分析。对于用户层面,解析了HTTP协议正文,包括JSON格式和HTML格式的数据,提取了B/S架构的软件和用户交互时包含的个人数据,设计了基于HTML格式数据的知识学习方法以扩充提取数据所依赖的知识库信息。在数据分析的基础上,设计了信息展示平台,以Web的形式提供服务,通过jQuery和Bootstrap等前端框架,实现了简洁易用友好的界面,采用Tomcat服务器,Mysql作为数据库,使用MyISAM存储引擎提高读写效率。最后,本文设计了对该系统进行功能测试和性能测试的方法,在测试阶段进行了程序调优,基本满足了工业和警用的要求,整个系统可投入实际使用运行。
[Abstract]:With the development of economy and society, the network profoundly changes and influences people's life habits, including social, electronic commerce, reading, knowledge learning and so on.Advances in technology have also led to an increase in network terminal access devices, making it easier for individual users to access the network.The rise of the new economy of the Internet leads to people's network activities as a microcosm of real life.The amount of information on the network is increasing explosively. In this context, by collecting the network traffic log, analyzing the network condition under a certain IP address, we can depict the portrait of the user.To understand the information of each IP address and the user data that may exist under it.This system proposes an analysis system based on HTTP protocol data, analyzes the current network status from three aspects of HTTP protocol traffic information, message information and text information, and describes the network model from IP address to terminal equipment to user.The IP traffic information in HTTP protocol is analyzed and a naive Bayesian classification model for IP address partitioning based on application services is designed. The network condition is described at the IP address level.Referring to the changing history of User_Agent field in HTTP protocol, the method of extracting browser information, device name and operating system from User_Agent field is put forward, and the analysis of terminal device level is realized.For the user level, the text of HTTP protocol is analyzed, including the data of JSON format and HTML format, and the software of B / S architecture and the personal data of user interaction are extracted.A knowledge learning method based on HTML format data is designed to expand the knowledge base on which the data is extracted.On the basis of data analysis, the information display platform is designed to provide service in the form of Web. Through the front-end framework such as jQuery and Bootstrap, a simple and user-friendly interface is realized. Tomcat server MySQL is used as the database.Use MyISAM storage engine to improve read and write efficiency.Finally, this paper designs the method of function test and performance test for the system, and optimizes the program in the test stage, which basically meets the requirements of industry and police, and the whole system can be put into practical use.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.52

【相似文献】