HTTP协议采集系统的设计实现与应用

发布时间：2018-07-24 12:55

【摘要】：在网络与计算机普及并渗透千家万户的当今社会,我们迎来了大数据DT时代,因网络中大部分访问报文由HTTP协议承载,高效实时的采集HTTP数据并挖掘其中的信息有助于更好的认知真实网络数据蕴藏的价值以及了解网民的行为习惯,进而可以优化相关网络服务并可以更有针对性的为用户提供服务。本文首先讲述了万维网中HTTP协议的工作原理及报文特点并介绍了分布式计算大数据平台的原理,为后文系统开发与大数据分析处理奠定理论基础。然后设计并开发了 HTTP协议采集系统,该系统可以实时的从网卡接收数据,对乱序到达的报文进行重组,结合多模匹配算法根据用户定义字段去解析报文,生成格式化话单。完成开发后,系统通过了功能测试及性能测试。为了使系统更好的适配真实网络环境,在分析了采集系统性能影响因素后,本文从软件结构和报文解析两个角度对系统进行优化提升。随后将HTTP协议采集系统部署于核心网及校园网中,实时采集数据,分别针对核心网和校园网的话单数据进行分析,了解不同群体上网产生的流量特征及HTTP报文特点。此外,本课题还利用话单数据对斗鱼软件进行用户留存及操作行为分析,将HTTP大数据分析应用于商业软件分析中,帮助软件更好的了解用户群体。最后论文对HTTP协议采集系统的设计实现与应用进行了总结与展望。
[Abstract]:In today's society where the network and computer are popularized and permeated by thousands of households, we have ushered in the era of big data DT, because most of the access packets in the network are carried by the HTTP protocol. Efficient real-time HTTP data collection and mining the information in it is helpful to better understand the value of the real network data and understand the behavior habits of Internet users. Furthermore, it can optimize related network services and provide more targeted services to users. This paper first describes the working principle and message characteristics of HTTP protocol in the World wide Web, and introduces the principle of distributed computing big data platform, which lays a theoretical foundation for the later development of the system and the analysis and processing of big data. Then the HTTP protocol acquisition system is designed and developed. The system can receive the data from the network card in real time, reorganize the incoming packets in disorder order, analyze the packets according to the user-defined fields combined with the multi-mode matching algorithm, and generate the formatted voice sheet. After the development, the system passed the function test and performance test. In order to better adapt the system to the real network environment, after analyzing the factors affecting the performance of the acquisition system, this paper optimizes the system from the two aspects of software structure and packet analysis. Then the HTTP protocol acquisition system is deployed in the core network and campus network to collect the data in real time. The data of the core network and the campus network are analyzed respectively to understand the traffic characteristics and the characteristics of HTTP message generated by different groups of Internet. In addition, this paper also uses the data of telephone bill to carry on the user retention and the operation behavior analysis to the Douyu software, applies the HTTP big data analysis to the commercial software analysis, helps the software to understand the user group better. Finally, the design and application of HTTP protocol acquisition system are summarized and prospected.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.04

【相似文献】