网络流量存储与分析平台的设计与实现

发布时间：2018-05-21 12:59

本文选题：网络流量 + NetFlow　；参考：《山东大学》2017年硕士论文

【摘要】：目前社会各领域的数据都在以爆炸式的速度增长,伴随着计算机科学与技术的发展使得数据的传递、存储与处理方式产生了巨大的变化,由于网络信息技术的蓬勃发展,个人和互联网上的程序以及应用都产生了大量的数据,数据的飞快增长对数据的存储和高速访问都提出了新的挑战。正是因为网络数据的快速增长,不同网络应用程序都需要拥有扩展存储容量的性能,并且应用程序应该能够实现存储节点的动态加入,并且保证历史数据及应用程序产生的最新数据在不同存储节点的分布均匀,从而实现整个平台系统的运行负载均衡。根据现阶段网络数据存储技术的发展,应用程序的动态扩容成本较高,而分布式存储技术可以良好的解决海量数据的低廉存储及高效的检索。本文介绍了网络流量存储与分析平台的开发与实现,针对业务需求,结合网络流量分析的相关技术,对网络流量存储与分析平台开展功能性需求分析和非功能性需求分析,并根据需求分析结果对系统功能架构和技术架构完成设计与实现。网络数据的采集分析业务使用C++语言基于nprobe改造完成,使用gRPC框架实现采集分析子系统与元数据管理子系统的数据通信传输功能,元数据管理系统负责维护、更新存储节点的信息并为存储请求、查询请求分配节点信息。查询业务使用C++语言实现,首先向元数据管理系统发送查询请求,获得存储节点信息后多线程完成各个存储节点的查询业务,并将结果数据返回客户端,由客户端完成结果汇聚后展示给用户或供第三方应用使用。根据业务需求,选择fastbit作为网络流量存储与分析平台的数据库,针对业务逻辑,设计数据库分区结构及存储路径。根据当前网络的实际运行环境,本文设计一套完整的流量采集分析和存储方案,并在此基础上实现了网络流量存储与分析平台。根据平台数据的分析结果,我们能够掌握在特定时刻下当前网络的整体状态;掌握不同的应用在网络中的资源占用情况,使得资源不足的应用能够得到更多的带宽;掌握每一个用户使用网络的具体情况,并根据结果信息促进管理人员更加有效地分配相关资源;利用NetFlow的相关数据,对运行网络的不同特征如数据流向等开展解析过程,从而获得网络的运行特征从而抓取潜在的流量异常。本文通过实验测评,验证了网络流量存储与分析平台在海量流量下系统的采集、存储与检索能力,在海量历史数据下的系统查询性能的提升以及系统运行的稳定性与可扩展性。
[Abstract]:At present, the data in all fields of society are increasing at an explosive rate. With the development of computer science and technology, great changes have taken place in the way of data transmission, storage and processing, because of the vigorous development of network information technology. Personal and Internet programs and applications have generated a lot of data, and the rapid growth of data has posed new challenges to data storage and high-speed access. Because of the rapid growth of network data, different network applications need to have the capability of expanding storage capacity, and applications should be able to dynamically join storage nodes. It also ensures that the historical data and the latest data generated by the application are evenly distributed among different storage nodes, thus realizing the load balance of the whole platform system. According to the development of network data storage technology at present, the dynamic expansion cost of application program is high, while distributed storage technology can solve the problem of low cost storage and efficient retrieval of mass data. This paper introduces the development and implementation of the network traffic storage and analysis platform. According to the business requirements and the related technologies of network traffic analysis, the functional requirement analysis and non-functional requirement analysis are carried out for the network traffic storage and analysis platform. According to the results of requirement analysis, the system functional architecture and technical architecture are designed and implemented. The network data collection and analysis business is transformed by C language based on nprobe, and the data communication and transmission function between the collection and analysis subsystem and the metadata management subsystem is realized by using the gRPC framework. The metadata management system is responsible for the maintenance. Updates the information of the storage node and assigns node information to the storage request, query request. The query service is implemented in C language. Firstly, the query request is sent to the metadata management system. After obtaining the information of the storage node, the query service of each storage node is completed by multi-threading, and the result data is returned to the client. After the client completes the result aggregation, it is displayed to the user or the third party application. According to the business requirement, fastbit is chosen as the database of network traffic storage and analysis platform, and the database partition structure and storage path are designed according to the business logic. According to the actual running environment of the current network, this paper designs a set of complete traffic collection, analysis and storage scheme, and realizes the network traffic storage and analysis platform on this basis. According to the analysis results of the platform data, we can master the whole state of the current network at a particular time, grasp the resource occupation of different applications in the network, make the application with insufficient resources get more bandwidth. Master the specific situation of each user using the network, and according to the result information to promote the managers to allocate the relevant resources more effectively; using the relevant data of NetFlow, the different characteristics of running the network, such as the flow of data, etc. In order to obtain the characteristics of the network operation, and grab the potential traffic anomalies. This paper verifies the ability of collecting, storing and retrieving the network traffic storage and analysis platform under the massive traffic, the improvement of the system query performance under the massive historical data, and the stability and expansibility of the system running by the network traffic storage and analysis platform.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【参考文献】