高性能网络流量采集和分析技术的实现

发布时间：2018-11-20 19:03

【摘要】：21世纪是信息技术的时代,随着网络技术的提高,信息技术的发展,计算机科学的进步,信息技术渗透到人们生活中的各处角落,为人们的工作,学习,生活带来了巨大的便利。近年来,互联网迅猛发展,网络带宽爆发式增大,高速网络发展为人们的生活带来了便利,促进了行业的发展,同时日益发展的高速网络也为软件系统的设计提出了难题。首先是高速网络数据流量采集的问题,为了对网络状态进行统计,需要对流量进行抓取分析,这项技术被广泛应用在网络监控,信息安全等方面,然而随着网络流量的爆发式增长,传统的数据包捕获方式的弊端开始显现出来,为了改善这种状况,国内外相关领域的学者进行了不懈的努力。近年来,DPDK(Intel Data Plane Development Kit)高性能报文处理框架技术的发展使高速网络下的数据包捕获成为了可能。本文中将对DPDK技术进行简要介绍。在各项高性能数据分析技术中,哈希算法一直占有一席之地。传统哈希算法在存储方面已经表现出高性能的特点,然而在面对高速网络复杂多变的情况下,经典哈希算法的性能和健壮性有时会略显不足。本文将重点对高速率高并发网络下的哈希算法进行优化。本文会在三方面对已有的哈希算法进行优化,首先提出具有普适性的两种Cache算法,PBS-CPE和Cuckoo_Cache。这两种算法能够在不改变哈希算法均匀性的基础上提升查询_效率。继而针对布谷鸟哈希算法的插入性能进行优化,增加哈希函数的个数,设置溢出栈。最后针对分段哈希算法在高速网络环境下的性能表现提出了两种优化方法:最小下标值法和布鲁姆过滤器的迭代更新算法,根据这两种优化提出了具有skew特性的多段哈希算法,该算法在高速率高并发网络环境下相较于分段哈希算法来说具有更加优异的性能表现。在本论文中,设计了高速率高并发网络流管理系统,该系统利用DPDK技术对数据包进行采集,采用各种不同的哈希算法对数据包进行存储,在对数据包进行处理的同时生成该种哈希算法的性能数据,在统计结束后将该性能数据写入到本地日志中,以便后续的性能对比。同时,在本论文中,会对各项哈希算法及论文中提出的各项优化方面进行对比,通过实验数据可知,PBS-CPE和Cuckoo_Cache缓存算法能够提升哈希算法的查询效率。针对布谷鸟哈希算法的两种优化方法均能极大提高布谷鸟哈希表容量不高时其算法的插入性能和健壮性。布鲁姆过滤器的迭代更新法能够减小布鲁姆过滤器的假阳性率,因此在一定程度上能够提升查询性能。具有skew特性的多段哈希算法在高速网络的特定环境下相较于分段哈希算法来说具有更佳的查询性能。
[Abstract]:The 21st century is the era of information technology. With the improvement of network technology, the development of information technology and the progress of computer science, information technology permeates into every corner of people's life, and it can work and learn for people. Life brings great convenience. In recent years, with the rapid development of the Internet and the explosive increase of the network bandwidth, the development of high-speed network has brought convenience to people's life and promoted the development of the industry. At the same time, the increasingly developing high-speed network has also posed a difficult problem for the design of software system. First of all, the problem of high-speed network data flow collection. In order to statistics the network status, it is necessary to grasp and analyze the traffic. This technology has been widely used in network monitoring, information security, and so on. However, with the explosive growth of network traffic, the malpractice of traditional packet capture has begun to appear. In order to improve this situation, scholars at home and abroad have made unremitting efforts. In recent years, the development of, DPDK (Intel Data Plane Development Kit) high performance packet processing framework makes it possible to capture packets in high speed networks. In this paper, the DPDK technology will be briefly introduced. Among the high performance data analysis techniques, the hash algorithm has always occupied a place. The traditional hash algorithm has shown high performance in storage. However, in the face of complex and changeable high-speed network, the performance and robustness of the classical hash algorithm are sometimes slightly inadequate. This paper focuses on the optimization of hash algorithm in high speed and high speed concurrent networks. In this paper, we will optimize the existing hash algorithms in three aspects. First, we propose two universal Cache algorithms, PBS-CPE and Cuckoo_Cache.. These two algorithms can improve the query _ efficiency without changing the uniformity of the hash algorithm. Then the insertion performance of the cuckoo hashing algorithm is optimized to increase the number of hash functions and set the overflow stack. Finally, two optimization methods are proposed for the performance of piecewise hash algorithm in high-speed network: minimum subscript method and iterative updating algorithm of Bloom filter. According to these two optimization methods, a multi-segment hash algorithm with skew characteristics is proposed. The performance of this algorithm is better than that of piecewise hash algorithm in high speed and high concurrency network environment. In this paper, a high speed and high concurrency network flow management system is designed. The system uses DPDK technology to collect data packets, and uses various hash algorithms to store data packets. The performance data of the hashing algorithm is generated at the same time when the packet is processed, and the performance data is written to the local log at the end of the statistics for the subsequent performance comparison. At the same time, in this paper, the hashing algorithms and the proposed optimization aspects are compared. Through the experimental data, we can see that PBS-CPE and Cuckoo_Cache cache algorithms can improve the query efficiency of the hash algorithm. The two optimization methods for the cuckoo hashing algorithm can greatly improve the insertion performance and robustness of the cuckoo hashing algorithm when the capacity of the cuckoo hash table is not high. The iterative updating method of the Bloom filter can reduce the false positive rate of the Bloom filter, so it can improve the query performance to some extent. Multi-segment hash algorithm with skew characteristics has better query performance than segmented hash algorithm in the specific environment of high-speed network.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.0;TP274.2

【相似文献】