移动互联网审计平台数据处理方案的改进与实现

发布时间：2018-05-17 23:11

本文选题：移动互联网 + 安全审计　；参考：《北京交通大学》2017年硕士论文

【摘要】：随着第四代移动通讯技术的崛起和智能终端的迅速发展,移动设备的网络连入量占据互联网的份额也越来越大,在移动互联网时代,数据业务渐变成了用户的核心需求,数据价值也将会成为运营商的最宝贵价值所在。因此移动互联网业务数据的分析与审计就会变得非常必要且重要。然而,也正因为移动设备的普及和移动应用的全方位覆盖,使得数据随时随地都在产生,移动网络数据包小且杂、同一时刻的实时活跃连接数量庞大、智能终端的应用种类繁多成了移动互联网的主要特点,对于数据分析,处理速度要求也变的更高,高效率的实时流量处理才能令移动业务数据产生相应的价值。原有移动互联网审计平台是单进程多线程架构的系统,采用传统的Libpcap工具来捕获数据包,单纯地采用DPI技术进行数据包解析,这样的审计系统性能问题诸多,频繁的中断请求和数据拷贝,耗时的DPI业务识别,串行的单任务进程等原因严重限制了移动互联网审计平台的性能,当流量速率足够大时,原平台的处理效率令人堪忧。本文分析指出原系统严重限制数据处理效率的原因,为提高审计平台数据处理的能力以满足实时高速的移动业务数据细粒度审计的需求,提出了数据处理过程中新的阶段性工作改进方案,在数据采集阶段采用基于PF_RING+NAPI的数据包捕获方案,并结合零拷贝思想做出相应的改进措施,在数据解析阶段采用基于DPI/DFI相结合的数据解析方式,并提出了基于缓存命中的预处理方法,大幅度降低了 DPI处理压力。同时,采用多核多进程架构重新设计审计平台的数据分析模块,在充分利用硬件资源的同时,提升审计平台的整体性能。通过阶段化的验证试验,对数据采集阶段和数据解析阶段的方案分别做了性能测试,并展示多进程架构下的系统运行状态。实验表明,本文方案完成了高速移动网络环境下,实时稳定的业务数据处理审计任务,达到了预期研究目标,可助力运营商开展移动网络数据价值转换和移动互联网安全监管等后续工作。
[Abstract]:With the rise of the fourth generation mobile communication technology and the rapid development of intelligent terminals, the amount of network connection of mobile devices occupies an increasingly large share of the Internet. In the era of mobile Internet, data services have gradually become the core needs of users. Data value will also become the most valuable value of operators. Therefore, the analysis and audit of mobile Internet business data will become very necessary and important. However, because of the popularity of mobile devices and the omni-directional coverage of mobile applications, the data is generated anytime and anywhere, the mobile network data packets are small and miscellaneous, and the number of real-time active connections at the same time is huge. The application of intelligent terminals has become the main characteristic of mobile Internet. For data analysis, processing speed also changes higher, high efficiency real-time traffic processing can make mobile business data produce corresponding value. The original mobile Internet audit platform is a single-process multi-threaded system. The traditional Libpcap tools are used to capture data packets, and the DPI technology is used to resolve the data packets. There are many problems in the performance of the audit system. Frequent interrupt requests and data copy, time-consuming DPI service identification, serial single-task process and other reasons seriously limit the performance of the mobile Internet audit platform. When the traffic rate is large enough, the processing efficiency of the original platform is worrying. This paper analyzes and points out that the original system seriously limits the efficiency of data processing. In order to improve the capability of data processing in the audit platform to meet the needs of real-time and high-speed fine-grained audit of mobile business data, In this paper, a new stage work improvement scheme in the process of data processing is put forward. In the data acquisition stage, the packet capture scheme based on PF_RING NAPI is adopted, and the corresponding improvement measures are made in combination with the zero-copy idea. In the stage of data parsing, the data parsing method based on DPI/DFI is adopted, and the preprocessing method based on cache hit is proposed, which greatly reduces the processing pressure of DPI. At the same time, the multi-core and multi-process architecture is used to redesign the data analysis module of the audit platform, which makes full use of the hardware resources and improves the overall performance of the audit platform. The performance of the schemes in the data acquisition stage and the data analysis stage are tested by the stage verification test, and the system running state under the multi-process architecture is shown. The experimental results show that the scheme has completed the audit task of real-time and stable service data processing under the high-speed mobile network environment, and has achieved the expected research goal. Can help operators to carry out mobile network data value conversion and mobile Internet security supervision and other follow-up work.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TN929.5;TP393.01;TP274

【相似文献】