网络实时分析系统的分析平台设计与实现

发布时间：2018-03-31 00:11

本文选题：计算机海量数据　切入点：流计算　出处：《北京邮电大学》2014年硕士论文

【摘要】：随着网络信息的急剧膨胀,给网络的管理也带来了前所未有的挑战。传统的网络管理软件有扩展性差,分析网络数据方式固定化,用户无法自定义数据分析方式等问题。针对以上问题,本文提出了一种支持海量数据实时处理的网络分析系统和方法来弥补这个领域的不足。针对现有网络分析软件存在的缺陷,本文提出一种基于“流水线”思想的网络分析系统平台的设计方案。同时,该方案结合P2P技术和事件处理机制等进行设计和实现。该平台的工作方式类似于“流水线”,每个数据分析流程由若干个相对独立的数据分析节点来一环扣一环的逐步分析完成；其底层的各数据分析逻辑间通过统一的事件机制来通信协作。同时,为了实现用户自定义数据分析方式,系统增加逻辑管理模块,用户通过扩展通用的模板接口,可以非常方便的完成自定义数据处理的需求。为实现上述设计,本文通过两种数据节点完成系统的功能：数据适配节点和数据分析节点。数据适配节点主要完成数据和内部事件之间的适配。数据分析节点是来完成数据分析任务,我们使用数据分析节点的可扩展性和可自定义处理逻辑性来完成流水线的搭建,完成数据的监视分析等操作。本文通过类图和流程图两种方式展开实现的阐述。最后通过网络数据包超时的实例测试,完成功能的校验及性能的测试。本文阐述的方案特色：1、用户可以根据自己的要求,通过我们提供的接口自定义数据分析逻辑,将其注入容器,即可生效；2、本系统可以通过动态扩展节点来提高性能的要求。由于有流水线的作业方式和事件路由机制提供的支持,只需要积木式的增加服务器,数据适配节点就会动态的将负载转移到新增加的服务器上,从而到达动态扩展节点来提高性能的要求。
[Abstract]:With the rapid expansion of network information, it brings unprecedented challenges to network management. To solve the above problems, a network analysis system and method to support real-time processing of massive data is proposed to make up for the shortcomings in this field. In view of the defects of the existing network analysis software, this paper proposes a design scheme of the network analysis system platform based on pipeline's thought. This scheme is designed and implemented with P2P technology and event handling mechanism. The working mode of this platform is similar to that of "pipeline". Each data analysis process is completed by several independent data analysis nodes step by step. At the same time, in order to realize the user-defined data analysis mode, the system adds the logical management module, and the user extends the common template interface. You can easily complete the requirements of custom data processing. In order to implement the above design, In this paper, two kinds of data nodes are used to complete the functions of the system: the data adaptation node and the data analysis node. The data adaptation node mainly completes the adaptation between the data and the internal events. The data analysis node is to complete the data analysis task. We use the extensibility and customizable processing logic of the data analysis node to complete pipeline's construction. Through class diagram and flow chart, the implementation is described. Finally, the function verification and performance test are completed by the example test of network data packet timeout. The solution described in this article features: 1, the user can customize the data analysis logic through the interface we provide, and inject it into the container according to his own request. This system can improve the performance requirements by dynamically extending nodes. With the support of pipeline's operation mode and event routing mechanism, only building blocks are needed to increase the server. The data adaptation node will dynamically transfer the load to the newly added server, thus reaching the requirement of dynamic expansion node to improve the performance.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2014
【分类号】：TP393.07

【参考文献】