恶意代码分析平台的设计与实现

发布时间：2018-04-08 15:01

本文选题：Elasticsearch　切入点：数据仓库　出处：《北京交通大学》2016年硕士论文

【摘要】：恶意代码数量和种类在不断的增加,并具备大数据的特点。当面对海量的恶意代码数据,传统的分析工具难以发挥作用。恶意代码在分析过程中会产生很多分析日志,目前,针对这些分析日志并没有一个合适的方案来进行管理、搜索、分析。本文针对于恶意代码分析产生的分析日志,设计并实现了恶意代码分析平台,利用大数据手段挖掘样本直接的联系,提高恶意代码分析的效率。本文首先阐述了恶意代码分析平台的背景和意义,然后根据业务需求提出功能性和非功能性的需求分析,并对系统设计提出解决方案。该平台是基于Spark、 Hive和Elasticsearch的框架来实现数据的实时分析、离线分析和全文搜索的,并分为四层结构：采集层、大数据层、业务层和展示层。分析日志通过Kafka推送入采集层并经过抽取、清理和转化后进入大数据层。大数据层是基于Spark进行的数据分析,并为业务层提供服务。业务层是基于SpringMVC+Hiberate构建的REST架构的Web Service。业务层通过跟大数据层交互来实现样本分析、样本搜索、报表分发、实时统计等功能,是本文所介绍的重点。作者参与完成了以下工作：(1)参与高可用的Hadoop集群的构建和Elasticserach的构建。(2)设计用户认证授权方式并实现用户管理模块。(3)参与邮件分发模块的业务功能设计与实现。(4)参与样本搜索的功能的设计与实现。(5)实现样本分析模块业务层的功能。本系统已经通过测试。本人设计实现的功能模块均能够正常运行,并且各模块的运行情况与用户需求相符合。
[Abstract]:Malicious code in the number and type of increasing, and with big data's characteristics.In the face of a large amount of malicious code data, the traditional analysis tools are difficult to play a role.Malicious code in the analysis process will produce a lot of analysis logs, at present, there is no suitable solution for these analysis logs to manage, search, analyze.Aiming at the analysis log generated by malicious code analysis, this paper designs and implements a malicious code analysis platform, and uses big data to mine the direct connection of samples to improve the efficiency of malicious code analysis.This paper first describes the background and significance of malicious code analysis platform, then proposes functional and non-functional requirements analysis according to business requirements, and puts forward a solution to the system design.The platform is based on the framework of Spark, Hive and Elasticsearch to realize real-time data analysis, offline analysis and full-text search, and is divided into four layers: acquisition layer, big data layer, business layer and display layer.The analysis log is sent into the collection layer by Kafka, then extracted, cleaned up and transformed into big data layer.Big data layer is based on Spark data analysis, and provide services for the business layer.The business layer is the Web Services of the REST architecture built on SpringMVC Hiberate.The business layer interacts with big data layer to realize the functions of sample analysis, sample search, report distribution, real-time statistics and so on, which is the focus of this paper.The author is involved in the following work: 1) participating in the construction of highly available Hadoop cluster and Elasticserach. 2) designing user authentication authorization mode and realizing user management module. 3) designing and implementing the business function of mail distribution module.Design and implement the function of sample search. 5) realize the function of business layer of sample analysis module.The system has passed the test.The functional modules designed and implemented by me can run normally, and the operation of each module is in accordance with the user's requirements.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.52;TP309

【相似文献】