基于Spark的情报大数据可视化分析

发布时间：2018-01-01 17:15

本文关键词：基于Spark的情报大数据可视化分析　出处：《山东大学》2017年硕士论文　论文类型：学位论文

【摘要】：信息化时代数据量激增,同时由于情报部门等特殊需求部门多年来对信息的积累,存储了大量结构化、半结构化数据,由于受技术水平、创新意识、支撑保障等诸多因素影响,出现不同程度的信息壁垒,信息共享度、利用率不高,如何利用这些情报数据掌握社会动向、分析事态的演化趋势,进而提前预警预测,并为决策者提出决策性的建议,尚有大量知识需要研究。大数据技术经过多年的发展和完善已经趋于成熟,使用大数据技术进行情报分析,可有效的利用数据,为决策者提供可靠的指引。通过对信息的采集、整合并辅以大数据技术,大力推进相关部门的信息化建设。利用Spark运行在内存中的特点,为使用者提供高效的、交互式的查询和计算,快速展现数据内在信息,提高情报机关工作效率。本系统以Spark、Hadoop大数据技术为基础,面向特殊应用信息库、各情报资源信息库等已有数据库或其他多种类型数据文件,辅以GraphX图计算框架、SparkRDD、SparkSQL等工具,进行快速、高效的信息查询和多种图形化展示,并为使用者提供对社区人群的分析、通话分析、人员关联查询等多种功能。本文主要工作内容如下:1、研究情报系统业务模型,以及情报、特殊部门的具体需求分析,研究Spark、Hadoop大数据分析与存储技术、基于J2EE的前端系统、分布式消息系统以及数据清洗和数据库使用等,并设计了情报大数据分析系统的一种实现方法。2、开发前端系统使用J2EE技术架构,搭载Spring、SpringMVC、Mybatis三大框架作为前端展示系统。其中视图层采用FreeMarker、JQuery EasyUI、ECharts等组件提供多种形式、直观的数据展示。使用Oracle数据库,为面端组件、用户名、密码等信息提供持久化服务。3、分布式计算和传输系统开发。前端展示系统通过Apache Kafka集群与Spark集群进行实时交互。Spark集群主要负责对数据进行计算、分析。通过使用Spark RDD、SparkSQL、GrpahX、GraphFrame等工具进行社区发现、重点人员查找、话单分析、人群分析等功能,并将结果通过Kafka集群实时的反馈给前端展示系统。4、数据清洗和导入工作。数据存储采用分布式系统存储,并支持多种数据来源的导入,如关系型数据库、文本文档、CSV文件等。经过数据清洗后统一存放在HDFS系统中。通过Sqoop实现对存储系统定时更新的功能,在固定时间间隔将外部数据导入到存储系统中,保证数据的实时有效性。大数据分析系统通过将分散在不同业务部门的信息通过汇总、提取、计算实现了部门之间的信息交换,打破信息壁垒,并提供高效、直观、多样的数据可视化处理,为情报机关工作效率和能力的提高发挥积极作用。
[Abstract]:At the same time, due to the accumulation of information by intelligence departments and other special needs departments over the years, a large number of structured, semi-structured data are stored, due to the level of technology, innovation awareness. Many factors, such as support and security, appear different degree of information barriers, information sharing degree, low utilization rate, how to use these information data to grasp the social trends and analyze the evolving trend of the situation. There is still a lot of knowledge to study. Big data technology has matured after many years of development and improvement. Big data technology is used to carry out information analysis. Data can be used effectively to provide reliable guidance for decision makers. Through the collection of information, integration and supplemented by big data technology. Vigorously promote the information construction of relevant departments, make use of the characteristics of Spark running in memory, provide users with efficient, interactive query and calculation, and quickly display the internal information of the data. This system is based on the technology of Sparku Hadoop big data and faces the special application information database. The existing databases or other kinds of data files, such as information base of intelligence resources, are carried out quickly with tools such as SparkRDDD Spark SQL, a framework for calculating GraphX diagrams. Efficient information query and a variety of graphical display, and provide users with community crowd analysis, call analysis, personnel association query and other functions. The main work of this paper is as follows: 1. This paper studies the business model of information system, as well as the analysis of the specific requirements of intelligence and special departments, and studies the analysis and storage technology of Spark Hadoop big data, the front-end system based on J2EE. Distributed message system, data cleaning and database usage, etc., and designed an implementation method of intelligence big data analysis system. The front-end system uses J2EE technology architecture and carries Spring. Spring MVC / Mybatis is used as the front end display system, in which the view layer adopts FreeMarker JQuery EasyUI. ECharts and other components provide a variety of forms of visual data display. Use Oracle database to provide persistence services for facet components, user names, passwords and other information. Distributed computing and transmission system development. The front-end display system through Apache Kafka cluster and Spark cluster real-time interaction. Spark cluster is mainly responsible for data calculation. Analysis. Through the use of Spark RDDN Spark SQL Grpah XN GraphFrame and other tools for community discovery, key people to find, voice analysis. Crowd analysis and other functions, and the results through the Kafka cluster real-time feedback to the front-end display system. 4, data cleaning and import work. Data storage using distributed system storage. And supports the import of various data sources, such as relational databases, text documents. After data cleaning, the CSV file is stored in the HDFS system. Through Sqoop, the storage system is updated periodically, and the external data is imported into the storage system at fixed time intervals. To ensure the real-time validity of the data. Big data analysis system through the aggregation of information scattered in different business departments through extraction, computing to achieve information exchange between departments, break down information barriers, and provide efficiency. Visual and diversified data visualization plays an active role in improving the efficiency and ability of intelligence agencies.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】