寄递大数据分析系统的设计与实现

发布时间：2018-09-07 10:39

【摘要】：随着网络和信息技术的不断发展,网上购物以其方便、快捷的特点受到越来越多用户的青睐,随之带来寄递数据量的激增。海量的寄递数据给企业带来了新的挑战。企业传统的分析系统是将数据存储在关系数据中,这种系统架构在分析海量寄递数据时,会遇到瓶颈。而且,随着数据量的增长,这种系统的扩展代价昂贵。因此,本课题致力于构建一种全新的寄递数据分析系统。本文的研究内容主要包括以下几个方面:(1)学习和研究现有的大数据处理技术。现有的大数据处理技术以批量计算和流式计算两种模式为主导,通过对比分析,本文选取Hadoop和Storm作为本系统分析计算的核心组件。通过对Hadoop、Hive、Flume、Kafka、Storm等开源软件的学习,深入理解大数据处理技术的两种模式。(2)分析现有“安监”系统瓶颈,提出本系统的建设目标。现有“安监”系统的瓶颈,主要集中在一些较为复杂的分析统计功能的性能上。当面单数据的量达到5000(万)条时,业务量运行查询统计、业务量趋势预测等较为复杂查询统计功能的分析时间超过500秒,系统超时。本系统的建设目标是通过构建新分析系统,克服“安监”系统分析瓶颈,并在新系统上实现上述复杂的分析统计功能。(3)给出本系统的架构设计,并根据架构设计对系统进行实现。系统包括数据采集模块、数据预处理模块、数据存储与分析模块和数据展示模块等4个模块。数据采集模块是系统建设的基础。其中“安监”系统采用的是Log4j记录系统日志。寄递大数据分析系统使用Flume日志收集工具将日志文件进行采集,并统一写入HDFS中。针对“安监”系统数据库(关系数据库)中的结构化数据,本系统采用Java程序定期抽取。针对实时采集数据,本系统采用Flume将Log4j日志消息直接推入Kafka中。数据预处理模块是系统重要组部本分之一。本系统通过数据预处理模块将数据处理成“干净”可靠的数据。数据存储和分析模块是系统的核心模块。针对业务需求,本文系统使用Hive、MapReduce和Storm这3种不同的数据分析处理技术对数据进行分析。数据展示模块是将系统结果进行展示。此模块采用了主流J2EE架构和MVC编程模式进行设计和实现,可向用户提供友好的展示界面。(4)搭建系统环境,并对本系统进行测试和验证。本课题搭建了一个20个节点机器的Hadoop集群和5个节点机器节点的Storm集群,对系统进行测试和验证。实验表明,当面单数据的规模为5000(万)时,系统的分析处理时间减少到100秒左右,完全满足设计要求。本课题设计并实现了寄递大数据分析系统,克服传统的关系数据库分析系统对海量寄递数据分析的性能瓶颈问题。并通过实验,证明了该系统在分析海量的寄递数据方面具有明显优势。
[Abstract]:With the continuous development of network and information technology, online shopping is favored by more and more users because of its convenience and quickness, which brings about a surge in the amount of data sent and delivered. Massive data delivery brings new challenges to enterprises. The traditional enterprise analysis system stores the data in the relational data. Moreover, as the amount of data grows, the expansion of the system is costly. Therefore, this subject is devoted to the construction of a new data analysis system. The research contents of this paper mainly include the following aspects: (1) study and study the existing big data processing technology. The existing big data processing technology is dominated by two modes: batch computing and flow computing. Through comparison and analysis, this paper selects Hadoop and Storm as the core components of the analysis and calculation of the system. Through the study of open source software such as Hadoop,Hive,Flume,Kafka,Storm, we deeply understand the two modes of big data's processing technology. (2) analyze the bottleneck of the existing "safety supervision" system, and put forward the construction goal of this system. The bottleneck of the existing safety monitoring system is mainly focused on the performance of some complicated statistical functions. When the number of single data reaches 5000, the analysis time of the more complex query statistical functions, such as running query statistics, forecasting the trend of traffic and so on, exceeds 500 seconds, and the system times out. The goal of this system is to overcome the bottleneck of "safety supervision" system analysis by building a new analysis system, and to realize the complex analysis and statistics function mentioned above on the new system. (3) the architecture design of the system is given. The system is implemented according to the architecture design. The system includes four modules: data acquisition module, data preprocessing module, data storage and analysis module and data display module. Data acquisition module is the foundation of system construction. One of the "safety monitoring" system is the use of Log4j logging system. Big data analysis system uses Flume log collection tool to collect log files and write them into HDFS. In view of the structured data in the database of "safety supervision" system (relational database), the system adopts Java program to extract regularly. In view of the real-time data acquisition, the system adopts Flume to push the Log4j log message directly into the Kafka. Data preprocessing module is one of the important parts of the system. The system processes the data into clean and reliable data through the data preprocessing module. Data storage and analysis module is the core module of the system. According to the business requirements, this paper uses three different data analysis and processing techniques, Hive,MapReduce and Storm, to analyze the data. The data display module displays the system results. This module uses the mainstream J2EE architecture and MVC programming mode to design and implement, can provide a friendly display interface to users. (4) build the system environment, and test and verify the system. In this paper, a Hadoop cluster with 20 nodes and a Storm cluster with 5 nodes are built, and the system is tested and verified. The experimental results show that when the scale of single data is 5000, the analysis and processing time of the system is reduced to about 100 seconds, which fully meets the design requirements. In order to overcome the bottleneck problem of traditional relational database analysis system, this paper designs and implements the analysis system of sending big data. Through experiments, it is proved that the system has obvious advantages in analyzing massive data.
【学位授予单位】：中国科学院大学(中国科学院工程管理与信息技术学院)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13

【相似文献】