电商流量大数据处理系统设计与开发

发布时间：2018-02-24 23:30

本文关键词： Hadoop Elasticsearch 数据处理电商流量　出处：《浙江工业大学》2016年硕士论文　论文类型：学位论文

【摘要】：计算机技术的发展和互联网的日益普及,使得电子商务飞速发展,电商企业不断涌现,用户数量成倍增加,导致电商网站的多源流量数据也随之快速攀升。传统的电商流量数据处理方式存储数据量有限,支持的数据源格式少,亟需引入现代化的电商流量大数据处理系统,以满足企业流量数据处理的需求。论文基于Hadoop技术和Elasticsearch技术,设计并开发电商流量大数据处理系统,实现海量电商流量原始数据的快速提取、转化、加载和管理,最终计算得到电商网站的页面点击量、独立访客数、页面转化率、页面跳出率、页面导出率等流量指标。论文主要工作如下:1.在文献查阅的基础上,综述了论文研究背景与意义,以及电商流量数据处理的国内外现状,详细介绍Elasticsearch中的搜索引擎Lucene和分布式存储技术、ETL技术中的分布式文件存储系统HDFS和编程模型MapReduce以及开源框架SpringMVC、Spring和Mybatis技术。2.从市场需求、社会经济效益、功能需求等方面对系统进行总体分析,基于原始数据ETL模块和流量统计服务模块的详细需求分析,设计系统的数据导入、数据ETL、数据管理和参数统计四个功能模块,并详细介绍了各个模块的功能。3.原始数据ETL系统采用Hadoop集群中的分布式文件系统HDFS存储数据,采用MapReduce编程模型处理流量数据,同时支持Oracle、MySql和PostgreSql等关系型数据库、数据接口以及其它类型的原始数据源的处理。4.流量统计服务系统采用Elasticsearch集群作为流量数据源,可根据需要进行集群扩容,采用Oracle数据库存储系统常规数据,采用SSM框架搭建系统后台,采用Bootstrap、Jquery和Ajax等技术进行前端开发。5.原始数据ETL系统实现原始流量数据的提取、处理和管理,以及将处理后的流量数据导入到Elasticsearch集群中;流量统计服务系统基于Elasticsearch集群中的流量数据和Oracle数据库中的常规数据实现了多种流量参数的统计,同时实现了人员管理、流量报警和定时邮件等常用功能。目前,电商流量大数据处理系统已经在国内某著名大型电商企业投入运营,系统运行稳定、反应良好。
[Abstract]:With the development of computer technology and the increasing popularity of the Internet, e-commerce has developed rapidly, e-commerce enterprises have been emerging, and the number of users has multiplied. As a result, the multi-source traffic data of e-commerce websites is also rising rapidly. Traditional e-commerce traffic data processing methods have limited storage data and less data source formats, so it is urgent to introduce a modern e-commerce traffic big data processing system. In order to meet the demand of enterprise traffic data processing, based on Hadoop technology and Elasticsearch technology, this paper designs and develops an e-commerce traffic big data processing system, which can quickly extract, transform, load and manage the raw data of mass e-commerce traffic. Finally, we calculate the number of page hits, number of independent visitors, page conversion rate, page jump rate, page export rate, etc. The main work of this paper is as follows: 1. On the basis of literature review, the main work of this paper is as follows: 1. This paper summarizes the research background and significance of the paper, as well as the domestic and foreign status of e-commerce traffic data processing. This paper introduces in detail the search engine Lucene in Elasticsearch, the distributed file storage system HDFS and programming model MapReduce in distributed storage technology, and the open source framework Spring Lucene and Mybatis technology. 2. Based on the detailed requirement analysis of the original data ETL module and the flow statistics service module, the design of the system data import, data ETL, data management and parameter statistics four functional modules, The function of each module is introduced in detail. The original data ETL system uses HDFS, a distributed file system in Hadoop cluster to store data, uses MapReduce programming model to process traffic data, and supports relational databases such as Oracle SQL and PostgreSql. Data interface and other types of original data source processing .4.The traffic statistics service system uses Elasticsearch cluster as the traffic data source, can expand the capacity of the cluster according to the need, and uses the Oracle database to store the conventional data of the system. SSM framework is used to build the background of the system, and Bootstrapper Jquery and Ajax are used to develop the front-end. 5.The raw data ETL system can extract, process and manage the original traffic data, and import the processed traffic data into the Elasticsearch cluster. The traffic statistics service system realizes the statistics of various traffic parameters based on the traffic data in the Elasticsearch cluster and the regular data in the Oracle database. At the same time, it realizes the common functions such as personnel management, traffic alarm and regular mail. The e-commerce flow big data processing system has been put into operation in a famous domestic e-commerce enterprise. The system runs stably and responds well.
【学位授予单位】：浙江工业大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13;TP274

【相似文献】