基于Hbase的大数据存储系统研究开发

发布时间：2018-08-06 09:17

【摘要】：随着大数据时代的到来,信息系统数据库所存储的数据量呈爆炸式增长,对于数据读写与查询的性能要求也越来越高,传统的关系型数据库已经无法满足大数据存储与查询的要求。为了探索海量数据的存储与查询技术,本文课题围绕典型的非关系型(NoSQL)数据库Hbase进行了研究开发。Hbase是GoogleBigTable的开源版本,具有高可靠性、高性能、面向列、可伸缩、一致性等特点,支持二级索引功能。利用Hbase技术可在廉价PC Server上搭建起大规模存储集群,实现大数据存储系统。本课题首先研究了大数据存储系统的体系结构、NoSQL数据库关键技术、Hbase数据库关键技术。接着在Spark大数据平台部署了 Hbase数据库系统,存储了流动人口数据库。由于Hbase数据库只支持基于主键的查询,我们对流动人口数据库增加了二级索引功能,大大提高了查询速度。在此基础上对基于Hbase的流动人口数据库的性能进行了分析评估,并使用雅虎公司开发的测试工具YCSB对Hbase的性能进行了测试。测试对象是基于某企业提供的实际数据而建立的Hbase数据表,总记录条数为3000万条。最后,基于Spark大数据平台和Hbase数据库系统,开发了海量流动人口数据管理原型系统。该系统具有流动人口数据获取、数据存储、数据管理、统计分析、系统管理等功能,其中存储的数据记录达3000万条,数据总量达到12.6GB,实现了流动人口海量数据的高效存储与快速查询。
[Abstract]:With the arrival of the big data era, the amount of data stored in the information system database is increasing explosively, and the performance requirements of data reading, writing and querying are becoming more and more high. The traditional relational database can no longer meet the requirements of big data storage and query. In order to explore the storage and query technology of massive data, this paper focuses on the research and development of typical non-relational (NoSQL) database Hbase. Hbase is an open source version of GoogleBigTable, which has the characteristics of high reliability, high performance, column oriented, scalable, consistent, and so on. Support for secondary indexing. A large scale storage cluster can be built on cheap PC Server by using Hbase technology, and the big data storage system can be realized. In this paper, the architecture of big data storage system is studied firstly, and the key technology of Hbase database is discussed. Then the Hbase database system is deployed on the Spark big data platform, and the floating population database is stored. Because the Hbase database only supports the primary key query, we add the secondary index function to the floating population database, which greatly improves the query speed. On this basis, the performance of floating population database based on Hbase is analyzed and evaluated, and the performance of Hbase is tested by YCSB, a testing tool developed by Yahoo Corporation. The test object is a Hbase data table based on the actual data provided by an enterprise. The total number of records is 30 million. Finally, based on Spark big data platform and Hbase database system, a prototype system of massive floating population data management is developed. The system has the functions of data acquisition, data storage, data management, statistical analysis, system management and so on. Among them, 30 million records are stored. The total amount of data reached 12.6 GB, which realized the efficient storage and fast query of massive data of floating population.
【学位授予单位】：西安理工大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP333

【参考文献】