基于MongoDB的传感器数据分布式存储的研究与应用

发布时间：2018-08-12 11:23

【摘要】：随着互联网技术的发展，尤其是移动互联网的业务的迅速普及，用户在日常的使用过程中产生了大量的数据。此外随着全球定位系统（GPS）、传感器、自动追踪器以及监控系统的普及，这些新的数据源也产生了大量数据，，它们给存储、分析和归档带来了新的机遇与挑战。在传感器大量使用的情况下，采用传统的关系数据库存储数据出现了新的问题，如：随着数据的急速增长，传统的数据库已无法满足海量数据的存储和管理需求；传感器通常以数秒的间隔采集数据，采集的数据具有很大的冗余等。因此寻求较高数据存储效率的传感器数据存储方案，成为了物联网应用中的研究热点。论文主要工作是针对传感数据高并发和大数据量的存储要求，采用MongoDB自动分片技术,设计并实现基于MongoDB数据库的数据存储接口。在接口实现中，通过对MongoDB数据库操作的封装、数据存储过程中加入数据压缩，将数据存储以虚拟接口的方式提供给用户。本文首先阐述了云数据管理技术，包括云计算技术、NoSQL系统和NoSQL数据库技术。其次研究了现在流行的NoSQL数据存储，在现有的数据存储技术基础上，分析了传感器数据存储的需求和特点，同时基于MongoDB文档型数据库，开发了适合传感器数据存储的WebService接口，以满足传感数据高并发、跨平台的存储以及快速查询的需求。最后研究了实时数据库中广泛使用的旋转门压缩算法,结合传感器数据海量的特点，在WebService接口中的数据压缩模块中采用此压缩算法和二次过滤方法，达到了较高压缩比和去除传感数据冗余的效果。
[Abstract]:With the development of Internet technology, especially the rapid popularity of mobile Internet services, users in the daily use of a large number of data. In addition, with the popularity of GPS (GPS), sensors, automatic trackers and monitoring systems, these new data sources also produce large amounts of data, which bring new opportunities and challenges to storage, analysis and archiving. In the case of extensive use of sensors, there are new problems in using traditional relational database to store data, such as: with the rapid growth of data, the traditional database can no longer meet the storage and management needs of mass data; Sensors usually collect data at intervals of several seconds, and the data collected have great redundancy and so on. Therefore, it has become a research hotspot in Internet of things applications to seek the sensor data storage scheme with high data storage efficiency. The main work of this paper is to design and implement the data storage interface based on MongoDB database by adopting MongoDB automatic slicing technology to meet the requirement of high concurrency and large amount of data storage of sensor data. In the implementation of the interface, the data storage is provided to the user in the form of virtual interface by encapsulating the operation of the MongoDB database and adding the data compression to the data storage procedure. This paper first describes cloud data management technology, including cloud computing technology NoSQL system and NoSQL database technology. Secondly, the current popular NoSQL data storage is studied. Based on the existing data storage technology, the requirements and characteristics of sensor data storage are analyzed. At the same time, based on the MongoDB document database, the WebService interface suitable for sensor data storage is developed. In order to meet the sensor data high concurrency, cross-platform storage and fast query requirements. Finally, the rotary gate compression algorithm, which is widely used in real-time database, is studied. According to the characteristics of sensor data, the compression algorithm and the secondary filtering method are used in the data compression module of WebService interface. The high compression ratio and the removal of sensor data redundancy are achieved.
【学位授予单位】：南京邮电大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP393.09;TP333

【引证文献】