基于Hadoop的安全云存储系统研究与实现

发布时间：2018-05-03 00:38

本文选题：Hadoop + 安全云存储　；参考：《电子科技大学》2013年硕士论文

【摘要】：随着信息时代的数据量以PB级飞速增长，传统存储模式在大数据存储方面的局限性逐渐暴露出来，，单一存储很容易使用户数据丢失，造成巨大损失，越来越多的企业和研究机构认识到必须有一种新的存储机制去改变存储行业现状。云存储技术的引入为存储模式的改变带来了契机，国内外机构和企业在云存储热潮下纷纷推出自己的云存储产品和解决方案，但由于对产品的安全性不够重视，各类安全事故频繁发生，给用户造成了巨大损失。在此背景下，云存储系统的安全性问题逐渐成为业界关注的热点。本文在假设云计算提供商可信的前提下，从应用安全的角度出发，通过大量关键技术的研究，构建了一种基于Hadoop的安全云存储系统。本文首先调研了国内外机构对云存储系统和云存储安全性的研究现状，在此基础上对构建云存储系统涉及到的关键技术进行了深入研究，主要包括HDFS数据读写原理、虚拟磁盘原理、文件系统过滤驱动原理、HTTPS安全传输原理、多线程传输原理。随后结合实际情况，对安全云存储系统从功能和性能两个方面进行了需求分析，并提出了系统的总体架构，然后根据总体架构从安全客户端、安全可靠传输和服务器端三个大的模块出发进行了详细设计，主要包括虚拟磁盘和文件系统过滤驱动的两级加密机制设计、多线程断点续传设计、服务端的存储架构及数据处理模块设计。接下来从代码实现的角度对系统各个模块的实现过程进行了详细阐述，并对某些重要功能进行了效果展示。最后对系统从功能上和性能上进行了测试，经测试构建的系统达到了预定的功能需求，满足响应性、可靠性、可用性三个方面的性能指标。本文设计和实现的基于Hadoop的安全云存储系统在客户端采用虚拟磁盘和文件系统过滤驱动两级加密机制，通过透明加解密的思想保证了用户数据安全；同步传输阶段采用HTTPS安全传输和多线程断点续传技术，保证了用户数据传输的安全性和可靠性；服务器端提供监控机制，通过对存储集群监控、文件同步过程监控、服务器性能监控，提高了系统的可维护性性。
[Abstract]:With the rapid growth of PB level in the information age, the limitations of traditional storage mode in large data storage are gradually exposed. Single storage is easy to use users to lose data and cause huge losses. More and more enterprises and research institutions have realized that there must be a new storage mechanism to change the status of storage industry. The introduction of storage technology has brought an opportunity for the change of storage mode. The domestic and foreign organizations and enterprises have launched their own cloud storage products and solutions in the cloud storage upsurge. However, due to the lack of attention to the security of the products, the frequent occurrence of various security accidents and the huge loss to the users. In this context, the cloud storage system is safe. The problem of full nature has gradually become a hot spot of concern in the industry.
In this paper, a security cloud storage system based on Hadoop is constructed from the perspective of application security and the research of a large number of key technologies. This paper first investigates the status of the research on cloud storage system and cloud storage security at home and abroad, and builds a cloud storage system on the basis of this. The key technologies involved are deeply studied, including the principle of HDFS data reading and writing, the principle of virtual disk, the principle of file system filtering and driving, the principle of HTTPS security transmission and the principle of multithreading transmission. Then, according to the actual situation, the requirements of the security cloud storage system from two aspects of function and ability are analyzed, and the system is put forward. The overall architecture is designed in detail based on the security client, safe and reliable transmission and the three large modules of the server. It mainly includes the design of the two level encryption mechanism of the virtual disk and file system filtering drive, the multi thread breakpoint renewal design, the storage architecture of the service end and the design of the data processing module. From the point of view of code realization, the realization process of each module of the system is elaborated in detail, and some important functions are displayed. Finally, the system is tested in function and performance, and the system has reached the desired functional requirements, full of responsiveness, reliability and availability in three aspects. It can be an indicator.
The security cloud storage system based on Hadoop is designed and implemented in this paper, which uses virtual disk and file system filtering to drive two level encryption mechanism, and ensures the user data security through the transparent encryption and decryption idea. The synchronization transmission phase adopts HTTPS secure transmission and multi thread break point renewal technology to ensure the user data transmission. Security and reliability; the server side provides monitoring mechanism, by monitoring the storage cluster, monitoring the process of file synchronization, monitoring the performance of the server, and improving the maintainability of the system.

【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333

【参考文献】