基于蓝光存储的异构云存储平台设计与研究

发布时间：2018-10-07 19:07

【摘要】：随着互联网在生活中的广泛应用,网络中的数据量呈指数级增长,IT行业已经从计算时代进入到存储时代。目前,许多数据中心都面临着系统能耗开销巨大、存储资源利用率较低以及存储成本不断上升等问题。有数据表明,存储的各种开支已经占数据中心总成本的30%-70%。而日常产生的数据中,经常使用的数据(热数据)只占到整体数据量的20%而极少使用或几乎不用的数据(冷数据)高达80%。在这种情况下,根据数据冷热特征进行分级存储,发挥各种存储设备优点,降低总存储成本,有十分广阔的研究和应用前景。对比目前存储介质技术的发展,光盘的单位容量价格远远低于磁盘,而其读写速度又优于磁带。因此,将光盘作为分级存储系统中的近线或离线存储设备具有现实意义。本文在分布式文件系统基础上,引入蓝光光盘库作为近线存储设备,建立一个“在线——近线”分级存储的云存储平台。在平台设计和实现的过程中主要工作包括如下内容:(1)对当前的数据区分技术进行研究与分析,在此基础上,通过对用户行为日志的分析,提取用户行为关键特征,建立数据价值模型。该模型采用指数平滑法充分考虑各个时刻的数据价值,并根据数据价值的高低区分冷热数据集,该模型能有效地提高云平台中用户访问数据时命中在线存储设备的概率。(2)研究数据迁移过程控制策略,实现数据在磁盘阵列和蓝光存储两种不同存储层次的设备之前自动的、不间断的迁移。重点分析基于蓝光光盘库的数据迁移过程,有效地保证数据在迁移过程中的一致性。(3)采用元数据服务器与应用服务器分离,前台页面与后台服务分离的分模块方式,实现平台各个功能,充分体现低耦合的软件设计思想。并且进行一系列测试,测试结果表明平台实现的数据迁移具有有效性和准确性,能够实现数据在蓝光光盘库与磁盘之间高效传输,保证数据的一致性,释放磁盘空间,并提高数据访问的速度。
[Abstract]:With the wide application of the Internet in daily life, the amount of data in the network has increased exponentially. The IT industry has entered the storage era from the computing era. At present, many data centers are faced with the problems of huge system energy consumption, low utilization of storage resources and rising storage costs. Data show that storage costs account for 30-70% of the total cost of the data center. In the daily data generation, the frequently used data (hot data) accounts for only 20% of the total amount of data, and the data (cold data) that is rarely used or rarely used (cold data) is as high as 80%. In this case, there is a broad research and application prospect for hierarchical storage based on the characteristics of data cooling and heating, to give full play to the advantages of various storage devices and to reduce the total storage cost. Compared with the current development of storage media technology, the unit capacity of CD-ROMs is much lower than that of disks, and the speed of reading and writing is better than that of magnetic tape. Therefore, it is of practical significance to use optical disk as near-line or off-line storage device in hierarchical storage system. Based on the distributed file system, this paper introduces the Blu-ray optical disk library as the near-line storage device, and establishes a "on-line -near-line" hierarchical storage cloud storage platform. In the process of platform design and implementation, the main contents are as follows: (1) Research and analysis of current data differentiation technology, on the basis of which, through the analysis of user behavior log, the key features of user behavior are extracted. Establish data value model. In this model, the exponential smoothing method is used to fully consider the data value at each time, and the cold and hot data sets are distinguished according to the value of the data. This model can effectively improve the probability of users hitting online storage devices when accessing data in cloud platform. (2) Research data migration process control strategy, realize data before disk array and Blu-ray storage two different storage levels of devices automatically, An uninterrupted migration. The data migration process based on Blu-ray optical disk library is analyzed in order to ensure the consistency of data migration effectively. (3) the metadata server is separated from the application server, and the foreground page is separated from the background service. Realize each function of the platform, fully embody the low coupling software design idea. And a series of tests are carried out. The test results show that the data migration implemented by the platform is effective and accurate. It can efficiently transfer the data between the Blu-ray disc library and the disk, ensure the consistency of the data, and release the disk space. And improve the speed of data access.
【学位授予单位】：电子科技大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP333

【参考文献】