论文分享系统中的海量图片存储研究

发布时间：2018-05-21 03:18

本文选题：分布式系统 + 可扩展性　；参考：《华中科技大学》2012年硕士论文

【摘要】：在当今互联网环境下，图片已经成为大多网站展示内容的一种主要手段，用户也常通过照片来分享自己的生活和心情。而随着互联网的高速发展，，传统的图片存储服务正面临着数据规模高速增长和数据访问高并发这两方面的挑战。这就需要图片存储系统具有良好的可扩展性和高性能。此外，如今的互联网应用越来越多样化，很难有一种通用的办法适用于所有的系统，我们需要针对系统的应用特征来设计相应的解决方案。以研究各大互联网公司的图片存储系统为基础，结合论文分享社会化网络的应用特点，提出一种日志型的基于哈希树和键值对（key-value）接口的存储模型，同时借鉴已有的解决系统可扩展性和可靠性的分布式架构设计理念，对传统的图片存储系统进行改进。使用日志型的key-value模型，将图片数据只增不减的写入大文件中，减少了文件数量，简化了操作系统的管理，同时使用哈希树结构，组织图片在大文件中的索引信息，实现图片的快速读取；采用分布式的架构设计，实现系统的可扩展性和可靠性；结合论文分享的特点，利用图片之间的相关性，进一步提高读取性能；采用了epoll网络模型和领导者-跟随者（leader-follower）线程模型，有效地提升了系统处理高并发请求的能力。然后根据提出的解决方案实现了一个针对论文分享的图片存储系统，并详细介绍了系统的架构、核心模块以及相应实现。最后对论文分享系统进行了性能测试，分别对采用日志型的key-value模型、采用哈希树组织索引信息、结合论文图片相关性以及使用epoll和leader-follower模型等方案进行了相应的测试，并与没有采用这些方案的系统进行对比，测试结果表明新系统的性能有了很大提升，验证了设计的针对论文分享的图片存储系统具有良好的性能。
[Abstract]:In today ' s Internet environment , images have become a major means of presenting content in most websites , and users often share their lives and mood through photos . With the rapid development of the Internet , traditional image storage services are facing the challenges of high data size and high data access . This requires a picture storage system with good scalability and high performance . In addition , today ' s Internet applications are becoming more diverse , and it is difficult to have a general approach to all systems , and we need to design the corresponding solutions for the application features of the system .

This paper presents a kind of log - based storage model based on hash tree and key - value interface based on the research of the image storage system of each big internet company , and proposes a kind of log - based storage model based on hash tree and key - value interface .
the distributed architecture design is adopted to realize the expandability and the reliability of the system ;
According to the characteristics of the sharing of the paper , the correlation between the pictures is utilized to further improve the reading performance ;
The epoll network model and leader - follower thread model are adopted to effectively improve the ability of the system to process high concurrency requests . Then a picture storage system for the paper sharing is realized according to the proposed solution , and the architecture , core module and corresponding implementation of the system are introduced in detail .

Finally , the performance test of the paper - sharing system is carried out , the key - value model of the log type is adopted , the index information of the hash tree is adopted , the correlation of the paper pictures is combined and compared with the system without the schemes , the test results show that the performance of the new system is greatly improved , and the design is verified to have good performance for the picture storage system shared by the paper .
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】