Hadoop平台下的海量数据存储技术研究

发布时间：2018-02-21 18:18

本文关键词： Hadoop 分布式文件系统海量数据存储　出处：《东北石油大学》2012年硕士论文　论文类型：学位论文

【摘要】：随着互联网的发展、互联网用户的增加，，互联网中的图片数据也急剧膨胀。企业的图片数据规模可达TB到几百TB。一般情况下使用分布式文件系统来处理这些图片。 Hadoop是近年来出现的处理海量数据的分布式系统。具有可靠性好、存储容量大、部署简单、可维护性更好等特点。基于以上两点，本文对Hadoop在图片存储方面的性能做了相应的研究，设计了针对中小企业的基于Hadoop平台的图片存储系统。具体研究内容如下： 1．Hadoop平台的工作原理从数据组织、数据流等几个方面比较深入地研究了Hadoop平台中数据的存放、读写原理。还研究了分布式计算框架MapReduce的工作过程。 2．基于Hadoop平台的图片存储系统的设计根据需求分析，将系统的功能设计为普通用户模块、管理员模块、日志分析模块、客户端和系统监控等模块。结合Hadoop、Tomcat、Mysql等软件完成了系统的架构设计。根据系统的功能设计了UML类图，最后设计了Mysql数据库的表。 3．图片存储系统的实现首先搭建了Hadoop集群，接下来完成了系统各部分的实现过程。重点实现了对Hadoop的操作和日志分析部分。 4．Hadoop平台和Web的整合本人在研究Hadoop和JSP的基础上，对二者进行了整合，即可以通过JSP页面来访问Hadoop存储的图片文件。最后综合本文的研究，提出相应的测试方法，来验证系统的性能和可靠性。
[Abstract]:With the development of the Internet and the increase of the Internet users, the picture data in the Internet is expanding rapidly. The scale of the picture data of the enterprise can reach to several hundred TBs. In general, distributed file system is used to process these images. Hadoop is a distributed system for processing massive data in recent years, which has the characteristics of good reliability, large storage capacity, simple deployment and better maintainability. Based on the above two points, this paper makes a corresponding study on the performance of Hadoop in picture storage, and designs a picture storage system based on Hadoop platform for small and medium-sized enterprises. The specific research contents are as follows:. 1. Working principle of Hadoop platform. The principle of data storage, reading and writing in Hadoop platform is studied in detail from the aspects of data organization and data flow, and the working process of distributed computing framework MapReduce is also studied. 2. Design of picture storage system based on Hadoop platform. According to the requirement analysis, the function of the system is designed as common user module, administrator module, log analysis module, The architecture of the system is designed with the software of Hadoop Tomcat and MySQL. The UML class diagram is designed according to the function of the system. Finally, the table of Mysql database is designed. 3. The realization of picture storage system. First, the Hadoop cluster is built, and then the implementation process of each part of the system is completed, and the operation and log analysis of Hadoop are mainly realized. 4. Integration of Hadoop platform and Web. On the basis of studying Hadoop and JSP, I integrate them, that is to say, we can access the image files stored by Hadoop through JSP pages. Finally, based on the research of this paper, a corresponding test method is proposed to verify the performance and reliability of the system.
【学位授予单位】：东北石油大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP333

【参考文献】