基于Hadoop平台的云计算构建与日志分析

发布时间：2018-05-27 01:21

本文选题：云计算 + Hadoop　；参考：《哈尔滨理工大学》2012年硕士论文

【摘要】：云计算是一种新型计算模型，它将计算任务分布在大量计算机构成的资源池里，使用户能够按需获取计算能力、存储空间和信息服务。与传统的数据处理模式相比，云计算技术可以有效解决大规模数据处理面临的性能瓶颈问题，提高了数据处理的可靠性及可扩展性，提高了数据处理能力的同时降低了计算对硬件设备的要求。本文对云计算概念、种类、关键技术等进行了重点研究。 Hadoop是一个开源分布式计算平台，它专为处理大规模数据和分布式计算而设计，是实现云计算的主要可选方式之一。Hadoop平台具有高效、可靠、扩展性强等特点，它的两个主要组成部分是Hadoop分布式文件系统HDFS和并行处理模型MapReduce。本文对HDFS的几个方面：设计前提与目标、体系结构、保障可靠性措施和提升性能措施以及MapReduce的几个方面：逻辑模型、编程模型、实现机制以及执行流程的细致分析和研究。在分析了原有海量数据处理系统之后，结合云计算技术和Hadoop的优势，建立了一个新的数据处理模型，依据该模型搭建了系统平台，并使用Web日志作为源数据进行了平台上的性能分析。通过实验对比，总结了利用云计算技术，使得日志分析过程在消耗时间上大大缩短，，并且随着数据量的增大，Hadoop平台的处理能力和数据存储能力也在适应着数据量的变化，恰恰体现了云计算技术在处理大规模数据时计算能力、存储空间等按需提高的优势。基于Hadoop平台的云计算环境在处理大规模数据方面解决了传统数据处理方法计算能力与存储能力的性能瓶颈问题，并且良好的可扩展性使得这种能力可以灵活的使用。
[Abstract]:Cloud computing is a new computing model, which distributes computing tasks in a large number of computer resource pools, enabling users to acquire computing power, storage space and information services on demand. Compared with the traditional data processing mode, cloud computing technology can effectively solve the performance bottleneck of large-scale data processing, and improve the reliability and scalability of data processing. The ability of data processing is improved and the requirement of computing hardware is reduced. This paper focuses on cloud computing concepts, categories, key technologies and so on. Hadoop is an open source distributed computing platform, which is specially designed to deal with large-scale data and distributed computing. It is one of the main options to implement cloud computing. Hadoop platform has the characteristics of high efficiency, reliability and expansibility. Its two main components are Hadoop distributed file system HDFS and parallel processing model MapReduce. This paper analyzes and studies several aspects of HDFS: design premise and goal, architecture, measures to guarantee reliability and improve performance, and several aspects of MapReduce: logical model, programming model, implementation mechanism and execution flow. After analyzing the original massive data processing system, combining the advantages of cloud computing technology and Hadoop, a new data processing model is established, and the system platform is built according to the model. Web log is used as the source data to analyze the performance of the platform. Through the comparison of experiments, this paper summarizes the use of cloud computing technology, which makes the log analysis process greatly shorten the consumption time, and with the increase of the amount of data, the processing capacity and data storage capacity of Hadoop platform are also adapted to the change of data volume. Cloud computing technology in processing large-scale data computing power, storage space and other advantages on demand. The cloud computing environment based on Hadoop platform solves the performance bottleneck problem of traditional data processing method computing ability and storage ability in dealing with large-scale data, and the good scalability makes this ability can be used flexibly.
【学位授予单位】：哈尔滨理工大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP3

【参考文献】