数据迁移云服务的设计与实现

发布时间：2018-01-02 06:35

本文关键词：数据迁移云服务的设计与实现　出处：《浙江大学》2017年硕士论文　论文类型：学位论文

【摘要】：大数据时代的到来,传统的数据存储和处理手段已经难以满足日益增长的需求,越来越多的数据需要迁移到hadoop计算平台进行存储和处理。数据迁移作为数据科学领域的重要研究方向和技术,也受到学术界、工业界更多研究人员的关注、研究。已有的数据迁移工具往往具有着单机性能低下、安装配置繁琐、不支持流式数据迁移等缺点。本文针对现有工具的缺点,结合已有研究成果,设计出了针对hadoop集群的数据迁移云服务。本文主要贡献如下:(1)设计并优化了基于数据库日志的流式数据提取、迁移技术。通过对数据库日志进行解析,提取增量数据,并将这些数据直接封装为消息发往hadoop集群。大大降低流式数据提取的I0、网络等开销。(2)将因子分析数学思想应用于负载均衡负载状态评估,将响应时间纳入负载均衡参数指标。该算法相对于传统的负载均衡算法,能够更有效地评估节点当前负载情况,更大地利用好集群资源。大大提高了数据迁移系统的吞吐量和集群计算能力。(3)将数据迁移系统上升到云计算的高度。针对业内已有迁移工具配置复杂、单机性能低下、容错性差等问题,本文提出的数据迁移云服务设计能够更好的提升系统整体迁移能力和吞吐。同时对于迁移任务具有一定的故障可恢复性。
[Abstract]:With the arrival of big data era, the traditional means of data storage and processing have been difficult to meet the increasing demand. More and more data need to be migrated to the hadoop computing platform for storage and processing. Data migration as an important research direction and technology in the field of data science is also by the academic community. Industry more researchers concern, research. The existing data migration tools often have the disadvantages of low performance, cumbersome installation and configuration, do not support streaming data migration and other shortcomings. This paper aims at the shortcomings of the existing tools. Combined with the existing research results, a data migration cloud service for hadoop cluster is designed. The main contributions of this paper are as follows: 1) the streaming data extraction based on database log is designed and optimized. Migration technology. Through the database log parsing, extract incremental data, and directly encapsulate these data as messages to the hadoop cluster, greatly reducing the I0 stream data extraction. This paper applies the mathematical idea of factor analysis to load balancing state evaluation, and takes response time into load balancing parameter index. The algorithm is compared with the traditional load balancing algorithm. Can more effectively evaluate the current load of the node. Better use of cluster resources. Greatly improve the throughput of data migration systems and cluster computing capabilities.) data migration system to the height of cloud computing. For the industry migration tools configuration is complex. The data migration cloud service design proposed in this paper can improve the overall migration capability and throughput of the system, and has a certain fault recoverability for migration tasks.
【学位授予单位】：浙江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.13;TP393.09

【相似文献】