大数据平台下数据缓存性能优化

发布时间：2019-03-31 00:02

【摘要】：借助互联网的发展,数据以爆炸性的方式增长。社交网络、物联网和车载网络的流行,使非结构化、半结构化数据在数据中心中所占的比重越来越大。大数据技术应运而生,发展迅速,深度影响着互联网产业的格局,是又一次信息革命的浪潮。大数据技术中具有典型代表是Hadoop开源框架,各种公司、开源团队、研究机构对于Hadoop的扩展研究非常活跃,推出了很多支持批处理、流处理、实时处理的大数据平台。本论文阐述了大数据实时查询平台的设计原理,重点描述了基于Mem cached的内存优化方案。针对Memcached存在的热点问题和单点失效问题,进行系统分析,并且提出基于热点的动态调整策略。本论文的研究成果具有重要的理论价值和实践价值。从理论上看,由热点引起的负载不均衡问题普遍存在于分布式系统中,例如在云计算系统中需要考虑各个节点的CPU和内存的利用是否均衡,论文提出的基于热点的动态调整策略对于分布式系统的优化具有学术参考价值。从实践上看,该策略的实现成功加速了我们设计的大数据平台的查询处理速度,证明了该策略对于构建高效的大数据平台具有实践参考价值。
[Abstract]:With the development of the Internet, the data grows explosively. The popularity of social networks, the Internet of things and in-vehicle networks makes unstructured, semi-structured data more and more important in data centers. Big data technology emerges as the times require, develops rapidly, deeply affects the pattern of Internet industry, is another tide of information revolution. The typical representative of big data technology is Hadoop open source framework, a variety of companies, open source teams, research institutions are very active in the expansion of Hadoop research, launched a lot of support batch processing, stream processing, real-time processing big data platform. This paper describes the design principle of big data real-time query platform, especially describes the memory optimization scheme based on Mem cached. Aiming at the hot spot problem and single point failure problem of Memcached, the system analysis is carried out, and the dynamic adjustment strategy based on hot spot is put forward. The research results of this paper have important theoretical and practical values. Theoretically, the problem of load imbalance caused by hot spots is widespread in distributed systems. For example, in cloud computing systems, it is necessary to consider whether the CPU and memory utilization of each node is balanced. The hot-spot-based dynamic adjustment strategy proposed in this paper is of academic reference value for the optimization of distributed systems. In practice, the implementation of this strategy has successfully accelerated the query processing speed of big data platform, and proved that the strategy has practical reference value for building an efficient big data platform.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】