基于Hadoop的大数据动态资源调节服务研究
发布时间：2018-07-18 08:20
【摘要】：随着技术革新，从传统互联网广泛应用到最近几年爆炸式增长的移动互联网及物联网的起步，依附在网络上的数据越来越庞大，据国际数据公司（IDC），EMC公司等最新研究表明，在移动网络设备（智能手机）及视频监控的推动下，目前全球的数据总量已经到达4870亿GB，而在2007年的相关报告表示当年的数据总量才1610亿GBN。这些数据中包括了大量的电话、邮件、照片、网络社交、新闻以及视频内容。如何有效利用这些数据，给用户提供优质的用户体验，在科学研究上，随着大量的带有GPS的数据设备采集的数据的汇集和研究都是急切需要得到技术支持。 在今后，随着物联网的进一步发展，有大量服务是基于位置的服务(Location Based Service，LBS)的数据产生，也将有大量请求服务基于LBS或个人偏爱。这也致使未来需要提供的服务应该同现在统一服务相区分，针对不同的用户属性提供不同的服务资源，在浩若烟海的数据中如何最快、最正确地提供能满足用户需求、并且是在低成本状态下完成变得尤为重要。本文的主要工作如下： 1.对现有的大数据存储进行分析研究，包括了GFS文件系统实现原理及Hadoop框架，为在大数据服务中的数据存储做技术支撑。 2.针对数据特性进行研究，并研究相关数据挖掘算法对非结构化数据进行归纳挖掘，同时针对结构化数据进行分析读取，实现数据热点权值的初始化。本文的主要成果如下： 1.在现有的Hadoop框架基础上，，二次开发，修改Hadoop的存储备份算法，通过资源权值列表来实现资源文件在修改后的框架中按照数据资源热点权值存储。 2.本文在Hadoop基础上设计出的资源文件权值初始化算法，当资源文件访问量增加或通过外部导入设置热点权值，实现对数据资源文件的存储节点计算并分发。 3.实现了在库资源文件的热点不断变化过程中的资源文件权值调整，并通该权值来实现数据资源文件的重新分发及调整，实现了服务资源的扩增算法和服务资源收缩算法。
[Abstract]:With the technological innovation, from the traditional Internet of Internet to the explosive growth of the mobile Internet in recent years and the start of the Internet of things, the data attached to the network is becoming larger and larger. According to the latest research, such as International data Corporation (IDC) and EMC, Driven by mobile network devices (smartphones) and video surveillance, the global data total has reached 487 billion GB, compared with 161 billion GBN reported in 2007. This data includes a lot of phone, email, photos, social networking, news and video content. How to effectively use these data to provide users with a high-quality user experience, in scientific research, with a large number of data with GPS data collection and research are urgently needed technical support. In the future, with the further development of the Internet of things, there will be a large number of services based on location based LBS (location based Service LBS) data generation, there will be a large number of request services based on LBS or personal preference. This also leads to the need for future services to be differentiated from the current unified services, to provide different service resources for different user attributes, and how to provide the fastest and most correctly available services to meet the needs of users in the vast sea of data. And it is particularly important to do so in a low-cost state. The main work of this paper is as follows: 1. This paper analyzes and studies the existing big data storage, including the implementation principle of big data file system and the framework of Hadoop, which provides technical support for data storage in big data services. 2. Based on the research of data characteristics, the related data mining algorithms are studied to induce and mine the unstructured data. At the same time, the structured data is analyzed and read to realize the initialization of the hot data weights. The main results of this paper are as follows: 1. Based on the existing Hadoop framework, the storage backup algorithm of Hadoop is redeveloped, and the resource file is stored in the modified framework according to the hot spot weight value of the data resource by the list of resource weights. 2. A resource file weight initialization algorithm based on Hadoop is designed in this paper. When the access of resource file is increased or the hot spot weight is set by external import, the storage node of data resource file can be calculated and distributed. This paper realizes the adjustment of resource file weight value in the process of changing hot spot of database resource file, and realizes the redistribution and adjustment of data resource file by using this weight value, and realizes the expansion algorithm of service resource and the algorithm of service resource contraction.
【学位授予单位】：成都理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333
本文编号：2131312
[Abstract]:With the technological innovation, from the traditional Internet of Internet to the explosive growth of the mobile Internet in recent years and the start of the Internet of things, the data attached to the network is becoming larger and larger. According to the latest research, such as International data Corporation (IDC) and EMC, Driven by mobile network devices (smartphones) and video surveillance, the global data total has reached 487 billion GB, compared with 161 billion GBN reported in 2007. This data includes a lot of phone, email, photos, social networking, news and video content. How to effectively use these data to provide users with a high-quality user experience, in scientific research, with a large number of data with GPS data collection and research are urgently needed technical support. In the future, with the further development of the Internet of things, there will be a large number of services based on location based LBS (location based Service LBS) data generation, there will be a large number of request services based on LBS or personal preference. This also leads to the need for future services to be differentiated from the current unified services, to provide different service resources for different user attributes, and how to provide the fastest and most correctly available services to meet the needs of users in the vast sea of data. And it is particularly important to do so in a low-cost state. The main work of this paper is as follows: 1. This paper analyzes and studies the existing big data storage, including the implementation principle of big data file system and the framework of Hadoop, which provides technical support for data storage in big data services. 2. Based on the research of data characteristics, the related data mining algorithms are studied to induce and mine the unstructured data. At the same time, the structured data is analyzed and read to realize the initialization of the hot data weights. The main results of this paper are as follows: 1. Based on the existing Hadoop framework, the storage backup algorithm of Hadoop is redeveloped, and the resource file is stored in the modified framework according to the hot spot weight value of the data resource by the list of resource weights. 2. A resource file weight initialization algorithm based on Hadoop is designed in this paper. When the access of resource file is increased or the hot spot weight is set by external import, the storage node of data resource file can be calculated and distributed. This paper realizes the adjustment of resource file weight value in the process of changing hot spot of database resource file, and realizes the redistribution and adjustment of data resource file by using this weight value, and realizes the expansion algorithm of service resource and the algorithm of service resource contraction.
【学位授予单位】：成都理工大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP333
【参考文献】
相关期刊论文 前3条
1 林鸿飞,马雅彬;基于聚类的文本过滤模型[J];大连理工大学学报;2002年02期
2 刘越;;云计算综述与移动云计算的应用研究[J];信息通信技术;2010年02期
3 孙广中;肖锋;熊曦;;MapReduce模型的调度及容错机制研究[J];微电子学与计算机;2007年09期
相关硕士学位论文 前4条
1 霍树民;基于Hadoop的海量影像数据管理关键技术研究[D];国防科学技术大学;2010年
2 曹风兵;基于Hadoop的云计算模型研究与应用[D];重庆大学;2011年
3 刘丛山;基于Hadoop的文本分类研究[D];上海交通大学;2012年
4 邱荣太;基于Hadoop平台的Map-Reduce应用研究[D];河南理工大学;2009年
本文编号：2131312
本文链接：https://www.wllwen.com/kejilunwen/jisuanjikexuelunwen/2131312.html
