面向天文学领域的大数据管理的研究与实现

发布时间：2017-12-31 04:07

本文关键词：面向天文学领域的大数据管理的研究与实现　出处：《哈尔滨工业大学》2013年硕士论文　论文类型：学位论文

【摘要】：近年来，随着云计算，云存储等概念的提出，在人们的日常生活及各方面的专业领域中会产生大量的数据。在天文学领域，近年来提出一个新的项目叫做大型综合巡天望远镜，旨在持续观测整个天空数十年，将相关数据记录下来以供分析研究。在项目持续的十年中，大约会将项目产生的60PB数据存储到服务器中供大众以及天文学家分析以揭示新的天空，对整个宇宙做出新的发现。相应地，60千万亿字节相对于现存的各种大型数据库管理系统来说是一项巨大的挑战。传统的主流数据库管理软件如Microsoft SQL Server，MySQL等已不适用于大数据处理业务，因此，出现了一些新的技术以提高对大数据的处理。一种可行的方案是将数个成本脚较低的计算机组成集群以提供并行服务如Oracle的Real Application Cluster，，IBM DB2，Hive，HadoopDB等，这种手段虽然具有相对较低的成本，但是却增加了运行环境的复杂程度，以及用于并行的额外开销。在分析此项目对于数据库管理系统的需求，以及学习，理解甲骨文数据库管理系统中提供的索引，分区，并行查询等技术后，本文将会对甲骨文真正应用程序集群的数据库性能进行一番优化，测试以观测现存的甲骨文真正应用程序集群是否能满足项目的需求。另外，本文还将对比查询语句在甲骨文真正应用程序集群以及在相同情况下其他数据库管理系统中的表现。本文还将就甲骨文真正应用程序集群的可扩展性进行一番测试，以观测甲骨文真正应用程序集群是否能很好的支持添加，扩展集群，以及其容错性，可伸缩性，可用性。相应地，为了更好的测试查询语句在甲骨文数据库管理系统中的性能，一个辅助的软件将会被开发出来以便简化测试查询性能的流程，减小工作量。在实际应用当中，本文会提供结合索引，分区，并行等技术的策略来对现存的数据库进行优化。而且，作为大型巡天望远镜项目的一部分，此部分优化策略对大型巡天望远镜现存的数据库管理系统来说也会有一定的意义。辅助测试数据库性能的软件也将可以被本项目组测试其他数据库系统性能的工程师直接使用。
[Abstract]:In recent years, with the introduction of cloud computing, cloud storage and other concepts, in the daily life of people and in all aspects of the professional field will produce a lot of data, in the field of astronomy. In recent years, a new project, called the large Integrated Sky Survey Telescope, has been proposed to continuously observe the entire sky for tens of years and record the relevant data for analysis and research. Approximately 60 PB of data generated by the project will be stored in servers for public and astronomer analysis to reveal new skies and make new discoveries across the universe. 60 trillion bytes is a huge challenge compared to existing large database management systems. The traditional mainstream database management software such as Microsoft SQL Server MySQL is no longer suitable for big data processing business. New technologies have emerged to improve the handling of big data. A viable solution would be to cluster several lower-cost computers to provide parallel services such as Oracle's Real. Application Cluster. Although IBM DB2Hiveer Hadoop DB and so on, this method has relatively low cost, but it increases the complexity of the running environment. After analyzing the requirements of the project for the database management system, and learning, understanding the Oracle database management system provided in the index, partition, parallel query and other technologies. This article will optimize the database performance of Oracle's real application cluster to test whether the existing Oracle real application cluster can meet the requirements of the project. This article will also compare the performance of query statements in Oracle real application cluster and other database management systems in the same situation. We will also test the extensibility of Oracle real application cluster. Try. To see if Oracle's real application cluster supports addition, extension, and fault tolerance, scalability, availability, and so on. In order to better test the performance of query statements in Oracle database management system, an auxiliary software will be developed to simplify the process of testing query performance and reduce the workload. In practical applications, this article will provide a combination of indexing, partitioning, parallelism and other techniques to optimize existing databases, and as part of a large survey telescope project. This part of the optimization strategy will also have some significance for the existing database management system of the large survey telescope. The software to assist in testing database performance will also be able to be used by the project team to test the performance of other database systems. The teacher uses it directly.
【学位授予单位】：哈尔滨工业大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：P11;TP311.52

【相似文献】