Antnest:一种支持多计算模型的分布式计算系统

发布时间：2018-04-06 20:39

本文选题：大数据　切入点：分布式计算　出处：《华中科技大学》2012年硕士论文

【摘要】：随着计算机和互联网的发展，信息量的增长已经达到了一个前所未有的速度，海量数据的日益累积，使人们迎来了大数据‖的时代。如何正确且高效地从这些数据中分析出有用信息并做出处理成为了一项极具挑战性的工作。面对这一问题，人们往往考虑采用分布式计算进行处理。然而对于大多数信息企业来说，往往存在着许多种不同需求的业务，这些业务所适用的分布式计算平台不尽相同，这就导致了企业分布式计算平台种类的繁多，学习和维护这些平台往往需要很高的成本，这给新型互联网企业带来了巨大的经济压力。因此，设计一个支持多计算模型的分布式计算系统成为了各企业的迫切需求。 Antnest将企业的各种基本业务需求抽象成三种不同的计算模型，，通过对比这些计算模型的特点，对它们的可等价性进行了分析，并提出了一种统一计算模型的方案，主要包括数据源的统一以及计算处理的统一。Antnest综合了批处理分布式计算、流式分布式计算、任务自动调度框架等三种平台的特点，对企业中最基本的三种业务——大规模静态数据处理业务、线上实时业务、后台定时业务提供了很好的支持，从而减少了企业因使用和维护多种平台所造成的开销。Antnest为用户提供了一个统一的任务创建模式，用户根据自己的实际业务需求采用相同的规则定义数据源及计算单元配置文件，并在对应的任务接口中实现计算单元的处理逻辑，接着便能将任务上传到该平台中执行。Antnest屏蔽了不同任务计算模型之间的差异。另外，底层的一些实现机制，如消息通信、心跳检测、容错、任务的划分及调度等，对于用户而言是完全透明的。经测试表明，Antnest能很好的支持不同计算模型任务，正确执行任务的各种基本操作，同时还对不同的数据库提供了支持。
[Abstract]:With the development of computer and Internet, the increase of information has reached an unprecedented speed, and the accumulation of massive data makes people usher in the era of big data.How to correctly and efficiently analyze and process useful information from these data has become a challenging task.Faced with this problem, people often consider using distributed computing for processing.However, for most information enterprises, there are many kinds of business with different requirements, and the distributed computing platforms for these services are not the same, which leads to a great variety of distributed computing platforms.Learning and maintaining these platforms often require high cost, which brings great economic pressure to new Internet enterprises.Therefore, the design of a distributed computing system supporting multiple computing models has become an urgent need of enterprises.Antnest abstracts various basic business requirements into three different computing models. By comparing the characteristics of these models, the equivalence of these models is analyzed, and a unified computing model is proposed.It mainly includes the unification of data sources and the unification of computing processing. Antnest synthesizes the characteristics of three kinds of platforms, such as batch distributed computing, flow distributed computing, task automatic scheduling framework, etc.It provides very good support for the most basic three kinds of business in the enterprise-large-scale static data processing service, on-line real-time service, backstage timing service, etc.This reduces the cost of using and maintaining multiple platforms in enterprises. Antnest provides a unified task creation model for users. Users define data sources and cell configuration files according to their actual business requirements with the same rules.The processing logic of the computing unit is implemented in the corresponding task interface, and then the task can be uploaded to the platform to execute .Antnest to shield the differences between different task computing models.In addition, the underlying implementation mechanisms, such as message communication, heartbeat detection, fault tolerance, task partitioning and scheduling, are completely transparent to users.The test results show that Antnest can support different computing model tasks, perform all kinds of basic operations correctly, and provide support for different databases.
【学位授予单位】：华中科技大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP338.8

【参考文献】