基于云计算技术的化合物相似性分析系统

发布时间：2018-11-24 07:22

【摘要】：随着整个社会各行各业信息化进程的发展,信息数据呈现了爆炸式地增长,对相关的信息存储、处理和管理技术也提出了更高的要求。而云计算通过网络将超大规模的计算和存储资源整合成计算资源池并以按需服务的方式提供给用户,有效地满足了海量信息存储和处理的需求。在药物设计领域,利用计算机虚拟筛选发现具有生物活性的化合物是一种重要的研究方法,其中一种筛选方法是将功能未知的分子与功能已知的分子进行结构比较,根据它们的结构相似性特征来预测该未知分子的生物活性,它大量应用于新药研发和新型化学物发现过程中。随着化合物数量的增加,药物筛选计算需求急剧增长,这就对计算系统处理技术提出了更高的要求,传统的单台机器处理方法具有很大的极限性,而云计算技术的特点为这个问题提供了潜在的解方案,在这种情况下,如何将云计算技术应用到计算机虚拟筛选中成为值得研究的课题。本文研究了云计算的相关理论、特点和关键技术,探索了分布式数据处理编程模型MapReduce以及其开源实现Hadoop的运行机制和原理,针对分子结构比较问题提出了一个分布式解决方案。主要做了如下研究： 1)研究了云计算相关技术和分子相似性比较算法,结合Hadoop云计算技术的优点,针对Hadoop无法直接应用于分子相似性比较问题,提出了基于索引文件的处理方法,将Hadoop云计算技术应用到分子相似性比较领域上。 2)通过实验验证了该方法的可行性。实验结果表明该系统在拥有16个CPU并行处理的情况下,本方案平均比单机处理系统快14.4倍,有效地满足了分子相似性比较系统海量计算的需求。 3)针对Hadoop的Partition算法可能存在的数据倾斜的问题,提出了一个均匀的轮询分区算法,减少了数据倾斜带来的影响,实验证明该算法可以使整个系统的效率提高5%以上。 4) MapReduce框架具备容错功能,也就是当某个计算节点出现故障时,主节点会自动把该节点计算任务分配到别的空闲机器上,但是该计算节点的数据块会被从新处理。我们对此进行了改进,使得该数据块从新被处理时从上一次中断的地方开始,避免了大量的重复计算。
[Abstract]:With the development of information technology in all kinds of industries in the whole society, the information data has been increasing explosively, and the related information storage, processing and management technology have been put forward higher requirements. Cloud computing integrates large-scale computing and storage resources into a computing resource pool through the network and provides users with on-demand services, which effectively meet the needs of massive information storage and processing. In the field of drug design, the use of computer virtual screening to find compounds with biological activity is an important research method, one of which is to compare the structure of unknown molecules with those with known functions. The bioactivity of the unknown molecule is predicted according to their structural similarity. It is widely used in the research and development of new drugs and the discovery of new chemicals. With the increase of the number of compounds, the demand for drug screening and calculation is increasing rapidly, which puts forward higher requirements for computing system processing technology. The traditional single machine processing method has a great limit. The characteristics of cloud computing technology provide a potential solution to this problem. In this case, how to apply cloud computing technology to computer virtual filtering has become a topic worth studying. This paper studies the theory, characteristics and key technologies of cloud computing, explores the distributed data processing programming model (MapReduce) and its open source implementation mechanism and principle of Hadoop, and puts forward a distributed solution to the problem of molecular structure comparison. The main contributions are as follows: 1) the related technologies and molecular similarity comparison algorithms of cloud computing are studied. Combined with the advantages of Hadoop cloud computing technology, Hadoop can not be directly applied to the problem of molecular similarity comparison. An index file based processing method is proposed, and Hadoop cloud computing technology is applied to the field of molecular similarity comparison. 2) the feasibility of the method is verified by experiments. The experimental results show that the proposed scheme is 14.4 times faster than the single processor system under the condition of 16 CPU parallel processing, which can effectively meet the requirement of mass computing in the molecular similarity comparison system. 3) aiming at the possible problem of data skew in Hadoop's Partition algorithm, a uniform polling partition algorithm is proposed, which reduces the effect of data skew. The experiment shows that the algorithm can improve the efficiency of the whole system by more than 5%. 4) the MapReduce framework has fault-tolerant function, that is, when a computing node fails, the master node will automatically assign the node computing tasks to other idle machines, but the data block of the computing node will be reprocessed. We have improved this so that the data block starts from the last time it was processed and avoids a lot of double computations.
【学位授予单位】：复旦大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP3;TQ460

【相似文献】