面向云平台的二代测序数据近似去重方法研究

发布时间：2018-04-07 16:50

本文选题：高通量测序　切入点：重复数据删除　出处：《计算机工程与应用》2017年23期

【摘要】：新一代测序因其数据量大、数据处理过程复杂、对计算资源要求高等特点,需要通过云计算进行处理。然而,云计算的处理方式要求先将测序数据上传到云平台中。但由于测序过程的随机性,使得同一样本的两次测序、两个相似样本分别测序后所产生的文件在二进制层面会有较大差别。目前已有的去重方法无法有效识别出这样的"重复"测序文件和测序结果中的"重复"内容。重复上传和存储这些重复数据,不仅消耗网络带宽,而且浪费存储空间。针对现存的重复数据删除方法仅仅基于文件的二进制特征,并未有效利用测序结果数据相似性特点的问题,提出一种面向云平台的海量高通量测序数据近似去重方法NPD(Near Probability Deduplication)。该方法对Fast Q中的序列和质量信息,使用Sim Hash计算分块指纹,采用客户端与云平台双布谷过滤器(Cukoo Filter)对指纹值进行快速存在性检测,最后由云平台使用近似算法对指纹值近似去重。实验结果表明,NPD方法在保证高效的同时,大幅提升了去重率,进而减少了网络流量,缩短了数据上传时间,能够支撑海量数据处理,具有良好的实用价值。
[Abstract]:The new generation sequencing needs to be processed by cloud computing because of its large amount of data, complex data processing process and high demand for computing resources.However, cloud computing requires that sequenced data be uploaded to the cloud platform first.However, because of the randomness of the sequencing process, the two similar samples are sequenced twice, and the files produced after the two similar samples are sequenced will be different in binary level.The existing methods can not effectively identify such "repeat" sequencing documents and sequencing results of "repeat" content.Uploading and storing these duplicate data repeatedly not only consumes network bandwidth, but also wastes storage space.In order to solve the problem that the existing methods of repeated data deletion are only based on the binary features of files and do not effectively utilize the similarity of sequencing results, a cloud platform-oriented approximate de-reduplication method for massive high-throughput sequencing data, NPD(Near Probability replication, is proposed.In this method, the sequence and quality information in Fast Q are calculated by using Sim Hash, and the existence of fingerprint is detected by using client and cloud platform double valley filter.At last, the approximate algorithm is used to remove the fingerprint value from the cloud platform.The experimental results show that the NPD method not only ensures high efficiency, but also greatly increases the weight removal rate, thus reducing the network traffic, shortening the time of data upload, and can support the massive data processing, which has good practical value.
【作者单位】：北京信息科技大学信息管理学院;首都医科大学附属北京地坛医院传染病研究所;
【基金】：国家自然科学基金(No.61572079) 北京市教育委员会科技计划一般项目(No.KM201711232018)
【分类号】：TP301.6

【相似文献】