基于子空间的两阶段离群挖掘算法及其并行化

发布时间：2018-04-21 20:53

本文选题：关联子空间 + 候选离群对象　；参考：《太原科技大学》2016年硕士论文

【摘要】：离群挖掘是数据挖掘领域中的主要研究内容之一,并已广泛地应用到金融分析、信用诈骗,垃圾邮件分析等领域。针对高维海量数据集,如何有效的降低维灾,提高数据挖掘的精度和效率,是离群挖掘中主要解决的问题。本文从提高离群数据挖掘效果的角度,对两阶段离群检测算法及其并行化进行了研究,其主要研究成果如下:(1)采用筛选候选离群对象删减计算量的思想,给出了一种两阶段离群挖掘算法。第一阶段,计算每个数据对象在每一维的密度比,所有维的密度比的乘积取对数平均作为密度系数,并选取候选离群对象;第二阶段,把候选离群对象的邻居在每一个关联子空间中的偏离程度之积作为偏差比,密度系数与偏差比的乘积作为离群系数,并确定离群数据对象。由于仅计算候选离群对象的离群系数,因此有效地提高挖掘效率;最后,采用UCI数据集,实验验证了该算法不仅保证挖掘结果精度,而且有效提高了挖掘效率。(2)采用Map Reduce编程模型,给出了基于子空间的两阶段离群数据并行挖掘算法。先将数据集的全部数据分配到每一个子节点上,在每一个节点上用一个map函数计算数据的密度系数。用Reduce函数将结果汇总到主节点得到候选离群数据集。为使每个节点负载均衡,对候选离群数据集中的对象重新编号,将计算任务均等的分配到每个子节点。计算候选离群数据集中每个对象的偏差比,用Reduce函数将结果汇总到主节点,计算候选离群对象的离群系数。最后排序,筛选出离群点。
[Abstract]:Outlier mining is one of the main research contents in the field of data mining, and has been widely used in financial analysis, credit fraud, spam analysis and other fields. For high-dimensional massive data sets, how to effectively reduce dimensionality disasters and improve the accuracy and efficiency of data mining is the main problem in outlier mining. In order to improve the effect of outlier data mining, the two-stage outlier detection algorithm and its parallelization are studied in this paper. A two-stage outlier mining algorithm is presented. In the first stage, the density ratio of each data object in each dimension is calculated, the product of the density ratio of all dimensions is taken as the logarithmic average as the density coefficient, and the candidate outliers are selected. The product of deviation degree of neighbor of candidate outlier in each associated subspace is regarded as deviation ratio, and the product of density coefficient and deviation ratio is taken as outlier coefficient, and the outlier data object is determined. Because the outlier coefficients of candidate outliers are only calculated, the mining efficiency is improved effectively. Finally, the UCI dataset is used to verify that the algorithm not only ensures the accuracy of mining results, but also improves the mining efficiency. Moreover, the efficiency of mining is improved effectively. (2) using Map Reduce programming model, a two-stage outlier parallel mining algorithm based on subspace is presented. First, all the data of the data set are assigned to each child node, and the density coefficient of the data is calculated by a map function on each node. Reduce function is used to summarize the results to the primary node to obtain candidate outlier data sets. In order to balance the load of each node, the objects in the candidate outlier dataset are renumbered and the computing tasks are assigned equally to each child node. The deviation ratio of each object in candidate outlier dataset is calculated, and the result is summarized to the primary node by Reduce function, and the outlier coefficient of candidate outlier is calculated. Finally, sorting out outliers.
【学位授予单位】：太原科技大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP311.13

【相似文献】