九种常用缺失值插补方法的比较

发布时间：2018-06-17 02:09

本文选题：两类缺失模式 + 随机缺失数据集　；参考：《云南师范大学》2017年硕士论文

【摘要】：数据集中缺失值的存在是一个常见但难于处理的问题,它会增大统计分析的复杂性和难度,导致分析结果的偏倚,降低统计工作的效率。虽然事前预防是避免缺失值最简单有效的方法,但是由于种种常规原因和现实条件,事前预防并不能完善地解决问题。因此,对缺失值的事后处理显得尤为重要,也越来越受到研究者们的高度重视。常见的缺失值的处理方法主要有直接删除法和数据插补法两种,鉴于直接删除法会导致数据信息的再次损失,所以本文主要从数据插补方面入手,先对统计学中常用的九种缺失值的插补方法——均值插补、随机插补、回归插补、多重插补、k最近邻插补、决策树插补、支持向量机插补和神经网络插补等方法的插补原理作了介绍;接着按数据量从小到大的顺序选取3个数据集salary、iris和Airfoil,以10%的样本量缺失率,在R中按单个变量随机缺失和多个变量随机缺失两种模式生成相应的缺失数据集(这里随机缺失的含义是将其中随机抽取的10%的数据替换为缺失值),然后用上述九种插补方法对两种缺失模式的数据集进行插补。为评价和比较各种插补方法的插补效果,本文从两方面进行比较:(1)从数据插补误差的角度,把单个变量缺失模式下的3个数据集的九种插补方法生成的诸插补值与对应的数据真值(随机缺失前)进行比较,计算对应的平均绝对误差(MAE)和均方误差(MSE),按这两个指标的大小评价比较这九种插补方法的优劣。(2)从模型的角度,用缺失前的完整数据集和先缺失再插补后的诸数据集各自建立多元线性回归模型,估计相应的回归系数(向量),计算出相应的判定系数,再进行比较和评价。最后指出了各插补方法的特点和异同,对本文的研究结果进行归纳总结,并说明有待改进之处和可以进一步研究的内容。
[Abstract]:The existence of missing values in data sets is a common but difficult problem, which will increase the complexity and difficulty of statistical analysis, lead to the bias of analysis results and reduce the efficiency of statistical work. Although pre-prevention is the simplest and most effective method to avoid missing value, due to various conventional reasons and practical conditions, pre-prevention can not solve the problem perfectly. Therefore, the post-processing of missing values is particularly important and has been paid more and more attention by researchers. There are two common methods to deal with missing values: direct deletion method and data interpolation method. In view of the fact that the direct deletion method will lead to the loss of data information again, so this paper mainly starts from the aspect of data interpolation. First, the interpolation methods of nine kinds of missing values commonly used in statistics are mean interpolation, random interpolation, regression interpolation, multi-interpolation nearest neighbor interpolation, decision tree interpolation, This paper introduces the interpolation principle of support vector machine interpolation and neural network interpolation, and then selects three data sets, Salarymiris and Airfoil, according to the order of data from small to large, and takes 10% sample loss rate. Generate the corresponding missing data set in R according to the two patterns of random deletion of single variable and random deletion of multiple variables (here the meaning of random deletion is to replace 10% of the randomly sampled data with the missing value), and then use the above. Nine interpolation methods are used to interpolate the data sets of two missing patterns. In order to evaluate and compare the interpolation effect of various interpolation methods, this paper compares the error of data interpolation from two aspects. The interpolation values generated by nine interpolation methods for three data sets in a single variable deletion mode are compared with the corresponding true values (before random deletion). Calculate the corresponding mean absolute error (mae) and mean square error (MSE), evaluate and compare the advantages and disadvantages of these nine interpolation methods according to the size of these two indexes) from the point of view of the model, The multivariate linear regression models are established by using the complete data sets before deletion and those before deletion and interpolation respectively. The corresponding regression coefficients (vectors) are estimated and the corresponding decision coefficients are calculated and compared and evaluated. Finally, the characteristics and similarities and differences of the interpolation methods are pointed out, the research results of this paper are summarized, and the points for improvement and the contents that can be further studied are explained.
【学位授予单位】：云南师范大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：O212.1

【相似文献】