稀疏矩阵插补及在大型问卷调查中的应用研究

发布时间：2018-08-02 17:07

【摘要】：自2012年以来,“大数据”一词越来越多地出现在人们的生活、工作和学习中。IBM公司曾进行过一项研究,研究结果显示从古至今我们人类世界的全部数据中有90%都产生于过去的两年,并且预计2020年后全人类范畴的数据量可能会达到目前数据量的44倍。在大量数据产生及扩展的过程中不完备数据的出现是不可避免的,而不完备数据中的缺失值又往往会对数据的可利用性产生重大的影响。网络购物平台的评价系统在收集大量不完备数据上起到了很大的作用。假如所有消费者均对自己所购买到的商品进行了评价,该网络购物平台的评分系统就能够将所有评分数据收集成一个含有大量缺失值的矩阵,我们称之为“稀疏矩阵”。如果一些消费者购买了商品,却没有对商品进行评价,则会提高该稀疏矩阵的缺失率。本文根据网络购物平台评分系统和美国Netflix在线影片租赁公司影片评价系统得到的数据结构为依据,联系当前随着大数据不断发展而扩增的实践调查数据,不难发现,以往简单的小型抽样调查已经不能满足当今社会对实践调查的要求,因此无论在问卷大小还是在样本量多少上,都需要有新的突破。针对含有大量问题的问卷调查,以往的做法通常是给予被调查者一定的奖励或回馈以获得被调查者的配合,该方法不但在人力、物力及财力上需要一定的保障,而且并不能保证问卷数据的质量。本文运用问卷分割法将调查中的大型问卷按照题量及问题之间的关联性分割为多个小型问卷,在调查过程中每个被调查者从中随机抽取特定数量的小型问卷进行作答,在保证样本量的前提下,收集并整理调查数据,最终会得到一个含有大量缺失值的稀疏矩阵。进而运用缺失值插补的方式对稀疏矩阵进行插补,以获到完整的研究数据。本文通过对一般数据插补方法、稀疏矩阵数据插补方法和大型问卷缺失数据插补方法的对照,采取随机数插补和多项逻辑模型插补两种插补方式,通过对插补成效的对照分析,得出相应的结论。由于人力及时间的限制,本文数据来自于R-Studio软件的模拟。首先,运用R-Studio软件生成模拟数据,由于每位被调查者回答的数据均以“单元”为单位,因此在进行数据缺失的过程中要实现成块缺失,即单元缺失,最终的稀疏矩阵中每个被调查者都回答了特定单元数的问题;其次,利用不同被调查者共同回答的问题作为铆题,计算不同被调查者在回答同一问题时的关联性,进而利用该关联性对其他未回答数据进行插补;最后,利用插补所得的数据与原始数据进行对比,验证问卷分割法及本文所用插补方式的可行性和准确性。由于本文数据采用R-Studio软件模拟生成,因此在理论上具有一定的理想化假设,虽然每个被调查者回答问卷的单元数可以在调查过程中进行人为的控制,但被调查者回答每个单元的问题数据需假设为内部无缺失,即整个数据矩阵只有“单元”缺失,没有个别缺失。全文包括五章的内容。第一章,介绍了文章的根本内容,包括选题背景和研究目的、文献综述、研究方法及论文创新之处;第二章,是缺失数据的处理方法简介,阐述了近年来学者们研究缺失数据插补时所用到的方法及其简单概念;第三章,作为本文的核心内容,从易到难、从数据的生成到缺失,再到插补,具体介绍了大型问卷分割法及缺失数据插补方法,并将完成插补的数据与原始数据进行比较;第四章,运用第三章研究的内容及R-Studio软件生成的大型稀疏矩阵进行进一步的分析,验证本文理论和方法的可行性和准确性;第五章,是对全文的总结以及对本文所研究内容发展前景的展望,同时,对本文的不足之处提出了改进方法。
[Abstract]:Since 2012, the word "big data" has appeared more and more in people's life. In work and study,.IBM has conducted a study. The results show that 90% of all the data in our human world have been produced in the past two years from ancient times to the present, and it is expected that the amount of data in the whole human category may reach the target after 2020. 44 times the amount of previous data. Incomplete data is inevitable in the process of generating and expanding a large number of data, and the missing values in incomplete data often have a significant impact on the availability of data. The evaluation system of the network shopping platform plays a great role in collecting a large number of incomplete data. Consumers are all evaluating what they have bought. The scoring system of the online shopping platform can collect all the scoring data into a matrix with a large number of missing values. We call it a "sparse matrix". If some consumers buy a commodity but do not evaluate the commodity, it will improve the sparse moment. Based on the data structure obtained by the network shopping platform scoring system and the film evaluation system of Netflix online film leasing company in the United States, this paper is not difficult to find out that the simple small sample survey can not meet the current social reality. As a result, a new breakthrough is needed both in the size of the questionnaire and in the size of the sample. In the past, the past practice usually gives the respondents a reward or feedback to obtain the cooperation of the respondents. The method not only needs a certain guarantee in human, material and financial resources. The quality of the questionnaire data is not guaranteed. In this paper, the questionnaire segmentation method is used to divide the large questionnaire in the survey into a number of small questionnaires according to the correlation between the questions and the questions. In the course of the investigation, a small number of small questionnaires are randomly selected from each of the respondents. After sorting out the survey data, a sparse matrix with a large number of missing values is finally obtained. Then the sparse matrix is interpolated with the missing value interpolation to obtain the complete data. Two interpolations are taken by random number interpolation and multiple logic model interpolation, and the corresponding conclusions are obtained by comparing the results of interpolation. The data of this paper are derived from the simulation of R-Studio software because of human and time constraints. First, R-Studio software is used to generate analog data, because the data each respondents answered is "unit". "As a unit, therefore, in the process of missing data, we have to realize the missing block, that is, the missing unit, and each of the investigators in the final sparse matrix answers the problem of the number of specific units. Secondly, the problem is used by different respondents as a riveting problem, and the correlation between the respondents in answering the same question is calculated. In the end, the data from the interpolation are compared with the original data to verify the feasibility and accuracy of the questionnaire segmentation method and the interpolation method used in this paper. Because the data used in this paper are simulated by R-Studio software, it has a certain idealization hypothesis in theory. The unit number of each respondents' answers to the questionnaire can be controlled artificially during the investigation, but the respondents' answer to each unit's problem data needs to be assumed to be internal, that is, the whole data matrix has only "unit" missing and no individual missing. The full text includes five chapters. Chapter 1 introduces the basic content of the article. Including the background and purpose of research, literature review, research methods and the innovation of the paper; the second chapter is the introduction of the missing data processing methods, and expounds the methods and simple concepts used by scholars in the absence of data interpolation in recent years. The third chapter, as the core content of this paper, is from easy to difficult, from data generation to missing, In the fourth chapter, the fourth chapter uses the contents of the third chapters and the large sparse matrix generated by the software of the third chapter to verify the feasibility and accuracy of the theory and method of this paper; Fifth Chapter one is the summary of the whole paper and the prospects for the development of the research content in this paper. At the same time, the paper puts forward the improvement methods for the deficiencies of this paper.
【学位授予单位】：河北经贸大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：O151.21

【相似文献】