基于分布式数据流的大数据分类模型和算法

发布时间：2018-08-24 21:29

【摘要】：大数据是需求驱动的概念.随着数据库系统的普及和因特网服务的扩张,企业或者个人可用的数据正在膨胀,已有的技术很难满足大数据时代的数据分析需求,因此需要探索新的理论和方法来支撑大数据的应用.虽然大数据的4V属性已经被广泛讨论,但是它们大多描述的仍然是大数据的表象,所以很难从中抽象出统一的数据格式,因而进一步寻找可用于数据格式化的技术特征是必要的.面向于以分布式和流动性为主要技术特征的大数据应用需求,文中以分布式数据流为数据表达载体,在此基础上设计对应的大数据分类模型和挖掘算子.同时针对大数据的分类挖掘需要解决的关键问题来构建关键步骤对应的算法.理论上证明了文中给出的微簇合并技术和样本数据重构方法的合理性.实验表明:文中提出的基于分布式数据流的大数据的分类模型及算法不仅能大幅度地减少网络节点间的通讯代价,而且可以获得平均10%左右的全局挖掘精度的提升(对比已有的典型算法DS-means);虽然时间花费略高于DS-means,但是两者在不同的数据容量测试下相差很小、且时间攀升趋势相当.
[Abstract]:Big data is a demand-driven concept. With the popularity of database systems and the expansion of Internet services, the data available to enterprises or individuals are expanding, and existing technologies are difficult to meet the data analysis needs of big data's time. Therefore, we need to explore new theories and methods to support big data's application. Although big data's 4V attribute has been extensively discussed, most of them still describe the representation of big data, so it is difficult to abstract a unified data format from it. Therefore, it is necessary to further search for the technical features that can be used for data formatting. In order to meet the requirements of big data, whose main technical features are distributed and fluidity, this paper uses distributed data stream as data expression carrier, and then designs the corresponding big data classification model and mining operator. At the same time, according to big data's classification mining need to solve the key problems to construct the corresponding algorithm of key steps. It is proved theoretically that the microcluster merging technique and the reconstruction method of sample data are reasonable. Experiments show that big data's classification model and algorithm based on distributed data flow can not only greatly reduce the communication cost among network nodes. The average global mining accuracy can be improved by about 10% (compared with the existing typical algorithm DS-means). Although the time cost is slightly higher than that of DS-means, the difference between them under different data capacity tests is very small, and the trend of time increase is similar.
【作者单位】：中央财经大学信息学院;
【基金】：国家自然科学基金(62173293) 中央财经大学学科建设基金(CUFE00100101)资助~~
【分类号】：TP311.13

【相似文献】