面向流数据的决策树分类算法并行化

发布时间：2019-02-11 10:23

【摘要】：随着云计算、物联网等技术的兴起,流数据作为一种新型的大数据形态广泛存在于电信、互联网、金融等领域.与传统静态数据相比,大数据环境下的流数据具有快速、连续和随时间变化等特点.同时数据流的隐含分布变化会带来概念漂移问题.为了适应大数据环境下流数据分类算法的要求,必须对传统的静态离线数据分类算法进行改进,提出基于分布式计算平台Storm的P-HT并行化算法.算法在满足Storm流处理平台要求基础上,通过滑动窗口机制、替代子树机制和并行化处理,提高了算法的灵活性和通用性,并且能良好地适应数据流的概念漂移.最后通过实验验证该算法的有效性和高效性,结果表明在与传统C4.5算法相比精度没有降低的情况下,改进的P-HT算法具有更大的吞吐量和更快的处理速度.
[Abstract]:With the rise of cloud computing, Internet of things and other technologies, streaming data as a new form of big data widely exists in telecommunications, Internet, finance and other fields. Compared with the traditional static data, the streaming data in big data environment is fast, continuous and time-varying. At the same time, the change of the implicit distribution of data flow will bring about the concept drift problem. In order to meet the requirements of stream data classification algorithm under big data environment, the traditional static off-line data classification algorithm must be improved, and a P-HT parallelization algorithm based on distributed computing platform Storm is proposed. On the basis of satisfying the requirements of Storm flow processing platform, the algorithm improves the flexibility and generality of the algorithm by sliding window mechanism, replacing subtree mechanism and parallelization processing, and can adapt to the conceptual drift of data flow well. Finally, the effectiveness and efficiency of the proposed algorithm are verified by experiments. The results show that the improved P-HT algorithm has higher throughput and faster processing speed than the traditional C4.5 algorithm.
【作者单位】：南京邮电大学计算机学院;江苏省无线传感网高技术研究重点实验室(南京邮电大学);南京邮电大学先进技术研究院;高维信息智能感知与系统教育部重点实验室(南京理工大学);
【基金】：国家自然科学基金项目(61170065) 江苏省自然科学基金优秀青年基金项目(BK20170100) 国家重点研发计划(2017YFB0202200) 江苏省重点研发计划项目(BE2017166)~~
【分类号】：TP311.13

【相似文献】