基于Spark平台的垃圾短信过滤系统的研究与实现

发布时间：2018-05-24 20:24

本文选题：垃圾短信 + 文本分类　；参考：《北京邮电大学》2016年硕士论文

【摘要】：近年来,随着网络的普及和手机用户的快速增长,垃圾短信泛滥的问题日益突出,已经严重影响了人民群众的生活,扰乱了社会秩序。为了解决这个问题,政府出台了相应的法律法规严厉打击垃圾短信,各大运营商也采取了各种治理手段。目前,常用的垃圾短信处理技术主要包括黑白名单过滤,基于短信长度和发送频率的算法过滤,基于文本分类技术的过滤等。单一的过滤方式往往只能覆盖某种类型的垃圾短信,而随着短信数量的增加,这些处理技术在性能上也显得捉襟见肘。针对以上问题,本文提出了基于Spark平台的垃圾短信过滤系统,该系统采用联合过滤的方式,并行的处理垃圾短信,能在提升过滤效果的同时,也提高应对大数据量短信的能力。本文具体所做的工作有如下几点:(1)本文对垃圾短信的背景进行了综述,对垃圾短信进行了定义和分类,概括了垃圾短信的特征和危害,并详细介绍了国内外垃圾短信治理的现状。本文对文本分类技术进行了深入研究,包括文本预处理,清洗与去噪,特征降维和文本分类算法等,介绍说明了 Simhash技术,对Hadoop平台和Spark平台进行了研究,并介绍了它们的工作原理。(2)设计实现了串行的垃圾短信过滤系统。本文首先对系统进行了需求分析和概要设计,然后对各模块进行了详细设计,并编码实现。串行过滤系统主要有短信处理模块、特性判定模块、Simhash模块和贝叶斯分类器模块组成,解决了单一过滤方式无法覆盖所有短信类别的问题,引入Simhash算法,不但提升了过滤效果,并且通过压缩样本库,提高了贝叶斯分类器的判定速度。(3)基于Spark平台的优势,对垃圾短信过滤系统进行了并行优化,包括策略提取模块、特性判定模块和贝叶斯分类器模块。本文详细介绍了并行优化的设计原则,包括适用性、可扩展性、有效性和并行,根据设计原则,对需要优化的模块进行了并行设计,然后进行了代码实现。另外,本文采取了并行策略提取的方式精简样本库,建立策略库,来解决样本数量过大,影响过滤效率的问题。最后通过搭建实验平台,对各个模块进行了实验,并对相关实验结果进行了分析总结。实验表明,本文提出的基于Spark的垃圾短信过滤系统能有效的对垃圾短信进行分类和过滤,并且处理大数量的短信能力突出,另外,该系统还具有良好的扩展性和实用性,给海量的垃圾短信处理提供了一种新的解决思路。
[Abstract]:In recent years, with the popularity of the network and the rapid growth of mobile phone users, the problem of spam message flooding has become increasingly prominent, which has seriously affected the lives of the people and disturbed the social order. In order to solve this problem, the government has issued the corresponding laws and regulations to crack down on spam messages. At present, the commonly used spam short message processing technology mainly includes black and white list filtering, algorithm filtering based on short message length and sending frequency, filtering based on text classification technology and so on. A single filtering method can only cover a certain type of spam messages, but with the increase of the number of SMS, these processing technologies are also overstretched in performance. Aiming at the above problems, this paper puts forward the spam short message filtering system based on Spark platform. The system adopts the method of joint filtering and parallel processing of spam short message, which can improve the filtering effect and improve the ability to deal with the large amount of short message at the same time. This paper summarizes the background of spam SMS, defines and classifies the spam SMS, and summarizes the characteristics and harm of spam SMS. And introduced in detail the domestic and foreign garbage short message management present situation. In this paper, the text classification technology is deeply studied, including text preprocessing, cleaning and denoising, feature reduction and text classification algorithm. The Simhash technology is introduced, and the Hadoop platform and Spark platform are studied. The design and implementation of serial spam short message filtering system are introduced. In this paper, the requirement analysis and outline design of the system are carried out first, and then each module is designed in detail, and the code is implemented. The serial filtering system is composed of short message processing module, Simhash module and Bayesian classifier module, which solves the problem that the single filtering method can not cover all short message categories. The introduction of Simhash algorithm not only improves the filtering effect. By compressing the sample base, the speed of Bayesian classifier is improved. (3) based on the advantage of Spark platform, the spam short message filtering system is optimized in parallel, including the module of policy extraction, the module of feature determination and the module of Bayesian classifier. This paper introduces the design principles of parallel optimization in detail, including applicability, extensibility, validity and parallelism. According to the design principles, the modules that need to be optimized are designed in parallel, and then implemented in code. In addition, the parallel policy extraction method is adopted to reduce the sample base and establish the policy database to solve the problem that the number of samples is too large and the filtering efficiency is affected. Finally, the experiment platform is built, and the experimental results are analyzed and summarized. The experiments show that the spam short message filtering system based on Spark can effectively classify and filter spam short messages, and the ability to deal with large number of short messages is outstanding. In addition, the system has good expansibility and practicability. It provides a new solution to the massive spam SMS processing.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP391.1

【相似文献】