基于语义的垃圾邮件过滤技术的研究

发布时间：2018-06-18 09:21

本文选题：垃圾邮件过滤 + 文本分类　；参考：《重庆大学》2016年硕士论文

【摘要】：电子邮件的兴起伴随着垃圾邮件的产生,随着互联网的普及和发展,电子邮件成为人们日常工作和生活中必不可少的交流方式,甚至已经成为文件和文档进行快速传输的主流载体。而垃圾邮件问题却日益严重并越来越深切的影响着人们的工作和生活,垃圾邮件不仅严重占用和浪费了网络带宽和计算资源,而且还带来各种各样的安全隐患,威胁着人们的信息安全。虽然人们已经提出了很多相应的解决对策和方法,但是垃圾邮件仍呈现出持续性的爆炸式增长的趋势。垃圾邮件的过滤拦截可以通过影响和控制电子邮件传递过程中的各个环节来实现。本文主要研究通过对电子邮件的正文内容进行语义识别来实现垃圾邮件过滤的方法和技术。垃圾邮件的内容和形式不管如何变化,必定会传达出某种语义信息,以这些语义信息作为垃圾邮件过滤和拦截的基础,会有效提高垃圾邮件拦截的效率和效果。传统的特征选择方法大多是基于统计学的,并且总是将文本中的字、词、短语拆分开来做为单独的特征项来处理,将文本单纯的看作一堆字词的堆砌,抛弃了不同语法结构、语言环境、词汇搭配所表现出的语义信息。本文提出了一种针对中文邮件的基于内容的垃圾邮件过滤技术,并在公共语料集TREC06c上进行了实验,获得了良好的结果。本文首先采用了逐层在文本上添加注释来提取语义信息的方法,然后将提取出的语义信息进行筛选后作为表示文本的特征项来构建决策树并最终生成分类器,实现了文本的识别和分类。这种方法的优势在于既能够准确表达文本的内容含义,又能够达到科学有效的降维效果。因此,直接选取词汇作为特征来表示文本,而提取文本的语义信息作为特征项来表示文本,是一种更加科学有效的特征选择方法。采用语义作为特征项有两个明显的优势。一是,对文本内容的反映更准确,更有利于实现准确的文本分类;二是,与传统方法相比,特征项的数量显著减少了,运算压力更小。本文主要研究的第二个问题是在科学有效的筛选出特征项的基础上,运用决策树方法来完成垃圾邮件的多分类任务,因为实现垃圾邮件的多分类更加有助于实现邮件过滤和拦截的个性化,注重了用户隐私。从自然语言处理的方面来看,本文中介绍的这一方法作为一种针对文本内容的多分类技术,在自然语言处理中也具有巨大潜力。
[Abstract]:With the popularity and development of the Internet, email has become an indispensable way of communication in people's daily work and life. It has even become the mainstream carrier for fast transmission of files and documents. However, the problem of spam is becoming more and more serious and deeply affects people's work and life. Spam not only occupies and wastes network bandwidth and computing resources seriously, but also brings all kinds of hidden dangers to security. Threatening people's information security. Although people have put forward a lot of corresponding solutions and methods, spam still shows a sustained explosive growth trend. The filtering and blocking of spam can be realized by influencing and controlling each link in the process of e-mail delivery. This paper mainly studies the method and technology of spam filtering by semantic recognition of the text of email. No matter how the content and form of spam changes, it will convey some semantic information. Using these semantic information as the basis of spam filtering and blocking will effectively improve the efficiency and effect of spam blocking. Most of the traditional feature selection methods are based on statistics, and they always separate the words, words and phrases in the text as separate feature items, and treat the text simply as a pile of words, leaving out different grammatical structures. Language environment, lexical collocation shows semantic information. In this paper, a content-based spam filtering technique for Chinese mail is proposed, and the experiment is carried out on TREC06c, a common corpus, and good results are obtained. In this paper, we first use the method of adding annotations to the text layer by layer to extract the semantic information, then filter the extracted semantic information as the feature items to represent the text to construct the decision tree and finally generate the classifier. The text recognition and classification are realized. The advantage of this method is that it can not only accurately express the meaning of the text, but also achieve a scientific and effective dimensionality reduction effect. Therefore, it is a more scientific and effective feature selection method to directly select words as features to represent text, and extract semantic information of text as feature items to represent text. There are two obvious advantages in using semantics as feature items. One is that the text content is more accurately reflected, which is more conducive to the realization of accurate text classification; second, compared with the traditional method, the number of feature items is significantly reduced, and the operation pressure is less. The second problem of this paper is to use the decision tree method to complete the multi-classification of spam on the basis of scientific and effective selection of feature items. Because the multi-classification of spam is more helpful to realize the personalization of email filtering and blocking, it pays more attention to user privacy. From the aspect of natural language processing, this method, as a multi-classification technique for text content, has great potential in natural language processing.
【学位授予单位】：重庆大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP393.098

【相似文献】