基于PSO-BP神经网络的Lucene搜索引擎的研究

发布时间：2019-02-23 09:37

【摘要】：Lucene是一个全文搜索体系架构，具有优异的索引结构、良好的系统架构以及高性能、可伸缩的信息搜索库等优点，但是对于中文分词以及多种文本格式的支持却很是不足。目前Lucene采用的中文分词算法有很多，包括Lucene自身提供的StandardAnalyzer和CJKAnalyzer，以及第三方提供的ChineseAnalyzer和IK_CAnalyzer等等很多种中文分词系统。其中，StandardAnalyzer是基于单字分词的，即在对中文文本进行分词时，以字为单位进行切分，其缺点是需要复杂的单字匹配算法，以及大量的CPU运算；CJKAnalyzer和ChineseAnalyzer采用的均是二分法，所谓二分法就是每每两个字当作一个词来切分；IK_CAnalyzer分词技术是基于分词词典的，采用了特有的正向迭代最细粒度切分算法和多子处理器分析模式。目前，Lucene搜索引擎并未实现基于理解的中文分词方法，因为计算机无法识别每个词在不同语境中的含义，所以基于理解的分词方法还未有实际的运用效果。针对Lucene对中文分词的不足，尤其是缺少基于理解领域的中文分词技术等缺陷，本文探讨了BP（Back Propagation）神经网络算法在中文分词中的应用研究，并针对BP神经网络应用中文分词具有收敛速度慢，容易陷入局部极小值以及速度和效率低等缺陷，提出了一种改进的微粒群优化算法（PSO, Particle SwarmOptimization）优化BP神经网络——PSO-BP神经网络，并将其运用于中文分词中，与传统的BP神经网络相比较，可以得出PSO-BP神经网络不仅解决了传统BP神经网络收敛速度慢的缺陷，同时也提高了分词的精度。然后，本文对Lucene提供的第三方中文分词组件的API进行了系统地研究与分析，将经PSO-BP神经网络优化后的中文分词技术成功应用于Lucene中，并与Lucene自带的中文分词技术进行比较，得出该技术明显优于自带的中文分词技术。最后，，本文采用包含PSO-BP神经网络中文分词组件的Lucene进行搜索引擎的设计和实现，从而实现搜索引擎的中文分词的智能化探索，为后续的工作和研究提供了一个良好的平台。
[Abstract]:Lucene is a full-text search architecture with excellent index structure, good system architecture and high performance, scalable information search library. However, the support for Chinese word segmentation and various text formats is very inadequate. At present, there are many Chinese word segmentation algorithms used in Lucene, including StandardAnalyzer and CJKAnalyzer, provided by Lucene itself and ChineseAnalyzer and IK_CAnalyzer provided by third parties. Among them, StandardAnalyzer is based on word segmentation, that is to say, word segmentation is based on word segmentation. Its disadvantage is that it needs complex word matching algorithm and a large number of CPU operations. CJKAnalyzer and ChineseAnalyzer use dichotomy, so called dichotomy is each word as a word to divide; The word segmentation technology of IK_CAnalyzer is based on the word segmentation dictionary, and adopts the special forward iterative finest granularity segmentation algorithm and the analysis mode of multiple sub-processors. At present, the Lucene search engine has not realized the Chinese word segmentation method based on understanding, because the computer can not recognize the meaning of each word in different context, so the word segmentation method based on understanding has no practical application effect. In view of the deficiency of Lucene in Chinese word segmentation, especially the lack of Chinese word segmentation technology based on understanding, this paper discusses the application of BP (Back Propagation) neural network algorithm in Chinese word segmentation. Aiming at the shortcomings of BP neural network in the application of Chinese word segmentation, such as slow convergence, easy to fall into local minima, and low speed and efficiency, an improved particle swarm optimization algorithm (PSO,) is proposed. Particle SwarmOptimization) optimizes BP neural network, PSO-BP neural network, and applies it to Chinese word segmentation. Compared with traditional BP neural network, PSO-BP neural network not only solves the problem of slow convergence speed of traditional BP neural network. At the same time, the accuracy of word segmentation is improved. Then, the API of the third-party Chinese word segmentation component provided by Lucene is systematically studied and analyzed in this paper. The Chinese word segmentation technology optimized by PSO-BP neural network is successfully applied to Lucene, and compared with the Chinese word segmentation technology provided by Lucene. The result shows that this technique is superior to the Chinese word segmentation technology. Finally, this paper uses Lucene which includes PSO-BP neural network Chinese word segmentation component to design and implement the search engine, so as to realize the intelligent exploration of Chinese word segmentation of search engine, which provides a good platform for the follow-up work and research.
【学位授予单位】：中国石油大学（华东）
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3;TP183

【参考文献】