藏文搜索和搜索结果聚类研究及系统实现

发布时间：2018-07-06 16:45

本文选题：藏文分词 + 藏文聚类　；参考：《西南交通大学》2013年硕士论文

【摘要】：藏文历史悠久,是藏族文化和藏族文明传承的载体,使用人数有600多万。藏文文献数目庞大,内容广泛。随着windows系统对藏文的支持,藏族同胞参入网络活动的热情日益高涨。然而当前尚无藏文搜索引擎,国内外各大著名搜索引擎也不提供藏文搜索,因而对藏文搜索系统的研究意义重大。本文围绕如何实现藏文搜索系统,研究了藏文分词,藏文文本收集,文本处理,编码转换,索引搜索及结果聚类等相关问题,旨在实现一个功能完善的藏文信息检索系统。本文的主要工作如下：第一,提出了一种AllCut藏文分词算法。藏文词间没有分隔符,因而需要分词。当前分词算法主要有基于统计概率、词性标注及语法规则等。然而这些算法或需要大量的语料训练学习,或实现起来很复杂,在当前情况下难以实现或实现效果并不好。因而本方案使用词典匹配,结合藏文的语法特性及格助词和接续性特征,同时使用细粒度切分,取得了很好的分词效果,为接下来工作提供了保障。第二,藏文聚类研究。本文首先研究了中藏聚类中文文本表示,藏文停词等相关问题：使用向量模型表示文档,使得文本可以很好的被计算机存储和处理；通过统计大量文档得到藏文停词,排除了这些词对聚类效果的干扰。最后系统研究了及划分法和层次法聚类算法对于藏文的聚类效果。第三,藏文信息检索研究及系统实现。藏文信息检索主要研究了藏文网页收集,藏文编码转换,藏文网页预处理,及藏文文本存储等,解决了计算机对藏文的处理和检索；然后以Lucene为基础,实现了该搜索系统,系统能够自动更发现更新藏文资源,提供藏文搜索功能,完成了藏文搜索引擎的功能。并结合藏文聚类对搜索结果聚类显示,提高了搜索结果的针对性和准确性。
[Abstract]:The Tibetan language has a long history and is the carrier of Tibetan culture and Tibetan civilization, with more than 6 million users. Tibetan literature is large in number and extensive in content. With the support of the windows system for Tibetan, Tibetan people's enthusiasm to participate in network activities is growing. However, there is no Tibetan search engine at present, and famous search engines at home and abroad do not provide Tibetan search, so the research on Tibetan search system is of great significance. This paper focuses on how to realize the Tibetan language search system, studies the Tibetan participle, the Tibetan text collection, the text processing, the coding conversion, the index search and the result clustering and so on. The purpose of this paper is to realize a perfect Tibetan information retrieval system. The main work of this paper is as follows: first, an all cut Tibetan word segmentation algorithm is proposed. There are no delimiters between Tibetan words, so participle is needed. Current word segmentation algorithms are mainly based on statistical probability, part of speech tagging and grammar rules. However, these algorithms require a lot of corpus training and learning, or they are very complex to implement, which are difficult to implement or not effective in the current situation. Therefore, this scheme uses dictionary matching, combines the grammatical characteristics of Tibetan and the features of case auxiliary and continuity, at the same time uses fine granularity segmentation, and achieves a good segmentation effect, which provides a guarantee for the next work. Second, the study of Tibetan clustering. In this paper, we first study the Chinese text representation, Tibetan word stopping and other related problems: using vector model to represent documents, so that the text can be well stored and processed by computer, through statistics a large number of documents to obtain Tibetan stop words, The interference of these words to the clustering effect is excluded. Finally, the clustering effect of partitioning and hierarchical clustering algorithm for Tibetan is studied systematically. Third, Tibetan information retrieval research and system implementation. Tibetan information retrieval mainly studies Tibetan web page collection, Tibetan coding conversion, Tibetan web page preprocessing, Tibetan text storage and so on, which solves the problem of computer processing and retrieval of Tibetan language, and then realizes the search system based on Lucene. The system can automatically discover and update Tibetan resources, provide Tibetan search function, and complete the function of Tibetan search engine. Combined with Tibetan clustering to display search results, improve the pertinence and accuracy of search results.
【学位授予单位】：西南交通大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.1

【参考文献】