基于最大熵模型的柬英平行网页获取

发布时间：2018-01-10 00:06

本文关键词：基于最大熵模型的柬英平行网页获取　出处：《计算机工程》2016年05期 　论文类型：期刊论文

【摘要】：由于平行网站的异构性和复杂性,如何自动有效获取双语平行网页以及提高平行网页的质量是构建语料库的关键问题。为此,应用最大熵模型,将平行网页的识别问题看作候选网页对的分类问题,对平行网页的获取方法进行改进。利用基于标题余弦相似性的方法或数据库查询的方法发现候选平行网页对。根据网页内容及候选网页对间余弦相似度特征和最大熵模型训练的分类器对平行网页进行识别。在特征选取上,提取网页的篇章结构特征、词汇化比例特征与页面元素特征等基本特征,并应用TF-IDF算法与余弦相似性提取文档向量的余弦相似度特征。实验结果表明,所提方法可有效提高双语网站中平行网页的召回率和准确率,所获取平行网页的准确率和召回率分别为98%,94%。
[Abstract]:Due to the heterogeneity and complexity of parallel websites, how to automatically obtain parallel pages automatically and improve the quality of parallel pages is a key problem in the construction of corpus. Therefore, the maximum entropy model is applied. The recognition problem of parallel web pages is considered as the classification problem of candidate page pairs. The method of obtaining parallel pages is improved. The candidate parallel page pairs are found by using the method based on title cosine similarity or database query. The features of cosine similarity and the most common features of candidate pages are obtained according to the content of the page and the feature of cosine similarity between candidate web pages. The classifier trained by large entropy model recognizes parallel pages and selects features. The text structure features, lexicalization scale features and page element features of the web page are extracted. The TF-IDF algorithm and cosine similarity are used to extract the cosine similarity of document vectors. The experimental results show that the proposed method can effectively improve the recall rate and accuracy of parallel pages in bilingual websites. The accuracy and recall rate of parallel pages are 98 and 94 respectively.
【作者单位】：上海师范大学语言研究所;云南省计算机技术应用重点实验室;云南民族大学东南亚南亚语言文化学院;昆明理工大学信息工程与自动化学院;
【基金】：国家自然科学基金资助项目“柬埔寨语命名实体识别及汉柬双语语料库构建方法研究”(61462055) 云南省计算机技术应用重点实验室开放基金资助项目“汉柬双语语料库构建及柬埔寨语词法分析方法研究”
【分类号】：TP391.1;TP393.092
【正文快照】： 中文引用格式:莫源源,潘丽同,严馨,等.基于最大熵模型的柬英平行网页获取[J].计算机工程,2016,42(5):194-200.英文引用格式:Mo Yuanyuan,Pan Litong,Yan Xin,et al.Khmer-English Parallel Web Page Extraction Based onM aximum Entropy M odel[J].Computer Engineering,2016

【相似文献】