中文网站抓取及编码转换研究

发布时间：2018-05-02 15:10

本文选题：汉字编码识别 + 特征选择　；参考：《新疆农业大学》2013年硕士论文

【摘要】：本文在分析GB2312、GBK、GB18030、Big5、UTF-8等中文字符编码特征的基础上，重点研究了网页文件的中文字符编码识别技术。通过对比不同的文本特征（布尔权重、词频权重、词频倒文档频权重）和机器学习方法（多元线性回归、朴素贝叶斯、K临近算法、支持向量机），提出了一种综合应用中文字符编码规则和网页文本特征的中文字符编码识别模型。该模型对于有着严格编码规则的UTF-8编码，按照其编码规则来判断。对于码位空间重合的GB系列和Big5编码采用网页文本特征来识别。试验表明，该模型在阈值（属于UTF-8编码的字符数比总字符数）等于1时，对UTF-8编码识别率为100%。特征值个数大于65个时，四种机器学习方法对GB系列和Big5编码识别率均为100%。为了实现农业垂直搜索引擎中中文统一编码的任务，本研究设计开发了农业网页中文字符编码自动识别和各类编码转换为UTF-8编码的通用模块。该模块以网络爬虫下载的网页为输入参数，首先抽取其中的中文，再根据编码规则判断是否采用UTF-8编码。若不是UTF-8编码，则根据实验得到的特征值，采用布尔权重和多元线性回归算法判断编码是GB系列编码还是Big5编码。最后采用iconv函数将网页统一转为UTF-8编码。
[Abstract]:Based on the analysis of the Chinese character encoding characteristics of GB2312 / GB18030 / Big5UF-8 and other Chinese characters, this paper focuses on the Chinese character coding and recognition technology of web pages. By comparing different text features (Boolean weight, word frequency weight, word frequency inverted document frequency weight) and machine learning methods (multiple linear regression, naive Bayesian K-proximity algorithm), Based on support vector machine (SVM), a Chinese character coding recognition model based on Chinese character encoding rules and web page text features is proposed. The model is judged by its encoding rules for UTF-8 codes with strict coding rules. For GB series and Big5 coding with coincident code space, web page text features are used to identify them. The experimental results show that when the threshold (the number of characters that belong to UTF-8 encode is equal to the total number of characters) is equal to 1, the recognition rate of UTF-8 coding is 100. When the number of eigenvalues is greater than 65, the recognition rate of the four machine learning methods for GB series and Big5 codes is 100. In order to realize the task of uniform Chinese coding in agricultural vertical search engine, a general module of automatic recognition of Chinese character encoding and conversion of all kinds of codes to UTF-8 coding for agricultural web pages is designed and developed in this paper. In this module, the web pages downloaded by the web crawler are taken as input parameters, the Chinese is extracted first, and then the UTF-8 encoding is judged according to the coding rules. If it is not a UTF-8 code, the Boolean weight and multivariate linear regression algorithm are used to determine whether the code is a GB series code or a Big5 code according to the eigenvalues obtained from the experiment. Finally, the iconv function is used to transform the web page into UTF-8 coding.
【学位授予单位】：新疆农业大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.1

【参考文献】