某电子商务网站搜索日志分析系统设计与实现

发布时间：2018-04-25 09:32

本文选题：日志分析 + 关键词　；参考：《中国科学院大学(中国科学院工程管理与信息技术学院)》2017年硕士论文

【摘要】：随着互联网的飞速发展以及网站数量的急剧增加,各网站对用户的争夺变得越来越激烈。为了更好的吸引和留住用户,需要更好地了解网站用户的使用行为,研究并分析搜索引擎的日志已成为在海量数据中获取用户行为有效数据的主要方法。基于此,为了更好地捕捉网站用户现实需求,了解用户意愿,本文设计和实现了一个网站搜索日志分析系统,从而帮助网站能更好的服务客户,并实现网站的快速发展。不同网站的搜索引擎针对的目标群体不同,本文的研究对象是某电子商务行业网站的搜索日志,通过建立日志分析系统来了解网站的用户行为模式,并挖掘其潜在需求。本系统设计中存在的最大困难之处在于如何搜索海量的日志数据,并实现搜索的高速性和准确性。主要研究内容如下:1,搜索日志的收集格式使用NCSA扩展日志格式,网站页面各分析项使用标签记录,使用开源Apache和Flume海量日志采集系统进行日志收集,使网站日志收集具有高效,准确,及时等特点,减轻了开发和测试的压力和负担,同时降低了风险。页面各统计项通过添加标签使分析日志具备了简单,准确的特点,降低了日志分析的负担。2,使用分布式处理平台Hadoop对日志进行分析,论文中主要分析基于HDFS文件存储和Map/Reduce的分布式处理的关键技术,对日志分析的实现过程进行了详细的描述和分析,通过使用Hadoop解决了海量日志分析处理的时效性和准确性的问题,并且代码开发非常简单,难度大幅度降低,项目推进的效率提升明显。3,设计并实现了用户行为的分析模型和用户信息质量的评分模型,通过这两个模型我们可以获知用户的网站浏览偏好以及用户信息的质量信息,以及关键词相关性的信息,建立了用户的偏好浏览模型和信息聚类模型,为信息聚合和个性化的搜索提供了数据支撑。最后,通过对上线系统运行两周后的结果分析,并且按照分析结果搜索重新进行排序设置以及聚类展示,很好地提升了使用效果,系统也达到了预期的目标。
[Abstract]:With the rapid development of the Internet and the rapid increase of the number of websites, the competition for users becomes more and more fierce. In order to attract and retain users better, it is necessary to understand the user's behavior better. The research and analysis of search engine log has become the main method to obtain the effective data of user behavior in the massive data. Based on this, this paper designs and implements a website search log analysis system in order to better capture the actual needs of website users and understand users' wishes, so as to help the website to better serve customers and realize the rapid development of the website. The search engine of different websites aims at different target groups. The research object of this paper is the search log of a website in an electronic commerce industry. Through the establishment of log analysis system, we can understand the user behavior pattern of the website and mine its potential demand. The biggest difficulty in the design of this system is how to search the massive log data and realize the high speed and accuracy of the search. The main research contents are as follows: the search log collection format uses NCSA extended log format, the analysis items of website pages use label recording, and the open source Apache and Flume massive log collection system are used for log collection, which makes the website log collection efficient. Accurate, timely and other characteristics, reduce the development and testing of the pressure and burden, while reducing the risk. Each statistical item on the page has simple and accurate features by adding tags, and reduces the burden of log analysis. 2. The distributed processing platform Hadoop is used to analyze the log. In this paper, the key technologies of distributed processing based on HDFS file storage and Map/Reduce are analyzed, and the implementation process of log analysis is described and analyzed in detail. Through the use of Hadoop to solve the problem of timeliness and accuracy of massive log analysis and processing, and the code development is very simple, the difficulty is greatly reduced, The efficiency of the project is improved obviously. 3. The analysis model of user behavior and the scoring model of user information quality are designed and implemented. Through these two models, we can get the user's preference for browsing website and the quality information of user information. The user preference browsing model and information clustering model are established, which provide data support for information aggregation and personalized search. Finally, by analyzing the results of the on-line system after two weeks' running, and reordering and clustering display according to the analysis result search, the system improves the use effect well, and the system also achieves the expected goal.
【学位授予单位】：中国科学院大学(中国科学院工程管理与信息技术学院)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP393.092;TP391.3

【相似文献】