面向用户个性化推荐的日志分析系统设计与实现

发布时间：2019-05-27 09:43

【摘要】：近年来,随着计算机的普及和互联网的发展,使用互联网的用户数和互联网上充斥的各类信息量突飞猛进,如何快速从互联网海量的数据信息中获取用户所关注的信息已成为用户最为关心的问题之一,也是互联网服务网站能够吸引用户使用的要点。同时网络视频业务也已成为互联网应用的最大热点,随着媒体事业的发展和台网联动的推广,提供网络视频业务的网站数量和互联网上视频数量也越来越多,通过关键字查询的搜索引擎显然已无法满足用户快速获取视频的需求,此时,基于主动推送的推荐引擎应运而生。互联网的用户数量和视频数量的急剧增长为推荐系统带来了新的问题。首先,海量用户日志信息的存储对推荐系统的存储模块提出了新的挑战,需保障存储数据的可靠性和存储容量的可扩展性；其次,海量日志信息的分析和推荐计算对系统处理数据的性能提出了更高的要求；最后,推荐引擎推送的视频信息要能够吸引用户,这就要求推荐引擎的推荐结果具有较高的准确有效性。本文针对推荐系统在面对海量数据信息时需要解决的各种问题,基于Hadoop分布式平台及其系列子项目提出了一种解决方案——面向用户个性化推荐的日志分析系统。该系统利用Hadoop分布式文件系统(HDFS)的可靠性和可扩展性的特点,使用基于HDFS的数据仓库Hive作为海量日志信息的存储平台,保证了用户日志信息存储的可靠性和可扩展性；利用Hadoop并行计算编程模型Map/Reduce的高效性特点,使用能够将SQL语句转换为Map/Reduce任务执行的Hive作为用户日志信息分析的平台,并使用借助Map/Reduce实现并行推荐计算的机器学习算法库Mahout封装的协同过滤算法来实现推荐引擎,实现了高性能的日志分析和推荐计算；最后通过分析Mahout源码和视频推荐的特点,优化了Mahout的推荐算法源码,以期提高推荐结果的准确有效性。为了对系统进行验证,本文设计了详细的测试方案。首先,从功能上验证了面向用户个性化推荐的日志分析系统的可用性和系统日志存储模块的可靠性、可扩展性；其次,从性能上验证了日志分析和推荐计算效率的提升；接着验证了优化后的推荐算法对推荐结果准确有效性的提升；最后通过搭建真实的实验环境,验证了系统的实际工作效果。
[Abstract]:In recent years, with the popularity of computers and the development of the Internet, the number of users using the Internet and all kinds of information filled with the Internet have advanced by leaps and bounds. How to quickly obtain the information concerned by users from the massive data information of the Internet has become one of the most concerned issues for users, and it is also the key point that Internet service websites can attract users to use. At the same time, the network video service has become the biggest hot spot of the Internet application. With the development of the media industry and the promotion of the network linkage, the number of websites providing network video service and the number of videos on the Internet are also increasing. The search engine through keyword query obviously can not meet the needs of users to get video quickly. At this time, the recommendation engine based on active push emerges as the times require. The rapid increase in the number of users and videos on the Internet has brought new problems to the recommendation system. First of all, the storage of massive user log information poses a new challenge to the storage module of the recommendation system, and it is necessary to ensure the reliability of storage data and the scalability of storage capacity. Secondly, the analysis and recommendation calculation of massive log information puts forward higher requirements for the performance of the system in processing data. Finally, the video information pushed by the recommendation engine should be able to attract users, which requires the recommendation results of the recommendation engine to have high accuracy and effectiveness. In order to solve all kinds of problems that recommendation system needs to solve in the face of massive data information, this paper proposes a solution based on Hadoop distributed platform and its series of subprojects-user personalized recommendation log analysis system. The system makes use of the reliability and expansibility of Hadoop distributed file system (HDFS), and uses the data warehouse Hive based on HDFS as the storage platform of massive log information, which ensures the reliability and scalability of user log information storage. Taking advantage of the efficient characteristics of Hadoop parallel computing programming model Map/Reduce, Hive, which can convert SQL statements into Map/Reduce tasks, is used as the platform for user log information analysis. The collaborative filtering algorithm packaged by Mahout, which realizes parallel recommendation computing with the help of Map/Reduce, is used to realize the recommendation engine, and the high performance log analysis and recommendation calculation are realized. Finally, by analyzing the characteristics of Mahout source code and video recommendation, the source code of Mahout recommendation algorithm is optimized in order to improve the accuracy and effectiveness of the recommendation results. In order to verify the system, a detailed test scheme is designed in this paper. Firstly, the availability of the user-oriented personalized recommendation log analysis system and the reliability and scalability of the system log storage module are verified from the functional point of view. Secondly, the performance of log analysis and recommendation computing efficiency is verified. Then the optimized recommendation algorithm is verified to improve the accuracy and effectiveness of the recommendation results. Finally, the actual working effect of the system is verified by building a real experimental environment.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2013
【分类号】：TP391.3

【相似文献】