基于协同过滤的习题推荐系统设计与实现

发布时间：2018-06-23 02:43

本文选题：协同过滤 + Hadoop　；参考：《长江大学》2017年硕士论文

【摘要】：近几年来,随着互联网、移动设备、传感器设备等信息技术快速发展,各行各业的数据量正以TB甚至PB级的数量迅猛增长。推荐系统的产生帮助用户在众多的数据中,寻找对他们感兴趣或是重要的信息,提高了用户办事的效率,也让他们有了明确的方向。在电子商务时代,推荐系统被广泛应用于商品购物、新闻、饮食、阅读和音乐等网站,给人们的生活带来了方便,提高了用户的体验。在教育领域,传统的数字校园建设模式已经满足不了现在的需求,物联网、大数据、云计算随之兴起,为教育信息化建设带来了崭新的思路。近年来提出“智慧校园”的概念,将教育和先进的科学技术结合起来,不管是从安全、教育、学习、生活方面,都有很大的提高。在线考试系统的产生,方便了老师和同学,减轻了老师的工作量,同时也方便了学生在线答题。但是,对于个性化推荐服务,还有待加强。信息技术的发展彻底打破了教学的时空界限,将推荐系统应用到教育领域,也是一种有益的尝试。推荐系统的产生解决了用户如何在海量数据中快速寻找有用的信息,推荐系统的产生为消费者和使用者提供了更友好的用户界面,提升了体验度。但是推荐系统也有其不足,会出现推荐不精准或是错误的推荐。推荐系统的发展主要在用户信息的获取、处理、推荐算法研究、推荐的效果和影响这5个方面。而在推荐算法中,协同过滤(Collaborative Filtering,简称CF)是最常用的算法之一,但是冷启动和稀疏问题一直是协同过滤需要解决的难题。在推荐系统发展至今,比较流行的推荐算法主要有协同过滤推荐算法、基于内容的推荐算法、混合推荐算法等,它们各有其利弊。综合学者的论文可以发现,基于推荐算法的优化,用得最多的就是混合推荐,将不同的算法取长补短,优化推荐效果。本文在前人的基础上,针对推荐系统中用户首次登录后由于没有历史数据,如何为新用户推荐合适的习题内容进行非个性化的推荐,以及如何为老用户推荐合适的习题进行个性化的推荐展开研究。分析影响用户和习题的主要因素,根据这些因素进行合理的处理,通过对协同过滤算法的优化,把相对有效的习题推荐给用户。首先,根据用户的历史记录收集数据并将数据作数字化、分类、归一化和减噪处理。找到影响习题推荐的影响因素,并通过一个权重公式,通过皮尔逊相关系数求解相似度。接着为了解决协同过滤的冷启动问题,即新用户首次登录系统,由于没有历史数据作为推荐依据进行个性化推荐,根据用户的学号将用户进行分组,找到用户所在的班级和专业,并根据所在班级和专业的历史数据进行分析,得到易错题排行榜、易错知识点排行榜、考试常见习题排行榜等,为用户进行非个性化的推荐。然后为了解决协同过滤的数据稀疏问题,将协同过滤的推荐方法进行优化,通过对基于用户的推荐和基于内容的推荐结果进行合并,将具有相同结果的值优先推荐给用户,把不同结果的值根据相似度进行排序,依次推荐给用户。这种推荐方法综合两种推荐算法的结果,为用户进行推荐,提高了准确性。在数据稀疏的情况下,这种方法的优点就更为突出。通过对协同过滤算法的优化,得到最终的推荐结果并保存到HDFS文件系统中。为了能将结果进行展示,使用Sqoop连接传统数据库MySQL与HDFS分布式文件系统,使它们能做到自动传输数据。最后将MySQL中的结果进行读取,并使用WEB的方式展示,形成最终的习题推荐系统。将推荐系统应用到习题中,从教育领域上说,具有一定的激励作用,提高了教学的灵活性。从老师的角度上说,大大减少了老师的工作量。从学生的角度来说,免去了从海量信息中选择合适题目的时间,提高了学生的学习效率和学习的热情,解决了学生之间的差异问题。将高新科技与教育相结合,体现了教育的公平、高效性和多样性,具有一定的实际意义。
[Abstract]:In recent years, with the rapid development of information technology such as Internet, mobile devices and sensor devices, the amount of data in all walks of life is growing rapidly in the number of TB and even PB. In the era of e-commerce, the recommendation system is widely used in commodity shopping, news, diet, reading and music, which bring convenience to people's life and improve the experience of users. In the field of education, the traditional digital campus construction model is not full of the needs of the present, the Internet of things, big data, cloud computing In recent years, the concept of "intelligent campus" has been put forward to combine education with advanced science and technology, regardless of its safety, education, learning and life. The production of the online examination system has facilitated the teachers and students and reduced the teacher's workload. At the same time, it is also convenient for students to answer the questions online. However, the personalized recommendation service still needs to be strengthened. The development of information technology has completely broken the time and space limit of teaching, and it is also a useful attempt to apply the recommended system to the field of education. The production of the recommended system solves the user how to find useful information in massive data quickly. The recommendation system provides a more friendly user interface for consumers and users and improves the degree of experience. However, the recommendation system has its shortcomings, and there will be recommendations for inaccuracy or error. The development of the recommendation system is mainly in the 5 aspects of user information acquisition, processing, recommendation algorithm research, recommended effects and effects. In the recommendation algorithm, Collaborative Filtering (CF) is one of the most commonly used algorithms, but cold start and sparsity has been a difficult problem to be solved by collaborative filtering. In the development of the recommended system, the popular recommendation algorithms are mainly collaborative filtering recommendation algorithm, content based recommendation algorithm, mixed recommendation algorithm. They have their own advantages and disadvantages. The thesis of the comprehensive scholars can find that the best use of the optimization based on the recommendation algorithm is the mixed recommendation, which makes the different algorithms make up the short and optimizes the recommendation effect. The content of the appropriate exercises is not personalized, and how to recommend the appropriate exercises for the old users to carry out the personalized recommendation. The main factors that affect the user and the problem are analyzed. According to these factors, the relative effective exercises are recommended to the users through the optimization of the collaborative filtering algorithm. According to the historical records of the user, the data is collected and the data are digitization, classification, normalization and noise reduction. The influence factors that affect the recommendation of the exercises are found, and a weight formula is used to solve the similarity through the Pearson correlation coefficient. Then, to solve the cold start problem of collaborative filtering, the new user first login system is not a calendar. Historical data as a recommendation basis for personalized recommendation, according to the user's school number of users to group, find the class and specialty of the user, and according to the class and professional historical data analysis, get the wrong list, the wrong list of mistakes, the list of common exam questions, etc., for the user to do non personalized In order to solve the problem of data sparsity in collaborative filtering, the recommendation method of collaborative filtering is optimized. By merging the user based recommendation and content based recommendation results, the values of the same results are recommended to the user first, and the values of the results are sorted according to the similarity, which is recommended to the users in turn. The recommendation method combines the results of two recommended algorithms to improve the accuracy of the user. In the case of sparse data, the advantages of this method are more prominent. By optimizing the collaborative filtering algorithm, the final recommendation results are obtained and saved to the HDFS file system. In order to display the results, use the Sqoop connection. The traditional database MySQL and HDFS distributed file system enable them to automatically transmit data. Finally, it reads the results in MySQL and displays it with WEB, and forms the final exercise recommendation system. The recommended system is applied to the exercises. In the field of education, it has a certain incentive effect and improves the flexibility of teaching. From the teacher's point of view, the teacher's workload is greatly reduced. From the point of view of the students, the time to choose the right topic from the mass information is avoided, the students' learning efficiency and enthusiasm are improved and the difference between the students is solved. The combination of high and new technology and education embodies the fairness, efficiency and more of education. It has certain practical significance.
【学位授予单位】：长江大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP391.3

【相似文献】