面向移动新闻的日志收集分析系统的设计与实现

发布时间：2018-03-14 09:54

本文选题：新闻客户端　切入点：日志收集　出处：《北京交通大学》2017年硕士论文　论文类型：学位论文

【摘要】：随着智能手机与移动互联网的快速发展,传统新闻门户网站的业务受到了移动性与便捷性的多重挑战。智能手机凭借其移动、便捷等优势,已发展成为重要的大众传播媒体,成为媒介融合的新平台,越来越多的手机用户开始选择使用移动新闻客户端来实时获取新闻。目前市场上存在多款新闻客户端APP,激烈的市场竞争使得各家厂商在不断优化自身移动新闻业务的同时,还开始尝试为用户提供个性化的新闻推荐。业务的优化离不开对于新闻服务数据的对比分析,而个性化推荐则需要从用户使用新闻客户端时的行为中分析用户的偏好。可以看到新闻客户端的发展离不开海量多维度数据的支持,因此收集新闻客户端的业务日志并从中挖掘出有价值的信息便成为了一项非常有意义的工作。针对新闻客户端设计的日志收集分析系统面临着许多挑战,考虑到日志收集分析系统整体需要面临的数据吞吐量,本系统采用了大量分布式框架来实现系统的各个功能,利用Flume、Kafka、Storm和HBase等分别实现了日志收集、缓存、传输、分析与存储等功能,再考虑到日志收集分析系统在新闻客户端业务中发挥的重要作用,系统还建设了完善的监控模块与数据展示模块为业务人员提供实时的系统监控与数据展示服务。本论文所阐述的解决方案来源于搜狐新闻移动客户端,从实际业务需求出发从研究背景、技术选择、系统架构等方面设计并实现了一套基于分布式框架的日志收集分析系统,重点阐述了日志收集、日志分发、流式处理、系统监控和数据展示等模块。本文在实际开发的基础上分析了系统的可靠性与实用性,提出了一套可行的移动新闻客户端日志收集分析方案,并展示了系统的部分功能界面。
[Abstract]:With the rapid development of smart phone and mobile Internet, the business of traditional news portal has been challenged by mobility and convenience. With its advantages of mobility and convenience, smart phone has developed into an important mass media. Become a new platform for media convergence, More and more mobile phone users choose to use the mobile news client to get news in real time. At present, there are many news clients in the market, and the fierce market competition makes each manufacturer optimize their mobile news business at the same time. It also tries to provide personalized news recommendation to users. Business optimization is inseparable from the comparative analysis of news service data. The personalized recommendation needs to analyze the user's preference from the user's behavior when using the news client. It can be seen that the development of the news client can not be separated from the support of massive multi-dimensional data. Therefore, collecting the business log of the news client and digging out the valuable information from it has become a very meaningful task. The log collection and analysis system designed for the news client faces many challenges. Considering the data throughput that the log collection and analysis system needs to face, this system adopts a large number of distributed framework to realize the various functions of the system, and uses Flumeer Kafka Storm and HBase to realize log collection, cache and transmission, respectively. Analysis and storage and other functions, and then considering the log collection and analysis system in news client business play an important role, The system also built a perfect monitoring module and data display module to provide real-time system monitoring and data display services for business personnel. The solution described in this paper comes from Sohu news mobile client. This paper designs and implements a set of log collection and analysis system based on distributed framework from the aspects of research background, technology selection, system architecture and so on, which focus on log collection, log distribution, flow processing, etc. This paper analyzes the reliability and practicability of the system, puts forward a set of feasible mobile news client log collection and analysis scheme, and shows some functional interfaces of the system.
【学位授予单位】：北京交通大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【参考文献】