基于Storm框架的微博用户潜在需求实时分析评估系统

发布时间：2019-05-17 22:53

【摘要】：随着互联网的普及与发展,微博作为一个开放的信息交流和分享平台,每天可以产生上亿级别的数据。从这些海量数据中挖掘出用户潜在的购买行为并加以分析会对企业产生巨大的经济价值。但目前的研究分析方法存在以下缺点:对微博分析的实时性不足,导致分析结果存在一定的滞后;当前微博分析的针对性不足,没有充分挖掘出特定群体的价值。本文针对现有的挖掘微博用户潜在购买行为的分析方法所存在的问题,设计并实现了一个高效的、实时的基于Storm的微博用户行为分析评估系统。具体工作包括:先提出Storm现有的调度策略所存在的任务分布不均的问题并通过实验加以验证,接着提出基于CPU权值的自适应调度模型,以解决由内部节点间的时间延时和消息的本地特性造成效率较低的问题;其后为实时分析系统的设计及实现:分为数据来源模块、数据接入模块、数据分析模块和数据展示模块:数据来源通过爬虫和新浪API获取微博的数据;数据接入模块通过搭建Kafka集群,解决数据流延时问题;实现Storm的Spout和Bolt接口,实现数据分析模块,利用中文分词技术对数据进行分词,用K-means对数据进行聚会分析;利用数据存储模块和Hbase对数据进行持久化保存、利用SpringMVC和ECharts完成数据展示模块的实现。实验表明,改进后的调度策略性能明显优于现有的调度策略,尤其是在CPU密集型的调度任务方面,性能明显了提高50%左右。该实时分析系统可以实时分析出用户的潜在购买行为,企业可以根据分析出的行为特征,进而进行相关的研究和营销。
[Abstract]:With the popularity and development of the Internet, Weibo, as an open information exchange and sharing platform, can generate hundreds of millions of levels of data every day. Mining the potential purchase behavior of users from these massive data and analyzing it will produce great economic value to the enterprise. However, the current research and analysis methods have the following shortcomings: the real-time analysis of Weibo is insufficient, resulting in a certain lag in the analysis results; at present, Weibo analysis is not targeted enough, and the value of specific groups has not been fully excavated. Aiming at the problems existing in the existing analysis methods for mining the potential purchase behavior of Weibo users, an efficient and real-time Weibo user behavior analysis and evaluation system based on Storm is designed and implemented in this paper. The specific work includes: firstly, the problem of uneven task distribution in the existing scheduling strategies of Storm is proposed and verified by experiments, and then an adaptive scheduling model based on CPU weights is proposed. In order to solve the problem of low efficiency caused by the time delay between internal nodes and the local characteristics of messages. Then it is the design and implementation of the real-time analysis system: it is divided into data source module, data access module, data analysis module and data display module: the data source obtains Weibo data through crawler and Sina API; The data access module solves the problem of data flow delay by building Kafka cluster, realizes the Spout and Bolt interface of Storm, realizes the data analysis module, uses Chinese word segmentation technology to segment the data, and uses K-means to analyze the data. The data storage module and Hbase are used to save the data, and SpringMVC and ECharts are used to realize the data display module. The experimental results show that the performance of the improved scheduling strategy is obviously better than that of the existing scheduling strategies, especially in CPU-intensive scheduling tasks, the performance of the improved scheduling strategy is obviously improved by about 50%. The real-time analysis system can analyze the potential purchase behavior of users in real time, and enterprises can carry out related research and marketing according to the analyzed behavior characteristics.
【学位授予单位】：北京邮电大学
【学位级别】：硕士
【学位授予年份】：2016
【分类号】：TP393.092;TP311.13

【参考文献】