网络广告效果分析系统的设计及实现

发布时间：2018-02-11 18:36

本文关键词： 广告分析系统海量数据处理分布式计算高并发服务器热力图　出处：《上海交通大学》2012年硕士论文　论文类型：学位论文

【摘要】：随着互联网的普及，越来越多的企业使用网络广告来进行内容推广。广告主为了衡量广告的投入产出价值，需要进行全面的广告效果分析，因此诞生了广告分析系统。这类广告分析系统需要承受大量访问请求，同时需要对记录着用户访问行为的海量数据进行统计分析。设计和实现一套能支持高并发访问，同时能够对海量数据进行分析的系统，成为这类分析系统成功的关键。本文针对大规模网络广告分析需求设计了基于Linux操作系统平台的分布式、大容量、可扩展的广告分析系统。实现了网络广告的数据采集，数据分析，数据展示三大基础功能，为广告投放人员提供了一个易用、快速、直观的广告分析平台。本系统采用了Brower/Server结构，用C++语言作为基础的程序设计语言，基于Linux平台Epoll网络模型实现用于记录用户行为日志采集服务器。采集服务器通过http协议与网页中javascript脚步进行交互，，在采集服务器前端通过lvs进行负载均衡转发，保证了系统的高可用性以及横向扩展能力。同时本文对常用的基于CGI协议的web应用开发方法和基于HTTP服务器的WEB应用开发所能承受的并发数进行了对比测试，证明了这种方式可以支持更高的并发访问。通过对开源的hadoop分布式计算平台以及MapReduce编程模型进行了介绍和研究，本系统改变了之前采用关系型数据库，通过存储过程分析数据的方式，采用hadoop作为计算平台，实现了针对海量数据的分析，为后续进一步对数据的深入挖掘打下良好的基础。系统在分析结果展示中，除了采用常规的表格分析展示外，创新的采用了热力图进行更直观的展示。本文对热力图的实现方法进行了详细的阐述。同时对生成热力图在性能上的做了多种优化尝试，并取得较好的效果。本文通过广告分析系统几个关键技术点的研究，介绍一种可以对大型日志进行采集、分析、展示的可行的解决方案，并通过实践证明了其可行性。
[Abstract]:With the popularity of the Internet, more and more enterprises use online advertising to promote content. Advertisers need to carry out comprehensive analysis of advertising effects in order to measure the input-output value of advertisements. Therefore, advertising analysis system is born. This kind of advertisement analysis system needs to bear a large number of access requests, at the same time, it needs to carry on the statistical analysis to the massive data which records the user's access behavior, designs and implements a set of can support the high concurrent access. At the same time, the system which can analyze the massive data is the key to the success of this kind of analysis system. In this paper, a distributed, large capacity and extensible advertising analysis system based on Linux operating system is designed to meet the demand of large scale network advertisement analysis, which realizes three basic functions: data acquisition, data analysis and data display. For advertising personnel to provide an easy to use, fast, intuitive advertising analysis platform. The system adopts Brower/Server structure, uses C language as the basic programming language, and uses Epoll network model based on Linux platform to record the user behavior log acquisition server. The acquisition server interacts with the javascript step in the web page through the http protocol. Load balancing and forwarding through lvs in the front end of the acquisition server, At the same time, this paper compares the common web application development method based on CGI protocol with the concurrency number of WEB application development based on HTTP server. It is proved that this method can support higher concurrent access. Through the introduction and research of the open source hadoop distributed computing platform and MapReduce programming model, the system changed the relational database, analyzed the data through the stored procedure, and adopted hadoop as the computing platform. The analysis of massive data is realized, which lays a good foundation for further data mining. In the display of the analysis results, in addition to the use of conventional table analysis display, In this paper, the realization method of thermodynamic diagram is described in detail. At the same time, a variety of optimization attempts on the performance of generating thermodynamic diagram are made, and good results are obtained. Through the research of several key technology points of advertising analysis system, this paper introduces a feasible solution that can collect, analyze and show large scale log, and proves its feasibility through practice.
【学位授予单位】：上海交通大学
【学位级别】：硕士
【学位授予年份】：2012
【分类号】：TP311.52

【参考文献】