基于SolrCloud大数据平台日志管理系统的设计与实现

发布时间：2018-03-05 10:41

本文选题：日志管理　切入点：SolrCloud　出处：《中国科学院大学(中国科学院工程管理与信息技术学院)》2017年硕士论文　论文类型：学位论文

【摘要】：信息化时代的发展随之而来的是数据的爆炸性增长,各种大数据计算框架在这样的环境下应运而生(如Hadoop,Spark,Strom等)。与此同时,大数据集群由于其服务器规模大,生态圈复杂等原因,给集群的运维管理带来很大挑战,日志管理系统作为集群运维管理,用户行为审计的重要辅助工具,就变得尤为重要。本文完成的主要工作包括:(1)针对大数据平台下组件多样,日志结构多样,整体结构分散,设计了一个集中化采集管理,统一存储,并且提供多种查询方式,以精确定位系统问题,快速了解系统运行情况;(2)采用Solr搜索引擎的分布式模式SolrCloud,来分担集群的索引压力,查询压力,采用HDFS分布式存储来分担存储压力;(3)使用spring boot设计系统业务的逻辑,做到维护简洁,部署方便;(4)本文本着人机友好的原则设计了日志查询,查询结果反馈,用户行为审计等用户界面,方便用户快速上手。本文设计并实现了基于SolrCloud的大数据平台上的日志管理系统,现已在多个厂家投入生产并使用。解决了大数据平台日志分散,类型多样,难以集中化管理的问题,并且实现多种查询方式以达到快速精确定位日志的目的,也提供了用户行为统计分析作为审计依据,降低了运维的难度。
[Abstract]:With the development of information age, the explosive growth of data comes into being, and various big data computing frameworks come into being in such an environment (such as Hadoop Sparkor Strom etc.) at the same time, due to the size of its servers and the complexity of the ecological circle, so on. Log management system, as an important auxiliary tool of cluster operation and maintenance management and user behavior audit, becomes particularly important. The main work accomplished in this paper includes: 1) aiming at the diversity of components under big data platform, log management system is a very important tool for cluster operation and maintenance management. The log structure is diverse and the whole structure is scattered. A centralized collection management, unified storage, and various query methods are designed to accurately locate the system problems. We use the distributed model of Solr search engine, Solr Cloud, to share the index pressure and query pressure of cluster, and use HDFS distributed storage to share storage pressure. We use spring boot to design the logic of system business. In this paper, the user interface such as log query, query result feedback, user behavior audit and so on are designed based on human-computer friendly principle. This paper designs and implements a log management system based on big data platform based on SolrCloud, which has been put into production and used in many manufacturers. It is difficult to centralize the management, and many query methods are implemented to locate the log quickly and accurately. The statistical analysis of user behavior is also provided as the audit basis, and the difficulty of operation and maintenance is reduced.
【学位授予单位】：中国科学院大学(中国科学院工程管理与信息技术学院)
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP311.52

【相似文献】