企业数据中心数据采集与建模

发布时间：2018-01-23 11:20

本文关键词： 数据数据中心 HADOOP　出处：《山东大学》2017年硕士论文　论文类型：学位论文

【摘要】：本论文包含某企业数据中心数据采集与建模部分设计与实现,数据中心包含采集层、数据服务层、应用层、访问层等,本文论述采集层与服务层的建设。数据在本系统中分为结构化与非结构化,其中结构化的数据存储在GBASE库中,非结构化的数据存储在HADOOP环境中,数据规模在PB级别,搭建成本比较低的分布式存储环境来存储与计算。开发使用OCDP大数据开发平台,数据存储采用HADOOP与GBASE数据库,流程控制采用BDPE工具进行配置管理。数据中心接入移动公司原有的三大业务支撑系统,将分散于各业务系统不同数据库、不同格式、不同类别的数据按照业务类型划分为7大主题域数据,通过统一数据接口采集到数据中心,依照移动公司现有的业务需求,将数据抽象为贴近业务需求的数据模型,实现将底层数据与上层的依赖于各业务的具体应用之间的松耦合,简化上层程序开发人员的开发难度。本文介绍数据采集层与数据服务层的开发工作,采集层将分布于不同系统的数据接入到数据中心,数据服务层将数据抽象为与业务相关的模型,为具体应用解决底层数据问题。
[Abstract]:This paper includes the design and implementation of data acquisition and modeling in a certain enterprise data center. The data center includes acquisition layer, data service layer, application layer, access layer and so on. This paper discusses the construction of collection layer and service layer. Data is divided into structured and unstructured in this system, in which structured data is stored in GBASE library. The unstructured data is stored in the HADOOP environment and the data scale is in the PB level. The distributed storage environment with relatively low cost is built to store and compute. The development uses the OCDP big data development platform. Data storage uses HADOOP and GBASE database, process control uses BDPE tools for configuration management. The data center is connected to the original three business support systems of mobile company. The data scattered in different databases, different formats and different types of data are divided into 7 subject domain data according to the business types, and the data center is collected through the unified data interface. According to the existing business requirements of mobile companies, the data is abstracted as a data model close to the business requirements to realize the loose coupling between the underlying data and the specific applications that depend on each business. This paper introduces the development of data acquisition layer and data service layer, which connects the data distributed in different systems to the data center. The data service layer abstracts the data as a business-related model to solve the underlying data problems for specific applications.
【学位授予单位】：山东大学
【学位级别】：硕士
【学位授予年份】：2017
【分类号】：TP308;TP274.2

【相似文献】